This is one of the more curious tonal shifts in the poem. Up to this point, the speaker's characterization of the man has been mostly empathetic. But this interjection carries a bitter chastisement and a far more diminutive pity. The purpose of this appears to be to echo a voice of parental authority and perhaps even the words of those beyond the door which caught the man listening in.