Check for recv == -1 as well.

Needed to catch some sort of hangup error (EPIPE?) from
clients leaving.