Cleanly remove disconnected clients.

Also fixes a bug where responses for a single client would be sent to
every connected client.