Factor out proper client removal code

Listener channel was not closed when a client was removed in case of
error (typically after a disconnection).