[Server] Make sure disconnected handlers are removed from memory