When a server disconnects, remove all servers on the remote side of it