handle corner case when uninviting disappearing clients.