Only stop if the disconnected client is ourselves