Fixed possible runtime error.

If the world is rebooted or shut down, server_manager.logger would get
deleted before clients get deleted, causing a runtime error when
attempting to log a client's deletion.