Don't crash when host leaves the lobby, fixes #2980

This fixes #2980. I overlooked a NullReferenceException there.