Don't crash when host leaves the lobby, fixes #2980

I overlooked a NullReferenceException there.