drewww/unhangout — pull request #177 by drewww (2013-10-17 02:26:38+00:00)
Title: Heartbeat delay compensation

Okay, I have high hopes that these changes will make a big difference to #154.

I found good evidence that part of how we get screwed up is that heartbeat messages will sometimes get delayed. I've seen evidence in the logs of delays > 60 seconds. That's way too much latency to push the timeout threshold that far out; it would introduce HUGE delay in detecting legitimate shutdowns, plus there's no real guarantee that we wouldn't still get occasional situations where the leader fails to send a heartbeat.

So - story here is that we think the rogue sessions are coming from situations where a session appears to time out, as far as the server is concerned, so it lets someone create a new hangout. But really, there was still a valid session running. To detect this case, this pull does two things:

* allows heartbeats from sessions that were thought to be shutdown, to more gracefully repopulate their participant list and url, without forcing a full session restart. This deals well with the case where a hangout would not send heartbeats for a while, then send them again, and no one tries to start a hangout in the intervening period.
* in situations where an existing (old, populated) hangout stops sending heartbeats, a new hangout gets created, and then the old one phones home again, the server will inform the NEWER hangout (based on hangout start time) that it is in conflict with an older one, and that the user should shut down immediately. It then gives priority to the older one, and uses its participant list and url, so future joiners will get directed to the right one, even if the newer one phoned home more recently.

I'm not 100% convinced this fixes #154, but it would explain a number of our issues, particularly with participant lists getting cleared (caused by heartbeat delays, which historically would reset participant list) and with multiple sessions existing and people getting sent to the wrong place (newer sessions always overwrote older ones, once they failed to phone home for a period). Seems as plausible an explanation of our issues as I've seen over the past few weeks.

Other minor note: this behavior has been super observable in permalinks for some time, which will hopefully show it working. The test case there is if we stop seeing running permalink sessions with 0 participants, which we've seen ALL THE TIME for quite a while.