neo4j/neo4j — pull request #2090 by jakewins (2014-03-04 14:52:40+00:00)
Title: Fix issue with cluster bricking if certain messages lost when master lea...

...ves.

o When a master leaves, if the leave learned message did not reach the machine with the
lowest id, the other instances would wait for that instance to start an election
indefinitely. The instance with the lowest id would realize it was out of date, but
try and contact the (now dead) master to catch up, and would not retry when that failed.

o This implementation fixes this by preferring instances more likely to be live and to have
the paxos instance id we are interested in (see CommonContextStateTest and LearnerStateTest).
o It also introduces proper retries if the catchup fails.
o Two refactorings are done in this commit as well, that should've been their own commits, sorry.
1) InstanceId has been renamed ClusterInstanceId, to avoid confusing it with the similarily named
class InstanceId.
2) The duplicated, unused, MultiPaxosContext has been removed, and the tests running against its
subclasses have been moved to test the live code.