ryanlecompte/redis_failover — pull request #34 by ryanlecompte (2012-08-31 02:45:25+00:00)
Title: be smart about picking a master on node manager startup

This pull request covers a couple ideas that I discussed with @eric today:

When a node manager starts up and the previously known redis master is unknown (i..e, nil in znode config or znode doesn't exist), we will attempt to find the right master. However, we don't want to just pick the first redis node that claims to be a master (i.e., not slaved to another node) since it could generate false positives. The Node Manager now does the following in this pull request:

- if it can't talk to any of the configured redis nodes, it will raise a NodeUnavailableError
- if it can talk to all the nodes but they all claim to be slaves, it will raise a NoMasterError
- if it can talk to all the nodes but it found more than one master, it will raise a MultipleMastersError

In the above cases, the Node Manager will log the error and attempt to retry the discovery process after a 3 second delay.

When the node manager is in this state, the user should do one of the following:

- perform a manual failover of a specific node via RedisFailover::Client#manual_failover
- manually log into your redis boxes and pick a master using slaveof

Proper synchronization has been added to make sure that we don't have race conditions between discovering nodes, handling a manual failover, and handling a normal state report from a node watcher.