ryanlecompte/redis_failover — pull request #31 by ryanlecompte (2012-08-21 20:16:42+00:00)
Title: 1st pass at coordinated failover

1st pass at coordinated failover. The basic idea is:

Client-side:

- Each client joins the 'client presence' ZK group.
- When a client gets a znode deletion for the redis node config, it joins the 'client failover ack group'.
- When the client subsequently gets a znode creation callback, it leaves the 'client failover ack group'.

Node manager side:
- When the NM wants to promote a new master, it grabs references to the ZK 'client presence' and 'client failover ack' groups.
- It figures out how many clients are currently in the 'client presence' group.
- It then deletes the znode.
- It then waits for the clients to all move to the 'client failover ack' group.
- Once all of the clients have moved to the ack group or the timeout is reached, the failover is performed.

A new configuration can be set that indicates the timeout for waiting for all clients to move over to the ACK group.