basho/riak-java-client — pull request #108 by paulbarry (2012-02-22 11:40:34+00:00)
Title: Failover client - comments?

I've added a new config to current client to fix a problem of nodes disappearing temporarily, for example during maintenance restarts.
The riak cluster is still active even if one node is gone, but the current java cluster client will start getting errors when the downed node is picked round-robin. It retries on a single node according to Retrier, but does not switch nodes after the target node has been selected.

In this branch, there's a new high-level client registered, with corresponding config. If all nodes are running then behaviour should be identical to ClusterClient - it does a simple round-robin load balancing.

If a node generates an error (EOF on socket for example), it's marked as inactive, and next node is selected and operation retried. Inactive nodes are ignored for time period (currently hardcoded to 15 seconds) and retried again after that.
During the 15 seconds, only known good nodes are selected as targets. If all active nodes fail, then operation retries for time period rather than specific number of retries. The meaty bit is in FailoverClusterClient around line 280

It's still a hack right now, with hardcoded values, only implemented for PB, and no tests. It currently retries for any error, which is wrong - need to copy over selected exceptions from retrier code, but I don't understand all the use cases, only my limited use (simple bucket ops, fetch, store and MR)

I'm testing it using my app and local cluster - seems to be working OK, but is definitely not full coverage. I have three node riak, with two nodes configured for use by this client. Requests continue to work even if one of the two nodes is down temporarily (tested by stopping the node)

This is not really a request for merge yet, more request for comments, and asking whether this is proper direction.