mrb/riakpbc — pull request #43 by boj (2013-08-06 08:35:29+00:00)
Title: addresses issue #41, among other things

This issue fixes #41 as per the comment I made.

It also fixes another issue that needed addressed, which was redialing valid nodes that happened to be down on client restart instead of outright deleting them from the pool.

@mrb Unfortunately this ended up removing your fancy `decaying.go` code. I consider this a much more elegant solution which has simplified the codebase quite a bit, but would be happy to hear your thoughts on it.

1. Node redialing happens internal to a node, and only when an error is triggered outside of the last known lastError time threshold, as opposed to continuously for all nodes in `client.go`. The initial retry time is 0.5s, and will scale +0.5s up to a total of 5.0s. I figure 5 second retries is a decent enough heartbeart rate in this case, although we should probably set these parameters to be configurable instead of a constant in the future.
2. If all nodes happen to be down, ErrAllNodesDown gets returned back to the client.

All tests pass, no race conditions are present. The simple code test @bobobo1618 gave us runs as intended, but here are some gotchas we need to document.

1. If a node goes down mid query you get an `EOF` error.
2. Because commands are queued into Riak as a request/response pair, old query responses will come back the next time the node comes up as `0: Unknown message code.` errors. This should probably be checked by the client and tossed out when dialing/redialing nodes, although I'm not entirely sure how to approach it at the moment.

@bobobo1618 If you would like to test this branch, cd into `$GOPATH/src/github.com/mrb/riakpbc` and do a`git pull` followed by `git checkout node-down-fix`, then force a rebuild of your code.