basho/riak_cs — pull request #519 by reiddraper (2013-04-02 21:41:26+00:00)
Title: Gracefully handle Riak disconnects

Previously, Riak CS would not start if it was unable to connect
to Riak. Furthermore, it would eventually crash if it was disconnected to Riak.
This was due to the fact that the `poolboy` workers would die and poolboy would
not try and restart them. This leads to the `poolboy` supervisor eventually
crashing and then it bubbles up to the `riak_cs_sup` crashing. The main
change here is that we pass the `auto_reconnect` option into
`riakc_pb_socket`, which allows it to return from `start_link` even if
it can't connect to Riak. Attempts to use `riakc_pb_socket` when it's in
this state will return `{error, disconnected}`. I've tried to handle as
many of the cases where this might happen, and return 503 to the end
user. There are cases where we might be far enough in the response
(headers already sent over the wire), where for now I just Let It Crash.
Originally, I had thought about adding a 'health' thread that would
monitor Riak availability, for the purposes of informing the deployer
via log messages. However, for now, I think the log messages are clear
enough when Riak gets disconnected that it should be obvious. The sudden
spike in 503 responses should also be a pretty good indicator. Other smaller
changes are detailed below.

* Allow `riak_cs_storage_d` to start even if it can't immediately connect to
Riak
* Refactor `riak_cs_wm_common:forbidden/2` to handle disconnected Riak
when retrieving the user record

___
Fixes #262
___

## Testing!

Here are some ideas for testing, beyond the usual bits:

* Start Riak CS when Riak is not running
* Start Riak CS when it's configured to talk to the wrong Riak port
* Kill/stop Riak while Riak CS is running, verify that requests start seeing 503 errors (I _think_ I eliminated all the cases where they'd see, 403, so it's a bug if you see 403).
* Verify that things that working again when Riak comes back to life