basho/riak_cs — pull request #262 by reiddraper (2013-04-02 21:37:28+00:00)
Title: Riak CS should handle the absence of Riak more gracefully

## Problem

When starting Riak CS, especially for the first time, it is not immediately obvious to the user if the system is setup correctly. For example, if Riak is not running the user experience is as follows:

```bash
$ riak-cs start
$ echo &?
0
$ curl http://localhost:8080/user --data "email=foobar@foobar.com&name=foo%20bar"
curl: (7) couldn't connect to host
```

At this point, the user is only aware that Riak CS is not responding to requests on the expected port. Possible problems include:
* Riak CS is configured with a different IP or Port
* Riak CS is misconfigured in such a way that it crashed
* Riak CS could not connect to Riak

A review of the Riak CS log reveals that poolboy crashed when it couldn't connect to Riak and brought down the Riak CS node.

```log
2012-08-08 16:01:02.289 [error] <0.90.0> CRASH REPORT Process <0.90.0> with 0 neighbours exited with reason: {tcp,econnrefused} in gen_server:init_it/6
2012-08-08 16:01:02.290 [error] <0.89.0> Supervisor poolboy_sup had child riak_moss_riakc_pool_worker started with {riak_moss_riakc_pool_worker,start_link,undefined} at <0.90.0> exit with reason {tcp,econnrefused} in context child_terminated
```

## Ideal Behavior

Ideal behavior for Riak CS in this scenario:

* Start successfully
* Output a clearer log message indicating that it could not connect to Riak
* Respond to requests with an appropriate error code (e.g. 502)

```bash
$ riak-cs start
$ echo &?
0
$ curl http://localhost:8080/user --data "email=foobar@foobar.com&name=foo%20bar"
InternalErrorWe encountered an internal error. Please try again./user4442587FB7D0A2F9
```

Unfortunately [S3 error codes](http://docs.amazonwebservices.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList) only include 500 and 501. A 502, 503, or 504 seems more appropriate for this scenario.

## TODO

- [ ] handle Riak not available on startup
- [ ] handle Riak not available on background services (gc, storage, access daemons)
- [ ] notify the user via log messages that Riak is not available
- [ ] Riak CS successfully automatically reconnects shortly after Riak becomes available
- [ ] Return 503 in as many places as possible if `riakc_pb_socket` returns `{error, disconnected}`, instead of crashing