basho/yokozuna — pull request #188 by beerriot (2013-09-26 21:08:40+00:00)
Title: Move wait_for_solr out of yz_solr_proc:init

Solr startup may take several seconds. All `gen_server:init` functions must finish within five seconds, though. So, waiting for Solr to become available in `yz_solr_proc:init` is not practical.

This PR splits the startup into two stages.

1. In `init`, we still make sure the data directory is setup correctly, and also spawn the JVM process.
1. After returning from init, `yz_solr_proc` checks once per second whether Solr has finished starting. If it hasn't finished by the set timeout, `yz_solr_proc` exits with a message about the timeout.

In order to support this new startup timeout, the application supervisors have also been rearranged. The `yz_solr_proc` process now lives under its own `yz_solr_sup` supervisor, while all other servers now live under a `yz_general_sup` supervisor. This is to allow for different restart policies for the children of each. Both of these new supervisors now live under the top-level `yz_sup`, where the restart strategy sets max restarts to zero. The idea is that if any child of the `solr` or `general` supervisor misbehaves enough that one of those supervisors exits, the whole app should tear down.

The restart strategy of `yz_solr_sup` attempts to match Solr's startup time. This PR sets things so that `yz_solr_sup` is not allowed to restart more than once within a span three times the length of the configured startup timeout. The idea is that if Solr falls over, the supervisor will attempt to restart it once, but if it fails to restart fast enough, the restart is not tried again.

A too-slow initial startup looks a little odd: `yz_solr_sup` exits with a message about the startup timeout, and then the supervisor tries to restart it. Typically this restart happens too quickly for the JVM to die and release network resources, though, so the restart will fail immediately when Jetty gets an error about ports already being in use. The biggest problem I foresee with this is that users are likely to send us log excerpts of the wrong error message. I'm not sure I see a clean fix yet, though.

The final commit of this PR also moves the registration with the `riak_core_node_watcher` service to `yz_solr_proc` so that it doesn't happen until Solr has become available. This also ties the service's status to the `yz_solr_proc` process instead of the yz supervisor. Since Erlang application startup no longer waits for Solr, this seemed the right thing to do.

The test for this change is `yz_solr_start_timeout`. It's a little bit of a tricky test. What it needs to do is prevent Solr from starting, but allow the JVM to start. This will keep `yz_solr_proc` in its wait state until it reaches its timeout. The best way I've found to make this happen so far is to delete `solr.war` from the node under test. An error is printed in the Solr log, but Jetty still hangs around anyway. This makes testing a little sticky for two reasons:

1. If yz_solr_start_timeout fails, the dev node will be left in a state that will never correctly start solr, because solr.war is missing.
1. I had to change `verify.sh` to use `devrel` instead of `stagedevrel`. With `stagedevrel`, the `solr.war` is contained in a symlinked yokozuna directory. The git-reset done by riak_test between tests does not reach into symlinked directories, so `solr.war` would never be restored.

A possible better test would be to use intercepts to override `yz_solr_proc:build_cmd/3` to make it return any command other than Solr that would stick around but not respond to Solr pings. Using the "manual" test-specific method of installing intercepts is not fast enough to get one installed before the process starts, though. The global config version may be, but I didn't try because the correct location of the intercept module was also uncertain.