move wait out of yz_solr_proc:init

The init function must return in five seconds or less, but solr startup
may take longer. We should be able to start the JVM (after checking the
data dir) in init - if that takes more than five seconds, we likely have
serious other problems.

The yz_solr_proc process still periodically pings Solr until it responds
successfully, or until the set timeout expires. It's just done after
init returns.

Removing the wait from init means that parts of yokozuna may try to
contact solr before it's up. This should be okay, because yokozuna should
be prepared for solr operations to fail anyway.

The idea behind yz_solr_proc checking that Solr does eventually become
available, even if it doesn't prevent other parts from attempting to use
it in the meantime, is to have an arbiter decide that Solr has been
unavailable for too long (since JVM startup), and will therefore likely
never become available.

It may be a good idea to continue checking Solr's availability even after
the first success. This might give an early warning on a Solr that has
gone too lunch, and a better signal than simply lots of requests failing
or timing out.
setup supervisor restart strategies to properly handle yz_solr_proc

YZ's solr management process waits to see if Solr eventually becomes
available after being started. If it doesn't seen Solr reply successfully
in a set amount of time, the management process exits.

The set amount of time is relatively long - 15 seconds by default, but
may be set higher by the user. To get this to behave correctly under a
supervisor, the restart strategy has to be both narrow, and also decided
at startup.

This patch moves yz_solr_proc under its own supervisor, yz_solr_sup, to
get the restart strategy necessary (no more than 1 restart in 3x the
startup wait time).

In addition, all other processes that were started by yz_sup are moved
underneath a new supervisor, yz_general_sup, where they have the same
restart strategy as always.

The new supervisors are started by the top-level yz_sup, which now has
a *very* restrictive restart policy: neither of its children are ever
allowed to restart. This was necessary to allow yz_solr_proc to be able
to cause the whole yokozuna application to tear down - if it was under
a supervisor that had the same restart policy as yz_general_sup, it would
have just been restarted, negating the point of its own restart policy.
test for solr startup timeout

This test cripples yokozuna such that jetty will start, but solr will not.
This allows us to test that yz_solr_proc does, in fact, tear down the
riak node if solr doesn't start up before the set timeout.
git commit fix solr start timeout & verify to reset nodes correctly

If verify.sh builds stagedevrel, then the lib/yokozuna directories in the
dev nodes are symlinks to the yokozuna built in the work space. Since they
are symlinks, their contents are not put under git control. Thus, the git-reset
between tests does not replace the the solr.war deleted in yz_solr_start_timeout.

This patch uses devrel instead of stage devrel to make the git-reset work.
don't mark yz up until solr is up

Since YZ can't really do anything without solr, tie its riak_core service
availability to solr instead of the app in general.
a better way to make solr startup "fail" for testing

Being unable to use stagedevrel for testing was inconvenient, so this
patch brings it back.

Instead of deleting solr.war, the test configures the YZ data directory
to a well-known string that causes yz_solr_proc:build_cmd to return something
other than jetty/solr/java.

A catch was also added to the port_close command in terminate, because the
fake-solr (grep) dies much faster than the real solr, so that port_close call
will fail, obscuring the reason for the process exit.
correct notes about yz_solr_start_timeout

the test no longer deletes solr.war, so the note was incorrect
match largest shutdown time under yz_general_sup

yz_index_hashtree_sup has an 'infinity' shutdown time, so its
supervisor should as well
Merge PR #188 from branch 'refactor/bwf/jvm-init' into develop