ctdb-tests: Try harder to avoid failures due to repeated recoveries

About a year ago a check was added to _cluster_is_healthy() to make
sure that node 0 isn't in recovery.  This was to avoid unexpected
recoveries causing tests to fail.  However, it was misguided because
each test initially calls cluster_is_healthy() and will now fail if an
unexpected recovery occurs.

Instead, have cluster_is_healthy() warn if the cluster is in recovery.

Also:

* Rename wait_until_healthy() to wait_until_ready() because it waits
  until both healthy and out of recovery.

* Change the post-recovery sleep in restart_ctdb() to 2 seconds and
  add a loop to wait (for 2 seconds at a time) if the cluster is back
  in recovery.  The logic here is that the re-recovery timeout has
  been set to 1 second, so sleeping for just 1 second might race
  against the next recovery.

* Use reverse logic in node_has_status() so that it works for "all".

* Tweak wait_until() so that it can handle timeouts with a
  recheck-interval specified.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
ctdb-daemon: Debugging for tickle updates

This was useful for debugging the race fixed by commit
4f79fa6c7c843502fcdaa2dead534ea3719b9f69.  It might be useful again.

Also fix a nearby comment typo.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jun 20 02:07:48 CEST 2014 on sn-devel-104
ctdb-eventscripts: Ensure $GANRECDIR points to configured subdirectory

Check that the $GANRECDIR symlink points to the location specified by
$CTDB_GANESHA_REC_SUBDIR and replace it if incorrect.  This handles
reconfiguration and filesystem changes.

While touching this code:

* Create the $GANRECDIR link as a separate step if it doesn't exist.
  This means there is only 1 place where the link is created.

* Change some variables names to the style used for local function
  variables.

* Remove some "ln failed" error messages.  ln failures will be logged
  anyway.

* Add -v to various mkdir/rm/ln commands so that these actions are
  logged when they actually do something.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jun 20 05:40:16 CEST 2014 on sn-devel-104