bootstrap: Return recovery error until hsm is ready

When constructing HSM, a background thread is started, invoking
lvm.bootstrap and other initialization tasks.  Since lvm bootstrap is
performed in a background thread, it may not be done when vdsm is asked
to connect to storage pool or manipulate images.  This may lead to a
race when lvm bootstrap thread is deactivating a logical volume after
engine asked to activate it.

I considered moving lvm bootstrap into the main thread; However this may
delay startup by a minute or more when we have huge amount of lvs. While
server is waiting for bootstrap to finish, engine does not get any
response from vdsm, and if vdsm is currently the SPM, engine cannot
switch SPM to another host.

This patch uses the recovery mechanism to respond with a recovery error
until hsm background thread is done. From the user point of view, the
host is "Initializing" until hsm is ready. If initializing host is the
SPM, engine can switch SPM to another host as soon as vdsm is up.

If a task in hsm background thread has a fatal error, hsm will never
become ready, and the host will be considered initializing. This patch
ensures that we cannot use storage and cause data corruption of storage
initialization failed. This change may require engine support, fencing a
host in this state.

Change-Id: Id74468917c5b7c05d4183854e2f1255de98325dc
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-on: http://gerrit.ovirt.org/21530
Reviewed-by: Dan Kenigsberg <danken@redhat.com>
Tested-by: Dan Kenigsberg <danken@redhat.com>