tests: check message order between vdi creation and object recovery

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
cluster/corosync: order unblock messsage with confchg events

Multicast messages after block operations also must be ordered with
confchg events.  For example, if an unblock event of SD_OP_NEW_VDI is
not ordered with confchg events, post_cluster_new_vdi() may not be
called on newly added nodes.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: wait for vdi copies to be updated in recovery process

If node membership changes between process_work and process_main of
SD_OP_NEW_VDI, we cannot get the vdi copies in recovery process.  In
that case, we need to wait for post_cluster_new_vdi to finish.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
store: return SD_RES_NO_OBJ when object size is zero

If clients read objects which is just being created, the size of read
data can be zero and clients get a SD_RES_EIO error.  Sheep leaves
cluster when the SD_RES_EIO error happens, so we should return
SD_RES_NO_OBJ in this case.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: check return value of get_vdis_from

A newly added node sends SD_GET_VDI_COPIES to only one node, so if the
node is not available, the added node cannot get vdi copies info.  We
should try another node in such case.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
collie: fix uninitialized variables

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
local: remove checksum

Currently, local driver doesn't update both queue and checksum data
atomically, so if the process exits unexpectedly after updating queue
and before doing checksum, a newly started process can regard the
queue as invalid.

This patch removes checksum feature completely from local driver.
Though we should introduce something like journaling to update queue
safely, it looks overkill because local driver is only for testing
purpose.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
local: check process at shorter intervals

Some testcases regards that sheep can detect node failure in less than
1 second.  This patch changes the interval of process check from 1
second to 200 milliseconds.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
local: use a different queue for block operations

This is the same change which was done in corosync driver to avoid
deadlock of vdi creations.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
local: avoid calling sd_block_handler twice

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
local: don't process events until joining to cluster

This patch also removes the master node from the local driver.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
local: remove blocking event when its sender goes away

This avoids a dead lock.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
local: add more debug outputs

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
corosync: give up cpg_initialize after retrying CPG_INIT_RETRY_CNT times

It seems that, with corosync 1.x.x, cpg_initialize returns
CS_ERR_TRY_AGAIN when the corosync daemon is not running.

This patch avoids an infinite loop to limit the retry count.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>