tests: check message order between vdi creation and object recovery

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
cluster/corosync: order unblock messsage with confchg events

Multicast messages after block operations also must be ordered with
confchg events.  For example, if an unblock event of SD_OP_NEW_VDI is
not ordered with confchg events, post_cluster_new_vdi() may not be
called on newly added nodes.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: wait for vdi copies to be updated in recovery process

If node membership changes between process_work and process_main of
SD_OP_NEW_VDI, we cannot get the vdi copies in recovery process.  In
that case, we need to wait for post_cluster_new_vdi to finish.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
store: return SD_RES_NO_OBJ when object size is zero

If clients read objects which is just being created, the size of read
data can be zero and clients get a SD_RES_EIO error.  Sheep leaves
cluster when the SD_RES_EIO error happens, so we should return
SD_RES_NO_OBJ in this case.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: check return value of get_vdis_from

A newly added node sends SD_GET_VDI_COPIES to only one node, so if the
node is not available, the added node cannot get vdi copies info.  We
should try another node in such case.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: fix uninitialized variables

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
corosync: give up cpg_initialize after retrying CPG_INIT_RETRY_CNT times

It seems that, with corosync 1.x.x, cpg_initialize returns
CS_ERR_TRY_AGAIN when the corosync daemon is not running.

This patch avoids an infinite loop to limit the retry count.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: remove unused variable in cluster_make_fs

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
test: fix random spurious failure by using _wait_for_sheep helper

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
object cache: fix crash when deleting object cache

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
local: remove checksum

Currently, local driver doesn't update both queue and checksum data
atomically, so if the process exits unexpectedly after updating queue
and before doing checksum, a newly started process can regard the
queue as invalid.

This patch removes checksum feature completely from local driver.
Though we should introduce something like journaling to update queue
safely, it looks overkill because local driver is only for testing
purpose.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
local: check process at shorter intervals

Some testcases regards that sheep can detect node failure in less than
1 second.  This patch changes the interval of process check from 1
second to 200 milliseconds.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
local: use a different queue for block operations

This is the same change which was done in corosync driver to avoid
deadlock of vdi creations.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
local: avoid calling sd_block_handler twice

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
local: don't process events until joining to cluster

This patch also removes the master node from the local driver.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
local: remove blocking event when its sender goes away

This avoids a dead lock.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
local: add more debug outputs

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: ensure that sheeps are not running if necessary

This ensures that sheeps are not running when
 - starting a testcase
 - unmounting devices
 - starting a sheep daemon

This removes a lot of false test errors.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests/040: wait for sheeps to finish starting up before listing vdis

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: add valgrind support

If you add '-valgrind' to command options, check script wraps all
sheep calls with valgrind.  This makes it easier to debug problems and
detect memory leaks.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>