test: add a test for sockfd keepalive

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
net: add a send timeout for sockfd

This fixes the bug found by 035.

This is kind of a subtle case:

'sendmsg' of A  would fill its sock buffer to full whlile the other end B is failed
in the middle of the 'sendsmg'. The sock buffer of A is full, then kernel block
the thread awaiting on the buffer drain, which will never happen because B is
down. Keepalive won't be triggered because the kernel detect that sock buffer
has data to send. So the final result is that, the blocked thread will never be
woke up.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: cleanup string format for epoch

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fix a bug that hangs the cluster during recovery

During recovery, a VDI creation request may waits for recovery
to complete, and VDI creation request is a cluster request which
prevent other cluster requests being processed, when recovery comes
to notify_recovery_completion_work, it issues another cluster request
with SD_OP_COMPLETE_RECOVERY which is blocked by VDI creation, and
as result, notify_recovery_completion_work blocks the recovery_wqueue,
if a new recovery comes, it's blocked, at the same time, a VDI creation
request may waits for this recovery to complete, so it's a dead lock.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: fix error output of 'collie node info'

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
test: refine 026 to catch more bugs

With this change, current master will fail most of time.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix dead lock for create_and_write request

This dead lock can be reprodiced by 026.

We should always service CREATE_AND_WRITE request instead of queueing it
on wait queues while in recovery. The recovery can be finished without any
any objects in the list (rw->count == 0 in some special case), in which case
that no one calls resume_wait_recovery_requests() or other flusher on rw_list or
obj_list.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
collie: remove cluster cleanup command

We for now automatically clean up useless objects generated from
recovery process.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
recovery: fix wrong index in oid_in_recovery()

rw->done is index of the original next object to be recovered and also the
number of objects already recovered. So rw->done - 1 identify an already
recovered object. A another caller case is that, if rw->done == 0, we'll end
up with a -1 valule, which might cause seg fault in the array.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: no need to set SD_FLAG_CMD_WRITE for local request

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: handle SD_FLAG_CMD_WRITE in cluster requests

If SD_FLAG_CMD_WRITE is set, req->data means sent data, so we don't
need to set response data in sd_notify_handler().  Otherwise, it means
receive data buffer, so we don't need to set multicast data in
prepare_cluster_msg().

This also fixes some local requests which don't set SD_FLAG_CMD_WRITE.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
cluster/corosync: retry when CS_ERR_TRY_AGAIN is returned

cpg_initialize often fails with a CS_ERR_TRY_AGAIN error.  We should
retry in such case.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
test: fix 003 for local driver

Local driver detect membership change by signal in 1 s internal, so kill &sleep 1
might have no effect for local driver.

This might cause random spurious failure of test.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>