tests: sleep 1 second before unmount

Seems that umount just after stopping sheeps fails occasionally.
Let's add 1 second sleep before umount again and see how it works.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: use wait instead of _wait_for_collie

When we use -valgrind option, there is a delay before collie starts
up.  In the worst case, we could call _wait_for_collie before collie
starts.  To solve the problem, this simply waits for background
processes to finish rather than iterating pgrep and sleep.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: update help message for collie cluster recover

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: make create operations atomic

Currently, CREATE_AND_WRITE operations are processed even if recovery
works of their target objects are on going (commit b3442e66).  This
causes a problem in the following situation (and it acutually happens
with test 008):

  1. There are three nodes, A, B, and C.
  2. CREATE_AND_WRITE operations are sent to them, and at the same
     time, a node D is added to cluster.
  3. The node D recovers a object from node A, but the object is just
     being created.  The node D reads a zero filled object; it is
     fallocated, but data is not written yet.
  4. The CREATE_AND_WRITE operation is retried to node D, and it is
     successfully finished.
  5. The recovery process on node D overwrite the object with the zero
     filled data.

Instead of adding another complex check in request_in_recovery, this
patch makes all CREATE_AND_WRITE operations atomic to keep code
simplicity.  With this patch, store->write is splited into
store->write and store->create_and_write, and store->atomic_put is
removed.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: open files with O_EXCL when creating objects

There is a race condition in default_create_and_write_obj because if
node membership changes during object creation, gateway and recovery
process could send CREATE operations to the same object at the same
time.  This causes, for example, the following problem.

  1. gateway request creates a tmp_path file
  2. recovery process also creates the same tmp_path file
  3. gateway request renames the tmp_path to a correct path
  4. recovery process also tries to rename, but it fails because the
     tmp_path doesn't exist

To avoid this problem, this patch uses a O_EXCL option to create
objects.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests/037: avoid false detection of network partion

We shouldn't stop more than half of nodes at the same time.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests/028: show md5 hash values

The VDI contents of this test are not random, so it's better to show
their md5 hash values to know which VDIs are corrupted.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: remove redundant ternary operator

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
store/plain_store: use epoch numbers as decimals

This makes stale object more human readable because we use decimal
numbers to print epochs in other places.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
store/plain: use link to recover local objects

It is not safe to rename stale objects to the working directory.  When
reading objects for recovery, default_read reads the working directory
first, and then, searches stale objects.  However, if the objects are
renamed from the stale directory to the working directory on the node
at the same time, the recovery read request fails to find the object.

To fix this race condition, this patch uses link/2 to keep stale
objects.  This fixes test failure of tests/028.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fix double put_vnode_info call

If cluster status is not SD_STATUS_OK after the request is retried,
queue_request doesn't set req->vinfo and free_request calls
put_vnode_info again.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fix vnode_info leak

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: use stdout and stdin for vdi backup and restore

This helps us to use backup/restore via ssh or pipe.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: exit program when EIO occurs

When EIO occurs, a sheep is expected to become a gateway node, but it
doesn't work now.  To fix this, we need to support changing virtual
nodes without stopping sheeps.  This patch makes it a future work, and
stops the program simply on the EIO error.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: introduce helpers to create loopback devices

This prepares for the next patches

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: add test of disk full handling

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: handle no space error of object I/Os

This handles no space error of object I/Os.  Handling this error when
writing a config or epoch logs is left for future work.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
store/plain: unlink temporary files when create_and_write fails

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
update bash_completion file

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
recovery: reorder recover_object_from_replica()

Move valloc() after store->link() operation will save us one glibc call for
recovering local objects.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>