sheep: use cached sockfd for all connections

This will especially boost recovery performance a bit.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: add a helper to send req on top of sockfd cache

This will greatly reduce the duplicated and lenghty code.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
gateway: clean up gateway_read_obj()

- use gateway_init_fwd_hdr() to init forward header.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests: sleep 1 second before unmount

Seems that umount just after stopping sheeps fails occasionally.
Let's add 1 second sleep before umount again and see how it works.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests: use wait instead of _wait_for_collie

When we use -valgrind option, there is a delay before collie starts
up.  In the worst case, we could call _wait_for_collie before collie
starts.  To solve the problem, this simply waits for background
processes to finish rather than iterating pgrep and sleep.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Revert "collie: add cluster recover info command"

With the new manual recovery design, we need to increment epoch even
when object recovery is disabled, so the command "cluster recover
info" is no longer required.

This reverts commit d652f24f00bef5045233083ff704b252a54beffe.

Conflicts:

	include/internal_proto.h
	sheep/ops.c

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Revert "sheep: do the real work of dsiable/enable recovery"

It requires quite big changes to extend this patch to handle leave
nodes.  The next patch will implements manual recovery with much
simpler approach.

This reverts commit a9186fd8f0e133ba7065a579da1062650991e89a.

Conflicts:

	sheep/group.c
	sheep/sheep_priv.h

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: add support for manual recovery

This implements disbling object recovery.  When recoery is disabled,
sheep will suspend the recovery process after it recovers objects in
the prio_oids queue.  The suspended recovery is resumed after new
objects are pushed into the prio_oids queue.  This means that
unaccessed objects are not recovered at all.

Note that sheep increments epoch even when recovery is disabled.  If
sheep receives a write request, it will place the updated object based
on the current node membership.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests: rewrite 010 to test new manual recovery

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: make create operations atomic

Currently, CREATE_AND_WRITE operations are processed even if recovery
works of their target objects are on going (commit b3442e66).  This
causes a problem in the following situation (and it acutually happens
with test 008):

  1. There are three nodes, A, B, and C.
  2. CREATE_AND_WRITE operations are sent to them, and at the same
     time, a node D is added to cluster.
  3. The node D recovers a object from node A, but the object is just
     being created.  The node D reads a zero filled object; it is
     fallocated, but data is not written yet.
  4. The CREATE_AND_WRITE operation is retried to node D, and it is
     successfully finished.
  5. The recovery process on node D overwrite the object with the zero
     filled data.

Instead of adding another complex check in request_in_recovery, this
patch makes all CREATE_AND_WRITE operations atomic to keep code
simplicity.  With this patch, store->write is splited into
store->write and store->create_and_write, and store->atomic_put is
removed.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>