sheep: purge directory when plugging the new disk

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: abort when disk store is meta-store

This can stop sheep from rebooting when disk store is a meta-store, such as in a
disordered path parameter

$ sheep /disk1,/disk2,/meta-store # abort

We can only reboot when first component is meta-store to avoid mess-up.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
reconnect in case of EINTR

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: calculate buffer size from object type

The current vdi check doesn't use the correct buffer size for vdi objects.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: show progress bar for vdi check and vdi create -P

These operations take a long time to complete.  Showing a prograss bar helps
users.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
[don't merge] tests: valgrind option for lower memory machine

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/008: read vdi in paralell

We can read each vdi in parallel for better performance.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests: don't use /dev/urandom for better performance

/dev/urandom is too slow.  We don't need good random values to fill vdis.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: don't try to recover from working directory

It is obvious that sheep doesn't have the object in the working
directory.  In this case, recover_object_from_replica() should return
SD_RES_NO_OBJ and try the next node asap.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: recover from local if possible

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: move objects to stale direcotry before starting recovery

Currently, if Sheepdog crashes before finishing recovery, stale objects can
exist in the working directory.  This confuses us when we recover Sheepdog
manually when the cluster completely goes wrong because we cannot know which
objects are the latest ones.

This patch moves the stale objects to .stale directory before starting object
recovery and adds the correct suffix when they became stale.  After this change,
we can guarantee that

 - All the objects in the working directory are the latest ones.
 - If the stale object has the newer suffix, it is the newer data.  For example,
   if node A has 0000.1 and node B has 0000.2 in the stale directories, 0000.2
   is a newer object.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
md: kick recovery outside of md_lock context

Now we may call for_each_object_in_wd() in kick_recovery() and get a read lock
of md_lock in it.  To avoid a dead lock, kick_recovery() must be called outside
the write lock context.  Even if we don't get the lock inside kick_recovery(),
it is better to release the write lock asap to avoid blocking long time.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: try to recover only from the target epoch

Now we have a stale object with an appropriate suffix, so no need to try to read
from the newer epoch.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/010: remove stale objects from working directory contents

The working directory doesn't contain stale objects any more.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: recover objects in the ordered worker thread

The current recovery code are not expected to work in multi threads.  However,
cluster_enable_recover() can queue a new recovery work even if there is a
running one.  To avoid this problem, this patch uses an ordered work queue for
the recovery work.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: split struct recovery_work

Currently, sheepdog recovery code has a race problem because the worker thread
can access recovery_work members which are not expected to be read outside of
the main thread.  For example, the worker thread can read rw->oids in
do_recover_object() while the main thread reallocates it in
finish_schedule_oids().  The root cause is that the main thread passes
recovering_work to the worker thread even though the variable is not thread safe.

This patch splits recovery_work into four types to remove the race condition:

 - recovery_info      : This was recover_work and can be accessed only
                        in the main thread.
 - recovery_list_work : This is used for preparing object list in the worker
                        thread.
 - recovery_obj_work  : This is used for recovering objects
                        in the worker thread.
 - recovery_work      : This is a super class of recovery_list_work and
                        recovery_obj_work, and is also used for notifying
                        recovery completion in the worker thread.

I think this also improves code readability and makes it easier to extend the
current recovery code to multi-threaded in future.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: show error message when object may be lost

From the point of view of block storage semantics, sheepdog must not show the
old data to clients.  If sheep may break the data consistency, sheep should
alert it so that users can know the problem.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
[tmp] sheep: add support get hash

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
[tmp] collie: use hash for collie check

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
[tmp] try to recover from local as far as we can

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>