sheep: do breadth first search in recover_object_from_replica()

This prepares for the succeeding patches.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: don't try to recover from working directory

It is obvious that sheep doesn't have the object in the working
directory.  In this case, recover_object_from_replica() should return
SD_RES_NO_OBJ and try the next node asap.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: recover from local if possible

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: move objects to stale direcotry before starting recovery

Currently, if Sheepdog crashes before finishing recovery, stale objects can
exist in the working directory.  This confuses us when we recover Sheepdog
manually when the cluster completely goes wrong because we cannot know which
objects are the latest ones.

This patch moves the stale objects to .stale directory before starting object
recovery and adds the correct suffix when they became stale.  After this change,
we can guarantee that

 - All the objects in the working directory are the latest ones.
 - If the stale object has the newer suffix, it is the newer data.  For example,
   if node A has 0000.1 and node B has 0000.2 in the stale directories, 0000.2
   is a newer object.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
md: kick recovery outside of md_lock context

Now we may call for_each_object_in_wd() in kick_recovery() and get a read lock
of md_lock in it.  To avoid a dead lock, kick_recovery() must be called outside
the write lock context.  Even if we don't get the lock inside kick_recovery(),
it is better to release the write lock asap to avoid blocking long time.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: try to recover only from the target epoch

Now we have a stale object with an appropriate suffix, so no need to try to read
from the newer epoch.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests/010: remove stale objects from working directory contents

The working directory doesn't contain stale objects any more.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: recover objects in the ordered worker thread

The current recovery code are not expected to work in multi threads.  However,
cluster_enable_recover() can queue a new recovery work even if there is a
running one.  To avoid this problem, this patch uses an ordered work queue for
the recovery work.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: split struct recovery_work

Currently, sheepdog recovery code has a race problem because the worker thread
can access recovery_work members which are not expected to be read outside of
the main thread.  For example, the worker thread can read rw->oids in
do_recover_object() while the main thread reallocates it in
finish_schedule_oids().  The root cause is that the main thread passes
recovering_work to the worker thread even though the variable is not thread safe.

This patch splits recovery_work into four types to remove the race condition:

 - recovery_info      : This was recover_work and can be accessed only
                        in the main thread.
 - recovery_list_work : This is used for preparing object list in the worker
                        thread.
 - recovery_obj_work  : This is used for recovering objects
                        in the worker thread.
 - recovery_work      : This is a super class of recovery_list_work and
                        recovery_obj_work, and is also used for notifying
                        recovery completion in the worker thread.

I think this also improves code readability and makes it easier to extend the
current recovery code to multi-threaded in future.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: show error message when object may be lost

From the point of view of block storage semantics, sheepdog must not show the
old data to clients.  If sheep may break the data consistency, sheep should
alert it so that users can know the problem.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>