test: tests/check failed wiih error 'invalid range end'

tests/check failed on Centos 6.x because 'sed' can't
work with '[A-z]'.
It would fail with 'invalid range end' error message.

Fixed it by replace [A-z] with [A-Za-z].

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: do breadth first search in recover_object_from_replica()

This prepares for the succeeding patches.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: don't try to recover from working directory

It is obvious that sheep doesn't have the object in the working
directory.  In this case, recover_object_from_replica() should return
SD_RES_NO_OBJ and try the next node asap.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: recover from local if possible

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: move objects to stale direcotry before starting recovery

Currently, if Sheepdog crashes before finishing recovery, stale objects can
exist in the working directory.  This confuses us when we recover Sheepdog
manually when the cluster completely goes wrong because we cannot know which
objects are the latest ones.

This patch moves the stale objects to .stale directory before starting object
recovery and adds the correct suffix when they became stale.  After this change,
we can guarantee that

 - All the objects in the working directory are the latest ones.
 - If the stale object has the newer suffix, it is the newer data.  For example,
   if node A has 0000.1 and node B has 0000.2 in the stale directories, 0000.2
   is a newer object.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
md: kick recovery outside of md_lock context

Now we may call for_each_object_in_wd() in kick_recovery() and get a read lock
of md_lock in it.  To avoid a dead lock, kick_recovery() must be called outside
the write lock context.  Even if we don't get the lock inside kick_recovery(),
it is better to release the write lock asap to avoid blocking long time.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: try to recover only from the target epoch

Now we have a stale object with an appropriate suffix, so no need to try to read
from the newer epoch.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests/010: remove stale objects from working directory contents

The working directory doesn't contain stale objects any more.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: recover objects in the ordered worker thread

The current recovery code are not expected to work in multi threads.  However,
cluster_enable_recover() can queue a new recovery work even if there is a
running one.  To avoid this problem, this patch uses an ordered work queue for
the recovery work.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: split struct recovery_work

Currently, sheepdog recovery code has a race problem because the worker thread
can access recovery_work members which are not expected to be read outside of
the main thread.  For example, the worker thread can read rw->oids in
do_recover_object() while the main thread reallocates it in
finish_schedule_oids().  The root cause is that the main thread passes
recovering_work to the worker thread even though the variable is not thread safe.

This patch splits recovery_work into four types to remove the race condition:

 - recovery_info      : This was recover_work and can be accessed only
                        in the main thread.
 - recovery_list_work : This is used for preparing object list in the worker
                        thread.
 - recovery_obj_work  : This is used for recovering objects
                        in the worker thread.
 - recovery_work      : This is a super class of recovery_list_work and
                        recovery_obj_work, and is also used for notifying
                        recovery completion in the worker thread.

I think this also improves code readability and makes it easier to extend the
current recovery code to multi-threaded in future.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: show error message when object may be lost

From the point of view of block storage semantics, sheepdog must not show the
old data to clients.  If sheep may break the data consistency, sheep should
alert it so that users can know the problem.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
collie/node: correct md info output

We actually return 'Free' instead of 'Size'

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: rename bs_thread_request_done() -> worker_thread_request_done()

The prefix "bs_" of the name bs_thread_request_done() would've come
from the naming scheme of tgt (the "backing store"). This prefix is
unclear so this patch rename it.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
md: purge directory only when md_add_disk() succeed

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/057: fix 057 and 057.out

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
cluster/local: don't process leave event of gateway node

We must not call sd_leave_handler() when the gateway node leaves cluster.  It is
because we already processed the event when the node became gateway.

The local driver can pass tests/048 with this patch.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: check whether there is a pending recovery correctly

Comparing current_rinfo->epoch and sys->epoch is not enough because sheep may
start recovery without incrementing sys->epoch (e.g. unplugging a md disk).
Checking next_rinfo is correct and more straightforward.

This fixes an error of tests/056.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
md: correct partial plug failure handling

If we plug multiple disks and some of them fail to purge directory, just call
md_del_disk and keep processing because successful plugging already add some
disks in the array.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: clean vdi_state when formatting cluster

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
zk: actively remove itself after leave_cluster() is called

actively remove itself from zookeeper than waitting for timeout.

Signed-off-by: ChenZhe <cz@de3eb.cn>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>