tests/008: read vdi in paralell

We can read each vdi in parallel for better performance.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: remove directory surely

rm -rf fails if new files are created before rm is completed.  This
patch renames the directory beforehand so that sheep cannot creates
new objects in it any more.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trivial: misc fixes of comments

This patch fixes some incorrect comments:
1. group.c: colosync -> corosync
2. sockfd_cache.c: vnode -> node
3. sheep_priv.h: past -> part

1 is a very trivial mistake. 2 wouldn't cause a serious confusion but
the description is not correct. vnode is a concept of consisntent
hashing and it is not a something connect to. 3 would be
confusing. Because the way of dealing with exceptional nodes is subtle
and hard to understand.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
[don't merge] tests: valgrind option for lower memory machine

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests: don't use /dev/urandom for better performance

/dev/urandom is too slow.  We don't need good random values to fill vdis.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: do breadth first search in recover_object_from_replica()

This prepares for the succeeding patches.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: don't try to recover from working directory

It is obvious that sheep doesn't have the object in the working
directory.  In this case, recover_object_from_replica() should return
SD_RES_NO_OBJ and try the next node asap.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: recover from local if possible

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: move objects to stale direcotry before starting recovery

Currently, if Sheepdog crashes before finishing recovery, stale objects can
exist in the working directory.  This confuses us when we recover Sheepdog
manually when the cluster completely goes wrong because we cannot know which
objects are the latest ones.

This patch moves the stale objects to .stale directory before starting object
recovery and adds the correct suffix when they became stale.  After this change,
we can guarantee that

 - All the objects in the working directory are the latest ones.
 - If the stale object has the newer suffix, it is the newer data.  For example,
   if node A has 0000.1 and node B has 0000.2 in the stale directories, 0000.2
   is a newer object.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
md: kick recovery outside of md_lock context

Now we may call for_each_object_in_wd() in kick_recovery() and get a read lock
of md_lock in it.  To avoid a dead lock, kick_recovery() must be called outside
the write lock context.  Even if we don't get the lock inside kick_recovery(),
it is better to release the write lock asap to avoid blocking long time.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: try to recover only from the target epoch

Now we have a stale object with an appropriate suffix, so no need to try to read
from the newer epoch.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/010: remove stale objects from working directory contents

The working directory doesn't contain stale objects any more.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: recover objects in the ordered worker thread

The current recovery code are not expected to work in multi threads.  However,
cluster_enable_recover() can queue a new recovery work even if there is a
running one.  To avoid this problem, this patch uses an ordered work queue for
the recovery work.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: split struct recovery_work

Currently, sheepdog recovery code has a race problem because the worker thread
can access recovery_work members which are not expected to be read outside of
the main thread.  For example, the worker thread can read rw->oids in
do_recover_object() while the main thread reallocates it in
finish_schedule_oids().  The root cause is that the main thread passes
recovering_work to the worker thread even though the variable is not thread safe.

This patch splits recovery_work into four types to remove the race condition:

 - recovery_info      : This was recover_work and can be accessed only
                        in the main thread.
 - recovery_list_work : This is used for preparing object list in the worker
                        thread.
 - recovery_obj_work  : This is used for recovering objects
                        in the worker thread.
 - recovery_work      : This is a super class of recovery_list_work and
                        recovery_obj_work, and is also used for notifying
                        recovery completion in the worker thread.

I think this also improves code readability and makes it easier to extend the
current recovery code to multi-threaded in future.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: show error message when object may be lost

From the point of view of block storage semantics, sheepdog must not show the
old data to clients.  If sheep may break the data consistency, sheep should
alert it so that users can know the problem.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
[tmp] sheep: add support get hash

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
[tmp] collie: use hash for collie check

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
[tmp] try to recover from local as far as we can

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
[tmp] clean up

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
[tmp] tests: shutdown while node changes

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>