tests/043: refine it to cache more bugs

Current master couldn't pass this refined version

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: rename some stuff about local request handling

This is a prepare patch for later patch.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: use single list for request wait queue

Multi-queues aims to provide more finer requests retry control but end up being
much more error prone. We don't need multi-queues because
 1) VM mostly just issue several IO requests in one go and won't progress until
    getting response of those previous IO requests, so most likely we don't have
    huge number of requests on the wait queues.
 2) single queue is very easy to make correct

This patch 
 a) fixes a bug found by refined 043
 b) use single list instead of multi-lists for wait queue
 c) add comment for requests retry
 d) refactor the code and rename the helpers a bit

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: don't start recovery for leave node

There is no sense for leave node (gateway only) to recover objects

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: prepare and cleanup err_to_stderr() for MD recovery

MD recovery relies on the err_to_stderr to catch EIO and do its own EIO handling
tricks.

We don't handle EIO for object cache and journal file gracefully and use
err_to_stderr() to translate errno is non-sense for now because MD recovery
don't care about other path except backend stores.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
work queue: don't call worker func if not assigned

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
recovery: start tgt_epoch with rw->epoch instead of 'rw->epoch - 1'

MD recovery will start recovery without epoch lifted. This will cause troulbe
when sys_epoch = 1, which will result in tgt_target = 0, and wrap around later
to be a very large but valid epoch to the recovery code and cause trouble.

This is a prepare patch for MD recovery.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
md: implement automatic recovery

I choose to take advantage of current node recovery mechanism because:
 1. It supports multiple failure events
 2. It provide nice race-free and wait queue mechanism for VM IO requests and
    recovery IO requests.
 3. already-made logic that can recover objects from other nodes. This is very
    crutial to md recovery because we distribute objects across disks without
    extra copies.
 4. Maintan one mechanism is always better than two

So disk failure will simply be seen as a node recovery without epoch lifted and
md recovery has all the merits of node recovery:
 1. multiple disk failures handling
 2. serve VM IO requests while in recovery
 3. can hot-plug and hot-unplug the disks (missing part yet)

Besides, another nice spin-off is that with unified recovery, both node and disk
recovery can suspend and supersede each other (missing part yet)

The missing parts will be implemented in later patch.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: modify _start_sheep helper with one more disk

This is a prepare patch.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: add a test 055 for md

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
md: try hard to link stale objects during node events

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: add 056 to test disk failure with node event change

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>