sheep: queue cluster and local requests directly

Now that we can process events and requests in parallel, and events aren't
queued anymore we don't need to queue requests either.

Start with offloading local and cluster requests to the workqueues directly
in this patch and leave the slightly more complex handling of I/O requests
to the next patch.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: remove the request_queue

Offload I/O requests to the worker threads as soon as we get them in
queue_request, or when walking one of the lists of delayed requests.

Also move the code related to queueing I/O requests from group.c and
recovery.c into sdnet.c to have it in a single place.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: remove dead code in get_cluster_status

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: consolidate checks for enough available zones

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: store nodes in sys->nodes in log_last_epoch

We'll need to generate a vnode list for this special case later on, so
remove the special case where we log a node list from outside of sys.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: store the node list in struct
This will allow us to keep the previous node list around for recovery, and thus
avoid reading the node list when starting recovery.

I've decided to make struct vnode_info public as part of this even if I'm not
entirely happy about it, but adding accessors for the node array seemed like
an even uglier alternative.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: move more join code to update_cluster_info

This keeps the successful join code in one place, and makes the changes in
the next patch a lot easier.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: refactor read/write_object()

It would be nice if all the read/write goes to the forward_read/write_req_obj()
This also cleans up the code a bit. But this patch goes further than refactor:

- for now only vdi opreation calls read/write_object(), which originally use
  connect_to() but forward_read/write_req_obj() does a timeout read/write.

- I'm planing to use these functions for object cache writethrough mode.

- write_inode_cache() -> write_object_cache()

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: further refactor functions to use forward_read_obj_req()

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: let forward_read_obj_req() read random copy

Read random copy from cluster for better load balance, useful for
reading base VM's COW objects

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
corosync: do not overload event types

Use different types for join requests vs responses, and block vs notify
events intead of using the blocked field to overload the type.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
make recovery not to retry when recover_object_from_replica() fail

Since we make sheep to wait to retry when epoch is inconsistent,
recover_object_from_replica() will never get a response with
SD_RES_NEW_NODE_VER, because the peer node will retry the request
itself locally until epoch gets consistent.

If epoch of request sender is old than the receiver, it would get
SD_RES_OLD_NODE_VER, in this case, it means the epoch it's to increment
and soon a new recovery work would replace the current one, we should
not waste time recovering for the out-of-date recovery work, what we
should do is to make the current recovery work cease to wait for replacement.

As for SD_RES_NETWORK_ERROR, currently, recover_object_from_replica() will
get SD_RES_NEWWORK_ERROR only if there's an EIO when reading the object,
in this case we should not make recovery retry, because next time it may
get an EIO either and so that make the recovery work hang there retrying
constantly, we should make it retry another copies or in another epoch.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>