sheep: set keepalive in all client socket fd

We have set keepalive in cached socket fd, but uncached fd should also
need to set this option. So I move set_keepalive() from get_sheep_fd()
to connect_to().

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: set keepalive in server-side socket fd

It's important to set keepalive option in server-side socket fd, this
can help the server to clear deaded connection quickly when its client
crashed for unexpected reason (eg: client reboot or be killed).

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: simplify the cluster driver interface for blocking events

Let sd_block_handler handle the fine details of how to handle an incoming
blocking event.  By passing the sender node structure we can easily handle
ignoring it on other nodes, and by keeping a local operation in progress
flag in group.c we can replace the callbacked flag in the on the wire
events with a way simpler mechanism.

The only slightly complicated bit is that zk_notify_blocked in the zookeeper
backend can now go negative for a short period of time, so we explicitly
have to check for it beeing positive in two places.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: factor leave_list processing

Factor handling of the leave list into a few helpers to make it more obvious.
Also rename the leave name component to failed/failed_nodes to make the use
case more obvious.  Finally, remove the nodes/leave_nodes union in favor of
a single array, as an union of two fields with the same type is fairly
confusing.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: return CJ_ values directly from cluster_sanity_check

Don't return SD_RES_ values from cluster_sanity_check first and then translate
them to the CJ_ values later.  This also shows that the result field in struct
join_message wasn't needed and can be removed entirely.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: remove SD_RES_JOIN_FAILED

We never set this status, so remove it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: refactor get_cluster_status

Redo the way get_cluster_status and cluster_sanity_check works.

The new get_cluster_status is a switch statement over all possible system
states, which then branches out into helper functions for states where
the checks are non-trivial.  This makes the intention of the checks a lot
more obvious.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: handle CJ_RES_MASTER_TRANSFER inside cluster_wait_for_join_check

Instead of fixing up the return value down in sd_check_join_cb handle this
special case directly inside cluster_wait_for_join_check.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: handle new nodes during WAIT_FOR_JOIN state

There is no problem letting new nodes join a cluster in WAIT_FOR_JOIN state as
long as they aren't counted towards the quorum of nodes requires to restart
the cluster.  For this repurpose the CJ_RES_JOIN_LATER return value that is
currently equivalent to CJ_RES_FAIL to indicate we have a node that could
join, but wasn't part of the cluster before shutdown.  A new nr_delayed_nodes
counter and list of delayed nodes is also added to the join message to make
sure all nodes know about this kind of delayed node.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: pass a node list to snap_file_write

This will allow passing in a node list from recovery later.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: pass vnode info to end_recovery

This allows farm to reuse the old epoch node list instead of re-reading it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: dont rewrite epoch log for the epoch before a node joins

Due to the way the cluster status and inc_epoch works this can get us
different historic epochs for different sheep in a cluster. Instead
create node list for the previous generation in memory if we need to.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: remove WORK_ORDERED and related code

This flag is entirely unused, and just complicates the workqueue code.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix strlen in log_enqueue

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: print thread id in logfile which could help debuging

Main thread's id is 0, other thread's id will be [1..nr_thread].

The output format looks like these:
 Jun 17 02:30:31|0|queue_request(387) 3
 Jun 17 02:30:31|1|do_gateway_request(280) 3, 80a5d05d00000000 , 1
                 ^
                 |
                 --- Here is thread id

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>