sheep: sheep aborted during startup, still joined in cluster

Currently, create_cluster() is called before any thread is created.
Once any one of the following steps during startup is failed,
sheep calls exit() or abort() directly so that leave_cluster() is not called.
Other nodes would consider that that one should be alived.
This will cause many problems.

This is a reproducible case of using journal file. Hot fix also submit.
But, should we re-arrange the startup steps? Should we avoid panic()
because it is dangerous?

Steps:

1. Start three sheeps. Cluster manager is zookeeper.
for i in `seq 0 2`; do
    sheep/sheep /tmp/sd$i -y 127.0.0.1 -c zookeeper:127.0.0.1:2181 -z $i \
        -p 700$i -j size=64M -n
done

2. Format the cluster and create a vdi. Data object file 007c2b2500000000
is always written into sd1 according to sheepdog hash algorithm.
$ dog cluster format -c 1
$ dog vdi create test 4M -P

3. Write the vdi continuously.
for i in `seq 0 4194303`; do
    echo $i
    echo "a" | dog vdi write test $i 1
done

4. Kill all sheeps in another terminal during writing vdi.
$ killall sheep

5. Sometimes journal files are not cleaned up when sheeps exit.
If they are not found, try step 3 and step 4 again.
$ ls /tmp/sd*/journal*
/tmp/sd0/journal_file0  /tmp/sd0/journal_file1
/tmp/sd1/journal_file0  /tmp/sd1/journal_file1
/tmp/sd2/journal_file0  /tmp/sd2/journal_file1

6. Remove data object file to simulate WAL is finished
but data object file is not created.
$ rm /tmp/sd1/obj/007c2b2500000000

7. Start the three sheeps again. We found sd0 and sd2 is up, but sd1 is down.

8. By the program log (sheep.log), we can see the sheep process of sd1
is already aborted.

 INFO [main] md_add_disk(337) /tmp/sd1/obj, vdisk nr 261, total disk 1
 INFO [main] send_join_request(787) IPv4 ip:127.0.0.1 port:7001
 INFO [main] replay_journal_entry(159) /tmp/sd1/obj/007c2b2500000000, ...
ERROR [main] replay_journal_entry(166) open No such file or directory
EMERG [main] check_recover_journal_file(262)
    PANIC: recoverying from journal file (new) failed
EMERG [main] crash_handler(268) sheep exits unexpectedly (Aborted).
EMERG [main] sd_backtrace(833) sheep() [0x406157]
...

9. However, dog command shows the node is still in the cluster!

$ dog cluster info
Cluster status: running, auto-recovery enabled

Cluster created at Mon May  5 10:33:26 2014

Epoch Time           Version
2014-05-05 10:33:26      1 [127.0.0.1:7000, 127.0.0.1:7001, 127.0.0.1:7002]

$ dog node list
  Id   Host:Port         V-Nodes       Zone
   0   127.0.0.1:7000      	128          0
   1   127.0.0.1:7001      	128          1
   2   127.0.0.1:7002      	128          2

----------

Complement:

If we want to cleanup resources gracefully, more changes are needed.

1. assigned an exit code stand for exception, normally it is 1, is not 0.
2. adjust the cleanup steps, for example, pid file is created later than
   other initial steps. I think it should be cleaned up first.
3. adjust some initial steps.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheepfs/volume: remove set_nodelay in function setup_socket_pool

because connect_to() is called in the function, and it already calls
set_nodelay()

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add new option for configure

Add new configure option '--enable-diskvnodes' so we can
use sheep cluster by using disk as vnodes.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add disk information into sd_node

Add disk information into sd_node therefore it will be sended
to cluster shared region (corosync/zookeeper, etc.) and be
aware of by all nodes in this cluster.

The new size of sd_node is 592 bytes and the max size of zookeeper
is 983080, so the SD_MAX_NODES is 983080/(592 * 2) = 830.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: change method of generating vnodes

The sheepdog use node (include network ip and port) to generate
vnoes in hashing-ring at present. Every node generate about 128 vnods.
When comes a oid, sheep will find the corresponding nodes by searching
the vnodes and send oid to these nodes.

Imaging a disk has corrupted in a cluster using erasure-code, the node
will try to recovery all data in corrupted disk by fetching from other
nodes. This progress will take a very long time because it need to fetch
m copies from remote to re-generate (erasure code algorithm) 1 copy in
m:n erasure-code cluster, which will costs large mount of network traffic.

To solve this problem, we use disk to generate vnodes; therefore when a
disk is corrupted, all the nodes in cluster will join the recovery progress
and the progress will be much shorter.

In our test environment, the recovery time is 10 minutes when using old
vnodes algorithm, but it reduces to only 2 minutes by using disk to
generate vnodes. The test cluster has 10TB data.

The disadvantage of this disk mode is that unlimited number of local disks
has been narrowed down to DISK_MAX which is only 32 currently.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/md: change the method of generating vnodes in md

Because the method of generating vnodes in cluster is changed, the
method of generating vnodes in md should changed also.

After changing generate item from nodes to disks, adding new disk
or removing old disk should also cause whole cluster to do recovery.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: cache vnode_info when doing recovery

When sheepdog doing recovery in same low-performance machines, the CPU is
very high. After using perf tools to check the hot point of performance in
sheep daemon, we find out that the "alloc_vnode_info()" function cost lots
of CPU circyles because the rollback_vnode_info() rebuilds the vnode_info
by calling alloc_vnode_info() too frequently.

The solution is to cache result of alloc_vnode_info() for specific 'epoch'
in the recovery context.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog: add information of disks in cluster info

Add number of disks for every node in cluster info and
also show type of backend and vnode mode (disk or node)
when using '-v' option.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: change a prototype of process_main() for obtaining sender information
revive lock operation
use vdi_info in dog