sheep: sheep aborted during startup, still joined in cluster

Currently, create_cluster() is called before any thread is created.
Once any one of the following steps during startup is failed,
sheep calls exit() or abort() directly so that leave_cluster() is not called.
Other nodes would consider that that one should be alived.
This will cause many problems.

This is a reproducible case of using journal file. Hot fix also submit.
But, should we re-arrange the startup steps? Should we avoid panic()
because it is dangerous?

Steps:

1. Start three sheeps. Cluster manager is zookeeper.
for i in `seq 0 2`; do
    sheep/sheep /tmp/sd$i -y 127.0.0.1 -c zookeeper:127.0.0.1:2181 -z $i \
        -p 700$i -j size=64M -n
done

2. Format the cluster and create a vdi. Data object file 007c2b2500000000
is always written into sd1 according to sheepdog hash algorithm.
$ dog cluster format -c 1
$ dog vdi create test 4M -P

3. Write the vdi continuously.
for i in `seq 0 4194303`; do
    echo $i
    echo "a" | dog vdi write test $i 1
done

4. Kill all sheeps in another terminal during writing vdi.
$ killall sheep

5. Sometimes journal files are not cleaned up when sheeps exit.
If they are not found, try step 3 and step 4 again.
$ ls /tmp/sd*/journal*
/tmp/sd0/journal_file0  /tmp/sd0/journal_file1
/tmp/sd1/journal_file0  /tmp/sd1/journal_file1
/tmp/sd2/journal_file0  /tmp/sd2/journal_file1

6. Remove data object file to simulate WAL is finished
but data object file is not created.
$ rm /tmp/sd1/obj/007c2b2500000000

7. Start the three sheeps again. We found sd0 and sd2 is up, but sd1 is down.

8. By the program log (sheep.log), we can see the sheep process of sd1
is already aborted.

 INFO [main] md_add_disk(337) /tmp/sd1/obj, vdisk nr 261, total disk 1
 INFO [main] send_join_request(787) IPv4 ip:127.0.0.1 port:7001
 INFO [main] replay_journal_entry(159) /tmp/sd1/obj/007c2b2500000000, ...
ERROR [main] replay_journal_entry(166) open No such file or directory
EMERG [main] check_recover_journal_file(262)
    PANIC: recoverying from journal file (new) failed
EMERG [main] crash_handler(268) sheep exits unexpectedly (Aborted).
EMERG [main] sd_backtrace(833) sheep() [0x406157]
...

9. However, dog command shows the node is still in the cluster!

$ dog cluster info
Cluster status: running, auto-recovery enabled

Cluster created at Mon May  5 10:33:26 2014

Epoch Time           Version
2014-05-05 10:33:26      1 [127.0.0.1:7000, 127.0.0.1:7001, 127.0.0.1:7002]

$ dog node list
  Id   Host:Port         V-Nodes       Zone
   0   127.0.0.1:7000      	128          0
   1   127.0.0.1:7001      	128          1
   2   127.0.0.1:7002      	128          2

----------

Complement:

If we want to cleanup resources gracefully, more changes are needed.

1. assigned an exit code stand for exception, normally it is 1, is not 0.
2. adjust the cleanup steps, for example, pid file is created later than
   other initial steps. I think it should be cleaned up first.
3. adjust some initial steps.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheepfs/volume: remove set_nodelay in function setup_socket_pool

because connect_to() is called in the function, and it already calls
set_nodelay()

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add new option for configure

Add new configure option '--enable-diskvnodes' so we can
use sheep cluster by using disk as vnodes.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add disk information into sd_node

Add disk information into sd_node therefore it will be sended
to cluster shared region (corosync/zookeeper, etc.) and be
aware of by all nodes in this cluster.

The new size of sd_node is 592 bytes and the max size of zookeeper
is 983080, so the SD_MAX_NODES is 983080/(592 * 2) = 830.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: change method of generating vnodes

The sheepdog use node (include network ip and port) to generate
vnoes in hashing-ring at present. Every node generate about 128 vnods.
When comes a oid, sheep will find the corresponding nodes by searching
the vnodes and send oid to these nodes.

Imaging a disk has corrupted in a cluster using erasure-code, the node
will try to recovery all data in corrupted disk by fetching from other
nodes. This progress will take a very long time because it need to fetch
m copies from remote to re-generate (erasure code algorithm) 1 copy in
m:n erasure-code cluster, which will costs large mount of network traffic.

To solve this problem, we use disk to generate vnodes; therefore when a
disk is corrupted, all the nodes in cluster will join the recovery progress
and the progress will be much shorter.

In our test environment, the recovery time is 10 minutes when using old
vnodes algorithm, but it reduces to only 2 minutes by using disk to
generate vnodes. The test cluster has 10TB data.

The disadvantage of this disk mode is that unlimited number of local disks
has been narrowed down to DISK_MAX which is only 32 currently.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/md: change the method of generating vnodes in md

Because the method of generating vnodes in cluster is changed, the
method of generating vnodes in md should changed also.

After changing generate item from nodes to disks, adding new disk
or removing old disk should also cause whole cluster to do recovery.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: cache vnode_info when doing recovery

When sheepdog doing recovery in same low-performance machines, the CPU is
very high. After using perf tools to check the hot point of performance in
sheep daemon, we find out that the "alloc_vnode_info()" function cost lots
of CPU circyles because the rollback_vnode_info() rebuilds the vnode_info
by calling alloc_vnode_info() too frequently.

The solution is to cache result of alloc_vnode_info() for specific 'epoch'
in the recovery context.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog: add information of disks in cluster info

Add number of disks for every node in cluster info and
also show type of backend and vnode mode (disk or node)
when using '-v' option.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
change the redundancy level of existing cluster and vdi

This is a new feature to allow user to change the redundancy level of
formatted cluster and created vdi. Only simple replica is supported now.

1. To set the replica number of formatted cluster:

    dog alter cluster-copy -c <copies>

2. To set the replica number of standalone vdi who has neither
   parent nor children:

    dog alter vdi-copy -c <copies> <vdiname>

3. To set the replica number of shared vdi who has parent or children,
   to avoid shared data inconsistency, please run dog vdi clone command
   with -n (--no-share) option first:

    dog vdi clone -s <snapshot> -n <src vdi> <dst vdi>

   It will deep copy a brand new standalone vdi base on source vdi
   so that we can run dog alter vdi-copy to change destinate vdi's
   replica number later.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog/vdi: stop preallocation for no share option in vdi cloning

Previous patch add a -n (--no-share) option in dog vdi clone.
Preallocation is also enabled by this option. However, it is
unnecessary to preallocate data object. In fact, we only need to
copy parent's data objects into its own.

This patch fix this problem to carry out thin-provision.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: pad hole of opcodes

Cc: Ruoyu <liangry@ucweb.com>
Singed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
dog: judge vdi is standalone or not by vdi family graph

SD_MAX_CHILDREN is removed in the future because the new GC algorithm
resolves the limitation of a maximum number of children. So "dog alter
copy-vdi" command shouldn't depend on it. This patch lets the command
judge VDI is standalone or not by vdi family graph like "dog vdi
tree".

Cc: Ruoyu <liangry@ucweb.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: introduce generational reference counting for object reclaim

Generational reference counting is an algorithm to reclaim data
efficiently without race conditions on distributed system.  This
extends vdi objects structure to store generational reference counts,
and increments the counts when creating snapshots.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: introduce ledger objects

This introduces ledger objects, which keeps track of the number of
outstanding references of each generation.  Sheep decrements a
generational reference count with a gateway request SD_OP_DECREF_OBJ,
and reclaims objects when there is no generational reference.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: decrement generational reference count on vdi deletion

This removes old vdi deletion code, which reclaims objects only when
all relevant snapshots are deleted, and uses a generational reference
counting algorithm.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: decrement generational reference count on copy-on-write

This decrements a reference count of the old data object when
allocating a new data object on CoW.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: remove max children limit

We no longer need it.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: introduce sparse objects

We don't need to care about a fragmentation when
 - the object is unlikely to be accessed sequentially, and
 - the object is read-only.

In that sense, we can make the objects sparse if they are not data
objects and writable ones.

This fixes the problem that sheepdog consumes many disk spaces for
deleted vdi objects if your filesystem supports FALLOC_FL_PUNCH_HOLE.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheepdog proto: recycle old child_vdi_id for storing btree_counter

Now the area isn't used and clients (QEMU and tgt) don't care about
it, we can recycle it for storing the btree_counter.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: get_request() for deferred response

With the new function, response message to clients can be deferred
safely without blocking the main thread.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>