Refactor zookeeper driver

This patch tries to refactor zookeeper driver.

It have changed so much code that it's difficult to
explain how it work in detail.

There were several problems in the old zookeeper driver:
 1. Each zookeeper message contains whole member list,
    it make network become heavy and slow.
 2. Before sending each message, get_nodes function was call.
    We used this function to get the number of members by comparing
    each znode in /sheepdog/queue. This function will become
    more and more slower as znodes increasing in /sheepdog/queue.
 3. The processing method of LEAVE EVENT is too lengthy. When a sheep
    droped from the cluster, zookeeper will remove the corresponding
    znodes in /sheepdog/member automatically, and each sheep will be
    notified by the registered watcher fcuntion. Normally, sheep should
    process this leave event when it got it, but it didn't, it created
    a new leave event message and send it to /sheepdog/queue, untill all
    sheep received this new message again.
 4. There are so many unnecessary zookeeper locks. These locks greatly
    reduce zookeeper driver's performance.

This patch tries to fix all problems memtioned above:
 1. Reduce message size, remove member list from message content
    and save it in two places:
    a) In /sheepdog/member. When a sheep joins into cluster, it put it's
       information into /sheepdog/member/<node_to_str(myself)>, and get
       other sheep's information from there.
    b) In eache sheep's private memory. Each sheep will maintain an array
       to contain the newest member list so that it can access this content
       as much as quickly, and needn't to call get_nodes function when it send
       a new message.
 2. Optimize the processing method of LEAVE EVENT. Now, sheep will process
    leave event once it received it.
 3. Remove unncecessary zookeeper locks.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Optimize the size of buffer send to zookeeper

We send the whole buf[MAX_EVENT_BUF_SIZE] to zookeeper regardless the
real content is so small.

Now, we only send the real content in the buffer, not waste anymore.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
retry again when zoo_* api return ZCONNECTIONLOSS/ZOPERATIONTIMEOUT error

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Fix two bug:

1. Change pthread lock in zk_dispatch/zk_block so that
it will not block IO request between sheep.
2. If one node pointed to by queue_pos was send by leaver,
and it have blocked whole cluster, we should ignore it.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Use atomic builtins to replace pthread mutex locks

There were two pthread mutex locks in zookeeper dirver:
1. pthread mutex lock in zk_dispatch/zk_lock, this lock was used to
prevent zk_lock poping data before zk_dispatch have pushed it.
2. pthread mutex lock in zk_queue_pop/add_event, this lock was used to
protected leave event list.

This patch use GNU atomic builtins(__sync_xxx) to replace these two
pthread mutex locks. It makes zookeeper driver more lighter and
and faster.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Rewatch znode in /sheepdog/member after it changed

If we not rewatch znode in /sheepdog/member after it changed, we
will lose all events of it in the future, such as DELETE event.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
If previous zookeeper session exists, shutdown sheep

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Add code to handle sequence number overflow

The range of sequence number in zookeeper is: [-2^31 ~ (2^31-1)],
We store it in queue_pos variable, and use queue_pos++ to refer
next znode in /sheepdog/queue.

When sheep creates the first znode in /sheepdog/queue, zookeeper
initials the sequence number with 0. Each time we create a new
znode, it's value plus 1. So it will overflow and become a negative
in the future. In fact, it's value just like a circle.

This patch try to handle this problem.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Fix bug: zk_leave doesn't work

Now we just need to delete znode in /sheepdog/member,
and zookeeper will notify LEAVE event to all other
sheeps automatically.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Fix bug: leave event lost in zookeeper driver

If one sheep joined and left too quickly,
it may lead to leave event lost.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Remove zk_lock from zk_join

Use ip address and port number to sort member list instead of sequence
number from /sheepdog/queue.

Benefit from these changes, the order of member list no longer depends
on the order of joining, then need not to take following steps into
one transaction:
- get sequence number from /sheepdog/queue
- create znode in /sheepdog/member/
- send join message to cluster

As a result, we can remove lock from zk_join, and move the creation of
znode in /sheepdog/member/ into zk_dispatch.

I use binary tree to store member list so that it can make the sort more
quickly. Methods with node_btree_xxx prefix are used to operate this
binary tree.

==*Note*==:
When sheep startups, it fetchs member list by reading /sheepdog/member/*,
if the result is empty, sheep will consider itself as *master*.

Now we have removed lock from zk_join，if we start multiple sheeps
simultaneously, one problem arises: there may exist more than one *master*,
this is bad.

To prevent this problem, we can start multiple sheeps like this:
- start the fist sheep alone, and sleep 2 seconds:
  $ sheep -d /store/0 -z 0 -p 7000 -c zookeeper:localhost:2181
  $ sleep 2

- start other sheeps simultaneously(need not to sleep between them):
  $ for i in {1..100}; do sheep -d /store/$i -z $i -p $((7000 + $i)) \
   -c zookeeper:localhost:2181

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix get_nr_copies()

We should never return anything greater than sys->nr_copies.

- use min() to get the better readability
- change sys->nr_copies as 'int' to stay in line with nr_zones

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Make connection information more readable

Add ipstr and port in connection structure, and fill these values
in create_client(). We can dprintf them when we need to show us where
the client connection comes from.

To help us debuging, I dprintf these info in client_rx_handler() and
destroy_client().

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
logger: fix pid confusion

We need to store the logger (child) pid in a variable as getppid() fails
inside the death handler.  Instead of going back to using the pid variable
like it was done before "logger: improve death messages" use two different
variable for the sheep and logger, and comment on why we perform this
confusing dance, so that it doesn't get accidentally removed again.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: refactor is_access_local()

- let is_access_local to calculate copies itself
- fix a nasty bug in void io_op_done(), we should not pass 0
  or leave_cluster() will never be called.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
farm: clean up farm_end_recover()

- print what the epoch we reach instead of 'iocb->epoch - 1'
- use epoch instead of 'iocb->epoch - 1'

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
farm: refactor trunk_file_write()

Since we already have a dedicated funciton trunk_file_write_recovery()
for system recovery handling, we don't have 'user' flag for
trunk_file_write() and simply rename it as trunk_file_write_user()

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
object_cache: fix a race condition

create_cache_object() need to get a lock to avoid race with
object_cache_rw().

This is to address below bug in the log:
Apr 28 17:48:12 do_io_request(931) 2, 47c33f00003f80 , 1
Apr 28 17:48:12 object_cache_rw(321) 00003f80, len 4096, off 1396736
Apr 28 17:48:12 client_rx_handler(427) HEAD, fd:280
Apr 28 17:48:12 client_rx_handler(432) DATA_INIT, fd:280
Apr 28 17:48:12 client_rx_handler(470) END, fd:280
Apr 28 17:48:12 queue_request(275) 2
Apr 28 17:48:12 do_io_request(931) 2, b1f0b000003f9b , 1
Apr 28 17:48:12 object_cache_pull(423) [local] 00003f9b
Apr 28 17:48:12 object_cache_pull(423) [local] 00003f9b
Apr 28 17:48:12 create_cache_object(352) 00003f9b already created
Apr 28 17:48:12 object_cache_rw(321) 00003f9b, len 16384, off 2940928
Apr 28 17:48:12 read_cache_object(307) size =0; count= 16384
Apr 28 17:48:12 do_io_request(956) failed: 2, b1f0b000003f9b , 1, 3
Apr 28 17:48:12 io_op_done(151) leaving sheepdog cluster

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: move read/write/remove_object() into store.c

- thus we can make read/write_object_local() as local static
  functions.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: move store_* operations into ops.c

- move stat_sheep() into ops.c too

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>