sheep: make invalid_vnode() type safe

Comparing sd_node and sd_vnode with node_id_cmp should be avoided.  It
is because node_id must be the first field of both structures but we
don't have any checks for it.

This also fixes a compile error in debug build.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
minor: fixed indention problem in sheep/journal.c

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/functional: use kill_zk_session to _kill_sheep with zk

- fix kill_zk_session.c that if ZNONODE returns, we simply return success

In this way we don't blind sleep for _kill_sheep with zk driver. This fixes
some false failure of tests with SSD drive and also boost tests a little bit

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: don't check of the existing object before slice_write

Currently, we stores sliced objects for the farm store, so
sha1_file_exist() fails always.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
util: add find_zero_blocks()

This adds a helper function which does the similar thing as
trim_zero_blocks() but doesn't update the given buffer.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
add sha1_from_buffer()

This adds a helper function to calculate a unique sha1 digest based on
the given buffer.

This also fixes a bug that we don't use an original buffer length to
calculate the hash value and the result is not unique.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tool: rename kill_zk_session as zk_control

This is prepared to add more methods

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
zk_control: refactor the code

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
zk_control: add remove method

With remove method, we can remove nasty 5s sleep in the _kill_sheep(), will
boost tests a lot with zk.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: remove xattr code in sha1.c

refcount of sha1 file for deletion makes use of xattr, but currently we don't
support snapshot deletion yet.

This patch remove xattr code completely because:
- deletion snapshots can be exercised by
  1. export the snapshots you want to keep to another snapshot storage
  2. delete the old snapshot storage.

- Assume storage for storing snapshots to support xattr is very lame, e.g., we
  we can't rely on Amazon's S3 storage to support kind of xattr.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: remove unused code in sha1.c

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: remove unnecessary _wait_for_sheep_stop

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: move zk node remove in _wait_for_sheep_stop

Then both _kill_sheep and _kill_all_sheep will make use of it

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: use _wait_for_sheep to make sure that sheep starts

This fixes a false alarm of zk tests on my environment.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: implement a correct detection of object cache pushing completion

Current implementation for pushing object cache would have a bug. In
object_cache_push(), the variable oc->push_count is incremented in the
loop of queuing work. This can produce a non deterministic bug when
a sequence of scheduling like below happens:

1. object_cache_push() creates a work and queue it to
   sys->oc_push_wqueue (oc->push_count == 1)
2. Before queuing next work, a worker thread of oc_push_wqueue wakes
   up and pushes the object
3. After completion of pushing, the worker thread decrements
   oc->push_count atomically. The thread calls eventfd_write() because
   oc->push_count is 0.
4. The thread which calls object_cache_push() detects an invalid
   completion after its queuing iteration.

This sequence can produce an invalid early finish of flush request
from VMs, especially when a number of cached object is large.

Cc: Luca Lazzeroni <luca@gvnet.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: remove master node

The current procedure to handle sheep join is as follows.

 1. The joining node sends a join request.
 2. The master node accepts the request.
 3. All the nodes update cluster members.

This procedure has some problems:

 - The master election is too complex to maintain.
   It is very difficult to make sure that the implementation is
   correct.

 - The master node can fail while it is accepting the joining node.
   The newly elected master has to take over the process, but it's
   usually difficult to implement because we have to know what the
   previous master did and what it did not before its failure.

This patch changes the sheep join procedure to the following.

 1. The joining node sends a join request.
 2. Some of the existing nodes accept the request.
 3. All the nodes update cluster members.

It is allowed for the multiple nodes to call sd_join_handler() against
the same join request, but at least one node must have to do it.  With
this change, we can eliminate a master, and node failure while
accepting node join is also allowed.

Removing a master from zookeeper is not easy since it doesn't expect
that multiple nodes send EVENT_ACCEPT.  I'll leave this for another
day.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
recovery: wait until getting the previous epoch's vdi bitmap

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: optimize getting vdi bitmap

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
zookeeper: allow non-master nodes to call sd_join_handler

Master election is still necessary to handle concurrent join when
sheepdog starts up.  I wrote it as a TODO in the source code.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: wait_for_sheep_recovery outside of simulate_sheep_..

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>