tests/functional: add .gitignore

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
codestyle: fixed code style errors in include/bitops.h

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
zookeeper: remove unneccessary zk operation

Current implementation of zk_handle_unblock() sends an NOTIFY event to
all sheep. All sheep who receive NOTIFY event will call
sd_notify_handler() to do unblock.
However, this is unncessary, we can call sd_notify_handler() directly
in zk_handle_unblock().
And this how corosync driver handle unblock.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: handle error of notify_vdi_add()

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
zookeeper: clean up sd_iprintf and sd_eprintf

These information aren't useful for users.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
cluster/local: fix spurious failure 065

PROCESS_CHECK_INTERVAL is too large on my box that

_kill_sheep 0 # kill the sheep nicely but local driver failed to process event
_start_sheep 0

will be actually executed similar to

_start_sheep 0
_kill_sheep 0

Set it to 50 ms solve the failure on my box.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: clean up #include headers

Current #include are messy duplicating #include everythere, since sheep_priv.h
already have most of system #include, it is convenient to have most common
system includes in it. Then source files in sheep/ will mostly need only
include sheep_priv.h.

Rare header files like <fcgiapp.h> aren't included in sheep_priv.h.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix a warning of shadowing global declaration

ops.c:402:20: warning: declaration of ‘log’ shadows a global declaration [-Wshadow]

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
minor: modified tests/functional/.gitignore

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
.gitignore: add tools/kill_zk_session to .gitignore

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: handle block/unblock/notify error

In group.c, it uses 3 broadcast operations: block, unblock and notify.
These broadcast operations are implemented by cluster drivers.
For example, corosync implements it by cpg_mcast_joined() while zookeeper by
sequential node.
And they can fail if network is unavailable for a while.

However, current group.c doesn't handle errors of block/unblock/notify events
and just ignore them.

This patch add a new error SD_RES_CLUSTER_ERROR to indicate these errors.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
cluster driver: handle pending block/notify event during reconnect

Current implementation of reconnection doesn't handle pending block/notify
event.

It is easy to handle notify event by calling sd_notify_handler() manually
to finish it.

However, it is a little bit complex for block event.
This is because a block event need 4 steps.
1. in queue_cluster_request(), send block event by sys->cdrv->block(), and
  add to pending_block_list.
2. in sd_block_handler(), queue the event to work queue of 'block' thread.
3. in cluster_op_done(), send unblock event by sys->cdrv->unblock().
4. in sd_notify_handler(), remove it from pending_block_list.

And step 1 and 3 contains broadcast operations.
So we have to know which step has been done for a pending block event.

If step 1 has been done, we can re-queue it simply. (Any block event which sent
by this node have been removed due to the leave event)
If step 2 has been done, the event is handling by another thread. We have to mark
it as 'drop' so that it will be dropped when cluster_op_done() is called later.
if step 3 has been done, we should call sd_notify_handler() manually to finish
it.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests: add a DynamoRIO client for testing the jounaling mechanism

This patch adds a DynamoRIO (often called DR) client for testing the
jounaling mechanism. Because of its nature, the recoverying path is
the most important and hard to test part of the journaling
mechanism. They need to be tested well.

But testing targetted recovery paths with traditional tests/ stuff is
hard because:
1. killing sheeps with kill commands doesn't take into account the
   internal state
2. inserting exit()s into sheep manually is a painful work

So this patch implements a fault injection mechanism with DR. DR
provides rich functionalities of transparent dynamic
instrumentation. One of the functionalities makes inserting function
calls before and after system calls possible. With this mechanism, the
fault injection mechanism lets sheep exit at suitable timings for
testing recovery paths of the journaling.

How to use:
0. preparation
   $ cd
   $ svn checkout http://dynamorio.googlecode.com/svn/trunk/ dynamorio
   $ cd dynamorio
   $ mkdir build
   $ cd build
   $ cmake ..
   $ make

(This patch assumes the source code of DR is store in $HOME/dynamorio,
and the build is done in $HOME/dynamorio/build)

1. build the DR client
   $ cd tests/dynamorio/journaling/
   $ cmake .
   $ make

2. run tests with preset scenarios
   $ ./01.sh 	  # for testing recovery of object store
     		  # after this, actual completion of recovery can be
		  #  checked via sheep.log

The fault injection implemented with this patch is so slack and not
capable for exhaustive testing. This is only a supoprted care. But I
believe it is useful.

With this patch, I tested the recovery path for object store and
checked it work well.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
util: solve cyclic dependency of util.h and logger.h

Current sheepdog specialized assert() in util.h cannot be used by
util.h itself because it depends on logger.h and logger.h depends on
util.h, too.

This patch solves the cyclic dependency by introducing new header
file compiler.h. logger.h doesn't require whole util.h so, the
required part is extracted as compiler.h.

In addition, this patch modifies lib/strbuf.c because it is a assert()
user but it didn't include util.h.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
util: introduce a new data type refcnt_t for reference counting

Using raw int for reference counting directly is dangerous because it
cannot detect ++ or -- for these counters at compilation time. So this
patch implements a new data type refcnt_t for safe reference counting
which can be done by multiple threads in parallel.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: use refcnt_t in various data structures

This patch lets various data structures employ refcnt_t for its
reference counting of cache entries. Changed structs are: vnode_info,
client_info, request, and object_cache_entry.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: initialize reference count of local request

The patches in this series detected a bug in reference counting of
request. alloc_local_request() doesn't initialize a reference count of
newly allocated request. This patch fixes this problem.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: rework update node logic

The new logic is
1. size-varied node call cdrv->update_node() to send a non-blocking message to
   all nodes
2. all nodes in the cluster
 - update cluster driver's internal state of sender's size if any
 - cluster driver call sd_update_node_handler() to update sender's size in sheep

SD_OP_UPDATE_SIZE is no longer used and removed.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
cluster/local: add add_event_lock helpe

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
cluster/corosync: rename .ent as .node

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>