remove extra newline character from fgets() result

This adds chomp() to remove an extra "\n".

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
recovery: initialize recovery_state

This prevents dog from reading uninitialize data and fixes a valgrind
error of functional tests.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
script: fix gen_bash_completion errors with the old perl version

The current script causes errors on debian squeeze.  This fixes it.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
read/write data from clients in worker threads

Currently, we read/write data from clients in the main thread in
the non-blockng way.  This has the following problems:

 - The implementation is much more complex than doing it in the
   blocking way.
 - The main thread is a bottleneck when doing a large data I/Os
   (e.g. object recovery does a lot of 4 MB reads and writes).

This patch creates one worker thread for each connection, makes socket
descriptors blocking, and reads/writes data from clients in the worker
threads in blocking way.

With this patch, client_handler() doesn't print warnings of the
loop_check tracer any more.  I did the following benchmark to emulate
a VM I/O while object recovery, and could see a lot of performance
improvement.

[benchmark]

  Simulate object recovery with 10 concurrent 4 MB reads, and simulate
  VM I/Os with 4 KB reads.

  $ ./script/vditest -C 10 -B 4M -T 10 test > /dev/null & \
    ./script/vditest -B 4k -T 10 test

[result (before)]

  options: -B 4096:4096 -c writethrough -C 1 -D 100:0 -o 0
           -p linear -s 1376684853 -S 0:4294967296 -T 10 -f 0
  Total read throughput: 638156.8B/s (623.2K/s), IOPS 155.8/s.

[result (after)]
  options: -B 4096:4096 -c writethrough -C 1 -D 100:0 -o 0
           -p linear -s 1376684811 -S 0:4294967296 -T 10 -f 0
  Total read throughput: 4237312.0B/s (4.0M/s), IOPS 1034.5/s.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog: add command to check and repair cluster

This patch adds a 'dog cluster check' command to check all the
objects' consistency.  While 'dog vdi check' is used when VM stops
unexpectedly because of the gateway failure, 'dog cluster check' is
expected to be used for fully checking of cluster status, e.g., after
total node failure happens or sheepdog hangs because of bugs.

With the current implementation, running 'dog cluster check' is same
as doing 'dog vdi check' against all the vdi images.  However, this is
usually very wasteful because vdis can share the same objects
especially when there are many snapshots.  I think of improving it to
skip redundant checks.

I'm planning to add other check commands later:

 - 'collie cluster check reference' : fix reference counts for object reclaim
 - 'collie cluster check epoch'     : fix a broken epoch file

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
make list iterator functions safe against removal of entry

Removing entries in non-safe list iterators causes severe bugs and
they are usually difficult to find.  This makes all the iterator
functions safe against removal of list entries in exchange for a very
small overhead.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
split list_head into list_head and list_node

Currently, we are abusing struct list_head for both list head and list
elements.  This patch introduces struct list_node and uses it for a
list element to make list operations easier to understand.  E.g. we
are using list_del_init() to remove a list element and list_empty() to
check the element is linked to a list.  With this patch, we use
list_del() to remove an element and list_linked() to check the element
is linked.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/functional: add _requirement() to filter testcases

We have several restrictions on which tests can be run.  E.g. 011
cannot be run without a root privilege, 042 cannot be run with -md
option, some cluster tests cannot be passed with shepherd, etc.

This patch generalizes _need_to_be_root() to _requirement() and
enables us to control which tests should be run more flexibly.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: use _requirements to control md tests

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: add support for client cache mode

This enables us to test the object cache feature.

E.g.

 $ sudo env SHEEP_EXTRA_OPTIONS="-w size=100" ./check
 $ sudo env SHEEP_EXTRA_OPTIONS="-w size=100" ./check -writethrough
 $ sudo env SHEEP_EXTRA_OPTIONS="-w size=100" ./check -direct

Note that writeback is default.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: add a script for throughout test

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>