sheep: remove last slash for new->path in md_add_disk()

Follow these steps:

  1. sheep /meta/,/data1/,/data2/...... (add slash before data path)
  2. upload data
  3. remove /data1/*
  4. get data

then the sheep daemon will go to dead loops and print logs like:

  ERROR [io 1234] err_to_sderr(96) /data1/ corrupted
  ERROR [io 1234] err_to_sderr(96) /disk5/ corrupted
  ...

The reason is when the sheep daemon start, it use "/data1/" as disk->path, but
when it occur a error-io, sheep will call md_remove_disk() with argument disk which
contain path of "/data1" (without slash), so sheep can not actually unplug "/data1/"
in md ring and it will try to forward request forever.

We can fix this by remove last slash when sheep call md_add_disk().

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Meng An <isolate000@gmail.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog/farm: fix closing negative fd

When executing dog cluster snapshot command, valgrind report
"Warning: invalid file descriptor -1 in syscall close()".

This patch fix the problem.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/recovery: reduce network overhead of preparing object list

In the stage of preparing object list, every node tries to fetch
entire object lists from all nodes of the cluster, and then,
screens out objects that don't belong to it's own. This behavior
transfers quite a lot of unnecessary object id over the network.

This patch resolve the issue by:
1. receiver node prepare objects that belong to the sender node.
2. sender node don't screen out objects any longer because it is
   already done by the receiver, but merging object lists fetched
   from different nodes is essential.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
test: add test case for data path with slash

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
test: modify _cleanup_devices() to umount loop device by correct order

When using _make_device like:

 _make_device $STORE/0/
 _make_device $STORE/0/d1
 _make_device $STORE/0/d2

the _cleanup_devices() will umount '$STORE/0/' first and then '$STORE/0/d1',
which will cause error message like:

 umount: /tmp/sheepdog/095/0/d1: not found
 umount: /tmp/sheepdog/095/1/d0: not found
 umount: /tmp/sheepdog/095/1/d1: not found
 umount: /tmp/sheepdog/095/2/d0: not found
 umount: /tmp/sheepdog/095/2/d1: not found

So we have to modify _cleanup_devices() to umount long path first.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/func: patch test cases since recently update

As titled.
Test case 028 and 035 are correlated with dog vdi object location.
Test case 030 is correlated with dog cluster info -v.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/ops: fix incorrect error log on remove_epoch

If epoch file is not existed due to the node had left, cluster format
command will cause the error in sheep.log

ERROR [main] remove_epoch(246) failed to remove
/tmp/sd9/epoch/00000002: Operation not permitted

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/md: fix guest file system read-only error

If using md, it is easy to re-produce the error when sheepdog is
doing recovery. The error message is just like as below.

ERROR [io 30895] md_move_object(630) failed to create
                 /mnt/sheepdog/obj2/0082aba600000056
ERROR [io 30895] md_check_and_move(661) move old
                 /mnt/sheepdog/obj/0082aba600000056 to new
                 /mnt/sheepdog/obj2/0082aba600000056 failed

If the above target object is being written unfortunately by other
thread at that time, atomic_create_and_write return -1, and then,
md_exist return false so that guest file system catch it.

Therefore, I think we should check errno if atomic_create_and_write
is failure. If the temp file exist, in another word, someone is
writting it, just return success to avoid the error.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/request: straightforward error message in sheep_exec_req

Current message as below in sheep_exec_req is rather confused.

ERROR [rw 6471] sheep_exec_req(1133) failed No object found

In fact, I don't know which node sends the request and what it
wants to do. They are the minimum requirements for any investigation.

I think this is very helpful to diagnose an issue.

ERROR [rw 16160] sheep_exec_req(1135) failed No object found,
remote address: 127.0.0.1:7002, op name: GET_HASH

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
unittest: fix coredump bug for unit-test

Test case in test_hash.c will call create_vdisks() which need to
check sys->cinfo.flags, but variable 'sys' is uninitialized in
unit-test environment. Therefore we need to init 'sys' before
calling create_vdisks().

Reported-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/md: temp file cannot be removed

d_name is only the file's name without absolute path. To unlink it
successfully, we must give the full name.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
update MAINTAINERS

I'm not working for NTT lab now.

Also, let's add Hitoshi to the master branch maintainers.  Unfortunately, I
have no time for sheepdog development currently.  He is the maintainer of the
stable branch and will do a good job instead of me.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@gmail.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
lib/event: fix and unify error message

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep/request: fix potential incorrect cluster info refcnt

For fear that req->data is not allocated successfully, move
increament of client info refcnt after memory alloc is reasonable.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Revert "sheep/recovery: reduce network overhead of preparing object list"

This reverts commit f56aceee959baa1127de5e0babbacb9e8971ceaa.

Quote from Ruoyu

==================================================================
Although the probability is rather low, the patch will cause the vm
file system crashed.

Suppose that there are 3 nodes in a cluster: A, B and C. A is fast and
B is slow. Now C is left accidently. Let us focus on node B, if
GET_OBJ_LIST request of A is received prior to the zookeeper
EVENT_LEAVE message, B will return object list on the old epoch to A.
However, what node A want to get is the object list on the new epoch.
Therefore, some objects cannot be recovered. That is the problem.

To simplify the scenario, I think it is better to use the old
algorithm, getting all object list first, and then, checking which
objects are belong to the node itself.
dog: add elapsed time to measure the program is fast or not

To be compatible with OpenStack glance and cinder driver, we cannot
show elapsed time by default. Therefore, the new option -t (--time)
is added in dog command line to achieve it. For example:

dog cluster info -t
dog vdi check test -t

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: fix sheep sometimes cannot be killed successfully

I am sure this is a bug because the variable nr_outstanding_reqs
does not reset to zero sometimes after client cancelling the
request. Once nr_outstanding_reqs is not zero, sheep process
never be killed successfully, neither using kill <pid> nor
using dog node kill command.

But I am not sure whether the bug is fixed perfectly because
I am not familiar with the sheepdog networking logic. I have to
add some error messages to every doutful statements.

The result is, I caught one of them. So, I call the function
clear_client_info in that place. It seems every thing is fine
after the modification.

Does anyone help to investigate and fix it?

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
add Robin as a subsystem maintainer of HTTP interface

Robin introduced big contributions (e.g. hypervolume) to sheepdog and
is a heavy user of the feature of HTTP interface for object storage at
Taobao, inc. So this patch adds him to MAINTAINERS as a subsystem
maintainer of HTTP interface (sheep/http/*).

Of course, current HTTP interface subsystem isn't separated cleanly
from the core components of sheepdog. Changes related to the subsystem
will affect the core part often. But I want to increase productivity
of the community and it will be helpful if Robin can be a maintainer
of the subsystem even his territory is limited. I'd like to pull
changes of HTTP interface from Robin's repository directly.

Cc: Robin Dong <robin.k.dong@gmail.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
dog/vdi: add hexadecimal support for index option

It is convenient if the user already knows a oid, no need to
convert it to decimal index first.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
configure.ac: fix building failed in jenkins

Because clock_gettime was introduced, we should check it first.
Otherwise, compilation may be failed in some Linux distribution.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>