sheep: remove master node

The current procedure to handle sheep join is as follows.

 1. The joining node sends a join request.
 2. The master node accepts the request.
 3. All the nodes update cluster members.

This procedure has some problems:

 - The master election is too complex to maintain.
   It is very difficult to make sure that the implementation is
   correct.

 - The master node can fail while it is accepting the joining node.
   The newly elected master has to take over the process, but it's
   usually difficult to implement because we have to know what the
   previous master did and what it did not before its failure.

This patch changes the sheep join procedure to the following.

 1. The joining node sends a join request.
 2. Some of the existing nodes accept the request.
 3. All the nodes update cluster members.

It is allowed for the multiple nodes to call sd_join_handler() against
the same join request, but at least one node must have to do it.  With
this change, we can eliminate a master, and node failure while
accepting node join is also allowed.

Removing a master from zookeeper is not easy since it doesn't expect
that multiple nodes send EVENT_ACCEPT.  I'll leave this for another
day.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
checkpath: fix block comment false alarm

+* blabla.
+* blabla.
+*/

Above in the patch file is valid block comments

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
lib: move sockfd cache from sheep to lib

On some subcommands, collie also issues lots of request to sheeps. So
collie can enjoy sockfd caching. For this purpose, this patch moves
sockfd from sheep to lib and generalize the interfaces of sockfd.

This patch doesn't change anything related to IO NIC in sockfd
cache. Because collie can know address and port of IO NIC so
potentially use them.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: check an error during issuing request correctly

Current collie checks error during issuing request by a return value
of collie_exec_req(). But it doesn't work well because
collie_exec_req() returns result code of the request. The code doesn't
represent that the request itself succeeded or not.

This patch lets collie_exec_req() return 0 or -1 for checking the
error which is caused during request issue. And also fixes invalid
error checking done in node_recovery().

In addtion, the last return statement of collie_exec_req() is changed
like this: return ret ? -1 : 0; This ternay operator is required
because exec_req() returns 1 when it fails. Some part of collie check
an error of collie_exec_req() by if (ret < ), and other part check by
if (ret). The above ternay operator is for avoiding return 1 and let
these error handlers work correct.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
lib: correct a way of freeing sockfd_cache_entry

Previous implementation of the sockfd caching mechanism has two memory
leaks. The leak is caused when sockfd_cache_destroy(),
sockfd_cache_add_nolock() try to free a sockfd cache entry. They
forget to free entry->fds.

This patch adds a new function, free_cache_entry() for safe freeing of
sockfd cache entries. And let the above two functions to use it.

Also this patch lets sockfd_cache_add() use free_cache_entry() for
consistency.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: let collie use sockfd cache

This patch lets collie use sockfd cache. Because the sockfd caching
mechanism is now in libsheepdog, collie can enjoy it on some
subcommands which issue many requests to sheep.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: use gcc builtin to get stack frame pointer

This remove hardcoded assembly which is not portable.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: retire sheep/sockfd_cache.c

Now the bulk of its sockfd cache code is in the library, so no need to keep
a dedicated file for it.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
collie: use xread and xwrite instead of read and write

The following code in vdi_read() is wrong

   write(STDOUT_FILENO, buf + (len - remain), len)

The number of bytes to be written should be 'remain'.

Rather than fixing it, this patch replaces them with xread() and
xwrite(), which don't return a short length on success.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
introduce eventfd_xread and eventfd_xwrite

Currently, we don't check the return value of eventfd_read and
eventfd_wriete in many places.  This introduces a helper function
which retries on EINTR and calls panic() on unrecoverable errors.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
introduce intcmp to simplify codes

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
debian: add debian/po

This file was dropped when I imported the debian directory, but looks
like a necessary file.  Without this patch, installing the created deb
package will fail.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: count recovered objects with uint64_t

In theory, a number of recovered objects (or their candidate) would be
larger than 2 ^ 32. This patch lets the recovery process use uint64_t
for counting the numbers.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: enhance STAT_RECOVERY for prividing information of recovery state

Current SD_OP_STAT_RECOVERY only provides an information that
indicates the node is doing recovery or not. For providing more useful
information about progress of recovery, this patch enhances this
request.

The enhanced SD_OP_STAT_RECOVERY returns the information with struct
recovery_state. The struct contains these information:
1. a flag which indicates a sheep process is in progress or not
2. state of recovery. With this information, collie can know which
   phase of progress sheep is in.
3. Number of copied object during a recovery process.
4. Number of entire objects which should be copied during a recovery
   process.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: add a new option --progress to "node recovery" for showing recovery progress

This patch adds a new option --progress (or -P) to the node recovery
subcommand. With this subcommand, users can show a progress of
recovery process.

Example:
$ sudo collie node recovery --progress
 99.7 % [==============================================>] 7047 / 7068

The denominator (7068 in the above case) indicates a number of entire
object which should be checked. The numerator (7047 in the above case)
indicates a number of objects which is already checked or copied.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
trivial: let git ignore .deb files

Generated .deb files should be ignored by git.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: push dirty bits correctly for successive FLUSH req

This will fix a bug that 'qemu-img convert' fail to convert images intact with
default unsafe mode.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: add test for reading/writing partially zeroed data

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
util: fix calculation of offset and length in find_zero_blocks()

This fixes bugs introduced by the below commits:

 - cfd1a566: the length of a fraction at the beginning of buffer was
             wrong.

 - 6169a0da: this commit removes memmove() from find_zero_blocks(), so
             offset calculation should have been changed.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/unit: fixed unittest error

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>