improve type check of xbsearch

It is possible that 'key' is a const variable and 'base' is not, or
vice versa.  This patches allows such cases.

This also makes xbsearch() return the type of 'base' instead of a void
pointer.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
farm: do cluster-wide snapshot by work queue

Currently, loading snapshot uses dynamic queue where saving snapshot uses ordered queue.
This is because there is a risk that parallelized saving snapshot could slow down cluster's performance.
We can easily parallelize this by using dynamic queue.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
farm: remove sha1_hdr

This is inspired by speed up saving snapshot.

We can speed up saving snapshot by reading sha1 value firstly.
Because 'farm' saves data with file name of sha1 value, so it is easy to check wether an object with the sha1 value already exists in local path.

However, the currently algorithm of computing sha1 is different between 'sheep' and 'farm' in two ways:
1. farm save data with a sha1_hdr, but sheep don't
2. sheep trims data before calculate sha1

This patch is to remove sha1_hdr from sha1 file.

There are 2 other reasons that support this idea:
1. Currently, sha1_hdr contains a tag which is to distinguish file type of data, trunk and snap.
   However, this is not neccessary. We can find snap by snap_log, and trunk by snap.
   We don't need to find a type of file by reading header.
2. Different type of data should not share the same hdr which has different meaning in different places.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
farm: trim data before calculating sha1

sha1.c trims data before calculating sha1.
So that collie and sheep use the same algorithm to calculate sha1 value.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
farm: add sha1_file_exist() to sha1_file.c

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: add sd_read_object_sha1() to common.c

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
farm: avoid of reading object which already exists

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
build: add collie/farm/farm.h as noinst_HEADERS

Without this, 'make dist' would miss collie/farm/farm.h file.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: correct help message for 'cluster snapshot'

Signed-off-by: Valerio Pachera <sirio81@gmail.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
build: include man/*.in to EXTRA_DIST

Currently, there will be an error in the distribution directory.
After a 'make clean', 'make' will fail becuase it cannot find *.in files in
directory 'man'.

This patch fixed this.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: remove unnecessary casting

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Add more macros for sha1.h

- use SHA1_DIGEST_SIZE instead of home-grew SHA1_LEN
- move snap_log into farm.h

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sockfd_cache: guard fds_count with sockfd_cache.lock

Otherwise, fds_count can increase while sockfd_cache_add() initializes
new->fds[i].fd, and it leads to severe memory corruption.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
zookeeper: exit program on unrecoverable error

This adds panic() where sheep receives unrecoverable error from
ZooKeeper.  It's not good to abort the program on error, but much
better than destroying membership information on each node.  It is
future work to remove those panic().

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
add helper to do linear search of an array

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests: add wrapper _cluster_format

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>