sheep: don't update vdi_iocb in vdi_* functions

We are using vdi_iocb to get a snapid from a vdi object, but it looks
like abusing of vdi_iocb.  This patch makes how to use vdi_iocb and
vdi_info clear like as follows.

 - vdi_iocb is used to pass parameters from clients.
 - vdi_info is used to get info from sheepdog.

To prevent from abusing them again, this patch makes some vdi_iocb
variables constant where possible.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/functional/059: make sure write completion before live snapshot

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
logger: use gdb commands to dump stack and variable

This patch uses gdb features to dump stack and variable, and
simplifies logger.c a lot.  With this patch, gdb outputs will be
logged to gdb.txt instead of sheep.txt.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: remove finish_join()

Just call sockfd_cache_add_group directly.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: return default copy number as formated

We should return cinfo.nr_copies, which is the number as formated, for
get_vdi_copy_number() if copy number isn't found during init.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: split create_vdi_obj() into 4 functions

create_vdi_obj() is very complex because it contains four kinds of
operations:

 - create fresh vdi
 - clone vdi
 - snapshot vdi
 - rebase vdi (rollback)

Splitting them apart makes the code much easier to read.

This patch is just a cleanup and doesn't change its logic at all.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
trace: call tracer->exit only when the paird trace->enter was called

This fixes the result of graph tracer when we use other tracers
(thread_checker and/or loop_checker) at the same time.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: don't start recovery if cluster was shutdowned

Add a new field in the config file to indicate shutdown status. The layout of
config file isn't modified, just find a free slot to hold shutdown status.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: remove unused get_max_copy_number()

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
make: add 'check-ununsed' command

'make check-unused' will try to find the possible unused functions that is
defined but used functions.

E.g

yliu@ubuntu-precise:~/sheepdog$ make check-unused
recalculate_vnodes
object_cache_try_to_reclaim
default_cleanup
default_create_and_write
default_exist
default_format
default_get_hash
default_init
default_link
default_purge_obj
default_read
default_remove_object
default_update_epoch
default_write
get_max_copy_number
...

functions listed above are either
1) defined but not used by other c files, indicating that we should define them
static functions
2) defined but not used at all, indicating that we should remove them completely

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
mark the functions static as much as possible

These functions are found by 'make check-unused' and this patch doesn't try to
deal with default_xxx functions in plain.c and object_tree_print, which are
expected to be called by future users.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
add helper to get a self path

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
optimize fnv hash

FNV hash author says that a expression of shifts and adds is faster
than FNV_prime multiply (*).  However, with the recent GCC and
hardware, it looks like the FNV_prime multiply generates much faster
code.

This patch uses FNV_64_PRIME to generate codes, and adds fnv_64a_64 to
create a hash value from 64 bit integers faster.  Note that this patch
is just for optimization and will not change the generated hash
values.

(*) http://www.isthe.com/chongo/tech/comp/fnv/#gcc-O3

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
add sd_hash to generate more dispersed hash values

I noticed that adding one more fnv_64a calculation like

  hval = fnv_64a_64(hval, hval)

generates much more dispersed hash values.

This patch introduces sd_hash(), which does the above calculation
before returning the result.  This also add sd_hash_next(), which is
intended to be used for virtual nodes and virtual disks.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: use sd_hash instead of fnv_64a_buf

This patch uses sd_hash instead of fnv_64a_buf for better object
dispersion.

This also fixes the problem that we are using the number of disks for
calculating virtual disk ids.  Without this patch, virtual disk ids of
the existing disks will be changed for each we plug/unplug a disk; a
lot of object movement will happen.

Please note that this patch breaks backward compatibility because this
changes a hash function of our consistent hashing algorithm, which
means that object location will be also changed.  The upgrade code for
the existing users will be appeared before releasing the next sheepdog
release.

I've not changed some hash calculation which is used for getting
object file names (vdi object name and attribute object name).  It is
because changing them makes it very hard to implement the upgrade code
and forces us to modify qemu client codes too.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
increase the default number of virtual nodes

We gave up backward compatibility in the previous patch, so it's a
good chance to increase the default number of virtual nodes.

The number means the default granularity of how much data will be
moved after node change events.  E.g., if the number of virtual nodes
is 5, at least one-fifth data will be moved when object movement
happens.

There is no good rational reason for the number 128.  The bigger
number requires more memory space.  I've chose it just because it's
the same number as the current default virtual disks.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: update store version

There is no compatibility after changing a hash function for the
consistent hash ring.

The upgrade code is not implemented yet.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/functional: update tests for new hash function

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
test/unit: add test to check hash function

This unittest checks

 - the existing virtual node ids after node join/leave events.
 - the existing virtual disk ids after disk plug/unplug events.
 - dispersion of sd_hash() with a chi-squared test.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/func: refine 072

We should use '_kill_all_sheep' instead of 'cluster shutdown' to simulate a
cluster crash

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>