object cache: introduce background pusher

Background pusher will mitigate the following problems
1 take a long time to flush a largely accumulated dirty objects set
2 some old kernel might not issue flush request periodically and get a long
  wait while shutdown the VM

We kick background pusher when there are more than MAX_DIRTY_OBJECT_COUNT dirty
object. At anytime, we only allow one pusher running, be it background pusher or
synchronous pusher triggered by VM flush request.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests: move tests to tests/functional

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
build: add 'make coverage' target

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
unittest: add unittest for collie/common.c

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
unittest: add unittest for sheep/vdi.c

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
add helper to do linear search and removing key in an array

This adds a type-safe version of lfind() and a helper to remove key in
an array, and simplifies complex manipulation of node arrays in the
cluster drivers.

This also fixes a memory boundary bug in do_leave_sheep().

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
functional/047: fix md test

If we use a different WD for this test, the objects can be stored to
other disks than d2.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
functional: add wrapper _cluster_format

This introduces a helper _cluster_format to wait for all the sheeps to
be running.  This removes sleep after 'collie cluster format' and
reduces the possibility of causing a false alarm.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
local: fix return type of lnode_cmp

The local driver cannot pass the functional tests without this.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sockfd_cache: guard fds_count with sockfd_cache.lock

Otherwise, fds_count can increase while sockfd_cache_add() initializes
new->fds[i].fd, and it leads to severe memory corruption.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: introduce xfallocate and xftruncate helpers

These helpers retry the syscalls while the error is retry-able.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
use 4KB for trim_zero_sectors instead of 512B

Most filesystems use 4KB for their block sizes, and in that case
trimming 512B may be meaningless.  It's better to use the block size
than the sector size for faster comparison.

This also allows for the caller to specify the offset of buffer so
that it can get the block-aligned offset and length.  It is in
preparation for allowing sparse objects.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: introduce generational reference counting for object reclaim

Generational reference counting is an algorithm to reclaim data
efficiently without race conditions on distributed system.  This
extends vdi objects structure to store generational reference counts,
and increments the counts when creating snapshots.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: introduce ledger objects

This introduces ledger objects, which keeps track of the number of
outstanding references of each generation.  Sheep decrements a
generational reference count with a gateway request SD_OP_DECREF_OBJ,
and reclaims objects when there is no generational reference.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: decrement generational reference count on vdi deletion

This removes old vdi deletion code, which reclaims objects only when
all relevant snapshots are deleted, and uses a generational reference
counting algorithm.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: decrement generational reference count on copy-on-write

This decrements a reference count of the old data object when
allocating a new data object on CoW.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: remove max children limit

We no longer need it.  This also reduces the vdi header size.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: introduce sparse objects

We don't need to care about a fragmentation when
 - the object is unlikely to be accessed sequentially, and
 - the object is read-only.

In that sense, we can make the objects sparse if they are not data
objects and writable ones.

This fixes the problem that sheepdog consumes many disk spaces for
deleted vdi objects if your filesystem supports FALLOC_FL_PUNCH_HOLE.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests: show ledger objects in list

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: add test for object reclaim

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>