tests/064: exclude non-deterministic line from result

The content of store 2 is not deterministic since we kill sheep 2
while reweighting is ongoing.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: flush cache for snapshot creation

This fixes a panic for such case with object cache enabled:

1 VM is running and take a live snapshot by qemu-img or collie
2 after snapshoting, there are some dirty objects in the cache
3 sheep will try to push these dirty objects but sheep gateway return READONLY
4 sheep gat paniced because pusher will assume it never fails

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
collie: remove 'vdi flush' command

Use 'vdi cache flush' to flush the vdi only.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix explanation of SD_RES_OLD_NODE_VER and SD_RES_NEW_NODE_VER

Sheep returns SD_RES_OLD_NODE_VER when the sent request contains an
older epoch, and SD_RES_NEW_NODE_VER when it contains an newer one.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
zookeeper: return the found path in zk_find_seq_node

This is necessary for the zookeeper driver to get the queue position
on the first push.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
zookeeper: exit program on the unrecoverable error

This adds panic() where sheep receives unrecoverable error from
ZooKeeper.  It is future work to remove those panic().

This also removes the ZOOAPI modifiers from the function definitions,
which are not ZooKeeper APIs.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: gateway cannot cleanup stale objects

sd_store can be NULL if this node is a gateway, so we need the NULL
check before calling sd_store->cleanup.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: introduce xfallocate and xftruncate helpers

These helpers retry the syscalls while the error is retry-able.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
use 4KB for trim_zero_sectors instead of 512B

Most filesystems use 4KB for their block sizes, and in that case
trimming 512B may be meaningless.  It's better to use the block size
than the sector size for faster comparison.

This also allows for the caller to specify the offset of buffer so
that it can get the block-aligned offset and length.  It is in
preparation for allowing sparse objects.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: introduce generational reference counting for object reclaim

Generational reference counting is an algorithm to reclaim data
efficiently without race conditions on distributed system.  This
extends vdi objects structure to store generational reference counts,
and increments the counts when creating snapshots.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: introduce ledger objects

This introduces ledger objects, which keeps track of the number of
outstanding references of each generation.  Sheep decrements a
generational reference count with a gateway request SD_OP_DECREF_OBJ,
and reclaims objects when there is no generational reference.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: decrement generational reference count on vdi deletion

This removes old vdi deletion code, which reclaims objects only when
all relevant snapshots are deleted, and uses a generational reference
counting algorithm.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: decrement generational reference count on copy-on-write

This decrements a reference count of the old data object when
allocating a new data object on CoW.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: remove max children limit

We no longer need it.  This also reduces the vdi header size.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: introduce sparse objects

We don't need to care about a fragmentation when
 - the object is unlikely to be accessed sequentially, and
 - the object is read-only.

In that sense, we can make the objects sparse if they are not data
objects and writable ones.

This fixes the problem that sheepdog consumes many disk spaces for
deleted vdi objects if your filesystem supports FALLOC_FL_PUNCH_HOLE.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
doc: add documentation of object reclaim algorithm

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>