update CHANGELOG.md

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/functional: reorder tests

The commit acbb4ebca1c174 added a new entry of test with invalid
order. This patch fixes it.

Cc: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Cc: Valerio Pachera <sirio81@gmail.com>
Cc: Alessandro Bolgia <alessandro@extensys.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: remove invalid error logs produced by dog vdi setattr

This bug is reported in https://bugs.launchpad.net/sheepdog-project/+bug/1273080
To remove invalid error logs, I changed to skip sd_read_object() for existence check in sheep/vdi.c
if -x option is not specified to dog vdi setattr.

=== Bug Description ===
I got error logs when I added new attribute on vdi.
It seems that sheep searched for existing attr and couldn't find. Then logged error but created attr normaly.
I think these log should be INFO or WARN level.

Environment and versions
sheepdog : tag 0.7.6 (4 nodes)
operating system : CentOS 6.4

Here's the operations.

$ dog vdi list
  Name Id Size Used Shared Creation time VDI id Copies Tag
$ dog cluster format
using backend plain store
$ dog cluster format
using backend plain store
$ dog vdi create disk1 10m
$ dog vdi list
  Name Id Size Used Shared Creation time VDI id Copies Tag
  disk1 0 10 MB 0.0 MB 0.0 MB 2014-01-23 18:51 e8b18f 3
$ dog vdi setattr disk1 key1 value1
$ dog vdi getattr disk1 key1
value1

This returned 'value1' for key1 normally.

Here's the sheep.log

Jan 23 18:53:15 ERROR [gway 12101] sheep_exec_req(1062) failed No object found
Jan 23 18:53:15 ERROR [gway 12101] sheep_exec_req(1062) failed No object found
Jan 23 18:53:15 ERROR [gway 12101] sheep_exec_req(1062) failed No object found
Jan 23 18:53:15 ERROR [block] read_backend_object(414) failed to read object 20e8b18fc3b4ab67, No object found

No output on log when 'dog vdi getattr'.

Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Reported-by: Yoshinori Matsuo <matsuo.yoshinori@lab.ntt.co.jp>
Signed-off-by: Yuichi Bando <bando.yuichi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/http: add APPEND operation for PUT request

We allow append write for PUT operation. When 'FLAG: append' is specified in the
http PUT request header, we append the new data at the tail of the existing object
instead of a 'delete-then-create' semantic. When we append objects, we mark them
as ONODE_INIT. When all the append operations are done, we specify 'FLAG: eof'
in the PUT request hearder to finalize the whole transaction, which mark the
objects as ONODE_COMPLETE.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/http: add variable 'data_len' for o_extent[]

Since users could use append-operation to add 1MB data each times, the
'start' and 'count' in o_extent[] is not enough to describe the exact
length of data stored in o_extent (each o_extent may store only 1MB
data, which is small than SD_DATA_OBJ_SIZE).

Therefore we add variable 'data_len' in o_extent[] to identify the
length of data.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
test: add test case for append-operation in http service

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheepfs/http: read next cache if data of current cache has been read out

When we use command 'tar' to extract a tar-file in http interface of sheepfs,
it report error and quit. The reason is that if a read operation has almost
reach the end of current cache, it will only return the size which has not
been read in CURRENT CACHE at present.
For example, if a read operation want to read 1MB and it already reach the
offset of (63MB + 500K) in current cache, the read() will return 524K even
the file has more data to read. The tool of 'tar' read 10KB every time, so if
its read() return size below 10KB, it will think the file is end and quit
which cause problems.

The solution is simple: if current cache has almost been read out, try to
read data of next cache.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
test: add test-case for nog-aligned read() on http interface of sheepfs

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/md: add main_fn marker to for_each_object_in_wd()

We should add the main_fn marker to for_each_object_in_wd() because
the function for_each_object_in_wd() could only be called in main
threads at present.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/md: called get_vnode_info() only once in for_each_object_in_wd()

get_vnode_info() should be called only once in for_each_object_in_wd()
(in the main thread) and the result should be passed to
thread_process_path() as a pthread argument.
This avoids lots of calls to get_vnode_info() and is necessary to pass
our thread checker of the sheepdog tracer.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog/farm: kill snap file

snap file is used to store [idx, trunk_sha1] pairs, but since we already have
sha1 information in struct snap_log, we don't need it at all and we can directly
store trunk sha1 in snap_log.

Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add a operation to get cluster info

This is a preparation patch.

Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog/farm: add snap log header

We don't store any previous cluster state for farm, so we can't restore old state
like redundancy scheme.

This patch introduce a header for snap file and as a side effect, we add a magic
and version check capability for cluster snapshot.

This will break the compatibility of old cluster snapshot. For simplicity, we just
error out if we are operating on the old snapshot data.

Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog: restore default redundancy scheme for 'cluster snapshot load'

Previously we don't honor the default redundancy scheme after loading cluster
snapshot and we always set it as '3'. With the help of snap log header, we can
restore the default redundancy automatically.

Reviewed-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/functional: use dog instead of qemu-img in 016

It seems that qemu-img command has a bug of using cluster's nr_copies
instead of vdi's original nr_copies when it creates snapshot. This
patch lets 016 not use qemu-img.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
dog: use nr_copies of inode object instead of command line parameter

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
dog: forbid creating snapshot of hypervolume

Currently it is not supported.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: introduce generational reference counting for object reclaim

Generational reference counting is an algorithm to reclaim data
efficiently without race conditions on distributed system.  This
extends vdi objects structure to store generational reference counts,
and increments the counts when creating snapshots.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: introduce ledger objects

This introduces ledger objects, which keeps track of the number of
outstanding references of each generation.  Sheep decrements a
generational reference count with a gateway request SD_OP_DECREF_OBJ,
and reclaims objects when there is no generational reference.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: decrement generational reference count on vdi deletion

This removes old vdi deletion code, which reclaims objects only when
all relevant snapshots are deleted, and uses a generational reference
counting algorithm.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>