sheep/cluster: delete znodes when leaving cluster

A sheep node call zk_lock() to create a znode in zookeeper and become the
owner of the lock, then if all disks under this sheep node is broken
suddenly, the node will have to leave cluster.Now, the znode will stay
in "/lock" and make other sheep nodes waiting it forever.

So we should delete znodes which created by this sheep node when leaving
cluster.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog/farm: register vdi for 'snapshot load operation'

This will git rid of need that we have to manually check copy_policy of requests
in the plain store.

Reviewed-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: kill is_erasure_obj()

Since we already register vdi for 'cluster snapshot load', we don't need
ugly is_erasure_obj() any more. Now we can use is_erasure_oid() to reliably
check if oid is ec object.

Reviewed-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: don't call heavy md_get_info init_obj_path()

This will iterate all the objects of the MD before vdi states setup, causing
data inconsistency problems.

Reviewed-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: store ec_index in the pathname instead of in xattr

This patch solves a performance issue that in the recovery, update_epoch() is
called to iterate all the objects to move stale objects into stale directories
this is fast for replication objects because oid_stale() doesn't access disks
at all. But for erasure objects, we have to read xattr for each one. This is
considerably slow for massive data deployment. For e.g, it might take more than
one hour for 6T data in 6 nodes cluster. Since update_epoch() is required to
run in the main thread, this means we can't process any requests during the
first phase of recovery.

Instead of storing 'ec_index' in the xattr of the object, we directly store it
in the pathname of ec objects, e.g, xxxoidxxx_1 indicate its ec index is 1 and
it is an ec object. By this way, we

- simplify the read/write path a bit
- no need to read disks at all for update_epoch()

Thus largely boost the update_epoch() and also boost normal read/write a bit.

With this patch, the disk layout for ec objects is changed from

oid       --> oid_index       [data  object]
oid.epoch --> oid_index.epoch [stale object]

Reviewed-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
dog/vdi: rework 'vdi object location' and 'vdi track'

Add a SD_OP_EXIST to check if an object exists in the specific node.

Reviewed-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/recovery: fix a sefault when ->stop is set true

For multi-threaded recovery, if one thread set ->stop true and release rinfo,
other threads might null-deferenece rinfo and get segfault.

We should simply ignore ->stop in main thread and wait for run_next_rw() to be
called and safely run into the next recovery.

Reviewed-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: reload config after successful migration

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: migrate format from v3 to v4

The commit 79706e07a068 introduced incompatibility of store format
(store ec index in path instead of xattr). This patch produces
migration function for this change.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/http: add macro ONODE_HDR_SIZE

Add macro ONODE_HDR_SIZE to replace BLOCK_SIZE in http with which the
code will be more easier to understand.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/http: add Created-Time for onode

At present, we allocate space for an onode and then write data for it,
so we should record two times: 'ctime' means when the onode is created
and allocated space, 'mtime' means when the data of onode have all be
written done.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/cluster: use wrapper function to create ephemeral znode

We use zk_create_node() before, but when operation of creating ephemeral
zonde has been timeout, we will see multi-znodes in parent znode because
zk_create_node() will create znodes again if there is ZOPERATIONTIMEOUT.
Multi-znodes in same directry will make zk_lock() be dead loop.

The best solution for this problem is make value of 'timeout' for
zookeeper_init() bigger, but we still need to use zk_create_seq_node() to
replace zk_create_node() for which we will see "operation timeout" fail
in sheep log.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/cluster: change output format in sheep log

We have print debug information for 'lock_id' in octal when calling
zk_lock(), so is the zk_unlock().

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
using xmalloc() to allocate sd_inode instead of using memory on stack

'struct sd_inode' may be very large, so putting it on stack is not safe.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheepfs: fix memory leak in object_create_entry()

We should free all args after using split_path().

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheepfs: free memory of strbuf 't'

Variable 't' is allocated by sheepfs_run_cmd() and should be released
after using it.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/http: only read name of kv_onode when iterating

In object_iterater() it read whole length of SD_DATA_OBJ_SIZE data for
onode which is not necessary and will hurt performance badly.
We only need to read out the 'name' of onode.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/func: Add test case to test data integrity after changing my address of sheep process

If modifying the "my address" of sheep parameter without intention,
the cluster treates it as a new node so that we know the operation is wrong.
However, when trying to revert, we find out the prior created vdi is lost.

Signed-off-by: Ruoyu <liangry@ucweb.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheepfs: add cache for http interface

At present, every read operation to files on sheepfs for http interface will
make a new http request which is bad for performance, so we add a cache: read
big chunk of data from http interface and then give it to read operation
on local.

To implement it, we need to add 'release' interface in fuse.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheepfs: make fetching data for cache become async operation

Currently, we have add cache for http interface of sheepfs.But it will
only fetch data from http request when fuse operation has read out of
the cache.

For better performance, we use 'double buffer' tech: fuse is reading
one buffer and a new created thread could read future data onto another
buffer at the same time. It will make reading operation more smoothly
and faster.

We use two pointer: 'ready' and 'prepare' to point the double buffers
and use classic 'consumer and producer model' to avoid race condition.

Signed-off-by: Robin Dong <sanbai@taobao.com>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>