collie: cleanup callbacks of collie command when they send header only requests

There are several callbacks of collie command send requests which only contain
header. So let's abstract the common part of the them into a new function:
send_light_req() which could make code more compactness.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: add more error msg for send_light_req()

- also move it out of lib/net.c since only collie use it, thus we can use
  fprintf(stderr, ...) inside to indicate which phase to fail.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: use cmd argument -w to specify a max cache size

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
object cache: add object cache tree for every VDI

Add object cache tree for every VDI to keep track of all the
objects cached by the VDI, for the reclaiming work.

When sheep starts, we should also read the cached objects in
disk which is created by the previous running, otherwise, these
cache objects may cause a disk leak.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
object cache: merge active and inactive dirt_tree/list

Since we will share the same entry in both object_tree and dirty_tree,
it would make thing more complicated to use two dirty tree/list, so
merge them together, and use lock when flushing.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
object cache: use rwlock to replace mutex lock for per-vdi cache

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
object cache: schedule the object cache in a lru list

We put all the cached object into a global lru list, when
the object cache is referenced(read/write), we move the object
to the head of the lru list, then when cache reaches the max
size we can reclaim it from the end of the lru list.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
object cache: reclaim cached objects when cache reaches the max size

This patch do reclaiming work when the total size of cached objects
reaches the max size specified by user, I did it in the following way:

1. check the object tree for the object entry to determine whether the
   cache entry is exist and whether it's reclaiming, if it's reclaiming
   we make sheep ingore the cache.
2. In object_cache_rw() we search the cache entry, after passed the sanity
   check, we increment its refcnt to tell the reclaiming worker that this
   entry is being referenced, we should not reclaim it now.
3. In add_to_object_cache(), when the cached size reaches the max size,
   we start a reclaiming thread, only one such thread can be running at
   one time.
4. In reclaim_work(), we reclaim cached objects until the cache size reduced
   to 80% of the max size.
5. In reclaim_object(), we start to reclaim an object, before this, we check
   that if the cache is flushing, we don't reclaim it, and if the refcnt of
   the object is not zero, we also don't reclaim it.
   If the cached object is dirty, we flush it by push_cache_object(), and
   then try to remove the object.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
object cache: refactor object_cache_remove()

Since all the cache entry are not stored in memory, we can not
only remove entry from dirty tree/list, we should also remove it
from the object tree/list.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
object cache: add a object_list for cache entry for cache deleting

When deleting an entire VDI cache, a object list is easy for traversing
comparing to a rb-tree

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: let all sheeps with smaller epoch added into delayed_nodes list

Since sheeps in delayed_nodes list won't cause recovery when the cluster in
WAIT_FOR_JOIN state, it's safe to put sheeps with smaller epoch into
delayed_nodes list, regardless of whether it once belonged to the cluster.

Benifit from this change, we needn't to restart sheep in the following scenario:

1) Start [0,1,2,3] sheeps:
   epoch of [0,1,2,3] sheeps = 1

2) Kill [0] sheep, and then Shutdown [1,2,3] sheeps
   epoch of [0] sheep      = 1
   epoch of [1,2,3] sheeps = 2

3) Start [1,2] sheeps:
   epoch of [0] sheep      = 1
   epoch of [1,2,3] sheeps = 2
   cluster status = WAIT_FOR_JOIN (waits [3] sheep)

4) Start [0] sheep:
   [0] sheep will be added into delayed_nodes list, needn't to restart
   epoch of [0] sheep      = 1
   epoch of [1,2,3] sheeps = 2
   cluster status = WAIT_FOR_JOIN (waits [3] sheep)

5) Start [3] sheep:
   epoch of [0,1,2,3] sheeps = 3
   cluster status = OK
   Now cluster start working...

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Small grammar fixes and removal of duplicate loglevel line in sheep.8 man page.

Signed-off-by: Matthew Law <matt@webcontracts.co.uk>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Small grammar fixes in collie.8 man page

Signed-off-by: Matthew Law <matt@webcontracts.co.uk>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: check whether extended attributes are available with getxattr

On some filesystems (e.g. ext3), listxattr doesn't return an error
even if extended attributes are not available.  This patch uses
getxattr instead of listxattr for the xattr check.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: check return values of store.init and store.format

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: list available stores only when the wrong driver is specified

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
object cache: don't panic in object_cache_read/wrie()

The caller of object_cache_read/write() can handle the 'NO_CACHE' case
gracefully.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
object cache: remove RECLAIM flag from reclaim path

Since we hold lock while calling find_cache_entry() and
reclaiming the cache entry, there's no necessary for RECLAIM flag

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: add node cache command to set max cache size

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: set max cache size when receive SD_OP_SET_CACHE_SIZE

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>