sheep: drive object cache to work

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: add red black tree library

Red black tree is more scalable than hash lists if we
cannot predict the size of the data set.

This is excerpted from Linux kernel.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: object cache proper

Object cache caches data and vdi objects on the local node. It is at
higher level than backend store. This extra cache layer translate gateway
requests into local requests, largely reducing the network traffic and highly
improve the IO performance.

Dirty objects will be flushed to cluster storage by 'sync' request from
guest OS.

- use red-black tree to track dirty objects
- use file lock to avoid RW race on object granularity
- use hash lists to maintain vdi space.
- each vid has its own independent object cache

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: teach sheep to use object cache

We only intrude IO code for gateway requests. Object IO path from
recovery logic is intact.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: add flush_vdi operation

This is supposed to be initiated by Guest OS, but our collie
friend might also like it.

Flush operation is operated on vdi basis, that is, when one guest
flush its own dirty data, other guests are not affected.

- use forward_write_obj_req() to flush dirty objects

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: let object cache better interact with recovery logic

If the targeted object of the gateway request is cached already,
we can simply operate on it despite of node changes.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: flush the vdi before searching the object

'collie vdi object' need to read latest vdi object from backend
store to assist it to locate the object. Since the vdi object
is also cached in object cache, we need to flush it beforehand to
cluster.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: add object list cache implemented by rb-tree

Added object list cache implemented by red-black tree,
everytime we creates a new object, we write the oid of the
object into the object list cache.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: object list cache applied to farm and simple store

Applied the object list cache to get_obj_list() which
walk through the rb-tree cache and return the cached list,
instead of calling sd_store->get_objlist(),
make it much faster to get the object list.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
farm: avoid unnecessary IO operation when recovering

When the cluster is recovering, we only need to write the objects
which no longer belong to the node to the snapshot, instead of writing
all the objects, by which we decreased the IO operation.

When we try to read an object, we first read it from the local
object directory, if not found, then read it from the snapshot.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Revert "simple store: fix preallocation object size"

This reverts commit 6e88ebe7258469542fa29ba719b22d5d0a048c5d.
Revert "farm: fix preallocation object size"

This reverts commit 70461948cc23e4808542b8944758ed4904224608.