sheep: use struct vdi_iocb to simplify the vdi_create api

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: add -c for 'collie vdi create' to specify redundancy level for per-vdi

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: rename ctime to create_time in vdi.c

'ctime' shadows a variable in pthread.h, in order to use pthread in
vdi.c, so we need to rename ctime to get rid of the compile warning

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: add vdi_copy tree to store copies number for every VDI

The normal requests from QEMU include the copies number in its header,
QEMU stores the nr_copies the first time it reads the inode data, but
many other local requests such as read_copy_from_replica and
recover_object_from_replica don't know the copies number for every object,
so this tree is necessary to keep the copies number

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fetch vdi copy list after sheep joins the cluster

The new joined node doesn't have the vdi copy list, or have
incomplete vdi copy list, so we need to fetch the copy list
data from other nodes

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: use the specified copies number for IO requests

In gateway_read{write,create_and_write}_obj, and read{write}_object,
we should use different copies number for different requests and objects,
instead of using the global copies number sys->nr_copies or calculated
from vnodes

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: remove some unused copies related functions

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: show copy number in the output of 'collie vdi list'

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: add private options to collie's command

Now, all collie's command share the same global collie_options, it will
lead to option's name conflict among commands if they use the same options
but with different description.

By moving the private options into individual structure of each command, and
make collie_options only contain the common part of them, we can solve this
problem.

Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
test: add a test description to output

With this patch, the new look is:

tailai.ly@taobao:~/sheepdog/tests$ sudo ./check 27
PLATFORM      -- Linux/x86_64 taobao 2.6.35-32-generic

027 Last Used:7s. Test sheep recovery logic breadth-first algorithm
Passed all 1 tests

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: return the copy number of the request object in peer_read_obj

collie uses this return value in the output of 'collie vdi object', without
this, we can not pass test 028

Signed-off-by: levin li <xingke.lwp@taobao.com>
sheep: fix cloning VDI fail when cloned VDI has greater nr_copies than the base

Signed-off-by: levin li <xingke.lwp@taobao.com>
test: add a test case for different VDI redundancy level

Signed-off-by: levin li <xingke.lwp@taobao.com>
test: allow more free style test description

23-25 doesn't begin with 'Test'. Instead of modifying it, we'd better allow more
free style description

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fix vdi exsitence check

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: add plain store driver

This introduces a storage driver 'plain' based on the current storage
interface.  The design of the plain store is similar to one of the
farm driver.  The main difference is that the current farm uses the
sha1 based backend store for stale objects, but simple_store uses a
flat directory for them.  With this design, plain store can move
objects from the working directory to the backend store efficiently
with rename(2).

Here are pros vs cons of the plain store against the current farm.

Pros:
 - faster recovery
 - smaller and simpler
 - would be a good example to introduce other storage drivers

Cons:
 - cluster snapshot is not supported
 - stale objects are not deduplicated
 - there is no sha1 verification

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: don't send vdi_inuse in get_vdis

Since we already send vdi copy list in get_vdis(), there's no
need to send the vdi_inuse bitmap any more, without it we can
send less data as the length of vdi copy list is variable and
it contains the vid just as vdi_inuse does.

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
collie: make `collie vdi list` not read vdi_inuse bitmap

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: use plain implementation for core IO path

This greately reduce the complexity of farm recovery handling. With this patch,
Farm will operate entirely the same as plain store for IO and recovery handling.
Farm and Plain store will share core IO functions completely.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
plain: fix init_objlist_and_vdi_bitmap()

We should add vdi copy number to copy list.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>