sheep: support specifying the initial count of virtual nodes

With this patch, you can set the weight of each node with the -v
option.

Example:

  $ ./sheep/sheep /store/0 -p 7000 -z 0 -v 1000
  $ ./sheep/sheep /store/1 -p 7001 -z 1 -v 2000
  $ ./sheep/sheep /store/2 -p 7002 -z 2 -v 3000
  $ ./collie/collie cluster format -c 1
  using backend simple store
  $ ./collie/collie node list
  M   Id   Host:Port         V-Nodes       Zone
  -    0   10.68.14.1:7000        1000          0
  -    1   10.68.14.1:7001        2000          1
  -    2   10.68.14.1:7002        3000          2
  $ ./collie/collie vdi create image 600M -P
  $ ./collie/collie node info
  Id      Size    Used    Use%
   0      188 GB  100 MB    0%
   1      188 GB  200 MB    0%
   2      188 GB  304 MB    0%
  Total   564 GB  604 MB    0%

  Total virtual image size  600 MB

If you specify zero to the number of virtual nodes, no data will be
stored to the node.

Reviewed-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
varies printf type typo

I found several printf related type errors in 32-bit linux environment.
Here's the trivial patch to fix them.

- fix a warning (Liu Yuan)

Signed-off-by: huxinwei <huxinwei@huawei.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fix the memmove bug in del_cpg_node

The size is not right, maybe lead to error

Signed-off-by: HaiTing Yao <wujue.yht@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: Improve the verbose error message in log.

Yet another trivial patch.
I just copied over the comments for these results.
It helps me to figure out the situation more easiler.

Signed-off-by: huxinwei <huxinwei@huawei.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
farm: Fix to put_sha1_file

I found an panic in sheep today.

Mar 02 23:57:28 trunk_file_write(315) try delete stale snapshot object 8071d67a00000000...
Mar 02 23:57:28 put_sha1_file(92) No such file or directory

It happens when the sheep is killed after omap_file_init but before
omap_file_final, and some of the staled snapshot have been unlinked.
Then sheep will always panic every time afterward.

The patch is a proposal to make put_sha1_file tolerant not-exist-file.

Signed-off-by: huxinwei <huxinwei@huawei.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
farm: fix lookup_trunk_entry

It should return NULL when 1. not found and 2. create = 0.
Cleanup the log function

The log_sigsegv function in logger.c actually serves two purposes.
1, the logger itself exit on illegal memory access.
2, the sheep exit, which makes the kernel deliver a SIGSEGV to the logger process

I think it'll be a good idea to make distinguish between these 2.

Signed-off-by: Xinwei Hu <huxinwei@huawei.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: remove unnecessary code

This code is never visited and acted as a safe guard, but the
uppper layer code should take responsibility of initializing
hdr->copies to avoid this redundant check in the hot path.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
simple store: fix preallocation object size

We should allocate different sizes for vdi and data object, instead
of fixed size for both.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
farm: fix preallocation object size

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: use pointer to avoid extra copy

forward_write_obj_req() is kind of hot code path, so this kind of
extra copy is nasty.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fix broken data length passed to store

Object size for vdi_attr obj is never right, though it didn't
cause problem, just bigger than it really needs.

- use SD_INODE_SIZE for inode size
- fix object size passed to store_create_and_write()

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: modify 'buf' member of deletion work

1, make it more readable

2, malloc size is not right, not multiply size of uint32_t

Signed-off-by: HaiTing Yao <wujue.yht@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fix recovery logic

It can be cases that, in some epoch, sheepdog cannot maintain the
required copies of replications. When recovering from such epoch,
we'd better be conservative and double check.

Signed-off-by: Xinwei Hu <huxinwei@huawei.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
farm: fix put_entry() of trunk

There is race when trunk entry is put for hash list.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: reduce snapshot COW read/write

Doing snapshot COW:

1, If new writing request occurs and need COW for sanpshot, now read old
object to buffer, then write the buffer to new object, then write the
request data to new object. We can merge the latter two writing request.

2, If new writing request covers whole object, no need to read old
object.

After the modification, pass bigger buffer to do_write_obj when doing
COW, but it will not add the burden. COW is never for inode object, so
it will not use the journal.

Signed-off-by: HaiTing Yao <wujue.yht@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Revert "sheep: use pointer to avoid extra copy"

This reverts commit ebe6d5289978fc9316d2e4d8330a5b1c6d3ca5bb.

Reason from the commit 32ae69d898721887253868ebe3d78b6f927ea6f9

   We cannot overwrite the request header in forward_*_obj_req() because
   its fields are reffered in __done().
sheep: drive object cache to work

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: add red black tree library

Red black tree is more scalable than hash lists if we
cannot predict the size of the data set.

This is excerpted from Linux kernel.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: object cache proper

Object cache caches data and vdi objects on the local node. It is at
higher level than backend store. This extra cache layer translate gateway
requests into local requests, largely reducing the network traffic and highly
improve the IO performance.

Dirty objects will be flushed to cluster storage by 'sync' request from
guest OS.

- use red-black tree to track dirty objects
- use file lock to avoid RW race on object granularity
- use hash lists to maintain vdi space.
- each vid has its own independent object cache

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>