sheep, dog: add "block_size_shift" option to cluster format command

This patch changes for specifying cluster default block_size_shift.

Signed-off-by: Teruaki Ishizaki <ishizaki.teruaki@lab.ntt.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep, dog: add selectable object_size support of VDI operation

Data object size was fix to 4MB and not selectable.
This patch add feature to select data object size of VDI.

If you want to use 8MB(2^23) data object_size, specify the
block_size_shift bit num to 23.

ex) dog vdi create -z 23 testvdi 100M

Signed-off-by: Teruaki Ishizaki <ishizaki.teruaki@lab.ntt.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
dog: revert the change for output of "dog vdi list" manually

This patch is a temporal thing. The previous commit (ae3c78b9651cc2)
introduced change of "dog vdi list" command, it added a new field of
block size shift.

It breaks existing tests/fucntional. So this patch remove the
change. After the tests updated correctly, this patch can be reverted
simply.

Cc: Teruaki Ishizaki <ishizaki.teruaki@lab.ntt.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
func/test: change functional test output for __vdi_list

Change output of functional test using __vdi_list
in assosiation with adding block_size_shift information
to vdi list command.

Signed-off-by: Teruaki Ishizaki <ishizaki.teruaki@lab.ntt.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
Revert "dog: revert the change for output of "dog vdi list" manually"

This reverts commit 0dbbc3cf45e603d144f31a6bba736dbc11ee19d6.

Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: fix bug for not saving block_size_shift to cluster config

This patch fixes bugs that block_size_shift info was forgotten
after cluster shutdown and start sheepdog.

Add block_size_shift info to cluster config file.

Signed-off-by: Teruaki Ishizaki <ishizaki.teruaki@lab.ntt.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
tests/functional: test a case of orphan object revival

Tested-by: Valerio Pachera <sirio81@gmail.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: forbid revival of orphan objects

Current recovery process can cause revival of orphan objects. This
patch solves this problem.

How to: exclude a newly joining sheep from recovery process if a
cluster is already in SD_STATUS_OK. It emans the cluster has every
replica already.

Tested-by: Valerio Pachera <sirio81@gmail.com>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep, dog: check cluster is formatted or not during vdi creation

Current dog prints an odd error message in a case of vdi creation
before cluster formatting like below:
$ dog/dog vdi create test 16M
VDI size is larger than 1.0 MB bytes, please use '-y' to create a hyper volume with size up to 16 PB bytes or use '-z' to create larger object size volume

This patch revives previous behavior.

Reviewed-by: Teruaki Ishizaki <ishizaki.teruaki@lab.ntt.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: let vdi state have parent VID

This is a preparation patch.

Cc: Saeki Masaki <saeki.masaki@po.ntts.co.jp>
Cc: Yuka Kawasaki <kawasaki.yuka@po.ntts.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: construct data structures for representing family tree

This patch adds new data structres which represent VDI family tree. It
is different from the existing vdi_state_entry tree. The purpose of
these new data structres is efficient tracking of family relation
between VDIs. It is required for garbage collecting needless VIDs.

Cc: Saeki Masaki <saeki.masaki@po.ntts.co.jp>
Cc: Yuka Kawasaki <kawasaki.yuka@po.ntts.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
libsheepdog: provide atomic_clear_bit

The new function is required because the bitmaps (vdi_inuse and
vdi_deleted) can be cleared now.

Cc: Saeki Masaki <saeki.masaki@po.ntts.co.jp>
Cc: Yuka Kawasaki <kawasaki.yuka@po.ntts.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>
sheep: garbage collect needless VIDs

Current sheepdog never recycles VIDs. But it will cause problems
e.g. VID space exhaustion, too much garbage inode objects.

Keeping deleted inode objects is required because living inodes
(snapshots or clones) can point objects of the deleted inodes. So if
every member of VDI family is deleted, it is safe to remove deleted
inode objects.

Cc: Saeki Masaki <saeki.masaki@po.ntts.co.jp>
Cc: Yuka Kawasaki <kawasaki.yuka@po.ntts.co.jp>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi@lab.ntt.co.jp>