collie: use xmalloc instead of malloc

collie can fail in a OOM case without causing any problem.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
md: rename get_object_path as md_get_object_path

This is a preparation patch.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
plain store: refactor default_get_hash

We only need to get object path once in the first place.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
collie: show 'auto-recovery' status for 'cluster info'

nr_copies is removed because it isn't used at all

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: change default store driver from "farm" to "plain"

1. modified collie/cluster.c to change default store driver to plain
2. remove tests/030 and tests/030.out, because snapshot is not available to plain store
3. modified tests/*.out, replace "farm" with "plain"

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: don't compile sheep/farm

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: remove snapshot from cluster subcommand

collie is not able to do "collie cluster snapshot".

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: remove farm logic from sheep

1. sheep/ops.c: remove snapshot operations
2. sheep/store.c: check_path_len() don't need to consider farm
3. sheep/sheep_priv.h: remove snapshot, restore and get_snap_file from struct store_driver

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: store.c don't include farm.h

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/farm: remove sheep/farm/farm.h

farm.h will be added to collie/farm later.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
script: remove script/simple2farm

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie/farm: implement object_tree

object_tree is a red black tree which is used to de-duplicate oid.
So that objects with the same oid in the cluster will be read only once when doing cluster snapshot.

When user sends cluster snapshot command, collie will:
1. parse vdis
2. save readonly object's id into object_tree, duplicate id will be saved only once
3. for each oid in object_tree, read the object and save to farm

object_tree has 5 public interfaces:
1. object_tree_size()
   returns current object number in the tree
2. object_tree_insert()
   insert a object id into the tree
3. object_tree_free()
   free memory
4. object_tree_free()
   print each node of tree for debug
5. for_each_object_in_tree()
   foreach node in the tree, read object from cluster and call the function.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie/farm: implement sha1_file

moved sha1_file from sheep/farm to collie/farm

Changes:
1. move function get_object_directory() to farm.h.
   It will be moved to farm.c later.
2. fix style problem
3. replace sd_xprintf(...) with fprintf(...)

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie/farm: implement snap object

moved snap.c from sheep/farm to collie/farm

Changes:
1. Snap also saves a "tag" to snap_log, so that user can load a snapshot by specify a tag
2. Epoch info is not necessary, use index instead. User can use either tag or index to select a snapshot to load
3. Construct snap_log_path during snap_init(). snap_init() will be called by farm_init().
4. Fix style problems.
5. Replaced sd_xprintf(...) with fprintf(...)

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie/farm: implement trunk object

moved trunk.c from sheep/farm to collie/farm

Changes:
1. trunk don't need to read file from plain_store.
2. farm.c will responsible for read objects and fill trunk entry.
3. add for_each_object_in_tree().

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add a flag to let notify_vdi_add set bitmap if needed

Add a uint8_t flag "set_bitmap" to sd_req.vdi_state.
Function notify_vdi_add will set bitmap if "set_bitmap" is not 0.

This feature will be used for recovering vdi state and vdi bitmap
on all nodes after loading snapshot to a new cluster.

Currenly, post_cluster_new_vdi() is called only after a SD_OP_NEW_VDI request.
And we won't send SD_OP_NEW_VDI, because it will do a lot things that we don't need.
We use SD_OP_NOTIFY_VDI_ADD request for setting both bitmap and vdi_state.

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie/farm: implement farm

moved farm.c from sheep/farm to collie/farm

* farm.c have changed a lot *

Farm is responsible to save/load cluster snapshot.

During save snapshot:
0. construct object_rb_tree (this is done in cluster.c)
1. parse object_rb_tree, for each object, call fill_trunk_entry().
2. write trunk entries to trunk file.
3. write trunk sha1 to snap file.
4. write snap sha1 and snapshot tag to snap log file.

During load snapshot:
1. read snap log file.
2. find snap sha1 by tag or index.
3. read snap file and get trunk sha1.
4. read trunk file.
5. for each entry in trunk, read object from sha1 file and write to new cluster
6. create active vdi for each vdi chain based on the last snapshot

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: fix collie failure when sub-subcommand has more than 2 arguments

Support command like this:
$collie command subcommand sub-subcommand arg1 arg2

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: implement "collie cluster snapshot" subcommand

Command example:
*Usage*
Save all readonly objects to local path.
"tag" is used to describe a snapshot.
$collie cluster snapshot save tag /localpath

List all cluster snapshot saved in local path.
$collie cluster snapshot list /localpath

Load a snapshot to a cluster.
This will format cluster firstly.
User can use tag or index to select a snapshot.
$collie cluster snapshot load tag|idx /localpath

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
test: add tests/030 for cluster snapshot

Signed-off-by: Kai Zhang <kyle@zelin.io>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>