sheep: use eprintf instead of fprintf(stderr)

These fprintfs will be called in the context sheep, so we have to use
eprintf for err messages.

 - coroutine: use panic() instead of fprintf() & abort()

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
makefile: add cscope support

It is nice to get cscope.out by single line 'make cscope'

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: refactor recovery logic

Current recovery logic is elusive and it is not easy to be understood.
I hope this work would ease the headache.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: rename sheepdog_{,v}node_list_entry into sd_{,v}node

This naming mostly force us to use multiple lines for function paras.
We'd better have them terse.

This patch is generated by utility 'find & sed'.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: abstract out 'all or nothing' write operation

In recovery, sheep needs to atomically write the object to the store
that is being recovered. This operation assume underlying store layout,
so we need to abstract it out.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: refactor fill_obj_list()

- rename __start_recovery to explicitly say it work in worker thread
- rename recover_done. It is not really done for most cases when it
is called.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix cluster multicast

Cluster ops with process_main() will be multicasted, so we need to set
req->data to msg->data in such cases.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: stop serving non-force requests when in halt

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix SD_FLAG_NOHALT collision

SD_FLAG_NOHALT and SD_FLAG_CMD_WRITE use the same value. This is wrong.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix recovery logic

We should check if join_msg->inc_epoch before starting recovery after
a new jode joins.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: hide some minor store layout aware operations

We need this to decouple the sheep from kv-store.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
store: add dynamic mechanism to chain the available backend stores.

 - change global store structure to a pointer
 - use a list to maintain the stores.
 - use /obj/.store to remember backend store persistently.
 - now we can specify the backend store in the command
	collie cluster format -b farm #use farm
   if no store specified, currently sheep will use 'simple' store.
   if specified store not available, collie will return a list of
   available stores.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: transfer store backend for newly joined node

When the new node joins the cluster, it doesn't know what kind of
backend store the cluster uses, so we need to transfer this information
by master node in the join phase.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: modify the configures to run farm.

Drive farm to work!

add two new lib dependency:
 - libssl

Dongsu Park suggests that we should check crypto lib for SHA1_Init
with newer libssl.

Currently libssl-1.0 and libssl-0.9.8 works with current configuration.

For debian based systems, you can install them by the name libssl-dev.

Signed-off-by: Dongsu Park <dongsu.park@profitbricks.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: check object directory path at start-up

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: add sha1_file operations

All the objects(snap, trunk, data) in the farm is based on the
operations of the sha1_file.

sha1_file provide us some useful features:

- Regardless of object type, all objects are all in deflated with zlib,
  and have a header that not only specifies their tag, but also size
  information about the data in the object.  It's worth noting that the
  SHA1 hash that is used to name the object is always the hash of this
  _compressed_ object, not the original data.

- the general consistency of an object can always be tested independently
  of the contents or the type of the object: all objects can be validated
  by verifying that
	(a) their hashes match the content of the file and
  	(b) the object successfully inflates to a stream of bytes that
  	forms a sequence of <sha1_file_hdr>  + <binary object data>

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: add trunk object

trunk object is meta data that describes the structure of the data objects
at the timeline of snapshot being taken.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: remove stale object in backend store

For storing one object into backend store, either

a) no content change, then point to the same old sha1_file (no stale object)
or
b) content updated, then will point to a new object with a new sha1.

So we need to remove stale object in case b), only in the assumption it is the
object generated by recovery code. [*]

When we try store new snapshot object into the backend store, it is safe and
good timing for us to remove the old object with the same object ID.

[*] Here I assume we don't need to restore to 'sys epoch' state.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
farm: add snapshot object

snap object is the meta data that describes the snapshot.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: add begin_recover() hook to store.

We need to notify underlying store when the recovery begins.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>