sheep: rework send join message for corosync and local drivers

They can't send the whole struct cluster_info. Instead of doing tricks in
send_join_request(), we just min(SD_MAX_EVENT_BUF_SIZE, join_msg_len) for
corosync and local drivers.

This patch also fixes a bug that corosync can't work with node join because
not enough buffer allocated for join message.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
lib/fec: remove neglected leftover init_fec() in fec_new

This is no longer needed since we call init_fec in the sheep init main function.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: use ec_index to remember strip placement

ec_index is used to tag the replica for data striping since for erasure coding,
each replica are indexed as different strip holders for the object.

We have no room for ec_index in the struct sd_req so we make use of
'uint64_t offset' which effectively we only use its first 32 bits. By squeezing
ec_index into this field, we won't change the layout of sd_req.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: make stale object erasure aware

Replica with different ec_index should be considered stale.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
lib/fec: rework ec_decode

fec_decode has some assumptins about the passed on buffer arrays and indexes.
The reworked ec_decode will handle all the cases internally and we can rebuild
whatever strip, be it data strip or parity strip, we want from this helper
function.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: add basic recovery support for erasure code

With this patch we can support single node event (join/leave).

The recovery strategy for erasure object:

- if object isn't lost, we will read it from targeted live node in its stale dir
- if object is lost, we will read enough replica and rebuild it using RS algorithm

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
recovery: cleanup erasure object handling functions

With this patch we completely use different code path for erasure recovery and
not mess up with replication object recovvery code and make erasure code more
isolated since replication and erasure doesn't share any common logic.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: update object list cache in default_create_and_write()

No need to call it everywhere else.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
recovery: mark last gathered epoch

The last gathered epoch is the epoch at which all the nodes complete the
recovery and purge the stale objects. So this means we don't need to go
the epoch history below last_gathered_epoch.

This is a preparation patch.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
recovery: handle multiple node events for erasured vdi

1. read the lost object from its track in epoch history vertically because
   every replica is unique
2. if not found in 1, then tries to rebuild it with RS algorithm
   2.1 read enough other replica from their tracks in epoch history
   2.2 rebuild the lost object from the content of replica read at 2.1

The subtle case is number for available zones is less than SD_EC_DP or the
requested index of lost object:
    1 we need to make sure nr_zones >= SD_EC_DP to avoid panic of
      oid_to_node(s) helpers.
    2 we have to search all the available zones when we can't get idx. Its
      okay to do a mad search when number of available zones is small

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
dog: fix parse_objs

This allows 'vdi object' to work with erasured vdi.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: support start/kill sheep with number bigger than 9

Current script can't distinguish, e.g, 1 and 10.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
tests/functional: teach it to support erasure code

usage:
$ ./check -ec # will trigger the erasure tests on most test scripts

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: get object size correctly

For backend store, the object size is determined by copy_policy.

Also add a helper to get store object size for future use.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix cluster_notify_vdi_add

We should set copy_policy too for snapshot vdi

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: rework cow object handling

cow handling should be handled in the gateway instead of backend store because
copy-on-write's copy(read) and write should be object type agnostic. With this
patch, we can add cow capability to erasured vdi.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: don't set copies unnecessarily

For gateway request, we don't need to set copy number.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: cleanup preallocate

Since trim/untrim is removed from sheep, we don't need to extend buffer.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: fix default_create_and_write()

Cluster snapshot's create request will have copy_policy set in the sd_req but
without vdi registered. So we have to pass copy_policy down and check it in the
default_create_and_write.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
dog/farm: fix copy_policy for cluster snapshot operation

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>