rbtree: make rb_for_each_entry and rb_for_each more generic

This allows us to call rb_xxx helpers inside the rb_for_xx helpers.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: remove local_idx in struct sd_rsp

This field is never used and since it is used internally by dog and sheep, it is
safe to remove it completely from the structure as we currently always have dog
and sheep out of the same repo.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
rbtree: add rb_copy helper

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: check only joining nodes in sd_accept_handler

Only the joining node need to perform cluster_join_check. ANd remove the epoch
check code which only checks the latest epoch to avoid epoch inconsistency.
Rationale is included in the source file.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: add option to disable shepherd driver

This is useful for people that don't want to compile any other drivers in the
sheep binary. Also is useful for driver patch to test drivers one by one.

Since it is not mature yet, we'd better disable it as default

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
rbtree: make rb_node aligned to 64 bytes

Because sd_node has it.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: use rbtree to manage struct sd_node

No algorithm nor functionality changes, but involes a lot of api changes that
assume a static array of sd_nodes.

With this patch, cluster driver will pass a rbtree of nodes directly to sheep
handlers and all the way down we'll play with rbtree instead of the old static
array.

I do not yet move static sd_nodes array in struct epoch_log and struct
cluster_info to dynamic structures. So basically we are still limited by max
node roof, but we are way close to get rid of it.

Shepherd driver is consdiered to be changed later because it might be redesigned
in the future and not stale for now.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
group: add node to sockfd_cache as early as possible

get_vdis will try to connect to all the nodes include joining node, so before
calling get_vdis, we'd better add joining node in the sockfd cache. This is not
a fatal problem and sockfd cache can actually sort it out with the old code, but
this patch will safe us unnecessary revalidating node after grab failure.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: refactor send_join_request()

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
cluster driver: don't use SD_MAX_NODES for local and corosync

They can't scale to SD_MAX_NODES, just order them to use private marco of max
nodes to support. (both as 1024)

This is a preparation patch.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: bump SD_MAX_NODES from 1024 to 6144

This actually means we can support SD_MAX_NODES with zookeeper driver.

For simplicity, local and corosync drivers support at most 1024 nodes.

We are still tied to a fixed number sheepdog can support, but now with a much
larger value.

The real limit is determined by the cluster driver with the current node
management code, that is, how much payload of broadcast message the driver can
send determined SD_MAX_NODES.

With 6144(6*1024), zookeeper's largest message is 983080 bytes, close to 1M that
is its max allowed message size. This means we can actually increase
SD_MAX_NODES beyound 6144 a little bit.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
remove trim/untrim functions

Why?
1. it makes code very complicated because trim/untrim is scattered everywhere in
the code
2 it is hard to pass correct parameters to them
3. only deleted inode can benefit from it for the recovery code, but deleted
inodes are relatively small group compared to data objects

We need a more lower level and transparent mechanism to reduce io network
because sheepdog data management shouldn't be aware of it thus greatly simplify
the code for read.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
group: fix compile error

----
roup.c:361:45: warning: argument to ‘sizeof’ in ‘memcpy’ call is the same expression as the destination; did you mean to dereference it? [-Wsizeof-pointer-memaccess]
    memcpy(timestamp, buf + nodes_len, sizeof(timestamp));
----

We should pass the real length instead of length of pointer

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>