ops: add NULL check of req->op

req->op can be NULL when the requested opcode is invalid.  It's safe
to add NULL checks before accessing req->op in ops.c.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add helper function to make sure that req->data is string

There is no guarantee that req->data is a string.  Actually, the
current code can cause a buffer overrun when, e.g.,
SD_OP_FORCE_RECOVER is requested.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
tests/functional: fix valgrind test

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
recovery: fix buffer overflow of rinfo->oids

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: initialize object cache info

This silences valgrind errors.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/plain_store: move displaced object from read/write operation

This is a defensive patch, but solve a problem we found on the test cluster. The
problem is tricky:

init_vdi_state() errorr out as "failed to read inode header xxx, 0".

After a detailed log analyzing, we found that the cause of this problem is
mismatch of object of path between get_obj_path() and its real path, e.g,

- disk1/oidA (real location)
- disk3/oidA (get_obj_path() location)

So default_read() fails.

To solve this problem and any future problems caused by misplaced object (if any
by any unexpected reason), we call default_exist() before default_{read, write}.
By this means, all the misplaced objects will be placed properly before
any read/write operations.

Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep/recovery: multi-threading recovery process

Rationale for multi-threaded recovery:

1. If one node is added, we find that all the VMs on other nodes will get
   noticeably affected until 50% data is transferred to the new node.

2. For node failure, we might not have problems of running VM but the
   recovery process boost will benefit IO operation of VM with less
   chances to be blocked for write and also improve reliability.

3. For disk failure in node, this is similar to adding a node. All
   the data on the broken disk will be recovered on other disks in
   this node. Speedy recoery not only improve data reliability but
   also cause less writing blocking on the lost data.

Our oid scheduling algorithm is intact and simply add multi-threading onto top
of current recovery algorithm with minimal changes.

- we still have ->oids array to denote oids to be recovered
- we start up 2 * nr_disks threads for recovery
- the tricky part is that we need to wait all the running threads to
  completion before start next recovery events for multiple nodes/disks events

This patch passes "./check -g md -md" on my local box

Signed-off-by: Liu Yuan <namei.unix@gmail.com>