sheep: handle SD_FLAG_CMD_WRITE in cluster requests

If SD_FLAG_CMD_WRITE is set, req->data means sent data, so we don't
need to set response data in sd_notify_handler().  Otherwise, it means
receive data buffer, so we don't need to set multicast data in
prepare_cluster_msg().

This also fixes some local requests which don't set SD_FLAG_CMD_WRITE.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
cluster/corosync: retry when CS_ERR_TRY_AGAIN is returned

cpg_initialize often fails with a CS_ERR_TRY_AGAIN error.  We should
retry in such case.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
test: fix 003 for local driver

Local driver detect membership change by signal in 1 s internal, so kill &sleep 1
might have no effect for local driver.

This might cause random spurious failure of test.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: initialize reclaim work queue only when cache is enabled

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: use short thread for recovery

Most time sheep isn't in recovery state, so the two recovery work
queue recovery_wqueue/recovery_notify_wqueue stay there useless,
since we have short thread, we can save resource by creating the
thread when recovery needs

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: don't fetch node list if not use

Signed-off-by: levin li <xingke.lwp@taobao.com>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
plain_store: print more information on error path

This makes debugging easier.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: add check of the number of running sheep processes

Without this patch, check script cannot return from _wait_for_sheep()
when one of sheep processs exits unexpectedly.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: add tests to check message order of format operation

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
lib: add a helper to strip out strbuf

This patch prepares for trace patch set.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
logger: add a helper to get the thread name

This patch prepare for trace patch set.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: rework the trace mechnasim to work with current code

There are mainly two changes:
 1 remove dumb ring_buffer and use strbuf to store trace items
 2 use another mechanism to suspend/resume worker threads for dynamic patching.

As before, if users don't configure with --enable-trace, there is actually zero
impact on the code base. All the tracer funtions will be complied out.

With reworks, add a new field to trace output: thread name (worker_name + worker_idx).

For e.x, we can only filter the trace output and get the gateway thread trace:

$ collie/collie debug trace start
... cluster is operating
$ collie/collie debug trace stop
$ collie/collie debug trace cat > t
$ grep 'gway 27' t | > t1
$ cat t1
gway 27             |            |   do_process_work() {
gway 27             |            |      gateway_write_obj() {
gway 27             |            |         gateway_forward_request() {
gway 27             |            |            gateway_to_peer_opcode() {
gway 27             |       0.793|            }
gway 27             |            |            get_sd_op() {
gway 27             |       0.598|            }
gway 27             |            |            get_req_copy_number() {
gway 27             |       0.547|            }
gway 27             |            |            sheep_get_sockfd() {
gway 27             |            |               addr_to_str() {
gway 27             |       1.904|               }
gway 27             |            |               sockfd_cache_search() {
gway 27             |       0.675|               }
gway 27             |            |               xmalloc() {
gway 27             |       1.36 |               }
gway 27             |      19.773|            }
gway 27             |            |            send_req() {
gway 27             |      29.548|            }
gway 27             |            |            sheep_get_sockfd() {
gway 27             |            |               addr_to_str() {
gway 27             |       2.499|               }
gway 27             |            |               sockfd_cache_search() {
gway 27             |       5.33 |               }
gway 27             |            |               xmalloc() {
gway 27             |       1.859|               }
gway 27             |      35.478|            }
gway 27             |            |            send_req() {
gway 27             |      27.173|            }
gway 27             |            |            sheep_do_op_work() {
gway 27             |            |               peer_write_obj() {
gway 27             |            |                  do_write_obj() {
gway 27             |            |                     strbuf_addf() {
gway 27             |       5.809|                     }
gway 27             |            |                     jrnl_begin() {
gway 27             |            |                        xzalloc() {
gway 27             |            |                           xmalloc() {
gway 27             |       1.143|                           }
gway 27             |       3.141|                        }
gway 27             |            |                        xpwrite() {
gway 27             |   41206.762|                        }
gway 27             |            |                        xpwrite() {
gway 27             |   27106.822|                        }
gway 27             |            |                        xpwrite() {
gway 27             |   16433.765|                        }
gway 27             |   84822.558|                     }
gway 27             |            |                     default_write() {
gway 27             |            |                        get_obj_path() {
gway 27             |       1.873|                        }
gway 27             |            |                        xpwrite() {
gway 27             |   37716.486|                        }
gway 27             |   37739.304|                     }
...

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: rename worker threads for gateway and revoery

Because they use short threads, their name is actually combination of
name string and idx. The idx might be big, so we try to keep short name
for short threads.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: rework debug trace funtions to use subcommand mechanism

The new look is:

tailai.ly@taobao:~/sheepdog$ collie/collie debug trace
Usage: collie debug trace {start|stop|cat} [-a address] [-p port] [-h]
Available subcommands:
  start                   start the trace
  stop                    stop the trace
  cat                     cat the trace
Options:
  -a, --address           specify the daemon address (default: localhost)
  -p, --port              specify the daemon port
  -h, --help              display this help and exit

The change is that 'stop' command actually read all the buffer to /tmp/tracefile
then we can 'cat' this file until it is overridden by next trace.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: change the opcode name

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: fix ret_stack_index type

Sometimes it can be negative, in this case it will be too big as unsigned.

Also mark tow functions as notrace because the are producing too high volume of
trace.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: use qemu cache option to use writeback/writethrough mode

When writeback is enabled, SD_FLAG_CMD_CACHE flag is set to I/O
requests.  This patch uses it for write mode.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: refine cache option of vdi write

What we should set when writing objects is not "whether we use object
cache" but "whether we use a writeback mode".

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: use new cache options

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: fix compile error without --enable-trace

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>