lib: add a helper to strip out strbuf

This patch prepares for trace patch set.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
logger: add a helper to get the thread name

This patch prepare for trace patch set.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: rework the trace mechnasim to work with current code

There are mainly two changes:
 1 remove dumb ring_buffer and use strbuf to store trace items
 2 use another mechanism to suspend/resume worker threads for dynamic patching.

As before, if users don't configure with --enable-trace, there is actually zero
impact on the code base. All the tracer funtions will be complied out.

With reworks, add a new field to trace output: thread name (worker_name + worker_idx).

For e.x, we can only filter the trace output and get the gateway thread trace:

$ collie/collie debug trace start
... cluster is operating
$ collie/collie debug trace stop
$ collie/collie debug trace cat > t
$ grep 'gway 27' t | > t1
$ cat t1
gway 27             |            |   do_process_work() {
gway 27             |            |      gateway_write_obj() {
gway 27             |            |         gateway_forward_request() {
gway 27             |            |            gateway_to_peer_opcode() {
gway 27             |       0.793|            }
gway 27             |            |            get_sd_op() {
gway 27             |       0.598|            }
gway 27             |            |            get_req_copy_number() {
gway 27             |       0.547|            }
gway 27             |            |            sheep_get_sockfd() {
gway 27             |            |               addr_to_str() {
gway 27             |       1.904|               }
gway 27             |            |               sockfd_cache_search() {
gway 27             |       0.675|               }
gway 27             |            |               xmalloc() {
gway 27             |       1.36 |               }
gway 27             |      19.773|            }
gway 27             |            |            send_req() {
gway 27             |      29.548|            }
gway 27             |            |            sheep_get_sockfd() {
gway 27             |            |               addr_to_str() {
gway 27             |       2.499|               }
gway 27             |            |               sockfd_cache_search() {
gway 27             |       5.33 |               }
gway 27             |            |               xmalloc() {
gway 27             |       1.859|               }
gway 27             |      35.478|            }
gway 27             |            |            send_req() {
gway 27             |      27.173|            }
gway 27             |            |            sheep_do_op_work() {
gway 27             |            |               peer_write_obj() {
gway 27             |            |                  do_write_obj() {
gway 27             |            |                     strbuf_addf() {
gway 27             |       5.809|                     }
gway 27             |            |                     jrnl_begin() {
gway 27             |            |                        xzalloc() {
gway 27             |            |                           xmalloc() {
gway 27             |       1.143|                           }
gway 27             |       3.141|                        }
gway 27             |            |                        xpwrite() {
gway 27             |   41206.762|                        }
gway 27             |            |                        xpwrite() {
gway 27             |   27106.822|                        }
gway 27             |            |                        xpwrite() {
gway 27             |   16433.765|                        }
gway 27             |   84822.558|                     }
gway 27             |            |                     default_write() {
gway 27             |            |                        get_obj_path() {
gway 27             |       1.873|                        }
gway 27             |            |                        xpwrite() {
gway 27             |   37716.486|                        }
gway 27             |   37739.304|                     }
...

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: rename worker threads for gateway and revoery

Because they use short threads, their name is actually combination of
name string and idx. The idx might be big, so we try to keep short name
for short threads.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: rework debug trace funtions to use subcommand mechanism

The new look is:

tailai.ly@taobao:~/sheepdog$ collie/collie debug trace
Usage: collie debug trace {start|stop|cat} [-a address] [-p port] [-h]
Available subcommands:
  start                   start the trace
  stop                    stop the trace
  cat                     cat the trace
Options:
  -a, --address           specify the daemon address (default: localhost)
  -p, --port              specify the daemon port
  -h, --help              display this help and exit

The change is that 'stop' command actually read all the buffer to /tmp/tracefile
then we can 'cat' this file until it is overridden by next trace.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: change the opcode name

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: fix ret_stack_index type

Sometimes it can be negative, in this case it will be too big as unsigned.

Also mark tow functions as notrace because the are producing too high volume of
trace.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: use qemu cache option to use writeback/writethrough mode

When writeback is enabled, SD_FLAG_CMD_CACHE flag is set to I/O
requests.  This patch uses it for write mode.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
collie: refine cache option of vdi write

What we should set when writing objects is not "whether we use object
cache" but "whether we use a writeback mode".

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: use new cache options

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: fix compile error without --enable-trace

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
test: consolidate 012 to git rid of spurious failure

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
sheep: split notification messages into two queues

Currently, the corosync driver uses two queues, confchg_list and
notify_list.  However, this causes problems because some messages
(e.g. format, shutdown, cluster snapshot, ...) needs to be ordered
with confchg events.

This patch splits multicast messages into two queues, block_list and
nonblock_list.  All block messages (e.g. vdi creation, etc) are linked
into block_list queue, and other notification messages and confchg
events are linked into nonblock_list.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: save store driver name outside storage driver

Storage drivers shouldn't care about config file.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: save config before logging current epoch

If sheep exits after writing epoch info and before saving config in
cluster_make_fs(), it will send a wrong config to master and fail to
join Sheepdog when it restarts.

This patch saves config before logging epoch info to avoid the
problem.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
tests: ensure that sheep processes exit after calling pkill

After send a signal, there is a delay until sheep exits.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: don't do patching while short_thread_running

When the code patcher wakes up, there is still chances that short thread is
running. We simply return to wait for next eventfd notification until we are
safe to do patching.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
trace: run trace cat as a worker

The subtle case is that when tracer is in patching, we should return AGAIN to
buffer reader to ask him to try again until patching is down.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
trace: fix random crash for enable command

We should wait for all the worker's signal handler running like old code, or we
will get random crash when signal handler is called after patching.

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
sheep: fix eventfd_read() error handling

This is motivated by a crash log:
...
Sep 01 21:53:50 [block] suspend(64) going to resume
Sep 01 21:53:50 [block] exec_local_req(449) event fd read error Interrupted system call
Sep 01 21:53:50 [block] do_process_work(1245) failed: 12, 0 , 1, 8
Sep 01 21:53:50 [main] cluster_op_done(283) LOCK_VDI (0xc7a790)
Sep 01 21:53:50 [gway 4] do_process_work(1238) 83, c7c850, 32580
...

that eventfd_read() get EINTR without proper handling and then mess up the next
process.

- retry read in exec_local_req() (worker thread)
- return in enable/disable_tracer() (main thread)

Signed-off-by: Liu Yuan <tailai.ly@taobao.com>