move suspend_worker_threads and resume_worker_threads to work.c

Currently, we are using trace_lock to prevent a new thread from being
created while suspending worker threads.  However, there is still a
race condition like the following case:

 1. create_worker_threads() creates a new worker thread.
 2. Before wq_create_cb() is called, suspend_worker_threads() sends
    SIGUSR2 signals to the threads in the worker list.  The newly
    created thread is not in the list yet, so it doesn't receive the
    signal.
 3. The newly created thread starts working before
    resume_worker_threads() is called.

We need worker_info->pending_lock to avoid this problem.  This patch
moves suspend_worker_threads() and resume_worker_threads() to work.c
to use the lock.  This also does the following changes:

 - Use bitmap instead of list to manage the worker threads.  With this
   change, we can delete threads more efficiently.

 - Use eventfd instead of pthread_lock to suspend threads.  Basically,
   pthread_mutex_lock() is not a signal-safe function and should be
   avoided here.  In addition, this change removes a busy loop in
   suspend_worker_threads().

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
trace: create a caller list in tracer_init()

In tracer_enable(), we replace all the mcount() calls in the caller
list with trace_caller().  However, we create the caller list
dynamically; we update the list when sheep calls a new function which
is not in the list for the first time.  This means that we cannot
trace the functions which were not called before we call
tracer_enable().

To include all the function into the tracer targets, this patch
creates a complete caller list just after sheep starts up.  This
achieves it with BFD (a part of binutils-dev) and makes the current
tracer code much simpler.

This can also remove trace.ld, and allows us to see a line number when
debugging with GDB.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
trace: guard strbuf with pthread_mutex_lock

strbuf is not thread-safe and we have to guard a buffer with a mutex.
Using a different buffer for each cpuid is not enough because the cpu
can switch to another thread in strbuf functions.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
trace: add support for stackable tracers

This adds support for calling multiple tracers at entry/exit points of
functions.  The benefits of this change is as follows:

 - This moves complicated codes of tracers into trace/tracer.c and
   simplifies trace/graph.c.  It makes it very easy to add another
   tracer.

 - I'm going to add a thread checker on top of the tracer
   infrastracture.  Supporting stackable tracers allows us to use both
   graph tracer and thread checker at the same time.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
trace: move nop5 to trace.c

No need to write it with an assembly code.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
add notrace to only entry point and exit point

It is difficult to know which function needs a notrace directive.  We
are using notrace to avoid starting tracer recursively while tracing,
but we can do the similar thing by using a thread local variable.
With this patch, notrace is necessary only to the entry point
(trace_function_enter) and the exit point (trace_function_exit).  This
patch removes notrace compiler.h so that we don't abuse the directive
any more.

This patch also allows us to use tracer with debug build.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
configure: support tracer with debug build

Now we can use tracer with debug build.  This patch enables it by
default when --enable-debug is specified.  Let's see whether it should
be enabled in the release build too.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add a checker for thread type

This introduces a two maker "main_fn" and "worker_fn".  If the
function must be called in the main thread, add "main_fn" to it.  If
the function must be called in the worker thread, add "worker_fn" to
it.  The markers are used only when we enable tracer infrastructure.
The check is done at the entry point of the function.

Adding the marker is simpler and easier than writing assert.  In
addition, programmers can easily understand which function must be
called in the main/worker thread.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
sheep: add event_loop checker

We shouldn't spend too much time in the main thread.  This checker
prints a warning message when sheep uses more than 1 ms in the main
thread.

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>
collie: rename debug to trace

This renames 'collie debug trace' to 'collie trace', and cleanups sub
commands.  Available trace commands are:

  trace enable <tracer>   enable the specified tracer
  trace disable <tracer>  disable the spcified tracer
  trace status            show tracer statuses
  trace cat               cat the output of tracers if any

Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Signed-off-by: Liu Yuan <namei.unix@gmail.com>