sched: add atomic reset() operation to timer_base

It is needed by the new async framework.
sched: introduce thread migration lock

This can be useful when there's a need to perform operations on
per-CPU structure(s) and all need to be executed on the same CPU but
there is code in between which may sleep (eg malloc).

For example this can be used to ensure that dynamically allocated
object is always freed on the same CPU on which it was allocated:

  WITH_LOCK(migration_lock) {
    auto _owner = *percpu_owner;
    auto x = new X();
    _owner->enqueue(x);
  }
core: introduce deferred work framework

The design behind timer_task

timer_task was design for making cancel() and reschedule() scale well
with the number of threads and CPUs in the system. These methods may
be called frequently and from different CPUs. A task scheduled on one
CPU may be rescheduled later from another CPU. To avoid expensive
coordination between CPUs a lockfree per-CPU worker was implemented.

Every CPU has a worker (async_worker) which has task registry and a
thread to execute them. Most of the worker's state may only be changed
from the CPU on which it runs.

When timer_task is rescheduled it registers its percpu part in current
CPU's worker. When it is then rescheduled from another CPU, the
previous registration is marked as not valid and new percpu part is
registered. When percpu task fires it checks if it is the last
registration - only then it can fire.

Because timer_task's state is scattered across CPUs some extra
housekeeping needs to be done before it can be destroyed.  We need to
make sure that no percpu task will try to access timer_task object
after it is destroyed. To ensure that we walk the list of
registrations of given timer_task and atomically flip their state from
ACTIVE to RELEASED. If that succeeds it means the task is now revoked
and worker will not try to execute it. If that fails it means the task
is in the middle of firing and we need to wait for it to finish. When
the per-CPU task is moved to RELEASED state it is appended to worker's
queue of released percpu tasks using lockfree mpsc queue. These
objects may be later reused for registrations.
core: introduce serial_timer_task

This is a wrapper of timer_task which should be used if atomicity of
callback tasks and timer operations is required. The class accepts
external lock to serialize all operations. It provides sufficient
abstraction to replace callouts in the network stack.

Unfortunately, it requires some cooperation from the callback code
(see try_fire()). That's because I couldn't extract in_pcb lock
acquisition out of the callback code in TCP stack because there are
other locks taken before it and doing so _could_ result in lock order
inversion problems and hence deadlocks. If we can prove these to be
safe then the API could be simplified.

It may be also worthwhile to propagate the lock passed to
serial_timer_task down to timer_task to save extra CAS.
net: add tracepoints for inpcb life cycle
net: remove dead code
net: convert in_pcb lock from struct mtx to mutex

The new async API accepts lock of type 'mutex' so I need to convert
in_pcb lock type, which will be used to synchronize callbacks.
net: replace callouts with the new async framework

The callout subsystem is using a shared global lock for most of its
operations. This became a bottleneck after merging net channels.
Explanation for this phenomena is that before net channels
were merged packet processing on the receive side was done form one
(virtio) thread and there was no contention on that lock. After the
merge the packets started to be processed from many CPUs which made
taking the lock expensive.

The new framework uses only per-timer locks, the worker is lock-free.

Below are measurements of the improvement. The measurements (both
before and after) were taken with Nadav's per-CPU rcu_defer()
improvement applied, because it was also a bottleneck.

The value measured was HTTP request-response throughput of tomcat
server as reported by the wrk tool. Server and client on different
machines, 4 vCPUs, 3g of guest memory.

=== 16 connections ===

Before:

  avg = 39272.61
  stdev = 3611.84

After:

  avg = 52701.82
  stdev = 3953.76

Improvement: 34%

=== 256 connections ===

Before:

 avg = 35225.19
 stdev = 2504.27

After:

 avg = 50576.67
 stdev = 3533.39

Improvement: 43%

One challenge in integrating the new framework with TCP stack was a
proper teardown of timers. Current code assumed that after calling
callout_cancel() it is safe to free the timer's memory. This was not
correct because the timer may have already fired and will try to
access memory which has been freed. The TCP stack had a workaround for
this race, each timer checked the inp field of the tcpcb block for
NULL, which was supposed to indicate that the block has been freed. It
still was not perfect though because the timer may have performed the
check before the field was nulled-out in the tcp_discardcb and then
block on a mutex which will be promptly freed. The solution I went for
is to delegate the release of memory into an async deferred task,
which will be executed as soon as possible but in a safe context, in
which we can wait until all timers are done and then free the
memory.
Merge branch 'async' of https://github.com/tgrabiec/osv

"After net channel merge in commit 2828ef505f398b18e0a6a973b890278ee31046a5
the performance of tomcat benchmark dropped significantly. Investigation
revealed that the biggest bottleneck was the callout subsystem, which was
using global mutex to protect its operations. This series improves
the performance by replacing use of callouts inside the TCP stack with
a new framework which is supposed to scale better."

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>