build.mk: unify boost-tests list construction method

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
run.py: Fix API user networking port forwarding

The API is listening to port 8000, not 8080.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
virtio-net: fix handling of bad buffers

When we discard a bad buffer, we don't advance to the next one, so we'll free
it again on the next pass, resulting in a double-free.

Tested-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: Add alphasort() function

Don Martin writes:

  I'm experimenting with Capstan, trying to rule out some things for
  getting a bigger application working. I have a simple program to
  recursively list directory contents, here...

    https://github.com/dmarti/mini-ls

  It works on the host, but when I build the .so, the "capstan build" step
  works, and then when I do "capstan run"...

  $ capstan run
  OSv v0.05-501-gdbdf4f7
  Failed looking up symbol alphasort

This is missing libc functionality in OSv. To fix that, add the
alphasort function from musl.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Doxygen: exclude drivers/libtsm

drivers/libtsm includes a bunch of Doxygen comments, and worst of all, a
"@mainpage" tag controlling Doxygen's main page. Since libtsm isn't really
part of the API we intend to advertise, let's exclude this directory from
being scanned by doxygen.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
kbd: don't reset modifier key state on every readkey()

Fixes #219.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
kbd: fix type definition of shift

This variable is for keeping value of uint32_t shiftcode[n], so it
should be uint32_t as well.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Fix type and documentation of sched::thread::id()

After commit 5c652796fe8d08b7ecc87739025492505c9c1b5a, our thread id
is 32 bit, not 64 bit. This patch fixes the return type of the id()
function, and replaces an incorrect comment (which claimed that the
ids are unique during the lifetime of the system) with a more accurate
description on this function, using a doxygen comment (so it can be
seen in our doxygen API documentation).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
mmu: Use thread_handle instead of thread pointer in tlb

Thread pointed to by tlb_flush_waiter may exit before tlb_flush_ipi()
calls wake() on it which will cause access to freed memory. Fix that by
using thread_handle which was introduces to handle exactly those kind of
races.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Use git submodule update --init --recursive

In a case of nested submodules, like swagger-ui inside mgmt inside osv,
only the --recursive update all the submodules, all the way down.

Signed-off-by: Tzach Livyatan <tzach@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc function availability checker for a shared library

This script list up missing libc function called from a shared library.
Usage:
./scripts/check-libcfunc-avail.sh release apps/memcached/memcached.so

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
signal handling: support SA_RESETHAND

Add support for SA_RESETHAND signal handler flag, which means that the signal
handler is reset to the default one after handling the signal once.

I admit it's not a very useful feature (our default handler is powering off
the system...) but no reason not to support it.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched: add atomic reset() operation to base_timer
sched: change timer state inside atomic context

This is prone to races. Example: cancel() may attempt to remove the
timer before it was inserted into timer tree.
sched: introduce thread migration lock

This can be useful when there's a need to perform operations on
per-CPU structure(s) and all need to be executed on the same CPU but
there is code in between which may sleep (eg malloc).

For example this can be used to ensure that dynamically allocated
object is always freed on the same CPU on which it was allocated:

  WITH_LOCK(migration_lock) {
    auto _owner = *percpu_owner;
    auto x = new X();
    _owner->enqueue(x);
  }
percpu: support for passing cpu pointer to object constructor

Some per-CPU structures will need to be aware about which CPU they
belong to. Since they may be constructed from a different CPU than the
one to which they belong we need a way to pass sched::cpu* to the
constructor.

The solution is to allow passing copy-able arguments to dynamic_percpu
constructor, which will be then passed to the per-CPU object
constructor. There is a special 'tag' object which will besubstitued
for sched::cpu*.

Example:

 struct MyObj {
    MyObj(sched::cpu* cpu, int config) {}
 };

 auto my_percpu = make_dynamic_percpu<MyObj>(cpu_tag(), 13);
core: introduce deferred work framework

This was done with intent to replace the current callout framework
which our networking stack uses for TCP timers. The current
implementation has scalability issues which has been exposed by merge
of the net channels work.

timer_task was design for making frequent cancel() and reschedule()
scale well with the number of threads and CPUs in the system. These
methods may be called frequently and from different CPUs, even on the
same timer object. To avoid expensive coordination between CPUs a
lockfree per-CPU worker was implemented.

Every CPU has a worker (asnc_worker) which has task registry and a
thread to execute them. Most of the worker's state may only be changed
from the CPU on which it runs.

When timer_task is rescheduled it registers its per-CPU part in
current CPU's worker. When it is then rescheduled from another CPU, it
does not do anything to the previous registration but registers a new
percpu part in the new worker. If task is rescheduled on a CPU on which it
already has a registration then that registration is atomically
replaced with a new registration.  timer_task keeps track which worker
has the last registration and this determines which percpu part is
eligible to fire. When percpu task fires it checks the owner field and
fires only if it is supposed to. In this scheme, canceling a task is
as simple as setting owner to nullptr.

Because timer_task's state is scattered across CPUs some extra
housekeeping needs to be done before it can be destroyed. This
operation is called shutdown(). We need to make sure that no percpu
task will try to access timer_task object after it is destroyed. Upon
shutdown, we walk the list of percpu tasks which belong to given
timer_task and atomically flip its state from ACTIVE to RELEASED. If
that succeeds it means the task is now revoked and the worker will not
try to execute it. If that fails it means the task is in the middle of
firing and we need to wait for it to finish. After the percpu task is
finally in RELEASED state we enqueue it to the worker's queue of
released percpu tasks using lockfree mpsc queue. These objects may be
later reused for registrations.
net: add tracepoints for inpcb life cycle
net: comment-out dead code
net: replace callouts with the new async framework

The callout subsystem is using a shared global lock for most of its
operations. This became a bottleneck after merging net channels.
Explanation for this phenomena is that before net channels
were merged packet processing on the receive side was done form one
(virtio) thread and there was no contention on that lock. After the
merge the packets started to be processed from many CPUs which made
taking the lock expensive.

The new framework uses only per-timer locks, the worker is lock-free.

Below are measurements of the improvement. The value measured was HTTP
request-response throughput of tomcat server as reported by the wrk
tool. Server and client on different machines, 4 vCPUs, 3g of guest memory.

=== 16 connections ===

Before:

  avg = 39272.61
  stdev = 3611.84

After:

  avg = 52701.82
  stdev = 3953.76

Improvement: 34%

=== 256 connections ===

Before:

 avg = 35225.19
 stdev = 2504.27

After:

 avg = 50576.67
 stdev = 3533.39

Improvement: 43%

One challenge in integrating the new framework with TCP stack was a
proper teardown of timers. Current code assumed that after calling
callout_cancel() it is safe to free the timer's memory. This was not
correct because the timer may have already fired and will try to
access memory which has been freed. This is even less correct with the
new framework in which timers have state distributed over many CPUs
and an explicit shutdown() or timer_task destructor need to be
run. The TCP stack had a workaround for this race, each timer checked
the inp field of the tcpcb block for NULL, which was supposed to
indicate that the block has been freed. It still was not perfect
though because the timer may have performed the check before the field
was nulled-out in the tcp_discardcb and then block on a mutex which
will be promptly freed. The solution I went for is to delegate the
release of memory into an async deferred task, which will be executed
as soon as possible but in a safe context, in which we can wait until
all timers are shut down and then free the memory.