net: don't close the tcp connection within tcp_do_segment()

tcp_do_segment() has a nasty habit of sometimes closing the tcp connection.
This plays havoc with the locking rules (as updated for net channels), since
tcp_do_segment() is supposed to return with the lock held, yet tcp_close()
will unlock if the connection is disassociated from a socket.

Fix by having tcp_do_segment() not close the connection, and instead indicate
its desire to do so to its caller.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
virtio: switch to page-sized external buffers

Curently virtio uses MCLBYTES (2KB) buffers for virtio.  This hits a weakness
in our UMA and memory allocation implementations:

- UMA appends a 4-byte reference count to the allocation, resulting in an
  2052 byte allocation request
- Our allocator only services requests up to 2048 bytes from the mempool
  allocator; larger requests are serviced by the large allocator, which is
  significately slower, and is also serialized by a mutex.

Switch to an EXT_EXTREF mbuf with the buffer and refcount allocated
separately.  This also allows us to switch to 4K allocations serviced by
the page allocator (memory::alloc_page()), which is our fastest allocator,
and halves the lengths of the mbuf chains on streaming workloads.

Increases iperf stream (host->guest) results from ~12Gbps to ~34Gbps on my
desktop.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
trace: fix prof -h

Currently it fails with:

Traceback (most rTraceback (most recent call last):
  File "scripts/trace.py", line 433, in <module>
    args = parser.parse_args()
...
  File "/usr/lib/python2.7/argparse.py", line 601, in _expand_help
    return self._get_help_string(action) % params
ValueError: incomplete format
Introduce sampling profiler
Merge branch 'sampler' into async

Conflicts:
	build.mk
	loader.cc
Per-CPU rcu_defer()

The existing rcu_defer() used a mutex to protect the list of deferred work,
and woke up the cleanup thread every time. This made rcu_dispose() much
slower than a regular delete: On my machine a new/delete pair takes 23 ns,
while a new/rcu_dispose takes a whopping 330 ns, and worse - when we have
4 threads on 4 vcpus in a tight new/rcu_dispose loop, the mutex contention
and the fact we free the memory on the "wrong" cpu brings the measurement
to as much as 12,000 ns.

With this patch the new/rcu_dispose numbers are down to 83 ns on a single
thread, and 105 ns on 4 concurrent threads (on 4 cpus). This is a x4 to x120
speedup :-)

This patch replaces the single list of functions with a per-cpu list.
rcu_defer() can add more functions to this per-cpu list without a mutex,
and instead of the single garbage-collection thread running all deferred work
on a single CPU, it just decides when to do the work, and then each CPU
carries out the work from its own list. This per-cpu work is particularly
effective for free() work (i.e., rcu_dispose()), because it is faster to
free memory on the same CPU where it was allocated.

Additionally, there is no need to wake() the garbage collection thread on
every rcu_defer(), like the previous code did. Because this thread often
goes back to sleep (to wait for the different CPUs to end their quiescent
state) spurious wakeups cause context switches and significantly hurt
performance. Instead, we only need to wake the thread once when we put the
first work item in the list - and not wake it up again until it finally
caused the list to be cleared.

The size of the (per-CPU) queue is currently set to 2000. Increasing the
queue size from 2000 to 10,000 continues to improve performance: single
thread new/rcu_dispose goes down from 83 ns to 58 ns. But the tradeoff
is that it increases memory use: these lists take memory, and of course
the more memory whose freeing is being defered, the more memory we waste.
So in the future we should reconside this issue.

How much batching actually happens depends on the scheduler's decisions:
Once the garbage collection thread is woken, It takes some time for all CPUs
to reach the RCU quiescent thread, and all that time new deferred work is
being batched; If the queue becomes full, further rcu_defer() calls wait.
On the other hand, we do not wait for the work queue to entirely fill
before handling it, so there is no risk of unlimited delay of deferred
work. Even if there was just one call to rcu_defer(), the garbage
collection thread will be woken, and will soon cause a cleanup of this
single item of work.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Merge remote-tracking branch 'avi/net_channel' into async

Conflicts:
	bsd/sys/netinet/tcp_input.cc