mempool: drop redundant assert
mempool: erase page range using an iterator rather than a reference

free_page_ranges is an intrusive set. erasing via a reference requires
iteration over reference equal_range under the hood, which means
traversing the tree to the leafs. Whereas erasing via an iterator
requires no such lookups so should be faster.
sched: turn _migration_lock_counter into __thread local variable

This way migration-lock can be used unconditionally in memory
allocator which is called very early, before the first thread is
created.
mempool: prevent across-CPU page stealing

This addresses the race which is mentioned in the comment. If the
thread doing add_page() is migrated before the preempt_lock is taken
it may feed the pool with a page coming from different CPU's page
buffer.

The fix is to embrace the body with migration-lock.
lockfree: introduce unordered_queue_mpsc

It's like queue_mpsc with two improvements:

 * consumer and producer links are cache line aligned to avoid false
   sharing. I was tempted to apply this to queue_mpsc too but then
   discovered that this queue is embedded in a mutex, and doing so
   would greatly bloat mutex size, so I gave up on this idea.

 * The contract of pop() is relaxed to return items in no particular
   order so that we can avoid the cost of reversing the chain.
lockfree: introduce unordered_queue_spsc

It is meant to provide both the speed of a ring buffer and
non-blocking properties of linked queues by combining the two. Unlike
for ring_spsc, push() is always guaranteed to succeed.
tests: introduce latch.is_released()
tests: introduce latch.await_for()
tests: add test for free() throughput across CPUs

There is one allocating threads and two freeing threads. Each thread
is allocated on a different core. The test measures throughput of
objects freed by both threads.
mempool: remove contention inside free_different_cpu()

Fixes #308.

When the per-cpu-pair ring fills up, the freeing thread is blocked and
enters a synchronous object hand-off. That synchronous hand-off is
cause of contention. Instead of having a bounded ring we can use a
unordered_queue_spsc which is a hybrid approach with a ring buffer and
a fall-back lock-free queue. In this implementation push() always
succeeds.

Dead objects are collected by per-CPU worker thread (same as
before). The thread is woken up once every 256 frees. That threshold
was chosen so that the behavior would more or less correspond to what
was before.

In a test which allocates 1K blocks on once CPU and having two threads
freeing them on two other CPUs, there is a 50% improvement of free()
throughput, the rate is also much smoother:

Before:

[tgrabiec@muninn osv]$ scripts/run.py -m10g -e "/tests/misc-free-perf.so"
OSv v0.08-174-g6053797
eth0: 192.168.122.15
4321806.50 obj/s
3832555.75 obj/s
4151844.50 obj/s
3996754.25 obj/s
4195447.00 obj/s
3834015.50 obj/s
4117833.25 obj/s
3817401.25 obj/s
4265755.50 obj/s
3748168.50 obj/s
4113812.75 obj/s
3966778.75 obj/s
3986076.75 obj/s
3956971.25 obj/s
3956712.00 obj/s
3951243.00 obj/s
3875920.25 obj/s
3893898.50 obj/s
3752529.75 obj/s
3739663.00 obj/s
3259523.50 obj/s
3259888.75 obj/s
2760378.25 obj/s
2745481.50 obj/s
2652068.75 obj/s
2638404.75 obj/s
2712665.25 obj/s
2702836.25 obj/s
3275618.00 obj/s
3246408.00 obj/s
1442684.00 obj/s
1429990.50 obj/s
957848.19 obj/s
857501.56 obj/s
760831.69 obj/s
694792.12 obj/s
337268.16 obj/s
816486.69 obj/s
0.00 obj/s
1005887.06 obj/s
Freed 58901562 in 10.37 s = 5679389.48 per second

After:

[tgrabiec@muninn osv]$ scripts/run.py -m10g -e "/tests/misc-freec-perf.so"
OSv v0.08-175-g4f45c63
eth0: 192.168.122.15
4273591.50 obj/s
4273824.00 obj/s
4326075.50 obj/s
4326214.00 obj/s
4414123.00 obj/s
4413920.50 obj/s
4409198.50 obj/s
4409229.00 obj/s
4349843.00 obj/s
4349563.50 obj/s
4395483.00 obj/s
4395509.50 obj/s
4384853.00 obj/s
4385046.50 obj/s
4397949.00 obj/s
4397754.50 obj/s
4399109.00 obj/s
4399075.00 obj/s
4402658.50 obj/s
4402843.00 obj/s
4400586.00 obj/s
4400337.50 obj/s
4365298.50 obj/s
4365690.00 obj/s
4390448.50 obj/s
4390114.50 obj/s
Freed 61310550 in 7.00 s = 8758626.73 per second

mempool
mempool: remove contention inside unfill_page_buffer()

This change reduces page buffer pressure caused by virtio-net
allocations, showign up as contention inside unfill_page_buffer() and
refill_page_buffer(). The virtio-net subsystem allocates pages on one
CPU which are then scattered across all other CPUs and freed to their
per-CPU buffers. To mitigate that, this change enables caching of
pages used by virtio-net.
mempool: do not take preempt_lock when freeing from different cpu