virtio::net: Set infinite priority to worker threads

This patch utilizes the priority in order to ensure the fast response of
a low latency thread.

In addition it prevents the context switch ping-pong for Rx and Tx
worker threads when they wake an application threads in the context of a
netchannel and per-CPU-Tx flows correspondingly.

The above ping-pong may happen since the runtime of each worker is
usually greater than a runtime of a single application thread and
scheduler will most likely switch to the application thread when it's
woken by a worker.

This patch:

   - Improves the performance of a ycsb workload-C cassandra test on a
     KVM Guest with 4 vCPU OSv Guest by 7.47%, 17.02%, 29.24% and 24.45%
     for 64, 128, 200 and 400 threads correspondingly.

   - Degrades the performance of the above test on single vCPU KVM Guest
     by ~10% for 64 and 128 threads and improves by 18.3% and 34.2% for
     200 and 400 threads.

   - Degrades the redis-benchmark LRANGE_500 test result (50 sockets +
     16 pilelines) on a KVM 2 vCPU Guest by 35%.

The Cassandra SMP case is the main beneficiary of this patch.  It's
because it makes sure that the virtio threads - running on one CPU -
always get a chance to quickly distribute work to the other CPUs instead
of letting another thread on this single CPU run for a long time and
cause the 3 other CPUs to become idle.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
percpu_xmit::poll_until(): Beautify the code a bit

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
percpu_xmit::poll_until(): Call a yield() when trying to bypass a control to the "next" tx-worker

This significantly improves a chance to the control to actually pass to
the next worker.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched: yield() don't overwrite the runtime of a yielding thread

Our yield() function guarantees that if there is another runnable
thread, we unconditionally switch to the other thread. However, this was
implemented by overwriting the yielding thread's "runtime" measurement
with a copy of the next thread's runtime - causing the scheduler to
prefer that other thread in the course of its normal flow.

Deleting the yield()ing thread's runtime history may create a lot of
weird and unpredictable situations, e.g. almost not running but
yield()ing thread may not be getting a fair share of the CPU's time when
it would want to run because there is another frequently running thread
on on the same CPU; or otherwise, frequently running and yield()ing
thread may "steal" the CPU time in an unfair way from the less
frequently running thread on the same CPU.

Namely calling yield() may cause our scheduler to become very unfair!

The solution is to change yield()'s implementation: Instead of changing
the yielding thread's runtime so that the scheduler's regular flow will
take care of the yielding, we leave the runtime unmodified and pass a
"yielding" flag to the scheduler.

When the scheduler sees this flag, it behaves differently: It doesn't
leave the current thread running even if it still has the lowest
runtime.

Also, when we switch to another thread, we need to pick some preemption
time (so we'll eventually return to the yielding thread). This time is
passed to the scheduler as a parameter.

Note: This patch alone when applied to master HEAD 5c48ba8fa4567
      introduce the regression of ~35% in redis LRANGE_500 (50 sockets,
      16 pipelines) test on 2 vCPUs KVM Guest.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched::yield(): Get rid of extra dereferencing

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>