Fix thread::thread_clock() on a running thread

The thread_clock() function, returning the amount of CPU time spent by a
thread, under-estimated when called for a different running thread. In that
case, it only accounted for full time slices. This underestimate was
usually forgivable (the load balancer wakes up 10 times a second, so keeps
time slices under 0.1 seconds), but as explained in issue #366, we noticed
this problem as idle percentage in "top.py" hovering between 95%-105%,
instead of being at 100% as expected.

The problem in estimating a the cpu time used by a thread running on a
different CPU, is that we need a consistent set of two different values
(running_since and total_cpu_time), but each may become visible at a
different time, and at an unknown order. Additionally, running_since is
currently per-cpu, not per-thread.

Rather than adding additional memory fences to the context-switch path,
I went with the solution in this patch. It is not pretty, but it is the
most efficient solution I could find (surprisingly, I couldn't even measure
a nanosecond of slowdown of the context switch benchmark).
If anybody can suggest a prettier solution with similar performance,
please let me know.

The good performance of this patch happens because all it adds in to a
context switch is a single write, when switching into a thread. There are
no memory fences, or locked instructions, and the written memory is in a
cache-line which should be in the cache anyway. Additionally, there is no
extra code when switching out of a thread.

The idea is that when switching in a thread, we atomically write a
64-bit thread field comprised of both running_since and total_cpu_time,
32 bits of each. To make 32 bits enough we give up 10 bits of accuracy
(so thread_clock() on a running thread is accurate up to a microsecond),
and 22 bits of range (so the range is 2200 seconds - way more than our
context switches). This code, cpu_estimator_set(), is very simple and quick
and I couldn't even measure one nanosecond of slowdown to the context switch.

Fixes #366.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>