sched: separate thread statistics

"class thread" is big and messy, and as we add more per-thread statistics
in the following patch, will become even messier. This patch simply moves
the members related to statistics together to the end of the class.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched: per-thread switch statistics

We currently have a "sched_switch" tracepoint which can count (e.g.,
using trace.py) the frequency of context switches in the entire system.
This patch further adds a per-thread counter for the number of times
that this thread was switched in.

The performance cost of the extra counter is very small (a non-atomic
increment to a variable in the same cache-line already touched for the
cpu-time statistics), and tests/misc-ctxsw.so shows no measurable slowdown
of context switches.

Note that dividing the thread's runtime (t->thread_clock()) by the number
of switches results in another interesting statistic - the average length
of time this thread ran before being switched.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
top.py: add "-s" option for context-switch statistics

This patch adds to top.py the ability to show per-thread statistics
on the number of times it was context-switched in, and the average
cpu time it got per switch, over the last refresh period. For
example:

   ID CPU  %CPU    TIME     sw     sw/s us/sw NAME
  478   0   2.9    0.39    630     38.0   763 >>>>>>>java.so

"sw", 630, is the total number of times this thread was ever switched
in. "sw/s", 38.0, is the number of switches per second in the last
refresh period. In each of the time slices in the last refresh period,
the thread ran for an average of 763 microseconds before being either
preempted or voluntarily giving up the CPU (by waiting or yielding).

To reduce users' information overload, the switching information is not
displayed by top.py by default, and the user needs to pass the "-s"
(or "--switches" option) to enable it.

This feature can be useful in understanding the scheduling behavior of
a workload and the function of individual threads. For example, in one
particular Cassandra read-only workload on one CPU, I see the virtio-net-rx
thread being switched in 13,000 times a second running only 2 microseconds
each; I then see 12 Java worker threads each being scheduled only 30 times
a second and doing a fairly large amount of work (around 800 microseconds);
I then see dozens of other Java threads each scheduled very frequently
(over 600 times a second) and doing very little work (20 microseconds) -
presuably the latter threads just read a request and dispatch it to the
worker threads.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
scripts: clean spaces behavior to follow a bit of pep8

Signed-off-by: William Dauchy <william@gandi.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
scripts: clean brackets in run.py

no need for them; make the thing more "pythonic"

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: William Dauchy <william@gandi.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>