mozilla/rust — pull request #11325 by alexcrichton (2014-01-05 17:10:26+00:00)
Title: Don't always wake up sleeping schedulers

I created a benchmark recently of incrementing a global variable inside of a
`extra::sync::Mutex`, and it turned out to be horribly slow. For 100K
increments (per thread), the timings I got were:

1 green thread: 10.73ms
8 green threads: 10916.14ms
1 native thread: 9.19ms
8 native threads: 4610.18ms

Upon profiling the test, most of the time is spent in `kevent()` (I'm on OSX)
and `write()`. I thought that this was because we were falling into epoll too
much, but after implementing the scheduler only falling back to epoll() if there
is no work or active I/O handles, it didn't fix the problem.

The problem actually turned out to be that the schedulers were in high
contention over the tasks being run. With RUST_TASKS=1, this test is blazingly
fast (78ms), and with RUST_TASKS=2, its incredibly slow (3824ms). The reason
that I found for this is that the tasks being enqueued are constantly stolen by
other schedulers, meaning that tasks are just getting ping-ponged back and forth
around schedulers while the schedulers spend *a lot* of time in `kevent` and
`write` waking each other up.

This optimization only wakes up a sleeping scheduler on every 8th task that is
enqueued. I have found this number to be the "low sweet spot" for maximizing
performance. The numbers after I made this change are:

1 green thread: 13.96ms
8 green threads: 80.86ms
1 native thread: 13.59ms
8 native threads: 4239.25ms

Which indicates that the 8-thread performance is up to the same level of
RUST_TASKS=1, and the other numbers essentiallyt stayed the same.

In other words, this is a 136x improvement in highly contentious green programs.