open-mpi/ompi — pull request #312 by hjelmn (2014-12-09 22:45:42+00:00)
Title: Introduce call to idle a thread for a short period of time.

Good idea: when hitting contention on the lifo or fifo idle the thread.

Bad idea: use sched_yield to idle a thread.

This commit adds a new abstraction to the opal thread interface:
opal_thread_idle (). This call uses nanosleep to sleep the calling thread
for a very short amount of time (currently 5 ns). Using this function
when contention is detected in either the opal_lifo_t or opal_fifo_t
greatly improves threading performance.

Ex opal_lifo threading test (tested on a Retina Macbook Pro running 10.9.5):

w/ opal_thread_yield

pn1246003% ./opal_fifo
Single thread test. Time: 0 s 2864 us 2 nsec/poppush
Atomics thread finished. Time: 0 s 20909 us 20 nsec/poppush
Atomics thread finished. Time: 0 s 175782 us 175 nsec/poppush
Atomics thread finished. Time: 0 s 191908 us 191 nsec/poppush
Atomics thread finished. Time: 0 s 192573 us 192 nsec/poppush
Atomics thread finished. Time: 0 s 193216 us 193 nsec/poppush
Atomics thread finished. Time: 0 s 195594 us 195 nsec/poppush
Atomics thread finished. Time: 0 s 195880 us 195 nsec/poppush
Atomics thread finished. Time: 0 s 197659 us 197 nsec/poppush
Atomics thread finished. Time: 0 s 198213 us 198 nsec/poppush
All threads finished. Thread count: 8 Time: 0 s 198343 us 24 nsec/poppush
Exhaustive atomics thread finished. Popped 649662 items. Time: 0 s 174389 us 268 nsec/poppush
Exhaustive atomics thread finished. Popped 670632 items. Time: 0 s 175257 us 261 nsec/poppush
Exhaustive atomics thread finished. Popped 692673 items. Time: 0 s 181763 us 262 nsec/poppush
Exhaustive atomics thread finished. Popped 681356 items. Time: 0 s 185193 us 271 nsec/poppush
Exhaustive atomics thread finished. Popped 663348 items. Time: 0 s 185895 us 280 nsec/poppush
Exhaustive atomics thread finished. Popped 695761 items. Time: 0 s 187089 us 268 nsec/poppush
Exhaustive atomics thread finished. Popped 712464 items. Time: 0 s 187513 us 263 nsec/poppush
Exhaustive atomics thread finished. Popped 712244 items. Time: 0 s 188061 us 264 nsec/poppush
All threads finished. Thread count: 8 Time: 0 s 188164 us 23 nsec/poppush
SUPPORT: OMPI Test Passed: opal_fifo_t: (8 tests)

w/out opal_thread_yield:

Single thread test. Time: 0 s 2850 us 2 nsec/poppush
Atomics thread finished. Time: 0 s 21451 us 21 nsec/poppush
Atomics thread finished. Time: 1 s 448048 us 1448 nsec/poppush
Atomics thread finished. Time: 1 s 453196 us 1453 nsec/poppush
Atomics thread finished. Time: 1 s 458207 us 1458 nsec/poppush
Atomics thread finished. Time: 1 s 460438 us 1460 nsec/poppush
Atomics thread finished. Time: 1 s 466642 us 1466 nsec/poppush
Atomics thread finished. Time: 1 s 467402 us 1467 nsec/poppush
Atomics thread finished. Time: 1 s 467713 us 1467 nsec/poppush
Atomics thread finished. Time: 1 s 467856 us 1467 nsec/poppush
All threads finished. Thread count: 8 Time: 1 s 468034 us 183 nsec/poppush
Exhaustive atomics thread finished. Popped 616810 items. Time: 1 s 40889 us 1687 nsec/poppush
Exhaustive atomics thread finished. Popped 617083 items. Time: 1 s 41604 us 1687 nsec/poppush
Exhaustive atomics thread finished. Popped 617736 items. Time: 1 s 53726 us 1705 nsec/poppush
Exhaustive atomics thread finished. Popped 614711 items. Time: 1 s 63529 us 1730 nsec/poppush
Exhaustive atomics thread finished. Popped 612983 items. Time: 1 s 64248 us 1736 nsec/poppush
Exhaustive atomics thread finished. Popped 615214 items. Time: 1 s 65507 us 1731 nsec/poppush
Exhaustive atomics thread finished. Popped 615318 items. Time: 1 s 66123 us 1732 nsec/poppush
Exhaustive atomics thread finished. Popped 619068 items. Time: 1 s 66113 us 1722 nsec/poppush
All threads finished. Thread count: 8 Time: 1 s 66253 us 133 nsec/poppush
SUPPORT: OMPI Test Passed: opal_fifo_t: (8 tests)

I see similar performance benefits under Linux with kernel 3.13. I will
also test kernel 2.6.32.