OpenImageIO/oiio — pull request #589 by lgritz (2013-05-04 18:18:29+00:00)
Title: spinlock tweaks that finally make it as good or better than TBB.

I dove really deep into the code and finally found the reason why (in particular, on my work machines which are Linux 64 bit gcc 4.4) my own spin locks continued to be not quite as good as TBB. Actually, it was worse... for many threads, mine were faster, but for <= 4 threads, TBB was faster, and of course those are important cases:

threads TBB No TBB
1 0.4 0.9
2 0.5 1.1
4 1.6 2.2
8 5.6 4.8
12 10.3 5.9
16 8.9 6.2
20 8.0 6.1
24 7.2 5.2
28 6.9 5.7
32 7.1 5.7

After a LOT of experimentation, I was able to narrow it down to one teeny thing... the gcc intrinsic __sync_lock_release (&m_locked) isn't anywhere near as efficient as "by hand" having a release barrier and then an assignment. I'm not sure why. I certainly don't care why. But there it is:

No TBB
threads TBB No TBB new
1 0.4 0.9 0.4
2 0.5 1.1 0.5
4 1.6 2.2 1.5
8 5.6 4.8 3.2
12 10.3 5.9 4.2
16 8.9 6.2 4.3
20 8.0 6.1 4.2
24 7.2 5.2 4.0
28 6.9 5.7 3.6
32 7.1 5.7 4.0

Crazy, no? Our home-spun spinlock is now as fast as TBB for <= 4 threads, and handily beats it for >= 8 threads. (It's a 12 core machine, incidentally, which is why performance flattens out at some point, but I still like to time it over-threaded.)

I also did some other cleanups, like removing the unused (badly performing) apple special cases and some other things that made debugging easier for me.

I'm really only able to test on 64 bit Linux and OSX.

So, dear readers, please please do me the favor of downloading this patch and doing the following on whatever platforms you have handy:

make nuke ; make USE_TBB=1
build/ARCH/libOpenImageIO/spinlock_test --wedge --threads 32 --trials 5
# clip the resulting output
make nuke ; make USE_TBB=0
build/ARCH/libOpenImageIO/spinlock_test --wedge --threads 32 --trials 5
# clip the resulting output

And post the timings here.

If it seems to hold for everybody, across various platforms and compilers (and I'm eager to especially hear about Windows [all flavors] and 32-bit Linux, none of which I'm unable to test myself), then I'll augment the patch to get rid of the remaining TBB code entirely, as we'll no longer want or need it.