ning/compress — Issue #14: Parallel LZF
Author: whoschek
Created: 2011-11-14 22:40:43+00:00
State: open

Hi Tatu,

I think using wide instructions via Unsafe looks very promising for decompression. I was wondering to what extent this might also be applied to compression. Compression is becoming more and more the bottleneck.

I am also thinking it might be good to use the plentiful cores available on today's production systems. Here we typically use servers with 12 cores each (2 sockets with 6 cores each). It's a bit of a pity to see only one out of 12 cores being used for compression. When the 20 core Intel CPUs (2 sockets with 10 cores each) become cheaper folks will switch to those. Using many cores for compression might work very well. Decompression is already so fast that it might not benefit as much.

Here, we are already using pigz - a parallel gzip implementation (really: drop-in replacement for gzip) for some things like backup - http://www.zlib.net/pigz/. Using this with many cores the I/O subsystem becomes the bottleneck even with gzip compression, which is, as you know, computationally much more demanding than LZF compression.

Here is an idea for your project. Perhaps ning compress could have one reader thread, one writer thread, and N worker threads. The reader feeds a queue of buffers. The worker threads grab a buffer from the queue, compress it (or uncompress it) and hand an output buffer to another queue where a writer thread grabs it, serializes the order in which output buffers are written (PriorityQueue), and then writes the logically next buffer to the output stream once that logically next buffer becomes available. Each buffer could contain an LZF chunk. To keep the CPUs busy the reader queue might maintain a read-ahead of 2 buffers for each of the N worker threads. Once the output buffer is written it is recycled via a buffer pool or queue (no gc). Same recycling for input buffers. Perhaps N=1 could be special cased to use no threads at all.

The pigz C source code available at http://www.zlib.net/pigz/ contains good comments in this direction.

Regards,
Wolfgang.