kevinweil/hadoop-lzo — Issue #41: Very slow compression of uncompressable data
Author: Shmuma
Created: 2012-01-12 14:21:06+00:00
State: open

I performed a small benchmark of LzoCodec (benchmark hadoop job is here: https://gist.github.com/1600759).

The result is quite suprising: LZO is more than 3 times slower on random data compared to plain storage or Snappy compressor.

On our cluster I get these numbers:
- without compression: 122 seconds
- Snappy: 126 seconds
- LZO: 390 seconds

I use latest revision of hadoop-lzo and lzo 2.04. Do you have any suggestion how to fix that?