twitter/hadoop-lzo — pull request #94 by themodernlife (2014-09-25 15:09:48+00:00)
Title: Demonstrate race condition to discuss how to fix #91

I was kind of glad to see https://github.com/twitter/hadoop-lzo/issues/91, because I too have ran into this issue while processing LZO files (from S3) via Spark using local config with multiple threads.

I dug into this and the issue seems to be in https://github.com/twitter/hadoop-lzo/blob/e8c11c2be93b965abb548411379b203dabcbce79/src/main/java/com/hadoop/compression/lzo/LzopInputStream.java#L349.

When using Spark, the exception occurs because the `TextInputFormat` creates a `LineRecordReader` that loads the decompressor from the pool. When it gets closed, the `LineRecordReader` adds it back to the pool... the trouble is that underlying `LzopInputStream` *also* adds the decompressor back to the pool, which means the *same* decompressor is now in the pool twice.

This means that the next 2 calls to `getRecordReader` are going to use the same decompressor, and that's where you get the race condition.

I attached a unit test that demonstrates the error. The fix here is just to remove the bit that adds the decompressor back to the pool on 349.

Can someone have a look at this as well and let me know what you think? I can clean up the PR and resubmit, but just wanted to add some info about the bug.

Cheers!