kevinweil/elephant-bird — Issue #160: Why is Elephant-Bird offering its own LZO Input/Output Formats?
Author: QwertyManiac
Created: 2012-03-06 05:46:30+00:00
State: open

Kevin's project "hadoop-lzo" (Which EB depends on anyway) already offers LZO I/O formats for Text, etc. - just not protobuf.

Why does EB choose to maintain a forked copy?

Would it be acceptable if I contributed a patch that makes EB classes extend the hadoop-lzo one (with tests to ensure we not break users of it)?

Some users of EB classes reported that it does not properly gel with Hive's CombineHiveInputFormat usage (which is now the default I/F).

For the same, simple "README.lzo" text file input that spans four blocks (worth 512 bytes each):

hive> describe extended lzoblocks;
OK
r	string

Detailed Table Information	Table(tableName:lzoblocks, dbName:default, owner:root, createTime:1329727318, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:r, type:string, comment:null)], location:hdfs://localhost/user/hive/warehouse/lzoblocks, inputFormat:com.hadoop.mapred.DeprecatedLzoTextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}), partitionKeys:[], parameters:{transient_lastDdlTime=1329727393}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
Time taken: 0.078 seconds

hive> describe extended ebblocks;
OK
r	string

Detailed Table Information	Table(tableName:ebblocks, dbName:default, owner:root, createTime:1329730707, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:r, type:string, comment:null)], location:hdfs://localhost/user/hive/warehouse/ebblocks, inputFormat:com.twitter.elephantbird.mapred.input.DeprecatedLzoTextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}), partitionKeys:[], parameters:{transient_lastDdlTime=1329732671}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
Time taken: 0.07 seconds

Doing a simple select of the hadoop-lzo class based table works fine, while the latter fails - when used with CDH3u3. The error the ebblocks table gets on a select * is [0].

This may be MAPREDUCE-1597 but I still think EB should be reusing classes rather than duplicating code and diverging in behavior.

[0] - 2012-02-17 11:04:07,598 WARN com.hadoop.compression.lzo.LzopInputStream: IOException in getCompressedData; likely LZO corruption.
java.io.IOException: Compressed length 439750499 exceeds max block size 67108864 (probably corrupt file)
at com.hadoop.compression.lzo.LzopInputStream.getCompressedData(LzopInputStream.java:285)
at com.hadoop.compression.lzo.LzopInputStream.decompress(LzopInputStream.java:255)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:63)
at com.twitter.elephantbird.util.StreamSearcher.search(StreamSearcher.java:47)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.skipToNextSyncPoint(BinaryBlockReader.java:102)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.parseNextBlock(BinaryBlockReader.java:107)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.setupNewBlockIfNeeded(BinaryBlockReader.java:139)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.readNextProtoBytes(BinaryBlockReader.java:75)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.readNext(BinaryBlockReader.java:63)
at com.twitter.elephantbird.mapreduce.io.ProtobufBlockReader.readProtobuf(ProtobufBlockReader.java:42)
at com.twitter.elephantbird.mapred.input.DeprecatedLzoProtobufBlockRecordReader.next(DeprecatedLzoProtobufBlockRecordReader.java:84)
at com.twitter.elephantbird.mapred.input.DeprecatedLzoProtobufBlockRecordReader.next(DeprecatedLzoProtobufBlockRecordReader.java:27)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:98)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:42)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
at org.apache.hadoop.hive.shims.Hadoop20SShims$CombineFileRecordReader.next(Hadoop20SShims.java:208)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:208)