kevinweil/elephant-bird — Issue #110: Hive queries failed with big files
Author: isaiah
Created: 2011-12-16 09:37:28+00:00
State: open

If the size of the lzo file is over 64m, hive queries will fail with the following exception (full stacktrace attached at the end):

java.io.IOException: Compressed length 1602367537 exceeds max block size 67108864 (probably corrupt file)

One may advise indexing the files, but seems the index is not recoginzsed, either. However, pig script works fine with or without index.

Full stacktrace:

java.io.IOException: Compressed length 1602367537 exceeds max block size 67108864 (probably corrupt file)
at com.hadoop.compression.lzo.LzopInputStream.getCompressedData(LzopInputStream.java:286)
at com.hadoop.compression.lzo.LzopInputStream.decompress(LzopInputStream.java:256)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:63)
at com.twitter.elephantbird.util.StreamSearcher.search(StreamSearcher.java:47)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.skipToNextSyncPoint(BinaryBlockReader.java:102)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.parseNextBlock(BinaryBlockReader.java:107)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.setupNewBlockIfNeeded(BinaryBlockReader.java:139)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.readNextProtoBytes(BinaryBlockReader.java:75)
at com.twitter.elephantbird.mapreduce.io.BinaryBlockReader.readNext(BinaryBlockReader.java:63)
at com.twitter.elephantbird.mapreduce.io.ProtobufBlockReader.readProtobuf(ProtobufBlockReader.java:42)
at com.twitter.elephantbird.mapred.input.DeprecatedLzoProtobufBlockRecordReader.next(DeprecatedLzoProtobufBlockRecordReader.java:85)
at com.twitter.elephantbird.mapred.input.DeprecatedLzoProtobufBlockRecordReader.next(DeprecatedLzoProtobufBlockRecordReader.java:28)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:98)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:42)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
at org.apache.hadoop.hive.shims.Hadoop20SShims$CombineFileRecordReader.next(Hadoop20SShims.java:208)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:208)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:193)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)