kevinweil/elephant-bird — pull request #239 by rangadi (2012-08-09 06:16:58+00:00)
Title: extend TDeserializer to set read-size for TBinaryProtocol

corrupt records are handled more gracefully.

Before this, corrupt records could result in OutOfMemoryError since the Thrift first tries to allocate random number of bytes before realizing there is not that much data left.

e.g. exceptions :

before

com.twitter.elephantbird.mapreduce.io.ThriftConverter: failed to deserialize org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 1 bytes, but only got 0 bytes.
...
com.twitter.elephantbird.mapreduce.input.LzoRecordReader: Error while reading an input record (4 out of 7674197 so far ): java.lang.OutOfMemoryError: Java heap space

after

[first one is same as before. second one is TException rather than OutOfMemoryError]

com.twitter.elephantbird.mapreduce.io.ThriftConverter: failed to deserialize org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 1 bytes, but only got 0 bytes.
...
com.twitter.elephantbird.mapreduce.io.ThriftConverter: failed to deserialize org.apache.thrift.TException: Message length exceeded: 1925492992