ScaleUnlimited/cascading.avro — pull request #2 by edwardnewett (2013-08-21 20:22:16+00:00)
Title: Only include data from the valid range of a BytesWritable

This issue came up when I noticed that serializing the resulting avro included additional bytes at the end that increase every time the object is deserialized/updated/re-serialized.

I found that BytesWritable automatically resizes its internal buffer capacity to be 50% larger when set() is called, and the method getBytes() returns the entire buffer. From the Javadoc of BytesWritable.getBytes():
/**

Get the data from the BytesWritable.
@return The data is only valid between 0 and getLength() - 1. */
An example of where a BytesWritable gets set with additional capacity is in
org.apache.hadoop.madred.JobClient$RawSplit (at least in versions 0.20 and 1.03)

This pull request includes two tests that demonstrate the issue and the fix to only include the valid range.