amplab/shark — pull request #329 by chiragaggarwal (2014-05-06 11:06:12+00:00)
Title: unnecessary serialize/deserialize for cases where no conversion was performed

If partition schema does not match table schema, the row (formed by deserializing through partition serde) is converted to match the table schema. If conversion was performed, convertedRow will be a standard Object, but if conversion wasn't necessary, it will still be lazy.

We can't have both (standard and lazy objects) across partitions, so we serialize and deserialize again to make it lazy.

This extra serialize/deserialize is being performed irrespective of the fact that whether conversion was done or not.

There are two effects of this serialization / deserialization:

1) Extra serialization / deserilization cost for cases, where no conversion happened.

2) If a table is created using ThriftDeserializer, the non-availability of serialize function in it makes it unusable in this context.

The fix done is that in case conversion was not done (when partition and table serde match), then this serialization and deserialization step should be skipped, since it is not required as the object would still be lazy. This shall also allow users to be able to use ThriftDeserializer in such a case.