cloudera/cdk — pull request #31 by prazanna (2014-01-17 01:00:10+00:00)
Title: ReadRCFileBuilder used to work only with FSDataInputstream now works with Generic java.io.InputStream

Issue: https://issues.cloudera.org/browse/CDK-282

Changes from previous Logic.

Earlier: I made a full pass on the Inputstream to emit all the rows for a given column and re-made passes for every column. This made sense to think of column-wise reading would emit all rows for column before moving on to the next column. This required Backward seeking which means I cannot work with Generic java.io.InputStream

Now: RC File is read in blocks (row-splits) and for each required column, all row values in that split is emitted in one record. This would mean I get one bunch of row values for column1, column2 and then a bunch of row values for column1,column2 again.

The documentation has been updated.

Let me know if anything is unclear. I am a little concerned about have a List in memory, but RCFile.Reader anyways has the entire block in-memory. Any other concerns please let me know.

Thanks
Prasanna