Merge pull request #16 from facebook/release

Release
[DWRF] Move compression classes to a separate package

Summary:
@public
same as title

Test Plan: ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1451554
[DWRF] Fix "Can't read header" bug in CompressedStream

Summary:
@public
There is a bug in the CompressedStream class where if the stream is initialized from a ByteBuffer (instead of a file) and seek is called before any data is read currentChunk would be set to 0.  Then if all the data is read from the stream and available() is called, it would throw a "Can't read header" error.  This is because available only returns 0 if all the data from the current compressed chunk has been read *and* the currentChunk is >= the number of chunks (in this case 1).

The fix I chose was to add a check in seek, if none of the data has been read and file is null (the data is coming from a ByteBuffer) don't change the value of currentChunk and don't call readData.  This means currentChunk will keep it's original value of 1.  readData immediately checks if file is null and returns so there's no real change there.

Test Plan: Added a test case which failed before the fix.

Reviewers: tejasp

Reviewed By: tejasp

Differential Revision: https://phabricator.fb.com/D1466979

Tasks: 4809068
[DWRF] Remove redundant stuff from TestFileDump

Summary:
@public
We already create conf object in @Before which is invoked before *every* individual test case so no need to have multiple objects

Test Plan: ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1450904

Tasks: 4738072
[DWRF] Use capacity based constructor for ArrayLists whenever possible

Summary:
@public
same as title

Test Plan: Ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1450437

Tasks: 4738072
[maven-release-plugin] prepare release 0.12
[maven-release-plugin] prepare for next development iteration
seekToRow in RecordReaderImpl does not update rowInStripe which can cause readers to read off the end of streams because a new stripe may not be read when the end of the stripe is reached.

Added a test case to demonstrate this.

Also, modified another test case to call seekToRow to skip <n> rows instead of calling seekToRow for each row regardless of whether the row is read or not.
Added documentation to RunLengthByte reader and writer. Did minor refac

Summary:
@public
Added documentation to RunLengthByte reader and writer. Did minor refac

Test Plan: Ran unit tests

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Differential Revision: https://phabricator.fb.com/D1427856
[DWRF] Add a testcase to make sure that setting offset in Reader.rows to a stripe offset will not skip that stripe.

Summary:
@public
I was a little nervous that this wouldn't work, fortunately it does, thought it'd be good to add a test case to make sure it continues to in the future.

The test case writes to stripes and reads them using Reader.rows setting offset and length to those of the respective stripes, and makes sure it reads the correct stripe and only the correct stripe.

Test Plan: The tests pass.

Reviewers: tejasp

Reviewed By: tejasp

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1427818
[DWRF] Changes needed for FileCat in Hive

Summary: Made OrcRecordReader public and exposed its ObjectInspector

Test Plan: Ran 'mvn clean install'

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1434624

Tasks: 4634790
[DWRF] Move compression classes to a separate package

Summary:
@public
same as title

Test Plan: ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1451554
[DWRF] Fix "Can't read header" bug in CompressedStream

Summary:
@public
There is a bug in the CompressedStream class where if the stream is initialized from a ByteBuffer (instead of a file) and seek is called before any data is read currentChunk would be set to 0.  Then if all the data is read from the stream and available() is called, it would throw a "Can't read header" error.  This is because available only returns 0 if all the data from the current compressed chunk has been read *and* the currentChunk is >= the number of chunks (in this case 1).

The fix I chose was to add a check in seek, if none of the data has been read and file is null (the data is coming from a ByteBuffer) don't change the value of currentChunk and don't call readData.  This means currentChunk will keep it's original value of 1.  readData immediately checks if file is null and returns so there's no real change there.

Test Plan: Added a test case which failed before the fix.

Reviewers: tejasp

Reviewed By: tejasp

Differential Revision: https://phabricator.fb.com/D1466979

Tasks: 4809068
[DWRF] Remove redundant stuff from TestFileDump

Summary:
@public
We already create conf object in @Before which is invoked before *every* individual test case so no need to have multiple objects

Test Plan: ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1450904

Tasks: 4738072
[DWRF] Use capacity based constructor for ArrayLists whenever possible

Summary:
@public
same as title

Test Plan: Ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1450437

Tasks: 4738072
[DWRF] addStripe() should copy the RawDataSize field of StripeInformation

Summary:
@public

I found that the RawDataSize field is always 0 in the stripes of a merged ORC file. Then I traced back to this function addStripe().
It seems to be a bug that it forgot to copy over the field RawDataSize, as I don't see why this is intentional. This function is called
from OrcMergeMapper.java in Hive.

Test Plan: Ran all the existing test cases and no new test cases added.

Reviewers: tejasp, kevinwilfong

Reviewed By: kevinwilfong

Subscribers: dilipj

Differential Revision: https://phabricator.fb.com/D1471921
[DWRF] Check memory on each row

Summary:
@public
One major source of OOM's is that we only check the memory usage once every 5000 rows.

The reason we check so infrequently is because to determine the amount of memory we're using requires an expensive traversal of the tree of writers.

I changed it so that as we use more memory we increment a counter, so that checking the memory usage is as cheap as checking the value of that counter.  This makes it so we can afford to check it on every row with minimal performance consequences.

This should help to significantly reduce the number of OOM's.

Test Plan:
The current testcases verify that memory is still being calculated appropriately, that stripes are getting flushed when they should be, and the MemoryManager is still performing as expected.

I also verified that at the 5000 row mark where we used to check memory, in each of the test cases the memory counters we're using now match the value we had before (accounting for a few of the changes I made in calculating memory).

Reviewers: tejasp

Reviewed By: tejasp

Differential Revision: https://phabricator.fb.com/D1476964
[maven-release-plugin] prepare release 0.13
[maven-release-plugin] prepare for next development iteration