seekToRow in RecordReaderImpl does not update rowInStripe which can cause readers to read off the end of streams because a new stripe may not be read when the end of the stripe is reached.

Added a test case to demonstrate this.

Also, modified another test case to call seekToRow to skip <n> rows instead of calling seekToRow for each row regardless of whether the row is read or not.
Added documentation to RunLengthByte reader and writer. Did minor refac

Summary:
@public
Added documentation to RunLengthByte reader and writer. Did minor refac

Test Plan: Ran unit tests

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Differential Revision: https://phabricator.fb.com/D1427856
[DWRF] Add a testcase to make sure that setting offset in Reader.rows to a stripe offset will not skip that stripe.

Summary:
@public
I was a little nervous that this wouldn't work, fortunately it does, thought it'd be good to add a test case to make sure it continues to in the future.

The test case writes to stripes and reads them using Reader.rows setting offset and length to those of the respective stripes, and makes sure it reads the correct stripe and only the correct stripe.

Test Plan: The tests pass.

Reviewers: tejasp

Reviewed By: tejasp

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1427818
[DWRF] Changes needed for FileCat in Hive

Summary: Made OrcRecordReader public and exposed its ObjectInspector

Test Plan: Ran 'mvn clean install'

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1434624

Tasks: 4634790
[DWRF] Move compression classes to a separate package

Summary:
@public
same as title

Test Plan: ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1451554
[DWRF] Fix "Can't read header" bug in CompressedStream

Summary:
@public
There is a bug in the CompressedStream class where if the stream is initialized from a ByteBuffer (instead of a file) and seek is called before any data is read currentChunk would be set to 0.  Then if all the data is read from the stream and available() is called, it would throw a "Can't read header" error.  This is because available only returns 0 if all the data from the current compressed chunk has been read *and* the currentChunk is >= the number of chunks (in this case 1).

The fix I chose was to add a check in seek, if none of the data has been read and file is null (the data is coming from a ByteBuffer) don't change the value of currentChunk and don't call readData.  This means currentChunk will keep it's original value of 1.  readData immediately checks if file is null and returns so there's no real change there.

Test Plan: Added a test case which failed before the fix.

Reviewers: tejasp

Reviewed By: tejasp

Differential Revision: https://phabricator.fb.com/D1466979

Tasks: 4809068
[DWRF] Remove redundant stuff from TestFileDump

Summary:
@public
We already create conf object in @Before which is invoked before *every* individual test case so no need to have multiple objects

Test Plan: ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1450904

Tasks: 4738072
[DWRF] Use capacity based constructor for ArrayLists whenever possible

Summary:
@public
same as title

Test Plan: Ran test cases

Reviewers: kevinwilfong

Reviewed By: kevinwilfong

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1450437

Tasks: 4738072
[DWRF] addStripe() should copy the RawDataSize field of StripeInformation

Summary:
@public

I found that the RawDataSize field is always 0 in the stripes of a merged ORC file. Then I traced back to this function addStripe().
It seems to be a bug that it forgot to copy over the field RawDataSize, as I don't see why this is intentional. This function is called
from OrcMergeMapper.java in Hive.

Test Plan: Ran all the existing test cases and no new test cases added.

Reviewers: tejasp, kevinwilfong

Reviewed By: kevinwilfong

Subscribers: dilipj

Differential Revision: https://phabricator.fb.com/D1471921
[DWRF] Check memory on each row

Summary:
@public
One major source of OOM's is that we only check the memory usage once every 5000 rows.

The reason we check so infrequently is because to determine the amount of memory we're using requires an expensive traversal of the tree of writers.

I changed it so that as we use more memory we increment a counter, so that checking the memory usage is as cheap as checking the value of that counter.  This makes it so we can afford to check it on every row with minimal performance consequences.

This should help to significantly reduce the number of OOM's.

Test Plan:
The current testcases verify that memory is still being calculated appropriately, that stripes are getting flushed when they should be, and the MemoryManager is still performing as expected.

I also verified that at the 5000 row mark where we used to check memory, in each of the test cases the memory counters we're using now match the value we had before (accounting for a few of the changes I made in calculating memory).

Reviewers: tejasp

Reviewed By: tejasp

Differential Revision: https://phabricator.fb.com/D1476964
[DWRF] Add some additional logging to debug NPE

Summary:
@public
Adding a log message when a writer is added to the MemoryManager, and one when it is removed.  This should be low overhead because it only adds two logs per writer (per file not per row).

Also, we throw a clearer error message instead of an NPE in shouldFlush, which should also help with debugging.  This *should* be false all the time so it should also be low overhead.

Test Plan: Ran the tests, verified the new log messages appear.

Reviewers: tejasp

Reviewed By: tejasp

Subscribers: edunov

Differential Revision: https://phabricator.fb.com/D1508828

Tasks: 4923088
[DWRF] MemoryManager to use ConcurrentHashMap

Summary:
@public

Multithreaded clients are running into issues with the MemoryManager which seems to be due to the face we are not using a thread safe HashMap, fixing that.

Test Plan: Ran one of the multi threaded clients on it successfully, unit tests pass.

Reviewers: tejasp

Reviewed By: tejasp

Subscribers: edunov, majakabiljo

Differential Revision: https://phabricator.fb.com/D1512434
[DWRF] Better exception when a non ORC file is attempted to be read as ORC

Summary:
@public
When a non ORC file is read by ORC reader, we get IndexOutOfBoundsException exception while creating a reader. Caught that exception and checked the file header to see if the input file was ORC or not. If its not ORC, throw a NotAnORCFileException with the file attempted to be reading (thus helping to figure out which table-partition was being read.

Test Plan: TBD

Reviewers: dheeraj, jingweil, dilipj

Reviewed By: dilipj

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1560431

Tasks: 4690285
[DWRF] Better comment for the phantom issue

Summary:
@public
Adding more details about the phantom problem as per suggestion by Kevin Wilfing in prev. patch

Test Plan: None. This is simply adding a comment.

Reviewers: dilipj, kevinwilfong

Reviewed By: dilipj

Subscribers: dilipj, hive-devel@

Differential Revision: https://phabricator.fb.com/D1564629

Tasks: 4690285
[DWRF] Display the full path to file for which the record reader creation failed

Summary:
@public
Would give better stack traces

Test Plan: Ran on a test table

Reviewers: dilipj, jingweil, dheeraj

Reviewed By: dheeraj

Subscribers: hive-devel@

Differential Revision: https://phabricator.fb.com/D1597549

Tasks: 5309581