PARQUET-2: Adding Type Persuasion for Primitive Types

Original from the old repo: https://github.com/Parquet/parquet-mr/pull/410
JIRA: https://issues.apache.org/jira/browse/PARQUET-2

These changes allow primitive types to be requested as different types than what is stored in the file format using a flag to turn off strict type checking (default is on). Types are cast to the requested type where possible and will suffer precision loss for casting where necessary (e.g. requesting a double as an int).

No performance penalty is imposed for using the type defined in the file type.  A flag exists to

A 6x6 test case is provided to test conversion between the primitive types.

Author: Daniel Weeks <dweeks@netflix.com>

Closes #3 from dcw-netflix/type-persuasion and squashes the following commits:

97f4e9a [Daniel Weeks] Added documentation as suggested by code review
1c3c0c7 [Daniel Weeks] Fixed test with strict checking off
f3cb495 [Daniel Weeks] Added type persuasion for primitive types with a flag to control strict type checking for conflicting schemas, which is strict by default.
Minor fix

Spell and comment issue.

Author: WangTao <barneystinson@aliyun.com>

Closes #10 from WangTaoTheTonic/minorFix and squashes the following commits:

0727a8f [WangTao] Minor fix
PARQUET-6: Create documentation on how to contribute.

Author: Julien Le Dem <julien@twitter.com>

Closes #8 from julienledem/master and squashes the following commits:

aa0d751 [Julien Le Dem] correct mailling-list
285b39c [Julien Le Dem] Update CONTRIBUTING.md
bca9bc4 [Julien Le Dem] Create CONTRIBUTING.md
PARQUET-9: Filtering records across multiple blocks

Update of the minimal fix discussed in https://github.com/apache/incubator-parquet-mr/pull/1, with the recursive call changed to to a loop.

Author: Tom White <tom@cloudera.com>
Author: Steven Willis <swillis@compete.com>

Closes #9 from tomwhite/filtering-records-across-multiple-blocks and squashes the following commits:

afb08a4 [Tom White] Minimal fix
9e723ee [Steven Willis] Test for filtering records across multiple blocks
PARQUET-4: Use LRU caching for footers in ParquetInputFormat.

Reopening https://github.com/Parquet/parquet-mr/pull/403 against the new Apache repository.

Author: Matthieu Martin <ma.tt.b.ma.rt.in+parquet@gmail.com>

Closes #2 from matt-martin/master and squashes the following commits:

99bb5a3 [Matthieu Martin] Minor javadoc and whitespace changes. Also added the FileStatusWrapper class to ParquetInputFormat to make sure that the debugging log statements print out meaningful paths.
250a398 [Matthieu Martin] Be less aggressive about checking whether the underlying file has been appended to/overwritten/deleted in order to minimize the number of namenode interactions.
d946445 [Matthieu Martin] Add javadocs to parquet.hadoop.LruCache.  Rename cache "entries" as cache "values" to avoid confusion with java.util.Map.Entry (which contains key value pairs whereas our old "entries" really only refer to the values).
a363622 [Matthieu Martin] Use LRU caching for footers in ParquetInputFormat.
PARQUET-21: Fix reference to 'github-apache' in dev docs

Author: Tom White <tom@cloudera.com>

Closes #20 from tomwhite/git-remote-name-fix and squashes the following commits:

c7f8a18 [Tom White] Change 'github-apache' to 'apache-github' in docs to match script.
PARQUET-18: Fix all-null value pages with dict encoding.

TestDictionary#testZeroValues demonstrates the problem, where a page of
all null values is decoded using the DicitonaryValuesReader. Because
there are no non-null values, the page values section is 0 byte, but the
DictionaryValuesReader assumes there is at least one encoded value and
attempts to read a bit width. The test passes a byte array to
initFromPage with the offset equal to the array's length.

The fix is to detect that there are no input bytes to read. To avoid
adding validity checks to the read path, this sets the internal decoder
to one that will throw an exception if any reads are attempted.

Author: Ryan Blue <rblue@cloudera.com>

Closes #18 from rdblue/PARQUET-18-fix-nulls-with-dictionary and squashes the following commits:

0711766 [Ryan Blue] PARQUET-18: Fix all-null value pages with dict encoding.
PARQUET-22: Backport of HIVE-6938 adding rename support for parquet

This patch was included in hive after the moving the Serde to hive (included in hive 0.14+).  Backport is required for use with previous versions.

Author: Daniel Weeks <dweeks@netflix.com>

Closes #13 from dcw-netflix/backport-hive-6938-rename and squashes the following commits:

453367b [Daniel Weeks] Backport of HIVE-6938 adding rename support for parquet
PARQUET-25. Pushdown predicates only work with hardcoded arguments.

Pull request for Sandy Ryza's fix for PARQUET-25.

Author: Sandy Ryza <sandy.ryza@cloudera.com>

Closes #22 from tomwhite/PARQUET-25-unbound-record-filter-configurable and squashes the following commits:

a9d3fdc [Sandy Ryza] PARQUET-25. Pushdown predicates only work with hardcoded arguments.
Column index access support

This patch adds the ability to use column index based access to parquet files in pig, which allows for rename capability similar to other file formats.  This is achieved by using the parametrized loader with an alternate schema.

Example:
# File Schema: {c1:int, c2:float, c3:chararray}
p = LOAD '/data/parquet/' USING parquet.pig.ParquetLoader('n1:int, n2:float, n3:chararray', 'true');

In this example, the names from the requested schema will be translated to the column positions from the file and will produce tuples based on the index position.

Two test cases are included that exercise index based access for both full file reads and column projected reads.

Note:  This patch also disables the enforcer plugin on the pig project per discussion at the parquet meetup.  The justification for this is that the enforcer is too strict for internal classes and results in dead code because duplicating methods is required to add parameters where there is only one usage of the constructor/method.  The interface for the pig loader is imposed by LoadFunc and StoreFunc by the pig project and the implementations internals should not be used directly.

Author: Daniel Weeks <dweeks@netflix.com>

Closes #12 from dcw-netflix/column-index-access and squashes the following commits:

1b5c5cf [Daniel Weeks] Refactored based on rewview comments
12b53c1 [Daniel Weeks] Fixed some formatting and the missing filter method sig
e5553f1 [Daniel Weeks] Adding back default constructor to satisfy other project requirements
69d21e0 [Daniel Weeks] Merge branch 'master' into column-index-access
f725c6f [Daniel Weeks] Removed enforcer for pig support
d182dc6 [Daniel Weeks] Introduces column index access
1c3c0c7 [Daniel Weeks] Fixed test with strict checking off
f3cb495 [Daniel Weeks] Added type persuasion for primitive types with a flag to control strict type checking for conflicting schemas, which is strict by default.
PARQUET-19: Fix NPE when an empty file is included in a Hive query that uses CombineHiveInputFormat

Make sure the valueObj instance variable is always initialized.  This change is neeeded when running a Hive query that uses the CombineHiveInputFormat and the first file in the combined split is empty.  This can lead to a NullPointerException because the valueObj is null when the CombineHiveInputFormat calls the createValue method.

Author: Matthieu Martin <ma.tt.b.ma.rt.in+parquet@gmail.com>

Closes #19 from matt-martin/fix_for_empty_files_NPE_with_CombineHiveInputFormat and squashes the following commits:

6c3a7f5 [Matthieu Martin] Make sure the valueObj instance variable is always initialized.  This change is neeeded when running a Hive query that uses the CombineHiveInputFormat and the first file in the combined split is empty.  This can lead to a NullPointerException because the valueObj is null when the CombineHiveInputFormat calls the createValue method.
Add a unified and optionally more constrained API for expressing filters on columns

This is a re-opened version of:
https://github.com/Parquet/parquet-mr/pull/412

The idea behind this pull request is to add a way to express filters on columns using DSL that allows parquet visibility into what is being filtered and how. This visibility will allow us to make optimizations at read time, the biggest one being filtering entire row groups or pages of records without even reading them based on the statistics / metadata that is stored along with each row group or page.

Included in this api are interfaces for user defined predicates, which must operate at the value level by may opt in to operating at the row group / page level as well. This should make this new API a superset of the `parquet.filter` package. This new api will need to be reconciled with the column filters currently in the `parquet.filter` package, but I wanted to get feedback on this first.

A limitation in both this api and the old one is that you can't do cross-column filters, eg: columX > columnY.

Author: Alex Levenson <alexlevenson@twitter.com>

Closes #4 from isnotinvain/alexlevenson/filter-api and squashes the following commits:

c1ab7e3 [Alex Levenson] Address feedback
c1bd610 [Alex Levenson] cleanup dotString in ColumnPath
418bfc1 [Alex Levenson] Update version, add temporary hacks for semantic enforcer
6643bd3 [Alex Levenson] Fix some more non backward incompatible changes
39f977f [Alex Levenson] Put a bunch of backwards compatible stuff back in, add @Deprecated
13a02c6 [Alex Levenson] Fix compile errors, add back in overloaded getRecordReader
f82edb7 [Alex Levenson] Merge branch 'master' into alexlevenson/filter-api
9bd014f [Alex Levenson] clean up TODOs and reference jiras
4cc7e87 [Alex Levenson] Add some comments
30e3d61 [Alex Levenson] Create a common interface for both kinds of filters
ac153a6 [Alex Levenson] Create a Statistics class for use in UDPs
fbbf601 [Alex Levenson] refactor IncrementallyUpdatedFilterPredicateGenerator to only generate the parts that require generation
5df47cd [Alex Levenson] Static imports of checkNotNull
c1d1823 [Alex Levenson] address some of the minor feedback items
67a3ba0 [Alex Levenson] update binary's toString
3d7372b [Alex Levenson] minor fixes
fed9531 [Alex Levenson] Add skipCurrentRecord method to clear events in thrift converter
2e632d5 [Alex Levenson] Make Binary Serializable
09c024f [Alex Levenson] update comments
3169849 [Alex Levenson] fix compilation error
0185030 [Alex Levenson] Add integration test for value level filters
4fde18c [Alex Levenson] move to right package
ae36b37 [Alex Levenson] Handle merge issues
af69486 [Alex Levenson] Merge branch 'master' into alexlevenson/filter-api
0665271 [Alex Levenson] Add tests for value inspector
c5e3b07 [Alex Levenson] Add tests for resetter and evaluator
29f677a [Alex Levenson] Fix scala DSL
8897a28 [Alex Levenson] Fix some tests
b448bee [Alex Levenson] Fix mistake in MessageColumnIO
c8133f8 [Alex Levenson] Fix some tests
4cf686d [Alex Levenson] more null checks
69e683b [Alex Levenson] check all the nulls
220a682 [Alex Levenson] more cleanup
aad5af3 [Alex Levenson] rm generated src file from git
5075243 [Alex Levenson] more minor cleanup
9966713 [Alex Levenson] Hook generation into maven build
8282725 [Alex Levenson] minor cleanup
fea3ea9 [Alex Levenson] minor cleanup
9e35406 [Alex Levenson] move statistics filter
c52750c [Alex Levenson] finish moving things around
97a6bfd [Alex Levenson] Move things around pt2
843b9fe [Alex Levenson] Move some files around pt 1
5eedcc0 [Alex Levenson] turn off dictionary support for AtomicConverter
541319e [Alex Levenson] various cleanup and fixes
08e9638 [Alex Levenson] rm ColumnPathUtil
bfe6795 [Alex Levenson] Add type bounds to FilterApi
6c831ab [Alex Levenson] don't double log exception in SerializationUtil
a7a58d1 [Alex Levenson] use ColumnPath instead of String
8f11a6b [Alex Levenson] Move ColumnPath and Canonicalizer to parquet-common
9164359 [Alex Levenson] stash
abc2be2 [Alex Levenson] Add null handling to record filters -- this impl is still broken though
90ba8f7 [Alex Levenson] Update Serialization Util
0a261f1 [Alex Levenson] Add compression in SerializationUtil
f1278be [Alex Levenson] Add comment, fix tests
cbd1a85 [Alex Levenson] Replace some specialization with generic views
e496cbf [Alex Levenson] Fix short circuiting in StatisticsFilter
db6b32d [Alex Levenson] Address some comments, fix constructor in ParquetReader
fd6f44d [Alex Levenson] Fix semver backward compat
2fdd304 [Alex Levenson] Some more cleanup
d34fb89 [Alex Levenson] Cleanup some TODOs
544499c [Alex Levenson] stash
7b32016 [Alex Levenson] Merge branch 'master' into alexlevenson/filter-api
0e31251 [Alex Levenson] First pass at values filter, needs reworking
470e409 [Alex Levenson] fix java6/7 bug, minor cleanup
ee7b221 [Alex Levenson] more InputFormat tests
5ef849e [Alex Levenson] Add guards for not specifying both kinds of filter
0186b1f [Alex Levenson] Add logging to ParquetInputFormat and tests for configuration
a622648 [Alex Levenson] cleanup imports
9b1ea88 [Alex Levenson] Add tests for statistics filter
d517373 [Alex Levenson] tests for filter validator
b25fc44 [Alex Levenson] small cleanup of filter validator
32067a1 [Alex Levenson] add test for collapse logical nots
1efc198 [Alex Levenson] Add tests for invert filter predicate
046b106 [Alex Levenson] some more fixes
d3c4d7a [Alex Levenson] fix some more types, add in test for SerializationUtil
cc51274 [Alex Levenson] fix generics in FilterPredicateInverter
ea08349 [Alex Levenson] First pass at rowgroup filter, needs testing
156d91b [Alex Levenson] Add runtime type checker
4dfb4f2 [Alex Levenson] Add serialization util
8f80b20 [Alex Levenson] update comment
7c25121 [Alex Levenson] Add class to Column struct
58f1190 [Alex Levenson] Remove filterByUniqueValues
7f20de6 [Alex Levenson] rename user predicates
af14b42 [Alex Levenson] Update dsl
04409c5 [Alex Levenson] Add generic types into Visitor
ba42884 [Alex Levenson] rm getClassName
65f8af9 [Alex Levenson] Add in support for user defined predicates on columns
6926337 [Alex Levenson] Add explicit tokens for notEq, ltEq, gtEq
667ec9f [Alex Levenson] remove test for collapsing double negation
db2f71a [Alex Levenson] rename FilterPredicatesTest
a0a0533 [Alex Levenson] Address first round of comments
b2bca94 [Alex Levenson] Add scala DSL and tests
bedda87 [Alex Levenson] Add tests for FilterPredicate building
238cbbe [Alex Levenson] Add scala dsl
39f7b24 [Alex Levenson] add scala mvn boilerplate
2ec71a7 [Alex Levenson] Add predicate API
Only call put() when needed in SchemaCompatibilityValidator#validateColumn()

This is some minor cleanup suggested by @tsdeng

Author: Alex Levenson <alexlevenson@twitter.com>

Closes #24 from isnotinvain/alexlevenson/columnTypesEncountered and squashes the following commits:

7f05d90 [Alex Levenson] Only call put() when needed in SchemaCompatibilityValidator#validateColumn()
PARQUET-56: Added an accessor for the Long column type.

I noticed there was a missing accessor for the Long column type in the example Group.

Author: James Scott <jim.scott@urbanairship.com>

Closes #25 from scottjab/getLong and squashes the following commits:

f96bb83 [James Scott] Added support for getting Longs in the sample group object.
PARQUET-57 - Update dev README to clarify two points

Author: Brock Noland <brock@apache.org>

Closes #26 from brockn/PARQUET-57 and squashes the following commits:

ffb2a9a [Brock Noland] Prompt for jira user/pass if not set
0135cdd [Brock Noland] Fix spelling
171d714 [Brock Noland] PARQUET-57 - Update dev README to clarify two points
PARQUET-13: The `-d` option for `parquet-schema` shouldn't have optional argument

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #11 from liancheng/fix-cli-arg and squashes the following commits:

85a5453 [Cheng Lian] Reverted the dummy change
47ce817 [Cheng Lian] Dummy change to trigger Travis
1c0a244 [Cheng Lian] The `-d` option for `parquet-schema` shouldn't have optional argument
PARQUET-59: Fix parquet-scrooge test on hadoop-2.

Author: Ryan Blue <rblue@cloudera.com>

Closes #27 from rdblue/PARQUET-59-fix-scrooge-test-on-hadoop-2 and squashes the following commits:

ac34369 [Ryan Blue] PARQUET-59: Fix parquet-scrooge test on hadoop-2.
[maven-release-plugin] prepare release parquet-1.6.0rc1
[maven-release-plugin] prepare for next development iteration
PARQUET-73: Add support for FilterPredicates to cascading schemes

Author: Alex Levenson <alexlevenson@twitter.com>

Closes #34 from isnotinvain/alexlevenson/filter-cascading-scheme and squashes the following commits:

cd69a8e [Alex Levenson] Add support for FilterPredicates to cascading schemes