IMPALA-913: Revisit the use of FNV Hash in exchange

FNV hash has the property that the least significant bit of the hashed value
is just the XOR of the LSBs of its input bytes. This results in poor
distribution of rows when the partition keys are duplicated -- for example,
if the partition key is (l_orderkey, l_orderkey). A recommended technique
to mitigate this is to generate a larger hash and use XOR-folding to reduce
it to the desired length.

In this patch FnvHash has been modified to use generate a 64-bit hash and
fold the result down to 32-bits. It has been renamed FnvHash64to32 to make
this explicit.

Change-Id: Ie12ad3f863fca15092803d3e4d616a654cb8d244
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2101
Reviewed-by: Srinath Shankar <sshankar@cloudera.com>
Tested-by: jenkins
Adding TRUNC builtin.

Includes additions to builtin UDF registration to support prepare/close.

Change-Id: I22668fa7ee033b3fa37050b7bccee935571ac453
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2141
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Decimal implementation.

This patch implements decimal support for text based formats.

Change-Id: I8e2c9e512ed149fe965216a72cb21fffd4f18e75
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1669
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
Close all queries in test_cancellation

The queries in test_cancellation are currently cancelled but not closed, causing some test
queries to eventually time out because the admission controller limits are passed. This
patch ensures that all queries issued in test_cancellation are closed.

Change-Id: I65b26672155e31889bb6f43d3ac87be0f7b4eb72
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2187
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Dynamically scale down mem usage in scanners and io mgr.

This patch scales down the amount of buffering in the io mgr and the number
of scanner threads if the query is close to mem limits.

Change-Id: I68ef247a68642939b98ec7c429dfd393b23a20d2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1906
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Update decimal tests to only run on text/none.

Change-Id: I9a35f9e1687171fc3f06c17516bca2ea4b9af9e1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2217
Tested-by: jenkins
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Migrate DataErrors tests to Python test framework, re-enable subset of tests

This re-enables a subset of the stable data errors tests and updates them to
work in our test framework. This includes support for updating results via --update_results.

This also lets us remove a lot of old code that was there only to support these disabled
tests.

Change-Id: I4c40c3976d00dfc710d59f3f96c99c1ed33e7e9b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1952
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Replace single process mini-dfs with multiple processes

This should allow individual service components, such as a single nodemanager,
to be shutdown for failure testing. The mini-cluster bundled with hadoop is a
single process that does not expose the ability to control individual roles.
Now each role can be controlled and configured independently of the others.

Change-Id: Ic1d42e024226c6867e79916464d184fce886d783
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1432
Tested-by: Casey Ching <casey@cloudera.com>
Reviewed-by: Casey Ching <casey@cloudera.com>
Decimal: switch out the boost 128_t int with the c++ standard one.

The c++ standard int128_t is exactly what we want. It is 16 bytes, stored as 2's
complement little endian (the exact extension of the native int types). It out
performs the boost library we were using (see benchmark) and looking at the assembly
for some of the operators, I doubt we can do better. This also seems like the kind
of thing hardware might be able to do natively in the future if we stuck with the
standard implementation.

This requires minimal changes to the rest of our code so the multi int library is
abstracted away.

The standard only added int128 and not 96 or any others. We still will need to use
the boost library for some cases but nothing in the hot path. We might want to revisit
implementing an int96 in the future that is of the same format to get some space
and efficiency savings but I think we can live with just int128 for a while.

Change-Id: I137ef7be812675036dd9b6e5b48dfc5c7aa9ab37
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2200
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Update bin/make* -notests to include other artifacts for packages.

Change-Id: I95e95f0a2e2131875b95d6676620bec7117b7f8a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2237
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Add the return type as a column for SHOW FUNCTIONS.

Also includes some misc pattern matching cleanup.

Change-Id: I6c9ec78b094a73864b4d669afbd75a48c9bf9585
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2199
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
IMPALA-949: Fix scan range initial queue capacity.

Change-Id: I289c61587da75b318ba5a543d31010920a9cffe9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2242
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-945: Fix column reordering with SELECT expressions

Previously, to produce the correct output expressions for the root plan
fragment before a table sink, InsertStmt would reorder the result
expressions for the query statement at the plan root. This had stopped
working for SelectStmts (and test coverage didn't catch that).

Now InsertStmt produces its own output expressions that can substitute
for the originals from the query statement, and the planner uses those
instead.

All query tests for column reordering have been duplicated to use SELECT
expressions.

Change-Id: Ib909fe35d27416b33ba2e5ac797aa931e1fe43f9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2204
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Fix active time reporting in runtime profiles.

- A few places didn't have total timer at the beginning.
- Async build thread for blocking join nodes really messed things up (sum of
  children was more than the time in the join node).

Change-Id: I9176ce37cf22f2bcebea21b117e45cce066dbc1d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2229
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
IMPALA-912: Enforce slot equivalences at the lowest possible plan node.

The reported issue is that we can have redundant hash expressions in exchanges.
The underlying cause is that we fail to remove redundant join predicates.
This patch enforces slot equivalences based on our computed equivalence classes
at the lowest possible plan node by generating new equality predicates.
Each plan subtree now has a minimal set of equality predicates that express
all known equivalences between slots belonging to tuples materialized at that
plan node.
As a result, eliminating redundant join predicates becomes trivial: It is
sufficient to pick a single representative predicate of each relevant equivalence
class. All predicates beyond that are redundant.

Change-Id: I7998fe8d7bdf84cc8eb129d32c86269bedeab68e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2177
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-839: Update tests to be more thorough when run exhaustively

Some tests have constraints that were there only to help reduce runtime which
reduces coverage when running in exhaustive mode. The majority of the constraints
are because it adds no value to run the test across additional dimensions (or
it is invalid to run with those dimensions). Updates the tests that have
legitimate constraints to use two new helper methods for constraining the table format
dimension:
create_uncompressed_text_dimension()
create_parquet_dimension()

These will create a dimension that will produce a single test vector, either
uncompressed text or parquet respectively.

Change-Id: Id85387c1efd5d192f8059ef89934933389bfe247
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2149
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
IMPALA-914: Map null type to boolean in JDBC to be compatible with Hive.

Change-Id: I5831ae7d5dcb03aecea4138d0b13487898951068
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2279
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Silenced ReportBadFormat() that overflowed logs

Change-Id: I72334d418d4bf910a323e8361aba58aee622281b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2281
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-956: RequestPoolService should use short username of principal

We should be using the short name of a Kerberos principal (e.g.
user/fully.qualified.domain@realm.com) or LDAP username (e.g. user@domain)
when checking group membership in RequestPoolService. Right now we call
UserGroupInformation.createRemoteUser() with the full user name and it
will throw an exception.

Change-Id: I39d849627cb49760807504d66109c05b7a399482
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2288
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
IMPALA-951: Throw parser error if no partition spec specified in ALTER TABLE ADD/DROP PARTITION

Change-Id: I876423e39d858d602ed0fbe8369a6714c82639d8
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2295
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins