IMPALA-845: Transfer predicates to 2nd phase merge agg in some cases.

Having predicates need to be transferred to the 2nd phase merge agg
for distinct + non-distinct aggregates without group by.
For distinct + non-distinct aggregates with group by, it is correct
to evaluate the predicates at the 2nd phase (non-merge) agg.

Change-Id: I71d73c4ef92becbb81e142bc0cb5f54e790b1fb5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1743
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-822,IMP-1271: Binding predicates on an aggregation now properly trigger slot materialization.

The bug was that the number of materialized agg-tuple slots did not correspond to the number
of materialized agg functions, due to binding predicates against an AggNode causing slot
materialization after SelectStmt.materializeRequiredSlots().
This patch fixes the issue by taking binding predicates (bound to a slot in an agg tuple)
into consideration in SelectStmt.materializeRequiredSlots().
I added a new sanity check in AggregationNode.toThrift() surfaced another issue with slot
materialization that is also fixed in this patch. The ordering exprs must be marked before
the agg exprs in SelectStmt.materializeRequiredSlots() because the odering exprs may contain
agg exprs that are only referenced inside the ORDER BY clause.

Change-Id: I1bdc0466f583907bed625ce6608938e59faee83f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1639
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
More robust initialization of ScannerContext

This commit is in conjunction with the "Fix missing status check in
lzo scanner." commit in the LZO repo. It provides a test case for the
LZO fix, and changes the ScannerContext initialization so it will fail
more gracefully instead of crashing.

Change-Id: Idcafeb3679a8fa54322d1ec31c6f1aba860e4e4f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1680
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
IMPALA-844: Catch TException rather than TTransportException during RPCs

Change-Id: I9a0df4ed01047860504efbe20ff56d9d2dc191cb
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1732
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
UDF API additions

This patch introduces the ability to specify a prepare and close
function for a UDF, as well as FunctionContext methods for maintaining
state across UDF invocations within a query. Many of the changes are
related to adding an Expr::Open() function which calls the UDF's
prepare function, if specified (it has to be called in Open() since
the LLVM module must be compiled first).

Change-Id: I581d90d03dff71f7ff5d4a6bef839ba6bc46b443
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1693
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Admission controller: user to pool resolution, authorization, and pool configs

Adds RequestPoolUtils which exposes user to pool resolution, authorization,
and relevant pool configurations by wrapping Yarn classes that provide that
functionality. (To support CDH4, those Yarn classes will come from
thirdparty/cdh4-extras.) RequestPoolUtils is created once by the backend and
the instance lives for the duration of the process.

Change-Id: I53db075555578614356d33f9d939c5378b9ec797
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1566
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Remove RM pool configuration and yarn_pool query option/profile property

Admission control adds support for configuring pools via a fair scheduler
allocation configuration, so the pool configuration mechanism is no longer
needed. This also renames the "yarn_pool" query option to the more general
"request_pool" as it can also be used to configure the admission controller
when RM/Yarn is not used. Similarly, the query profile shows the pool as
"Request Pool" rather than "Yarn Pool".

Change-Id: Id2cefb77ccec000e8df954532399d27eb18a2309
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1668
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Admission control tests: Increase thread join timeout and remove unnecessary locking

In some rare cases on overloaded machines, the thread join timeout of 10 seconds isn't
long enough. Also, taking the lock at that time isn't necessary because the main thread
will not attempt to cancel a thread unless it is already in the list of running threads.
Threads are added to that list only after they submit their query.

Change-Id: I23a67d726bc25221f0e9331ca1a3e9f5363f821d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1744
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
IMPALA-759: Parquet writer: all NULL columns should use PLAIN encoding.

Change-Id: I35d5997d7623239f6cfce7f97a5802545126ea4c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1750
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
IMPALA-816: Fix JDBC getFunctions API.

Change-Id: I452306be05363cfb1502dd6a2f82cdf84dfad35b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1749
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Fix UDF query test files.

Change-Id: Idea277ea2d20c47b2a81b0f2f06c48455de2ea45
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1772
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Limit test_udfs to always run with a single exec_option test vector

Change-Id: If3ff1f5f17a95cce88282f9dc165fe5ce85200b9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1781
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-824: Generalize getBoundPredicates() to handle multi-slot predicates.

The bug: Multi-slot predicates bound to a single
outer-joined tuple were not marked as materialized.
In addition, such predicates were not picked up by nodes
under the join via getBoundPredicates() even if it
would be correct to do so.
The fix: Always mark slots of predicates that must be
evaluated by a join in SelectStmt.materializeRequiredSlots(),
regardless of whether the predicate can also be safely
evaluated below the join.
This patch also generalizes getBoundPredicates() to handle
multi-slot predicates and fixes some issues with redundant
predicate assignment. Still, the new approach has several
limitations which are documented in the predicate propatation
planner test to ease future improvements.

Change-Id: If5da0354a83c00a9766fc63b7780ed4d5a9c46e5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1717
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Fix Hdfs sink using wrong partition descriptor when partition keys are
NULL

This bug was unconvered after the fix to IMPALA-624. If two partitions
have NULL as their first partition key, and an INSERT's partition clause
mentions that key statically as NULL, then the last partition processed
would be used, even if it was the partition.

The reason that this didn't come to light before was that the only way
that partitions differ from one another is either in their output
format, or in their location. We have no test for this situation with
different output formats, and prior to IMPALA-624 the location of all
partitions is dynamically determined from their partition keys. After
IMPALA-624, partitions with NULL keys have explicitly set location
fields, and since a random partition was being chosen, that location was
often incorrect.

This is covered by tests in alter_table.test.

Change-Id: I6035f092ccb9c8dc8c947af351642b51474747b6
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1666
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
IMPALA-624: Inserts should respect changes in partition location

Impala would ignore changes in a partition's location (by ALTER TABLE
... SET LOCATION ...).

Change-Id: I9fdc1f09f9d848aa1a4ade3d4f35f8de9cbd18a5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1647
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Admission controller: Add mem limit to tests

Change-Id: Ieae5c25e0d034317113f97ed66b8971cd80e0bae
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1705
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Exit HBaseTestDataRegionAssignment forcefully after success to avoid HDFS-6057.

Change-Id: I125c09c0bf817f3427302adf89e17c419200907c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1768
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-719: Output correct SQL in SelectStmt.toSql() if the stmt has not been analyzed.

Change-Id: I76ec5916352ace6aa9de1f90f47c4bbf6d369b93
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1172
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
[CDH4] CDH-17085: Remove import of Parquet Hive SerDe class that no longer exists

This change is to unblock Impala on CDH5. The Hive Parquet SerDe has changed Java
packages from:

"parquet.hive.serde.ParquetHiveSerDe"
to
"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"

Because of how our build/packaging is configured, this doesn't cause a compile
time error, because the old class is available at build time, but causes a runtime
failure because the class doesn't exist in an actual CDH5 installation. The fix
is to just use a string as the class name, instead of doing an import.

There is some additional work that needs to happen to verify compatibility
of the new/old SerDe and add support for any new INPUT/OUTPUT format classes.
That is tracked by CDH-17085.

Note: This commit also exists in CDH5 (a57497), but later decided to backport to CDH4.

Change-Id: I9cd0a2d8c2aae03d7bbb9d53d5b34747af69cd12
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1398
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1798
Tested-by: jenkins
IMP-1201: Update thirdparty/squeasel to the latest version (e640905)

This change disables support for HTTP operations PUT, DELETE and others
which are viewed as security risks.

Change-Id: I1fa5e69d206f11c7221edf893a41bd77e253d1d9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1796
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins