Silenced ReportBadFormat() that overflowed logs

Change-Id: I72334d418d4bf910a323e8361aba58aee622281b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2281
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-956: RequestPoolService should use short username of principal

We should be using the short name of a Kerberos principal (e.g.
user/fully.qualified.domain@realm.com) or LDAP username (e.g. user@domain)
when checking group membership in RequestPoolService. Right now we call
UserGroupInformation.createRemoteUser() with the full user name and it
will throw an exception.

Change-Id: I39d849627cb49760807504d66109c05b7a399482
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2288
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
IMPALA-951: Throw parser error if no partition spec specified in ALTER TABLE ADD/DROP PARTITION

Change-Id: I876423e39d858d602ed0fbe8369a6714c82639d8
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2295
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
External Data Source: Public API

Adds the thrift structures for the public external data source API
and a new maven project containing the Java ExternalDataSource
interface and the generated Java thrift classes.

The ExternalDataSource.thrift structures can evolve in a backward
compatible way. The ExternalDataSource Java interface will always
contain a version number in the namespace (e.g.
com.cloudera.impala.extdatasource.v1 for V1) so we can potentially
make breaking changes to the interface in the future but still
support older versions.

A trivial implementation of the ExternalDataSource API is also
added for testing purposes.
TODO: Make the sample data source implementation realistic.

Change-Id: I827d6420a87ed7a2bce34e050362ca98ddc5dbcc
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2241
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
IMPALA-939: Regex should match anywhere in string.

Change-Id: I8dcd337c3b06b632017270670a4f199ec7ada648
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2296
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
Increase default_pool_max_requests in run-all-tests.

Temporarily increase the cap on max requested queries while running tests to unblock
builds. Currently, the exhaustive runs always fails, and there are some intermittent
failures in the core runs.

Change-Id: I26b9ce343d72bab7687e49f7dbd7bf3bf655a294
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2323
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
IMPALA-888: Drop union operands with constant conjuncts evaluating to false.

This patch simplifies the complex slot materialization logic for unions by
making the materialization independent of conjuncts assigned to MergeNodes.
When 'pushing down' predicates into union operands, we drop union operands
with constant predicates evaluating to false. Constant predicates that
evaluate to true are simply ignored.

Change-Id: I0e7ccfb206bed29db2b5d667e2bb61310980e80a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2270
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
CDH-18751: Impala should use a unique Kerberos credentials cache location

Kerberos, by default, uses a location for its credentials cache (where
tickets are saved to avoid having to re-authenticate on each connection)
that is unique only to the user. This means that if several programs
running as the same user try and authenticate concurrently, they may
race to access the cache, and an error might occur.

This patch causes Impala's demons to use a path that is unique to
process name and process id (e.g. /tmp/krb5_impalad_4324). The new flag
--krb_credential_cache can be used to set the location explicitly.

Any Kerberos connections initiated from Java will still use the system
default cache. The catalog service is the only such service. It's
important that the cache locations for Java and C++ are different (to
avoid the same bug being exposed in a single process, which is actually
its most likely cause). The Java location can be changed by setting the
KRB5CCNAME environment variable.

Change-Id: I4d975a3a7cf5ff6ee4c107833f5292115cd33b03
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2308
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Fix two unused variable warnings

Change-Id: Ib970d8300ba289455fcbc6843001c8ec6844e009
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2340
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Adding EXTRACT builtin.

Change-Id: I6de20f336ecdfa3acd8d3a9166cff4a062baaacc
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2247
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
IMPALA-934: Refresh cached UDF library when creating a new function

This change adds the ability to refresh a local cache entry, causing
the old cache entry to be dropped and the library to be reloaded from
HDFS. This is used in ResolveSymbolLookup(), which is called by the
frontend when creating a new a function, and in ImpalaServer when
receiving a "create function" heartbeat. This change also makes sure
the FE calls into the backend for jars, so jars get refreshed as well.

Change-Id: I5fd61c1bc2e04838449335d5a68b61af8b101b01
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2286
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
CDH-18416: Don't inline ReadWriteUtil::ReadZLong()

For wide Avro tables, ReadZLong() would get inlined many times into a
single function body, causing LLVM to crash. Not inlining doesn't seem
to have a performance impact on narrow tables, and helps with wide
tables.

This change also adds tests over wide (i.e. many-column) tables. The
test tables are produced by specifying shell commands to generate test
tables in functional_schema_template.sql, which are executed in
generate-schema-statements.py. In the SQL templates, sections starting
with a ` are treated as shell commands. The output of the shell
command is then used as the section text. This is only a starting
point; it isn't currently implemented for all sections, and may have
to be tweaked if we use this mechanism for all tables.

Change-Id: Ife0d857d19b21534167a34c8bc06bc70bef34910
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2206
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
CDH-18563: Speed up the computation of transitive value transfers.

The issue: Computing the full transitive closure for all slots can be very
expensive (10s of seconds for >2k slots, minutes for >4k slots).
Queries with many views and/or unions were affected most because each
union/view adds a new tuple with slots, increasing the total number of slots.

The fix: The new algorithm exploits the sparse structure of the value transfer
graph for a significant speedup (>100x). The high-level steps are:
1. Identify complete subgraps based on bi-directional value transfers, and
   coalesce the slots of each complete subgraph into a single slot.
2. Map the remaining uni-directional value transfers into the new slot domain.
3. Identify the connected components of the uni-directional value transfers.
   This step partitions the value transfers into disjoint sets.
4. Compute the transitive closure of each partition from (3) in the new slot
   domain separately. Hopefully, the partitions are small enough to afford
   the O(N^3) complexity of the brute-force transitive closure computation.

Change-Id: I35b57295d8f04b92f00ac48c04d1ef1be4daf41b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2321
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Clarify prevention of value transfers into outer-joined inline views with a limit.

Change-Id: I1cd239314bb844f5eeaaf808610dda99a9206e35
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2361
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
External Data Source: Thrift structure changes

A few changes to the external data source thrift types:
* Change RowBatch to return entire columns. Adds Data.TColumnData to
  represent an entire column.
* Makes all fields in ExternalDataSource (except for status fields on
  the result structures) optional in case fields become deprecated in
  the future.
* Adds a limit parameter to the TOpenParams structure in case the
  data source needs to apply the limit itself.

Change-Id: I62db68bfb64d2190dfdd0c84be5925ad5db031ef
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2345
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reduce mini dfs logging verbosity.

Currently, the default log level is set to DEBUG. This produces approximately 10-20 GB of
logs per build, which is unacceptable.

Change-Id: Ibbb48876fc72faa23d76f32166f31f0257a7a3a0
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2386
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: jenkins
Add max / min_*int() builtins

It would have been convenient today to know the largest values that
Impala accepts for its integer types. This patch adds max and min
builtins for our numeric types.

[localhost:21000] > select max_bigint(), max_int(), max_smallint(),
max_tinyint();
Query: select max_bigint(), max_int(), max_smallint(), max_tinyint()
+---------------------+------------+----------------+---------------+
| max_bigint()        | max_int()  | max_smallint() | max_tinyint() |
+---------------------+------------+----------------+---------------+
| 9223372036854775807 | 2147483647 | 32767          | 127           |
+---------------------+------------+----------------+---------------+

Change-Id: I6df6df2728197529c6375dbb1b7d3c9ddb9833d2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2381
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
[CDH4] Disable admission control to work around HUE-994

On CDH4, Hue does not close queries (HUE-994), so queries
submitted through Hue will never be released in the admission controller
and eventually all queries will be rejected. This disables admission
control on CDH4 to work around this problem.

Change-Id: I1bb1707b1d2c4f8c041b6521e715aa9f7fabc182
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2399
Tested-by: jenkins
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>