IMPALA-939: Regex should match anywhere in string.

Change-Id: I8dcd337c3b06b632017270670a4f199ec7ada648
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2296
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
Increase default_pool_max_requests in run-all-tests.

Temporarily increase the cap on max requested queries while running tests to unblock
builds. Currently, the exhaustive runs always fails, and there are some intermittent
failures in the core runs.

Change-Id: I26b9ce343d72bab7687e49f7dbd7bf3bf655a294
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2323
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
IMPALA-888: Drop union operands with constant conjuncts evaluating to false.

This patch simplifies the complex slot materialization logic for unions by
making the materialization independent of conjuncts assigned to MergeNodes.
When 'pushing down' predicates into union operands, we drop union operands
with constant predicates evaluating to false. Constant predicates that
evaluate to true are simply ignored.

Change-Id: I0e7ccfb206bed29db2b5d667e2bb61310980e80a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2270
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
CDH-18751: Impala should use a unique Kerberos credentials cache location

Kerberos, by default, uses a location for its credentials cache (where
tickets are saved to avoid having to re-authenticate on each connection)
that is unique only to the user. This means that if several programs
running as the same user try and authenticate concurrently, they may
race to access the cache, and an error might occur.

This patch causes Impala's demons to use a path that is unique to
process name and process id (e.g. /tmp/krb5_impalad_4324). The new flag
--krb_credential_cache can be used to set the location explicitly.

Any Kerberos connections initiated from Java will still use the system
default cache. The catalog service is the only such service. It's
important that the cache locations for Java and C++ are different (to
avoid the same bug being exposed in a single process, which is actually
its most likely cause). The Java location can be changed by setting the
KRB5CCNAME environment variable.

Change-Id: I4d975a3a7cf5ff6ee4c107833f5292115cd33b03
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2308
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Fix two unused variable warnings

Change-Id: Ib970d8300ba289455fcbc6843001c8ec6844e009
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2340
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Adding EXTRACT builtin.

Change-Id: I6de20f336ecdfa3acd8d3a9166cff4a062baaacc
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2247
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
IMPALA-934: Refresh cached UDF library when creating a new function

This change adds the ability to refresh a local cache entry, causing
the old cache entry to be dropped and the library to be reloaded from
HDFS. This is used in ResolveSymbolLookup(), which is called by the
frontend when creating a new a function, and in ImpalaServer when
receiving a "create function" heartbeat. This change also makes sure
the FE calls into the backend for jars, so jars get refreshed as well.

Change-Id: I5fd61c1bc2e04838449335d5a68b61af8b101b01
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2286
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
CDH-18416: Don't inline ReadWriteUtil::ReadZLong()

For wide Avro tables, ReadZLong() would get inlined many times into a
single function body, causing LLVM to crash. Not inlining doesn't seem
to have a performance impact on narrow tables, and helps with wide
tables.

This change also adds tests over wide (i.e. many-column) tables. The
test tables are produced by specifying shell commands to generate test
tables in functional_schema_template.sql, which are executed in
generate-schema-statements.py. In the SQL templates, sections starting
with a ` are treated as shell commands. The output of the shell
command is then used as the section text. This is only a starting
point; it isn't currently implemented for all sections, and may have
to be tweaked if we use this mechanism for all tables.

Change-Id: Ife0d857d19b21534167a34c8bc06bc70bef34910
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2206
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
CDH-18563: Speed up the computation of transitive value transfers.

The issue: Computing the full transitive closure for all slots can be very
expensive (10s of seconds for >2k slots, minutes for >4k slots).
Queries with many views and/or unions were affected most because each
union/view adds a new tuple with slots, increasing the total number of slots.

The fix: The new algorithm exploits the sparse structure of the value transfer
graph for a significant speedup (>100x). The high-level steps are:
1. Identify complete subgraps based on bi-directional value transfers, and
   coalesce the slots of each complete subgraph into a single slot.
2. Map the remaining uni-directional value transfers into the new slot domain.
3. Identify the connected components of the uni-directional value transfers.
   This step partitions the value transfers into disjoint sets.
4. Compute the transitive closure of each partition from (3) in the new slot
   domain separately. Hopefully, the partitions are small enough to afford
   the O(N^3) complexity of the brute-force transitive closure computation.

Change-Id: I35b57295d8f04b92f00ac48c04d1ef1be4daf41b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2321
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Clarify prevention of value transfers into outer-joined inline views with a limit.

Change-Id: I1cd239314bb844f5eeaaf808610dda99a9206e35
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2361
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
External Data Source: Thrift structure changes

A few changes to the external data source thrift types:
* Change RowBatch to return entire columns. Adds Data.TColumnData to
  represent an entire column.
* Makes all fields in ExternalDataSource (except for status fields on
  the result structures) optional in case fields become deprecated in
  the future.
* Adds a limit parameter to the TOpenParams structure in case the
  data source needs to apply the limit itself.

Change-Id: I62db68bfb64d2190dfdd0c84be5925ad5db031ef
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2345
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Reduce mini dfs logging verbosity.

Currently, the default log level is set to DEBUG. This produces approximately 10-20 GB of
logs per build, which is unacceptable.

Change-Id: Ibbb48876fc72faa23d76f32166f31f0257a7a3a0
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2386
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: jenkins
Add max / min_*int() builtins

It would have been convenient today to know the largest values that
Impala accepts for its integer types. This patch adds max and min
builtins for our numeric types.

[localhost:21000] > select max_bigint(), max_int(), max_smallint(),
max_tinyint();
Query: select max_bigint(), max_int(), max_smallint(), max_tinyint()
+---------------------+------------+----------------+---------------+
| max_bigint()        | max_int()  | max_smallint() | max_tinyint() |
+---------------------+------------+----------------+---------------+
| 9223372036854775807 | 2147483647 | 32767          | 127           |
+---------------------+------------+----------------+---------------+

Change-Id: I6df6df2728197529c6375dbb1b7d3c9ddb9833d2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2381
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
[CDH4] Disable admission control to work around HUE-994

On CDH4, Hue does not close queries (HUE-994), so queries
submitted through Hue will never be released in the admission controller
and eventually all queries will be rejected. This disables admission
control on CDH4 to work around this problem.

Change-Id: I1bb1707b1d2c4f8c041b6521e715aa9f7fabc182
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2399
Tested-by: jenkins
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
IMPALA-970 The kinit renewal command should specify the credential cache file.

All Impala services store their kerberos ticket credentials in a custom file. This
PATCH updates the ticket renewal command to use the custom credentials file.

Change-Id: I4a77a9b58e9747c0c73fa7a53a85e1214c7a31f7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2409
Reviewed-by: Henry Robinson <henry@cloudera.com>
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2413
Adding STDDEV builtin.

Change-Id: I79e5aee1e9e879aa2d09078ab45bc149675e1d4a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2341
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
Decimal: Read/Write to parquet.

This adds support for the FIXED_LENGTH_BYTE_ARRAY parquet type and
encoding for decimals.

Change-Id: I9d5780feb4530989b568ec8d168cbdc32b7039bd
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1727
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
External Data Source: Few small API changes

* Rename getStats() to prepare()
* Adds TRowBatch.num_rows to indicate number of rows when no cols are
  materialized
* Changes api and sample poms to produce source jars

Change-Id: I02dcc89e27716978708386cfc3f7940ee5dbc023
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2406
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
IMPALA-923: Boolean slotrefs not marked as assigned in inline views

A boolean slotref predicate that could be pushed into an inline view
would not be correctly marked as assigned, leading to an extra select
node being introduced to evaluate it. This was because the id of the
expression after substitution would change (see createInlineViewPlan()),
but only the post-substitution conjunct IDs were marked as assigned.

This bug only affected standalone slotrefs; other exprs (like casts, or
explicit predicates referencing a slotref) would not change their ID
under substitution.

Change-Id: I4127528b4aec25c966a4d186ddc98a68502b90c1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2430
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Support SSL / TLS connections to LDAP servers

Secure connections to LDAP require a client-side certificate which is
used to confirm that the LDAP server really is who we expect it to
be. This patch adds support for --ldap_ca_certificate to tell LDAP where
that file is located.

Tested with both ldaps:// to port 636, and ldap://-over-TLS to port
389 (the latter with --ldap_tls).

Change-Id: I69e2a7c267bd471124ae9ccfd6fffca5d3d4f123
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2401
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins