CDH-21036: Make --quiet flag suppress warning messages in impala-shell

Change-Id: Ia57652840b3e3483a06c0bbef283c60a229db27f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4092
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
Fix return status of summary within impala shell

Follow up fix to IMPALA-1153. Ensure that the correct
CmdStatus is returned by the summary command. (ERROR for
invalid queries and SUCCESS even if summary is not available.)

Change-Id: Icf67164dc82f202ec15071541f6ed3b26e3ad7fb
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4089
Reviewed-by: Abdullah Yousufi <abdullah.yousufi@cloudera.com>
Tested-by: jenkins
IMPALA-1098 Updated bad test name from CR

Change-Id: Ic7df7d37e8ca9e74231e760dcd411c528d02f730
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4124
Reviewed-by: Alex Leblang <alex.leblang@cloudera.com>
Tested-by: jenkins
Analytic functions tests

Added several tests for analytical functions

Tests for the following have not been added because it's not implemented yet:
- Lag, Lead functions
- Window clauses

Change-Id: I34546c967a6d29c97327f4cba405006a50867dcb
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4307
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: jenkins
IMPALA-1235: Get a new Impala Client instance every time a user reconnects.

Earlier, if the user connected via the command line, the shell would only attempt to
create a new Impala Client instance if a connection did not exist. This resulted in the
shell not connecting to the user specified Impalad.

Change-Id: I74c291256d0c063f6324b01aa7336282e6969a4e
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4392
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
IMPALA-1178: fix Impalad crashes during query cancellation

There are several variants of races that can occur if a query is unregistered
or cancelled before it's fully running.  Normally, this will only happen when
the session or connection is closed, because the 16-byte query-id won't yet be
available to a client to cancel/close the specific query until after
ExecuteStatement and friends have returned the query_id.  However, it could
also be triggered via the web interface or if a user were able to "guess" the
query-id before it's returned by ExecuteStatement (e.g. by scraping the web
interface where queries show up "early", though for HS2 could implement secret
to prevent that).

Here's some examples of the race (the first leads to the DCHECK reported in
the jira).

* If session is closed and query unregistered before exec_state->WaitAsync()
  completes, then the async thread would be freed while still executing.
* If the query is cancelled before coord_ is set, then exec_state->Cancel()
  won't be effective.
* If Coordinator::Cancel() is called before executor_ is set, then fragment
  cancel won't be effective.
* Lot's of other points with inconsistent state (but not safe to hold the
  lock) where correctness is hard to reason about.

The session->inflight_queries already tracks which queries to unregister when
a session is closed.  Delay adding queries to this set until after it is safe
to cancel.  This handles the session/connection close case.

Then, add a boolean arguement to UnregisterQuery and CancelInteral that
indicates whether these should return error if the query is not yet in-flight.
When UnregisterQuery() is used to clean up after an error on the query issue
path (when the query is not yet in-filght) will pass false.  Otherwise, the
other paths will pass true, and this in-flight check will guard against the
race from the CancelOperation RPC path.

Also, fix a race in query expiration.  Previously queries could be processed
by ExpireSessions() before last_active_time_ was set (though in practice it'd
be hard to hit this race because timeouts are specified in seconds) and before
it's safe to cancel.

Testing:
- verified this fixes the crashes/hangs using the steps to reproduce given in
  IMPALA-1178.
- Attempted to add a pytest case for this but so far I've had no luck
  reproducing it programmatically.  Will try some more and follow up with a
  separate change when that's successful.

Change-Id: Iec6b977e28fbec2d8bcc2734d1b7c17604e44180
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4245
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
Tested-by: jenkins
IMPALA-1297: Results of NOT IN may not be correct if subquery results in
NULL

This commit fixes a bug in the implementation of the null-aware anti
join that resulted in wrong results being returned from NOT IN correlated
subqueries in the presence of nulls.

Change-Id: I6f2eb326ec7e40d80ec8da94ba33946b9ac9b115
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4506
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
Char Partition Fix

Fixed bug CHAR and VARCHAR partition columns. Also, disables CHAR and VARCHAR for UDAs
and UDFs.

Change-Id: I67ccd746cb4c063f8a7a984df9564fa9122fdf43
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4493
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Analytic fn query test for IMPALA-1280 needs VERIFY_IS_EQUAL_SORTED

Change-Id: Iefb232a501a97a0f0351fd0794c7a3bfc279f98c
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4513
Reviewed-by: Ippokratis Pandis <ipandis@cloudera.com>
Tested-by: jenkins
IMPALA-1296: Fix DCHECK failure for unnecessary buffered tuple in AnalyticNode

The AnalyticEvalNode had a DCHECK that expected the buffered tuple to
only be set when it was needed (i.e. when there are partition exprs or
order by exprs). However, the FE creates a buffered tuple for an entire
sort group when any AnalyticEvalNodes in the sort group need it and that
tuple is set for all nodes. This reverses the logic so that we DCHECK
the buffered tuple is set when it is needed.

Change-Id: If54b303bc439f235da06a542b46a35c61da9e1bd
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4489
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Minor cleanups to the commit for IMPALA-1178.

These addressed review feedback from Henry, but I forgot to do a
git add/commit before submitting.  All changes are trivial cleanups.

Change-Id: Ic300a463d4ea6de882dff67ea72033901cac5a25
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4517
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
Tested-by: jenkins
Add mechanism to the BufferedBlockMgr to allow pinning multiple blocks at once.

The callers of the block mgr will want to Pin() multiple blocks at once
(e.g. PinStream()). Typically, if it can't pin all the blocks, it needs to
unpin them and fall back to something else (e.g. repartitioning).

There's no point in pinning half the blocks (forcing writes to disk).

Change-Id: I0e05b8243a27e8b99f5692f781c4b498d600284d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4495
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
IMPALA-1293: Fix DCHECK failure with window ROWS BETWEEN UNBOUNDED PRECEDING

Change-Id: I4e92e9593402f4341826c6940e23e493c7d23641
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4487
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Char PARQUET, AVRO, and TEXT tests

Adds fixes and tests for Hive CHAR & VARCHAR compatibility.
Also fixes a bug in tuple materialization for VARCHAR and non in-lined CHAR.

Change-Id: I400b089cb8ddba2e264ef9f2e37956b2ceaaf9fb
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4054
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
IMPALA-1012: lower the default parquet file size to 256 MB

Parquet files cannot be split into multiple scan ranges.  So, the
advantage of smaller files is more potential parallelism for queries.
Also, less memory is used to write smaller files.

However, we don't want to go too small, or we could suffer on the I/O
performance during a scan. To verify that 256 MB is not too small, I ran
some scan heavy queries on the 10-node cluster using the 10000 GB TPCH
lineitem table. The baseline is 1 GB files, and even at that size the
cluster is small enough that there should be plenty of parallalism.  So,
the experiment is to determine whether scans (and in particular I/O
efficiency) suffers at smaller file size.  Results (best, worse
execution times in seconds over four runs):

(a) select l_comment from lineitem where l_comment is null

1 GB files:    157.05, 191.42
256 MB files:  129.51, 132.29
64 MB files:   127.60, 148.79

(b) select l_orderkey from lineitem where l_orderkey is null;

1 GB files:    34.27,  41.97
256 MB files:  35.30,  38.86
64 MB files:   42.86,  61.03

Also, I took a look at the profiles and nothing bad stands out at 256
MB.  So, 256 MB looks like a good compromise.

Change-Id: Ifa8945d300141b2b2bbf66a3f8ce02644ddf14b3
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4519
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
IMPALA-1301: Invalid rewrite of uncorrelated scalar subquery when outer
expr is in both sides of binary predicate

This commit fixes the issue where an invalid rewrite is performed for
scalar uncorrelated subqueries that participated in a binary predicate
in which columns from the outer query block appear in both sides. We fix
this issue by rewritting the subquery using a cross join and placing
binary predicate in the outer's WHERE clause.

Change-Id: Ic5234bd2ee704d5ddead9217a636259e694e3eda
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4512
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Fix hash table buckets to allocate memory from the BlockMgr.

This was always a TODO. We want memory to come from the block mgr and trigger spilling.

Change-Id: I07f1f79fbbb33068fb2df64510a80a9b008ef73d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4466
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Fix data loading to unblock gvm

Change-Id: I5e145f1e8497d340cb72a8112c247e63b1c79362
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4537
Reviewed-by: Nong Li <nong@cloudera.com>
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: Victor Bittorf <victor.bittorf@cloudera.com>
Add DECODE builtin

This adds DECODE functionality into the existing CaseExpr class. There
will be no separate backend impementation for DECODE, it will be sent to
the backend as a CASE expr so the existing codegen function can be used.

Because Oracle does cast checking during execution and Impala cast
checking during analysis, some uses of DECODE that are valid in Oracle
are invalid in Impala.

Ex:

  SELECT DECODE(foo, bar, int_col, baz, string_col_containing_only_ints)
  FROM ...

  would be run on Oracle. If string_col_containing_only_ints actually
  contained non-INTs, an error would be thrown during execution and no
  results would be returned. In Impala an error is thrown during analysis.
  If a CAST was added to the STRING column, a cast failure would result in
  NULL.

Change-Id: Ia08cc2389abb6f843bba117e7091c659ad25ff41
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4334
Tested-by: jenkins
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Casey Ching <casey@cloudera.com>
Implement disabling spilling.

Change-Id: I07951b9c8bcce74a9cb94e1abf9ff36b71e575fc
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4535
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>