IMPALA-1308: estimate scanner thread memory requirements based on file size

We use a constant memory overhead when estimating if we have enough
memory to start a new scanner thread. This doesn't work well for
compressed text files, which need to be entirely decompressed in
memory. Our constant estimate was often too low in this case, which
could cause threads to OOM. Instead, for compressed text files, base
the overhead on the maximum file size.

This is tested in test_mem_usage_scaling.py, which failed on text/gzip
before this patch.

Change-Id: I151704df6b4ead2b7ebeb1b2103af056a8f0b42d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4545
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Free local UDF/UDA allocations

UDF/UDAs use FunctionContext::AllocateLocal() to allocate memory that
is owned by Impala and can be freed at any time. This is used to
return string data to Impala. However, we weren't freeing local
allocations before the end of the query, even if the returned data was
no longer needed. This meant that queries like "select
min(lower(string_col)) from tbl" would never free the memory allocated
by lower(), effectively accumulating the entire dataset in memory.

This patch adds calls to FunctionContext::FreeLocalAllocations() so we
don't accumulate unneeded memory. We don't want to call this too often
(i.e. for every row evaluated) for performance reasons, so the
allocations are freed in RuntimeState::QueryMaintenance() (renamed
from CheckQueryState()).

I'm not sure how to test this, but ad-hoc testing confirms that this
does reduce peak memory usage.

Change-Id: I1b973a027151a86521057756f4fe9bf7954881bd
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4536
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
CDH-20211, IMPALA-1152: Ensure that /var/tmp has drwxrwxrwt permissions

Kerberos uses /var/tmp as the location for the replay cache.  If the
permissions aren't such that any user can't write files there,
Kerberos authentication will fail in a manner that is quite difficult
to debug.  This fix will fail to start kerberized daemons if the
permissions aren't correct, and the failure string contains
instructions on how to fix the permissions.

Change-Id: I84a2329b6f311eaccb881bd2f6dc125ba99cb71f
IMPALA-1100: AnalysisException when CTAS has a WITH clause and inline
view

This commit fixes the issue where a CTAS statement with a WITH clause and
an inline view throws an AnalysisException. The problem was that during
the analysis of the WITH clause, the globalState of the analyzer used in
the CTAS was poluted because the same conjuncts were registered multiple
times.

Change-Id: I4f27aeb0f2107d565c80ac9e3eef7d0387865bdc
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4574
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-1271: Query with LAG(...) analytic fn hits DCHECK: ctx->IsArgConstant(2)

Change-Id: I74b4c78944078a9c50935ccc5ade22db09052bad
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4580
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
BufferedBlockMgr: bug fixes for stress.

Change-Id: I084569e10595fd359c7c83be731c4378156185d4
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4576
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPAL-1312: Fix num_pinned tracking in BufferedTupleStream.

Change-Id: I04264ef25ba8d43826e65f98d34135e7b3593f8b
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4582
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
IMPALA-1299: Analytic should be allows in correlated EXISTS subquery

With this commit we enable correlated and uncorrelated EXISTS
subqueries with grouping and/or aggregation including analytic
functions. Furthermore, we enable correlated EXISTS subqueries
with a LIMIT clause.

Change-Id: I36c33f80b152b7f175bf803cbe920ce1983d7162
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4583
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
Fix subqueries test for HBase

Change-Id: I8d3c10d29a198135e87ab848ba206c2662166760
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4597
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
Silence most of data loading spew by redirecting it to log files

Change-Id: I256a3970ce52bbcac816178029f703095fec388f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4610
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
IMPALA-1303: Fix setting of intermediate agg tuples for num_nodes=1.

Change-Id: Ic6270443287be330f2a70d3037a96d7c98a380ec
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4601
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
length(CHAR(*)) function ignores spaces

Fixed length to behave like postgres for CHAR(*).

Change-Id: I817c74a4af73a3c0a9f5ea1d8ee36b520936395f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4572
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: Victor Bittorf <victor.bittorf@cloudera.com>
Proper handling of NULL tuples by buffered-tuple-stream.

Adding a bitstring at the head of each block in the TupleStream that indicates which
tuples of the appended rows in the block are NULLs. When reading the stream, through
GetNext() or GetTupleRow() calls, the NULL tuples are stitched back to their correct
position.

This fixes crashes in PHJ of bushy plans with NULLs on the build side(s) as well as
similar crashes in PAGG and the analytic node.
For example, it fixes IMPALA-1204, IMPALA-1223, and IMPALA-1249.
Also, adds regression tests for IMPALA-1175, IMPALA-1204, IMPALA-1223, IMPALA-1249
and IMPALA-1306.

Change-Id: I30ad0dbd4dfeabcda8fae444d1c6ec9291f38398
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4596
Reviewed-by: Ippokratis Pandis <ipandis@cloudera.com>
Tested-by: jenkins
Add remote fragment start latency counter to profile

Change-Id: I4deb1bf489223d4128357cfde0269e3519e56c8f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4613
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
IMPALA-1316: crash on VARCHAR join

Fixed codegen issue casing some VARCHAR joins to crash.

Change-Id: Ib2674199a3b2c3c5a5fd63cfae0b64e3b1ca158b
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4616
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
IMPALA-1278: Basic cardinality estimation for semi joins.

Change-Id: I353c9c581f7a54e0b42bdd1e89cf99b93d6e18de
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4634
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Limit the size of a single hash table.

Limit the size of the hash table to the number of bits we are using in the
hash. We use 32-bit hashes with some of the bits used for partitioning. We
can't reuse the bits we picked for partitioning to index into the hash table.

Change-Id: I9c1f8d2185f219f71416ea61255635ee19672cf1
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4629
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Fix appx_median to return correct result type

Change-Id: Ifc54e1069e2f7a46242229d710943e921633a920
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4625
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Call Function::eraseFromParent() when codegen fails

Aborting codegen mid-function without erasing the function can result
in a corrupt module.

Change-Id: I64b518c8286195d78e68c2df5397a8843f6badc3
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4632
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
IMPALA-1210: Log Java version at impalad/catalogd startup

Change-Id: I4ab39231996522decce043482fcdd860fd02ada3
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4611
Reviewed-by: Juan Yu <jyu@cloudera.com>
Tested-by: jenkins