AnalyticEvalNode logging improvements and fix transferring resources

Adds the node id to log output because queries may have many nested
analytic nodes running on the same thread. Makes level 2 logging
less verbose and other misc improvements to some log output.

Also fixes a potential problem transferring resources between
curr_tuple_pool_ and prev_tuple_pool_ where we could potentially
allocate more than the 8MB maximum (MAX_TUPLE_POOL_SIZE) from
curr_tuple_pool_ while processing a row batch. In that case, we
would have transferred more resources from curr_tuple_pool_ to
prev_tuple_pool_ before waiting for those resources to be
transferred to the output row batch. In theory, that would mean
that we could accumulate more than the 8MB max, though that would
not cause any other issues as the resources would just be held
longer than necessary. This was tested manually by reducing
MAX_TUPLE_POOL_SIZE and evaluating analytics over very large
string columns. The additional logging was checked to verify
the resources were transferred as expected.

Change-Id: I3dc88df4bc51d57179193c8e21d7ac43b42288dd
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4649
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-1319: Fixed CHAR padding for numeric casts
IMPALA-1322: Crash on VARCHAR/CHAR join

Fixed 2 issues:
  (1) Disabled codegen for CHAR in hash join equality
  (2) fixed memory layout for CHAR
  (3) Fixed a regression where space padding could be dropped for numeric casts.

Change-Id: I6475fd527ca0d67c7d4d5ec7e561549e43fbc336
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4640
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
Don't remove RM vcore callback if it was never added

It's legal to call Close() without ever calling Open(). The HDFS scan
node only attaches a 'vcore' callback in Open(), but always tries to
remove it in Close() which can hit a DCHECK.

Change-Id: I2481a24cecff9c86083a837e2d1b53eb68df7d81
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4619
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
(cherry picked from commit 0a8ee5448acea0d7fa98b6a17de968d818ee9af2)
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4668
Reviewed-by: Henry Robinson <henry@cloudera.com>
IMPALA-1323: Issue distinct error messages for unsupported subquery
syntax

This commit improves the error message for the case where a UNION query
contains subqueries.

Change-Id: Ic489b6a027b88e2a62c96dc4967a5aa2f54fcf8e
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4667
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
Don't check vcore subscription levels if scanner thread is reserved

Each scan node needs at least one thread to make progress. In a query
with a lot of scan nodes, the number of threads required becomes very
high, and for certain exec nodes (e.g. joins and unions) it's not ok to
run scan nodes serially, since they become blocked on the input from a
single scan that may not have yet run, and can't consume the input from
nodes that currently have an active scan thread.

The ThreadResourceMgr handles this case by allowing reserved thread
tokens that are always granted. However, with Llama enabled, we would
always check the thread-to-vcore ratio before allowing a scanner thread
to start without taking into account whether the thread was reserved or
not. This could lead to starvation, and ultimately a deadlocked query.

This patch allows threads to tell whether or not a token has come from a
reservation or not. The scanner threads can then use this information to
decide whether to check the subscription ratio before running the
thread.

Of course, if the subscription ratio really is that high, there'll be
other performance problems but at least the query should make (slow) progress.

Change-Id: I1ff0c94d919ce3a2b29380741a62f7b8368c6c59
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4614
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
(cherry picked from commit a988e4342e5d6bf2c892ad7071f4de7b3c2fbbcf)
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4622
Tested-by: jenkins
CDH-22027: Remove default Llama hosts

There are two flags that allow us to specify Llama hosts - one was added
for HA, while the other is in the process of being deprecated. Both had
default values which interfered with each other if they weren't
overwritten. This patch removes the defaults, and Impala won't start if
neither are set.

Change-Id: Ibba771ab91ee13e5b5d1a725e385c964f2f9a132
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4652
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-1336: Set the shell's environment locale setting for utf-8 compatibility.

Change-Id: I120af07e7d8878561552977054a1e3837f470dea
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4665
Reviewed-by: Sean Mackrory <sean@cloudera.com>
Tested-by: jenkins
Fix CHAR/VARCHAR Aggregation

This fixes an issue where VARCHAR and CHAR could error in some aggregations.
The cause of the problem is that the BE currently does not support CHAR/VARCHAR as
arguments to aggregates, they require an implicit cast to string first.
The resolution is to have these operators return STRING instead of CHAR(*) or VARCHAR(*).
Note that the CHAR(*) comparisons still ignore spaces for min/max.

This takes advantage of the fact that STRING, VARCHAR(*), and CHAR(*) values are all
handled as a StringVal for exprs. The STRING aggregates are registered as CHAR(*) and
VARCHAR(*) aggregates and the front end converts the return type to a STRING in all cases.

Also includes a fix for a TODO about casting between CHAR and VARCHAR.

Change-Id: I1d3a9cc48e426286ce63677324a8c680e67b005a
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4573
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
Free local UDF/UDA allocations, take 2

UDF/UDAs use FunctionContext::AllocateLocal() to allocate memory that
is owned by Impala and can be freed at any time. This is used to
return string data to Impala. However, we weren't freeing local
allocations before the end of the query, even if the returned data was
no longer needed. This meant that queries like "select
min(lower(string_col)) from tbl" would never free the memory allocated
by lower(), effectively accumulating the entire dataset in memory.

This patch adds calls to FunctionContext::FreeLocalAllocations() so we
don't accumulate unneeded memory. We don't want to call this too often
(i.e. for every row evaluated) for performance reasons, so the
allocations are freed in a new ExecNode::QueryMaintenance() function,
which replaces most calls to RuntimeState::CheckQueryState(). Local
allocations are freed per exec node so we only need to worry about
concurrent expr evaluation and freeing local allocations within a
single node.

I'm not sure how to test this, but ad-hoc testing confirms that this
does reduce peak memory usage.

Change-Id: Id84e747c1c10f475749cd96540cfbec2c24b0551
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4630
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
IMPALA-1317: Explicitly close alter, use and drop ddl queries in the shell.

Change-Id: I034d9dcfd76666cdbfa622d3c8cdc59ea85a75c9
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4655
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Reduce min PA/PHJ mem requirement.

Update PA/PHJ to use small (< io sized buffers) initially. Without this we would
not be able to run at the QPS that we need just due to the buffering requirements
of these operators.

Change-Id: Ic8a777d147893567c9590fbab17f561eadb6ee19
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4623
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Change BufferedBlockMgr's "BlockWritesIssued" counter to "BytesWritten"

Change-Id: Icdc8499179821b9cd88eb17878fbf169c884b7dd
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4677
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Allow forcing principals / usernames to be lower case for authorisation

This patch adds a temporary workaround for the issue where user
principals or usernames are in upper case, but our authorisation
components (Sentry, admission control) might expect lower case
shortnames.

The usual way to do this is via Kerberos' auth_to_local rules, but these
are not available to us in C++ as the MIT KRB implementation doesn't
support the lower case flag. We could, instead, use Hadoop's
UserGroupInformation class and set up auth_to_local rules there: this is
probably the best long-term fix but we need this patch more quickly.

Instead, if a user sets --force_lowercase_usernames, all authenticated
principals will have their shortnames mapped to lower-case before being
passed to Impala. Unauthenticated names are unaffected.

Change-Id: I2b802bcbe26d01e9998e39ab19e8699bfa27b2a2
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4650
Reviewed-by: Michael Yoder <myoder@cloudera.com>
Tested-by: jenkins
Support specifying KW_ROLE in GRANT/REVOKE privilege stmts for Hive compatibility

Allows KW_ROLE to be specified in a GRANT/REVOKE PRIVILEGE statements. Most SQL engines
do not require this keyword, but the Hive implementation does. The syntax is now:

GRANT/REVOKE <privilegeSpec> TO/FROM [ROLE] <roleName>

Change-Id: I4341ff064d8bfb60dc6a887a4ca62a4ac1c64558
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4680
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
IMPALA-1335: Wrong subquery rewrite for correlated scalar subqueries
with complex exprs

This commit fixes the issue where, for the case of scalar subqueries,
complex exprs in a correlated predicate may result in a wrong subquery
rewrite.

Change-Id: Ib6f14a37ca7a74e25daf3b31f86766ff9032d7fd
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4674
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
IMPALA-1322: Fix related issue

There is an issue related to IMPALA-1322. The expression list when laying out memory
was being improperly index.

Change-Id: I2eef84a812b451d87ecb8afd304e765aff1f5a6b
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4675
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins