Fix spilling with aggregated exprs with string slots.

This patch also does some clean up on agg-fn-evaluator to support this case.

Change-Id: If7e5c8663c7d371b2666acaab2966a2cd7bdccf9
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4256
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
IMPALA-1230: Uncorrelated non-scalar subquery accepted in binary
predicate

This commit fixes the issue where an uncorrelated non-scalar subquery
with distinct in the select list is accepted in a binary predicate. The
issue is fixed by adding a proper check for a distinct clause in the
select list of a subquery when evaluating the subquery's return type
(scalar or non-scalar).

Change-Id: I7a207c89502a6db589d5288aeaf4e54b43122c41
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4316
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
Fix char test to only run on test/none.

Change-Id: I8f5ac5a6e7399ce2fdbe78d07ae24deaa1d7532d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4326
Tested-by: jenkins
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Adding support in PHJ for right-{semi,anti} joins.

Changes needed for PHJ to support RIGHT {SEMI, ANTI} JOINs. Codegen works as well.
Basic parser tests and minimal (end-to-end) query tests.
Need to add analyzer tests and add more query tests.
Note that in the case of right-{semi,anti} and perhaps also on {right,full}-outer we
should not be broadcasting the build side.

Change-Id: I6854ee9e4640f809f0350229bcc00811fa474f07
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4288
Reviewed-by: Ippokratis Pandis <ipandis@cloudera.com>
Tested-by: jenkins
IMPALA-1231 Analytic query with HAVING: IllegalStateException

By including the actual FunctionCallExpr as a child of an AnalyticExpr, we would end
up substituting it if it also happened to show up as a regular/non-analytic aggregate
expr in the query. The solution is only to include the function call parameters as
children of the AnalyticExpr, not the function call itself.

Plus, fixed up partition-by/order-by less-than predicates in AnalyticPlanner.

Change-Id: Ib5d33baa2f257f7a8a21dd536332d11c55fbdbca
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4327
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Speed up PHJ.

This patch does a few things to speed up probe performance by ~50% in a fact to dim
table join on a single int64 key.

- Change IR function to return int instead of status. This doesn't change performance
but makes the IR much easier to read.
- Switch from FNV to murmur2 hash. I had done some experiments before and this function
is much faster than FNV and produces better hashes.
- Remove some unnecessary state in HashTableCtx that was not necessary and generating
load/stores.
- Cache the per partition hash table instead of walking the hash_partitions_ structure.

This takes us from roughly 2x slower from v1.4 to ~35% slower. If I switch the hash
function to CRC (like in v1.4) the perf difference is <5%. In a follow up patch, we'll
probably have to use CRC as the first level to get perf parity in the non-spilling
case.

Change-Id: I7481a35ed767d4817e78d06fe85016408620a85a
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4325
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Fix AVG codegen

We weren't returning the right merge function for decimal in
GetAvgFunction(). Someday the functions will be registered in the FE
like for scalar functions.

Change-Id: I1153ef8570b0e78f0925b7d3d58ec3b0fbb2c589
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4336
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Codegen SUM(decimal)

On a simple local benchmark that sums a decimal column, this improves
query time by ~4x.

Change-Id: I4968a31c66f94ac55b1d3a6e51707f72b81b662a
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4341
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
CDH-21520: Better error message for invalid HdfsScanNode cardinality_

Change-Id: I6aaf22826d9108cd8a6e32416d636806f0631523
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4339
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Add null check for database in HS2 MetadataOp

It is possible the database was dropped between the call to getDbNames() and
getDb(dbName) causing a NPE.

Change-Id: Icc4d0f5c54995783dd59d73f2db3050369a6f944
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4343
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
[CDH4] Disable TestCancellationFullSort (see IMPALA-1240)

Change-Id: Id85aa2792b7e3bbf61837bb5399623d2810be72e
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4354
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-1251: Fix hang for high-offset queries

If a row batch exceeded its memory capacity without containing any rows,
ExchangeNode::GetNextMerging() would spin in its loop which erroneously
expected at least one row to be produced by its child on every
iteration. The fix is to check for capacity once every time through the
loop and return to let operators up the line deal with the condition.

Change-Id: I2c015a4c91a7c1cf8b78f911273bb4f1ba87c9af
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4356
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-995: Add plan hints embedded in comments and preserve them in views.

This patch adds two new hint styles:
1. Traditional commented hint: /* +hint1,hint2,hint3 */
2. End-of-line commented hint: -- +hint1,hint2,hint3\n

We now preserve hints when creating views. We always use the
end-of-line commented hint style to allow Hive to read
hinted views created by Impala. Hive does not support
traditional /* */ comments, and attempts to parse /*+ */ as
hints, failing with a parse error on unrecognized hints.

This patch also changes Impala to only issue a warning
for unrecognized hints instead of throwing an error. This
allows Impala to run against hinted views created by Hive.

Change-Id: I6e8352442e763c0029f72c17363caa087572dca0
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4235
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-1151: Fix use-after-free in webserver

It's an error to add a temporary string directly to a RapidJson document
without wrapping it in a Value object first, because the latter will
copy the data whereas the former only moves it. Unfortunately, RapidJson
doesn't do anything to prevent this usage (which is, at least, safe for
literals).

Webserver::RootHandler() had several of these errors. I don't think we
ever saw a related crash, but if someone had hammered
http://<impalad>:25000/ there may have been an issue.

Change-Id: Ic3d5f6bdd7e60f4660f4eb713385664e76fb565f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4360
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
Tested-by: jenkins
IMPALA-1257: Fix asan issue with string literals.

Change-Id: I31a24a3323081b6486bf7baf450f8fb2fdc56ec5
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4363
Reviewed-by: Ippokratis Pandis <ipandis@cloudera.com>
Tested-by: jenkins
IMPALA-1105 : TRUNC(TIMESTAMP) malformed input from cast

Fixed crash when TRUNC(TIMESTAMP) was passed malformed timestamp values. The fix
is for CAST(STRING to TIMESTAMP), returns NULL when STRING is malformed.

Change-Id: Ie13395d740b2587dfb59c2f677b3ed3d0d4e992f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3583
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
Buffered Tuple Stream fixes.

This patch fixes two issues:
  - Add API to buffered block mgr to allow an atomic Unpin and GetNewBlock. This has
    the semantics of unpinning a block and giving the buffer to the new block. This
    is necessary for the tuple stream to make sure another thread does not grab the
    unpinned block in between.
  - Buffer management reading an unpinned stream. Before moving onto a new block (and
    unpinning the current), we need to make sure all the tuples returned from the
    current block are returned up the operator tree.

Change-Id: I95ee58d1019dd971f6a7dc19ecafdfa54cdbf942
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4333
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Fixed TPCH Query 15

Rewrote TPCH Query 15 to use an inline view.

Change-Id: I6755cc054ead14b032f7f59ea0f519b45e404b7d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4287
Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Tested-by: jenkins
IMPALA-1240: TestCancellationFullSort hits mem limit of 300MB with tpch_parquet

The test sets a mem_limit of 300MB and does:

   delay(t)
   assert(query_state != EXCEPTION)
   cancel()

If the mem_limit is hit before the delay completes, the assert will fail.  So,
the test is racy.  It appears we are right on the 300 MB memory limit (on my
machine, the query completes with peak mem usage in the high 200 MB).

I'm not sure why the mem_limit dimension was added, but given that it makes
the test racy and doesn't seem to add value, let's remove it and re-enable the
test.

Change-Id: I63767161bec6d7d6cd3de2e4563425297ffccb0d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4366
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Properly initialize variables in DecimalAvgGetValue.

This was causing the ASAN failures.

Change-Id: I31b7ea4059d2ac6be295b2f5c19339b79ed3b30e
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4374
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>