[CDH4] Bump version to v1.4.0-cdh4-INTERNAL

Change-Id: Iac935f003d9be2d5ce76b5a1d8f1a374bde55c02
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2140
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
Cleanup and re-enable some tests in TestPartitionMetadata

Partition metadata tests were marked as xfail because of IMPALA-624. Additionally, we had
to invoke hive to insert into two partitions pointing to the same location (this
limitation is now removed). This patch changes the test to use Impala exclusively,
removes the xfail tag and adds a teardown method to the test class.

Change-Id: I15fa97bef4f8714d0873a9c713627a198f3388ad
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2086
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
IMPALA-904: Make TupleIsNullPredicate work on non-nullable tuples.

We wrap certain exprs substituted from outer-joined inline view in an expr that
evaluates to NULL if the underling tuple(s) are NULL. We do this for exprs that evaluate
to non-NULL values if their slots are NULL, i.e., we must then distinguish tuples that are
NULL from slots that are NULL (otherwise evaluating an expr against a tuple that is NULL
due to the outer join may incorrectly return a non-NULL value.)

The bug: Exprs referring to an outer-joined inline view may appear in various places
in the outer query block. For example, they could appear in an On-clause or be
placed into scans/aggregates due to predicate propagation. In such cases, the underlying
tuples may not be nullable yet because they only become nullable after the outer join.
We had a DCHECK in tuple-is-null-predicate.cc requiring the tuples to be nullable.
The fix: Remove the DCHECK. The fix is not elegant but practical. It would be rather
difficult to fix the inline view expr substitution such that a TupleIsNullPredicate
never references a non-nullable tuple, esp. due to predicate propagation.

Change-Id: I180f75f14173f356abfeec751e6b2d419378a9a7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2146
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
IMPALA-935: Make PlanFragment.getDestFragment() return null if no destination is set.

Change-Id: I269a7f552d7ff67ff4d65e86e8c6df9c41d0fca1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2153
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-901: Fix grouping with NULLs when codegen is disabled

The standard implementation of HashTable::Equals() did not correctly
check the NULL bit when the argument row did not evaluate to NULL for a
given probe expr. In the rare circumstance that this gave rise to a
false positive (more on that below), two rows with different grouping
values would be considered equal, and one would be excluded from the
final aggregation output.

HashTable::EvalRow() fills an expression value buffer with the values of
either probe or build exprs evaluated for the argument row. These cached
values are used to determine row equality in Equals(). In order to avoid
a lot of false collisions, an 'unlikely' value is written to that buffer
for NULL values, chosen to be HashUtil::FNV_SEED. So without correct
NULL-bit checking in Equals(), two single-slot rows are considered to be
equal if one of them has NULL for its slot, and the other has a value
equal to HashUtil::FNV_SEED truncated to the size of the slot.

For tinyint columns, this value is -59. As it happens, our random
generator happened to create a table with one tinyint column and which
contained NULL and -59 as values. In order to trigger this bug, the rows
must also have been written to disk in order such that the scanners
returned -59 *first*, and then NULL to the aggregation node; the bug is
not symmetric and works in the opposite case.

Change-Id: I17d43eaeee62b2ac01b67dd599bc4346b012a074
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2130
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
IMPALA-888: Materialize union slots referenced by constant predicates.

To keep the predicate assignment/propagation logic simple, we assign conjuncts
whose underlying base table exprs are constant in at least one union operand
to the evaluating MergeNode, and not in the operand(s) whose corresponding base
table exprs are constant.
The JIRA describes two different bugs:
The first bug was that the slots required for evaluating such predicates in the
MergeNode were not marked as materialized. The second bug was that predicates
'pushed' into union operands did not get re-analyzed after substituting the
predicate's exprs with the result exprs of that union operand. Missing casts
lead to a crash. The new test covers both bugs.

Change-Id: I0f5b8a366b32f7d4b2587e13793b6103cdf7e8b3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2135
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-13: Use SSE string functions that take an explicit length

This patch modifies DelimitedTextParser and StringValue to work with
data containing null characters by using SSE instructions that take a
length, rather than expecting null-terminated strings. It also adds
some other minor changes to correctly handle data with nulls and to
faciliate testing. I checked the execution time of a count(*) and a
select(*) limit 1 query locally, and saw no difference for either text
or sequence files.

Change-Id: Ia920b35bea7048aa286f39ec83e313c2a39251d1
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2110
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
Add support for SHOW PARTITIONS <table name>

This statement returns info on all partitions for the given table. It is implemented as
an alias for SHOW TABLE STATS, with some extended analysis checks (such as throwing if
the statement targets an unpartitioned table).

Change-Id: I19154a9d90314de18f86ba355aa5dbed808f147f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2145
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
Mark test_non_codegen_tinyint_grouping as execute_serially

The test contains an INSERT and some DDL, which is racy if performed in parallel.

Change-Id: I2b88533f45756fcf6372d6ee4eb7edd474087048
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2167
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
Break up locking in DiskIoMgr::ScanRange.

Currently, the entire object is protected by one lock. Unfortunately this
lock is taken during calls into libhdfs. This means it is impossible for the
scan node to pull off ready buffers while the disk thread is reading from
this scan range. Whoops. The lock is taken while in libhdfs to facilitate cleanup
so it's very easy to split the big lock up.

Change-Id: Idbf34cdba0cf860a90f9cad016d1ec133f923d85
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2143
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Fix buffer overflow bug in StringCompare()

Includes benchmark for comparing different StringCompare() implementations.

Change-Id: Ib4623b3ae6c99977af332ce5161da66af3cae9e5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2168
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
Testing: Generate queries and compare results against other databases

This is the intital commit and is a work in progress. See the README for a
list of possible improvements.

As an overview of how the files are related:

  model.py: This is the base upon which the other files are built. It
      contains something like a grammer for queries.

  query_generator.py: Generates random permutations of the model.

  model_translator.py: Produces SQL based on the model

  discrepancy_searcher.py: Uses the above to generate, run, and compare
      query results.

Change-Id: Iaca6277766f5a86568eaa3f05b99c832942ab38b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1648
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: Casey Ching <casey@cloudera.com>
IMPALA-932: evaluate concat/concat_ws children once

Change-Id: Id22a6c1dfb57cf659a1e24af4de6e5a2336cafa4
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2152
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
IMPALA-922: Return helpful errors with Yarn group rules

When the -fair_scheduler_allocation_path is configured with a policy that uses
the "primaryGroup" Yarn queue allocation rule, Yarn throws an error if the user
is not on the local OS. Currently the user will get an error message that says:
"java.io.IOException: No groups found for user <username>". We now return a more
helpful error message.

Change-Id: I014ac15ef607e473957752f23af94d0cc4efec0f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2078
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
IMP-1329,IMPALA-924: Make ExchangeNode::Open() block until rows are available.

The bug: Coordinator::Wait() is supposed to block until rows become available for
consumption by the client. We rely on Wait() to determine when to advance the query
status to a 'ready' state and signal to the client that rows can be fetched.
Long fetch times can trigger client timeouts at various levels (socket, app, etc.).
Coordinator::Wait() simply opens the coordinator fragment's plan tree.
For most plan nodes, Open() does work to prepare the plan tree, s.t., GetNext()
returns quickly. However, for ExchangeNodes Open() used to not wait
until rows are obtained form the underlying stream receiver.
The fix: Make ExchangeNode::Open() block until rows are available.

Change-Id: I7b197eea11d21fd732414d96c899a17b2d99631c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2128
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMP-1339: Fix crash in rcfile scanner from mem-pool tracking bug.

In the case where the MemPool fails in FindChunk, we were not properly
updating the MemPool's state.

Change-Id: I3ed9bd7ee9505cfaf4c7812304c1da85ae06f72f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2160
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-913: Revisit the use of FNV Hash in exchange

FNV hash has the property that the least significant bit of the hashed value
is just the XOR of the LSBs of its input bytes. This results in poor
distribution of rows when the partition keys are duplicated -- for example,
if the partition key is (l_orderkey, l_orderkey). A recommended technique
to mitigate this is to generate a larger hash and use XOR-folding to reduce
it to the desired length.

In this patch FnvHash has been modified to use generate a 64-bit hash and
fold the result down to 32-bits. It has been renamed FnvHash64to32 to make
this explicit.

Change-Id: Ie12ad3f863fca15092803d3e4d616a654cb8d244
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2101
Reviewed-by: Srinath Shankar <sshankar@cloudera.com>
Tested-by: jenkins
Adding TRUNC builtin.

Includes additions to builtin UDF registration to support prepare/close.

Change-Id: I22668fa7ee033b3fa37050b7bccee935571ac453
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2141
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Decimal implementation.

This patch implements decimal support for text based formats.

Change-Id: I8e2c9e512ed149fe965216a72cb21fffd4f18e75
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1669
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
Close all queries in test_cancellation

The queries in test_cancellation are currently cancelled but not closed, causing some test
queries to eventually time out because the admission controller limits are passed. This
patch ensures that all queries issued in test_cancellation are closed.

Change-Id: I65b26672155e31889bb6f43d3ac87be0f7b4eb72
Reviewed-on: http://gerrit.ent.cloudera.com:8080/2187
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins