Remove --disk_spill_integrity; merge into --disk_spill_encryption

There appears to be little reason to separate disk spill integrity and
encrytion into two different flags, so this submittal removes the
integrity check flag and makes that functionality available when disk
spill encryption is turned on.

Change-Id: I046966c33422736cb7315f7f378af06b9808a83f
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4547
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-1310: Handle creating parquet files with many columns

The parquet writer assumes a minimum size per column, but for very wide
tables with enough columns (> ~1300 columns for 256 MB parquet file, >
~2600 for 512 MB file, etc)), this will not be true. If the user did not
set the parquet_file_size query option, then automatically adjust the
file size to satisfy the minimum size per column constraint.  Otherwise,
if the query option was set, always use that value but return an error
if the size is not sufficient.

Finally, set the default parquet file size back to 256 MB now that the
test has been fixed.

Change-Id: Ie66cf50d429993d205e278df75f73a04a80b0009
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4558
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Fix HBase region splitting for tests.

It appears that HBase sometimes ignores an admin.splitRegion() RPC,
which made our region splitting fail. As a workaround, this patch adds
another retry loop such that the split/wait sequence is attempted
multiple times.

Change-Id: I9aa8ab87bba79ea11b79c50f15328b8be844924d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4557
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
IMPALA-1308: estimate scanner thread memory requirements based on file size

We use a constant memory overhead when estimating if we have enough
memory to start a new scanner thread. This doesn't work well for
compressed text files, which need to be entirely decompressed in
memory. Our constant estimate was often too low in this case, which
could cause threads to OOM. Instead, for compressed text files, base
the overhead on the maximum file size.

This is tested in test_mem_usage_scaling.py, which failed on text/gzip
before this patch.

Change-Id: I151704df6b4ead2b7ebeb1b2103af056a8f0b42d
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4545
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Free local UDF/UDA allocations

UDF/UDAs use FunctionContext::AllocateLocal() to allocate memory that
is owned by Impala and can be freed at any time. This is used to
return string data to Impala. However, we weren't freeing local
allocations before the end of the query, even if the returned data was
no longer needed. This meant that queries like "select
min(lower(string_col)) from tbl" would never free the memory allocated
by lower(), effectively accumulating the entire dataset in memory.

This patch adds calls to FunctionContext::FreeLocalAllocations() so we
don't accumulate unneeded memory. We don't want to call this too often
(i.e. for every row evaluated) for performance reasons, so the
allocations are freed in RuntimeState::QueryMaintenance() (renamed
from CheckQueryState()).

I'm not sure how to test this, but ad-hoc testing confirms that this
does reduce peak memory usage.

Change-Id: I1b973a027151a86521057756f4fe9bf7954881bd
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4536
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Codegen-enabled operators should create codegen object immediately

ScalarFnCall exprs will use codegen if the codegen object is already
created, even if the operator calling the exprs does not codegen. This
is good for performance. After the lazy codegen intitialization
change, ScalarFnCalls in e.g. scan nodes were not using codegen
because the fragment's codegen object would be created later in the
Prepare() phase, hurting performance. To fix this, have
codegen-enabled operators create the codegen object immediately before
preparing child nodes / exprs.

Change-Id: Id10e97d3fa2f1e5cfe9236fa1c83bea3384334b0
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4566
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
[CDH4] Bump version to v2.1.0

Change-Id: I8950804efd1df67684dfe6e8c4a280904cf958ef
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4569
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>