IMPALA-161: Add Impala support for CREATE TABLE AS SELECT

This adds support for CREATE TABLE AS SELECT to Impala. It supports all functionality a
regular CREATE TABLE statement includes, except it does not allow for for specifying
partition columns. Hive also has this limitation and it wouldn't be too hard to support
in the future.

Change-Id: I4ca3c3b8f1576441b8bb5ed9dc521d7dfa96ab74
Reviewed-on: http://gerrit.ent.cloudera.com:8080/157
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
Create partitions in parallel in the frontend

This patch parallelises the creation of partitions as a result of an
INSERT query. The result is a reduction from ~4s to ~1s in the time to
insert into 100 existing partitions (with 16 worker threads) on a single
machine.

Change-Id: Ib8f84ba08c90d002a2b47c33187fc178f4493119
Reviewed-on: http://gerrit.ent.cloudera.com:8080/243
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Add wrapper for boost threads.

* Allows access to the thread ID for assignment to cgroups and getting
  stats via /proc
* Adds /threadz page to debug webserver that shows threads by group, and
  breaks down CPU usage by thread

Change-Id: Id3b9aae92f4ae5c01ed1aab2185bbaa99fce4385
Reviewed-on: http://gerrit.ent.cloudera.com:8080/94
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
(IMPALA-422) Add string concatenation function

Implements a group_concat() function which concatenates all the values in a group together.

The format is group_concat(str_col, [separator]). The default separator is ', '. NULLs
are ignored.

Change-Id: If152df6f528401117dba81d66ef691bfb548cc7d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/117
Reviewed-by: Aaron Davidson <aaron.davidson@cloudera.com>
Tested-by: Aaron Davidson <aaron.davidson@cloudera.com>
Always signal HdfsScanNode::row_batch_added_cv_ while holding row_batches_lock_

HdfsScanNode::GetNext waits on row_batch_added_cv_ while holding row_batches_lock_, but
the error path in ScannerThreadHelper does not take row_batches_lock_ while setting the
associated condition (done_ == true) or signalling the cv itself. This leads to the
possibility of a race when GetNext might be spuriously awake and between checking done_ ==
true and rewaiting for the cv. Holding the lock while setting done_ also guarantees
visibility between the two threads.

Change-Id: Iad050fcd4302240cae2074ae4f2a1d64f3eaafe3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/286
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
IMPALA-544: Fix debug counter tracking in io mgr.

Change-Id: I8ca972b6752c011edada67d1ee39cdf8025ca36a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/287
Tested-by: jenkins
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Vary number of build threads based on number of cores

Simply makes buildall.sh and the make_*.sh commands use 2 * ncores
build threads. ncores includes logical CPUs.

Change-Id: Ib3fbf1f1c8362c5bd3afab61f4d3030a50c51c10
Reviewed-on: http://gerrit.ent.cloudera.com:8080/288
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Add membership size as a impalad metric.

We'd like CM to validate this is reasonable.

Change-Id: Ic249af0ecb46200e87c21a6a57769998c3f3fbd3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/291
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
IMPALA-549: Failure during Prepare() of TopNNode causes segfault due to NULL ptr access

Change-Id: I5756195dcc818db0cbd8a28434363dae93aa77a5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/296
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Convert audit log flush threads to impala::Thread

Change-Id: Idf30e053adb7845719cc73afdd73f24c98bf27f0
Reviewed-on: http://gerrit.ent.cloudera.com:8080/283
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Cleanup start-impala-cluster and use exec to start processes

This change cleans up start-impala-cluster to remove all the uneeded log4j setup code.
As part of this change, updated the start-impalad script to "exec" the impala binaries,
which removes the .sh wrapper script from the list of running processes.

Change-Id: I5dee49b72ff51012bf43ab9d2a3a21fd2b841ff5
Reviewed-on: http://gerrit.ent.cloudera.com:8080/270
Tested-by: jenkins
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Ishaan Joshi <ishaan@cloudera.com>
External sorting

This is an experimental implementation of external sorting. This patch includes the following additions:
(1) creation and implementation of the Sorter interface, which can sort Impala Tuples.
(2) normalization of Tuples to allow memcmp-able sorting.
(3) a testing framework for the Sorter,
(4) a benchmark to compare the current state of the Sorter with other sorts,
(5) an implementation of a Vector which can store data whose size is only known at runtime,
(6) a sorting algorithm (basically a dumbed down STL sort) which can operate over such a vector,
(7) implementation of a simple in-memory Merger, and
(8) logic to stream blocks of memory in and out of memory for the actual external merging.

I have a local branch for experimental optimizations and benchmarking -- this should be considered
a "basic", working sort.

The following optimizations have been implemented:
(i)   Optionally extracting keys instead of writing them in place.
(ii)  Optionally opportunistically parallelize run building (sorting & prepare for output).
(iii) Maximize disk IO and minimize buffer recycling by writing buffers out, but also keeping
      them in memory until right when they're needed.
(iv)  Prepare auxililary data backwards so the buffers can be released as we go, and still
      go out in an order which preserves the first buffers of the run.
(v)   Always merge maximum number of runs at a time, taking from the next merge level if
      available.

Change-Id: I1d7304d54d73152da929b1efffc1e851e5fb8fd4
Reviewed-on: http://gerrit.ent.cloudera.com:8080/126
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Aaron Davidson <aaron.davidson@cloudera.com>
Handle malformed HS2 THandleIdentifiers without crashing

THandleIdentifier is required to have two 16-byte members, but the
client supplies this data structure and Impala would crash (either by
DCHECK or by corrupting memory) if they were too short. This patch turns
that error into an exception returned to the client.

Change-Id: I843362cc9c9bd6c8bf88d97e80c56cd3c4074470
Reviewed-on: http://gerrit.ent.cloudera.com:8080/315
Reviewed-by: Alan Choi <alan@cloudera.com>
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Fix packaging build.

Change-Id: Ib30232086f62d90b531fd7045f8fb3b098c61fda
Reviewed-on: http://gerrit.ent.cloudera.com:8080/328
Tested-by: jenkins
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Add counter cleanup in HdfsScanNode::Close()

If UpdateCounters is not called, the periodic counters don't get unregistered and
we are at the mercy of the OS scheduler for proper object cleanup order. We want
to call this in GetNext() if possible but it's good practice to have mirroring
cleanup code for everything created in Prepare()/Open().

I think in the case of this issue, it was because of the RETURN_IF_CANCELLED in
GetNext() which meant UpdateCounters() was never called.

Change-Id: Iae38da18563898d2e16dd7c540de27d676d8dfee
Reviewed-on: http://gerrit.ent.cloudera.com:8080/337
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Set compact_data appropriately.

Change-Id: I3f2602a2b48ac7e0747433aa427cf6ff3be7cef2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/338
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Add a few client tests for HS2

These HiveServer2 client tests (in tests/hs2) are intended to check the
HS2 API implementation in the following ways:

* API tests: Can all the supported API endpoints be called successfully?
* Query lifecycle tests: does calling the API in an unusual sequence
  result in reasonable behaviour?
* Malformed query tests: does sending an incorrectly constructed request
  correctly result in an error?

This patch adds a few simple tests as a starting point for a larger test suite.

Change-Id: I4b926d1639c640317ea3478bdeb0aa4b5a9286ee
Reviewed-on: http://gerrit.ent.cloudera.com:8080/320
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
IMPALA-541: Remove race for QueryExecState::query_events()

EventSequence counters are not thread-safe. This patch fixes a race
between UnregisterQuery and Wait in ImpalaServer that causes both to try
to mark an event at the same time.

Change-Id: Ie0518c19b7d2158fefd9e4a687c6f3febaf05a3f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/340
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Fix unclosed session in TestHS2

The HS2 server does not close sessions when the associated socket
connection fails (unlike Beeswax); if a session is not deliberately
closed it persists forever.

Change-Id: If2d0bfade26a27225023f4f80482bf34132c55c2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/341
Tested-by: jenkins
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Avoid calling Init() twice.

CreateCompressor() and CreateDecompressor() call Init() on the
returned codec, so Init() should not be called elsewhere in order to
avoid memory leaks. This patch also cleans up the Codec interface to
enforce that Init() is always called exactly once via
CreateDe/compressor().

Change-Id: I1ff82db009b2fde5c9ec2874a1e16ea9f6f4db9e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/331
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins