Accept spaces in query IDs for backwards compatibility with CM4.x

Change-Id: I809f46b1a12519da6dcd0fe45a9d6d9c21eb538e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/799
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
Fix block queue test.

Change-Id: I0bf1cb09f9c21f18e9d417c902e7b3ed607c5694
Reviewed-on: http://gerrit.ent.cloudera.com:8080/779
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Bump version to 1.2.1-Internal.

Change-Id: I8b8ff39799652031d6d625da1d9a1d20e8acecb3
Reviewed-on: http://gerrit.ent.cloudera.com:8080/760
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
IMPALA-598: Order by behavior for NULLs should be revisited

This change modifies that behavior of NULL ordering such that nulls always
compare greater than other values, but "nulls first" or "nulls last" can be used
to explicitly specify if nulls should be sorted first or last regardless of the
asc/desc.

Change-Id: I92feda1e7f42249de4009afd39f8395a0a32a2f8
Reviewed-on: http://gerrit.ent.cloudera.com:8080/812
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Update KeyNormalizer to take a vector<bool> for nulls first/last, per-expr

In IMPALA-598 we added support for NULLS FIRST/NULLS LAST, which allows
for ordering nulls first/last per-column. This updates the KeyNormalizer
to order nulls first/last on a per-expr basis. This is currently only
used by the sorting experiments. The tests for the sorting experiments
were not working, so a few small changes were needed in the sorting code
just to get the tests passing.

Change-Id: I10e640650eefdbac0435f12d340b452c3732e365
Reviewed-on: http://gerrit.ent.cloudera.com:8080/822
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
IMPALA-531: Allow arithmetic expressions for LIMIT

Change-Id: Ic1901e9dbaeee5fb0aef72a278b4aa262a2abcd7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/829
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Add alias in impala-config.sh to gerrit-merge-verify script

This alias makes it easy to verify and merge Gerrit changes

Change-Id: Idb0b5c3e6c825721e375bdf0c86b5975df1ed4b9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/836
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: Lenni Kuff <lskuff@cloudera.com>
Utilities to deal with millisecond timestamps since the epoch

Despite having access to both TimestampValue and boost::posix_time, we
have no simple utilities to deal with timestamp values represented by
milliseconds since the Unix epoch. These are extremely useful to
have for the following reasons:

* They do not wrap (unlike local_time()::milliseconds()), and thus are
  less prone to comparison bugs.
* Arithmetic on them is extremely quick
* Their units are extremely explicit - *always*
  milliseconds. The provided SleepForMs() always takes milliseconds as
  well, leading to fewer conversion bugs when using usleep directly.

There's nothing here that you can't do with posix_time, but the simpler
interface makes common bugs harder to write. TimestampValue and
posix_time are still preferred for more complex scenarios.

Change-Id: Iee525d50a24ce61ce7d7cfb026af5c4dd2c70cf7
Reviewed-on: http://gerrit.ent.cloudera.com:8080/835
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Added SHOW TABLE/COLUMN STATS command.

Fixed the following stats-related bugs:
- Per-partition row count was not distributed properly via CatalogService
- HBase column stats were not loaded and distributed properly

Enhancements to test framework:
- Allow regex specification of expected row or column values
- Fixed expected results of some tests because the test framework
  did not catch that they were incorrect

Change-Id: I1fa8e710bbcf0ddb62b961fdd26ecd9ce7b75d51
Reviewed-on: http://gerrit.ent.cloudera.com:8080/813
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Add metrics for tcmalloc

The behaviour of tcmalloc causes quite some confusion amongst our
users. Handily, it exposes a number of key metrics that we can ourselves
expose so that CM or other tools can retrieve them, and make sense of
them for our users.

To do this, this patch adds a callback for each metric that is invoked
every time the metric's value is retrieved. This allows subclasses to
implement UpdateValue() themselves and calculate a value on
demand. Since Metric::value() itself is templated, it can't be virtual
and therefore can't be overridden to get the same behaviour.

Change-Id: I01daf751573a57fd00672ed2a80ed10221c4cf40
Reviewed-on: http://gerrit.ent.cloudera.com:8080/846
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Re-enable process failure tests + add simple failure tests for catalogd

This brings back online the process failure tests and adds a basic failure
test for the catalog service. The timeouts had to be adjusted to account for the
extra time it takes to load the the catalog and also there is an additional state
store subscriber. Note: the statestore 'live.backends' metric which is used in these
tests needs to be renamed, it really means 'live.subscribers'. However, it requires some
coordination with other teams to make the change.
Also updated start-impala-cluster to check the catalog.ready flag to ensure the impalad
catalog is ready to accept queries.

Change-Id: If22e25dba7dc83aa40bec937b5f82b815bed4645
Reviewed-on: http://gerrit.ent.cloudera.com:8080/730
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Update mem pool to support TryAllocate and RcFileScanner to use it.

We've got users exceeding the mem limit by a large amount (e.g. 20GB). This
is likely because we allow the limit to be exceeded and check after. This
usually works pretty well except for some allocations that we know can be
very large. The RcFile scanner has one such place and we can extend using
TryAllocate() to more places in the future.

Change-Id: Idd62b9be621be05008a8b6ea812e9763504e3527
Reviewed-on: http://gerrit.ent.cloudera.com:8080/758
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
IMPALA-395: Impala "show create table" statement

Adds support for "show create table", a DDL statement that outputs a DDL statement that
creates the specified table.

In general, the output DDL works in Impala, so a user can copy the output and execute it
to create the same table. However, there are a few special cases that output Hive DDL
because we do not support creating some tables in Impala: HBase tables and tables with
LZO compressed text. When we do support creating these tables in Impala, users should
be able to execute the DDL in Impala as well.

Change-Id: I8c130297a657810dea5b994bf99d72b0e61b847b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/842
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
IMPALA-620: Fix re-registration starvation bug in statestore

This patch fixes a slightly pathological state that occurs when the
statestore is under heavy load. The result of the bug is that
subscribers cannot successfully re-register because the statestore never
marks them as failed.

The exact sequence of events is as follows:

1. Subscriber registers with state-store.
2. Statestore does not send heartbeats in timely fashion to
   subscriber. Subscriber times-out.
3. Subscriber is restarted quickly. Statestore does not detect
   restart.
4. Subscriber's RegisterSubscriber() call fails, because statestore
   detects duplicate registration.
5. Subscriber restarts again. Since state-store is slow to send
   heartbeats, the state-store has not detected the restart and the
   subscriber receives a heartbeat message from the statestore and
   does not reject it.
6. Statestore continues to believe subscriber is alive, since the
   heartbeats are not being rejected.

To fix this, we add a registration ID to each successfully registered
subscriber that is known to both subscriber and statestore. If the
subscriber should restart and re-register, it receives a new
registration ID. Whenever a heartbeat arrives, it compares its
registration ID to that sent by the statestore with the heartbeat, and
rejects the heartbeat if they do not match.

We also allow re-registration of existing subscribers (getting rid of
the dreaded "Duplicate subscription" message). A new registration
overwrites an old one.

Change-Id: Ie32df3a586ccb375375ebfbcbec1aaeb930b6bfe
Reviewed-on: http://gerrit.ent.cloudera.com:8080/778
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
IMPALA-470 (part 1/2): Expire idle sessions

This patch monitors the last time that sessions were accessed via
RPC. If that time exceeds a new parameter --idle_session_timeout, the
session is 'expired', meaning that all incoming queries to that session
are rejected, and all running queries associated with that session are
cancelled.

The second half of this patch will deal with idle query expiration.

Change-Id: I2d892f71876d2d627225153637e3bb0b15a76fc2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/604
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Fix BitUtil::Log2()

Change-Id: I25debeebe6864ede2ea19f15739201111f0432e0
Reviewed-on: http://gerrit.ent.cloudera.com:8080/889
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-636: Remove entry from backend_map_ if there are no more backends with that IP.

Change-Id: I4e38b47133cc2879f45c001764082893ce47c7b2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/891
Reviewed-by: Alan Choi <alan@cloudera.com>
Tested-by: jenkins
Add user and query_id to FunctionContext.

Change-Id: I0ae1523191edd23e2f17f942129ed0fd49912506
Reviewed-on: http://gerrit.ent.cloudera.com:8080/911
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Fix mem tracker reporting on error.

Change-Id: I4f2de591d03093d8eca5180c98d2f38d3ef709f9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/931
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
Allow different Kerberos principals for front- and back- end services.

This patch allows different principals for client and server
services. The server principal is specified by --be_principal (unless
not set, in which case --principal is used). Kinit is run for the server
principal only.

This patch also cleans up a few errors in the way Kinit was run, so that
failure can be detected more accurately.

Change-Id: I30bf747666caca17772ad30a7d14eab03ce799d6
Reviewed-on: http://gerrit.ent.cloudera.com:8080/890
Reviewed-by: Alan Choi <alan@cloudera.com>
Tested-by: jenkins