Skip generation of version and impala-ir .cc files in buildall.sh if -noclean is specified.

Before this patch the -noclean option had almost no effect on the BE build time because
some source files were re-generated with .py scripts regardless.
This change allows ./buildall -skiptests -noclean to do a true incremental rebuild.

Change-Id: Ib3af85db05bdc96a2279a22c1d49d735f2cabd4e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1394
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Fix race when invalidating catalog metadata and loading a new table

There was race when the catalog was invalidated at the same time a table
was being loaded. This is because an uninitialized Table was being returned
unexpectedly to the impalad due to the concurrent invalidate.

This fixes the problem by updating the CatalogObjectCache to load when
a catalog object is uninitialized, rather than load when null. New items can
now be added in a initialized or uninitialized state; uninitialized objects
are loaded on access.

Also adds a stress test for invalidate metadata/invalidate metadata <table>/refresh

In addition, it cleans up the locking in the Catalog to make it more
straight forward. The top-level catalogLock_ is now only in CatalogServiceCatalog
and this lock is used to protect the catalogVersion_. Operations that need to
perform an atomic bulk catalog operation can use this lock (such as when the
CatalogServer needs to take a snapshot of the catalog to calculate what delta to send
to the statestore). Otherwise, the lock is not needed and objects are protected by the
synchronization at each level in the object heirarchy (Db->[Function/Table]). That is,
Dbs are synchronized by the Db cache, each Db has a Table Cache which is synchronized
independently.

Change-Id: I9e542cd39cdbef26ddf05499470c0d96bb888765
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1355
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
IMPALA-781: Block Impala from reading RC file tables created using the LazyBinaryColumnarSerDe

With Hive .12, the default RC file format can be configured to be ColumnarSerDe or
LazyBinaryColumnarSerDe. Impala does not yet support the LazyBinaryColumnarSerDe. This change
verifies it is properly disabled.

Change-Id: Ia84495868237ce2c89a9706ad75e0f7eb8499057
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1416
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Reduce duplicate/unused string data from FileDescriptor/FileBlock thrift structs

Made the following changes:

1) Removed filePath from FileBlock. It wasn't used anywhere.
2) Update FileDescriptor to use file name rather than file path. The full path to the file
   can be found by prepending the parent partition directory the file name.
3) Removed fileLength from FileBlock. This was the same length used in FileDescriptor.

Testing these changes out on two table on a 20 node cluster. One table had ~100K blocks
and one table had ~15K blocks.

In both cases I saw the following improvements:

* After making change 1) the total serialized size of the table dropped by ~30%
* After making change 1) and 2) the total serialized size of the table dropped by ~52%
* After making change 1), 2), and 3) the total serialized size of the table dropped by ~55%

Change-Id: Ic85b3cbcf775569f69b7303bec4adc52593fc35c
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1351
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Remove duplication of datanode network addresses in HdfsTable

This removes duplication TNetworkAddresses in the block location metadata of HdfsTable.
Each HdfsTable now contains a list of TNetworkAddress and the BlockLocations
just reference an index in this list to specify the host, rather than duplicating the
TNetworkAddress.

For a table with 100K blocks, this reduces the size of the THdfsTable struct by an additional
~50+% (on top of the duplicate file path changes). This takes the total size of the table
from:
21.1MB -> 9.4MB (file path duplication) -> 4.2MB (host duplication) = ~80% total improvement.

Change-Id: If7f11764dc0961376f9648779d253829f4cd83a2
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1367
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
IMPALA-774: Fix runtimestate setup when evaluating expr from FE.

We weren't initializing the udf mem pool causing UDFs to return strings to crash if used as part
of a constant expression.

Change-Id: Ic3a0e556aec8ce03a9e59f3ccf6980c682046b50
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1393
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
[CDH4] Revert subscriber name changes for compatibility with CDH4 CM

Change-Id: I26f2773869821a72821d52f926d16145e733cf4d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1425
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Cleanup for IMPALA-774 fix.

Change-Id: I47bce71c482b3576957e88980f764c30f45229a9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1454
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-770: Fix crash in aggregation node with zero-width tuple

The select exprs of an inline view may not always be materialised, yet
the output tuple itself may be. This patch fixes a crash in this
situation in the backend aggregation node which assumed its output tuple
would always have at least one materialised slot.

The cause was a couple of too-conservative DCHECKs that failed if the
tuple was NULL. In fact, the code was robust to this possibility without
the checks, so this bug didn't affect release builds of Impala.

Change-Id: If0b90809d30fcd196f55197953392452d1ac9c4f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1431
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
[CDH4] Updates to IoMgr from adding cached read support in C5.

To minimize the merge cost in the IoMgr code going forward, this is a
cherry-pick of the CDH5 changes with the minimal amount of code removed
to run on CDH4.

Change-Id: I66cb8d0db75e7cad678d3bb4fcf11b338c1446ae
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1455
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Query admission controller, part 1

This adds a simple admission control mechanism which is able to make
localized admission decisions and can queue some number of queries
that are not able to execute immediately. In this change, there is
a single pool and the maximum number of concurrent queries and the
maximum number of queued queries are configurable via flags, but the
data structures all support multiple pools so that we can later add
support for getting pools and per-pool configs from Yarn and Llama
configs (i.e. fair-scheduler.xml and llama-site.xml).

Each impalad keeps track of how many queries it has executing and
how many are queued, per pool. The statestore is used to disseminate
local pool statistics to all other impalads. When topic updates are
received, a new cluster-wide total number of currently executing
queries and total number of queued queries are calculated for each
pool. Those totals are used to make localized admission decisions in
the AdmissionController.

There are a number of per-pool metrics which are used in automated
testing (in a separate commit) and there are many assertions and
debugging logging which I've been using to verify manual testing.

Change-Id: I68f92c789108336fca33c2148a4e14534c77e9f0
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1347
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Break up total network time into send and receive time.

CM has requested this and it's I think a very good way to
summarize what is taking time for each fragment. Each of
the counters in this group are non-overlapping and they
tell you what hardware resource the fragment is bottlenecked on.

Change-Id: I34a42361749c0b9c5fec39f48b588e27ca402bbb
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1472
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
IMPALA-735. Fix codegen bug affecting outer joins.

Change-Id: I99ca45b558fb2ed694f261a22e7e91e59f1ad675
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1407
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Admission controller: protect queue access and mark pools for updates

This fixes two bugs:
1) AdmitQuery() accesses the request_queue_map_ without holding the
   lock and the unordered_map is not thread-safe.
2) Dequeued requests were not causing pool stats to be sent.

Change-Id: I6d21dd69a6c8b59da0506d90f97ad0d996d8751b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1488
Reviewed-by: Henry Robinson <henry@cloudera.com>
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Fix broken udf test case. Should not specify DB.

Change-Id: I5f6343cbef9f52d349130360e029b38b23d0187a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1494
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Add a time constraint while evaluating regressions in performance reports.

For very short queries, using percentage as a strict bound for detecting regression does
not cut it. This introduces a flag through which the user can set a value (in seconds),
which will only attempt to detect a regression if the time difference is equal to or
greater than the value.

Change-Id: I95bde65afb8fd564666223fe069537e2483ec98e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1498
Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
Tested-by: jenkins
Skip the invalidate metadata stress test until the race in the catalog server is resolved.

Change-Id: I71911078d274f894f5a28c0e7123e5e5ac8dc940
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1507
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
IMPALA-786: Drop function should clear library cache.

We were previously only clearing the cache in the catalog service
update loop so the impalad the drop was issued to was not doing the
right thing.

Change-Id: I6bee228e8c0d565cea4ea61cbf64240d83a45a7d
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1445
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-736: invalidate metadata <table name> should add/remove table from catalog if it does/does not exist in the metastore

The invalidation logic is:
- If table exists in the metastore, add it to the catalog as an IncompleteTable.
  If the table's parent database does not exist in the catalog, it will also be added.
- If the table does not exist in the metastore, remove it from the catalog cache.
- If we are unable to determine whether the table exists in the metastore (there was
  an exception thrown), invalidate any existing entry by replacing it with an
  IncompleteTable.

Change-Id: If64f07950324a1bec186f9c9ce829197cad87044
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1301
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
IMP-1133: Add pause before renewing Kerberos ticket for the first time.

See https://issues.cloudera.org/browse/hue-640 for a full description of the issue. The
summary is that kinit -R can't come less than a second after the initial kinit.

In order to separate out two separate calls this patch adds a method to run a shell
command and gather the output.

Change-Id: I65b79150c1df0d3f08f4200f59088eab644b0f35
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1387
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins