Bump version to 1.2.2-INTERNAL.

Change-Id: I256ef47b6e957a2723422e606d1b87f4e800bbf9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1032
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Nong Li <nong@cloudera.com>
IMP-1158: Dropping a database in Impala does not cleanup the db's HDFS directory

We need to pass a flag to the metastore for the cleanup to happen. Previously we
were passing 'false' when we need to pass 'true' to get the same behavior as Hive
when dropping databases. Added a test case to validate the cleanup when dropping
databases and tables.

Change-Id: I500a3d3ac52c1b2031fae842403a670cfe43fa98
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1035
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Update DDL to support 'STORED AS PARQUET' and 'STORED AS AVRO' syntax

This change updates our DDL syntax support to allow for using 'STORED AS PARQUET'
as well as 'STORED AS PARQUETFILE'. Moving forward we should prefer the new syntax,
but continue to support the old.  I made the same change for 'AVROFILE', but since
we have not yet documented the 'AVROFILE' syntax I left out support for the old syntax.

Change-Id: I10c73a71a94ee488c9ae205485777b58ab8957c9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1053
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Drop thirdparty cyrus-sasl back to version 2.1.23

Some packaging problems ensue when building against 2.1.25, and the
functionality we had previously depended on in 2.1.25 is no longer used.

Change-Id: I6c95d30d4a78574f60540e0d91890cb4de7c2774
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1046
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
Build changes to use cyrus-sasl-2.1.23

Change-Id: Ie87e35945b6a415b0383cb75ffcae2fe35755623
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1047
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
Add openldap to thirdparty/

Openldap is required for LDAP authentication.

Change-Id: Ifcf4fd24a31c9a622e307be3a97129416729906b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1048
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
Build Openldap as a thirdparty package

Change-Id: Ifbb0f468a23186f4160fceb462953bc321469c27
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1049
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Henry Robinson <henry@cloudera.com>
Native LDAP authentication

Due to some difficulties with SASL's LDAPDB plugin connecting to Active
Directory, we're forced to implement our own mechanism for
authentication with LDAP servers.

To do this, we implement a callback that SASL defines called
'checkpass'. This method tries to bind with a user-configured LDAP
server, and fails the authentication process if not successful.

This callback is only invoked in the context of an auxiliary-properties
plugin ('auxprop'). We add a dummy auxprop plugin that does nothing but
provide the calling context for the checkpass callback.

All connections to the LDAP server are not encrypted. To enable
encryption at the transport level, users can set --ldap_tls to force
Impala to attempt a TLS connection, failing authentication if that's not
possible.

Finally, to allow users to provide a custom SASL configuration, they may
set --ldap_manual_config to bypass all the automatic configuration
provided by this patch.

Change-Id: Ia672ebba74d2513f1163d3beeef32f8e9be93b2f
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1050
Tested-by: jenkins
Reviewed-by: Henry Robinson <henry@cloudera.com>
Fix symbol mangling of BufferVal type.

Change-Id: I9e6232046341544f47918a00ec5579bf84ef68d9
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1015
Tested-by: jenkins
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Add per-topic memory consumption statistics to statestore's /topics page

The memory consumption is broken down by keys, values and the total, for
each topic. Three new metrics are added:

* statestore.total-key-size-bytes
* statestore.total-value-size-bytes
* statestore.total-topic-size-bytes

which reflect the total consumption across all topics.

Change-Id: I6861022870956069c76341f8a395c04ab1418034
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1051
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Fix subexpr elimination IR rewriting.

Change-Id: Iabdcc1686951e71136a603ed30f9d16fb1c1ec46
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1056
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Allow ALTER TABLE to set per-partition serde and table properties.

The main motivation is to allow users to set the per-partition number
of rows for manual incremental stats maintenance, as well as a means
to 'drop' stats that may have caused undesirable plan changes.

Change-Id: Iff38317a993e5d7952ea4df839947f5ec341e930
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1010
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Add analysis error for UDAs if intermediate type is not return type.

Change-Id: I6929a5e103d37246647ddd94a7fbf6c90f4df482
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1061
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
IMPALA-679: Append hash of HDFS path to filename in CopyHdfsFile() to avoid collisions.

Change-Id: Ia84fa81fe043a9604248d66ed963ef3f91b0601e
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1018
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Fixup error message returned to user for tables with incomplete metadata

When there was an error loading the table metadata the Catalog Server
includes (among other things) the table loading error message(s) and
the call stack where the error happened. The error messages returned
to the user previously included the call stack, but with this change
the call remote (catalog service) call stack is only output to the impalad
log when the table is accessed by the user. This is done by annotating
and appending the remote call stack to the local call stack.

Change-Id: Icf46a3d7fc15d8b0a8e59564722fdb991b074618
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1063
Tested-by: jenkins
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Improve table loading error message for table with unsupported serde library

Change-Id: I95f43ae31fd7f8714c61b505c27a22aecd31e3df
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1067
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins
Fix checking wrong variable in hdfs scanner thread's start new thread logic.

We're trying to check that we haven't already started one thread for each
remaining scan range but the variable we are checking is wrong.

Change-Id: I2d5f1902728fc9ccdd38a9d0b142d57d9a732a06
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1062
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
IMPALA-692: Fix data corruption with dictionary encoded values.

We weren't clearing the state in the dictionary when rolling over to a new
page. The memory for the dictionary (built from the first file) was cleared
but the dictionary entires were not.

This also had a minor side effect that unused dictionary entries from the first
page were still being written out for subsequent pages, although in practice,
this is unlikely to affect the file size much.

Change-Id: I8e11fc4723dc23d21c5de8a42def13d8238c137b
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1072
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Update ResolveSymbolLookup error message to be more helpful.

These are the errors that are returned when we unsuccessfully attempt
to mangle a function name to the full symbol. I find that in practice,
this usually fails due to not providing the correct arguments.

Old message:
Query: create function udf_count(bigint) returns int location '/test-warehouse/libudasample.so' symbol='CountUpdate'
ERROR: AnalysisException: Could not find symbol 'CountUpdate' in: /test-warehouse/libudasample.so
Could not find symbol.

New UDF message:
Query: create function udf_count(bigint) returns int location '/test-warehouse/libudasample.so' symbol='CountUpdate'
ERROR: AnalysisException: Could not find function CountUpdate(BIGINT) in: hdfs://localhost:20500/test-warehouse/libudasample.so
Check that symbol and argument types are correct.

New UDA message:
Query: create aggregate function udf_count(bigint) returns int location '/test-warehouse/libudasample.so' update_fn='CountUpdate'
ERROR: AnalysisException: Could not find function CountUpdate(BIGINT) returns INT in: hdfs://localhost:20500/test-warehouse/libudasample.so
Check that function name, arguments, and return type are correct.

New message with full symbol:
Query: create aggregate function udf_count(bigint) returns int location '/test-warehouse/libudasample.so' update_fn='_Zasdf'
ERROR: AnalysisException: Could not find symbol '_Zasdf' in: hdfs://localhost:20500/test-warehouse/libudasample.so

Change-Id: I602db6d3786f8351364ce8272b7dd930e04f5b51
Reviewed-on: http://gerrit.ent.cloudera.com:8080/1009
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Predicate propagation plus join order optimization.

Introduces STRAIGHT_JOIN keyword to prevent join order optimization.

Structural changes to the planning framework:
- slot materialization: the decision whether to materialize a slot now happens *prior* to
  plan generation. This is needed in order to be able to generate accurate cost estimates
  at plan generation time. see QueryStmt.materializeRequiredSlots()
- added PlanNode.init(), which initializes the entire state of a PlanNode; this subsumes
  finalize()
  * computeMemLayout() now happens per-tuple in the corresponding ScanNode's init()
  * init() calls computeStats() by default; also marks slots as materialized and calls
    TupleDescriptor.computeMemLayout()
- added PlanNode.tblRefIds_
- restructured UnionStmt and union plan generation to fit pred propagation model:
  all tuples are created (and equiv predicates registered) prior to plan generation
- added Expr.isAuxExpr

Change-Id: I475c1645bfca9e84ae6e5f529e7781d9532e5c9a
Reviewed-on: http://gerrit.ent.cloudera.com:8080/955
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Reviewed-by: Lenni Kuff <lskuff@cloudera.com>
Tested-by: jenkins