FE Cleanup: Use analyzeNoThrow() and investigate Expr.substitute() TODO.

I looked at those places that call Expr.substitute/List() and I think
that for all of them we do not want preserveRootType, so I simply
removed the TODO.

Change-Id: I7c98e2aee8f11b14781ec49840e82e69adc6b751
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4841
Tested-by: jenkins
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Alex Behm <alex.behm@cloudera.com>
Fix exhaustive test run: Modify test to produce identical results on HBase.

Change-Id: I7187f9aca63f61ea1686820b3cbec277240da191
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4866
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
IMPALA-1300: Allow subqueries in UNION operands

This enables the existing subquery rewrite rules to rewrite UNION
statements. UNION rewriting is easily done by simply calling the
rewriter for each operand in the UNION. At least one TPC-DS query
requires this functionality (IMPALA-1365).

The more difficult case of a UNION within a subquery is still not
supported.

Change-Id: I7f83eed0eb8ae81565e629f09f6918a4ba86ee13
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4859
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: jenkins
Reintroduce SEQUENCEFILE writer tests

The sequence writer test had an issue with zlib on certain cluster machines, making
this a flaky test. This has passed several times locally and in private builds. This
re-enables the test because the failures could not be produced in private builds.

Change-Id: I0aeea3a2d000e711e5a84427a7b40592e1eef75b
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5077
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
IMPALA-1412: Create view as select produces incorrect results

This commit fixes the issue where querying a view produces incorrect
results if the view definition statement references the same column
multiple times in the select list. In that case, a predicate referencing
the same slot is generated when equivalences among view slots are
enforced, thereby causing null values to be rejected.

Change-Id: I3d13656141fb41d232ddd38562cbde277f2a1264
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5031
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: jenkins
IMPALA-1134: Use copyBytes() to get value from o.a.h.io.Text

This affects java UDFs. Previously it was possible that the length of
the string returned from a java udf didn't match the actual data. Per the
Text.getBytes() documentation "... only data up to getLength() is
valid.". Impala just needs to use copyBytes() which is a convenience
function for this situation. The same should be done for BytesWritable.

Before:

Query: select length(echo('12345678901234567890'))
+-------------------------------------------+
| length(java.echo('12345678901234567890')) |
+-------------------------------------------+
| 22                                        |
+-------------------------------------------+

After:

Query: select length(echo('12345678901234567890'))
+-------------------------------------------------+
| length(functional.echo('12345678901234567890')) |
+-------------------------------------------------+
| 20                                              |
+-------------------------------------------------+

Change-Id: If9671278df8abf7529d3bc470c5f9d037ac3da1b
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4897
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: jenkins
IMPALA-1456: Hive UDFs with String args would crash impalad

The wrong buffer was being used.

Change-Id: I18bf9040eaeda871d1d0baee2e276749a3a38615
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5185
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: jenkins
Move fragment management out from ImpalaServer

ImpalaServer is still pretty big. One reason was that it was conflating
several separate responsibilities, including management of query
execution, management of fragment execution and responding directly to
Thrift RPCs.

This patch moves fragment execution into a separate FragmentMgr class,
and adds a (now-needed) proxy class to forward ImpalaInternalService
RPCs to either ImpalaServer or the FragmentMgr. Fragment management is
pretty self-contained (look at how small the resulting class had to be),
and so it was a good candidate to be broken out of ImpalaServer.

Change-Id: I24efb1a71a4225f9f85610565bf557a2e677e629
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/4854
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Improved zlib error message

Sometimes writing to compressed files can result in an unhelpful error message:
"zlib deflate failed:"
Which does not contain details because a message field is not given to us by zlib in all
cases. This patch addresses one of those cases mentioned in the zlib documentation, namely
when the output buffer is too small to hold the compressed input.

Change-Id: I4aa53083f79caf424c47ed6978789ad09ac82b63
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5159
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
Disable compressed test to unblock code coverage.

Change-Id: Ida584920dde75a27882e217933eba97344219192
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5241
Reviewed-by: Victor Bittorf <victor.bittorf@cloudera.com>
Tested-by: jenkins
IMPALA-1429: Avoid ambiguous col refs after subquery rewrite

Prior to this commit, unqualified column refs may become ambiguous
during the rewriting of subqueries. As part of rewrite, the subquery is
transformed into an inline view and since inline views provide
candidates during column resolution, there may be multiple candidates
after the rewrite.

This commit adds a layer to separate column labels exposed by an inline
view from column labels used to construct the query within the inline
view.

Example:
  SELECT 1 FROM alltypes WHERE int_col IN (SELECT 1 AS int_col)

  would have been rewritten as

  SELECT 1
  FROM alltypes
  SEMI JOIN (SELECT 1 AS int_col) `$a` ON `$a`.int_col = int_col

  making the final int_col ambiguous. The rewrite is now

  ...
  SEMI JOIN (SELECT 1 AS int_col) `$a` (`$c1`) ON `$a`.`$c1` = int_col

The ability to override column labels is not available to users.

Change-Id: I9be0a331b7dbf24e4d66dc44ec109042d7250ed9
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5230
Reviewed-by: Casey Ching <casey@cloudera.com>
Tested-by: jenkins
Fix zlib maximum compressed size estimate

Some compressed file formats in Impala, such as SEQUENCEFILE and TEXT, have run into
errors with zlib compression. This has been tracked to situations where the output
buffer is too small because of an error in zlib.

There is a known issue that zlib 1.2.3 does not include the size of the
gzip wrapper when estimating the size of compressed output.
This is has been fixed in zlib 1.2.3.1: http://www.zlib.net/ChangeLog.txt

"Take into account wrapper variations in deflateBound()"

Mark, maintainer of zlib, has stated that 12 needs to be added to result for gzip,
http://compgroups.net/comp.unix.programmer/gzip-compressing-an-in-memory-string-usi/54854

To have a safe upper bound for "wrapper variations", we add 32 to estimate.

Change-Id: I4f17506e880d2d1cdeca8e770434efb202be69ff
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5249
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: jenkins
Fix lock contention in fragment mgr.

The issue is that in the current code, the last reference to the
FragmentExecState is dropped when we remove the state from the
fragment exec state map. This is done when the global lock is taken.
Cleaning up the fragment exec state is not cheap and this causes bad
lock contention with high concurrency.

Change-Id: I7461e0ac5a034ba8da98183435f6cefa85c9d94e
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5251
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
IMPALA-1392: Fix crash from UDFs that throw exceptions.

Change-Id: Ic8775d6344aba9655511f99c0a1760e8e148d0cf
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5243
Reviewed-by: Nong Li <nong@cloudera.com>
Tested-by: jenkins
Fix leaking FragmentExecStates.

The previous attempt to fix this resulted in the FragmentExecState never being
deleted.

Change-Id: I81afe9c60e557119d0451335219379fedb2021c8
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/5273
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins