BufferedTupleStream: allow interleaved reads/writes and return rows FIFO

Adds support to read from the BufferedTupleStream before all rows have been
added. To do this, at least two pinned buffers may be needed so the c'tor
takes a parameter to indicate if this behavior is enabled.

The stream will also now return rows in the order in which they were inserted.
Instead of maintaining separate lists of pinned and unpinned blocks, we just
keep a single list to easily maintain the FIFO ordering.

This also attempts to clarify the buffer management configuration by separating
pinning behavior from PrepareForRead() and the delete-on-read behavior. Now
pinning is set with Pin()/Unpin() (not PrepareForRead()/Unpin()) and there is
a separate c'tor parameter to specify if blocks should be deleted after reading.

Change-Id: Ic7628ff13ad4329461de0742c80cd91d1f385089
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3785
Reviewed-by: Matthew Jacobs <mj@cloudera.com>
Tested-by: jenkins
Adding support for {anti,left semi,left outer} joins to the partitioned hash join implementation.

Adding the "anti join" keyword in the frontend and the corresponding backend paths for the
partitioned hash join implementation.  Adding some basic testing for this new join (the
other types have already tests).

Also, fixing a bug in the tuple stream when it was handling strings.

Change-Id: Ied8cff96b2bca284a5f66f7d11df5c5b5ec789cc
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3805
Reviewed-by: Ippokratis Pandis <ipandis@cloudera.com>
Tested-by: jenkins
CDH-18432: Fix assignment of On-clause predicates of semi-/anti-joins.

Change-Id: Iad4b772f170ea70ded0747ce55972b0a5194f88a
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3852
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
Fixed LIKE predicate %% pattern error

Before:
Query: select count(*) from functional.alltypes where date_string_col
like "%%"
+----------+
| count(*) |
+----------+
| 0        |
+----------+
After:
Query: select count(*) from functional.alltypes where date_string_col
like "%%"
+----------+
| count(*) |
+----------+
| 7300     |
+----------+

Change-Id: Ic2f047ba3b03eeebe9477892da95a1d06dc27da7
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3768
Reviewed-by: Paden Tomasello <paden.tomasello@cloudera.com>
Tested-by: jenkins
IMPALA-1131: Measure time in sink::Close() in fragments' total time.

Change-Id: I811c6da0573f0ff58eea67455a78d0aecfd15d04
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3742
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
IMPALA-1021: Fix loading of views with decimal and complex-typed columns.

Change-Id: I8b63c31be47dd64f1e13fb29be3105b0f7e245dc
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3820
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
 Performance improvements: Alter table add partition and drop partition

    This patch improves the performance of the DDL queries
    "Alter table add partition" and "Alter table drop partition"
    as the number of partitions is scaled up.

    The issue was that every time a partition was added or dropped,
    the entire block metadata for that table was reloaded. This
    operation was highly expensive especially as the number
    of partitions became larger.

    This patch handles this by adding/dropping only the added/dropped
    partition's metadata to the hdfsTable (adding/dropping it to/from
    the internal partition list), and incrementally updating the
    corresponding data structures instead of refreshing them from scratch.

    The following are the time improvements observed.

    Number of partitions    Time taken to add/drop     Time to add/drop
    (existing)              a new partition (before)   a new partition (now)

                       1          1.02s                    1.02s
                      10          0.27s                    0.27s
                     100          0.14s                    0.14s
                     500          0.35s                    0.35s
                    1000          0.91s                    0.51s
                   10000          11.72s                   0.85s
                   20000          21.92s                   0.87s

Out of this total time (for the worst case), around 0.50s is spent in
adding and dropping the partition to the hive meta store and rest of the
time is spent in updating the catalog.

Change-Id: I359ab0af921543c0fdcb975c14b05f80f93fe803
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3291
Reviewed-by: Anusha Dasarakothapalli <anusha.dasarakothapalli@cloudera.com>
Tested-by: jenkins
IMPALA-491: Fixed worker process failure test and clients-in-use metric accounting.

Fixing the worker process failure test revealed a bug in our clients-in-use metrics
accounting in the client cache in the presence of node failures.
The pathological sequence causing leading to an incorrect metric was:
1. ClientConnection c'tor checks out an existing client and updates metrics
2. Existing client connection has broken pipe (or some other TException)
3. Recover by calling ClientConnection::Reopen()
4. Reopen() internally tries to create a new client connection, but the connection
   fails because the remote side is dead.
5. ClientConnection::client_ is set to NULL and therefore its d'tor
   does not properly decrement the clients-in-use for the client
   checked out in step 1.

Change-Id: I911210b23c4a024a4a8a84365a3d12268767b031
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3804
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: jenkins
IMPALA-958: Long query plan serialization time when querying huge tables.

One easy thing to do, as mentioned by Ippo in the jira, is to avoid
sending the file_descs to the BE.  The BE doesn't use this info, but
instead reconstructs it from the scanranges.  When the blocks dominate
the TQueryExecRequest size, this reduces the size of serialized
TQueryExecRequest by around 30%.

I considered a couple of alternative approaches.  The first is to keep
sending the file_descs, but then make THdfsFileSplit reference them,
so that each scanrange doesn't carry the info that's common to a file.
The second would be to have the frontend just send the THdfsFileBlocks
rather than scanranges.  However, both of these get a little messy
because then more backend components need to know the difference
between Hdfs and HBase.

So, let's first see how much we can trim things down before making a
more drastic change.

Change-Id: I7738d17af5e78952d600edd52b655dd365ec05b7
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3815
Reviewed-by: Daniel Hecht <dhecht@cloudera.com>
Tested-by: jenkins
Render all webpages via the template engine

This patch changes all remaining non-templated webpages to use the
templating engine to produce their HTML and text output. The main
motivation for this change is to make it much harder for unescaped output
to appear in HTML pages, where it might be interpreted by the browser as
e.g. JavaScript. The template engine automatically escapes all text.

As a side-effect, moving everything to the templating engine allows us
to remove the 'HTML' path from the webserver. Every callback now
produces Json and is rendered via a template.

This patch also changes the mechanism via which pages are presented to
the browser as text, rather than HTML. Any URL that includes the
argument '?raw' will automatically be rendered as text, not HTML, but
still passes through the template process. Any callback that wishes to
ensure that is only ever rendered as text (e.g. /jsonmetrics) has only
to set the special '__raw__' key in its Json output.

Finally, this patch removes the 'is_styled' parameter to
Register*Callback(); this logic belongs in the template, not inside the
webserver itself.

Change-Id: Ia33df77c52ebfa8125c5cf1fbcf40ea4b1da5497
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3810
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Add tests for escaped / plain text webpages

Change-Id: Ia0adeff41a84508d170f12d51bbd944662763977
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3837
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
Reorganize/reduce end to end test time.

This patch does a few things:
1) Move the metadata tests into their own folder under tests/. I think it's useful to
loosely categorize them so it's easier to run a subset of the tests that are most
useful for the changes you are making.

2) Reduce the test vectors for query_tests. We should have identical coverage in
the daily exhaustive runs but the normal runs should be much better. In particular,
deemphasizing scanner tests since that code is more stable now.

3) Misc test cleanup/consolidate python test files/etc.

Change-Id: I03c2f34877aed192c2a50665bd5e15fa85e12f1e
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3831
Tested-by: jenkins
Reviewed-by: Nong Li <nong@cloudera.com>
Optionally dynamically link Impala executables

This patch adds two new flags to make_impala.sh:

-build_shared_libs: Impala libraries (excluding thirdparty ones) will be
 built as shared objects (.so), and linked dynamically.

-build_static_libs: Impala libraries will be built as archive
 files (.a), and linked statically. This was the behaviour before this
 patch, and is still the default for make_impala.sh.

The speedup from dynamic linking for a clean build is significant:

make_impala.sh -clean -build_static_libs: 11m48.676s
make_impala.sh -clean -build_shared_libs:  5m46.943s

make_debug.sh now passes -build_shared_libs by
default. make_[asan|release].sh still builds with statically linked libraries. All
automated builds will be statically linked for now; we can move them to
dynamic linking on a case-by-case basis.

Change-Id: Icfd8101bf8e85cadd61d8995ae8864f8730297ea
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3828
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: jenkins
IMPALA-958: Don't store full paths in scan ranges.

When there are a lot of scan ranges per partition, we can reduce the
TQueryExecRequest footprint by storing only file names in the scan
ranges.  The directory path can be inferred by the partition id.

How much this reduces the footprint is dependent on the number of scan
ranges per parition and the length of the absolute path, but in our
test cluster for a partition with many files (~1000), this change
reduces the scan range size by about 30%.

Change-Id: I48fc72433dfd8ad7105c18694eae909c232fe8d7
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3844
Reviewed-by: Marcel Kornacker <marcel@cloudera.com>
Tested-by: jenkins
Rewrite decimal functions/operators as UDFs.

This patch also adds a GetReturnType() method to FunctionContext. This
is staging for the expr refactoring.

Change-Id: I854e79ded409e151663c4ec99c4e08631ad9e03e
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3234
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
Rewrite operators as UDFs.

This patch removes gen_operators.py, replacing it with operators.h/cc
and cast-functions.h/cc. Case cannot be implemented as a UDF so is
still expressed in original interface. This will be updated when the
main expr refactoring patch goes in.

Change-Id: I69df3b4546608ef641b8b845e9ec36d14e8c189b
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3406
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
Convert math functions to the UDF interface.

Also adds FunctionContext::GetNumArgs() method to the public UDF API.

Change-Id: I76e21814e423f075a0a22b4e924c1d3ec26daba7
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3410
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
Convert string functions to UDF interface

This also switches to using the re2 library for regular expression
functions instead of boost.

Change-Id: I4c3ae72ff2f7cbd5b1a2be719275f1b2e25f8ab2
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3412
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
Expr refactoring

This patch changes the interface for evaluating expressions, in order
to allow for thread-safe expression evaluations and easier
codegen. Thread safety is achieved via the ExprContext class, a
light-weight container for expression tree evaluation state. Codegen
is easier because more expressions can be cross-compiled to IR.

See expr.h and expr-context.h for an overview of the API
changes. See sort-exec-exprs.cc for a simple example of the new
interface and hdfs-scanner.cc for a more complicated example.

This patch has not been completely code reviewed and may need further
cleanup/stylistic work, as well as additional perf work.

Change-Id: I3e3baf14ebffd2687533d0cc01a6fb8ac4def849
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3459
Reviewed-by: Skye Wanderman-Milne <skye@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>
Fix expr refactoring rebase

Change-Id: Ice4846c65945f3fc06a2c18db136482966c632dc
Reviewed-on: http://gerrit.sjc.cloudera.com:8080/3858
Reviewed-by: Henry Robinson <henry@cloudera.com>
Tested-by: Skye Wanderman-Milne <skye@cloudera.com>