HIVE-4854 : testCliDriver_load_hdfs_file_with_space_in_the_name fails on hadoop 2 (Gunther Hagleitner via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1503304 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4852 : -Dbuild.profile=core fails (Gunther Hagleitner via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1503309 13f79535-47bb-0310-9956-ffa450edef68
HIVE-3603 Enable client-side caching for scans on HBase (Navis Ryu via EGC)

Submitted by:	Navis Ryu
Reviewed by:	Edward Capriolo


git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1503544 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4845 : Correctness issue with MapJoins using the null safe operator (Brock Noland via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1503818 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4769 : Vectorized ORC reader does not handle absence of column present stream correctly. (Sarvesh Sakalanaga via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/branches/vectorization@1503883 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4820 : webhcat_config.sh should set default values for HIVE_HOME and HCAT_PREFIX that work with default build tree structure (Eugene Koifman via Jianyong Dai)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1503940 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4848 : std, stddev and stddev_pop aggregates on double/float fail to vectorize (Remus Rusanu via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/branches/vectorization@1504216 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4865 - HiveLockObjects: Unlocking retries/times out when query contains ":" (Gunther Hagleitner via Brock Noland)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1504225 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4859 : String column comparison classes should be renamed. (Jitendra Nath Pandey via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/branches/vectorization@1504228 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4685 : query using LIKE does not vectorize (Eric Hanson via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/branches/vectorization@1504263 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4787 : make vectorized LOWER(), UPPER(), LENGTH() work end-to-end; support expression input for vectorized LIKE (Eric Hanson via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/branches/vectorization@1504265 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4873 Sort candidate functions in case of UDFArgumentException (Xuefu Zhang via egc)

Submitted by:	Xuefu Zhang
Reviewed by:	Edward Capriolo


git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1504361 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2206 [jira] add a new optimizer for query correlation discovery and optimization
(Yin Huai via Ashutosh Chauhan)

Summary:
update test results

This issue proposes a new logical optimizer called Correlation Optimizer, which is used to merge correlated MapReduce jobs (MR jobs) into a single MR job. The idea is based on YSmart (http://ysmart.cse.ohio-state.edu/). The paper and slides of YSmart are linked at the bottom.

Since Hive translates queries in a sentence by sentence fashion, for every operation which may need to shuffle the data (e.g. join and aggregation operations), Hive will generate a MapReduce job for that operation. However, for those operations which may need to shuffle the data, they may involve correlations explained below and thus can be executed in a single MR job.

	Input Correlation: Multiple MR jobs have input correlation (IC) if their input relation sets are not disjoint;
	Transit Correlation: Multiple MR jobs have transit correlation (TC) if they have not only input correlation, but also the same partition key;
	Job Flow Correlation: An MR has job ﬂow correlation (JFC) with one of its child nodes if it has the same partition key as that child node.

The current implementation of correlation optimizer only detect correlations among MR jobs for reduce-side join operators and reduce-side aggregation operators (not map only aggregation). A query will be optimized if it satisfies following conditions.

	There exists a MR job for reduce-side join operator or reduce side aggregation operator which have JFC with all of its parents MR jobs (TCs will be also exploited if JFC exists);
	All input tables of those correlated MR job are original input tables (not intermediate tables generated by sub-queries); and
	No self join is involved in those correlated MR jobs.

Correlation optimizer is implemented as a logical optimizer. The main reasons are that it only needs to manipulate the query plan tree and it can leverage the existing component on generating MR jobs.

Current implementation can serve as a framework for correlation related optimizations. I think that it is better than adding individual optimizers.

There are several work that can be done in future to improve this optimizer. Here are three examples.

	Support queries only involve TC;
	Support queries in which input tables of correlated MR jobs involves intermediate tables; and
	Optimize queries involving self join.

References:
Paper and presentation of YSmart.
Paper: http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-11-7.pdf
Slides: http://sdrv.ms/UpwJJc

Test Plan: EMPTY

Reviewers: JIRA, ashutoshc

Reviewed By: ashutoshc

CC: brock

Differential Revision: https://reviews.facebook.net/D11097

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1504395 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4675 Create new parallel unit test environment (Brock Noland via egc)

Submitted by: Brock Noland	
Reviewed by: Edward Capriolo	


git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1504759 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4874 Identical methods PTFDeserializer.addOIPropertiestoSerDePropsMap(), PTFTranslator.addOIPropertiestoSerDePropsMap() (Edward Capriolo via Brock Noland

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1504959 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4818: SequenceId in operator is not thread safe (Edward Capriolo via Brock Noland)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1504976 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4730 : Join on more than 2^31 records on single reducer failed (wrong results) (Navis via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1504983 13f79535-47bb-0310-9956-ffa450edef68
Enable parallel execution of various E2E tests (deepeshk via gates)


git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1505028 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4724 Better detection of non-ORC files in the ORC reader (omalley)


git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1505187 13f79535-47bb-0310-9956-ffa450edef68
HIVE-4877 : In ExecReducer, remove tag from the row which will be passed to the first Operator at the Reduce-side (Yin Huai via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1505233 13f79535-47bb-0310-9956-ffa450edef68