HIVE-2711 : Make the header of RCFile unique (Owen Omalley via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1325442 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2767 [jira] Optionally use framed transport with metastore
(Travis Crawford via Ashutosh Chauhan)

Summary:
Add support for optionally using the thrift framed transport, enabling
integration with environments where that is necessary.

Users may want/need to use thrift's framed transport when communicating with the
Hive MetaStore. This patch adds a new property
hive.metastore.thrift.framed.transport.enabled that enables the framed transport
(defaults to off, aka no change from before the patch). This property must be
set for both clients and the HMS server.

It wasn't immediately clear how to use the framed transport with SASL, so as
written an exception is thrown if you try starting the server with both options.
If SASL and the framed transport will indeed work together I can update the
patch (although I don't have a secured environment to test in).

Test Plan:
Tested locally that client and server can connect, both with and
without the flag. Tests pass.

Reviewers: JIRA, ashutoshc

Reviewed By: ashutoshc

Differential Revision: https://reviews.facebook.net/D2661

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1325446 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2910 : Improve the HWI interface (Hugo Trippaers via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1325492 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2948 [jira] HiveFileFormatUtils should use Path.SEPARATOR instead of
File.Separator

Summary:
https://issues.apache.org/jira/browse/HIVE-2948

Simple patch which replaces File.separator to Path.SEPARATOR since separator for
hdfs path is Path.SEPARATOR not java's File.SEPARATOR

Because its munging hdfs paths and not OS paths.

Test Plan: EMPTY

Reviewers: JIRA, kevinwilfong

Reviewed By: kevinwilfong

Differential Revision: https://reviews.facebook.net/D2763

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1325615 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2936 : Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory (Rohini via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1325791 13f79535-47bb-0310-9956-ffa450edef68


git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1325793 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2937. TestHiveServerSessions hangs when executed directly. (navis via kevinwilfong)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1325830 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2942. substr on string containing UTF-8 characters produces StringIndexOutOfBoundsException (Kevin Wilfong via pauly)


git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1326444 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2530. Implement SHOW TBLPROPERTIES. (leizhao via kevinwilfong)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1327189 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2930 [jira] Add license to the Hive files
(Owen O'Malley via Ashutosh Chauhan)

Summary:
<<Enter Revision Title>>

Fixing rat warnings

We need to clean up the RAT report to 0. Apache projects aren't supposed to
release until they have shown that all of their files have proper headers.

Note that although most of the files are just missing headers, some of them
explicitly say copyright by facebook and released under the Thrift (not Apache)
license. I'll generate a list of them, but I'd really appreciate it if someone
from facebook could in verify that they intend to license them to Apache.

Test Plan: EMPTY

Reviewers: JIRA, ashutoshc

Reviewed By: ashutoshc

Differential Revision: https://reviews.facebook.net/D2835

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1327205 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2946 [jira] Hive metastore does not have any log messages while shutting
itself down.
(Vandana Kasavajjala via Ashutosh Chauhan)

Summary:
https://issues.apache.org/jira/browse/HIVE-2946

Addressed Ashutosh's comments.

When hive metastore is started, the event is logged with good amount of
information. But when it is stopped ( using a kill) command, no information is
being written to the logs. It will be good if we can add a shutdown event to the
HiveMetastore class.

Test Plan: EMPTY

Reviewers: JIRA, ashutoshc

Reviewed By: ashutoshc

Differential Revision: https://reviews.facebook.net/D2787

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1327323 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2959 [jira] TestRemoteHiveMetaStoreIpAddress always uses the same port
(Kevin Wilfong via Ashutosh Chauhan)

Summary:
https://issues.apache.org/jira/browse/HIVE-2959

TestRemoteHiveMetaStoreIpAddress now uses the standard way of finding a free
port using Java's ServerSocket class.

TestRemoteHiveMetaStoreIpAddress always uses the same port, meaning that if
another process happens to be using that port, the tests cannot succeed.

There seems to be a standard way of finding a free port using Java's
ServerSocket class, this should be used instead.

Test Plan: Ran TestRemoteHiveMetaStoreIpAddress and
TestRemoteUGIHiveMetaStoreIpAddress, the two tests which would be affected by
this change.  I verified they passed and did not use port 39083.

Reviewers: JIRA, njain, ashutoshc

Reviewed By: ashutoshc

Differential Revision: https://reviews.facebook.net/D2841

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1327591 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2958 [jira] GROUP BY causing ClassCastException [LazyDioInteger cannot be
cast LazyInteger]
(Navis Ryu via Ashutosh Chauhan)

Summary:
DPAL-1111 GROUP BY causing ClassCastException [LazyDioInteger cannot be cast
LazyInteger]

This relates to https://issues.apache.org/jira/browse/HIVE-1634.

The following work fine:

CREATE EXTERNAL TABLE tim_hbase_occurrence (
  id int,
  scientific_name string,
  data_resource_id int
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH
SERDEPROPERTIES (
  "hbase.columns.mapping" = ":key#b,v:scientific_name#s,v:data_resource_id#b"
) TBLPROPERTIES(
  "hbase.table.name" = "mini_occurrences",
  "hbase.table.default.storage.type" = "binary"
);
SELECT * FROM tim_hbase_occurrence LIMIT 3;
SELECT * FROM tim_hbase_occurrence WHERE data_resource_id=1081 LIMIT 3;

However, the following fails:

SELECT data_resource_id, count(*) FROM tim_hbase_occurrence GROUP BY
data_resource_id;

The error given:

0 TS
2012-04-17 16:58:45,693 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Initialization Done 7 MAP
2012-04-17 16:58:45,714 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Processing alias tim_hbase_occurrence for file
hdfs://c1n2.gbif.org/user/hive/warehouse/tim_hbase_occurrence
2012-04-17 16:58:45,714 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
forwarding 1 rows
2012-04-17 16:58:45,714 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0
forwarding 1 rows
2012-04-17 16:58:45,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1
forwarding 1 rows
2012-04-17 16:58:45,723 FATAL ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"id":1444,"scientific_name":null,"data_resource_id":1081}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548)
	at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
	at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.lazydio.LazyDioInteger cannot be cast to
org.apache.hadoop.hive.serde2.lazy.LazyInteger
	at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:737)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
	... 9 more
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.lazydio.LazyDioInteger cannot be cast to
org.apache.hadoop.hive.serde2.lazy.LazyInteger
	at
org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyIntObjectInspector.copyObject(LazyIntObjectInspector.java:43)
	at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:239)
	at
org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.deepCopyElements(KeyWrapperFactory.java:150)
	at
org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.deepCopyElements(KeyWrapperFactory.java:142)
	at
org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.copyKey(KeyWrapperFactory.java:119)
	at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:750)
	at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:722)
	... 18 more

Test Plan: EMPTY

Reviewers: JIRA, ashutoshc

Reviewed By: ashutoshc

Differential Revision: https://reviews.facebook.net/D2871

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1328157 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2965 : Revert HIVE-2612 (hashutosh)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1328469 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2966 :Revert HIVE-2795  (Thejas Nair via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1328568 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2970: several jars in hive tar generated are not required (Thejas Nair via Ashutosh Chauhan)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1329314 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2646. Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs (Andrew Bayer and Thomas Weise)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1329381 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2928. Support for Oracle-backed Hive-Metastore (longvarchar to clob in package.jdo) (Mithun Radhakrishnan and Andrew Bayer via cws)

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1329416 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2883 [jira] Metastore client doesnt close connection properly

Summary:
https://issues.apache.org/jira/browse/HIVE-2883

Here is the patch which gets rid of this problem as per my hypothesis. I also
tested it and the above noted stack trace doesnt occur anymore.

While closing connection, it always fail with following trace. Seemingly, it
doesnt have any harmful effects.

12/03/20 10:55:02 ERROR hive.metastore: Unable to shutdown local metastore
client
org.apache.thrift.transport.TTransportException: Cannot write to null
outputStream
	at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
	at
org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
	at
org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
	at
com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:421)
	at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415)
	at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310)

Test Plan: EMPTY

Reviewers: JIRA, cwsteinbach

Reviewed By: cwsteinbach

Differential Revision: https://reviews.facebook.net/D2613

git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1329461 13f79535-47bb-0310-9956-ffa450edef68
HIVE-2703 ResultSetMetaData.getColumnType() always returns VARCHAR(string) for partition columns
irrespective of partition column type (tamtam180 via namit)



git-svn-id: https://svn.apache.org/repos/asf/hive/trunk@1329492 13f79535-47bb-0310-9956-ffa450edef68