Project: ClickHouse (App Definition and Development / Database)
File: hive.md

slug: /en/engines/table-engines/integrations/hive sidebar_position: 84 sidebar_label: Hive The Hive engine allows you to perform `SELECT` queries on HDFS Hive table. Currently it supports input formats as below: Text: only supports simple scalar column types except `binary` ORC: support simple scalar columns types except `char`; only support complex types like `array` Parquet: support all simple scalar columns types; only support complex types like `array` ``` sql CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] ( name1 [type1] [ALIAS expr1], name2 [type2] [ALIAS expr2], ... ) ENGINE = Hive('thrift://host:port', 'database', 'table'); PARTITION BY expr ``` See a detailed description of the query. The table structure can differ from the original Hive table structure: Column names should be the same as in the original Hive table, but you can use just some of these columns and in any order, also you can use some alias columns calculated from other columns. Column types should be the same from those in the original Hive table. Partition by expression should be consistent with the original Hive table, and columns in partition by expression should be in the table structure. Engine Parameters `thrift://host:port` Hive Metastore address `database` Remote database name. `table` Remote table name. We strongly advice you to enable local cache for remote filesystems. Benchmark shows that its almost 2x faster with cache. Before using cache, add it to `config.xml` ``` xml <localcacheforremotefs> <enable>true</enable> <rootdir>localcache</root_dir> <limitsize>559096952</limitsize> <bytesreadbeforeflush>1048576</bytesreadbeforeflush> </localcacheforremotefs> ``` enable: ClickHouse will maintain local cache for remote filesystem(HDFS) after startup if true. root_dir: Required. The root directory to store local cache files for remote filesystem. limit_size: Required. The maximum size(in bytes) of local cache files. bytesreadbefore_flush: Control bytes before flush to local filesystem when downloading file from remote filesystem. The default value is 1MB. When ClickHouse is started up with local cache for remote filesystem enabled, users can still choose not to use cache with `settings uselocalcacheforremotestorage = 0` in their query. `uselocalcacheforremotestorage` is `1` by default. ``` text hive > CREATE TABLE `test`.`test_orc`( `f_tinyint` tinyint, `f_smallint` smallint, `f_int` int, `f_integer` int, `f_bigint` bigint, `f_float` float, `f_double` double, `f_decimal` decimal(10,0), `f_timestamp` timestamp, `f_date` date, `f_string` string, `f_varchar` varchar(100), `f_bool` boolean, `f_binary` binary, `farrayint` array<int>, `farraystring` array<string>, `farrayfloat` array<float>, `farrayarray_int` array<array<int>>, `farrayarray_string` array<array<string>>, `farrayarray_float` array<array<float>>) PARTITIONED BY ( `day` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://testcluster/data/hive/test.db/test_orc' OK Time taken: 0.51 seconds hive > insert into test.testorc partition(day='2021-09-18') select 1, 2, 3, 4, 5, 6.11, 7.22,

currenttimestamp(), current_date(), 'hello world', 'hello world', 'hello world', true, 'hello world', array(1, 2, 3), array('hello world', 'hello world'), array(float(1.1), float(1.2)), array(array(1, 2), array(3, 4)), array(array('a', 'b'), array('c', 'd')), array(array(float(1.11), float(2.22)), array(float(3.33), float(4.44))); OK Time taken: 36.025 seconds hive > select * from test.test_orc; OK 1 2 3 4 5 6.11 7.22 8 2021-11-05 12:38:16.314 2021-11-05 hello world hello world hello world true hello world [1,2,3] ["hello world","hello world"] [1.1,1.2] [[1,2],[3,4]] [["a","b"],["c","d"]] [[1.11,2.22],[3.33,4.44]] 2021-09-18 Time taken: 0.295 seconds, Fetched: 1 row(s) ``` Table in ClickHouse, retrieving data from the Hive table created above: ``` sql CREATE TABLE test.test_orc ( `f_tinyint` Int8, `f_smallint` Int16, `f_int` Int32, `f_integer` Int32, `f_bigint` Int64, `f_float` Float32, `f_double` Float64, `f_decimal` Float64, `f_timestamp` DateTime, `f_date` Date, `f_string` String, `f_varchar` String, `f_bool` Bool, `f_binary` String, `farrayint` Array(Int32), `farraystring` Array(String), `farrayfloat` Array(Float32), `farrayarray_int` Array(Array(Int32)), `farrayarray_string` Array(Array(String)), `farrayarray_float` Array(Array(Float32)), `day` String ) ENGINE = Hive('thrift://202.168.117.26:9083', 'test', 'test_orc') PARTITION BY day ``` ``` sql SELECT * FROM test.testorc settings inputformatorcallowmissingcolumns = 1\G ``` ``` text SELECT * FROM test.test_orc SETTINGS inputformatorcallowmissing_columns = 1 Query id: c3eaffdc-78ab-43cd-96a4-4acc5b480658 Row 1: f_tinyint: 1 f_smallint: 2 f_int: 3 f_integer: 4 f_bigint: 5 f_float: 6.11 f_double: 7.22 f_decimal: 8 f_timestamp: 2021-12-04 04:00:44 f_date: 2021-12-03 f_string: hello world f_varchar: hello world f_bool: true f_binary: hello world farrayint: [1,2,3] farraystring: ['hello world','hello world'] farrayfloat: [1.1,1.2] farrayarray_int: [[1,2],[3,4]] farrayarray_string: [['a','b'],['c','d']] farrayarray_float: [[1.11,2.22],[3.33,4.44]] day: 2021-09-18 1 rows in set. Elapsed: 0.078 sec. ``` ``` text hive > CREATE TABLE `test`.`test_parquet`( `f_tinyint` tinyint, `f_smallint` smallint, `f_int` int, `f_integer` int, `f_bigint` bigint, `f_float` float, `f_double` double, `f_decimal` decimal(10,0), `f_timestamp` timestamp, `f_date` date, `f_string` string, `f_varchar` varchar(100), `f_char` char(100), `f_bool` boolean, `f_binary` binary, `farrayint` array<int>, `farraystring` array<string>, `farrayfloat` array<float>, `farrayarray_int` array<array<int>>, `farrayarray_string` array<array<string>>, `farrayarray_float` array<array<float>>) PARTITIONED BY ( `day` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'hdfs://testcluster/data/hive/test.db/test_parquet' OK Time taken: 0.51 seconds hive > insert into test.testparquet partition(day='2021-09-18') select 1, 2, 3, 4, 5, 6.11, 7.22, 8.333, currenttimestamp(), current_date(), 'hello world', 'hello world', 'hello world', true, 'hello world', array(1, 2, 3), array('hello world', 'hello world'), array(float(1.1), float(1.2)), array(array(1, 2), array(3, 4)), array(array('a', 'b'), array('c', 'd')), array(array(float(1.11), float(2.22)), array(float(3.33), float(4.44))); OK Time taken: 36.025 seconds hive > select * from test.test_parquet; OK 1 2 3 4 5 6.11 7.22 8 2021-12-14 17:54:56.743 2021-12-14 hello world hello world hello world true hello world [1,2,3] ["hello world","hello world"] [1.1,1.2] [[1,2],[3,4]] [["a","b"],["c","d"]] [[1.11,2.22],[3.33,4.44]] 2021-09-18 Time taken: 0.766 seconds, Fetched: 1 row(s) ``` Table in ClickHouse, retrieving data from the Hive table created above: ``` sql CREATE TABLE

( `f_tinyint` Int8, `f_smallint` Int16, `f_int` Int32, `f_integer` Int32, `f_bigint` Int64, `f_float` Float32, `f_double` Float64, `f_decimal` Float64, `f_timestamp` DateTime, `f_date` Date, `f_string` String, `f_varchar` String, `f_char` String, `f_bool` Bool, `f_binary` String, `farrayint` Array(Int32), `farraystring` Array(String), `farrayfloat` Array(Float32), `farrayarray_int` Array(Array(Int32)), `farrayarray_string` Array(Array(String)), `farrayarray_float` Array(Array(Float32)), `day` String ) ENGINE = Hive('thrift://localhost:9083', 'test', 'test_parquet') PARTITION BY day ``` ``` sql SELECT * FROM test.testparquet settings inputformatparquetallowmissingcolumns = 1\G ``` ``` text SELECT * FROM test_parquet SETTINGS inputformatparquetallowmissing_columns = 1 Query id: 4e35cf02-c7b2-430d-9b81-16f438e5fca9 Row 1: f_tinyint: 1 f_smallint: 2 f_int: 3 f_integer: 4 f_bigint: 5 f_float: 6.11 f_double: 7.22 f_decimal: 8 f_timestamp: 2021-12-14 17:54:56 f_date: 2021-12-14 f_string: hello world f_varchar: hello world f_char: hello world f_bool: true f_binary: hello world farrayint: [1,2,3] farraystring: ['hello world','hello world'] farrayfloat: [1.1,1.2] farrayarray_int: [[1,2],[3,4]] farrayarray_string: [['a','b'],['c','d']] farrayarray_float: [[1.11,2.22],[3.33,4.44]] day: 2021-09-18 1 rows in set. Elapsed: 0.357 sec. ``` ``` text hive > CREATE TABLE `test`.`test_text`( `f_tinyint` tinyint, `f_smallint` smallint, `f_int` int, `f_integer` int, `f_bigint` bigint, `f_float` float, `f_double` double, `f_decimal` decimal(10,0), `f_timestamp` timestamp, `f_date` date, `f_string` string, `f_varchar` varchar(100), `f_char` char(100), `f_bool` boolean, `f_binary` binary, `farrayint` array<int>, `farraystring` array<string>, `farrayfloat` array<float>, `farrayarray_int` array<array<int>>, `farrayarray_string` array<array<string>>, `farrayarray_float` array<array<float>>) PARTITIONED BY ( `day` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://testcluster/data/hive/test.db/test_text' Time taken: 0.1 seconds, Fetched: 34 row(s) hive > insert into test.testtext partition(day='2021-09-18') select 1, 2, 3, 4, 5, 6.11, 7.22, 8.333, currenttimestamp(), current_date(), 'hello world', 'hello world', 'hello world', true, 'hello world', array(1, 2, 3), array('hello world', 'hello world'), array(float(1.1), float(1.2)), array(array(1, 2), array(3, 4)), array(array('a', 'b'), array('c', 'd')), array(array(float(1.11), float(2.22)), array(float(3.33), float(4.44))); OK Time taken: 36.025 seconds hive > select * from test.test_text; OK 1 2 3 4 5 6.11 7.22 8 2021-12-14 18:11:17.239 2021-12-14 hello world hello world hello world true hello world [1,2,3] ["hello world","hello world"] [1.1,1.2] [[1,2],[3,4]] [["a","b"],["c","d"]] [[1.11,2.22],[3.33,4.44]] 2021-09-18 Time taken: 0.624 seconds, Fetched: 1 row(s) ``` Table in ClickHouse, retrieving data from the Hive table created above: ``` sql CREATE TABLE test.test_text ( `f_tinyint` Int8, `f_smallint` Int16, `f_int` Int32, `f_integer` Int32, `f_bigint` Int64, `f_float` Float32, `f_double` Float64, `f_decimal` Float64, `f_timestamp` DateTime, `f_date` Date, `f_string` String, `f_varchar` String, `f_char` String, `f_bool` Bool, `day` String ) ENGINE = Hive('thrift://localhost:9083', 'test', 'test_text') PARTITION BY day ``` ``` sql SELECT * FROM test.testtext settings inputformatskipunknownfields = 1, inputformatwithnamesuseheader = 1, datetimeinputformat = 'besteffort'\G ``` ``` text SELECT * FROM test.test_text SETTINGS inputformatskipunknownfields = 1, inputformatwithnamesuseheader = 1, datetimeinputformat = 'best_effort' Query id: 55b79d35-56de-45b9-8be6-57282fbf1f44 Row 1: f_tinyint: 1 f_smallint: 2 f_int: 3 f_integer: 4 f_bigint: 5 f_float: 6.11 f_double: 7.22 f_decimal: 8 f_timestamp: 2021-12-14 18:11:17 f_date: 2021-12-14 f_string: hello world f_varchar: hello world f_char: hello world f_bool: true day: 2021-09-18 ```