Installing Apache Hive 3.1.2 on Windows 10
By: Hadi Fadlallah
Tags: Hadoop 3, Hadoop, Apache Hive, Big Data, Hive

Installing Apache Hive 3.1.2 on Windows 10

A step-by-step guide to install Apache Hive 3.1.2 on Windows 10 operating system

While working on a project, we were asked to install Apache Hive on a Windows 10 operating system. Many guides were found online but unfortunately, they didn’t work. For this reason, I decided to write a step-by-step guide to help others.

The starting point of this guide was from a great video I found on Youtube which provides a working scenario for Hive 2.x without much detail.

1. Prerequisites

1.1. 7zip

In order to extract tar.gz archives, you should install the 7zip tool.

1.2. Installing Hadoop

To install Apache Hive, you must have a Hadoop Cluster installed and running: You can refer to our previously published step-by-step guide to install Hadoop 3.2.1 on Windows 10.

1.3. Apache Derby

In addition, Apache Hive requires a relational database to create its Metastore (where all metadata will be stored). In this guide, we will use the Apache Derby database 4.

Since we have Java 8 installed, we must install Apache Derby 10.14.2.0 version (check downloads page) which can be downloaded from the following link.

Once downloaded, we must extract twice (using 7zip: the first time we extract the .tar.gz file, the second time we extract the .tar file) the content of the db-derby-10.14.2.0-bin.tar.gz archive into the desired installation directory. Since in the previous guide we have installed Hadoop within “E:\hadoop-env\hadoop-3.2.1\” directory, we will extract Derby into “E:\hadoop-env\db-derby-10.14.2.0\” directory.

1.4. Cygwin

Since there are some Hive 3.1.2 tools that aren’t compatible with Windows (such as schematool). We will need the Cygwin tool to run some Linux commands.

2. Downloading Apache Hive binaries

In order to download Apache Hive binaries, you should go to the following website: https://downloads.apache.org/hive/hive-3.1.2/. Then, download the apache-hive-3.1.2.-bin.tar.gz file.

Figure 1 — apache-hive.3.1.2-bin.tar.gz file

When the file download is complete, we should extract twice (as mentioned above) the apache-hive.3.1.2-bin.tar.gz archive into “E:\hadoop-env\apache-hive-3.1.2” directory (Since we decided to use E:\hadoop-env\” as the installation directory for all technologies used in the previous guide.

3. Setting environment variables

After extracting Derby and Hive archives, we should go to Control Panel > System and Security > System. Then Click on “Advanced system settings”.

Figure 2 — Advanced system settings

In the advanced system settings dialog, click on “Environment variables” button.

Figure 3 — Opening environment variables editor

Now we should add the following user variables:

Figure 4 — Adding User variables

HIVE_HOME: “E:\hadoop-env\apache-hive-3.1.2\”

DERBY_HOME: “E:\hadoop-env\db-derby-10.14.2.0\”

HIVE_LIB: “%HIVE_HOME%\lib”

HIVE_BIN: “%HIVE_HOME%\bin”

HADOOP_USER_CLASSPATH_FIRST: “true”

Figure 5 — Adding HIVE_HOME user variable

Besides, we should add the following system variable:

HADOOP_USER_CLASSPATH_FIRST: “true”

Now, we should edit the Path user variable to add the following paths:

%HIVE_BIN%

%DERBY_HOME%\bin

Figure 6 — Editing path environment variable

4. Configuring Hive

4.1. Copy Derby libraries

Now, we should go to the Derby libraries directory (E:\hadoop-env\db-derby-10.14.2.0\lib) and copy all *.jar files.

Figure 7 — Copy Derby libraries

Then, we should paste them within the Hive libraries directory (E:\hadoop-env\apache-hive-3.1.2\lib).

Figure 8 — Paste Derby libraries within Hive libraries directory

4.2. Configuring hive-site.xml

Now, we should go to the Apache Hive configuration directory (E:\hadoop-env\apache-hive-3.1.2\conf) create a new file “hive-site.xml”. We should paste the following XML code within this file:

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration><property> <name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:derby://localhost:1527/metastore_db;create=true</value>

<description>JDBC connect string for a JDBC metastore</description>

</property><property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>org.apache.derby.jdbc.ClientDriver</value>

<description>Driver class name for a JDBC metastore</description>

</property>

<property>

<name>hive.server2.enable.doAs</name>

<description>Enable user impersonation for HiveServer2</description>

<value>true</value>

</property>

<property>

<name>hive.server2.authentication</name>

<value>NONE</value>

<description> Client authentication types. NONE: no authentication check LDAP: LDAP/AD based authentication KERBEROS: Kerberos/GSSAPI authentication CUSTOM: Custom authentication provider (Use with property hive.server2.custom.authentication.class) </description>

</property>

<property>

<name>datanucleus.autoCreateTables</name>

<value>True</value>

</property>

</configuration>

5. Starting Services

5.1. Hadoop Services

To start Apache Hive, open the command prompt utility as administrator. Then, start the Hadoop services using start-dfs and start-yarn commands (as illustrated in the Hadoop installation guide).

5.2. Derby Network Server

Then, we should start the Derby network server on the localhost using the following command:

E:\hadoop-env\db-derby-10.14.2.0\bin\StartNetworkServer -h 0.0.0.0

6. Starting Apache Hive

Now, let try to open a command prompt tool and go to the Hive binaries directory (E:\hadoop-env\apache-hive-3.1.2\bin) and execute the following command:

hive

We will receive the following error:

'hive' is not recognized as an internal or external command, operable program or batch file.

This error is thrown since the Hive 3.x version is not built for Windows (only in some Hive 2.x versions). To get things working, we should download the necessary *.cmd files from the following link: https://svn.apache.org/repos/asf/hive/trunk/bin/. Note that, you should keep the folder hierarchy (bin\ext\util).

You can download all *.cmd files from the following GitHub repository

Now if we try to execute the “hive” command, we will receive the following error:

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)

at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)

at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:518)

at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:536)

at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:430)

at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)

at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104)

at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)

at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.util.RunJar.run(RunJar.java:318)

at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

This error is thrown due to a Bug mentioned in the following Hive issue link: HIVE-22718.

As mentioned in the comments, this issue can be solved by replacing the guava-19.0.jar stored in “E:\hadoop-env\apache-hive-3.1.2\lib” with Hadoop’s guava-27.0-jre.jar found in “E:\hadoop-env\hadoop-3.2.1\share\hadoop\hdfs\lib”.

Note: This file is also uploaded to the GitHub repository mentioned above.

Now, if we run hive command again, then Apache Hive will start successfully.

Figure 9 — Starting Apache Hive

7. Initializing Hive

After ensuring that the Apache Hive started successfully. We may not be able to run any HiveQL command. This is because the Metastore is not initialized yet. Besides HiveServer2 service must be running.

To initialize Metastore, we need to use schematool utility which is not compatible with windows. To solve this problem, we will use Cygwin utility which allows executing Linux command from windows.

7.1. Creating symbolic links

First, we need to create the following directories:

E:\cygdrive

C:\cygdrive

Now, open the command prompt as administrator and execute the following commands:

mklink /J E:\cygdrive\e\ E:\

mklink /J C:\cygdrive\c\ C:\

These symbolic links are needed to work with Cygwin utility properly since Java may cause some problems.

7.2. Initializing Hive Metastore

Open Cygwin utility and execute the following commands to define the environment variables:

export HADOOP_HOME='/cygdrive/e/hadoop-env/hadoop-3.2.1'

export PATH=$PATH:$HADOOP_HOME/bin

export HIVE_HOME='/cygdrive/e/hadoop-env/apache-hive-3.1.2'

export PATH=$PATH:$HIVE_HOME/bin

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*.jar

We can add these lines to the “~/.bashrc” file then you don’t need to write them each time you open Cygwin.

Now, we should use the schematool utility to initialize the Metastore:

$HIVE_HOME/bin/schematool -dbType derby -initSchema

7.3. Starting HiveServer2 service

Now, open a command prompt and run the following command:

hive --service hiveserver2 start

We should leave this command prompt open, and open a new one where we should start Apache Hive using the following command:

hive

7.4. Starting WebHCat Service (Optional)

In the project we are working on, we need to execute HiveQL statement from SQL Server Integration Services which can access Hive from the WebHCat server.

To start the WebHCat server, we should open the Cygwin utility and execute the following command:

$HIVE_HOME/hcatalog/sbin/webhcat_server.sh start

8. References