Installing Apache Hive 3.1.2 on Windows 10 | by Hadi Fadlallah | Towards Data Science

While working on a project, we were asked to install Apache Hive on a Windows 10 operating system. Many guides were found online but unfortunately, they didn’t work. For this reason, I decided to write a step-by-step guide to help others.
The starting point of this guide was from a great video I found on Youtube which provides a working scenario for Hive 2.x without much detail.
This article is a part of a series that we are publishing on TowardsDataScience.com that aims to illustrate how to install Big Data technologies on Windows operating system.
Other published articles in this series:
Installing Hadoop 3.2.1 Single node cluster on Windows 10
Installing Apache Pig 0.17.0 on Windows 10
In order to extract tar.gz archives, you should install the 7zip tool.
To install Apache Hive, you must have a Hadoop Cluster installed and running: You can refer to our previously published step-by-step guide to install Hadoop 3.2.1 on Windows 10.
In addition, Apache Hive requires a relational database to create its Metastore (where all metadata will be stored). In this guide, we will use the Apache Derby database 4.
Since we have Java 8 installed, we must install Apache Derby 10.14.2.0 version (check downloads page) which can be downloaded from the following link.
Once downloaded, we must extract twice (using 7zip: the first time we extract the .tar.gz file, the second time we extract the .tar file) the content of the db-derby-10.14.2.0-bin.tar.gz archive into the desired installation directory. Since in the previous guide we have installed Hadoop within “E:\hadoop-env\hadoop-3.2.1\” directory, we will extract Derby into “E:\hadoop-env\db-derby-10.14.2.0\” directory.
Since there are some Hive 3.1.2 tools that aren’t compatible with Windows (such as schematool). We will need the Cygwin tool to run some Linux commands.
In order to download Apache Hive binaries, you should go to the following website: https://downloads.apache.org/hive/hive-3.1.2/. Then, download the apache-hive-3.1.2.-bin.tar.gz file.
When the file download is complete, we should extract twice (as mentioned above) the apache-hive.3.1.2-bin.tar.gz archive into “E:\hadoop-env\apache-hive-3.1.2” directory (Since we decided to use E:\hadoop-env\” as the installation directory for all technologies used in the previous guide.
After extracting Derby and Hive archives, we should go to Control Panel > System and Security > System. Then Click on “Advanced system settings”.
In the advanced system settings dialog, click on “Environment variables” button.
Now we should add the following user variables:
HIVE_HOME: “E:\hadoop-env\apache-hive-3.1.2\”
DERBY_HOME: “E:\hadoop-env\db-derby-10.14.2.0\”
HIVE_LIB: “%HIVE_HOME%\lib”
HIVE_BIN: “%HIVE_HOME%\bin”
HADOOP_USER_CLASSPATH_FIRST: “true”
Besides, we should add the following system variable:
HADOOP_USER_CLASSPATH_FIRST: “true”
Now, we should edit the Path user variable to add the following paths:
%HIVE_BIN%
%DERBY_HOME%\bin
Now, we should go to the Derby libraries directory (E:\hadoop-env\db-derby-10.14.2.0\lib) and copy all *.jar files.
Then, we should paste them within the Hive libraries directory (E:\hadoop-env\apache-hive-3.1.2\lib).
Now, we should go to the Apache Hive configuration directory (E:\hadoop-env\apache-hive-3.1.2\conf) create a new file “hive-site.xml”. We should paste the following XML code within this file:
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration><property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:derby://localhost:1527/metastore_db;create=true</value> <description>JDBC connect string for a JDBC metastore</description></property><property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.apache.derby.jdbc.ClientDriver</value> <description>Driver class name for a JDBC metastore</description></property><property> <name>hive.server2.enable.doAs</name> <description>Enable user impersonation for HiveServer2</description><value>true</value></property><property><name>hive.server2.authentication</name> <value>NONE</value><description> Client authentication types. NONE: no authentication check LDAP: LDAP/AD based authentication KERBEROS: Kerberos/GSSAPI authentication CUSTOM: Custom authentication provider (Use with property hive.server2.custom.authentication.class) </description></property><property><name>datanucleus.autoCreateTables</name><value>True</value></property></configuration>
To start Apache Hive, open the command prompt utility as administrator. Then, start the Hadoop services using start-dfs and start-yarn commands (as illustrated in the Hadoop installation guide).
Then, we should start the Derby network server on the localhost using the following command:
E:\hadoop-env\db-derby-10.14.2.0\bin\StartNetworkServer -h 0.0.0.0
Now, let try to open a command prompt tool and go to the Hive binaries directory (E:\hadoop-env\apache-hive-3.1.2\bin) and execute the following command:
hive
We will receive the following error:
'hive' is not recognized as an internal or external command, operable program or batch file.
This error is thrown since the Hive 3.x version is not built for Windows (only in some Hive 2.x versions). To get things working, we should download the necessary *.cmd files from the following link: https://svn.apache.org/repos/asf/hive/trunk/bin/. Note that, you should keep the folder hierarchy (bin\ext\util).
You can download all *.cmd files from the following GitHub repository
https://github.com/HadiFadl/Hive-cmd
Now if we try to execute the “hive” command, we will receive the following error:
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)Vat org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:518)at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:536)at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:430)at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104)at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.util.RunJar.run(RunJar.java:318)at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
This error is thrown due to a Bug mentioned in the following Hive issue link: HIVE-22718.
As mentioned in the comments, this issue can be solved by replacing the guava-19.0.jar stored in “E:\hadoop-env\apache-hive-3.1.2\lib” with Hadoop’s guava-27.0-jre.jar found in “E:\hadoop-env\hadoop-3.2.1\share\hadoop\hdfs\lib”.
Note: This file is also uploaded to the GitHub repository mentioned above.
Now, if we run hive command again, then Apache Hive will start successfully.
After ensuring that the Apache Hive started successfully. We may not be able to run any HiveQL command. This is because the Metastore is not initialized yet. Besides HiveServer2 service must be running.
To initialize Metastore, we need to use schematool utility which is not compatible with windows. To solve this problem, we will use Cygwin utility which allows executing Linux command from windows.
First, we need to create the following directories:
E:\cygdrive
C:\cygdrive
Now, open the command prompt as administrator and execute the following commands:
mklink /J  E:\cygdrive\e\ E:\mklink /J  C:\cygdrive\c\ C:\
These symbolic links are needed to work with Cygwin utility properly since Java may cause some problems.
Open Cygwin utility and execute the following commands to define the environment variables:
export HADOOP_HOME='/cygdrive/e/hadoop-env/hadoop-3.2.1'export PATH=$PATH:$HADOOP_HOME/binexport HIVE_HOME='/cygdrive/e/hadoop-env/apache-hive-3.1.2'export PATH=$PATH:$HIVE_HOME/binexport HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*.jar
We can add these lines to the “~/.bashrc” file then you don’t need to write them each time you open Cygwin.
Now, we should use the schematool utility to initialize the Metastore:
$HIVE_HOME/bin/schematool -dbType derby -initSchema
Now, open a command prompt and run the following command:
hive --service hiveserver2 start
We should leave this command prompt open, and open a new one where we should start Apache Hive using the following command:
hive
In the project we are working on, we need to execute HiveQL statement from SQL Server Integration Services which can access Hive from the WebHCat server.
To start the WebHCat server, we should open the Cygwin utility and execute the following command:
$HIVE_HOME/hcatalog/sbin/webhcat_server.sh start
Stackoverflow.com QA WebsiteApache Hive official WebsiteYoutube: EASY HIVE INSTALLATION ON WINDOWSKontext.tech: Apache Hive 3.0.0 Installation on Windows 10 Step by Step Guide
Stackoverflow.com QA Website
Apache Hive official Website
Youtube: EASY HIVE INSTALLATION ON WINDOWS
Kontext.tech: Apache Hive 3.0.0 Installation on Windows 10 Step by Step Guide