Installing Hadoop 3.1.0 multi-node cluster on Ubuntu 16.04 Step by Step | by Hadi Fadlallah | Towards Data Science

There are many links on the web about install Hadoop 3. Many of them are not working well or need improvements. This article is taken from the official documentation and other articles in addition of many answers from Stackoverflow.com
Note: All prerequisites must be applied on name node and data nodes
First, we need to install SSH and few software installation utilities for Java 8:
sudo apt install \openssh-server \software-properties-common \python-software-properties
Then we need to install Oracle’s Java 8 distribution and update the current OS.
sudo add-apt-repository ppa:webupd8team/javasudo apt updatesudo apt install oracle-java8-installer
To verify the java version you can use the following command:
java -version
We will use a dedicated Hadoop user account for running Hadoop applications. While that’s not required but it is recommended because it helps to separate the Hadoop installation from other software applications and user accounts running on the same machine (security, permissions, backups, etc).
sudo addgroup hadoopgroupsudo adduser --ingroup hadoopgroup hadoopusersudo adduser hadoopuser sudo
You can check the groups and users using the following commands:
compgen -gcompgen -u
Hadoop requires SSH access to manage its different nodes, i.e. remote machines plus your local machine.
First you need to login as Hadoopuser
sudo su -- hadoopuser
The following commands are used for generating a key value pair using SSH
ssh-keygen -t rsa -P “” -f ~/.ssh/id_rsa
Copy the public keys form id_rsa.pub to authorized_keys.
cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keyschmod 0600 ~/.ssh/authorized_keys
Make sure hadoopuser can ssh to its own account without password. ssh to localhost from hadoopuser account to make sure it is working fine.
ssh localhost
Note: If you get error: ssh: connect to host localhost port 22: Connection refused, then, please try to install ssh-server using below command.
sudo apt-get install ssh
In this article, we will install Hadoop on three machines:
The First machine will act as the name node (Master) and a data node (slave), the other machines are data nodes (slaves)
On each machine we have to edit the /etc/hosts files using the following command
sudo gedit /etc/hosts
Each file must contain these rows:
127.0.0.1 localhost10.0.1.1 hadoop-namenode10.0.1.2 hadoop-datanode-210.0.1.3 hadoop-datadnode-3
Note: if the /etc/hosts file contains the following line
127.0.1.1 <Hostname>
Then you have to delete this line.
We are going to install all the software under the /opt directory and store HDFS’ underlying data there as well. Below we will create the folders with a single command.
sudo mkdir -p /opt/{hadoop/{logs},hdfs/{datanode,namenode},yarn/{logs}
The layout of the folder will looks like
/opt/├── hadoop│ ├── logs├── hdfs│ ├── datanode│ └── namenode├── yarn│ ├── logs
You can download the hadoop-3.1.0.tar.gz using the following command:
wget -c -O hadoop.tar.gz http://www-eu.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz
To decompress the Hadoop package you can use the following command:
sudo tar -xvf hadoop.tar.gz \ --directory=/opt/hadoop \--strip 1
The binary release of Hadoop 3 is 293 MB compressed. Its decompressed size is 733 MB with 400 MB of small documentation files that may take a long time to decompress. You can skip these files by adding the following line to the command above:
--exclude=hadoop-3.1.0/share/doc
Note: These Steps must be done on the Name node and Data nodes as well.
There are environment settings that will be used by Hadoop, Hive and Spark and are shared by both root and the regular user accounts. To centralize these settings I’ve stored them in /etc/profile and created a symbolic link from /root/.bashrc to this file as well. That way all users will have centrally-managed settings.
sudo gedit /etc/profile
The /etc/profile must looks like:
if [ “$PS1” ]; thenif [ “$BASH” ] && [ “$BASH” != “/bin/sh” ]; then# The file bash.bashrc already sets the default PS1.# PS1=’\h:\w\$ ‘if [ -f /etc/bash.bashrc ]; then. /etc/bash.bashrcfielseif [ “`id -u`” -eq 0 ]; thenPS1=’# ‘elsePS1=’$ ‘fififiif [ -d /etc/profile.d ]; thenfor i in /etc/profile.d/*.sh; doif [ -r $i ]; then. $ifidoneunset ifiexport HADOOP_HOME=/opt/hadoopexport PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbinexport HADOOP_CONF_DIR=/opt/hadoop/etc/hadoopexport HDFS_NAMENODE_USER=rootexport HDFS_DATANODE_USER=rootexport HDFS_SECONDARYNAMENODE_USER=rootexport JAVA_HOME=/usr/lib/jvm/java-8-oracleexport HADOOP_MAPRED_HOME=/opt/hadoopexport HADOOP_COMMON_HOME=/opt/hadoopexport HADOOP_HDFS_HOME=/opt/hadoopexport YARN_HOME=/opt/hadoop
The following command will create a symbolic link between ~/.bashrc and /etc/profile and apply the changes made to /etc/profile
sudo ln -sf /etc/profile /root/.bashrcsource /etc/profile
Update /opt/hadoop /etc/hadoop/ hadoop-env.sh file and set JAVA_HOME variable and HADOOP_HOME, HADOOP_CONF_DIR AND HADOOP_LOG_DIR variables
export JAVA_HOME=/usr/lib/jvm/java-8-oracleexport HADOOP_HOME=/opt/hadoopexport HADOOP_CONF_DIR=/opt/hadoop /etc/hadoopexport HADOOP_LOG_DIR=/opt/hadoop/logs
Log out and re-login to your hadoopuser account and check Hadoop installation using below command.
hadoop -version
First, we have to update the hdfs-site.xml file located a /opt/Hadoop/etc/Hadoop/ to define the name node and data node on this machine and define the replication factor and other settings:
sudo gedit /opt/hadoop/etc/hadoop/hdfs-site.xml
The file must looks like:
<configuration><property><name>dfs.namenode.name.dir</name><value>file:///opt/hdfs/namenode</value><description>NameNode directory for namespace and transaction logs storage.</description></property><property><name>dfs.datanode.data.dir</name><value>file:///opt/hdfs/datanode</value><description>DataNode directory</description></property><property><name>dfs.replication</name><value>3</value></property><property><name>dfs.permissions</name><value>false</value></property><property><name>dfs.datanode.use.datanode.hostname</name><value>false</value></property><property><name>dfs.namenode.datanode.registration.ip-hostname-check</name><value>false</value></property></configuration>
Then we have to update core-site.xml file located at /opt/hadoop/etc/hadoop and let Hadoop distribution know, where the name node is located:
sudo gedit /opt/hadoop/etc/hadoop/core-site.xml
The file must looks like:
<configuration><property><name>fs.defaultFS</name><value>hdfs://hadoop-namenode:9820/</value><description>NameNode URI</description></property><property><name>io.file.buffer.size</name><value>131072</value><description>Buffer size</description></property></configuration>
Then we have to update yarn-site.xml file located at /opt/hadoop/etc/hadoop/
sudo gedit /opt/hadoop/etc/hadoop/yarn-site.xml
The file must looks like:
<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>Yarn Node Manager Aux Service</description></property><property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property><property><name>yarn.nodemanager.local-dirs</name><value>file:///opt/yarn/local</value></property><property><name>yarn.nodemanager.log-dirs</name><value>file:///opt/yarn/logs</value></property></configuration>
Then we have to update mapred-site.xml file located at /opt/hadoop/etc/hadoop/
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value><description>MapReduce framework name</description></property><property><name>mapreduce.jobhistory.address</name><value>hadoop-namenode:10020</value><description>Default port is 10020.</description></property><property><name>mapreduce.jobhistory.webapp.address</name><value> hadoop-namenode:19888</value><description>Default port is 19888.</description></property><property><name>mapreduce.jobhistory.intermediate-done-dir</name><value>/mr-history/tmp</value><description>Directory where history files are written by MapReduce jobs.</description></property><property><name>mapreduce.jobhistory.done-dir</name><value>/mr-history/done</value><description>Directory where history files are managed by the MR JobHistory Server.</description></property></configuration>
Now we have to format the name node
hdfs namenode –format
Finally, we have to add your data node(s) (Slaves) to workers file located in /opt/hadoop/etc/hadoop
10.0.1.110.0.1.210.0.1.3
After configuring data nodes you have to make sure that the name node has a password less access to them:
ssh-copy-id -i /home/hadoopuser/.ssh/id_rsa.pub hadoopuser@10.0.1.2ssh-copy-id -i /home/hadoopuser/.ssh/id_rsa.pub hadoopuser@10.0.1.3
Note: Instead of downloading Hadoop you can copy the Hadoop.tar.gz file from the name node to data nodes and extract it. You can use the following command:
scp hadoop.tar.gz hadoop-datanode-2:/home/hadoopuserscp hadoop.tar.gz hadoop-datanode-3:/home/hadoopuser
On each data node you must do the following steps:
We have to update hdfs-site.xml, core-site.xml, yarn-site.xml and mapred-site.xml located at /opt/hadoop/etc/hadoop directory as the following:
hdfs-site.xml
<configuration><property><name>dfs.datanode.data.dir</name><value>file:///opt/hdfs/datanode</value><description>DataNode directory</description></property><property><name>dfs.replication</name><value>3</value></property><property><name>dfs.permissions</name><value>false</value></property><property><name>dfs.datanode.use.datanode.hostname</name><value>false</value></property></configuration>
core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://hadoop-namenode:9820/</value><description>NameNode URI</description></property></configuration>
yarn-site.xml
<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>Yarn Node Manager Aux Service</description></property></configuration>
mapred-site.xml
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value><description>MapReduce framework name</description></property></configuration>
After finishing the steps above, from the name node we have to execute the following command to start the Name node, data nodes and secondary name node:
start-dfs.sh
It will give the following output:
Starting namenodes on [hadoop-namenode]Starting datanodesStarting secondary namenodes [hadoop-namenode]
Also to start the resource manager and node managers we have to execute the following command:
start-yarn.sh
It will give the following output:
Starting resourcemanagerStarting nodemanagers
After that, to ensure that Hadoop started successfully, we must run the jps command on name node and data nodes must give the following output:
On Name node (ignore process ids):
16488 NameNode16622 DataNode17215 NodeManager17087 ResourceManager17530 Jps16829 SecondaryNameNode
On Data nodes (ignore process ids):
2306 DataNode2479 NodeManager2581 Jps
If you get similar output then all Hadoop daemons started successfully.
Note: You may check logs at /opt/hadoop/logs and check if everything is alright, by running hdfs dfsadmin -report command (it must return Live datanodes (3)).
Namenode
Access to the following URL: https://hadoop-namenode:9870/
ResourceManager
Access to the following URL: https://hadoop-namenode:8088/
[1] F. Houbart, “How to install and set up hadoop cluster,” Linode, 16 October 2017. [Online]. Available: https://www.linode.com/docs/databases/hadoop/how-to-install-and-set-up-hadoop-cluster. [Accessed 20 July 2018].
[2] “Stack overflow Q&A,” Stack overflow, [Online]. Available: https://www.Stackoverflow.com.
[3] “Apache Hadoop Documentation,” Apache, [Online]. Available: https://www.hadoop.apache.org. [Accessed 01 07 2018].
[4] G. Bansal, “Install Hadoop 3.0.0 multi-node cluster on Ubuntu,” 31 August 2017. [Online]. Available: http://www.gaurav3ansal.blogspot.com/2017/08/installing-hadoop-300-alpha-4-multi.html. [Accessed 16 07 2018].
[5] M. Litwintschik, “Hadoop 3 Single-Node Install Guide,” 19 March 2018. [Online]. Available: http://www.tech.marksblogg.com/hadoop-3-single-node-install-guide.html. [Accessed 10 06 2018].