Installing Hadoop 3.1.0 multi-node cluster on Ubuntu 16.04 Step by Step
By: Hadi Fadlallah
Tags: Big Data, Hadoop, Ubuntu, Hadoop 3, Hadoop Cluster

There are many links on the web about install Hadoop 3. Many of them are not working well or need improvements. This article is taken from the official documentation and other articles in addition of many answers from Stackoverflow.com

1. Prerequisites

Note: All prerequisites must be applied on name node and data nodes

1.1. Installation of JAVA, SSH and other software utilities

First, we need to install SSH and few software installation utilities for Java 8:

sudo apt install \ openssh-server \ software-properties-common \ python-software-properties

Then we need to install Oracle’s Java 8 distribution and update the current OS.

sudo add-apt-repository ppa:webupd8team/java sudo apt update sudo apt install oracle-java8-installer

To verify the java version you can use the following command:

java -version

1.2. Creation of a dedicated user and group for Hadoop

We will use a dedicated Hadoop user account for running Hadoop applications. While that’s not required but it is recommended because it helps to separate the Hadoop installation from other software applications and user accounts running on the same machine (security, permissions, backups, etc).

sudo addgroup hadoopgroup sudo adduser --ingroup hadoopgroup hadoopuser sudo adduser hadoopuser sudo

You can check the groups and users using the following commands:

compgen -g compgen -u

1.3. SSH Configuration

Hadoop requires SSH access to manage its different nodes, i.e. remote machines plus your local machine.

First you need to login as Hadoopuser

sudo su -- hadoopuser

The following commands are used for generating a key value pair using SSH

ssh-keygen -t rsa -P “” -f ~/.ssh/id_rsa

Copy the public keys form id_rsa.pub to authorized_keys.

cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keys chmod 0600 ~/.ssh/authorized_keys

Make sure hadoopuser can ssh to its own account without password. ssh to localhost from hadoopuser account to make sure it is working fine.

ssh localhost

Note: If you get error: ssh: connect to host localhost port 22: Connection refused, then, please try to install ssh-server using below command.

sudo apt-get install ssh

2. Download and Configure Hadoop

In this article, we will install Hadoop on three machines:

The First machine will act as the name node (Master) and a data node (slave), the other machines are data nodes (slaves)

On each machine we have to edit the /etc/hosts files using the following command

sudo gedit /etc/hosts

Each file must contain these rows:

127.0.0.1 localhost 10.0.1.1 hadoop-namenode 10.0.1.2 hadoop-datanode-2 10.0.1.3 hadoop-datadnode-3

Note: if the /etc/hosts file contains the following line

127.0.1.1 <Hostname>

Then you have to delete this line.

2.1. Download Hadoop

We are going to install all the software under the /opt directory and store HDFS’ underlying data there as well. Below we will create the folders with a single command.

sudo mkdir -p /opt/{hadoop/{logs},hdfs/{datanode,namenode},yarn/{logs}

The layout of the folder will looks like

/opt/ ├── hadoop │ ├── logs ├── hdfs │ ├── datanode │ └── namenode ├── yarn │ ├── logs

You can download the hadoop-3.1.0.tar.gz using the following command:

To decompress the Hadoop package you can use the following command:

sudo tar -xvf hadoop.tar.gz \ --directory=/opt/hadoop \ --strip 1

The binary release of Hadoop 3 is 293 MB compressed. Its decompressed size is 733 MB with 400 MB of small documentation files that may take a long time to decompress. You can skip these files by adding the following line to the command above:

--exclude=hadoop-3.1.0/share/doc

2.2. Hadoop Common Configuration

Note: These Steps must be done on the Name node and Data nodes as well.

There are environment settings that will be used by Hadoop, Hive and Spark and are shared by both root and the regular user accounts. To centralize these settings I’ve stored them in /etc/profile and created a symbolic link from /root/.bashrc to this file as well. That way all users will have centrally-managed settings.

sudo gedit /etc/profile

The /etc/profile must looks like:

if [ “$PS1” ]; then if [ “$BASH” ] && [ “$BASH” != “/bin/sh” ]; then # The file bash.bashrc already sets the default PS1. # PS1=’\h:\w\$ ‘ if [ -f /etc/bash.bashrc ]; then . /etc/bash.bashrc fi else if [ “`id -u`” -eq 0 ]; then PS1=’# ‘ else PS1=’$ ‘ fi fi fi if [ -d /etc/profile.d ]; then for i in /etc/profile.d/*.sh; do if [ -r $i ]; then . $i fi done unset i fi export HADOOP_HOME=/opt/hadoop export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop export HDFS_NAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export JAVA_HOME=/usr/lib/jvm/java-8-oracle export HADOOP_MAPRED_HOME=/opt/hadoop export HADOOP_COMMON_HOME=/opt/hadoop export HADOOP_HDFS_HOME=/opt/hadoop export YARN_HOME=/opt/hadoop

The following command will create a symbolic link between ~/.bashrc and /etc/profile and apply the changes made to /etc/profile

sudo ln -sf /etc/profile /root/.bashrc source /etc/profile

Update /opt/hadoop /etc/hadoop/ hadoop-env.sh file and set JAVA_HOME variable and HADOOP_HOME, HADOOP_CONF_DIR AND HADOOP_LOG_DIR variables

export JAVA_HOME=/usr/lib/jvm/java-8-oracle export HADOOP_HOME=/opt/hadoop export HADOOP_CONF_DIR=/opt/hadoop /etc/hadoop export HADOOP_LOG_DIR=/opt/hadoop/logs

Log out and re-login to your hadoopuser account and check Hadoop installation using below command.

hadoop -version

2.3. Master Node Configuration

First, we have to update the hdfs-site.xml file located a /opt/Hadoop/etc/Hadoop/ to define the name node and data node on this machine and define the replication factor and other settings:

sudo gedit /opt/hadoop/etc/hadoop/hdfs-site.xml

The file must looks like:

<configuration> <property> <name>dfs.namenode.name.dir</name> <value>file:///opt/hdfs/namenode</value> <description>NameNode directory for namespace and transaction logs storage.</description> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///opt/hdfs/datanode</value> <description>DataNode directory</description> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.datanode.use.datanode.hostname</name> <value>false</value> </property> <property> <name>dfs.namenode.datanode.registration.ip-hostname-check</name> <value>false</value> </property> </configuration>

Then we have to update core-site.xml file located at /opt/hadoop/etc/hadoop and let Hadoop distribution know, where the name node is located:

sudo gedit /opt/hadoop/etc/hadoop/core-site.xml

The file must looks like:

<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop-namenode:9820/</value> <description>NameNode URI</description> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> <description>Buffer size</description> </property> </configuration>

Then we have to update yarn-site.xml file located at /opt/hadoop/etc/hadoop/

sudo gedit /opt/hadoop/etc/hadoop/yarn-site.xml

The file must looks like:

<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>Yarn Node Manager Aux Service</description> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>file:///opt/yarn/local</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>file:///opt/yarn/logs</value> </property> </configuration>

Then we have to update mapred-site.xml file located at /opt/hadoop/etc/hadoop/

<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> <description>MapReduce framework name</description> </property> <property> <name>mapreduce.jobhistory.address</name> <value>hadoop-namenode:10020</value> <description>Default port is 10020.</description> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value> hadoop-namenode:19888</value> <description>Default port is 19888.</description> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/mr-history/tmp</value> <description>Directory where history files are written by MapReduce jobs.</description> </property> <property> <name>mapreduce.jobhistory.done-dir</name> <value>/mr-history/done</value> <description>Directory where history files are managed by the MR JobHistory Server.</description> </property> </configuration>

Now we have to format the name node

hdfs namenode –format

Finally, we have to add your data node(s) (Slaves) to workers file located in /opt/hadoop/etc/hadoop

10.0.1.1 10.0.1.2 10.0.1.3

After configuring data nodes you have to make sure that the name node has a password less access to them:

ssh-copy-id -i /home/hadoopuser/.ssh/id_rsa.pub hadoopuser@10.0.1.2 ssh-copy-id -i /home/hadoopuser/.ssh/id_rsa.pub hadoopuser@10.0.1.3

2.4. Configuring Data nodes

Note: Instead of downloading Hadoop you can copy the Hadoop.tar.gz file from the name node to data nodes and extract it. You can use the following command:

scp hadoop.tar.gz hadoop-datanode-2:/home/hadoopuser scp hadoop.tar.gz hadoop-datanode-3:/home/hadoopuser

On each data node you must do the following steps:

We have to update hdfs-site.xml, core-site.xml, yarn-site.xml and mapred-site.xml located at /opt/hadoop/etc/hadoop directory as the following:

hdfs-site.xml

<configuration> <property> <name>dfs.datanode.data.dir</name> <value>file:///opt/hdfs/datanode</value> <description>DataNode directory</description> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.datanode.use.datanode.hostname</name> <value>false</value> </property> </configuration>

core-site.xml

<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop-namenode:9820/</value> <description>NameNode URI</description> </property> </configuration>

yarn-site.xml

<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>Yarn Node Manager Aux Service</description> </property> </configuration>

mapred-site.xml

<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> <description>MapReduce framework name</description> </property> </configuration>

2.5. Start Hadoop

After finishing the steps above, from the name node we have to execute the following command to start the Name node, data nodes and secondary name node:

start-dfs.sh

It will give the following output:

Starting namenodes on [hadoop-namenode] Starting datanodes Starting secondary namenodes [hadoop-namenode]

Also to start the resource manager and node managers we have to execute the following command:

start-yarn.sh

It will give the following output:

Starting resourcemanager Starting nodemanagers

After that, to ensure that Hadoop started successfully, we must run the jps command on name node and data nodes must give the following output:

On Name node (ignore process ids):

16488 NameNode 16622 DataNode 17215 NodeManager 17087 ResourceManager 17530 Jps 16829 SecondaryNameNode

On Data nodes (ignore process ids):

2306 DataNode 2479 NodeManager 2581 Jps

If you get similar output then all Hadoop daemons started successfully.

Note: You may check logs at /opt/hadoop/logs and check if everything is alright, by running hdfs dfsadmin -report command (it must return Live datanodes (3)).

2.6. Accessing Hadoop on Browser

Namenode

Access to the following URL: https://hadoop-namenode:9870/

ResourceManager

Access to the following URL: https://hadoop-namenode:8088/

3. References