Installing Hadoop on a Mac
By: Siphu Langeni
Tags: Data Science, Hadoop, Data Engineering, Big Data, Mac Os X

Is the only thing standing between you and Hadoop just trying to figure out how to install it on a Mac? A quick internet search will show you the lack of information about this fairly simple process. In this brief tutorial, I will show you how you can very easily install Hadoop 3.2.1 on a macOS Mojave (version 10.14.6) using Terminal for a single node cluster in pseudo-distributed mode.

To begin, you will need to have installed several packages that need to be placed in the appropriate directories. The HomeBrew website has made this a very simple task, automatically determining what is needed on your machine, installing correct directories and symlinking their files into /user/local. Additional documentation may also be found on their website.

Install HomeBrew

Copy the command at the top of the page and paste into a new terminal window. You will be notified of what will be installed. Pressing RETURN initiates the process:

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Confirm you have the correct version of java (version 8) on your machine. If it returns anything other than 1.8., be sure to install the correct version.

$ java -version $ brew cask install homebrew/cask-versions/adoptopenjdk8

Install Hadoop

Next, you will install the most current version of Hadoop at the path: /usr/local/Cellar/hadoop. This happens to be 3.2.1 at the time of the writing of this article:

$ brew install hadoop

Configure Hadoop

Configuring Hadoop will take place over a few steps. A more detailed version can be found in the Apache Hadoop documentation for setting up a single node cluster. (Be sure to follow along with the correct version installed on your machine.)

Updating the environment variable settings Make changes to core-, hdfs-, mapred- and yarn-site.xml files Remove password requirement (if necessary) Format NameNode

Open the document containing the environment variable settings :

$ cd /usr/local/cellar/hadoop/3.2.1/libexec/etc/hadoop

$ open hadoop-env.sh

Make the following changes to the document, save and close.

Add the location for export JAVA_HOME

export JAVA_HOME= “/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home”

You can find this path by using the following code in the terminal window:

$ /usr/libexec/java_home

Replace information for export HADOOP_OPTS

change export HADOOP_OPTS=”-Djava.net.preferIPv4Stack=true”

to export HADOOP_OPTS = ”-Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=”

Make changes to core files

$ open core-site.xml <configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

Make changes to hdfs files

$ open hdfs-site.xml <configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

Make changes to mapred files

$ open mapred-site.xml <configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>

</property>

</configuration>

Make changes to yarn files

$ open yarn-site.xml

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.env-whitelist</name>

<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>

</property>

</configuration>

Remove password requirement

Check if you’re able to ssh without a password before moving to the next step to prevent unexpected results when formatting the NameNode.

$ ssh localhost

If this does not return a last login time, use the following commands to remove the need to insert a password.

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

$ chmod 0600 ~/.ssh/authorized_keys

Format NameNode

$ cd /usr/local/cellar/hadoop/3.2.1/libexec/bin

$ hdfs namenode -format

A warning will tell you that a directory for logs is being created. You will be prompted to re-format filesystem in Storage Directory root. Say Y and press RETURN.

Run Hadoop

$ cd /usr/local/cellar/hadoop/3.2.1/libexec/sbin

$ ./start-all.sh

$ jps

After running jps, you should have confirmation that all the parts of Hadoop have been installed and running. You should see something like this:

66896 ResourceManager

66692 SecondaryNameNode

66535 DataNode

67350 Jps

66422 NameNode67005 NodeManager

Open a web browser to see your configurations for the current session.

http://localhost:9870

Information about your current Hadoop session.

Close Hadoop

Close Hadoop when you are all done.

$ ./stop-all.sh

I hope this short article has helped you get over the hurdle of installing Hadoop on your macOS machine!