Hadoop : How to install in 5 Steps in Windows 10
By: Shashank Singhal
Tags: Hadoop, Big Data, Data Analytics, Data, Data Analysis

4. Editing Hadoop files

Once we have configured the environment variables next step is to configure Hadoop. It has 3 parts:-

4.1 Creating Folders

We need to create a folder data in the hadoop directory, and 2 sub folders namenode and datanode

Fig. 13:- Creating Data Folder

Create DATA folder in the Hadoop directory

Fig. 14:- Creating Sub-folders

Once DATA folder is created, we need to create 2 new folders namely, namenode and datanode inside the data folder

inside the data folder These folders are important because files on HDFS resides inside the datanode.

4.2 Editing Configuration Files

Now we need to edit the following config files in hadoop for configuring it :-

(We can find these files in Hadoop -> etc -> hadoop)

* core-site.xml

* hdfs-site.xml

* mapred-site.xml

* yarn-site.xml

* hadoop-env.cmd

4.2.1 Editing core-site.xml

Right click on the file, select edit and paste the following content within <configuration> </configuration> tags.

Note:- Below part already has the configuration tag, we need to copy only the part inside it.

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

4.2.2 Editing hdfs-site.xml

Right click on the file, select edit and paste the following content within <configuration></configuration>tags.

Note:- Below part already has the configuration tag, we need to copy only the part inside it.

Also replace PATH~1 and PATH~2 with the path of namenode and datanode folder that we created recently(step 4.1).

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>PATH~1

amenode</value>

<final>true</final>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>PATH~2\datanode</value>

<final>true</final>

</property>

</configuration>

4.2.3 Editing mapred-site.xml

Right click on the file, select edit and paste the following content within <configuration> </configuration> tags.

Note:- Below part already has the configuration tag, we need to copy only the part inside it.

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

4.2.4 Editing yarn-site.xml

Right click on the file, select edit and paste the following content within <configuration> </configuration> tags.

Note:- Below part already has the configuration tag, we need to copy only the part inside it.

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<!-- Site specific YARN configuration properties --> </configuration>

4.2.5 Verifying hadoop-env.cmd

Right click on the file, select edit and check if the JAVA_HOME is set correctly or not.

We can replace the JAVA_HOME variable in the file with your actual JAVA_HOME that we configured in the System Variable.

set JAVA_HOME=%JAVA_HOME%

OR

set JAVA_HOME="C:\Program Files\Java\jdk1.8.0_221"

4.3 Replacing bin

Last step in configuring the hadoop is to download and replace the bin folder.

* Go to this GitHub Repo and download the bin folder as a zip.

* Extract the zip and copy all the files present under bin folder to %HADOOP_HOME%\bin

Note:- If you are using different version of Hadoop then please search for its respective bin folder and download it.

5. Testing Setup

Congratulation..!!!!!

We are done with the setting up the Hadoop in our System.

Now we need to check if everything works smoothly…

5.1 Formatting Namenode

Before starting hadoop we need to format the namenode for this we need to start a NEW Command Prompt and run below command

hadoop namenode -format

Fig. 15:- Formatting Namenode

Note:- This command formats all the data in namenode. So, its advisable to use only at the start and do not use it every time while starting hadoop cluster to avoid data loss.

5.2 Launching Hadoop

Now we need to start a new Command Prompt remember to run it as administrator to avoid permission issues and execute below commands

start-all.cmd

Fig. 16:- start-all.cmd

This will open 4 new cmd windows running 4 different Daemons of hadoop:-

* Namenode

* Datanode

* Resourcemanager

* Nodemanager

Fig. 17:- Hadoop Deamons

Note:- We can verify if all the daemons are up and running using jps command in new cmd window.

6. Running Hadoop (Verifying Web UIs)

6.1 Namenode

Open localhost:50070 in a browser tab to verify namenode health.

Fig. 18:- Namenode Web UI

6.2 Resourcemanger

Open localhost:8088 in a browser tab to check resourcemanager details.

Fig. 19:- Resourcemanager Web UI

6.3 Datanode

Open localhost:50075 in a browser tab to checkout datanode.

Fig. 20:- Datanode Web UI

7. Congratulations..!!!!🎉

Congratulation! We have successfully installed Hadoop.

There are chances that some of us might have faced some issues… Don’t worry its most likely due to some small miss or incompatible software. If you face any such issue please visit all the steps once again carefully and verify for the right software versions.

If you still can’t get Hadoop up and running, Don’t hesitate describe your problem below in the comment section.

8. Special Note 🙏

Don’t forget to Like and share it with your friends and colleagues.

Also, don’t miss the opportunity to share your views and suggestions for this or new articles.

You can know more About Me HERE..

Happy Learning… !!! 🙂