HDFS Simple Docker Installation Guide for Data Science Workflow | by Paras Varshney | Towards Data Science

In this small article, we would be discussing how to set up the Docker-based Hadoop distributed file system on your computer and will discuss a simple example to demonstrate the use-case. Also once the setup is ready to be with on your machine, you can start building your own map-reduce jobs to play around with Hadoop DFS.
First, you need to clone the following Git repository into your desired directory in your system. I prefer cloning it into the home for demonstration and easy access.
git clone https://github.com/big-data-europe/docker-hadoop
then enter the project repository using the change directory command:
cd docker-hadoop
Now on doing an “ls -l” command you can see all the files inside this repository as shown below.
After that, you need to spin up the containers required to setup HDFS on your system. To do that, there is a docker-compose.yml file in the project directory, you need to use the “docker-compose up” command to download and install the required images from the Docker hub and configure the containers based on the docker-compose.yml file. The “-d” flag runs the containers in detached mode. If the images are not found locally, Docker downloads them from DockerHub, but if you want to manually download them you can use “docker-compose pull”.
Note: Use “sudo ” as the prefix of these commands, if you get a permission error.
# download images required for setting up HDFS and spin up necessary # containers.docker-compose up -d
The above command will download all the necessary Docker images from the docker hub for setting up HDFS containers. It might take a little while to download images depending on your internet speed.
Now, to have a look at your current running Docker containers, use the command to list all active containers.
# List all the available running docker containers.docker container ls
Once you have all your files copied inside your Hadoop cluster in a directory(note: in this example, I have copied files in /tmp), now you would go inside the namenode using the following command in an interactive terminal mode in bash mode.
# Enter inside namenode and open its bashdocker exec -it namenode /bin/bash
Example: sudo docker cp my_input.txt namenode:/tmp/
Now we need to copy the jar files which contains our map-reduce jobs and copy them inside the namenode (which will be running your jobs) in HDFS using the following Docker command:
docker cp <file_name> namenode:/<path>
Once you enter the name node in an interactive terminal, use the following HDFS commands to interact with the namenode.
# HDFS list commands to show all the directories in root "/"hdfs dfs -ls /# Create a new directory inside HDFS using mkdir tag.hdfs dfs -mkdir -p /user/root# Copy the files to the input path in HDFS.hdfs dfs -put <file_name> <path># Have a look at the content of your input file.hdfs dfs -cat <input_file>
Now you can run your map-reduce job using the following command:
# Run map reduce job from the path where you have the jar file.hadoop jar <jar_file_name> <class_name> input output
Example: hadoop jar word_counter.jar org.apache.hadoop.examples.WordCount input output
Once this command runs successfully, you will notice that the map-reduce job completes its execution with some information on the console about the process.
Once the job is executed successfully, you can check your output using the cat command in HDFS:
# Check the content of the output file after running the jobhdfs dfs -cat <output_file>
You will see the word frequency of your word counter job should be printed on the console.
Congratulations! You have successfully configured and created your first Hadoop HDFS map-reduce job!
You can access the HDFS namenode’s UI dashboard on your localhost at port 9870. Use the following link:
http://<your_ip_address>:9870
So basically the setup of HDFS was straightforward for simple data science workflow and you have learned after reading this article that how simple it is to set up a local Docker-based HDFS set up and start writing your own map-reduce jobs to execute various tasks. If you are not very familiar with the map-reduce jobs so I have attached a few useful links. Enjoy!
https://en.wikipedia.org/wiki/MapReducehttps://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.htmlhttps://hadoop.apache.org/docs/r1.2.1/hdfs_user_guide.html
https://en.wikipedia.org/wiki/MapReduce
https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
https://hadoop.apache.org/docs/r1.2.1/hdfs_user_guide.html
Hope you found the article helpful!Thank You!