A Beginner’s Guide to Hadoop’s Fundamentals | Towards Data Science

Literally, Hadoop was the name of a toy elephant — specifically, the toy elephant of Doug Cutting’s (Hadoop’s co-founder) son. But you’re not here to learn how, or from where, Hadoop got its name! Broadly speaking, Hadoop is a general-purpose, operating system-like platform for parallel computing.
I am sure I do not need to mention the severe limitations of a single system when it comes to processing all the big data floating around us — it is simply beyond the processing capacity of a single machine. Hadoop provides a framework to process this big data through parallel processing, similar to what supercomputers are used for.
But why can’t we utilize supercomputers to parallelize the processing of big data:
There is no standardized operating system (or an operating system like-framework) for supercomputers — making them less accessible to small and mid-sized organizations
High cost of both the initial purchase and regular maintenance
Hardware support is tied to a specific vendor, i.e., a company cannot procure the various individual components from different vendors and stack them together
In most cases, custom software needs to be developed to operate a supercomputer based on the specific use case
Not easy to scale horizontally
Hadoop comes to the rescue as it takes care of all the above limitations: it’s an open-source (with strong community support and regular updates), operating system-like platform for parallel processing that does not rely on specific hardware vendors for ongoing hardware support (works with commodity hardware) and does not require any proprietary software.
There have been three stable releases of Hadoop since 2006: Hadoop 1, Hadoop 2, and Hadoop 3.
Let’s now look at Hadoop’s architecture in more detail — I will start with Hadoop 1, which will make it easier for us to understand Hadoop 2’s architecture later on. I will also assume some basic familiarity with the following terms: commodity hardware, cluster & cluster node, distributed system, and hot standby.
Following are the major physical components of the Hadoop 1 architecture:
Master Nodes:
Name Node: Hadoop’s centralized file system manager, that keeps track of the number of blocks a data file was broken into, the block size, and which data nodes will save and process each file block — without saving any data within itself
Secondary Name Node: Backup for the Name Node, but not on hot standby
Job Tracker: Hadoop’s centralized job scheduler that is responsible for scheduling the execution of a job on data nodes
Each of the above nodes represents an individual machine in a production environment, working in the master mode, that are usually placed in different racks in a production setup (to avoid the failure of one rack bringing down multiple master nodes).
Slave Nodes:
Data Nodes: Individual machines/systems where the working files, in the form of data blocks, are stored and processed upon
Task Tracker: A software service to monitor the state of the Job Tracker, keep track of activities being performed by the Data Node, and report the status to the Job Tracker. One Task Tracker for each Data Node
The slave nodes cannot function without the master nodes and are fully dependant on the instructions that they receive from the master nodes before undertaking any kind of processing activities. To ensure continuous uptime, slave nodes send a heartbeat signal to the name node once every three seconds to confirm that they’re up and active.
All the above master and slave nodes are interconnected through networking infrastructure to each other to form a cluster.
In terms of processing capacities, Job Tracker is more powerful than the Name and Secondary Name Nodes with neither requiring substantial storage capacity. However, the data nodes are the most powerful machines within the cluster with substantial RAM and processing capabilities.
Following are the three primary deployment or configuration modes supported by Hadoop:
Standalone Mode: All Hadoop services (i.e., each of the Name Node, Secondary Name Node, Job Tracker, and Data Nodes) run locally on a single machine within a single Java Virtual Machine (JVM). However, the standalone mode is seldom used nowadaysPseudo Distributed Mode: All Hadoop services run locally on a single machine but within different JVMs. Pseudo Distributed Mode is usually used during development and testing activities and for educational purposesFully Distributed Mode: Used in a production setup, where all Hadoop services run on separate and dedicated machines/servers
Standalone Mode: All Hadoop services (i.e., each of the Name Node, Secondary Name Node, Job Tracker, and Data Nodes) run locally on a single machine within a single Java Virtual Machine (JVM). However, the standalone mode is seldom used nowadays
Pseudo Distributed Mode: All Hadoop services run locally on a single machine but within different JVMs. Pseudo Distributed Mode is usually used during development and testing activities and for educational purposes
Fully Distributed Mode: Used in a production setup, where all Hadoop services run on separate and dedicated machines/servers
A Job in the Hadoop ecosystem is analogous to a Python script/program that one can execute in order to perform a certain task(s). Just like a Python script, Hadoop’s job is a program(s), typically as a JAR file, that is submitted to the Hadoop cluster in order to be processed and executed on the input (raw) data that resides on the data nodes and the post-processed output is saved at a specified location.
Now let’s move on from Hadoop’s physical infrastructure to its software components. The core software components of Hadoop are:
Hadoop Distributed File System (HDFS) used for data storage and retrievalMapReduce, a parallel processing Java-based framework, is Hadoop’s programming arm that processes the data made available by the HDFS
Hadoop Distributed File System (HDFS) used for data storage and retrieval
MapReduce, a parallel processing Java-based framework, is Hadoop’s programming arm that processes the data made available by the HDFS
MapReduce is further comprised of:
A user-defined Map phase, which performs parallel processing of the input data
A user-defined Reduce phase, that aggregates the output of the Map phase
Just to be clear, Hadoop is a parallel processing platform providing the hardware and software tools to allow parallel processing) that then makes available the MapReduce framework (i.e., a bare-bone skeleton that can be customized based on the user requirements) for parallel processing. But MapReduce is not the only framework supported by Hadoop — Spark is another.
HDFS is the file-management component of the Hadoop ecosystem that is responsible for storing and keeping track of large data sets (both structured and unstructured data) across the various data nodes.
In order to understand the working of HDFS, let consider an input file of size 200MB. As explained earlier, in order to facilitate parallel processing on data nodes, this single file will be broken down into multiple blocks and saved on the data nodes.
The default split size (that is a global setting and can be configured by the Hadoop administrator) in HDFS is 64MB. Therefore, our sample input file of 200MB will be split into 4 blocks — where 3 blocks will be of 64MB and the 4th block will be 8MB. The splitting of the input file into individual blocks and saving them on specific data nodes is taken care of by HDFS.
One critical aspect to take note of here is that the splitting of the input file by HDFS happens on the client machine that is outside the Hadoop cluster and the name node decides the placement of each data block into the specific data nodes, based on a specific algorithm. So the client machine directly writes the data blocks to the data nodes once the name node has provided it with the block placement strategy.
The name node, acting as the Table of Contents of a book, remembers the placement of each data block within the various name nodes, together with other information, e.g., block size, hostname, etc., in a file table called the File System Image (FS Image).
So what happens in case of failure of a data node? The failure of even one data node will result in the entire input file being corrupted — since one piece of our puzzle has gone missing! In a typical production setup, where we are usually dealing with data blocks of hundreds of gigabytes, it is highly inefficient and time-consuming to push the original data file back into the Hadoop cluster.
To avoid any potential data loss, backup copies of data blocks on each data node is kept on an adjacent data node. The number of backup copies to be made of each data block is controlled by the Replication Factor. By default, the replication factor is set at 3, i.e., every block of data on each data node is saved on 2 additional backup data nodes so that the Hadoop cluster will have 3 copies of each data block. This replication factor can be configured on a per-file basis at the time of pushing the source data file into HDFS.
The backup data node will kick-in as soon as any data node fails to send a heartbeat signal to the name node. Once a backup data node is up and running, the name node will initiate another backup of the data block so that the replication factor of 3 holds throughout the cluster.
In Hadoop 1.0, the Secondary Name Node acts as a backup of the Name Node’s FS image. However, and this was one of Hadoop 1.0’s primary limitations, the Secondary Name Node does not operate in a hot standby mode. Therefore, in the event of the Name Node’s failure, the entire Hadoop cluster will go down (data will be present in the Data Nodes, however, it will be inaccessible since the cluster has lost the FS image), and the contents of the Secondary Name Node need to be manually copied to the Name Nome.
We will go over this later — but this limation was addressed with the release of Hadoop 2.0, where the Secondary Name Node acts as a Hot Standby.
Hadoop 2.0 is also sometimes known as MapReduce 2 (MR2) or Yet Another Resource Negotiator (YARN).
Let’s try to understand the salient architectural differences between Hadoop 1.0 and Hadoop 2.0. Remember that, in Hadoop 1.0, the Job Tracker acts as a centralized job scheduler that splits up a specific job into multiple jobs before passing them on to individual data nodes — where the individual tasks on the data nodes are monitored by the Task Tracker that then reports back the status to the Job Tracker. In addition to its job scheduling responsibilities, the Job Tracker also allocates the system resources to each data node in a static mode (that is, the system resources are not dynamic).
Hadoop 2.0 replaces the Job Tracker with YARN while the underlying file system remaining as HDFS. In addition to MapReduce, YARN also supports other parallel processing frameworks, e.g., Spark. YARN can also support up to 10,000 data notes, compared to only 4,000 data nodes supported by Hadoop 1.0’s Job Tracker.
YARN has 2 components: Scheduler and Applications Manager. Both these tasks were managed single-handedly by the Job Tracker in Hadoop 1.0. Separating these distinct responsibilities into YARN’s individual components allows better utilization of system resources.
Further, the task trackers on each data node were replaced by a single Node Manager (that works in the slave mode) in Hadoop 2.0. Node Manager communicates directly with YARN’s Applications Manager for resource management.
As alluded to earlier, in addition to a Secondary Name Node, Hadoop 2.0 also has a Hot Standby Name Node that seamlessly kicks-in in case of Name Node’s failure. The Secondary Name Node comes in handy in case of the failure of both the Name and the Hot Standby Name Node.
As the name suggests, MapReduce is comprised of the following 2 stages with each stage having 3 further sub-stages:
All 3 sub-stages of the Map stage are performed or acted upon in each of the data blocks residing in the individual data nodes — this is where parallelization kicks-in within Hadoop.
Record Reader
The Record Reader is pre-programmed to process one line at a time from the input file and produces 2 outputs:
Key: a number
Value: the entire line
Mapper
Mapper is programmable to process each key-value pair output from the Record Reader one at a time based on any required logic or the problem statement. It outputs additional Key-Value pairs based on a user-defined function it was programmed to perform.
Sorter
The output from Mapper is fed into the Sorter that lexicographically sorts (obviously! 😊) the keys from the Mapper’s output. In case the keys are numeric, then the Sorter will perform a numerical sorting. The Sorter is pre-programmed and the only configuration possible is to implement sorting on values.
At the end of the Map stage, we will have multiple Mapper outputs, one from each of the data nodes. All these outputs will be transferred to a separate, single data node where the Reduce operation will be implemented on them.
The 3 sub-stages of the Reduce operation are:
Merge
Intermediary outputs from each Map operation are appended to one another to result in a single merged file.
Shuffler
Shuffler is another pre-programmed built-in module that aggregates together the duplicate keys as present in its input, resulting in a list of values for each unique key.
Reducer
Shuffler’s output is fed to the Reducer, which is the programmable module of the Reduce stage — similar to the Mapper. Reducer produces output in key-value pairs based on what it is programmed to perform as per the problem statement.
I will use a very simple, non-ML problem statement to try and explain the mechanics and the workflow of MapReduce. Consider an input file with just 2 statements as follows:
Processing big data through Hadoop is easyHadoop is not the only big data processing platform
Our task is to find the frequency of words in the input file, the expected output being:
Processing    2big           2data          2through       1Hadoop        2is            2easy          1not           1the           1only          1platform      1
Going through the MapReduce stages explained above:
the output of Record Reader after reading the first line will be:Key: 0 (file/line offset — the starting position)Value: Processing big data through Hadoop is easy
We can program the Mapper to do the following:Step 1: Ignore the input keyStep 2: Extract each word from the line (tokenization)Step 3: Produce the output in key-value pairs where the key is each word of the line and value is the frequency of that word in the input lineAccordingly, Mapper’s output after processing both lines will be something like this:
Processing     1big            1data           1through        1Hadoop         1is             1easy           1Hadoop         1is             1not            1the            1only           1big            1data           1processing     1platform       1
The output from the Sorter will be something like this:
big            1big            1data           1data           1easy           1Hadoop         1Hadoop         1is             1is             1not            1only           1platform       1processing     1Processing     1the            1through        1
Shuffler’s output will be something like this:
big            1, 1data           1, 1easy           1Hadoop         1, 1is             1, 1not            1only           1platform       1processing     1, 1the            1through        1
Reducer can be programmed to do the following:Step 1: Take the key-value pair from Shuffler’s outputStep 2: Add up the list values for each keyStep 3: Output the key-value pairs where the key remains unchanged and the value is the sum of numbers in the list from Shuffler’s outputStep 4: Repeat the above steps for each key-value pair received from the ShufflerAccordingly, Reducer’s output will be:
big            2data           2easy           1Hadoop         2is             2not            1only           1platform       1processing     2the            1through        1
Certain industry use case of MapReduce include:
searching for keywords in big datasets
Google used it for wordcount, Adwords, page rank, indexing data for Google Search, article clustering for Google News (recently Google has moved on from MapReduce)
Text algorithms such as grep, text-indexing, reversing indexing
Data mining
Facebook uses it for data mining, ad optimization, spam detection
Analytics by financial services providers
Batch, non-interactive analysis
Right, so this was a very high-level and non-technical introduction to the world of Hadoop and MapReduce. Obviously, there are several other Hadoop components that I have not even touched upon here, e.g., Hive, Zookeeper, Pig, HBase, Spark, etc.
Please feel free to reach out to me if you want to discuss the above content, or that in any of my previous posts, or anything in general related to data analytics, machine learning, and financial risk.
Till next time, rock on!