Hadoop Distributed File System
By: Prathamesh Nimkar
Tags: Hadoop, Hdfs, Data Engineering, Data Science, Big Data

The data nodes send an activity update to the “Active” Name Node (every 5 seconds at minimum — configurable). This metadata is synced to the “Stand by” Name Node in real-time. Thus, when the “Active” fails, the “Stand by” has all the necessary metadata to switch over.

The Zookeeper, through its Fail-over Controller, monitors the health of the Active and Stand by Name Nodes through a heart beat or instant notification it receives from each NN (every 5 seconds, again configurable). It also has the information of all the stand by name nodes available (Hadoop 3.x allows for multiple stand by name nodes).

Thus, a connectivity between the data nodes, name nodes and zookeeper is established. The moment an active name node fails, the Zookeeper elects an appropriate stand by name node and facilitates the automatic switch over. The Stand by becomes the new Active Name Node and broadcasts this election to all the data nodes. The data nodes now send their activity updates to the newly elected Active Name Node within a few minutes.

What is NameNode Metadata?

NameNode Metadata

The name node (NN) metadata consists of two persistent files, namely, FsImage — namespace and Edit logs — transaction logs (insert, append)

Namespace & FsImage

In every file system, there is a path to the required files — On Windows: C:\Users\username\learning\BigData

amenode.txt and on Unix: /usr/username/learning/BigData/namenode.txt. HDFS follows the Unix way of namespace. This namespace is stored as part of the FsImage. Every detail of the file i.e. who, what, when, etc. is also stored in the FsImage snapshot. The FsImage is stored on the disk for consistency, durability and security.

Edit logs

Any real-time changes to all files are logged in what is known as “Edit logs”. These are recorded in-memory (RAM) and contain every little detail of the change and the respective file/block.

On HDFS startup, the metadata is read from the FsImage and the changes are written to Edit Logs. Once the data is recorded for the day in Edit Logs, it is flushed down onto the FsImage. This is how the two work in tandem.

As an aside, the FsImage and Edit Logs are not human readable. They are binary-compressed (serialized) and stored in the file system. However, for debugging purposes it can be converted into an xml format to be read using Offline Image Viewer.

How does NameNode Metadata Sync?

As you can imagine or see in the image ‘HDFS High Availability Architecture’, the name node metadata is a single point of failure, hence this metadata is replicated to introduce redundancy and enable high-availability (HA).

Shared Storage

Shared Storage Sync

We now know that there exists an Active Name Node and a Standby Name Node. Any change in the active is synced in real-time to the shared folder/storage i.e. network file system (NFS). This NFS is accessible to the standby, which downloads all of the relevant incremental information in real-time to maintain the sync between the Namenodes. Thus, in the event of a failure of the active, the standby name node already has all the relevant information to continue “business as usual” post fail-over. This is not used in the Production environment.

Quorum Journal Node (QJN)

QJN Sync

“Quorum” means minimum required to facilitate an event. The term is generally used in politics; it is the minimum number of representatives required to conduct proceedings in the house.

Here, we use this concept to determine the minimum number of journal nodes aka quorum that is needed to establish a majority and maintain metadata sync.

The image shows three (always odd) journal nodes (process threads not physical nodes) that help to establish metadata sync. When an Active NN receives a change, it pushes it to majority of the QJ Nodes (follow a single colour). The Standby NN, in real-time, requests the majority number of QJ Nodes for the required metadata to establish the sync.

The minimum number for QJN to function is 3 and the quorum/majority is determined by the following formula:

Q = (N+1)/2

where N = total number of Journal Nodes

For example, if we have N=5, the quorum/majority would be established by (5+1)/2 i.e. 3. The metadata change would be written to 3 journal nodes.

The QJN is the preferred Production method of metadata sync as it is also “highly available”. In the event of a failure of any of the QJ nodes, any of the remaining nodes are available to provide the required data to maintain metadata sync. Thus, the standby already has all the relevant information to continue “business as usual” post fail-over.

This brings us to the end of my comprehensive guide on HDFS and it’s inner workings.