Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
Tags: Hadoop, Lvm

In this blog, I am going to discuss how we can integrate LVM with Hadoop to provide elasticity to the Datanode storage. Let me first explain about the task which Im going to perform.

Task Description :

In this task, we have to make the datanode storage elastic so that whenever we required to change the size of datanode storage we can achieve it without shutting down the datanode or without stopping the service of the datanode.

For this task, there are some prerequisites

Prerequisite :

Basic knowledge of Hadoop

Knowledge of Linux Partition and LVM

Hadoop :

Hadoop is an open-source framework that allows to store and process of big data in a distributed storage environment across the cluster of computers. It provides massive storage for any kind of data, enormous processing power, and the ability to handle virtually limitless concurrent tasks or jobs.

What is Elasticity ?

Elasticity is the concept which we can use to increase or decrease the volume of Hadoop Data Node. Hadoop data nodes shared storage can’t be static so LVM is used to make it dynamic.

Logical Volume Management :

LVM is a tool for logical volume management which includes allocating disks, striping, mirroring, and resizing logical volumes. With LVM, a hard drive or set of hard drives is allocated to one or more physical volumes. LVM physical volumes can be placed on other block devices which might span two or more disks. The physical volumes are combined into logical volumes.

Now let’s begin our practical with a simple Hadoop cluster of one data node only.

step-1) Create a Hadoop Cluster.

Configure NameNode -

Install Java and Hadoop in NameNode System then configure the hdfs-site.xml file-

hdfs-site.xml (NameNode)

Configure the core-site.xml file-

core-site.xml (NameNode)

Now we format the NameNode and start the Service-

Configure Data Node -

Install Java and Hadoop in DataNode System then configure the hdfs-site.xml file-

hdfs-site.xml (DataNode)

Configure the core-site.xml file-

core-site.xml (DataNode)

Now start the Service of DataNode -

Here, we can see that DataNode is providing all 20GB storage to NameNode using command

hadoop dfsadmin -report

Step-2) Now Create Partition:

Attach Volume to DataNode System -

Create a volumes -

Volume 1

Volume 2

Here, I have attached both the volume to DataNode.

Now Check that volume (Hard Disk) is attached or not by using command-

fdisk -l

Here two volume of 10GB and 9GB have attached.

Step-3) Now create Physical Volume for both Volumes-

Physical volume for first volume-

pvcreate /dev/sdb

pvdisplay /dev/sdb

Physical volume for second volume-

pvcreate /dev/sdc

pvdisplay /dev/sdc

Step-4) Now create a Volume Group for both the physical volumes.

vgcreate V_Group /dev/sdb /dev/sdc

vgdisplay V_Group

Here Volume Group of 18.99Gib is created.

Step-5) Create Logical Volume of 5GB from above volume group.

lvcreate --size %G --name lv1 V_Group

Now format the Logical Volume-

mkfs.ext4

Format logical volume

Now create a directory to mount this Logical Volume -

mkdir /mydb

Now mount the above Logical Volume to this directory-

mount /dev/V_Group/lv1 /mydb

Now check that logical volume is mounted or not-

df -h

Step-6) Now update this directory to hdfs-site.xml file of DataNode.

hdfs-site.xml (DataNode)

Now Start the Service of DataNode.

hadoop-daemon.sh start datanode

Now check the report of hadoop cluster in NameNode-

hadoop dfsadmin -report

Here, Now DataNode is only Contributing 5GB to NameNode.

Increase the Logical Volume:

Step-7) Now increase the logical volume size .

I am increasing the size of logical volume by 2GB-

lvextend --size +2G /dev/V_Group/lv1

Now We have to format 2Gb extra part, to format the Non-partition portion from the complete LV.

resize2fs /dev/V_Group/lv1

lvdisplay V_Group/lv1

Now check the hadoop cluster report in NameNode-

Here, we have extended the logical volume successfully.

In the Same way we can also reduce the LV size.

lvreduce -L 3G /dev/V_Group/lv1

Now check the size of logical volume-

lvdisplay V_Group/lv1

Here, LV size is reduced from 7GB to 3GB.

In this way, Task is Completed Successfully !!

Conclusion

In this way, we can integrate LVM with a Hadoop Cluster to take the advantage of Elasticity.

Thank You For Reading:)