Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
By: Mangesh Ghorpade
Tags: Hadoop, Lvm

✨Task Description —

7.1: Elasticity Task

◆Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

◆Increase or Decrease the Size of Static Partition in Linux

In this blog, I am going to discuss how we can integrate LVM with Hadoop to provide elasticity to the Data node storage. Let me first explain about the task which Im going to perform.

For this task, there are some prerequisites

✍Task Description :

In this task, we have to make the data node storage elastic so that whenever we required to change the size of data node storage we can achieve it without shutting down the data node or without stopping the service of the data node.

✍Prerequisite :

●Basic knowledge of Hadoop

●Knowledge of Linux Partition and LVM

Hadoop :

Hadoop is an open-source framework that allows to store and process of big data in a distributed storage environment across the cluster of computers. It provides massive storage for any kind of data, enormous processing power, and the ability to handle virtually limitless concurrent tasks or jobs.

What is LVM ?

LVM is a tool for logical volume management which includes allocating disks, striping, mirroring and resizing logical volumes.

With LVM, a hard drive or set of hard drives is allocated to one or more physical volumes. LVM physical volumes can be placed on other block devices which might span two or more disks. LVM provides features like the ability to add disk space to a logical volume and its filesystem. LVM helps to provide the Elasticity to the storage Device.

What is Elasticity ?

Elasticity is the concept which we can use to increase or decrease the volume of Hadoop Data Node. Hadoop data nodes shared storage can’t be static so LVM is used to make it dynamic.

Now let’s begin our practical with a simple Hadoop cluster of one data node only.

Step-1) Create a Hadoop Cluster.

Configure NameNode -

Install Java and Hadoop in NameNode System then configure the hdfs-site.xml file-

Configure the core-site.xml file-

Now format the NameNode and start the Service-

Configure Data Node -

Install Java and Hadoop in DataNode System then configure the hdfs-site.xml file-

Configure the core-site.xml file-

Now start the Service of DataNode

hadoop dfsadmin -report

Here, we can see that DataNode is providing all 50GB storage to NameNode.

Step-2) Now Create Partition:

Attach Volume to DataNode System -

Create a volume of 10GB-

In the Same way, I am creating one more volume of 9GB.

Here, I have attached both the volume to my DataNode.

Now Check that volume (Hard Disk) is attached or not by using ‘fdisk -l’ command-

fdisk -l

Here two volume of 10GB and 9GB have attached.

Step-3) Now create Physical Volume for both Volumes-

Physical volume for first volume-

pvcreate /dev/sdb

pvdisplay /dev/sdb

Physical volume for second volume-

pvcreate /dev/sdc

pvdisplay /dev/sdc

Step-4) Now create a Volume Group for both the physical volumes.

vgcreate V_Group /dev/sdb /dev/sdc

vgdisplay V_Group

Here Volume Group of 18.99Gib is created.

Step-5) Create Logical Volume of 5GB from above volume group.

lvcreate --size %G --name lv1 V_Group

Now format the Logical Volume-

mkfs.ext4

Now create a directory to mount this Logical Volume -

mkdir /mydb

Now mount the above Logical Volume to this directory-

mount /dev/V_Group/lv1 /mydb

Now check that logical volume is mounted or not-

Step-6) Now update this directory to hdfs-site.xml file of DataNode.

Now Start the Service of DataNode.

hadoop-daemon.sh start datanode

Now check the report of hadoop cluster in NameNode-

hadoop dfsadmin -report

Here, Now DataNode is only Contributing 5GB to NameNode.

Increase the Logical Volume:

I am increasing the size of logical volume by 2GB-

lvextend --size +2G /dev/V_Group/lv1

Now We have to format 2Gb extra part, to format the Non-partition portion from the complete LV.

resize2fs /dev/V_Group/lv1

lvdisplay V_Group/lv1

Now check the hadoop cluster report in NameNode-

Here, we have extended the logical volume successfully.

In the Same way we can also reduce the LV size.

Now check the size of logical volume-

lvdisplay V_Group/lv1

Here, LV size is reduced from 7GB to 3GB.

In this way, Task is Completed Successfully !!

!! Thank You for Reading the Article !!