Increasing the limit of the storage of the Datanode to the Hadoop Cluster dynamically
By: Sachin Kumar
Tags: Aws Ec2, Lvm, Storage, Hadoop, AWS

Increasing the limit of the storage of the Datanode to the Hadoop Cluster dynamically

Let’s see how we can increase or distribute the size of storage of the Datanode to the Hadoop Cluster dynamically using the Logical Volume Management Concept.

Before starting the main part of this article I would like to give some overview of those concepts and terminologies which we will use to increase the storage of Datanode to the Hadoop Cluster dynamically. And the more important thing, we will use most of the concepts which are related to my previous article where I have explained how we can distribute the storage to the Hadoop Cluster in the Static form. So I would like to suggest you please see this article first then after you can move on to this article so that you can relate every bit of this. Now I am assuming that you have read my article. So let’s start with Logical Volume Management without any delay.

Logical Volume Management

LVM, or Logical Volume Management, is a storage device management technology that gives users the power to pool and abstract the physical layout of component storage devices for easier and flexible administration. Utilizing the device-mapper Linux kernel framework, the current iteration, LVM2, can be used to gather existing storage devices into groups and allocate logical units from the combined space as needed.

The main advantages of LVM have increased abstraction, flexibility, and control. Logical volumes can have meaningful names like “databases” or “root-backup”. Volumes can be resized dynamically as space requirements change and migrated between physical devices within the pool on a running system or exported easily. LVM also offers advanced features like snapshotting, striping, and mirroring.

In this guide, we will briefly discuss how LVM works and then demonstrate the basic commands needed to get up and running quickly.

LVM Architecture and Terminology

Before we dive into the actual LVM administrative commands, it is important to have a basic understanding of how LVM organizes storage devices and some of the terminology it employs.

LVM Architecture

LVM Storage Management Structure

LVM functions by layering abstractions on top of physical storage devices. The basic layers that LVM uses, starting with the most primitive, are.

LVM Storage Management Structure

Physical Volume:

LVM utility prefix : pv...

: Description: Physical block devices or other disk-like devices (for example, other devices created by device-mapper, like RAID arrays) are used by LVM as the raw building material for higher levels of abstraction. Physical volumes are regular storage devices. LVM writes a header to the device to allocate it for management.

Volume Group:

LVM utility prefix : vg...

: Description: LVM combines physical volumes into storage pools known as volume groups. Volume groups abstract the characteristics of the underlying devices and function as a unified logical device with a combined storage capacity of the component physical volumes.

Logical Volume:

LVM utility prefix : lv... (generic LVM utilities might begin with lvm... )

: (generic LVM utilities might begin with ) Description: A volume group can be sliced up into any number of logical volumes. Logical volumes are functionally equivalent to partitions on a physical disk, but with much more flexibility. Logical volumes are the primary component that users and applications will interact with.

In summary, LVM can be used to combine physical volumes into volume groups to unify the storage space available on a system. Afterward, administrators can segment the volume group into arbitrary logical volumes, which act as flexible partitions.

Hopefully, now you have cleared about LVM and its terminology because we will use these concepts to increase the storage to the Hadoop cluster dynamically in our practical part. Before starting the main part, I want to give you the actual scenario of what we are doing exactly forward.

Problem Statement

Here Problem is that we are providing the storage to the Hadoop cluster through our DataNode. But we want to distribute only limited storage to the Cluster. Also, that scenario has already been solved in the above-mentioned article Statically. But now we want that this scenario is taken place dynamically. As we require more or less storage for the Datanode to providing to the Cluster. At that time We can do all these things on the fly. So let’s start with practical then you have a more clear idea of what exactly we are going to achieve.

Note: Here I am using the orange-colored terminal for NodeNode and white-colored terminal for Datanode as well as I used aqua-colored terminal for creating and attaching the volume with in datanode on the top of AWS.

Also, I am not going here how we configure the Hadoop cluster. So I am assuming you know how to configure the Hadoop cluster if you don't know then please go through my below-mentioned article.

Note: I will do all the steps of practical on the top of AWS Redhat Instance So please make sure these things also. I don't want you will face any challenge.

Hopefully, now you have a clear idea about all the above mentioned so let’s start the Hadoop cluster and see using WebUI how much storage is provided by Datanode to the cluster.

So you can see Datanode is providing 9.99GB to the cluster in the below-mentioned screenshot. But we want to give the limit storage so let’s see all these things.

Note: Using pubicip_of_namenode:50070 eg. ( 13.233.128.206:50070 ), you can access the WebUI of the Hadoop cluster.

WebUI Dashboard of Hadoop Cluster

I'm also showing you how many devices Datanode has before attaching any extra volume and fdisk -l is a command which is used to see how many disk your system has.

So, first of all, create 2 EBS volumes using the below-mentioned command so we can perform the LVM concept in the Hadoop.

Note: I am performing all these things on the AWS but you can do on your local system or vm using same approach wherever you want.But if you want to AWS and you don’t know how AWS CLI then you can go through my AWS CLI Aricle.

aws ec2 create-volume --availability-zone ap-south-1b --size 1

Volumes have created as you can see in the above screenshot So now we have to attach these volumes with the Datanode using the below command.

aws ec2 attach-volume --volume-id <value> --instance-id <value> --device /dev/sdf

Note: device name should be diffrent of both volumes.

Attach another created volume with the same Datanode using the above-mentioned command.

Now Both volumes have been attached with Datanode as you have seen above in both screenshots. But you can also confirm with fdisk -l command they have attached or not.

So you can see /dev/xvdf and /dev/xvdg have attached with 1–1 GiB size with the Datanode in the above screenshot. Now we have to apply the LVM concept here but first, install the LVM in your Datanode using the below command.

yum install lvm2 -y

After installing lvm, make these volumes as Physical volume using pvcreate devicename command and check all the details of this Physical volume (PV) using the pvdisplay devicename command as you can see in the below screenshot.

Now do the same thing with /dev/xvdg as you have done above.

Now Both PVs are ready for use. So, create Volume Group (VG) for them using the below-mentioned command and vgdisplay vgname command is used to see all the details of the Volume Group (VG)

vgcreate vgname /first_device_name /second_device_name

Volume Group had created with the name of hadoopvg. So now you have to create a Logical Volume using the below-mentioned command if you want to use this volume for your Datanode and lvdisplay vgname/lvname command is used to display the details of Logical Volume.

Note: Here lvname is presenting for logical volume name as well as vgname for Volume Group.

lvcreate --size <value> --name lvname vgname

After creating the Logical Volume (LV) you have to format it. So, Use the below-mentioned command to format the Logical Volume (LV).

mkfs.ext4 /dev/vgname/lvname

LV has formated as you can see in the above screenshot. So now mount it with Datanode folder /dn2 .which is responsible for providing the storage to the Hadoop Cluster using the below-mentioned command.

Note: /dn2 folder has created during the creation of cluster inside the Datanode. So you can give anyname when you create a hadoop cluster.

mount /dev/vgname/lvname /dn2

Logical Volume has been mounted but you can check it has been mounted or not in your Namenode as well as Datanode using the below-mentioned command.

hadoop dfsadmin -report

Same thing you can check in your WebUI of Hadoop Cluster also.

So you have seen in the above two screenshots Hadoop cluster has 1.93 GB configured Capacity instead of 9.99 GB now which you have already seen in the first screenshot. Hence it has been proved that we can contribute storage of Datanode to the Hadoop Cluster as much we want. But one problem has arisen here that if the storage has been fulled then how we can increase the storage at that time. So let’s see the solution to it.

Create one more volume to increase the storage of Datanode to the cluster using the same which has been used above. But I am mentioning the same below also.

aws ec2 create-volume --availability-zone ap-south-1b --size 1

Now you have to attach this created volume with Datanode so use the below command to it.

aws ec2 attach-volume --volume-id <value> --instance-id <value> --device /dev/sdh

Create a Physical volume for attaching with Volume Group using the pvcreate devicename command.

Attach this Physical Volume with your pre-created Volume Group using vgextend vgname /devivename the command.

Now you can extend the size of the storage of Datanode using the below command.

lvextend --size <value> /dev/vgname/lvname

Logical Volume has resized successfully so you have to just mount it with /dn2 folder using the below command.

resize2fs /dev/vgname/lvname

Now everything has taken place but you can check it through your cluster WebUI.

As you can see in the above screenshot Cluster Configured Capacity has been 2.82 GB instead of 1.93GB.

Conclusion

As you can see above you can distribute the amount of storage of Datanode to the Hadoop Cluster dynamically and extend it on the fly whenever and how much you want. Also, you have a good idea of Logical Volume Management. it helps you to configure this setup dynamically.

I tried to explain as much as possible. Hope You learned Something from here. Feel free to check out my LinkedIn profile mentioned below and obviously feel free to comment. I write Cloud Computing, Machine Learning, Bigdata, and Web, etc. blogs so feel free to follow me on Medium.