Providing Elasticity to the HDFS Cluster
By: Ankush Chavan
Tags: Hadoop, Hdfs, Big Data, Storage, Lvm

Confirm that the Physical Volumes are created using the pvdisplay command.

$ pvdisplay /dev/xvdf $ pvdisplay /dev/xvdg

Step 4: Create a Volume Group

To get the space from separate Physical Volumes we have to combine their space. To achieve this, we have to create a Volume Group that has space from both /dev/xvdf1 and /dev/xvdg1 physical volumes.

$ vgcreate contribdata /dev/xvdf1 /dev/xvdg1

This will create a volume group of size 5.5 GiB (512 MiB + 5 GiB).

View the created Volume Group using the following command

$ vgdisplay contribdata

Step 5: Create a Logical Volume

Now, we have Volume Group ready. We have to create a logical volume from this volume group, so that, we can mount the logical volume to use it.

$ lvcreate --size 3G --name ankush contribdata

The above command will create a logical volume of size 3 GiB and named ankush. This logical volume will take the storage space from the volume group named contribdata.

We can display the logical volume using the command lvdisplay.

$ lvdisplay

The command above will also show the exact path/name of the logical volume. For this logical volume, the exact name of the logical volume will be /dev/contribdata/ankush.

So, finally, we have a logical volume ready. And as this volume is the Logical Volume, we can increase or decrease its space on the go without losing any of the stored data.

Step 6: Mount the Logical Volume

As we already had configured HDFS cluster and our datanode will share the storage space to the HDFS cluster from /data directory. So, we have to mount the logical volume to the logical volume at /data directory. But, before mounting, we will format the logical volume to ext4 partition type.

$ mkfs.ext4 /dev/contribdata/ankush

Now, the logical volume is ready to use. We will mount the logical volume named /dev/contribdata/ankush at /data directory.

$ mount /dev/contribdata/ankush /data

Step 7: Start the Datanode

Now, we are all set to start the Datanode. This Datanode will share the storage to the HDFS cluster from the logical volume.

Before starting the Datanode, I will go to the Namenode and check how much storage it currently has.

$ hadoop dfsadmin -report

You can see in the above screenshot that the HDFS cluster does not have any Datanode connected and hence the current storage space is 0 GiB.

Now, Go back to the Datanode and start the Datanode.

$ hadoop-daemon.sh start datanode

After that, go to the Namenode again and check the storage space.

You can see that, after starting the Datanode, the HDFS cluster now has 2.89 GiB(approx. 3 GiB) of storage space. All this space is coming from Datanode and hence from the logical volume.

Now, Let’s see the power of LVM in the real scenario.

Let us assume that we are about to run out of the space of 3 GiB and we have to increase the space of the Datanode instead of adding a new Datanode. This problem may in the real world and copying all of the data to the new harddisk of more size is not the optimized solution. So, here the role of the LVM comes into play.

Now, we just have to extend the space of the logical volume and that's it! We can increase the storage without losing any past data.

Let’s see this in action.

Increasing the storage space of the Datanode

For increasing the space of the Datanode storage, we have to increase the size of the logical volume. This size can only be possible to extend if the volume group of the logical volume has unallocated storage space. As we had used 3 GiB for the logical volume out of 5.5 GiB total size of the volume group, we can extend our logical volume by up to 2.5 GiB.

Here, we will extend the logical volume by 1 GiB.

So, let’s begin.

Step 1: Stop the Datanode and unmount the logical volume

As the logical volume is mounted to /data directory, we will unmount it in order to increase its space.

$ hadoop-daemon.sh stop datanode

$ umount /data

Step 2: Extend the size of the logical volume

To extend the size of the logical volume, run the following command.

$ lvextend --size +1G /dev/contribdata/ankush

This will increase the size of the logical volume by 1 GiB. But this size is increased logically. If we check the size of the logical volume, it will show it 4GiB but after mounting the volume, the actual size is 3 GiB. This is because, we had extended the size of the logical volume but inorder to get the extended storage, we have to format only that part of the volume that we had added newly.

Step 3: Format the newly added storage to the logical volume

If we try to format the volume by using the regular method, it will recreate the whole partition table. And thus, we will lose the data stored in the volume. But, we have to format only that part of the volume that we had added newly. To do this, we will use the command resize2fs.

$ e2fsck -f /dev/contribdata/ankush

$ resize2fs /dev/contribdata/ankush

The command above will format the un-formated part of the logical volume.

Now, we have the size of 4 GiB ready to use.

Step 4: Mount the logical volume and start the datanode

Mount the logical volume to the /data directory.

$ mount /dev/contribdata/ankush /data

Start the datanode again.

$ hadoop-daemon.sh start datanode

Now, go back to the Namenode and check the HDFS storage.

In the above screenshot, you can see that the available storage space is 3.87 GiB(approx. 4 GiB).

So, this will prove that, we had successfully increased the storage space of the datanode on the go.

In the same way, we can reduce the size of the Datanode storage using lvreduce command.

Reducing the storage space of the datanode

For, reducing the Datanode storage space, run the following commands.

Step 1: Unmount logical volume

$ unmount /data

Step 2: Resize the LVM to 2 GiB

$ e2fsck -f /dev/contribdata/ankush

$ resize2fs /dev/contribdata/ankush 2G

Step 3: Reduce the size using lvreduce

$ lvreduce -L 2G /dev/contribdata/ankush

$ e2fsck -f /dev/contribdata/ankush

Step 4: Mount the LVM to the /data

$ mount /dev/contribdata/ankush /data

Creating a Python script to automate the elasticity of the HDFS cluster

The script above will automate the process of providing elasticity to the HDFS cluster.

That's it for this article!