How to Setup Hadoop Cluster on AWS using Ansible
By: Ajay Pathak
Tags: Hadoop, Automation, Ansible, DevOps, AWS

Hey everyone,

In this article, I will show you an interesting Automation in which we will Setup the Hadoop Cluster (HDFS) on top of AWS Cloud (EC2) and we will do everything using a tool called Ansible which is one of the best Automation tools. And the best thing in this article is we will use Ansible’s Dynamic Inventory and Roles concepts to make the whole process more Dynamic.

I expect that you have some familiarity with AWS, maybe you’ve launched an EC2 instance, or similar. And also have some basic knowledge about Ansible and Hadoop. You can easily find information about these technologies on the internet and in this article, our main aim is to integrate all these technologies.

So let’s understand what we actually gonna do to setup this automation.

A genda!!!

In the real world, we generally deal with BigData and the size of this BigData is in Peta or ExaBytes or more. And to handle and process this big data we use technologies like Hadoop. But to store such huge data we need more and more storage devices or resources and it will increase our cost. But we can use Cloud Storage to store this bigdata as Cloud providers provide almost unlimited Storage like AWS S3, EBS, EFS.

So in this, we will launch EC2 instances to setup our Hadoop Cluster i.e., NameNode and DataNodes. And then we will create and attach EBS Volume with DataNodes to store the bigdata that the HDFS Cluster processes. And to setup all these things we will use Ansible to automate our work.

So Let’s do the Hands-on Practical!

To know the installation of Ansible refer to my previous article

Setup AWS Ansible Dynamic Inventory

Now first we have to launch EC2 Instances using Ansible. But before doing this we have to setup some more things in ansible so that we can able to do Provisioning on AWS using Ansible.

First, we have to setup the Ansible Dynamic Inventory, so that it can able to fetch all EC2 Instances and then can able to configure those instances.

Dynamic inventory is an ansible plugin that makes an API call to AWS to get the instance information in the run time. It gives you the ec2 instance details dynamically to manage the AWS infrastructure.

To setup a dynamic inventory in ansible the most common way is to use the pre-created EC2 python files. But instead of this, we will use an ansible dynamic inventory plugin that is aws_ec2.

So aws_ec2 plugin is a great way to manage AWS EC2 Linux instances without having to maintain a standard local inventory. This will allow for easier Linux automation, configuration management, and infrastructure as code of AWS EC2 instances.

In order to use Ansible for AWS, we have to install boto, boto3, and botocore Python libraries

pip3 install boto boto3

To enable the aws_ec2 plugin we have to add the following statement to the ansible.cfg file:

enable_plugins = aws_ec2

So our ansible.cfg file looks like this

ansible.cfg file example

In the above image, you can see that my inventory path is ansible_plugins, so this is a handy way to manage all ansible plugins like if we have multiple AWS accounts or different cloud accounts.

Now after this we have to create a configuration file for the aws_ec2 plugin, here I have created aws_ec2.yaml file as shown below

---

plugin: aws_ec2

aws_access_key: <YOUR-AWS-ACCESS-KEY-HERE>

aws_secret_key: <YOUR-AWS-SECRET-KEY-HERE> regions:

- ap-south-1 strict: False

keyed_groups:

- key: tags

prefix: tag

- key: placement.region

prefix: aws_region

This is a simple configuration file and you can modify it according to your needs. So in the above file, notice keyed_groups keyword under which we basically write that how we want to group the instances like according to the tags, region, instance type, etc. And here I used tags to group my HDFS Cluster NameNode and DataNode instances.

Test the dynamic inventory by listing all the EC2 instances like this

ansible-inventory --list

Create Ansible Roles

In Ansible, the Role is basically a way to groups all the things we write in a playbook or it is the primary mechanism for breaking a playbook into multiple files. So Roles are the collection of variables, tasks, files, templates, and modules. And this simplifies writing complex playbooks, and it makes them easier to reuse.

In ansible to create a Role directory structure, we can use the ansible-galaxy command like this

ansible-galaxy init <role name>

Or we can also manually create this structure if we have limited groups.

Now in this, we will create 3 roles that are :

For launching EC2 Instances to setup HDFS Cluster

To configure an instance as a Hadoop Master (NameNode)

To configure instances as Hadoop Slaves (DataNodes)

Creating Role to Launch EC2 Instances

In this role, we will create a tasks file and a vars file.

Here is the tasks file which has all the tasks that ansible will perform

In the above file, you can see I have used variables and some user-defined variables too like number of instances and tags, which we will used to launch instances as namenode and datanodes with a single role. And notice that I used a module named meta which will automatically refresh our dynamic inventory as soon as we launch some instance.

Variable file for this role is

---

region: "ap-south-1"

ami_id: "ami-0a9d27a9f4f5c0efc"

instance_type: "t2.micro"

subnet_id: "subnet-2e8ee562"

sg_id: "sg-007e984dff1d14721"

key: "taskoskey"

The security group I attached to all the instances allow only SSH and HDFS services running on 9001 port.

Creating Roles to Configure HDFS Cluster

To configure a HDFS Cluster using ansible, you need to have some basic knowledge like how to setup a HDFS Cluster manually. But if you are new to Hadoop and don’t know how to setup HDFS then you can check out my this article which is a step-by-step guide to setup HDFS locally.

So Let’s see how to setup HDFS with Ansible

Creating Role for NameNode

In this role, we mainly have 3 directories i.e., tasks, templates, and vars.

Here is the tasks file which has all the tasks that we want Ansible to perform in order to configure NameNode and it typically performs these tasks

Install Hadoop and java

Configure instance as NameNode and start its services

Mount the EBS Volume with its NameNode Data directory

Now in the templates directory, there are 2 Hadoop files i.e., hdfs-site.xml and core-site.xml which ansible will copy in the NameNode’s /etc/hadoop directory and replace some variables with their values. And these files are also known as jinja templates. So the content in these files are

hdfs-site.xml file

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property>

<name>dfs.{{val}}.dir</name>

<value>{{hdfs_dir}}</value>

</property> </configuration>

core-site.xml file

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property>

<name>fs.default.name</name>

<value>hdfs://{{master_ip}}:{{hdfs_port}}</value>

</property> </configuration>

And here are the variables for this role

master_ip: "0.0.0.0"

val: "name"

hdfs_dir: "/master_hdfs"

hadoop_dir: "/etc/hadoop"

hdfs_port: 9001

Creating Role for DataNodes

The structure of the role to configure DataNode is same as the NameNode role and there are some minor changes in the tasks file and Variable file.

Since in Hadoop we do not format DataNodes, so we have to remove the task that does formatting and also we use a different command to start DataNode services, so we also have to remove this task. So typically we just have to remove 2 tasks from the NameNode tasks file and add the below task only and the other tasks remains the same.

- name: Starting Hadoop DataNode services

command: "hadoop-daemon.sh start datanode"

ignore_errors: true

And the Variable file for this role is

master_ip: "{{ groups.tag_Hadoop_namenode[0] }}"

val: "data"

hdfs_dir: "/slave_hdfs"

hadoop_dir: "/etc/hadoop"

hdfs_port: 9001

Now all the required Roles have been created and the Dynamic Inventory is also ready. And now we just have to create one final Ansible Playbook that will run all these roles and setup our HDFS Cluster on AWS.

And here is that Playbook

Now we just have to run this playbook like this

ansible-playbook setup.yml

Just this one command will do everything for us from launching EC2 instances to configuring them as NameNode and DataNodes.

Output of this command