Create a Hadoop Cluster by Ansible !!
By: Lalit Yadav
Tags: Ansible, Hadoop, Vimal Daga, Automation, Righteducation

After reading this Article, you also will be able to create a Hadoop cluster by using Ansible. Now lets see how to integrate these amazing technologies.

Ansible:

Ansible is an open-source automation tool, or platform, used for configuration management, application deployment, intra service setup, and provisioning. Automation is the need of today, with IT environments that are too complex and often need to scale too quickly for system administrators and developers to keep up if they had to do everything manually. Automation simplifies complex tasks. In other words, it frees up time and increases efficiency. And Ansible, as noted above, is rapidly rising to the top in the world of automation tools.

It uses no agents and no additional software, so it’s easy to deploy .

It uses a very simple language (YAML, in the form of Ansible Playbooks) that allow you to describe your automation jobs (e.g. make and edit files ,transfer file and folder, install packages, start and stop service)

How ansible works?

There is a controller node(a system having ansible software) and others are the managed nodes(all the systems which we want to automate).In controller node ,we write a script or code called playbook(written in YAML format). Inventory file is the file in which we give the IP’s of the managed nodes which we want to automate.

STEPS TO PERFORM THE TASK:-

If you want to launch your cluster on AWS, plz follow step 1 also and if not you can skip this step and just play the ansible role.

Launch Instances:-

I am creating my Hadoop Cluster on the top of AWS ec2 instance. So first launch instances and should be pingable. During launching of datanodes it will ask for no of slave nodes also.

- hosts: localhost

vars:

tag_name: datanode

vars_prompt:

— name: num

prompt: “Enter the no of slave nodes”

private: no

To know how to launch instances by using ansible , plz go through my previous article :

Running instances.

2. Download and Install the Hadoop & JDK software:-

I have uploaded jdk software in S3 bucket and using the object URL, it will download by get_url module.



# tasks file for hadoop

- name: downloading the file

get_url:

url: “{{ item }}”

dest: /home/ec2-user/

loop:

— https://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/hadoop-1.2.1-1.x86_64.rpm

— https://lalitbucket6033.s3.ap-south-1.amazonaws.com/jdk-8u171-linux-x64.rpm — -# tasks file for hadoop- name: downloading the fileget_url:url: “{{ item }}”dest: /home/ec2-user/loop: - name: installing softwares

command:

cmd: "{{ item }}"

args:

warn: no

loop:

- rpm -ivh /home/ec2-user/jdk-8u171-linux-x64.rpm

- rpm -ivh /home/ec2-user/hadoop-1.2.1-1.x86_64.rpm --force

3. Copy the files.

Copy the core-site.xml and hdfs-site.xml files to the target node. It automatically copy respective files to both namenode as well as datanode.

- name: copying the core-site.xml

template:

src: /etc/ansible/roles/hadoop/templates/{{node_name}}/{{item}}

dest: /etc/hadoop/{{item}}

loop:

— core-site.xml

— hdfs-site.xml

4. Create the folder.

- name: creating directory

file:

path: “{{ direc_name }}”

state: directory

5. Format the namenode.

Here, I am using when conditions to format the namenode only.

- name: formatting namenode

shell: “echo Y|hadoop namenode -format”

args:

warn: no

register: x

when:

node_name == “namenode”

- name: debug

debug:

var: x

It will run the formatting block only when the node_name is namenode. In case of datanode it will skip the block.

6. Start the Service.

Start the namenode and datanodes and by using “jps” commad , it will also print the output by using debug.

- name: starting the services

shell: hadoop-daemon.sh start “{{ node_name }}”

args:

warn: no

register: result1

- debug:

var: result1 - name: check the status

command: jps

register: result

- debug:

var: result

7. Check the Report of Cluster.

To check how many datanodes are present in the cluster. Use ‘hadoop dfsadmin -report ’ command to check the report and then debug the result.

- name: check the status

shell: “hadoop dfsadmin -report”

register: report

when:

node_name == “datanode”

- name: debug

debug:

var: report

Snapshots of the playbook O/P

I have created a role for this task. You have to only run the ‘tests/test.yml’ playbook of the role.

In the last Fig. you can see the 2 datanodes are available. Hence by using Ansible we can setup a Cluster automatically.

Link for Github URL: https://github.com/yadav12122000/ansible/tree/main/hadoop

✨✨✨✨✨✨✨✨✨✨✨✨✨✨

For any query, feel free to contact me on Linkedin.

https://www.linkedin.com/in/lalit-yadav-4018b61b2/

Hope you found the article useful and informative!!👍