Configure Hadoop and start the cluster services using the Ansible
By: Sachin Kumar
Tags: Cluster, Hadoop, Ansible, AWS, Automation

Configure Hadoop and start the cluster services using the Ansible

Let’s see how we can configure the Hadoop cluster and start through Ansible

In this article, you will see the configuration of the Hadoop cluster but you have some idea about Hadoop and Ansible so I am also writing some introduction part of them. So that you have some idea why we are using ansible here to configure the Hadoop cluster and how Ansible will help in the configuration. So let’s start with an introduction to both of them.

Redhat Ansible

Ansible is an open-source automation tool, or platform, used for IT tasks such as configuration management, application deployment, intra-service orchestration, and provisioning. Automation is crucial these days, with IT environments that are too complex and often need to scale too quickly for system administrators and developers to keep up if they had to do everything manually. Automation simplifies complex tasks, not just making developers’ jobs more manageable but allowing them to focus attention on other tasks that add value to an organization. In other words, it frees up time and increases efficiency. And Ansible, as noted above, is rapidly rising to the top in the world of automation tools. Let’s look at some of the reasons for Ansible’s popularity.

Hadoop Cluster

Hadoop is an Apache open-source framework written in java that allows distributed processing of large datasets across clusters of computers using simple programming models. The Hadoop framework application works in an environment that provides distributed storage and computation across clusters of computers. Hadoop is designed to scale up from a single server to thousands of machines, each offering local computation and storage.

Hadoop Architecture

Hopefully, now you have a little bit of an idea why we are using Ansible to configure the Hadoop cluster. If you are using Ansible to configure anything then you have a clear manual approach that will help you to understand the Automation. So if you are not comfortable with the installation part of the Hadoop cluster then you can read the below-mentioned article where you will get a manual approach to it. Even you will see here lots of basic and advanced of the Hadoop.

Now I hope you have a clear manual approach to the configuration of the Hadoop cluster. So you can go forward to set up your configuration with an automation approach. But for whom your Ansible setup should be ready and I am skipping here how we can configure the Ansible setup. If you are not comfortable with the installation of Ansible then you can see the below-mentioned article.

If your ansible setup has configured then you can go forward So I hope everything is ready for use. if in case you are facing any problem yet then you can ping me on my LinkedIn account which is mentioned at the bottom of this article. So I am starting the main part of this article. Hopefully, you will enjoy it.

Hadoop with Ansible

If you have read the Hadoop Article then you have a good idea of the Hadoop manual approach. But let’s assume you are working for a company that needs many Hadoop clusters then you would like to go with a manual approach? Of course not, you will like to go with an automation approach. So here Ansible plays a vital role to automate the Hadoop Cluster. So let’s see how to configure the Hadoop Cluster and start its services. But You have an idea about variables and playbooks, etc in the Ansible.

Ansible Playbook

Playbooks are the files where the Ansible code is written. Playbooks are written in YAML format. YAML stands for Yet Another Markup Language. Playbooks are one of the core features of Ansible and tell Ansible what to execute. They are like a to-do list for Ansible that contains a list of tasks.

Playbooks contain the steps which the user wants to execute on a particular machine. Playbooks are run sequentially. Playbooks are the building blocks for all the use cases of Ansible.

Ansible Variables

Ansible uses variables to manage differences between systems. With Ansible, you can execute tasks and playbooks on multiple different systems with a single command. To represent the variations among those different systems, you can create variables with standard YAML syntax, including lists and dictionaries. You can define these variables in your playbooks, in your inventory, in re-usable files or roles, or at the command line. You can also create variables during a playbook run by registering the return value or values of a task as a new variable.

After you create variables, either by defining them in a file, passing them at the command line, or registering the return value or values of a task as a new variable, you can use those variables in module arguments, in conditional “when” statements, in templates, and in loops. The ansible-examples github repository contains many examples of using variables in Ansible.

Ansible Modules

Modules (also referred to as “task plugins” or “library plugins”) are discrete units of code that can be used from the command line or in a playbook task. Ansible executes each module, usually on the remote managed node, and collects return values. In Ansible 2.10 and later, most modules are hosted in collections.

Hopefully, now you have an idea about Ansible variables, modules, and playbooks.

Note: I will attach the link to the whole demo of the Hadoop Cluster configuration at the bottom of the Article. So when you stuck somewhere then you can see it.

Namenode Configuration

In the section, I am writing the whole playbook for Namenode to configure. So I divided the whole configuration of Namenode into two playbooks, one is for variables second contains the main code.

Note: If you are going to follow the below playbooks then make sure the controller node should contain Hadoop and JDK softwares

Namenode Variables

Make a workspace for Hadoop configuration on your controller node and create a variable playbook to Namenode then write the above code inside it.

Now the playbook of the Namenode is given below with respect to variables.

Namenode Playbook

This playbook is contained all the steps which are necessary to configure the Namenode. So, create a playbook for it and write the below code inside it.

Running Namenode Playbook

After completing the above playbooks, run the namenode.yml playbook using the below command.

ansible-playbook namenode.yml

After configuring the Namenode let’s jump on the configuration of Datanode.

Datanode Configuration

Same I am dividing the entire configuration of Datanode into two playbooks. One will contain all the variables and another one is for the main configuration. So let’s see how to configure it.

Datanode Variables

Create a playbook for the Datanode’s variables and write the below code inside it.

Now the playbook of the Datanode is given below with respect to variables.

Datanode Playbook

This playbook is contained all the step which are necessary to configure the Datanode. So, create a playbook for it and write the below code inside it.

Running Datanode Playbook

Run the datanode.yml playbook using the below command to configure the Datanode.

ansible-playbook datanode.yml

After configuration the Datanode, now you can configure your client node to use or upload the data on the Hadoop cluster.

Client Node

I am also dividing the complete configuration of the client node into two playbooks, one is for variable and the second is for the main steps of configuration

Client Node Variables

This playbook is containing all the variables which will use in the client.yml playbook. So, create a playbook for variable and write the below code inside it.

Now the playbook of the Client node is given below with respect to variables.

Datanode Playbook

This playbook is contained all the step which are necessary to configure the Client node. So, create a playbook for it and write the below code inside it.

Running Client Node Playbook

Run the client.yml playbook using the below command to configure the Client node.

ansible-playbook client.yml

Hadoop cluster has been configured so let’s check through WebUI.

Conclusion

Hopefully, you learn something new from the article as well as enjoy it. But there is any doubt left then I also attached the Github repo which contains the whole code of it as well as the Video Demo link that might be helpful for you.

I tried to explain as much as possible. Hope You learned Something from here. Feel free to check out my LinkedIn profile mentioned below and obviously feel free to comment. I write Cloud Computing, Machine Learning, Bigdata, DevOps and Web, etc. blogs so feel free to follow me on Medium.

Thanks, Everyone for reading. That’s all… Signing Off… 😊