Deploy a Kafka Cluster with Terraform and Ansible | by Marcos Lombog | Towards Data Science

In my previous post — Install a Kafka Cluster on Ubuntu in AWS, I have detailed the steps to manually set up your messaging backbone in AWS. This is a great step to understand how Kafka works, but it takes a long time and is not scalable.
A Kafka Cluster has several components such as Brokers, Zookeepers, Workers running on multiple instances like the example below. The simplest way to provision, deploy and manage your platform is by using tools that will automate this task.
On this post, I will explain how can we use Terraform to quickly provision multiple instances and then install Kafka on these instances using Ansible to set up your Cluster.
First, let’s quickly understand what is Terraform and Ansible in a high-level view
Terraform is an open-source infrastructure as code software tool created by HashiCorp. It enables users to define and provision infrastructure using a high-level configuration language known as Hashicorp Configuration Language (HCL), or optionally JSON.
Terraform supports a number of cloud infrastructure providers such as Amazon Web Services, Google Cloud Platform, Microsoft Azure.
Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.
Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.
Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications — automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.
First, let’s install Terraform and Ansible
Terraform
You can follow the instructions below or if you are on Mac, the simplest way is using brew:
brew install terraform
learn.hashicorp.com
Confirm Terraform is correctly installed with:
terraform -v
Ansible
Install Ansible using pip command. PIP is a package manager for Python packages. macOS doesn’t have pip installed by default. You have to install it first.
sudo easy_install pip
Use pip to install Ansible
sudo pip install ansible
Let’s verify Ansible is installed
ansible --version
For other OS, please use the reference below:
docs.ansible.com
Congratulations! Setup completed!
Now, let’s build the Kafka infrastructure using Terraform HCL.
The setup is based on the Enterprise Confluent Kafka using a public VPC and subnet.
All code is available in my GitHub
git clone git@github.com:mlomboglia/kafka-cluster-infra.git
Inside the terraform folder, there are the following files:
gateway.tf → Internet Gateway creation. There is no need to change this file
instances.tf → Instances to be created. There is no need to change this file
There are five types of instances with a short name in parenthesis:
Kafka Brokers (broker) → Host the Kafka broker service and maintain topic data
Zookeeper (zookeeper) →Host the quorum management service and manage topic metadata
Kafka Connect REST API (connect) → Kafka Connect, an open-source component of Kafka, is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.
REST Proxy (rest) → REST and HTTP/HTTPS access to additional Confluent Platform services
KSQL Server REST API (ksql) →KSQL is the streaming SQL engine that enables real-time data processing
Schema Registry REST API (schema) →The Schema Registry REST server uses content types for both requests and responses to indicate the serialization format of the data as well as the version of the API being used
Control Center (control_center) → Confluent Control Center is a web-based tool for managing and monitoring Apache Kafka®. Control Center facilitates building and monitoring production data pipelines and streaming applications.
public.tf → Subnet creation. For this example, I am using a public subnet
route.tf → Route table information
security_group.tf → Security group settings to allow for necessary network traffic into the instances. There is no need to change this file for the example.
Please keep in mind that ingress allows 0.0.0.0/0, so this is open to the public internet. You will need to restrict this to be used in a production environment.
vpc.tf → VPC creation
variables.tf contain the parameters for our terraform. You can modify this. For my example, I will provision 7 instances in AWS.
“rest” = 1 “connect” = 1 “ksql” = 1 “schema” = 1 “control_center” = 1 “broker” = 1 “zookeeper” = 1
With a prefix “staging”
Intance Type “t2.small” .
Important: Confluent recommends a minimum instance type “t2.large” to run Kafka, so if you modify this it will be out of AWS Free Tier and it will cost you money
Key Pair.
You will need an AWS Key Pair to create your instances. If you don’t have one yet, log in to your AWS account
EC2 -> Create Key Pair
Give a name and click Create.
Download your key, copy to a safe folder and update permissions
chmod 400 ~/.ssh/<Key Name>.pem
Update the section below of the variables.tf with your Key Name
variable "key_name" {  description = "Key Pair"  default     = "<Your Key Name>"}
Now, we are good to go.
Terraform init will download your dependencies
cd kafka-cluster-infraterraform init
Terraform plan will create your deployment plan
terraform plan
Now, be careful on the next step. It will cost you money depending on your configuration.
terraform apply
Confirm with “yes”
if there is an error: ami is not found, go to your EC2 console, launch instance and update with the latest ami label:
Now you have provisioned all the instances in AWS.
Go your AWS and see all your instances running and take note of the public DNS (IPv4) of each instance. You will need this in the next section.
Now, let’s proceed to install Kafka using Ansible
To install Kafka, I will use the Confluent version.
1. Create a copy of the hosts_example.yml file.
cd ..cd cp-ansiblecp hosts_example.yml hosts.yml
2. Use your favourite IDE and edit hosts.yml file entering the hosts from your AWS Terraform installation above following the labels for each instance type, looking similar to this with your Public DNS.
Be very careful to use correct indentation on YML files:
Uncomment the following line in the hosts.yml file adding your key path
ansible_ssh_private_key_file: '~/.ssh/<Key Path>.pem'
3. Confirm Ansible can connect over SSH
ansible -i hosts.yml all -m ping
Type “yes” to trust all servers.
If there is an issue connecting to an instance, try to SSH manually with the command below:
ssh -i <Key Path> ec2-user@<Instance Public DNS>
4. Run the all.yml playbook
ansible-playbook -i hosts.yml all.yml
The Ansible playbook will perform the Kafka installation on all your instances.
After some time, you should see this message above. Everything installed!
Now, let’s test the installation:
You can SSH to any of your instances to check the services are running using:
You can SSH to any of your instances to check the services are running using:
systemctl status confluent*
To continue you can build Your First Kafka Application with my other post:
towardsdatascience.com
Don’t forget to destroy your Cluster and terminate your instances running the command below: