Real Time Data Streaming From MySQL to Elasticsearch using Kafka
By: Srinivas Choudhury
Tags: Elasticsearch, Kibana, MySQL, Kafka, Kafka Connect

This article is about how to implement a pipeline to stream data from MySQL database to Elasticsearch in the form of JSON using Confluent Kafka, Kafka JDBC Connector and Kafka Connector for Elasticsearch. So you should be familiar with these tools/applications before you implement such model.

Model/Pipeline Overview

We are going to implement the model in two phases.

1. Pulling data from MySQL to Kafka Topic using Kafka JDBC Connector.

2. Pushing data from Kafka Topic to Elasticsearch using Kafka Elasticsearch Connector.

Phase 1 can also be achieved using Debezium Kafka Connector. However, this connector uses Change Data Capture(CDC) and it has to be enabled from source(MySQL) by Database Administrator(DBA). This approach might degrade the performance of databases. However, it doesn’t matter if the volume of data is not huge. Another side, if we are using JDBC approach, our source table must have either a primary key or a timestamp column in order to ingest data incremental basis.

Now let’s implement the phase 1. It is believed that MySQL, Confluent Kafka and Elasticsearch are already installed on your machine or setup with docker.

Let’s create a table Student in MySQL database under a test database testdb with following code where ID is the primary key.

create database testdb;

USE testdb;

Create Table Student

(

ID int primary key,

Name varchar(255),

Subject varchar(255),

Mark varchar(255)

)

Now insert 2–3 records/rows to the newly created table just to verify whether our table is ready for streaming or not ;)

Insert into Student values (101, ‘Bhavik’, ‘Science’, 73);

Insert into Student values (102, ‘Drashti’, ‘Physics’, 89);

Select * from Student;

Now if everything what you have done so far is correct, then you probably would be seeing a screen like this.

MySQL Table Implementation

Now MySQL part is done. Let’s move to Kafka part. If you haven’t downloaded Confluent version Kafka, you can download it from here.

I am using Mac OS. If you are using Ubuntu or other OS, then commands might be little different for your case. Let’s start our confluent Kafka with start command. If your Confluent started successfully, you would see a screen like below.

Confluent Start Page

I believe you have already installed the Kafka JDBC Connector. If not, you simply run this confluent hub command to install it.

confluent-hub install confluentinc/kafka-connect-jdbc:latest

For JDBC connection establishment, we need to put database specific JDBC driver in the plugin path of Kafka node. As we are using MySQL databases here, I have downloaded mysql-connector-java-8.0.18.jar and placed at CONFLUENT_HOME/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib

Now the next part is very important i.e. configuration of the connector. I have created a configuration file in JSON format i.e. MySQL_Config.json. Here we have to put details of both MySQL and Kafka Topic. You can refer to my file as shown below.

MySQL_Config.json

Most people make typos in the above file. So I request you to pay special attention while configuring your connector. I have kept my MySQL_Config.json under CONFLUENT_HOME/bin so that I can easily run the command from here only. Now please run the following command from your bin folder.

Running Kafka JDBC Connector

After this, you would probably get screen like below.

Connector Start Screen

Just to confirm whether our connector is running or not, try the below command as shown.

Connector Running Status

So our connector is running perfectly. Now let’s check the list of topics in Kafka and verify our topic is created or not. Just check the below image.

Kafka List Topic

You can also check the same using any web browser by hitting http://localhost:9021/.

Kafka List Topic(UI)

So everything is good so far and our data is getting streamed from MySQL to Kafka topic. To check our MySQL data in Kafka, you can run kafka-producer command for that topic from beginning as shown below.

Kafka Topic Data

Now let’s insert a row to MySQL Student table to verify the real time streaming capability. Let’s fire some SQL queries.

Insert into Student values (103, ‘Nishant’, ‘Chemistry’, 62);

Select * from Student;

So by now, our tables looks like below.

Updated Student Table in MySQL

As soon as you insert a record to your MySQL table, the same(newly inserted) record is getting streamed to Kafka. Your previous Kafka Topic Data or Kafka-producer screen is now showing the new record in JSON format. You don’t need to type anything, just check the terminal only.

Kafka Topic Data(Updated)

Cool…You have successfully implemented the Phase 1 of the whole model and we have completed almost 70% of implementation. Now let’s go for Phase 2(Stream the Kafka topic to Elasticsearch).

The primary task you need to perform is to install the Kafka Elasticsearch Connector to your Confluent Hub Components. You can try the below command from your terminal.

confluent-hub install confluentinc/kafka-connect-elasticsearch:latest

Let’s create a Configuration file for this connector similar MySQL_Config.json.

Here is my ElasticSearch_Config.json. Again, please pay a special attention to this as well. You need to put details Elasticsearch as per your configuration. In my case, Elasticsearch is running on port number 32769.

Elasticsearch_Config.json

Now let’s run the Kafka Elasticsearch connector using the config file just you had created. Please refer the attached image below.

Running Elasticsearch Connector

After this, if you want to check the status of the connector whether it is running or not, fire the status query for the connector as shown below.

I am using Kibana Dashboard to work on Elasticsearch. I believe you are familiar with this as well. Let’s click the developer mode and fire the query for the topic we have mentioned in the config file. Your screen probably should look like this.

Kibana Dashboards Screen

If you observe the right side panel, you will get to see your MySQL Student table data or Kafka_Connect topic data is present in JSON format. Now let’s insert a to MySQL table with the following sql query and re-run the GET command in Kibana.

Insert into Student values (104, ‘Pratiksha’, ‘Mathematics’, 91);

Kibana Dashboard Updated

As soon as you insert the record in MySQL, that record data got streamed to Elasticsearch which is visible in Kibana as shown above.

BINGO!, With this you have successfully implemented the pipeline/model i.e. Real Time Data Streaming from MySQL to Elasticsearch using Kafka. Mostly, I have attached the images of my screen here. I wanted to attach the codes and config files as well. But I want you to code yourself and create your own config files. CTRL+C and CTRL+V are not gonna help you for learn.

I wish you implement this pipeline end to end without any ERROR or issues. If you face such, please go through the code and config files where you have put details. Possible issues(typos) could be present there only. All The Best :) !!!