Streaming Real-time data to AWS Elasticsearch using Kinesis Firehose
By: Janitha Tennakoon
Tags: Big Data, Kinesis, AWS, Elasticsearch, Data

Explore how we can deliver real-time data using data streams to Elasticsearch service using AWS Kinesis Firehose.

Elasticsearch is an open-source solution that is used by many companies around the world for analytics. By definition, Elasticsearch is an open-source, RESTful, distributed, indexed search, and analytics solution.

The first part of the definition is that it is an open-source solution, which means it is a community-driven solution and it is free to use widely. Next, it is RESTful, which means all the communication and configurations can be done through simple REST HTTP API calls. Elasticsearch has developed a feature-rich REST API framework for use by clients in order to consume from. Next distributed and indexed, now this is where Elasticsearch gets differentiated from a common search solution that we tend to implement on top of our existing databases. Elasticsearch is distributed, which means the functionality and data being stored are using multiple resources and it will use all of these resources to do these functionalities which makes them very efficient meanwhile providing high availability as well. Next, it is indexed, which improves significantly data retrieval. Elasticsearch uses Apache Lucene for indexing, which is known to be an extremely fast indexing solution that makes search or analytics extremely fast.

There are many other concepts that we need to talk about Elasticsearch. But since this article is going to address a solution for stream data into Elasticsearch and not on how to use Elasticsearch I will not try to go and write about those concepts. If you are a complete beginner at Elasticsearch I recommend you to first learn at least the basic concepts that are used in Elasticsearch before continuing through this article. There are many valuable posts, videos on the internet about Elasticsearch for beginners which I am sure you will be able to find. For this article, we are going to use the AWS Elasticsearch service because it is fully managed by AWS so we do not need to worry about deployments, security, and making a fail-proof high available solution.

AWS Kinesis

Amazon Kinesis is a service provided by AWS for processing data in real-time. In this scenario that we are going to talk about data that are coming to Kinesis as a stream where Kinesis will execute all kinds of functionalities based on our requirements. There are many other data stream processing solutions in the community as well. Apache Kafka, Apache Spark are some of the examples that are used currently in the community. The main reason that I chose Kinesis is that our Elasticseach solution will also be created as an AWS service, but also since all the servers, failures are fully managed by AWS themselves it is really easy us to more focus on the outputs rather than managing the infrastructure. I think before going to discuss about Kinesis let us first try to understand what are data streams and some examples for data streams.

Data Streams

Data streams are data that are generated and sent continuously from a data source. This data source is called as data producers in the streaming world. The main feature that distinguishes data streams with other form of data sources is that it is generated continuously, like a river. So rather than sending data in batches, in streaming, we expect that every time data will be given to us in a means of a stream. Data streams mostly surface with the Big Data world. We may this that at a given time frame if we inspect data it might be small, so why are we referring to Big Data for data streams. But here we are talking about continuously sending data every time, which means a large quantity of data at the end of the day. Below are some examples of data streams that we can find in the real world.

Log files generated by a software application

Financial stock market data

User interaction data from a web application

Device and sensor output from any kind of IoT devices

Thus what we do with these data streams is either we can do analytics in real-time and provide analytical data as the output or we can store these data and do analytics on top of the data. The first scenario is used in many solutions where real-time analytics are required like identifying traffic congestions, identifying different patterns, etc. For the second scenario what we do is we store these data in an analytical platform and later to analytics on top of that. Elasticsearch is such kind of analytical platform where we can perform analytical tasks. But how can we handle and send the data to Elasticsearch using the data stream that we are getting from data producers? thus comes the Kinesis into the use case of ours.

So as mentioned earlier Kinesis is a service that is used to process data streams. So what kind of solutions does AWS Kinesis provides to handle these kinds of data streams? for that currently, AWS Kinesis provides four types of solutions at the time I am writing this article.

Kinesis Data Streams — used to collect and process large streams of data records in real-time

— used to collect and process large streams of data records in real-time Kinesis Data Firehose — used to deliver real-time streaming data to destinations such as Amazon S3, Redshift, Elasticsearch, etc..

— used to deliver real-time streaming data to destinations such as Amazon S3, Redshift, Elasticsearch, etc.. Kineses Data Analytics — used to process and analyze streaming data using standard SQL

— used to process and analyze streaming data using standard SQL Kinesis Video Streams — used to fully manage services that use to stream live video from devices

So out of the four solutions provided, we can see that for our use case which s to load data into Elaslticsearch we will be going to use Kinesis Data Firehose.

Amazon Kinesis Data Firehose

Kinesis Data Firehose is one of the four solutions provided by AWS Kinesis service. So as Kinesis service, it is also fully managed by AWS which means we do not need to worry about deployments, availability, and security at every level. Kinesis Firehose is used to deliver real-time streaming data to pre-defined service destinations. At the writing of this article, AWS supports four destinations.

Amazon S3 — an easy to use object storage

Amazon Redshift — petabyte-scale data warehouse

Amazon Elasticsearch Service — open-source search and analytics engine

Splunk — operational intelligent tool for analyzing machine-generated data

As mentioned in the title itself in this we are going to look at how we can load data into Elasticsearch destination. Before implementing and designing or solution let us first look at some of the basic concepts of Kinesis Firehose.

Kinesis Data Firehose delivery stream — the main component of the firehose, which is the main delivery stream which will be sent to our destinations

the main component of the firehose, which is the main delivery stream which will be sent to our destinations Data producer — the entity which sends records of data to Kinesis Data Firehose. These will be the main source for our data streams

— the entity which sends records of data to Kinesis Data Firehose. These will be the main source for our data streams Record — the data that our data producer sends to the Kinesis Firehose delivery stream. In s data stream these records will be sent continuously. Usually, these records will be very small with the max value as 1000KB

— the data that our data producer sends to the Kinesis Firehose delivery stream. In s data stream these records will be sent continuously. Usually, these records will be very small with the max value as 1000KB Buffer size and buffer interval — the configurations which determine how much buffering is needed before delivering them to the destinations.

In order to process data stream data producers should continuously send data to our Firehose. Thus now the question should be how do these data producers send data to our Firehose. There are several ways for data producers to send data to our Firehose.

Kinesis Data Firehose PUT APIs — PutRecord() or PutRecordBatch() API to send source records to the delivery stream.

Amazon Kinesis Agent — Kinesis Agent is a stand-alone Java software application that offers an easy way to collect and send source records. In order to load data in the Kinesis Agent, this agent should be available in our data producer system.

AWS IoT — Create AWS IoT rules that send data from MQTT messages.

CloudWatch Logs — If we are going to use cloudwatch logs we can use subscription filters to deliver a real-time stream of log events.

CloudWatch Events — Create rules to indicate which events are of interest to your application and what automated action to take when a rule matches an event.

In our solution, we are going to use the demo data stream provided by Kinesis Firehose. In demo data stream data will be sent in the following format.

{"ticker_symbol":"QXZ", "sector":"HEALTHCARE", "change":-0.05, "price":84.51}

Since now we have a basic understanding about the components and services we are going to use let us now begin to design our solution. Our scenario is to deliver data stream which are going to be created by data producers to our analytical platform, which is Elasticsearch service. Before implementing the solution we might be asking these following questions based on our requirements.

Does the data need to be transformed before stored into Elasticsearch?

If so what kind of transformation is required?

Do we need to store raw data even after transformation for future purposes or as a backup?

Let’s assume for our scenario answers for all three questions above is Yes. Then we need functionalities to transform our data into a different format and also save the raw data into some location. Fortunately, Kinesis Firehose already provides these functionalities by using AWS Lambda functions.

According to the diagram above before sending data to the Elasticsearch service a Lambda function will transform our data according to our requirements. In the meantime, our raw data before transformation will be sent to an AWS S3 bucket. In there using lifecycle rules we can transfer them to either AWS Galcier or other S3 categories according to our requirements. Now since we have the design ready let’s go ahead and try to implement our designed solution in AWS.

Creating Elasticsearch service

We are going to create our Elasticsearch service only for development and learning purposes so the configurations that we will select for our domain will not be the most secure configurations and should not be followed for an enterprise solution.

First , Go to AWS Elasticservice service and create a new domain. Since we are using this domain for only learning select Deployment type as Development and testing and click next.

Next, provide a domain name and after that select the instance type as t2.small.elasticsearch because that is the only instance type that is available for the free tier if you are already using a free tier account. You can leave all the other options as default values without any change.

On the next screen we will be asked to configure security for our elasticsearch domain. The recommended is to use VPC access where our domain will be in a private network with only instances within our VPC network that will have access. But in our case make it as Public access.

Below on the same page you will see where we are setting access policy for our domain. Since this is only for learning add Allow open access to the domain which will make any IP address have access. (we are using this only for learning purpose) Here we can specify either an IAM role or specific AWS user accounts as well.

Now that is it for creating our testing elasticsearch domain. After a couple of minutes our elasticsearch domain will be up and running. So creating our Elasticsearch service is done now.

Creating Kinesis Data Firehose

Go to AWS Kinesis service and select Kinesis Firehose and create a delivery stream. In the next screen give a stream name and select the source as Direct PUT or other sources. The other option here is to select the source as a Kinesis Data Stream. In our scenario, we do not have a Kinesis data stream and we send the data to the Firehose directly from our producer.

We have the option to enable server-side encryption for data we stream. If we want to encryption for data enable it and provide the necessary encryption key. But for this article let’s keep server-side encryption as disabled.

Next we will be asked whether we need to transform the source records. Here enable data transformation. This will prompt us to give an Lambda function which will handle the data transformation. Since we have not created a Lambda function yet select Create new.

This will open a box where AWS will prompt us with already available lambda blueprints. Select the first option General Kinesis Data Firehose Processing for our use case since we are going to do a custom function.

This will guide us to the Lambda function creation page. Before creating the Lambda function first create an IAM role for our lambda function with permission for AWS Kinesis services.

Here we are providing full access to Firehose, but we only need permission for “firehose:PutRecordBatch” . But since this is only for learning purposes let’s give full access. Make sure to add AWSLambdaExecute permission as well which will give execution permission for our function as well as log permission to create logs to cloudwatch service.

After creating our role let’s again go to our Lambda function creation page. There give our function a name and select the IAM role we created earlier.

As mentioned earlier our data stream will have the following format.

{"ticker_symbol":"QXZ", "sector":"HEALTHCARE", "change":-0.05, "price":84.51}

So for this article's purpose let’s assume that we do not need change property to be stored in our Elasticsearch service. Also let’s rename the ticker_sysmbol property to ticker_id to be stored as in Elasticsearch. Below is the lambda function that I have created using node.js in order to fulfill our transformation. As you know you can use any supported language in Lambda to create your function.

exports.handler = async (event, context) => {

/* Process the list of records and transform them */

const output = event.records.map((record) => {

const payload =record.data;

const resultPayLoad = {

ticker_id : payload.ticker_symbol,

sector : payload.sector,

price : payload.price,

};



return{

recordId: record.recordId,

result: 'Ok',

data: (resultPayLoad),

} });

console.log(`Processing completed. Successful records ${output.length}.`);

return { records: output };

};

As mentioned above we are doing a simple transformation of renaming a property and removing a unwanted property. Go ahead and create this Lambda function with the above code.

Now go back to our Firehose creation page and select our created Lambda function.

This will give us a warning saying that our Lambda function timeout is only 3 seconds and increase it at least for 1 minute. This warning is given because we are dealing with streaming data it may take some time to execute and complete this function and the default 3 seconds timeout will not be enough. We can do that by going to our function’s basic settings. Lambda supports up to 5 minutes of timeout.

After that, we have also have an option to convert record format to either Apache Parquet format for Apache ORC format rather than using a JSON format. These converting is done using AWS Glue service by defining our schemas there. But in our scenario, we do not need that functionality.

Next, we will be asked for the destination of our data stream. Select Elasticsearch from there and select our domain we created earlier and with the index that we are going to store our data with.

Next we have the option of taking backups of our raw data. Here we can either select to backup only failed records or all records for future purposes. Here we need to select a S3 bucket, if you don’t have it already created we can create a new bucket using Create New. We can also append a prefix to the data stream that will be save on the S3 bucket so we can easily categorize our saved data inside the S3 bucket. Kinesis Data Firehose automatically appends the “YYYY/MM/DD/HH/” UTC prefix to delivered S3 files. Apart from that, we can add a custom prefix as well according to our requirements.

The next page will display several configurations that we can modify. First is Elasticsearch buffer conditions. Firehose will buffer data before sending them to store in Elasticsearch, we can determine this buffer using two metrics, buffer size and buffer interval. So when either of these conditions fulfilled Kinesis Firehose will deliver our data to Elasticsearch. Since we are going to use Lamda functions to transform our data stream we need to comply with the AWS Lambda function payload limit, which is 6 MB.

Next configuration is S3 compression and encryption. These configurations are related to the backup S3 bucket that we are going to use for raw data. Here we can either compress data in order to make them smaller or make them secure by encrypting data before strong in S3. Next Error loggings will be enabled by default which will log errors on Cloudwatch, lastly, we need to create a new IAM role for our Firehose service. For that, we can go ahead and create a new IAM role.

After that we have done all the configurations so go ahead and create our stream. It will take a couple of seconds for AWS to create our firehose stream. The status of our stream will become Active when the stream is fully created.

Now in order to test our system working according to our design we can go to our stream and Test with demo data.

This will send demo test data continuously to our Kinesis Firehose. We can first go to the Elasticsearch Kibana dashboard and verify that our data is loaded accordingly with the appropriate transformation we have done. Confirm that all the raw data is available on our S3 buckets as well.

Now we have done our configurations and made sure they are working using the demo stream. That is all that I was hoping to go through using this article, but there are many more other functionalities provided by both AWS Kinesis and Elasticsearch services so make sure to explore more on those services. Thank you for reading this article. :)