AWS Streaming Data Solution for Amazon Kinesis
By: Adit Modi
Tags: AWS, Streaming, Big Data

[Article 6 of 6 in series Big Data Applications on AWS]

Previous Article in the Series

The AWS Streaming Data Solution for Amazon Kinesis automatically configures the AWS services necessary to easily capture, store, process, and deliver streaming data. This solution helps you solve real-time streaming use cases, for example, capturing high volume application logs, analyzing clickstream data, continuously delivering to a data lake, and more. To provide flexibility to adapt to different use cases and business needs, this solution offers four AWS CloudFormation templates. Each template enables you to combine different components and extend them to meet your needs. You can use this solution to test new service combinations as the basis for your production environment, or to improve existing applications.

This solution shortens your development time by removing or reducing the need for you to: model and provision resources using AWS CloudFormation; set up Amazon CloudWatch alarms, dashboards, and logging; and manually implement streaming data best practices in AWS. This solution is data and logic agnostic, enabling you to start with boilerplate code and quickly start customizing. After deployment, you can use the solution’s monitoring capabilities to easily transition to production.

The AWS Streaming Data Solution for Amazon Kinesis uses templates where data flows through producers, streaming storage, consumers, and destinations. Producers continuously generate data and send it to streaming storage where it is durably captured and made available for processing by a data consumer. Data consumers process the data and then send it to a destination.

My Background: Cloud and Big Data Enthusiast | 5x AWS Certified | 3x Oracle Cloud Certified | 3x Azure Certified | Big Data Certified | AWS Community Builder.

AWS Streaming Data Solution for Amazon Kinesis solution architecture

AWS CloudFormation template using Amazon API Gateway, Kinesis Data Streams, and AWS Lambda

This AWS CloudFormation template deploys a reference architecture that includes the following:

An Amazon API Gateway ingests streaming data and processes that data with an AWS Lambda function.

An Amazon API Gateway REST API acts as a proxy to Amazon Kinesis Data Streams, adding either an individual data record or a list of data records.

Stores records in Kinesis Data Streams. An AWS Lambda function processes the records from the data stream.

Errors and failed records that occur during AWS Lambda processing are annotated, and the events are stored in Amazon Simple Queue Service (Amazon SQS).

An Amazon Simple Queue Service queue stores metadata for failed batch records and Lambda errors, enabling customers to retrieve these records and determine the next steps to resolve them.

Figure 1: AWS CloudFormation template using Amazon API Gateway, Kinesis Data Streams, and AWS Lambda reference architecture

This solution captures data from non-AWS environments (for example, mobile clients). This option uses an Amazon API Gateway as a layer of abstraction, which enables you to implement custom authentication approaches for data producers, control quotas for specific producers, and change the target Kinesis stream. This template uses AWS Lambda as the data consumer, which is best suited for use cases that don’t require internal state like filtering, business event processing, and data cleansing. AWS Lambda offers a small surface area for error scenarios and is simple to scale and operate.

Cost estimate for running the solution using the CloudFormation template that deploys API Gateway, Kinesis Data Streams, and AWS Lambda

Steps:

Before you launch this template, review the architecture and other considerations discussed in this guide. Follow the step-by-step instructions in this section to configure and deploy the solution into your account.

Time to deploy: Approximately five minutes

Launch the Stack

Sign in to the AWS Management Console and use the button below to launch the aws-streaming-data-solution-for-kinesis-using-api-gateway-and-lambda AWS CloudFormation template.

Alternatively, you can download the template as a starting point for your own implementation. The template launches in the US East (N. Virginia) Region by default. To launch this solution in a different AWS Region, use the Region selector in the console navigation bar. On the Create stack page, verify that the correct template URL shows in the Amazon S3 URL text box and choose Next. On the Specify stack details page, assign a name to your solution stack. For information about naming character limitations, refer to IAM and STS Limits in the AWS Identity and Access Management User Guide. Under Parameters, review the parameters for the template and modify them as necessary. This solution uses the following default values. Choose Next. On the Configure stack options page, choose Next. On the Review page, review and confirm the settings. Check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources. Choose Create stack to deploy the stack. You can view the status of the stack in the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in approximately five minutes.

You can customize or replace the demo application that is included with this solution can be customized or replaced to meet your business needs. The source code is available from the solution’s GitHub repository. For information about the demo producer application and customizing the demo application or replacing it with your own application, refer to the README.md file in the GitHub repository.

Demo consumer application

A consumer is an application that processes data from a Kinesis data stream. The aws-streaming-data-solution-for-kinesis-using-api-gateway-and-lambda template includes a demo consumer application, which is a Node.js function that logs the data being published. The customizable source code is available from the solution’s GitHub repository and can be customized to your business needs.

CloudWatch dashboards and alerts

This template deploys an Amazon CloudWatch dashboard that monitors the health of the data stream. You can customize the dashboards and alerts using Amazon CloudWatch or the source code from the solution’s GitHub repository.

Figure 2: Kinesis Data Streams metrics on the CloudWatch dashboard

Figure 3: AWS Lambda metrics on the CloudWatch dashboard

You can uninstall the AWS Streaming Data Solution for Amazon Kinesis using the AWS Management Console or the AWS Command Line Interface (AWS CLI). However, the Amazon Simple Storage Service (Amazon S3) bucket and Amazon CloudWatch Logs created by this solution must be manually deleted.

Using the AWS Management Console

Sign in to the AWS CloudFormation console. On the Stacks page, select the solution stack. Choose Delete.

Using AWS Command Line Interface

Determine whether AWS Command Line Interface (AWS CLI) is available in your environment. For installation instructions, see What Is the AWS Command Line Interface in the AWS CLI User Guide. After confirming the AWS CLI is available, run the following command.

$ aws cloudformation delete-stack --stack-name <cloudformation-stack-name>

Replace <cloudformation-stack-name> with the name of your CloudFormation stack.

Deleting the Amazon S3 buckets

This solution is configured to retain the Amazon S3 buckets if you choose to delete the AWS CloudFormation stack to prevent against accidental data loss. After uninstalling the solution, you can manually delete the S3 buckets if you do not need to retain the data. Use the following procedure to delete the Amazon S3 buckets.

Sign in to the Amazon S3 console. Choose Buckets from the left navigation pane. Locate the <stack-name> S3 buckets. Select one of the S3 buckets and choose Delete.

Repeat the steps until you have deleted all the <stack-name> S3 buckets.

Alternatively, you can configure the AWS CloudFormation template to delete the Amazon S3 buckets automatically. Before deleting the stack, change the deletion behavior in the AWS CloudFormation DeletionPolicy attribute.

Deleting the CloudWatch Logs

This solution retains the CloudWatch Logs if you decide to delete the AWS CloudFormation stack to prevent against accidental data loss. After uninstalling the solution, you can manually delete the logs if you do not need to retain the data. Use the following procedure to delete the CloudWatch Logs.

Sign in to the Amazon CloudWatch console. Choose Log Groups from the left navigation pane. Locate the log groups created by the solution. Select one of the log groups. Choose Actions and then choose Delete.

Repeat the steps until you have deleted all the solution log groups.

hope all of you Understood AWS Architecture and How to Use Big Data with AWS. If you have any doubt regarding the examples provided or are unable to understand how AWS Services make use of Big Data Applications you can contact me.

LinkedIn : https://www.linkedin.com/in/adit-modi-2a4362191/

Github : https://github.com/AditModi

You can view my badges on:

https://www.youracclaim.com/users/adit-modi/badges

Hope you enjoyed this AWS Big Data Series !! Do Clap this Article and Series if you like it.