Perfect Guide To Starting Your Journey Into Cloud Computing with AWS and Big Data
By: Adit Modi
Tags: AWS Lambda, Aws Kinesis, Cloud Computing, Big Data, AWS

cloud computing means storing and accessing data and programs over the Internet instead of your computer’s hard drive.

Big Data is a phrase used to mean a massive volume of both structured and unstructured data that is so large it is difficult to process using traditional database and software techniques. In most enterprise scenarios the volume of data is too big or it moves too fast or it exceeds current processing capacity.

Why use Amazon Web Services ?

How to Use Big Data for Cloud Computing using Amazon Web Services ?

There are lots of doubts regarding how does Big Data Applications like Hadoop work in AWS.

AWS provides lot of services that can work with big data applications.

My Background: Cloud and Big Data Enthusiast | 4x AWS Certified | 3x Oracle Cloud Certified | 3x Azure Certified | Big Data Certified.

Big Data Courses Done by me include

Udemy Course by Frank Kane.

Coursera Course By University of California San Diego.

IBM Courses.

People who are new to AWS , refer to my article on medium . This article will help you Understand AWS and it will help you develop a career in AWS. It Includes the path I used to get AWS Certified and It has ways of preparing for AWS Certified Exams.

For people who are familar with AWS but are new to Big Data , you can refer to my article on medium. This article explains Different Big Data Applications and It helps you understand Hadoop Platform and Applications Framework.

What are the AWS services that use Big Data ?

Amazon Athena : Interactive analytics

Amazon EMR : Big data processing

Amazon Redshift : Data warehousing

Amazon Kinesis : Real-time analytics

Amazon Elasticsearch Service : Operational analytics

Amazon Quicksight : Dashboards and visualizations

Thus there are a lot of AWS Services that use Big Data and Analytics in general.

We will discuss an application of AWS that uses Big Data. This Application is among the most popular use-cases of Big Data on AWS.

Analyzing-Twitter-in-real-time-with-Kinesis-Lambda-Comprehend-and-ElasticSearch

We are going to set a system to evaluate in real time the sentiment of all Tweets made with a specific Twitter hashtag. You can use it to evaluate your next smartphone based on other people opinion or you can know what people say about your company and products.

Let’s take a look to the architecture and pieces to use:

We are going to use a basic Python script to obtain real time Tweets thanks to the Twitter API, from the script we’ll put the Tweets directly in a Kinesis Firehose delivery stream where we have a transformation Lambda function, in that moment we are going to obtain the sentiment information using Amazon Comprehend and obtain a clean Twitter comment, finally the Tweet and its sentiment data will be stored in an Elasticsearch domain where we can see real time information using custom charts.

Ready?! Lets see and configure each piece:

Twitter API

First we’ll need credentials to access the Twitter API so if you don’t have them this is where you can start: https://apps.twitter.com/

Amazon Elasticsearch

Amazon Elasticsearch Service, is a fully managed service that makes it easy for you to deploy, secure, operate, and scale Elasticsearch to search, analyze, and visualize data in real-time.

Lambda Function

Next it’s time to create the Lambda function responsible of data transformation in the delivery stream and add the sentiment information using Amazon Comprehend.

Amazon Kinesis Firehose

Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data stores and analytics tools.

The Entire Source Code is Available on Github.

Twitter API

First we’ll need credentials to access the Twitter API so if you don’t have them this is where you can start: https://apps.twitter.com/

Amazon Elasticsearch

Amazon Elasticsearch Service, is a fully managed service that makes it easy for you to deploy, secure, operate, and scale Elasticsearch to search, analyze, and visualize data in real-time.

Lets create a new domain:

First lets set a domain name and an Elasticsearch version:

This time we’ll choose a small instance with not too much storage, don’t need more for this demo:

Just for accelerating things and make it easier we are setting a public access:

Remember!! don’t try this at home!

And we are done! After a few minutes the domain will be created and we will have an Elasticsearch endpoint with a Kibana URL for access the information stored.

Lambda Function

Next it’s time to create the Lambda function responsible of data transformation in the delivery stream and add the sentiment information using Amazon Comprehend.

Before continue to the next step lets take a closer look to the Lambda function and whats happening there.

First, we get the Tweet’s text from each record decoding the information sent from the Python script (we are just sending the text and the Firehose stream is responsible of encoding it inside the parameter ‘data’):

dict_data = base64.b64decode(record['data']).decode('utf-8').strip() print(dict_data)

Then we have to invoke the Amazon Comprehend Service using the ‘detect_sentiment’ method. After that we’ll get the sentiment type (POSITIVE, NEGATIVE, NEUTRAL or MIXED) and the score based in the positive and negative confidence:

comprehend = boto3.client(service_name='comprehend', region_name='eu-west-1')

sentiment_all = comprehend.detect_sentiment(Text=dict_data, LanguageCode='en')

sentiment = sentiment_all['Sentiment']

print(sentiment)

positive = sentiment_all['SentimentScore']['Positive']

negative = sentiment_all['SentimentScore']['Negative']

total = positive - negative

print(total)

I wrote that score because most of the Tweets have a NEUTRAL sentiment, however they have an amount of positive and negative sentiment; with this score even if they are mostly a neutral Tweet we can identify that little amount of positive or negative inside them.

Finally we’ll create the data structure for the Tweet and sentiment information encoded inside the required data structure for Kinesis Firehose output:

data_record = {

'message': dict_data,

'sentiment': sentiment,

'total': total

}

print(data_record)

output_record = {

'recordId': record['recordId'],

'result': 'Ok',

'data': base64.b64encode(json.dumps(data_record).encode('utf-8')).decode('utf-8')

}

print(output_record)

Note: Remember to give the function an execution role with permissions to access Amazon Comprehend and CloudWatch logs.

Amazon Kinesis Firehose

Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data stores and analytics tools. It can capture, transform, and load streaming data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today.

The place where the previous steps come together is the Delivery Stream:

Write a name for the new stream:

And set Direct PUT as the source because we’ll use a custom script for it:

Now its time to choose the previously created Lambda function enabling the record transformation inside the stream:

And set Amazon Elasticsearch Service as destination giving the information of the domain previously created:

Just in case, let’s set a S3 bucket where failed records can be stored for future analyze:

And now we are ready to use all this recently created services with real world information coming from Twitter!

Python Script

You can find the script in the same link that the Lambda function:

https://github.com/AWS-Big-Data-Projects/Analyzing-Twitter-in-real-time-with-Kinesis-Lambda-Comprehend-and-ElasticSearch