Real-Time Sentiment Analysis For Twitter Data Using AWS
By: Adit Modi
Tags: AWS, Cloudformation, Sentiment Analysis, Machine Learning, Serverless

The solution enables you to gain insights into your customer’s conversations and deepen brand awareness by analyzing social media interactions.

This solution automatically provisions and configures the AWS services necessary to capture multi-language tweets in near real-time, translate them, and store both the raw and enriched datasets durably in the solution’s data lake. You can then analyze this data and create meaningful dashboards powered by Amazon QuickSight to visualize and understand customer sentiment.

My Background: Cloud and Big Data Enthusiast | 5x AWS Certified | 3x Oracle Cloud Certified | 3x Azure Certified | Big Data Certified | AWS Community Builder.

Implementation overview

AI-Driven Social Media Dashboard monitors and ingests specified tweets using stream processing and leverages a serverless architecture and ML services (Amazon Translate and Amazon Comprehend) to translate and extract insights from those tweets. The diagram below presents the architecture you can deploy in minutes using the solution’s implementation guide and accompanying AWS CloudFormation template.

Implementation architecture

This solution deploys an Amazon Elastic Compute Cloud (Amazon EC2) instance running in an Amazon Virtual Private Cloud (Amazon VPC) that ingests tweets from Twitter. An Amazon Kinesis Data Firehose delivery stream loads the streaming tweets into the raw prefix in the solution’s Amazon Simple Storage Service (Amazon S3) bucket. Amazon S3 invokes an AWS Lambda function to analyze the raw tweets using Amazon Translate to translate non-English tweets into English, and Amazon Comprehend to use natural-language-processing (NLP) to perform entity extraction and sentiment analysis.

A second Kinesis Data Firehose delivery stream loads the translated tweets and sentiment values into the sentiment prefix in the Amazon S3 bucket. A third delivery stream loads entities in the entities prefix using in the Amazon S3 bucket.

The solution also deploys a data lake that includes AWS Glue for data transformation, Amazon Athena for data analysis, and Amazon QuickSight for data visualization. AWS Glue Data Catalog contains a logical database which is used to organize the tables for the data on Amazon S3. Athena uses these table definitions to query the data stored on Amazon S3 and return the information to an Amazon QuickSight dashboard.

Prerequisites

Before you launch this solution, you must have a Twitter consumer key (API key) and secret, and a Twitter access key and secret. If you do not already have these keys, you must create an app in Twitter.

You must also have an Amazon Elastic Compute Cloud (Amazon EC2) key pair. If you do not already have a key pair, see Creating a Key Pair Using Amazon EC2 in the Amazon EC2 User Guide for Linux Instances.

Steps

The procedure for deploying this architecture on AWS consists of the following steps. For detailed instructions, follow the links for each step.

Step 1. Launch the Stack

This automated AWS CloudFormation template deploys the AI-Driven Social Media Dashboard on the AWS Cloud. Please make sure that you have completed the prerequisites before launching the stack.

You are responsible for the cost of the AWS services used while running this solution. See the Cost section for more details. For full details, see the pricing webpage for each AWS service you will be using in this solution.

Sign in to the AWS Management Console and click the button below to launch the ai-driven-social-media-dashboard AWS CloudFormation template.

You can also download the template as a starting point for your own implementation. The template is launched in the US East (N. Virginia) Region by default. To launch the solution in a different AWS Region, use the region selector in the console navigation bar.

This solution uses Amazon QuickSight, Amazon Athena, Amazon Translate, Amazon Comprehend and AWS Glue which are currently available in specific AWS Regions only. Therefore, you must launch this solution in an AWS Region where these services are available. For the most current service availability by region, see AWS service offerings by region. On the Select Template page, verify you selected the correct template and choose Next. On the Specify Details page, assign a name to your solution stack. Under Parameters, review the following parameters for the template and modify them, as necessary. This solution uses the following default values. Select Next. On the Options page, select Next. On the Review page, review and confirm the settings. Be sure to check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources resources. Select Create to deploy the stack.

You can view the status of the stack in the AWS CloudFormation console in the Status column. You should see a status of CREATE_COMPLETE in approximately five minutes.

Step 2. Build the Queries

Navigate to the Amazon Athena console. In the navigation pane under Database, select ai_driven_social_media_dashboard . Use the Athena Query Editor query pane to run queries on the data. The following table contains a list of queries and their descriptions

FROM tweets limit 10; Returns information for the last 10 tweets.

SELECT * FROM tweet_sentiments limit 10; Returns sentiment information for the last 10 tweets.

SELECT * FROM tweet_entities limit 10; Returns entity information for the last 10.

SELECT type, count(*) cnt FROM tweet_entities GROUP BY type ORDER BY cnt desc Returns a count of the number of entities for each entity type.

SELECT lang, count(*) cnt FROM tweets GROUP BY lang ORDER BY cnt desc Returns a count of the number of tweets in each language.

Step 3. Create the Data Source

Navigate to the Amazon QuickSight console. Select Manage data. Select New data set. Select Athena. For Data source name, enter a name (for example, tweet_sentiment ) and select Create data source.

After a few minutes, a success message should appear that shows that the Amazon Athena data imported into Amazon QuickSight. Select the ai_driven_social_media_dashboard database. When the list of available tables populates, select the tweet_sentiments table. Select Edit/Preview Data. Enter the following custom query to count the distinct tweet IDs:

SELECT s.*, e.entity, e.type, e.score, t.lang as language, coordinates.coordinates[1] AS lon, coordinates.coordinates[2] AS lat , place.name, place.country, t.timestamp_ms / 1000 AS timestamp_in_seconds, regexp_replace(source, '\<.+?\>', '') AS src FROM ai_driven_social_media_dashboard.tweets t JOIN ai_driven_social_media_dashboard.tweet_sentiments s ON (s.tweetid = t.id) JOIN ai_driven_social_media_dashboard.tweet_entities e ON (e.tweetid = t.id)

10. Select Finish.

You should be able to see the sampled data.

11. In the navigation pane, select Fields, timestamp_in_seconds, Change data type and select Date.

12. Select Save and Visualize.

Step 4. Build the Dashboard

On the All analyses tab of the Amazon QuickSight start page, choose New analysis. You are taken to the Your Data Sets page. Choose the data set and then choose Create analysis. Choose Add on the application bar, and then choose Add visual. In the Fields list pane, select the drop-down menu and select type and tweetid. If the Fields list isn’t visible, choose Visualize to display it.

The field wells display the fields that are visualized. Select Field wells to access the options. In the Fields list pane, drag the tweetids field to the Value field wells. Select tweetid, Aggregate, Count Distinct. For Visual types, select the pie chart icon.

Amazon QuickSight creates the visual. Select + Add to build the second visual for the dashboard. Resize the second visual so it matches the size of the first visual and position it next to the first visual.

11. In the Fields list pane, select the drop-down menu and select sentiment and timestamp_in_seconds.

12. In the Field wells, zoom either in or out of the time element until hours is shown.

Amazon QuickSight creates the second visual.

13. Optional: Open the context menu (right-click) for the neutral line and select Exclude NEUTRAL.

Do this when there are a disproportionate number of neutral tweets affecting the visual display of the data range of the other tweets.

14. Select the + Add option to build the third visual for the dashboard.

15. Position the third visual in the bottom half of the workspace.

16. For Visual types, select the table icon.

17. From the Fields list, select the following (to add the translated tweets to the visual):

language

text

originalText

18. On the toolbar, select Filter.

19. Select One: language.

20. Select Custom filter, Does not equal, and enter en .

Now you can view graphical representations of your tweet entities, tweet sentiment over time, and translated tweets.

You can create additional visuals to create a more robust visualization of your tweet data.

hope all of you Understood AWS Implementation Architecture and How to Use Sentiment Analysis with AWS. If you have any doubt regarding the examples provided or are unable to understand how AWS Services work and how to implement different AWS Solutions do reach out to me.

LinkedIn : https://www.linkedin.com/in/adit-modi-2a4362191/

Github : https://github.com/AditModi

You can view my badges on:

https://www.youracclaim.com/users/adit-modi/badges