Twitter Sentiment Analysis Dashboard On Event Management
By: Rajendran S P
Tags: Dashboard, Python, Data Integration, NLP, Sentiment Analysis

In this blog post, we will see how to effectively use Syncfusion Dashboard and Data Integration Platforms in analyzing the sentiments of the attendees of a workshop through effective processing of live Twitter tweets.

Recently, Syncfusion had an opportunity to give a workshop on big data and dashboards on the cloud at the Microsoft campus in Chennai. It was an all-day event at which we encouraged the attendees to post tweets with two hashtags (#syncfusion, #dashboardcloud) with their honest feedback on the session and products on Twitter. Attendees from various organizations at varied levels came to this workshop. In it, we focused on providing introductory content for our Big Data and Data Integration Platforms and then showcased the data in the Syncfusion Dashboard Cloud. Finally, we showed them how effectively we can bring in data from Twitter in real time, process it in Spark, integrate it within SQL Server, and finally showcase it in the Dashboard Platform during the hands-on session.

Here, I am going to briefly explain the tweet processing, how it helped us gauge the sentiment of the attendees on the overall workshop, and the insights we gained as a feedback tool.

Steps involved:

Using Syncfusion Data Integration Platform:

Bring in real-time tweets with the hashtags #syncfusion and #dashboardcloud. Clean and process JSON data into required flat schema. Perform sentiment analysis over the tweeted text using Stanford CoreNLP. Move the final processed data along with sentiment score into a SQL database.

Using Syncfusion Dashboard:

5. Create a dashboard to showcase real-time Twitter sentiment analysis.

For steps 1 to 4, we will be defining a data-flow in the Data Integration Platform as shown in the following image.

Data flow in Data Integration Platform

Step 1: Use the Get twitter component to bring in real-time tweets with the hashtags #syncfusion and #dashboardcloud.

Ensure that you get the consumer key, consumer secret, access token, and access token secret from the Twitter developer site by referring to this guide before creating the dataflow. Refer to the following configurations, where you can provide your own hashtags under the Terms to filter on property.

Step 2: Data preparation — clean and process JSON data into required fields (attributes) using the processors — Evaluate Json Path and Update attribute.

Data preparation workflow

Evaluate Json Path is used to filter the fields like user details, tweeted text, created date, retweet details, language, friends count, followers count, favourites count, and location.

Note: These attribute and property names will be used as column names in the SQL table.

Property names as column in SQL table

In the update attribute processor,

We use a conditional expression to fetch the exact tweeted text based on the following conditions. We then add hashtags as an additional attribute. To learn more about tweets, refer to the Twitter documentation.

Also cleansed the data for extracting created date suitable for creating dashboard.

Step 3: Perform Sentiment analysis over the processed tweet using Stanford core NLP.

We use the Execute Stream command processor to run the Python script with Stanford CoreNLP service to process the tweeted text and evaluate its sentiment (mood and the score).

To configure the environment to run Python sentiment analysis script within DIP, follow these steps:

Install Stanford NLP package from this location. Start the server using the following command.

C:\<installed location>\stanford-corenlp-full-2018-10-05> java -mx5g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -timeout 10000

Note: Java in the command refers to JAVA_PATH, If the JAVA_PATH is set already just run the above command or else mention the JAVA_PATH in place of java used in the command. Install the “pycorenlp” python package using below command.

- pip install pycorenlp We use the following Python script to perform sentiment analysis. It is a very basic one that uses default settings in CoreNLP to gauge sentiment.

from pycorenlp import StanfordCoreNLP

import sys

import re

#handle the invalid characters in input data

text = re.sub('[^a-zA-Z0-9

\.]','',sys.argv[1])

res = nlp.annotate(text,

properties={

'annotators': 'sentiment',

'outputFormat': 'json',

'timeout': 1000,

}) nlp = StanfordCoreNLP(' http://localhost:9000' #handle the invalid characters in input datatext = re.sub('[^a-zA-Z0-9

\.]','',sys.argv[1])res = nlp.annotate(text,properties={'annotators': 'sentiment','outputFormat': 'json','timeout': 1000,}) for s in res["sentences"]:

print("'%s', %s , %s" % (" ".join([t["word"] for t in s["tokens"]]),

s["sentimentValue"],s["sentiment"]))

Save the previous code in a file named (sentiment.py) and provide this file location as command line arguments for the Execute Stream Command processor as depicted in the following screenshot.

Command Arguments: <python script file location> <tweeted text>

Command Path: Python exe (installed location)

Step 4: Move the processed data along with sentiment score into a SQL table.

We use following processors in this step:

Extract Text — Extract the sentiment results from the Python script into an attribute. Update Attribute — Update sentiment and sentiment score attributes from the sentiment results. Attributes to Json — Create JSON out of the attributes (fields) we want to track in the dashboard. ConvertJsontoSQL — Convert the JSON string into SQL insert statements. PutSQL — Execute the insert statements generated.

Data flow — Moving predicted output to SQL server

Note: Make sure you have created a SQL table (tweetssentiment) using the following query and create a controller service for it in the Data Integration Platform. For more details on controller settings, refer to our documentation.

CREATE TABLE [dbo].[tweetssentiment](

[tweetid] [bigint] NULL,

[userid] [bigint] NULL,

[username] [varchar](500) NULL,

[screenname] [varchar](500) NULL,

[tweetedtext] [varchar](500) Null,

[language] [varchar](500) NULL,

[location] [varchar](500) NULL,

[created_at] [varchar](500) NULL,

[hashtag] [varchar](500) NULL,

[retweetcount] [int] NULL,

[favouritecount] [int] NULL,

[friendscount] [int] NULL,

[followerscount] [int] NULL,

[sentiment] [varchar](500) NULL,

[sentimentscore] [varchar](500) NULL

) ON [PRIMARY]

GO

The data integration workflow can be scheduled in real time by setting it as “0 sec” so that it looks for input tweets constantly, or it can be scheduled in intervals.

Step 5: Final step is to create a Twitter sentiment analysis dashboard like the following in the Syncfusion Dashboard Cloud.

To learn the basics of creating a dashboard, refer to these links:

You can follow the steps covered in these links to create your own dashboards easily.

Twitter sentiment analysis dashboard

Conclusion:

With the final dashboard, we can understand that, overall, the experience of the workshop ended on a positive note. We also got valuable feedback asking us to slow down and suggesting a few more aspects to cover.

This is just one of the sample use cases gauging sentiments using Dashboard Cloud and using the Data Integration Platform. You can further extend usage of this to any number of social media sites, online feedback surveys for any event, or you can track TV shows for positive remarks on relevant sites like IMDB, Twitter, or Facebook. When you think out of the box, you can always find any number of practical applications to review the sentiment for products, events, TV shows, companies, and industries. You can then make better-informed decisions to improve your services, product recommendations, and other aspects of your business.

Apart from sentiment analysis, you can also run other machine-learning algorithms using extensible Python libraries and showcase the results in a dashboard with more insights for decision-makers.

References:

You can find the Data Integration Platform sample template and the Python script file in this GitHub location.

Tweets and access tokens:

Stanford CoreNLP: https://stanfordnlp.github.io/CoreNLP/

Data Integration Platform:

Dashboard Cloud: