Sentiment Analysis of Tweets. How to analyse the sentiment of tweets... | by Alan Jones | Towards Data Science

Sentiment Analysis, or Opinion Mining, is often used by marketing departments to monitor customer satisfaction with a service, product or brand when a large volume of feedback is obtained through social media.
Gone are the days of reading individual letters sent by post. Today’s customers produce vast numbers of comments on Twitter or other social media.
Such a large amount of data cannot be reasonably analysed individually, so what is produced electronically has to be analysed electronically.
There are two fundamental Sentiment Analysis solutions: first, there are rule based systems that use a lexicon of words and rules to classify a particular piece of text and, second, there are systems that use machine learning techniques that analyse a set of texts that are already labelled with a particular classification (typically, positive or negative) and predict a classification of a new text based upon this.
Machine learning techniques are used by the well-known Python library NLTK (Natural Language Toolkit) and, another NLP library, Textblob, provides both types. But, for our purposes, we are going to use a rule based system that is particularly aimed at social media texts and can not only classify text but also embedded emoticons and shorthand such as OMG.
The Python library that we will use is called VADER and, while it is now incorporated into NLTK, for simplicity we will use the standalone version.
To quote the README file from their Github account: “VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media.” And since our aim is to analyse Tweets, this seems like a good choice.
So the first thing to do is to install it, for example:
pip3 install vaderSentiment
or
conda install vaderSentiment
VADER is very easy to use — here is how to create an analyzer:
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzeranalyzer = SentimentIntensityAnalyzer()
The first line imports the sentiment analyser and the second one creates an analyser object that we can use.
Next we define three strings. They portray attitudes that might have been expressed during Britain’s Brexit debate. The first is likely to have come from someone who is against Brexit, the second is an objective statement of fact, and the third could have come from someone who is a Brexit enthusiast.
s1 = "Britain’s trade will be much worse if it doesn’t have a good trade deal."s2 = "Britain will have to abide by WTO rules if it doesn’t have a trade deal."s3 = "Britain will be very successful whether or not it has a trade deal."
Now we are going to run the sentiment analyzer on the three sentences by passing each string to the polarity_scores method of the analyser
vs = analyzer.polarity_scores(s1)print("{}... {}".format(s1[:30], str(vs)))vs = analyzer.polarity_scores(s2)print("{}... {}".format(s2[:30], str(vs)))vs = analyzer.polarity_scores(s3)print("{}... {}".format(s3[:30], str(vs)))
Printing out the result, we get this:
Britain’s trade will be much w... {'neg': 0.315, 'neu': 0.685, 'pos': 0.0, 'compound': -0.6711}Britain will have to abide by ... {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0}Britain will be very successfu... {'neg': 0.0, 'neu': 0.746, 'pos': 0.254, 'compound': 0.624}
The first part of the string is displayed followed by the result of the analysis. This result is a dictionary with four elements: neg, neu, pos, and compound.
These represent the negative, neutral and positive sentiment as measured by the analyser plus a combination of the scores to produce an overall sentiment value. The first three parts are in the range 0 to 1, whereas the compound result is between -1 and +1: a negative value represents a negative sentiment value and positive value means a positive sentiment. A value around zero obviously means that the sentiment expressed is fairly neutral.
Looking at the compound results for our three strings we can see that they are what we might expect, the first is negative, the second neutral and the last one is positive. Well done, VADER!
Now we need to get some tweets. What you want to look for obviously depends on who you are but the solution below allows you to specify your own search term and look at the tweets that mention that. For this example we will search for “vaccine” but you might want to search for a product name or brand.
I’m going to use the Twitter library but there are other libraries that will do a similar job. You’ll need to sign up for a Twitter developer account to use this and when you do, you will be given the various codes that will need to be inserted below.
import twitterCONSUMER_KEY = '####################'CONSUMER_SECRET = '###############################################'OAUTH_TOKEN = '###############################################'OAUTH_TOKEN_SECRET = '###########################################'auth =       twitter.oauth.OAuth(OAUTH_TOKEN,OAUTH_TOKEN_SECRET,CONSUMER_KEY,    CONSUMER_SECRET)twitter_api = twitter.Twitter(auth=auth)
Now we are ready to download some tweets.
count = 20query = "vaccine"tweets = twitter_api.search.tweets(q=query, count=count,    lang='en',tweet_mode="extended")
As you can see, I’ve asked for 20 tweets that contain the search term “vaccine” and the return tweets are in JSON format with the actual tweet text in tweets[‘statuses’][‘full_text’]. In the code below I iterate through the tweets, analyse the text from each one and construct a list of dictionaries containing the text of the tweet and the overall sentiment value (the compound value returned from the analyzer).
tweetsWithSent = []for t in tweets['statuses']:   text = (t['full_text'])   ps = analyzer.polarity_scores(text)   tweetsWithSent.append({'text':text, 'compound':ps['compound']})
Next I convert the dictionary to a Pandas dataframe and plot the result as a bar graph
import pandas as pdtweetdf = pd.DataFrame(tweetsWithSent)tweetdf.plot.bar(figsize=(15,5),width=1)
You can see that sentiment is fairly evenly distributed — where bars do not appear the value is zero, meaning neutral sentiment.
You will want to use your own search term in order to judge the sentiment of whatever interest you but to give you an idea of the results that I got, here is a screenshot:
(Note that I have anonymised and shortened the text of the tweets above, normally they would contain Twitter handles along with the text.)
If these tweets were comments on your product or service you might be happy to read the positive ones but you would be better spending your time looking at those with a negative sentiment to find out what the problem is.
It would be simple enough to filter out the tweets that are particularly negative for special attention.
Of course, it is not impossible that all of your feedback will be positive — but in the real world that is unlikely. Sentiment Analysis allows you to get an overview of how your customers feel and can allow you to spot problems before they get out of hand.
And that is about it, so I’ll just sign off with the following:
ps = analyzer.polarity_scores("Have a safe 2021")print(ps){'neg': 0.0, 'neu': 0.508, 'pos': 0.492, 'compound': 0.4404}
See a demo of Sentiment Analysis here.
To learn more about the academic background to VADER, you can read this paper:
Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.