Sentiment Analysis on Twitter data regarding 2020 US Elections
By: Pritam Guha
Tags: Data Analytics, Sentiment Analysis, US Elections, Python, Naturallanguageprocessing

Data Wrangling

In order to use the Twitter API, we need to have a Twitter Developer Account which gives us access to an access token, an access token secret, an api key and an api secret key. I have noted these parameters inside a properties file which I placed inside one of my drive folders. I used python’s configparser to read those data in order to use the twitter API.

import configparser

import os

import tweepy as tw

import pandas as pd config = configparser.RawConfigParser()

config.read('/content/drive/My Drive/Colab Notebooks/twitter.properties') accesstoken = config.get('twitter','accesstoken')

accesstokensecret = config.get('twitter','accesstokensecret')

apikey = config.get('twitter','apikey')

apisecretkey = config.get('twitter','apisecretkey') auth = tw.OAuthHandler(apikey, apisecretkey)

auth.set_access_token(accesstoken, accesstokensecret)

api = tw.API(auth, wait_on_rate_limit=True) search_words = "Joe Biden" # This will be changed to Donald Trump when we retrieve tweets related to him.

date_since = "2020-10-27"

Twitter uses pagination a lot in their API development. In order to perform pagination, Twitter supplies a page/cursor parameter with each of their requests. As a result, this requires a lot of boiler plate code just to manage the pagination loop. To help make pagination easier and require less code the Twitter API or Tweepy has the Cursor object.

Now, since the Cursor is passed callable, we can not pass the parameters directly into the method. Instead the parameters are passed into the Cursor constructor method. For our api to search tweets, we pass the following parameters:

q: This is the keyword to be searched in the tweet. For our project, we pass the name of the candidate (Donald Trump/Joe Biden). lang: This is the language of the tweets we want to retrieve from the API. Since USA is largely an English-speaking country with English also being the official language, we retrieve tweets made in English. since: This is the date from which we want to retrieve tweets. For our purposes, we are interested in the current political discourse to understand the current mood ahead of the elections. So we pass this as last Tuesday, which is one week before the elections are to be held.

Another factor which needs to be kept in mind is the tweepy search api retrieves a maximum of 1500 tweets at a time followed by a cooldown period of 15 minutes.

tweets = tw.Cursor(api.search, q=search_words, lang="en", since=date_since).items(100)

tweets

The result of the search api call is an Iterator object.

The result of the search api call is a cursor item iterator object. From this iterator object, we iterate through pick up the geo, text, username and location details. Some details like geo and location details are dependent on whether the particular user is sharing these details or not. If the user is not sharing these details, we will get blank data for these columns.

Afterwards, we convert our list to a pandas dataframe.

tweet_details = [[tweet.geo, tweet.text, tweet.user.screen_name, tweet.user.location] for tweet in tweets]

tweet_df = pd.DataFrame(data=tweet_details, columns=["geo","text","user","location"])

tweet_df.head()

Looking at the tweet texts, we notice that there are lot of characters which do not add any value towards sentiment analysis. Many of the tweets are preceeded with RT which indicates the tweet was retweeted. The tweets are also preceeded with the name of the user from whom the tweet was retweeted which is also an information we are not concerned. We also remove any html website links which maybe present inside the tweet. All of these data cleaning are done with the help of the ‘re’ built-in package of Python.

import re

def clean_tweets(text):

text = re.sub("RT @[\w]*:","",text)

text = re.sub("@[\w]*","",text)

text = re.sub("https?://[A-Za-z0-9./]*","",text)

text = re.sub("

","",text)

return text tweet_df['text'] = tweet_df['text'].apply(lambda x: clean_tweets(x))

tweet_df['Subject'] = search_words

tweet_df.head(20)

In this analysis, we are mostly concerned with finding the political mood of each and every US state currently from the twitter data. In order to do this, we need to do some more filtering with the original data which we extracted from the tweepy API. As we discussed before, the tweepy API provides us with the location of the user who made the tweet provided he/she is sharing this information with us. This location data is mostly blank. For others, it may consist of the country name only, city name only or the city and state separated by a comma and so on.

In the next part, we form a list of US states and another list of US state codes in order to extract tweets which contain these in the location record.

P.S. This is a very naive way to do this cleansing. A much better way would have been to use the Google Maps API and get the state information from the location data provided it was in United States of America. However, due to problems with my Google Maps Billing Account, I had to contend myself with the naive way.

states = ['Alabama', 'Alaska', 'Arizona', 'Arkansas', 'California', 'Colorado', 'Connecticut', 'Delaware', 'Florida', 'Georgia', 'Hawaii', 'Idaho', 'Illinois', 'Indiana', 'Iowa', 'Kansas', 'Kentucky', 'Louisiana', 'Maine', 'Maryland', 'Massachusetts', 'Michigan', 'Minnesota', 'Mississippi', 'Missouri', 'Montana', 'Nebraska', 'Nevada', 'New Hampshire', 'New Jersey', 'New York', 'New Mexico', 'North Carolina', 'North Dakota', 'Ohio', 'Oklahoma', 'Oregon', 'Pennsylvania', 'Rhode Island', 'South Carolina', 'South Dakota', 'Tennessee', 'Texas', 'Utah', 'Vermont', 'Virginia', 'Washington', 'West Virginia', 'Wisconsin', 'Wyoming'] stateCodes = ['AL', 'AK', 'AZ', 'AR', 'CA', 'CO', 'CT', 'DE', 'FL', 'GA', 'HI', 'ID', 'IL', 'IN', 'IA', 'KS', 'KY', 'LA', 'ME', 'MD', 'MA', 'MI', 'MN', 'MS', 'MO', 'MT', 'NE', 'NV', 'NH', 'NJ', 'NM', 'NY', 'NC', 'ND', 'OH', 'OK', 'OR', 'PA', 'RI', 'SC', 'SD', 'TN', 'TX', 'UT', 'VT', 'VA', 'WA', 'WV', 'WI', 'WY'] stateMapping = {'AL': 'Alabama', 'AK': 'Alaska', 'AZ': 'Arizona', 'AR': 'Arkansas', 'CA': 'California', 'CO': 'Colorado', 'CT': 'Connecticut', 'DE': 'Delaware', 'FL': 'Florida', 'GA': 'Georgia',

'HI': 'Hawaii', 'ID': 'Idaho', 'IL': 'Illinois', 'IN': 'Indiana', 'IA': 'Iowa', 'KS': 'Kansas', 'KY': 'Kentucky', 'LA': 'Louisiana', 'ME': 'Maine', 'MD': 'Maryland', 'MA': 'Massachusetts', 'MI': 'Michigan', 'MN': 'Minnesota', 'MS': 'Mississippi', 'MO': 'Missouri', 'MT': 'Montana', 'NE': 'Nebraska', 'NV': 'Nevada', 'NH': 'New Hampshire', 'NJ': 'New Jersey', 'NY': 'New York', 'NM': 'New Mexico', 'NC': 'North Carolina', 'ND': 'North Dakota', 'OH': 'Ohio', 'OK': 'Oklahoma', 'OR': 'Oregon', 'PA': 'Pennsylvania', 'RI': 'Rhode Island', 'SC': 'South Carolina', 'SD': 'South Dakota', 'TN': 'Tennessee', 'TX': 'Texas', 'UT': 'Utah', 'VT': 'Vermont', 'VA': 'Virginia', 'WA': 'Washington', 'WV': 'West Virginia', 'WI': 'Wisconsin', 'WY': 'Wyoming'} tweet_copied_df = tweet_df

for index, row in tweet_df.iterrows():

flag = 0

if row.location:

locationSplit = row.location.split(',')

for word in locationSplit:

word_stripped = word.strip()

if word_stripped in states:

flag = 1

row['state'] = word_stripped

elif word_stripped in stateCodes:

flag = 1

row['state'] = stateMapping[word_stripped]

if flag == 0:

tweet_copied_df = tweet_copied_df.drop(index=index)

else:

tweet_copied_df.loc[index, 'state'] = row['state']

At the end, we append the cleaned data to an existing csv file to create a corpus of twitter data on which we will use our sentiment analyser. For the purposes of this project, I ran the above lines of code in a loop with an interval of 15 minutes in between every iteration.