Sentiment Analysis Of Collected #F1 Tweets
By: Pratik Parija
Tags: Assignment 3

Tweet Extraction

In a Jupyter notebook, we can use the Tweepy Python library to connect with our Twitter credentials and stream real-time tweets related to a term of interest and then, save them into a “.txt” file. I used a simple Twitter stream listener to collect 1000 tweets with the hashtag “f1” in it. I had the stream directly save to the “.txt” file.

class StreamSaver(tweepy.StreamListener):

def __init__(self, filename, max_num_tweets=1000, api=None):

self.filename = filename



self.num_tweets = 0



self.max_num_tweets = max_num_tweets



tweepy.StreamListener.__init__(self, api=api)





def on_data(self, data):

#print json directly to file



with open(self.filename,'a') as tf:

tf.write(data) self.num_tweets += 1 if self.num_tweets%100 == 0:

print(self.num_tweets) if self.num_tweets > self.max_num_tweets:

return False





def on_error(self, status):

print(status) saveStream = StreamSaver(filename='f1tweetsen.txt', max_num_tweets=1000)

mySaveStream = tweepy.Stream(api.auth, saveStream) mySaveStream.filter(languages=['en'], track=['#f1'])

mySaveStream.disconnect()

After that, we read the data stored in the “.txt” file, into a pandas DataFrame. When you’ve collected your data, you set up a pandas DataFrame containing information of interest: who tweeted, how many followers they have, was it a retweet, who it was a retweet of, if it was one, and to whom it was a reply, if it was one, etc. For this purpose of this post, we are mostly interested in the tweet itself. The other information can be pretty much disregarded.

tweets_data_path = 'f1tweetsen.txt' tweets_data = []

tweets_file = open(tweets_data_path, "r") ... # When we've processed all the tweets, build the DataFrame from the rows

# we've collected

tweets = pd.DataFrame(rows_list) print('Columns are:', tweets.columns)

When we print out the columns this is what we have. Our focus is “text”, which contains the tweets itself.

Columns are: Index(['author', 'reply_to', 'age', 'followers', 'retweet_of', 'rtfollowers',

'rtage', 'text'],

dtype='object')

Clean Up

Now that we have the data we need to clean it up at least that is the crucial step in sentiment analysis.

First I removed the punctuations from my tweet texts.

string.punctuation

In Python, string.punctuation will give all sets of punctuation. Using that:

def remove_punct(text):

text = "".join([char for char in text if char not in string.punctuation])

text = re.sub('[0-9]+', '', text)

return text tweets['text_punct'] = tweets['text'].apply(lambda x: remove_punct(x))

This will remove all those punctuations in our collected tweet texts.

Then I tokenized the texts, turning all the words into list-objects.

def tokenization(text):

text = re.split('\W+', text)

return text tweets['text_tokenized'] = tweets['text_punct'].apply(lambda x: tokenization(x.lower()))

Next, I remove the stop words. A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore, both when indexing entries for searching and when retrieving them as the result of a search query.

stopword = nltk.corpus.stopwords.words('english') def remove_stopwords(text):

text = [word for word in text if word not in stopword]

return text tweets['text_nonstop'] = tweets['text_tokenized'].apply(lambda x: remove_stopwords(x))

After removing the stock words I get into stemming and lematization. Stemming technique looks at the form of the word whereas lemmatization technique looks at the meaning of the word. It means after applying lemmatization, we will always get a valid word.

ps = nltk.PorterStemmer() def stemming(text):

text = [ps.stem(word) for word in text]

return text tweets['text_stemmed'] = tweets['text_nonstop'].apply(lambda x: stemming(x)) wn = nltk.WordNetLemmatizer() def lemmatizer(text):

text = [wn.lemmatize(word) for word in text]

return text tweets['text_lemmatized'] = tweets['text_nonstop'].apply(lambda x: lemmatizer(x)) tweets.head()

When I try to look at the dataframe this is what we have now:

We have the original “text” and then the cleaned up “text_stemmed” & “text_lemmatilized”

Sentiment Analysis

The next step is sentiment analysis. Sentiment analysis is a text analysis method that detects polarity (e.g. a positive or negative opinion) within the text. Sentiment analysis aims to measure the attitude, sentiments, evaluations, attitudes, and emotions of a speaker/writer based on the computational treatment of subjectivity in a text.

Sentiment Analysis sounds easy enough but it is much more complicated than that. A text may contain multiple sentiments all at once.

For the purpose of sentiment analysis, I needed a new library, NLTK. NLTK(Natural Language Toolkit) is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries, and an active discussion forum.

This part was tricky for me as I was trying to figure out how I should go on about doing this. This is when I learned about VADER from a friend.

VADER ( Valence Aware Dictionary for Sentiment Reasoning) is a model used for text sentiment analysis that is sensitive to both polarity (positive/negative) and intensity (strength) of emotion. It is available in the NLTK package and can be applied directly to unlabeled text data.

VADER sentimental analysis relies on a dictionary that maps lexical features to emotion intensities known as sentiment scores. The sentiment score of a text can be obtained by summing up the intensity of each word in the text.

It is useful for analyzing short documents especially tweets. Considers emojis and capitalization of words too.

VADER’s SentimentIntensityAnalyzer() takes in a string and returns a dictionary of scores in each of four categories:

negative

neutral

positive

compound (computed by normalizing the scores above)

# Instantiate new SentimentIntensityAnalyzer

sid = SentimentIntensityAnalyzer() # Generate sentiment scores

sentiment_scores = tweets['text_punct'].apply(sid.polarity_scores) sentiment = sentiment_scores.apply(lambda x: x['compound'])

I used the VADER sentiment analyzer on our cleaned up tweet text to generate scores. I try to print out a positive and negative tweet after they have been scored.

# Print out the text of a positive tweet

print(tweets[sentiment > 0.6]['text_punct'].values[0]) # Print out the text of a negative tweet

print(tweets[sentiment < -0.6]['text_punct'].values[0])

This is what I get:

Now I’ll add columns to the original DataFrame to store polarity_score dictionaries, extracted compound scores, and new “pos/neg” labels derived from the compound score. Then I take the compound score and sort of round it to be either 1 (positive), 0 (neutral), or -1(negative) so I can plot it.

tweets['scores'] = tweets['text_punct'].apply(lambda x: sid.polarity_scores(x)) tweets['compound'] = tweets['scores'].apply(lambda score_dict: score_dict['compound']) tweets['comp_score'] = tweets['compound'].apply(lambda c: 'pos' if c >=0 else 'neg') # Creating function for calculating positive, negative and neutral

# More than 1 --> Positive, equal to 0 --> neutral and less than 0 --> Negative

def ratio(x):

if x > 0:

return 1

elif x == 0:

return 0

else:

return -1 tweets['analysis'] = tweets['compound'].apply(ratio) tweets.head()

So now I have got a complete analysis of every review as either positive or negative. This is how it looks:

In the end, I try to visualize the sentiment to see how much of each emotion I analyzed from my collected tweets:

# Plotting

tweets['analysis'].value_counts().plot(kind = 'bar')

Formula 1 is a popular motorsport event and in the end, as I suspected, most of the tweets under the hashtag “f1” were positive with very few actually being negative. People seem to be enjoying the 70th-anniversary season which has been quite an interesting one as the world tries to cope with COVID-19. VADER also did a great job with the results and was a lot simpler to use than I had expected. The only real weakness I have encountered is that it isn’t really good at analyzing non-English text.