Creating A Twitter Sentiment Analysis Program in Python with Naive Bayes Classification | by Anas Al-Masri | Towards Data Science

Sentiment Analysis is a term that you must have heard if you have been in the Tech field long enough. It is the process of predicting whether a piece of information (i.e. text, most commonly) indicates a positive, negative or neutral sentiment on the topic. In this article, we will go through making a Python program that analyzes the sentiment of tweets on a particular topic. The user will be able to input a keyword and get the sentiment on it based on the latest 100 tweets that contain the input keyword.
Also known as “Opinion Mining”, Sentiment Analysis refers to the use of Natural Language Processing to determine the attitude, opinions and emotions of a speaker, writer, or other subject within an online mention.
Essentially, it is the process of determining whether a piece of writing is positive or negative. This is also called the Polarity of the content.
As humans, we are able to classify text into positive/negative subconsciously. For example, the sentence “The kid had a gorgeous smile on his face”, will most likely give us a positive sentiment. In layman’s terms, we kind of arrive to such conclusion by examining the words and averaging out the positives and the negatives. For instance, the words “gorgeous” and “smile” are more likely to be positive, while words like “the”, “kid” and “face” are really neutral. Therefore, the overall sentiment of the sentence is likely to be positive.
A common use for this technology comes from its deployment in the social media space to discover how people feel about certain topics, particularly through users’ word-of-mouth in textual posts, or in the context of Twitter, their tweets.
Basic programming knowledge:
Although Python is highly involved in this mini-project, it is not required to have a deep knowledge in the language, as long as you have basic programming knowledge.
Installed tools:
For this program, we will need Python to be installed on the computer. We will be using the libraries twitter, nltk, re, csv, time, and json. You are likely to have to install the first two libraries. The rest already come with the Python interpreter. It doesn’t hurt to check that they’re up-to-date though.
Data set splitting concept:
This is critical to fully understand the process pipeline. You only need to know the difference between Training and Test data sets, and in what context each one is used.
Basic RESTful API knowledge:
This is not crucial, but it could help. We will be using the Twitter API here and there in the code, making normal calls to the API and dealing with the JSON objects it returns. In case you need it, you can find the official Twitter API documentation here.
Section A: Preparing The Test Set
Step A.1: Getting the authentication credentials
Step A.2: Authenticating our Python script
Step A.3: Creating the function to build the Test set
Section B: Preparing The Training Set
Section C: Pre-processing Tweets in The Data Sets
Section D: Naive Bayes Classifier
Step D.1: Building the vocabulary
Step D.2: Matching tweets against our vocabulary
Step D.3: Building our feature vector
Step D.4: Training the classifier
Section E: Testing The Model
Twitter can sometimes take a few days to approve your application to use the Twitter API. However, it usually takes less that 24 hours.
It can take 10+ hours to download the Training set (this will be explained later on).
The tutorial was adopted from the Udemy course: From 0 to 1: Machine Learning, NLP, and Python-cut to The Chase.
As our task of Sentiment Analysis is one that focuses heavily on textual data, one would expect there to be a lot of text processing. This is definitely correct. In fact, both our Test and Training data will merely comprise of text.
I chose to start with the Test set in order to get you all warmed up for the Training set extraction part, as it will rely more on the API. Here is a bit of an overview of what we are about to do:
1- Register Twitter application to get our own credentials.
2- Authenticate our Python script with the API using the credentials.
3- Create function to download tweets based on a search keyword.
Registering an application with Twitter is critical, as it is the only way to get authentication credentials. As soon as we get our credentials, we will start writing code. Step 3 is where the Test set lies. We will be downloading tweets based on the term that we are trying to analyze the sentiment on.
Step A.1: Getting the authentication credentials
First off, we need to visit the Twitter Developer website and log into our account through the following link:
developer.twitter.com
On the top-right corner, click on the Apps button, Create an App, Apply and then Continue, as shown below:
Next, we will choose the “I am requesting access for my own personal use” option:
On the same web page, scroll-down a bit and input your Account name and Country of operation then click Continue, and you will be redirected to the next web page. Here, you can choose any Use Cases you’re interested in. For our case, I chose the following:
After you make your choice, scroll-down and fill out the use case interest paragraph required. Twitter takes this very seriously now (guess they learned from Facebook’s mistakes xD), so make sure you emphasize on the application being a self-learning/academic-related project. Choose “No” for the government involvement question, and press “Continue”.
On the next web page, read the Terms and Conditions list, Agree to them then Submit Application.
Next, open your email and verify your Twitter Developer account through the link included in the email sent to you. Finally, you will get a message similar to the following:
All you can do now is wait for a few hours for the application to be approved (which almost certainly will, if your explanation did not violate Twitter’s terms and conditions).
When you get the approval email, click on the login link it contains. You will be redirected to the following web page, where you should choose “Create an app”:
On the next web page, click “Create an app” from the top-right corner. After you are redirected, fill out the required app details, including — if you’d like — that it is for self-learning purposes. Click “Create”.
The next web page will include the app details that you just input, access tokens and permissions. Proceed to the “Keys and tokens” tab. Copy the API key as well as the API secret key into a safe place (a text file, if you’d like), as we will be using them in a bit.
Once you’re all set, click “Create” to generate the Access token credentials. Copy the Access token and the Access token secret into a safe place as well. We’re done with the credential acquisition part!
Step A.2: Authenticating our Python script
Since we now have our Twitter Developers login credentials (i.e. API keys and Access token), we can proceed to authenticating our program. First, we need to import the Twitter library, then create an Twitter.API object with the credentials from the “safe” place we talked about, as follows:
The last line in the previous code snippet is only there to verify that our API instance works. This will be determined based on the output we get. Run the above code and you should get something like the following JSON response:
{"created_at": "Tue Feb 12 17:48:27 +0800 2019" 'default_profile": true ............} 
That is nothing crazy but some data about the access made to the API through your Twitter account. If you reach this, you’re good to go.
Step A.3: Creating the function to build the Test set
Now we can start on making a function that downloads the Test set that we talked about. Basically, this is going to be a function that takes a search keyword (i.e. string) as an input, searches for tweets that include this keyword and returns them as twitter.Status objects that we can iterate through.
The caveat here, though, is that Twitter limits the number of requests you can make through the API for security purposes. This limit is 180 requests per 15-minute window.
This means, we can only get up to 180 tweets using our search function every 15 minutes, which should not be a problem, as our Training set is not going to be that large anyway. For the sake of simplicity, we will limit the search to 100 tweets for now, not exceeding the allowed number of requests. Our function for searching for the tweets (i.e. Test set) will be:
As you might have expected, this function will return a list of tweets that contain our search keyword.
Note that we coupled — into a JSON object — every tweet’s text with a label that is NULL for now. This is merely because we are going to classify each tweet as Positive or Negative later on, in order to determine whether the sentiment on the search term is positive or negative, based on the majority count. This is how Sentiment Analysis pragmatically works.
Before we move on, let’s test out our function by adding the following code after the function body:
This should print out five tweets that contain our search keyword on the Terminal of your IDE (if you’re using one). Now everything is set. We have our Test set and we can move on to building our Training set.
Note that so far, we haven’t written a lot of code. This is attributed to the beauty of Python’s succinctness in syntax as well as the use of external program-ready libraries, such as RESTful APIs (Twitter API in our case).
In this section, we will also be using our Twitter API instance from the last section. However, we need to get some things out the way first. We will be using a downloadable Training set. The tweets of which were all labeled as positive or negative, depending on the content. This exactly what a Training set is for.
A Training set is critical to the success of the model. Data is which needs to be labeled properly with no inconsistencies or incompleteness, as training will rely heavily on the accuracy of such data and the manner of acquisition.
For this task, we will be using the amazing Niek Sanders’ Corpus of over 5000 hand-classified tweets, which makes it quite reliable. There’s also a catch here. Twitter does not allow storing tweets on a personal device, even though all such data is publicly available. Therefore, the corpus includes a keyword (topic of the tweet), a label (pos/neg) and a tweet ID number for every tweet (i.e. row in our CSV corpus). You can get the file containing the corpus from the original site, or through this link of a personal repository.
Let’s backtrack for a bit. Remember the Twitter API limit we talked about? This will also apply here, as we will be using the API to get the actual tweet text through each tweet’s ID number included in the Corpus we have. This means, to download 5000 tweets, we will need to follow:
max_number_of_requests = 180time_window = 15 minutes = 900 secondsTherefore, the process should follow:Repeat until end-of-file: {    180 requests -> (900/180) sec wait}
Let’s now write the code that does exactly that. Let’s not forget to save the tweets we retrieve through the API into a new CSV file so that we don’t have to download them every time we run the code. Our function will be as follows:
This was a handful, but it’s fairly simple if we break it down into parts. Firstly, we define the function to take two inputs, both of which are file paths:
corpusFile is the string path to the Niek Sanders’ CSV corpus file we downloaded. This file, as mentioned earlier, includes the tweet’s topic, label and id.
tweetDataFile is the string path to the file we would like to save the full tweets in. In contrast to corpusFile, this file will include every tweet’s text as well as topic, label and id.
Next, we started with an empty list corpus. We then opened the file corpusFile and appended every tweet from the file to the list corpus.
The next segment of the code deals with getting the text of tweets based on the IDs. We loop through the tweets in corpus, calling the API on every tweet to get the Tweet.Status object of the particular tweet. Afterwards, we use that same object (status) to get the text associated with it and push it into the trainingDataSet then sleep (i.e. pause execution) for five minutes (900/180 seconds) in order to abide by the request limit we talked about.
Now let’s take the time to leave our script download the tweets (which will take hours) following our last function. We can do this using the following snippet:
As soon as the code finishes executing, you will have your tweetDataFile CSV file full of tweets (~5000, as a matter of fact). If you got this far, CONGRATULATIONS !— it took me, the first time, a substantial amount of time to reach here without problems. Now we’re done with the relatively boring part. Let’s get ourselves hyped up for the upcoming section.
Before we move on to the actual classification section, there is some cleaning up to do. As a matter of fact, this step is critical and usually takes a long time when building Machine Learning models. However, this will not be a problem in our task, as the data we have is relatively consistent. In other words, we know exactly what we need from it. I will express on this matter later on.
Let’s talk about what matters and what doesn’t matter in Sentiment Analysis. Words are the most important part (to an extent that we will talk about in the upcoming section). However, when it comes to things like punctuation, you cannot get the sentiment from punctuation. Therefore, punctuation does not matter to Sentiment Analysis. Moreover, tweet components like images, videos, URLs, usernames, emojis, etc. do not contribute to the polarity (whether it is positive or negative) of the tweet. However, this is only true for this application. For instance, in another application, you could have a Deep Learning image classifier that learns and predicts whether this image that the tweet contains stands for something positive (e.g. a rainbow) or negative (e.g. a tank). When it comes to the technicality, both Sentiment Analysis and Deep Learning fall under Machine Learning. In fact, you can perform Sentiment Analysis through Deep Learning, but that’s a story for another day.
So we know what we need to keep in the tweets we have and what we need to take out. This applies to both Training and Test sets. So let’s make a our pre-processor class:
That was a handful, so let’s break it down into parts. We start off by our imported libraries. re is Python’s Regular Expressions (RegEx) library, which takes care of parsing strings and modifying them in an efficient way without having to explicitly iterate through the characters comprising the particular string. We also imported ntlk, is the Natural Processing Toolkit, which is one of the most commonly used Python libraries out there. It takes care of any processing that we need to perform on text to change its form or extract certain components from it. The class constructor removes stop words. This is a relatively big topic that you can read up on later, as it is more into Natural Language Processing and less related to our topic.
The processTweets function just loops through all the tweets input into it, calling its neighboring function processTweet on every tweet in the list. The latter does the actual pre-processing by first making all the text in lower-case letters. This is merely because, in almost all programming languages, “cAr” is not interpreted the same way as “car”. Therefore, it is better to normalize all characters to be lower-case across all our data. Secondly, URLs and usernames are removed from the tweet. This is for the reasons we disclosed earlier in the article. Afterwards, the number sign (i.e. #) is removed from every hashtag, in order to avoid hashtags being processed differently. Last but not least, duplicate characters are rid off of, in order to ensure that no important word goes unprocessed even if it is spelled out in an unusual way (e.g. “caaaaar” becomes “car”). Finally, the tweet’s text is broken into words (tokenized) in order to ease its processing in the upcoming stages.
Let’s take an example. The following tweet could be present in the data set:
"@person1 retweeted @person2: Corn has got to be the most delllllicious crop in the world!!!! #corn #thoughts..."
Our pre-processor will result in the tweet looking like:
“AT_USER rt AT_USER corn has got to be the most delicious crop in the world corn thoughts”
And finally, the tokenization will result in:
{“corn”, “most”, “delicious”, “crop”, “world”, “corn”, “thoughts”}
Note that our code removed duplicate characters in words as we metioned earlier (i.e. “delllllicious” became “delicious”). However, it did not remove duplicate words (i.e. “corn”) from the text, but rather kept them. This is because duplicate word play a role in determining the polarity of the text (as we will see in the upcoming section).
We are all set to use our Pre-processor class. First, we will create a variable that refers to it (an object), and then call it on both the Training and Test sets as we discussed earlier:
Now we can move on to the most exciting part — classification. But first, let’s brush up (or touch) on our algorithm for this task: Naive Bayes Classifier.
Without a doubt, one of the most important concepts in Computer Science and Machine Learning. It can be frustrating to get into the math of it head-first. Therefore, I will not be explaining it from the mathematical perspective.
You don’t need to know the math to be a Computer Scientist. However, you do need to know the math to become a really good one.
I want to emphasize on the fact that I will only briefly explain Naive Bayes Classification here, as the in-depth explanation deserves its own lengthy post.
Naive Bayes Classifier is a classification algorithm that relies on Bayes’ Theorem. This theorem provides a way of calculating a type or probability called posterior probability, in which the probability of an event A occurring is reliant on probabilistic known background (e.g. event B evidence). For example, if Person_X only plays tennis when it is not raining outside, then, according to Bayesian statistics, the probability of Person_X playing tennis when it is not raining can be given as:
P(X plays | no rain) = P(no rain | X plays)*P(x plays)/P(no rain)
following Bayes’ theorem:
P(A|B) = P(B|A)*P(A)/P(B)
All you need to know for our task is that a Naive Bayes Classifier depends on the ever-famous Bayes’ theorem. Before we move on, let’s give a quick overview of the steps we will be taking next:
1- Build a vocabulary (list of words) of all the words resident in our training data set.
2- Match tweet content against our vocabulary — word-by-word.
3- Build our word feature vector.
4- Plug our feature vector into the Naive Bayes Classifier.
This might seem like a lot, but don’t worry. It is actually fairly simple and as short as it can be. Let’s take it down step-by-step.
Step D.1: Building the vocabulary
A vocabulary in Natural Language Processing is a list of all speech segments available for the model. In our case, this includes all the words resident in the Training set we have, as the model can make use of all of them relatively equally — at this point, to say the least. The code will look something like this:
This is just creating a list of all_words we have in the Training set, breaking it into word features. Those word_features are basically a list of distinct words, each of which has its frequency (number of occurrences in the set) as a key.
Step D.2: Matching tweets against our vocabulary
This step is crucial, as we will go through all the words in our Training set (i.e. our word_features list), comparing every word against the tweet at hand, associating a number with the word following:
label 1 (true): if word in vocabulary is resident in tweetlabel 0 (false): if word in vocabulary is not resident in tweet
This is fairly simple to code:
Given the last snippet, for every word in the word_features, we will have the JSON key ‘contains word X’, where X is the word. Every key of those will have the value True/False, according to what we said earlier about the labels — True for ‘present’ and False for ‘absent’.
Step D.3: Building our feature vector
Let’s now call the last two functions we have written. This will build our final feature vector, with which we can proceed on to training.
The NTLK built-in function apply_features does the actual feature extraction from our lists. Our final feature vector is trainingFeatures.
Step D.4: Training the classifier
We have finally come to the most important — and ironically the shortest — part of our task. Thanks to NLTK, it will only take us a function call to train the model as a Naive Bayes Classifier, since the latter is built into the library:
We’re almost done! All we have left is running the classifier training code (i.e. nltk.NaiveBayesClassifier.train()) and testing it. Note that this code could take a few minutes to execute.
Moment of truth! Let’s finish up our work by running the classifier (i.e. NBayesClassifier) on the 100 tweets that we downloaded from Twitter, according to our search term, and getting the majority vote of the labels returned by the classifier, then outputting the total positive or negative percentage (i.e. score) of the tweets. This is also going to be very simple:
That’s it! Congratulations. You have created a Twitter Sentiment Analysis Python program. Note that we did not touch on the accuracy (i.e. evaluate the model) because it is not our topic for the day. There will be a post where I explain the whole model/hypothesis evaluation process in Machine Learning later on. Let’s take a final look at the full code we wrote for this task:
Sentiment Analysis is an interesting way to think about the applicability of Natural Language Processing in making automated conclusions about text. It is being utilized in social media trend analysis and, sometimes, for marketing purposes. Making a Sentiment Analysis program in Python is not a difficult task, thanks to modern-day, ready-for-use libraries. This program is a simple explanation to how this kind of application works. This is only for academic purposes, as the program described here is by no means production-level.