Creating A Twitter Sentiment Analysis Program in Python with Naive Bayes Classification
By: Anas Al-Masri
Tags: Sentiment Analysis, Machine Learning, Python, Naive Bayes

Section A: Preparing The Test Set

As our task of Sentiment Analysis is one that focuses heavily on textual data, one would expect there to be a lot of text processing. This is definitely correct. In fact, both our Test and Training data will merely comprise of text.

I chose to start with the Test set in order to get you all warmed up for the Training set extraction part, as it will rely more on the API. Here is a bit of an overview of what we are about to do:

1- Register Twitter application to get our own credentials. 2- Authenticate our Python script with the API using the credentials. 3- Create function to download tweets based on a search keyword.

Registering an application with Twitter is critical, as it is the only way to get authentication credentials. As soon as we get our credentials, we will start writing code. Step 3 is where the Test set lies. We will be downloading tweets based on the term that we are trying to analyze the sentiment on.

Step A.1: Getting the authentication credentials

First off, we need to visit the Twitter Developer website and log into our account through the following link:

On the top-right corner, click on the Apps button, Create an App, Apply and then Continue, as shown below:

Next, we will choose the “I am requesting access for my own personal use” option:

On the same web page, scroll-down a bit and input your Account name and Country of operation then click Continue, and you will be redirected to the next web page. Here, you can choose any Use Cases you’re interested in. For our case, I chose the following:

After you make your choice, scroll-down and fill out the use case interest paragraph required. Twitter takes this very seriously now (guess they learned from Facebook’s mistakes xD), so make sure you emphasize on the application being a self-learning/academic-related project. Choose “No” for the government involvement question, and press “Continue”.

On the next web page, read the Terms and Conditions list, Agree to them then Submit Application.

Next, open your email and verify your Twitter Developer account through the link included in the email sent to you. Finally, you will get a message similar to the following:

All you can do now is wait for a few hours for the application to be approved (which almost certainly will, if your explanation did not violate Twitter’s terms and conditions).

When you get the approval email, click on the login link it contains. You will be redirected to the following web page, where you should choose “Create an app”:

On the next web page, click “Create an app” from the top-right corner. After you are redirected, fill out the required app details, including — if you’d like — that it is for self-learning purposes. Click “Create”.

The next web page will include the app details that you just input, access tokens and permissions. Proceed to the “Keys and tokens” tab. Copy the API key as well as the API secret key into a safe place (a text file, if you’d like), as we will be using them in a bit.

Once you’re all set, click “Create” to generate the Access token credentials. Copy the Access token and the Access token secret into a safe place as well. We’re done with the credential acquisition part!

Step A.2: Authenticating our Python script

Since we now have our Twitter Developers login credentials (i.e. API keys and Access token), we can proceed to authenticating our program. First, we need to import the Twitter library, then create an Twitter.API object with the credentials from the “safe” place we talked about, as follows:

The last line in the previous code snippet is only there to verify that our API instance works. This will be determined based on the output we get. Run the above code and you should get something like the following JSON response:

{"created_at": "Tue Feb 12 17:48:27 +0800 2019" 'default_profile": true ............}

That is nothing crazy but some data about the access made to the API through your Twitter account. If you reach this, you’re good to go.

Step A.3: Creating the function to build the Test set

Now we can start on making a function that downloads the Test set that we talked about. Basically, this is going to be a function that takes a search keyword (i.e. string) as an input, searches for tweets that include this keyword and returns them as twitter.Status objects that we can iterate through.

The caveat here, though, is that Twitter limits the number of requests you can make through the API for security purposes. This limit is 180 requests per 15-minute window.

This means, we can only get up to 180 tweets using our search function every 15 minutes, which should not be a problem, as our Training set is not going to be that large anyway. For the sake of simplicity, we will limit the search to 100 tweets for now, not exceeding the allowed number of requests. Our function for searching for the tweets (i.e. Test set) will be:

As you might have expected, this function will return a list of tweets that contain our search keyword.

Note that we coupled — into a JSON object — every tweet’s text with a label that is NULL for now. This is merely because we are going to classify each tweet as Positive or Negative later on, in order to determine whether the sentiment on the search term is positive or negative, based on the majority count. This is how Sentiment Analysis pragmatically works.

Before we move on, let’s test out our function by adding the following code after the function body:

This should print out five tweets that contain our search keyword on the Terminal of your IDE (if you’re using one). Now everything is set. We have our Test set and we can move on to building our Training set.