Twitter Sentiment Analysis in Python | by Przemyslaw Jarzynski | Towards Data Science

Sentiment analysis is one of the most common tasks in Data Science and AI. In this article, we will use Python, Tweepy and TextBlob to perform sentiment analysis of a selected Twitter account using Twitter API and Natural Language Processing.
We will use Twitter to perform sentiment analysis of the written text. We will use Twitter in this example but this can be also used in a business context to analyse different social media accounts, reviews of your company, reviews of your products and services, analysing support tickets, emails or free text from surveys to get an idea of the mood that is coming from people engaging with you and your business online.
You will learn how to perform basic sentiment analysis using TextBlob; powerful Natural Language Processing library for Python. We will also use the WordCloud library to visualise some of our findings and we will also work with a Twitter API. Familiarizing with APIs is a useful skill for data scientists. It is a very common method of getting hold of the data from the internet.
Our task is to analyse the Tweets of an individual Twitter account in terms of Subjectivity and Polarity. We will identify individual tweets as positive, negative and neutral and calculate the percentage of positive tweets. We will use the WordCloud library to display a word cloud of the most positive words from the tweets.
Creating a virtual environment
We want to separate Twitter specific Python libraries from other libraries on our computer. That is why we will create a virtual environment for this project.
You should have a folder on your computer where you would have all your virtual environments stored. There should be nothing else in this folder, no project files etc., just different virtual environments. You can call this folder “virtualenvironments”.
Once you that folder, create your virtual environment:
$ virtualenv twitter_venv
then activate it:
$ source twitter_venv/bin/activate
then install Tweepy and other libraries:
(twitter_venv) $ conda install tweepy(twitter_venv) $ conda install textblob(twitter_venv) $ conda install wordcloud
Once the library is successfully installed you can deactivate it:
(twitter_venv) $ deactivate
Then you need to add this virtual environment to your jupyter notebook configuration:
$ python -m ipykernel install — user — name=twitter_venv
Once you have completed this step you are ready to continue with the project.
Create a project folder
Create a folder for a project on your computer called “Twitter-Sentiment-Analysis”:
$ mkdir “Twitter-Sentiment-Analysis”
Creating Twitter Applications Keys
For this project, we will be accessing information from twitter using Twitter API. In order to do this, we need a Twitter developer account.
Twitter has introduced a few new requirements for developers and there are few steps to complete this process.
First, you need to go to developer.twitter.com. If you do not have a standard Twitter account you need to create one and log in.
Standard Twitter account will allow you to login but it will not let you do anything else and you need to apply for developer access by going to this address developer.twitter.com/en/apply-for-access and click on Apply for developer account.
You will have to answer a few questions. For the reason of using Twitter developer tools, you can select Exploring the API.
You will have to answer a few more questions about how you will be using Twitter API. You can answer by explaining that you will be learning data science and not planning to build any products using API (subject you are only creating Twitter Developer Account for the purpose of this exercise, if you are planning to use Twitter Developer Account for anything else, you need to provide more details).
At the end, you have to agree to Developer Agreement & Policy and you can submit your application.
Once the application is submitted you will need to confirm your email address by clicking on the link that Twitter will send to you. After confirming your email address your access to Twitter Developer Account should be granted almost immediately. In case you receive an email from Twitter asking for additional information or clarification about your application, you need to provide all the necessary information.
Once your access is granted go to this url https://developer.twitter.com/en/apps and Create an app.
You will have to provide a name, description, URL and the reason you’re creating the app.
Once your app is created go to Keys and tokens tab. You will have API key and API secret key already created there and you need to generate an Access token and Access token secret. You can do it by clicking Generate button. You will need all four values for your Twitter Sentiment Analysis project. Copy all of them now and keep them somewhere safe in the file. We will use them later.
Start a new notebook
Enter the project folder and start Jupyter Notebook by typing a command in the Terminal/Command Prompt:
$ cd “Twitter-Sentiment-Analysis”
then
$ jupyter notebook
Click new in the top right corner and select twitter_venv virtual environment.
This will open a new jupyter notebook in your browser. Rename the Untitled project name to your project name and you are ready to start.
If you are using Google Colab, open a new notebook.
Loading libraries
Importing Tweepy, TextBlob, WordCloud, Pandas, Numpy and Matplotlib.
Let’s load the config file (make sure you edit the config file and enter your Twitter API details as described in the introduction).
Config file config.csv has the following format:
twitterApiKey,twitterApiSecret,twitterApiAccessToken,twitterApiAccessTokenSecretenter-your-value,enter-your-value,enter-your-value,enter-your-value
And you can find an example of it here.
Now we need to set all Twitter API config values required for authentication with Tweepy. These values will be read from the config.csv file to Python variables.
We are making an authentication call with Tweepy so we can call a function to retrieve the latest tweets from the specified Twitter account.
We are using elonmusk Twitter account as an example but feel free to change the Tweeter account to a different one; even your own Twitter account.
Now we are going to retrieve the last 50 Tweets & replies from the specified Twitter account.
And we are going to create Pandas Data Frame from it.
Let’s see what is in the Data Frame by calling the head() function.
Before we start our sentiment analysis it is a good idea to clean up each tweet from an unnecessary data first.
We are going to create a cleanUpTweet function that will:
remove mentions
remove hashtags
remove retweets
remove urls
And now we are going to apply it for all Tweets in our Pandas Data Frame.
We are also going to build a couple more functions to calculate the subjectivity and polarity of our tweets using TextBlob.
And now we are going to apply these functions to our Data Frame and create two new features in our Data Frame Subjectivity and Polarity.
Now, let’s see how our data frame looks now.
We have some empty rows here so before further processing let’s remove those.
The below command will remove all the rows with the Tweet column equals to “”.
We can see that we have calculated score for the subjectivity and polarity in our Data Frame.
Now let’s build a function and categorize our tweets as Negative, Neutral and Positive.
And apply this function and create another feature in our Data Frame called Score.
Here is our Data Frame with our Tweets, Subjectivity, Polarity and Score for all our Tweets.
Let’s now take all positive tweets and calculate the percentage of positive tweets from all the tweets in our Data Frame.
57.446808510638306 % of positive tweets
We can now visualise Positive, Negative, Neutral tweets using Matplotlib.
We can see how Negative, Neutral and Positive tweets spread on this account.
We can also visualise the same information by displaying the exact values of subjectivity and polarity on the graph.
We can also calculate the percentage of objective tweets.
27.659574468085108 % of objective tweets
At the end, we can also generate a word cloud to see the themes and most common words used in the tweets we were analysing.
Doing sentiment analysis of the tweets enabled us to calculate numerical values of subjectivity and polarity.
This could help us to understand better this Twitter account in terms of the language that is being used.
Combining this with additional information about likes and comments can be very useful from marketing point of view and can enable us to find some correlation between subjectivity, polarity and the engagement our the users for a specified Twitter account.
We encourage you to experiment more with this example and come up with some more ideas how this can be used in practice.
If you would like to learn more and experiment with Python and Data Science you can look at another of my articles Analysing Pharmaceutical Sales Data in Python, Introduction to Computer Vision with MNIST and Image Face Recognition in Python.
To consolidate your knowledge consider completing the task again from the beginning without looking at the code examples and see what results you will get. This is an excellent thing to do to solidify your knowledge.
Full Python code in Jupyter Notebook is available on GitHub:https://github.com/pjonline/Basic-Data-Science-Projects/tree/master/8-Twitter-Sentiment-Analysis
Happy coding!
Not subscribing to Medium yet? Consider signing up to become a Medium member. It’s only $5 a month and it will give you unlimited access to all stories on Medium. Subscribing to Medium supports me and other writers on Medium.