Using NLP to determine the Sentiments of Lagos-tweets
By: Samson Afolabi
Tags: NLP, Data Science, Plotly, Tweepy, Machine Learning

Africa’s Most Populous City. Lagos, Nigeria.

Everyone talks about Lagos being the city where dreams are fulfilled and all aspirations are graciously met. These thoughts have brought about the former capital being overwhelmingly flooded with its 20 Local Government Areas being currently inhabited by over 16 million people.

I performed a sentiment analysis of tweets on twitter originating from this city using Natural Language Processing tools. In this article, I will walk you through my process. The link to the full code on Github is available at the bottom.

There is no data science without data. First, we need to get the data. To do this we need to open a twitter developer’s account, through this link. You might have to answer some questions before the account is created. This link also explains more.

Twitter Developers account home page

After the account has been created, we will use the our access codes(which I advice you never share online) to stream on twitter. For my work, I streamed for tweets over a period of 4 days generating about 38,500 tweets. Installing a tweepy library allows us to stream on twitter. Note also that, the API-search only gives access to 1000 tweets every 15 minutes. Another option is to use the API-stream which generates randomly 10% of tweets. Read more here.

Data Generator

The wildcard in the image above returns all tweets while the geocode ensures we get tweets directly from Lagos only.

Now that we have our data, we move next to do some analysis. Here we will need the TextBlob,newspaper3k and nltk libraries for our analysis.

First, we need to clean up the tweets. This involves removing URLS, turning the tweets into lower case and splitting the tweets into words. Using nltk, we can create a counter. We may also include stopwords. Stopwords are common words, which are not interesting to us. In my case, I added some Nigerian pidgin stopwords, since some of the tweets were in pidgin.

Clean up and word counter

From this we may also generate a Word Cloud using the word cloud library to display the frequently used words. Here is the image I generated from that.

word cloud image of frequently used words

Now, the Image above reveals that the words like,don’t, know,one,love,people were used more during the period these data were collected.

We might also generate a pie chart using plotly to visualize the sentiment analysis of tweets. Here we use the TextBlob library. This returns a float between -1 and 1 indicating the severity of negativity to positivity. The chart shows a lot of neutral emotions. The measure of negativity or positivity is determined by many pretrained words. So for example ‘I am sad’ or ‘I am disgusted’ could be -0.2 and ‘I am very sad and annoyed’ could be -0.6. Basically, the algorithm reads through the tweets and tries to determine the mood of the writer from the words used.

Pie chart showing sentiment analysis

We can also use Textblob to return to us how subjective or objective these tweets. For my example, I got this. Indicating that Lagosians are very objective people.

Plotly Pie Chart showing measure of Lagos Twitter Subjectivity

We may also try out the plotly histogram to show us the counts alongside.

Plotly Histogram showing measure of Lagos Twitter Subjectivity

While the results are quite interesting, the non-availability of Pidgin stopwords was a little bit of headache as this had to be manually inputted. I really do hope that in NLP advancements, more African languages will be considered. Link to the full code on github is here.