Sentiment Analysis on Twitter Dataset — Positive, Negative, Neutral Clustering
By: Nikhil Ahuja
Tags: Clustering, Machine Learning, Sentiment Analysis, Twitter, Data Science

Abstract

Text Analytics is the process of converting unstructured text data into meaningful insights to measure customer opinion, product reviews, sentiment analysis, customer feedback. Our domain of expertise in this solution is driven towards sentiment analysis.

A sentiment analysis system for text analysis combines natural language processing and machine learning techniques to assign weighted sentiment scores to the entities, topics, themes and categories within a sentence or phrase.

In this blog, we will be trying to do sentiment analysis on Twitter dataset and categorizing them into positive, negative and neutral behaviour of people.

Architecture

The diagram shows the architecture design and approach that we will be following for our capstone project. The solution is divided broadly into the following categories.

Data Collection/Retrieval Data Generation Data Preprocessing Data Exploratory Analysis Data Model building Data Clustering

Data Collection/Retrieval

In this step we have collected the data from public twitter dataset which are csv files with a variety of tweets information. Also to give it an extra edge we have implemented integration with twitter using its REST API using which we can collect streaming real time data.

We have used Tweepy a highly used python library for getting the tweets on a specific search topic. Below are the details of the API Spec

Twitter open source search API:

Request URL: https://api.twitter.com/1.1/search/tweets.json

Data Preprocessing

This is one of the most crucial and important exercise that is performed on the final dataset. As we would be getting raw data, it is expected the data quality will be very low, in short NOISY DATA. During our research on techniques used to clean the data we have decided to use some extremely common and highly accurate techniques to clean the data.

Lets see them in detail.

Data Filtering

Lower Case conversion — The first pre-processing step which we will do is transform our tweets into lower case. This avoids having multiple copies of the same words.

Removing Punctuations — As it doesn’t add any extra information while treating text data. Therefore removing all instances of it will help us reduce the size of the training data.

Stop Words Removal — Commonly occurring words)should be removed from the text data.

Rare Words Removal — Remove rarely occurring words from the text. Because they’re so rare, the association between them and other words is dominated by noise. You can replace rare words with a more general form and then this will have higher counts.

Spelling correction — We’ve all seen tweets with a plethora of spelling mistakes. Our timelines are often filled with hasty sent tweets that are barely legible at times. For this we would be using TEXT BLOB python library.

Tokenization — Tokenization refers to dividing the text into a sequence of words or sentences.

Lemmatization — It converts the word into its root word, rather than just stripping the suffices. It makes use of the vocabulary and does a morphological analysis to obtain the root word.

Data Association

It’s an additional step in data pre processing that will help to build our model more accurately. This step involves extracting features from the dataset using some of the most widely used NLP techniques.

N — Grams — N-grams are the combination of multiple words used together. Ngrams with N=1 are called unigrams. Similarly, bigrams (N=2), trigrams (N=3) and so on can also be used.

Unigrams do not usually contain as much information as compared to bigrams and trigrams. The basic principle behind n-grams is that they capture the language structure, like what letter or word is likely to follow the given one. The longer the n-gram (the higher the n), the more context you have to work with. Optimum length really depends on the application — if your n-grams are too short, you may fail to capture important differences. On the other hand, if they are too long, you may fail to capture the “general knowledge” and only stick to particular cases.

TF_IDF — It works by penalizing these common words by assigning them lower weights while giving importance to words like Messi in a particular document.

Bag of Words — the representation of text which describes the presence of words within the text data. The intuition behind this is that two similar text fields will contain similar kind of words, and will therefore have a similar bag of words. Further, that from the text alone we can learn something about the meaning of the document

Modelling

After all the data filtering and association process, we will implement the Sentimental Analysis on the cleaned tweets.

First, we will implement the rule-based sentiment analysis approach where we will use vaderSentiment package to label the tweets as positive, negative and neutral.Vader Sentiment is a lexicon based analysis tool which does behaviour analysis on social media content and works well on special characters like emojis, punctuation, etc. Once we have the labelled data, we will implement the TF-IDF as before implementing clustering we need to create the tf-idf matrix. TF-IDF refers to Term Frequency–Inverse Document Frequency and it is a product of two statistics that reflect how important a word is to a document in a collection or corpus: K-Means — We will then implement the k-means clustering model with 3 as the number of clusters.

You can refer to github link for the the code.