Yet Another Twitter Sentiment Analysis Part 1 — tackling class imbalance | by Ricky Kim | Towards Data Science

I finished an 11-part series blog posts on Twitter sentiment analysis not long ago. Why do I want to do the sentiment analysis again? I wanted to extend further and run sentiment analysis on real retrieved tweets. And there are other limits to my previous sentiment analysis project.
The project stopped at the final trained model and lacks application of the model to retrieved tweetsThe model was trained on only positive and negative class, so it lacks the ability to predict a neutral class
The project stopped at the final trained model and lacks application of the model to retrieved tweets
The model was trained on only positive and negative class, so it lacks the ability to predict a neutral class
Regarding neutral class, it might be possible to set a threshold value for negative, neutral, positive class, and map the final output probability value to one of three classes, but I wanted to train a model with training data, which has three sentiment classes: negative, neutral, positive.
Since I already wrote quite a lengthy series on NLP, sentiment analysis, if a concept was already covered in my previous posts, I won’t go into the detailed explanation. And also the main data visualisation will be with retrieved tweets, and I won’t go through extensive data visualisation with the data I use for training and testing a model.
*In addition to short code blocks I will attach, you can find the link for the whole Jupyter Notebook at the end of this post.
In order to train my sentiment classifier, I need a dataset which meets conditions below.
Preferably tweets text data with annotated sentiment label
with 3 sentiment classes: negative, neutral, positive
big enough to train a model
While googling to find a good data source, I learned about renowned NLP competition called SemEval. “SemEval (Semantic Evaluation) is an ongoing series of evaluations of computational semantic analysis systems, organized under the umbrella of SIGLEX, the Special Interest Group on the Lexicon of the Association for Computational Linguistics.”
You might have already heard of this if you’re interested in NLP. Highly-skilled teams from all around the world compete on a couple of tasks such as “semantic textual similarity”, “multilingual semantic word similarity”, etc. One of the competition tasks is the Twitter sentiment analysis. It also has a couple of subtasks, but what I would want to focus on is “Subtask A. : Message Polarity Classification: Given a message, classify whether the message is of positive, negative, or neutral sentiment”.
Luckily the dataset they provide for the competition is available to download. The training data consists of SemEval’s previous training and test data. What’s even better is they provide test data, and all the teams who participated in the competition are scored with the same test data. This means I can compare my model performance with 2017 participants in SemEval.
I first downloaded full training data for SemEval 2017 Task 4.
There are 11 txt files in total, spanning from SemEval 2013 to SemEval 2016. While trying to read the files into a Pandas dataframe, I found two files cannot be properly loaded as tsv file. It seems like there are some entries not properly tab-separated, so end up as a chunk of 10 or more tweets stuck together. I could have tried retrieving them with tweet ID provided, but I decided to first ignore these two files, and make up a training set with only 9 txt files.
import pandas as pd  import numpy as npimport matplotlib.pyplot as pltplt.style.use('fivethirtyeight')%matplotlib inline%config InlineBackend.figure_format = 'retina'
Once I import basic dependencies, I’ll read the data to a Pandas dataframe.
import globpath ='Subtask_A/'all_files = glob.glob(path + "/twitter*.txt")frame = pd.DataFrame()list_ = []for file_ in all_files:    df = pd.read_csv(file_,index_col=None, sep='\t', header=None, names=['id','sentiment','text','to_delete'])    list_.append(df.iloc[:,:-1])df = pd.concat(list_)df = df.drop_duplicates()df = df.reset_index(drop=True)df.tail()
The dataset looks fairly simple with individual tweet ID, sentiment label, and tweet text.
df.info()
There are total 41,705 tweets. As another sanity check, let’s take a look at how many words are there in each tweet.
df['token_length'] = [len(x.split(" ")) for x in df.text]max(df.token_length)
df.loc[df.token_length.idxmax(),'text']
OK, the token length looks fine, and the tweet for maximum token length seems like a properly parsed tweet. Let’s take a look at the class distribution of the data.
df.sentiment.value_counts()
The data is not well balanced, and negative class has the least number of data entries with 6,485, and the neutral class has the most data with 19,466 entries. I want to rebalance the data so that I will have a balanced dataset at least for training. I will deal with this after I define the cleaning function.
Data cleaning process is similar to my previous project, but this time I added a long list of contraction to expand most of the contracted form to its original form such as “don’t” to “do not”. And this time, instead of Regex, I used Spacy to parse the documents, and filtered numbers, URL, punctuation, etc. Below are the steps I took to clean the tweets.
Decoding: unicode_escape for extra “\” before unicode character, then unidecodeApostrophe handled: there are two characters people use for contraction. “’”(apostrophe) and “‘“(single quote). If these two symbols are both used for contraction, it will be difficult to detect and properly map the right expanded form. So any “’”(apostrophe) is changed to “‘“(single quote)Contraction check: check if there’s any contracted form, and replace it with its original formParsing: done with SpacyFiltering punctuation, white space, numbers, URL using Spacy methods while keeping the text content of hashtag intactRemoved @mentionLemmatize: lemmatized each token using Spacy method ‘.lemma_’. Pronouns are kept as they are since Spacy lemmatizer transforms every pronoun to “-PRON-”Special character removalSingle syllable token removalSpell correction: it is a simple spell correction dealing with repeated characters such as “sooooo goooood”. If the same character is repeated more than two times, it shortens the repetition to two. For example “sooooo goooood” will be transformed as “soo good”. This is not a perfect solution since even after correction, in case of “soo”, it is not a correct spelling. But at least it will help to reduce feature space by making “sooo”, “soooo”, “sooooo” to the same word “soo”
Decoding: unicode_escape for extra “\” before unicode character, then unidecode
Apostrophe handled: there are two characters people use for contraction. “’”(apostrophe) and “‘“(single quote). If these two symbols are both used for contraction, it will be difficult to detect and properly map the right expanded form. So any “’”(apostrophe) is changed to “‘“(single quote)
Contraction check: check if there’s any contracted form, and replace it with its original form
Parsing: done with Spacy
Filtering punctuation, white space, numbers, URL using Spacy methods while keeping the text content of hashtag intact
Removed @mention
Lemmatize: lemmatized each token using Spacy method ‘.lemma_’. Pronouns are kept as they are since Spacy lemmatizer transforms every pronoun to “-PRON-”
Special character removal
Single syllable token removal
Spell correction: it is a simple spell correction dealing with repeated characters such as “sooooo goooood”. If the same character is repeated more than two times, it shortens the repetition to two. For example “sooooo goooood” will be transformed as “soo good”. This is not a perfect solution since even after correction, in case of “soo”, it is not a correct spelling. But at least it will help to reduce feature space by making “sooo”, “soooo”, “sooooo” to the same word “soo”
OK now let’s see how this custom cleaner works with tweets.
pd.set_option('display.max_colwidth', -1)df.text[:10]
[spacy_cleaner(t) for t in df.text[:10]]
It looks like it’s doing what I intended it to do. I’ll clean the “text” column and create a new column called “clean_text”.
df['clean_text'] = [spacy_cleaner(t) for t in df.text]
By running the cleaning function I can see it encountered some “invalid escape sequence”. Let’s see what these are.
for i,t in enumerate(df.text):    if '\m' in t:        print(i,t)
The tweets that contain ‘\m’ were actually containing an emoticon ‘\m/’ I didn’t know about this until I googled it. Apparently ‘\m/’ stands for the horn sign you make with your hand. This hand sign is popular in metal music. Anyway, this is just a warning and it is not an error. Let’s see how the cleaner deals with this.
df.text[2064]
spacy_cleaner(df.text[2064])
Again it seems like to be doing what I intended it to do. So far so good.
“The class imbalance problem typically occurs when, in a classification problem, there are many more instances of some classes than others. In such cases, standard classifiers tend to be overwhelmed by the large classes and ignore the small ones.”
As I have already realised, the training data is not perfectly balanced, ‘neutral’ class has 3 times more data than ‘negative’ class, and ‘positive’ class has around 2.4 times more data than ‘negative’ class. I will try fitting a model with three different data; oversampled, downsampled, original, to see how different sampling techniques affect the learning of a classifier.
The simple default classifier I’ll use to compare performances of different datasets will be the logistic regression. From my previous sentiment analysis project, I learned that Tf-Idf with Logistic Regression is a pretty powerful combination. Before I apply any other more complex models such as ANN, CNN, RNN etc, the performances with logistic regression will hopefully give me a good idea of which data sampling methods I should choose. If you want to know more about Tf-Idf, and how it extracts features from text, you can check my old post, “Another Twitter Sentiment Analysis with Python-Part5”.
In terms of validation, I will use K-Fold Cross Validation. In my previous project, I split the data into three; training, validation, test, and all the parameter tuning was done with reserved validation set and finally applied the model to the test set. Considering that I had more than 1 million data for training, this kind of validation set approach was acceptable. But this time, the data I have is much smaller (around 40,000 tweets), and by leaving out validation set from the data we might leave out interesting information about data.
from sklearn.pipeline import Pipelineoriginal_pipeline = Pipeline([    ('vectorizer', tvec),    ('classifier', lr)])lr_cv(5, df.clean_text, df.sentiment, original_pipeline, 'macro')
With data as it is without any resampling, we can see that the precision is higher than the recall. If you want to know more about precision and recall, you can check my old post, “Another Twitter sentiment analysis with Python — Part4”.
If we take a closer look at the result from each fold, we can also see that the recall for the negative class is quite low around 28~30%, while the precisions for the negative class are high as 61~65%. This means the classifier is very picky and does not think many things are negative. All the text it classifies as negative is 61~65% of the time really negative. However, it also misses a lot of actual negative class, because it is so very picky. We have a low recall, but a very high precision. The intuition behind this precision and recall has been taken from a Medium blog post by Andreas Klintberg.
There is a very useful Python package called “imbalanced-learn”, which helps you deal with class imbalance issues, it is compatible with Scikit Learn, and easy to implement.
Within imbalanced-learn, there are different techniques you can use for oversampling. I will use below two.
RandomOverSamplerSMOTE (Synthetic Minority Over-Sampling Technique)
RandomOverSampler
SMOTE (Synthetic Minority Over-Sampling Technique)
There is one more point to consider if you are cross-validating with oversampled data. Oversampling the minority class can result in overfitting problems if we oversample before cross-validating. Why is that so? Because by oversampling before cross validation split, you are leaking the information of validation data already to your training set. As they say “What has been seen, cannot be unseen.”
If you want more detailed explanation, I recommend this Youtube video “Machine Learning — Over-& Undersampling — Python/ Scikit/ Scikit-Imblearn”
Luckily cross-validation function I defined above as “lr_cv()” will fit the pipeline only with the training set split after cross-validation split, thus it is not leaking any information of validation set to the model.
Random over-sampling is simply a process of repeating some samples of the minority class and balance the number of samples between classes in the dataset.
from imblearn.pipeline import make_pipelinefrom imblearn.over_sampling import ADASYN, SMOTE, RandomOverSamplerROS_pipeline = make_pipeline(tvec, RandomOverSampler(random_state=777),lr)SMOTE_pipeline = make_pipeline(tvec, SMOTE(random_state=777),lr)
Before we fit each pipeline, let’s see what the RadomOverSampler does. In order to make it easier to see I defined some toy text data below, and the target sentiment value for each text.
sent1 = "I love dogs"sent2 = "I don't like dogs"sent3 = "I adore cats"sent4 = "I hate spiders"sent5 = "I like dogs"testing_text = pd.Series([sent1, sent2, sent3, sent4, sent5])testing_target = pd.Series([1,0,1,0,1])
My toy data has 5 entries in total, and the target sentiments are three positives and two negatives. In order to be balanced, this toy data needs one more entry of negative class.
One thing is over sampler won’t be able to handle raw text data. It has to be transformed into a feature space for over sampler to work. I’ll first fit TfidfVectorizer, and oversample using Tf-Idf representation of texts.
tv = TfidfVectorizer(stop_words=None, max_features=100000)testing_tfidf = tv.fit_transform(testing_text)ros = RandomOverSampler(random_state=777)X_ROS, y_ROS = ros.fit_sample(testing_tfidf, testing_target)pd.DataFrame(testing_tfidf.todense(), columns=tv.get_feature_names())
pd.DataFrame(X_ROS.todense(), columns=tv.get_feature_names())
By running RandomOverSampler, now we have one more entry at the end. The last entry added by RandomOverSampler is exactly same as the fourth one (index number 3) from the top. RandomOverSampler simply repeats some entries of the minority class to balance the data. If we look at the target sentiments after RandomOverSampler, we can see that it has now a perfect balance between classes by adding on more entry of negative class.
y_ROS
lr_cv(5, df.clean_text, df.sentiment, ROS_pipeline, 'macro')
Compared to the model built with original imbalanced data, now the model behaves in opposite way. The precisions for the negative class are around 47~49%, but the recalls are way higher at 64~67%. Now we have a situation of high recall, low precision. What this means is that the classifier thinks a lot of things are negative. However, it also thinks a lot of non-negative texts are negative. So from our set of data we got a lot of texts classified as negative, many of them were in the set of actual negative, however, a lot of them were also non-negative.
But without resampling, the recall rate was as low as 28~30% for negative class, the precision rate for the negative class I get from oversampling is more robust at around 47~49%.
Another way to look at it is to look at the f1 score, which is the harmonic average of precision and recall. The original imbalanced data had 66.51% accuracy and 60.01% F1 score. However with oversampling, we get a slightly lower accuracy of 65.95%, but a much higher F1 score of 64.18%
SMOTE is an over-sampling approach in which the minority class is over-sampled by creating “synthetic” examples rather than by over-sampling with replacement.
According to the original research paper “SMOTE: Synthetic Minority Over-sampling Technique” (Chawla et al., 2002), “synthetic samples are generated in the following way: Take the difference between the feature vector (sample) under consideration and its nearest neighbour. Multiply this difference by a random number between 0 and 1, and add it to the feature vector under consideration. This causes the selection of a random point along the line segment between two specific features. This approach effectively forces the decision region of the minority class to become more general.” What this means is that when SMOTE creates a new synthetic data, it will choose one data to copy, and look at its k nearest neighbours. Then, on feature space, it will create random values in feature space that is between the original sample and its neighbours.
Once you see the example with the toy data, it will become clearer.
smt = SMOTE(random_state=777, k_neighbors=1)X_SMOTE, y_SMOTE = smt.fit_sample(testing_tfidf, testing_target)pd.DataFrame(X_SMOTE.todense(), columns=tv.get_feature_names())
The last entry is the data created by SMOTE. To make it easier to see, let’s see only the negative class.
pd.DataFrame(X_SMOTE.todense()[y_SMOTE == 0], columns=tv.get_feature_names())
The top two entries are original data, and the one on the bottom is synthetic data. You can see it didn’t just repeat original data. Instead, the Tf-Idf values are created by taking random values between the top two original data. As you can see, if the Tf-Idf values for both original data are 0, then synthetic data also has 0 for those features, such as “adore”, “cactus”, “cats”, because if two values are the same there are no random values between them. I specifically defined k_neighbors as 1 for this toy data, since there are only two entries of negative class, if SMOTE chooses one to copy, then only one other negative entry left as a neighbour.
Now let’s fit the SMOTE pipeline to see how it affects performance.
lr_cv(5, df.clean_text, df.sentiment, SMOTE_pipeline, 'macro')
SMOTE sampling seems to have a slightly higher accuracy and F1 score compared to random oversampling. With the results so far, it seems like choosing SMOTE oversampling is preferable over original or random oversampling.
How about downsampling. If we oversample the minority class in the above oversampling, with downsampling, we try to reduce the data of majority class, so that the data classes are balanced.
from imblearn.under_sampling import NearMiss, RandomUnderSamplerRUS_pipeline = make_pipeline(tvec, RandomUnderSampler(random_state=777),lr)NM1_pipeline = make_pipeline(tvec, NearMiss(ratio='not minority',random_state=777, version = 1),lr)NM2_pipeline = make_pipeline(tvec, NearMiss(ratio='not minority',random_state=777, version = 2),lr)NM3_pipeline = make_pipeline(tvec, NearMiss(ratio=nm3_dict,random_state=777, version = 3, n_neighbors_ver3=4),lr)
Again, before we run the pipeline, let’s apply this to the toy data to see what it does.
rus = RandomUnderSampler(random_state=777)X_RUS, y_RUS = rus.fit_sample(testing_tfidf, testing_target)pd.DataFrame(X_RUS.todense(), columns=tv.get_feature_names())
pd.DataFrame(testing_tfidf.todense(), columns=tv.get_feature_names())
Compared with the original imbalanced data, we can see that downsampled data has one less entry, which is the last entry of the original data belonging to the positive class. RandomUnderSampler reduces the majority class by randomly removing data from the majority class.
lr_cv(5, df.clean_text, df.sentiment, RUS_pipeline, 'macro')
Now the accuracy and the F1 score has significantly dropped. But the characteristic of low precision and high recall is as same as oversampled data. Only its overall performance dropped.
According to the documentation of “imbalanced-learn”, “NearMiss adds some heuristic rules to select samples. NearMiss implements 3 different types of heuristic which can be selected with the parameter version. NearMiss heuristic rules are based on nearest neighbors algorithm.”
There is also a good paper on resampling techniques. “Survey of resampling techniques for improving classification performance in unbalanced datasets” (Ajinkya More, 2016)
I borrowed the explanation of three different versions of NearMiss from More’s paper.
In NearMiss-1, those points from majority class are retained whose mean distance to the k nearest points in minority class is lowest. Which means it will keep the points of majority class that’s similar to the minority class.
nm = NearMiss(ratio='not minority',random_state=777, version=1, n_neighbors=1)X_nm, y_nm = nm.fit_sample(testing_tfidf, testing_target)pd.DataFrame(X_nm.todense(), columns=tv.get_feature_names())
pd.DataFrame(testing_tfidf.todense(), columns=tv.get_feature_names())
We can see that NearMiss-1 has eliminated the entry for the text “I adore cats”, which makes sense because both words “adore” and “cats” are only appeared in this entry, so makes it the most different from minority class in terms of Tf-Idf representation in feature space.
lr_cv(5, df.clean_text, df.sentiment, NM1_pipeline, 'macro')
It seems like both the accuracy and F1 score got worse than random undersampling.
In contrast to NearMiss-1, NearMiss-2 keeps those points from the majority class whose mean distance to the k farthest points in minority class is lowest. In other words, it will keep the points of majority class that’s most different to the minority class.
nm = NearMiss(ratio='not minority',random_state=777, version=2, n_neighbors=1)X_nm, y_nm = nm.fit_sample(testing_tfidf, testing_target)pd.DataFrame(X_nm.todense(), columns=tv.get_feature_names())
pd.DataFrame(testing_tfidf.todense(), columns=tv.get_feature_names())
Now we can see that NearMiss-2 has eliminated the entry for the text “I like dogs”, which again makes sense because we also have a negative entry “I don’t like dogs”. Two entries are in different classes but they share two same tokens “like” and “dogs”.
lr_cv(5, df.clean_text, df.sentiment, NM2_pipeline, 'macro')
Both accuracy and F1 score got even lower compared to NearMiss-1. And we can also see that all the metrics fluctuate from fold to fold quite a lot.
The final NearMiss variant, NearMiss-3 selects k nearest neighbours in majority class for every point in the minority class. In this case, the undersampling ratio is directly controlled by k. For example, if we set k to be 4, then NearMiss-3 will choose 4 nearest neighbours of every minority class entry.
Then we’ll end up with either more or fewer samples of majority class than minority class depending on n neighbours we set. For example, with my dataset, if I run NearMiss-3 with default n_neighbors_ver3 of 3, it will complain and the number of neutral class(which is majority class in my dataset) will be smaller than negative class(which is minority class in my dataset). So I explicitly set n_neighbors_ver3 to be 4, so that I’ll have enough majority class data at least the same number as the minority class.
One thing I’m not completely sure is that what kind of filtering it applies when all the data selected with n_neighbors_ver3 parameter is more than the minority class. As you will see below, after applying NearMiss-3, the dataset is perfectly balanced. However, if the algorithm simply chooses the nearest neighbour according to the n_neighbors_ver3 parameter, I doubt that it will end up with the exact same number of entries for each class.
lr_cv(5, df.clean_text, df.sentiment, NM3_pipeline, 'macro')
NearMiss-3 produced the most robust result within NearMiss family, but slightly lower than RandomUnderSampling.
from collections import Counternm3 = NearMiss(ratio='not minority',random_state=777, version=3, n_neighbors_ver3=4)tvec = TfidfVectorizer(stop_words=None, max_features=100000, ngram_range=(1, 3))df_tfidf = tvec.fit_transform(df.clean_text)X_res, y_res = nm3.fit_sample(df_tfidf, df.sentiment)print('Distribution before NearMiss-3: {}'.format(Counter(df.sentiment)))print('Distribution after NearMiss-3: {}'.format(Counter(y_res)))
5-fold cross validation result (classifier used for validation: logistic regression with default setting)
Based on the above result, the sampling technique I’ll be using for the next post will be SMOTE. In the next post, I will try different classifiers with SMOTE oversampled data.
Thank you for reading and you can find the Jupyter Notebook from the below link: