Re-sampling Imbalanced Training Corpus for Sentiment Analysis
By: Mustafa Abusalah
Tags: Machine Learning, Sentiment Analysis, Corpora, Imbalanced Data, Data Science

In this article I will discuss techniques for over-sampling, under-sampling and combination between both in order to balance an imbalanced training data-set to solve an online Twitter Sentimental Analysis challenge. The challenge aims to detect hate and abuse speech in testing corpus that is manually judged. Challenge including training and testing data is available here.

Re-sampling Imbalanced Data-set will definitely improve the Classification

The training corpus contains tweets judged manually and those which are considered abuse tweets are labeled 1 and other normal tweets are labeled 0.

Let us first import necessary libraries to carry on:

#import the necessary libraries for dataset preparation, feature engineering, model training

from sklearn import model_selection, preprocessing, metrics, linear_model, svm

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

from imblearn.over_sampling import BorderlineSMOTE, SMOTE, ADASYN, SMOTENC, RandomOverSampler

from imblearn.under_sampling import (RandomUnderSampler,

NearMiss,

InstanceHardnessThreshold,

CondensedNearestNeighbour,

EditedNearestNeighbours,

RepeatedEditedNearestNeighbours,

AllKNN,

NeighbourhoodCleaningRule,

OneSidedSelection,

TomekLinks)

from imblearn.combine import SMOTEENN, SMOTETomek

from imblearn.pipeline import make_pipeline

import pandas as pd, numpy, string

from nltk.tokenize import WordPunctTokenizer

from nltk.stem import PorterStemmer

from bs4 import BeautifulSoup

#Remove Special Charactors

import re

from nltk.tokenize import WordPunctTokenizer

from nltk.stem import PorterStemmer

from bs4 import BeautifulSoup

Now we need to import the training and testing csv files and have a partial look at the text:

#Import Training and Testing Data

train = pd.read_csv('train.csv')

print("Training Set:"% train.columns, train.shape, len(train))

test = pd.read_csv('test_tweets.csv')

print("Test Set:"% test.columns, test.shape, len(test)) Training Set: (31962, 3) 31962

Test Set: (17197, 2) 17197

The training set consists of 32K tweets and the testing set consists of 17K tweets. Let is explore the top records of both sets:

train.head()

id label tweet

1 0 @user when a father is dysfunctional and is 2 0 @user @user thanks for #lyft credit i can't 3 0 bihday your majesty

4 0 #model i love u take with u all the time i 5 0 factsguide: society now #motivation\ test.head()

id tweet

31963 #studiolife #aislife #requires #passion #dedic 31964 @user #white #supremacists want everyone to s... 31965 safe ways to heal your #acne!! #altwaystohe... 31966 is the hp and the cursed child book up for res. 31967 3rd #bihday to my amazing, hilarious #nephew...

If you notice that the corpus replaced the mentions of usernames to @ user also the text includes # tags, special characters and numbers. Let us see the percentage of abuse tweets to normal ones in the training corpus:

#Percentage of Positive/Negative

print("Positive: ", train.label.value_counts()[0]/len(train)*100,"%")

print("Negative: ", train.label.value_counts()[1]/len(train)*100,"%")

The negative tweets are 93% of the data set and the negative is 7%:

Positive: 92.98542018647143 %

Negative: 7.014579813528565 %

This means we have imbalanced training data set that will definitely affect the accuracy of the predictions. To make this argument valid we will first predict negative and positive tweets by splitting the judged training data and then compare the predicted labels to the human judged labels.But before we start let us clean the tweets from numbers, html/xml tags , special characters, white spaces also stem the words, this is all done in the below block of code:

porter=PorterStemmer()

tok = WordPunctTokenizer()

pat1 = r'@[A-Za-z0-9]+'

pat2 = r'https?://[A-Za-z0-9./]+'

combined_pat = r'|'.join((pat1, pat2)) def tweet_cleaner(text):

soup = BeautifulSoup(text, 'lxml')

souped = soup.get_text()

stripped = re.sub(combined_pat, '', souped)

try:

clean = stripped.decode("utf-8-sig").replace(u"\ufffd", "?")

except:

clean = stripped

letters_only = re.sub("[^a-zA-Z]", " ", clean)

lower_case = letters_only.lower()

# During the letters_only process two lines above, it has created unnecessay white spaces,

# I will tokenize and join together to remove unneccessary white spaces

words = tok.tokenize(lower_case)

#Stemming

stem_sentence=[]

for word in words:

stem_sentence.append(porter.stem(word))

stem_sentence.append(" ")

words="".join(stem_sentence).strip()

return words

nums = [0,len(train)]

clean_tweet_texts = []

for i in range(nums[0],nums[1]):

clean_tweet_texts.append(tweet_cleaner(train['tweet'][i]))

nums = [0,len(test)]

test_tweet_texts = []

for i in range(nums[0],nums[1]):

test_tweet_texts.append(tweet_cleaner(test['tweet'][i]))

train_clean = pd.DataFrame(clean_tweet_texts,columns=['tweet'])

train_clean['label'] = train.label

train_clean['id'] = train.id

test_clean = pd.DataFrame(test_tweet_texts,columns=['tweet'])

test_clean['id'] = test.id

Let us split the Training Data set into training and testing:

#split the dataset into training and validation datasets

train_x, valid_x, train_y, valid_y = model_selection.train_test_split(train_clean['tweet'],train_clean['label'])

#label encode the target variable

encoder = preprocessing.LabelEncoder()

train_y = encoder.fit_transform(train_y)

valid_y = encoder.fit_transform(valid_y)

In the previous code kindly note that I had to transform the label column into an array as I will now convert the tweets into a weighted vector of terms using term frequency TF and inverse document frequency IDF, read more about them here.

We have more than one option to value the terms importance, one of the best methods is to use TFIDF, you could try to use bag of words BOW for the sake of argument.

# word level tf-idf

tfidf_vect = TfidfVectorizer(analyzer='word', token_pattern=r'\w{1,}', max_features=100000)

tfidf_vect.fit(train_clean['tweet'])

xtrain_tfidf = tfidf_vect.transform(train_x)

xvalid_tfidf = tfidf_vect.transform(valid_x)

Evaluation Metrics

Accuracy is a great measure but only when you have balanced data set where labels 0 and 1 are decomposed almost equally. Therefore, we need to look at other parameters to evaluate the performance of the model. In our model, we could got high accuracy as 0s are about 93% of the corpus.

For this model I used F1 score — which is the weighted average of Precision and Recall.

F1 Score = 2*(Recall * Precision) / (Recall + Precision)

More details about evaluation techniques can be found here.

#Return the f1 Score

def train_model(classifier, feature_vector_train, label, feature_vector_valid):

# fit the training dataset on the classifier

classifier.fit(feature_vector_train, label)



# predict the labels on validation dataset

predictions = classifier.predict(feature_vector_valid) return metrics.f1_score(valid_y,predictions)

Classifiers

I’m going to use linear classifier (Logistic Regression LR) and Support Vector Machine (SVM) Classifier to predict the negative and positive tweets. LR measures the relationship between the categorical dependent variable and one or more independent variables by estimating probabilities using a logistic/sigmoid function.

SVM algorithm finds the hyperplane that has the maximum margin in an N-dimensional space (N — the number of features) that distinctly classifies the data points. More about both classifiers can be found here.

Let us see the baseline result (without re-sampling):

accuracyORIGINAL = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),xtrain_tfidf, train_y, xvalid_tfidf)

print ("LR Baseline, WordLevel TFIDF: ", accuracyORIGINAL)

accuracyORIGINAL = train_model(svm.LinearSVC(), xtrain_tfidf, train_y, xvalid_tfidf)

print ("SVM Baseline, WordLevel TFIDF: ", accuracyORIGINAL) #LR Baseline, WordLevel TFIDF: 0.534131736526946

#SVM Baseline, WordLevel TFIDF: 0.6991701244813279

Re-sampling the Training Corpus

To balance the training data set there are many methods, we will discuss in this article: under-sampling, over-sampling and combine both over and under sampling.

Under-sampling balances the data set by reducing the size of the majority class. This method is used when quantity of data is sufficient. By keeping all samples in the minority class and randomly selecting an equal number of samples in the majority class. In our model there is a huge gap between the positive and negative tweets, only 7% of the tweets are negative so under-sampling might not produce great results. We will see some scenarios in this article.

Over-sampling is used when the quantity of data is insufficient. It attempts to balance the data-set by increasing the size of minority samples. Rather than getting rid of majority samples, new minority samples are generated by using: repetition, bootstrapping, SMOTE (Synthetic Minority Over-Sampling Technique) or ADASYN (Adaptive synthetic sampling).

Combine both methods is another way to resample the data-set. Selecting re-sampling technique depends on the model itself, for our model, we will evaluate the three methods.

Random Over-sampling

The below code block uses Scikit learn over-sampling method to randomly repeat some minority samples and balance the number of samples between the data-set.

#Random Over Sampling

ros = RandomOverSampler(random_state=777)

ros_xtrain_tfidf, ros_train_y = ros.fit_sample(xtrain_tfidf, train_y) accuracyROS = train_model(linear_model.LogisticRegression(random_state=0, solver=’lbfgs’,multi_class=’multinomial’),ros_xtrain_tfidf, ros_train_y, xvalid_tfidf)

print (“LR ORIGINAL, WordLevel TFIDF: “, accuracyROS)

accuracyROS = train_model(svm.LinearSVC(),ros_xtrain_tfidf, ros_train_y, xvalid_tfidf)

print ("SVM ROS, WordLevel TFIDF: ", accuracyROS) #LR ROS, WordLevel TFIDF: 0.6822066822066821

#SVM ROS, WordLevel TFIDF: 0.6995744680851064

It is clear from the f1 score result that with this trivial approach is already less biased toward the majority class. But if we apply this to the real testing data would we achieve this result? You may try that yourself.

SMOTE Over-sampling

SMOTE is a Synthetic minority over-sampling approach in which the minority class is over-sampled by creating “synthetic” samples rather than creating new random minority samples from existing minority classes.

#SMOTE

sm = SMOTE(random_state=777, ratio = 1.0)

sm_xtrain_tfidf, sm_train_y = sm.fit_sample(xtrain_tfidf, train_y) accuracySMOTE = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),sm_xtrain_tfidf, sm_train_y, xvalid_tfidf)

print ("LR SMOTE, WordLevel TFIDF: ", accuracySMOTE)

accuracySMOTE = train_model(svm.LinearSVC(),sm_xtrain_tfidf, sm_train_y, xvalid_tfidf)

print ("SVC SMOTE, WordLevel TFIDF: ", accuracySMOTE) #LR SMOTE, WordLevel TFIDF: 0.6848436246992782

#SVC SMOTE, WordLevel TFIDF: 0.693288020390824

We can see that SMOTE scored a little better than ROS but this is for the Training/Testing data, but for the real testing data this might, try it using test_clean data-set.

ADASYN: Adaptive synthetic sampling (Over-sampling)

The advantage of ADASYN over SMOTE is the use a weighted distribution for different minority class samples according to their level of difficulty in learning, where more synthetic data is generated for minority class samples that are harder to learn compared to those minority samples that are easier to learn. More details available here.

#ADASYN

ad = ADASYN(random_state=777, ratio = 1.0)

ad_xtrain_tfidf, ad_train_y = ad.fit_sample(xtrain_tfidf, train_y)

accuracyADASYN = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),ad_xtrain_tfidf, ad_train_y, xvalid_tfidf)

print ("LR ADASYN, WordLevel TFIDF: ", accuracyADASYN)

accuracyADASYN = train_model(svm.LinearSVC(),ad_xtrain_tfidf, ad_train_y, xvalid_tfidf)

print ("SVM ADASYN, WordLevel TFIDF: ", accuracyADASYN) #LR ADASYN, WordLevel TFIDF: 0.6666666666666666

#SVM ADASYN, WordLevel TFIDF: 0.6818181818181819

Again the f1 score result was less than both previous approaches, still this is on training/testing corpus.

In the 3 previous over-sampling examples, the RandomOverSampler over-samples by duplicating some of the original samples of the minority class, while SMOTE and ADASYN generate new samples in the corpus by interpolation. However, the samples used to generate new synthetic samples differ. ADASYN approach focuses on generating samples next to the original samples which are wrongly classified using a k-Nearest Neighbors classifier (Hard to learn) while the basic implementation of SMOTE will not make any distinction between easy and hard to learn samples to be classified using the nearest neighbors rule.

It is worth mentioning that SMOTE might connect inliers and outliers, so this approach offers three additional options to generate samples. Those methods focus on samples near of the border of the optimal decision function and will generate samples in the opposite direction of the nearest neighbors class. I have experimented BorderlineSMOTE and SMOTENC.

SMOTE-NC uses SMOTE approach by synthesizing new minority samples but slightly change the way a new sample is generated by performing something specific for the categorical features. In fact, the categories of a new generated sample are decided by picking the most frequent category of the nearest neighbors present during the generation:

#SMOTENC

smnc = SMOTENC(categorical_features=[0, 2], random_state=0)

smnc_xtrain_tfidf, smnc_train_y = smnc.fit_sample(xtrain_tfidf, train_y)

accuracySMOTENC = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),smnc_xtrain_tfidf, smnc_train_y, xvalid_tfidf)

print ("LR SMOTENC, WordLevel TFIDF: ", accuracySMOTENC)

accuracySMOTENC = train_model(svm.LinearSVC(),smnc_xtrain_tfidf, smnc_train_y, xvalid_tfidf)

print ("SVM SMOTENC, WordLevel TFIDF: ", accuracySMOTENC) #LR SMOTENC, WordLevel TFIDF: 0.534131736526946

#SVM SMOTENC, WordLevel TFIDF: 0.6978193146417445

The borderline SMOTE will use the samples in danger (weakly judged) to generate new samples:

#Borderline SMOTE

bsm = BorderlineSMOTE()

bsm_xtrain_tfidf, bsm_train_y = bsm.fit_sample(xtrain_tfidf, train_y)

accuracyBSMOTE = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),bsm_xtrain_tfidf, bsm_train_y, xvalid_tfidf)

print ("LR Borderline SMOTE, WordLevel TFIDF: ", accuracyBSMOTE)

accuracyBSMOTE = train_model(svm.LinearSVC(),bsm_xtrain_tfidf, bsm_train_y, xvalid_tfidf)

print ("SVM Borderline SMOTE, WordLevel TFIDF: ", accuracyBSMOTE) #LR Borderline SMOTE, WordLevel TFIDF: 0.6799999999999999

#SVM Borderline SMOTE, WordLevel TFIDF: 0.6947368421052631

SMOTE-NC and Borderline SMOTE seemed to improve the performance of both classifiers, but highly improving the LR.

Random Under-sampling

Random under-sampling is a controlled under-sampling method, where the number of samples to be selected can be defined. This method randomly reduces the majority classes in order to make the balance, let us see the evaluation:

# Random Under Sampling

rus = RandomUnderSampler(random_state=0, replacement=True)

rus_xtrain_tfidf, rus_train_y = rus.fit_sample(xtrain_tfidf, train_y)

accuracyrus = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),rus_xtrain_tfidf, rus_train_y, xvalid_tfidf)

print ("LR RUS, WordLevel TFIDF: ", accuracyrus)

accuracyrus = train_model(svm.LinearSVC(),rus_xtrain_tfidf, rus_train_y, xvalid_tfidf)

print ("SVC RUS, WordLevel TFIDF: ", accuracyrus) #LR RUS, WordLevel TFIDF: 0.5045045045045046

#SVC RUS, WordLevel TFIDF: 0.5091093117408907

So the f1 score was less than all over-sampling methods used. Let us try something else.

NearMiss

NearMiss method is part of the imbalance learn library, NearMiss implements heuristic rules in order to select samples. It performs under-sampling of samples in the majority class based on their distance to other samples in the same class. This method uses three different variants:

NearMiss-1 retain samples from the majority class for which the average distance of the k ( tunable hyperparameter) nearest samples of the minority class is the smallest.

NearMiss-2 keeps the samples from majority class whose mean distance to the k furthest samples in the minority class is lowest.

NearMiss-3 selects k nearest neighbors in majority class for every sample in the minority class. So under-sampling ratio is directly controlled by k and is not separately tuned.

#NearMiss

for sampler in (NearMiss(version=1),NearMiss(version=2),NearMiss(version=3)):

nm_xtrain_tfidf, nm_train_y = sampler.fit_sample(xtrain_tfidf, train_y)

accuracysm = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),nm_xtrain_tfidf, nm_train_y, xvalid_tfidf)

print ("LR NearMiss(version= {0}), WordLevel TFIDF: ".format(sampler.version), accuracysm)



for sampler in (NearMiss(version=1),NearMiss(version=2),NearMiss(version=3)):

nm_xtrain_tfidf, nm_train_y = sampler.fit_sample(xtrain_tfidf, train_y)

accuracysm = train_model(svm.LinearSVC(),nm_xtrain_tfidf, nm_train_y, xvalid_tfidf)

print ("SVC NearMiss(version= {0}), WordLevel TFIDF: ".format(sampler.version), accuracysm) #LR NearMiss(version= 1), WordLevel TFIDF: 0.2734271303424476

#LR NearMiss(version= 2), WordLevel TFIDF: 0.5236625514403291

#LR NearMiss(version= 3), WordLevel TFIDF: 0.5541591861160982

#SVC NearMiss(version= 1), WordLevel TFIDF: 0.3597020219936148

#SVC NearMiss(version= 2), WordLevel TFIDF: 0.5224948875255624

#SVC NearMiss(version= 3), WordLevel TFIDF: 0.5594149908592323

As you can see NearMiss-3 over performed the other two versions. But still orver all it is below the over-sampling results.

Tomek Link Removal

A pair of samples is called a Tomek link if they belong to different classes and are each other’s nearest neighbors. Under-sampling can be done by removing all tomek links from the dataset. An alternate method is to only remove the majority class samples that are part of a Tomek link. Reference 1, Reference 2

# Under-Sampling TomekLinks

tl = TomekLinks()

tl_xtrain_tfidf, tl_train_y = tl.fit_sample(xtrain_tfidf, train_y)

accuracy = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),tl_xtrain_tfidf, tl_train_y, xvalid_tfidf)

print ("LR TomekLinks, WordLevel TFIDF: ", accuracy)

accuracy = train_model(svm.LinearSVC(),tl_xtrain_tfidf, tl_train_y, xvalid_tfidf)

print ("SVC TomekLinks, WordLevel TFIDF: ", accuracy) #LR TomekLinks, WordLevel TFIDF: 0.5358851674641147

#SVC TomekLinks, WordLevel TFIDF: 0.6998961578400832

It looks that removing nearest neighbors from different classes really improved the SVC classifier to over-perform all previous methods.

Edited Nearest Neighbor (ENN)

ENN under samples the majority class by removing samples which their class differs from the one of their nearest neighbors. If this step is repeated then we drive a new method RepeatedEditedNearestNeighbours

Another driven method: AllKNN is slightly different from the RepeatedEditedNearestNeighbours by changing the k parameter of the internal nearest neighbors algorithm, increasing it at each iteration. The last two algorithms require higher processing time.

#ENN - EditedNearestNeighbours

for sampler in (EditedNearestNeighbours(),

RepeatedEditedNearestNeighbours(),

AllKNN(allow_minority=True)):

enn_xtrain_tfidf, enn_train_y = sampler.fit_sample(xtrain_tfidf, train_y)

accuracy = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),enn_xtrain_tfidf, enn_train_y, xvalid_tfidf)

print ("LR {0}, WordLevel TFIDF: ".format(sampler), accuracy)

for sampler in (EditedNearestNeighbours(),

RepeatedEditedNearestNeighbours(),

AllKNN(allow_minority=True)):

enn_xtrain_tfidf, enn_train_y = sampler.fit_sample(xtrain_tfidf, train_y)

accuracy = train_model(svm.LinearSVC(),enn_xtrain_tfidf, enn_train_y, xvalid_tfidf)

print ("SVM {0}, WordLevel TFIDF: ".format(sampler), accuracy) #LR EditedNearestNeighbours: 0.5480093676814988

#LR RepeatedEditedNearestNeighbours: 0.5727170236753101

#LR AllKNN: 0.5547785547785548

#SVM EditedNearestNeighbours: 0.7101303911735205

#SVM RepeatedEditedNearestNeighbours: 0.7046332046332046

#SVM AllKNN: 0.7131474103585658

The K-Nearest-Neighbors KNN algorithm is a robust and versatile classifier that is often used as a benchmark for more complex classifiers such as Artificial Neural Networks (ANN) and Support Vector Machines (SVM). Despite its simplicity, KNN can outperform more powerful classifiers

AllKNN appied KNN algorithm for under-sampling and with SVM classifier it over-performed all re-sampling methods. The KNN has a memory cost and requires processing time.

Condensed Nearest Neighbor (CNN)

CNN makes use of a 1st nearest neighbor (1-NN) to iteratively decide if a sample should be kept in a data-set or not. Under-sampling via CNN can be slower compared to other methods since it requires many passes over the training data. CNN is considered noise sensitive and preserves the noisy samples. Another driven method OneSidedSelection also used the 1-NN and use TomekLinks discussed above to remove the samples considered noisy. The NeighbourhoodCleaningRule which is also driven from CNN uses EditedNearestNeighbours to remove some sample. Additionally, they use a 3 nearest-neighbors to remove samples which do not agree with this rule:

#CNN - CondensedNearestNeighbor

for sampler in (CondensedNearestNeighbour(random_state=0),

OneSidedSelection(random_state=0),

NeighbourhoodCleaningRule()):

nm_xtrain_tfidf, nm_train_y = sampler.fit_sample(xtrain_tfidf, train_y)

accuracysm = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),nm_xtrain_tfidf, nm_train_y, xvalid_tfidf)

print ("LR {0}, WordLevel TFIDF: ".format(sampler), accuracysm) for sampler in (CondensedNearestNeighbour(random_state=0),

OneSidedSelection(random_state=0),

NeighbourhoodCleaningRule()):

nm_xtrain_tfidf, nm_train_y = sampler.fit_sample(xtrain_tfidf, train_y)

accuracysm = train_model(svm.LinearSVC(),nm_xtrain_tfidf, nm_train_y, xvalid_tfidf)

print ("SVM {0}, WordLevel TFIDF: ".format(sampler), accuracysm) #LR CondensedNearestNeighbour: 0.4289156626506024

#LR OneSidedSelection: 0.5358851674641147

#LR NeighbourhoodCleaningRule: 0.5541327124563447 #SVM CondensedNearestNeighbour: 0.48195030473511485

#SVM OneSidedSelection: 0.6998961578400832

#SVM NeighbourhoodCleaningRule: 0.7075376884422111

Combining Under and Over Sampling

There are other methods to Over and Under Sample, but how about combining both? Ajinkya More claims in this paper that combining SMOTE and ENN produced the best result for his re-sampling experiments, let us see if we will achieve the same:

# Re-Sampling SMOTEENN

se = SMOTEENN(random_state=42)

se_xtrain_tfidf, se_train_y = se.fit_sample(xtrain_tfidf, train_y)

accuracy = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),se_xtrain_tfidf, se_train_y, xvalid_tfidf)

print ("LR SMOTEENN: ", accuracy)

accuracy = train_model(svm.LinearSVC(),se_xtrain_tfidf, se_train_y, xvalid_tfidf)

print ("SVC SMOTEENN: ", accuracy) #LR SMOTEENN: 0.39669421487603307

#SVC SMOTEENN: 0.47154471544715443

But another approach SMOTE + Tomek Link Removal worked well for our experiment:

# Re-Sampling SMOTETomek

se = SMOTETomek(random_state=42)

se_xtrain_tfidf, se_train_y = se.fit_sample(xtrain_tfidf, train_y) accuracy = train_model(linear_model.LogisticRegression(random_state=0, solver='lbfgs',multi_class='multinomial'),se_xtrain_tfidf, se_train_y, xvalid_tfidf)

print ("LR SMOTETomek: ", accuracy)

accuracy = train_model(svm.LinearSVC(),se_xtrain_tfidf, se_train_y, xvalid_tfidf)

print ("SVC SMOTETomek: ", accuracy) #LR SMOTETomek: 0.6756756756756755

#SVC SMOTETomek: 0.6876061120543293

Conclusions

Selecting the correct re-sampling algorithm for imbalanced corpus is important to feed the classifier with the best set of training data. In this long example, I have discussed variety of over-sampling, under-sampling and combined over and under sampling methods and came to a conclusion that the best fit method for our model was to apply AllKNN under sampling algorithm. When testing the code on the competition testing data the f1 score was: 0.729857819905213

I have used ensemble Extremely Randomized Trees for the same exercise, which over-performed this method.

Full running code is available on github.