Multi Label Classification using Bag-of-Words (BoW) and TF-IDF | by Snehal Nair | Towards Data Science

For this study, we are using Kaggle data for Toxic Comment Classification Challenge. Lets load and inspect the data. This is a multilabel classification problem where comments are classified by the level of toxicity: toxic / severe_toxic / obscene / threat / insult / identity_hate
import pandas as pddata = pd.read_csv('train.csv')print('Shape of the data: ', data.shape)data.head()
y_cols = list(data.columns[2:])is_multilabel = (data[y_cols].sum(axis=1) >1).count()print('is_multilabel count: ', is_multilabel)
From the above data we can see that not all comments have a label.
Its Multilabel data (each comment can have more than one label)
Add a label, ‘non_toxic’ for comments with no label
Lets also check the explore how balanced is the classes.
# Add a label, ‘non_toxic’ for comments with no labeldata['non_toxic'] = 1-data[y_cols].max(axis=1)y_cols += ['non_toxic']# Inspect the class balancedef get_class_weight(data):    class_weight = {}    for num,col in enumerate(y_cols):        if num not in class_weight:            class_weight[col] = round((data[data[col] == 1][col].sum())/data.shape[0]*100,2)    return class_weightclass_weight = get_class_weight(data)print('Total class weight: ', sum(class_weight.values()), '%\n\n', class_weight)
We can see that the data is highly imbalanced. Imbalanced data refers to classification problems where the classes are not represented equally for e.g., 89% comments are classified under the newly built ‘non_toxic’ label.
Any given linear model will handle class imbalance very badly if it uses squared loss for binary classification. We will not discuss the techniques to tackle the imbalance problem in this project. Let's focus on preprocessing the text data before converting it to numeric data using BoW and tf-idf.
from sklearn.model_selection import train_test_splitX, X_test, y, y_test = train_test_split(X_data, y_data, test_size=0.2, train_size=0.8)X_train, X_val, y_train, y_val = train_test_split(X,y,test_size = 0.25,train_size =0.75)X_train[:1]
Lets take a closer look at one the comments. Please note the text will vary for you since the dataset is split randomly. Please use seed in the split if you aim to reproduce the results.
From the above example, we can see that the text requires preprocessing, i.e., converting it into same cas (lower), remove symbols, numbers and stop words before the text is converted into tokens. For preprocessing the text you will need to download specific libraries.
import reimport numpy as nimport nltknltk.download('stopwords')from nltk.corpus import stopwordsnltk.download('punkt')
For machine learning models, the textual data must be converted to numeric data. This can be done in various ways like BoW, tf-idf, Word embeddings, etc. In this project, we will be focusing on BoW and tf-idf.
In the BoW model, a text (such as a sentence or a document) is represented as the bag (multiset) of its words, disregarding grammar and even word order but keeping multiplicity.
We will restrict ourselves to N popular words to limit the size of the matrix. Moreover, including unpopular words will only introduce sparsity without adding too much information. For this project lets work with 10000 popular words.
# Lets take a look at top 10 popular wordsPOPULAR_WORDS[:10]
For each comment in the corpora create a zero vector with N dimension and for the words found in the comment increase the values in the vector by 1 for e.g., If a word appear twice, that index in the vector will get 2.
For efficient storage, we will convert this vector into a sparse vector, one that leverages sparsity and actually stores only nonzero entries.
from scipy import sparse as sp_sparse
In information retrieval, tf–idf or TFIDF, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.
This method is an extension to Bag-of-Words where the total frequency of the word is divided by total words in the document. This penalizes too frequent word by normalizing it over the entire document.
from sklearn.feature_extraction.text import TfidfVectorizer
We have the datasets prepared using two different techniques BoW and tf-idf. We can run classifiers on both the datasets. Since this is multilabel classification problem, we will be using a simple OneVsRestClassfier logistic regression.
from sklearn.multiclass import OneVsRestClassifierfrom sklearn.linear_model import LogisticRegression
You can experiement with different regularization techniques, L1 and L2 with different coefficients (e.g. C equal to 0.1, 1, 10, 100) till you are happy with the result, this is called hyperparameter tuning. This can be acheived by cv grid search, random search and bayesian optimisation. We are not covering this topic in this article. If you would like to learn more about this, please refer to this post.
We will using metrics like accuracy score and f1 score for evaluation.
Accuracy Score: In multilabel classification, this function computes subset accuracy: the set of labels predicted for a sample must exactly match the corresponding set of labels in y_true.
F1 score : The F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. F1 score = 2 * (precision * recall) / (precision + recall)
'F1 score micro': Calculate metrics globally by counting the total true positives, false negatives and false positives.
'F1 score macro': Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
'F1 score weighted': Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.
from sklearn.metrics import accuracy_scorefrom sklearn.metrics import f1_scorefrom sklearn.metrics import roc_auc_score from sklearn.metrics import average_precision_scorefrom sklearn.metrics import recall_score
F1 score weighted and macro which accounts for data imbalance look good. Lets check the output, predicted label and actual label. We will need to replace on hot encoded labels with actual label for interpretation. Lets run prediction on tf-idf model.
test_predictions = classifier_tfidf.predict(X_test_tfidf)
for i in range(90,97):    print('\ny_label: ', test_labels[i], '\ny_pred: ', test_pred_labels[i])print('\ny_label: ', test_labels[i], '\ny_pred: ', test_pred_labels[i])
The results are not too bad, but it can do better. Please try to experiment hyper parameter tuning and try different classifiers to check the performance of the model. Hope you enjoyed reading.
I have left the code for building word cloud image, if you are interested.
comments_join = ' '.join(POPULAR_WORDS)from scipy.misc import imreadfrom wordcloud import WordCloud, STOPWORDStwitter_mask = imread('twitter.png', flatten=True)    wordcloud = WordCloud(                      stopwords=STOPWORDS,                      background_color='white',                      width=1800,                      height=1400,                      mask=twitter_mask            ).generate(comments_join)plt.figure(figsize = (12, 12), facecolor = None) plt.imshow(wordcloud)plt.axis("off")plt.savefig('twitter_comments.png', dpi=300)plt.show()
For jupyter notebook with codes, please click here.