Algorithms for Text Classification — Part 1 | by Hailey Huong Nguyen | Towards Data Science

When you check news about Natural Language Processing (NLP) these days, you will see a lot of hype surrounding language models, transfer learning, OpenAI, ULMFit, etc. Catching up with the current state-of-art in NLP is great, though I still believe that one shall be strong in understanding the classic algorithms, such as Naive Bayes and Logistic Regression. Why? Because the company that you might work with might not be specialized in building chatbots or working on a text-generating machine! Most of the time, starting with simple models might give you great results without banging your head to the wall trying to explain your methodology to your business partners. So this blog post is the beginning of a series about text classification’s methods, starting with the basic. I will try to explain the theory as well as how to use the algorithm in practice.
Why the name? “Bayes” is named from the famous Bayes’ Theorem in probability, and “Naive” is because the assumptions of this algorithm are very simple and most of the time not true.
The general idea of Naive Bayes:
Represent a document X as a set of (w, a frequency of w) pairs.For each label y, build a probabilistic model P(X| Y = y) of documents in class y.To classify, select label y which is most likely to generate X:
Represent a document X as a set of (w, a frequency of w) pairs.
For each label y, build a probabilistic model P(X| Y = y) of documents in class y.
To classify, select label y which is most likely to generate X:
Assumptions:
The order of the words in document X makes no difference but repetitions of words do.Words appear independently of each other, given the document class.
The order of the words in document X makes no difference but repetitions of words do.
Words appear independently of each other, given the document class.
Based on these assumptions, we have the following equations to estimate P(X|y):
There are problems with these equations:
For equation (1), if our document has more than 100 words, P(w1,..., w_n|Y = y) will be a product of very small word probabilities ( < 0.1), leading to the UNDERFLOW problem => Working with logarithms is desirable to maintain numerical stability.
For equation (2), if we have a new word w in the new text that we need to classify, P(W = w | Y = y) = 0 as w has never appeared in our training data. => one solution is to smooth the probabilities. Assume we have m examples with P(w|y) = p. This use of m and p is a Dirichlet prior for the multinomial distribution. Please note that there are many smoothing methods.
Putting all these together, we have the following algorithm:
Now, let’s work on a hypothetical example to understand the algorithm:
Suppose we have 3 documents:
X1 = “The government shutdown” with label y1 = news
X2 = “Federal employees are protesting shutdown” with label y2 = news
X3 = “Turn melancholy forth to funerals” with label y3 = poetry
and a new document to classify:
X_new = “The shutdown affects federal employees benefit”
We then can have this count table from our training data
For the purpose of simplicity, I did not exclude stopwords, but in practice, you definitely should. Besides, to prevent the underflow problem, I define the smoothing parameters: p = 0.5 and m = 1. We can then calculate the score of each label for the new document X_new as following.
We can see that the score for label “news” is higher than the score for label “poetry”, so we will classify X_new as “news”.
Next, let’s see how to run this algorithm using Python with real data:
import pandas as pdimport numpy as npspam_data = pd.read_csv('spam.csv')spam_data['target'] = np.where(spam_data['target']=='spam',1,0)print(spam_data.shape)spam_data.head(10)
from sklearn.model_selection import train_test_split#Split data into train and test setsX_train, X_test, y_train, y_test = train_test_split(spam_data['text'],spam_data['target'],random_state=0)from sklearn.feature_extraction.text import CountVectorizerfrom sklearn.naive_bayes import MultinomialNBfrom sklearn.metrics import roc_auc_score#Train and evaluate the modelvect = CountVectorizer().fit(X_train)X_train_vectorized = vect.transform(X_train)clfrNB = MultinomialNB(alpha = 0.1)clfrNB.fit(X_train_vectorized, y_train)preds = clfrNB.predict(vect.transform(X_test))score = roc_auc_score(y_test, preds)print(score)#0.9720812182741116
Our AUC score is 0.97, which is not bad for a simple model like this. We will compare this performance with other algorithms’ performance using the same dataset in later blog posts.