Text Classification Using Naive Bayes: Theory & A Working Example | by Serafeim Loukas | Towards Data Science

IntroductionThe Naive Bayes algorithmDealing with text dataWorking Example in Python (step-by-step guide)Bonus: Having fun with the modelConclusions
Introduction
The Naive Bayes algorithm
Dealing with text data
Working Example in Python (step-by-step guide)
Bonus: Having fun with the model
Conclusions
Naive Bayes classifiers are a collection of classification algorithms based on Bayes’ Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle, i.e. every pair of features being classified is independent of each other.
Naive Bayes classifiers have been heavily used for text classification and text analysis machine learning problems.
Text Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols (i.e. strings) cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length.
In this article I explain a) how Naive Bayes works, b) how we can use text data and fit them into a model after transforming them into a more appropriate form. Finally, I implement a multi-class text classification problem step-by-step in Python.
Let’s get started !!!
Naive Bayes classifiers are a collection of classification algorithms based on Bayes’ Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle, i.e. every pair of features being classified is independent of each other.
The dataset is divided into two parts, namely, feature matrix and the response/target vector.
The Feature matrix (X) contains all the vectors(rows) of the dataset in which each vector consists of the value of dependent features. The number of features is d i.e. X = (x1,x2,x2, xd).
The Response/target vector (y) contains the value of class/group variable for each row of feature matrix.
Naive Bayes assumes that each feature/variable of the same class makes an:
independent
equal
contribution to the outcome.
Side Note: The assumptions made by Naive Bayes are not generally correct in real-world situations. In-fact, the independence assumption is often not met and this is why it is called “Naive” i.e. because it assumes something that might not be true.
Bayes’ Theorem finds the probability of an event occurring given the probability of another event that has already occurred. Bayes’ theorem is stated mathematically as follows:
where:
A and B are called events.
P(A | B) is the probability of event A, given the event B is true (has occured). Event B is also termed as evidence.
P(A) is the priori of A (the prior independent probability, i.e. probability of event before evidence is seen).
P(B | A) is the probability of B given event A, i.e. probability of event B after evidence A is seen.
Given a data matrix X and a target vector y, we state our problem as:
where, y is class variable and X is a dependent feature vector with dimension d i.e. X = (x1,x2,x2, xd), where d is the number of variables/features of the sample.
P(y|X) is the probability of observing the class y given the sample X with X = (x1,x2,x2, xd), where d is the number of variables/features of the sample.
Now the “naïve” conditional independence assumptions come into play: assume that all features in X are mutually independent, conditional on the category y:
Finally, to find the probability of a given sample for all possible values of the class variable y, we just need to find the output with maximum probability:
One question that arises at this point is the following:
How are we going to use the raw text data to train the model ? The raw data is a collection of strings !
Text Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols (i.e. strings) cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length.
In order to address this, scikit-learn provides utilities for the most common ways to extract numerical features from text content, namely:
tokenizing strings and giving an integer id for each possible token, for instance by using white-spaces and punctuation as token separators.
counting the occurrences of tokens in each document.
In this scheme, features and samples are defined as follows:
each individual token occurrence frequency is treated as a feature.
the vector of all the token frequencies for a given document is considered a multivariate sample.
from sklearn.feature_extraction.text import CountVectorizercorpus = [    'This is the first document.',    'This document is the second document.',    'And this is the third one.',    'Is this the first document?',]vectorizer = CountVectorizer()X = vectorizer.fit_transform(corpus)print(vectorizer.get_feature_names())[‘and’, ‘document’, ‘first’, ‘is’, ‘one’, ‘second’, ‘the’, ‘third’, ‘this’]print(X.toarray())[[0 1 1 1 0 0 1 0 1] [0 2 0 1 0 1 1 0 1] [1 0 0 1 1 0 1 1 1] [0 1 1 1 0 0 1 0 1]]
In the above toy example, we have a collection of strings stored into the variable corpus. Using the text transformer, we can see that we have a specific number of unique strings (vocabulary) in our data.
This can be seen by printing the vectorizer.get_feature_names() variable. We observe that we have 9 unique words.
Next, we printed the transformed data (X) and we observe the following:
We have 4 rows in X as the number of our text strings (we have the same number of samples after the transformation).
We have the same number of columns (features/variables) in the transformed data (X) for all the samples (this was not the case before that transformation i.e. the individual strings had different lengths).
The values 0,1,2, encode the frequency of a word that appeared in the initial text data.
E.g. The first transformed row is [0 1 1 1 0 0 1 0 1] and the unique vocabulary is [‘and’, ‘document’, ‘first’, ‘is’, ‘one’, ‘second’, ‘the’, ‘third’, ‘this’], thus this means that the words “document”, “first”, “is”, “the” and “this” appeared 1 time each in the initial text string (i.e. ‘This is the first document.’).
Side note: This is the counting approach. The term-frequency transform is nothing more than a transformation of a count matrix into a normalized term-frequency matrix.
Hope everything is clear now. If not, read this paragraph as many times as it is needed in order to really grasp the idea and understand this transformation. It is a fundamental step.
Now that you understood how the Naive Bayes and the Text Transformation work, it’s time to start coding !
As a working example, we will use some text data and we will build a Naive Bayes model to predict the categories of the texts. This is a multi-class (20 classes) text classification problem.
Let’s start (I will walk you through). First, we will load all the necessary libraries:
import numpy as np, pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltfrom sklearn.datasets import fetch_20newsgroupsfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.naive_bayes import MultinomialNBfrom sklearn.pipeline import make_pipelinefrom sklearn.metrics import confusion_matrix, accuracy_scoresns.set() # use seaborn plotting style
Next, let’s load the data (training and test sets):
# Load the datasetdata = fetch_20newsgroups()# Get the text categoriestext_categories = data.target_names# define the training settrain_data = fetch_20newsgroups(subset="train", categories=text_categories)# define the test settest_data = fetch_20newsgroups(subset="test", categories=text_categories)
Let’s find out how many classes and samples we have:
print("We have {} unique classes".format(len(text_categories)))print("We have {} training samples".format(len(train_data.data)))print("We have {} test samples".format(len(test_data.data)))
The above prints:
We have 20 unique classesWe have 11314 training samplesWe have 7532 test samples
So, this is a 20-class text classification problem with n_train = 11314 training samples (text sentences) and n_test = 7532 test samples (text sentences).
Let’s visualize the 5th training sample:
# let’s have a look as some training dataprint(test_data.data[5])
As mentioned previously, our data are texts (more specifically, emails) so you should see something like the following printed out:
The next step consists of building the Naive Bayes classifier and finally training the model. In our example, we will convert the collection of text documents (train and test sets) into a matrix of token counts (I explained how this works in Section 3).
To implement that text transformation we will use the make_pipeline function. This will internally transform the text data and then the model will be fitted using the transformed data.
# Build the modelmodel = make_pipeline(TfidfVectorizer(), MultinomialNB())# Train the model using the training datamodel.fit(train_data.data, train_data.target)# Predict the categories of the test datapredicted_categories = model.predict(test_data.data)
The last line of code predicts the labels of the test set.
Let’s see the predicted categories names:
print(np.array(test_data.target_names)[predicted_categories])array(['rec.autos', 'sci.crypt', 'alt.atheism', ..., 'rec.sport.baseball', 'comp.sys.ibm.pc.hardware', 'soc.religion.christian'], dtype='<U24')
Finally, let’s build the multi-class confusion matrix to see if the model is good or if the model predicts correctly only specific text categories.
# plot the confusion matrixmat = confusion_matrix(test_data.target, predicted_categories)sns.heatmap(mat.T, square = True, annot=True, fmt = "d", xticklabels=train_data.target_names,yticklabels=train_data.target_names)plt.xlabel("true labels")plt.ylabel("predicted label")plt.show()print("The accuracy is {}".format(accuracy_score(test_data.target, predicted_categories)))The accuracy is 0.7738980350504514
Let’s have some fun using the trained model. Let’s classify whatever sentence we like 😄.
# custom function to have fundef my_predictions(my_sentence, model):    all_categories_names = np.array(data.target_names)    prediction = model.predict([my_sentence])    return all_categories_names[prediction]my_sentence = “jesus”print(my_predictions(my_sentence, model))['soc.religion.christian']my_sentence = "Are you an atheist?"print(my_predictions(my_sentence, model))['alt.atheism']
We inserted the string “jesus” to the model and it predicted the class “[‘soc.religion.christian’]”.
Change the “my_sentence” into other stings and play with the model 😃.
We saw that Naive Bayes is a very powerful algorithm for multi-class text classification problems.
Side Note: If you want to know more about the confusion matrix (and the ROC curve) read this:
towardsdatascience.com
From the above confusion matrix, we can verify that the model is really good.
It is able to correctly predict all 20 classes of the text data (most values are on the diagonal and few are off-the-diagonal).
We also notice that the highest miss-classification (value off-the-diagonal) is 131 (5 lines from the end, last column at the right). The value 131 means that 131 documents that belonged to the “religion miscellaneous ” category were miss-classified as belonging to the “religion christian” category.
Interesting thing that these 2 categories are really similar and actually one could characterize these as 2 subgroups of a larger group e.g. “religion” in general.
Finally, the accuracy on the test set is 0.7739 which is quite good for a 20-class text classification problem 🚀.
That’s all folks! Hope you liked this article.
If you liked and found this article useful, follow 👣 me to be able to see all my new posts.
Questions? Post them as a comment and I will reply as soon as possible.