Naive Bayes Algorithm for Classification | by Idil Ismiguzel | Towards Data Science

Classification is one of the most used forms of prediction where the goal is to predict the class of the record. For binary classification, we aim to predict whether a record is a 1 or a 0 such as spam/not spam or churn/not churn and for multiclass classification, we aim to predict the class of a record such as classifying a mail as primary/social/promotional, etc.
In addition to predicting the class of our records most of the time, we also want to know the predicted probability of belonging to the class of interest. These probability values are also called propensity scores. We can set a cutoff probability for the class of interest and for the records having propensity score below we consider the record belonging to that class, and vice versa.
For classification problem that is a form of supervised learning we start with a labeled data where the class of the records is known, we train our model using this data, and then apply the model to the new data where the class is unknown.
In this article, we will see how to use Naive Bayes algorithm for multiclass classification problem by implementing in Python.
I encourage anyone to check out the Jupyter Notebook on my GitHub for the full analysis and code.
Bayes theorem provides a way to calculate the probability of a hypothesis given our prior knowledge.
Prior Probability is the probability of an event before new data is collected i.e. P(spam) is the probability of spam mails before any new mail is seen.
Marginal Likelihood also called evidence is the probability of the evidence event to occur i.e. P(money) is the probability of mails include “money” in the text.
Likelihood is the probability of the evidence happen given that event is true i.e. P(money|spam) is the probability of mail includes “money” given that the mail is spam.
Posterior Probability is the probability of an outcome after the evidence information has been incorporated i.e. P(spam|money) is the probability of the mail is spam given that mail includes “money” in the text.
We can see that we are calculating the posterior probability P(spam|money) from the prior probability P(spam) with P(money) and P(money|spam).
Maximum a posteriori (MAP) is the hypothesis with the highest posterior probability. After calculating the posterior probability for several hypotheses we select the hypothesis with the highest probability.
Example: If P(spam|money) > P(not spam|money) then we can say that the mail can be classified as spam. This is the maximum probable hypothesis.
The answer is the naive solution.
The exact Bayesian classification is technically impractical since we have many evidence variables (predictors) in our dataset. When the number of predictors increases, many records that we want to classify will not have an exact match.
The above equation shows only the case where we have 3 evidence variables and even with only 3 of them it is not easy to find an exact match.
The naive assumption introduces that the variables are independent given the class. So we can calculate the conditional probability as follows:
By assuming the conditional independence between variables we can convert the Bayes equation into a simpler and naive one. Even though assuming independence between variables sounds superficial, the Naive Bayes algorithm performs pretty well in many classification tasks.
Let’s look at an example 👀
The multinomial Naive Bayes classifier is suitable for classification with discrete features such as word counts for text classification. It requires integer feature counts such as bag-of-words or tf-idf feature extraction applied to text.
For this example, I will use the dataset called “Twenty Newsgroups”, which is a collection of approximately 20,000 newsgroup documents partitioned evenly across 20 different newsgroups.
from sklearn.datasets import fetch_20newsgroups# import datasetdata = fetch_20newsgroups()data.target_names
For the simplicity of our example, I will select some categories to filter the dataset.
# Selected categoriescategories = ['talk.politics.misc', 'talk.religion.misc', 'sci.med', 'sci.space', 'rec.autos']# Create train and test dataset train = fetch_20newsgroups(subset='train', categories=categories)test = fetch_20newsgroups(subset='test', categories=categories)
Let’s look at an example data:
print(train.data[5])>>>From: rts@nwu.edu (Ted Schreiber)Subject: Opinions on recent Alfa SpidersNntp-Posting-Host: mac178.mech.nwu.eduOrganization: Mechanical EngineeringLines: 15Anyone have any opinions on fairly recent model Alfa Spiders (86-> ) Reliablilty for what their class ( I'm not talking Alfa vs. Toyota corrola which is more relaible ) As far as I can discern, in recent years, there are 3 levels with all basically the same mechanical aspects.Please email any responses Ted SchreiberMechanical Enginering Northwestern UniversityTel: 708.491.5386 FAX 708.491.3915 Email: rts@nwu.edu
To use the text data for any machine learning task, we need to convert each string into a vector of numbers. This step is called feature engineering and one way to do is using tf-idf vectorizer. We can create a pipeline that attaches the tf-idf vector to a multinomial naive Bayes classifier.
from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.naive_bayes import MultinomialNBfrom sklearn.pipeline import make_pipeline# Create a pipelinemodel = make_pipeline(TfidfVectorizer(), MultinomialNB(alpha=1))
📢 alpha in multinomialNB represents the additive smoothing parameter. If it is 0, then there will be no smoothing.
⚡ Why smoothing? Smoothing solves the zero probability problem in Naive Bayes algorithm, which is the problem of assigning probability equal to zero for every new data point in the test set.
# Fit the model with training setmodel.fit(train.data, train.target)#Predict labels for the test setlabels = model.predict(test.data)
After fitting the model with the training set and predicting the labels for the test set, we can plot the confusion matrix to evaluate our model.
from sklearn.metrics import confusion_matriximport seaborn as snsimport matplotlib.pyplot as plt# Create the confusion matrixconf_mat = confusion_matrix(test.target, labels, normalize="true")# Plot the confusion matrixsns.heatmap(conf_mat.T, annot=True, fmt=".0%", cmap="cividis", xticklabels=train.target_names, yticklabels=train.target_names)plt.xlabel("True label")plt.ylabel("Predicted label")
From the confusion matrix, we see that naive Bayes classifier performed quite well for some news categories such as autos, med, and space where the true labels are correctly predicted for 98%, 95%, and 97% of the time. However, it achieved lower results for politics and religion categories as you can imagine..
If you are interested in text classification you can read my article below where I showed text mining techniques and used logistic regression to classify Amazon’s fine food reviews dataset.
medium.com
Naive Bayes classifiers are generally good for the initial baseline classification. They are fast, easy to implement and interpret. Having said so, they do not perform well for sophisticated models such as not having well-separated categories.
Even though Naive Bayes is a simple classifier, it is used in many text-data based applications such as text classification (as we did in the example), sentiment analysis (i.e. understanding if the person thinks positively or negatively), and recommender systems (i.e. in collaborative filtering whether the customer is interested in a particular product group or not).
To sum up we can list several advantages and disadvantages of Naive Bayes.
✅ Fast for training and prediction
✅ Easy to implement having only a few tunable parameters such as alpha
✅ Easy to interpret since they provide a probabilistic prediction
❌ Naive assumption of all predictors are independent do not hold in real life
❌ Zero probability problem can introduce wrong results when the smoothing technique wasn't used well.
❌ It can create highly biased estimations.
I hope you enjoyed following this article and learning about Naive Bayes algorithm.
If you liked this article, you can read my other articles here and follow me on Medium. Let me know if you have any questions or suggestions.✨
Enjoy this article? Become a member for more!