How I was using Naive Bayes (Incorrectly) till now — Part-1 | by GOWTHAM CH | Towards Data Science

Just like Naive Bayes makes a naive assumption that the features we give to the model are independent, I have also made several naive assumptions about how NB works.
I assumed that MultinomialNB works for any given data. (And I use that for whatever data I have) → Explained in detail in the below post, why this is a bad practice.
Gaussian naive Bayes make an assumption that the features are gaussian. →The features are not assumed to be gaussian it assumes the likelihood probabilities follow Gaussian distribution.
Calculating of likelihood probability is same in MultinomialNB and CategoricalNB → No, there is a slight difference in calculating the probability. In multinomial, the denominator will the total number of words in that particular class, while in CategoricalNB it is a total number of data points with a particular class.
To implement naive Bayes from scratch in python is very hard. → It just takes 15 lines of code to implement multinomial Naive Bayes.
If you are new to Naive Bayes or want a quick revision, please check my notes to get started.
1. Why in Naive Bayes do we assume the features are conditional independent?
2. Sklearn has GaussianNB, MultinomialNB, CategoricalNB, BernoulliNB → Given data that has categories, numerical, binary features which model will you choose?
3. How do you implement Multinomial Naive Bayes from scratch for text data and match the results with Sklearn MultinomialNB?
4. How do you implement Categorical Naive Bayes from scratch for categorical data and match the results with Sklearn CategoricalNB?
5. How do you implement Gaussian Naive Bayes from scratch for Numerical data and match the results with Sklearn GaussianNB?
6. What are the training and test → time and space complexities for Naive Bayes?
7. Is Naive Bayes affected by Imbalanced data, if yes how to resolve it?
8. How is Naive Bayes affected by outliers?
9. Is Naive Bayes interpretable, can we tell which features helped us predicting a particular class?
10. Is Naive Bayes a linear classifier, Can it solve a non-linear decision boundary?
11. How to avoid overfitting or underfitting problems in Naive Bayes?
In this post, you will find all the answers for the first 4 questions (In bold), As the post was getting longer I moved those questions to part — 2, you can check the link here.
gautigadu091.medium.com
Let's get started,
This is the naive Bayes equation for calculating the posterior probability of a target value given an input X (without assuming any conditional independence)
Assume you have three features then in order to calculate the likelihood probability, you have to follow the below steps.
Assume your training data is only having three data points (0,1,0),(1,0,0) and (1,0,1) with target values 0,0 and 1 respectively. Now let us compute the likelihood probability with and without assuming conditional independence
Without conditional independence, we get likelihood probability as zero — But why?
Because we do not have enough combinations of the data that will help us compute the likelihood probability.
So, how many data points minimum you need to have?
Assume we are solving binary classification and the input is also a binary feature. Let us say we have 3 features — Then all possible values of input X can have 2*2*2 = 8 combinations and we need these 8 combinations for each class 8*2 = 16.
For d-dimension data, we need to have 2^(d+1) data points. If d = 30 then it is close to 500 billion data points. Which is practically not feasible.
With conditional independence, this comes to 2*d combinations. With just a simple assumption the time complexity is significantly reduced.
2. Sklearn has GaussianNB, MultinomialNB, CategoricalNB, BernoulliNB → Given data that has categories, numerical, binary features which model will you choose?
This is something I overlooked I usually apply Multinomial NB given any kind of data I have, but when I read more about each and every algorithm I understand that each algorithm is expecting different types of data.
GaussianNB → When you have continuous features.
CategoricalNB →When you have categorical data.
MultinomialNB → Applied to text data.
So, given your data has continuous features, categorical features and text data what algorithm you will use?
The fundamental assumption of each and every algorithm is that it assumes the features are conditionally independent. Fit the categorical features on categoricalNB, continuous features on GaussianNB and text data on MultinomialNB and get the likelihood probabilities of each model(For each data point now we will have 3 likelihood probabilities) and multiply them to get the overall likelihood probability.
Note: You have to multiply the prior probability to the final likelihood probability to get the final posterior probability
Steps:
Convert the data into a vector using BOW.Calculate the counts based on the class.Calculate all the likelihood probabilities.Calculate the prior probability.Calculate the posterior probability.
Convert the data into a vector using BOW.
Calculate the counts based on the class.
Calculate all the likelihood probabilities.
Calculate the prior probability.
Calculate the posterior probability.
Let us take a sample data:
Convert the text into BOW using CountVectorizer:
Convert the text into BOW using CountVectorizer:
This is straightforward and self-explanatory.
2. Calculate the counts based on classes
First, do a one-hot encoding of the target values. I am using LabelBinarizer here. Check the sample output in the below code.
The shape of y now will be (n_classes*n_datapoints) and the shape of X is (n_datapoints*n_features).
To get the count based on the class, it is simple enough to multiply the transpose of y with X.
Confused about how the count_matrix is working, let us take an example and make it clear:
Hope the above example is clear, if not try to do it on a piece of paper to understand the matrix multiplications.
3. Calculating probabilities:
For example to calculate P(‘are’/y=’question’) we use the below formula. Laplace smoothing is added to avoid a zero probability case.
Steps in calculating feature_log_probabilities:
We already have the counts → Just add alpha to them for doing laplace smoothing.Now, to this calculate the sum row-wise to get the number of words in a particular class.
We already have the counts → Just add alpha to them for doing laplace smoothing.
Now, to this calculate the sum row-wise to get the number of words in a particular class.
When alpha = 1:
Adding one to all the values in count_matrix.Calculating row-wise sum. As per the formula to the denominator, we have to add 21*alpha (21 unique words), are we doing this?
Adding one to all the values in count_matrix.
Calculating row-wise sum. As per the formula to the denominator, we have to add 21*alpha (21 unique words), are we doing this?
Yes, here in the numerator to each and every value we have added alpha so when we do just sum we are basically doing row sum (count_matrix)+ 21*alpha.
4. Calculate prior probabilities:
This is the simplest step in the algorithm and the below code is self-explanatory.
4. Predict function:
Now, we have the probabilities and a query point. We can simply do a weighted sum of probabilities and query point using matrix multiplications.
Wrapping up all the code,
Let us compare our results with that of sklearn’s implementation
With just 15 lines of code, We have implemented the “Naive Bayes classifier”.
4. How do you implement Categorical Naive Bayes from scratch for categorical data and match the results with Sklearn CategoricalNB?
Preprocessing the data.Calculate the counts/presence of each feature based on class.Calculate likelihood probability.Calculate prior probability.Calculate posterior probability for a given query point → Predict function
Preprocessing the data.
Calculate the counts/presence of each feature based on class.
Calculate likelihood probability.
Calculate prior probability.
Calculate posterior probability for a given query point → Predict function
For this exercise, I am taking simple Categorical data:
Preprocessing the data:
Converting the categorical data into a numerical form using ordinal encoding. The features are converted to ordinal integers. This results in a single column of integers (0 to n_categories — 1) per feature.
Apply one-hot encoding on target values (As we did in the Multinomial NB)
X,y,classes = preprocess()X.shape, y.shape
Output shape after encoding is: X →((18, 3),y → (18, 2))
2. Calculate the counts/presence of each feature based on class.
Below is an example of what we will try to achieve for each and every feature.
Our main aim in this part is to generate this count matrix for each and every feature.
For each feature — Extract that column from the data — X_feature
For a class → Identify where the class will be zero and where it will be one and convert it into a boolean.
Now mask your input feature and do simply counting.
Count_matrix will give an output this way, For each of the features you have 2D -array(The first row corresponding to No and the second row corresponding to Yes).
3. Calculating Likelihood probabilities:
This is similar to what we did for MultinomialNB, Here
For each feature, we are adding alpha(Laplace smoothing) if provided.
Doing row-wise sum, as discussed in the image above.
Finding log-probabilities — log(num/den) → log(num)-log(den).
4. Calculate prior probabilities:
This is exactly similar to MultinomialNB and is the easiest step in the algorithm.
5. Calculating Posterior probability:
Get the corresponding log_probs from each feature.
Add these probabilities with the prior probability to get our final posterior probability.
Comparing the results of Sklearn and our manual implementation.
For the remaining questions, do check the part-2 in the below link:
gautigadu091.medium.com
Thank you so much 💖 for reading this post and I hope you got a clear and depth understanding of how naive Bayes works.
You can find the entire code in this GitHub link. You can also connect with me on LinkedIn.
https://scikit-learn.org/stable/modules/naive_bayes.html#naive-bayesApplied AI course
https://scikit-learn.org/stable/modules/naive_bayes.html#naive-bayes
Applied AI course