Machine Learning Basics:Naive Bayes Algorithm-Part II
By: Madhu Ramiah
Tags: Supervised Learning, Machine Learning, Text Classification, Naive Bayes

In this blog we will be looking into how to handle text data with Naive Bayes algorithm, its pros and cons. We will also look into Bernouli Naive Bayes algorithm.

How to handle text data?

Spam vs Ham

Naive Bayes algorithm can be used for solving problems like detecting if an email is spam/not spam, classifying reviews as positive or negative, or classifying any text content or for sentiment analysis. Let us consider the reviews below.

Restaurant Reviews

We would need to find all the words in the reviews We need to convert to lower case We need to remove the stop words

After stop words are removed

4. We need to create a vector with all the words like below where 1 means the word is present and 0 means the word is not present.

Vector for bag of words

5. We need to compute the probabilities as below

P(food)=3/17

P(amazing)=1/17

P(worst)=1/17

P(service)=2/17

P(chicken)=1/17 and so on

6. We need to calculate the conditional probabilities as below,

P(food|positive)=2/11

P(food|negative)=1/6

P(amazing|positive)=1/11

P(amazing|negative)=0/6

7. We can see that P(amazing|negative)=0/6, so when we calculate the probability of “amazing food” being negative, it is 0 as below, because of one value being 0. We want to avoid such a scenario and so do some smoothing called “Laplace smoothing”

P(negative|"amazing food")=P(amazing|negative)*P(food|negative)*P(negative)/P(amazing)*P(negative)=0

What is Laplace Smoothing?

Laplace smoothing is adding 1 to every value in the vector, so that none of the conditional probability values will be 0. If we did Laplace smoothing for the table above, then it would result in something like this

Now the P(amazing)=8/108,P(negative)=29/108 and P(amazing|negative)=3/29. Now, no probabilities will be 0.

What is Bernoulli Naive Bayes?

In Bernoulli Naive Bayes, the vector values are always 1 even though the number of occurrences may be greater than 1. Though we don’t use Bernoulli Naive Bayes widely, it may be useful in some places. Let’s say you want to classify emails as spam/ham(not spam). In this case, we don’t want any ham emails to be classified as spam because we would lose many important emails. When we use the Naive Bayes algorithm, we may get more False Positives in some cases. In those cases, it is better to try Bernoulli Naive Bayes. But using it all depends on the type of problem we are trying to solve.

Advantages of Naive Bayes:

It is very simple and can be implemented with less training data It is very fast Can handle both numeric and text data It is highly scalable, works well even if the number of data points increase Can be used for both binary and multi-class classification Handling missing values becomes lot easier Can also make probabilistic predictions

Disadvantage of Naive Bayes:

It assumes conditional independence, so it is very difficult to understand the relation between features While handling continuous values, you may have to bin the values properly Smoothing is surely needed, else we may get many probability values to be 0

Useful Links