Comparing a variety of Naive Bayes classification algorithms | by Pavlo Horbonos | Towards Data Science

Naive Bayes algorithm is one of the well-known supervised classification algorithms. It bases on the Bayes theorem, it is very fast and good enough for text classification. I believe that there is no need to describe the theory behind it, nevertheless, we will cover a few concepts and after that focus on the comparing of different implementations.
Firstly, we take the Bayes formula itself:
What does it mean? Since it may be confusing, we will switch a look. We will talk about the classes of events in terms of spam messages. So now we have two classes of events: spam message and not-spam message.
Let’s change A to an event of getting spam (y) and B to a message which consists of a set of words (x1, x2, ...). So:
Now it is more obvious: the probability of a message with words (x1, x2,...) to be spam equals the general probability of getting spam, multiplied with the probability of spam to contain the mentioned words and divided by the probability of any incoming message to contain the given words.
Naive Bayes theorem says, that the probabilities of all the events in (x1, x2, ...) set may be treated as independent, so:
This is still a bit complex formula, but we can simplify it. Obviously, we need to calculate the probability of a message to be spam and the probability of a message to not be spam. As we need to compare both values and since they are in the same field and have equal divider (P(x1, x2, ...)) we can work only with the numerator:
We can find the general probability of getting spam from a dataset just from the distribution. So, the main problem is to find the conditional probabilities of every word to appear in the spam message (P(xi | y)). That is almost the single difference between the implementations of the Naive Bayes algorithm.
I have already built the implementation of an algorithm from scratch:
towardsdatascience.com
Though, let’s now investigate the abilities of the sklearn library. For our purpose, we will use the collection of SMS messages, which was put together by Tiago A. Almeida and José María Gómez Hidalgo. It is free and can be downloaded from the UCI Machine Learning Repository. It contains 5572 records of different messages together with 747 spam messages.
We split every message to a set of words and encode the labels. After that, we split the dataset to the train and the test parts.
The next step is the most important: we turn the train part to the vocabulary and count the repetitions of every vocabulary word in every message (in both train and test parts):
Now the dataset is ready for the model creation.
This approach is built on the assumption of a normal distribution of probabilities. It means, that spam and not-spam classes of messages have frequencies of the words from vocabulary distributed by the Gaussian law:
The formula is based on the mean (μ) and Bessel corrected variance (σ) of the frequency of each word in the class of messages.
cl_gauss = sklearn.naive_bayes.GaussianNB()res_gauss = cl_gauss.fit(X_train_voc, y_train).predict(X_test_voc)metrics.accuracy_score(y_test, res_gauss) * 100
The result is 91.03% of model accuracy. It is good enough, but shows, that words have not completely Gaussian distributions.
Multinomial classification suits best for the discrete values like word counts. So we expect it to show the best accuracy. In this case distribution of probabilities for each event bases on the formula:
Ny is the total number of features of the event y (total number of words in all spam messages), Nyi — count of each feature (summary number of repetitions of a word in all spam messages), n — the number of features (number of words in the vocabulary) and α is a smoothing Laplace parameter to discard the influence of words absent in the vocabulary. The same formula applies to the set of not-spam messages.
cl_multi = sklearn.naive_bayes.MultinomialNB()res_multi = cl_multi.fit(X_train_voc, y_train).predict(X_test_voc)metrics.accuracy_score(y_test, res_multi) * 100
Model accuracy is 99.19%, which is (spoiler) the highest among the tested models including the custom one.
This approach is almost the same as the Multinomial, though now we count the occurrences of a word in the complement to the class. For example, for the spam message we will count the repetitions of each word in all the non-spam messages:
Nc — total number of words in the opposite class (for the spam parameter — number of non-spam words), Nci — repetitions of a word in the opposite class (for a word from spam message — the number of repetitions in all non-spam messages). We also use the same smoothing parameters. After the calculation of basic values we start working with the real parameters:
It is the weight for each word in the message of k words. The final decision is calculated by the formula:
So, the classification result is the class with the minimum value of the sum of weights for each word in the message.
cl_compl = sklearn.naive_bayes.ComplementNB()res_compl = cl_compl.fit(X_train_voc, y_train).predict(X_test_voc)metrics.accuracy_score(y_test, res_compl) * 100
We have got 98.12% of correct predictions, which is a significantly high result.
Bernoulli formula is close to the multinomial one, though the input is the set of boolean values (the word is present in the message or not) instead of the set of frequencies.
So, the algorithm explicitly penalizes the non-occurrence of a feature (word in the message is absent in the vocabulary) while the multinomial approach uses the smoothing parameter for the absent values. sklearn Bernoulli algorithm binarizes input values, so, no additional actions required.
cl_bern = sklearn.naive_bayes.BernoulliNB()res_bern = cl_bern.fit(X_train_voc, y_train).predict(X_test_voc)metrics.accuracy_score(y_test, res_bern) * 100
We have got 98.30% of correct predictions
Categorical Naive Bayes is suitable for the categorical values — if the example has the set of features or not. In our case, it means, that the vocabulary is treated as the set of features, and the occurrence of a word in the message is treated as the matching with the feature. All formulas are the same as for the multinomial approach but with the occurrences instead of repetitions.
Since the algorithm needs categorical values, we convert the frequencies of words to the presence of words: 1 — the message contains the word, 0 — the word is absent in the message.
X_train_voc_cat= X_train_voc.applymap(lambda el: 1 if el > 0 else 0)X_test_voc_cat = X_test_voc.applymap(lambda el: 1 if el > 0 else 0)
Now we apply the model to the updated dataset:
cl_cat = sklearn.naive_bayes.CategoricalNB()res_cat=cl_cat.fit(X_train_voc_cat,y_train).predict(X_test_voc_cat)metrics.accuracy_score(y_test, res_cat) * 100
Accuracy is pretty high: 98.30%.
In the article we have acquainted with several Naive Bayes algorithms, have understood the math behind them and have tested the formulas. The results talk by themselves. You can find Jupyter notebook with all the working models on my Github:
github.com
Also, take a look at the implementation of a custom algorithm from scratch without any library with models: