Naïve Bayes vs. SVM For Text Classification
By: Shantam Kar
Tags: Machine Learning, Naive Bayes Classifier, Svm, Data Science, Text Classification

Introduction

In this article, I will highlight the various aspects of the Support vector machine that makes it different from the Naïve Bayes approach for text classification.

But first thing first!

What is Text Classification?

The classification of text into predefined categories using some automated techniques such as machine learning is known as text classification. An example includes — classifying Emails into spam or hams, tweets into different polarities like Positive, Negative, Disgust, Anger, Fear, Joy, etc. Classification Algorithms like Naïve Bayes, SVM, and even Neural Networks can be used for the above process.

First, it was your secretaries in the offices that used to separate out the important letters, priorities of your meeting based on context, now it’s the machine enabled secretaries that separates the spams out for you. The only downside is that you won’t be able to take this secretary for dinners.

Any who.

Let’s move on to,

The pipeline required to apply the above-mentioned machine learning algorithms :

The Machine learning models need some preprocessing steps as it does not understand or learn from raw text data. Instead, the following methods are used to convert the text into numeric : (aka word2vec)

The simplest common method is:

A Bag of Words Approach (in a nutshell) :

Need to first create tokens using tokenization that is turning a string or documents into the token (smaller chunks), the first step in preparing a text for NLP, you can use nltk.tokenize import word_tokenize

Lemmatization / Stemming: Shorten the words to their root stems

Removing Stop words: Removing punctuations, or unwanted token that doesn’t carry a lot of meaning based on the context

Lowercasing the words also helps

Count up all the tokens (which is known as term frequency)

The more the frequent a word, the more important it might be

Can be a great way to determine the significant words in a text

Note: Every use case has a different pipeline and the above mentioned are just basic preprocessing.

In some cases, to understand the language it is important to keep the raw text of data.

But there is another method that overcomes a problem with Bag of words ( where only term frequency is taken into account ) that is TF-IDF: stands for term frequency-inverse document frequency :

What’s the importance of TF-IDF?

Well the idea behind TF- IDF is that each corpus may have shared words beyond just stop words, these words are like stopwords, and need to be removed or down-weighted in importance, example is in an Article regarding Astronomy that word “Sky” might often pop up but it’s not important.

This ensures that these common words do not show up as the keywords and keeps the document-specific frequent words weighted high. Along with that, some rare words such as “liger” in cat articles will be removed as well.

TF-IDF formula:

1st the weight will be low if the term does not appear that often because the tf variable will then be low. However, the weight will also be low if the log value is near to zero meaning the internal equation is low.

Meaning that if the term appears in most or all the documents and when we divide the frequency by the total number of documents it will be close to one which will result in a log value close to zero. So, words that occur across in many or all documents will have very low weight but on the contrary, if the words across rarely that weight would be high.

We can use gensim.model.tfidfmodel import TfidfModel for the purpose

Ok, now let’s come back to our model selection:

Naive Bayes :

Probability of observing features f1 and f2

The posterior probability after the independence assumption

As the name suggests that Naive Bayes must be performing some naive operations for classification, going down into the math of Naive Bayes, it assumes in the posterior probability that every element is Independent of each other provided the evidence. Well, it works surprisingly well for most of the texts where the corpus is small. ( such as tweets, or quotes )

But what comes as a boon acts as imprecation for the algorithm since in most cases the assumption fails to hold its significance. The price paid for such ease is it cannot learn interactions between features because of the class-condition independence assumption made above which is rarely true in most real-world applications. Resulting in the loss of accuracy.

For example: what the Naive Bayes Test case can’t learn is that although you love cats, you hate big cats which might have an association with “Tigers” or “Lions”

Now let’s talk about the elephant in the room!

Support Vector Machine :

Support Vector Machine is considered to be one the best out of box model of all time. Because rather than taking a probabilistic approach SVM works on the geometric interpretation of the problems. The text being a high dimension problem fits right into its core since the model is independent of dimensions (ability to learn is independent of the dimensionality of the features space).

Also, due to preprocessing steps of text classification the corpus now contains very few irrelevant features and a good classifier would like to also take into account the interaction between the term to understand a “dense” concept which Naive Bayes fails due to its assumption but since SVM has the ability to plug a number of kernels which include Linear kernel can easily process on such predictors and separate out the independent term to max distance. ( try to relate the cat example here ).

Let’s come back to the document term matrix that we created earlier in the preprocessing step and let us understand its significance. The number of unique words in a given document is a small, small fraction of the total number of unique words in the corpus ( more zeros than one)… So the document vectors are sparse. SVM thrives on sparse instances.

Next, The above image shows how using RBF Kernel does not affect the performance when compared to linear kernel even if there are a lot of features, this is significant in concluding that most of the text classification problem can be solved by the linear kernel. (but does not hold true for all cases)

Moving on to some exploration over the web I came across a very resourceful guide based on which I was able to formulate this article:

Let me share with you the result of an experiment conducted by the Computer Science Department of Cornell University. Where they have used some traditional methods (including Naïve Bayes) and compared it with SVM.

As per the experiment done on Reuters corpus. The Precision/Recall Break-Even Point is used as a measure of performance and micro averaging is applied to get a single performance value for overall binary classification tasks. You can get more information about the experiment from here: http://www.cs.cornell.edu/people/tj/publications/joachims_98a.pdf

So the Conclusion Now :

Which algorithm is better for text classification, well there is no single answer and there will never be since it all comes down to use cases, for example in telemedicine it is important to classify the medicine effectively from the rest of the corpus of data. Based on the dosage we can predict whether the patient is suffering from mild or severe conditions. Thus using SVM would be a safe bet because it is not prone to catastrophic failures along with that the algorithm is able to correlate with other elements within the corpus that help to understand the dense features in NLP resulting in sentimental analysis or machine translation, but in case of Naive Bayes, the results are not consistent enough. Though it is more of a generalized algorithm that works effectively and cheaply when we want to classify a small corpus of data having a relatively small amount of input features, you don’t expect the inputs to be meaningfully correlated. If an algorithm is not able to capture the correlation between different worlds and its intensity then how can you expect it to learn “Word Order” Also, removing stop words and performing TD-IDF adds on to the misery of NB as its takes away a good chunk of raw text. There are other forms of Naïve Bayes that require hyperparameter tuning that results in much better results.

References :

Note: This article is written by a beginner for beginners. Please do comment if you find any contradicting statements or false statements. Would be happy to rectify.