Sentiment Classification with Naive Bayes classifier
By: Praveen Reddy
Tags: Machine Learning, Naive Bayes Classifier, Sentiment Analysis, Naturallanguageprocessing, Data Science

In the following, I take a stab at explaining my understanding of how Naive Bayes Classifier can be used for sentiment classification.

What is a probabilistic classifier?

Sum of probabilities (0.3 + 0.5 + 0.2)= 1

A classifier that outputs a probability distribution over a set of output classes. Let's say an event has 3 possible outcomes. The probabilistic classifier predicts the chance for each of these outcomes. Sum of these probabilities for an event equals to 1.

These distributions are called conditional distributions. Which means, for a particular value of input in set X(this is the condition), output is a set of probabilities of all outcomes. It is represented as Pr(Y|X). Outcome that has maximum probability among the possible outcomes is considered as prediction.

What is Naive Bayes classifier?

Bayes theorem states or describes the probability of an event based on other related events. For example, probability of a random driver stopped by a cop being drunk changes depending on time of a day. If a driver is stopped by a cop at 2AM on Friday night, probability increases. If it is 8AM in the morning on a weekday the chances of driver being drunk is less.

Now a naive bayes classifier is a probabilistic classifier that applies bayes theorem with an assumption. The assumption is that the related events that influence the probability of an output are independent of each other. This is assumption is the ‘naive’ part of the naive bayes classifier.

In order to understand this assumption, let's take a task of predicting if I will be late to office today. There are many factors or related events that could impact my being in office on time. Will the alarm go off, traffic on the way, condition of the vehicle I use to commute and many more. Though we may factor in all these events to calculate the probability, we assume that these events are independent of each other and have no correlation between them. Alarm going off on time has nothing to do with the condition of my vehicle and such.

Getting down to mathematical representation of the above when applied to the task of document or text classification. Lets each document (like a movie review or product review) is represented as document d and its classification as class c (like favorable review or unfavorable review).

Probability of a document d belonging to a class c can be represented as:

P(c|d) = P(d|c) * P(c) / P(d)

P(c): is called prior probability. Probability of an outcome without considering any of the related events.

P(d|c): is called likelihood which expresses how likely the document d with all its features belong to a particular class C.

P(d): is called evidence. This can be dropped during the probability calculations as evidence does not vary based on the class.

How to classify text using Naive Bayes classifier?

Even before classification can being, the text needs to be represented in the numeric form that preserves some properties of text such that any predictions made on the numerical form are applicable on the text as well. This form of representation is called is called vectorization.

Bag-of-words representation

In this particular task, I am using bag-of-words representation to get vector representation of the words/text. In this representation the text is (literally) represented as bunch of words in a bag ignoring grammar and order of the words while retaining the count of each word occurance. Vectors generated are features for the classification task.

Within bag of words model, there are multiple ways to represent each word.

Count: Just count the occurance of each word. If a word, ex: ‘good’ occurs 5 times, use 5 to represent ‘good’ as a vector.

Binary: Each word is represented by 1 if it exists in the document regardless of how many number of times it occurs. 0 if it does not occur.

TF-IDF (Term Frequency — Inverse Document Frequency): This is a product of two terms where each term captures an aspect of the word. Term fequency(TF) — How often the term occurs in the document. Inverse document frequency(IDF), gives higher weight to words that only occur in few documents. IDF is the fraction of total number of documents in which the word appears.

tf-idf = (TF) * (IDF)

Frequency: How often a particular word occurs as a fraction of all other words.

With enough theory, lets take a look at an example. The following example is created following an exising example from Speech and Language Processing book.

Annotated reviews

Here we have 5 documents (each review is a document). Of the five, 3 are negative and 2 are positive.

2 possible outcomes — Positive and Negative.

Using these reviews we need to classify a new review to be either positive or negative.

New review

Prior probabilities P(c):

Calculate prior probabilities using the traning data.

P(Negative) = 3/5

P(Positive) = 2/5

Likelyhood probabilities P(c|d):

Calculate the both, probability of positive review given the document and probability of negative review given the document. This is where we use vectors we created using bag-of-words model.

The way we do this is, take each word’s vector from ‘new’ document and calculate the probability of it belonging to each class.

Note: In order to avoid division with zeros and zero probabilities, Laplace smoothing is applied, adding 1 to the word count(s).

Source: Speech and Language Processing by Daniel Jurafsky and James H. Martin

Likelyhood for positive classification:

P(‘Great’|+) = 1+1 / 12+8

P(‘Music’|+) = 0+1 / 12+8

P(‘and’|+) = 1+1 / 12+8

P(‘sensible’|+) = 1+1 / 12+8

P(‘acting’|+) = 1+1 / 12+8

So P(+|New Document) = (2/5) * (2*1*2*2*2/20⁵)

Similarly, likelyhood for negative classification:

P( — |New Document) = (3/5) * (1*1*2*1*1/(15+14)⁵)

Hence from the calculations from above, the new document can be classified as positive review.