Multinomial Naive Bayes Classifier for Text Analysis (Python) | by Syed Sadat Nazrul | Towards Data Science

One of the most popular applications of machine learning is the analysis of categorical data, specifically text data. Issue is that, there are a ton of tutorials out there for numeric data but very little for texts. Considering how most of my past blogs on Machine Learning were based on Scikit-Learn, I decided to have some fun with this one by implementing the whole thing on my own.
In this blog, I will cover how you can implement a Multinomial Naive Bayes Classifier for the 20 Newsgroups dataset. The 20 newsgroups dataset comprises around 18000 newsgroups posts on 20 topics split in two subsets: one for training (or development) and the other one for testing (or for performance evaluation). The split between the train and test set is based upon a messages posted before and after a specific date.
First, let us import the libraries needed for writing the implementation:
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport operator
First, we calculate the fraction of documents in each class:
#Training labeltrain_label = open('20news-bydate/matlab/train.label')#pi is the fraction of each classpi = {}#Set a class index for each document as keyfor i in range(1,21):    pi[i] = 0    #Extract values from training labelslines = train_label.readlines()#Get total number of documentstotal = len(lines)#Count the occurence of each classfor line in lines:    val = int(line.split()[0])    pi[val] += 1#Divide the count of each class by total documents for key in pi:    pi[key] /= total    print("Probability of each class:")print("\n".join("{}: {}".format(k, v) for k, v in pi.items()))
Let’s first create the Pandas Dataframe
#Training datatrain_data = open('20news-bydate/matlab/train.data')df = pd.read_csv(train_data, delimiter=' ', names=['docIdx', 'wordIdx', 'count'])#Training labellabel = []train_label = open('/home/sadat/Downloads/HW2_210/20news-bydate/matlab/train.label')lines = train_label.readlines()for line in lines:    label.append(int(line.split()[0]))#Increase label length to match docIdxdocIdx = df['docIdx'].valuesi = 0new_label = []for index in range(len(docIdx)-1):    new_label.append(label[i])    if docIdx[index] != docIdx[index+1]:        i += 1new_label.append(label[i]) #for-loop ignores last value#Add label columndf['classIdx'] = new_labeldf.head()
For calculating our probability, we will find the average of each word for a given class.
For class j and word i, the average is given by:
However, since some words will have 0 counts, we will perform a Laplace Smoothing with low ɑ:
where V is an array of all the words in the vocabulary
#Alpha value for smoothinga = 0.001#Calculate probability of each word based on classpb_ij = df.groupby(['classIdx','wordIdx'])pb_j = df.groupby(['classIdx'])Pr =  (pb_ij['count'].sum() + a) / (pb_j['count'].sum() + 16689)    #Unstack seriesPr = Pr.unstack()#Replace NaN or columns with 0 as word count with a/(count+|V|+1)for c in range(1,21):    Pr.loc[c,:] = Pr.loc[c,:].fillna(a/(pb_j['count'].sum()[c] + 16689))#Convert to dictionary for greater speedPr_dict = Pr.to_dict()Pr
Stop words are words that show up a lot in every document (e.g. prepositions and pronouns).
#Common stop words from onlinestop_words = ["a", "about", "above", "across", "after", "afterwards", "again", "all", "almost", "alone", "along", "already", "also",    "although", "always", "am", "among", "amongst", "amoungst", "amount", "an", "and", "another", "any", "anyhow", "anyone", "anything", "anyway", "anywhere", "are", "as", "at", "be", "became", "because", "become","becomes", "becoming", "been", "before", "behind", "being", "beside", "besides", "between", "beyond", "both", "but", "by","can", "cannot", "cant", "could", "couldnt", "de", "describe", "do", "done", "each", "eg", "either", "else", "enough", "etc", "even", "ever", "every", "everyone", "everything", "everywhere", "except", "few", "find","for","found", "four", "from", "further", "get", "give", "go", "had", "has", "hasnt", "have", "he", "hence", "her", "here", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", "him", "himself", "his", "how", "however", "i", "ie", "if", "in", "indeed", "is", "it", "its", "itself", "keep", "least", "less", "ltd", "made", "many", "may", "me", "meanwhile", "might", "mine", "more", "moreover", "most", "mostly", "much", "must", "my", "myself", "name", "namely", "neither", "never", "nevertheless", "next","no", "nobody", "none", "noone", "nor", "not", "nothing", "now", "nowhere", "of", "off", "often", "on", "once", "one", "only", "onto", "or", "other", "others", "otherwise", "our", "ours", "ourselves", "out", "over", "own", "part","perhaps", "please", "put", "rather", "re", "same", "see", "seem", "seemed", "seeming", "seems", "she", "should","since", "sincere","so", "some", "somehow", "someone", "something", "sometime", "sometimes", "somewhere", "still", "such", "take","than", "that", "the", "their", "them", "themselves", "then", "thence", "there", "thereafter", "thereby", "therefore", "therein", "thereupon", "these", "they","this", "those", "though", "through", "throughout","thru", "thus", "to", "together", "too", "toward", "towards","under", "until", "up", "upon", "us","very", "was", "we", "well", "were", "what", "whatever", "when","whence", "whenever", "where", "whereafter", "whereas", "whereby","wherein", "whereupon", "wherever", "whether", "which", "while", "who", "whoever", "whom", "whose", "why", "will", "with","within", "without", "would", "yet", "you", "your", "yours", "yourself", "yourselves"]
Now, let’s create the vocabulary dataframe
vocab = open('vocabulary.txt') vocab_df = pd.read_csv(vocab, names = ['word']) vocab_df = vocab_df.reset_index() vocab_df['index'] = vocab_df['index'].apply(lambda x: x+1) vocab_df.head()
Getting the counts of each word in the vocabulary and setting stop words to 0:
#Index of all wordstot_list = set(vocab_df['index'])#Index of good wordsvocab_df = vocab_df[~vocab_df['word'].isin(stop_words)]good_list = vocab_df['index'].tolist()good_list = set(good_list)#Index of stop wordsbad_list = tot_list - good_list#Set all stop words to 0for bad in bad_list:    for j in range(1,21):        Pr_dict[j][bad] = a/(pb_j['count'].sum()[j] + 16689)
Multinomial Naive Bayes Classifier
Combining probability distribution of P with fraction of documents belonging to each class.
For class j, word i at a word frequency of f:
In order to avoid underflow, we will use the sum of logs:
One issue is that, if a word appears again, the probability of it appearing again goes up. In order to smooth this, we take the log of the frequency:
Also, in order to take stop words into account, we will add a Inverse Document Frequency (IDF)weight on each word:
Even though the stop words have already been set to 0 for this specific use case, the IDF implementation is being added to generalize the function.
#Calculate IDF tot = len(df['docIdx'].unique()) pb_ij = df.groupby(['wordIdx']) IDF = np.log(tot/pb_ij['docIdx'].count()) IDF_dict = IDF.to_dict()def MNB(df, smooth = False, IDF = False):    '''    Multinomial Naive Bayes classifier    :param df [Pandas Dataframe]: Dataframe of data    :param smooth [bool]: Apply Smoothing if True    :param IDF [bool]: Apply Inverse Document Frequency if True    :return predict [list]: Predicted class ID    '''    #Using dictionaries for greater speed    df_dict = df.to_dict()    new_dict = {}    prediction = []        #new_dict = {docIdx : {wordIdx: count},....}    for idx in range(len(df_dict['docIdx'])):        docIdx = df_dict['docIdx'][idx]        wordIdx = df_dict['wordIdx'][idx]        count = df_dict['count'][idx]        try:             new_dict[docIdx][wordIdx] = count         except:            new_dict[df_dict['docIdx'][idx]] = {}            new_dict[docIdx][wordIdx] = count    #Calculating the scores for each doc    for docIdx in range(1, len(new_dict)+1):        score_dict = {}        #Creating a probability row for each class        for classIdx in range(1,21):            score_dict[classIdx] = 1            #For each word:            for wordIdx in new_dict[docIdx]:                #Check for frequency smoothing                #log(1+f)*log(Pr(i|j))                if smooth:                     try:                        probability=Pr_dict[wordIdx][classIdx]                                 power = np.log(1+ new_dict[docIdx][wordIdx])                             #Check for IDF                        if IDF:                            score_dict[classIdx]+=(                               power*np.log(                               probability*IDF_dict[wordIdx]))                        else:                            score_dict[classIdx]+=power*np.log(                                                   probability)                    except:                        #Missing V will have log(1+0)*log(a/16689)=0                         score_dict[classIdx] += 0                                        #f*log(Pr(i|j))                else:                     try:                        probability = Pr_dict[wordIdx][classIdx]                                power = new_dict[docIdx][wordIdx]                                       score_dict[classIdx]+=power*np.log(                                           probability)                         #Check for IDF                        if IDF:                            score_dict[classIdx]+= power*np.log(                                   probability*IDF_dict[wordIdx])                     except:                        #Missing V will have 0*log(a/16689) = 0                        score_dict[classIdx] += 0                  #Multiply final with pi                     score_dict[classIdx] +=  np.log(pi[classIdx])                                  #Get class with max probabilty for the given docIdx         max_score = max(score_dict, key=score_dict.get)        prediction.append(max_score)            return prediction
Comparing the effects of smoothing and IDF:
regular_predict = MNB(df, smooth=False, IDF=False)smooth_predict  = MNB(df, smooth=True, IDF=False)tfidf_predict   = MNB(df, smooth=False, IDF=True)all_predict     = MNB(df, smooth=True, IDF=True)#Get list of labelstrain_label = pd.read_csv('20news-bydate/matlab/train.label',                          names=['t'])train_label= train_label['t'].tolist()total = len(train_label) models = [regular_predict, smooth_predict,           tfidf_predict, all_predict] strings = ['Regular', 'Smooth', 'IDF', 'Both']  for m,s in zip(models,strings):    val = 0    for i,j in zip(m, train_label):        if i != j:            val +=1        else:            pass       print(s,"Error:\t\t",val/total * 100, "%")
As we can see, IDF has little effect as we removed the stop words. Smoothing, however, makes the model more accurate.
Hence, our optimal model is:
Now that we have out model, let’s use it to predict our test data.
#Get test datatest_data = open('20news-bydate/matlab/test.data')df = pd.read_csv(test_data, delimiter=' ', names=['docIdx', 'wordIdx', 'count'])#Get list of labelstest_label = pd.read_csv('/home/sadat/Downloads/HW2_210/20news-bydate/matlab/test.label', names=['t'])test_label= test_label['t'].tolist()#MNB Calculationpredict = MNB(df, smooth = True, IDF = False)total = len(test_label)val = 0for i,j in zip(predict, test_label):    if i == j:        val +=1    else:        passprint("Error:\t",(1-(val/total)) * 100, "%")