Title: Is there anything incorrect about my implementation of my Naive Bayes Classifier?
Tags: python,machine-learning,nlp,text-classification,naivebayes

Question: I'm building a text classifier using Naive Bayes for a school project. My accuracy on the testing set that my professor provided us is 89.4% which is reasonably high but my professor said that if I correctly implemented Naive Bayes that it should be around a point higher. Is there anything incorrect with my implementation of Naive Bayes or if there's maybe any normalization technique that I should try applying to the documents? Below are the two main functions for my classifier. There are other functions and without them the program won't run but they're mostly related to tuning the model. Please let me know if you need me to clarify anything else or if you want the full version of the code.
Also the training and testing set consists of two columns with the first column being the file path to the document and the second one the class that that document belongs to.
def getStatistics(trainSet):
  #here we're going to calculate the probability of a document being in a certain
  #category by first looking at the fraction of documents in the training set that
  #are in that category

  probCateg = trainSet["Category"].value_counts()

  probCateg = probCateg/probCateg.sum()

  #we create a dictionary that maps a category to the probability of a document being
  #in that category

  probCateg = probCateg.to_dict()


  #to calculate p(t|c) we need to find the probability of a term occuring in 
  #a category

  #So to quickly get the probability of a word appearing in a document of a certain category
  #I used a dictionary which maps the word to another dictionary which in turn maps the category
  #to "C(t|c)". This might not be as efficient as having the dictionary map to a list of 
  #prob values (lookup is still O(1) but hashing the key is probably more computationally expensive than retrieving 
  #element index) but in my opinion it makes the data structure work very well with pandas 
  #so that training my classifier only takes a few lines of code.
  wordFreq = {}


  lemmatizer = WordNetLemmatizer()

  #for each data point in our training set
  for index, row in trainSet.iterrows():

      #we read in the document
      with open(row["Filename"], 'r') as f:
          text = f.read()

          f.close()

      #and split the document into tokens
      #then we're going to reduce all of the words in the document
      #to their base form and then make all of the letters lower case. 
      #This helps make our assumption that the probabilities of each individual
      #word appearing in the document are independent of one another more valid
      #as if you see the word robbery in a document then you're also likely to see
      #words like robbed, robbing, and rob in the document. Additionally, this will
      #make training and classification of our model much faster as there are significantly 
      #less words that it has to calculate probabilities for

      #I also removed stop words and punctuation from the documents since they only seemed
      #to hurt the model by adding noise and it will also make training and classification
      #faster as well

      tokens = nltk.word_tokenize(text)

      #I found that for the third corpus, counting the number of times the word appears in a document
      #instead of just if it occurs in the document signifcantly increases the accuracy by around 4%
      
      tokens = list(set(tokens))
      

      tokens = [token.lower() for token in tokens]
      tokens = [token for token in tokens if (token not in punctuation) and (token not in stopwords)]
      tokens = [(lemmatizer.lemmatize(word)) for word in tokens]

      #we then convert the tokens to a set and then back to a list so that we only 
      #have a list of unique words in the document
      

      tokens = list(set(tokens))
        
      #for each word
      for word in tokens:
        #if the word isn't yet in the dictionary 
        if not (word in wordFreq):
        
          #initialize an entry in the dictionary that maps that word
          #to a dictionary that maps the names of the categories to 0s
          #since so far none of the categories have had that word
          wordFreq[word] = dict(zip(probCateg.keys(), [0]*len(probCateg.keys())))
        
        
        #then if the word is in that document. We increment the # of occurences of that term
        #in this category by 1
        wordFreq[word][row["Category"]] += 1

      

  return wordFreq, probCateg
#in this function, we classify the document
#the function takes in the name of the file in which the document is in
#and a tunable paramter that's used to account for words that don't
#appear in the training set but are present in the testing set
#The function returns the category that the document is most likely
#to be
def classifyDoc(wordFreq, probCateg, filename, eps, trainSize):

  #open and read in the file
  with open(filename, 'r') as f:
        text = f.read()

        f.close()
  
  #and we do the same text processing that we did for the documents in the training set
  lemmatizer = WordNetLemmatizer()

  tokens = nltk.word_tokenize(text)


      
  tokens = list(set(tokens))

  tokens = [token.lower() for token in tokens]
  tokens = [token for token in tokens if (token not in punctuation) and (token not in stopwords)]
  tokens = [(lemmatizer.lemmatize(word)) for word in tokens]


  #we then convert the tokens to a set and then back to a list so that we only 
  #have a list of unique words in the document

  tokens = list(set(tokens))
        
  #since we're going to multiply several very small numbers together, we run 
  #the risk of the value rounding to 0. To avoid this, we take the log probability
  #instead

  #we're going to calculate the log probability for each category
  #so we can hold these values in a dictionary that maps the categories to 
  #their probabilities for that word
  logProb = dict(zip(probCateg.keys(), [math.log(probCateg[key]) for key in probCateg.keys()]))
  
  secCorp = "O" in probCateg.keys()

  #for each category
  for categ in probCateg.keys():
    #to get p(t|c) we divide by total number of documents that are of that category

    denom = trainSize*probCateg[categ]

    #for the second corpus i found that the full form for laplace smoothing gives better results
    if secCorp:
      denom += len(wordFreq)*eps

    #for each word in the document
    for word in tokens:
      
      #if we encountered the word before
      if word in wordFreq:
        #we get C(t|c) from the dictionary and add the log of it to the corresponding
        #log probability plus the smoothing parameter.
        logProb[categ] += math.log(wordFreq[word][categ] + eps)
        
              
      #However, if the word isn't in that category, that would mean that p(t|c) = 0
      #which would mean that p(c|d) = 0 which is unreasonable. Instead,
      #we add a small constant, eps, to all of the term frequencies for all of the categories
      else:

        logProb[categ] += math.log(eps)

      #and then here we divide by denom to get p(t|c) which for log prob is the same as subtracting
      #log(denom)
      logProb[categ] -= math.log(denom)
        


  #after we have calculated the log probabilities we return the category with 
  #the highest probability
  return max(logProb, key=logProb.get)

Answer: So I think I figured out why my classifier wasn't performing as well as I expected it to and it's actually pretty funny and also confusing. When I didn't perform lemmatization on the documents it actually increased my accuracy to 90.3% which feels counterintuitive to me. I always thought that lemmatization was just one of those things you always do when you're processing a text. I think the reason for this jump in performance is that the verb tense can be used to distinguish categories in this corpus. The categories in this corpus are the categories of news articles and they are crime, discovery, politics, struggle, and other. In politics, we talk about statistics and trends whereas the crime category mostly covers individual things that have happened. So the crime section would be more likely to have singular words like robbery and murder whereas the politics section would have the plural forms of those words: robberies and murders.