Language Modeling With NLTK. Building and studying statistical…
By: Ruthu S Sanketh
Tags: Language Model, Statistical Modeling, Naturallanguageprocessing, NLP, Ngrams

Language Modeling With NLTK

Building and studying statistical language models from a corpus dataset using Python and the NLTK library.

To get an introduction to NLP, NLTK, and basic preprocessing tasks, refer to this article. If you’re already acquainted with NLTK, continue reading!

What are Language Models?

A language model learns to predict the probability of a sequence of words. But why do we need to learn the probability of words?

In machine translation, we take in a bunch of words from a language and convert these words into another language. Now, there can be many potential translations that a system might give us and we will want to compute the probability of each of these translations to understand which one is the most accurate. For example, looking at word orderings -

p(the lady is beautiful) > p(beautiful the is lady)

In the above example, we know that the probability of the first sentence will be more than the second. That’s how we arrive at the right translation.

This ability to model the rules of a language as a probability gives great power for NLP related tasks. Language models are used in speech recognition, machine translation, part-of-speech tagging, parsing, optical character recognition, handwriting recognition, information retrieval, and many other daily tasks.

Neural Networks in NLP — Image by Flickr on Unsplash

Types of Language Models

There are primarily two types of language models -

Statistical Language Models - These models use traditional statistical techniques like n-grams, Hidden Markov Models (HMM) and certain linguistic rules to learn the probability distribution of words. Neural Language Models - These models surpass statistical language models in their effectiveness. They use different kinds of neural networks to model language.

“You shall know the nature of a word by the company it keeps.” - John Rupert Firth

Now that we have a basic idea about language models, let’s build them. In this article, we will look at statistical language analysis using n-gram models, with and without smoothing. Then we will look at next-word prediction using different n-gram models and their relative performance.

What are N-gram Models?

Given a sequence of n-1 words, an n-gram model predicts the most probable word that might follow this sequence. It’s a probabilistic model that’s trained on a corpus of text. An n-gram model is built by counting how often word sequences occur in corpus text and then estimating the probabilities. Since a simple n-gram model has limitations, improvements are often made via smoothing or interpolation.

Consider two sentences - ‘There was heavy rainfall’ vs ‘There was huge rainfall’. From experience, we know that the former sentence sounds better. Ideally, an n-gram model will tell us that ‘heavy rainfall’ occurs much more often than ‘huge rainfall’ in the training corpus. Thus, the first sentence is more probable and will be selected by the model.

A model that simply relies on how often a word occurs without looking at previous words is called a unigram model. If a model considers only the previous word to predict the current word, then it’s called bigram. If two previous words are considered, then it’s a trigram model, and so on.

An n-gram model for the above example would calculate the following probability -

p(There was heavy rainfall) = p(There, was, heavy, rainfall) = p(There)p(was|There)p(heavy|There was)p(rainfall|There was heavy)

Since it’s impractical to calculate these conditional probabilities, using the Markov assumption, we can approximate this to a bigram model where the occurrence of each word only depends on the occurrence of the previous word giving the probability as -

p(There was heavy rainfall) ~ p(There) p(was|There)p(heavy|was)p(rainfall|heavy)

Let us calculate these probabilities using code.

Initial Steps

First we import the dataset and the required libraries. The dataset used in this article can be accessed here.

# import libraries needed, read the dataset import nltk, re, pprint, string

from nltk import word_tokenize, sent_tokenize

string.punctuation = string.punctuation +'“'+'”'+'-'+'’'+'‘'+'—'

string.punctuation = string.punctuation.replace('.', '')

file = open('corpus.txt', encoding = 'utf8').read()

We have imported string punctuation and added some other special characters present in the dataset which we would like to remove. The period character has been removed from string punctuation so that we can count the number of sentences in the dataset, after which we can remove periods from the dataset as well. We now preprocess the data.

#preprocess data

file_nl_removed = ""

for line in file:

line_nl_removed = line.replace("

", " ") #removes newlines

file_nl_removed += line_nl_removed file_p = "".join([char for char in file_nl_removed if char not in string.punctuation]) #removes all special characters

Now that we have removed special characters and newlines, we can get some statistics about our data.

sents = nltk.sent_tokenize(file_p)

print("The number of sentences is", len(sents))

#prints the number of sentences words = nltk.word_tokenize(file_p)

print("The number of tokens is", len(words))

#prints the number of tokens average_tokens = round(len(words)/len(sents))

print("The average number of tokens per sentence is",

average_tokens)

#prints the average number of tokens per sentence unique_tokens = set(words)

print("The number of unique tokens are", len(unique_tokens))

#prints the number of unique tokens

Statistics of the dataset

Building the N-gram Models

We now have some statistical parameters about the data such as the number of unique tokens which can be used while defining the vocabulary size in a model. Next we create the following language models on the training corpus -

1. Unigram

2. Bigram

3. Trigram

4. Fourgram

We list the top 5 bigrams, trigrams, four-grams without smoothing. We remove those which contain only articles, prepositions, determiners, for example, ‘of the’, ‘in a’, etc. These are called stopwords and can be removed by using an inbuilt list from NLTK. Stopwords are the English words which do not add much meaning to a sentence and can safely be ignored without sacrificing the meaning of the sentence.

from nltk.util import ngrams

from nltk.corpus import stopwords

stop_words = set(stopwords.words('english')) unigram=[]

bigram=[]

trigram=[]

fourgram=[]

tokenized_text = [] for sentence in sents:

sentence = sentence.lower()

sequence = word_tokenize(sentence)

for word in sequence:

if (word =='.'):

sequence.remove(word)

else:

unigram.append(word)

tokenized_text.append(sequence)

bigram.extend(list(ngrams(sequence, 2)))

#unigram, bigram, trigram, and fourgram models are created

trigram.extend(list(ngrams(sequence, 3)))

fourgram.extend(list(ngrams(sequence, 4))) def removal(x):

#removes ngrams containing only stopwords

y = []

for pair in x:

count = 0

for word in pair:

if word in stop_words:

count = count or 0

else:

count = count or 1

if (count==1):

y.append(pair)

return(y) bigram = removal(bigram)

trigram = removal(trigram)

fourgram = removal(fourgram) freq_bi = nltk.FreqDist(bigram)

freq_tri = nltk.FreqDist(trigram)

freq_four = nltk.FreqDist(fourgram) print("Most common n-grams without stopword removal and without add-1 smoothing:

")

print ("Most common bigrams: ", freq_bi.most_common(5))

print ("

Most common trigrams: ", freq_tri.most_common(5))

print ("

Most common fourgrams: ", freq_four.most_common(5))

Most common n-grams without stopword removal

We can also remove stopwords entirely from our dataset and find the n-gram models. Let us find the most common n-grams in the dataset after removing all the stopwords from the dataset.

#stopwords = code for downloading stop words through nltk from nltk.corpus import stopwords

stop_words = set(stopwords.words('english')) #prints top 10 unigrams, bigrams after removing stopwords print("Most common n-grams with stopword removal and without add-1 smoothing:

")

unigram_sw_removed = [p for p in unigram if p not in stop_words]

fdist = nltk.FreqDist(unigram_sw_removed)

print("Most common unigrams: ", fdist.most_common(10)) bigram_sw_removed = []

bigram_sw_removed.extend(list(ngrams(unigram_sw_removed, 2)))

fdist = nltk.FreqDist(bigram_sw_removed)

print("

Most common bigrams: ", fdist.most_common(10))

Most common n-grams after stopword removal

In order to get insights into the most common words used in a dataset, stopwords are generally removed to get meaningful results.

What is Add-1 Smoothing?

We have used Maximum Likelihood Estimation (MLE) for training the parameters of an n-gram model. The problem with MLE is that it assigns zero probability to unknown or unseen words. This is because MLE uses a training corpus. If the word in the test set is not available in the training set, then the count of that particular word is zero and it leads to zero probability.

To eliminate this zero probability, we can do smoothing. Smoothing involves taking some probability mass from the events seen in training and assigning it to unseen events. Add-1 smoothing or Laplace smoothing is a simple smoothing technique that adds 1 to the count of all n-grams in the training set before normalizing them into probabilities.

Add-1 Smoothing

We can create a dictionary where each element is a list corresponding to a particular n-gram, and store every word and its associated probability as elements of the list. Add-1 smoothing is performed considering all the unique words in the tokenized text of our dataset as the vocabulary.

#Add-1 smoothing is performed here.



ngrams_all = {1:[], 2:[], 3:[], 4:[]}

for i in range(4):

for each in tokenized_text:

for j in ngrams(each, i+1):

ngrams_all[i+1].append(j); ngrams_voc = {1:set([]), 2:set([]), 3:set([]), 4:set([])} for i in range(4):

for gram in ngrams_all[i+1]:

if gram not in ngrams_voc[i+1]:

ngrams_voc[i+1].add(gram) total_ngrams = {1:-1, 2:-1, 3:-1, 4:-1}

total_voc = {1:-1, 2:-1, 3:-1, 4:-1}

for i in range(4):

total_ngrams[i+1] = len(ngrams_all[i+1])

total_voc[i+1] = len(ngrams_voc[i+1])



ngrams_prob = {1:[], 2:[], 3:[], 4:[]}

for i in range(4):

for ngram in ngrams_voc[i+1]:

tlist = [ngram]

tlist.append(ngrams_all[i+1].count(ngram))

ngrams_prob[i+1].append(tlist)



for i in range(4):

for ngram in ngrams_prob[i+1]:

ngram[-1] = (ngram[-1]+1)/(total_ngrams[i+1]+total_voc[i+1]) #add-1 smoothing

This is a general method to perform add-1 smoothing for any n-gram (the range can be increased from 4 if necessary). Let us see how this can be used to obtain the most common n-grams and their smoothed probabilities for unigrams, bigrams, trigrams and fourgrams.

#Prints top 10 unigram, bigram, trigram, fourgram after smoothing print("Most common n-grams without stopword removal and with add-1 smoothing:

")

for i in range(4):

ngrams_prob[i+1] = sorted(ngrams_prob[i+1], key = lambda x:x[1], reverse = True)



print ("Most common unigrams: ", str(ngrams_prob[1][:10]))

print ("

Most common bigrams: ", str(ngrams_prob[2][:10]))

print ("

Most common trigrams: ", str(ngrams_prob[3][:10]))

print ("

Most common fourgrams: ", str(ngrams_prob[4][:10]))

Most common n-grams after add-1 smoothing

We can also obtain the smoothed probabilities of each n-gram after removal of stopwords, according to the requirement in context of the dataset.

Next Word Prediction

Using the above bigram, trigram, and fourgram models that we just experimented with, we can predict the next word (top 5 probable) given the previous n-gram for the sentences below.

For str1, str2, we predict the next 2 possible word sequences using the trained smoothed models. For example, for the string ‘He looked very’ the next words can be -

(1) ‘He looked very’ *anxiously*

(2) ‘He looked very’ *uncomfortable*

str1 = 'after that alice said the'

str2 = 'alice felt so desperate that she was'

We use the pre-trained smoothed models that were formed without removal of stopwords. We obtain n-grams of the two strings to use as input to get the next word predictions.

#smoothed models without stopwords removed are used token_1 = word_tokenize(str1)

token_2 = word_tokenize(str2) ngram_1 = {1:[], 2:[], 3:[]} #to store the n-grams formed

ngram_2 = {1:[], 2:[], 3:[]}

for i in range(3):

ngram_1[i+1] = list(ngrams(token_1, i+1))[-1]

ngram_2[i+1] = list(ngrams(token_2, i+1))[-1] print("String 1: ", ngram_1,"

String 2: ",ngram_2)

N-grams of the 2 strings

Now we use the n-grams of the strings to get the next word predictions based on the highest probabilities.

for i in range(4):

ngrams_prob[i+1] = sorted(ngrams_prob[i+1], key = lambda x:x[1], reverse = True)



pred_1 = {1:[], 2:[], 3:[]}

for i in range(3):

count = 0

for each in ngrams_prob[i+2]:

if each[0][:-1] == ngram_1[i+1]:

#to find predictions based on highest probability of n-grams



count +=1

pred_1[i+1].append(each[0][-1])

if count ==5:

break

if count<5:

while(count!=5):

pred_1[i+1].append("NOT FOUND")

#if no word prediction is found, replace with NOT FOUND count +=1 for i in range(4):

ngrams_prob[i+1] = sorted(ngrams_prob[i+1], key = lambda x:x[1], reverse = True)



pred_2 = {1:[], 2:[], 3:[]}

for i in range(3):

count = 0

for each in ngrams_prob[i+2]:

if each[0][:-1] == ngram_2[i+1]:

count +=1

pred_2[i+1].append(each[0][-1])

if count ==5:

break

if count<5:

while(count!=5):

pred_2[i+1].append("\0")

count +=1

Now we can call the above method for each n-gram to get the next word predictions according to each model.

print("Next word predictions for the strings using the probability models of bigrams, trigrams, and fourgrams

") print("String 1 - after that alice said the-

") print("Bigram model predictions: {}

Trigram model predictions: {}

Fourgram model predictions: {}

" .format(pred_1[1], pred_1[2], pred_1[3])) print("String 2 - alice felt so desperate that she was-

") print("Bigram model predictions: {}

Trigram model predictions: {}

Fourgram model predictions: {}" .format(pred_2[1], pred_2[2], pred_2[3]))

Next word predictions for each n-gram model

We see that str1 does now show any next words based on the fourgram model. This is because of the insufficient length of the string. We also observe that the fourgram model predictions are more accurate and contextually suitable than the trigram model predictions, which are in turn better than the bigram model predictions, and so on. This further shows that predictions get better when more adjacent words are considered while predicting the next word.

Conclusion

I hope this article clearly explains language models, how to construct them, train them, and use them to get next word predictions. Apart from n-gram language models, there are many other statistical language models that may give better results, such as exponential models, HMM, and also neural network models. The entire code used in this article, as well as the dataset can be accessed here.

Further Reading