Exploring the Next Word Predictor!
By: Dhruvil Shah
Tags: NLP, Python, Markov Chains, Data Science, Deep Learning

Introduction to Language Prediction

How does the keyboard on your phone know what you would like to type next? Language prediction is a Natural Language Processing - NLP application concerned with predicting the text given in the preceding text. Auto-complete or suggested responses are popular types of language prediction. The first step towards language prediction is the selection of a language model. This article shows different approaches one can adopt for building the Next Word Predictor you have in apps like Whatsapp or any other messaging app.

There are generally two models you can use to develop Next Word Suggester/Predictor: 1) N-grams model or 2) Long Short Term Memory (LSTM). We will go through every model and conclude which one is better.

N-grams approach

If you’re going down the n-grams path, you’ll need to focus on the ‘Markov Chains’ to predict the likelihood of each following word or character based on the training corpus. Below is the snippet of the code for this approach. In this approach, the sequence length of one is taken for predicting the next word. This means we will predict the next word given in the previous word.

Importing necessary modules: word_tokenize, defaultdict, Counter

import re

from nltk.tokenize import word_tokenize

from collections import defaultdict, Counter

Creating the class MarkovChain containing methods:

class MarkovChain:

def __init__(self):

self.lookup_dict = defaultdict(list) def _preprocess(self, string):

cleaned = re.sub(r’\W+’, ' ', string).lower()

tokenized = word_tokenize(cleaned)

return tokenized def add_document(self, string):

preprocessed_list = self._preprocess(string)

pairs = self.__generate_tuple_keys(preprocessed_list)

for pair in pairs:

self.lookup_dict[pair[0]].append(pair[1]) def __generate_tuple_keys(self, data):

if len(data) < 1:

return

for i in range(len(data) - 1):

yield [ data[i], data[i + 1] ]



def generate_text(self, string):

if len(self.lookup_dict) > 0:

print("Next word suggestions:", Counter(self.lookup_dict[string]).most_common()[:3])

return

When we create an instance of the above class a default dictionary is initialized. There is a method to preprocess the training corpus that we add via the .add_document() method. When we add a document with the help of the .add_document() method, pairs are created for each unique word. Let’s understand this with an example: if our training corpus was “How are you? How many days since we last met? How are your parents?” our lookup dictionary, after preprocessing and adding the document, would be:

{ 'how': ['are', 'many', 'are'], 'are': ['you', 'your'],

'you': ['how'], 'many': ['days'], 'days': ['since'],

'since': ['we'], 'we': ['last'], 'last': ['met'], 'met': ['how'],

'your': ['parents']}

Each unique word as a key and its following words’ list as a value is added to our lookup dictionary lookup_dict.

When we input a word it will be looked up in the dictionary and the most common words in its list of following words will be suggested. Here, the maximum number of word suggestions is three like we have in our keyboards. Below is the running example of this approach for the sequence length of one. The output contains suggested words and their respective frequency in the list. When we enter the word ‘how’, it is looked up in the dictionary and the most common three words from its list of following words are chosen. Here, ‘many’ word appears 1531 times meaning the word sequence ‘How many’ appears 1531 times in the training corpus.

The output of the n-grams model (sequence length of one)

Further in the n-grams approach:

Further, in the above-explained method, we can have a sequence length of 2 or 3 or more. For this, we will have to change some of the code above.

class MarkovChain:

"""

Previous code continued

"""

def add_document(self, string):

preprocessed_list = self._preprocess(string)

pairs = self.__generate_tuple_keys(preprocessed_list)

for pair in pairs:

self.lookup_dict[pair[0]].append(pair[1])

pairs2 = self.__generate_2tuple_keys(preprocessed_list)

for pair in pairs2:

self.lookup_dict[tuple([pair[0], pair[1]])].append(pair[2])

pairs3 = self.__generate_3tuple_keys(preprocessed_list)

for pair in pairs3:

self.lookup_dict[tuple([pair[0], pair[1], pair[2]])].append(pair[3])



def __generate_tuple_keys(self, data):

if len(data) < 1:

return

for i in range(len(data) - 1):

yield [ data[i], data[i + 1] ]



#to add two words tuple as key and the next word as value

def __generate_2tuple_keys(self, data):

if len(data) < 2:

return

for i in range(len(data) - 2):

yield [ data[i], data[i + 1], data[i+2] ]



#to add three words tuple as key and the next word as value

def __generate_3tuple_keys(self, data):

if len(data) < 3:

return

for i in range(len(data) - 3):

yield [ data[i], data[i + 1], data[i+2], data[i+3] ]



def oneword(self, string):

return Counter(self.lookup_dict[string]).most_common()[:3] def twowords(self, string):

suggest = Counter(self.lookup_dict[tuple(string)]).most_common()[:3]

if len(suggest)==0:

return self.oneword(string[-1])

return suggest def threewords(self, string):

suggest = Counter(self.lookup_dict[tuple(string)]).most_common()[:3]

if len(suggest)==0:

return self.twowords(string[-2:])

return suggest



def morewords(self, string):

return self.threewords(string[-3:]) def generate_text(self, string):

if len(self.lookup_dict) > 0:

tokens = string.split(" ")

if len(tokens)==1:

print("Next word suggestions:", self.oneword(string))

elif len(tokens)==2:

print("Next word suggestions:", self.twowords(string.split(" ")))

elif len(tokens)==3:

print("Next word suggestions:", self.threewords(string.split(" ")))

elif len(tokens)>3:

print("Next word suggestions:", self.morewords(string.split(" ")))

return

Let’s break the code. Methods .__generate_2tuple_keys() and .__generate_3tuple_keys() are to store the sequences of length two and three respectively and their following words’ list. Now, our code has the strength to predict words based on up to three previous words. Let’s look at our new lookup dictionary lookup_dict for the example: “How are you? How many days since we last met? How are your parents?”

{

"""

Same as before

"""

('how', 'are'): ['you', 'your'],

...

('how', 'many'): ['days'],

('many', 'days'): ['since'],

...

('how', 'are', 'you'): ['how'],

...

('how', 'many', 'days'): ['since'],

...

}

New pairs are added to the dictionary compared to the previous one. The class MarkovChain that we created above handles any length of a sequence we input. If we input one word then the method ‘oneword’ will be called and this will be the same as the previous one. For input length two or three the methods ‘twowords’ and ‘threewords’ will be called respectively. What these methods do is that they look for the most common three words from the lookup dictionary, given the input words. When input words are more than four then the last three will be processed. When encountered an unknown word, that word will be ignored and the rest of the string will be processed. Look at the figure below to clear any doubts. This figure is based on a different training corpus. The left side shows the input and the right side, the output.

Various cases of input for our n-grams model

Below is the running output of this approach:

The output of the n-grams model (sequence length of more than one)

The above output is based on a different and bigger dataset that was used for this approach. GitHub’s link for this approach is this. You can find the above code there.

Limitations:

Markov chains do not have memory. There are many limitations to adopting this approach. Take an example, “I ate so many grilled …” next word “sandwiches” will be predicted based on how many times “grilled sandwiches” have appeared together in the training data. As we are getting suggestions based only on the frequency, there are many scenarios where this approach could fail.

Long Short Term Memory (LSTM) approach:

A more advanced approach, using a neural language model, is to use Long Short Term Memory (LSTM). LSTM model uses Deep learning with a network of artificial “cells” that manage memory, making them better suited for text prediction than traditional neural networks and other models.

“ The grass is always … ” The next word is simply “green” and could be predicted by most models and networks. But for the sentence, “ It’s winter and there has been little sunlight, the grass is always … ”, we need to know the context from further back in the sentence to predict the next word “brown”.

Standard RNNs and other language models become less accurate when the gap between the context and the word to be predicted increases. Here’s when LSTM comes in use to tackle the long-term dependency problem because it has memory cells to remember the previous context. You can learn more about LSTM networks here.

Let’s start coding and define our LSTM model. In building our model, first, an embedding layer, two stacked LSTM layers with 50 units each are used.

Keras offers an embedding layer that can be used for neural networks on text data. The Embedding layer is initialized with random weights and learns embeddings for all of the words in the training dataset. It requires the input data in an integer encoded form. This data preparation step can be performed with the help of Tokenizer API also provided by Keras. Learn more about Embedding layer here.

The two LSTM layers are followed by two fully connected or dense layers. The first layer has 50 units and the second dense layer is our output (softmax) layer and has the number of units equal to the vocabulary size. As for each input, the model will predict the next word from our vocabulary based on the probability. Categorical cross-entropy is used as a loss function.

Preprocessing the data:

For input to the Embedding layer, we first have to use Tokenizer from keras.processing.text to encode our input strings. What we are doing in preprocessing is simple: We first create features dictionary sequences. Then we encode it into the integer form with the help of the Tokenizer.

from keras.preprocessing.text import Tokenizer

import nltk

from nltk.tokenize import word_tokenize

import numpy as np

import re

from keras.utils import to_categorical

from doc3 import training_doc3 cleaned = re.sub(r'\W+', ' ', training_doc3).lower()

tokens = word_tokenize(cleaned)

train_len = 4

text_sequences = [] for i in range(train_len,len(tokens)):

seq = tokens[i-train_len:i]

text_sequences.append(seq) sequences = {}

count = 1 for i in range(len(tokens)):

if tokens[i] not in sequences:

sequences[tokens[i]] = count

count += 1 tokenizer = Tokenizer()

tokenizer.fit_on_texts(text_sequences)

sequences = tokenizer.texts_to_sequences(text_sequences) #vocabulary size increased by 1 for the cause of padding

vocabulary_size = len(tokenizer.word_counts)+1

n_sequences = np.empty([len(sequences),train_len], dtype='int32') for i in range(len(sequences)):

n_sequences[i] = sequences[i] train_inputs = n_sequences[:,:-1]

train_targets = n_sequences[:,-1]

train_targets = to_categorical(train_targets, num_classes=vocabulary_size)

seq_len = train_inputs.shape[1]

Let’s understand what is happening in the code above with an example: “How are you? How many days since we last met? How are your parents?”. We first clean our corpus and tokenize it with the help of Regular expressions, and word_tokenize from nltk library. What does the ‘sequences’ dictionary do? Below is the ‘sequences’ dictionary before using the tokenizer.

{'how': 1, 'are': 2, 'you': 3, 'many': 4, 'days': 5, 'since': 6, 'we': 7, 'last': 8, 'met': 9, 'your': 10, 'parents': 11}

Our ‘text_sequences’ list keeps all the sequences in our training corpus and it would be:

[['how', 'are', 'you', 'how'], ['are', 'you', 'how', 'many'], ['you', 'how', 'many', 'days'], ['how', 'many', 'days', 'since'], ['many', 'days', 'since', 'we'], ['days', 'since', 'we', 'last'], ['since', 'we', 'last', 'met'], ['we', 'last', 'met', 'how'], ['last', 'met', 'how', 'are'], ['met', 'how', 'are', 'your']]

After using tokenizer we have the above sequences in the encoded form. The numbers are nothing but the indexes of the respective words from the ‘sequences’ dictionary before re-assignment.

[[1, 2, 9, 1], [2, 9, 1, 3], [9, 1, 3, 4], [1, 3, 4, 5], [3, 4, 5, 6], [4, 5, 6, 7], [5, 6, 7, 8], [6, 7, 8, 1], [7, 8, 1, 2], [8, 1, 2, 10]]

Once we have our sequences in encoded form training data and target data is defined by splitting the sequences into the inputs and output labels. As for this example, we are going to predict the next word based on three previous words so in training we use the first three words as input and the last word as a label that is to be predicted by the model. Our ‘training_inputs’ would now be:

[[1 2 9] [2 9 1] [9 1 3] [1 3 4] [3 4 5] [4 5 6] [5 6 7] [6 7 8] [7 8 1] [8 1 2]]

Then, we convert our output labels into one-hot vectors i.e into combinations of 0’s and 1. One-hot vectors in ‘train_targets’ would look like:

[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

...

[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]

For the first target label “how”, the index was ‘1’ in sequence dictionary so in the encoded form you’d expect ‘1’ at the place of index 1 in the first one-hot vector of ‘train_targets’.

Note: The above code is explained for the text “How are you? How many days since we last met? How are your parents?” for a simpler explanation. But in reality, a bigger dataset is used.

Building the model:

Now we train our Sequential model that has 5 layers: An Embedding layer, two LSTM layers, and two Dense layers. In the input layer of our model i.e. Embedding layer, the input length is set to the size of a sequence that is 3 for this example. (Note: We split the data for training inputs and training targets as 3 to 1, so when we give input to our model for prediction we will have to provide 3 length vector.)

from keras.models import Sequential

from keras.layers import Dense

from keras.layers import LSTM

from keras.layers import Embedding

model = Sequential()

model.add(Embedding(vocabulary_size, seq_len, input_length=seq_len))

model.add(LSTM(50,return_sequences=True))

model.add(LSTM(50))

model.add(Dense(50,activation='relu'))

model.add(Dense(vocabulary_size, activation='softmax')) # compiling the network

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(train_inputs,train_targets,epochs=500,verbose=1)

Predicting words:

After our model is trained we can give input in the encoded form and get the three most probable words from the softmax function as shown below.

from keras.preprocessing.sequence import pad_sequences input_text = input().strip().lower()

encoded_text = tokenizer.texts_to_sequences([input_text])[0]

pad_encoded = pad_sequences([encoded_text], maxlen=seq_len, truncating='pre')

print(encoded_text, pad_encoded) for i in (model.predict(pad_encoded)[0]).argsort()[-3:][::-1]:

pred_word = tokenizer.index_word[i]

print("Next word suggestion:",pred_word)

In the above code, we use padding because we trained our model on sequences of length 3, so when we input 5 words, padding will ensure that the last three words are taken as an input to our model. What happens when we input less than 3 words? We will not get the best results! The same happens when we input an unknown word as the one-hot vector will contain 0 in that word’s index. What we can do in the future is we add sequences of length 2(inputs) to 1(target label) and 1(input) to 1(target label) as we did here 3(inputs) to 1(target label) for best results.

Below is the final output of our model predicting the next 3 words based on the previous words.

The output of the LSTM model

The above output shows the vector form of the input along with the suggested words. [6, 4, 3] is the ‘encoded_text’ and [[6, 4, 3]] is the ‘pad_encoded’.

Note: Here we split our data as 3(inputs) to 1(target label). Therefore, we must input three words.

GitHub’s link for the above code is this. You can find the code of the LSTM approach there.

Conclusion:

Above, we saw that the n-grams approach is inferior to the LSTM approach as LSTMs have the memory to remember the context from further back in the text corpus. While starting a new project, you might want to consider one of the existing pre-trained frameworks by looking on the internet for open-source implementations. This way, you will not have to start from scratch and you don’t need to worry about the training process or hyperparameters.

Photo by Josh Riemer on Unsplash

Much recent work within Natural Language Processing domain includes the development and training of the neural models to approximate the way our human brains exert towards language. This deep learning approach enables computers to mimic the human language in a far more efficient way.