Language Model Concept behind Word Suggestion Feature | by Vitou Phy | Towards Data Science

You may have seen word suggestion feature on your smartphone. As you type in one word, the keyboard understands what you are typing and suggests the next relevant word. If you select the suggested word, it will then suggest another one, and another and another until you end up having these funny sentences.
This is a simple concept, known as language modeling. So what language modeling does is it reads the written text and it attempts to assign probability to the next word. Let’s look at an example.
Intuitively, suppose we have 100 words to be considered for the next word, language model takes previous word into consideration and give probability to those 100 words. Then, it can give top 3 options to the users as part of word suggestion feature. Now, let’s give a formal definition to LM.
Commonly, language model is known as a way to give probability to a sentence.
But wait! Isn’t earlier we mentioned LM assign probability to the next word? Let’s break down the equation above to investigate this further.
As you can see in Equation 1, LM not only can assign probability to the next word given but also can give the probability to the whole sentence. Then, the LM can understand which sentence might sound good and which might not.
Awesome, right? But what’s under the hood of such awesome function? There are many ways to create this language model. In this post, we will explore a simple method, called n-gram.
One key idea is ngram looks only at n words at a time. Let’s work with the example “the cat sat on the mat”.
Suppose n=2 (also known as bigram), it will try to predict the second word based on the first one. For example:
the cat: Bigram tries to remember that after “the”, it should be “cat”.
cat sat: after “cat”, it should be “sat”.
sat on: after “sat”, it’s “on”
The same thing goes with n=3. For the words “the cat sat”, trigram knows “sat” comes after “the cat”. I guess you get the idea.
This concept is scalable to n words (n-gram). So, how does ngram know all of these pattern? It’s probability theory.
For simplicity, we will look the case of bigram. From the example above, we know that “cat” comes after “the”, which can be denoted as P(cat | the).
P(cat | the) : the probability that the next word is “cat” given the word “the”.
To compute this probability, simply use this the formula below.
where c(the cat) is number of occurrence of “the cat” in the entire corpus. The same goes for c(the). Intuitively, this equation says that: Among all of the words “the”, how many of them are followed by “cat”? And voila, now we have P(cat | the).
Bigram is simple and quite powerful but there are many cases it will fail to perform. For instance:
In the beginning of the sentence, the model sees “the” followed by “cat”. If we use bigram and rely only on one previous word, it is very difficult to guess that the underline word is “mat” since it probably generates “cat” again. Fortunately, trigram or 4-gram can resolve this issue. i.e. If we see “sat on the”, it is more likely that the next word is “mat”.
However, that does not mean the longer sequence we consider, the better the prediction. For 6-gram, for instance, it might not be very common to see the exact “the cat sat on the”. Thus, it is not very efficient.
Most of the time, bigram and trigram are the common choices.
There are also times where your dataset is small and 3-gram fails to recognize some exact words; thus, resulting in the probability of 0. That would make your entire 3-gram implementation not that useful anymore.
Easily you can do a simple if-else, and use 2-gram in case 3-gram fails. Or use 1-gram in the case of 2-gram fails. There is called, back-off.
Another technique, interpolation, is considering different ngrams at the same time. The idea is to give weight (λ) to your n-gram so your 3-gram model can also look at the value of 2-gram and 1-gram. i.e.
60% on 3-gram
35% on 2-gram
05% on 1-gram
Config the weight manually until you feel it’s right. You can also use some automatic method, but we won’t go over that in this post. You can refer to this for further info on how to properly set these weights.
Now let’s try to use this LM to generate a sentence based on random initial word. For this task, we will use Harry Potter and the Sorcerer’s Stone Movie Script to train our ngram. For the code below, you can find on this kernel as well.
Then, we can count the frequency of unigram, bigram and trigram. This will be used later to compute probability.
Now that we can the count of the n-gram, we can compute its probability. We will use linear interpolation of 0.01, 0.4, and 0.5 respectively for 1-gram, 2-gram and 3-gram.
Let’s attempt to generate 20 words. The initial will be randomized, and choose the best next word.
So, how will this perform? Below are the results of each generation.
Generation #1:protecting the stone , but i think i 'll be a wizard . . . . . . . .Generation #2:platform 9 3/4 ? think you 're going to be the same . . . . . . . Generation #3:sacrifice himself , he 's got a little bit . . . . . . . . . . .
N-gram is a simple yet powerful technique to do language modeling by just looking at 2 to 3 words. It is quite practical in many applications, for instance, word suggestion on mobile device, thanks to its easy and fast in computation. N-gram, however, is not robust when it comes to generating long sentence where the current word depends on the very first word in the sentence. This long term dependency is more suitable for the model like neural network, but that is a topic for later post.