Build your LSTM language model with Tensorflow
A language model is a machine learning model that we can use to estimate how grammatically accurate some pieces of words are. This kind of…
Category: Machine Learning
Author: MilkKnight

Build your LSTM language model with Tensorflow
A language model is a machine learning model that we can use to estimate how grammatically accurate some pieces of words are. This kind of model is pretty useful when we are dealing with Natural Language Processing(NLP) problems. And in speech recognition tasks, the model is essential to be here to give us prior knowledge about the language your recognition model is based on.
As you may have known already, for most of the traditional statistical language models, they are enlightened by Markov property. For example, this is the way a bigram language model works:
The memory length of a traditional language model is not very long .You can see that in a bigram model, the current word only depends on one previous word. And in a trigram model, the current word depends on two preceding words. From my experience, the trigram model is the most popular choice, some big companies whose corpus data is quite abundant would use a 5-gram model. Nevertheless, you can see that even the memory of a 5-gram model is not that long.
Except for the short-term memory of statistical language models, another defect of traditional statistical language models is that they hardly decern similarities and differences among words. For instance, P(dog, eats, veg) might be very low if this phrase does not occur in our training corpus, even when our model has seen lots of other sentences contain “dog”. The model just can’t understand words.
So, this is when our LSTM language model begin to help us. Firstly, it can definitely memorize a long-term memory. More important, it can seize features of words, this is a valuable advantage we can get from an LSTM model.
In this tutorial, we will build an LSTM language model with Tensorflow together. At the end of this tutorial, we’ll test a 5-gram language model and an LSTM model on some gap filling exercise to see which one is better.

Prepare Data
I’m going to use PTB corpus for our model training; you can get more details on this page. PTB is good enough for our experiment, but if you want your model to perform better, you can feed it with more data.
The preprocessing of your raw corpus is quite necessary. Typically, every first step of an NLP problem is preprocessing your raw corpus. This step sometimes includes word tokenization, stemming and lemmatization. But, in here, we just simply split sentences since the PTB data has been already processed.
Here is the code.

First, we generate our basic vocabulary records. One important thing is that you need to add identifiers of the begin and the end of every sentence, and the padding identifier can make LSTM skip some input data to save time, you can see more details in the latter part.
Then, we turn our word sequences into index sequences. This processing is very similar to how we generate vocabularies. We set the OOV (out of vocabulary) words to _UNK_ to deal with certain vocabularies that we have never seen in the training process.
Build the LSTM Model
You can see the code on github. Here, I am going to just show some snippets.
The first step is to feed our model inputs and outputs. One thing important is that you need to tell the begin and the end of a sentence to utilize the information of every word in one sentence entirely. As you can see in Fig.1, for sequence “1 2605 5976 5976 3548 2000 3596 802 344 6068 2” (one number is one word), the input sequence is “1 2605 5976 5976 3548 2000 3596 802 344 6068,” and the output sequence is “2605 5976 5976 3548 2000 3596 802 344 6068 2”. “1” indicates the beginning and “2” indicates the end if you remember the way we symbolize our raw sentence.
Fig.1
You may have noticed the dots in fig.1, they mean that we are processing sequences with different lengths. There are many ways to deal with this situation. An intuitive solution is zero-padding, which is to append zeros to some sequences to get a bunch of sequences with the same lengths (We sometimes call it “max_time”). You can see it in Fig.2. However, we need to be careful to avoid padding every sequence in your data set. For example, if you have a very very long sequence with length like 1000, and the lengths of all you other sequences are just about 10, if you did zero-padding on this whole data set, every sequence length would be 1000, and apparently, you would waste your space and computation time. So, doing zero-padding for just a batch of data is more appropriate. This process sounds laborious, luckily, Tensorflow offers us great functions to manipulate our data.
Fig.2
We are going to use tf.data to read data from files directly and also feed zero-padded data to LSTM model (more convenient and concise than FIFOQueue I think).
In the code above, we use placeholders to indicate the training file, the validation file, and the test file. And then, we can do batch zero-padding by merely using padded_batch and Iterator.
But, we still have a problem. It is weird to put lonely word indices to our model directly, isn’t it? Yes! You may have seen a terminology like “embedding” in certain places. This is what we’ll talk about in our next step.
The reason we do embedding is to create a feature for every word. One advantage of embedding is that more affluent information can be here to represent a word, for example, the features of the word “dog” and the word “cat” will be similar after embedding, which is beneficial for our language model.
Fig.3
Embedding itself is quite simple, as you can see in Fig.3, it is just mapping our input word indices to dense feature vectors. The decision of dimension of feature vectors is up to you. For example, we have a 10*100 embedding feature matrix given 10 vocabularies and 100 feature dimension. Then, we get a sequence “1, 9, 4, 2”, all we have to do is just replace “1” with the 1st row of the feature matrix (don’t forget that the 0th row is prepared for “_PAD_”), then, turn “9” to the 9th row of the matrix, “4” to the 4th, “2” to the second, just like the way when you are looking up a word in the dictionary. In Tensorflow, we can do embedding with function tf.nn.embedding_lookup.
Then, we start to build our model, below is how we construct our cell in LSTM, it also consists of dropout. We can use that cell to build a model with multiple LSTM layers.
Next step, we build our LSTM model. As usual, Tensorflow gives us a potent and simple function to do this.
The dynamic_rnn can unfold nodes automatically according to the length of the input and be able to skip zero-padded nodes; these properties are valuable for us to cope with variable-length sequences. All it needs is just the lengths of your sequences. Here I write a function to get lengths of a batch of sequences.
The form of outputs from dynamic_rnn is [batch_size, max_time_nodes, output_vector_size] (default setting), just what we need! At this step, feature vectors corresponding to words have gone through a model and become new vectors that eventually contain information about words, context, etc. However, Since we have converted input word indices to dense vectors, we have to map vectors back to word indices after we get them through our model. A nonlinear transformation is enough to do this thing. First, we define our output embedding matrix (we call it embedding just for symmetry, cause it is not the same processing as the input embedding).
Fig.4
OK, we’ve got our embedded outputs from LSTM. What’next? Of course, we are gonna to calculate the popular cross-entropy losses. But before we move on, don’t forget that we are processing variable-length sequences, so, we need to dispense with the losses which are calculated from zero-padding inputs, as you can see in Fig.4. On the other hand, keep in mind that we have to care about every output derived from every input (except zero-padding input), this is not a sequence classification problem.
In the code above, we first calculate logits with tf.map_fn, this function can allow us to multiply each LSTM output by the output embedding matrix, and add the bias obviously.
Then, we reshape the logit matrix (3d, batch_num * sequence_length * vocabulary_num) to a 2d matrix. This reshaping is just to calculate cross-entropy loss easily.
The last thing we have missed is doing backpropagation. As always, Tensorflow is at your service.
Evaluate our model

In fact, when we want to evaluate a language model, the perplexity is more popular than cross entropy, why? You can see a good answer in this link. Here, I am gonna just quote:
Remember that, while entropy can be seen as information quantity, perplexity can be seen as the “number of choices” the random variable has.
So how to get perplexity? It is quite simple and straight; perplexity is equal to e^(cross-entropy). One more thing, you may have noticed that in some other places, they said that perplexity is equal to 2^(cross-entropy), this is also right because we just use different bases. In Tensorflow, we use natural logarithm when we calculate cross entropy whose base is e. So, if you calculate cross entropy function with base 2, the perplexity is equal to 2^(cross-entropy). In addation, I prove this equation if you have interest to look into.

Test our modelvim
Now, let’s test how good our model can be.
First, we compare our model with a 5-gram statistical model. Here, I chose to use SRILM, which is quite popular when we are dealing with speech recognition and NLP problems. Two commands have been executed to calculate the perplexity:
As you can see, we get the ppl and ppl1. According to SRILM documents, the ppl is normalized by the number of words and sentences while the ppl1 is just normalized by the number of words. Thus, the ppl1 is the score that we want to compare with the ppl comes from our RMMLM model.
Yes! Our model gets a much better score!
However, just one ppl score is not very fun, isn’t it? So, I’m going to use our model to do gap filling exercise for us!

You can find the questions in this link. The way we choose our answer is to pick the one with the lowest ppl score.
First, we utilize the 5-gram model to find answers.
We can add “-debug 1” to show the ppl of every sequence.
The answers of 5-gram model are:
1. everything that he said was wrong (T)
2. what she said made me angry (T)
3. everybody is arrived (F)
4. if you should happen to finish early give me a ring (T)
5. if she should be late we would have to leave without her (F)
6. the thing that happened next horrified me (T)
7. my jeans is too tight for me (F)
8. a lot of social problems is caused by poverty (F)
9. a lot of time is required to learn a language (T)
10. we have got only two liters of milk left that is not enough (T)
11. you are too little to be a soldier (F)
12. it was very hot that we stopped playing (F)
The accuracy rate is 50%. Remember, we have removed any punctuation and converted all uppercase words into lowercase.
The answers of rnnlm are:
1. everything that he said was wrong (T)
2. what she said made me angry (T)
3. everybody has arrived (T)
4. if you would happen to finish early give me a ring (F)
5. if she should be late we would have to leave without her (F)
6. the thing that happened next horrified me (T)
7. my jeans is too tight for me (F)
8. a lot of social problems are caused by poverty (T)
9. a lot of time is required to learn a language (T)
10. we have got only two liters of milk left that is not enough (T)
11. you are too small to be a soldier (T)
12. it was too hot that we stopped playing (F)
The accuracy rate is 66.7%
Our model gets a better score, obviously. :)
Conclusion
In this tutorial, we build an LSTM language model, which has a better performance than a traditional 5-gram model. The model in this tutorial is not very complicated; If you have more data, you can make your model deeper and larger.