Simple Text Generation. How to teach a machine to “speak” | by Jeremy Chow | Towards Data Science

If you’d like to see the entire code, here is the Github link.
In this article, I’ll briefly go over a simple way to code and train a text generation model in Python using Keras and Tensorflow. Our goal is to train a model to emulate the speaking style of the text it is trained on. In this case, our data set is 7009 sentences from Edgar Allen Poe horror stories.
The first step to training any NLP model is the tokenization of words. Here we use the Keras Tokenizer, which does the following:
Removes punctuationSets all text to lower caseSplits the words up into individual elements in a list, then assigns a unique integer to each wordReplaces all instances of that word with the integer.
Removes punctuation
Sets all text to lower case
Splits the words up into individual elements in a list, then assigns a unique integer to each word
Replaces all instances of that word with the integer.
Tokenization is necessary for preparing data for embedding layer (see model architecture section below)
Setting up training data using sliding windows. We are using a fairly rudimentary strategy of using the previous 19 words to predict the 20th word. These numbers were not selected with any concrete reasoning in mind and can definitely be tinkered with to improve.
1. Embedding layer
An embedding layer is a key layer to any sort of deep learning model that seeks to understand words. What an embedding layer does from a mathematical standpoint is take a vector from a higher dimensional space (tens of thousands or more, the original size of our vocab) to a lower dimensional space (the amount of vectors we want to represent our data in, typically 100–300 in models like Word2Vec, Fasttext).
However, it does this in such a way that words with similar meanings have similar mathematical values and exist in spaces that correspond to their meaning. Mathematical operations can be performed on these vectors, for example, ‘king’ minus ‘man’ may equal ‘royalty’.
2. Two Stacked LSTM layers
I’m not going to go into the specifics of LSTM’s here, but there are plenty of good articles to check out how they work. Some things to note:
Stacked LSTMs may add more depth than additional cells in a single LSTM layer according to these folks at Cornell when applied to speech recognition. While our application is not identical, it is similar in its use of LSTM’s to try to identify language patterns, so we will try this architecture.
The first LSTM layer must have return sequences flag set to True in order to pass sequence information to the second LSTM layer instead of just its end states
3. Dense (regression) layer with ReLU activation
The output of an LSTM is the “hidden layer”. It is common to apply a dense layer following an LSTM to further capture linear relationships, but it is not explicitly necessary.
4. Dense layer with Softmax activation
This layer is necessary to convert the output of the above layers into the actual word probabilities across our entire vocabulary. It uses the softmax activation function, which is a function that converts all the input word probabilities from (-∞,∞ ) to (0,1). This allows us to select or generate the most probable words.
Once we have set up our model architecture, we can train it!
(Note that we can use checkpoints to maintain our progress in case our training gets interrupted.)
With training complete, we now have a model that can generate text. However, we need to give it a starting point. To do this, we write a function that takes a string input, tokenizes it, then pads it with zeroes so it fits into our 19 long prediction window.
The following is the result given the input string listed. Note that we didn’t train our model with any punctuation, so sentence divisions are left up for interpretation by the reader.
Input String:  First of all I dismembered the corpse.Model  3 :first of all i dismembered the corpse which is profound since in the first place he appeared at first so suddenly as any matter no answer was impossible to find my sake he now happened to be sure it was he suspected or caution or gods and some voice held forth upon view the conditions
While it doesn’t make the most sense, it actually seems readable and has the correct word types in the right places. Given a larger set of training data, this approach and model can be used to generate much more understandable text (most likely with some parameter tweaking, of course).
To quantify performance, you can implement a typical train test split with the original data and rate predictions on the cosine similarity between predictions and target.
That’s all for today, thanks for reading. If you want to see more, here are some other machine learning projects I’ve done that you can read about:
Quantifying Chatroom Toxicity
Growth Oriented Recommender System For Streamers