Text Generation Using Recurrent Neural Networks | by Donald Dong | Towards Data Science

Text generation is a popular problem in Data Science and Machine Learning, and it is a suitable task for Recurrent Neural Nets. This report uses TensorFlow to build an RNN text generator and builds a high-level API in Python3. The report is inspired by @karpathy ( min-char-rnn) and Aurélien Géron ( Hands-On Machine Learning with Scikit-Learn and TensorFlow ). This is a class project in CST463 — Advanced Machine Learning at Cal State Monterey Bay, instructed by Dr. Glenn Bruns.
Dataset, RNNTextGenerator, and ModelSelector are the three main modules. For the documentation please check out this project on Github.
Defined in src/dataset.py
Creates a text dataset contains the one-hot encoded text data. It produces batches of sequences of encoded labels. We split the text data into batches are used to train the RNN, and we sample a random chuck of the text (with given length) to evaluate the performance of our model.
Defined in src/text_generator.py
Creates a recurrent neural network with a TensorFlow RNN cell (which performs dynamic unrolling of the inputs). It has an output projection layer which produces the final probability for each character class. It generates the text by sampling the next character based on the probability distribution of the last character of the current sequence.
We use Alice in Wonderland as the text.
We can see the accuracy and loss have not converged. Each fit call will run one epoch (we passed epoch=1 to the constructor).
We want to continue fitting the model with more epochs for 30 more seconds, and we’re interested in how the model improves over time.
test acc: 0.535040020942688, test loss: 1.592130422592163Yes, but much towe this NOT, aft open') said 'Letger comimpie hone,'U VERY up in aborious, it with the adge,-----------------------test acc: 0.5360000133514404, test loss: 1.5756181478500366Yes, but the Habembood cacs must, was the lang donant as perpen my his huril about harriered me shreep), whre-----------------------test acc: 0.5430399775505066, test loss: 1.5500394105911255Yes, but to Alice.So thenhep datee went orling,' said Alice,' said had geined,' she you not!' said thenil-----------------------test acc: 0.5302400588989258, test loss: 1.6037070751190186Yes, but the could mould cobl over howner? Now that oplo-thing to her orverewn.''Why, toherself!The Qu-e-----------------------test acc: 0.539199948310852, test loss: 1.6024020910263062Yes, but the Queen home way the at the placsarnd of the sat a it,' the QUED INK Affece herge how eacushing ov-----------------------test acc: 0.5388799905776978, test loss: 1.538255214691162Yes, but did they rossan that may the sing,' sayeing, round queer thatled thing's she went on, I'm she can wa-----------------------test acc: 0.5424000024795532, test loss: 1.5805484056472778Yes, but first's word.' She was byself, if our it was en!' seemes you seleasions of the doous rige out oldes-----------------------test acc: 0.5455999970436096, test loss: 1.5576822757720947Yes, but the other-- Whereet heam who her notone and it didn't knowling e s-----------------------
We can see the generated text does become better over time. There are more actual words in the sentence. The test accuracy and loss are still improving.
Now, we will try different hyperparameters.
Defined in src/model_selector.py
Performs randomized search and rank the models by accuracy. It selects the best ranking models and allows lengthy searching (for hours/days).
We define a large search space for the model selector and give it 24 hours to search. We hope it would discover an excellent model for the Alice in Wonderland text.
The best model the selector has seen uses a GRU Cell with 320 neurons and leaky_relu as the activation function. It uses an AdamOptimizer with a learning rate of 0.006. It's trained in 45 epochs, with 91 sequences per batch (batch_size). It has an average accuracy of 0.6245 and loss of 1.25 over 5 randomly sampled test sequences.
Yes, but to talk about this of ketched him in the puppy juging said to the tell messave off, things very unfusts, and that put on hesserriely I’ll make the pu
We consider epoch as a hyperparameter of the model since a big number of epochs can allow the neural net to overfit and perform poorly on the test sequences. Thus, the best model from the selector is the final model we will use.
The best text generator is deployed as a simple web application (https://www.ddong.me/alice-text-gen) for educational purposes.