Sequence Models and Literature. Tales of the Alhambra.
By: Faycal Zouine
Tags: NLP, Machine Learning, TensorFlow, Sequence Model, Data Science

A quick hands-on demonstration of sequences models using BRNN/LSTM on Tensorflow. I do not attempt to go too deep into the roots of the different algorithms used for this case but try to give an understanding of the overall process, where [1]data consist of sequences of (x, y) pairs with significant sequential correlation. Or, in other words, nearby x and y values are expected to be related to each other. This is designed for someone who has some coding experience.

Most of our daily lives are made up by sequential data and by “sequential data” I mean data where the order in which the datum appears matters. DNA sequencing is one of the most well-known examples. Human-computer interactions, where the order of keystrokes may elude to fraud is another; as is the field of meteorology, where sequential observations, over time, predict weather. Or, in our case, the very words that you are reading now, where your interpretation comes from the order of the words that I have written; and these words are written in a way that they are dependent on the preceding words.

The focus of this piece is how to predict sequential data in text. For this I have chose to use Washington Irvings’ Tales of the Alhambra, which I selected after having visited the Alhambra situated in Granada, Spain (a stunning place to visit).

The Tales of the Alhambra is a collection of essays and sketches, written in the 1800s, set to the backdrop of Granada (Andalusia, Spain). The data comes from The Project Gutenberg. You can find the original text here.

The Albaicín (Albayzín) in Grenada, looking back at the Alhambra.

The most effective model to predict sequential data is a deep neural network as it is highly efficient at capturing non-linearity. In this post we will be covering BRNN (Bidirectional Recurrent Neural Network, also sometimes referred to as Bidirectional Recursive Neural Network)) and LSTM (Long Term Short-Term Memory). This is as they have been proven to be very efficient neural network architectures[2].

An Overview of BRNN and LSTM

BRNN

A layered neural network, [3]BRNN has been around since 1997, and is a form of generative learning. An evolution of the regular recurrent neural network (RNN) cells states shift in a positive time direction, forwards (future) and backwards in a negative time direction (past), so that cells states from a later training set can impact the earlier. These are connected to the same output layer. The general structure of RNN and BRNN is depicted in the diagram below.

By using two different time directions, input information from the past and future of the current time frame can be used, unlike with the standard RNN which requires the delays for including future information.

LSTM

In the same year that BRNN came along, so did the gradient-based approach of [4]long short-term memory (LSTM). As we know, storing data over prolonged time intervals with recurrent backpropagation takes a long time. LSTM learns to bridge negligible time lags by introducing a memory unit, called the cell, into the network. This means that LSTM is able to remember a lot over prolonged periods of time.

Below is diagram of a LSTM building block. There is very nice blog here >> by Colah, who has an impressive with Google Brain and is now working on OpenAI. As well, I recommend the lessons of Andrew Ng (LSTM on Coursera) and Laurence Moroney’s Introduction to Tensorflow for AI, also on Coursera.

Getting Started

You can copy and paste the Python code below, directly into your Jupyter Notebook or Colab. Open and start playing!

import numpy as np

import tensorflow as tf

print(tf.__version__)

from tensorflow import keras

from keras.preprocessing.text import Tokenizer

from keras.preprocessing.sequence import pad_sequences

from keras.models import Sequential

from keras.layers import Dense, Dropout, Embedding, LSTM, Bidirectional,Activation

from keras.layers.recurrent import SimpleRNN

from keras.optimizers import Adam

2.3.0