RNNs from theory to PyTorch
By: Tiago Miguel
Tags: Rnn, Recurrent Neural Network, Theory, Pytorch

RNNs from theory to PyTorch

Let me show you what RNNs are, where they are used, how they forward and backward propagate and how to use them in PyTorch. Tiago Miguel Dec 19, 2020·13 min read

Most types of neural networks are built to make predictions on samples whose targets they have been trained on. A prime example is the MNIST dataset. A regular neural network like the MLP knows there are 10 digits and it is only based on them that it makes predictions even if the images are very different from those the network has been trained on.

Now, imagine we could leverage such a network with a sequential analysis by providing a sequence of 9 ordered digits and let the network guess the 10th. The network would not only know how to discriminate the 10 digits but would also know that, in a sequence of 0 to 8, the next digit is most likely a 9.

When analyzing sequenced data, we understand that most often than not, the elements within that sequence are somehow related, meaning they depend on one another. Because of this, we need to take into account each element to understand the sequence’s idea.

The Cambridge University Press defines a sequence as either “the order in which things or events follow each other” or, and most importantly, “a series of related things or events”. To adjust this definition into the scope of Deep Learning, sequence is a set of data that contains trainable context, and removing some elements may render it useless.

But what does a sequence contain? What pieces of grouped data could have context? And how can we extract context to leverage the power of neural networks? Before getting into the neural network itself, let me show you two types of problems frequently solved with Recurrent Neural Networks (RNNs).

Timeseries Forecasting

This first example is a Timeseries Forecasting problem, where we train a neural net with a sequence of existing numerical values (blue) in order to predict future time steps (red).

If we make a sequence out of the monthly energetic expenditure of a household throughout the years, we can see that there is a rising sinusoidal trend with a sudden fall.

The sinusoidal part’s context may be the different energetic requirements throughout the year from summer to winter and back to summer again. The rising energetic expenditure could come from using more electrical appliances and devices or switching to more powerful ones that may require more energy. The sudden fall’s context could mean one person grew old enough to leave home and the energy required by that person is no longer there.

The better you understand the context, the more information you can feed the network with to help it understand the data, generally through concatenation of the input vector. In this case, for every month, we could concatenate three more values to the energy that would be the number of electric appliances and devices, their energetic efficiency and the number of people the household contains.

Natural Language Processing

Mary rides the bycicle, the bycicle is ____.

A second example is a Natural Language Processing problem. This is also a good example because the neural net must take into consideration the context provided by the existing sentence to complete it.

Let’s say our network is trained to complete sentences with possessive pronouns. A well trained network would understand the sentence is built in the third person singular and that Mary is most likely a feminine name. So, the predicted pronoun should be “hers” instead of the masculine “his” or the plural “theirs”.

Now that we have seen two examples of sequenced data, let’s explore the network going about its forward and backward propagation.

RNN configurations

As we have seen, RNNs extract information out of sequences to improve their predictive power.

Simple recurrent network diagram. Figure by author.

A simple RNN diagram is represented above. The green node lets in some input x^t and outputs some value h^t that is also fed to the node again containing information gathered from the input. Whatever pattern there is in what is being fed to the node, it learns it and keeps that information for the next input. The superscript t stands for a time step.

Recurrent network configurations. Figure by author.

There are some variations to the neural network’s configuration based on the shape of the input or output, we’ll understand what happens inside the nodes later.

Many-to-one configuration is when we feed several inputs in different time steps to get one output, which could be the sentiment analysis captured in various frames of a movie scene.

One-to-many uses one input to obtain several outputs. For instance, we could encode a poem conveying a certain emotion using a many-to-one configuration and use the one-to-many configuration to create new lines of the poem with that same emotion.

Many-to-many uses several inputs to obtain several outputs like using a sequence of values like in the energetic usage and forecast twelve months in the future instead of only one.

The stacked configuration is just a network with more than one hidden layer of nodes.

RNN forward pass

To understand what’s happening inside the nodes of the neural network, we’ll use a simple dataset as a Timeseries Forecasting example. Bellow is the full sequence of values and its restructuring as a training and a testing dataset.

I took this example from this website which is a great resource for Deep Learning in general. Now, let’s separate the datasets into batches.

I don’t show it here, but don’t forget the dataset should be normalized. It is important since neural networks are sensitive to the magnitude of the dataset values.

The idea is to predict one value into the future. So, say we pick the first row of the batch: [10 20 30], after training our network we should get the value 40. To test the neural net, one could feed the vector [70 80 90] and expect to obtain a value close 100 if the network is well trained.

We’ll use the many-to-one configuration feeding the three time steps of each sequence separately. When using recurrent networks, the input is not the only value that goes into the network, there is also a hidden array that is the structure that will transport the context of the sequence from node to node. We initialise it as an array of zeros and it is concatenated to the input. Its dimensionality (1 x 2) is a personal choice, just to use a different dimension from the step input of 1 x 1.

Recurrent forward pass. Figure by author.

Taking a closer look, we can see that the weight matrix is separated into two parts. The first deals with the input creating two outputs and the second deals with the hidden array creating also two outputs. These two sets of outputs are then added together and a new hidden array is obtained, containing information from the first input (10), and is fed to the next time step input (20). It should be noted that the weight and bias matrices are the same from time step to time step.

The global input vector X^t, the weight matrix W and the bias matrix B and the calculation of the hidden arrays are represented above. Only one step missing to finish the forward pass and that is using the last hidden array to predict the next time step into the future with a linear layer to calculate the final result. The full network has the following form:

Full recurrent forward pass. Figure by author.

Can you see the many-to-one configuration? We feed three inputs from a sequence, their context is captured by the weight and bias matrices and stored in a hidden array that is updated with new information at every time step. Finally, that context stored in the final hidden array goes through another set of weights and biases and one value is outputed after all time steps of the sequence were inputed into the network.

We can see the linear form of the final hidden state and the weight and bias matrices of the linear layer as well as the prediction value calculation (y_hat).

Now, this was the forward propagation of the RNN but we still haven’t seen the backward pass.

RNN backward pass

The backward propagation is a very important step of the training of every neural network. Here, the error between the predicted output and the real value is propagated towards the neural net with the goal of improving the weights and biases in order to obtain better predictions with every iteration.

Most times, this step gets overlooked because of its complexity. I’ll present you an explanation as simple as I can make it while mentioning the important bits.

The backward pass is a series of derivations using the chain rule of calculus from the loss until all the weight and bias parameters. This means that we ultimately want the following values (or arrays if they are multi dimensional as is the case):

I recommend reading about Gradient Descent if you are not familiar with the mathematical meaning of first derivatives but in essence, when the first derivative is a zero, generally it means we found a minimum value in our system and ideally we wouldn’t be able to improve it any further.

A little side note here: the zero could also be maximum values, which are unstable and an optimization shouldn’t go there, or a saddle point which in itself isn’t very stable position as well. And the minimum may be global (lowest minimum of the function) or local. This doesn’t really matter to my explanation, but if you want to know more about it you can look it up!

Gradient Descent example. Two balls rolling down the hill. Figure by author.

What you see in the picture are two balls rolling down the valley. Visually, the first derivative gives us a magnitude of the inclination of the hills. If we travel the direction in which the W axis increases (left to right), the inclination will be negative (going down) for the green ball and positive (going up) for the red ball.

Read the next paragraph carefully and go back to the figure as needed.

If we want the loss to be minimal, we want the balls to go to the lowest point in the valley. W represents the values of the weights and biases so if we are in the position of the green ball we will subtract some portion of the negative derivative (making it positive) to the W position of the green ball moving it right and subtract some portion of the positive derivative (making it negative) to the W position of the red ball moving it left in order to approach both balls to the minimum.

Mathematically what we have is the following:

The η adjusts the proportion of the derivative we use to update the weights and biases.

Now, to get on with the backward pass issue. I’ll present the chained derivatives from the loss to all the parameters and we’ll see what each derivative represents. It is important to have the equations from the layers presented above along with their parameter’s matrices in mind.

One thing to bear in mind is that the shapes of the four first derivative arrays we are looking for need to be the same as the parameters we will update. For example, the shape of the array dL/dW_h must be the same as the weight array W_h. The superscripted T means the matrix is transposed.

We have found the derivatives of the final linear layer’s parameters. Knowing that dL/dy_hat has shape 1 x 1 and h³ 2 x 1 and that for matrix multiplication to happen the inner dimensions must be equal, h³ needs to be transposed. We obtain a matrix that is 1 x 2 and since W_h is 2 x 1, we need a second transposition.

Now, the recurrent backward pass is slightly different. Despite the parameters are the same in all the three time steps, we still back propagate through all of them and for each hidden node, we calculate a new dL/dW_x and dL/dB_x and add to them together.

The numbered superscript in the derivatives’ denominator relates to the time step. We calculated the parameters for the third time step and now we need to go back to the second time step.

We have gone back to dL/dh², this means we back propagated an entire recurrent hidden node and for that we needed to deconcatenate the input and the hidden state from the global input derivative. The next two hidden states’ calculations are the same so I won’t go over them.

Now we add the derivatives together and apply the Gradient Descent equation we saw earlier to update the parameters and the model is ready for another iteration. Let’s see how to build a simple RNN with PyTorch.

RNN with PyTorch

Using PyTorch makes it very simple since we don’t really need to worry about the backward pass. However, I still believe it is important to know how it works even if we don’t use it directly.

Moving on, if we refer to PyTorch’s documentation, we can see they already have an RNN object ready to be used. When defining it, there are two essential parameters:

input_size — The number of expected features in the input x hidden_size — The number of features in the hidden state h

The input_size is 1 since we are using one time step of each sequence (eg. 10 from the sequence 10, 20, 30) at a time and the hidden_size is 2 since we obtain a hidden state containing two values.

Defining the n_layers parameter to 2 would mean we have a stacked RNN with two hidden layers.

Also we are going to define the parameter batch_first to True . This means the batch dimension in input and output comes first (don’t mistake with Input and Output )

Inputs: input, h_0 input of shape (seq_len, batch, input_size): tensor containing the features of the input sequence. h_0 of shape (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for each element in the batch.

The inputs of the RNN should be an input array of shape 1 x 3 x 1. The sequence contains three time steps that are 10, 20 and 30 for the first batch of the dataset. From each batch, an input of size one will be fed three times to the network being the three time steps of the sequence.

The hidden state h_0 is our first hidden array that we feed the network along with the first time step input of shape 1 x 1 x 2.

Outputs: output, h_n output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the RNN, for each t. h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len.

The output contains all the hidden states calculated by the neural network at each time step of shape 1 x 3 x 2 and h_n is the hidden state from the last time step. This is useful to keep because, if we choose to use a stacked recurrent network, this will be the hidden state that will be fed at the first time step and has shape 1 x 1 x 2.

All these arrays are represented in the example above and can be seen in the RNN diagram. One more thing to note is, using recurrent networks and the specific example of Timeseries Forecasting, setting the num_directions to 2 would mean predicting into both the future and the past. That type of configuration will not be considered here.

I’ll leave a piece of code where I implement the RNN and how to train it. I’ll also leave it to you to use it as you need with the dataset you want. Don’t forget to normalise your data and to create a dataset and a dataloader before using the network.

Closing thoughts

To end this story with a brief summary of what was discussed here, we started by seeing two types of problems commonly solved with recurrent networks being Timeseries Forecasting and Natural Language Processing.

Later we saw examples of some typical configurations and an actual example whose objective was to predict one time step into the future using the many-to-one configuration.

In the forward pass we understood how the inputs and the hidden states interact with the weights and biases of the recurrent layers and how to use the information contained in the last hidden state to predict the next time step value.

The backward pass is just the application of the chain rule from the loss gradient with respect to the predictions until it becomes with respect to the parameters we want to optimize.

Finally, we went through part of the documentaion of PyTorch on RNNs and discussed the most important bits in order to build a basic recurrent network.

Thank you for reading! Maybe you got something out of this lengthy story. I write them to help me understand new concepts and hopefully help others too.