Recurrent Neural Networks
By: Ismail Mebsout
Tags: Rnn, Time Series Forecasting, Attention Mechanism, Data Science, Lstm

2. RNN model

RNNs represent a special case of neural networks where the parameters of the model, as well as the operations performed, are the same throughout the architecture. The network performs the same task for each element in a sequence whose output depends on the input and the previous state of the memory .

The graph below shows a neural network of neurons having a single layer of hidden memory:

Equations

The variables in the architecture are:

where:

h(−1)​ is randomly initialized, ϕ and ψ are non-linear functions, U, V, and W are parameters of the various linear regressions, preceding the nonlinear activations.

It is important to note that they are the same throughout the architecture.

Applications

Recurrent neural networks have significantly improved sequential models, in particular :

NLP Tasks, Modeling and Text Generation

Translation machine

Voice recognition

We summarize the above applications in the following table:

Learning algorithm

As in classical neural networks, learning in the case of recurrent networks is done by optimizing a cost function with respect to U, V and W. In other words, we aim to find the best parameters that give the best prediction y^i, starting from the input xi​, of the real value yi​.

For this, we define an objective function called the loss function and denoted J which quantifies the distance between the real and the predicted values on the overall training set.

We minimize J following two major steps:

Forward Propagation : we propagate the data through the network either in entirely or in batches, and we calculate the loss function on this batch which is nothing but the sum of the errors committed at the predicted output for the different rows.

: we propagate the data through the network either in entirely or in batches, and we calculate the loss function on this batch which is nothing but the sum of the errors committed at the predicted output for the different rows. Backward Propagation Through Time : consists of calculating the gradients of the cost function with respect to the different parameters, then apply a descent algorithm to update them. It is called BPTT, since the gradients at each output depend both on the elements of the same instant and the state of the memory at the previous instant.

We iter the same process a number of times called epoch number . After defining the architecture, the learning algorithm is written as follows:

(∗) The cost function L evaluates the distances between the real and predicted value on a single point.

Forward propagation

Let us consider the prediction of the output of a single sequence through the neural network.

At each moment t, we compute:

Until reaching the end of the sequence.

Again, the parameters U, W and V remain the same all along the neural network.

When dealing with an m-row data set, repeating these operations separately for each line is very costly. Hence, we truncate the dataset in order to have sequences described in the same timeline, i.e:

We can use linear algebra to parallelize it as follows:

Backpropagation Through Time

The backpropagation is the second step of the learning, which consists of injecting the error committed in the prediction (forward) phase into the network and update its parameters to perform better on the next iteration . Hence, the optimization of the function J, usually through a descent method.

We can now apply a descend method as detailed in my previous article.

Memory problem

There are several fields where we are interested in predicting the evolution of a time serie based on its history: music, finance, emotions…etc.

The intrinsic recurrent networks described above, called ‘Vanilla’, suffer from a weak memory unable to take into account several elements of the past in the prediction of the future.

With this in mind, various extensions of the RNNs have been designed to trim the internal memory: bi-directional neural networks, LSTM cells, attention mechanisms…etc. Memory enlargement can be crucial in certain fields such as finance where one seeks to memorize as much history as possible in order to predict a financial series.