Time Series Predictions with LSTM
Tags: Time Series Forecasting, Long Short Term Memory, TensorFlow, Deep Learning, Machine Learning

Time-series predictions play a major role in machine learning which is often neglected. Nonetheless, there are lots of machine learning algorithms we could use for these problems. The major machine learning algorithms involving Statsmodels and Econometric models etc. Today we will take a look at how to use and apply Deep learning algorithms to predict the time series Data.

Why use a Deep Learning Algorithm?

With the data volume growing enormous day by day we shouldn’t confine ourselves to only the standard ML algorithms. Deep learning algorithms help us to handle large volumes of data and without leaving the key insights and by tuning the model within the right way gives us the maximum yield i.e., in our cause maximum accuracy 😊 . The model also determines if our prediction is better or worse from its own neural network architecture.

For this Time series forecasting we will use Long- Short Term Memory unit (LSTM).

What is LSTM ?

Long Short Term Memory unit(LSTM) was typically created to overcome the limitations of a Recurrent neural network (RNN). The Typical long data sets of Time series can actually be a time-consuming process which could typically slow down the training time of RNN architecture. We could restrict the data volume but this a loss of information. And in any time-series data sets, there is a need to know the previous trends and the seasonality of data of the overall data set to make the right predictions.

Before going into the brief explanation of LSTM cell, Let us see how the LSTM cell looks like :

The Architecture may look little complicated on the first glance, but it is pretty neat and clear and easily understandable if we break it into parts.

Lets first start understanding what are our inputs and outputs. The typical input if you see on the left-hand side of the diagram Ct-1 which is the previous cell state and ht-1 which is the output from the previous cell and Xt which is the input of the present cell.

The output of the cell is Ct and ht which are the corresponding cell state and output of the present cell.

The first step of an LSTM is the forget gate layer (f) where we determine what are we going to forget from the previous cell state. This typically takes the input ht-1 and Xt and make a linear transformation with some weights and bias terms and pass into the sigmoid function. As we are aware the output of a sigmoid function is always between 0 and 1. Here 0 will be considered as to forget it and 1 will represent to keep it

Forget gate later=> f = Sigmoid ( Weights (ht-1,Xt) + bias)

The second step is a two-part process and this is the step which tells us actually processing within this layer. Here in the first part we take the same inputs as before the ht-1 and Xt and make a linear transformation with some weights and biases and pass on to a sigmoid function. And the second part we will make a linear transformation again between ht-1 and Xt with some weights and biases but this time its going to be a hyperbolic tangent function (tanh). At the end of this step, we will get vectors of values which can be new candidate values for this present cell.

First part => I = sigmoid( Weights (ht-1,Xt) + bias)

Second part => II = tanh( Weights (ht-1,Xt) + bias)

The third step is the update step which helps us in deriving the new cell state Ct using our previous steps. First, we will multiply the previous cell state with the forget gate layer and add the vectors we got from the second step which forms the new cell state Ct of the present cell at t.

Update layer => Ct = Ct-1* f + I* II

The final step is another main output of the cell, for this, we will directly form a linear transformation with the previous output ht-1 and input of the present cell Xt with some bias and weight terms and pass on to a sigmoid layer. Finally, now we will multiply this output to the new cell state Ct which is passed on to a hyperbolic tangent function. This gives us the present output ht.

Final layer =>

i = sigmoid ( Weights (ht-1,xt) + bias)

final ht = i * tanh(Ct)

So, i hope we have a clear understanding of the step by step dissection of the LSTM layer. Let's see how we apply our LSTM cell into a time series data.

How to ?

Let's consider a basic data set for time series such as a sine series. For that let's use numpy to create this data set :

And when we plot the input it looks like this :

lets start initializing our placeholder variables and our LSTMcell can be invoked from tf.contrib.rnn.LSTMCell ( maybe in future it will be under tf.compat.v1.keras.layers.LSTMCell). And let’s encompass that under an output projection wrapper for our output size.

Let's invoke our session for 2000 epochs/iterations to get a better result.

we will predict some points within our training range so see how our model is predicting with the actual.

As we now have our predictions, let's visualize the output.

Well done!!! we can see our actual and predicted are very close with a very basic model for 2000 epochs. In the overall plot, we can see the predictions of how it goes with the seasonality. Try taking a different input set and predict the same. We could see the predictions holds good for all of those.

Hurray !! we now have a good idea of LSTM cell and how to implement them to get a time-series predictions. Give a pat on your back as a token of appreciation :) :)

Please find the full code for the same below :