LSTM to Predict Stock Prices — Time-Series Data | by Sarit Maitra | Towards Data Science

PREDICTION of financial series is always a complex task and has attracted major attention from academics, investment companies, banks etc. Price prediction are typically evaluated on the basis of statistical criteria, such as mean error, mean absolute error (MAE), or root mean squared error (RMSE). We have already seen the relationships among multiple time series (Gold prices, Silver prices, Crude Oil prices, Stock index , Interest Rate and USD rate) and how they influence each other.
Let’s use the same data set and try to solve our prediction problem with Recurrent Neural network (RNN) algorithm. Neural networks are well known for their efficiency with multiple input variables.
Let’s load the data which is already processed earlier; if there is any data error, hopefully our model will learn to ignore.
Let’s understand the mathematical intuition of feed-forward network.
where, x1, x2....xk are input series, ŷ is output series and in between a bunch of hidden layers. However, here the issues are-
information does not account for time ordering
inputs are processed independently
no mechanism to keep the past information
“Feed-Forward network architecture does not have a memory built-in. Moreover, it appears a situation where a deep multilayer feed-forward network is unable to propagate useful gradient information from the output end of the model back to the layers near the input end of the model”
To overcome these difficulties, RNN was introduced which comes with a network architecture that can-
retain the past information
track the state of the world
update the state of the world as the network moves faster
“RNN handles variable-length sequence by having a recurrent hidden state whose activation at each time is dependent on that of the previous time”
Below diagram and equations give a clear idea about how RNN works —
where, xt represents inputs, ht expresses the hidden layers from input to output, yt is output. On the top of that, it has a recurrent loop which loop back to the past to express that, the output is not only a function of new input but also a function of past hidden layer and that way the network keeps growing. However, it has a problem of exploding and vanishing gradient. Recurrent Neural Networks (LSTM) addresses this issue. They are networks with loops in them, allowing information to persist. The most important concept for LSTM is the state of the cell, and this information is transmitted to each other between cells with yt data embedded on the top of the cell.
In the above diagram, a chunk of neural network, A, looks at some input xt and outputs a value ht. A loop allows information to be passed from one step of the network to the next. These loops make recurrent neural networks kind of mysterious object. However, they aren’t all that different than a normal Neural Network. A Recurrent Neural Network is the multiple copies of the same network, each passing a message to a successor. The cell state works like a conveyor belt and is carried through the entire chain. Cells may make small changes to this information. There are gates inside the cells. These gates contain a sigmoid function and open and close the sigmoid function according to the output value. Then the gate output value and yt value are subjected to multiplication. The sigmoid function outputs can take values between 0 and 1, while a value of zero closes the information transition and moves a value across the entire information. If the loop is unrolled as shown in below diagram-
This chain-like nature reveals that recurrent neural networks are intimately related to sequences and lists. They’re the natural architecture of neural network to use for such data. Let’s consider the hidden state ht at time step t. The first thing the LSTM cell needs to decide is to report the cell status. This decision is made by the forget gate layer. The forget gate layer generates a value between 0 and 1 for each yt−1 by looking at ht−1 and xt. 1 means that data is stored and 0 means that it will be forgotten. The mathematical expression of the process performed by forget gate layer :
The next step is about whether to record the new information in the cell state. Input gate layer decides which value to update. The input gate layer and the output of the input gate layer can be formulated as :
A non-linear function layer (tanh) which generates a vector of new candidate values, ŷt which could be added to the state.
The old cell state is then updated and the yt value representing the new cell state is determined. The below equation can be written for obtaining the new yt value.
Finally, to ensure that the output value is between -1 and 1 and filter the output, the output functions can be specified as :
LSTM suffers from vanishing gradients as well, but not as much as the basic RNN. The difference is for the basic RNN, the gradient decays with wσ′(⋅) while for the LSTM the gradient decays with σ(⋅). For the LSTM, there’s is a set of weights which can be learned such that σ(⋅)≈1. Assuming vt + k = wx for some weight w and input x, then Neural Network can learn a large w to prevent gradients from vanishing. For the basic RNN, there is no set of weights which can be learned such that :
The LSTM model in Keras assumes that the data is divided into input (x) and output (y) components. This can be achieve this by using the observation from the last time step (t-1) as the input and the observation at the current time step (t) as the output in a time series.
Let’s add a “pred” column as our output and shift it.
Scaling and normalizing is an important activity in the formation of Neural Network Architecture.
If we check the data, it has a wide range of values (1.336, 13592.79) and neural network do not perform well in this kind of range.
We have defined a function to specify the look back interval (60 time steps) and the predicted column as below.
Here, I have split the data in 80/20 ratio. Moreover, we need to reshape the data to 3D array for LSTM to read and train the model.
We can see the training set contains 4103 data points and testing set has 966 data points.
Now, we add the LSTM layers and few Dropout layers to prevent overfitting. I have added 3 extra layers in the network. The following arguments are considered for the network architecture:
50 units which is the dimensionality of the output space,
return_sequences = True, which determines whether to return the last output in the output sequence, or the full sequence,
input_shape as the shape of the training set,
The Dropout layers specified as 0.2, i.e. 20% of the layers will be dropped,
Thereafter, I have added the Dense layer that specifies the output of 1 unit,
Next, the model was fit to run on 100 epochs with a batch size of 32. Instead of training the RNN on the complete sequences of observations, we have used a batch of shorter sub-sequences (32) picked at random from the training-data.
model_lstm = tf.keras.Sequential()model_lstm.add(tf.keras.layers.LSTM(75, return_sequence = True, input_shape = (xtrain.shape[1], xtrain.shape[2])))model_lstm.add(tf.keras.layers.LSTM(units=30, return_sequence=True))model_lstm.add(tf.keras.layers.LSTM(units=30))model_lstm.add(tf.keras.layers.Dense(units=1))model_lstm.Compile(loss = 'mae', optimizer = 'adam')model_lstm.summary()history_lstm = model_lstm.fit(xtrain, ytrain, epochs = 15,        batch_size=32, validation_data = (xtest, ytest), shuffle=False)
A line plot of model accuracy on the train and test sets is created, showing the change in performance over all 15 training epochs. The line plot below of the test data (blue) vs the predicted values (orange) providing the context for the model skill.
The model is fit to the training data; therefore, it can be used to make forecasts. Now, used the trained model to predict the “pred” values in test dataset. I have used fixed approach here for prediction.
Fixed approach : We can decide to fit the model once on all of the training data, then predict each new time step one at a time from the test data
Dynamic approach: We can re-fit the model or update the model each time step of the test data as new observations from the test data are made available.
We must invert the scale on forecasts to return the values back to the original scale so that the results can be interpreted and a comparable error score can be calculated.
Now we can see, we are not quite there yet. The result is not that impressive. We can check the difference between these two and compare the results in various ways & optimize the model before we build the trading strategy.
Let’s see if we can improve the model performance by shifting 10 time steps and doing some hyper parameter adjustments.
Here the line plot visually looks better than the previous two on both training and testing set. The model appears to rapidly learn the problem, converging on a solution in about 20 epochs. The loss metrics e.g. RMSE, MAE can be obtained similar way as shown earlier.
Though there is some improvement, but in terms of prediction, we still have rooms for improvement. An important factor here is that, we have used daily prices in this model so the data points are really less only 5,129 data points for Neural Network Architecture. I would recommend to use more than 100,000 data points using minute or tick data for training the model while building ANN or any other Deep Learning model to be most effective.
Six different time series were used in order to measure the performance of the network fairly. We have found these series are causing each other by using Granger’s Causality. The estimation results obtained are compared with the graphs. RMSE and R values were examined as predictive success criteria. However, it is possible to achieve more successful results with more data points and by changing the hyper parameters of the LSTM network.
I can be reached here.