My Experiments with Truth- Stock Prediction
By: Ajay Mukund
Tags: Yahoo Finance, Predictive Analytics, Stock Prediction, Recurrent Neural Network, Lstm

These are the steps followed and my learnings from Hackathon 2020

It was a beautiful day when I saw the data for how stocks work. I had my own preconceptions that predicting stock prices are as simple as predicting any other machine learning problems’ outcome but I was in for a surprise which made me write this blog post to inform some of the people in the data science community who have the same suppositions like mine.

I was introduced in the hackathon about the library ‘yfinance’ (stands for yahoo finance) which has up to date data about stocks in various banks such as HDFC, ICICI, SBI, Kotak, Axis Bank etc. We will be able to simply download the data of stock prices from yfinance using the ticker of that particular bank for which you want to find the stock prices. Some of the ticker codes are as follows:

HDFC- ‘HDFCBANK.NS’

ICICI- ‘ICICIBANK.NS’

SBI- ‘SBIN.NS’

Kotak- ‘KOTAKBANK.NS’

Axis- ‘AXISBANK.NS’

About the Dataset from Yahoo Finance

The data consists of 8 numerical columns namely Date at which the particular observation has been recorded, Opening stock price on that particular day (minute), Closing stock price on that particular day (minute), High and low which specifies the highest price of stock and the lowest price of stock on that particular day (minute) respectively to provide the insight of stock price varying range, Volume of stocks, Dividends and Stock Splits.

A typical Stock Prices Data- IBM

Data can be obtained for every day over the past years (approximately 5 years) or it can also be obtained minute by minute for the past seven days. My aim was to predict the stock prices for the months November 2019 and December 2019 which I took as Test set based on the data available from January 1996 to October 2019 which I took as the Training set. Intuitively, I thought that using Recurrent Neural Networks with Long Short Term Memory (LSTM) will give me the best results. I wanted to predict the binary outcome. That is, on a particular given day, will the closing stock price increase (denoted by 1) or will it decrease (denoted by 0).

I wrote a Chi-Square Test function to delete or drop the columns that were not affecting (independent) the values in the closing stock price. Running the Chi-Square Test, I was in for the first surprise of my day when the results proved that all columns in the dataset affects the values present in the closing stock price column.

This is the Chi-Sqaure Function which I wrote

I moved on to my Spyder3 to implement my code. First I took the data from January 1996 to October 2019 as my training set and from that I took only the column “Close” which denotes the closing stock price of that particular day. Then I scaled the values of closing stock prices in the range 0 to 1 using the MinMaxScaler

Obtaining the 4th column which is the Closing Stock Price Column and scaling using MinMaxScaler

Then, I came up with the task of creating a data structure with 60 timesteps and 1 output. Timesteps is the amount of data that is being taken into account given that you want to predict for a particular day. Approximately, there exists 20 trading days in a month. So, 60 timesteps means 3 months taken into account. Therefore, here the assumption is that for a particular day say x, we will be taking into account the days from x- 60 to x- 1. That forms the data structures X_train and y_train (output).

X_train with 60 Timesteps and 5885 observations (days)- Creation of X_train and y_train

To give X_train as an input to the Recurrent Neural Network, it is necessary for X_train to be reshaped as a 3-D data structure.

The next step is to start building our Recurrent Neural Network. My intuition behind this binary prediction was that, if I am able to predict the closest real value pertaining to the closing stock price, I can simply compare the closing stock price of the previous day and the predicted closing stock price to find the binary output whether the price has increased or decreased (1 or 0 accordingly).

Building the Recurrent Neural Network

So, I started building a regression model when a classification model was required. The particular I built was a stacked LSTM with each LSTM layer containing 50 units each and a Dropout regularization value of 0.2. Dropout rate is the rate of the neurons which you want to drop or that you want to ignore in the layers. Here we are dropping 20% of the neurons from every layer of LSTM. The stacked LSTM contains 4 LSTM layers. The last step for building the Recurrent Neural Network is to add the output layer of the neural network. Since the output layer is not an LSTM layer but a fully connected layer, we use the “Dense” class and the number of neurons present in this layer which is 1. Since we are predicting a real value, corresponding to the closing stock price, the output has only one dimension and this one dimension is exactly what we should input as the units (number of neurons) which is 1 (one).

Architecture of the Recurrent Neural Network with 4 stacked LSTMs and an output layer with 1 neuron

Compiling and Fitting the Recurrent Neural Network

That makes the architecture of the Recurrent Neural Network. There are two more steps. One is compiling the RNN with a powerful optimizer and the loss. The second is to fit this model on the training set. There are so many optimizers that can be used for Recurrent Neural Network. In particular, “rmsprop” optimizer is recommended for RNNs but for this particular problem, I found using parameter tuning that “adam” optimizer performs better than rmsprop and adagrad. The second parameter is the loss function and since we are not doing a classification, the loss is not going to be “Binary Crossentropy”. Since we are dealing with a regression problem, the loss is the “Mean-Squared Error”. This makes perfect sense as the error can be measured by the mean of the squared differences between the predictions and the target.

Compiling the Recurrent Neural Network with “adam” optimizer

Fitting of the model happens by inputting the X_train (the input) and y_train (the ground truth). The next argument is the number of epochs which specifies that in how many iterations should the neural network be trained. I specified the number of epochs to be 100 which means that the whole training data is forward propagated inside the neural network and back propagated 100 times to update the weights. Instead of every stock price being forward propagated inside the neural network, we send the inputs batch wise and here we send a batch of size 32 every single time.

Fitting the RNN to the training set with number of epochs 100 and batch size 32

The next step is to import the test set which consists of data available from November 2019 and December 2019 and make an array real_stock_price which takes all the closing stock prices of the test set. Then, we are going to get the predicted_stock_price for those two months. We trained our model to predict the closing stock price at time t+1 based on the 60 previous closing stock prices and therefore for predicting the stock price of each financial day, we will be using both the training set and the test set. We concatenate both the training set and the test set and scale them.

Get the closing stock prices from column 4 from the Test dataset and concatenate test and training

Now we are ready to predict the closing stock prices. The range is from 60 (number of timesteps) to 101 (60+ the number of financial days in the test set whcih is 41). Create another data structure X_test which will have all the inputs of test set and reshape it into 3-D format. Taking the regressor, we predict the predicted_stock_price using the X_test. We would have got the predicted_stock_price values in the range of 0 to 1 and therefore, we have to inverse the scaling using the inverse_transform method to get the original predicted_stock_price.

Predict and Inverse Transform the ouputs

I have then created a pred_array which gives the binary output of whether the closing stock price will increase or decrease based on the predicted_stock_price array comparing the current value with its previous value. Same way, we get the actual_array from the real_stock_price array.

Conversion of Regression into a Classification problem

We got an accuracy of 56% which is indeed very bad. This suggests that the use of LSTMs is not a very good method to predict the stock prices

Conclusion

My intuition in changing the Binary classification problem went wrong. The LSTMs view the long term effects of the stock and thus does not react immediately to changes and therefore gives a very smooth curve of prediction which was not at all useful in the binary classification as the predictions start going down which means that all are zeroes (0) and then starts, moving upwards and results in the predictions being all ones (1) and thus doesnot work with binary classification. Some of the models that can give better results are Decision Trees, Logistic Regression and Linear Regression. Still one cannot assure that the model’s prediction is state of art as there are so many factors that affect the stock prices like recent news which cannot be taken into consideration and thus stock predictions remain an area where some more work has to be done and there is scope for improvement.

I guess that this blog post would have cleared many misconceptions that people would have in the data science community about predicting the prices of stock. And yes! we all know that if there exists a model that can predict stocks accurately, all data scientists would end up being Billionaires!

The link to the code present in Github is as follows: https://github.com/AjayMukundS/Hackathon-2020