Predicting Stock Price with LSTM. Machine learning has found its... | by Asutosh Nayak | Towards Data Science

Machine learning has found its applications in many interesting fields over these years. Taming stock market is one of them. I had been thinking of giving it a shot for quite some time now; mostly to solidify my working knowledge of LSTMs. And finally I have finished the project and quite excited to share my experience.
I will write about my experience over a series of blogs. The purpose of this series is not to explain the basics of LSTM or Machine Learning concepts. Hence, I will assume the reader has begun his/her journey with Machine Learning and has the basics like Python, familiarity with SkLearn, Keras, LSTM etc. The reason is that there are already excellent articles on topics like “How LSTMs work?” by people who are much more qualified to explain the maths behind it. But I will be sharing links to such articles, wherever I feel like background knowledge might be missing. While there are lots of articles out there to tell you how to predict stock prices given a dataset, mostly authors don’t reveal/explain how they reached that particular configuration for a Neural Network or how did they select that particular set of Hyperparameters. So the real purpose of this article is to share such steps, my mistakes and some steps that I found very helpful. As such, this article is not limited to Stock Price Prediction problem.
Here are the things we will look at :
Reading and analyzing data. (Pandas)Normalizing the data. (SkLearn)Converting data to time-series and supervised learning problem.Creating model (Keras)Fine tuning the model (in the next article)Training, predicting and visualizing the result.Tips & tools that I found very helpful (last article of the series)
Reading and analyzing data. (Pandas)
Normalizing the data. (SkLearn)
Converting data to time-series and supervised learning problem.
Creating model (Keras)
Fine tuning the model (in the next article)
Training, predicting and visualizing the result.
Tips & tools that I found very helpful (last article of the series)
Please note that this first article talks about preprocessing steps and terminologies of LSTM. If you are fairly confident about these steps, you can skip to next article.
Let’s begin!
I will be using the historical stock price data for GE for this post. You can find the data in my kaggle site here. I don’t remember the source of data since I had downloaded it long back. We can read the data into frame as shown below :
df_ge = pd.read_csv(os.path.join(INPUT_PATH, "us.ge.txt"), engine='python')df_ge.tail()
As you can see there are around 14060 items, each representing a day’s stock market attributes for the company. Lets see how does it look on a plot :
from matplotlib import pyplot as pltplt.figure()plt.plot(df_ge["Open"])plt.plot(df_ge["High"])plt.plot(df_ge["Low"])plt.plot(df_ge["Close"])plt.title('GE stock price history')plt.ylabel('Price (USD)')plt.xlabel('Days')plt.legend(['Open','High','Low','Close'], loc='upper left')plt.show()
It seems the prices — Open, Close, Low, High — don’t vary too much from each other except for occasional slight drops in Low price.
Now let’s check out the plot for volume :
plt.figure()plt.plot(df_ge["Volume"])plt.title('GE stock volume history')plt.ylabel('Volume')plt.xlabel('Days')plt.show()
Huh. Did you see something interesting? There is quite a surge in the number of transactions around 12000th day on the timeline, which happens to coincide with the sudden drop of stock price. May be we can go back to that particular date and dig up old news articles to find what caused it.
Now let’s see if we have any null/Nan values to worry about. As it turns out we don’t have any null values. Great!
print("checking if any null values are present\n", df_ge.isna().sum())
The data is not normalized and the range for each column varies, especially Volume. Normalizing data helps the algorithm in converging i.e. to find local/ global minimum efficiently. I will use MinMaxScaler from Sci-kit Learn. But before that we have to split the dataset into training and testing datasets. Also I will convert the DataFrame to ndarray in the process.
This is quite important and somewhat tricky. This is where the knowledge LSTM is needed. I would give a brief description of key concepts that are needed here but I strongly recommend reading Andre karpathy’s blog here, which is considered one of the best resources on LSTM out there and this. Or you can watch Andrew Ng’s video too (which by the way mentions Andre’s blog too).
LSTMs consume input in format [ batch_size, time_steps, Features ]; a 3- dimensional array.
Batch Size says how many samples of input do you want your Neural Net to see before updating the weights. So let’s say you have 100 samples (input dataset) and you want to update weights every time your NN has seen an input. In that case batch size would be 1 and total number of batches would be 100. Like wise if you wanted your network to update weights after it has seen all the samples, batch size would be 100 and number of batches would be 1. As it turns out using very small batch size reduces the speed of training and on the other hand using too big batch size (like whole dataset) reduces the models ability to generalize to different data and it also consumes more memory. But it takes fewer steps to find the minima for your objective function. So you have to try out various values on your data and find the sweet spot. It’s quite a big topic. We will see how to search these in somewhat smarter way in the next article.
Time Steps define how many units back in time you want your network to see. For example if you were working on a character prediction problem where you have a text corpus to train on and you decide to feed your network 6 characters at a time. Then your time step is 6. In our case we will be using 60 as time step i.e. we will look into 2 months of data to predict next days price. More on this later.
Features is the number of attributes used to represent each time step. Consider the character prediction example above, and assume that you use a one-hot encoded vector of size 100 to represent each character. Then feature size here is 100.
Now that we have some what cleared up terminologies out of the way, let’s convert our stock data into a suitable format. Let’s assume, for simplicity, that we chose 3 as time our time step (we want our network to look back on 3 days of data to predict price on 4th day) then we would form our dataset like this:
Samples 0 to 2 would be our first input and Close price of sample 3 would be its corresponding output value; both enclosed by green rectangle. Similarly samples 1 to 3 would be our second input and Close price of sample 4 would be output value; represented by blue rectangle. And so on. So till now we have a matrix of shape (3, 5), 3 being the time step and 5 being the number of features. Now think how many such input-output pairs are possible in the image above? 4.
Also mix the batch size with this. Let’s assume we choose batch size of 2. Then input-output pair 1 (green rectangle) and pair 2 (blue rectangle) would constitute batch one. And so on. Here is the python code snippet to do this:
‘y_col_index’ is the index of your output column. Now suppose after converting data into supervised learning format, like shown above, you have 41 samples in your training dataset but your batch size is 20 then you will have to trim your training set to remove the odd samples left out. I will look for a better way to go around this, but for now this is what I have done:
Now using the above functions lets form our train, validation and test datasets
x_t, y_t = build_timeseries(x_train, 3)x_t = trim_dataset(x_t, BATCH_SIZE)y_t = trim_dataset(y_t, BATCH_SIZE)x_temp, y_temp = build_timeseries(x_test, 3)x_val, x_test_t = np.split(trim_dataset(x_temp, BATCH_SIZE),2)y_val, y_test_t = np.split(trim_dataset(y_temp, BATCH_SIZE),2)
Now that our data is ready we can concentrate on building the model.
We will be using LSTM for this task, which is a variation of Recurrent Neural Network. Creating LSTM model is as simple as this:
Now that you have your model compiled and ready to be trained, train it like shown below. If you are wondering about what values to use for parameters like epochs, batch size etc., don’t worry we will see how to figure those out in the next article.
Training this model (with fine tuned hyperparameters) gave best error of 3.27e-4 and best validation error of 3.7e-4. Here is what the Training loss vs Validation loss looked like:
This is how the prediction looked with above model:
I have found that this configuration for LSTM works the best out of all the combinations I have tried (for this dataset), and I have tried more than 100! So the question is how do you land on the perfect (or in almost all the cases, close to perfect) architecture for your neural network? This leads us to our next and important section, to be continued in the next article.
You can find all the complete programs on my Github profile here.
NOTE: A humble request to the readers — You all are welcome to connect with me on LinkedIn or Twitter, but if you have a query regarding my blogs please post it on comments section of respective blog instead of personal messaging, so that if someone else has the same query, they would find it here itself, and I wouldn’t have to explain it individually. However, you are still welcome to send queries unrelated to blogs or general tech queries, to me personally. Thanks :-)
UPDATE 13/4/19
It has come to my knowledge, since I have written this article, that my model used for this blog may have been overfitted. While I have not confirmed it, it’s likely. So please be careful while implementing this in your projects. You could try out things like lesser epochs, smaller network, more dropout etc.I have used Sigmoid activation for last layer which may suffer from limitation of not being able to predict a price greater than ‘max’ price in dataset. You could try ‘Linear’ activation for last layer to solve this.Fixed a typo in “converting data to time-series” section.
It has come to my knowledge, since I have written this article, that my model used for this blog may have been overfitted. While I have not confirmed it, it’s likely. So please be careful while implementing this in your projects. You could try out things like lesser epochs, smaller network, more dropout etc.
I have used Sigmoid activation for last layer which may suffer from limitation of not being able to predict a price greater than ‘max’ price in dataset. You could try ‘Linear’ activation for last layer to solve this.
Fixed a typo in “converting data to time-series” section.
Thanks to the readers for bringing these to my attention.
UPDATE 21/1/2020
As mentioned in some of the comments, I was exploring other ways to approach the stock prediction problem. I have finally got it working. Interested readers can read about that here.