How (NOT) To Predict Stock Prices With LSTMs | by Viraf | Towards Data Science

Not so recently, a brilliant and ‘original’ idea suddenly struck me — what if I could predict stock prices using Machine Learning. After all, a time series can be easily modeled with an LSTM. I could see myself getting rich overnight! If this is so easy, why hasn’t anyone done it yet?
Very excited at my bright prospects, I powered up my laptop, opened Google, and keyed in “predict stock prices LSTM python.” The results poured in — and very quickly, I realized that my idea was not very original. Well, so much for that. Anyway, I went through many of these articles, and each one of them seemed to have gotten surprisingly good results. But the tutorial ends there — no one put it to test with real money. Why aren’t these people millionaires yet? Something was fishy.
As a disclaimer, note that none of the contents of this article is financial advice and is purely educational. Quite obvious, I know, but needs to be said. On the contrary, I am trying to educate you on how to not be fooled by this, and start using your head.
Nowadays, with Stocks and Machine Learning both becoming so hands-on, easy-to-use, and accessible, it has become really hard to not fall into this trap, where an incomplete, half-baked knowledge of both fields can lead you into serious trouble.
Let’s get started on how to NOT use an LSTM for predicting stock prices. The flow of this article is as follows:
A simple introduction to LSTMs.
Get historical stock data in python.
Create a dataset in a format suitable for the LSTM model.
Build and train the LSTM model with TensorFlow Keras.
Predict and interpret the results.
Long-Short-Term-Memory (LSTM) networks are a type of neural network commonly used to predict time series data. In simple words, they have a memory/cache functionality which helps them learn the long term dependencies and relations in the data. So, looking at the previous N data points, they can predict the next (or next few) points by learning the patterns.
I am going to keep this part quite simple. If you want more clarity on how an LSTM functions, I recommend going through this, but you should pretty much understand the main gist and purpose of this article even without an in-depth understanding.
There are multiple options to get access to historical stock prices in python, but one of the simplest libraries is yfinance. Quite convenient and free, it gets the job done by scraping data from yahoo finance.
!pip install yfinance# Import the required librariesimport yfinance as yfimport pandas as pdimport matplotlib.pyplot as pltfrom pandas.plotting import register_matplotlib_convertersregister_matplotlib_converters()
For this article, I will take up the stock prices of ‘Reliance,’ but this is valid for all other instruments as well.
The following piece of code downloads stock price data for Reliance over 15 years with a resolution of 1 day and stores it in a pandas dataframe. You can vary these parameters as you deem fit for your experiments.
Printing out the head of the pandas dataframe, you can see the various parameters available for the stock data.
Plotting out the ‘close’ price to visualize the data, see how nicely the stock has risen in the last few years. The steep drop in March 2020 (due to COVID) is also visible. But it has seemingly beaten this downfall and risen to a new high yet again.
A question to ask yourself here — do you think an ML model will be able to capture all this randomness?
Let’s fix our problem statement now — the LSTM model shall see the close prices for the last 10 days (called the time_step) and predict the close price for the next day.
For simplicity, let’s test on only the final 200 days. We can train on the rest of the data. We have 3486 data points to train on (will increase depending on the day you execute the code).
(Note that you can vary this according to your requirements, but I feel this should be sufficient to put forth my point).
# Get Close datadf = data[['Close']].copy()# Split data into train and testtrain, test = df.iloc[0:-200], df.iloc[-200:len(df)]print(len(train), len(test))>>> 3486 200
Now, we need to normalize our data. One common mistake people make at this stage is that they normalize the test data separately as well. But in a practical scenario, the test data will be in real-time, so you won’t know the minimum or maximum or average values beforehand! If you provide the model with this information from the future, you basically end up providing a trend or guideline that the predicted prices need to follow. I have seen so many authors make this error, and I don’t blame them since it took me a while to realize this too.
To tackle this issue, we will use the minimum and maximum of the training data to normalize the test data as well. This should work as a decent approximation (of course, there are better ways to do it, for example, take min and max values of only last N prices).
Finally, let’s structure the data so that our LSTM model can easily read them. The LSTM requires input in the form of [samples, time_steps, features]. The following code creates the dataset in this needed format.
For our model, we will be using the TensorFlow Keras library in python. The simple sequential model has an LSTM layer followed by a dropout (to reduce over-fitting) and a final dense layer (our output prediction). We shall use the ever-dependable ‘Adam’ optimizer with Mean Absolute Error loss. Train it for about 50 epochs. Remember to keep shuffle = False (since the sequence of samples is important — kind of the whole point here :P).
The summary of the model is as shown in the picture below. The model is very minimal, but you can try and play around with a few more layers and whatnot. Be careful not to overfit the data though (even with this small model the data overfits).
Post-training, plot the loss as:
plt.plot(history.history['loss'], label='train')plt.legend();plt.show()
The training loss has clearly reduced over the epochs (with some ripples). The model seems to have trained well. Or has it? Let’s evaluate it.
Pass the test set through the model. Rescale the results (remember, we had normalized it earlier) and finally plot them out to see how it performed visually.
Look at the predictions in the image below! They are shockingly good! The model seems to have predicted the COVID fall as well! This is brilliant! We can make millions in the stock market now!
Most tutorials end here. The authors’ final words being ‘The results look promising, tweak the parameters, alter the model, etc.’ But wait, before you go quit your job to become a full-time trader, let’s zoom into the results once!
Can you see what is happening now? The predicted price is trailing the true price by 1 time-step. The model is just predicting a value close to the previous price it sees, as this is the best prediction for the next price.
Let me make it easier for you. Let’s plot the true price value with a lag of 1 day in the image below. See how the shapes of the plots match — now it is clearer to see that our model is just trying to mimic the last price it sees.
What seems to be the problem? The LSTM is not doing anything wrong! As you know, stocks are quite dynamic and random (for all practical purposes). The model is giving the best possible guess for a random walk problem, which turns out to be the previous day’s price. But this means that the model is not predicting anything useful for trading/investing. And that is why blindly using this in practice is very dangerous.
I am not saying that the task of predicting stock prices is impossible. It definitely is possible — there are several algorithms and trading bots out there, which employ some amount of machine learning — but it is not going to be so damn straightforward.
A little more effort will be required to formulate the problem statement, inputs, and model. Some good direction to head in would be predicting changes in value (a derivative or second derivative) instead of the value itself. Predicting only the stock price movements (binary classification problem) could be another option. Or to use an ensemble of models to achieve combined/different goals. I’d recommend that you try to gain stronger domain knowledge in these fields first, followed by explorations — don’t restrict yourself to these ideas or other ideas you see on the internet.
I am not saying I am an expert in these fields — I am just putting forward the conclusions I got to after my explorations with this topic, so feel free to point out my errors or add anything I missed. Thank you for your time.
Can an ML model literally read visual stock price charts? Check out my article below to see the results!
towardsdatascience.com
Why waste time and effort in trying to predict the stock market when you can do this...
medium.com
Or, check out some of my other machine learning articles. I’m sure you’ll find them useful...