Predicting Future Stock Market Trends with Python & Machine Learning | by Lucas Rea | Towards Data Science

Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
With the recent volatility of the stock market due to the COVID-19 pandemic, I thought it was a good idea to try and utilize machine learning to predict the near-future trends of the stock market. I’m fairly new to machine learning, and this is my first Medium article so I thought this would be a good project to start off with and showcase.
This article tackles different topics concerning data science, namely; data collection and cleaning, feature engineering, as well as the creation of machine learning models to make predictions.
Note: I previously had the look-ahead bias in my code for this article, which produced some extremely good results (suspiciously good). However they were misleading, and I now aim to try and fix that with cross-validation.
Author’s disclaimer: This project is not financial or investment advice. It is not a guarantee that it will provide the correct results most of the time. Therefore you should be very careful and not use this as a primary source of trading insight.
You can find all the code on a jupyter notebook on my github:
github.com
To begin, we include all of the libraries used for this project. I used the yfinance API to gather all of the historical stock market data. It’s taken directly from the yahoo finance website, so it’s very reliable data.
We then define some constants that used in data retrieval and data processing. The list with the indicator symbols is useful to help use produce more features for our model.
Here’s a link where you can find the actual names of some of these features.
github.com
Now we pull our historical data from yfinance. We don’t have many features to work with — not particularly useful unless we find a way to normalize them at least or derive more features from them.
We see that our data above is rough and contains lots of spikes for a time series. It isn’t very smooth and can be difficult for the model to extract trends from. To reduce the appearance of this we want to exponentially smooth our data before we compute the technical indicators.
We can see that the data is much more smoothed. Having many peaks and troughs can make it hard to approximate, or be difficult to extract tends when computing the technical indicators. It can throw the model off.
Now it’s time to compute our technical indicators. As stated above, I use the finta library in combination with python’s built in eval function to quickly compute all the indicators in the INDICATORS list. I also compute some ema’s at different average lengths in addition to a normalized volume value.
I remove the columns like ‘Open’, ‘High’, ‘Low’, and ‘Adj Close’ because we can get a good enough approximation with our ema’s in addition to the indicators. Volume has been proven to have a correlation with price fluctuations, which is why I normalized it.
Index(['close', 'RSI', 'MACD', 'SIGNAL', '14 period STOCH %K','MFV', '14 period ATR', 'MOM', '14 period MFI', 'ROC', 'OBV_x', 'OBV_y', '20 period CCI', '14 period EMV', 'VIm', 'VIp', 'ema50', 'ema21', 'ema14', 'ema5', 'normVol'], dtype='object')
Right before we gather our predictions, I decided to keep a small bit of data to predict future values with. This line captures 5 rows corresponding to the 5 days of the week on July 27th.
live_pred_data = data.iloc[-16:-11]
Now comes one of the most important part of this project — computing the truth values. Without these, we wouldn’t even be able to train a machine learning model to make predictions.
How do we obtain truth value? Well it’s quite intuitive. If we want to know when a stock will increase or decrease (to make a million dollars hopefully!) we would just need to look into the future and observe the price to determine if we should buy or sell right now. Well, with all this historical data, that’s exactly what we can do.
Going back to the table where we initially pulled our data, if we want to know the buy (1) or sell (0) decision on the day of 1993–03–29 (where the closing price was 11.4375), we just need to look X days ahead to see if the price is higher or lower than that on 1993–03–29. So if we look 1 day ahead, we see that the price increased to 11.5. So the truth value on 1993–03–29 would be a buy (1).
Since this is also the last step of data processing, we remove all of the NaN value that our indicators and prediction generated, as well as removing the ‘close’ column.
Because we used Pandas’ shift() function, we lose about 15 rows from the end of the dataset (which is why I captured the week of July 27th before this step).
Right before we train our model we must split up the data into a train set and test set. However, due to the nature of time-series’, we need to handle this part carefully. If we randomize our train-test set, we could encounter a look-ahead bias which is not good for predicting the stock market. It’s caused when you train your model on data it would’ve already seen.
To prevent this we are going to be training the model using a different technique called cross-validation. The image below illustrates how we are going to partition our data and test the accuracy of the model.
First, we’re going to use multiple classifiers to create an ensemble model. The goal here is to combine the predictions of several models to try and improve on predictability. For each sub-model, we’re also going to use a feature from Sklearn, GridSearchCV, to optimize each model for the best possible results.
First we create the random forest model.
Then the KNN model.
And now finally we create the voting classifier
Once we set up our models, we can put it together with the cross-validation. We start by writing some code that will allow us to iterate over our data with many evenly sized chunks.
Printing out the indices of our dataframe, we have successfully partitioned our data like in the sliding window image.
10 5020 6030 7040 8050 9060 100...6820 68606830 6870
Now we add in the code that will split the partitions into train and test sets. It’s very important to set shuffle=False in the train_test_split function —it’s how you avoid the look-ahead bias.
Finally we incorporate our models.
A sample run of this looks like... the results are exaggerated.
rf prediction is  [1. 1. 1. 0. 0. 1. 1. 1. 1. 1. 1. 1.]knn prediction is  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]ensemble prediction is  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]truth values are  [1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1.]0.9166666666666666 0.9166666666666666 0.9166666666666666
And then the final results gives us
RF Accuracy = 0.6732649071358748KNN Accuracy = 0.67460899315738Ensemble Accuracy = 0.6925708699902248
We can see that we gain slightly more accuracy by using ensemble modelling. Compared to not using cross-validation, the resulting accuracy is much less. However this is a more correct way of going about this problem... and about a 70% accuracy isn’t bad either!
For the next step we’re going to predict how the S&P500 will behave with our predictive model. I’m writing this article on the weekend of August 17th. So to see if this model can produce accurate results, I’m going to use the closing data from this week as the ‘truth’ values for the prediction. Since this model is tuned to have a 15 day window, we need to feed in the input data with the days in the week of July 27th.
July 27th -> August 17th
July 28th -> August 18th
July 29th -> August 19th
July 30th -> August 20th
July 31st -> August 21st
We saved the week we’re going to use in live_pred_data.
live_pred_data.head()
Here are the five main days we are going to generate a prediction for. Looks like the models predicts that the price will increase for each day.
Lets validate our prediction with the actual results.
del(live_pred_data['close'])prediction = ensemble_model.predict(live_pred_data)print(prediction)[1. 1. 1. 1. 1.]
Results
July 27th : $ 322.78 — August 17th : $ 337.91
July 28th : $ 321.74 — August 18th : $ 338.64
July 29th : $ 323.93 — August 19th : $ 337.23
July 30th : $ 323.95 — August 20th : $ 338.28
July 31st : $ 325.62 — August 21st : $ 339.48
As we can see from the actual results, we can confirm that the model was correct in all of its predictions. However there are many factors that go into determining the stock price, so to say that the model will produce similar results every time is naive. However, during relatively normal periods of time (without major panic that causes volatility in the stock market), the model should be able to produce good results.
To summarize what we’ve done in this project,
We’ve collected data to be used in analysis and feature creation.We’ve used pandas to compute many model features and produce clean data to help us in machine learning. Created predictions or truth values using pandas.Utilized cross-validation to avoid the look-ahead bias. Trained many machine learning models and then combined them using ensemble learning to produce higher prediction accuracy.Ensured our predictions were accurate with real world data.
We’ve collected data to be used in analysis and feature creation.
We’ve used pandas to compute many model features and produce clean data to help us in machine learning. Created predictions or truth values using pandas.
Utilized cross-validation to avoid the look-ahead bias. Trained many machine learning models and then combined them using ensemble learning to produce higher prediction accuracy.
Ensured our predictions were accurate with real world data.
I’ve learned a lot about data science and machine learning through this project, and I hope you did too. Being my first article, I’d love any form of feedback to help improve my skills as a programmer and data scientist.
Thanks for reading :)