Beating the S&P500 Using Machine Learning | by Thomas Rochefort-Beaudoin | Towards Data Science

The S&P 500 index is one of the most followed market indexes and is often used as a benchmark for the overall US equity market. While one could consider investing in a passive fund replicating the S&P500, many will turn towards fund managers and will prefer investing in active funds. Unfortunately, more and more studies and data have shown that it is excessively hard to “beat the market” (outperform the benchmark). One can go here and here to read more on the subject matter and this excellent book by Burton G. Malkiel is also highly recommended. One of the conclusions of those studies is that 91.62 % of the actively managed equity funds in the US underperformed the S&P 500 on the 15 year-period ending in December 2018.
In this article, a systematic approach based on machine learning is presented, which is able to vastly outperform the S&P 500 in the same period.
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
A machine learning algorithm written in Python was designed to predict which companies from the S&P 1500 index are likely to beat the S&P 500 index on a monthly basis. To do so, a random forest regression based algorithm, taking as input the financial ratios of all the constituents of the S&P 1500, was implemented. The following workflow was used in this project:
Data harvesting and preparationData cleaningFeature selectionTraining and backtesting the random forest regressor
Data harvesting and preparation
Data cleaning
Feature selection
Training and backtesting the random forest regressor
*Please note that the datasets used in this project came from proprietary sources (Standard & Poor’s and Bloomberg) and therefore cannot be published. We will therefore skip step 1 in this article. Those with access to the datasets through the required subscriptions can instead refer to the complete notebook hosted on the following Github project: SP1500StockPicker.
The random forest method is based on multiple decision trees. A single decision tree is constructed from a top-down method, where the dataset follows a series of boolean splits based on variable values which result in the most accurate separation at each node. Each unique decision tree is trained on different samples of the training set with replacement (meaning the sample is replaced into the larger datasets). The random forest regression averages multiple decision trees together to emit a prediction. This article will not go too deep into the details of this technique, but more information can be found on the random forest method in these excellent articles by Prashant Gupta and George Seif.
The following code was written in Python in a Jupyter Notebook. The JupyterHub environment was used to host the code on a computing server. Most of the data manipulation was made using the Pandas and NumPy libraries. The machine learning algorithm is based on the extensive Scikit-learn library. For the parallelization of some sections of the code, the multiprocessing package from Python was used. Finally, the feature selection process was based on the feature-selector package.
One of the beauties of Python is the immense amount of packages and libraries which are open-source and available to anyone. Leveraging the power of existing tools to create innovative solutions allows for fast and efficient problem solving.
The starting dataset is composed of 86 variables. A quick look at the first 5 entries is presented in the next figure:
Some of the most important variables are shown in the table below. The other variables are all defined in the “variables.txt” file in the Github repo.
One can see how many missing entries we have for each variables using the following pandas command:
dataset.isna().sum()trt1m                5141exchg                   0tpci                    0adate                 441qdate                   0public_date             0CAPEI                2332bm                   4764evm                  1942pe_op_basic          9984pe_op_dil           74009pe_exi              10594pe_inc              10261ps                    475roe                  5294roce                 3336                    ...  dltt_be              5698debt_assets           532debt_capital         5328de_ratio              535intcov              63530intcov_ratio        63542cash_ratio          56652quick_ratio         56634curr_ratio          56635sprtrn                  0win                     0decision_date           0SP1500                  0Length: 86, dtype: int64
We can see that some variables have a significant amount of missing entries, knowing that the total length of the dataframe is 413 012 entries. In order to address this problem, the following steps were taken to clean the data:
The rows missing the monthly return of the stock were deleted since we cannot trade missing returns.The variables where more than 50% of the rows were empty were removed.For the missing financial ratios, we used the SimpleImputer package from Scikit-learn to replace the missing values with the average value of the variable for the entire database. This approach is extremely lazy and will add modeling errors into the data. The more appropriate approach would be to use the average value for a period of n months for each specific stocks. Still, the lazy approach was used and the following code summarizes the 3 steps enumerated here.
The rows missing the monthly return of the stock were deleted since we cannot trade missing returns.
The variables where more than 50% of the rows were empty were removed.
For the missing financial ratios, we used the SimpleImputer package from Scikit-learn to replace the missing values with the average value of the variable for the entire database. This approach is extremely lazy and will add modeling errors into the data. The more appropriate approach would be to use the average value for a period of n months for each specific stocks. Still, the lazy approach was used and the following code summarizes the 3 steps enumerated here.
#Dropping rows with now "trt1m" values:database = database[np.isfinite(database['trt1m'])]# Dropping columns with more than 50% of the rows missing:database = database.loc[:, database.isnull().mean() < .5]#Looping through the variables to replace NaNs with the average:imp=SimpleImputer(missing_values=np.nan, strategy="mean" )impute_columns=database.columns[database.isna().any()].tolist()for i in impute_columns:    database[i] = imp.fit_transform(database[[i]])
Auxiliary steps were taken to finish the cleaning process and can be looked up in the complete notebook inserted at the end of this article or hosted on my Github.
The package feature_selector from WillKoehrsen was used. One can read his article here for a comprehensive tour of its functions.
Basically, a feature selection algorithm will help identify meaningless (low importance) and also collinear features in order to trim the starting dataset from unnecessary data variables. Removing those collinear variables will reduce the dataset from entries that do not contribute, hence speeding up the overall machine learning process. This feature selection will also help in improving the generalization capabilities of the algorithm when trying to identify non-linear relationships between variables.
Identifying highly correlated features and printing out the correlation matrix is very simple with the feature_selector package:
fs.identify_collinear(correlation_threshold=0.975)fs.plot_collinear(plot_all=True)
From the above plot, we can see that some features have high correlation (more red) and 5 features with correlation higher than 0.975 were identified by the function.
Next, we want to identify features with zero importance. Those features will not be used to split any nodes of a decision tree and can therefore be removed to decrease training time. The feature_selector package implements a gradient boosting machine from the LightGBM library. The function will average the feature importances of 10 training runs to reduce variance.
fs.identify_zero_importance (task = 'regression', eval_metric = 'auc', n_iterations = 10, early_stopping = True)1 features with zero importance after one-hot encoding.
Low importance features can also be identified with the feature_selector package. The method uses the results from the zero importance feature selection to identify features that are not required to obtain 99% of the total importance.
fs.identify_low_importance(cumulative_importance = 0.99)69 features required for cumulative importance of 0.99 after one hot encoding.5 features do not contribute to cumulative importance of 0.99.
We can now remove all of the selected features from the 3 steps above to obtain the final dataframe:
all_to_remove = fs.check_removal()database = database.drop(columns = all_to_remove)Total of 9 features identified for removal:['ptpm', 'tpci', 'debt_invcap', 'debt_ebitda', 'opmad', 'aftret_equity', 'dltt_be', 'cfm', 'capital_ratio']
The machine learning algorithm presented in this section will predict the stocks which are most likely to outperform the S&P500 for an entire month. The prediction will be made on the first day of a specified month. We also want to be able to backtest the algorithm for various periods in time to validate the performance of the model. We will define a function called “rfstockpicker_backtest” which will take the following entries:
The first section of the function takes the decision date and identifies the previous month. It then creates a training set ranging from the previous month down to 7 years earlier. This duration was chosen because it is known that the average duration of the past 33 economic cycles was 56 months. The current expansion cycle has lasted for 10 years. 7 years allows us to get approximately an entire economic cycle for each of our training set. The test set will be the month on which we want to emit predictions. We will be using a basic iteration to go back in time, month after month, in order to backtest over several months. Instantiating our variables at each iterations ensures that no information is propagated back in time (see entire code for details).
After dropping non-necessary data for training and testing, the function instantiates aRandomForestRegressor object from the sklearn library and initialize it using the number of trees (n_estimators) and available processes (n_jobs) supplied as input parameters. The function then trains the algorithm on the training set and emits predictions for every stock of the SP1500 on the test month.
The last section of the function returns the various feature importances for further analysis.
For example, one can test the function for the month of December 2018:
results,feature_imp,importances,train_feature_list=rfstockpicker_backtest(database,("2018-12-01"),7,1,100,24)
This call to the “rfstockpicker_backtest” function will return the following information:
For example, let’s say that for each month, it is decided to buy the top 10 stocks with the highest chance of beating the index. These can be displayed using the following command:
portfolio=results[0].nlargest(10, 'predictions')portfolio
Averaging those returns with equal weights (arithmetic averaging) gives a return of 2.09% over the month of December 2018, compared to the SP500 which returned 1.79% over the same period.
The model can be improved by looking at feature importance:
Although in this figure, many features are compressed on the x-axis, it is still possible to see that some of them have very low importance. The features with importances lower than 0.02 can be removed using the following command:
df_importance = pd.DataFrame(feature_imp, columns = ['Feature', 'Importance']) feature_to_drop=df_importance.Feature[df_importance['Importance']<0.02]new_db=database.drop(columns=feature_to_drop)
This new database allows us to increase the monthly return for December 2018, returning 3.57%, an improvement of +1.48%.
Now let’s backtest our model over a longer period of 15 years, from December 2003 to December 2018:
results,feature_imp,importances,train_feature_list=rfstockpicker_backtest(new_db,("2018-12-01"),7,(12*15),100,24)
The main iteration loop will start on December 2018, and go back in time 1 month for 12*15 or 180 months to cover the entire backtest period. Over the test period of 180 months, a compounded return of +1421 % was obtained for our model, compared to +261 % compounded return for the actual SP500 index. The model, therefore, generated +1160% of excess return over the 15 years period.
Three main hypothesis were made to simplify the project:
Every stock bought on the previous month were sold and 10 new stocks were bought for each month tested. It is, therefore, possible that a stock could have been sold at the end of the month and then bought again on the next day.No transaction costs were accounted for, which would impact the compounded return of the model. For every month tested, 10 stocks were purchased and 10 stocks were sold.No dividends were accounted for, which would impact also the compounded return of the model.
Every stock bought on the previous month were sold and 10 new stocks were bought for each month tested. It is, therefore, possible that a stock could have been sold at the end of the month and then bought again on the next day.
No transaction costs were accounted for, which would impact the compounded return of the model. For every month tested, 10 stocks were purchased and 10 stocks were sold.
No dividends were accounted for, which would impact also the compounded return of the model.
A lot of improvements could also be implemented into the code. Additional machine learning techniques and neural networks will be explored in future articles. Stay tuned!
If you have any comments or questions, feel free to write them below! I will do my best to answer them. The code for the article is made available in a Jupyter Notebook from the project’s Github Repo.
Thank you for your time!