Predicting Returns with Fundamental Data and Machine Learning in Python
By: Nate Cibik
Tags: Quantitative Finance, Data Science, Python, Machine Learning, Finance

Feature Selection and Modeling

This section will be split into our two major modeling tasks: asset price modeling and modeling of returns. For each, we will perform feature engineering, then compare the efficacy of imputation techniques, train machine learning algorithms, and then interpret our modeling results.

Modeling Asset Price

The purpose of this task is to model the prices of the securities using the fundamental data related to the companies, in order to perform a pseudo-fundamental analysis of intrinsic value to develop a value investing-style trading strategy based on the residuals of our model, which represent margin of safety, or in other words, how over or under valued a given security is compared to what the model estimates its value to be. The hypothesis is that the residuals will be correlated to the returns of the securities over the six month period since the data were scraped, as overvalued stocks would see their prices move down as the market adjusts toward their true value, and undervalued stocks would move up.

In financial analysis, intrinsic value is at least partially subjective, as different analysts will arrive at different estimates through construction of their own individual proprietary pricing models, meaning that in this experiment we are actually trying to model a hidden variable using the current trading price. The logic behind why this may work is that markets are at least partially efficient, meaning that the current prices of assets reliably reflect their value, with some error and noise present. So, while we are using market price as a target variable, the hope is that the model we build finds the way that the features contribute to prices across the market, and that our residuals will reflect deviations from actual (intrinsic) value.

Feature Selection:

As discussed above, using features which contain information about the current price of the securities will cause target leakage, and undermine our goal of estimating intrinsic value. For example, price to earnings ratio (P/E) is calculated as the price of a security divided by its earnings per share (EPS). Since we have EPS and P/E present in our features, the model could easily factor out the current price of the security, and too accurately reflect the current trading prices, rather than the intrinsic value. Therefore, we must be careful to remove or modify all features which would allow such target leakage.

The first step is to remove all features which are directly related to price, such as anything to do with periodic high/low, percent above/below these marks, and any open/close/last/bid/ask related features. The price ratios can have the prices factored out by dividing the current price of the securities by the ratio itself, leaving only the earnings/book/sales/cash flow behind, which represent fundamental information about the company. We will also remove any columns with highly incomplete data, and columns which are datetime, since none of these are useful in this task. Annual Dividend % and Annual Dividend Yield represent the same thing, and the latter is missing more values, so it will be dropped.

<class 'pandas.core.frame.DataFrame'>

Index: 501 entries, A to ZTS

Data columns (total 65 columns):

% Held by Institutions 495 non-null float64

5yr Avg Return 501 non-null float64

Annual Dividend % 394 non-null float64

Beta 488 non-null float64

Change in Debt/Total Capital Quarter over Qua...473 non-null float64

Days to Cover 495 non-null float64

Dividend Change % 405 non-null float64

Dividend Growth 5yr 354 non-null float64

Dividend Growth Rate, 3 Years 392 non-null float64

EPS (TTM, GAAP) 490 non-null float64

EPS Growth (MRQ) 487 non-null float64

EPS Growth (TTM) 489 non-null float64

EPS Growth 5yr 438 non-null float64

FCF Growth 5yr 485 non-null float64

Float 495 non-null float64

Gross Profit Margin (TTM) 422 non-null float64

Growth 1yr Consensus Est 475 non-null float64

Growth 1yr High Est 475 non-null float64

Growth 1yr Low Est 475 non-null float64

Growth 2yr Consensus Est 497 non-null float64

Growth 2yr High Est 497 non-null float64

Growth 2yr Low Est 497 non-null float64

Growth 3yr Historic 497 non-null float64

Growth 5yr Actual/Est 497 non-null float64

Growth 5yr Consensus Est 497 non-null float64

Growth 5yr High Est 497 non-null float64

Growth 5yr Low Est 497 non-null float64

Growth Analysts 497 non-null float64

Historical Volatility 501 non-null float64

Institutions Holding Shares 495 non-null float64

Interest Coverage (MRQ) 389 non-null float64

Market Cap 501 non-null object

Market Edge Opinion: 485 non-null object

Net Profit Margin (TTM) 489 non-null float64

Operating Profit Margin (TTM) 489 non-null float64

P/E Ratio (TTM, GAAP) 451 non-null object

PEG Ratio (TTM, GAAP) 329 non-null float64

Price/Book (MRQ) 455 non-null float64

Price/Cash Flow (TTM) 468 non-null float64

Price/Earnings (TTM) 448 non-null float64

Price/Earnings (TTM, GAAP) 448 non-null float64

Price/Sales (TTM) 490 non-null float64

Quick Ratio (MRQ) 323 non-null float64

Return On Assets (TTM) 479 non-null float64

Return On Equity (TTM) 453 non-null float64

Return On Investment (TTM) 437 non-null float64

Revenue Growth (MRQ) 487 non-null float64

Revenue Growth (TTM) 489 non-null float64

Revenue Growth 5yr 490 non-null float64

Revenue Per Employee (TTM) 459 non-null float64

Shares Outstanding 495 non-null float64

Short Int Current Month 495 non-null float64

Short Int Pct of Float 495 non-null float64

Short Int Prev Month 495 non-null float64

Short Interest 495 non-null float64

Total Debt/Total Capital (MRQ) 460 non-null float64

Volume 10-day Avg 500 non-null float64

cfra 479 non-null float64

creditSuisse 337 non-null object

ford 493 non-null float64

marketEdge 484 non-null float64

marketEdge opinion 484 non-null object

newConstructs 494 non-null float64

researchTeam 495 non-null object

theStreet 496 non-null object

dtypes: float64(58), object(7)

memory usage: 258.3+ KB

Things are looking cleaner. The next step for this task is to get rid of the analyst ratings, since these are not really fundamental. Another issue to deal with is that we have duplicates of P/E ratio. The column named ‘P/E Ratio (TTM, GAAP)’ is of object data type, and we have the same thing below with ‘Price/Earnings (TTM, GAAP)’ with a numeric dtype. We also have ‘Price/Earnings (TTM)’ that incorporates what is called non-GAAP earnings, which purposefully leave out any large nonrecurrent expenses that the company has had recently which may obfuscate financial analysis, and are considered favorable for purposes such as ours. Thus, we will drop the GAAP earnings, and keep the non-GAAP earnings. PEG ratio is the P/E ratio divided by the annual EPS growth rate, both of which we already have, so it can be dropped.

Things are getting cleaner, but we have more considerations to make. The Market Cap column is still encoded in string format, but it also contains information about the prices, and will cause target leakage. Market capitalization is calculated as the number of outstanding shares times the current price of the security, and since we have a Shares Outstanding column, we need to get rid of Market Cap altogether. Also, we have not checked our dataset for nan’s evil stepbrother: inf. Let’s drop Market Cap and perform this check.

True

We have indeed located some culprits. Not to worry, since we were already planning to deal with missing data, we will just re-encode these as nans to be dealt with later.

And it is done. We need to check now to make sure that there are no companies in our feature set that aren’t present in our target data. We can perform a check with a list comprehension as follows:

['AGN', 'ETFC']

These are familiar faces, they are the companies which have been acquired by others over the period of study. We need to drop them from our feature set.

Finally, we need to remove the pricing information from the pricing ratios by dividing the current price by each ratio. This can be done like so:

<class 'pandas.core.frame.DataFrame'>

Index: 499 entries, A to ZTS

Data columns (total 52 columns):

% Held by Institutions 493 non-null float64

5yr Avg Return 499 non-null float64

Annual Dividend % 392 non-null float64

Beta 486 non-null float64

Change in Debt/Total Capital Quarter over Qua...471 non-null float64

Days to Cover 493 non-null float64

Dividend Change % 403 non-null float64

Dividend Growth 5yr 354 non-null float64

Dividend Growth Rate, 3 Years 392 non-null float64

EPS (TTM, GAAP) 488 non-null float64

EPS Growth (MRQ) 485 non-null float64

EPS Growth (TTM) 487 non-null float64

EPS Growth 5yr 437 non-null float64

FCF Growth 5yr 483 non-null float64

Float 493 non-null float64

Gross Profit Margin (TTM) 420 non-null float64

Growth 1yr Consensus Est 473 non-null float64

Growth 1yr High Est 473 non-null float64

Growth 1yr Low Est 473 non-null float64

Growth 2yr Consensus Est 494 non-null float64

Growth 2yr High Est 495 non-null float64

Growth 2yr Low Est 494 non-null float64

Growth 3yr Historic 495 non-null float64

Growth 5yr Actual/Est 495 non-null float64

Growth 5yr Consensus Est 494 non-null float64

Growth 5yr High Est 495 non-null float64

Growth 5yr Low Est 494 non-null float64

Growth Analysts 495 non-null float64

Historical Volatility 499 non-null float64

Institutions Holding Shares 493 non-null float64

Interest Coverage (MRQ) 388 non-null float64

Net Profit Margin (TTM) 487 non-null float64

Operating Profit Margin (TTM) 487 non-null float64

Quick Ratio (MRQ) 322 non-null float64

Return On Assets (TTM) 477 non-null float64

Return On Equity (TTM) 451 non-null float64

Return On Investment (TTM) 435 non-null float64

Revenue Growth (MRQ) 485 non-null float64

Revenue Growth (TTM) 487 non-null float64

Revenue Growth 5yr 488 non-null float64

Revenue Per Employee (TTM) 457 non-null float64

Shares Outstanding 493 non-null float64

Short Int Current Month 493 non-null float64

Short Int Pct of Float 493 non-null float64

Short Int Prev Month 493 non-null float64

Short Interest 493 non-null float64

Total Debt/Total Capital (MRQ) 458 non-null float64

Volume 10-day Avg 498 non-null float64

Book (MRQ) 453 non-null float64

Cash Flow (TTM) 466 non-null float64

Earnings (TTM) 447 non-null float64

Sales (TTM) 488 non-null float64

dtypes: float64(52)

memory usage: 210.5+ KB

There we have it, a clean feature set with all continuous numeric variables. It should be noted at this point that multicollinearity is present among the features here, but since we will not be concerning ourselves with feature importances in this task of modeling current price, we do not need to address it, since the accuracy of the model will not be negatively impacted. We will, however, be dealing with multicollinearity in the next task of modeling returns, where feature importances will be examined.

Imputing Missing Data:

Scikit-learn offers a variety of effective methods of imputation between their SimpleImputer, KNNImputer, and IterativeImputer classes. The SimpleImputer can utilize multiple strategies to fill nans: using the mean, median, mode, or a constant value. The KNNImputer utilizes K Nearest Neighbor modeling to estimate the missing values using the other data columns as predictors. The IterativeImputer is experimental, and allows for any estimator to be passed into it, which it uses to estimate missing values in a round-robin fashion. It is highly recommended that the reader check out the documentation for these classes in the previous links.

The best choice between these options depends on the data, task, and algorithm at hand, which is why it is generally best practice to train a model instance with each imputation method and compare the results. Below, I will establish some helper functions which are modified versions of the code found in the scikit-learn documentation links above, which will help us compare these imputation methods in the context of our problem. Note that the IterativeImputer requires importing a special item called ‘enable_iterative_imputer’ in order to work. Let’s import what we need and make our functions, which will generate cross validation scores for each of our imputation methods, and give us results we can use to compare them.

Notice that scaling is built into this process. It is now helpful to combine all off the above functions into a wrapper function that will make testing and graphing the scores of all the imputation methods for various regressors and tasks easy without having to repeat any code.

Excellent, now we have a framework for testing imputations methods for any estimator that we choose, and returning convenient axes objects to view them. The notebooks in the repository go into a comparison of many different regressors for the task of modeling current price, but to save space here, we will focus on the regressor which demonstrated the best performance for this task: scikit-learn’s GradientBoostingRegressor. First, we instantiate an out-of-the-box regressor, and pass it into the wrapper function with our data to see how the imputation methods compare. Note that the compare_imputer_scores function takes in a list of estimators to be used with the IterativeImputer, which can have variable parameters, and we need to make this list to pass it into the function.

The top chart represents the scores from the SimpleImputer, KNNImputer, and out-of-the-box IterativeImputer. The bottom chart compares the performance of the IterativeImputer used with each of the estimators in the estimators list. We can see some solid R squared scores all around, but the best performance is happening with the KNNImputer. It is always relieving to see a less computationally intense imputer (ie not the IterativeImputer) with the highest score before performing a grid search, because the IterativeImputer takes much longer, and when fitting thousands of models this does make a noticeable difference.

Modeling:

Now that we have a selection for the best regressor/imputer combo, we can do a grid search to find the optimal hyperparameters for our model, and then move on with our investigation of the residuals.

Awesome, after a while we have our best tuned model. Let’s see the optimal parameters and the predictive accuracy they produced.

Best Model:

r-squared: 0.8738820298532683 {'imputer__n_neighbors': 5,

'regressor__learning_rate': 0.1,

'regressor__max_depth': 2,

'regressor__n_estimators': 1000,

'regressor__subsample': 0.7}

Nice, we can see that the model has a solid R squared score of .87, using 1000 weak learners with a max depth of only 2 levels per tree, subsampling 70% of the data with a learning rate of 0.1.

Now that we have an asset pricing model trained with our data, we can see how the actual current prices deviate from the model’s predictions in the residuals, then look to see if these residuals are correlated with the returns since the date of the scrape. As a reminder, the hope is that the higher above the model estimate an asset price is, the more it can be expected to move downwards, or the lower below the model estimate an asset price is, the more it would be expected to move upward toward the estimated value over time. Since residuals are calculated as actual minus predicted, the residuals should be negatively correlated with the returns, if our hypothesis is correct. Let’s first generate the residuals, and take a look at them.

We can see that the residuals have a thin, fairly symmetrical distribution around zero, with some big outliers. Let’s see if we can see any linear relationship between them. I will be removing outliers past three standard deviations and fitting a linear regression model to do this. To see this coded out, refer to the repository, it is just a simple linear regression of the log returns using the residuals as the independent variable.

This linear regression, while showing a slight negative slope, offers an R squared of .004, and a p-value of .186 for the coefficient relating the residuals to the returns, so this is not indicating a powerful linear relationship. Thus, the hypothesis that the residuals of our asset pricing model would be correlated to the returns over the six month period since the scrape is not supported. Disappointing though this may be, this is the nature of science. Investigating this on longer time periods would likely be appropriate, since this hypothesis was derived from a value-investing perspective, and value investors typically plan to hold assets for much longer time periods, generally over a year at the very least. Incidentally, for shorter time periods such as the one used in this study, one could argue that overvalued or undervalued securities would be experiencing price momentum, attracting traders to trade with the current trend, thereby feeding it, and thus these securities may be likely to move more in the same direction in the short term before the market corrects itself toward their underlying value. Additionally, the residuals used above have a very tight distribution, because the regression was trained using all of the samples for which residuals were calculated. It may be worthwhile to repeat this process on a holdout set and compare the results.

Modeling Returns

Now we move on to our second task: modeling the returns over the six month period since the date of the scrape. This can be done a number of ways, here we will do three. The first will be to try and regress the continuous values of the returns since the scrape using the data as predictors, the second will be to create a binary classifier to predict gainers/losers (returns above or less than or equal to zero), and the third will be to classify stocks which over or underperformed the market (returns above or less than or equal to the average of returns of the index). The reason for the two different classification tasks, as mentioned above, is that a bull (rising) market will cause all stocks within an index to move upward on average, as similarly a bear (falling) market will cause them all to move downward on average. By subtracting the mean of returns from the returns for this particular time period, the model may be better generalized to application on a different time period. Unfortunately, we do not have data to test another time period, so we will not be able to test that theory, nor will be be able to evaluate the performance of a model trained on one time period on predicting the returns of another. We will be able to test the model on holdout sets of securities, albeit in the same time period as the training. Despite this unfortunate detail, a model using our features which can successfully predict winners and losers over any time period is an indication that the features have predictive power, and that further study on different time periods in the future is merited. We will also be able to observe the feature importances of the models, which may provide insight into which of the features contributes the most in predicting returns.

Let’s keep our target variables in a convenient data frame:

Feature Selection:

Since we are not predicting the prices of securities this time, having features which contain information about the price is no longer an issue, which means we can include features that we left out last time. However, since we want to get an accurate look at feature importances, we are going to need to deal with multicollinearity among the features to make that possible. Sometimes a model can get more accuracy by leaving multicollinearity alone, so it is generally best to train a separate model each way and compare them, using the most accurate for predictions, and using the one trained with multicollinearity managed for analysis. In this case, it was found that the classification models were even more accurate after multicollinearity was managed, but that the (already disappointing) performance of the regression models suffered as a result of removing multicollinearity. In order to save space here, I will briefly touch on the results of regression, which were not very impressive, and then move on to removing multicollinearity and performing classification.

First thing to do for any of the upcoming tasks is to drop features we will definitely not be using, and see what next steps should be taken to clean the data. We are starting again from the totally raw data frame.

<class 'pandas.core.frame.DataFrame'>

Index: 501 entries, A to ZTS

Data columns (total 83 columns):

% Held by Institutions 495 non-null float64

52-Wk Range 501 non-null object

5yr Avg Return 501 non-null float64

5yr High 501 non-null float64

5yr Low 501 non-null float64

Annual Dividend % 394 non-null float64

Ask 494 non-null float64

Ask Size 492 non-null float64

B/A Ratio 492 non-null float64

B/A Size 501 non-null object

Beta 488 non-null float64

Bid 499 non-null float64

Bid Size 492 non-null float64

Change in Debt/Total Capital Quarter over Qua...473 non-null float64

Closing Price 500 non-null float64

Day Change % 501 non-null float64

Day High 501 non-null float64

Day Low 501 non-null float64

Days to Cover 495 non-null float64

Dividend Change % 405 non-null float64

Dividend Growth 5yr 354 non-null float64

Dividend Growth Rate, 3 Years 392 non-null float64

EPS (TTM, GAAP) 490 non-null float64

EPS Growth (MRQ) 487 non-null float64

EPS Growth (TTM) 489 non-null float64

EPS Growth 5yr 438 non-null float64

FCF Growth 5yr 485 non-null float64

Float 495 non-null float64

Gross Profit Margin (TTM) 422 non-null float64

Growth 1yr Consensus Est 475 non-null float64

Growth 1yr High Est 475 non-null float64

Growth 1yr Low Est 475 non-null float64

Growth 2yr Consensus Est 497 non-null float64

Growth 2yr High Est 497 non-null float64

Growth 2yr Low Est 497 non-null float64

Growth 3yr Historic 497 non-null float64

Growth 5yr Actual/Est 497 non-null float64

Growth 5yr Consensus Est 497 non-null float64

Growth 5yr High Est 497 non-null float64

Growth 5yr Low Est 497 non-null float64

Growth Analysts 497 non-null float64

Historical Volatility 501 non-null float64

Institutions Holding Shares 495 non-null float64

Interest Coverage (MRQ) 389 non-null float64

Last (size) 501 non-null float64

Market Cap 501 non-null object

Market Edge Opinion: 485 non-null object

Net Profit Margin (TTM) 489 non-null float64

Operating Profit Margin (TTM) 489 non-null float64

P/E Ratio (TTM, GAAP) 451 non-null object

PEG Ratio (TTM, GAAP) 329 non-null float64

Prev Close 501 non-null float64

Price/Book (MRQ) 455 non-null float64

Price/Cash Flow (TTM) 468 non-null float64

Price/Earnings (TTM) 448 non-null float64

Price/Earnings (TTM, GAAP) 448 non-null float64

Price/Sales (TTM) 490 non-null float64

Quick Ratio (MRQ) 323 non-null float64

Return On Assets (TTM) 479 non-null float64

Return On Equity (TTM) 453 non-null float64

Return On Investment (TTM) 437 non-null float64

Revenue Growth (MRQ) 487 non-null float64

Revenue Growth (TTM) 489 non-null float64

Revenue Growth 5yr 490 non-null float64

Revenue Per Employee (TTM) 459 non-null float64

Shares Outstanding 495 non-null float64

Short Int Current Month 495 non-null float64

Short Int Pct of Float 495 non-null float64

Short Int Prev Month 495 non-null float64

Short Interest 495 non-null float64

Today's Open 501 non-null float64

Total Debt/Total Capital (MRQ) 460 non-null float64

Volume 500 non-null float64

Volume 10-day Avg 500 non-null float64

Volume Past Day 501 non-null object

cfra 479 non-null float64

creditSuisse 337 non-null object

ford 493 non-null float64

marketEdge 484 non-null float64

marketEdge opinion 484 non-null object

newConstructs 494 non-null float64

researchTeam 495 non-null object

theStreet 496 non-null object

dtypes: float64(73), object(10)

memory usage: 348.8+ KB

B/A Size is encoded as strings, but we have numeric features for Bid Size and Ask Size, so we do not need the B/A Size feature at all, and it can be dropped. There is also the duplicate P/E Ratio (TTM, GAAP) feature in string format that we found earlier that can be dropped again. There are two columns, Market Edge Opinion and marketEdge opinion which are the same, and since there is a numeric counterpart to this analyst rating in the marketEdge column, we can drop both of the market edge opinion columns. The Volume Past Day feature is a categorical variable in string format that tells us whether the previous trading day had light, below average, average, above average, or heavy volume compared to a typical trading day for the security. This feature could be one hot encoded, but since there is a Volume 10-day Avg feature which is numeric and more descriptive of recent volume trend, we will just drop Volume Past Day. That leaves us with five features with an object dtype to manage before we begin the imputation step: 52-Wk Range, Market Cap, creditSuisse, researchTeam, and theStreet. Let’s drop the unneeded columns, then take a closer look at these 5 columns.

It looks as though we can modify the strings in 52-Wk Range to give us two numeric columns: 52-Wk Low and 52-Wk High. Then, we can move on to converting the Market Cap column, but we will need to know all of the unique letters that it contains, which represent some order of magnitude for the number they are associated with. Let’s treat 52-Wk Range, and check our result.

Alright, this has worked. Now we need to determine what letter suffixes the numbers in the Market Cap column have, so we can build a function to convert this column into a numeric dtype. We can also drop the old 52-Wk Range column.

array(['B', 'T', 'M'], dtype=object)

Here we can see that the column has B, T, and M suffixes expressing amounts in billions, trillions, and millions, respectively. Knowing this, we can make a function to treat this column.

Excellent, another success story. The old column can be dropped. Now we need to check if there are any companies in our feature set that aren’t present in the log_returns. Recalling from earlier, we should expect to find AGN and ETFC in this list.

['AGN', 'ETFC']

Just as suspected. We need to drop these from the index of our feature set. Since we also know from earlier that we have infs that need to be re-encoded as nans, we will do that in this step as well.

Now we have just one more chore before we can move on to dealing with multicollinearity: we need to one hot encode the categorical analyst ratings that were encoded as strings. Since we have fixed all of the other columns, we can simply call the pandas get_dummies method, and make sure that we set drop_first to True, in order to avoid the dummy variable trap.

<class 'pandas.core.frame.DataFrame'>

Index: 499 entries, A to ZTS

Data columns (total 82 columns):

% Held by Institutions 493 non-null float64

5yr Avg Return 499 non-null float64

5yr High 499 non-null float64

5yr Low 499 non-null float64

Annual Dividend % 392 non-null float64

Ask 492 non-null float64

Ask Size 490 non-null float64

B/A Ratio 490 non-null float64

Beta 486 non-null float64

Bid 497 non-null float64

Bid Size 490 non-null float64

Change in Debt/Total Capital Quarter over Qua...471 non-null float64

Closing Price 498 non-null float64

Day Change % 499 non-null float64

Day High 499 non-null float64

Day Low 499 non-null float64

Days to Cover 493 non-null float64

Dividend Change % 403 non-null float64

Dividend Growth 5yr 354 non-null float64

Dividend Growth Rate, 3 Years 392 non-null float64

EPS (TTM, GAAP) 488 non-null float64

EPS Growth (MRQ) 485 non-null float64

EPS Growth (TTM) 487 non-null float64

EPS Growth 5yr 437 non-null float64

FCF Growth 5yr 483 non-null float64

Float 493 non-null float64

Gross Profit Margin (TTM) 420 non-null float64

Growth 1yr Consensus Est 473 non-null float64

Growth 1yr High Est 473 non-null float64

Growth 1yr Low Est 473 non-null float64

Growth 2yr Consensus Est 495 non-null float64

Growth 2yr High Est 495 non-null float64

Growth 2yr Low Est 495 non-null float64

Growth 3yr Historic 495 non-null float64

Growth 5yr Actual/Est 495 non-null float64

Growth 5yr Consensus Est 495 non-null float64

Growth 5yr High Est 495 non-null float64

Growth 5yr Low Est 495 non-null float64

Growth Analysts 495 non-null float64

Historical Volatility 499 non-null float64

Institutions Holding Shares 493 non-null float64

Interest Coverage (MRQ) 388 non-null float64

Last (size) 499 non-null float64

Net Profit Margin (TTM) 487 non-null float64

Operating Profit Margin (TTM) 487 non-null float64

PEG Ratio (TTM, GAAP) 329 non-null float64

Prev Close 499 non-null float64

Price/Book (MRQ) 453 non-null float64

Price/Cash Flow (TTM) 466 non-null float64

Price/Earnings (TTM) 447 non-null float64

Price/Earnings (TTM, GAAP) 447 non-null float64

Price/Sales (TTM) 488 non-null float64

Quick Ratio (MRQ) 322 non-null float64

Return On Assets (TTM) 477 non-null float64

Return On Equity (TTM) 451 non-null float64

Return On Investment (TTM) 435 non-null float64

Revenue Growth (MRQ) 485 non-null float64

Revenue Growth (TTM) 487 non-null float64

Revenue Growth 5yr 488 non-null float64

Revenue Per Employee (TTM) 457 non-null float64

Shares Outstanding 493 non-null float64

Short Int Current Month 493 non-null float64

Short Int Pct of Float 493 non-null float64

Short Int Prev Month 493 non-null float64

Short Interest 493 non-null float64

Today's Open 499 non-null float64

Total Debt/Total Capital (MRQ) 458 non-null float64

Volume 498 non-null float64

Volume 10-day Avg 498 non-null float64

cfra 478 non-null float64

ford 491 non-null float64

marketEdge 482 non-null float64

newConstructs 492 non-null float64

52-Wk Low 499 non-null float64

52-Wk High 499 non-null float64

marketCap 499 non-null float64

creditSuisse_outperform 499 non-null uint8

creditSuisse_underperform 499 non-null uint8

researchTeam_hold 499 non-null uint8

researchTeam_reduce 499 non-null uint8

theStreet_hold 499 non-null uint8

theStreet_sell 499 non-null uint8

dtypes: float64(76), uint8(6)

memory usage: 303.1+ KB

Regression

Excellent, we have a clean data frame to perform regression with. The study found that the variety of regression algorithms from scikit-learn and xgboost that were tested had consistently lackluster performance in the task of using these data to regress the returns since the date of the scrape. The best performance was provided by the RandomForestRegressor from scikit-learn, with an average r-squared score of just below .21 after either zero or mean imputation, with a wide variance of scores. Below we can see these results.

These scores are not particularly impressive, but they are high enough to indicate that the model is effective on some level. R-squared tells us the proportion of the variance of the target which is explained by the model, and even though a large proportion of this variance is not being explained, a significant portion of it is, meaning there is some predictive power being provided by our features. Although the results from regression are somewhat disappointing, the upcoming classification tasks were quite fruitful. As mentioned above, the performance of classification was actually aided by the removal of multicollinearity, so from here I will demonstrate that removal first, then move on to the classification model building.

Removing Multicollinearity:

Since we want to get meaningful insights from the feature importances of the models that we train, we need to now deal with multicollinearity among the features. To start this process, we need to generate a heatmap to visualize the correlation matrix of all of the features. We can do this with a combination of the pandas .corr method and seaborn’s heatmap.

The first thing to notice is that Annual Dividend % has a negative correlation with many price based features, especially with 5yr Avg Return. We can look at a scatter plot of this relationship to get a more detailed understanding.

We can see a strong negative correlation here. It can also be noticed that some of these annual dividends are quite large, all the way up to 17.5% of the price per share. Whenever a dividend is paid out to shareholders, it is subtracted from the price of the share on the date of payment. Thus, if an annual dividend is 5%, then the stock will need to have gained 5% that year in order to break even after the all dividends are paid out. This explains the negative correlation seen above: the higher the dividend payouts, the more share price is reduced as a result, and the less the gains in price per share will be for the year. We will remove the Annual Dividend % feature.

Next, we can see that all of the price related features are highly correlated, which is surprising to no one. Although features like 5yr High, 5yr Low, Ask, Bid, Closing Price, Day High, Day Low, etc. are all expressing distinct things about price, they are so closely correlated that there will be no way to determine their individual effects on the model, and since we will be looking at feature importances, we need to cull our price-related features. Additionally, the growth estimates have some strong multicollinearity, so we will drop some of these as well.

Another strong correlation exists between Float and Shares Outstanding. These are very similar features. Float Represents the number of shares available for public trade, and Shares Outstanding is the total number of shares that a company has outstanding. These are bound to be highly related, but we can see just how related with another scatter plot.

Above we can see what verifies our understanding of these features: a company has a certain number of shares outstanding, but some of those shares may not be available for the public to trade. Thus we see that Float is never above Shares Outstanding, and the relationship is mostly perfectly linear along the line of slope 1 (identity function). We can drop the Float feature in favor of Shares Outstanding.

Return on Assets and Return on Investment are very strongly correlated. As explained in this article, cross-industry comparison of Return on Assets may not be meaningful, and it is better to use Return on Investment in these cases, including the one we find ourselves in. Thus, we will drop Return on Assets.

Net Profit Margin is Operating Profit Margin minus taxes and interest, and therefore the two are highly correlated. Since the prior contains more information, and they have the same number of missing values, we can drop Operating Profit Margin.

Volume is highly correlated to Volume 10-day Avg, and since the latter contains more information, we can drop the prior. Historical Volatility has strong correlation with Beta, which is a similar metric expressing relative volatility to the market. We will drop Historical Volatility in favor of Beta.

Price/Earnings (TTM) and Price/Earnings (TTM, GAAP) are highly correlated. As mentioned above, the non-GAAP metric is considered more useful in quantitative financial analysis, since it leaves out large non-recurrent costs that may have appeared on recent financial statements, so we will drop the feature calculated with GAAP earnings.

Dividend Growth Rate, 3 Years, and Dividend Growth 5yr are highly correlated, since they contain a lot of similar historical information. The 3 year growth rate should be informative enough to predict returns over a six month period, so we will drop Dividend Growth 5yr.

We can also seem some collinearity occurring among the Short Interest related features. Let’s view these features together to see what they look like.

We can see that the Short Int Pct of Float and Short Interest columns are the same, but with more resolution in the prior, so we can drop the latter. Short Int Current Month and Short Int Prev Month are highly correlated with one another, and it stands to reason that the Current Month feature is more valuable looking forward, so we can drop the Short Int Prev Month.

This is looking much better. Although there is still some correlation present between variables, we have dealt with most of the redundant features and multicollinearity. This is a good place to move on to our imputation and modeling phases.

Classification — Class 1: Gainers/Losers

Now we move on to our classification tasks, both of which have binary categorical targets. The first class will be gainers/losers, the second will be over/under performers relative to the market

Imputing Missing Data (Class 1: Gainers/Losers):

This study showed that the XGBClassifier from the xgboost package had superior predictive performance on this target, so we will investigate it here. For full performative comparison of various classifiers, see the notebooks.

We can see that the SimpleImputer is giving us decent performance, though with a wide variance. The IterativeImputer with the DecisionTreeRegressor is doing the best, and with the KNeighborsRegressor it is doing nicely with a tighter variance, which sometimes can be preferable for robustness. One thing to always keep in mind when preparing to do a grid search using these imputers is that the IterativeImputer makes the process take a lot longer, for reasons implicit in the name. In this study, both were compared, and it turned out that the SimpleImputer was both faster and produced a better model, so we will demonstrate with this imputation method here.

Modeling (Class 1: Gainers/Losers):

We can now construct a pipeline and a grid search to build an optimal model for this task, which we can then analyze.

After a while, we have our completed grid search. Let’s look at the best cross validation score and parameters that it produced.

Best CV roc_auc score: 0.7014277550220106 {'clf__colsample_bylevel': 1,

'clf__colsample_bytree': 1,

'clf__learning_rate': 0.001,

'clf__max_depth': 5,

'clf__n_estimators': 1000,

'clf__reg_lambda': 0.5,

'clf__subsample': 1}

The best auc score from the cross validation was a .701, which is neither awesome nor terrible. Let’s see how the predicted probabilities for the test set relate to the log returns.

The r-squared for this simple linear regression between the mode’s predicted probabilities and log returns is very low at 0.059, but the coefficient for the probabilities is significant with a p-value of 0.015. We can see the imbalance of the classes in the scatter plot above, since as the market was uptrending during the period of study, the majority of stocks in the index had gains, and are therefore members of the gainer class. Let’s look now at what the probability of selecting a gainer at random would have been by dividing the number of securities in the gainer class by the total number of securities in the index.

0.7294589178356713

Here we can see that an investor would have had a 73% chance of randomly picking a gainer during the time period of study due to the uptrending market. We can now start to see why it does not make much sense to model gainers/losers if we intend to gain insights or make a predictive model which will be useful during different time periods, because the proportion of gainers/losers (class membership) will change depending on what the behavior of the market is, and this model may be attributing gains to the predictive features that had less to do with those features and more to do with the movement of the market. In a moment, we will remove the average return of the market to adjust for this, and model over/under performers relative to the market, but first let’s take a quick look at the accuracy and roc curve for this model.

Accuracy Score (test set): 0.81

We can see that the model predicted the correct class 81% of the time, which sounds impressive, but knowing that the probability of randomly selecting a gainer was 73%, it isn’t quite so, and we know that this accuracy could vary widely in another time period where the market trended differently. The auc score on the test set is 0.68, which is not super impressive. Let’s move on to modeling our last target, now that we know that it will give us the most robust insight into the performance of stocks relative to each other.

Classification — Class 2: Over/Under Performers

As we can see from the modeling and analysis above, there is a significant disadvantage to modeling gainers/losers over a given time period, because the category that a stock falls into is highly dependent on the behavior of the overall market during that time period. This can lead the model to be biased in how it evaluates the contributions of features to stock performance because some of that performance can be attributed to the market’s behavior, and not to the underlying features of the stock itself. In order to adjust for this, we can subtract the average return of the market over the time period from the return of each security, thereby excluding the influence of the overall market movement from the target, and focusing solely on the differences in performance among the securities in the index. By modeling relative performance, an investor can construct a hedged portfolio using a long/short equity strategy based off of the model predictions, which should then be robust to varying market conditions.

Imputing Missing Data (Class 2: Over/Under Performers):

The XGBRFClassifier from xgboost was found to be the most effective classifier for this task. Let’s take a look at how the imputation methods get along with this classifier.

We can see that the best performance is being achieved using the SimpleImputer, and although these results are not particularly exciting, some hyperparameter tuning of the classifier will help. Let’s move on to a grid search.

Modeling (Class 2: Over/Under Performers):

After running the cell above, we have a fitted grid search object that will have our optimal estimator fit to the training set. Let’s inspect some of the features of this estimator.

Best CV training score: 0.690298076923077 {'clf__colsample_bylevel': 0.8,

'clf__colsample_bynode': 1.0,

'clf__colsample_bytree': 0.6,

'clf__learning_rate': 0.001,

'clf__max_depth': 7,

'clf__n_estimators': 100,

'clf__reg_lambda': 0.75,

'clf__subsample': 0.8}

The best cross-validation score in the grid search on the training set was an roc_auc of 0.69, which is neither great nor terrible. We can now use the best estimator from the grid search to generate predicted probabilities for the holdout set, then see if there is a linear relationship between these probabilities and the actual log returns of the securities.

We can see that the numeric range of these predicted probabilities is extremely slim, but that the classifier is still effective. The r-squared for this simple linear regression of the log returns using the predicted probabilities is very low at .088, but the coefficient for the probability is significant with a p-value of .003, meaning that there is a linear relationship between the log returns and the predicted probabilities generated by the model. What this means for us is that rather than simply choosing securities above/below a threshold probability, one could create a more conservative portfolio by creating a window around a threshold probability and only selecting stocks to long or short that are outside of that window of probabilities. For simplicity, here we will look at just longing or shorting on either side of our chosen threshold, but first we need to figure out what the optimal threshold for this model is by looking at the roc curve.

We can see that the roc_auc score for the holdout set is .70, which isn’t too bad. We can also see that there appears to be a sweet spot right around a True Positive Rate of just under 0.8 and a False Positive Rate around 0.4. We can create a data frame of these rates with their thresholds using the roc_curve function, and use it to determine what the best threshold to use for our model is.

Great, now we can slice into this data frame to find the probability threshold that corresponds with the sweet spot we saw on the roc curve above.

And there we have it, the sweet spot between tpr and fpr appears to be at index 21. We can get the full resolution of the threshold by indexing the column, and use it to manually generate predictions using this threshold. It will be useful to compare the predictive accuracy using our chosen threshold with the accuracy attained by using a standard probability threshold of 0.5.

Accuracy Score with Standard Threshold: 0.62

Accuracy Score with Selected Threshold: 0.69

As we can see, fine tuning our classification threshold has given us another 7% on our overall predictive accuracy, bringing us up to 69%. This is considerably better than a random guess, which for this target had a 53% chance of randomly picking an over performer. We can see what the probability of correctly picking an overperforming stock by dividing the number of overperforming stocks by the total number of stocks in the index.

0.5270541082164328

We can see that the model is indeed giving us a predictive advantage over randomly selecting an overperforming stock. Now that we have our predictions, we can set about constructing a portfolio using a long/short equity strategy, and see how the returns of this portfolio would compare to buying and holding the market index over the same time period.

Before we move on to portfolio construction, however, let us take a look at the information on feature importances provided to us by the model. There are two ways to look at this: the first is to look at the feature importances from the training process, and the second is to look at permutation importances on the holdout set. Comparing the two can provide interesting insights into how the model behaves with the two sets of data. First, let’s look at the feature importances of the training process.

Above we can see the relative importances of the features in the model fit, which it uses to make predictions. Interestingly, the leading feature is the analyst rating from Ford Equity Research, perhaps they would be pleased to know. Beneath this we see the list of our fundamental features, all of which are shown to be playing an important role in estimation, apart from some of the dummy variables. This is very informative, but it only gives us a view into how the model built itself to best fit the training set. In order to see how these features contribute to the predictive accuracy of the holdout set, we need to use permutation importances. Permutation importances are generated by iteratively shuffling each feature (breaking the relationship to the target) and measuring the loss of predictive performance caused by doing so. If shuffling a certain feature leads to a major loss in performance, then it can be said to be an important feature in prediction. Let’s look at this below.

Above we can see the distributions of how much the model performance was affected by 10 shuffles of each feature. We can notice that the order of the importances has been shuffled around somewhat, although there are some consistencies. This is due to the fact that what is important in predicting the target for the securities of the training set may be different than what is important in predicting the target for securities found in the testing set. Features which are found toward the top of both lists can be assumed to be important overall, such as Growth 1yr High Est, Revenue Per Employee (TTM), ford, marketCap, and Beta. In the case of some of the permutation importances toward the bottom of the chart, random shuffling of the features actually improved the predictive accuracy on the holdout set!

Although we know that certain features are important to the model, their relationships to the target variable may not be totally clear to a human being, as we can see below by plotting the log returns over the (apparently highly valuable) ford ratings.

Ford is an important feature with an unclear relationship to the target.

Constructing a Portfolio

To demonstrate how our model can be useful to an investor, we will construct a basic long/short equity portfolio using the model’s predictions of over/under performers. The benefit of the long/short equity strategy is that the investor is hedged against the market by taking an equal amount of long and short positions. This way, the portfolio is not affected by overall upward or downward movement of the market, and instead is only affected by how well the investor has predicted the relative performance of securities within it. Our model from above is correctly identifying over and under performers with an accuracy of 69%, which isn’t perfect, but when this predictive power is combined with a long/short equity strategy, it can lead to a consistently profitable trading strategy by both diversifying the risk of the portfolio by taking many positions and also hedging against the market. If the market moves up, the investor will gain from their long positions while losing on their short positions, and conversely if the market moves down, they will gain on their short positions while losing on their long positions. By having predicted with better than random accuracy which securities were set to over or underperform the market, the gains from the winning side of the portfolio should average out to be bigger than the losses from the losing side, no matter which way the market moves. This is why it was so important to subtract the average return of the market from our target variable, because it led to developing a model which focused only on relative performances, ignoring the impact of the market.

We can simulate how such a portfolio would have performed using the securities of the test set by using the class predictions for each security to alter the sign of their respective log returns, and averaging all of these adjusted returns together. This would be the equivalent of the investor making equal dollar investments into each security in the test set, going short in any stock predicted by the model to underperform, and going long in any stock predicted to overperform. Since being short in a stock turns losses into gains and vice-versa, we will take all securities with predicted class 0 and reverse the signs.

Now we are ready to determine what the returns of a portfolio which took equal dollar value positions in each stock of the holdout set, going short in those predicted to underperform, and long in those predicted to overperform. We can do this as follows:

Portfolio Return (test set): 0.10186008839757786

Market Return (test set): 0.14362567579115593

We can see that the portfolio has underperformed the market considerably, but this is the nature of a hedging strategy. This is due to the fact that the market itself did very well over the time period studied, so the short side of the portfolio had losses which subtracted from the potential gains of buying and holding the market index. The sacrifice in comparative performance to such a booming market is made in order to protect the portfolio in the event that the market behaves poorly. To see this in action, let’s repeat this comparison, but with a simulated bear (falling) market, which we can create by subtracting the average market return twice from each security, thereby making the overall movement of the market the opposite of what actually happened. We can then compare how the portfolio would compare to buying and holding the market index under these averse circumstances.

Now we can repeat the process above to see how our portfolio performance would have compared to buying and holding this simulated bear market.

Portfolio Return (test set): 0.050753389653326986

Market Return (test set): -0.06931890230988941

Here we can see where the hedged portfolio design truly shines: where someone who had bought and held the market would have suffered losses, the hedged portfolio actually saw gains due to the short positions, thus demonstrating how the long/short equity portfolio strategy reduces market risk, and leads to more consistent profitability.

Visualizing the Portfolio Performance:

We can get a deeper understanding of what is happening by visualizing the performance of the portfolio vs the market. In this case, we are actually looking at the portion of the market which is within the holdout set. We can make a quick visual to see how the overall market differs from our test set, after we create a data frame representing the daily returns of all of the securities in the index by calling pandas .diff() method on the log_close data frame, which will calculate the daily change in log price, also known as log returns.

We can use the cumulative sum to represent the cumulative returns over time. Beneath we will look at how the test set compares to the entire index.

We can see above that the portion of the S&P 500 that is in the test set noticably outperformed the index, but that the overall shape is almost exactly the same. Recall above that the average for the holdout set was .14, while the index had an overall return of .11. Now, we can use the predictions of under and over performers given to us by the model to split the securities in the test set into long and short positions. The portfolio will go long in all companies predicted to outperform, and short in all that were predicted to underperform. To construct this portfolio, we will split the log_returns_full frame into the long side and short side, reverse the sign of the short side, and recombine the two. Then, we will look at a plot comparing the returns of a buy & hold strategy of the test set vs the long/short portfolio we have made.

Now we can really see much more detail of what is going on here. We see that the final return for the test set is at .14, and the total return for the portfolio is at .10, just as we calculated earlier, but now we can see the history leading up to that point. Notice that the portfolio is much less volatile than the buy & hold strategy, with the peaks and troughs much more subtle. This is due to the hedging. To get an even better idea of what is happening, we can look at the returns of the long side of the portfolio and the short side of the portfolio separately.

Here we see the mechanics behind our strategy most clearly. The green line corresponds to the collective returns from our long positions, and the red line shows that of our short positions. Notice how they almost look like reflections of each other, but that the green line goes further up than the red line goes down. This was the entire goal of our strategy in action! The mirrored peaks and troughs are what combine to create the smoother line of the portfolio vs the buy & hold strategy, since the market movements experienced by all of the stocks have been mostly canceled out by this hedging. The fact that the red line doesn’t lose as much as the green line gains is thanks to the help of our model, which successfully helped us pick stocks that were more likely to over or underperform the market. Since the short positions lose the investor money when the prices increase, the fact that these positions as a whole underperformed the market means that the losses will be minimized, while the positions that the investor is long in were likely to maximize gains.

To explain why we are willing to sacrifice profits for our hedging strategy, let’s create the simulated bear market, and see how the same portfolio would have behaved in this environment. To create the bear market, we can subtract the mean of each day’s log returns from each company’s log return that day twice, effectively reversing the flow of the market over this time period. Let’s do this, and create a visualization to verify that it has worked as expected.

We can see that this math has mirrored the cumulative returns of the index, and given us a simulation of a market downturn. Let’s see how the same portfolio positions, as determined by the predictions of our model, would have performed vs the test set in these market conditions.

Here we can see the beauty of our strategy. In these averse conditions, the profits generated from the short positions have outweighed the losses from the long positions, so that the portfolio has actually made gains while the buy & hold investor would have had losses, and without all of the dramatic swings to boot. Let’s look again at the two halves of the portfolio separately.

We can see that the red line representing the short positions is driving the profits, and that the predictive power of the model has again led to an overall win in these conditions by allocating our portfolio based on the securities most likely to over or underperform the market. Notice that the green line ends up just about breaking even, whereas the test set went down to -.06, meaning that our long positions have indeed overperformed the market. Since the short positions have collectively provided a cumulative log return of .16, they actually saw losses at -.16, meaning that they drastically underperformed the market during these hard times, according to our plan.