Machine Learning and Trading on Sentiment | by Steven Downey, CFA, CMT | Towards Data Science

What You Will Learn and Skills to Take Away
How to build multiple classification machine learning models into a composite ensemble model
Use time-series split and randomized cross-validation for hyperparameter tuning
Explore important classification metrics, such as F1 score and accuracy, in accessing the quality of the models
Portfolio performance and construction code that can be applied across multiple strategies factoring in real-world transaction costs
How to build a feature set using One Hot Encoding
Discover whether short term price movements can be predicted from basic machine learning models
In my previous article, I explored if it was possible to use a single sentiment indicator and a simple linear model to systematically outperform the US stock market. Unsurprisingly, I found that a simple system did not work by a long shot. But maybe a machine learning model that combines different sentiment signals could create a profitable strategy?
With the advent of social media, analysts have used social media analytics and natural language processing to distill emotional sentiment around specific companies, with firms like Ravenpack finding alpha in long high positive sentiment companies and short negative sentiment firms. They found a 50 bps to 100 bps return on a long/short top decile bottom decile portfolio over a 2–3 day period.
Shlaefer found that there was 1 to 12 months underreaction (stock price goes higher after good news and lower after bad news) revealing that it takes time for investors to process new information, but that over 3–5 years investors overreact and pay too much for a string of positive earnings and ignore the mean-reverting nature of company fundamentals.
Baker and Stein found that a period of high liquidity and small bid-ask spreads result in lower future returns. Since the irrational investors tend to make the market more liquid, measures of liquidity provide an indicator of the relative presence or absence of these investors, and hence of the level of prices relative to fundamentals.
Baker and Wurgler hypothesize that stocks most sensitive to investor sentiment will be those of companies that are younger, smaller, more volatile, unprofitable, non-dividend paying, distressed or with extreme growth potential, or having analogous characteristics. Whereas “bond-like” stocks will be less driven by sentiment. They found that sentiment can be a strong short term contrarian indicator with small-cap, speculative stocks that are young, highly volatile, near financial distress, and unprofitable. Specifically, within this group of companies, their monthly return was on average -.34% when sentiment was one standard deviation below their historical average and 1.18% when their sentiment is above by one standard deviation.
Borovoka and Dijkstra realized a 60% accuracy in their deep neural network model in forecasting the EURO STOXX 50 price movement based on high-frequency news sentiment around the individual index companies.
Tetlock found that media sentiment, as measure by the Wall Street Journal’s Abreast of the Market column, was a confirming indicator over a few days with regards to the overall stock market returns, and especially so with small stocks. Also, he found that unusually high or low market pessimism led to high trading volume.
None of the research investigated sector-specific effects driven by overall investor sentiment, so I was curious to explore it.
I utilized the 9 SPDR Sector ETFs (XLU, XLK, XLB, XLI, XLV, XLF, XLE, XLP, XLY) that had their inception in December 1998, providing over 21 years of daily OHLC and volume data from Yahoo Finance. Note that free data, such as Yahoo Finance, is not always the cleanest price data but I kept it as is so you can utilize the code with minimum cost in case you would like to add different ETF or stock tickers to test the model.
I also utilized data from Sharadar, the St. Louis Federal Reserve, Duke University, the Chicago Board of Options Exchange, the University of Michigan, Economic Policy Uncertainty, and
Combining multiple classification machine learning models from the scikit-learn python library into an ensemble classification, I hope that a diversified model will perform well out of sample compared to any individual model.
The dataset into a training, validation, and test set:
Training/Validation (In Sample) — Beginning of data until the end of 2013, with training comprising 80% and validation the remaining 20% of In Sample data
Test (Out of Sample) — January 2014 — May 2020
The questions I will try to answer:
Over what time frame are Sector returns most predictable based on sentiment? Since high OHLC and volume data, I will only be able to buy on the open of the next day and close out any position on n days later.
What models should be used to combine in a final ensemble model?
What hyperparameters are optimal for predicting out of sample data?
After combining the best models, will the machine learning portfolio outperform after transaction costs, since this would be a high turnover strategy?
Can the model outperform a passive buy and hold strategy?
Throughout the article, I will share parts of the code and not all due to readability, but you can access the data (except Sharadar due to licensing restrictions) and python files on GitHub for your personal use.
The following compose the feature set from which the algorithm will predict the desired value, namely whether the security had a positive or negative return over n days.
Market Turnover (% of total shares outstanding traded in a day)Advance/Decline Ratio (total number of shares increasing/shares decreasing)Simple Moving Average/Price iterated over rolling 2 to 22 trading daysAnnualized Standard Deviation iterated over rolling 2 to 22 trading daysVIXSahm Rule — which has been useful in predicting recessions in real-timeInvestment Grade — High Yield Corporate Bond SpreadPolitical Uncertainty IndexDuke CFO Optimism IndexPut/Call RatioAAII Investor Sentiment IndexUniversity of Michigan Consumer Sentiment IndexYale/Shiller Investor Sentiment Index
Market Turnover (% of total shares outstanding traded in a day)
Advance/Decline Ratio (total number of shares increasing/shares decreasing)
Simple Moving Average/Price iterated over rolling 2 to 22 trading days
Annualized Standard Deviation iterated over rolling 2 to 22 trading days
VIX
Sahm Rule — which has been useful in predicting recessions in real-time
Investment Grade — High Yield Corporate Bond Spread
Political Uncertainty Index
Duke CFO Optimism Index
Put/Call Ratio
AAII Investor Sentiment Index
University of Michigan Consumer Sentiment Index
Yale/Shiller Investor Sentiment Index
I was curious if the individual sector ETFs performed differently under the same Feature set, and thus I used One Hot Encoding to create columns for each ETF that had a 1 if used and 0 otherwise (see below picture).
The final feature and value pandas dataframe looks like the following that is then converted to a NumPy array.
Credit Spread  US Shiller Valuation Index Indiv (Lag)  \date                                                                2003-10-17           3.95                                   66.24   2003-10-17           3.95                                   66.24   2003-10-17           3.95                                   66.24   2003-10-17           3.95                                   66.24   2003-10-17           3.95                                   66.24US Shiller Valuation Index Inst (Lag)               ...date                                                ...2003-10-17                                  76.64   ...2003-10-17                                  76.64   ...2003-10-17                                  76.64   ...2003-10-17                                  76.64   ...2003-10-17                                  76.64   ...            XLK  XLP  XLU  XLV  XLY  date                                 2003-10-17  0.0  0.0  0.0  0.0  0.0  2003-10-17  0.0  1.0  0.0  0.0  0.0  2003-10-17  1.0  0.0  0.0  0.0  0.0  2003-10-17  0.0  0.0  0.0  0.0  1.0  2003-10-17  0.0  0.0  0.0  0.0  0.0
You can easily create a train and validation set, note the need to place shuffle=False to prevent a shuffling in the data that would leave to a model overfit since the financial data is not independent. What I like to do is keep my in sample data (train and validation) in one python file and keep the out of sample (test) data in a separate file to prevent any temptation to cheat and look at the future.
############Creating Training/Validation Set################# Labels are the values we want to predictlabels = np.array(features['value'])# Remove the labels from the feature and the datefeatures_array = features.drop(['value','index'], axis=1)# Saving feature names for later usefeature_list = list(features_array.columns)# Convert to numpy arrayfeatures_array = np.array(features_array)# Using Skicit-learn to split data into training and testing sets, but in this case it is a validation settest_ratio = .2 #variable for the percentage of train data to leave as validation# Split the data into training and testing sets, with simple splittrain_features, validation_features, train_labels, validation_labels = train_test_split(features_array, labels, test_size=test_ratio, random_state=42, shuffle=False)number_features = train_features.shape[1]
The accuracy test was conducted over a 1–30 day holding period using the following classifiers:
AdaBoost
Random Forest
Gradient Boosting
Gaussian Process
K Nearest Neighbors
MLP Neural Network
Support Vector Machine (SVC)
I choose these in particular as I was navigating step by step through the scikit-learn algorithm cheat-sheet in the classification bubble.
Each model was tested based on the F1 Score, accuracy score (percent correctly predicted), and the average of the two. I chose to eliminate the SVC and MLP Neural Network model since they had the lowest two combined scores of .516 and .521 respectively. Next, I chose to focus on the 3-day holding period due to the algorithms' ability to predict that time horizon well and that, assuming the model is good, a higher number of trading opportunities would serve it well, albeit constrained by transaction costs. The table below uses conditional formatting to visually pick put the high and low scores, with the standard deviation having a separate formula using red. What I want to see is a high average score and low standard deviation between all models, indicating to me that there is a signal to be extracted in the feature set and the algorithms are able to access it with reliability.
I will use the time series split and random cross-validation search function within the sci-kit learn library for hyperparameter tuning. The time series split, also known as the walk forward method, is designed for time series data as it is often not independent and identically distributed.
The challenge, however, as De Prado has raised in Advances in Machine Learning is that the cross-validation function equal weights all the scores even though some are trained on a small fraction of the overall data. To manage this problem, I decided to weigh the cross-validation scores based on the fraction of folds use in total (i.e. with a 5 fold cross-validation, the score of the last test would weigh 5 / (5+4+3+2)).
Another issue, De Prado raises is the need to purge the data near the edges of the train/validation split in the cross-validation approach to minimize data from the test and train sets leaking into each other. I tried to utilize his purgedKfoldCV function but was unable due to the fact there were multiple data points at each time point, due to the fact we are looking at 9 sector ETFs simultaneously.
The F1 score is the primary metric, composed of the harmonic mean of precision and recall, by which the accuracy of the model will be tested during cross-validation.
With the Randomized CV Search function, I will give a list of parameters to randomly choose from, and then isolate the best scoring combination to then use on the validation set. The algorithms that have the highest, and hopefully uncorrelated, prediction power will be combined in a majority voting system to create the final forecast system.
Below is an example of the random search parameters for the Random Forest algorithm I took from an article by Will Koehrsen and adapted to my purposes.
#### Create 5 K Fold Split Time Series ####number_of_splits = 5tscv = TimeSeriesSplit(n_splits=number_of_splits)#####Randomized CV for Random Forest using Timseries split #######   # Number of trees in random forestn_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]# Number of features to consider at every splitmax_features = ['auto', 'sqrt']# Maximum number of levels in treemax_depth = [int(x) for x in np.linspace(10, 110, num = 11)]max_depth.append(None)# Minimum number of samples required to split a nodemin_samples_split = [2, 5, 10]# Minimum number of samples required at each leaf nodemin_samples_leaf = [1, 2, 4]# Method of selecting samples for training each treebootstrap = [True, False]# Create the random gridrandom_grid = {'n_estimators': n_estimators,                'max_features': max_features,                'max_depth': max_depth,                'min_samples_split': min_samples_split,                'min_samples_leaf': min_samples_leaf,                'bootstrap': bootstrap}print(random_grid)# Use the random grid to search for best hyperparameters# First create the base model to tunerf = RandomForestClassifier(n_jobs=-1)# Random search of parameters, using time series split, # search across 100 different combinationsrf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 100, cv = tscv, verbose=2, random_state=42, \                                    scoring='f1')# Fit the random search modelsearch_rf = rf_random.fit(train_features, train_labels)search_rf.best_params_search_rf.cv_results_search_rf.best_score_search_rf.best_estimator_search_rf.scorer_
Some of the random searches took a while, such as the Random Forest at 77 minutes and the Gradient Boosting at 144 minutes on my Mac (16GB Ram and 2.9 GHz Dual-Core Intel Core i5). You could try using the Histogram-based Gradient Boosting Classification Tree instead which is designed to be faster on large feature sets.
With Random Forests and Gradient Boosting, you are able to see the relative importance in percentage terms for each feature.
General Observations:
On the far right-hand side, we see that the models found very little use in whether the ETF was a specific sector ETF
The volatility features that had the highest prediction over the 3-day return was, no surprise, the 2 and 3-day volatility
The overall market dynamics, such as the Advance/Decline and Market Turnover, were the most useful in predicting the future returns over 3 days
The five models combined did provide diversification as seen with the correlation matrix below, which makes sense as they utilize different mathematical algorithms to classify based on the feature set.
After combining the forecasts of all 5 models, I created a majority vote system. Next, I run the strategy over the training and validation data set and compare versus an equal weight investment strategy.
Set up the variables needed to implement the portfolio
#########  Portfolio Construction  ########Predictions_DF = features.iloc[:,(-len(tickers)-1):]#Using the same in sample dates here and for equal weight benchmarkf_date = datetime.datetime.strptime(In_Sample_Start, '%Y-%m-%d')l_date = datetime.datetime.strptime(In_Sample_End,  '%Y-%m-%d')delta = l_date - f_date#Choose the number of periods (days in range / the forecasted return days)period_delta = np.floor(delta.days/(Number_days_to_hold))period_delta = int(period_delta)first_period = In_Sample_Start #using f_datereturns_df_portfolio = pd.DataFrame()row_length = Predictions_DF.shape[0]Portfolio_Turnover = pd.DataFrame()
Create the for loop that runs through the data and executes the trades (the below code looked quite atrocious when I tried to embed).
As you can see, the in sample is highly overfit as to be expected vs. the validation set, so I parsed out the graphs to illustrate the difference.
The strategy had a higher risk-adjusted return indicated by the higher Sharpe Ratio, but not materially better otherwise.
Below are the statistics around the classification of the model along with the last row indicating what the results would be if I assume the markets will always be up, which is akin to being fully invested always.
The “Markets Always Go Up” buy and hold orientation was the general winner.
As seen in the confusion matrix below, the ensemble model did a good job at predicting when the future return would be positive (lower right) more so than predicting when there would be a negative return (top left).
Here is a chart of the actual performance of the strategy using the test data (January 2014- May 2020).
The model executed 1074 one-way trades, and with an assumption of $1 trade commission and 15 bps cost due to market impact and slippage, the real-world costs would have subtracted 17.9 % and $1,074 from the total gross performance of a $10,000,000 portfolio, resulting in an annualized return of 1.3%, much less than the unrealistic 6.1% annualized return.
The ensemble model did a good job at predicting up days, but after the high transaction costs it did not result in any superior risk-adjusted performance
The model did do better than random chance, as evidence by the accuracy of 53.4%, but the market was up 54.8% of the time, and thus if you had a simple thesis “the market will go up” then you would have a higher accuracy score.
The machine learning models were able to extract some signal from market internal features, like market turnover, but in the end, did not create a predictive model superior a naïve “always be invested” model.
Based on other relevant research mentioned earlier, I would expect a machine learning model to be able to extract a more meaningful signal from stock-specific social media analysis, small-cap and microcap stocks, and improve the overall accuracy with a deep neural network.
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.