Machine learning for stock prediction. A quantitative approach | by Gianluca Malato | Towards Data Science

Using machine learning for stock price predictions can be challenging and difficult. Modeling the dynamics of stock price can be hard and, in some cases, even impossible.
In this article, I’ll cover some techniques to predict stock price using machine learning. We’ll see some models in action, their performance and how to improve them. All the calculations will be made in Python using the famous Pandas, NumPy and Scikit-learn libraries.
For this example, I’ll use Microsoft’s daily stock price for the last 10 years downloaded in CSV format from Yahoo Finance. This dataset is made by the High, Low, Open and Adjusted close price, in addition to the volume.
First of all, let’s load our dataset. For the seek of simplicity, I’ll replace the “Close” price with the “Adjusted Close” price.
df = pd.read_csv("MSFT.csv",index_col=0)df['Close'] = df['Adj Close']df = df.drop("Adj Close",axis=1)
The price itself is not enough to produce useful predictions. For this example, I’ll add the following indicators that will be used as inputs for our models:
Moving averages (the average price of the last N days)
Bollinger bands (the moving average plus and minus a certain amount of standard deviations)
Donchian channels (the rolling maximum and minimum prices of the last N days)
Let’s add simple moving averages with periods 5, 10, 20, 50, 100 200.
for sma_period in [5,10,20,50,100,200]:    indicator_name = "SMA_%d" % (sma_period)    df[indicator_name] = df['Close'].rolling(sma_period).mean()
Now let’s add Bollinger bands with the following settings:
20 periods, 2 standard deviations
20 periods, 1 standard deviation
10 periods, 1 standard deviation
10 periods, 2 standard deviations
df['BollingerBand_Up_20_2'] = df['Close'].rolling(20).mean() + 2*df['Close'].rolling(20).std()df['BollingerBand_Down_20_2'] = df['Close'].rolling(20).mean() - 2*df['Close'].rolling(20).std()df['BollingerBand_Up_20_1'] = df['Close'].rolling(20).mean() + df['Close'].rolling(20).std()df['BollingerBand_Down_20_1'] = df['Close'].rolling(20).mean() - df['Close'].rolling(20).std()df['BollingerBand_Up_10_1'] = df['Close'].rolling(10).mean() + df['Close'].rolling(10).std()df['BollingerBand_Down_10_1'] = df['Close'].rolling(10).mean() - df['Close'].rolling(10).std()df['BollingerBand_Up_10_2'] = df['Close'].rolling(10).mean() + 2*df['Close'].rolling(10).std()df['BollingerBand_Down_10_2'] = df['Close'].rolling(10).mean() - 2*df['Close'].rolling(10).std()
Finally, let’s add Donchian channels with the same periods of the moving averages.
for channel_period in [5,10,20,50,100,200]:    up_name = "Donchian_Channel_Up_%d" % (channel_period)    down_name = "Donchian_Channel_Down_%d" % (channel_period)        df[up_name] = df['High'].rolling(channel_period).max()    df[down_name] = df['Low'].rolling(channel_period).min()
Now we want to use all these features in several time lags (i.e. the last day, the last 2 days and so on), so we need to shift them all by some lag. For this example, we’ll shift them from 1 to 10 days.
newdata = df['Close'].to_frame()for lag in [1,2,3,4,5,6,7,8,9,10]:    shift = lag    shifted = df.shift(shift)    shifted.columns = [str.format("%s_shifted_by_%d" % (column ,shift)) for column in shifted.columns]    newdata = pd.concat((newdata,shifted),axis=1)
taking a look at the dataset we’ll now notice features like SMA_5_shifted_by_3. It means “5 periods moving average calculated 3 days before”.
Finally, we’ll create the target variable, which is the number we want to predict. For this example, let’s say we want to predict the close price of 5 days in the future.
forward_lag = 5newdata['target'] = newdata['Close'].shift(-forward_lag)newdata = newdata.drop('Close',axis=1)newdata = newdata.dropna()
We have now a huge amount of data (2302 rows × 311 columns) as displayed in the following figure.
As usual, we’ll split our dataset into training and test sets. The training set will be used to train our models and tune the hyperparameters, while the test set will be used only for performance calculations.
The split is performed randomly.
X = newdata.drop("target",axis=1)Y = newdata['target']X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=42)
310 features are a great number. It’s very useful to take only the most important ones. For this example, we’ll sort the features by the absolute value of the Pearson’s correlation coefficient of each feature and the target. The idea is to take only the 50 most correlated features to the target.
correlations = np.abs(X_train.corrwith(y_train))features =  list(correlations.sort_values(ascending=False)[0:50].index)X_train = X_train[features]X_test = X_test[features]
Now we can start with the models.
Let’s start with the simplest regression model ever, which is the linear regression.
First, we’ll train the model.
lr = LinearRegression()lr.fit(X_train,y_train)
Then we calculate the predictions of the model.
y_pred = lr.predict(X_test)
The predictions can be compared with the real test set data to calculate the prediction error. For this example, I’ll use the mean absolute error.
mean_absolute_error(y_test,y_pred)
The error is 1.325
We can finally plot the real values of the test set and the predicted value to check if they lie in a 45° line passing through 0
plt.scatter(y_test,y_pred)plt.xlabel("Real")plt.ylabel("Predicted")plt.title("Linear regression")plt.show()
As you can see, real and predicted values are pretty similar.
The next model is a Random Forest Regressor from sklearn library.
Since Random Forest model has many hyperparameters, it’s useful to fine-tune them by a random search. The hyperparameters we are going to tune are the number of trees and the number of features.
We’ll perform a 20 steps random search with a 5-fold cross-validation, calculating the mean absolute error for each fold and then averaging among the 5 folds. The hyperparameters’ values that minimize the error (i.e. maximize the error with the minus sign) are the optimal values.
rf = RandomizedSearchCV(RandomForestRegressor(),param_distributions =  {                  'n_estimators':np.arange(10,500,5),                  'max_features':np.arange(1,10,1)               },                  cv=5, n_iter = 20,                  iid=False,random_state=0,refit=True,                  scoring="neg_mean_absolute_error")rf.fit(X_train,y_train)
The best values for the hyperparameters are:
The mean absolute error, calculated with the same code used for the linear regression, is 0.868
This is the scatter plot.
Random forest shows a lower error than linear regression.
We can now try with the Gradient Boosting tree regressor, which uses the boosting technique to improve model accuracy.
We still have two hyperparameters, which are the number of trees used in the boosting sequence and the maximum number of features.
gb = RandomizedSearchCV(GradientBoostingRegressor(),param_distributions =  {               'n_estimators':np.arange(10,500,5),               'max_features':np.arange(1,10,1)            },          cv=5, n_iter = 20,          iid=False,random_state=0,refit=True,          scoring="neg_mean_absolute_error")gb.fit(X_train,y_train)
The best values for the hyperparameters are:
The mean absolute error is 0.916 and here follows the scatter plot.
GBR is a nice model but has a slightly higher error than Random Forest.
KNN is a very powerful model in many application and we’ll use it in this example. We’ll change the nearest neighbors from 1 to 20 and the scoring formula from “uniform” (each neighbor has the same weight) to “distance” (each neighbor is weighted by the inverse of the distance between it and the input vector).
Since this hyperparameter space is quite small, we can easily use a grid search instead of a random search.
knn = GridSearchCV(KNeighborsRegressor(),param_grid =  {            'n_neighbors':np.arange(1,20,1),            'weights':['distance','uniform']            },          cv=5,           iid=False,refit=True,          scoring="neg_mean_absolute_error")knn.fit(X_train,y_train)
The best values are:
The mean absolute error is 0.830 and here’s the scatterplot.
So far, this is the model that produces the lowest error.
We’ll finish our model scouting with a single-layer Artificial Neural Network, with 5000 maximum epochs, an adaptive learning rate, and using the Stochastic Gradient Descent optimization algorithm.
Our hyperparameters are the number of neurons of the hidden layer (which we’ll span from 1 to 50) and their activation function (which will span among logistic and ReLU).
Before giving training data to an ANN we must scale them. We’ll use a MinMaxScaler to scale each feature into the 0–1 interval.
scaler = MinMaxScaler()scaler.fit(X_train)nnet = RandomizedSearchCV(MLPRegressor(max_iter=5000,learning_rate = 'adaptive',solver='sgd'),param_distributions =  {     'hidden_layer_sizes':[(x,) for x in np.arange(1,50,1)],     'activation':['logistic','relu']},cv=5, n_iter = 20,iid=False,random_state=0,refit=True,scoring="neg_mean_absolute_error")
These are the best values:
The mean absolute error is 1.407 and here’s the scatterplot.
So far, everything seems to tell us that KNN is the most powerful model. In reality, it isn’t.
Everything depends on how we perform the training/test split. If we split the records randomly, some records in the test set can be very close to other records in the training set. That’s why KNN and trees work very well.
What happens if we take the first 70% of data as training and the last 30% as test? The test records are now different from the training ones, so things can change dramatically.
We can perform this split via the following code, that replaces the previous train_test_split code
X = newdata.drop("target",axis=1)Y = newdata['target']train_size = int(X.shape[0]*0.7)X_train = X[0:train_size]y_train = Y[0:train_size]X_test = X[train_size:]y_test = Y[train_size:]
We can now repeat the training procedure for each model. Before calculating the errors, let’s take a look at the scatter plots.
It’s clear that KNN and trees now perform very badly. Linear regression and neural networks still resist.
Here are the mean absolute errors for each model:
Linear Regression: 2.379
Random Forest: 42.591
Gradient boosting regressor: 42.594
K Nearest Neighbors: 42.885
Neural Network: 2.366
Now the best model is the neural network.
What happened?
Well, remember that we are using the classical machine learning approach to model a phenomenon which is, actually, a time series. Records are not completely uncorrelated from each other so a random train/test split may be wrong. That’s why trading systems are usually trained taking the first part of the historical data as the training set and only the most recent data as the test set.
Moreover, KNN and trees can only predict outputs that are similar to those they have been trained with. Linear regression and neural networks are parametrical formulas, so they can predict any possible value with no limitations, once the parameters have been fixed by the training process.
The stock price can change from 0 to infinity and that’s why we need a mathematical model instead of a topological or geometrical model.
We all love neural networks, but let’s face a simple fact: in this example, ANN performs slightly better than linear regression. It’s not a dramatic performance improvement.
I think that machine learning works better when the models are simple. That’s why, for this purpose, I’d rather choose the linear regression.
Is there a way to boost its performance? Of course, yes.
Remember Random Forest and Gradient Boosting Regressor? Their predictive power relies on Bagging and Boosting techniques. The former creates multiple training dataset resampling the original records and choosing a subset of features randomly, while the latter creates a sequence of models that learns from the errors of the previous ones.
These techniques can be adapted to any model using the BaggingRegressor and AdaBoostRegressor modules of sklearn.
To apply Bagging to a linear regression we’ll use the following code. It performs a hyperparameter optimization on the number of models and the number of features.
lr_bag = RandomizedSearchCV(BaggingRegressor(LinearRegression()),param_distributions =  {               'n_estimators':np.arange(10,500,5),               'max_features':np.arange(1,10,1)          },          cv=5, n_iter = 20,          iid=False,random_state=0,refit=True,          scoring="neg_mean_absolute_error")lr_bag.fit(X_train,y_train)
The best values are:
The mean absolute error is 2.343 and here follows the scatter plot.
Finally, we can use the Boosting technique optimizing the number of models (we can use a simple grid search since we have a single hyperparameter).
Here’s the code:
lr_boost = GridSearchCV(AdaBoostRegressor(LinearRegression()),param_grid =  {               'n_estimators':np.arange(20,500,5)      },      cv=5,       iid=False,refit=True,      scoring="neg_mean_absolute_error")lr_boost.fit(X_train,y_train)
The best value is:
The mean absolute error is 2.580 and the scatter plot is:
The best result performed so far has been achieved by the Linear Regression with bagging.
Predicting stock price is hard and very difficult. It should be done frequently in order to learn from recent price fluctuations and try to better predict future ones.
Linear regression shows the best performance if helped by the Bagging technique, which reduces overfitting and tries to reduce collinearity between input features.
The next step could be using Recurrent Neural Networks, particularly Long Short Term Memory (LSTM) models. They have gained popularity in the last years and have proven their ability to model non-linear problems.
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.