A limitation of Random Forest Regression | by Ben Thompson | Towards Data Science

Random Forest is a popular machine learning model that is commonly used for classification tasks as can be seen in many academic papers, Kaggle competitions, and blog posts. In addition to classification, Random Forests can also be used for regression tasks. A Random Forest’s nonlinear nature can give it a leg up over linear algorithms, making it a great option. However, it is important to know your data and keep in mind that a Random Forest can’t extrapolate. It can only make a prediction that is an average of previously observed labels. In this sense it is very similar to KNN. In other words, in a regression problem, the range of predictions a Random Forest can make is bound by the highest and lowest labels in the training data. This behavior becomes problematic in situations where the training and prediction inputs differ in their range and/or distributions. This is called covariate shift and it is difficult for most models to handle but especially for Random Forest, because it can’t extrapolate.
For example, let’s say that you’re working with data that has an underlying trend over time such as stock prices, home values or sales. If you’re training data is missing any time periods, your Random Forest model will under or over predict, depending on the trend, examples outside of the time frames in your training data. This will be very noticeable if you plot your model’s predictions against their true values. Let’s take a look at this by creating some data.
import numpy as npimport matplotlib.pyplot as pltfrom sklearn.linear_model import LinearRegressionfrom sklearn.ensemble import RandomForestRegressor%matplotlib inline#make fake data with a time trendX = np.random.rand(1000,10)#add time feature simulating years 2000-2010time = np.random.randint(2000,2011,size=1000)#add time to XX = np.hstack((X,time.reshape(-1,1)))#create target via a linear relationship to Xweights = np.random.rand(11)y = X.dot(weights)#create test data that includes years#not in training data 2000 - 2019X_test = np.random.rand(1000,10)time_test = np.random.randint(2000,2020,size=1000)X_test = np.hstack((X_test,time_test.reshape(-1,1)))y_test = X_test.dot(weights)
Let’s see how well a Random Forest can predict the test data.
#fit and score the data using RFRF = RandomForestRegressor(n_estimators=100)RF.fit(X,y)RF.score(X_test,y_test)>>0.5872576516824577
That’s not very good. Let’s plot our predictions against their known values to see what is going on.
#plot RF as trees increase#set starting point for subplotsindex = 1#set the size of the subplot to something largeplt.figure(figsize=(20,20))#iterate through number of trees in model#and plot predictions v actualfor i in [1,5,10,100]:    plt.subplot(2, 2, index)    RF_plot = RandomForestRegressor(n_estimators=i)    RF_plot.fit(X,y)    #split data btw vals RF can interploate vs. data    #it needs to exptrapolate    interpolate_index = X_test[:,10]<=2010    extrapolate_index = X_test[:,10]>2010    X_interpolate = X_test[interpolate_index]    X_extrapolate = X_test[extrapolate_index]    y_interpolate = y_test[interpolate_index]    y_extrapolate = y_test[extrapolate_index]    #plot predictions vs. actual    plt.scatter(RFplot.predict(X_interpolate),                y_interpolate,                color="g",label="interpolate")    plt.scatter(RFplot.predict(X_extrapolate),                y_extrapolate,                color="b",label="extrapolate")    plt.xlabel('Predicted')    plt.ylabel('Actual')    plt.title('Random Forest with {} trees'.format(i))    plt.subplots_adjust(wspace=.4, hspace=.4)    plt.legend(loc="best")    index += 1
This plot makes it clear that the highest value the model can predict is around 961, while the underlying trend in the data pushes more recent values as high as 966. Unfortunately, the Random Forest can’t extrapolate the linear trend and accurately predict new examples that have a time value higher than that seen in the training data (2000–2010). Even adjusting the number of trees doesn’t fix the problem. In this situation, since we forced a perfectly linear relationship on the data, a model like Linear Regression would be a better choice and will have no problem detecting the trend in the data and making accurate predictions for data outside of the time ranges in the training data.
#fit the data using Linear RegressionLR = LinearRegression()LR.fit(X,y)LR.score(X_test,y_test)>>1.0#plot predictions of Linear Regression against actualplt.figure(figsize=(7,7))plt.xlabel('Predicted')plt.ylabel('Actual')plt.title('Linear Regression - Test Data')_ = plt.scatter(LR.predict(X_interpolate),y_interpolate,               color="g",label="interpolate")_ = plt.scatter(LR.predict(X_extrapolate),y_extrapolate,               color="b",label="extrapolate")plt.legend(loc="best")
While Random Forest is often an excellent choice of model, it is still important to know how it works, and if it might have any limitations given your data. In this case, because it is a neighborhood based model it prevented us from making accurate predictions for time frames outside of our training data. If you find yourself in such a situation it would be best to test other models, such Linear Regression or Cubist, and/or consider using the Random Forest in an ensemble of models. Happy predicting!