DecisionTreeRegressor — Stop Using For Future Projections! | by Kaushik Choudhury | Towards Data Science

Scikit-learn is one of the most famous machine learning library in Python. It offers several classification, regression and clustering algorithms and its key strength, in my opinion, is seamless integration with Numpy, Pandas and Scipy.
Scikit-learn is so well written by the developers, that with a couple of lines of code we can check the model predictions by many different algorithms. I feel sometimes, this strength of Scikit-learn inadvertently works to its disadvantage. Machine learning developers esp. with relatively lesser experience implements an inappropriate algorithm for prediction without grasping particular algorithms salient feature and limitations.
In this article, I will discuss the reason we should not use the decision tree regression algorithm in making a prediction involving extrapolating the data.
Objective
We have the iron, calcium and protein content of peas since the time it is picked from the farm until 1142 days. Let us assume that it is easier and economical to measure the iron and calcium content compare to protein content.
We will use this data to train the DecisionTreeRegressor algorithm and then predict the protein content based on new data points related to iron content, calcium, and days passed.
Sample Data File
I think that the data file is pretty much self-explanatory. The rows show the iron, calcium and protein content of peas with days since harvesting.
Step 1- We will import the packages pandas, matplotlib, and DecisionTreeRegressor and NumPy which we are going to use for our analysis.
from sklearn.tree import DecisionTreeRegressorimport pandas as pdimport matplotlib.pyplot as pltimport numpy as np
Step 2- Read the full data sample data excel file into the PandasDataframe called “data”.
data= pd.read_excel("Peas Nutrient.xlsx")
I will not focus on preliminary data quality checks like blank values, outliers, etc. and respective correction approach in this article, and assuming that there are no data series related to the discrepancy.
Step 3- We will split the full data set into two parts viz. training and testing set. As the name suggests, we will be using the training dataset to train the decision tree regressor algorithm and compare the protein predictions with actual content based on data in the testing set.
In the below code, data records from day 1 to day 900 are sliced as training data and data records from day 901 to 1142 as testing data.
Training_data= data[:900]Test_data=data.loc[901:1142]
Step 4- “Days passed”, “iron content” and “calcium content” are independent variables used for prediction.”Protein content” predicted is the dependent variable. Generally, the independent variable is denoted with “X “and the dependent variable with “y”.
In the code below, “Protein content” data column is dropped from the DataFrame and remaining, data i.e independent variables datapoints is declared as X_train. Similarly, all the data columns except “Protein content” is dropped and declared as y_train.
X_train=Training_data.drop(["Protein Content "], axis=1)y_train=Training_data.drop(["Days Passed", "Iron Content " ,"Calcium Content "], axis=1)
The same process is repeated in the below code for the testing data set i.e. values from day 901 to day 1142,
X_test=Test_data.drop(["Protein Content "], axis=1)y_test=Test_data.drop(["Days Passed", "Iron Content " ,"Calcium Content "], axis=1)
Step 5- DecisionTreeRegressor model is trained with the training dataset. Further, the score is checked to understand how well the algorithm is trained on this data.
tree_reg = DecisionTreeRegressor().fit(X_train, y_train)print("The model training score is" , tree_reg.score(X_train, y_train))
A perfect score of 1.0 itself indicates the overfitting of the model.
Step 5- To address the overfitting due to unconstrained depth of tree during training the model, we will put a constraint of the max depth of 4.
tree_reg = DecisionTreeRegressor(max_depth=6).fit(X_train, y_train)print("The model training score is" , tree_reg.score(X_train, y_train))
This solves the overfitting of the model on training data, and the model is ready to predict the protein content based on test data points.
Step 6- In the below code, “protein content” of test data set i.e. from days 901 to 1142 is predicted based on respective “days passed”, “iron content” and “calcium content” data.
y_pred_tree = tree_reg.predict(X_test)
Step 7- We will plot the predicted protein content by the decision tree regression model and compare with actual protein content in the test dataset from day 901 to 1142.
plt.plot(X_test["Days Passed"],y_test, label="Actual Data")plt.plot(X_test["Days Passed"],np.rint(y_pred_tree), label="Predicted Data")plt.ylabel("Days Passed")plt.xlabel('Protin Content (in Grams)')plt.legend(loc='best')plt.show()
We can see that the decision tree regressor model, which is trained quite well in training dataset with 0.93 score fails miserably to predict the protein content on test data. The model predicts the same protein content of ~ 51.34 for all days.
We should not use the Decision Tree Regression model for prediction involving extrapolating the data. This is just an example, and the main takeaway for us machine learning practitioners are to consider the data, prediction objective, algorithms strengths and limitations before starting modelling.
We can make similar mistakes while selecting the independent variables for Machine Learning Supervised Algorithms. In the article, “How to identify the right independent variables for Machine Learning Supervised Algorithms? ” I have discussed a structured approach to identify the appropriate independent variables to make accurate predictions.