Regression Tree in Python From Scratch | by Eligijus Bujokas | Towards Data Science

This article aims to present to the readers the code and the intuition behind the regression tree algorithm in python. I find looking through the code of an algorithm a very good educational tool to understand what is happening under the hood. I hope that the readers will this useful too.
The algorithm that is explained is the regression tree algorithm. It is used to model the relationship between a continuous variable Y and a set of features X:
Y = f(X)
The function f is a set of rules of features and feature values that does the “best” job of explaining the Y variable given features X.
This article is a continuation of the previous article about the decision trees:
towardsdatascience.com
I will assume that the reader will be familiar with the concept of a Node, splitting and the level of a tree. All of these concepts are explained in the previous article.
In the previous article, the Y variable was a binary variable containing two values — 0 and 1.
Regression trees are estimators that deal with a continuous response variable Y. For example, height, salary, clicks, etc. The algorithm is coded and implemented (as well as with a complimentary notebook) in my GitHub repository:
github.com
The Node class for regression is very similar to the binary classification tree class from the previous article:
Each Node, regardless of its level, has the following main attributes:
All of the above attributes will be used in the splitting procedure.
The preparation of features before a split is identical as in the classification decision tree:
For each of the numeric features in our dataset, we sort the feature values and get the means of two neighbouring values.
For example, suppose our feature_1 is the following:
feature_1 = [7, 5, 9, 1, 2, 8]
Sorted:
feature_1 = [1, 2, 5, 7, 8, 9]
Means of neighbours:
feature_1 = [1.5, 3.5, 6.5, 7.5, 8.5]
We do the same for all the numeric features.
For categorical features like
feature_2 = [‘cat’, ‘dog’, ‘dog’, ‘cow’, ‘cat’, ‘cow’] 
We find the unique values of the feature:
feature_2 = [‘cat’, ‘dog’, ‘cow’]
With such preprocessing, for each feature and each feature value we split the data of the node into two subsets, left and right:
For numeric features (for example, feature_p):
left = X[X[feature_p]<value]right = X[X[feature_p]≥value]
For categorical features (for example, feature_c):
left = X[X[feature_c]==feature_c]right = X[X[feature_c]!=feature_c]
In the classification tree, we needed to calculate the number of observations in each of the datasets (left and right). In regression trees, we lump all the residuals together of both the datasets and calculate the joint mean squared error statistic. The feature and the feature value which gives the lowest mse is the splitting criteria.
For example, let’s consider two possible splits, X at 4 and X at 6.5:
The residuals in the first example of the left node are [3.5, 4.5]
The residuals in the first example of the right node are [2, 0, -2]
The residuals in the second example of the left node are [-0.67, 0.33, 0.33]
The residuals in the second example of the right node are [1, -1]
Thus, the residuals of the first split are:
r1 = [3.5, 4.5, 2, 0, -2]
And the second:
r2 = [-0.67, 0.33, 0.33, 1, -1]
We then find the mse statistic of both the residual sets and compare them:
As we can see, the second mse is much smaller than the first. Thus, between the two possible splits, the second one (X ≥ 6) is better.
Let us try out the custom NodeRegression class with an example.
We want to create a tree f such that:
mpg = f(horsepower, weight)
Here
mpg — miles per gallon
horsepower — the number of horsepower in a car; engine power
weight — the weight of a car in kilograms
The implementation is very similar to the previous tree method:
# Reading datad = pd.read_csv("data/regression/auto-mpg.csv")# Subsettingd = d[d['horsepower']!='?']# Constructing the X and Y matricesfeatures = ['horsepower', 'weight']# Ensuring the correct types for ft in features:    d[ft] = pd.to_numeric(d[ft])# Constructing the X and Y matricesX = d[features]Y = d['mpg'].values.tolist()
Node initiation and growing the tree:
# Initiating the Noderoot = NodeRegression(Y, X, max_depth=2, min_samples_split=3)# Growing the treeroot.grow_tree()
And finaly printing the tree:
# Printing treeroot.print_tree()
We can compare the results with the scikit-learn implementation of the regression tree https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html with the same hyperparameters:
As we can see, the split values and features are all the same.
The last thing to note is that the forecast of the node is the mean of the Y observations in the node. In the classifier decision tree, the forecast is the class that has the highest number of observations in the node.
In the above-grown trees, if we follow the rules:
weight ≤2764.5 →horsepower ≤70.5
We would get that that node has 69 observations in it. If we average the mpg variable of those variables, we would get that these cars, on average, have an mpg value of 33.68. This is the prediction for all the future cars, which follow the above-mentioned path.
To conclude, I would highly recommend going through the code of the NodeRegressor class because I tried to write it very explicitly. By loading the data from my GitHub repository and going through the code line by line you will soon see that the regression tree is a very simple and intuitive algorithm.
Feel free to fork, copy, clone the code and create pull requests in my git repository, which was mentioned in the first paragraph.
If any of the concepts are still unclear, feel free to leave a comment.
Thanks for reading and happy coding!