Linear Tree: the perfect mix of Linear Model and Decision Tree
By: Marco Cerliani
Tags: Linear Models, Editors Pick, Decision Tree, Data Science, Machine Learning

Decision Trees are well-known algorithms in the field of statistics and machine learning. For sure, every data scientist has ever tried to fit Decision Trees. They are one of the first topics in every course of data mining where we learn how to deal with supervised and unsupervised tasks. When fitting a Decision Tree, the goal is to create a model that predicts the value of a target by learning simple decision rules based on several input variables.

The predictions of a Decision Tree are simple constant approximations obtained at the end of the optimal data splitting process. This base feature made Decision Trees widely adopted. They are simple to understand and interpret. Not require particular assumptions about the input data (no data preparation, no constraints on the features types). On the contrary, they tend to overfit easily if the growth is not monitored. The instability can be a serious problem if some unseen samples fall in low explored sections during the training phase.

Not everybody knows simple yet effective variations of the Decision Tree algorithm. These are known as Model Trees. They learn an optimal splitting of the training data, as in the standard Decision Tree, with the main difference that the goodness of splits is evaluated fitting models. A particular case of Model Trees is known as Linear Tree. This implies having Linear Models in the leaves instead of simple constant approximations. It can be seen as a simple improvement/alternative from the standard Decision Tree preserving at the same time the interpretation ability.

In this post, we introduce how to easily create a Linear Tree using the linear-tree package. linear-tree is a python library to build Model Trees with Linear Models at the leaves.

linear-tree is developed to be fully integrable with scikit-learn. LinearTreeRegressor and LinearTreeClassifier are provided as scikit-learn BaseEstimator. They are wrappers that build a Decision Tree on the data fitting a linear estimator from sklearn.linear_model . All the models available in sklearn.linear_model can be used as linear estimators.

In other words, we only have to choose a Linear Model to build our Linear Tree. The linear-tree wrapper simply develops a Decision Tree structure on the data. At each node, it searches for an optimal split simply evaluating if the weighted loss of the children is lower than the loss of the parent. The losses involved are derived from the fitting of Linear Models on the data sections.

The splits thresholds in each feature are derived through quantile binning to make the process faster. The only parameters involved are the loss criterion , max_depth , min_samples_split , min_samples_leaf and max_bins (which are self explainable and the same as in classical Decision Trees). Let’s see it in action.

LINEAR TREE FOR REGRESSION

In this section, we use a Linear Tree to model a regression task. To make it understandable and visually explainable, we fit a 1D time-series data.

1D sinusoidal data (image by the author)

We operate a fit at various depths to see how the Linear Tree splits the input space and made predictions.

Linear Tree Regressor at various depths (image by the author)

It’s clearly visible that the Linear Tree operates a linear approximation in the splits. This is in contrast with a classical Decision Tree which operates constant approximations on the same data.