CART: Classification and Regression Trees for Clean but Powerful Models
By: Saul Dobilas
Tags: Getting Started, Plotly, Machine Learning, Gini, Decision Tree Classifier

While in this story, I focus on CART for classification, the regression case is very similar except for using a different method to calculate the best splits in the tree.

How do classification and regression trees work?

Example

Let’s start with a simple example. Assume you have a bunch of oranges and mandrins with labels on them, and you want to identify a set of simple rules that you can use in the future to distinguish between these two types of fruit.

Photo by Philippe Gauthier on Unsplash

Typically, oranges (diameter 6–10cm) are bigger than mandarins (diameter 4–8cm), so the first rule found by your algorithm might be based on size:

Diameter ≤ 7cm.

Next, you may notice that mandarins tend to be slightly darker in color than oranges. So, you use a color scale (1=dark to 10=light) to split your tree further:

Color ≤5 for the left side of the sub-tree

Color ≤6 for the right side of the sub-tree

Your final result is a tree that consists of 3 simple rules that help you to correctly distinguish between oranges and mandarins in the majority of the cases:

Decision Tree for identifying oranges vs. mandarins. Image by author.

How does CART find the best split?

Several methods can be used in CART to identify the best splits. Here are two of the most common ones for classification trees:

Gini Impurity

where p_i is the fraction of items in the class i.

Using the above tree as an example, Gini Impurity for the leftmost leaf node would be:

1 - (0.027^2 + 0.973^2) = 0.053

To find the best split, we need to calculate the weighted sum of Gini Impurity for both child nodes. We do this for all possible splits and then take the one with the lowest Gini Impurity as the best split.

Calculating Gini Impurity. Image by author.

Important note: If the best weighted Gini Impurity for the two child nodes is not lower than Gini Impurity for the parent node, you should not split the parent node any further.

Entropy

The entropy approach is essentially the same as Gini Impurity, except it uses a slightly different formula:

To identify the best split, you would have to follow all the same steps outlined above. The split with the lowest entropy is the best one. Similarly, if the entropy of the two child nodes is not lower than that of a parent node, you should not split any further.

How to build CART Decision Tree models in Python?

We will build a couple of classification decision trees and use tree diagrams and 3D surface plots to visualize model results. First, let’s do some basic setup.

Setup

We will use the following data and libraries:

Let’s import all the libraries:

Then we get the Australian weather data from Kaggle, which you can download following this link: https://www.kaggle.com/jsphyg/weather-dataset-rattle-package.

Once you have saved the data on your machine, ingest it with the below code. Note, we also do some simple data manipulation and derive a few new variables for later use in our models.

A snippet of Kaggle’s Australian weather data with some modifications. Image by author.

To reduce the amount of repeated code, we will create a couple of functions that we can reuse throughout the analysis.

This first function performs the following actions:

Splits the data into train and test samples

Fits the model

Predicts the label on a test set

Generates model performance evaluation metrics

Creates a decision tree graph

The second function will be used to plot 3D scatter graphs with the test data and model prediction surface:

CART classification model using Gini Impurity

Our first model will use all numerical variables available as model features. Meanwhile, RainTomorrowFlag will be the target variable for all models.

Note, at the time of writing sklearn’s tree.DecisionTreeClassifier() can only take numerical variables as features. However, you can also use categorical ones as long as you encode them with an encoding algorithm such as sklearn’s Ordinal Encoder or any other appropriate method to convert categorical values into numerical ones.

Let’s use our fitting function to build the model with the tree depth limited to 3 and minimum leaf size being 1,000 observations. Limiting tree depth and leaf size help us to avoid overfitting. In a later example, we will look at how much tree complexity increases once we remove some of those constraints.

Here is the output generated by the fitting function:

Model 1 — CART model performance metrics. Image by author.

We can see that the model performs relatively well in predicting dry days. However, the performance is worse on predicting rainy days, with precision on test data being 0.76 and recall 0.34.

Precision means that it will actually rain tomorrow in 76% of those cases where the model predicts a rainy day.

means that it will actually rain tomorrow in 76% of those cases where the model predicts a rainy day. Meanwhile, recall means that for all the rainy days in the test data, the model only identified 34% of them.

The difference in performance across the two class labels is largely driven by an imbalance in the data, with many more dry days available than rainy days.

Next, let’s look at the tree graph generated by our fitting function: