How to Prune Decision Trees to Make the Most Out of Them | by Soner Yıldırım | Towards Data Science

Decision trees are supervised machine learning algorithms that work by iteratively partitioning the dataset into smaller parts. The partitioning process is the most critical part of building decision trees.
The partitions are not random. The aim is to increase the predictiveness of the model as much as possible at each partitioning so that the model keeps gaining information about the dataset.
For instance, the following is a decision tree with a depth of 3.
The first partition is based on feature 6 (X[6]) and able to put all data instances that belong to the first class (59) on the right side of the tree.
This clearly shows that partitions that maximize the information gain are prioritized. The information gain can be quantified by Entropy or Gini impurity. In general, the information gain is proportional to the increase in purity (or decrease in impurity).
The purity of a node is inversely proportional to the distribution of different classes in that node.
Consider a case where you pick a data instance and randomly label it. Gini impurity measures how often your random label would be incorrect. If all data points had the same label, then the label would always be correct and Gini impurity would be zero. On the other hand, if data points were equally distributed among multiple labels, the random labels would often be incorrect. Thus, Gini impurity increases with randomness.
Entropy is another measure of uncertainty or randomness. The more randomness a variable has, the higher the entropy is.
Partitions that result in more pure nodes are chosen. Thus, when selecting a feature to partition the dataset, the decision tree algorithm tries to achieve:
More predictiveness
Less impurity
Lower entropy
The decision trees need to be carefully tuned to make the most out of them. Too deep trees are likely to result in overfitting. Scikit-learn provides several hyperparameters to control the growth of a tree.
We will see how these hyperparameters achieve using the plot_tree function of the tree module of scikit-learn.
I will use the wine dataset available under the datasets module of scikit-learn. Let’s first import the libraries and load the dataset.
import numpy as npimport pandas as pdfrom sklearn import treefrom sklearn.datasets import load_wineimport matplotlib.pyplot as plt%matplotlib inlineX, y = load_wine(return_X_y=True)
The dataset 178 observations that belong to 3 different classes. There are 13 features that describe the observations.
X.shape(178,13)np.unique(y, return_counts=True)(array([0, 1, 2]), array([59, 71, 48]))
We can now start building decision trees with different hyperparameter values. The obvious one is the max_depth which stops tree growth at the specified depth level.
clf = tree.DecisionTreeClassifier(criterion='gini', max_depth=2)\.fit(X, y)plt.figure(figsize=(16,8))tree.plot_tree(clf, filled=True, fontsize=16)
The partitions are selected based on Gini impurity and the depth of the tree is 2. Max_depth provides a simple way to control tree growth which may not be enough in more complex cases.
The min_samples_split specifies the minimum number of samples in a node to be further split.
clf = tree.DecisionTreeClassifier(criterion='gini', min_samples_split=10).fit(X, y)plt.figure(figsize=(20,8))tree.plot_tree(clf, filled=True, fontsize=16)
I only put a part of the visualization where we see the effect of the current hyperparameter. The node at the right is not further split because there are only 5 samples in it. Without min_samples_split=10, it’d be further split as follows.
Another hyperparameter to control tree growth is min_impurity_decrease which sets a threshold on the impurity decrease to consider a partition. It is a more educated way than the max depth because it takes into account the quality of a partition.
clf = tree.DecisionTreeClassifier(criterion='gini', min_impurity_decrease=0.1).fit(X, y)plt.figure(figsize=(10, 6))tree.plot_tree(clf, filled=True, fontsize=16)
All the partitions achieved a decrease of more than 0.1 on the impurity. When setting this value, we should also consider the criterion because Gini impurity and Entropy have different values.
The max_leaf_nodes can also be used to control tree growth. It limits the number of leaf nodes a decision tree can have. The leaf nodes are the nodes at the end of a decision tree.
The tree keeps growing in the best-first fashion until the maximum number of leaf nodes is reached. The best partitions are chosen based on the decrease in impurity.
clf = tree.DecisionTreeClassifier(criterion='gini', max_leaf_nodes=5).fit(X, y)plt.figure(figsize=(14, 8))tree.plot_tree(clf, filled=True, fontsize=16)
We have covered 5 different hyperparameters that can be used to control tree growth.
Criterion
Max_depth
Min_samples_split
Min_impurity_decrease
Max_leaf_nodes
We have adjusted them one by one to see the individual effects. However, in real-life cases, these hyperparameters need to be tuned together in order to create a robust and accurate model.
If we grow a tree more than we should, we are likely to end up having an overfit model. Overfitting occurs when a tree fits the training set too well. Having very different accuracies on training and test sets is a strong indication of overfitting. In such cases, we should control the tree growth to obtain a well-generalized model.
Thank you for reading. Please let me know if you have any feedback.