4 Useful techniques that can mitigate overfitting in decision trees
By: Rukshan Pramoditha
Tags: Hyperparameter Tuning, Decision Tree, Data Science, Machine Learning, Overfitting

Overfitting is the second worse problem for data scientists, after missing values and outliers. Compared to other machine learning models, overfitting in decision trees can easily happen as their learning algorithms can produce large and complex decision trees that perfectly fit training instances. In general, overfitting is a problem that happens when the learning algorithm perfectly fits the training data but fails to generalize for new unseen data (test/validation data). In overfitting, the model is memorizing the noise in the dataset and fails to learn important patterns in the data. Overfitting affects the accuracy (or any other model evaluation matric) of predictions made from the samples which are not part of the training set.

One can build a perfect decision tree model on the training data with 100% accuracy, but with significantly low accuracy on test data. Have a look at the following Python code which builds such a decision tree model on the “heart_disease” dataset.

Code snippet 1: Decision tree base model (Wait till loading!)

Output 1 (Image by author)

As you can see in the output, the training accuracy is 100% and the test accuracy is 71% which is much lower than the training accuracy. Therefore, it is very clear that our decision tree model is overfitting. Throughout this article, we’ll use this model as the base model so that it can be compared with other models.

There are plenty of available general techniques that can mitigate or prevent the problem of overfitting in machine learning models. However, in this article, we’ll discuss 4 techniques which are specific to decision trees.

Note: If you want to learn more about using decision trees for regression tasks, read this post written by me. However, this is not a prerequisite for today's content.

Overview of the techniques

Here is an overview of the techniques that we’ll discuss in this post.

(Image by author)

The general idea of pruning in decision trees

In the general context, pruning is a process which involves the removal of specific parts of a tree or plant to improve its growth. The same thing applies to decision trees which are not actual trees, but machine learning models. In decision trees, pruning is a process which is applied to control or limit the depth (size) of the trees. By default, decision tree model hyperparameters were created to grow the tree into its full depth. These trees are called fully-grown trees which are always overfitting. However, we can tune those hyperparameters to generate trees that are not fully grown and not overfitting!

There are two types of pruning techniques: Pre‐pruning and Post‐pruning

Pre‐pruning

Pre-pruning is an early stopping rule which stops the growth of a decision tree at an early stage. It generates a tree with fewer branches. Pre-pruning can be applied by tuning the following hyperparameters in the Scikit-learn DecisionTreeClassifier() or DecisionTreeRegressor() classes. These hyperparameters specify threshold values for the stopping conditions.

max_depth: Specifies the maximum depth of the tree. This controls the complexity of branching (i.e. the number of times the splits are made). If None (default), then nodes are expanded until all leaves are pure (i.e. fitting the model with 100% accuracy). Decreasing this value prevents overfitting.

Specifies the maximum depth of the tree. This controls the complexity of branching (i.e. the number of times the splits are made). If None (default), then nodes are expanded until all leaves are pure (i.e. fitting the model with 100% accuracy). Decreasing this value prevents overfitting. min_samples_leaf: Determines the minimum number of samples required to be at a leaf node. The default is 1. Increasing this value prevents overfitting.

Determines the minimum number of samples required to be at a leaf node. The default is 1. Increasing this value prevents overfitting. min_samples_split: Specifies the minimum number of samples required to split an internal node. The default is 2. If we set its value to 5, no further splits are permitted for nodes with five samples or fewer.

Among these hyperparameters, max_depth is the most important one. We can easily apply pre-pruning to decision trees by tuning the max_depth hyperparameter alone. Now, we’ll see the influence of max_depth alone while keeping other pre-pruning hyperparameters on their default values.

Technique 1: Tuning max_depth alone and see the influence of the single hyperparameter

Without doing grid search or randomized search, it is sometimes useful to find out the influence of the single max_depth hyperparameter on the training and test data. The validation curve is a very useful technique that can be used to measure the influence of a single hyperparameter. The following code creates the validation curve using the above decision tree base model. Note that the following code continues from code snippet 1 (see above).

Code snippet 2: Plotting the influence of max_depth (Wait till loading!)

Output 2 (Image by author)

Note that, in the graph, the accuracy score of the train set is marked as the “Training Score” and the accuracy score of the test set is marked as the “Cross-Validation Score”. At the max_depth value of 3, the model is just right (not overfitting or underfitting). The model fits the training data very well and it is also generalizable to new unseen data. Let’s build the decision tree again, but now with max_depth=3 and compare the predictions with the base model’s predictions.

Code snippet 3: Creating decision tree with max_depth=3 (Wait till loading!)

Output 3 (Image by author)

Compared to the base model, the test accuracy has been improved from 0.71 to 0.74. In addition to that, the number of false negatives (classifying a true heart disease patient into a non-heart disease person — which is much more dangerous!) have significantly been reduced. So, the model behaves in the way that we want.

Note: If you want to learn more about plotting and interpreting the validation curve, read this post written by me.

Technique 2: Tuning multiple hyperparameters at once

Sometimes, tuning a single hyperparameter alone may not be much efficient. Another pre-pruning technique is finding a new combination of multiple hyperparameter values that mitigate the overfitting. The validation curve cannot be used in this case as it was designed to plot the influence of a single hyperparameter. Here, we can use Grid Search combined with Cross-validation to tune multiple hyperparameters at once. The following code performs hyperparameter tuning with cross-validation using the above base model. Note that the following code continues from code snippet 1 (see above).

Code snippet 4: Tuning multiple hyperparameters at once (Wait till loading!)

Output 4 (Image by author)

Compared to the base model and the previous model with max_depth=3, the test accuracy has been improved significantly. In addition to that, both false positives and false negatives have significantly been reduced. Now, our model is not overfitting and performance on test data has been improved.

Note: If you want to learn more about k-fold cross-validation and hyperparameter tuning techniques, read this post written by me.

Post‐pruning

The Post-pruning technique allows to grow the decision tree in full and then removes parts of it.

Technique 3: Cost-complexity pruning

Cost complexity pruning (ccp) is one type of post-pruning techniques. It provides another option to control the tree size. It can be performed by finding the right value for the alpha which is often referred to as ccp_alpha in the Scikit-learn decision tree classes. Greater values of ccp_alpha increase the number of nodes pruned. By default, the value is zero meaning that no pruning is performed.

Scikit-learn DecisionTreeClassifier class provides a method called cost_complexity_pruning_path which returns a dictionary-like object containing the effective values of alpha (accessed by ccp_alphas attribute) and the corresponding impurities (accessed by impurities attribute) of subtrees at each step of the pruning process. We can obtain these alpha values of our base decision tree model by executing:

path = dtclf.cost_complexity_pruning_path(X_train, y_train)

ccp_alphas = path.ccp_alphas

ccp_alphas

Output 5 (Image by author)

Now, we can use these values of alpha to prune our decision tree. By looping over the ccp_alphas array, we pass each alpha value to the ccp_alpha hyperparameter of the DecisionTreeClassifier and then calculate the train and test accuracy scores at each iteration and store them in two lists.

Code snippet 5 (Wait till loading!)

Then we plot the train and test accuracy scores.

Code snippet 6: Plotting ccp_alpha vs accuracy (Wait till loading!)

Output 6 (Image by author)

By looking at the plot, we can decide the optimal alpha value that mitigates overfitting in our decision tree model is 0.06.

Now, we create the decision tree again by setting ccp_alpha=0.06.

Code snippet 7: Creating a decision tree with ccp_alpha=0.06 (Wait till loading!)

Output 7 (Image by author)

Compared to the base model, the test accuracy has been improved from 0.71 to 0.77. In addition to that, both false positives and false negatives have been reduced! Now, our model is not overfitting and performance on test data have been improved.

We can visualize the pruned decision tree.

Code snippet 8: Visualize the decision tree (Wait till loading!)

Output 8 (Image by author)

Now, we realize that post-pruning happened behind the scenes so that if we choose the second large alpha value (or a value closer to that), the tree was trimmed in the bottom-up fashion until the maximum depth of 2. The greater the ccp_alpha value, the more the number of nodes pruned!

Note: If you want to learn more about different ways of visualizing decision trees, read this post written by me.

Creating ensembles

Another technique for reducing overfitting in decision trees is creating multiple decision trees from subsets of the training data and features. This collection of trees is called an ensemble (group).

Technique 4: Creating a random forest

Random forests, as the name suggests, is an ensemble of decision trees. A large ensemble of uncorrelated decision trees in a random forest can produce more accurate and stable results than any of the individual decision trees. In a normal decision tree, the algorithm searches for the best feature out of all the features when it wants to split a node. In contrast, each tree in a random forest searches for the best feature out of a random subset of features. This creates extra randomness when growing the tress inside a random forest.

Note: If you want to learn more about random forests, read this post written by me.

The following code creates a random forest with 100 decision trees.

Code snippet 9: Creating a random forest with 100 trees (Wait till loading!)

Output 9 (Image by author)

Even without tuning any of the hyperparameters, the random forest model performs well compared to the base model! However, this model is still overfitting. It is recommended to apply some hyperparameter tuning techniques to further mitigate the overfitting.

Summary

All the techniques that we’ve discussed today can only mitigate overfitting in decision trees, but they cannot fully prevent the problem of overfitting. The decision tree is the base learner for other tree-based learners such as Random Forest, XGBoost. Therefore, the techniques that we’ve discussed today can almost be applied to those tree-based learners too.

Overfitting in decision trees can easily happen. Because of that, decision trees are rarely used alone in model building tasks. However, if you apply proper hyperparameter tuning techniques or make ensembles using multiple trees, you can mitigate overfitting in decision trees and even use them alone in model building tasks.

The model evaluation metric that we’ve used here is the “Accuracy” metric which is not the best option when the target variable has a class imbalance. We can check whether the target variable of our “heart_disease” dataset has a class imbalance or not:

import seaborn as sns

sns.catplot("target", data=df, kind="count",

aspect=1.0, height=4)

(Image by author)

There is no apparent class imbalance in our target variable.

Note: A complete list of model evaluation metrics and scoring methods used in Scikit-learn can be found here.

Thanks for reading!

Read my other articles at https://rukshanpramoditha.medium.com

All the images except the cover image, code samples, diagrams and all the content are copyrighted by the author. Special credit goes to Marissa&Eric on Unsplash who are the owners of the cover image. Thank you very much, Marissa&Eric, for providing me with an excellent image for the cover image in this post!

2021–05–16