Hyperparameters of Decision Trees Explained with Visualizations | by Soner Yıldırım | Towards Data Science

Decision tree is a widely-used supervised learning algorithm which is suitable for both classification and regression tasks. Decision trees serve as building blocks for some prominent ensemble learning algorithms such as random forests, GBDT, and XGBOOST.
A decision tree builds upon iteratively asking questions to partition data. For instance, the following figure represents a decision tree used as a model to predict customer churn.
Decision trees are prevalent in the field of machine learning due to their success as well as being straightforward. Some of the features that make them highly efficient:
Easy to understand and interpret
Can handle both numerical and categorical data
Requires little or no preprocessing such as normalization or dummy encoding
On the downside, decision trees are prone to overfitting. They can easily become over-complex which prevents them from generalizing well to the structure in the dataset. In that case, the model is likely to end up overfitting which is a serious issue in machine learning.
To overcome this issue, we need to carefully adjust the hyperparameters of decision trees. In this post, we will try to gain a comprehensive understanding of these hyperparameters using tree visualizations.
We will use one of the built-in datasets of scikit-learn. The wine dataset contains 13 features (i.e.columns) on three different wine classes. There are 178 samples (i.e. rows) in the dataset.
import numpy as npimport pandas as pdfrom sklearn.datasets import load_wineX, y = load_wine(return_X_y = True)
Let’s start with a decision tree classifier without any hyperparameter tuning.
from sklearn import treeclf = tree.DecisionTreeClassifier()clf.fit(X, y)
All of the hyperparameters are set with the default settings. We can plot our model using plot_tree function.
import matplotlib.pyplot as plt%matplotlib inlineplt.figure(figsize=(24,14))tree.plot_tree(clf, filled=True, fontsize=14)
The model keeps splitting the nodes until all the nodes are pure (i.e. contain samples from only one class). Let’s first understand what the information in a
box tells us. The first line indicates the name of the feature (i.e. column). Since we did not name the columns, the index of the column is shown. Samples indicates the number of observations (i.e. rows) and value shows the distribution of these samples according to the target variable.
Gini is a measure of impurity. As stated on wikipedia, “Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset”. It basically means that impurity increases with randomness. For instance, let’s say we have a box with ten balls in it. If all the balls are same color, we have no randomness and impurity is zero. However, if we have 5 blue balls and 5 red balls, impurity is 1. If you take a look the leaf nodes (the nodes at the end of tree), you will see that gini is equal to zero.
The other function to evaluate the quality of a split is entropy which is a measure of uncertainty or randomness. The more randomness a variable has, the higher the entropy is.
We can select gini or impurity using criterion parameter. The default value is gini.
We usually do not want a tree with all pure leaf nodes. It would be too specific and likely to overfit.
When the algorithm performs a split, the main goal is to decrease impurity as much as possible. The more the impurity decreases, the more informative power that split gains. As the tree gets deeper, the amount of impurity decrease becomes lower. We can use this to prevent the tree from doing further splits. The hyperparameter for this task is min_impurity_decrease. It is set to zero by default. Let’s change it and see the difference.
clf = tree.DecisionTreeClassifier(min_impurity_decrease=0.2)clf.fit(X, y)plt.figure(figsize=(18,10))tree.plot_tree(clf, filled=True, fontsize=14)
We now have a much smaller tree. Consider the green node at the bottom. It contains 65 samples and 61 of them belong to one class. There is no need to further split that node because we can afford to have 4 misclassified samples out of 65. If we keep splitting that node, the model will probably be overfitting. Min_impurity_split parameter can be used to control the tree based on impurity values. It sets a threshold on gini. For instance, if min_impurity_split is set to 0.3, a node needs to have a gini value that is more then 0.3 to be further splitted.
Another hyperparameter to control the depth of a tree is max_depth. It does not make any calculations regarding impurity or sample ratio. The model stops splitting when max_depth is reached.
clf = tree.DecisionTreeClassifier(max_depth=3)clf.fit(X, y)plt.figure(figsize=(20,10))tree.plot_tree(clf, filled=True, fontsize=14)
Max_depth is less flexible compared to min_impurity_decrease. For instance,
we probably should not make the split on the left. It only distinguishes 2 samples and decreases the impurity by less than 0.1. This actually brings us to another hyperparameter which is min_samples_leaf. It indicates the minimum number of samples required to be at a leaf node. We need to be careful when using hyperparameters together. For instance, if we set min_samples_leaf to 3 in the previous tree, the split that separates 2 samples will not occur because we cannot have a leaf node with less than 3 samples. Well, it’s not quite right. Let’s see what really happens.
clf = tree.DecisionTreeClassifier(max_depth=3,min_samples_leaf=3)clf.fit(X, y)plt.figure(figsize=(20,10))tree.plot_tree(clf, filled=True, fontsize=14)
In this case, min_samples_leaf is actually harmful for the model. It did not prevent the model from doing that final split. Furthermore, it caused extra misclassifications. Thus, it is not wise to use min_samples_leaf in this way.
We can also limit the number of leaf nodes using max_leaf_nodes parameter which grows the tree in best-first fashion until max_leaf_nodes reached. The best split is decided based on impurity decrease.
clf = tree.DecisionTreeClassifier(max_leaf_nodes=5)clf.fit(X, y)plt.figure(figsize=(20,10))tree.plot_tree(clf, filled=True, fontsize=14)
We end up having a tree with 5 leaf nodes.
Another important hyperparameter of decision trees is max_features which is the number of features to consider when looking for the best split. If not specified, the model considers all of the features. There are 13 features in our dataset. If we set max_features as 5, the model randomly selects 5 features to decide on the next split. Max_features parameter also helps preventing the model from overfitting but it is not enough just to use max_features. If we let the model to become too deep, it will end up using all the features anyway.
The hyperparameters need to be carefully adjusted in order to have a robust decision tree with a high out-of-sample accuracy. We do not have to use all of them. Depending on the task and the dataset, a couple of them could be enough. Please pay extra attention if you use multiple hyperparameters together because one may negatively effect the other.
Thank you for reading. Please let me know if you have any feedback.