Construct a Decision Tree and How to Deal with Overfitting | by Jun M. | Towards Data Science

A decision tree is an algorithm for supervised learning. It uses a tree structure, in which there are two types of nodes: decision node and leaf node. A decision node splits the data into two branches by asking a boolean question on a feature. A leaf node represents a class. The training process is about finding the “best” split at a certain feature with a certain value. And the predicting process is to reach the leaf node from root by answering the question at each decision node along the path.
The term “best” split means that after split, the two branches are more “ordered” than any other possible split. How do we define more ordered? It depends on which metric we choose. In general, there are two types of metric: gini impurity and entropy. The smaller these metrics are, the more “ordered” the dataset is.
The difference between the two metrics is pretty subtle. To learn more, you may read this post. In most applications, both metric behave similarly. Below is the code to calculate each metric.
The training process is essentially building the tree. A key step is determining the “best” split. The procedure is as follows: we try to split the data at each unique value in each feature, and choose the best one that yields the least disorder. Now let’s translate this procedure into Python code.
Before we build the tree, let’s define decision node and leaf node. A decision node specifies the feature and value upon which it will split. It also points to its left and right children. A leaf node includes a dictionary similar to a Counter object showing how many training examples for each class. This is useful to calculate the accuracy for training (although it is not necessary since we could get the accuracy by predicting each example after the model is trained). In addition, it leads to the resulting prediction for each example that reaches this leaf.
Given its structure, it is most convenient to construct the tree by recursion. The exit of recursion is a leaf node. This happens when we cannot increase the purity of the data through splitting. If we could find a “best” split, this becomes a decision node. Next we do the same recursively to its left and right children.
Now we can predict an example by traversing the tree until a leaf node.
It turns out that the training accuracy is 100% and the decision boundary is weird looking! Clearly the model is overfitting the training data. Well, if you think about it, a decision tree will overfit the data if we keep splitting until the dataset couldn’t be more pure. In other words, the model will correctly classify each and every example if we don’t stop splitting! The training accuracy is 100% (except when there are examples with different classes for exactly same features) without surprise.
From previous section, we know the behind-scene reason why a decision tree overfits. To prevent overfitting, there are two ways: 1. we stop splitting the tree at some point; 2. we generate a complete tree first, and then get rid of some branches. I am going to use the 1st method as an example. In order to stop splitting earlier, we need to introduce two hyperparameters for training. They are: maximum depth of the tree and minimum size of a leaf. Let’s rewrite the tree building part.
Now we can retrain the data and plot the decision boundary.
Next, we will visualize a decision tree by print out its nodes. The indentation of a node is proportional to its depth.
|---feature_1 <= 1.87|   |---feature_1 <= -0.74|   |   |---feature_1 <= -1.79|   |   |   |---feature_1 <= -2.1|   |   |   |   |---Class: 2|   |   |   |---feature_1 > -2.1|   |   |   |   |---Class: 2|   |   |---feature_1 > -1.79|   |   |   |---feature_0 <= 1.62|   |   |   |   |---feature_0 <= -1.31|   |   |   |   |   |---Class: 2|   |   |   |   |---feature_0 > -1.31|   |   |   |   |   |---feature_1 <= -1.49|   |   |   |   |   |   |---Class: 1|   |   |   |   |   |---feature_1 > -1.49|   |   |   |   |   |   |---Class: 1|   |   |   |---feature_0 > 1.62|   |   |   |   |---Class: 2|   |---feature_1 > -0.74|   |   |---feature_1 <= 0.76|   |   |   |---feature_0 <= 0.89|   |   |   |   |---feature_0 <= -0.86|   |   |   |   |   |---feature_0 <= -2.24|   |   |   |   |   |   |---Class: 2|   |   |   |   |   |---feature_0 > -2.24|   |   |   |   |   |   |---Class: 1|   |   |   |   |---feature_0 > -0.86|   |   |   |   |   |---Class: 0|   |   |   |---feature_0 > 0.89|   |   |   |   |---feature_0 <= 2.13|   |   |   |   |   |---Class: 1|   |   |   |   |---feature_0 > 2.13|   |   |   |   |   |---Class: 2|   |   |---feature_1 > 0.76|   |   |   |---feature_0 <= -1.6|   |   |   |   |---Class: 2|   |   |   |---feature_0 > -1.6|   |   |   |   |---feature_0 <= 1.35|   |   |   |   |   |---feature_1 <= 1.66|   |   |   |   |   |   |---Class: 1|   |   |   |   |   |---feature_1 > 1.66|   |   |   |   |   |   |---Class: 1|   |   |   |   |---feature_0 > 1.35|   |   |   |   |   |---Class: 2|---feature_1 > 1.87|   |---Class: 2
The examples above clearly shows one characteristic of decision tree: the decision boundary is linear in the feature space. While the tree is able to classify dataset that is not linearly separable, it relies heavily on the quality of training data and its accuracy decreases around decision boundaries. One way to address this drawback is feature engineering. Similar to the examples in logistic regression, we could expand the features to include non-linear terms. Here is one example if we add terms like x1x2, x12 and x22.
Unlike other regression models, decision tree doesn’t use regularization to fight against overfitting. Instead, it employs tree pruning. Selecting the right hyperparameters (tree depth and leaf size) also requires experimentation, e.g. doing cross-validation with a hyperparameter matrix.
For a complete workflow, including data generation and plotting decision boundary, you may visit my github.