Decision Trees. An Overview of Classification and... | by Jason Wong | Towards Data Science

This post will serve as a high-level overview of decision trees. It will cover how decision trees train with recursive binary splitting and feature selection with “information gain” and “Gini Index”. I will also be tuning hyperparameters and pruning a decision tree for optimization. The two decision tree algorithms covered in this post are CART (Classification and Regression Trees) and ID3 (Iterative Dichotomiser 3).
Decision trees are very popular for predictive modeling and perform both, classification and regression. Decision trees are highly interpretable and provide a foundation for more complex algorithms, e.g., random forest.
The structure of a decision tree can be thought of as a Directed Acyclic Graph, a sequence of nodes where each edge is directed from earlier to later. This graph flows in one direction and no object can be a child of itself. Take a look at the DAG above, we can see it starts with a root node, the best attributes become interior nodes, i.e., decision nodes. Then, the internal nodes check for a condition and perform a decision, partitioning the sample space in two. The leaf nodes represent a classification, when the record reaches the leaf node, the algorithm will assign the label of the corresponding leaf. This process is referred to as recursive partitioning of the sample space. Terminology when working with decision trees:
Parent Node - a node divided into sub-nodesChild Node - sub-nodes from a parent nodeRoot Node - represents the sample space/population that will be split into two or more sets (sub-nodes)Decision Node - sub-node that splits into more sub-nodes Leaf Node - nodes without splits (children)Branch - a subsection of a decision treePruning - reducing the size of a decision tree by removing nodes
Decision trees use some cost function in order to choose the best split. We’re trying to find the best attribute/feature that performs the best at classifying the training data. This process is repeated until a leaf node is reached and therefore, is referred to as recursive binary splitting. When performing this procedure all values are lined up and the tree will test different splits and select the one returning the lowest cost, making this a greedy approach.
Something to note, since the algorithm repeatedly partitions the data into smaller subsets, the final subsets (leaf nodes) consist of few or only one data points. This causes the algorithm to have low bias and high variance.
A widely used metric with decision trees is entropy. Shannon’s Entropy, named after Claude Shannon provides us with measures of uncertainty. When it comes to data, entropy tells us how messy our data is. A high entropy value indicates less predictive power, think of the entropy of a feature as the amount of information in that feature. Decision trees work to maximize the purity of the classes when making splits, providing more clarity of the classes in the leaf nodes. The entropy is calculated before and after each split. If the entropy increases, another split will be tried or the branch of the tree will stop, i.e., the current tree has the lowest entropy. If the entropy decreases, the split will be kept. The formula for calculating entropy of an entire dataset:
where n is the number of groups and (pi) is the probability of belonging to the ith group. Let’s say we have a dataset containing 462 positive (1) labels and 438 negative (0) labels. We can calculate the entropy of the dataset by:
Information gain uses entropy as a measure of impurity. It is the difference in entropy from before to after the split, and will give us a number to how much the uncertainty has decreased. It is also the key criterion used in the ID3 classification tree algorithm. To calculate the information gain:
When performing classification tasks, the Gini index function is used. From Corrado Gini, this function informs us of how “pure” the leaf nodes in the tree are. The gini impurity will always be a value from 0 to 0.5, the higher the value, the more disordered the group is. To calculate the gini impurity:
where (pi) is the probability of belonging to the ith group. The equation above states the gini impurity is 1 minus the sum of the different probabilities in each split.
When decision trees train by performing recursive binary splitting, we can also set parameters for stopping the tree. The more complex decision trees are, the more prone they are to overfitting. We can prune the tree by trimming it using the hyperparameters:
max_depth- determines how deep we want the tree to be
min_samples_leaf- minimum number of training samples in each leaf node
max_leaf_nodes- maximum number of leaf nodes
min_impurity_decrease- threshold to determine if a node will split or become a leaf
There are more parameters that can be changed, for a list and a more detailed explanation, take a look over the documentation.
Let’s build a decision tree classifier with sklearn. I will be using The Titanic dataset, with the target being the Survived feature. The dataset I’m loading in has previously been cleaned. For a description on the features in the dataset, see the data dictionary below.
Importing the necessary libraries
Loading in and previewing the dataset
Defining predictor and target features, performing train test split, and preprocessing data
Training a decision tree classifier
The decision tree classifier is performing better on the train set than the test set, indicating the model is overfit. Decision trees are prone to overfitting since the recursive binary splitting procedure will continue until a leaf node is reached, resulting in an overly complex model. This is where we would perform hyperparameter tuning and pruning to optimize the classifier.
Plotting the tree
It may be helpful to plot the tree in order to visually see the splits. We can plot the tree with a few extra libraries.
Feature Importance
If we want to check the feature importances of the model, we can use the .feature_importances_ attribute from the decision tree classifier. Feature importance is calculated using the gini importance.
Optimizing a decision tree classifier with grid search cv
By running a cross-validated grid search we can input a parameter dictionary containing different values for the decision tree hyperparameters. I have used the pruning hyperparameters mentioned above with the default 5 fold cross validation.
By running the cross-validated grid search, the best parameters improved our bias-variance tradeoff. The first model with default parameters performed 20% better on the train set than the test set, indicating low-bias and high variance in the tree. The decision tree with the hyperparameters set from the grid search shows the variance was decreased with a 5% drop-off in accuracy from the train and test sets.
Decision trees performing regression tasks also partition the sample place into smaller sets like with classification. The goal for regression trees is to recursively partition the sample space until a simple regression model can be fit to the cells. The leaf nodes in a regression tree are the cells of the partition. The simple regression models being fit to each partition take the mean of the dependent variable for the partition, i.e., the sample mean is used to make predictions.
We used entropy above as a measure of impurity for performing classification. With regression, the CART algorithm utilizes mean squared error as a measure of impurity.
Dm is the training subset in node m
nm is the number of training samples in node m
yi is the target value for the ith sample
ŷm is the predicted target value, mean of the sample
When evaluated the performance of the model, we will be looking at the root mean squared error (RMSE). This is just the square root of the mean of squared errors. By taking the square root we can measure the size of error that weights large errors more than the mean. The metric we will use for evaluating the goodness of fit for our model is the R-squared value. The r-squared tells us the percentage of the variance in the dependent variables explain collectively (Frost et al., 2020).
CART can be used for more than just regression. Here’s an interesting article from Neptune.ai where decision trees are used to detect outliers/anomalies in time series data.
Let’s go ahead and build a decision tree regressor with sklearn. I will be using the Ames Housing dataset retrieved from kaggle. For the purposes of this tutorial I will only be using 3 continuous features and the target feature.
Loading in the dataset, defining predictor and target features, and performing train test split
Training a decision tree regressor
Once again, the decision tree is overfitting to the train set. Similarly to classification, we can run a cross-validated grid search to optimize the decision tree.
Optimizing Decision tree regressor with grid search cv
By running the cross-validated grid search with the decision tree regressor, we improved the performance on the test set. The r-squared was overfitting to the data with the baseline decision tree regressor using the default parameters with an r-squared score of .9998. Using the parameters from the grid search, we increased the r-squared on the test set from .42 to .56. The train r-squared being .58 tells us the model is not overfitting to the training data and will perform similarly on the unseen test set.
Decision trees are great predictive models that can be used for both classification and regression. They are highly interpretable and powerful for a plethora of machine learning problems. While there are many similarities between classification and regression tasks, it is important to understand different metrics used for each. The hyperparameters for decision trees are helpful in combating their tendency to overfit to the training data. Something to note, while performing a grid search can help in finding optimal hyperparameters for a decision tree, they can also be very computationally expensive. You may have a grid search running for hours or even days depending on the possible parameters chosen.
I hope this post was helpful in gaining a better understanding to regression and classification trees. If there was anything left out or that I could’ve explained more clearly please feel free to leave feedback! Thanks so much for taking the time to check out the post.
1.10. Decision Trees. (n.d.). Retrieved from https://scikit-learn.org/stable/modules/tree.html#
Sklearn.tree.DecisionTreeClassifier. (n.d.). Retrieved from https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
Sklearn.tree.DecisionTreeRegressor. (n.d.). Retrieved from https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html
Ronaghan, S. (2019, November 01). The Mathematics of Decision Trees, Random Forest and Feature Importance in Scikit-learn and Spark. Retrieved from https://towardsdatascience.com/the-mathematics-of-decision-trees-random-forest-and-feature-importance-in-scikit-learn-and-spark-f2861df67e3
14.2 — Recursive Partitioning. (n.d.). Retrieved from https://online.stat.psu.edu/stat555/node/100/
Brownlee, J. (2020, August 14). Classification And Regression Trees for Machine Learning. Retrieved from https://machinelearningmastery.com/classification-and-regression-trees-for-machine-learning/
Frost, J., Lombardi, J., Hadero, M., Aksamitova, J., Lamessa, Laurie, . . . Dubey, D. (2020, November 03). How To Interpret R-squared in Regression Analysis. Retrieved from https://statisticsbyjim.com/regression/interpret-r-squared-regression/
RMSE: Root Mean Square Error. (2020, July 06). Retrieved from https://www.statisticshowto.com/probability-and-statistics/regression-analysis/rmse-root-mean-square-error/
Sklearn.metrics.mean_squared_error. (n.d.). Retrieved from https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html
Titanic — Machine Learning from Disaster. (n.d.). Retrieved from https://www.kaggle.com/c/titanic
Ames Housing Data Summer 2018. (n.d.). Retrieved from https://www.kaggle.com/c/ames-housing-data-summer-2018/data