Decision Trees: A step-by-step approach to building DTs | by Gokul S Kumar | Towards Data Science

Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. Decision trees are commonly used in operations research, specifically in decision analysis, to help identify a strategy most likely to reach a goal, but are also a popular tool in machine learning.
In this article, we will be discussing the following topics
What are decision trees in generalTypes of decision trees.Algorithms used to build decision trees.The step-by-step process of building a Decision tree.
What are decision trees in general
Types of decision trees.
Algorithms used to build decision trees.
The step-by-step process of building a Decision tree.
The above picture is a simple decision tree. If a person is non-vegetarian, then he/she eats chicken (most probably), otherwise, he/she doesn’t eat chicken. The decision tree, in general, asks a question and classifies the person based on the answer. This decision tree is based on a yes/no question. It is just as simple to build a decision tree on numeric data.
If a person is driving above 80kmph, we can consider it as over-speeding, else not.
Here is one more simple decision tree. This decision tree is based on ranked data, where 1 means the speed is too high, 2 corresponds to a much lesser speed. If a person is speeding above rank 1 then he/she is highly over-speeding. If the person is above speed rank 2 but below speed rank 1 then he/she is over-speeding but not that much. If the person is below speed rank 2 then he/she is driving well within speed limits.
The classification in a decision tree can be both categoric or numeric.
Here’s a more complicated decision tree. It combines numeric data with yes/no data. For the most part Decision trees are pretty simple to work with. You start at the top and work your way down till you get to a point where you cant go further. That’s how a sample is classified.
The very top of the tree is called the root node or just the root. The nodes in between are called internal nodes. Internal nodes have arrows pointing to them and arrows pointing away from them. The end nodes are called the leaf nodes or just leaves. Leaf nodes have arrows pointing to them but no arrows pointing away from them.
In the above diagrams, root nodes are represented by rectangles, internal nodes by circles, and leaf nodes by inverted-triangles.
There are several algorithms to build a decision tree.
CART-Classification and Regression TreesID3-Iterative Dichotomiser 3C4.5CHAID-Chi-squared Automatic Interaction Detection
CART-Classification and Regression Trees
ID3-Iterative Dichotomiser 3
C4.5
CHAID-Chi-squared Automatic Interaction Detection
We will be discussing only CART and ID3 algorithms as they are the ones majorly used.
CART is a DT algorithm that produces binary Classification or Regression Trees, depending on whether the dependent (or target) variable is categorical or numeric, respectively. It handles data in its raw form (no preprocessing needed) and can use the same variables more than once in different parts of the same DT, which may uncover complex interdependencies between sets of variables.
Now we are going to discuss how to build a decision tree from a raw table of data. In the example given above, we will be building a decision tree that uses chest pain, good blood circulation, and the status of blocked arteries to predict if a person has heart disease or not.
The first thing we have to know is which feature should be on the top or in the root node of the tree. We start by looking at how chest pain alone predicts heart disease.
There are two leaf nodes, one each for the two outcomes of chest pain. Each of the leaves contains the no. of patients having heart disease and not having heart disease for the corresponding entry of chest pain. Now we do the same thing for good blood circulation and blocked arteries.
We can see that neither of the 3 features separates the patients having heart disease from the patients not having heart disease perfectly. It is to be noted that the total no. of patients having heart disease is different in all three cases. This is done to simulate the missing values present in real-world datasets.
Because none of the leaf nodes is either 100% ‘yes heart disease’ or 100% ‘no heart disease’, they are all considered impure. To decide on which separation is the best, we need a method to measure and compare impurity.
The metric used in the CART algorithm to measure impurity is the Gini impurity score. Calculating Gini impurity is very easy. Let’s start by calculating the Gini impurity for chest pain.
For the left leaf,
Gini impurity = 1 - (probability of ‘yes’)2 - (probability of ‘no’)2              = 1 - (105/105+39)2 - (39/105+39)2Gini impurity = 0.395
Similarly, calculate the Gini impurity for the right leaf node.
Gini impurity = 1 - (probability of ‘yes’)2 - (probability of ‘no’)2              = 1 - (34/34+125)2 - (125/34+125)2Gini impurity = 0.336
Now that we have measured the Gini impurity for both leaf nodes, we can calculate the total Gini impurity for using chest pain to separate patients with and without heart disease.
The leaf nodes do not represent the same no. of patients as the left leaf represents 144 patients and the right leaf represents 159 patients. Thus the total Gini impurity will be the weighted average of the leaf node Gini impurities.
Gini impurity = (144/144+159)*0.395 + (159/144+159)*0.336              = 0.364
Similarly the total Gini impurity for ‘good blood circulation’ and ‘blocked arteries’ is calculated as
Gini impurity for ‘good blood circulation’ = 0.360Gini impurity for ‘blocked arteries’ = 0.381
‘Good blood circulation’ has the lowest impurity score among the tree which symbolizes that it best separates the patients having and not having heart disease, so we will use it at the root node.
Now we need to figure out how well ‘chest pain’ and ‘blocked arteries’ separate the 164 patients in the left node(37 with heart disease and 127 without heart disease).
Just like we did before we will separate these patients with ‘chest pain’ and calculate the Gini impurity value.
The Gini impurity was found to be 0.3. Then we do the same thing for ‘blocked arteries’.
The Gini impurity was found to be 0.29. Since ‘blocked arteries’ has the lowest Gini impurity, we will use it at the left node in Fig.10 for further separating the patients.
All we have left is ‘chest pain’, so we will see how well it separates the 49 patients in the left node(24 with heart disease and 25 without heart disease).
We can see that chest pain does a good job separating the patients.
So these are the final leaf nodes of the left side of this branch of the tree. Now let’s see what happens when we try to separate the node having 13/102 patients using ‘chest pain’. Note that almost 90% of the people in this node are not having heart disease.
The Gini impurity of this separation is 0.29. But the Gini impurity for the parent-node before using chest-pain to separate the patients is
Gini impurity = 1 - (probability of yes)2 - (probability of no)2              = 1 - (13/13+102)2 - (102/13+102)2Gini impurity = 0.2
The impurity is lower if we don’t separate patients using ‘chest pain’. So we will make it a leaf-node.
At this point, we have worked out the entire left side of the tree. The same steps are to be followed to work out the right side of the tree.
Calculate the Gini impurity scores.If the node itself has the lowest score, then there is no point in separating the patients anymore and it becomes a leaf node.If separating the data results in improvement then pick the separation with the lowest impurity value.
Calculate the Gini impurity scores.
If the node itself has the lowest score, then there is no point in separating the patients anymore and it becomes a leaf node.
If separating the data results in improvement then pick the separation with the lowest impurity value.
The process of building a decision tree using the ID3 algorithm is almost similar to using the CART algorithm except for the method used for measuring purity/impurity. The metric used in the ID3 algorithm to measure purity is called Entropy.
Entropy is a way to measure the uncertainty of a class in a subset of examples. Assume item belongs to subset S having two classes positive and negative. Entropy is defined as the no. of bits needed to say whether x is positive or negative.
Entropy always gives a number between 0 and 1. So if a subset formed after separation using an attribute is pure, then we will be needing zero bits to tell if is positive or negative. If the subset formed is having equal no. of positive and negative items then the no. of bits needed would be 1.
The above plot shows the relation between entropy and i.e., the probability of positive class. As we can see, the entropy reaches 1 which is the maximum value when which is there are equal chances for an item to be either positive or negative. The entropy is at its minimum when p(+) tends to zero(symbolizing x is negative) or 1(symbolizing x is positive).
Entropy tells us how pure or impure each subset is after the split. What we need to do is aggregate these scores to check whether the split is feasible or not. This is done by Information gain.
Consider this part of the problem we discussed above for the CART algorithm. We need to decide which attribute to use from chest pain and blocked arteries for separating the left node containing 164 patients(37 having heart disease and 127 not having heart disease). We can calculate the entropy before splitting as
Let’s see how well chest pain separates the patients
The entropy for the left node can be calculated
Similarly the entropy for the right node
The total gain in entropy after splitting using chest pain
This implies that if in the current situation if we were to pick chest pain for splitting the patients, we would gain 0.098 bits in certainty on the patient having or not having heart disease. Doing the same for blocked arteries , the gain obtained was 0.117. Since splitting with blocked arteries gives us more certainty, it would be picked. We can repeat the same procedure for all the nodes to build a DT based on the ID3 algorithm.
Note: The decision of whether to split a node into 2 or to declare it as a leaf node can be made by imposing a minimum threshold on the gain value required. If the acquired gain is above the threshold value, we can split the node, otherwise, leave it as a leaf node.
The following are the take-aways from this article
The general concept behind decision trees.The basic types of decision trees.Different algorithms to build a Decision tree.Building a Decision tree using CART algorithm.Building a Decision tree using ID3 algorithm.
The general concept behind decision trees.
The basic types of decision trees.
Different algorithms to build a Decision tree.
Building a Decision tree using CART algorithm.
Building a Decision tree using ID3 algorithm.
Refer to this playlist on youtube for more details on building Decision trees using CART algorithm.
Refer to this playlist on youtube for more details on building Decision trees using CART algorithm.
2. Refer to this playlist on youtube for more details on building Decision trees using ID3 algorithm.
PS:- I will be posting another article regarding Regression trees and Random Forests soon. Stay tuned :)
Do check out my other articles regarding Data Science and Machine Learning here. Feel free to reach out for a more in-depth discussion in the comments and on LinkedIn.