The Basics of Decision Trees
By: Arif R
Tags: Machine Learning, Artificial Intelligence, Decision Tree Classifier, Decision Tree, Decision Tree Regressor

Decision Tree Algorithms - Part 1

1. Introduction

Decision Trees is the non-parametric supervised learning approach, and can be applied to both regression and classification problems. In keeping with the tree analogy, decision trees implement a sequential decision process. Starting from the root node, a feature is evaluated and one of the two nodes (branches) is selected, Each node in the tree is basically a decision rule. This procedure is repeated until a final leaf is reached, which normally represents the target. Decision trees are also attractive models if we care about interpretability.

2. Various Decision Tree Algorithms

There are algorithms for creating decision trees :

ID3 (Iterative Dichotomiser 3) was developed in 1986 by Ross Quinlan. The algorithm creates a multiway tree, finding for each node (i.e. in a greedy manner) the categorical feature that will yield the largest information gain for categorical targets. Trees are grown to their maximum size and then a pruning step is usually applied to improve the ability of the tree to generalise to unseen data [1].

was developed in 1986 by Ross Quinlan. The algorithm creates a multiway tree, finding for each node (i.e. in a greedy manner) the categorical feature that will yield the largest information gain for categorical targets. Trees are grown to their maximum size and then a pruning step is usually applied to improve the ability of the tree to generalise to unseen data [1]. C4.5 was developed in 1993 by Ross Quinlan, is the successor to ID3 and removed the restriction that features must be categorical by dynamically defining a discrete attribute (based on numerical variables) that partitions the continuous attribute value into a discrete set of intervals. C4.5 converts the trained trees (i.e. the output of the ID3 algorithm) into sets of if-then rules. These accuracy of each rule is then evaluated to determine the order in which they should be applied. Pruning is done by removing a rule’s precondition if the accuracy of the rule improves without it [1].

was developed in 1993 by Ross Quinlan, is the successor to ID3 and removed the restriction that features must be categorical by dynamically defining a discrete attribute (based on numerical variables) that partitions the continuous attribute value into a discrete set of intervals. C4.5 converts the trained trees (i.e. the output of the ID3 algorithm) into sets of if-then rules. These accuracy of each rule is then evaluated to determine the order in which they should be applied. Pruning is done by removing a rule’s precondition if the accuracy of the rule improves without it [1]. C5.0 is Quinlan’s latest version release under a proprietary license. It uses less memory and builds smaller rulesets than C4.5 while being more accurate [1].

is Quinlan’s latest version release under a proprietary license. It uses less memory and builds smaller rulesets than C4.5 while being more accurate [1]. CART (Classification and Regression trees) is very similar to C4.5, but it differs in that it supports numerical target variables (regression) and does not compute rule sets. CART constructs binary trees using the feature and threshold that yield the largest information gain at each node [1].

scikit-learn uses an optimised version of the CART algorithm

3. Decision Tree Terminology

In keeping with the tree analogy, the terminology was adopted from the terminology of the tree [2].

Root node : is the first node in decision trees

: is the first node in decision trees Splitting : is a process of dividing node into two or more sub-nodes, starting from the root node

: is a process of dividing node into two or more sub-nodes, starting from the root node Node : splitting results from the root node into sub-nodes and splitting sub-nodes into further sub-nodes

: splitting results from the root node into sub-nodes and splitting sub-nodes into further sub-nodes Leaf or terminal node : end of a node, since node cannot be split anymore

: end of a node, since node cannot be split anymore Pruning : is a technique to reduce the size of the decision tree by removing sub-nodes of the decision tree. The aim is reducing complexity for improved predictive accuracy and to avoid overfitting

: is a technique to reduce the size of the decision tree by removing sub-nodes of the decision tree. The aim is reducing complexity for improved predictive accuracy and to avoid overfitting Branch / Sub-Tree : A subsection of the entire tree is called branch or sub-tree.

: A subsection of the entire tree is called branch or sub-tree. Parent and Child Node: A node, which is divided into sub-nodes is called parent node of sub-nodes whereas sub-nodes are the child of parent node.

4. Decision Tree Intuition

Let’s consider the following example where a decision tree to decide tree to predict an salary on a baseball player case :

We use the Hitters data set to predict a baseball player’s Salary (mean log salary) based on Years (the number of years that he has played in the major leagues) and Hits (the number of hits that he made in the previous year).

Based on the features, the decision tree model learns a series of splitting rules, starting at the top of the tree (root node).

The root node split into sub-node with observation rule having Years <4.5 to the left branch, which means the players in dataset with Years<4.5 having mean log salary is 5.107 and we make a prediction of e5.107 thousands of dollars, i.e. $165,174 for these players Players with Years>=4.5 are assigned to the right branch and then that group is further subdivided by Hits < 177.5 having mean los salary of 6. Players with Years>=4.5 are assigned to the right branch and then that group is further subdivided by Hits >= 177.5 having mean los salary of 6.74

In this case, it can be seen that the decision tree makes a segment into three regions where this region determines the salaries of baseball players and it can be said that the region is a decision boundary [3].

These three regions can be written as

R1 ={X | Years<4.5 }

={X | Years<4.5 } R2 ={X | Years>=4.5 ,Hits<117.5 }

={X | Years>=4.5 ,Hits<117.5 } R3 ={X | Years>=4.5 , Hits>=117.5 }.

From this intuition there is a process, how a decision tree splitting features to form a region that can predict salary of baseball players. This process will be explained in more detail in the next article (Decision Tree Algorithms - Part 2).

5. Splitting in Decision Trees

In order to split the nodes at the most informative features using the decision algorithm, we start at the tree root and split the data on the feature that results in the largest information gain (IG). Here, the objective function is to maximize the information gain (IG) at each split, which we define as follows:

f is the feature to perform the split, Dp and Dj are data set of the parent, j-th child node, I is our impurity measure, Np is the total number of samples at the parent node, and Nj is the number of samples in the j-th child node.

As we can see, the information gain is simply the difference between the impurity of the parent node and the sum of the child node impurities — the lower the impurity of the child nodes, the larger the information gain. however, for simplicity and to reduce the combinatorial search space, most libraries (including scikit-learn) implement binary decision trees. This means that each parent node is split into two child nodes, D-left and D-right.

impurity measure implements binary decisions trees and the three impurity measures or splitting criteria that are commonly used in binary decision trees are Gini impurity (IG), entropy (IH), and misclassification error (IE) [4]

5.1 Gini Impurity

According to Wikipedia [5],

Used by the CART (classification and regression tree) algorithm for classification trees, Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.

Mathematically, we can write Gini Impurity as following

where j is the number of classes present in the node and p is the distribution of the class in the node.

Simple simulation with Heart Disease Data set with 303 rows and has 13 attributes. Target consist 138 value 0 and 165 value 1

In order to build a decision tree from the dataset and to determine which separation is best, we need a way to measure and compare Gini Impurity in each attribute. The lowest Gini Impurity value on the first iteration will be the Root Node. we can write equation 3 as :

In this simulation, only use the sex, fbs (fasting blood sugar), exang (exercise induced angina), and target attributes.

How to measure Gini Impurity in Sex attribute

Gini Impurity — Left Node

Gini Impurity — Right Node

Now that we have measured the Gini Impurity for both leaf nodes. We can calculate the total Gini Impurity with weight average. Left Node represented 138 patient while Right Node represented 165 patient

Total Gini Impurity — Leaf Node

How to measure Gini Impurity in Fbs (fasting blood sugar) attribute

Gini Impurity — Left Node

Gini Impurity — Right Node

Total Gini Impurity — Leaf Node

How to measure Gini Impurity in Exang (exercise induced angina) attribute

Gini Impurity — Left Node

Gini Impurity — Right Node

Total Gini Impurity — Leaf Node

Fbs (fasting blood sugar) has the lowest Gini Impurity, so well use it at the Root Node

5.2 Entropy

Used by the ID3, C4.5 and C5.0 tree-generation algorithms. Information gain is based on the concept of entropy, the entropy measure is defined as [5]:

where j is the number of classes present in the node and p is the distribution of the class in the node.

In the same case and same data set, we need a way to measure and compare Entropy in each attribute. The highest Entropy value on the first iteration will be the Root Node.

We need calculate entropy in Target attribute first

How to measure Entropy in Sex attribute

Entropy — Sex = 0

Entropy — Sex = 1

Now that we have measured the Entropy for both leaf nodes. We take the weight average again to calculate the total entropy value.

Entropy — Sex

How to measure Entropy in Fbs attribute

Entropy — Fbs = 0

Entropy — Fbs = 1

Entropy — Fbs

How to measure Entropy in Exang attribute

Entropy — Exang = 0

Entropy — Exang = 1

Entropy — Exang

Fbs (fasting blood sugar) has the highest gini impurity, so we will use it at the Root Node, Precisely the same results we got from Gini Impurity.

5.3 Misclassification Impurity

Another impurity measure is the misclassification impurity , Mathematically, we can write misclassification impurity as following

In terms of quality performance, this index is not the best choice because it’s not particularly sensitive to different probability distributions (which can easily drive the selection to a subdivision using Gini or entropy)[6].

6. Advantages and Disadvantages of Trees

Advantages

Trees are very easy to explain to people. In fact, they are even easier

to explain than linear regression! Some people believe that decision trees more closely mirror human

decision-making than do the regression and classification approaches

seen in previous chapters. Trees can be displayed graphically, and are easily interpreted even by

a non-expert (especially if they are small). Trees can easily handle qualitative predictors without the need to

create dummy variables[3].

Disadvantages