Understanding Decision Trees
By: Vikas Solegaonkar
Tags: Decision Tree, Machine Learning, Statistical Learning

Decision Tree is an interesting concept that mimics a very common way our mind approaches a classification problem — of taking decisions one step at a time.

Suppose we want to predict the outcome of a cricket match, we would typically check out the star players; the other players; the stadium; pitch, crowd; season etc.. in an appropriate order. At each step, we consider if that factor is pro-victory or pro-defeat. Such a binary walk takes us to a reasonable prediction of the outcome.

This is the decision tree algorithm — we build a tree of decisions that lead us to the final decision.

This is different from the process of logistic regression — where we would try to come up to a number based on the different parameters, and then validate if the number is more or less than a threshold. Our mind does not always follow that process.

(Source Wikimedia)

In Mathematical Terms

We have two ways of looking at a machine learning problem. We can build a unified model for our training data set of n features. Or we can take up one feature at a time and classifying data elements of that feature — the two nodes thus obtained can be classified further based on the next feature and so on.

The latter gives us a tree structure — where each node represents a given data feature. As we perform this classification, we expect to reach a state where each node has true on one side and false on the other — thus completing our model training.

As one can see, the whole game is about identifying the order of features to split, and the split thresholds. Thankfully, we have strong libraries to help us with that. If you are interested in its details, refer to the section below, on “Understanding the Decision Tree”. But before that, let us get a feel by trying to implement stuff in code.

Implementation

Like most machine learning algorithms, you have a simple solution for the Decision Tree from SciKitLearn. You can work on the Iris flower database as below.

Start with the usual imports:

Now load the iris data set and split it into the train and test

Next, instantiate the Decision Tree classifier and try to fit the data into the model

Now, evaluate the model using the test set.

This is a clear example of over fitting. But, the test accuracy if 0.973 is not bad either; so we take this a good classification.

Understanding the Decision Tree

In order to understand the decision tree, let us try to implement one from scratch.

On a high level. a Decision Tree implementation can be thought of a simple if/else logic. Of course, we have important hyper-parameters — depth of the tree, the choice of order of features and the thresholds for each decision. There are several algorithms for identifying these. But, once they are chosen, the classification is not a difficult task.

Over the years, researchers have identified several algorithms that can help us with this choice.

Hunt’s algorithm

This is the most adopted for identifying the order of features. In essence, it tries to split the data over different combinations of features — to identify the one with highest correlation with the decision. It helps identify the features that impact the outcome more than the others. It makes sense to classify on these features before others.

Max Tree Depth

This is a user specified hyper parameter. It defines the maximum depth to which we grow the decision tree. Deeper the tree, more its complexity, and more the chances of overfitting. The leaf nodes in a tree are also called the Terminal Nodes.

Maximum Node Records

This is another indicator of the granularity of the decision tree. We should stop growing the tree at a given node, if the number of records there fall below this threshold. The point is that we should have enough data at the given node, so that we can train that node well.

Recursive Splitting

Having defined the process and limits for splitting the records on a node, we can recursively go on splitting the data records — growing the tree into more and more nodes

Gini Index

For choosing the individual thresholds on each of the nodes, we do not follow the typical regression process. But it is the same in essence — trying to reduce an error function.

The natural question here is, what is the error function? What do we try to minimise? The most popular among the different cost functions defined is the Gini Index — that is a measure of the amount of split on each node — Is there a meaningful difference between the data sets on either side of the split?

Gini Index score at a node is defined as: One minus the square of amount of partition at a node. If we have a perfect partition; if all elements of either class are on either side of the split, the Gini index is 0. Higher value means the data did not split well.

Code

Enough of theory, now we can get down to some real code. Let’s implement each of the components of the decision tree in simple Python code and then we can put them all together

Gini Index

Let us start by building python code to calculate the Gini Index for a set of groups and the classes.

Here, we loop through each row in each group. For each, we check the second element in the pair (the value), to find out the ratio of records with a given outcome on a given side of the split. This is followed by the arithmetic to aggregate this into the value of Gini Index.

Create a Data Split

The code to evaluate Gini Index is ready. Now, we move ahead, to identify the best threshold to split at the given node. Essentially, we try out several values in the available range. Among these, we pick up the one with minimum value for Gini Index.

We do not need to go through the regression at this point — because we pick up one feature at a time, and we are not dealing with continuous functions. We scan through the data set, to find the particular value in the data, that can can best serve as the point to split.

We can extend this to include the hyper-parameters we discussed above. Thus, we can create a method to split at a given node.

This recursively splits the nodes till each branch reaches the criteria for being the terminal node.

Construct the Decision Tree

Now we have all the building blocks required, and we can go ahead to build the decision tree. That is a simple task.

We can pass in the training data along with the hyper-parameters of max depth and minimum size of nodes. The code will go recursively through the above logic to construct and train the decision tree.

Prediction

With the tree in place, we can go ahead to test it with a prediction.

This method recursively drills down the tree to the terminal node that can give us the decision.