How are decision trees built?
By: Zolzaya Luvsandorj
Tags: Data Science, Supervised Learning, Machine Learning, Decision Tree, Classification

2. Build a classification tree 🔨

If you enjoy math, I encourage you to manually calculate alongside this guide to make most of this blog. In this section, those characters who are parents are abbreviated as pa and the non-parents are abbreviated as np for brevity.

2.1. Using gini impurity (a.k.a gini index or gini coefficient) ☝️

Decision trees are built by recursively splitting into binary nodes from top-down. We can find the best split for a node with the following steps:

STEP 1: Calculate gini impurity (here onwards gini) for the node to split from

STEP 2: Find all possible splits

STEP 3: Calculate gini for both nodes for each split

STEP 4: Calculate the weighted average gini for each split

STEP 5: Determine the best split: the one with lowest weighted average gini

STEP 6: Calculate information gain: split if information gain is positive

The very top node that includes everyone from the training data is known as root node. Let’s determine the best split for the root node with the steps.

🚪 STEP 1: Calculate the gini at a root node

➗ Formula:

two variations of the formula will give exact same result

We can simplify this generic formula to the following and calculate the gini:

this simplfied formula will be used here onwards

We know that there were 2 non-parents and 4 parents among the 6 characters. Using this information, we found the gini to be 0.444 at the root node. Root node can therefore be summarised as below:

🚪 STEP 2: Find all possible splits from the root node

There are three ways to split using any of the three features. For instance, we could split the 6 characters into 2 groups: one group for those with pets and another group for those without. Same goes for the other two features.

🚪 STEP 3: Calculate gini for both nodes for each split

Let’s calculate the gini for each node for the three splits.

📌Exercise: See if you can calculate gini for all 6 nodes before proceeding.

🔑 Answer: Using the same logic from step 1, we find the gini for each node as follows:

Hope these tiny figures are legible 👀 (if too small, try zooming in)

🚪 STEP 4: Calculate the weighted average gini for each split

Now, we need to find the weighted average gini, denoted as w_gini, for each split. Taking was_on_a_break as an example, we compute the w_gini as follows:

➗ Formula: left: node of the left, right: node on the right

📌Exercise: See if you can calculate the weighted gini for the other two.

🔑 Answer: Do your answers match to these?

🚪 STEP 5: Determine the best split (lowest weighted average gini)

By inspecting the results from the previous step, using is_married for splitting gives us the lowest weighted average gini. We can intuitively make sense of this decision if we look at the table from step 3 as well. When splitting on is_married, the tree is able to split half of the data into a pure node on the left. This node is pure because it only contains parents. Gini is 0 for the purest node and 0.5 for the impurest node (e.g. right nodes for the other splits).

🚪 STEP 6: Calculate information gain: if positive, 👉 split

We learnt that it’s best to use is_married if we were to split. Now let’s see if we gain any information from splitting. Information gain is defined as the difference between the gini at the top node to split from and the weighted average gini from the bottom nodes.

Information gain is positive. In other words, we gain information by splitting. Therefore, the correct decision is to split using is_married from the root node.

We have just learned how a split is determined. ⭐️ Let’s repeat the same steps to build the rest of the tree. Time to assess the split for the mustard nodes!

Perhaps we could start from the easier one: the left node containing married ones. We see that all married characters are parents and therefore gini=0. If we split from this node, we won’t gain any information even if the weighted gini is 0 from the split. Therefore, the correct decision is not to split. In this case, this node is considered as a terminal node, the one that doesn’t split further. On the other hand, we have some work to do for the non-married characters in the other node on the right. The best way to make this knowledge stick is to practice yourself, why not try applying what we just learned with the exercise below:

📌Exercise: Complete all the steps to find the correct split for the right node

🔑 Answer:

STEP 1: We already know the answer from previous split: 0.444

STEP 2: We could split either using was_on_a_break or has_pet

STEP 3 & STEP 4: See the image below

STEP 5: The best split is to use was_on_a_break with w_gini of 0

STEP 6: Information gain is 0.444 - 0=0.444, so the correct decision is to split.

Combining the outputs, the final decision tree looks like this:

Root node: A starting node that includes everyone

Internal nodes: All other nodes in between root and terminal nodes.

Terminal nodes: Nodes where decisions are reached

Yay❕ We have built a simple decision tree.

2.2. Using entropy ✌️

Let’s understand what would change if we used entropy instead of gini. The steps remain the same, except we calculate entropy each time.

➗ Formula:

We can simplify this generic formula to:

If we complete all the steps using entropy starting from the root node again, the outputs for steps 1, 3 and 4 change to those below:

With this output, it appears that is_married is yet again the best split from the root node. Information gain is assessed in a similar way:

It makes sense to split given that we gain information. Let’s continue the same steps for the bottom nodes like before.

Because the left node for married characters are pure, we don’t need to split from that anymore. But we will try to improve the right node by following the steps. Outputs to steps 1, 3 and 4 would look like below:

This is the final tree 🌴:

Did you notice that gini is 0.5 and entropy is 1 for the impurest case where the node is split evenly between the two classes whereas they are both 0 for the purest node that consists of only one class?

If you are keen to practice more to consolidate your learning, feel free to use your own small dataset to build a simple decision tree. You can check your tree against sklearn output using the sample script below:

Don’t forget to replace the data with the one you are going to use

3. Final remarks 💭

In practice, more robust algorithms that use decision tree as a building block are perhaps more commonly used as a predictive model than decision trees themselves.

Decision trees tend to overfit easily if you are not careful. We could argue that was_on_a_break was not a good feature to use because it happened to be a very specific feature that only applies to the records in the training data. Therefore, using this feature to build a model can lead the model overfit to the noise in the training data.

Regardless, I think it is still valuable to understand the underlying principles used in building decision trees. ✨