Decision Trees and Random Forests — Explained
By: Soner Yıldırım
Tags: Data Science, Machine Learning, Random Forest, Decision Tree

Decision Trees and Random Forests — Explained

Decision trees and random forests are supervised learning algorithms used for both classification and regression problems. These two algorithms are best explained together because random forests are a bunch of decision trees combined. There are ofcourse certain dynamics and parameters to consider when creating and combining decision trees. In this post, I will explain how decision trees and random forests work as well as the critical points to consider when using these models.

A decision tree builds upon iteratively asking questions to partition data. It is easier to conceptualize the partitioning data with a visual representation of a decision tree:

This represents a decision tree to classifiy animals. First split is based on the size of animal. Although the question seems to be “How big is the animal?”, it is asked in the form of “Is the animal bigger than 1m?” because we want to split the data points in two groups at each step. The questions get more specific as the tree gets deeper.

What you ask at each step is the most critical part and greatly influences the performance of decision trees. For example, assume your dataset has “feature A” ranging from 0 to 100 but most of the values are above 90. In this case, the first question to ask is “Is feature A more than 90?”. It does not make sense to ask “Is feature A more than 50?” because it will not give us much information about the dataset.

Our aim is to increase the predictiveness of the model as much as possible at each partitioning so that the model keeps gaining information about the dataset. Randomly splitting the features does not usually give us valuable insight about the dataset. Splits that increase purity of nodes are more informative. The purity of a node is inversely proportional to the distribution of different classes in that node. The questions to ask are chosen in a way that increases purity or decrease impurity.

There are two ways to measure the quality of a split: Gini Impurity and Entropy. They essentially measure the impurity and give similar results. Scikit-learn uses gini index by default but you can change it to entropy using criterion parameter.

Gini Impurity

As stated on wikipedia, “Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset”. It basically means that impurity increases with randomness. For instance, let’s say we have a box with ten balls in it. If all the balls are same color, we have no randomness and impurity is zero. However, if we have 5 blue balls and 5 red balls, impurity is 1.

Entropy and Information Gain

Entropy is a measure of uncertainty or randomness. The more randomness a variable has, the higher the entropy is. The variables with uniform distribution have the highest entropy. For example, rolling a fair dice has 6 possible outcomes with equal probabilities so it has a uniform distribution and high entropy.

Entropy vs Randomness

Splits that result in more pure nodes are chosen. All these indicate “information gain” which is basically the difference between entropy before and after the split.

When choosing a feature to split, decision tree algorithm tries to achieve