Decision Tree Algorithm in Python From Scratch | by Eligijus Bujokas | Towards Data Science

The aim of this article is to make all the parts of a decision tree classifier clear by walking through the code that implements the algorithm. The code uses only NumPy, Pandas and the standard python libraries.
The full code can be accessed via https://github.com/Eligijus112/decision-tree-python
As of now, the code creates a decision tree when the target variable is binary and the features are numeric. This is completely sufficient to understand the algorithm.
The golden standard of building decision trees in python is the scikit-learn implementation:
scikit-learn.org
When I tested out my code I wanted to make sure that the results are identical to the scikit-learn implementation.
The data used in this article is the famous Titanic survivor dataset. We will use two numeric variables — Age of the passenger and the Fare of the ticket — to predicting whether a passenger survived or not.
The goal is to create the “best” splits of the numeric variables. Just eyeballing the data, we could guess that one good split is to split the data into two parts: observations that have Age < 10 and observations that have Age ≥ 10:
Now some immediate questions may rise:
Is this a good split?
Maybe a split at the Fare = 200 is a better one?
How do we quantify the “goodness” of a split?
How does a computer search for the best split?
All of these questions will be answered by the end of this article.
A decision tree algorithm (DT for short) is a machine learning algorithm that is used in classifying an observation given a set of input features. The algorithm creates a set of rules at various decision levels such that a certain metric is optimized.
The target variable will be denoted as Y = {0, 1} and the feature matrix will be denoted as X.
Keywords to expand on:
Node
Gini impurity (a metric which we are optimizing)
Level
Splitting
A node is the building block in the decision tree. When viewing a typical schema of a decision tree (like the one in the title picture) the nodes are the rectangles or bubbles that have a downward connection to other nodes.
Each node has the following main attributes:
Gini impurity score
Number of observations
The number of observations belonging to each of the binary target classes.
The feature matrix X representing the observations that fall into the node.
The custom Node class in python (written by me):
The first node in a decision tree is called the root. The nodes at the bottom of the tree are called leaves.
If splitting criteria are satisfied, then each node has two linked nodes to it: the left node and the right node. For example, a very simple decision tree with one root and two leaves may look like this:
n — number of observationsy1 — number of first class elementsy2 — number of second class elements X — numeric feature for the observations
An input if the feature X less than 5 would go to the left node. A feature value of more or equal to 5 would go to the right node.
As mentioned above, each node has a GINI impurity score. To calculate the GINI impurity, all that is needed is the distribution of the target variable in the node or simply, how many Y=1 and Y=0 observations there are in a node.
The formal definition of GINI impurity is as follows:
Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labelled if it was randomly labelled according to the distribution of labels in the subset.
The algebraic definition:
For example, if we have in a node 10 observations that are survivors and 5 observations that are not survivors, then:
p(survivors) = 10 /(10 + 5) = 0.66..p(non-survivors) = 5 /(10 + 5) = 0.33..
Thus the GINI impurity can be calculated by squaring the two numbers, adding them up and subtracting from one:
gini impurity = 1 - (0.66..^2 + 0.33..^2) = 0.44.. 
In a binary case, the maximum Gini impurity is equal to 0.5 and the lowest is 0. The lower the value, the more “pure” the node is. No matter how many observations we have, we can calculate the share of one of the classes in terms of the whole dataset and draw a relationship:
We always search for a split where the GINI impurity will be the lowest.
If nothing else, the basic intuition is that the more one class observations there is in a node, the lower its impurity.
The level attribute defines how many splits were made before the node was created. So for example, the root node’s level is 0, then the left and the right nodes have a level of 1, and so on.
In the custom Node class, the maximum depth of the tree can be regulated with the hyperparameter max_depth.
The splitting procedure is a process where we search in each node which feature and which feature value is the best to divide the data into two smaller parts.
In the classification case, we want to maximize the Gini gain. Going back to the example before:
The root node has a Gini score of 0.48. The left node has a score of 0.27 and the right node has 0.5. How much Gini did we “gain”?
The formula to calculate this is:
Gini gain = Parent node Gini impurity subtracted by the weighted average of the Gini impurities of the left and right nodes.
In the example above, plugging in all the values we would get:
0.48 — (0.27 * 30 /100 + 0.5 * 70/100) = 0.049
The GINI gain is equal to 0.049. Any positive Gini gain is an improvement. This means that our decision would make the nodes more “pure”.
How does the algorithm search for the best split among numeric columns?
For each of the feature, we sort the feature values and get the means of two neighbouring values.
For example, suppose our feature_1 is the following:
feature_1 = [7, 5, 9, 1, 2, 8]
Sorted:
feature_1 = [1, 2, 5, 7, 8, 9]
Means of neighbours:
feature_1 = [1.5, 3.5, 6.5, 7.5, 8.5]
We then check what is the GINI gain with each of the values from the above vector. And, we do this for all the features in our dataset.
The final split value and split feature is the one that has the highest GINI gain.
The hyperparameters of the custom Node object that grows the tree (more will be added in the future) are the max_depth: int and min_samples_split: int variables.
The max_depth integer defines how deep should the tree grow. At the depth of max_depth, the searching for the best split feature and split feature values stops.
The min_samples_split integer defines the minimal number of observations in a node for the best split search to start. So for example, if the node has 51 observations but the min_samples_split = 55, then the growth of the tree stops.
So, how does the code work?
First of all, read the data:
# Loading datad = pd.read_csv(‘data/train.csv’)# Dropping missing valuesdtree = d[[‘Survived’, ‘Age’, ‘Fare’]].dropna().copy()# Defining the X and Y matricesY = dtree[‘Survived’].valuesX = dtree[[‘Age’, ‘Fare’]]# Saving the feature list features = list(X.columns)
Then we define the dictionary of hyperparameters.
hp = { ‘max_depth’: 3, ‘min_samples_split’: 50}
Then we initiate the root node:
root = Node(Y, X, **hp)
The main tree building function is the grow_tree() function.
root.grow_tree()
And that's it!
To view the results, we can invoke the print_tree() function.
root.print_tree()
The results:
The decision tree obtained from the scikit-learn implementation is identical:
|--- Fare <= 52.28|   |--- Fare <= 10.48|   |   |--- Age <= 32.50|   |   |   |--- class: 0|   |   |--- Age >  32.50|   |   |   |--- class: 0|   |--- Fare >  10.48|   |   |--- Age <= 6.50|   |   |   |--- class: 1|   |   |--- Age >  6.50|   |   |   |--- class: 0|--- Fare >  52.28|   |--- Age <= 63.50|   |   |--- Age <= 29.50|   |   |   |--- class: 1|   |   |--- Age >  29.50|   |   |   |--- class: 1|   |--- Age >  63.50|   |   |--- class: 0
Although, the scikit-learn implementation prints out less information than my implementation.
As it turns out, the best first initial split is the Fare feature at a value of 52.28 and not at the proposed Age feature at value 10.
The code that I have written builds the same trees as scikit-learn implementation and the predictions are the same. But the training time for the scikit-learn algorithm is much faster. But my goal was not to grow the trees faster. My goal was to write an understandable code for any machine learning enthusiast to have a better grasp of what is happening under the hood.
Feel free to create a pull request in this repo https://github.com/Eligijus112/decision-tree-python if you see any bugs or just want to add functionalities.