Random Forest Algorithm in Python from Scratch | by Eligijus Bujokas | Towards Data Science

This article aims to demystify the popular random forest (here and throughout the text — RF) algorithm and show its principles by using graphs, code snippets and code outputs.
The full implementation of the RF algorithm written by me in python can be accessed via: https://github.com/Eligijus112/decision-tree-python
I highly encourage anyone who stumbled upon this article to dive deep into the code because the understanding of the code will make any future documentation reading about RF much more straightforward and less stressful.
Any suggestions about optimizations are highly encouraged and are welcomed via a pull request on GitHub.
The building blocks of RF are simple decision trees. This article will be much easier to read if the reader is familiar with the concept of a classification decision tree. It is highly recommended to go through the following article before going any further:
towardsdatascience.com
Scikit-learn in python has an implementation of the RF algorithm which is fast and reviewed hundreds of times:
scikit-learn.org
In the documentation, the first hyperparameter which needs to be defined is the n_estimators parameter with a default value of 100. The explanation of this parameter is very elegant:
The number of trees in the forest.
This explanation is very romantic but a stubborn mathematician reading for the first time may find this a little bit puzzling: what kind of forest and what kind of trees are we talking about?
The intuition behind the random forest algorithm can be split into two big parts: the random part and the forest part. Let us start with the latter.
A forest in real life is made up of a bunch of trees. A random forest classifier is made up of a bunch of decision tree classifiers (here and throughout the text — DT). The exact amount of DTs that make up the whole forest is defined with the n_estimators variable mentioned earlier.
Each DT in an RF algorithm is completely independent of one another. If the n_estimators variable is set to 50 then the forest is made up of 50 decision trees that were grown completely independently from one another and do not share any information.
A binary classification decision tree can be viewed as a function that takes input X and outputs either 1 or 0:
DT: X → {0, 1}
The final prediction of RF is a majority vote of the predictions made from each individual DT.
If out of 50 trees, 30 trees label a new observation as a 1 and 20 trees label the same observation as a 0 the final prediction of the random forest will be 1.
In the article about simple classification decision trees it is clear that with the same input and the same hyperparameters, the same output and the same rules will be learnt by a decision tree. So why grow 50 (or 100, or 1000, or k) of them? This is where the second part of the RF intuition comes in: the random part.
The random part in RF can be split into two parts.
The first part is the bootstrapping of data. A data bootstrap is a subsample of its rows with replacement. The part of the python implementation that creates a bootstrapped sample:
For example, if the whole dataset d were 10 rows:
The response variable is the Churn column and the three other columns are the features.
Three independent bootstrapped datasets of d might look like:
As we can see from the picture above, in the first sample, the 5th and the 8th rows are the same. Additionally, not all original observations are present.
If we define that the RF has k decision trees (n_estimators = k) then there will be k different bootstrapped datasets created and each tree will be grown with a different dataset. Each dataset can have the same number of rows as the original or can have fewer rows than the original dataset.
Thus if RF is made of 50 decision trees then the high-level graph of the RF would look like this:
Each of the fifty decision trees would be grown with a unique dataset.
The second part of random in the random forest is in the training phase. When fitting a classification decision tree on data, during each of the splitting choices, the features stays the same.
During a random forest classifier creation, each decision tree is grown a bit differently than in the algorithm implemented here
https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier
The only difference is that during each of the split creation moments, a random subsample of features is drawn and the maximum Gini gain is calculated with those features.
https://gist.github.com/Eligijus112/4c47be2f7566299bb8c4f97c107d82c6f
For example, let us assume that we have a bootstrapped data sample d and X matrix with 100 initial features. We can define how many features to leave during each splitting phase with the argument X_features_fraction (max_features in scikit-learn implementation). Let us set it to 0.05, meaning, that at each split, 5 random features (or 5%) will be selected for the split.
In the first node, there are five X features: 1, 25, 28, 30 and 98. The best split is found when splitting the 25th feature by value x. The two below nodes have another 5 random features where the best split is searched for.
Thus, the RF algorithm is as follows:
Select a set of hyperparameters.
For 1 to k trees do:
Generate a random bootstrapped data sample d.
Fit a decision tree on the data d. During each of the split during the growth stage, select a random subsample of features to split on.
When forecasting, get the prediction of each of the k trees and then select the label using a majority vote.
Let's fit an RF classifier on some real data. The full notebook and data can be accessed via:
https://github.com/Eligijus112/decision-tree-python/blob/main/RandomForestShowcase.ipynb
The dataset which is in my GitHub repository has 3333 rows and we will be using the following columns:
The objective is to create a classifier that predicts whether a telecom customer will churn or not. Churn is an event where a customer no longer uses a companies product. If churn = 1 it means that the user has switched to a different service provider or for any other reason shows no activity.
Let's split the data into train and test splits. The training data will have 75% of the rows (2500) and the test set will have 25% of the rows (833).
Growing 5 decision trees with 25% of features in every split and each tree having a depth of 2 we get the following forest:
------ Tree number: 1 Root   | GINI impurity of the node: 0.27   | Class distribution in the node: {0: 2099, 1: 401}   | Predicted class: 0|-------- Split rule: DayCalls <= 113.5           | GINI impurity of the node: 0.24           | Class distribution in the node: {0: 1598, 1: 257}           | Predicted class: 0|---------------- Split rule: DayMins <= 287.7                   | GINI impurity of the node: 0.21                   | Class distribution in the node: {0: 1591, 1: 218}                   | Predicted class: 0|---------------- Split rule: DayMins > 287.7                   | GINI impurity of the node: 0.26                   | Class distribution in the node: {1: 39, 0: 7}                   | Predicted class: 1|-------- Split rule: DayCalls > 113.5           | GINI impurity of the node: 0.35           | Class distribution in the node: {0: 501, 1: 144}           | Predicted class: 0|---------------- Split rule: DayMins <= 225.0                   | GINI impurity of the node: 0.25                   | Class distribution in the node: {0: 431, 1: 76}                   | Predicted class: 0|---------------- Split rule: DayMins > 225.0                   | GINI impurity of the node: 0.5                   | Class distribution in the node: {1: 68, 0: 70}                   | Predicted class: 0------ ------ Tree number: 2 Root   | GINI impurity of the node: 0.26   | Class distribution in the node: {0: 2124, 1: 376}   | Predicted class: 0|-------- Split rule: OverageFee <= 13.235           | GINI impurity of the node: 0.24           | Class distribution in the node: {0: 1921, 1: 307}           | Predicted class: 0|---------------- Split rule: DayMins <= 261.45                   | GINI impurity of the node: 0.18                   | Class distribution in the node: {0: 1853, 1: 210}                   | Predicted class: 0|---------------- Split rule: DayMins > 261.45                   | GINI impurity of the node: 0.48                   | Class distribution in the node: {0: 68, 1: 97}                   | Predicted class: 1|-------- Split rule: OverageFee > 13.235           | GINI impurity of the node: 0.38           | Class distribution in the node: {1: 69, 0: 203}           | Predicted class: 0|---------------- Split rule: DayMins <= 220.35                   | GINI impurity of the node: 0.13                   | Class distribution in the node: {0: 186, 1: 14}                   | Predicted class: 0|---------------- Split rule: DayMins > 220.35                   | GINI impurity of the node: 0.36                   | Class distribution in the node: {1: 55, 0: 17}                   | Predicted class: 1------ ------ Tree number: 3 Root   | GINI impurity of the node: 0.25   | Class distribution in the node: {1: 366, 0: 2134}   | Predicted class: 0|-------- Split rule: DataUsage <= 0.315           | GINI impurity of the node: 0.29           | Class distribution in the node: {1: 286, 0: 1364}           | Predicted class: 0|---------------- Split rule: MonthlyCharge <= 62.05                   | GINI impurity of the node: 0.18                   | Class distribution in the node: {1: 144, 0: 1340}                   | Predicted class: 0|---------------- Split rule: MonthlyCharge > 62.05                   | GINI impurity of the node: 0.25                   | Class distribution in the node: {1: 142, 0: 24}                   | Predicted class: 1|-------- Split rule: DataUsage > 0.315           | GINI impurity of the node: 0.17           | Class distribution in the node: {0: 770, 1: 80}           | Predicted class: 0|---------------- Split rule: RoamMins <= 13.45                   | GINI impurity of the node: 0.12                   | Class distribution in the node: {0: 701, 1: 49}                   | Predicted class: 0|---------------- Split rule: RoamMins > 13.45                   | GINI impurity of the node: 0.43                   | Class distribution in the node: {0: 69, 1: 31}                   | Predicted class: 0------ ------ Tree number: 4 Root   | GINI impurity of the node: 0.26   | Class distribution in the node: {0: 2119, 1: 381}   | Predicted class: 0|-------- Split rule: DayCalls <= 49.5           | GINI impurity of the node: 0.49           | Class distribution in the node: {1: 8, 0: 6}           | Predicted class: 1|---------------- Split rule: MonthlyCharge <= 31.5                   | GINI impurity of the node: 0.0                   | Class distribution in the node: {0: 4}                   | Predicted class: 0|---------------- Split rule: MonthlyCharge > 31.5                   | GINI impurity of the node: 0.32                   | Class distribution in the node: {1: 8, 0: 2}                   | Predicted class: 1|-------- Split rule: DayCalls > 49.5           | GINI impurity of the node: 0.26           | Class distribution in the node: {0: 2113, 1: 373}           | Predicted class: 0|---------------- Split rule: DayMins <= 264.6                   | GINI impurity of the node: 0.21                   | Class distribution in the node: {0: 2053, 1: 279}                   | Predicted class: 0|---------------- Split rule: DayMins > 264.6                   | GINI impurity of the node: 0.48                   | Class distribution in the node: {1: 94, 0: 60}                   | Predicted class: 1------ ------ Tree number: 5 Root   | GINI impurity of the node: 0.24   | Class distribution in the node: {0: 2155, 1: 345}   | Predicted class: 0|-------- Split rule: OverageFee <= 7.945           | GINI impurity of the node: 0.15           | Class distribution in the node: {0: 475, 1: 43}           | Predicted class: 0|---------------- Split rule: AccountWeeks <= 7.0                   | GINI impurity of the node: 0.28                   | Class distribution in the node: {1: 5, 0: 1}                   | Predicted class: 1|---------------- Split rule: AccountWeeks > 7.0                   | GINI impurity of the node: 0.14                   | Class distribution in the node: {0: 474, 1: 38}                   | Predicted class: 0|-------- Split rule: OverageFee > 7.945           | GINI impurity of the node: 0.26           | Class distribution in the node: {0: 1680, 1: 302}           | Predicted class: 0|---------------- Split rule: DayMins <= 259.9                   | GINI impurity of the node: 0.2                   | Class distribution in the node: {0: 1614, 1: 203}                   | Predicted class: 0|---------------- Split rule: DayMins > 259.9                   | GINI impurity of the node: 0.48                   | Class distribution in the node: {0: 66, 1: 99}                   | Predicted class: 1------
Every tree is a bit different from one another. The precision and recall scores on the test set with the selected hyperparameters are:
Let's try to increase the accuracy metrics by growing a more complex random forest.
When growing a random forest with 30 trees, 50% of features at each split and max_depth of 4 the accuracy in the test set is:
If we grow 100 trees, with 75% of the features and the maximum depth of the trees would be 5 the results are:
The scikit learn implementation gives very similar results:
# Skicit learn implementationfrom sklearn.ensemble import RandomForestClassifier# Initiatingrf_scikit = RandomForestClassifier(n_estimators=100, max_features=0.75, max_depth=5, min_samples_split=5)# Fitting start = time.time()rf_scikit.fit(X=train[features], y=train[‘Churn’])print(f”Time took for scikit learn: {round(time.time() — start, 2)} seconds”)# Forecasting yhatsc = rf_scikit.predict(test[features])test[‘yhatsc’] = yhatscprint(f”Total churns in test set: {test[‘Churn’].sum()}”)print(f”Total predicted churns in test set: {test[‘yhat’].sum()}”)print(f”Precision: {round(precision_score(test[‘Churn’], test[‘yhatsc’]), 2) * 100} %”)print(f”Recall: {round(recall_score(test[‘Churn’], test[‘yhatsc’]), 2) * 100} %”)
Note that the difference between the sklearn and the custom made code results may vary because of the random nature of bootstrapping data for each tree and the random features that are used to find the best split. Even running the same implementation with the same hyperparameters we will get slightly different results every time.
In summary, the random forest algorithm is made up of independent simple decision trees.
Each decision tree is created using a custom bootstrapped dataset. At every split and in every decision tree a random subsample of features is considered when searching for the best feature and feature value which increases the GINI gain.
The final prediction of an RF classifier is a majority vote of all the independent decision trees.