Random Forest Classifier Implementation from scratch with Python
By: Chanrith Poleak
Tags: Random Forest, Random Forest Classifiers, Decision Tree, Sklearn, Machine Learning

Random Forest Classifier Implementation from scratch with Python

Soon to finish the third semester of graduate study, the overcautious me start to browse the internet on how I can land a job in machine learning field. After a few minutes of redditing /r/MachineLearning, I stumble upon an interesting blog of a guy discussing his road map to getting his job in AI research position. In the blog, he talked about the process from research the companies to salary negotiation, and also about the ML-related interview questions, which I am mostly interested in. As expected, the interview questions, while being heavily-based ML knowledge, the answers can be found in your ML course or, even easier, in google. However, there is one point that caught my attention, which related to the implementation of the Decision Tree and Random Forest, and thus, lead me to writing this blog on how to implement the random forest algorithm using python. Actually without actually implement it from scratch, there are many libraries like scikitlearn that already has build-in implementation this algorithm. However, since I want to know how it work behind the box, maybe code it line-by-line would be a good idea for my learning purpose.

Random Forest is one the most used machine learning algorithm that can be used in both the classification and regression tasks. Just like its name, “Forest”, Random Forest is a collection of many tree, specifically decision trees, and you try to improve your prediction accuracy by using the output from this decision tree. Therefore, to implement the Random Forest, Decision Tree is needed.

Decision Tree

Imagine AI as a person behind a mask, and when taking it off, it just a bunch of If-else statement (I’m sure you might have seen this image before). That is actually a decision tree, except that it requires more works than that. To find the long-list conditional statement for a decision tree, we need to find which feature at which value is the best for a condition, so that we can use it to split our dataset. To do so, we can use gini index as the cost function to find those best condition for the splits. By giving the splits, ‘gini’ will try to score by checking how the class is mix. If all the class are all perfectly split, it will give the score of 0.0. Beside from gini, we can also use other method such as Entropy or Chi Square to evaluate the splits. However, for this study, I will only implement the gini index.

Therefore, we can simplify the algorithm as followed:

find gini index for all possible splits

split the dataset using the feature and value at which the gini index is smallest

recursively build the decision tree based on the split data

After writing some required function needed for splitting our tree, we can finally build and visualize our decision tree as follow:

with all the code combine, we have completed the decision tree implementation. We can test the prediction result from our Tree using the built-in sklearn Iris dataset.

Here are what our decision tree look like:

Visualization of Iris Dataset Decision Tree in Terminal

Decision Tree maybe good and easy for visualization. Among all the machine learning algorithm, it’s also maybe the easiest method to interpret and understand. However, it’s also prune to overfitting, which is unwanted in any application. To solve the overfitting problem will required using randomforest, which is the main goal of this post. I will complete the full implementation in the next part. Cheer!!