Random Forest Algorithm From Scratch | by Vienna Bai | Towards Data Science

Implementing a random forest ML algorithm from scratch can sound like an intimidating task. It is easy to feel overwhelmed by merely the idea of how many details we have to get right. Or at least that is how I felt. Where do I start? What is the first step? How do I know if it is working?
The turning point was when I realized I needed to start by not thinking about the code. Imagine that I am standing inside the algorithm, at the root of one tree, what would I do? And then what? Asking these questions will help break down and simplify the problem. Answering these questions will build up an intuition of the inner workings of the algorithm. In this post, we will explore these questions and answers together, with the Titanic dataset.
Our agenda:
cover a high-level overview of what random forests dowrite the pseudo-code for a binary random forest classifieraddress some minimal data preprocessing requirementswrite the code (can be accessed in full here)check the results against the scikit-learn algorithm
cover a high-level overview of what random forests do
write the pseudo-code for a binary random forest classifier
address some minimal data preprocessing requirements
write the code (can be accessed in full here)
check the results against the scikit-learn algorithm
Please note that this post was inspired by the Fast.ai Machine Learning course, which I highly recommend for those who learn best by doing.
Let’s dive in!
If you had been onboard the Titanic, would you have survived?
“Mean” refers to the average survival rate of everyone in that group.
starting out, for everyone onboard, expected survival is 40%
then for all women, 75%
and for all men, 20% ...etc.
For me, I am a female, and say I had a ticket in the 3rd class. Following this decision tree, my probability of surviving the Titanic was 50/50.
Another person, male, also in the 3rd class, would have had a 10% chance of survival; unless he was under the age of 15, then his chance of survival was 90%.
This is one decision tree within a random forest. We had been provided with which feature to split on (i.e. Age), and at what level (i.e. under 15). To predict the survival chances of a new profile, we simply follow the branches.
Now let’s take away the branches. Imagine that we are at the root of the tree.
All we know that of all the samples, 40% survived. We have to decide which feature to split on, and at what level. How?
We will try every single feature and every single level, score each combination, and select the best.
for feature in list of features:      for (unique) row in data:           what is the score of we split here?           if the score is better what we have: update it 
But what is the score? How do we know which split is the best?
For classification problems, one commonly used metric is the Gini impurity (this is also the default for scikit-learn). Each split will have a Gini score:
Let’s see a simple example:
If I split at women vs men, what would the Gini impurity be?
x = [1, 1, 1, 1, 1, 2, 2, 2] #5 female & 4 maley = [1, 1, 1, 1, 0, 0, 0, 0] #4 survived & 4 died
Decision Tree:def check_features:      for feature in list of features:           find_best_split of this featuredef find_best_split(feature):     for (unique) row in data:          left = list of all rows <= current row                    right = list of all rows > current row           calculate the score                     if the score is better than what we have: update itdef find_gini(left_split, right_split, y):      p(left) = n_left/n_y     p(right) = n_right/n_y     for each outcome: #in this case, Survived or Died          p = probability of this outcome for the left           sum_left += p^2           p = probability of this outcome for the right          sum_right += p^2     gini = weighted average of (1-sum_left) and (1-sum_right)def prediction():       return array of predictions for this tree 
A forest is made of numerous decision trees. What about the random part? Let’s say that we have a dataset with 1000 samples and 5 features. And we want to build 10 trees. All the trees will split at the same place, ending up with 10 identical trees. This would not be very helpful to us.
What if we take a different portion of the dataset each time? Let’s say we want 100 trees, that would mean 10 samples per tree — which would lead to pretty high variance.
Is there a way we can use 1000 samples each time? And also have slightly different data for each tree? Yes: bootstrapping. We draw 1000 samples from our dataset, with replacement. On average, about 63% of the original observations will be included. This is important to reduce over-fitting.
Using this method, we will have different data for each tree. The goal of a random forest is to have trees that are as different from each other as possible, but be very good at predicting the data it is given. Then, if you have many uncorrelated trees, each with high predictive power, when you sum them up, the errors will sum up to zero. What remains will be the true relationship between the X and y variables.
Random Forest:def __init__ (x, y, n_trees, sample_size, min_leaf):     for numbers up till 1-n_trees:           create a tree def create_tree():     get sample_size samples, use np.random.choice w/ replacement     create an instance of the Decision Tree class def predict():     average of predictions from n_trees      return binary outcome 0 or 1 
We now have the pseudo-code for the 2 classes. Let’s turn this into code and compare the results against scikit-learn’s RandomForestClassifier.
Inputs and targets must be numeric
Categorical features are label-encoded (if ordinal) or one-hot-encoded
What about outliers and feature scaling? Turns out random forests are generally immune to these. Its decisions are ordinal. Remember that at each split consideration, we group samples into two groups: those larger than the split value, and those smaller than or equal to the split value. If we are considering splitting at Age 15: it doesn’t matter if your age is 16 or 60, both will be placed in one group. Random forests make very few assumptions about the distribution of your data, making it a great place to start.
Here are what the inputs look like for the preprocessed Titanic dataset:
For now, let’s focus on a subset of 2 features: Pclass and Sex.
We are back at the root of the first tree. Let’s write
class RandomForest():
    def __init__ (self, x, y, n_trees, sample_sz=None, min_leaf=5):
        np.random.seed(42) 
        if sample_sz is None:
            sample_sz=len(y)
        self.x, self.y, self.sample_sz, self.min_leaf = x, y, sample_sz, min_leaf
        self.trees = [self.create_tree() for i in range(n_trees)]
    
    def create_tree(self):
        idxs = np.random.choice(len(self.y), replace=True, size = self.sample_sz)      
        return DecisionTree(self.x.iloc[idxs], 
                            self.y[idxs],
                            idxs=np.array(range(self.sample_sz)),
                            min_leaf=self.min_leaf)
    
    def predict(self, x):
        percents = np.mean([t.predict(x) for t in self.trees], axis=0)
        return [1 if p>0.5 else 0 for p in percents]

def find_gini(left, right, y):
    classes  = np.unique(y)
    n = len(left) + len(right)
    s1=0; s2=0
    
    for k in classes:   
        p1 = len(np.nonzero(y[left] == k)[0]) / len(left)
        s1 += p1*p1 
        p2 = len(np.nonzero(y[right] == k)[0]) / len(right)
        s2 += p2*p2 
    
    gini = (1-s1)*(len(left)/n) + (1-s2)*(len(right)/n)
    
    return gini

class DecisionTree():
    def __init__(self, x, y, idxs=None, min_leaf=5):  
        if idxs is None: 
            idxs=np.arange(len(y))
        self.x, self.y, self.idxs, self.min_leaf = x, y, idxs, min_leaf   
        self.n, self.c = len(idxs), x.shape[1]
        self.val = np.mean(y[idxs])
        self.score = float('inf')
        self.find_varsplit()
        
    @property
    def split_name(self): return self.x.columns[self.var_idx]
    
    @property
    def split_cols(self): return self.x.values[self.idxs, self.var_idx]
    
    @property
    def is_leaf(self): return self.score == float('inf')
    
    def __repr__(self):
        s = f'n: {self.n}'
        if not self.is_leaf:
            s+= f'; gini:{self.score}; split:{self.split}; var: {self.split_name}'
        return s
            
    def check_features(self):
        for i in range(self.c): 
            self.find_better_split(i)
    
    def find_best_split(self, var_idx):
    
        x, y = self.x.values[self.idxs, var_idx], self.y[self.idxs]   
        sort_idx = np.argsort(x)
        sort_y = y[sort_idx]
        sort_x = x[sort_idx]

        for i in range(0, self.n-self.min_leaf-1):
            if i < self.min_leaf or sort_x[i] == sort_x[i+1]: continue 
            lhs = np.nonzero(sort_x <= sort_x[i])[0]
            rhs = np.nonzero(sort_x > sort_x[i])[0]
            if rhs.sum()==0: continue

            gini = find_gini(lhs, rhs, sort_y)

            if gini<self.score: 
                self.var_idx, self.score, self.split = var_idx, gini, sort_x[i]

tree = TreeEnsemble(x_sub, y_train, 1).trees[0]; tree

The scikit-learn tree tells us this is the first split.
Our tree made the same decision!
tree = RandomForest(x_sub, y_train, 1).tree[0][output] gini: 0.318; split:1, var: Sex
If we allow the scikit-learn tree to split one more time (depth =2):
After the first split, we have all the women in one group, all the men in another. For the next split, these two groups will effectively become the root of their own decision tree. For women, the next split is to group separate 3rd class from the rest. For men, the next split is to split 1st class from the rest.
Let’s alter our pseudo-code:
def check_features:      for feature in list of features:           find_best_split of this feature     left_hand_side = rows <= our split point      right_hand_side = rows > our split point      DecisionTree(left_hand_side)     DecisionTree(right_hand_side)
Results match those of scikit-learn:
tree = RandomForest(x_sub, y_train, 1).tree[0][output] gini: 0.318; split:1, var: Sextree.lhs [output] gini: 0.264; split:2, var: Pclasstree.rhs[output] gini: 0.272; split:1, var: Pclass
Let’s include more features and see how our model does:
Using 10 decision trees, our model had an accuracy score of 81% on the test set. The score is below that of scikit-learn’s. But not bad for an algorithm that we built from scratch!
So you see, building a random forest from scratch does not have to be intimidating. This part of the learning process allowed me to separate out which parts of the algorithm I truly understood, and which parts I still needed to work on. Sometimes, unless you try to implement it on your own, that clarity escapes you.
Thank you for allowing me to share what I had learned. I hope you found the post to be helpful. Questions and suggestions are more than welcome!
References:
J. Howard, Fast.ai Machine Learning, lesson 7
G. James, D. Witten, W. Trevor, and R. Tibshirani, Introduction to Statistical Learning, chapter 5
Until next time,