Random Forest Models: Why Are They Better Than Single Decision Trees?
By: Saul Dobilas
Tags: Random Forest Classifiers, Data Science, Supervised Learning, Machine Learning, Python

Side note, while I have put Neural Networks in a category of their own within the supervised learning branch, they can be used to solve a wide range of problems including classification and regression. The above chart is interactive so try clicking on different categories 👆.

How does the Random Forest classification algorithm work?

Under the hood, the random forest is essentially a CART algorithm (Classification and Regression Trees), except it creates an ensemble of many trees instead of just one. This provides several advantages such as:

Improved performance (the wisdom of crowds)

Improved robustness (less likely to overfit since it relies on many random trees)

If you are not familiar with CART, you can find everything about it in my previous story:

Wisdom of crowds — what is it and why is it good?

Random Forest algorithm uses majority agreement prediction for the class label, which means that each tree predicts whether the observation belongs to ‘Class 0’ or ‘Class 1’. If 55 trees out of a hundred predicted ‘Class 1’ and 45 predicted ‘Class 0’, then the final model prediction would be ‘Class 1’.

Multiple Decision Trees. Image by author.

The majority prediction from multiple trees is better than an individual tree prediction because the trees protect each other from their individual errors. This is, however, dependant on the trees being relatively uncorrelated with each other.

The process of building multiple trees

The question is — how do we build many trees from the same data pool while keeping them relatively uncorrelated? There are two parts to it:

Bootstrap aggregation (random sampling with replacement) Feature randomness

Bootstrap aggregation

Decision trees are so-called high-variance estimators, which means that small changes to the sample data can greatly impact the tree structure and its prediction.

Ideally, you want to turn it into a low-variance estimator by creating many trees and using them in aggregation to make the prediction.

Bootstrapping is a technique where instead of taking the whole training data to build one tree, you use sampling with replacement to create many samples of the same size. Then you can use these samples to build separate trees. For example:

Whole data (10 observations): [1,2,2,2,3,3,4,5,6,7] Bootstrap sample 1 (10 obs): [1,1,2,2,3,4,5,6,7,7]

Bootstrap sample 2 (10 obs): [1,2,2,2,2,3,3,3,3,6]

...

Bootstrap sample n (10 obs): [2,2,2,3,4,5,6,6,7,7] Note, how each of your samples consist of 10 observations.

In this case, you randomly pick one number from the whole sample and record what you have picked. Then you repeat the action until you have recorded 10 observations.

An equivalent would be randomly picking a ball from a bucket of red and blue balls, recording its color, and then returning it to the bucket before repeating the same action.

This method produces many samples with the same observations but different distributions. In the end, the aggregation of these trees leads to a reduction in the variance of your model prediction, turning it into a low-variance estimator.

Feature randomness

The second part in reducing the correlation between the trees comes from using a random selection of features when looking for the best split for each node.

For example:

Full list of features: [feat1, feat2, ..., feat10] Random selection of features (1): [feat3, feat5, feat8]

Random selection of features (2): [feat1, feat5, feat10]

... The split in the first node would use the most predcitive feature from a set of [feat3, feat5, feat8]

The splint in the second node would use the most predictive feature from a set of [feat1, feat5, feat10]

... Note, when building a Random Forest model, you can control the number/proportion of features that are used to find the best split for each node.

This process allows the creation of less correlated trees and, combined with bootstrap aggregation, produces a Random Forest.

Python examples of Random Forest classification models

Leaving theory behind, let us build a Random Forest model in Python.

Setup

We will use the following data and libraries:

Let’s import all the libraries:

Then we get the Australian weather data from Kaggle, which you can download following this link: https://www.kaggle.com/jsphyg/weather-dataset-rattle-package.

We ingest the data and derive a few new variables for usage in the models.

A snippet of Kaggle’s Australian weather data with some modifications. Image by author.

Next, let’s build a model by following these steps:

Step 1 — select model features (independent variables) and model target (dependent variable)

Step 2 — split data into train and test samples

Step 3 — set model parameters and train (fit) the model

Step 4 — predict class labels on train and test data using our model

Step 5 — generate model summary statistics

The above code generates the following output that summarizes the model performance.

Random Forest model performance. Image by author.

As we can see, the model generalized quite well with similar performance on test data when compared to training data. The number of estimators was 1000, which means that our Random Forest consists of 1000 individual trees. Here is what a tree 552 looks like. If you wish, you can generate tree diagrams for each one of them by changing the index.