Random Forest Models: Why Are They Better Than Single Decision Trees? | by Saul Dobilas | Towards Data Science

If you want to be a successful Data Scientist, you need to understand the nuances of different Machine Learning algorithms.
This story is part of a series where I provide an in-depth look at how such algorithms work. This includes simple examples, 3D visualizations, and complete Python code for you to use in your Data Science projects.
The category of algorithms Random Forest classification belongs to
An explanation of how Random Forest classification works and why it is better than a single decision tree
Python code examples and visualizations
Similar to some other algorithms, Random Forest can handle both classification and regression. While in this story, I focus on classification, the same logic largely applies to regression too.
A quick recap on the difference between classification and regression :
With classification, we attempt to predict a class label. In other words, it is used for problems where the output (target variable) takes a finite set of values, e.g., whether it will rain tomorrow or not. Corresponding algorithm: sklearn.ensemble.RandomForestClassifier
Meanwhile, regression is used to predict a numerical label. This means your output can take an infinite set of values, e.g., a house price. Corresponding algorithm:sklearn.ensemble.RandomForestRegressor
Both cases fall under the supervised branch of machine learning algorithms. The below graph is interactive, so make sure to click on different categories to enlarge and reveal more👇.
If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.
Under the hood, the random forest is essentially a CART algorithm (Classification and Regression Trees), except it creates an ensemble of many trees instead of just one. This provides several advantages such as:
Improved performance (the wisdom of crowds)
Improved robustness (less likely to overfit since it relies on many random trees)
If you are not familiar with CART, you can find everything about it in my previous story:
towardsdatascience.com
Random Forest algorithm uses majority agreement prediction for the class label, which means that each tree predicts whether the observation belongs to ‘Class 0’ or ‘Class 1’. If 55 trees out of a hundred predicted ‘Class 1’ and 45 predicted ‘Class 0’, then the final model prediction would be ‘Class 1’.
The majority prediction from multiple trees is better than an individual tree prediction because the trees protect each other from their individual errors. This is, however, dependant on the trees being relatively uncorrelated with each other.
The question is — how do we build many trees from the same data pool while keeping them relatively uncorrelated? There are two parts to it:
Bootstrap aggregation (random sampling with replacement)Feature randomness
Bootstrap aggregation (random sampling with replacement)
Feature randomness
Decision trees are so-called high-variance estimators, which means that small changes to the sample data can greatly impact the tree structure and its prediction.
Ideally, you want to turn it into a low-variance estimator by creating many trees and using them in aggregation to make the prediction.
Bootstrapping is a technique where instead of taking the whole training data to build one tree, you use sampling with replacement to create many samples of the same size. Then you can use these samples to build separate trees. For example:
Whole data (10 observations): [1,2,2,2,3,3,4,5,6,7]Bootstrap sample 1 (10 obs): [1,1,2,2,3,4,5,6,7,7]Bootstrap sample 2 (10 obs): [1,2,2,2,2,3,3,3,3,6]...Bootstrap sample n (10 obs): [2,2,2,3,4,5,6,6,7,7]Note, how each of your samples consist of 10 observations.
In this case, you randomly pick one number from the whole sample and record what you have picked. Then you repeat the action until you have recorded 10 observations.
An equivalent would be randomly picking a ball from a bucket of red and blue balls, recording its color, and then returning it to the bucket before repeating the same action.
This method produces many samples with the same observations but different distributions. In the end, the aggregation of these trees leads to a reduction in the variance of your model prediction, turning it into a low-variance estimator.
The second part in reducing the correlation between the trees comes from using a random selection of features when looking for the best split for each node.
For example:
Full list of features: [feat1, feat2, ..., feat10]Random selection of features (1): [feat3, feat5, feat8]Random selection of features (2): [feat1, feat5, feat10]...The split in the first node would use the most predcitive feature from a set of [feat3, feat5, feat8]The splint in the second node would use the most predictive feature from a set of [feat1, feat5, feat10]...Note, when building a Random Forest model, you can control the number/proportion of features that are used to find the best split for each node.
This process allows the creation of less correlated trees and, combined with bootstrap aggregation, produces a Random Forest.
Leaving theory behind, let us build a Random Forest model in Python.
We will use the following data and libraries:
Australian weather data from Kaggle
Scikit-learn library for splitting the data into train-test samples, building Random Forest models, and model evaluation
Plotly for data visualizations
Pandas and Numpy for data manipulation
Graphviz library to plot decision tree graphs
Let’s import all the libraries:
Then we get the Australian weather data from Kaggle, which you can download following this link: https://www.kaggle.com/jsphyg/weather-dataset-rattle-package.
We ingest the data and derive a few new variables for usage in the models.
Next, let’s build a model by following these steps:
Step 1 — select model features (independent variables) and model target (dependent variable)
Step 2 — split data into train and test samples
Step 3 — set model parameters and train (fit) the model
Step 4 — predict class labels on train and test data using our model
Step 5 — generate model summary statistics
The above code generates the following output that summarizes the model performance.
As we can see, the model generalized quite well with similar performance on test data when compared to training data. The number of estimators was 1000, which means that our Random Forest consists of 1000 individual trees. Here is what a tree 552 looks like. If you wish, you can generate tree diagrams for each one of them by changing the index.
While we cannot easily visualize all model predictions using 17 features, we can do it when we build a random forest with just 2 features. For this, we first need to build another model using the code above, remembering to select only 2 features:
X=df[['WindGustSpeed', 'Humidity3pm']]
After building a model, we use the following code to create a 3D visualization with Plotly:
The above generates the following graph.
From the above visualization, we can clearly see that chances of rain tomorrow increase as the humidity at 3 pm and wind gust speed increases. Also, while not completely smooth, the decision surface has fewer big step changes compared to using just one decision tree (see graph below).
Random Forest is amongst the best performing Machine Learning algorithms, which has seen wide adoption. While it is a bit harder to interpret than a single Decision Tree model, it brings many advantages, such as improved performance and better generalization.
Thanks for reading, and feel free to use the above code and materials in your own Data Science projects.
Cheers! 👏Saul Dobilas
If you have already spent your learning budget for this month, please remember me next time. My personalized link to join Medium is:
solclover.com
Also, you may be interested to see how Random Forest compares to AdaBoost:
towardsdatascience.com
Other Boosting Algorithms: