Tree-Based Machine Learning Algorithms Explained
By: Dhivya Ravindran
Tags: Random Forest, Decision Tree, Classification, Xgboost, Machine Learning

Machine Learning 🤖

Machine Learning is a branch of Artificial Intelligence based on the idea that models and algorithms can learn patterns and signals from data, differentiate the signals from the inherent noises in the data and use the learnings to predict over new, unseen data. As the model can scale and predict over vast new data quickly, efficiently and consistently, Machine Learning is also an automation technique. Whether we know it or not, we interact with a variety of Machine Learning algorithms every single day- from recommendation engines on Netflix, Spotify, YouTube and Amazon; social media feeds on LinkedIn, Instagram, Twitter and Facebook; spam classifiers on Gmail and Outlook; to search engines like Google and so on. I’ll try to explain the seemingly complex but popular Machine Learning algorithms in non-technical English without any complex mathematical equations or scary code.

The Three Flavors of Machine Learning

Machine Learning algorithms come in three flavors: Supervised Learning, Unsupervised Learning and Reinforcement Learning.

In Supervised Machine Learning, the data is labeled to tell the machine what patterns it should look for. Examples include a model built on the loan dataset from a bank with the loan default (yes or no) as the target/ label or a model built on the patient health dataset from a hospital with the diabetes positive (yes or no) as the target variable. In Unsupervised Machine Learning, the data is not labeled and so the machine does not know what patterns to look for but it can still find some patterns. Cybersecurity is one space where unsupervised ML can be applied. Reinforcement learning is an algorithm that learns by trial and error. It has an objective and it tries out a lot of different things and is rewarded or penalized depending on whether its behaviors help or hinder it from reaching the objective.

In this article, we’ll learn in brief about three tree-based supervised Machine Learning algorithms and my personal favorites- Decision Tree, Random Forest and XGBoost.

Decision Tree 🌲

Most of us would have looked at a decision tree at some point in our lives. It looks like something in the picture below. A decision tree has nodes and branches. Assume we have a loan dataset from a prominent bank that has attributes about the person that made the loan application and whether or not his/ her loan was approved which is the target or label. So the algorithm at the top node, selects one feature from the application, a feature that splits the dataset into two and provides maximum “information gain”, let’s say it picks if the person owns or rents the house he lives in. Then at the next two nodes, the algorithm picks another feature from the application that provides the maximum information gain, let’s say it picks if the person earns more than $80,000 or lesser than $80,000 in annual income. So as the tree grows deeper, the algorithm keeps picking features from the dataset and keeps learning.

Decision Tree. Icons made by Becris from Flaticon

Now the main problem here is that as the tree keeps growing, the algorithm increases in complexity and so it does not just learn from the signals but also learns the noises in the dataset. This problem is the classic “overfitting” problem in Machine Learning. A model that is overfitting to the data, will have extremely high accuracy but will not generalize well and will provide poor predictions when exposed to a new and unseen dataset as it has learned from the noises as well in addition to the signals. So the solution to this is called “tree pruning”, we can limit the maximum depth of the decision tree and also the number of leaf nodes in the tree. From experience, any decision tree which has a maximum depth greater than 4 levels, starts to overfit and produce nonsensical outputs. And in most real-world scenarios, we use CART (classification and regression trees) which is slightly different from a decision tree, where each leaf node comes with a score and not just a yes/ no type of decision.

Pros of Decision Trees:

Easy to understand and explain to non-technical stakeholders. Simple to implement and deploy into the product if used in “Product Data Science” as it is just a bunch of nested if-then-else statements. It can be the preferred go-to algorithm when the strategy is the delivery and not accuracy. Can process over continuous and categorical features/ variables, unlike regression models which cannot model over categorical variables unless one-hot encoded (converted to 1/0 dummy variables). The dataset does need not be pre-processed using methods like data normalization and can be used as-is. Can handle datasets with missing values without any treatment, unlike regression which requires missing values to be either removed or filled in or imputed using some technique. Does not require linear relationships between the features and the label/ target in the scatter plot and can handle non-linear relationships perfectly.

Cons of Decision Trees:

Can easily cause overfitting. Generally, has poor accuracy and needs to ensembled to improve accuracy. Adding or removing features can drastically change the overall tree structure and the algorithm.

Random Forest 🌲🌲🌲🌲

The solution to the overfitting problem in Decision Trees is “ensembling” which basically means putting many weak trees together in a strong forest and getting predictions from each of the trees and taking a vote to decide on the winning prediction. So in our example, if we build an ensemble of 1000 trees in a random forest and ‘700’ trees produce “yes” as the prediction for loan approval and “300” trees produce “no” as the prediction, then the democratic majority vote which belongs to “yes” becomes the output of the algorithm. Sounds simple, right? This is the concept behind the Random Forest algorithm.

Random Forest

It is based on the principle that the crowd is wiser than the individual, and even if a few weak trees/ learners in the forest produce nonsensical predictions, the other trees in the forest would correct for this error and so the forest/ model in its entirety would move toward the more accurate prediction. This method of having many parallel uncorrelated trees/ estimators is also famously called “Bagging”. The trees are required to be uncorrelated and diverse (from each other). If we ensemble similar correlated trees, then the output would only be as good as the output from a single decision tree. So to do this, the random forest algorithm will take samples of the dataset with replacement, the main point to emphasize here is that these samples are not subsets, they have the same sample size as the train dataset but they are samples made with replacements.

Another way to diversify the trees/ learners from one another is by using “feature randomness”. In Decision Trees, at each node of the tree, all features that are available in the dataset are considered in feature selection for maximum information gain but in Random Forest, only a subset of features are available for selection. This means that the models further diversify from one another and avoid overfitting. In practice, this can be achieved by tuning the max features parameter in the model.

Pros of Random Forest:

The overfitting problem in Decision Trees is addressed and the model is significantly more accurate in producing predictions compared to a single Decision Tree. Other advantages with Decision Trees are also applied here as in: can process over continuous and categorical features/ variables. The dataset does need not be pre-processed using methods like data normalization. Can handle datasets with missing values without any missing value treatment. Does not require linear relationships between the features and the label/ target in the scatter plot and can handle non-linear relationships perfectly. The most important advantage of Random Forest is the generation of something called “Feature Importance” or Variable Importance. This is basically just a list of model features with an Importance value assigned, this can be highly useful in finding the features that have the best predictive power or that contribute the most to the information gain or reducing impurity of the leaf nodes. Knowing feature importance can particularly help in feature selection when iterating over the model as most Machine Learning models require the Data Scientist/ ML Engineer to choose the fewest uncorrelated variables that can explain the most variance in the data.

Cons of Random Forest:

Unlike Decision Trees, Random Forest is not easy to understand and interpret and this can be a problem when the model needs to be explained to non-technical stakeholders before implementation and also when there are algorithm transparency regulations to adhere to. Deployment of the model into the product (for Product Data Science) is also tedious especially if old-school deployment strategies are followed like translating the model (in R/ Python) into the language in which the product is written(like Rails). So this necessitates the organization to invest in MLOps for deployment (MLOps is the ML equivalent of DevOps). Can cause immense strain on computation resources when learning especially for massive datasets.

XGBoost 🌲 🌲🌲🌲😜

While Random Forest performs significantly better than Decision Trees, it is “XGBoost” that is considered the mother of all tree-based algorithms and they’re the most evolved tree-based algorithm as well. In organizational settings, if we’re not building models for Product Data Science but for other touchpoints within an organization like Operations, Marketing, and so on and if we’re dealing with tabular structured data (and not text, images, speech), then XGBoost will almost always be the best performing and most accurate model. XGBoost stands for “Extreme Gradient Boosting” trees. While Random Forest was a Bagging-based algorithm, XGBoost is a Boosting-based algorithm.

Boosting: is the process where weak decision trees are ensembled in series to produce a strong learner/ estimator and each subsequent tree tries to learn from the errors of its predecessor and works towards minimizing those errors. Below is an in-depth comparison between bagging and boosting.

Bagging: 1. Weak learners/ decision trees are ensembled in parallel. 2. Learners do not learn from each other. 3. Learners do not boost the influence of high performing learners. 4. Since the trees are ensembled in parallel, the model learns quickly. 5. The dataset used to build the new learners are samples of the same sample size as the original dataset with replacement. 6. In Bagging, the prediction is based on the democratic majority votes or the simple average of outputs from all the learners. Boosting: 1. Weak learners/ decision treest are ensembled sequentially. 2. Since the learners are built sequentially, the learners work towards minimizing the errors of their predecessors (previous learners). 3. Learners boost the influence of high performing learners, hence the name: Boosting. 4. Since the trees are ensembled sequentially, they learn slowly. But in statistics, models that learn slowly perform better. 5. The dataset used to build the new learners are also samples of the same sample size as the original dataset with replacement but since the learners are ensembled sequentially, each data point is weighted and at the end of each learning, the weights are redistributed. So if the data is misclassified (erroneous), then the weights are increased to indicate their difficulty and subsequent learners will emphasise on them during their learning. 6. In Boosting, the output is the weighted average of all the learners ensembled using a second set of weights (not the one discussed in point 5). Some learners that have performed well will have a higher weightage compared to other weak learners.

Evolution of Tree-Based Algorithms. From Decision Trees to XGBoost. Infographic source from Vishal Morde.

Gradient Boosting: In gradient boosting when ensembling weak learners we also use a “gradient descent algorithm” to optimize or minimize the loss function. In this example where we are predicting whether a loan is approved or not, it is a classification problem and so “logarithmic loss” (log loss quantifies the accuracy of a classifier by penalizing wrong classifications and is the metric used to create the leaderboard in most Kaggle classification competitions) can be used. In the case of regression problems, for instance, if we’re predicting the amount of loan that is approved/ disbursed, then we use squared error (square of the difference between the predicted and the actual).

XGBoost combines the benefits of boosting, gradient boosting, and more to produce unbelievable accuracy and performance and that’s why it is called EXTREME gradient boosting. So a key limitation in running Machine Learning workloads is the strain on the computational resources. Random Forest and Gradient Boosting algorithms are extremely resource-hungry and cause a serious drain on your hardware infrastructure when learning over massive datasets and can cause frequent crashes of the infrastructure and also have a prolonged learning time. XGBoost successfully addresses this problem and has time and again been the most computationally efficient Machine Learning algorithm (even bettering the benchmark set by regression algorithms). Here is a comparison between Random Forest, GBM, and XGBoost in terms of accuracy/ predictive power and strain on computational resources while learning over a dataset generated from Scikit-Learn’s make_classification dataset that has 1 million rows and 20 features.

Performance Comparison between Random Forest, Gradient Boosting Trees, and XGBoost. Source from Vishal Morde.

Pros of XGBoost:

Uses a more regularized learner, we’ll learn more about regularization in another article, but regularization is the magic cure for model overfitting and so the model accuracy and performance tend to be the most superior compared to the other ML algorithms. Has in-built cross-validations, cross-validation also prevents model overfitting. Approaches serialized ensembling through parallelization which improves computational efficiency. The dataset does need not be pre-processed using methods like data normalization. Does not require linear relationships between the features and the label/ target in the scatter plot and can handle non-linear relationships perfectly. Can handle a sparse dataset with a lot of missingness and so no missing value treatment is required.

Cons of XGBoost:

Like Random Forest, XGBoost is not easy to understand and interpret and this can be a problem when the model needs to be explained to non-technical stakeholders before implementation and also when there are algorithm transparency regulations to adhere to. Deployment of the model into the product (for Product Data Science) is also tedious especially if old-school deployment strategies are followed like translating the model (in R/ Python) into the language in which the product is written(like Rails). So this necessitates the organization to invest in MLOps for deployment.

Why not just use XGBoost all the time and why even bother about the other algorithms? 😈

XGBoost is the best performing algorithm but its accuracy is highly dependent on the Data Scientist or the ML Engineer’s ability to tune its hyperparameters like learning rate (slower the learning, better is its performance), max depth of the trees/ learners, and so on. In Product Data Science, unless the organization invests in forward-thinking model deployment pipelines like MLOps, deployment of XGBoost algorithm into the product becomes a serious pain compared to other lightweight algorithms like Decision Trees and Logistic Regression. And in many business requirements, algorithm transparency and interpretability are key requirements. For instance, if an ML-based financial investment advisor is recommending me to make investment decisions based on a Machine Learning algorithm, I would love to know transparently on what basis this algorithm is making this decision. And so in these cases, it becomes difficult to explain the XGBoost algorithm to the end-users and stakeholders. So when choosing the right algorithm for a project, there are a lot of factors that need to be considered especially when model deployment is on the agenda.

Conclusion