Decision Trees and Ensembling Methods
By: Gabriel Prieto
Tags: Boosting, Bagging, Machine Learning, Data Science, Ensemble Learning

Decision Trees or Traditional Models?

If the relationship between predictors and target variable is approximately linear, then the a traditional approach will likely perform better on the data and outperform a variety of tree models.

On the other hand, if the model is very complex and follows a non-linear pattern, the decision tree might outperform traditional approaches. Moreover, decision trees might be preferred in case the project requires more interpretability and visualization, as those can be represented in a ‘flow-chart’ type diagram and are somewhat easy to be interpreted.

Advantages of Decision Trees:

Easier to explain than most models, including linear regression

Can be visualized on the form of a ‘flow’ or ‘organizational’ chart, what makes it easy to understand, even for laymen

Decision trees tend to simulate better the way a human actually thinks

Capability to handle qualitative variables without the need to create dummy variables

Disadvantages of Decision Trees:

Do not have the same predictive accuracy as other regression and classification models such as linear regression and logistic regression

Not very generalizable, a small change in the data can throw the final model off

Having in mind the disadvantages mentioned above, there are other techniques that can be applied to decision trees that could substantially increase their predictive power. Those are bagging, random forests and boosting.

Bagging:

Bagging or Bootstrap Aggregation is growing a n number of decision trees and training them with n samples of the data set (samples should be randomly pulled from a single data set. With replacement!).

While this method can improve the results of many estimators, it is particularly useful for decision trees. Due to their high variance, averaging a set of observations has proven to reduce variance without compromising bias!

Much like what is the so called ‘wisdom of the crowd’, while some underestimate and other overestimate, the sum of the errors tend to decrease when we average out those guesses.

Although Bagging is a great way of increasing the predictive accuracy of decision trees, it comes with the downside that we lose interpretability of the model. With a single tree it is quite easy to interpret and visualize best features used in the model, but after averaging hundreds or thousands of them we give up on the interpretability of our model…

Random Forests:

Much like bagged decision trees Random Forests consist on a set of bagged decision trees, but with a small tweak that can make a big difference in the outcome.

Instead of testing the bootstrapped training samples on the same Tree, a random sample of features (predictors) is chosen at every split of the decision tree. This guarantees that the models tested are different from one another as they will not always use the most important feature to do the first split of the tree.

Consequently the different models will be less correlated to each other, having a more impactful change once averaged and hence producing a less variable model.

The main difference between bagged decision trees and random forests is the choice of a predictor subset size m out of a total of p predictors. Typically for random forests we use m = √p which leads in to a reduction of the predictions’ error. When m = p we have a regular bagged decision tree model.

Boosting:

Boosting also works in a similar way, but trees are grown sequentially by using information on prediction errors from previous trees. Models are fit to modified versions of the data and as it iterates, each tree is forced to concentrate on those errors or misclassifications.

According to the performance of a tree it is given weights to indicate its performance. The weights will serve at the end to average results and come up with the best model.

Implementation of different techniques on Python’s Scikit-Learn: