Understanding AdaBoost for Decision Tree | by Valentina Alto | Towards Data Science

Decision Trees are popular Machine Learning algorithms used for both regression and classification tasks. Their popularity mainly arises from their interpretability and representability, as they mimic the way the human brain takes decisions.
In my former article, I’ve been introducing some ensemble methods for decision trees, whose aim is that of converting a bunch of weak classifiers into a stronger one. Here, I’m going to dwell a bit more on the Boosting method, which is the one where the algorithm ‘learns’ from its error at each iteration. More specifically, I’m going to explain step by step the idea behind Adaboost and how to implement it with R.
The main difference between Adaboost and bagging methods (including Random Forests) is that, at the end of the process, when all the classifiers built during the iterations will be asked to vote for the target of a new observation, there will be trees with a heavier vote than others. Those are the trees that performed the best during all the iterations (so, they showed very few misclassifications). Let’s say that, at the end of the day, some trees will say more than others. This ‘importance of say’ is measured (and updated) throughout the iteration procedure.
Besides that, the other important features of Adaboost, as anticipated, is its ability to learn from past errors, since, at each iteration, the next classifier is built based on the misclassification error of the past one.
Let’s now see how the algorithm concretely works.
Imagine we have a dataset with N observations, P covariates and one binary target variable y=1,-1. This will be our train set.
You can design any task of your interest with this framework (namely, whether an e-mail is spam or not, whether the weather tomorrow will be sunny or not...).
The first thing we do is assigning to each row a weight, which indicates how important is for that observation to be well classified. We initialize those weights equal to 1/N, so that at the first iteration, when training the first classifier, they won’t make the difference.
Now we train our first classifier G(x) on our dataset and then inspect which are the misclassified observations, since we want to compute the total error, which differs from the standard misclassification error since it is a weighted summation:
As you can see, we have a self-normalized summation of weighed errors (when the target is different from the fitted). Note that, because of the way it is built, the total error will always be bounded between 0 and 1. With this quantity, we can compute what we called ‘importance of say’:
As you can see, the higher the error, the less important will be the vote of that tree at the end of the day. We can also visualize it graphically:
error = seq(0,1,0.001)alpha = log((1-error)/error)plot(error,alpha, type='line')
With this metric alpha, we can now update the weights in a way such that, those corresponding to misclassified observations, will be higher in the next iteration. By doing so, when we will fit the next tree on the new weighted dataset, the cost of misclassifying those observations will be higher, and the tree will be very careful not to misclassify them again (since it would imply a higher cost). So, for all the observations i=1,2,..,N, we update the i-th weight as follows.
If we repeat this procedure for M iterations, at the end of the day we will have M classifiers, each with a weighted vote, and, if we want to predict the target of a new observation x*, the formula is:
So there will be some trees whose output will be highly determinant in the final decision (-1 or 1), while others will be neglectable.
Let’s now see a very brief implementation with R of this procedure:
data = data.frame( X=c(rnorm(100,0,1),rnorm(100,1,1)), Y=c(rep(0,100),rep(1,100) ) )data$Y =factor(data$Y)model =adaboost(Y~X, data, 10)pred =predict( model ,newdata=data)
Let’s see the summary of the model and its prediction error (which is equal to the misclassification error in the train set):
print(pred$error)print(model)
We can also retrieve the final ‘importance of say’ weights:
model$weights
As you can see, there are some trees (like the first one) whose final vote is very important, while others (like the last one) are less important.
Finally, we can compare its error with the one of a single tree:
single_model=tree(Y∼X, data)summary(single_model)
As you can see, our boosted classifier is stronger than the single tree one.
Ensemble methods are powerful techniques that can largely improve the predictive accuracy of decision trees. The caveat of those, however, is that they make a bit less easy to present and interpret the final results. Indeed, as we said at the very beginning, the most appreciable feature of decision trees is their interpretability and easiness to understand, which, in a world of algorithms which look like ‘black box’, is an important value. Nevertheless, some visual alternative options are available and, if ensembling multiple trees improve that much accuracy, it is for sure worth it.