How Do You Implement AdaBoost with Python? | by Rukshan Pramoditha | Towards Data Science

We’re continuing from Part 1 of Boosting algorithms in machine learning article series. Have you read Part 1? It gives you a short introduction to boosting and definitions of some key technical terms that are all important to understand today’s content.
Today, we’re going to learn one of the most popular boosting algorithms: AdaBoost (Adaptive Boosting). More emphasis will be given to the implementation part of the algorithm under the following topics.
Scikit-learn classes for AdaBoost
Train and evaluate an AdaBoost classification model on Wine data
Compare the AdaBoost model with a decision tree stump
Important hyperparameters in AdaBoost
Measure the effect of hyperparameter n_estimators
Measure the effect of hyperparameter learning_rate
Find the optimal hyperparameter values using Grid Search
All code samples will be included as GitHub gists so that you can easily work with them! At the end of the article, you will be able to implement the AdaBoost algorithm on a given dataset with optimal hyperparameter values.
Let’s get started!
The Python implementation of AdaBoost is fulfilled by two Scikit-learn classes: AdaBoostClassifier() for classification (both binary and multi-class) and AdaBoostRegressor() for regression. The import conventions are:
from sklearn.ensemble import AdaBoostClassifierfrom sklearn.ensemble import AdaBoostRegressor
By early stopping the tree growth with max_depth=1, we’ll build a decision stump on Wine data. This is to compare the decision stump with the AdaBoost model. We also use this stump model as the base learner for AdaBoost.
The following Python code creates a decision tree stump on Wine data and evaluates its performance.
The model’s performance is poor. This is not a surprise as we strictly regulated the tree growth by setting max_depth=1.
Now, we’re going to build an AdaBoost classification model on Wine data.
Ohh that score! The AdaBoost model is performing really well compared to the decision tree stump.
The following are the most important hyperparameters in both AdaBoostClassifier() and AdaBoostRegressor().
base_estimator: This is the base learner used in AdaBoost algorithms. The default and most common learner is a decision tree stump (a decision tree with max_depth=1) as we discussed earlier.
n_estimators: The maximum number of estimators (models) to train sequentially. The default is 50. We’ll measure the effect of this hyperparameter soon.
learning_rate: This determines the weight applied to each estimator in the boosting process. The default is 1. Smaller values such as 0.05, 0.1 force the algorithm to train slower but with high-performance scores. We’ll measure the effect of this hyperparameter soon.
We’ll manually measure the effect of the hyperparameter n_estimators by varying its values from 2 to 100 and plot the test scores given by the AdaBoost model.
After about 30 estimators, the accuracy score is constant at 0.972 that we obtained earlier. You can use any integer above 30. But keep in mind that if you increase the value, the algorithm will take much time for training. So, in this case, it is better to use the default value, 50.
We’ll manually measure the effect of the hyperparameter learning_rate by varying its values from 0.1 to 1 and plot the test scores given by the AdaBoost model.
The highest accuracy is given at 0.4, 0.5, 0.7 and 0.8. We’ve used 0.5 in the above model.
Both measurements were done manually. There is an easier way to find the optimal values for n_estimators and learning_rate.
Here, we automatically tune (find optimal values) both hyperparameters at the same time by using Grid Search. In contrast, in the previous steps, we tuned one hyperparameter at a time.
We can use these values to get the highest accuracy of the AdaBoost model. Some other possible combinations that give the same accuracy are:
learning_rate=0.5, n_estimators=50
learning_rate=0.8, n_estimators=50
learning_rate=0.8, n_estimators=100
learning_rate=0.4, n_estimators=50
Building AdaBoost models in Python is straightforward. All you have to do is to determine the type of problem (regression/classification) you want to solve and select the suitable AdaBoost class provided in Scikit-learn. However, the hyperparameter tuning procedure is a real challenge. You need to make some visualizations, do parallel computations for hyperparameter tuning. Actually, you’ll create thousands of models. Luckily, we can automate the tuning process using advanced methods as we discussed in this post.
The model’s outputs can be largely affected by random data splitting. You may get different accuracy scores depending on the integer specified in the random_state. It is always recommended to specify an integer to get static results at different executions.
Compared to other tree-based methods such as decision trees and random forests, boosting algorithms like AdaBoost perform really well. This is because tress in boosting are sequentially trained considering the errors created by initial tress in previous rounds.
In Part 3, we’ll discuss Gradient Boosting, another popular boosting algorithm. See you in the next story. Happy learning to everyone!
My readers can sign up for a membership through the following link to get full access to every story I write and I will receive a portion of your membership fee.
rukshanpramoditha.medium.com
Thank you so much for your continuous support!
Special credit goes to John Price on Unsplash, who provides me with a nice cover image for this post.