AdaBoost Explained From Its Original Paper
Source: tai_blog

This publication is meant to show a very popular ML algorithm in complete detail  how it works  the math behind it  how to execute it in Python and an explanation of the proofs of the original paper. There will be math and code  but it is written in a way that allows you to decide which are the fun parts.   A bit on the origins of the algorithm: It was proposed by Yoav Freund and Robert E. Schapire in a 1997 paper  A Decision-Theoretic Generalization of On-Line Learning and an Application to Boostinga beautiful and brilliant publication for an effective and useful algorithm.   Lets start with the pros  cons  and uses of AdaBoost.   Advantages: improves performance and achieves higher accuracy than a single model. It reduces overfitting compared to some other machine learning algorithms.   Disadvantages: AdaBoost can be sensitive to noisy data and outliers. It requires careful tuning  and the performance can depend on the choice of weak learners and the number of iterations. It cannot be parallelized (or only partially)  since each predictor can only be trained after the previous predictor has been trained and evaluated. As a result  it does not scale as well as bagging or pasting.   Applications: image recognition  text classification  fraud detection  predictive modeling.   Introduction  what is ensemble learning and boosting?Python script with an Ada Boost algorithm  lets go straight to using this toolAda Boost explanation  the math on how it worksAda Boost example  simplifying the math  an example of one iterationReferencesIntroductionLets talk a bit about the wisdom of the crowd. Wisdom of the crowd is a phenomenon that suggests that the collective judgment of a diverse number of people is often surprisingly accurate. This mainly occurs because of the central limit theorem  which states that when you take an average of a large number of independent observations  the distribution will center around the true value.   Lets explain this with an example. What if there was a competition where people had to guess how many bubble gum pieces were in a jar? Thousands of different (independent) people will guess; some might be close  and others will be quite far from the true number  but once we calculate the average of the guesses  we will be quite close to the actual number of bubble gum balls  this my friends is the wisdom of the crowd.   How does this apply to Machine Learning?   If we have many predictors (decision trees  other classifiers or regressors) and we aggregate the predictions of this group  they will often perform better than the best individual predictor. A group of predictors is called an ensemble  thus  this technique is called Ensemble Learning.   AdaBoost belongs to a method called boosting. Boosting refers to any ensemble method that combines several weak learners (simple models) into a strong learner (a more accurate model).   There are many boosting methods  the most popular by far are Ada Boosting and Gradient Boosting.   Ada Boost with Python and Scikit-LearnPart 1: data preparationWe create a dummy dataset and separate the data into train and test.   import numpy as np from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split # Generate a random dataset (for example purposes) X  y = make_classification(n_samples=100  n_features=2  n_informative=2  n_redundant=0  random_state=42) # Split the dataset into training and testing sets X_train  X_test  y_train  y_test = train_test_split(X  y  test_size=0.3  random_state=42)Part 2: AdaBoost with Decision Trees (1 branch)First  lets understand the possible parameters in Scikit-learns AdaBoostClassifier:   estimator: The base estimator from which the boosted ensemble is built. Usually  a decision tree with a max depth 1 (a weak learner).n_estimators: The maximum number of estimators at which boosting is terminated.learning rate: Weight applied to each classifier at each boosting iteration. A higher learning rate increases the contribution of each classifier.random_state: Controls the random seed given at each estimator at each boosting iteration.from sklearn.ensemble import AdaBoostClassifier from sklearn.tree import DecisionTreeClassifier # Create the AdaBoost classifier # Notice that the depth of the decision tree is 1 base_estimator = DecisionTreeClassifier(max_depth=1) ada_boost = AdaBoostClassifier(estimator=base_estimator  n_estimators=50  learning_rate=1.0  random_state=42) # Train the classifier ada_boost.fit(X_train  y_train) # Make predictions y_pred = ada_boost.predict(X_test)Part 3: Model evaluationWe measure the metrics of the model. Interpretation of these metrics will be seen in a different article.   from sklearn.metrics import accuracy_score  classification_report # Evaluate the classifier accuracy = accuracy_score(y_test  y_pred) print(f'Accuracy: {accuracy:.2f}') print('Classification Report:') print(classification_report(y_test  y_pred))Accuracy: 0.97 Classification Report: precision recall f1-score support 0 1.00 0.94 0.97 16 1 0.93 1.00 0.97 14 accuracy 0.97 30 macro avg 0.97 0.97 0.97 30 weighted avg 0.97 0.97 0.97 30Part 4: Plotting Resultsimport matplotlib.pyplot as plt # Plotting the decision boundary x_min  x_max = X[:  0].min() - 1  X[:  0].max() + 1 y_min  y_max = X[:  1].min() - 1  X[:  1].max() + 1 xx  yy = np.meshgrid(np.arange(x_min  x_max  0.01)  np.arange(y_min  y_max  0.01)) Z = ada_boost.predict(np.c_[xx.ravel()  yy.ravel()]) Z = Z.reshape(xx.shape) plt.contourf(xx  yy  Z  alpha=0.3) plt.scatter(X[:  0]  X[:  1]  c=y  edgecolors='k'  marker='o') plt.title('AdaBoost Decision Boundary') plt.xlabel('Feature 1') plt.ylabel('Feature 2') plt.show()Ada Boost ExplanationIn this section  we explain the key concepts and how an iteration works (a bit of math included  folks.   AdaBoost  short for Adaptive Boosting  is a machine learning algorithm that is used to improve the performance of other machine learning algorithms. We will define a few key concepts to explain how it works:   Weak Learners: models that perform slightly better than random guessing. Decision trees with one split are often used.Boosting: the process of combining multiple weak learners to form a strong learner. Each learner has a weight based on the performance of the previous learners.Weight Adjustment: First  all data points have equal weights. After each iteration  the weight of incorrectly classified points is increased; that way  the learner focuses more on the difficult cases.Combining Learners: the final model is a weighted sum of all the weak learners; each learners contribution to the final model is based on its accuracy  and more accurate learners are given higher weights.Algorithm stepsThe image shows how the algorithm improves on each iteration on separating between the blue and red dots. Lets find out how each step works.   Initialize Weights:Assign equal weights to all data points (each predictor).2. Train Weak Learner and Calculate Weighted Error   Train a weak learner on the weighted dataset (h_t).Calculate the error rate of the weak learner.3. Calculate Alpha  the learner's weight   4. Update weights   5. Combine weak learners   Example of one iterationInitialize weights   Train Weak Learner and Calculate Weighted Error   Calculate Alpha   Update weights   This process continues for each iteration.   ReferencesA Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting (1997)  Yoav Freund and Robert E. SchapireHands-On Machine Learning with Scikit-Learn  Keras  and TensorFlow (2019)  Aurelin GernSklearn documentation