Understanding Boosting Algorithms: A Mathematical and Python Implementation Guide
Source: tai_blog

A Deep Dive into the Mechanisms of Boosting with Step-by-Step Examples  Leading to the Development of Boosting in Machine Learning   Boosting is a powerful machine learning technique widely used to improve the performance of predictive models. Its a key component in many winning models on platforms like Kaggle. But what makes boosting so effective? How does it work? This article will break down the boosting algorithm both mathematically and practically.   Well start with the basics  explaining the mathematical foundation of the boosting algorithm in simple terms. Youll see how boosting iteratively improves predictions by correcting errors from previous models. This process is crucial for mastering and effectively applying boosting.   Next  well move to hands-on implementation. Instead of pre-built Python packages  well write the boosting algorithm from scratch  using decision trees as base learners. This approach will help you understand how boosting works step by step.   Finally  well introduce XGBoost  a popular gradient-boosting implementation. Well explain how XGBoost fits into the general boosting framework and guide you through creating a raw XGBoost model.   By the end of this article  youll understand how boosting works and how to implement and customize it for your predictive modeling tasks.   How Does Boosting WorkImagine we have a model represented by the equation:   f(x) is our models prediction  and y is the actual value. Our goal is to make our model as accurate as possible by minimizing the total error  known as the loss function:   To minimize the loss function  we split it into many smaller pieces. The loss function can often be complex or have no explicit form  so we express it as a sum of smaller components:   Each piece represents an error or gradient. This breakdown helps us manage and minimize the total error more effectively. The boosting method uses a model to predict each piece. We iteratively refine our final prediction by summing up all these predicted errors:   where m_i(x) are the models predicting each error piece.   In practice  when implementing the boosting method  we use the following Taylor expansion to approximate the loss function:   We can illustrate the boosting algorithm with the following example:   Just as a supermarket sells bread at varying discounts based on freshness to maximize sales  similarly  the boosting method handles the residuals of a loss function. Earlier residuals (lower-order terms) significantly reduce the loss value  while later residuals (higher-order terms) have a diminishing effect  akin to the decreasing value of less fresh bread. This process continues until no further reduction in loss can be achieved.   The boosting algorithm accumulates these contributions to minimize the total loss value  refining the models predictions iteratively. Each iteration builds on the previous one  incorporating the residuals to improve overall prediction accuracy.   An Intuitive Example of BoostingLets walk through a straightforward Boosting example using linear regression.   Imagine we predict y = 7 given x = 2. We now create a model that makes this prediction through iterative steps.   InitializationWe start with an initial prediction. For simplicity  lets assume our initial prediction is zero:   First IterationPerform Linear Regression: begin by fitting a simple linear regression model to our data point (x = 2  y = 7):Using x = 2 and y = 7  we solve for a and b:   Assume the model predicts p_1 = 4. The residual error e_1 is:   Update the Prediction: update the initial prediction with this new value:Second IterationFit Residuals: perform linear regression on the residual e_1:Using x = 2 and e_1 = 3  we solve for the new prediction p2:   Assume the model predicts p_2 = 2. The new residual e_2 is:   Update the Prediction: add this new value to our prediction:Third IterationFit Residuals: continue by fitting linear regression on the new residual e_2:Using x = 2 and e_2 = 1  we solve for the new prediction p_3:   Assume the model predicts p_3=1. The new residual e_3 is:   Update the Prediction: add this final value to our prediction:This example illustrates the basic mechanism of boosting using linear regression. But in practice more complex models like decision trees are utilized to predict residuals  leading to techniques such as Gradient Boosting Trees.   Gradient Boosting TreesWhy Not Use Linear Regression in Boosting?In our previous example  we used linear regression to predict gradients in each boosting step to demonstrate the basic concept of boosting. However  linear regression is not suitable due to the orthogonality of error and prediction.   In linear regression  the error (residual) is orthogonal to the predictions  meaning the residuals are uncorrelated with the predicted values:   This approach doesnt capture complex error patterns. Because of this orthogonality  fitting a linear regression model to the residuals multiple times is the same as fitting it once to the original data. Therefore  using linear regression in an iterative boosting framework doesnt add extra value over a single linear regression model.   Why Use Tree Models in Boosting?Boosting is an ensemble algorithm  meaning the final prediction is obtained by combining the outputs from multiple models. Interaction and bagging efficiency are important to the production of highly accurate results. Tree models meet these requirements for several reasons:   Non-linearity and Accurate Gradient Prediction Tree models can capture non-linear relationships between features and the target variable  accurately predicting gradients.   Local Approximation Tree models split data into regions and fit simple models (like constants) within these regions. This local approximation can precisely capture the datas patterns.   Handling Different Data Types for Robust Gradient Predictions Tree models can handle numerical and categorical variables without extensive preprocessing.   Robustness to Outliers Tree models are robust to outliers because splits are based on median values rather than means  reducing the influence of extreme values.   Greedy Splitting and Optimality Tree models use a greedy algorithm to find optimal splits that minimize the loss function. This approach can effectively reduce the error.   Examples of Gradient Boosting TreesWe have shown that using Tree Models in Boosting is effective. In this section  I will demonstrate how the Boosting Gradient Tree model operates using Python code with a simple dataset. We will use the raw boosting method (decision tree regression as the gradient prediction model combined with iterative steps) and the GradientBoostingRegressor from the SKLEARN package for comparison.   Python code implementation.   import numpy as np from sklearn.tree import DecisionTreeRegressor import matplotlib.pyplot as plt # Data X = np.array([[1]  [0]  [2]  [3]]) y = np.array([0  0  3  -1]) # Number of boosting iterations n_iterations = 200 # Learning rate learning_rate = 0.1 # Initial prediction (constant model) initial_prediction = np.mean(y) predictions = np.full(y.shape  initial_prediction) # Function to plot predictions def plot_boosting(X  y  predictions  title): plt.scatter(X  y  color='red'  label='Actual data') plt.plot(X  predictions  color='blue'  label='Predicted data') plt.xlabel('X') plt.ylabel('y') plt.title(title) plt.legend() plt.show() print(fInitial prediction: {predictions}) # Plot initial prediction plot_boosting(X  y  predictions  Initial Prediction) # Boosting iterations for i in range(n_iterations): # Calculate residuals residuals = y - predictions print(fIteration {i+1}: Residuals: {residuals}) # Fit decision tree to residuals tree = DecisionTreeRegressor(max_depth=1) tree.fit(X  residuals) prediction_update = tree.predict(X) print(fIteration {i+1}: Prediction Update: {prediction_update}) # Update predictions predictions += learning_rate * prediction_update print(fIteration {i+1}: Updated Predictions: {predictions}) # Plot updated predictions every 20 iterations if (i + 1) % 20 == 0: plot_boosting(X  y  predictions  fIteration {i+1} - Updated Predictions) print(fFinal Predictions: {predictions}) # Final plot plot_boosting(X  y  predictions  Final Predictions) GradientBoostingRegressor Method:    import numpy as np from sklearn.ensemble import GradientBoostingRegressor import matplotlib.pyplot as plt # Data X = np.array([[1]  [0]  [2]  [3]]) y = np.array([0  0  3  -1]) # Initialize and fit the model model = GradientBoostingRegressor(n_estimators=100  learning_rate=0.5  max_depth=1  random_state=0) model.fit(X  y) # Predictions predictions = model.predict(X) print(Predictions:  predictions) # Plotting the results plt.figure(figsize=(10  6)) plt.scatter(X  y  color='red'  label='Actual data') plt.plot(X  predictions  color='blue'  label='Predicted data') plt.xlabel('X') plt.ylabel('y') plt.title('Boosting Gradient Tree Model') plt.legend() plt.show()Here is the output of the raw method:   Here is the output of the GradientBoostingRegressor method:   By comparing the raw boosting with the GradientBoostingRegressor from SKLEARN  we can better understand the inner workings of the boosting algorithm and how it iteratively improves the models performance.   General Framework and Mathematical FoundationsBased on the example in the previous section. we summarize the following general boosting procedure:   1. Initialization:   The initial model is typically a simple model  such as predicting the mean   of the target values.   2. Iterative Process:   For each iteration i  the following steps are performed:   Calculate errors:   The errors e_i represent the discrepancies between the actual target values y and the predictions from the previous model. The error function can be Mean Absolute Percentage Error (MAPE)  Mean Squared Error (MSE)  or others.   Fit Model to Errors:   Update the Model:   In addition  the predictions are updated by adding the new models predictions  scaled by a learning rate   to the previous models predictions.   We now investigate the logistics for the boosting steps above; this is related to minimizing a loss function L(y  f(x)) that measures the difference between the actual target values y and the models predictions f(x).   A loss function is used in machine learning to see how close a models predictions are to the real data. Think of it as a way to measure the error or badness of the model predictions. The lower the loss function value  the better the model is performing. It helps to calculate the total error between the model and the sample data. The loss function can often be explicitly expressed as a math formula  such as linear and logistic regressions  but there might be no simple math form like a decision tree and neural networks.   The steps can be mathematically formalized as follows:   The initial model f_0(x) is chosen to minimize the loss function over the training data.   2. Gradient Descent on Errors: For each iteration i: Compute the Gradient and Hessian:   The gradient represents the direction of the steepest increase in the loss function. The Hessian provides information about the curvature of the loss function. Computing the gradient and Hessian is essential because we use them in building the decision tree model below.   Fit a Decision Tree:   In this step where we fit a Decision Tree  the tree model m_i(x) is trained to predict the gradient g_i with a regularization term that involves the Hessian h_i. Here  the objective function (i.e. the approximation of the loss function) combines the gradient and the Hessian to determine the optimal split and leaf in the decision tree. The formulation   ensures that the tree is trained to approximate the gradient and account for the curvature from the Hessian. Therefore  the tree model primarily predicts the gradient while being influenced by the Hessian to improve the robustness and stability of the model.   Regularization is used in the boosting method to prevent overfitting and ensure that the model generalizes well to new data. The regularization term (f) can include both L1 and L2 penalties:   where  and  are regularization parameters  and w_j are the weights of the leaf nodes in the decision tree. The final model after M iterations is given by:   Mathematically  this process can be seen as a series expansion where each term incrementally improves the approximation of the target function. By understanding these steps  data scientists can appreciate how boosting leverages simple models to build a highly accurate ensemble model.   How XGBoost Works?XGBoost (Extreme Gradient Boosting) is an advanced implementation of gradient boosting that includes features like regularization to prevent overfitting. In this post  I will explain the mathematical foundation of the XGBoost algorithm  focusing on error calculation  optimal weight determination in each iterative step  and the logic behind these calculations.   1. Initialization   The initial prediction for each input value is typically the mean of the target   values y:   2. Iterative Boosting Process   For each boosting iteration m from 1 to M:   a. Compute Residuals (Gradients and Hessians)   The following gradients represent the first derivatives of the loss function concerning the predictions:   The Hessians represent the second derivative of the loss function for the predictions:   b. Fit a Decision Tree to the Residuals   A decision tree is fitted to the gradients. The tree model h_m(x) is trained to minimize the loss function:   c. Optimal Leaf Weights   The optimal weight w_j for each leaf j of the tree is calculated to minimize the loss function. The weight for each leaf is given by:   where I_j is the set of data points in leaf j  g_i are the gradients  hi are the Hessians  and  is the regularization parameter.   This weight is used to adjust the gradient prediction of the decision tree model.   After completing all iterations  the final model is the sum of the initial model and all the subsequent prediction updates:   Lets construct a simple XGBoost algorithm using the raw boosting method  where decision tree regression is used as the gradient prediction model combined with iterative steps:   import numpy as np from sklearn.tree import DecisionTreeRegressor import matplotlib.pyplot as plt # Data X = np.array([[1]  [0]  [2]  [3]  [0]]) y = np.array([0  0  3  -1  1]) # Number of boosting iterations n_iterations = 700 # Learning rate learning_rate = 0.1 # Regularization parameters lambda_reg = 1.0 # L2 regularization term # Initial prediction (constant model) initial_prediction = np.mean(y) predictions = np.full(y.shape  initial_prediction) print(fInitial prediction: {predictions}) # Define the loss function and its gradient and hessian def loss(y  y_pred): return (y - y_pred) ** 2 def gradient(y  y_pred): return -2 * (y - y_pred) def hessian(y  y_pred): return 2 * np.ones_like(y_pred) # Boosting iterations for i in range(n_iterations): # Calculate gradients and Hessians gradients = gradient(y  predictions) hessians = hessian(y  predictions) # Fit a decision tree to the gradients tree = DecisionTreeRegressor(max_depth=1) tree.fit(X  gradients) prediction_update = tree.predict(X) # Update predictions with regularization predictions -= learning_rate * prediction_update / (hessians + lambda_reg) # Debugging output if (i + 1) % 20 == 0 or i == 0: print(fIteration {i+1}:) print(f Gradients: {gradients}) print(f Hessians: {hessians}) print(f Prediction Update: {prediction_update}) print(f Updated Predictions: {predictions}) # Plot updated predictions plt.figure() plt.scatter(X  y  color='red'  label='Actual data') plt.plot(X  predictions  color='blue'  label='Predicted data') plt.xlabel('X') plt.ylabel('y') plt.title(f'Iteration {i+1} - Updated Predictions') plt.legend() plt.show() print(fFinal Predictions: {predictions})Code Explanation:We first prepare the input data x and target values y.The initial prediction is set as the mean of the target values.Define the loss function along with its gradient and Hessian.Iterate to calculate the gradients and Hessians  fit a decision tree to these gradients  and update the predictions. Regularization is applied to ensure robustness.Output and visualize the predictions at regular intervals to observe the convergence.The final predictions will be displayed after 700 iterations  with intermediate results visualized every 20 iterations to show the models progress in learning from the residuals.   Here is the output:   Final ThoughtsUnderstanding the math behind boosting algorithms and building your own boosting models is important for effectively applying these techniques in machine learning. This knowledge lets you fine-tune parameters  optimize predictions  and achieve better results. It also enables you to innovate and develop new boosting methods tailored to specific needs.   Remember  the ability to adjust and iterate based on mathematical understanding makes a good data scientist stand out. Keep trying new things  fine-tuning  and learning  boosting has much to offer for those who understand it well.