Statistical Machine Learning: Gradient Boosting & AdaBoost from Scratch | by Andrew Rothman | Towards Data Science

Boosting is a family of ensemble Machine Learning techniques for both discrete and continuous random variable targets. Boosting models take the form of Non-Parametric Additive models and are most typically specified with additive components being “weak learners”. From an empirical risk decomposition perspective, where it can be easily shown the Mean Squared Error (MSE) of any arbitrary statistical estimator is the additive sum of the squared bias and sampling variance of said sampling estimator...
... a “weak learner” is a statistical estimator with high squared-bias but low sampling variance. Weak learners have the benefit of ease of model fitting from a computational perspective. Examples of weak learners include single decision and regression trees constrained to only a few layers of depth, possible even one layer (refereed to as “decision/regression stumps”).
The motivation of Boosting is to recover an additive ensemble of weak leaners that together can specify an arbitrarily complex model. In this piece, we will specify the theoretically framework and full mathematical derivation for Gradient Boosting and AdaBoost. We will also provide a full computational simulation of these methods from scratch, without use of Boosting computational packages.
The Table of Contents for this piece are as follows:
With Gradient Boosting, the model estimation procedure can be viewed as performing Gradient Descent in function-space over a space of proposal weak-learners (hence the name “Gradient Boosting”). Let’s jump into the mathematical specification and fitting procedure:
To understand the fitting procedure for Gradient Boosting Models, it is helpful to first recall the Taylor Expansion.
Let’s next discuss AdaBoost, which is in-fact a special case of the more general Gradient Boosting Model
With the information and notation outlined in section 3.1, let’s dive into the fitting procedure for AdaBoost
There is a slight alternative view of Boosting from a Non-Parametric Additive Model perspective. This is a view quite well understood in the Mathematical Statistics and Statistical Learning community, but one I rarely ever see covered in the Computer Science arm of the Machine Learning community. I think it’s a perspective worth understanding, and provides a more robust view of Boosting as a family of procedures.
There are a class of Statistical Models known as Non-Parametric Additive Models. This class of estimators is enormous, and are highly flexible. In a certain sense, these class of models is too flexible. As an an analyst leveraging a statistical model from this class, there are so many hyperparameters and design choices to choose from, specifying a the “right” model can be overwhelming.
For a given prediction problem, there is some underlying Non-Parametric Additive model that “works best” so-to-speak in a global sense. This “best” model we will refer to as the “true NPA model”. Given our sample data, we would like to do our best to specify and recover this “true NPA model”. But how can go about doing so in a structured and practical manner? The parameters space over which we need to search is enormous!
Let’s make an analogy for a moment. Imagine instead we had a Machine Learning problem for which we are leveraging Linear Regression. We have 100 features to choose from, and we know there is some true combination of features that provides the “best subset selection” Linear Regression Model. But we would have to test 2100 different models to recover the “best one”, which is a search space in the hundreds of trillions. Instead of attempting to search over that entire space, we could instead leverage a greedy algorithm like Forward-Selection where we will choose and add one variable to the final model at a time making our choices based on a greedy myopic heuristic. With this approach, we only need to test a few thousand models instead of hundreds of trillions. We know this Forward-Selection procedure is unlikely to recover the exact solution to the “best subset selection” problem; but it will recover a model that will likely work very well for our purposes.
The Forward-Selection scenario above is very similar to leveraging Boosting as a fitting procedure for Non-Parametric Additive models in a Machine Learning context. We would like to recover an estimate to the “true NPA model”, however the size of the search space is overwhelming. Boosting in this sense can be viewed as a family of greedy algorithmic procedures for estimating the solution to the “true NPA model” problem, similar to estimating the best subset-selection problem with Forward Selection.
Below is a fully working example (built from scratch) of fitting a Gradient Boosted Model on a continuous target, and an AdaBoost Model on a binary categorical target.
First, let’s load our needed libraries:
####################
## load libraries ##
####################
import numpy as np
import pandas as pd
np.random.seed(123456789)
from sklearn.tree import DecisionTreeRegressor
from sklearn.tree import DecisionTreeClassifier
Next, a function to simulate a dataset with a purposely complex model specification:
############################
## Toy Dataset Simulation ##
############################
def simulate_df(n=100, seed=123456, binary_flag=False):
    np.random.seed(seed)
    
    ## specify dataframe
    df = pd.DataFrame()

    ## specify variables L1 through L6
    L1_split = 0.52
    L2_split = 0.23
    L3_split = 0.38
    df['L1'] = np.random.choice([0, 1], size=n, replace=True, p=[L1_split, (1-L1_split)])
    df['L2'] = np.random.choice([0, 1], size=n, replace=True, p=[L2_split, (1-L2_split)])
    df['L3'] = np.random.choice([0, 1], size=n, replace=True, p=[L3_split, (1-L3_split)])
    df['L4'] = np.random.normal(0, 1, df.shape[0])
    df['L5'] = np.random.normal(0, 0.75, df.shape[0])
    df['L6'] = np.random.normal(0, 2, df.shape[0])
    
    theta_0 = 5.5
    theta_1 = 1.28
    theta_2 = 0.42
    theta_3 = 2.32
    theta_4 = -3.15
    theta_5 = 3.12
    theta_6 = -4.29
    theta_7 = -1.23
    theta_8 = -10.18
    theta_9 = 2.21
    theta_10 = 10.3
    
    if(binary_flag):
        Z = theta_0 + (theta_1*df['L1']) + (theta_2*df['L2']) + (theta_3*df['L3']) + (theta_4*df['L4']) + (theta_5*df['L5']) + (theta_6*df['L6']) + (theta_7*df['L2']*df['L4']) + (theta_8*df['L3']*df['L6']) + (theta_9*df['L5']*df['L5']) + (theta_10*np.sin(df['L5']))
        p = 1 / (1 + np.exp(-Z))
        df['Y'] = np.random.binomial(1, p)
        df.loc[df['Y']==0, 'Y'] = -1
    else:
        df['Y'] = theta_0 + (theta_1*df['L1']) + (theta_2*df['L2']) + (theta_3*df['L3']) + (theta_4*df['L4']) + (theta_5*df['L5']) + (theta_6*df['L6']) + (theta_7*df['L2']*df['L4']) + (theta_8*df['L3']*df['L6']) + (theta_9*df['L5']*df['L5']) + (theta_10*np.sin(df['L5'])) + np.random.normal(0, 0.1, df.shape[0])

    return(df)
Let’s recover a simulated dataset with a continuous outcome target Y, and fit a Gradient Boosting Model. Note, we will print the empirical realization of the Global Loss function with each weak learner added to the model. We will consider the model to have converged when the empirical loss < 1.
###############################################
## Gradient Boosting with Continuous Outcome ##
###############################################

## simulate the dataset with continuous outcome Y
df = simulate_df(n=1000, seed=123456)
df['Y_original'] = df['Y']

## fit Gradient Boosting Model with depth-3 Regression Trees as Weak Learners
## continue fitting model until loss function < 1
alpha = 100000
current_loss = sum((df['Y'])**2)
while(current_loss > 1):
    model = DecisionTreeRegressor(random_state=0, max_depth=3)
    model.fit(df[['L1', 'L2', 'L3','L4', 'L5', 'L6']], df['Y'])
    df['Y_hat'] = model.predict(df[['L1', 'L2', 'L3','L4', 'L5', 'L6']])
    df['Y_hat_squared'] = df['Y_hat']**2
    df['Y_hat_scaled'] = np.sqrt(df['Y_hat_squared'] / df['Y_hat_squared'].sum()) * np.sign(df['Y_hat'])
    loss_not_lowered_flag = True
    while(loss_not_lowered_flag):
        new_loss = sum((df['Y'] - (alpha*df['Y_hat_scaled']))**2)
        if(new_loss < current_loss):
            loss_not_lowered_flag = False
            current_loss = new_loss
            print('Current Loss: ' + str(current_loss))
            df['Y'] = df['Y'] - (alpha*df['Y_hat_scaled'])
        else:
            alpha = 0.99*alpha
    del model
print('model converged')
Let’s now recover a simulated dataset with a binary categorical outcome target Y, and fit an AdaBoost Model. Note, we will print the empirical realization of the Global Loss function with each weak learner added to the model. We will consider the model to have converged when all n=1000 training points are correctly classified by the ensemble model.
######################################################
## AdaBoost with Binary Categorical Outcome {-1, 1} ##
######################################################

## simulate the dataset with continuous outcome Y
df = simulate_df(n=1000, seed=123456, binary_flag=True)
df = df.rename(columns={'Y':'Y_original'})
df['w'] = df.shape[0]*[1/df.shape[0]]
df['Y'] = 0

## fit AdaBoost Model with depth-2 Decision Trees as Weak Learners
## continue fitting model until all n-1000 observations predicted correctly
count = True
while(count):
    model = DecisionTreeClassifier(random_state=0, max_depth=2)
    model.fit(df[['L1', 'L2', 'L3','L4', 'L5', 'L6']], df['Y_original'], sample_weight=df['w'])
    df['Y_hat'] = model.predict(df[['L1', 'L2', 'L3','L4', 'L5', 'L6']])
    df.loc[df['Y_hat']==df['Y_original'], 'w'] = 0
    epsilon = sum(df['w'])
    alpha = 0.5*(np.log((1-epsilon)/epsilon))
    
    df['Y'] = df['Y'] + (alpha*df['Y_hat'])
    
    current_loss = sum(np.exp(-df['Y_original']*df['Y']))
    psi = np.exp(-df['Y_original']*df['Y'])
    df['w'] = psi / current_loss
    
    print('Current Loss: ' + str(current_loss))
    
    df['Y_final'] = 1
    df.loc[df['Y']<0, 'Y_final'] = -1
    
    if(df.loc[df['Y_original']!=df['Y_final'], :].shape[0] == 0):
        print('results converged, all datapoints correctly classified')
        break
pd.crosstab(df['Y_original'], df['Y_final'])
For the complete code/notebook for the above computational simulation, please see the following github link.
I hope the above is insightful. As I’ve mentioned in some of my previous pieces, it’s my opinion not enough folks take the time to go through these types of exercises. For me, this type of theory-based insight leaves me more comfortable using methods in practice. A personal goal of mine is to encourage others in the field to take a similar approach. I will continue to write similar pieces in the future. Please subscribe and follow me here on Medium and on LinkedIn for updates!