Fitting GLMs by Hand. Using Maximum Likelihood and Gradient... | by Daniel Friedman | Towards Data Science

In ordinary linear regression, we treat our outcome variable as a linear combination of several input variables plus some random noise, typically assumed to be Normally distributed. While this modeling approach is easily interpreted, efficiently implemented, and capable of accurately capturing many linear relationships, it does come with several significant limitations. A simple extension of linear models, a Generalized Linear Model (GLM) is able to relax some of linear regression’s most strict assumptions. These assumptions include:
Linearity between the outcome and input variablesNormal distribution of error termsConstant variance of error terms.
Linearity between the outcome and input variables
Normal distribution of error terms
Constant variance of error terms.
Relaxing these assumptions allows us to fit much more flexible models to much broader data types.
GLMs can be easily fit with a few lines of code in languages like R or Python, but to understand how a model works, it’s always helpful to get under the hood and code it up yourself. This article shows how to implement GLMs from scratch using only Python’s Numpy package. For more on the basics and intuition on GLMs, check out this article or this book.
Fitting a GLM first requires specifying two components: a random distribution for our outcome variable and a link function between the distribution’s mean parameter and its “linear predictor”.
The first step to building our GLM is identifying the distribution of the outcome variable. If the data has a binary response, we might want to use the Bernoulli or Binomial distributions. If we are working with count data, a Poisson model might be more useful. This distribution is typically assumed to come from the Exponential Family of distributions, which includes the Binomial, Poisson, Negative Binomial, Gamma, and Normal.
Each of these models can be expressed in terms of its mean parameter, μ = E(Y). For instance, we specify a binomial model as Y ~ Bin(n, p), which can also be written as Y ~ Bin(n, μ/n). Once we estimate μ, we model Y as coming from a distribution indexed by μ̂ and our predicted value of Y is simply μ̂.
In a GLM, we estimate μ as a non-linear function of a “linear predictor” η, which itself is a linear function of the data. The non-linear function connecting μ to η is called the link function, and we determine it before model-fitting. The link function is written as a function of μ, e.g. η = g(μ).
Suppose we have the following training data where each x is a D-dimensional vector:
We first write η as a linear function of x for each observation n = 1, ... , N:
Then we connect η to μ with the link function:
To fit the GLM, we are actually just finding estimates for the βs: from these, we obtain estimates of η, which leads immediately to an estimate for μ, which then gives us an estimated distribution for Y! To estimate the βs, follow these steps:
Specify the distribution of Y as a function of μ.Specify the link function, η = g(μ).Identify a loss function. Here, we use the negative log-likelihood. We can decompose the loss function into a function of each of the linear predictors and the corresponding true Y values as shown in the image below.Find the β values to minimize the loss function, either through a closed-form solution or with gradient descent.
Specify the distribution of Y as a function of μ.
Specify the link function, η = g(μ).
Identify a loss function. Here, we use the negative log-likelihood. We can decompose the loss function into a function of each of the linear predictors and the corresponding true Y values as shown in the image below.
Find the β values to minimize the loss function, either through a closed-form solution or with gradient descent.
To reinforce our understanding of this structure, let’s first write out a typical linear regression model in GLM format. As step 1, let’s specify the distribution of Y. Recall that a typical linear model assumes
where β is a length-D vector of coefficients (this assumes we’ve added a 1 to each x so the first element in β is the intercept term). Note that the mean of this distribution is a linear combination of the data, meaning we could write this model in terms of our linear predictor by letting
Then for step 2, we need to find the function linking μ and η. In the case of linear regression, it’s simple. Since E(Y) = μ and the mean of our modeled Y is η, we have η = g(μ) = μ!
Step 3: let’s find the negative log-likelihood.
Note that our loss function is proportional to the sum of the squared errors.
Finally for step 4, let’s see if we can minimize this loss function analytically. Start by taking the derivative with respect to β and setting it equal to 0.
This gives the closed-form solution we know and love from ordinary linear regression.
Now for the simple coding. Let’s randomly generate some normally-distributed Y values and fit the model. The scatterplot below shows that our fitted values for β are quite close to the true values.
import numpy as npimport matplotlib.pyplot as plt%matplotlib inlineimport seaborn as snsnp.random.seed(123)## GENERATE STUFF ### Generate XN = 1000  # number of observationsD = 10 # dimension of each observationX = np.random.randn(N, D-1) # (minus 1 because we have to add the intercept)intercept = np.ones(N).reshape(N, 1) X = np.concatenate((intercept, X), axis = 1)# Generate true betabeta = np.random.randn(D)# Generate response as function of X and betay = X @ beta + np.random.randn(N)## FIT STUFF ##beta_hat = np.linalg.inv(X.T @ X) @ X.T @ y## PREDICT STUFF ##y_hat = X @ beta_hat## PLOT ##fig, ax = plt.subplots()sns.scatterplot(beta, beta_hat)ax.set(xlabel = 'True Betas', ylabel = 'Estimated Betas', title = 'Beta vs. Beta Hats');
In logistic regression, we model our outputs as independent Bernoulli trials. I.e. we assume
That completes step 1. For step 2, we must find a way to relate our linear predictor η to our parameter p. Since p is between 0 and 1 and η can be any real number, a natural choice is the log-odds. I.e.,
Inversely, we use the sigmoid function to get from η to p (which I will call S):
This wraps up step 2. For step 3, find the negative log likelihood.
This gives us our loss function and finishes step 3. For step 4, we find the values of β to minimize this loss. Unfortunately, in the logistic regression case, there is no closed-form solution, so we must use gradient descent. So, let’s find the derivative of the loss function with respect to β. First, note that S’(x) = S(x)(1-S(x)):
Then we can derive the entire gradient:
To speed up calculations in Python, we can also write this as
where
and
Now let’s fit the model using gradient descent. Again, the scatterplot below shows that our fitted values for β are quite close to the true values.
np.random.seed(123)## GENERATE Ys ### Generate response as a function of the same X and beta# only now it's a Bernoulli r.v. def sigmoid(x):    return 1/(1 + np.exp(-x))p = sigmoid(X @ beta)y = np.random.binomial(1, p)## FIT WITH GD ##nruns = 10**5learning_rate = 0.0001beta_hat = np.random.randn(D) # initialize beta_hat estimatesp_hat = sigmoid(X @ beta_hat) # initialize p_hatsfor run in range(nruns):    # get gradient     a, b = y*(1-p_hat), (1-y)*p_hat     grad = X.T @ (b-a)    # adjust beta hats    beta_hat -= learning_rate*grad    # adjust p hats    p_hat = sigmoid(X @ beta_hat)## PLOT ##fig, ax = plt.subplots()sns.scatterplot(beta, beta_hat)ax.set(xlabel = 'True Betas', ylabel = 'Estimated Betas', title = 'Beta vs. Beta Hats');
The Poisson is a great way to model data that occurs in counts, such as accidents on a highway or deaths-by-horse-kick.
Step 1: Suppose we have
Step 2, we specify the link function. The link function must convert a non-negative rate parameter λ to the linear predictor η ∈ R. A common function is
which of course has inverse
Now for step 3, find the negative log-likelihood.
And step 4, optimize for β.
Sadly, there is no closed-form solution, so again we turn to gradient descent, as implemented below. Once again, the estimated parameters are plotted against the true parameters and once again the model does pretty well.
np.random.seed(123)## GENERATE Ys ### Generate response as a function of the same X and beta# only now it's a Poisson r.v. beta = np.random.randn(D)/2lam = np.exp(X @ beta)y = np.random.poisson(lam = lam)## FIT WITH GD ##nruns = 10**5learning_rate = 0.00001beta_hat = np.random.randn(D)/2 # initialize beta_hat estimateslam_hat = np.exp(X @ beta_hat) # initialize lam_hatsfor run in range(nruns):    # get gradient     c = y - lam_hat    grad = X.T @ c    # adjust beta hats    beta_hat -= learning_rate*grad    # adjust lambda hats    lam_hat = np.exp(X @ beta_hat)## PLOT ##fig, ax = plt.subplots()sns.scatterplot(beta, beta_hat)ax.set(xlabel = 'True Betas', ylabel = 'Estimated Betas', title = 'Beta vs. Beta Hats');
When building GLMs in practice, R’s glm command and statsmodels’ GLM function in Python are easily implemented and efficiently programmed. But becoming familiar with the structure of a GLM is essential for parameter tuning and model selection. When it comes to modeling, often the best way to understand what’s underneath the hood is to build the car yourself.