Diagnose the Generalized Linear Models
Tags: Model, Linear Models, Regression, Data Science, R

Generalized Linear Model (GLM) is popular because it can deal with a wide range of data with different response variable types (such as binomial, Poisson, or multinomial).

Comparing to the non-linear models, such as the neural networks or tree-based models, the linear models may not be that powerful in terms of prediction. But the easiness in interpretation makes it still attractive, especially when we need to understand how each of the predictors is influencing the outcome.

The shortcomings of GLM are as obvious as its advantages. The linear relationship may not always hold and it is really sensitive to outliers. Therefore, it’s not wise to fit a GLM without diagnosing.

In this post, I am going to briefly talk about how to diagnose a generalized linear model. The implementation will be shown in R codes.

There are mainly two types of diagnostic methods. One is outliers detection, and the other one is model assumptions checking.

Residuals

Before diving into the diagnoses, we need to be familiar with several types of residuals because we will use them throughout the post. In the Gaussian linear model, the concept of residual is very straight forward which basically describes the difference between the predicted value (by the fitted model) and the data.

Response residuals

In the GLM, it is called “response” residuals, which is just a notation to be differentiated from other types of residuals.

The variance of the response is no more constant in GLM, which leads us to make some modifications to the residuals.

If we rescale the response residual by the standard error of the estimates, it becomes the Pearson residual.

Pearson residuals

You may see some people use the square root of the estimates instead of the standard error as the denominator in the equation above. That’s because they are using the Poisson model where the estimated variance equals the estimated mean. So, don’t be confused, they are the same thing.

One of the characters of the Pearson residuals is that the sum of squared Pearson residuals approximately follows the chi-squared distribution.

The Sum of squared Pearson residual approximately follows a chi-squared distribution

This character inspires researchers to use another type of residual named deviance residual, the sum of squared of which also follows the chi-squared distribution.

Note: The deviance of a model simply describes the goodness of fit, which specifically calculates the difference between the log-likelihood for the full (saturated) model and that for the model under consideration (the model you built for fitting). High deviance indicates a bad fitted model.

Each data point has its own contribution to the model’s deviance. If we assign a direction to each of these individual deviance based on the difference between the fitted value and the data, we get the deviance residual.

Deviance residual

Usually, the deviance residual is preferable to the other types of residuals in the diagnostics of GLMs.

In R, it’s simple to implement these different types of residuals using the ‘residuals’ function.

# R code

# Suppose your fitted model is mod residuals(mod, type = "response") ## response residuals

residuals(mod, type = "pearson") ## pearson residuals

residuals(mod, type = "deviance") ## deviance residuals

The default performance of the function ‘residuals’ is deviance residual, so don’t worry if you forget to pass the ‘type’ argument to the function.

Model Diagnostics

The plot of residuals against fitted values is the most important graphic in the diagnostics. The plot aims to check whether there is evidence of nonlinearity between the residuals and the fitted values.

One difference between the GLMs and the Gaussian linear models is that the fitted values in GLM should be that before the transformation by the link function, however in the Gaussian model, the fitted values are the predicted responses.

Let’s check the following Poisson model as an example. Remember the Poisson regression model is like this:

Poisson Model

Let’s plot the residuals against estimates (mu_i) first.

# R code

mod_pois = glm(Species ~ ., family=poisson,gala) # gala is the example dataset in package "faraway"

plot(residuals(mod_pois) ~ predict(mod_pois,type="response"),xlab=expression(hat(mu)),ylab="Deviance residuals",pch=20,col="red")



Deviance residuals against estimates

We can see that most points are squeezed at the left side of the plot, which makes it hard to interpret. Therefore, it’s better to check that without the link function’s transformation.

# R code

plot(residuals(mod_pois) ~ predict(mod_pois,type="link"),xlab=expression(hat(eta)),ylab="Deviance residuals",pch=20,col="blue")

Deviance residuals against \eta

If the model’s assumption is met, we do expect a constant variation in the plot because the deviance residuals should not have the nonconstant variation that is already rescaled out. If there’s an obvious nonlinearity in the plot, it will raise the warning sign. However, this plot looks OK without any nonlinearity.

Let’s say if there is an obvious nonlinearity in the plot, what should we do?

Usually, it’s not wise to start with changing the link function or transform the response variables. The selection of the link function is usually limited with the natural features of your response data, for example, the Poisson model will require the response variable to be positive. Similarly, the transformation of the response variable will violate the assumed distribution of the response.

It’s worth trying two things:

change the model type. For example, utilize a negative binomial model instead of the Poisson model. Do some transformation to the predictors, like log(), sqrt(), and so on…

Outliers Detection

Another important part of diagnostics on GLMs is to detect the outliers. It could be done either quantitatively or graphically.

In the quantitative way of detecting the outliers, the basic idea is to find those points that have an abnormally large influence on the model or those that the fitted model is most sensitive to.

Two metrics could be used on a fitted model: the leverage and the Cook’s distance. Here are the codes in R.

# R code

i_n = influence(mod_pois)$hat # calculate the influence of data points

which.max(i_n)

which yields,

## SantaCruz

## 25

It means the data point “SantaCruz” has the largest influence on the fitted Poisson model.

If we apply the Cook’s distance metric, it will yield the same result.

# R code

c_d = cooks.distance(mod_pois)

which.max(c_d)

the result is,

## SantaCruz

## 25

We can also detect outliers graphically, which uses the QQ plot. One difference from the Gaussian linear models’ diagnostics, we are not looking for a straight line in the QQ plot in GLM diagnostics because the residuals are not expected to be normally distributed. The only purpose of the QQ plot in GLM is to find the outliers in the data.

# R code

halfnorm((i_n))

QQ plot of the leverages

# R code

halfnorm((c_d))

QQ plot of Cook’s distances

These methods suggest further investigation on the data point “25 SantaCruz ”.

The Takeaways

It’s important to run the diagnostics in the GLMs. Two aspects need investigation: model assumptions and outliers. Transformation on predictors but not on response variables if necessary. Remove outliers that are detected both quantitatively and graphically.

Hope this post is helpful to you.

References:

Faraway, Julian J. Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models. CRC press, 2016.