Got GAM? a flexible modeling approach that preserves interpretability | by Onyi Lam | Towards Data Science

In many social science and business problems, it is often more important to explain why a phenomenon happens than improving the model’s predictability on the event happening. Having an interpretable model is, therefore, crucial in understanding how different factors interact with the outcome of interest.
The model’s interpretability is also important in highly regulated business environments, such as loan approval decisions. Even in situations where prediction accuracy is more important than the “why”, an interpretable model can help debug more complicated models and guide new approaches to feature engineering and data preprocessing.
In this context, generalized additive models (GAM) offer a middle ground between simple models, such as those we fit with linear regression, and more sophisticated machine learning models like neural networks that usually promise superior prediction performance to simple models. GAM can also be used in various tasks: regression, classification, binary choice.
In linear regression, we model outcome y as a function of 2 inputs X1 and X2 with the following:
y = β1X1 + β2 X2 + u
In GAM, the β Xi is replaced by f(Xi), where f() can be any arbitrary nonlinear functions. In other words, GAM is composed of a sum of smooth functions f() on the input. The idea is still that each input feature makes a separate contribution to the response, and these just add up, but these contributions don’t have to be strictly proportional to the inputs. The beauty of this approach is that, similar to what β represents in linear regression, the partial response function f() still captures the change on outcome y to change the input. The change in prediction depends on the initial value of Xi.
A common approach to deal with nonlinear relationships in regression models involves creating polynomial features. For the predictor in question, Xi, we add terms e.g. quadratic (Xi2), cubic (Xi3), etc to get a better fit. GAM encompasses this idea but includes an additional aspect: penalized estimation. The idea is similar to that of a ridge or lasso regression, where penalty terms are added to help avoid overfitting.
Another way of creating transformations of a variable is to cut the variable into distinct regions, and fit those regions separately. However, the different fits could be unconnected, leading to sometimes notably different predictions for values close together. GAM allows the user to specify the number ofknots in the variable, which are used to create sections where separate cubic polynomials are fit at each section and then joined to create a continuous curve.
A number of smooths are available with the mgcv package in R, and one can learn more via the help file for smooth.terms (link). The default is thin plate regression splines (TPRS), which works well in general in terms of performance. Cubic spline is also a common basis that mirrors adding polynomial terms for the covariates.
We use the mgcv package in `R` to implement gam and use a built-in dataset medcare from the package catdata to illustrate gam and compare with the performance using logistic regression.
The medcare data was collected on 4406 individuals, aged 66 and over, that were covered by a public insurance program. The outcome variable is healthpoor, a binary variable that equals 1 if the individual reports poor health and 0 otherwise. We are interested in knowing how ofp, the number of physician office visits, might be correlated with health outcomes. One can hypothesize a nonlinear relationship between the two variables: a healthy individual will have few physician visits, but having more visits also allow the patients to return to be more healthy. So we are interested to have a more flexible modeling approach.
The mgcv treats gam as a generalized version of glm, so one can directly call thegam method to use glm methods such as linear and logistic regression.
library(catdata)library(mgcv)library(Metrics)library(ggplot2)library(visreg)data(medcare)#### logit model (same as running glm with family set to "binomial")lm <- gam(healthpoor ~ male+ age + ofp, family = binomial, data = medcare)pred <- predict(lm, type="response")lm_auc <- Metrics::auc(medcare$healthpoor, pred)lm_auc#### generalized additive modelgam <- gam(healthpoor ~ male+s(age)+s(ofp), family = binomial, data = medcare)pred <- predict(gam, type="response")gam_auc <- Metrics::auc(medcare$healthpoor, pred)gam_auc
The s() terms in the gam formula indicate which terms are to be smoothed. There are several options you can pass on to the s() terms — for example, you can specify a different smooth function bs, and a different number of knots k.
If the number of unique values is less than the number of basis (For example, when you tried to add s() to a binary variable like male), the function would return the following error:
A term has fewer unique covariate combinations than specified maximum degrees of freedom.
Using visreg to visualize the fitted terms ofofp, we can see that gam produces a much more nuanced prediction of ofp while that of the logistic regression is much more linear.
visreg(gam, "ofp", jitter=TRUE, line=list(col="red"),       fill=list(col="green"))
The gam model also yields a slightly better AUC, 0.679, versus 0.673 of the logistic regression.
We are ultimately interested in estimating the change in ofp on the probability of healthpoor. To do that, we need to fix the other features’ values to their mean or mode(if the feature is a categorical variable). We then create a test dataframe that has a list of possible values of ofp.
# function to get mode of an array of valuesgetmode <- function(v) {  uniqv <- unique(v)  uniqv[which.max(tabulate(match(v, uniqv)))]}testdata = data.frame(ofp = seq(0, 100, length = 101),                      male = getmode(gam$model$male),                      age = mean(gam$model$age))
Using the predict function, we can then extrapolate the previously fitted gam model to the test data. The plot shows that ofp has a nonlinear relationship with the likelihood of reporting poor health, holding other variables constant. Furthermore, the confidence interval less tight in regions with higher ofp.
fits = predict(gam, newdata=testdata, type='response', se=T)### create a confidence interval for the fitspredicts = data.frame(testdata, fits) %>%   mutate(lower = fit - 1.96*se.fit,         upper = fit + 1.96*se.fit)ggplot(aes(x=ofp,y=fit), data=predicts) +  geom_ribbon(aes(ymin = lower, ymax=upper), fill='gray90') +  geom_line(color='#00aaff') +theme_bw()
For more information on GAM: