From Logistic Regression to Basis Expansions and Splines
Tags: Statistics, Editors Pick, Machine Learning, Mathematics, Data Science

From Logistic Regression to Basis Expansions and Splines

Nick Fewings via Unsplash

Introduction:

Logistic Regression is a common method used for fitting a binary or categorical response variable. But did you know that if you are not careful, logistic regression can miss out on important features?

In this in depth article, we will use the South African Heart Disease data, in combination with Chapters 4 and 5 of The Elements of Statistical Learning II to demonstrate why you need to be careful when applying logistic regression. We will also use this example as a starting point to introduce basis functions and splines.

In particular, we will learn about non-linearity, and how non-linear features can be incorporated into models. It will be fun, so, if you want to learn more intermediate statistical techniques, or are just looking for a review, please read along. The code used can be found on my github.

The Data:

The South African Heart Disease dataset comes from a real 1983 retrospective study and consists of 462 observations, with the response of interest being Myocardial Infarction, or coronary heart disease (chd) (0/1 or no/yes). The dataset includes several numeric predictors such as Systolic Blood Pressure (sbp), Total Kilograms of Tobacco smoked (tobacco), obesity, alcohol, ldl, and age. There is also a binary factor variable (famhist (Present or Absent)).

Which predictors do you think will be most significant when modelling? Try to choose some before continuing.

We will now demonstrate the general approach that one might take for creating a model for chd. We will start with the usual first steps, examining the data and checking out the predictors.

Linear Methods:

Firstly, you should always examine structured data if possible. R comes with the ‘View’ function which allows us to conveniently see what our data looks like. (Note, we remove predictors adiposity and typea as these were not used in the reviews/books).

Figure 1: The first 20 observations from the South African Heart Disease dataset. Image by Author.

After looking at the data in a basic form, we want to get an idea about how our predictors are correlated with one another, and how they are correlated with the response. We can do this using a pairs plot, which is also built into R with the ‘pairs’ function. We colour our points according to whether the patient was healthy (black points) or had a heart attack (red points).

Figure 2: Pairs plot of the predictors, black represents healthy individuals, red indicates those who had a heart attack (chd). Image by Author.

Logistic Regression:

To classify the response, chd, we are simply trying to classify a binary response. We thus attempt to model the probability that our response belongs to one group, given the predictors X.

We model the so called logit, this ensures that our estimates remain in the interval [0,1] as we are modelling a probability.

Figure 3: The general form of modelling using logit odds. Image by Author.

Figure 4: Taking exp of both sides. Image by Author.

Figure 5: Simplifying the terms. Image by Author.

Logistic Regression is the usual go to method for problems involving classification. R allows for the fitting of general linear models with the ‘glm’ function, and using family=’binomial’ allows us to fit a response. Logistic Regression models are often fit using maximum likelihood using iterated reweighed least squares.

Fitting Logistic Regression:

Logistic Regression can be fit using iterated reweighed least squares or minimisation of a cost function. In R, we can fit logistic regression for a binary response using the ‘glm’ function and specifying the family as ‘binomial’.

Here is the result of naively applying logistic regression to the heart data:

Figure 6: Applying logistic regression on the entire dataset provides these estimates and standard errors. Stars indicate significance. Image by Author.

Think back to the question I asked about choosing some important variables. It is very likely you chose famhist (that would be correct), but you also probably choose one of obesity or blood pressure (sbp). We know that those definitely have an impact on heart disease, so why is logistic regression indicating these are not significant predictors?

This example helps illustrate the problem of colinearity. According to Figure 6, obesity has a negative estimate and is not significant, which seems to go against all that we know from medical science. Also, according to the Figure 6, systolic blood pressure is not significant, which seems odd. The issue is, we have correlation. Modelled independently, sbp and obesity have negative signs and are significant.

Let us apply backward step selection, which is recommended when you believe there is correlated predictors. R has the built in ‘step’ function.

Figure 7 : Applying backward selection on AIC criteria leaves us with less predictors, but all are significant. Image by Author.

Our final model apparently does not include obesity or sbp!

Interpreting coefficients:

Before we move on, it is worth it to review how to interpret the summary table (Figure 7). The estimate for tobacco is 0.0807 with a standard error of 0.0255. Since tobacco was measured in total lifetime usage in kilograms, a 1 unit increase in tobacco represents an exp(0.0807)=1.084 increase in logit odds for chd. That is an 8.4% increase in heart attack! We can also calculate exp(0.0807+/ 2*0.255)=(1,03,1.14) to get the 95% confidence interval, so it ranges from a 3% to a 14% increase.

But are we done? Does obesity and sbp really not have an impact on heart attack? If we stopped there, our model still might be okay. But it seems something is going wrong. The reason is that above we fit linear logistic regression. But there are NON linear features. To understand this, we need to understand basis expansions. Come along.

Non-Linearity: Basis Expansions and Splines

Above we saw that simply using linear functions to model our response didn’t seem to fully address our concern. Moreover, ‘ reality’ or the ‘true function’ is almost never linear, so what are the chances that fitting a model using only linear methods will be sufficient? True, linear models are still wildly popular and useful. They are certainly necessary, but are they sufficient?

Basis Expansions:

When we fit linear models such as linear regression or linear logistic regression we modelled our function in the form:

Figure 8: The general form of a linear model. Image by Author.

To model non-linearity, we don’t mean our weights β. We instead refer to the terms, or the basis functions, multiplying the weights;

That is, we form models like this instead;

Figure 9: The general form of a model made using basis expansions. Image by Author.

Figure 10: Are basis function maps from a higher dimensional space to R. Image by Author.

We will see why this is useful and how it can be implemented.

Local Fitting:

Suppose X is a one-dimensional set of observations. By separating the domain of X into adjoining regions, and fitting a polynomial to each region separately, we can start to get at the idea of fitting more complicated functions.

For example, see the Figure below which shows the division of the domain of X into three regions, separated at points Ɛ1 and Ɛ2. These are often referred to as ‘knots’.

Figure 11: Image Citation: Elements of Statistical Learning II. This image shows piecewise constant models fit to each region with no constraints on continuity.

The function in blue represents some real function. The points around it were generated from the function and then adding noise. In practice, we receive noise and want to find the true underlying structure, the blue curve.

The green line indicate the model fit using the following three basis functions and least squares;

Figure 12: The most basic way we could fit the below curve, using piecewise constant fit and identity function. Image by Author.

Figure 13: The model would be represented like this, with h as above. Image by Author.

The green lines indicate the least squares solution in each interval which is;

Figure 14: The solution for the weights for the piecewise linear solved using linear regression. Image by Author.

That is a very simple case, and clearly the fit is not great. Moving on, we could perform simple linear regression in each interval. This requires an additional three basis functions that allow us to model the slope term;

Figure 15: The three additional basis functions to allow for slope. Image by Author.

Figure 16: Image Citation: The Elements of Statistical Learning II. This image shows simple linear regression applied in each interval.

And each solution is found using the simple linear regression formula for the weights as usual.

But we are still missing something. We want to enforce continuity. So we should make the constraints that we touch on the intervals;

Figure 17: Constraint 1. Image by Author.

And;

Figure 18: Constraint 2. Image by Author.

Which results in the following fit:

Figure 19 : Image Citation: The Elements of Statistical Learning II. Simple Linear Regression with enforced continuity.

Which still doesn’t look great.

Non-Linear:

Above we considered fitting linear functions locally in each interval, but the major benefit of basis functions is we can also model non linear functions.

Figure 20 : Image Citation: The Elements of Statistical Learning II.

The bottom right image is continuous and has continuous first derivative and a continuous second derivatives. This thus known as a CUBIC SPLINE. The following six basis functions define a cubic spline.

Figure 21 : The six basis functions that define the cubic spline. Image by Author.

The top left shows polynomial regression fit to each interval. The top right shows polynomial regression with enforced continuity. The bottom left shows polynomial regression with enforced continuity and enforced continuity of the first derivative. The bottom right, the cubic spline has enforced continuity of the second derivative as well.

We have discussed the constraints, there is another constraint that forms what is known as the natural cubic spline, this constraint is that the function is linear past the boundary knots, two constraints each in both boundary regions, a natural cubic spline with K knots is represented by K basis functions. We will fit our SA heart data with predictors with natural cubic splines.

Now that we have reviewed basis functions and non-linearity, we can return to the South African Heart disease data! While before we modelled the logit just using the X terms in isolation, we now repeat but allow for the X terms themselves to be modelled by a natural spline. Thus our estimate for chd will have the form;

Figure 22 : The form for coronary heart disease with basis functions of the predictors. Image by Author.

We fit each of our numeric predictors using a natural spline with four basis functions. Notice that sbp and obesity are non-linear.

Figure 23: Image Citation: The Elements of Statistical Learning II.

We then created a logistic regression using the expanded predictors, which included the non-linear terms.Instead of simply multiplying the coefficient by itself by a weight, we multiply a weight by the BASIS function of the parameter, with the basis function being a non-linear fit using splines of that variable! We then used the drop function to remove terms that were not significant. You can fit spline terms in R with the ‘ns’ function. Input the df to tell R how many basis functions you want the spline to fit.

Figure 24 : Final logistic regression model after applying backward stepwise selection to a model which included splines. Image by Author.

Our final model includes higher order terms, including obesity and sbp! It turns out, indeed they were important, but they were lost by not considering that they had a non-linear impact.

We can confirm they are important in the model by seeing what happens if we remove them from the model.

Figure 25: Indeed, we shouldn’t drop any of these features. Image by Author.

Figure 25 summaries the Likelihood ratio test when a spline term is deleted from the model. Notice, removing ‘no’ terms keeps the AIC the lowest. Removing any other spline terms raises the AIC. Hence keeping obesity and sbp is indeed useful.

Be careful using simple linear logistic regression! It may miss out on non-linear features, and it has issues with correlation among predictors. Splines are a way to fit non-linear features and the LRT (Likelihood Ratio Test) can tell you the impact of removing higher order splines from model.

So why did we see an increase in heart attack at low values of sbp and obesity? It turns out this was due to the method of the original study, it was a retrospective study, and measurements were made after a heart attack had occurred. Hence, patients had already begun to fix their lifestyle (weight and blood pressure). For example, they had a heart attack and so were extra diligent to lower obesity, this likely explains it!

Extras: Multidimensional Splines

Above we studied one-dimensional splines. But indeed, they can be expanded to multiple dimensions. The details are a bit much for an article, but involves forming the tensor product basis. Below is an example of fitting a thin plate spline to three of the heart disease risk factors and representing the contour plot, the green and red points are the knots.

Figure 26: Image Citation: The Elements of Statistical Learning II.

Summary:

In this article, we reviewed a common real dataset and saw how logistic regression can be fit to model a response variable. We also saw how logistic regression can fail to pick up on important features, especially when there is non-linearity or correlation in the predictors.

We also reviewed splines and basis functions, and saw how local polynomial regression can be used to fit better models. In our original model for heart disease, logistic regression missed out on obesity and sbp as important features. We had to consider non-linearity for our model to pickup the significance of these features.

Thank you for reading!

Also, if you liked this article, please check out my other Medium articles and consider leaving a clap or a comment, or sharing if you learnt something new.

If you want to connect, add me on LinkedIn.

Please see The Elements of Statistical Learning II for more. Images from the textbook are reprinted with permission from Trevor Hastie.

Code:

Github: https://github.com/Robby955/SAHeartDisease

Sources:

[1] Hastie, Tibshirani, Friedman (2009). The Elements of Statistical Learning, II edition.

[2] Rousseauw et al, (1983). South African Medical Journal.

[3] https://people.cs.clemson.edu/~dhouse/courses/405/notes/splines.pdf