Regression Splines in R and Python | by Yufeng | Towards Data Science

The linear model is named so because of the linear relationship between the input (independent variable) and the output (dependent variable). Even though we know it’s of a high probability that the real-world data shows nonlinearity, people usually keep regarding the linear model as one of the top choices.
The reasons for that are mainly two things. First, with acceptable approximation, the linear model is one of the simplest models to interpret. Second, the low complexity of the linear model makes it very unlikely to overfit the data, especially when you have small n (sample size) and large p ( variable number).
If the estimation bias from the linear model is too large to accept, we need to go beyond linearity in our model. It doesn’t mean we have to directly jump into Neural Networks, because a lot of simpler models are available there with less complexity (variance).
The most straightforward way to “go beyond linearity” is to simply raise the degree of the predictors. For example, a cubic regression uses X, X2, and X3 as predictors. “Cubic” means the highest power to variable X is 3 in the model. The training of a cubic regression model is also very simple, which estimates the coefficients of X, X2, and X3 as we do in a simple linear model with three independent variables. These nonlinear models that raise the power of the original predictors are called polynomial regression models.
Polynomial regression models are in fact a special case of a basis function approach. A basis function uses transformations applied to X: b1(X), b2(X), b3(X), ..., Bk(X) as the new variables instead of X in the linear model.
For polynomial regression, the basis functions are bj(x) = x^j.
In practice, one of the shortcomings of the polynomial regression model is that it fits a single polynomial function to the entire training data. In such a case, in order to follow most data points that span the entire range of X, the model curve would be super wiggly (we are talking about the situation where there is only one independent variable X). In another word, the single polynomial model requires high degree terms (eg. X5) to be that flexible. The higher degree of the polynomial model is, the more unstable the model is at the boundaries.
One of the solutions is to use piecewise polynomial models that cut the range of a variable into several distinct local regions, and within each region, a polynomial function is fit to the data. However, this type of piecewise polynomials has a major issue: it’s discontinuous at the knots (the local region boundaries).
If we force (constrain) the piecewise polynomials to join smoothly at the knots, we get a new methodology named regression splines. The smooth joint here means the two piecewise functions are continuous at the knot and also continuous in the first and second derivatives. One graph in “An Introduction to Statistical Learning” illustrates this concept very well as shown below.
In the plots above, we can see that the Regression Spline (the bottom left) yields a smooth join at the knots. If we only constrain the piecewise functions to be continuous, the joint point looks not that smooth, as shown in the top right plot.
Similar to the basis representation of polynomial regression, the regression spline can also be represented by the basis functions. Let’s use cubic spline with K knots as an example model. It can be framed as
The total number of basis functions is K+3 for cubic spline, where we use K+3 predictors in the least-squares regression. It has K extra predictors than a simple cubic model (X, X2, and X3 as the three predictors) because these extra functions are used to adjust to the changes at the knots.
One of the most frequently used basis functions in cubic splines is the truncated power basis function defined as,
where ξ is the knot.
So, in total, the cubic splines have the following predictors, X, X2, X3, h(X,ξ1), h(X,ξ2), ..., h(X,ξK).
Don’t be scared by the truncated power basis function. Its concept is very simple, which is just like a switch that is turned on when X is larger than its corresponding knot.
It’s hard to directly imagine the function in the cubic splines, so let’s look at the animation for a 1-degree spline. The GIF below is from this great post to illustrate how the truncated power functions are turned on.
The turning points of the black curve show exactly where the knots are. There are in total four knots in the plot above, therefore there are four extra truncated power functions highlighted by the four colored dashed lines.
It’s the same idea when we look at the truncated power functions in cubic splines.
There are also four “turning points” in the black curve and each truncated power function is turned on when X is larger than the corresponding knot.
One of the shortcomings of a regression spline is that the variance of the regression spline is huge when X is very small or very large (at the boundary regions). This can be addressed by adding boundary constraints to the model, where we force the model to be linear (1-degree of X) at the boundary regions. This is also called a natural spline. Since it’s not the focus of this post, we will talk about that in future articles.
Alright, after knowing the basic concepts of regression splines, let me show the implementations in R and Python.
Fortunately, there are already built functions for regression splines, so the standard implementation is very simple in both R and Python. I will only show the basic implementation, but you probably need to modify the codes specifically in real-world situations.
In R, we will use the example dataset Wage in the package ISLR. The R package to perform regression spline is splines.
library(splines)library(ISLR)attach(Wage)spl_mod <- lm(wage ~ bs(age, knots = c(30,50)), data = Wage)
where function bs() generates the entire matrix of basis functions for splines, knots takes the position of knots in X (here is the variable age), data = Wage specifies the data source, wage~ indicates the dependent variable is wage (variable name), and lm performs the least squared regression.
We can then check the fitting parameters by,
summary(spl_mod)
In Python, we cannot write it in one-line code as that in R, but we need to first generate the basis function matrix for splines, and then fit with the linear regression model.
We need several statistic model packages,
import pandas as pdimport numpy as npimport statsmodels.api as smfrom patsy import dmatrixdf = pd.read_csv('Wage.csv')basis_x = dmatrix("bs(df.age, knots=(30,50), degree=3, include_intercept=False)", {"df.age": df.age}, return_type='dataframe')
where df is the Wage dataset, basis_x is the matrix of the basis functions, and bs is the function to generate basis functions. The fitting step is,
spl_mod = sm.GLM(df.wage, basis_x).fit()
we can check the fitting parameters by,
spl_mod.params
That’s it.
If you fully understand the idea of regression splines, the R/Python codes are just a piece of cake because all you need is first to transform X to basis function matrix and second to fit a simple linear regression.
Cheers! I hope this short article is helpful.