Smoothing Splines in R. This post discusses basic knowledge... | by Yufeng | Towards Data Science

One of the most frequently asked questions about regression splines is “How to determine the number and positions of knots?” A lot of methods out there can address this topic, but there’s a shortcut to avoid considering such questions in the modeling. It’s the smoothing splines.
Instead of requesting a sequence of pre-selected knots, smoothing splines take every unique value of X as a knot. Wait! Does it mean it adds at most N (N = the number of observations) extra degree of freedom to the original model? Is it overfitting?
Don’t panic. In order to address the overfitting problem, smoothing splines brings in a penalty term to its loss function. The penalty term encourages a smoother line and penalizes variance.
Specifically, the loss function of smoothing splines equals the sum of the residual sum of squares (RSS) and the smoothing term,
where g is the model function, λ is a nonnegative tuning parameter, and g′′ 2 is the squared second derivatives.
We can see that the first term in the loss function above is simply the RSS. The second term, the penalty term, encourages a smoother fitting. Why does the sum of second derivatives associate with the smoothness of the fitting?
As we know, the first derivative at point A measures the slope of the function at A. And the second derivate at A measures the change in the slope at A. Then, we look at the loss function equation again, we can easily understand that a more wiggly curve will result in a larger smoothing term. Since we aim to minimize the entire loss function, we prefer a smaller smoothing term, which corresponds to a smoother line. Pretty straightforward, isn’t it?
Does this idea of smoothing splines remind you of anything? Yes, I’m sure you recall the Ridge or LASSO regression that uses a penalty term in the loss function to shrink the coefficients of the linear model. Similar to the Ridge or LASSO regression, the loss function of smoothing splines has both the “Bias” term and the “Variance” term. The RSS measures the bias of the model and the smoothing term measures the variance of the model.
See? The bias-variance tradeoff is everywhere in statistical modelings. All we want is to find a balance between them, which best “describes” the real data.
You may have already noticed that with such settings in smoothing splines, the smoothing parameter λ plays a very important role in balancing the bias and variance.
If λ is super huge, it penalizes the roughness of the fitting function a lot, then the model is simply a straight line because a straight line has zero second derivatives.
If λ is 0, it doesn’t penalize the roughness at all, then the model is perfectly overfitting and follows every training point.
A more straightforward way to visualize a large λ versus a small λ,
The red dashed curve is for a larger λ, and the blue/black curves are for smaller λs. You can see the differences in the roughness of the curves.
Even though we avoid the selection of the number and the positions of the knots, we still need to tune the smoothing parameter λ in smoothing splines.
Thankfully, it can be simply solved by cross-validation. The λ corresponding to the least test set error is the final smoothing parameter to use.
For most models, the leave-one-out cross-validation (LOOCV) sounds more time and resource-consuming than k-fold cross-validation because it needs (N - k) extra iterations in the parameter tuning.
However, LOOCV is the simplest method for smoothing parameter tuning in smoothing splines. The reason is that the sum of LOOCV’s RSS can be calculated using only one original fit to all the data. This trick doesn’t apply to all models. If you are interested in the theoretical part about why it’s the case, you can refer to the classic book, ESL II by Trevor Hastie, Robert Tibshirani, and Jerome Friedman.
Alright, after knowing the basic concepts of smoothing splines, let me show the implementations in R.
Fortunately, there are already built functions for smoothing splines, so the standard implementation is very simple in R. I will only show the basic implementation, but you probably need to modify the codes specifically in real-world situations.
In R, we will use the example dataset Wage in the package ISLR. The R package to perform smoothing spline is splines. We will use the function smooth.spline(),
library(splines)library(ISLR)attach(Wage)spl_mod <- smooth.spline(age, wage, cv= TRUE) 
where age and wage indicate the independent and dependent variable is age and wage (variable name), respectively. Here cv means the function performs the cross-validation to select the best smoothing parameter automatically.
That’s it.
I hope this short article helps you better understand splines in modeling nonlinear data. If you are interested in my articles about data and modeling, please subscribe to my Medium account. Thank you!
stats.stackexchange.com
James, Gareth, et al. An introduction to statistical learning. Vol. 112. New York: springer, 2013.