SVD in Machine Learning: Ridge Regression and Multicollinearity | by Kunyu He | Towards Data Science

This article discusses what is multicollinearity, how can it compromise least squares, and how ridge regression helps avoid that from a perspective of singular value decomposition (SVD). It is heavily based on Professor Rebecca Willet’s course Mathematical Foundations of Machine Learning and it assumes basic knowledge of linear algebra.
In this section, we will look at multicollinearity and how can it compromise least squares.
Consider a matrix X of shape n × p. For its columns X1, X2, ..., Xp ∈ Rn, we say they are linearly independent when ∑αiXi = 0 if and only if αi = 0 for i = 1, 2, ..., p. Intuitively, none of the columns in X can be written as a weighted sum of the others. The other way around, if it’s not the case for some columns, we call them linearly dependent. Assume rank(X) = r, then (p — r) columns of X are linearly dependent.
Multicollinearity, in regression terms, refers to a phenomenon where a predictor in a multiple regression model can be linearly predicted from the others with a substantial degree of accuracy. In other words, the predictor is approximately a linear combination of the others. Perfect multicollinearity indicates linear dependency in the feature matrix. Intuitively, it implies redundancy in our features that some of them fail to provide unique and/or independent information to the regression.
Multicollinearity matters not only theoretically, but also for the practice. The coefficient estimates may change erratically in response to small changes in the model or the data, and themselves do not make sense at all. Why is that? We’ll look at it from an SVD perspective. Before that, below is a quick recap on SVD.
For the matrix X, there always exists matrices U, Σ, V such that:
Where both U and V are orthogonal:
Σ is diagonal:
The diagonal elements of Σ are called singular values (σ1 ≥ σ2 ≥ ... ≥ σp ≥ 0). The number of non-zero singular values is the rank of the matrix X, and the columns of Σ are the basis for the rows of X.
Recall that for the feature matrix X and the target variable y, least squares attempts to approximate the solution of the linear system y = Xw by minimizing the sum of squares of the residuals ∥y — Xw∥2. The weights vector can be written with the normal equation:
Note that A is invertible if and only if n ≥ p and rank(X) = p. Now it’s not hard to see why perfect multicollinearity is a major problem for least squares: it implies that the feature matrix is not full-rank so we cannot find a proper set of coefficients that minimize the sum of squared residuals.
However, why multicollinearity, or strong multicollinearity in specific, is problematic, either? Let’s find out from an SVD perspective.
Consider the true weights w, we know that y = Xw + ε, where ε is some neglectable noise or error:
We can see that the least squares coefficients deviate from the true weights by ε multiplied by some inflation term. Take a closer look at the inflation term as in the previous post linked at the end of this article:
If all the columns of X are linearly independent, we still have p singular values and σ1 ≥ σ2 ≥ ... ≥ σp > 0. However, with the presence of multicollinearity, some σi, σp for example, will be close to zero. Then the diagonal element 1/σp will be huge, leading to a really large inflation term and therefore a great deviation in the least squares weights from the true weights. Intuitively, multicollinearity can compromise least squares as it leads to small singular values. The estimation errors of the coefficients are inflated by the reciprocals of those singular values and therefore become too large to be neglected.
How can we avoid this? One possibility is ridge regression.
In this section, we will understand how ridge regression helps avoid the problems caused by multicollinearity in linear regression through a formal derivation.
Ridge regression builds on least squares by adding a regularization term in the cost function so that it becomes ∥y — Xw∥2 + λ∥w∥2, where λ indicates the strength of regularization. We can write the cost function f(w) as:
Then we can compute the gradient and set it to zero:
Take a closer look at the middle part of the expression:
How does this help?
Consider σp ≈ 0, this time σp/(σp2 + λ) ≈ 0 if and only if λ ≠ 0. Therefore, with ridge regression, the coefficients of unimportant features will be close to zero (but will not be exactly 0 unless there is perfect multicollinearity) and the error term would not be inflated to an explosion. Note that when there is no regularization (λ = 0) things go back to least squares. Also, for most occasions (σp ≫ λ), σp/(σp2 + λ) ≈ 1/σp just like in least squares.
In this section, we’ll work on a sample dataset seatposto verify our previous findings.
Car drivers like to adjust the seat position for their own comfort. Car designers would find it helpful to know where different drivers will position the seat depending on their size and age. Researchers at the HuMoSim laboratory at the University of Michigan collected data on 38 drivers.
The dataset contains the following features:
We want to fit a linear model that predicts hipcenter based on all the other features in the dataset. From the descriptions, some features can be closely related to one another. For example, Weight and Ht, Ht and HtShoes. Plot the correlation matrix:
The plot above confirms our guess. We have strong multicollinearity in our feature matrix. The good news is that our target hipcenter is strongly correlated with most of the features and we can expect a good fit. Codes that produces the correlation plot above:
Use the following code chunk to:
Add an offset to the feature matrix;Split the dataset into training and test set;Normalize the feature matrix so that we can compare the coefficients, as we expect features with larger variations to have smaller coefficients, ceteris paribus
Add an offset to the feature matrix;
Split the dataset into training and test set;
Normalize the feature matrix so that we can compare the coefficients, as we expect features with larger variations to have smaller coefficients, ceteris paribus
For simplicity, first look at a model with only Ht and HtShoes as predictors.
X_train_sub = X_train_[:, 2:4]X_test_sub = X_test_[:, 2:4]ls = LinearRegression(fit_intercept=True)ls.fit(X_train_sub, y_train)print(ls.intercept_, ls.coef_)  -165.844 [54.745 -105.923]
Surprisingly, although Ht and HtShoes are nearly perfectly correlated, their partial effects on hipcenter have the opposite signs. This can be a result of strong multicollinearity. Fit a ridge regression model with λ = 10 instead.
ridge = Ridge(alpha=10)ridge.fit(X_train_sub, y_train)print(ridge.intercept_, ridge.coef_)  -165.844 [-21.593  -22.269]
The coefficients of ridge regression seem to make more sense. Compare its test RMSE with that of the least squares.
ls_rmse = mean_squared_error(y_test, ls.predict(X_test_sub))ridge_rmse = mean_squared_error(y_test, ridge.predict(X_test_sub))print("Least squares test RMSE: {:.3f}".format(ls_rmse))print("Ridge test RMSE: {:.3f}".format(ridge_rmse))  Least squares test RMSE: 643.260  Ridge test RMSE: 519.287
For the bivariate linear model, ridge regression results in a better ability to generalize. However, since ridge regression introduces a regularization term, its bias can be higher in exchange for a lower variance sometimes, which may lead to worse fit.
At last, let’s look at the full model and check how the magnitudes of the coefficients differ across least squares and ridge regression, and how they change with the strength of penalty, λ.
We can see that least squares weights differ greatly from ridge regression weights on Ht and HtShoes as expected. Ridge regression weights get closer to zero as the penalty gets stronger. Codes that produces the plot above:
This article is written by Kunyu He. Kunyu is a current master’s student at The University of Chicago. He finds it interesting to understand statistical modeling and machine learning techniques, apply them to real-world data, and help create end-to-end solutions in the financial services industry. Connect with Kunyu on LinkedIn! 🐷