Machine Learning: Ridge Regression in Detail | by Ashish Singhal | Towards Data Science

As there are already more than sufficient articles about Linear Regression here, I won’t write about it one more time. Instead I will write about one kind of normalized regression type - Ridge Regression - which solves problem of data overfitting.
Linear regression model is given by following equation:
Y = Σ WjHj(Xi)Here, Σ runs from j = 0 to j = D where D is the total number of features.Wj is the jth coefficient Hj is the jth feature function which takes Xi observationXi is the ith observation
The above equation gives the predicted value, provided we know the values of W coefficients.For simplification, let’s denote the above equation by F(X) where X is the observation.
Cost function of Linear Regression model is given by following equation:
Cost Function = RSS(W) = Σ [Yi — F(Xi)]2Here,Σ runs from i=0 to i = N where N is the total number of observations.Yi is the known value of ith observation.F(Xi) gives the predicted value of ith observation.RSS stands for Residual Sum of Squares
Cost function always work on the training data set.
The whole idea of linear regression model revolves around minimizing the above cost function’s value. Lower the cost function value, better the linear regression model.
Generally, to decrease the cost function, we increase the number of features in our model. As we keep on increasing the features in model, model starts fitting the training data set well and cost function value starts decreasing.
But, with increase in number of features; our equations becomes a higher order polynomial equation; and it leads to overfitting of the data.
Why overfitting of data is bad ?In an overfitted model the training error becomes almost zero resulting into saying that model is working perfectly on training data set. But does that model works perfectly on data sets other than training data set like real outside world data?Generally it is seen that an overfitted model performs worse on the testing data set, and it is also observed that overfitted model perform worse on additional new test data set as well.
From the above graph, we can see the overfitted model performs well on training data set and cost function is zero for training data set.
But when we test this model with the test data set in graph above, model isn’t performing well at all. For test data, model is predicting wrong values which are far from actual correct values. This is sufficient to label this model as unfit for use in industry.
How to catch overfitting?By visualising the model (like above), one can easily see the overfitting in the model(observe how model fits training data set well). But as our model’s complexity increases, it goes into higher dimension which makes it difficult to visualise on graphs(or some other tool).
Instead of trying to visualise the model always, we can also see overfitting by seeing the coefficients’ value ( W ). Generally when overfitting happens, these coefficients’ values becomes very huge.
Ridge regression is used to quantify the overfitting of the data through measuring the magnitude of coefficients.
To fix the problem of overfitting, we need to balance two things:1. How well function/model fits data.2. Magnitude of coefficients.
So,Total Cost Function = Measure of fit of model + Measure of magnitude                    of coefficientHere,Measure of fit of model = RSS(W)Measure of magnitude of coefficient = ||W||2If Measure of fit of the model is a small value that means model is well fit to the data.If Measure of magnitude of coefficient is a small value that means model is not overfit.Total Cost Function = RSS(W) + λ*||W||2We have added λ in total cost function as a tuning parameter to balance the fit of data and magnitude of coefficients.
Ridge Regression Cost = RSS(W) + λ*||W||2 = (Y - WH)*(Y - WH) + WW
In matrix notation it will be written as:Ridge Regression Cost = (Y - HW)t (Y - HW) + WtW
Taking gradient of above equation(differentiation):
Δ[RSS(W) + λ||W||]2= Δ{(Y - HW)t(Y - HW)} + λ Δ{WtW}= -2Ht(Y - HW)+2λW
Setting the above gradient to 0 we get
W = (HtH + λI)-1HtY
Thus, we know the values of W coefficients.
The given data set into three sets:1. Training Set2. Validation Set3. Test Set
Training SetThis data set will be used to get W coefficients’ value every value of λ. Let’s assume the value of W coefficients’ for each value of λ value be Wλ.
Validation SetThe different values of Wλ will be assessed on validation set. The one which has lower error value will get selected.
Test SetThe selected value of W coefficients’ will again be assessed by test data set.
The above method is used only if sufficient amount of data is present.
And this is how finally λ’s value is selected. The process is kind of brute force. But with smart guesses and experience, iterations to guess λ value can be reduced.
We have seen why overfitting is bad in machine learning and how it is identified in the model by looking at the model’s W coefficients’ values. Then we saw new cost function for linear regression which takes into account overfitting of data with the tuning parameter λ.
Then we saw formula for calculating W values with respect to new cost function and how to choose value for λ.