Explaining Subset Selection and Regularization Methods for Least-Squares Part-2
By: Adrià Serra
Tags: Math, Regression, Deep Learning, Data Science, Machine Learning

MACHINE LEARNING THEORY

Explaining Subset Selection and Regularization Methods for Least-Squares Part-2

In previous posts, we introduced least-squares and explained some subset selection techniques. Today we are here to introduce more subset selection and shrinkage models.

Least angle regression(LAR)

Least angle regression is a kind of forward stepwise that only enters as much of a predictor as it deserves. At the first step, it identifies the variable mos correlated with the response.

LAR adjusts the coefficient of a variable until another variable catches up in terms of correlation with the residual. Then the second variable is added and the coefficients are adjusted again. We repeat the process until all the variables are in the model.

LAR Algorithm

Standardize the predictions to have mean zero (μ= 0) and unit norm N(0,1) . Start with residual r = y-ȳ, β1,β2,…,βp = 0 .

and unit norm . Start with residual . Find the xj predictor most correlated with r .

predictor most correlated with . Move βj from 0 towards its least-squares coefficient <xj,r> until some other xk has more correlation with the residual than xj .

from 0 towards its least-squares coefficient until some other has more correlation with the residual than . Move the βj and βk in the direction defined by their joint least-squares coefficient of the current residual on (xj,xk) until some other competitor xe is more correlated with the current residual.

and in the direction defined by their joint least-squares coefficient of the current residual on until some other competitor is more correlated with the current residual. Keep doing this until all p competitors enter. After min(N-1,p) we arrive at the full least-squares solution.

How does it work?

Suppose Ak is the actual set of variables at the beginning of kth step, and let βAk be the coefficient vector for this variable at this step. There will be K-1 non-zero values and the one just entered will be 0. If rk = y-XAk βAk is the residual, then the direction of the step is:

The direction of each step for LAR, self-generated.

The coefficient then evolves as βAk(α) = βAk + αδk, follow this direction keeps the decreasement of the correlations. If the fit vector at the beginning of this step is fK, it evolves as fˆK(α)=fˆK + αyk, where u_k = XAkδk is the new fit direction.

Lasso modification of LAR

Go back to the algorithm described above and add this between the fourth and fifth points:

If a non-zero coefficient hits 0, drop its variable from the active set of variables and recomputes the current joint least-squares direction.

This modification is extremely efficient, requiring the same order of computation as a single least-squares fit using p predictors. It always takes p steps, meanwhile, the lasso can take more than p steps, it’s better especially when p ≥ N.

Degrees of freedom for LAR and Lasso

How many parameters(degrees of freedom) we use when at LAR we decide to stop at some number of steps k<p or at lasso we decide the bound t that constrains the full least-squares fit?

The degrees of freedom are the number of linear independent parameters that we want to obtain with a regression, to achieve the best k, we have to fit distinct values of k, so we end up using more than k degrees of freedom.

To generalize this, we define the degrees of freedom of the fitted vector as:

Degrees of freedom of LAR, self-generated.

Where Cov(y^i,yi) refers to the sampling variance between the predicted value and its corresponding outcome value yi. Then, the harder we fit the data, the larger this covariance becomes, and hence df(y^).

For a linear regression df(y^)=K, but for subset selection techniques it’s more challenging to calculate, for ridge regression we can obtain K by estimating Cov(y^i,yi)/σ² by simulations. In the case of LAR and LASSO, and these techniques are adaptative in a smoother way than subset selection.

Methods using derived input directions

Most of the time we have a lot of inputs and a lot of them are tightly correlated. To solve that, we can apply the following methods for regression.

Principal Component Regressor (PCR)

In this approach, we create the Zm principal components of our data. After Zm = Xvm, then, we regress y on z1,z2,…,zM for some M≤p. Since Zm are orthogonal, this regression is just a sum of univariate regressions.

Principal component regressor as a combination of univariate regressions, self-generated.

where Θ^_m = <zm,y>/<zm,zm>. As zm are linear combinations of xj.

Principal component regressor betas, self-generated.

As ridge regression, PCR depends on the input scales, so we should standardize our data. PCR discards the p-M smallest eigenvalues. Note that of p=M we got a simple linear regression.

Ridge regression shrinks the coefficients, principal components regressor(PCR) truncates them at M.

Partial Least-Squares

This alternative constructs a set of linear combinations of the inputs for regression, but it uses y in addition to X to do it. As PCR or Ridge, it’s sensible to the scale of the data so we will need to standardize it.

The algorithm looks like:

Partial least squares algorithm, self-generated.

Doing this we obtain a sequence of orthogonal inputs or directions z1,…,zm. Including y, PLS searches for high variance and high correlation directions, not as PCR that just searches for a high variance. But hight variance tends to dominate over correlation

Comparing all the models

The tuning parameters for ridge and lasso vary over a continuous range, meanwhile, best subset, PLS, and PCR tale just two discrete steps to the least-squares solution.

The shrinkage behaviour of each model is distinct:

Ridge shrinks all directions, but lower-variance directions more.

PCR leaves M hight-variance directions alone and discards the rest.

Partial least-squares interestingly tends to shrink the low-variance directions but can inflate the hight-variance ones. It can end up in instability and bigger error rates.

PLS, PCR, and Ridge tend to behave similarly. Ridge may be preferred because it shrinks smoothly. Lasso falls in the middle of Ridge and best subset selection.

Conclusion

In this post, we have seen more regression subset selection techniques. These techniques are better than dropping variables from the complete model because using it we still get some information from each variable.