Model Selection in Linear Regression
By: Nabil Madali
Tags: Data Science, Linear Regression, Machine Learning, Artificial Intelligence, Statistics

In many many practical problems, there are often many independent variables of the dependent variable Y. At this time, you may be troubled by not knowing which independent variables to choose for modeling! What kind of model is the best.

ANOVA Tabe.Image by Author

Suppose there is a linear relationship between variable Y and variables X1, X2,…, Xp which can be model as follow:

Linear regression model is a solution obtained by minimizing the residual sum of squares (SSE, Sum of Squared Error).

Intuitively speaking, for a good linear regression model, the predicted value should be close to the true value, so the SSE should be small.However, if we only use a small SSE as a measurement standard to select the final model, our model will always be a model with all variables added. Because adding arbitrary variables will reduce the SSE, even when the variables have nothing to do with the true value.

Therefore, simply using the residual sum of squares (SSE) as the only criterion for selecting variables will cause the following two major problems:

On the one hand, the model may be over-fitting: adding more variables will increase the variance of the model, and the amount of data is not enough to make the model exactly the same as the real situation.

On the other hand, the interpretation of the model is lost: we don’t know which variables will actually have an impact or which variables are most helpful for prediction .

From this point of view, our ideal model needs to have the lowest possible residuals and as few variables as possible.The actual model is not so perfect, so we need a way to balance both. From this we can lead to two types of basic variable selection logic.

Method 1: In the situation where only a small number of regressors are involved, to test the validity of a submodel m against a larger model, there is

two indices (or coefficients) whose computation and interpretation are quite immediate

A first possibility is to look at the coefficient of determination:

Where,

And,

This index therefore compares the predicted values of Y with the values observed through

The denominator corresponding in some way to a renormalization. Plus the coefficient R² will be close to 1, the greater the fit to the data of the chosen model. If we are led to choose between two explanatory models, so we are easily tempted to retain the one with the most important determination coefficient.

The adjusted coefficient of determinationmakes it possible to take into account the number of regressors retained and therefore proposes a compromise between the equation and the parameterization of the model.

This index is defined by:

where n represents the number of samples, k represents the number of selected variables.

Another approach consists in considering the model which maximizes the likelihood function,consider a simple linear regression model

Therefore the joint density of the independent random responses Y1,…,Yn is

The method of maximum-likelihood can be used to finds parameter values β and σ² that maximise the joint density .

There are two more commonly used metrics for model selection that use the likelihood function :

AIC

BIC

we denote by ln (Y, m, θ) the likelihood associated with the model m ,the AIC criterion is given by

And the BIC criterion is given by

The model retained by the intermediary of this criterion will maximize the likelihood which can be seen as a measure of the fit to the model while providing a control of the dimension of the model, the term | m | which can be seen as a term of penalization.

Method 2: Look at the prediction effect of new data. If it is difficult to find a suitable measurement standard, it is better to look directly at the predictability of the model .If the model’s predictions for the new data are very close, then we can select the variables in the model. At this time, our measurement standard is still the residual sum of squares (SSE), but the residual at this time is not the residual of the data that has been used to build the model, but the residual of the new data that has not been used.

However, this measurement method requires the support of a large amount of data. If the amount of data is small, some data will often make the model unable to learn correctly due to the lack of sample size.

References

1.CHAPMAN & HALL/CRC Texts in Statistical Science Series Series Editors C. Chatfield, University of Bath, UK J. Zidek, University of British Columbia, Canada