Ridge Regression and multicollinearity: An in-depth review

Multicollinearity is the phenomenon in which two or more identified predictor variables in a multiple regression model are highly correlated. The presence of this phenomenon can have a negative impact on the analysis as a whole and can severely limit the conclusions of the research study. This paper reviews and provides examples of the different ways in which multicollinearity can affect a research project, how to detect multicollinearity and how one can reduce it through Ridge Regression applications. This paper is intended for any level of SAS® user. DEFINITION DETECTION Examination of the Correlation Matrix • Large correlation coefficients in the correlation matrix of predictor variables indicate multicollinearity • If there is multicollinearity between any two predictor variables, then the correlation coefficient between those two variables will be near to unity (1.0000) • PROC CORR Variance Inflation Factor • Quantifies the severity of multicollinearity in an ordinary least-squares regression analysis • Consider equation: VIFj= 1/(1-Rj^2), for j= 1,2,. p-1 • Let Rj^2 denote the coefficient of determination when Xj is regressed on all other predictor variables in the model o VIFj= 1 when RJ^2 = 0 . When jth variable is not linearly related to the other variables o VIFj-> ∞ when Rj^2 -> 1 . When jth variable is linearly related to the other predictor variables • The VIF is an index which measures how much an estimated regression coefficient’s variance is increased due to multicollinearity o Example:  VIF for Xj is 5  Variance of estimated Bj is 5 times larger than if Xj was uncorrelated with other predictors • Note: If any of the VIF values exceeds 5 or 10 it implies that the associated regression coefficients are poorly estimated because of multicollinearity (Montgomery, 2001) Tolerance • Another way of looking at Variance Inflation Factor • Represented by 1/VIF Eigensystem Analysis of Correlation Matrix • The eigenvalues can also be used to measure the presence of multicollinearity • If multicollinearity is present in the predictor variables, one or more of the eigenvalues will be small (near to zero). o Let λ1. λp be the eigenvalues of correlation matrix. The condition number of correlation matrix is defined as follows: K = sqrt(λmax / λmin) & o Condition indices of correlation matrix are defined as: Kj= sqrt(λmax / λj), j=1,2,. ,p • Note: If one or more of the eigenvalues are small (close to zero) and the corresponding condition number is large, then it indicates multicollinearity (Montgomery, 2001) SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are trademarks of their respective companies. Definition / Explanation • A statistical phenomenon wherein there exists a perfect or exact relationship between predictor variables • Occurs in regression when several predictors are high correlated • Linear Dependence: Fit well into a straight line that passes through many data points • Another way to look at collinearity is co-dependence Consequence • Creates difficulty in creating reliable estimates of individual coefficients for the predictor variables • Results in incorrect conclusions about the relationship between outcome and predictor variables • As degree of multicollinearity increases, regression model estimates of the coefficients become unstable Consequence of Variance Inflation • Multicollinearity inflates the variances of the parameter estimates (Consider R-square = higher the value, better the model) • Collinearity results in inflation of variance, standard error, and parameter estimates • Can lead you to an over-specified model • Include predictor variables with low statistical significance • The presence of multicollinearity can cause serious problems with the estimation of B and its interpretation Explanatory vs Predictive Models • Collinearity is a problem when a model’s purpose is explanation and not prediction • More difficult to achieve significance of collinear parameters • Note: if estimates are statistically significant, they are as reliable as any other variable in the model • If they are not significant, the sum of the coefficient is likely to be reliable • In the case of a predictive model: just need to increase sample size • In the case of an explanatory model: further measures are needed Ridge Regression and Multicollinearity: An In-Depth Review Deanna Schreiber-Gregory Henry M Jackson Foundation for the Advancement of Military Medicine DETECTION: CORRELATIONS /* Examination of the Correlation Matrix */ Proc corr data=temp; Var cholesterolloss age weight cholesterol triglycerides hdl ldl height skinfold systolicbp diastolicbp exercise coffee ; Run; Tip: Keep in mind, while reviewing these results we want to check to see if any of the variables included have a high correlation – about 0.8 or higher – with any other variable. Result: As we can see, upon review of this correlation matrix, there seems to be some particularly high correlations between a few of the variables. Some relationships of note would be Cholesterol / LDL (0.96) and Weight / Height (0.70). DETECTION: COLLINEARITY /* Multicollinearity Investigation: VIF TOL COLLIN */