Collinearity Measures. Metrics and approaches towards... | by ShengJun | Towards Data Science

Feature selection is a process where the predictor variables that contribute most significantly towards the prediction/ classification of the target variable are selected. In feature selection for linear regression models, we are concerned with four aspects regarding the variables. Framed as a mnemonic “LINE”, these are:
Linearity. The selected variable possesses a linear relationship with the target variable.Independence of predictor variables. Selected variables to be independent of each other.Normality. Residuals generally follow a normal distribution (mean of zero).Equality of variance. The residual errors are generally consistent across the values of predictor variables (i.e. Homoscedasticity).
Linearity. The selected variable possesses a linear relationship with the target variable.
Independence of predictor variables. Selected variables to be independent of each other.
Normality. Residuals generally follow a normal distribution (mean of zero).
Equality of variance. The residual errors are generally consistent across the values of predictor variables (i.e. Homoscedasticity).
In cases where selected predictor variables are not independent of each other, we would not be able to clearly determine or attribute the contribution from the various predictor variables towards the target variable — interpretability of the model coefficients becomes an issue.
One approach in feature selection would be through the use of p-values; where variables with p-values above a certain threshold (typically +/- 0.05) are surfaced as not significantly contributing towards the target variable, and hence can be dropped to reduce model complexity. However, this approach has its own challenges when multicollinearity exists among the predictor variables as illustrated through the Boston housing dataset in sklearn.
# Original (full) set of variablesX_o = df_wdummy[['CRIM', 'ZN', 'INDUS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'CHAS_1.0']]X_o = sm.add_constant(X_o)y_o = df_wdummy['MEDV']# Baseline resultsmodel_o = sm.OLS(y_o, X_o)results_o = model_o.fit()results_o.summary()
Based on the p-values, we remove variables INDUS and AGE and review the updated p-values.
# Remove Age -> Remove INDUS, based on p-values. X_r0 = df_wdummy[['CRIM', 'ZN', 'NOX', 'RM', 'DIS', 'RAD', 'TAX',       'PTRATIO', 'B', 'LSTAT', 'CHAS_1.0']]X_r0 = sm.add_constant(X_r0)y_r0 = df_wdummy['MEDV']# resultsmodel_r0 = sm.OLS(y_r0, X_r0)results_r0 = model_r0.fit()results_r0.summary()
Though all p-values are lower than 0.05, multicollinearity still exists (between TAX and RAD variables.) The presence of multicollinearity can mask the importance of the respective variable contributions to the target variable, where the interpretability of p-values then becomes challenging. We could use correlation measures and matrices to help visualize and mitigate multicollinearity. Such an approach is fine until we need to use different correlation measures (i.e. Spearman, Pearson, Kendall) due to the inherent attributes of the variables. In the example above, the variable RAD (index of accessibility to radial highways) is an ordinal variable. TAX (full-value property-tax rate per $10,000) is a continuous variable (not normally distributed). Using the different correlation measures and matrices, one could potentially overlook correlation among different categories of variables.
Another approach to identify multicollinearity is via the Variance Inflation Factor. VIF indicates the percentage of the variance inflated for each variable’s coefficient. Beginning at a value of 1 (no collinearity), a VIF between 1–5 indicates moderate collinearity while values above 5 indicate high collinearity. Some cases where high VIF would be acceptable include the use of interaction terms, polynomial terms, or dummy variables (nominal variables with three or more categories). Correlation matrices enable identification of correlation among variable pairs while VIF enables the overall assessment of multicollinearity. The correlation matrix for most of the continuous variables is presented below to highlight the various collinear variable pairs. VIF can be calculated using the statsmodels package; the code block below presents the VIF values with collinear variables included (left) and removed (right).
# Setting the predictor variablesX_o = df_wdummy[['CRIM', 'ZN', 'INDUS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'CHAS_1.0']]X_r1 = df_wdummy[['CRIM', 'ZN', 'INDUS', 'RM', 'AGE', 'DIS', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'CHAS_1.0']]# from statsmodels.stats.outliers_influence import variance_inflation_factorvif = pd.Series([variance_inflation_factor(X_o.values, i) for i in range(X_o.shape[1])], index=X_o.columns,                 name='vif_full')vif_r = pd.Series([variance_inflation_factor(X_r1.values, i) for i in range(X_r1.shape[1])], index=X_r1.columns,                  name='vif_collinear_rvmd')pd.concat([vif, vif_r], axis=1)
The VIF values correspond with the correlation matrix; for example, variable-pair NOX and INDUS, the correlation coefficient is above 0.5 (0.61), and the respective VIF values are above 5. The removal of the collinear variables RAD and NOX improved the VIF figures. Dropping collinear variables solely based on the highest VIF figure is not a guaranteed way towards building the best performing model, as elaborated in the next section.
We build a baseline model by dropping all collinear variables identified in the correlation matrix (shown above), pending TAX and RAD to be dropped next.
# Baseline variables X_bl = df_wdummy[['INDUS', 'RM', 'AGE', 'RAD', 'TAX', 'PTRATIO', 'LSTAT']]y_bl = df_wdummy['MEDV']# Explore mitigating multi-collinearityvif_bl = pd.Series([variance_inflation_factor(X_bl.values, i) for i in range(X_bl.shape[1])], index=X_bl.columns,                 name='vif_bl')X_noTAX = X_bl.drop(['TAX'],axis=1)X_noRAD = X_bl.drop(['RAD'],axis=1)vif_noTAX = pd.Series([variance_inflation_factor(X_noTAX.values, i) for i in range(X_noTAX.shape[1])],                      index=X_noTAX.columns, name='vif_noTAX')vif_noRAD = pd.Series([variance_inflation_factor(X_noRAD.values, i) for i in range(X_noRAD.shape[1])],                      index=X_noRAD.columns, name='vif_noRAD')pd.concat([vif_bl, vif_noTAX, vif_noRAD], axis=1)
While it appears dropping the TAX variable based on VIF seems to be better, a prudent approach is to check via the adjusted R-squared metric (adjusted for the number of predictors, the metric increases only if the next added variable improves the model more than would be expected by chance).
# Without TAXmodel = sm.OLS(y, sm.add_constant(X_noTAX)).fit()print_model = model.summary()print(print_model)
# Without RADmodel = sm.OLS(y, sm.add_constant(X_noRAD)).fit()print_model = model.summary()print(print_model)
From the higher adjusted R-squared figure, we can infer that the model performs better with the RAD variable dropped! With multicollinearity issue addressed, the next step could be to explore the addition of interaction terms to potentially boost model performance.
In summary, the presence of multicollinearity can mask the importance of predictor variables to the target variable. The use of both Correlation matrices and VIF can help identify correlated variable pairs and assess multicollinearity among selected variables (features). While some iterations would still be necessary to assess model performance, with VIF and correlation matrices, we would be able to make a better-informed decision for feature selection.
Codes are hosted here: https://github.com/AngShengJun/petProj/tree/master/eda_viz
References:
Correlation methods 1 https://datascience.stackexchange.com/questions/64260/pearson-vs-spearman-vs-kendallCorrelation methods 2 https://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/regression/supporting-topics/basics/a-comparison-of-the-pearson-and-spearman-correlation-methods/
Correlation methods 1 https://datascience.stackexchange.com/questions/64260/pearson-vs-spearman-vs-kendall
Correlation methods 2 https://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/regression/supporting-topics/basics/a-comparison-of-the-pearson-and-spearman-correlation-methods/