R: Linear models with the lm function, NA values and Collinearity
Date: 2014-10-18 06:35:49

In my continued playing around with R I've sometimes noticed 'NA' values in the linear regression models I created but hadn't really thought about what that meant.

On the http://java.dzone.com/articles/r-first-attempt-linear#comment-120879[advice of Peter Huber] I recently started working my way through https://class.coursera.org/regmods-007[Coursera's Regression Models] which has a whole slide explaining its meaning:

image::{{<siteurl>}}/uploads/2014/10/2014-10-17_06-21-07.png[2014 10 17 06 21 07,600]

So in this case 'z' doesn't help us in predicting Fertility since it doesn't give us any more information that we can't already get from 'Agriculture' and 'Education'.

Although in this case we know why 'z' doesn't have a coefficient sometimes it may not be clear which other variable the NA one is http://en.wikipedia.org/wiki/Multicollinearity[highly correlated with].

____
*Multicollinearity* (also *collinearity*) is a statistical phenomenon in which two or more predictor variables in a multiple regression model are highly correlated, meaning that one can be linearly predicted from the others with a non-trivial degree of accuracy.
____

In that situation we can make use of the +++<cite>+++http://stat.ethz.ch/R-manual/R-devel/library/stats/html/alias.html[alias]+++</cite>+++ function to explain the collinearity as http://stats.stackexchange.com/questions/112442/what-are-aliased-coefficients[suggested in this StackOverflow post]:

[source,r]
----

library(datasets); data(swiss); require(stats); require(graphics)
z <- swiss$Agriculture + swiss$Education
fit = lm(Fertility ~ . + z, data = swiss)
----

[source,r]
----

> alias(fit)
Model :
Fertility ~ Agriculture + Examination + Education + Catholic +
    Infant.Mortality + z

Complete :
  (Intercept) Agriculture Examination Education Catholic Infant.Mortality
z 0           1           0           1         0        0
----

In this case we can see that 'z' is highly correlated with both Agriculture and Education which makes sense given its the sum of those two variables.

When we notice that there's an NA coefficient in our model we can choose to exclude that variable and the model will still have the same coefficients as before:

[source,r]
----

> require(dplyr)
> summary(lm(Fertility ~ . + z, data = swiss))$coefficients
                   Estimate  Std. Error   t value     Pr(>|t|)
(Intercept)      66.9151817 10.70603759  6.250229 1.906051e-07
Agriculture      -0.1721140  0.07030392 -2.448142 1.872715e-02
Examination      -0.2580082  0.25387820 -1.016268 3.154617e-01
Education        -0.8709401  0.18302860 -4.758492 2.430605e-05
Catholic          0.1041153  0.03525785  2.952969 5.190079e-03
Infant.Mortality  1.0770481  0.38171965  2.821568 7.335715e-03
> summary(lm(Fertility ~ ., data = swiss))$coefficients
                   Estimate  Std. Error   t value     Pr(>|t|)
(Intercept)      66.9151817 10.70603759  6.250229 1.906051e-07
Agriculture      -0.1721140  0.07030392 -2.448142 1.872715e-02
Examination      -0.2580082  0.25387820 -1.016268 3.154617e-01
Education        -0.8709401  0.18302860 -4.758492 2.430605e-05
Catholic          0.1041153  0.03525785  2.952969 5.190079e-03
Infant.Mortality  1.0770481  0.38171965  2.821568 7.335715e-03
----

If we call alias now we won't see any output:

[source,r]
----

> alias(lm(Fertility ~ ., data = swiss))
Model :
Fertility ~ Agriculture + Examination + Education + Catholic +
    Infant.Mortality
----