Understanding Regression Output in R: A Lesson for Absolute Beginners (Part 2)
By: Vivekananda Das
Tags: Output In R, R Squared, Regression, Adjusted R Squared, F Statistic

In this second part, we are going to discuss how to interpret the following three values shown at the bottom of the regression output in R:

(1) Multiple R-squared,

(2) Adjusted R-squared, and

(3) F-statistic

In case you missed the first part of this article (discusses how to interpret the coefficients of predictors and their statistical significance), here is the link:

A quick recap! We are trying to predict the quality of red wine depending on the alcohol level, pH level, and density. We fit the following multiple regression model:

quality = b0 + b1*alcohol + b2*pH + b3*density + u

The Output

Multiple R-squared

The multiple R-squared value of 0.2528 means: our regression model with three predictors (alcohol, pH, and density) explains 25.28% of the total variation in the wine quality (outcome).

The R-squared value ranges from 0 to 1, where 0 indicates a model that explains nothing (worst predictive model) of the total variation in the outcome and 1 indicates a model that explains everything (best predictive model) of the total variation in the outcome.

The key issue with the R-squared value is that if we keep adding more and more predictors, the R-squared value always goes up, even if some of these predictors do not increase the predictive ability of the model. This results in an overfitted model (which refers to a model that is more accurate than it should be given the small sample data; such a model may appear to predict the outcome quite accurately when applied to the sample data, but will perform poorly when applied to out-of-sample data).

Adjusted R-squared

The adjusted R-squared value increases with the addition of new predictors only if they improve the predictive ability of the model beyond the improvement expected by chance. By accounting for the inclusion of new predictors, the adjusted R-squared value tries to fix the key issue in the R-squared value.

The interpretation of the adjusted R-squared value is similar to the R-squared value. In our example, the adjusted R-squared value of 0.2513 means: our regression model with three predictors (alcohol, pH, and density) explains 25.13% of the total variation in the wine quality (outcome).

Notice, in our example, the adjusted R-squared value is slightly smaller than the R-squared value. Generally, adjusted R-squared value tends to be smaller than the R-squared value.

F-statistic

The F-statistic, right at the bottom of the R output, refers to a hypothesis test called the “Overall F-test”. In this hypothesis test, we are testing a null hypothesis that says: the coefficients of all the predictors are equal to zero.

In the context of our example, the null hypothesis would be: neither alcohol level, nor pH level, nor density have any association with the quality of wine.

If the p-value < 0.05, we reject the null hypothesis. If the p-value >= 0.05, we fail to reject the null hypothesis.

As we see a p-value of 2.2e^-16 (obviously smaller than 0.05), we reject the null hypothesis.

Note that, by rejecting the null hypothesis, all we can claim is: at least one of the three predictors has statistically significant association with the outcome. But the result of this test does not indicate anything regarding which of these predictors have significant association with the outcome! If we are interested in identifying the statistical significance of each of the predictors, we can look at the coefficients part of the R output and take a p-value approach as described in the part 1 of this article!

Data Courtesy: P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis.

Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547–553, 2009.