Understanding Regression Output in R: A Lesson for Absolute Beginners (Part 1)
By: Vivekananda Das
Tags: Statistical Significance, Regression, Interpretation, Coefficient, Linear Regression In R

Linear Regression is one of the most powerful weapons in the arsenal of a data analyst. Although the core concept of the simple linear regression seems intuitive to anyone who understands the equation of a straight line (y = b + mx), the inferential parts of regression may seem quite complicated in case the person never took an inferential statistics course before. Thanks to the wider availability of different software packages, fitting a regression model is no more a challenge; however, correctly interpreting the results is crucial as it is so easy to get it all (or partially) wrong!

In this two-part article, I will try to explain how to interpret the different parts of the output of a multiple regression model (i.e. a model with more than one predictor) in R. In this first part, we will discuss:

how to interpret the coefficients of predictors in a regression model and how to identify whether the coefficients are statistically significant

Data and Background

Using the Wine Quality Dataset provided by Cortez et al. (2009), we are building a model to “predict” the quality of red wine based on its alcohol content, pH level, and density.

Model and Some Basic Ideas

We are fitting the following model:

quality = b0 + b1*alcohol + b2*pH + b3*density + u

Here, b0 is the intercept term. It is the expected value (i.e. the value we would find on average) of quality if all three predictors added to the model are equal to 0 (which, by the way, is not useful in this example because we have no such observation in the dataset for which any of the three predictors is 0. Also, if alcohol = 0, the liquid cannot be red wine, right? :D Regardless, the intercept can be useful in other contexts).

Next, b1, b2, and b3 are the coefficients of alcohol, pH, and density respectively. These coefficients express the association between each of the predictors and the outcome.

Finally, u is the error term. Theoretically, u consists of all other predictors of quality except the ones (i.e. alcohol, pH, and density) added to the model.

In R, I write the following code:

m1<-lm(quality~alcohol+pH+density,data=winequality.red)

summary(m1)

Fitting a regression model takes just a couple of lines of code! Isn’t it easy? :D

The Output

First of all, please notice that the intercept and the coefficient of each predictor have an Estimate, a Standard Error, a t-value, and a p-value.

Generally, we are interested in the “Statistical Significance” of the coefficients of predictors. For that purpose, we can use either the t-value or the p-value. In this article, I am going to use the p-value approach as I find it more intuitive.

Question #1: What does the coefficient of alcohol = 0.39704 mean?

Answer: It means that holding the values of all other predictors (i.e. pH and density) constant, a one unit increase in alcohol is, on average, associated with 0. 39704 unit increase in the quality of red wine. More concisely, we can say that alcohol level has a positive association with the quality of red wine.

Question #2: Looking at the regression output, do you think the coefficient of alcohol is “statistically significant”?

Answer: Statistical significance is related to a hypothesis test. The finding of the hypothesis test helps us make an inference on the value of a true population parameter based on the sample estimate of the parameter. In the part of the regression output we are considering in this article, population parameter = coefficient of a predictor in the population and sample estimate = coefficient of the same predictor in our sample data.

For reaching a decision on the statistical significance, we need to decide on two things:

1) Hypothesized value of the coefficient: We need to have a hypothesized value of the true population coefficient. In the built-in R output (as we see in the figure), the hypothesized value of each of the coefficients is 0. That is: for each of these coefficients, we are testing the null hypothesis (the default hypothesis) that there is no association (coefficient = 0) between the predictor and the outcome in the true population.

2) Significance Level: Typically, we select a 5% significance level (but we could select 0.1%, 1%, 10%, etc.). This is just an arbitrary choice.

Importantly, the p-value expresses the probability of finding a coefficient at least as extreme as the sample coefficient (e.g. 0.39704 for alcohol) given the null hypothesis is true (e.g. the true population coefficient of alcohol = 0). If the p-value<0.05, we would reject the null hypothesis that the true population coefficient is 0 (and conclude: the coefficient is not 0 → there is some association between the predictor and the outcome → the coefficient is statistically significant) On the contrary, if the p-value≥ 0.05, we would fail to reject the null hypothesis that the true population coefficient is 0 (and conclude: the coefficient is 0 → there is no association between the predictor and the outcome → the coefficient is statistically insignificant)

Okay! In the output, we see that for the hypothesis test on the true population coefficient of alcohol = 0, the p-value is < 2e-16, which is definitely smaller than 0.05! This result indicates that given the true population coefficient of alcohol = 0, the probability of finding a sample coefficient of alcohol = 0.39704 is less than 0.0000000000000002. So, we have a very strong reason to believe that the sample coefficient of alcohol = 0.39704 does not belong to a population in which the coefficient of alcohol = 0.

Consequently, we reject the null hypothesis that the true population coefficient =0 (no association) and conclude that alcohol level has a significant positive association with the quality of red wine.

Can you interpret the coefficients of the other two predictors (i.e. pH and density) based on our above discussion? Are they statistically significant? Hopefully, you can answer both of these questions! 😊

Before we finish, you might be wondering: “does the positive association between alcohol level and the quality of wine mean that alcohol level has a positive effect on the wine quality?”

Great question! Although the positive association may indeed be an evidence of a positive effect, our current model is apparently not sufficient to make any causal conclusion. Please notice that in this example, we built the model only for predictive purpose. For inferring on causality, we need to worry about confounding variables (which means we need subject matter knowledge on the causal relationship). For a quick introduction to confounding variables, you can check my previous article:

In the second and final part of this article, we will discuss how to interpret some other important things in the R regression output. Here is the link of the second part:

Data Courtesy: P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis.

Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547–553, 2009.