Regression Modeling in R (Part 1)
By: Namitha Deshpande
Tags: Machine Learning, Linear Regression, R Programming, Data Science, Modeling

Simple Linear Regression

It is the simplest form of regression. The relationship between the dependent variable and independent variables is assumed to be linear.

In statistics, linear regression is a linear approach to modeling the relationship between an outcome(or dependent variable) and one or more predictor (or independent variables). — Wikipedia

The Dataset

As an example to start with, let’s look at the student survey data (the survey data frame in the package MASS) a little more closely. Install and load the “MASS” package as shown below :

install.packages("MASS")

library(MASS)

Turn your attention to the data frame survey, located in the MASS package. These data record particular characteristics of 237 first-year undergraduate statistics students collected from a class at the University of Adelaide, South Australia.

Use ?survey to explore the different variables present in the data frame.

Graphical Analysis

The aim here is to build a simple regression model that we can use to predict Height by establishing a statistically significant linear relationship with Handspan. But before jumping into the model, let's try to understand these variables graphically.

Firstly, plot the student heights on the y-axis and their handspans (of their writing hand) on the x-axis using the plot function.

plot(survey$Height~survey$Wr.Hnd,xlab="Writing handspan(cm)",ylab="Height (cm)")

A scatterplot of height against writing handspan for a sample of first-year statistics students

As one might expect, there’s a positive association between a student’s handspan and their height. That relationship appears to be linear in nature.

Correlation

Correlation is a statistical measure that suggests the level of linear dependence between two variables, that occur in pair — just like what we have here in Height and Handspan.

Correlation can take values between -1 and+1. If we observe for every instance where handspan increases, the height also increases along with it, then there is a high positive correlation between them and therefore the correlation between them will be closer to 1.

cor(survey$Wr.Hnd,survey$Height,use="complete.obs")

[1] 0.6009909

Defining a model

The purpose of a linear regression model is to come up with a function that estimates the mean of one variable given a particular value of another variable.

In terms of the student survey example, you might ask something like “What’s the expected height of a student if their handspan is 14.5 cm?”

Assume you’re looking to determine the value of response variable Y given the value of an explanatory variable X. The simple linear regression model states that the value of response is expressed as the following equation:

Parameters

Y is the output or response variable.

X is the input or predictor variable.

ε term represents random error.

ß0 is the intercept, interpreted as the expected value of the output variable when the input is zero.

ß1 is the slope, interpreted as the change in the mean response for each one-unit increase in the predictor.

Fitting the model with lm

A linear regression model can be calculated in R with the command lm . It takes many arguments as shown below:

lm(formula, data, subset, weights, na.action,

method = "qr", model = TRUE, x = FALSE, y = FALSE, qr = TRUE,

singular.ok = TRUE, contrasts = NULL, offset, …)

For example, the following line of code creates a fitted linear model object of the mean student height by handspan and stores it in your global environment in the model variable as:

model <- lm(Height~Wr.Hnd,data=survey)

The first argument is the now-familiar response ~ predictor formula, which specifies the desired model. The specific dataframe is provided in the second argument for the function to look for.

With the command summary(model) you can see detailed information on the model’s performance and coefficients.

summary(model) Call:

lm(formula = Height ~ Wr.Hnd, data = survey) Residuals:

Min 1Q Median 3Q Max

-19.7276 -5.0706 -0.8269 4.9473 25.8704 Coefficients:

(Intercept) 113.9536

Wr.Hnd 3.1166

Coefficients

In the summary square above, you can see the values of the intercept and the slope for the writing handspan variable. This reveals that the linear model for this scenario is estimated as follows:

If you evaluate the mathematical function for y at a range of different values for x, you end up with a straight line when you plot the results.

Considering the definition of intercept given earlier as the expected value of the response variable when the predictor is zero, in the current example, this would imply that the mean height of a student with a handspan of 0 cm is 113.9536 cm.

The slope is 3.1166. This states that, on average, for every 1 cm increase in handspan, a student’s height is estimated to increase by 3.1166 cm.

With all this knowledge, let’s plot the fitted linear regression model using the abline function which adds one or more straight lines through the current plot using the command below:

abline(model,lwd=2)

The simple linear regression line (solid, bold) fitted to the observed data.

Residuals

A good way to test the quality of the fit of the model is to look at the residuals or the differences between the real values and the predicted values.

When the parameters are estimated, as shown above, the fitted line is referred to as an implementation of least-squares regression because it’s the line that minimizes the average squared difference between the observed data and itself.

The idea here is that the sum of the residuals is approximately zero or as low as possible. In real life, most cases will not follow a perfectly straight line, so residuals are expected.

In the R summary of the lm function, you can see descriptive statistics about the residuals of the model, the following are the values for our fitted model.

Residuals:

Min 1Q Median 3Q Max

-19.7276 -5.0706 -0.8269 4.9473 25.8704

Coefficient of Determination

One measure very used to test how good is your model is the coefficient of determination or R². This measure is defined by the proportion of the total variability explained by the regression model.

The output of summary provides you with the values of Multiple R-squared and Adjusted R-squared, which are particularly interesting. Both of these are referred to as the coefficient of determination.

Multiple R-squared: 0.3612, Adjusted R-squared: 0.3581

For simple linear regression, the Multiple R-squared measure is simply obtained as the square of the estimated correlation coefficient. This tells you that about 36.1 per cent of the variation in the student heights can be attributed to handspan which is the Multiple R-squared values.

The adjusted measure is an alternative estimate that takes into account the number of parameters that require estimation. The adjusted measure is generally important only if you’re using the coefficient of determination to assess the overall “quality” of the fitted model in terms of a balance between the goodness of fit and complexity

This can seem a little bit complicated, but in general, for models that fit the data well, R² is near 1.

Conclusion

You made it to the end! Yes, this was a long one. Linear regression is a big topic. Hence I have decided to make a series of article on Regression Modeling in R. Stay tuned!

Thanks for reading :)