How least squares regression estimates are actually calculated | by Christian Lee | Towards Data Science

Ordinary least squares is a method used by linear regression to get parameter estimates. This entails fitting a line so that the sum of the squared distance from each point to the regression line (residual) is minimized. Let’s visualize this in the diagram below where the red line is the regression line and the blue lines are the residuals.
Below, I will first introduce an example then show how residuals are used to estimate the parameters of the linear model. I will then show how each of the values in the summary output is calculated. Together, this will reveal the inner-workings of ordinary least squares and demonstrate how it all fits together.
The goal is to predict a student’s ACT score given the student’s GPA. We begin by making a data frame of 15 ACT scores and GPAs. We then create the linear model using the lm function.
gpa = c(2.931,3.682,3.113,3.910,2.568, 3.292,3.684,3.835,3.180,3.516, 3.358,2.801,2.729,3.989,2.679)act = c(28,31,26,36,27,31,28,34,29,32,28,24,30,33,26)df = data.frame(cbind(gpa,act))mod = lm(act~gpa,data=df)summary(mod)
act = 11.877 + 5.376*gpa
We see that gpa is supported by a low p-value, suggesting it is a strong predictor of act. The estimate is 5.376, meaning for every 1.0 increase in gpa, we expect a 5.376 point increase in the ACT. Let’s look at where these different values in the summary table are coming from.
In the diagram below each point is yi, each predicted value is ŷi and the mean is ȳ.
Residual sum of squares (RSS/SSE)
ei = yi - ŷi
The ith residual is the difference between the ith actual value and the ith predicted value (blue lines). The sum of each residual squared is RSS. This is what is minimized to get our beta estimates.
Recall, ŷ = b0 + b1x
therefore, ei = yi - ŷi = yi - b0 - b1xi
RSS = Σ(yi - b0 - b1xi)2
We begin by taking the partial derivative of RSS with respect to b0 and b1 then setting them equal to zero.
∂RSS/∂b0 = -2Σ(yi - b0 - b1xi)
b0 = ȳ - b1x̄
∂RSS/∂b1 = -2Σ(yi - b0 - b1xi)xi
b1 = (Σ(xi - x̄)(yi - ȳ))/Σ(xi - x̄)2 = SXY/SXX
Let’s use the formulas above and see the results, starting with b1.
avg_act = mean(df$act)avg_gpa = mean(df$gpa)b1 = sum((df$gpa - avg_gpa)*(df$act - avg_act))/sum((df$gpa - avg_gpa)**2)
This yields 5.376. Notice this matches the lm estimate of gpa. Great, let’s repeat the process for b0.
b0 = avg_act - b1*(avg_gpa)
This yields 11.877, which matches the intercept estimate. Great, we have shown how to calculate parameter estimates but now we need to test their importance. Let’s begin by taking a look at two other values closely related to RSS.
Sum of squares regression (SSReg)
SSReg = Σ(ŷi - ȳ)2
This sums the squared difference between the predicted value and the mean. In words, this measures how much of the sum of squares is explained by the regression line. Refer back to the previous plot to visualize this.
Sum of squares total (SST)
SST = Σ(yi - ȳ)2
SST = RSS + SSReg
Let’s check that this equation holds true for our example.
rss = sum(((df$act - fitted.values(mod)))**2)ssreg = sum(((fitted.values(mod) - avg_act))**2)sst = sum(((df$act - avg_act))**2)all.equal((rss+ssreg),sst)
Residual standard error
Assume our errors are independent, homoscedastic and normally distributed with mean 0 and variance σ2. The residuals can be used to create an unbiased estimate of the error variance. The residual standard error is the square root of this σ2 estimator. This gives us an idea as to how far away our points lie from the regression line.
n = nrow(df)p = 1rse = sqrt(rss/(n-p-1))
Here, p is the number of non-intercept predictors. In our SLR example, we only have a single predictor so the denominator becomes n-2.
Std. error (of the parameter estimates)
Var(b0|X)= σ2[(1/n) + (x̄2/SXX)]
sxx = sum((df$gpa - avg_gpa)**2)sigma_squared = rss/(n-p-1)var_b0 = sigma_squared*((1/n)+(avg_gpa**2/sxx))se_b0 = sqrt(var_b0)
Var(b1|X) = σ2/SXX
var_b1 = (rss/(n-p-1))/(sxx)se_b1 = sqrt(var_b1)
t value
T-statistics are calculated by dividing the beta estimates by their standard errors. The numerators are standard normal variables and the denominators are square-roots of chi-square variables with n-p-1 degrees of freedom. Therefore, these statistics follow a T distribution with n-p-1 degrees of freedom.
t_b0 = b0/se_b0t_b1 = b1/se_b12*pt(t_b0,n-p-1,lower.tail=F)2*pt(t_b1,n-p-1,lower.tail=F)
Multiple R-squared
R2 = SSReg/SST = 1 - RSS/SST
In words, R2 is a measure of how much variance is explained by the regression line (SSReg) over the total variance (SST).
Adjusted R-squared
R2adj = 1 - (RSS/n-p-1)/(SST/n-1)
Rearranging the adjusted R2 equation will leave n-1 in the numerator and n-p-1 in the denominator. As a result, adjusted R2 is always smaller than multiple R2, even in the simple linear regression context.
F-statistic
f = MSReg/MSE
MS refers to mean square. These values are calculated by dividing SSReg and RSS by their respective degrees of freedom, 1 and n-p-1. Since this is the ratio of two chi-square variables the new statistic follows the F distribution with 1 and n-p-1 degrees of freedom.
f = (ssreg/1)/(rss/(n-2))pf(f,1,n-p-1,lower.tail=F)
We have looked at ordinary least squares and how it is used to calculate parameter estimates. Additionally, we have shown how the test statistics and diagnostics are calculated using values related to RSS.