Explaining least-squares. The basic linear regression model
By: Adrià Serra
Tags: Machine Learning, Deep Learning, Data Science, Mathematics, Linear Regression

MACHINE LEARNING THEORY

Explaining least-squares

The basic linear regression model

As we start with machine learning, for me, the first model to understand is the least squares. A simple model that is easy to perform and gives a lot of insights about your datasets.

This model assumes that the expected E(Y|X) values of the dependent variable are linear to the inputs X1,…, Xn.

Least Squares

It’s a supervised learning algorithm that takes an input vector, X^T = (X1, X2, …, Xp), and want to predict the output Y. The mathematical expression has the form

Linear regression model, self-generated.

The linear model assumes that the regression function E(Y|X) is linear or a reasonable approximation. The β are called parameters or coefficients and the variables X can be of different types:

Quantitative variables, that can be transformed with logs, polynomial representations, etc…

Numeric or “dummy” variables coding the distinct levels of a qualitative input.

Interactions between other variables.

Now you should be thinking that the model is not linear after applying logs, polynomial, and interaction transformation, and it’s true, but we’re keeping the linearity in the β parameters.

In the least-squares, we pick the β coefficients to minimize the error function of the sum of the residuals squared.

Least squares error function, self-generated.

The next image represents a regression, where we try to predict y using the vector X of one variable.

Regression example, Krishnavedala / CC BY-SA (https://creativecommons.org/licenses/by-sa/3.0)

In the image you have the prediction made by least squares in blue, the data values in red, and the distance between them or error in green, the error is calculated by adding all the error distances squared.

Minimizing the error function

We denote the matrix of input vectors of size (p+1) we added a 1 in the first position of each row by X. Meanwhile y will be the N-vector of outputs in the training set. Now the RSS can be rewritten as:

Vectorial form of RSS, self-generated.

To minimize it we will use what we learned in calculus posts, differentiation with respect to β:

Derivatives of RSS, self-generated.

Now, assuming that XTX is positive definite(X has full column rank), setting the first derivative to 0, we obtain the unique solution:

Unique solution for linear squares, self-generated.

We minimize RSS(β) = ||y-Xβ||² by choosing β^, which makes the residual vector orthogonal to this subspace. The matrix H is the part multiplicating y in the last equation and computes the orthogonal projection.

If the variables involved in the β calculation are correlated or redundant, then XTX is singular and the β^ becomes not singular, however, it still lives in the projection of y. Later we will talk about techniques to drop these correlations.

Assumptions made about the data

Notice that since now, the regression does not assume the validity of the model.

Distribution of the data

The first assumptions to make are the ones related to the distribution of the data, to use the properties of our parameters estimations, we have to assume that:

The values of the dependent variable y are uncorrelated.

are uncorrelated. y has constant variance σ².

The variance of the estimated parameters can be obtained from

The variance of the estimated parameters, self-generated.

To obtain the σ², as we usually work with samples and not populations, we typically use the next formula to obtain it

Variance estimation formula, self-generated.

Conditional expectations and error distribution

The next assumptions allow us to infer our results to new data, in other words, to be sure that our model predictions will work “correctly” on new data.

We have to assume that the equation of the lineal regression is the correct model for the mean, which means that the conditional expectation of Y is linear in X1, …, Xp.

We also assume that the deviations of Y around its expectation are additive and Gaussian.

Conditional expectations of Y, self-generated.

Where the error ε is a Gaussian random variable with expectation 0 and variance σ², written ε ∼ N(0, σ²). This allows us to demonstrate that our estimations are contained in a multivariate normal distribution with the following variance-covariance matrix.

Distribution of the estimators, self-generated.

Also a chi-squared distribution with N−p−1 degrees of freedom. Also, the estimations of the coefficients and the variances are statistically independent. We use these distributional properties to form tests of hypothesis and confidence intervals for the parameters βj.

Hypothesis testing

Single coefficient significance testing

To test the hypothesis that a particular coefficient βj = 0, we form the standardized coefficient or Z-score

Coefficient z scores calculation, self-generated.

where v_j is the diagonal element of (XTX)^-1. The null hypothesis of this test is that βj = 0, and z_j is distributed as t(N−p−1)(t-students distribution).

A large value of z_j will lead to rejecting the null hypothesis, so βj != 0 that’s the same as saying that βj is significant. We can use the t-student distribution because the tails of the distribution are similar to the normal distribution tails.

Multiple coefficient significance testing

Now that we checked if variables are significative one by one, we should check the significance of multiple variables together, an example of when we need to do that is the new test the significance of a categorical variable that had been converted to dummy variables and has more than one coefficient.

To test it we need to build two models, one including the variables to test (1)and one without it (0).

Here, we use the F statistic

F statistic formula, self-generated.

where RSS1 is the residual sum-of-squares for the least-squares fit of the bigger model with p1+1 parameters, and RSS0 the same for the nested smaller model with p0+1 parameters, having p1−p0 parameters constrained to be zero.

The F statistic measures the change in residual sum-of-squares per additional parameter in the bigger model, and it is normalized by an estimate of σ².

Under the Gaussian assumptions and the null hypothesis that the smaller model is correct, the F statistic will have an F(p1−p0, N−p1−1) distribution.

The null hypothesis is rejected if the F calculated from the data is greater than the critical value of the F-distribution.

The Gauss-Markov Theorem

This theorem demonstrates that the least-squares estimates of the parameters β have the smallest variance among all linear unbiased estimates. We will make this precise here, and also make clear that the restriction to unbiased estimates is not necessarily a wise one. In posterior posts, we will consider biased regression techniques.

To demonstrate this theorem for one variable(it’s easy to understand and to define it into the vector of parameters):

Consider the mean squared error of an estimator θ˜ in estimating θ:

Mean squared error, self-generated.

The first term is the variance and the second one is the squared bias.

Bias-Variance Tradeoff

The Gauss-Markov theorem implies that the least-squares estimator has the smallest mean squared error of all linear estimators with no bias. However, there may well exist a biased estimator with a smaller mean squared error, we will talk about distinct estimators that allow us to find a more equilibrated model in the relation variance-bias.

The following two techniques give some insights into how to apply the least-squares in distinct situations.

Multiple Regression from Simple Univariate Regression

This technique converts a multiple linear regression into a chain of univariate linear regression.

Suppose we have a univariate model with no intercept: Y = Xβ + ε. The least-squares estimate and residuals are

Residuals of single-variable regression, self-generated.

in vector notation expressed as <x,y≥ = x^Ty, so we can rewrite the formulas in vector notation.

Residuals of single-variable regression in vector notation, self-generated.

Now suppose that the input independent variables are orthogonal, that makes all <xj,xk> = 0 for all k!=j. Then, when the independent variables are orthogonal, they have no effects on the other parameter estimates in the model.

Orthogonal data is not easily found in real-world datasets, there’s always some type of correlation, noise, etc… To orthogonalize the data we will use the following, suppose that we have an intercept and a single input variable x:

orthogonalization for β, self-generated.

This procedure can be obtained by applying multiple univariate regressions:

First, regress x on the vector of N-ones(1) , with no intercept, to produce the residuals z = x-x̄1.

on the vector of , with no intercept, to produce the residuals Then regress y on the residuals z to obtain β^1.

The orthogonalization does not change the subspace spanned by x1 and x2, it simply produces an orthogonal basis for representing it.

Step by step multiple regression

Now that we converted orthogonalization into a process of regressions, we can adapt the process to multiple regressions.

Initialize z0 = x0 = 1 .

. For j = 1, 2, . . . , p:

Regress y on the residual zp to give the estimate βˆ p.

Note that the inputs z0, . . . , zj−1 in step 2 are orthogonal, hence the simple regression coefficients computed there are in fact also the multiple regression coefficients.

The result of this algorithm is

Resultant β for the algorithm, self-generated.

The multiple regression coefficient βˆ j represents the additional contribution of xj on y, after xj has been adjusted for x0, x1, . . . , xj−1, xj+1, . . . , xp.

We can calculate the variance using:

The variance of the estimators, self-generated.

In other words, the precision with which we can estimate βˆ p depends on the length of the residual vector zp; this represents how much of xp is unexplained by the other xk’s.

Multiple outputs Regression

Since now we used only one dependent variable in the regression, but what if we have multiple dependent variables Y1, Y2, . . . , YK?

To solve this, we first assume a linear model for each output variable,

Multiple output regression, self-generated.

in matrix notation Y = XB + E. To obtain the loss function we just have to perform a straightforward generalization of the univariate loss function,

The error of multiple-output regression, self-generated.

where the least squares estimates have the form of

Coefficients of the regression, self-generated.

Hence the coefficients for the kth outcome are just the least squares estimates in the regression of yk on x0, x1, . . . , xp. Multiple outputs do not affect one another’s least squares estimates.

If the errors are correlated, it’s better to modify in favor of a multivariate version. Suppose Cov(ε) = Σ, then the multivariate weighted loss function becomes:

Multivariate weighted loss function, self-generated.

Here f(x) is the vector function f1(x), . . . , fK(x), and yi the vector of K responses for observation i.

Conclusion

In this post, we introduced the most used linear regression machine learning technique, the least-squares. We explained distinct approaches to multiple linear regressions and regressions with multiple outputs.

We know that it’s super theoretical, but don't be scared, the next post will be about subset selection, a technique to reduce the dimensionality of the model and avoid overfitting. After this post, we will use it in practice using Python.