The Three Regressions
By: Gaurav Thapa
Tags: Regression, Towards Data Science, Mathematics, Data Science, Machine Learning

The Three Regressions

Photo by Scott Graham on Unsplash

When I first started my journey with Data Science and Machine Learning, I was overwhelmed by the sheer amount of information that was available on the internet. Overtime I realized that the majority of information was just the same thing being presented in different ways; most of the things that I actually wanted to learn or dive deep into were pretty scarce, if at all present. Thus, to help anyone who has the same questions as I had before, I will try to present discussions on topics which are often encountered but have relatively less detailed information available.

Here, we shall discuss the detailed mathematical background of Linear, Multiple, Ridge and Lasso regressions. Before we start, we shall recall some important definitions and properties which will be important for the said derivations.

Some Definitions and Properties

Random Variable (RV)

Let us consider a random experiment E and let S be its corresponding sample space. A random variable can be defined as a function that assigns a real number X(s) to every element s of S i.e. it assigns a value to every outcome of the experiment E.

Discrete Random

If a RV X can take either a finite or countably infinite number of values, then X is said to be a discrete RV.

Probability Mass Function (pmf)

If X is a discrete RV that takes the values x1, x2, x3, …, such that P(X=xi)=pi, where

then pi is called the probability mass function.

Continuous RV

If a RV X can take all the values in an interval, then X is said to be a continuous RV.

Probability Density Function (pdf)

If X is a continuous RV and

then f(x) is called the probability density function.

Mathematical Expectation or Arithmetic Mean

If X is discrete RV with pmf P(X=xi)=pi, i=1, 2, 3,…; the mathematical expectation or the arithmetic mean of X is

If X is a continuous RV with pdf f(x), then

Moments

Let X be a RV. The r-th order non-central moment about any value ‘a’ is given by

Such moments are called central moments.

Variance

The second order central moment is called the variance.

Property of Variance:

Proof: We know that,

Standard Deviation

The positive square root of variance is called the standard deviation.

Normal Distribution

A continuous RV X is said to follow a normal distribution with parameters mu and sigma if its pdf is given by

It can be symbolically denoted as

L1 and L2 norm

Let us consider an n-dimensional Euclidean space. Any vector x belonging to this space is in the form x=(x1, x2,…, xn). Here, the length of x is measured as

This is called the Euclidean norm or the L2 norm.

The L1 norm is given by

The distance derived using this norm is called the Manhattan distance.

Singular Matrix

An n*n square matrix X is said to be non-singular if det(X) is not 0 and singular if det(X)=0.

Transpose of a matrix

Let X be an m*n matrix. Then, the n*m matrix obtained by interchanging the rows and columns of X is called the transpose of X, denoted by X^T.

Transpose Properties

Linear Dependence and Independence

A set of vectors is said to be linearly dependent if at least one of the vectors can be represented as a linear combination of the others i.e. if one of them lies in the span of the others. If not, then it is said to be linearly independent.

Rank of a Matrix

The rank of a matrix is the maximum number of linearly independent column vectors in the matrix.

Collinearity and Super-collinearity

If some of the independent predictor variables in the data are highly linearly related, then we can say that the data has collinearity. When we have two or more independent variables which are perfectly linearly dependent, then the data possesses super-collinearity.

Method of Lagrange Multiplier

The method of Lagrange multipliers is used to find the local maxima and minima of a function subject to equality constraints. The method consists of reformulation of the given problem resulting in the creation of a function called the Lagrangian function which is then solved to find the stationary points.

Correlation Coefficient

Let X and Y be two RVs corresponding to a random experiment. Let the experiment be repeated n number of times so that we end up with pairs of values (x1, y1), (x2, y2),…, (xn, yn), which are plotted on a graph paper.

Equation of the Simple Regression Line

In the graph that we had obtained while deriving the correlation coefficient, let us suppose that Y is dependent on X. Let the equation of the regression line be

Error in Estimate of Y

Multiple Regression

Let us consider n number of samples with each sample having p characteristics, where n<p. Here, the n*p dimensional data matrix X is

Let Y be the response variable, where Y=(Y1, …, Yn)^T.

Using the linear regression form,

Maximum Likelihood (ML) Estimators

Although the likelihood can be calculated by taking this product, we prefer to use the log-likelihood instead. Either way is acceptable since the concavity of the logarithmic function ensures that the maximization of the likelihood coincides with the maximization of the log-likelihood.

Now, in order to get the value of beta that maximizes the log-likelihood, we shall first take the partial derivative of Eq. 15 with respect to beta,

Properties of the ML Estimators

Here,

Predictions

Need for Regularization

Let X be an n*p high dimensional data matrix. So, n<p.

Then, rank(X)≤=n.

Also, any subspace spanned by the columns of X has dimension≤n.

We know that n<p

Ridge Regression

It is a type of regularized regression which is used when the data suffers from multi-collinearity. It can be formulated as,

Properties of Ridge Estimators

Lasso Regression

We have seen that the parameter of the linear regression model in ridge regression are obtained by minimizing the term,

Here, the term

referred to as the ridge penalty, is a function whose choice is arbitrary. In other words, we can use a penalty of our choice in this term to be minimized. In lasso regression, the penalty is taken to be

which is called the lasso penalty. Thus, the difference between the ridge and lasso penalty is their respective use of the L2 and L1 norm i.e. in ridge, the penalty term is taken as lambda times the sum of the squared differences while in lasso, the penalty is taken as lambda times the absolute values of the regression parameters.

Unlike ridge regression, the lasso regression problem has no closed form solution except when X is orthogonal.

Let us suppose that X is orthogonal. Then,

Now, the lasso regression can be formulated as,