Least Square Estimation of Parameter
By: Praveen Yaragatti
Tags: Model, Linear Regression, Least Squares, Parameter, Multiple Linearregression

What Is the Least Squares Method

The “least squares” method is a type of mathematical regression analysis. used to determine the line that best fits your data set by providing a visual representation of the relationship between data points. Each data point represents the relationship between a known independent variable and unknown dependent variable.

What Does the Least Squares Method Tell You?

The least squares method provides the general rationale for placement of the best fit line between the data points under consideration. The most common application of this method, sometimes referred to as “linear” or “ordinary”, aims to create a straight line that minimizes the sum of squares of the errors generate by the associated results equations, such as squared residuals resulting from differences in observed value and expected value, based on that model.

This regression analysis method starts with a series of data points to be plotted on an x-axis and y-axis graph. An analyst who uses the least squares method it will generate a better fit line that explains the potential relationship between independent and dependent variables.

In regression analysis, dependent variables are shown on the vertical y-axis, while independent variables are shown on the horizontal x-axis. These designations will form the equation for the best-fit line, that is determined by the least squares method. The least squares method is presented under the forms of Simple linear regression, multiple linear model and non-linear models (Gauss-Newton method) Topics such as residual analysis, sample distribution of estimators, trust limits and ranges, etc.

SIMPLE LINEAR REGRESSION — LEAST SQUARES METHOD

Model

Consider the following variables and parameters:

Response or dependent variable = Y

Auxiliary or independent variable = X

Parameters = A, B

The response variable is linear with the parameters

Y = A+BX

Objective

The objective of the method is to estimate the parameters of the model, based on the observed pairs of values and applying a certain criterium function (the observed pairs of values are constituted by selected values of the auxiliary variable and by the corresponding observed values of the response variable), that is:

Estimation method

In the least squares method the estimators are the values of A and B which minimize the object function. Thus, one has to calculate the derivatives ∂Φ/∂A e ∂Φ/∂B, equate them to zero and solve the system of equations in A and B.

The solution of the system can be presented as:

Notice that the observed values y, for the same set of selected values of X, depend on the collected sample. For this reason, the problem of the simple linear regression is usually presented in the form:

y = A + BX + ε

where ε is a random variable with expected value equal to zero and variance equal to σ2.

So, the expected value of y will be Y or A+BX and the variance of y will be equal to the variance of ε.

To analyse the adjustment of the model to the observed data, it is necessary to consider the following characteristics:

Sum of squares of the residuals:

This quantity indicates the residual variation of the observed values in relation to the estimated values of the response variable of the model, which can be considered as the variation of the observed values that is not explained by the model.

Sum of squares of the deviations of the estimated values of the response variable of the model:

This quantity indicates the total variation of the observed values in relation to the mean

It is easy to verify the following relation:

SQtotal = SQmodel + SQresidual

or

r2 (coefficient of determination) is the percentage of the total variation that is explained by the model and

1-r2 is the percentage of the total variation that is not explained by the model.

2. MULTIPLE LINEAR REGRESSION — LEAST SQUARES METHOD

Model

Consider the following variables and parameters:

Response or dependent variable

= Y

Auxiliary or independent variables

= X1, X2,…, Xj,…, Xk

Parameters

= B1, B2,…, Bj,…, Bk

The response variable is linear with the parameters

Y = B1X1+B2X2+… + BkXk = Σ BjXj

Objective

The objective of the method is to estimate the parameters of the model, based on the observed n sets of values and by applying a certain criterium function (the observed sets of values are constituted by selected values of the auxiliary variable and by the corresponding observed values of the response variable), that is:

Observed values x1,i x2,i,…, xj,i,.., xk,i and yi for each set i, where i=1,2,…,i,…n

Values to be estimated B1,B2,…,Bj,…,Bk et (Y1,Y2,…, Yi,…, Yn)

The estimated values can be represented by:

Object function (or criterium function):

Estimation method

In the least squares method the estimators are the values of Bj which minimize the object function.

As with the simple linear model, the procedure of minimization requires equating the partial derivatives of Φ to zero in order to each parameter, Bj, where j=1, 2,…, k. The system is preferably solved using matrix calculus.

Matrix version

Matrix X(n,k) = Matrix of the n observed values of each of the k auxiliary variables

Vector y(n,1) = Vector of the n observed values of the response variable

Vector Y(n,1) = Vector of the values of the response variable given by the model (unknown)

Vector B(k,1) = Vector of the parameters

Vector

or b(k,1) = Vector of the estimators of the parameters Model

Y(n,1) = X(n,k). B(k,1) ou Y=X.B+ε

Object function

Φ(1,1) = (y-Y)T.(y-Y) ou Φ(1,1) = (y-X.B)T.(y-X.B)

To calculate the least squares estimators it will suffice to put the derivative dΦ/dB of Φ in order to vector B, equal to zero. dΦ/dB is a vector with components ∂Φ/∂B1, ∂Φ/∂B2,…, ∂Φ/∂Bk. Thus:

dΦ/dB(k,1) = -2.XT.(y-X.B) = 0

or XTy — (XT.X). B = 0

and b = (XT.X)-1. XTy

Example of the Least Squares Method

An example of the least squares method is an analyst who wishes to test the relationship between a company’s stock returns, and the returns of the index for which the stock is a component. In this example, the analyst seeks to test the dependence of the stock returns on the index returns. To achieve this, all of the returns are plotted on a chart. The index returns are then designated as the independent variable, and the stock returns are the dependent variable. The line of best fit provides the analyst with coefficients explaining the level of dependence.

Conclusion

It is necessary to know the sampling theory and statistical inference. This manual will use one of the general methods most commonly used in the estimation of parameters — the least squares method. In many cases this method uses iterative processes, which require the adoption of initial values. Therefore, particular methods will also be presented, which obtain estimates close to the real values of the parameters. In many situations, these initial estimates also have a practical interest. These methods will be illustrated with the estimation of the growth parameters and the S-R stock-recruitment relation.