Section B1: Linear Regression

B1.1. Introduction

Linear regression, also known as ordinary least square (OLS), is a method of analyzing linear 

relationships between variables. Particular methods used depend on the type of data set to be 

analyzed. Types of data sets used in regression analyses include the following:

• A cross-sectional data set comprises records for a sample of businesses, individuals, 

households, cities, states, etc. measured for a time period called a reporting or reference 

period (e.g., a day, week, or month). Differences between members of a cross-sectional 

dataset are treated as random.

• A time series data set comprises observations on a variable or collection of variables for a 

sequence of different reference periods, e.g., months or years.

• A pooled cross-sectional data set comprises records from two or more disjoint cross 

sectional data sets, where each cross sectional data set may have a different temporal 

reference period.

• A panel or longitudinal data set consists of a time series for a collection of crosssectional members. The cross-sectional members, sometimes called a cohort, remain the 

same for all reference periods.

The basic OLS estimation and diagnostic techniques were originally developed for use with 

cross-sectional data sets. Methods were extended for use with other types of data. Time series 

data presents special problems, e.g., it may be difficult to tell whether or not any changes over 

time can safely be treated as random. Specialized techniques have been developed for timeseries regression. These techniques exploit the across-time correlation in time series data and 

address concerns regarding “spurious regression” (the appearance of correlation between series 

that merely exhibit a similar trend). Specialized time-series regression techniques include cointegration analysis with error-correction models and dynamic linear regression models under the 

Bayesian paradigm.

B1.2. Estimating Regression Coefficients

We assume that a dependent (response) variable yy varies with a set of independent explanatory 

variables xx1, ... , xxmm. The variation in yy has two components:

• A systematic component that can be modeled as a linear function of the xx variables.

• A random component that is unexplained and unrelated to any variation in the xx

variables. The random component is represented by an error term εε.

The linear regression model has the general form

yy = ββ0 + ββ1xx1 + ⋯ + ββmmxxmm + εε, (B1.2.1)

where ββ0, ββ1, ... , ββmm are fixed, unknown parameters, and εε is a normal (Gaussian) random 

variable representing the random component of the variation in yy. Conditional on the 

explanatory variables xxjj, we assume the following:
1. The expected value EE(εεii) = 0 for all ii.

2. All of the εεii have the same variance, i.e., VVVVVV(εεii) = σσ2 for all ii.

3. For jj ≠ ii, the error terms εεii and εεjj are independent, i.e., EE�εεii, εεjj� = 0.

We also assume that the linear functional form B1.2.1 is appropriate and that none of the 

dependent variables xxjj may be expressed as a linear combination of the others.

We apply linear regression analysis to a set of nn observed values of a dependent variable yy and 

mm ≥ 1 associated independent variables xxjj to estimate the parameters ββ0, ββ1, ... , ββmm. For ii ∈

{1, ... , nn} we assume that the observed value yyii can be estimated as a linear function yy�ii of the 

independent variables and the estimated parameters: 

yy�ii = ββ̂

0 + ββ̂1xxii,1 + ⋯ + ββ̂mmxxii,mm. (B1.2.2)

When using the model for prediction, we compute yy�kk ∉ {yy�1, ... , yy�nn}, where yykk is unknown and 

the covariates xxkk,1, ... , xxkk,mm are known or predicted by other means. 

The vector of estimators ββ� that minimizes the sum of squared prediction errors ∑ (yyii − yy�ii) nn 2

ii=1 is 

the ordinary least squares (OLS) estimator of ββ. It can be shown that

ββ� = (XX′XX)−1XX′yy, (B1.2.3)

where yy = �

yy1

⋮

yynn

�, ββ� = �

ββ̂

0

⋮

ββ̂

mm

� , and XX = �

1

⋮

1

xx1,1 ⋯ xx1,mm

⋮ ⋱ ⋮

xxnn,1 ⋯ xxnn,mm

�.

Although the matrix XX′XX is always square, it need not have a unique inverse. When XX′XX is not of 

full rank, we cannot use OLS to estimate the parameters. In this case, we must use generalized 

matrix inverses, and it may not be possible to estimate all the parameters.

B1.3. Regression Diagnostics

Before using the results of a regression analysis, we compute test statistics that indicate whether 

or not the data support the assumptions embedded in the OLS model. For ii ∈ {1, ... , nn}, let yy�ii be 

as defined in equation B1.2.2. The difference εε̂

ii ≡ yyii − yy�ii is known as the ii

tth residual and plays 

an important role in model evaluation. Some fundamental test statistics are based on the 

partitioned sum of squares. Let

yy� = 1

nn�yyii

nn

ii=1

. (B1.3.1)

It can be shown that the sum of squares ∑ (yyii − yy�) nn 2

ii=1 can be partitioned into two components 

representing Model Error (∑ (yyii − yy�ii) nn 2

ii=1 ) and Residual Error (∑ (yy�ii − yy�) nn 2ii=1 ):
�(yyii − yy�)2

nn

ii=1

= �(yyii − yy�ii)2

nn

ii=1

+�(yy�ii − yy�)2

nn

ii=1

. (B1.3.2)

 Total Error = Model Error + Residual Error

The Model Error term ∑ (yyii − yy�ii) nn 2

ii=1 is the sum of the squared residuals.

Mean Squared Error

Hypothesis tests on the regression parameters use the mean squared error (MSE or ss2), which is 

the sum of the squared residuals divided by the model’s degrees of freedom:

ss2 = ∑ (yyii − yy�ii) nn 2

ii=1

nn − mm − 1 = ∑ εε�ıı nn 2

ii=1

nn − mm − 1

. (B1.3.3)

The model’s degrees of freedom is the sample size nn minus the number of parameters estimated,

mm + 1. To compute the MSE, we must have nn > mm + 1. It can be shown that the MSE is an 

unbiased estimator of σσ2, the variance of the εε’s.

Coefficient of Variation RR2

The coefficient of determination or R2 statistic for a regression model is

RR2 ≡ 1 − Model Error

Total Error = 1 − ∑ (yyii − yy�ii) nn 2

ii=1

∑ (yyii − yy�) nn 2

ii=1

. (B1.3.4)

RR2 (sometimes called multiple RR2) indicates the proportion of the total error that the independent 

variables explain. Because the RR2 statistic may be inflated by the use of a large number of 

explanatory variables, the adjusted RR2 (RRAA

2), which corrects for the number of degrees of 

freedom, is generally preferred:

RRAA

2 ≡ 1 −

∑ �

(yyii − yy�ii)2

(nn − mm − 1) � � nn

ii=1

∑ �

(yyii − yy�)2

(nn − 1) � � nn

ii=1

= 1 − (nn − 1)ss2

∑ (yyii − yy�) nn 2

ii=1

. (B1.3.5)

Variances of the Regression Coefficients ββjj

In order to compute test statistics, we estimate the variances and standard errors of the regression 

coefficients. When there is only one xx variable, the variances of ββ0 and ββ1 are estimated as 

follows:

ssββ�0

2 = ss2 ∑ xxii,1 nn 2

ii=1

nn ∑ �xxii,1 − xx̅

1�

2 nn

ii=1

, where xx̅

1 = 1

nn�xxii,1,

nn

ii=1

. (B1.3.6)

and
ssββ�1

2 = ss2

∑ �xxii,1 − xx1̅�

2 nn

ii=1

. (B1.3.7)

When there are multiple explanatory variables, we estimate the covariance matrix of the vector ββ�

as

ssββ�

22 = ss2(XX′

XX)−1. (B1.3.8)

Test for the significance of a parameter ββjj

For jj ∈ {0,1, ... , mm}, suppose we wish to test the hypothesis

HH0: ββjj = 0 vs. HHAA: ββjj ≠ 0. (B1.3.9)

For this two-sided hypothesis test, we may use the test statistic

ttjj = ββ̂

jj

ssββ�jj

. (B1.3.10)

For large samples, under the null hypothesis, the density of ttjj is approximately normal 

(Gaussian) with mean 0 and variance 1. For ZZ~NN(0,1), PP(|ZZ| > 1.96) ≈ 0.05. Thus when 

testing B1.3.9 with αα = 0.05, we generally reject HH0 when ttjj ≥ 2. This means that the 

explanatory variable xxjj explains a significant amount of the variation in the dependent variable 

yy.

Weighted Least Squares and Robust Standard Errors

The linear regression setup is based on several assumptions. We assume, for example, that the 

linear functional form is appropriate and that the model’s systematic component incorporates all 

of the important explanatory variables. Also, we assume that, for all ii ∈ {1, ... , nn}, 

a) EE(εεii) = 0 and

b) VVVVVV(εεii) = σσii

2 = σσ2.

The assumption of constant variances (assumption b above), is called homoscedasticity. When 

this assumption is violated, the data are heteroscedastic, and the following problems occur:

• The least squares estimator (B1.2.3) is not a Best Linear Unbiased Estimator (BLUE). It 

is still unbiased, but there is another estimator with lower variance.

• The standard errors of the estimated coefficients, calculated by formula B1.3.8, may be 

incorrect; this can lead to erroneous conclusions from hypothesis tests.

The Breusch-Pagen (BP) test is a χχ2 test that tests the null hypothesis of homoscedasticity 

against the alternative hypothesis that the data are heteroscedastic. Other homoscedasticity tests 
are also discussed in the statistics literature. For the BP test, we use an auxiliary regression 

equation to regress the residuals (εε̂

ii) on the independent variables and compute the RR2 statistic 

RR�2 from this regression. Under the null hypothesis, the test statistic nnRR�2 has a χχ2 distribution 

with mm degrees of freedom. If the pp-value from the χχ2 test falls below 0.05, we reject the null 

hypothesis and conclude that heteroscedasticity is present in the data.

To find the BLUE in the presence of heteroscedasticity, we may use weighted least squares 

(WLS), which is an example of generalized least squares (GLS). For ii ∈ {1, ... , nn}, let xxii denote 

the values of the explanatory variables for observation ii, and let wwii(xxii) be a function of xxii such 

that 

VVVVVV(εεii|xxii) = σσ2wwii(xxii). (B1.3.11)

Then 

EE � εεii

�wwii(xxii)

�xxii� = 0 and VVVVVV � εεii

�wwii(xxii)

�xxii� = σσ2. (B1.3.12)

Setting wwii = wwii(xxii), we divide equation B1.2.1 by �wwii to obtain, for ii ∈ {1, ... , nn},

yyii

�wwii

= ββ0

�wwii

+ ββ1 � xxii,1

�wwii

� + ⋯ + ββmm �

xxii,mm

�wwii

� +

εεii

�wwii

, (B1.3.13)

or

yy�ii = ββ0xx�ii,0 + ββ1xx�ii,1 + ⋯ + ββmmxx�ii,mm + εε̃

ii, (B1.3.14)

where yy�ii = yyii

�wwii

, xx�ii,0 = 1

�wwii

, εε̃

ii = εεii

�wwii

, and, for jj ∈ {1, ... , mm}, xx�ii,jj = xxii,jj

�wwii

. For the transformed 

equation B1.3.14, we have VVVVVV(εε̃

ii) = σσ2 for all ii ∈ {1, ... , nn}, so the homoscedasticity 

assumption is valid. The regression results, however, must be interpreted using the original 

model B1.2.1.

The use of WLS assumes the availability of an appropriate set of weights wwii. An alternative is to 

use robust standard errors (also called heteroscedasticity-consistent standard errors or HuberWhite standard errors) when testing B1.3.9. The robust variance is estimated as

ssHH,ββ�jj

2 = ∑ ̂

ii,jj

2 εε̂

ii nn 2ii=1

SSSS jj

2 , (B1.3.15)

where ̂

ii,jj is the ii

tth residual from the auxiliary regression with xxjj as the dependent variable and 

all the other xx’s as independent variables, and SSSS jj

2 is the sum of the squared residuals (εε̂

ii’s) 

from this auxiliary regression. For example, in a model with only one independent variable xx, 

the robust variance estimate for ββ̂

1 is
ssHH,ββ�1

2 = ∑ (xxii − xx̅)2εε̂

ii nn 2ii=1

∑ (xxii − xx̅) nn 2

ii=1

. (B1.3.16)

Diagnostics such as tt-statistics computed using the robust standard errors provide accurate

results in the presence of heteroscedasticity.

B1.4. Instrumental Variables and Two-stage Lease Squares (2SLS)Regression Diagnostics

In order for OLS estimator ββ� (shown in B1.2.3) to be consistent, the explanatory variables 

xx1, ... , xxmm must be exogenous, i.e., they must be uncorrelated with the error term εε. When one or 

more of the explanatory variables is correlated with εε, they are jointly determined with the 

dependent variable yy, and the use of two-stage least squares (2SLS) provides consistent 

parameter estimates for the model B1.2.1. The 2SLS technique, which employs instrumental 

variables to compensate for deficiencies in the OLS model, is also useful when important 

variables determining yy cannot be accurately measured. 

Consider the model

yy = ββ0 + ββ1xx1 + ββ2xx2 + εε, (B1.4.1)

and suppose that the explanatory variable xx2 is endogenous or cannot be accurately measured. If 

we perform OLS with xx2 omitted, it will be incorporated into the error term εε, and the 

coefficients ββ0 and ββ1 (computed as in B1.2.3) may be biased. A new variable zz that is 

correlated with xx1 but not with εε may be used as an instrumental variable (IV) for xx1. We test 

the correlation between xx1 and zz by fitting the model

xx1 = γγ0 + γγ1zz + ζζ, (B1.4.2)

where ζζ is a normally distributed random error term, and testing the hypothesis HH0: γγ1 = 0 vs. 

HHAA: γγ1 ≠ 0. 

The OLS model with xx2 omitted is 

yy = ββ0 + ββ1xx1 + εε. (B1.4.3)

We compute the covariance of each term in B1.4.3 with the instrumental variable zz: 

CCCCCC(yy, zz) = ββ1CCCCCC(xx1, zz) + CCCCCC(εε, zz), (B1.4.4)

where we have used the fact that the covariance of xx1 with the constant ββ0 is zero. Because we 

assume that CCCCCC(εε, zz) = 0, equation B1.4.4 yields the IV estimator of ββ1:

ββ̂

1 = CCCCCC(yy, zz)

CCCCCC(xx1, zz)

, (B1.4.5)
which we can estimate from the data. In the special case that xx1 = zz, the ββ̂

1 in B1.4.5 reduces to 

the usual OLS estimator. The IV technique may be extended to the multiple regression setting in 

a straightforward manner.

B1.5. The Two-stage Least Squares (2SLS) Procedure

In 2SLS, we assume that at least one instrumental variable is available for each endogenous 

explanatory variable in the OLS model (B1.2.1). Consider the model

yy = ββ0 + ββ1xx1 + ⋯ + ββmmxxmm + γγ1yÿ

1 + ⋯ + γγppyÿpp + εε, (B1.5.1)

where the variables xx1, ... , xxmm are exogenous, and the variables yÿ

1, ... , yÿpp are endogenous. For 

each endogenous explanatory variable yÿ, we combine the available IV’s into a single IV by 

running an auxiliary regression of the form

yÿ= γγ0 + γγ1zz1 + γγ2zz2 + ⋯ + γγqqzzqq + ζζ, (B1.5.2)

where qq is the number of IV’s for yÿand ζζ is a normally-distributed error term. In practice, we 

may include the exogenous explanatory variables in the OLS model B1.5.1, as well as all 

additional IV’s, as independent variables (zz’s) in equation B1.5.2. Fitting the model B1.5.2 for 

each endogenous explanatory variable is the first stage of 2SLS.

After fitting model B1.5.1, we compute the model’s fitted values for each observation:

yy�̈= γγ�0 + γγ�1zz1 + γγ�2zz2 + ⋯ + γγ�ppzzpp, (B1.5.3)

where we have suppressed the subscript indicating an individual observation. In the second 

stage of 2SLS, we fit model B1.5.1 with the estimated values yy�̈substituted for the observed yÿ

values of the exogenous explanatory variables:

yy = ββ0 + ββ1xx1 + ⋯ + ββmmxxmm + γγ1yy�̈

1 + ⋯ + γγppyy�pp̈ + εε. (B1.5.4)

Although this is the second stage of 2SLS, the procedure should not be performed “by hand” in 

two independent stages, because the second stage model fitting will yield residuals and variance 

estimates that don’t account for the variance of the error term ζζ in equation B1.5.2. Statistical 

software packages with 2SLS functions automatically correct these residuals and the associated 

diagnostic statistics.

Because of the additional random error term in the 2SLS procedure, the variances of 2SLS 

coefficients always exceed those of the OLS coefficients computed from formula B1.2.3. In the 

case of endogenous explanatory variables, however, 2SLS provides consistent parameter 

estimates, whereas the OLS estimates are inconsistent.
B1.6. Regression with Time Series Data

A time series is a set of observations xx1, ... , xxnn, where, for tt ∈ {1, ... , nn}, xxtt is the value or set of 

values observed for time period tt. We assume that the data are generated by a stochastic 

process, and thus contain random errors, and that the time intervals are equal and consecutive. 

Regression on time series data differs from regression on cross-sectional data because of the 

following considerations:

1. Unlike cross-sectional data, time series data are ordered.

2. While we assume that each cross-sectional observation is uncorrelated with the other 

observations, time series data are generally autocorrelated.

The linear time-series regression model has the general form

yytt = ββ0 + ββ1xxtt,1 + ⋯ + ββmmxxtt,mm + εεtt, (B1.6.1)

where, for tt ∈ {1, ... , nn}, the error term εεtt is independent of the explanatory variables xxtt for all tt, 

and εεtt~NN(0, σσ2), conditional on the explanatory variables xxtt,jj. 

In time series regression, all of the assumptions detailed in section B1.2 apply, and we add a 

further assumption regarding the error terms εεtt: we assume that, conditional on the sample data, 

error terms associated with different time periods are uncorrelated:

CCCC (εεtt, εεss) = 0 for all ss ≠ tt. (B1.6.2)

Time series regression also requires additional attention to independent variables that may be 

correlated with the error term. For an explanatory variable xxtt to be considered exogenous, it 

cannot react to any past or future changes in the dependent variable yytt. 

Under these assumptions, we may apply OLS to time series data, estimating coefficients and 

diagnostic statistics as described in section B1.

Dummy Variables

Time series regression models often include binary variables, known as dummy variables, that 

represent events. A dummy variable takes a value of 1 if the event occurred and a value of 0 

otherwise. Common uses of dummy variables include the following:

1. Reducing outlier effects by including a dummy variable that takes a value of 1 only in the 

time period coinciding with a series outlier.

2. Accounting for seasonality by including monthly or quarterly dummy variables.

3. Representing permanent changes, e.g., policy changes, that influence the dependent 

variable. In this case, the value of the dummy variable is 0 for all time periods prior to 

the change and 1 for all time period after the change.

Time Trends

Many time series contain trends that can cause regression diagnostics to erroneously indicate 

causal relationships between series. Two unrelated series with similar trends can appear 
correlated, leading to the “spurious regression problem.” In some cases we may alleviate this 

problem by including a trend term in the model. Adding a trend term αα to equation B1.6.1 

provides the model

yytt = αα + ββ0 + ββ1xxtt,1 + ⋯ + ββmmxxtt,mm + εεtt, (B1.6.3)

where αα is a trend parameter. When the series yytt contains both upward and downward trends, a 

quadratic trend term may be added to the model:

yytt = αα1tt + αα2tt2 + ββ0 + ββ1xxtt,1 + ⋯ + ββmmxxtt,mm + εεtt. (B1.6.4)

The addition of cubic and higher-ordered trend terms is allowed but usually not recommended, 

because too many polynomial trend terms may obscure the significance of the explanatory 

variables xxtt,jj.

Autocorrelation in Time-series Regression Residuals

To test that assumption B3.2.2 holds for a particular regression model, some regression software 

packages compute the Durbin-Watson (D-W) statistic:

dd = ∑ (εε̂

ii − εε̂ii−1) nn 2

ii=2

∑ εε̂

ii nn 2

ii=1

. (B1.6.5)

The D-W statistic compares the squared differences of adjacent residuals to the squared 

residuals. If the squared differences are small relative to the squared residuals, resulting in a low 

value of dd, there is evidence of first-order autocorrelation. Critical values for the D-W statistic 

depend on nn and mm. The test is for first-order autocorrelation only, i.e., it tests only for 

correlation between residuals from adjacent time periods.

One way to reduce the autocorrelation of time-series regression residuals is to include a lagged 

dependent variable as an independent variable in the model. Model B1.6.1 then becomes

yytt = ββ0 + ββ−1yytt−1 + ββ1xxtt,1 + ⋯ + ββmmxxtt,mm + εεtt. (B1.6.6)

For models that include a lagged dependent variable as an independent variable, the D-W 

statistic underestimates autocorrelation. For these models, Durbin proposed the h-statistic, 

which is a bias-adjusted version of the D-W statistic. It follows a normal distribution for large 

samples. Let ββ−1 be the regression coefficient of yytt−1. When the variance ssββ−1

2 <

1

nn

,

h = �1 − dd

2

� � nn

1 − nnssββ−1

2 . (B1.6.7)

Because the D-W statistic is difficult to interpret, Wooldridge (2018) suggests the alternative of 

performing a tt-test on the estimated first-order correlation coefficient of the OLS residuals εε̂

tt. 

That is, we first run the OLS regression B1.6.1 and then run a second regression of εε̂

tt on εε̂tt−1. 
The tt-statistic B1.3.10 of the coefficient of εε̂

tt−1 will indicate the significance of first-order 

autocorrelation in the OLS residuals. The tt-statistic is asymptotically normal and may be 

interpreted in the usual manner.