Lagged Variable Regressions and Truth
By: Ben Ogorek
Tags: Towards Data Science, Machine Learning, Math, Data Science, Statistics

Lagged Variable Regressions and Truth

Dynamic regression models offer vast representational power but also bias risk

Variables related to each other over adjacent time steps, originally in the context of dynamic Bayesian networks (Wikimedia user Guillaume.lozenguez, CC BY-SA 4.0)

Turn a nonlinear structural time-series model into a regression on lagged variables using rational transfer functions and common filters,

and See bias in an ordinary least squares lagged variable regression due to remaining serial correlation in the errors,

in an ordinary least squares lagged variable regression due to remaining serial correlation in the errors, Use generalized least squares to eliminate the bias and recover the process parameters, i.e., the “truth.”

“Near things are more related than distant things,” states Tobler’s (very believable) first law of geography. However little we know about an underlying data generating mechanism, we can often exploit sheer nearness — in time or space — to gain an advantage. But we also know that regressing on a time-shifted copy of the dependent variable, one that is affected by all the same factors, comes with caveats.

The biggest caveat is the potential for asymptotic bias in ordinary least squares (OLS) regression that no amount of additional data can alleviate. This happens when regressors are contemporaneously correlated with the error term. As explained by Peter Kennedy in A Guide to Econometrics, “This is because the OLS procedure, in assigning ‘credit’ to regressors for explaining variation in the dependent variable, assigns, in error, some of the disturbance-generated variation of the dependent variable to the regressor with which that disturbance is contemporaneously correlated.” In short, OLS can’t separate the effects of regressor and error when the two are correlated.

From a passive prediction standpoint, the bias might not matter, and “truth,” in this article’s title, refers to the true data generating process underlying a time series. Henceforth, our goal is quality estimation of the process parameters. For both an objective truth and a case where parameters have scientific meaning, I’ll turn to a nonlinear, nonstationary, structural time-series model, the “fitness-fatigue” model of athletic performance. This model relates training history to athletic performance via latent fitness and fatigue constructs and is tuned by scientifically meaningful parameters.

By rewriting this model using rational transfer functions of input to output, a set of “common filtering” operations will become apparent. Applying these filters will transform the model into a dynamic linear regression, but the filtering operations will also have a side effect: errors with an MA(2) correlation structure. We will see the destructive power of contemporaneous correlation between regressors and error in OLS regression, but in the end, we will get to the truth via a GLS regression that accommodates that structure.

A nonlinear structural time-series model

The “fitness-fatigue” model of athletic performance has the form:

where p_t is a numeric measure of (athletic) performance, w_t is training “dose,” and ϵ_t is i.i.d. Gaussian error. The model can be viewed as a linear regression with two nonlinear features,

latent representations of athletic “fitness” and “fatigue,” respectively. These features are convolutions of athletic training history with exponential decay, differing only in the decay rate. Typically, fatigue (g_t) is relatively transient and associated with a faster exponential decay, but also has a relatively large regression weight compared to fitness.

With a bit of algebra, fitness and fatigue can be put into the dynamic form:

where θ_1 = exp(-1 / τ_1) and θ_2 = exp(-1 / τ_2).

If we plug the dynamic representation of fitness and fatigue into the original model, we arrive at

which looks nicer but is not immediately useful as h_t and g_t depend on unknown parameters.

Rational transfer functions to the rescue

Working a little harder, we can use the “backshift operator” B defined by B y_t = y_{t-1} for any time-indexed variable y, and arrive at

Solving for h_t and g_t and plugging them back into the original model, we arrive at:

Thus we have two rational transfer functions operating on the exogenous input series w_t. With rational transfer functions, denominator terms of the form (1- θ B) correspond to an autoregressive impulse response, i.e., a process with a long memory, and this is a nuisance to us. There is an option to rid ourselves of the denominator component, but not one without a cost.

A “common filter,” as discussed in Identification of Multiple-Input Transfer Function Models (Liu & Hanssens, 1982) premultiplies the right and left-hand side of a time-series equation by (1- θB). It does not change the transfer function weights, so you can apply multiple common filters in succession, and besides causing complexity and losing some rows due to lags, you have not destroyed the relationship between the input and output series.

The point is, if we were to use the common filter (1- θ_1 B) (1 -θ_2 B), we rid ourselves of the autoregressive components:

The above still looks ugly, but after expanding the polynomials applying the backshift operations, we arrive at:

This is a regression of performance on two lagged versions of itself and on the external training input, but one with a twist. There is the issue of the MA(2) error structure induced by the common filters, and it has real consequences.

A simulated example in R

This section will use simulated data that can be reproduced from an R gist or downloaded directly as a csv file. To run the R code below, change the file path in the following R code block:

train_df <- read.csv("c:/devl/data/train_df.csv")

head(train_df)

The following calculations show the theoretical values of all the coefficients in the lagged variable model as simulated:

# True parameter values:

mu <- 496

theta1 <- exp(-1 / 60)

theta2 <- exp(-1 / 13)

k1 <- .07

k2 <- .27 #Theoretical coefficient for intercept is

(1 - theta1) * (1 - theta2) * mu #Theoretical coefficient for performance lagged once is

theta1 + theta2 #Theoretical coefficient for performance lagged twice is

-theta1 * theta2 #Theoretical coefficient for training lagged once is

k1 * theta1 - k2 * theta2 #Theoretical coefficient for training lagged twice is

-theta1 * theta2 * (k1 - k2)

And thus the goal is to recover the following equation:

Side note: To go the opposite direction, solve for θ_1 and θ_2 by substitution, which leads to a quadratic equation in one of the two parameters. They lead to the same two roots, so choose θ_1 to be the larger of the two. With θ_1 and θ_2 in hand, solving for μ is trivial and k_1 and k_2 constitute a linear system with two equations and two unknowns.

Fitting the lagged regression model

To prepare for the regression, the code below lags performance and training variables once and twice each. The dplyr package makes it easy:

library(dplyr) train_aug <- train_df %>%

mutate(perf_lag1 = lag(perf, n = 1, order_by = day),

perf_lag2 = lag(perf, n = 2, order_by = day),

train_lag1 = lag(w, n = 1, order_by = day),

train_lag2 = lag(w, n = 2, order_by = day))

Let’s see what happens when we regress performance on these lagged variables using OLS:

my_lm <- lm(perf ~ perf_lag1 + perf_lag2 + train_lag1 + train_lag2,

data = train_aug[3:nrow(train_aug), ])

summary(my_lm)

The above code results in the partial output:

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) 33.99821 12.09470 2.811 0.005327 **

perf_lag1 0.48066 0.05619 8.553 1.18e-15 ***

perf_lag2 0.46189 0.05504 8.393 3.45e-15 ***

train_lag1 -0.15602 0.04406 -3.541 0.000475 ***

train_lag2 -0.02346 0.04516 -0.520 0.603807

---

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1



Residual standard error: 8.518 on 252 degrees of freedom

Multiple R-squared: 0.9117, Adjusted R-squared: 0.9103

F-statistic: 650.4 on 4 and 252 DF, p-value: < 2.2e-16

There is clearly bias in the OLS coefficient estimates as a results of the serial correlation remaining in the residuals.