Adjusting for Autocorrelated Errors in Neural Networks for Time Series

1 Introduction . Time series data are increasingly ubiquitous as the cost of data collection continues to decrease . Analysts seek to model such data even when the underlying data-generating process ( DGP ) is unknown , in order to perform various tasks such as time series forecasting , classification , regression , and anomaly detection . For example , industrial internet of things ( IIoT ) sensors are being installed around manufacturing equipment to collect time series data that can help to increase the efficiency of day-to-day manufacturing operations . During the collection and modeling of time series , there are inevitably errors . It is common to assume that errors at different time steps are uncorrelated , especially for fitting neural networks ( NNs ) under the framework of maximum likelihood estimation ( MLE ) . However , errors are actually oftentimes autocorrelated due to the temporality of the data interacting with the three sources noted below . The first source is the omission of influential variables . In real-world situations , determining which variables to collect and include in the model is a complicated task . For example , temperature should be an important variable to consider if we want to forecast household electricity consumption . But publicly available electricity datasets , which many previous works are compared upon [ 37 , 38 , 54 , 57 , 64 ] , do not include such information . Even if we were to add temperature as a variable , there are still many other variables that might be influential . In many real-world cases , in order to have satisfactory prediction results , it is either too complicated to decide which and how many variables are required , or impossible to collect all the required variables . Omitting influential variables can result in autocorrelated errors . Secondly , measurement errors are almost unavoidable , and measurement errors in time series are usually autocorrelated due to the temporality of the data . For example , when taking time-sampled measurements in semiconductor fabrication equipment , sensor noise may exist due to drift , calibration 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . error , or environmental factors . If the rate of measurement is faster than the rate of fluctuation of the noise , then the errors are autocorrelated . The third important source is model misspecification . In general , it is difficult to formulate an exact model for the underlying DGP . Such model misspecification can cause errors to be autocorrelated even if an optimal fit within the limitation of the approximate model is reached . A simple example would be fitting a linear time-trend model on a sinusoidal signal . Neural networks ( NNs ) , as universal approximators , help alleviate this issue and enable us to achieve good approximations in many cases . However , even when an optimal fit is reached for a given NN structure , this does not imply that the remaining errors are uncorrelated . All three aforementioned sources can lead to autocorrelated errors [ 26 , 46 ] , in which errors at the current time step are correlated with errors at previous time steps . In ordinary least squares ( OLS ) , autocorrelated errors violate the assumption that the errors are uncorrelated , which implies that the Gauss-Markov theorem is not applicable . Specifically , the variance of the coefficient estimates increases but the estimated standard error is underestimated . Thus , if neglected , prediction accuracy is reduced , and an outcome that appears to be statistically significant may actually be insignificant . As for nonlinear data , autocorrelated errors impair the standard MLE and thus weaken model performance . Adjusting for autocorrelated errors in linear or nonlinear time series data has been studied extensively , especially in econometrics [ 4 , 13 , 20 , 21 , 24 , 29 , 49 ] . However , those methods are applicable only when the exact ( and correct ) form of the underlying system is known . On the other hand , NNs for time-series-related tasks [ 5 , 10 , 17 , 22 ] have become a popular research direction due to NNs ’ effectiveness in approximating unknown , nonlinear systems . However , to the best of our knowledge , none of the existing NN-based methods adjust for autocorrelated errors . In this work , we introduce a method to account for autocorrelated errors in NNs and show that the proposed method improves the performance in many time series tasks , especially time series forecasting . The implementation of our method can be found at https : //github.com/Daikon-Sun/AdjustAutocorrelation . Our main contributions are : • We propose to learn the autocorrelation coefficient jointly with model parameters via gradient descent in order to adjust for autocorrelated errors in NNs for time series . • Our large-scale experiments on time series forecasting show that our method improves performances across a wide range of real-world datasets and state-of-the-art NN architectures . • Based on these large-scale experiments , we identify the issue of autocorrelated errors and we suggest empirical critical values of the remaining autocorrelation in errors that act as a guideline to determine whether adjusting for autocorrelated errors is necessary . • By ablation study and grid-searching over autocorrelation coefficients and model hyperparameters , we validate the strength of our method . We also study the effect of model misspecification . • We apply our methods on other time series tasks to demonstrate the effectiveness of our method on a variety of additional time series tasks beyond forecasting . 2 Preliminaries . 2.1 Time series forecasting . In time series forecasting , we have an input matrix X = { X1 , . . . , Xt , . . . , XT } ∈ RT×N representing N variables sampled at the same rate at the same time for T time steps where Xt ∈ RN is the t-th sample . The goal is to forecast the value of Xt given the histories { X1 , . . . , Xt−1 } . In practice , only the W most recent histories { Xt−W , . . . , Xt−1 } are fed into a model . This is a common approach [ 2 , 37 , 50 , 57 ] that assumes older histories are less informative , establishes fair comparisons between different methods , and makes the memory usage plausible . Mathematically speaking , given the model f , we optimize the model parameters θ to minimize the mean squared error ( MSE ) : MSE = ∑ t ‖et‖22 = ∑ t ‖Xt − X̂t‖22 , ( 1 ) where X̂t = f ( Xt−1 , . . . , Xt−W ; θ ) is the model forecast , Xt = X̂t + et , and et is the error . 2.2 Autocorrelated Errors . In most machine learning literature and in our work , the errors are assumed to be uncorrelated across different series . Thus , for ease of understanding , we look at each series separately and assume et ∈ R. Usually , the errors et are assumed to be independent , identical , and normally distributed : Cov ( et−∆t , et ) = 0 , ∀∆t 6= 0 , ( 2 ) et ∼ N ( 0 , σ2 ) . ( 3 ) Thus , minimizing MSE is equivalent to maximizing likelihood . However , as discussed in Section 1 , there are often situations in which the assumption may be violated and the errors are thus autocorrelated . In general , a p-th order autocorrelated error has the form et = ρ1et−1 + · · ·+ ρpet−p + t , |ρi| < 1 , ∀i ( 4 ) where ρ1 , . . . , ρp are autocorrelation coefficients and t ∼ N ( 0 , σ2 ) is the uncorrelated error . Notice that the magnitude of ρi should be strictly smaller than 1 , as explained in Appendix A . Generally , the first-order autocorrelation is the single most significant term because it is reasonable to assume that the correlation between et and et−∆t decreases when ∆t increases . Thus , in this work , we only focus on the linear , first-order autocorrelation following previous work [ 4 , 13 , 16 , 29 , 49 ] and simplify the notation to et = ρet−1 + t. Nevertheless , our method can be extended to higher order in cases where other terms are important . It is possible that a nonlinear , more complex structure would be beneficial , but we didn ’ t discuss it here because ( 1 ) a linear model is a good choice for simple and basic modeling , ( 2 ) we follow the work in the field of econometrics , where most of the discussions about autocorrelated errors happen , and most importantly ( 3 ) if we use a NN for modeling the errors ( i.e. , replacing Xt − ρXt−1 with Xt − f ( Xt−1 ; θ ) ) , the overall model becomes deeper with many more parameters , then it becomes unclear whether the improvement comes from adjustment or from a bigger model . When there exists first-order autocorrelated errors , as derived in Appendix A , Cov ( et , et−∆t ) = ρ∆t 1− ρ2 σ2 , ∀∆t = 0 , 1 , 2 , . . . , ( 5 ) i.e. , errors are no longer correlated , and thus the standard MLE becomes untenable . Alternatively , one should use the following form of MLE : Xt − ρXt−1 = f ( Xt−1 , . . . , Xt−W ; θ ) − ρf ( Xt−2 , . . . , Xt−W−1 ; θ ) + t , ( 6 ) so that the remaining errors are uncorrelated . Practically , the true ρ value is unknown , so an estimate ρ̂ is used instead . Per Section 3.1 , there are several methods to obtain the estimate ρ̂ with linear or predetermined nonlinear models , but none for NNs . 3 Related Work . 3.1 Adjusting for Autocorrelated Errors . Adjusting for autocorrelated errors in linear models has been studied extensively , especially in econometrics . Typically , after collecting the data and determining the formulation of the underlying system , the Durbin-Watson statistic [ 16 ] is calculated to detect the presence of first-order autocorrelated errors . If there is statistical evidence that first-order autocorrelated errors exist , then one of the following methods can be applied . The Cochrane-Orcutt method [ 13 ] is the most basic approach . It first estimates the remaining autocorrelation in the errors , then transforms the series to weaken the autocorrelation and fits OLS to the transformed series . The procedure can be done once or iteratively until convergence ; both versions have the same asymptotic behavior [ 21 ] , but the performance may differ with finite samples . During the transformation of the time series , the first sample is discarded — a large information loss when the sample size is small . The Prais-Winsten method [ 49 ] solves this issue by retaining the first sample with appropriate scaling . Finally , the Beach-Mackinnon method [ 4 ] formulates the exact likelihood function that incorporates not only the first sample but also an additional term that constrains the autocorrelation coefficient to be stationary . These methods update the autocorrelation coefficient starting from 0 . However , multiple local minima might exist and the procedure might converge to a bad local minimum [ 14 , 15 , 48 , 55 ] . Thus , the Hildreth-Lu method [ 29 ] grid-searches over autocorrelation coefficients and picks the best one . For autocorrelation of higher orders , the Ljung-Box test [ 39 ] or Breusch-Godfrey test [ 9 , 25 ] can be applied to detect their presence . Aforementioned methods can then be extended for adjustment [ 1 , 8 ] . When dealing with nonlinear data , as long as the underlying system is known , prior methods are applicable by changing OLS to nonlinear least squares [ 21 ] or employing other nonlinear optimization techniques such as BFGS [ 18 ] . However , it is often difficult to assume or formulate a correct and exact model structure for real-world time series data , especially when there are multiple series . This is where NNs come into play . Neural networks ( NNs ) [ 31 ] are inherently designed to learn arbitrary nonlinear relationships between input-target pairs from the data . In [ 32 ] , it has been proven that a nonlinear NN with sufficient number of hidden units is capable of approximating any function to any degree of accuracy . Pairing this with the exponentially increasing availability of data , computational power , and new algorithms , NNs can learn complex functionalities . Although NNs show promising outcomes , as of today , there is no publication we are aware of that is dedicated to adjusting autocorrelated errors in NNs for time series .

Summary: The authors consider the problem of controlling for autocorrelated errors in neural networks. This is accomplished through the introduction of a new training scheme that jointly learns the first-order autocorrelation coefficient alongside model parameters.  This is backed up by a lot of experiments that, particularly for time series forecasting, demonstrate significantly improved performance almost across the board by implementing the new training procedure.