Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper argues for handling auto-correlated errors in time series forecasting, in particular for neural network and proposes methods to adjust for such auto-correlated errors via adjusting standard MSE to incorporate auto-correlated errors. In extensive empirical studies, the authors show that their approach improves the accuracy of methods (and as a by-product show the general applicability).
main_review: While I agree that taking auto-correlated errors into account is an important topic that hasn't received the attention it deserves in the ML literature and the authors build a convincing case for their technique, I'm concerned with a number of aspects that I'll try to outline in the following:

l. 85: "MSE is equivalent to MLE." This is only true if we assume Gaussianity and is at best sloppy writing if taken isolatedly. However, It points to a larger issue which is an apparent unfamiliarity of the authors with the state of the art in probabilistic neural forecasting. For example, much of the modern neural forecasting literature is concerned with non-Gaussian errors since the first published neural forecasting models such as https://www.sciencedirect.com/science/article/pii/S0169207019301888, there are quantile regression approaches 
https://arxiv.org/pdf/1711.11053.pdf http://proceedings.mlr.press/v89/gasthaus19a/gasthaus19a.pdf How does the method presented here extend to such non-MSE-based approaches? This needs a discussion since so much of the state of the art is concerned with lifting Gaussianity assumptions.

l. 88: there is quite some recent work on multivariate probabilistic forecasting models with neural networks:
https://arxiv.org/abs/2002.06103
https://papers.nips.cc/paper/2019/hash/0b105cf1504c4e241fcc6d519ea962fb-Abstract.html
https://arxiv.org/abs/2101.12072
https://proceedings.neurips.cc/paper/2020/hash/1f47cef5e38c952f94c5d61726027439-Abstract.html
at the very least, this work should be mentioned (the above arxiv links are published at ICLR 20 & 21 to the best of my knowledge)

l. 90: no, at least not in contemporary neural probabilistic forecasting models. A lot of work is concerned with non-IID errors and non-normal errors (see pointers to literature in the rest of this review; and apart from these there is quite some work on using normalizing flows or GANs for complex forecasting distributions).

in the related work, I would have expected a discussion of deep state space models which should address many of the issues, e.g., 
https://papers.nips.cc/paper/2018/hash/5cf68969fb67aa6082363a6d4e6468e2-Abstract.html
https://proceedings.neurips.cc/paper/2020/hash/afb0b97df87090596ae7c503f60bb23f-Abstract.html and references therein. How would the techniques that the authors present here extend to these cases? 

in the empirical comparisons, I would have expected to see some popular neural forecasting models. NBEATS is one such model that is missing. To the best of my knowledge implementations of NBEATS and DeepAR and other models exist in open source. A vanilla LSTM is, in my opinion, a less useful baseline for the empirical comparisons.

The authors limit themselves to linear, first-order auto-correlated errors. It's not immediately clear to me why such a strong restriction would yield such high benefits and why these simple error structures couldn't be compensated for via other approaches, such as increasing the capacity of the neural networks. Also, it would be interesting to discuss the benefits of a more complex auto-correlation structure since we are already in a neural network context. E.g., couldn't equation (4) itself be a neural network whose parameters we could attempt to learn? 


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a strategy for accounting for autocorrelated errors in time series that can be used with a variety of existing neural nets. Essentially we learn a neural net’s parameters jointly with an additional autocorrelation coefficient parameter that is based on a first-order autocorrelation error approximation.
main_review: This paper cleverly combines insights from how autocorrelation error is dealt with in econometrics with neural nets for time series analysis. Overall, I found the paper to be easy to follow, and the proposed method to be a major ML advance (since it’s able to improve a vast number of existing neural net models, making them handle autocorrelated errors across time) albeit with pieces that are not fully justified/well-understood. I’ve listed strengths and weaknesses below.

Strengths:
- This paper tackles an extremely common problem in time series analysis tasks (that errors across time are very commonly correlated rather than independent)
- This paper combines insights from econometrics with recent neural net advances in a clever way
- The experimental results are quite strong and suggest that accounting for autocorrelation error in time series analysis typically improves prediction accuracy

Weaknesses:
- Some of the algorithmic edits (lines 177-187) to the naive procedure described in Section 4 come off as somewhat ad hoc; while they have some justification, precise details of what is happening theoretically/under the hood is a bit unclear to me (could the problem setup have been formulated in a way where these edits just naturally appear as opposed to being introduced as modifications?). Also, perhaps an ablation study of sorts could be used to explain how much each of the edits helps improve results.
- I don’t think the last statement on page 9 is true in that if a neural net is used for some application that has negative societal impact (and where higher accuracy turns out to be correlated to more negative societal impact), then the proposed method could potentially exacerbate the negative impact.
limitations_and_societal_impact: See the second bullet point in "weaknesses" from my main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this study, authors propose a loss function that learns the autocorrelation coefficient of time series data jointly with the model parameters to adjust the autocorrelated errors that may present in neural network-based time series forecasting algorithms.
main_review: Strengths

•	The motivation of the study is clear with a clear introduction to the three sources of error in machine learning.

•	Authors have used six deep learning-based architectures (covering all major neural network architectures) for time series forecasting to apply the proposed autocorrelation adjustment and evaluated these methods using multiple publicly available benchmark datasets. Authors have also shown the statistical significance of the results.

•	The correct use of relative and a scale-invariant error measure to evaluate datasets where sets of many time series are available

•	An additional ablation study conducted to further analyse the effect of adjusting the input and output for autocorrelated errors.

•	Additional experiments have conducted (Section 5.5) to justify that there exist residuals in the baseline model, and the proposed adjustment procedure reduce the remaining autocorrelation error. 

Weaknesses

•	If possible, in addition to the neural network-based benchmarks, it’s always a good idea to have some statistical time series forecasting methods to include in your experiments (ARIMA, Prophet, or TBATS)

limitations_and_societal_impact: Limitations of the proposed method and the Broader Societal Impacts have discussed
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors consider the problem of controlling for autocorrelated errors in neural networks. This is accomplished through the introduction of a new training scheme that jointly learns the first-order autocorrelation coefficient alongside model parameters.

This is backed up by a lot of experiments that, particularly for time series forecasting, demonstrate significantly improved performance almost across the board by implementing the new training procedure. 
main_review: ## Strengths
- A nice, simple approach to solve a (surprisingly) little-studied problem of dealing with autocorrelated errors in time series with NN models achieving excellent performance improvements. 
- The quality of experiments and performances is extremely good, in particular for time series regression problems. The joint training of $\rho$ with model parameters seems to offer clear benefits and the additional studies are good. 
- Paper is generally clear and well written. 

## Weaknesses
**Method explanation** I found the discussion on the method section to be rather confusing. This could just be me, but step 2 made me scratch my head as to what was going on. I assume you are minimising the residuals after adjusting for the autocorrelated errors, that is: 

minimise over $\theta, \sum \quad X_t - \rho X_{t-1} - (f_{\theta, t} - \rho f_{\theta, t-1})$

I may have this wrong, but either way, it is not clear in the method section what is going on. I think it could do with a rewrite and would add a lot to the paper.

**Inclusion of $\bar{X}$** It is stated in lines **190-191** that $X_{t - W - 1}$ is removed since this would include an extra timestep not used in the comparison model. However, this is replaced by the training mean, $\bar{X}$, why is this done rather than removal of the step? Surely the point of removing $X_{t - W - 1}$ was so we can have a fair comparison to the standard model using $W$ time steps, but now more information has been added through $\bar{X}$ that is not seen in the 'w/o' model slightly muddying the comparison. 

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: All reviewers are overall positive, the paper addresses a problem that is important, and the authors have provided thoughtful and useful replies to the reviewers.

The paper proposes an elegant way to deal with autocorrelated errors in time series usable with a variety of existing neural nets. The idea is to learn the net’s parameters jointly with an autocorrelation coefficient parameter. The paper combines insights from how autocorrelation error is dealt with in econometrics with deep learning for time series analysis. The method may be a major advance, since it can improve many existing neural time series models.
Decision: Accept (Poster)