Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to regularize the training of time-series forecasting models for better generalization performance. It extends the recently proposed flooding method which prevents the training loss from dropping to zero by regularizing it to stay above a fixed threshold. The authors argue that such fixed threshold is not amenable to time-series forecasting as it regularizes the average of all future time-steps and features. To address this issue, WaveBound is proposed which dynamically enforces the threshold for each time-step and feature per mini-batch. This is done by keeping an additional copy (called target) of the forecaster whose weights are updated with exponential moving average (EMA) of the source network. The source network is then regularized to be in the epsilon neighborhood of the loss predicted by the target network, thus dynamically regularizing the source network. Experiments have been presented on several datasets showing the improvement WaveBound-regularization offers for multiple forecasting models.
strengths_and_weaknesses: [Strengths]

- This paper presents a novel extension of the previously proposed flooding regularization to time-series forecasting models. I like the innovative idea of using a slow network to dynamically set the flooding threshold.
- The proposed idea is technically sound. Key claims of the paper have been supported by systematic experiments. Several benchmark models have been studied on several datasets showing that WaveBound-regularization improves the generalization performance across these models. Qualitative results also solidify authors’ claims. Ablation experiments have also been conducted to highlight the importance of dynamically regularizing the model.
- The paper is well-written and easy to understand. I have some suggestions that could improve the readability and make the main text more complete.
    - Consider moving the pseudocode to the main text.
    - Include a proof-sketch for Theorem 1. In the current version, the theorem appears abruptly and isn’t followed-up with any discussion. Please consider adding discussion around the theorem to stress its significance.
- With its wide applicability, the proposed method and results are significant for the time-series community.

[Weaknesses]

- Discussion of some important prior work is completely missing from the paper. In section 5, the authors discuss statistical approaches and deep-learning-based approaches but complete miss on the important class of state-space models (and their combination with neural networks). A non-exhaustive list of recent works combining SSMs with NNs for time-series forecasting follows.
    - Kurle et al. “Deep Rao-blackwellised particle filters for time series forecasting.” In NeurIPS, 2020.
    - Rangapuram et al. “Deep state space models for time series forecasting.” NeurIPS, 2018.
    - de Bézenac et al. ”Normalizing kalman filters for multivariate time series analysis.” In NeurIPS, 2020.
    - Ansari et al. "Deep Explicit Duration Switching Models for Time Series." *In NeurIPS, 2021*.
- The proposed method is focused on a particular type (MSE) of loss function. Is the method applicable to other types of loss functions, e.g., those for generative models listed above?
- The method introduces additional training overhead as it requires another copy of the network. This weakness has also been acknowledged by the authors.
questions: - How was alpha, epsilon chosen?
- Do the authors have any comments on the applicability of the method to loss functions other than the MSE loss?
limitations: Limitations have been discussed by the authors in the appendix. While the method does introduce additional training overhead, there is no overhead at test time and the method improves the generalization performance significantly.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an effective regularization method called WaveBound for time series forecasting, targeting on the overfitting problem of deep networks. WaveBound uses two networks throughout the training phase. The first network provides lower bounds of errors for the second network, which improves the generalization ability of the second network. Experimental results show that WaveBound improves the performance of several time series forecasting models.
strengths_and_weaknesses: Strength:

1. This work adapts a regularization technique called flooding to TSF tasks, and it improves the generalization ability of existing time-series forecasting models. 

2. The proposed method seems to be simple yet effective.

Weakness:

1. The overfitting problem of existing TSF models is not well motivated. It is until the experimental section wherein Figures 3 and 4 show the training-testing error curves of some models used in the paper. It is essential to provide sound justifications that overfitting does exist in TFF models first. 

2. In time-series forecasting, a large generalization loss can also come from the distribution shift between training data and test data.  The Reversible Instance Norm (RIN) technique proposed in ICLR'22 is a simple way to mitigate this issue. Since both RIN and WaveBound try to reduce generalization errors, please compare them. Also, can these two methods be used at the same time and how would they perform together?

3. The evaluations are performed on long-term forecasting tasks only. As the long-term forecasting problem itself is not well studied yet, it weakens the effectiveness of the proposed solution. Consequently, it is essential to evaluate WaveBound in some mature experimental settings, i.e, well-developed models for short-term forecasting tasks. 

4. The proposed method heavily relies on the EMA technique, but no ablation study is performed from this aspect. 
questions: 1. Is overfitting a real issue for the time-series forecasting task? Can WaveBound work properly on short-term forecasting tasks? Please consider evaluating WaveBound on short forecasting tasks, i.e, exchange rate, ECL, traffic used in the paper but forecast for 3, 6, 12 steps. Some spatial-temporal time-series forecasting tasks are also quite popular, e.g., PEMS08, 07, 03, 04. These experimental settings are well studied in the literature, and their corresponding SOTA models are more mature. If WaveBound works on these tasks, the soundness of this paper will be largely improved. [[1](https://dl.acm.org/doi/pdf/10.1145/3394486.3403118), [2](https://arxiv.org/pdf/2106.09305.pdf), [3](https://arxiv.org/pdf/2205.08689.pdf)]

2. What is the benefit of using the source network to learn the lower bound defined by the target network? EMA model is proven to have a positive impact on test-time performance in other areas, such as semi-supervised learning. What if we only EMA without flooding? Please conduct an ablation study. 

3. How is WaveBound compared with RIN? Can we use these two methods together and achieve better results? 

4. Simple models are also effective for the long-term forecasting task. Is WaveBound effective on small-scale models, e.g., MLP? 

5. Since WaveBound runs two models at the same time, does WaveBound largely increase the computational and memory cost? 
limitations: The computational efficiency of WaveBound is not discussed in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposed WaveBound, a new regularization method to prevent unstable training and over-fitting for time series forecasting problems by encouraging a non-zero training loss. Compared with the original flooding bound, WaveBound consider the error bound for each individual feature and step. Besides, the error bound dynamically adjusts with the help of a target network. The paper show the efficacy of the method by extensive experiments.
strengths_and_weaknesses: Strongness: 

1. The paper introduces a new way to regularize the error terms individually. The error bound is automatically customized.
2. The experiments show the improvement of this regularization methods over many SOTA models.

Weakness
1. I’m confused about the inspiration to allow individual flooding for each time step and feature. It is understandable that time series forecasting shall have different learning difficulty/error patterns for each output element. However, in the testing phase, using either MAE or MSE does not bias the error and metric is fair for all the elements. Is that possible the target network decreases the learning difficulty for certain locations, therefore for certain element, both target and source network produces bad prediction, therefore leading to overall larger testing error? By intuition, if one feature receives larger error (for instance less predictable), it’s natural to give more “bias” to that feature in optimization which is reflected from the loss function. For tasks like image classification, self-supervised way is fine because individual probability is bounded. In regression tasks using MAE or MSE error, the error magnitude could vary a lot among different features, so it seems counter-intuitive for me to massively balance the “bias” in training. I hope the author could elaborate this in the paper.



questions: 1. In table 3, it shows WaveBound outperforms both constant flooding and original flooding. However, it seems like constant flooding along is not improving. I’m curious how will the model perform if you delete the individual error feature from the WaveBound, say using a loss function similar to original flooding but b term is from a target network. Also in Fig 4, when you have a univariate problem and small prediction length, the wavebound still performs better than the baseline.

2. In Fig 3b, there’s a jump of testing MSE for WaveBound at the beginning of training? Do you have any explanation for that?
limitations: there's no potential negative societal impact
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a regularization method for time series modeling aimed at preventing overfitting in deep learning methods. This method extends flooding regularization, which uses a constant target training risk, to use an exponential moving average of past models, yielding a level of training risk that is adaptive. The authors demonstrate that their improved formulation leads to better performance when using SOTA models (Autoformer, Pyraformer) on popular benchmarks datasets in the field (ett*, electricity, exchange).
strengths_and_weaknesses: Strengths:
- The presentation is generally pretty clear and I found the paper easy to read
- The evaluation shows that the method yields consistent improvements over the chosen baselines
- The proposed method seems to be quite general purpose, and could probably be applied to other domains. The method essentially seems like an extension of vanilla flooding to sequence-to-sequence models. 

Weaknesses:
- Section 3.2 could be better motivated. It’s not immediately obvious why bounding the risk to be similar for g and g* is desirable a priori or why it might lead to similar outcomes as vanilla flooding. Theorem 1, especially, should be motivated more, as it’s not clear what it implies practically. Do we just need to specify an appropriately sized epsilon and then suddenly we obtain lower MSEs across the board? Why wouldn’t we apply this to essentially every instance of ERM if there’s a MSE reduction? Clearly the conditions for the theorem to hold entail trade-offs that are not made obvious. 
- It’s not entirely clear to me that this method works better than other obvious regularization methods. For example, in the plots shown in Figure 3 it seems like early stopping would be almost as effective as flooding for preventing overfitting and obtaining good test error. Moreover, it’s not clear to me that the proposed flooding method wouldn’t start overfitting as well if you continued to run it for more epochs. The EMA slow network prevents the loss from quickly going to zero, but eventually the EMA network can also begin overfitting. 
- In general if the essential claim behind the paper is that overfitting is a major source of error in current deep learning methods for time series, it would be beneficial to investigate and support this claim more directly. For example you could examine how this phenomenon might change with dataset size or signal-to-noise ratio. Beyond proposing a specific technique, this analysis would be incredibly useful to the community, as deep learning still seems to struggle on time series and there’s no real consensus around what the most significant roadblocks are. 

questions: I’ve already enumerated some questions in the above section, but I include a few more questions/comments below:

It’s not clear to me why the design of the approach is actually tailored for time series data. You note that (1) vanilla flooding doesn’t deal well with inputs that have temporal and channel dimensions, and (2) vanilla flooding doesn’t gracefully handle inputs with lower signal-to-noise. (1) seems general to any sequence-to-sequence task, and I’m not sure if the method actually addresses (2) directly. If as practitioners we are worried about irreducible noise, why is the proposed adaptive method favorable over a fixed level of risk? The slow EMA network is also capable of overfitting to random noise. 

limitations: The paper suggests that stochasticity is a source of overfitting in applications of deep learning to time series, but the proposed method doesn’t really deal with stochasticity explicitly, which I think is potentially a major shortcoming. In many of the benchmark datasets used for evaluation, the signal-to-noise ratio is quite small (e.g. exchange rates data) and any deterministic method is going to struggle to fit the data. In some sense the objective is simply misspecified. Beyond simply trying to limit overfitting to this noise with regularization, explicitly modeling the stochasticity can probably be very helpful.
review_title: null

Meta-review: metareview: The reviewers highlight the novelty of the method, the clarity of writing, and the consistent performance improvements over baselines. Initial concerns by the reviewers related to missing related work, missing empirical evaluation within the more mature short-term forecasting experimental paradigm, and missing empirical comparisons (comparing to reversible instance norm, isolation of the effect of EMA) were addressed by the authors during the discussion period. Some concerns around the overall motivation behind the proposed approach remain, but are outweighed by the empirical effectiveness.
confidence: Less certain
Decision: Accept