Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a Transformer-based framework that iteratively forecast time series at different scales with shared weights. In particular, it proposes normalizing the downsampled time series to avoid distribution shift and using adaptive loss to deal with outliers. Experiments show the multi-scale framework can significantly improve the performance of various transformer-based forecasting models.
strength_and_weaknesses: Strengths:
+ The paper is well-organized and the essential ideas are easy to follow.
+ Extensive experiments are conducted with impressive improvement in main results.

Weakness
- The introduction of multi-scale structure is not well motivated. It would be nicer if a good motivating example of using multi-scale forecasting is provided other than the ablation study.
- The reason of using average only in cross-scale normalization is not well justified. Why would it be able to address distribution shift without standardization as well?
- The adaptive loss is introduced for outliers but there is no such demonstration. And the improvement of adaptive loss seems marginal in the ablation study (figure 4)
clarity,_quality,_novelty_and_reproducibility: The methodology is presented clearly and the empirical studies are thorough, but the motivation is not quite clear and the novelty is marginal. The work is expected to be easy to be reproduce as source code is provided.
summary_of_the_review: Overall I believe this paper make incremental contribution to forecasting community with a multi-scale framework. The empirical results are inspiring, however, for future research.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a general multi-scale framework for time series forecasting.
strength_and_weaknesses: Strengths
1. SOTA methods are compared in the paper. 
2. Comprehensive experiments are conducted.

Weaknesses:
1. Technical contribution is incremental.
clarity,_quality,_novelty_and_reproducibility: The technical contribution of this paper is incremental. Detailed implementation details are shared. The overall presentation could be improved with more clarification on the model description and training process. 
summary_of_the_review: In this paper, the authors proposed a  multi-scale framework that can apply directly to the existing time series forecasting framework. Detailed comments are listed below:
1. The technical contribution of this paper is a little bit incremental. The idea of leveraging multiple time resolutions of time series is not something new. The major contribution comes from cross-scale normalization.
2. How is the model trained with learnable parameters in the loss function? Also, how are alpha and c learned during the training?
3. Based on Figure 4, it seems that the adaptive loss doesn't have significant benefits until the multi-scale framework is used. Can the authors provide a move analysis and discussion?
4. It's observed that, in many cases (Tables 2 and 3), a single-scale model with mean normalization performs significantly better than the multi-scale version without normalization. Does that mean a simple mean normalization is more important than the multi-scale information in the time series? Can the authors provide more insight analysis?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a transformer-based framework for time-series forecasting for multi-scale frequency components. The solution is inspired by two recent state-of-the-art transformer architectures used as backbones by iteratively refining a forecasted time series at increasingly fine-grained scales and introducing a normalization scheme that minimizes distribution shifts between scales.
strength_and_weaknesses: Strengths:
* The novelty of the paper is interesting,
* The idea of time-series forecasting for multiple scales, by minimizing the distribution shifts between scales and windows using cross-scale normalization, is an interesting contribution to the different domains: Neuroscience, Stock Market, Weather Forecasting, etc.
* The paper is well written and is very thorough with the problem formulations (Both Architecture and its components are well explained with mathematical proofs).

Weaknesses:
* The Informer-MSA results are much closer to the actual time series compared to other methods in Figure 5; however, Fedformer and Autoformer display lower MSE and MAE in Tables 2 and 3. It is not clear to me why these models are performing poorly in the qualitative analysis compared to Informer-MSA.
* Figure 4 requires standard error bars for better comparison across the baseline models. Also, the authors could perform a statistical significance test to compare the results. 
* While the authors focused more on comparing the model performances, whether authors have tried if we have frozen the layer of the Transformer and only train at the final layer? How do the frozen model effects approximate dynamic time-series forecasting?
* It will be interesting if authors can present the layer-wise representations across the models and how these models learn the temporal aspect while forecasting the time series.

clarity,_quality,_novelty_and_reproducibility: * The paper is well written and is very thorough with the problem formulations.
* Most of the experiments are performed on standard time-series datasets. The framework needs to be validated on datasets like samples with chaotic time series, seasonality, and the time-series changes abruptly based on external factors.
* Since the authors released the Github code; hence all the experiments are reproducible.
summary_of_the_review: Overall, the authors propose a novel framework Transformer-based model for time series forecasting for multiple scales. However, the paper requires much discussion on why one model is superior to the other and interprets the layer-wise representations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduced a multi-scale forecasting training module to improve the forecasting performance of existing transformer models. The basic idea is laying on training scaling-shift aware fine-grind forecasting model. To achieve this goal, the authors introduce an across scale normalization module and a multi-resolution forecasting training structure. The result shows that the proposed module could improve the forecasting performance with various transformer deep learning models without adding additional burdens.

strength_and_weaknesses: Strength:
1. the paper is well written and the technique is simple yet intuitive
2. the contribution can contribute any transformer based forecasting model
3. Extensive experiments are conducted to justify the claim

Weakness
1. Introducing Adaptive loss seems unrelated to the major contribution of this paper
2. The multiscale forecasting is so-what similar to U-Net structure
clarity,_quality,_novelty_and_reproducibility: 1. My major concern is the similarity between the proposed multi-scale forecasting structure vs. the existing U-Net based forecasting approach (e.g. [1]) which is not cited. The technique differents between the proposed method and such types of approaches should be discussed.

2. Introducing adaptive loss seems unrelated to the major contribution of this paper. I personally think Table 6 in supplemental material is more important to justify the paper's claim.

3.  Eq. 7 and 8 normalize the mean. Why not normalize the standard deviation? Any reason?

4. Why solely using the proposed normalization cannot increase the performance of Autoformer? Any reason?

[1] Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting, ECML-PKDD




summary_of_the_review: Overall, this is an interesting paper, and the proposed module can be widely used in existing time series forecasting framework. Therefore, I believe this is an accept paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Deep learning models such as recurrent neural networks have been applied to time series forecasting. With the surge of research interests in transformer-based methods in recent years, it is a natural next step to apply them to time series forecasting as well. Nevertheless, a number of subtle issues have to be addressed. Instead of proposing yet another model, this paper proposes a multi-scale framework that can be applied to existing transformer-based time series forecasting models. Good results are obtained in the experiments. In addition, since it does not introduce much computational demand, we are convinced that the proposed framework will see adoption by practitioners to further enhance the performance of their preferred choice of SOTA transformer-based models. Nevertheless, the paper in its original form does have much room for improvement. We thank the authors for responding to our comments during the discussion period. We believe many of the concerns, such as those related to the motivation and presentation of the paper, can be addressed before publication. We highly recommend the authors to significantly revise their paper by incorporating the clarifications and changes made during the discussion period and also making further changes as deemed necessary.

justification_for_why_not_higher_score: Unless there is a mechanism in the review process that allows us to review the revised paper before publication, I don’t feel comfortable of recommending for spotlight acceptance.

justification_for_why_not_lower_score: There is one vote for rejection (5), but in fact the reviewer also praises this work as indicated in the strengths. The weaknesses are more related to presentation and further elaboration which can be fixed in the revision. In other words, they are not fatal.
Decision: Accept: poster