Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents exponential smoothing transformers for long-range time series forecasting. The key idea is to leverage a level-growth-seasonality decomposed transformer architecture and employ both exponential smoothing attention and frequency attention to reduce computational complexity. The experiment results showed the effectiveness of the proposed method.
strength_and_weaknesses: Strength:

1. This paper is well-written and organized.
2. Several transformer-based baselines have been used in empirical studies.

Weaknesses:

1. Combining exponential smoothing with neural networks to perform time series forecasting is not a new idea, for example, 

[1] A hybrid method of Exponential Smoothing and Recurrent Neural Networks for time series forecasting International Journal of Forecasting 2019

[2] A Hybrid Residual Dilated LSTM and Exponential Smoothing Model for Midterm Electric Load Forecasting. IEEE TNNLS 2021

[3] Time‑series analysis with smoothed Convolutional Neural Network

Especially In [2], the authors have thoroughly assessed various approaches (including LSTM, GNNs)+ ETS for time series forecasting. However, none of them were mentioned or compared in this paper. Also, applying ETS to the transformer seems straightforward to me and thus the technical novelty here is limited.

2. FEDformer has already shown that properly considering frequency-based attention in the transformer is useful for long-range forecasting. In the paper, I did not observe how the proposed FA is superior to FEDformer.

3. Overall, I feel this paper is a little bit add-hoc, i.e., combining the benefit of ETS (which has been validated in earlier works) and FA (similar ideas have been validated in FEDformer).  It is not clear which component contributes more to the performance boost.

4. The experiment results do not show the obvious superior performance of the proposed ETSformer over FEDformer and other baselines.

5. The discussion of the results in Table 1 is limited, especially for the cases when ETSformer cannot outperform baselines.

clarity,_quality,_novelty_and_reproducibility: The overall clarify is OK. However, the technical quality and novelty is limited. Reproducibility is OK since code is available.

summary_of_the_review: Overall, I have concerns over the novelty, related works, and experiment results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel exponential smoothing at- tention and frequency attention to replace the self-attention mechanism in vanilla Transformers.  They also redesign the Transformer architecture with modular decomposition blocks such that it can learn to decompose the time-series data into interpretable time- series components such as level, growth and seasonality. Experimental results show the efficacy of proposed method.
strength_and_weaknesses: Strengths:
1. It's novel to combine traditional exponential smoothing method with Transformer based model. Also it replaces the self-attention in vanilla Transformer with their designed frequency attention mechanism.
2. Exponential Smoothing Attention and Frequency Attention Mechanism are derived in detail.
3. Qualitative and quantitative experiments demonstrate the efficacy of proposed method.

Weaknesses:
1. Previously exponential smoothing has already been integrated with LSTM. The paper doesn't show the comparison of the proposed method with the integration with LSTM.
2. It's not clear that whether this special Exponential Smoothing Attention and Frequency Attention Mechanism could be integrated with other Transformer based model, e.g. Autoformer, FedFormer and etc.
3. The comparison is out-of-dated. It doesn't compare with FedFormer and NHITS.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The method novelty is marginal. It looks reproducible.
summary_of_the_review: This paper proposes a novel exponential smoothing at- tention and frequency attention to replace the self-attention mechanism in vanilla Transformers.  They also redesign the Transformer architecture with modular decomposition blocks such that it can learn to decompose the time-series data into interpretable time- series components such as level, growth and seasonality. Experimental results show the efficacy of proposed method.

However,  It's not clear that whether this special Exponential Smoothing Attention and Frequency Attention Mechanism could be integrated with other Transformer based model, e.g. Autoformer, FedFormer and etc. Also the integration of Exponential smoothing with some backbone neural network is not novel. (previously Uber did the integration with LSTM). It doesn't show the experimental results difference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a transformer architecture called ETSformer that leverage the advantage of the exponential smoothing method for more accurate and interpretable forecasts. The core contributions are the development of exponential smoothing attention (ESA) and frequency attention (FA) mechanisms that model the growth and seasonality information of time series data. These methods build connections with classical time series models and perform forecasts for each individual component of time series data, leading to better point forecast accuracy and interpretability. The overall transformer architecture is clear and intuitive. Empirical results have shown improved forecasting accuracy than other transformer-based forecasting baselines, and have shown better computational efficiency and interpretability. 
strength_and_weaknesses: Strength:
1. The novelty of this paper is clear and well-motivated. The exponential smoothing method is widely used in time series literature and the underlying supports of ETSformer, Holt-Winter's method, and its modifications are also theoretically sound.
2. To obtain the attention matrix, this paper uses an efficient $\mathcal{A}_{ES}$ that computes the attention in $\mathcal{O}(L^2)$ complexity, which reduces the computational time compared with baselines.
3. The empirical evaluation and ablation study are thorough and sound, providing good support for the proposed method.

Weakness:
1. Exponential smoothing also has some limitations in time series forecasting, such as (1) the forecasts it generates will be behind, (2) difficult to account for dynamic changes in the real world, and (3) constantly requires updating to respond to new information. The authors need to discuss how the proposed ETSformer can overcome these drawbacks compared with classical methods.
2. I am not sure what is the benefit of having $N$ layers/stacks together in the encoder/decoder. It seems that they are not time stamps. To better explain the intuition, it is good to have an e.g., layer-wise analysis to visualize/quantitatively show the difference of the information captured at each layer.
3. Although ETSformer is more efficient, it appears the improvement over FEDformer is not much.

Minor Questions:
1. Are the empirical evaluations single-step or multi-step forecasts?
2. In the first paragraph of section 3, what does the "dimension" mean? Is this a temporal dimension or a feature dimension?
clarity,_quality,_novelty_and_reproducibility: This paper is well written. The idea is novel and the contributions are clear. The authors have also included the code for reproducing the results.
summary_of_the_review: Overall, this paper provides a novel modification for the transformer architecture that can improve both the accuracy and interpretabilty of time series forecasts. The proposed method is proven to work well in multiple real datasets. As an improvement, the authors can better clarify how the proposed method can overcome classical exponential smoothing drawbacks and a more intuitive explanation of the model architectures.

**Update after rebuttal**

The authors have addressed my concerns and have demonstrated them with additional experiments. Based on the response and comments from other reviewers, I tend to keep my current evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel deep neural network architecture for multi-horizon time series forecasting, ETSformer, via improved representation learning of level-growth-seasonality relationships commonly found in time series datasets. While the paper refers to these as attention mechanisms, it in fact models features in each layer using tweaked exponential smoothing dynamics – with seasonal components isolated using the top-k Fourier bases, and broader trends extracted using level/growth models on intermediate features.
strength_and_weaknesses: While the model could be promising for time-series datasets where level-growth-seasonality relationships dominate, I do have several key questions:

1) Is the ETSformer only performant on time series datasets with level/trend/seasonality relationships?

A key benefit of neural forecasting models is to learn temporal dynamics in a purely data-driven way – without the users having to make any assumptions on the relationships present in the dataset. While exponential smoothing dynamics are commonly found in many time series dataset, many other types of behaviours exist (e.g. chaotic time series, event-driven behaviour, holiday seasonality, regime-switching dynamics etc.). The high-degree specialisation of the representations within the ETSformer hence raises the question is able to learn general time series relationships, or if it is confined to variants of exponential smoothing. 

The model could still be useful if this were the case, but multiple comparisons would need to be made against specialised models that capture these relationships – namely simple benchmarks such as the Holts-Winter’s additive model and structural time series models, and Neural forecasters ES-RNN (winner of M4 competition) and N-Beats which also make similar decompositions.

2) How useful is the ETSformer vs common standardisations from time-series datasets?

Another key claim the paper makes is that the model does away with the need for feature engineering on the inputs. Given that some standardisation appears to have been performed on the inputs (could the authors clarify specifics?), I wonder if the performance of other transformers would be improved using standardisations more appropriate for time-series datasets – e.g. simple log or Box-Cox transforms. In addition, the ETSformer should also be compared to other networks with implicit scale handling – e.g. DeepAR which has an input scaling layer.

3) How are benchmarks calibrated?

Has hyperparameter optimisation been performed on the other comparable models, and how are hyperparameters selected otherwise? 

4) Can the authors comment on how the frequency attention mechanism differs from that used by the Fedformer, and on the statistical significance of improvements vs it?

5) Is the ETSformer truly interpretable, or does it presume that the level/trend/seasonality relationships already exist?

I was a bit confused by the ESA attention plots – don’t the fixed “attention weights” pre-define the patterns to be found? In addition, are “FA weights” by definition periodic given that they are obtained by passing a limited number of frequencies through the inverse DFT? Given that none of these are learnt from the data, I am not sure that we can consider the model interpretable. Finally can the author comment on how the Real-world decomposed forecasts were obtained in the appendix? It’s not immediately clear how the components can be cleanly separated if multiple layers/stacks are present in the network.


clarity,_quality,_novelty_and_reproducibility: Clarity on the structure of the model and how interpretability works could be further improved -- see above section for more.
summary_of_the_review: Per the section above, I do have many key concerns regarding the paper, which would require clarification before it can be recommended for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: It is useful to develop time series forecasting models that are more interpretable or decomposable, especially when the models are deep learning models. It seems novel to combine the traditional exponential smoothing method with transformer-based models in particular, although exponential smoothing was combined with other neural network models before. This is a natural next step which has already happened to many other deep learning models. The attention mechanism of the proposed model is more efficient than other baselines. However, the technical novelty of this method is not very significant. Moreover, the proposed model can only handle level, trend and seasonality behaviors. When other behaviors co-exist with these (which is not uncommon in practice), can the proposed method still work well for these behaviors? Unless there is theoretical justification, extensive experimentation will be needed to demonstrate the robustness of the model even when other behaviors exist. While we generally agree that this work has merits, it is hard to champion acceptance of the paper, even in its revised form. We hope the authors will make substantial revision of their paper for future resubmission.

justification_for_why_not_higher_score: This paper has potential to cross the acceptance bar after substantial revision. Currently it is not up to the standard of ICLR.

justification_for_why_not_lower_score: N/A
Decision: Reject