Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors propose the tilDEq loss function that combines phase, amplitude, and shift sub-losses and claims to outperform MSE. The paper needs to improve its motivation and experiments.
strength_and_weaknesses: Strengths
L-norm minimization for machine learning and in particular forecasting, is a very important topic.
The authors identify some challenges of L-norm minimization within the forecasting task.

Weaknesses
1. Experimentation is not extensive; the authors are cutting by half the number of datasets from AutoFormer (Wu, 2021) experiments that used ETT, ECL, Exchange, Traffic, and Weather. Is it possible for only the datasets that align with the paper's research to be reported? Please report the results on all the missing datasets.
2. Motivation of the paper is lacking, see comments of Figure 1.
clarity,_quality,_novelty_and_reproducibility: Clarity
Figure 1. needs to improve; the chosen example signal can be easily forecasted because it has evident seasonal patterns. The Figure suggests to the reader that MSE/normality forecasting assumption is incapable of accurate predictions for such a scenario; this intuition goes against a century of work on the topic. Please refer to ARIMA/ETS, and also consider changing the example signal; it gives the impression of creating a research problem that does not exist.

Novelty
Paper's novelty is limited. Phase, amplitude and shift subcomponents have been extensively studied.
There are clear connections between the sub-losses and MSE please see:
https://en.wikipedia.org/wiki/Parseval%27s_theorem

Reproducibility
Given the quality of the experiments, reproducibility would be a second consideration, I would suggest to focus on improving them.

summary_of_the_review: Strong reject.
The importance of the study of loss functions tailored for the forecasting task starkly contrasts with this submission's quality.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work studied distortion handling in timer series forecasting problems. In particular, a novel loss function is proposed to guide models to consider amplitude, phase, and uniform amplification shifting invariance. 
strength_and_weaknesses: Strengths
1. The related work is comprehensive and shows a thorough analysis of different time series distortions.
2. The problem studied in this paper is important for the applications related to time series forecasting.
3. The paper is well-organized and easy to read.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, with high quality and high novelty. Furthermore, it could bring great value to the applications of time series forecasting. 
summary_of_the_review: This paper is well-written and provides a high-level of readability to most readers who work in the field of time series forecasting. The authors thoroughly explain why point-wise metrics could mislead the model and how models could benefit from learning shape-aware loss functions. The experiment shows the significant improvement the proposed loss function could bring, and a detailed experiment analysis is provided.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new loss function for time-series called TILDE-Q, which has 3 components. Each loss component is invariant to different distortions, including amplitude shifting, phase shifting, and uniform amplification. The paper starts with formal definitions of common time-series distortions and then designs 3 loss components that are invariant to 3 types of distortion. The experiment results suggest that the new loss function is model-agnostic. It can improve the accuracy of different existing time-series models, even on common existing metrics such as MSE or DTW.


strength_and_weaknesses: Strength:
  - The paper is self-contained. It provides a systematic study of different time-series distortions and builds up invariant loss components from there.
  - The paper is well-presented, with clear notations and figures. The appendix is also very useful with detailed experiment setup and qualitative results.
  - The new loss function yields significant accuracy improvements across different time-series models on various datasets.

Weaknesses:
  - The paper does not study the loss components' weights $\alpha$ and $\gamma$. How sensitive are those hyper-params? What happens if we change them, qualitatively and quantitively?
  - TILDE-Q itself is not invariant to any of the amplitude shifting, phase shifting, and uniform amplification distortions.
  - Maybe it is worth studying what happens if we add MSE to TILDE-Q loss.
  - The paper claims that "time-series forecasting models should aim to both accurately forecast the value for each time-step and the prediction should have similar shapes as those in original time-series". This is not necessarily true. In practice, time-series forecasts deliver value by allowing future planning. Therefore, the cost of a specific error or distortion in the forecast depends strongly on the kind of planning we use the forecast for.
clarity,_quality,_novelty_and_reproducibility: Even though the paper borrows some key ideas from prior works, such as normalized cross-correlation function or dominant Fourier coefficient, the TILDE-Q loss function construction is novel, as far as I can tell.
summary_of_the_review: The paper gives a good systematic review of common time-series distortions. The paper is well-presented with good support materials. The designs of the 3 loss components are elegant and effective. The experiment results are strong, which shows that TILDE-Q, the weighted sum of the 3 loss components, can improve the accuracy of different existing time-series models, even on common existing metrics such as MSE or DTW.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates distortions in amplitude and phase to understand shape-awareness and distortion invariances in time-series forecasting. This paper uses TILDE-Q that has invariances to many existing distortions and achieves shape-awareness and informative forecasting in a timely manner. Experimental results show the efficacy of proposed method.
strength_and_weaknesses: Strengths:

1. It's novel to investigates distortions in amplitude and phase to understand shape-awareness and distortion invariances
2. The formulation on time series distortions in transformation perspective is novel and interesting. 

Weaknesses:

1. The experiments are not compared with the state-of-the art methods, such as FedFormer, NHITS and etc.
2. It's not sure that whether this method is general enough for other Transformer-based models, such as FedFormer and etc.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The method is novel and looks reproducible.
summary_of_the_review: This paper investigates distortions in amplitude and phase to understand shape-awareness and distortion invariances in time-series forecasting. This paper uses TILDE-Q that has invariances to many existing distortions and achieves shape-awareness and informative forecasting in a timely manner. Experimental results show the efficacy of proposed method.

However,  the experiments are not compared with the state-of-the art methods, such as FedFormer, NHITS and etc. Also it's not sure that whether this method is general enough for other Transformer-based models, such as FedFormer and etc.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a new loss function (transformation invariant loss function with distance equilibrium) for time series forecasting. This new loss function helps capture the shape of time-series sequences. The authors tested the effectiveness of the new loss function on multiple datasets and models.
strength_and_weaknesses: Strength:
1. The authors have done a thorough literature review on the SOTA methods, models, and metrics on time series forecasting.
2. The authors also have experimental results to compare the proposed loss function and the other two metrics with 4 SOTA models on 5 datasets.

Weaknesses:
1. 'The output Y_hat should have similar temporal dynamics with Y.'
This is a strong assumption without much justification. It will be helpful to elaborate on why temporal dynamics are important and give examples of how they benefit time series forecasting in reality.
2. The definition of temporal dynamics, and distortion is quite fuzzy. It is necessary to give mathematical definitions for these two concepts.
3. The authors criticized that DILATE handles time scaling with DTW and phase shifting with TDI but did not explain the link between DILATE and the new loss function. It will be good to have a separate section to clearly explain the relationship between DILATE and TILDE-Q. Is it beneficial to merge these metrics into one?
4. The Fourier transform is one of the classical ways to analyze time series data and the authors used the Fourier coefficients for phase shifting. However, it is not clear how to define dominant frequency.
6. Ablation study is not complete.
It is not clear how the authors tuned hyperparameters alpha and gamma to generate results in Tables 1-4 which is critical to justify the effectiveness of the new metric. Thoroughly tuning these parameters can help get better results than roughly tuning compared loss functions like DILATE. It is encouraging that the authors showed some results of tuning alpha in the Appendix but still no result w.r.t. gamma is available.
clarity,_quality,_novelty_and_reproducibility: 1. The writing needs to be improved significantly to better present the work.
2. The authors did not give enough details to reproduce all results in this work.
summary_of_the_review: The paper assembles too many strong assumptions without very good justification. It is a clear rejection.

-----------
I have read the authors' responses and would like to keep the same score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission has been assessed by five knowledgeable reviewers. Two of them rated this paper as acceptable, one as marginally acceptable, while the other two reviewers voted for rejection (one reject and one strong reject). The supportive comments highlight clarity of presentation and good empirical performance on the benchmark data selected. The negative comments bring up several limitations, including limited choice of benchmark data, limited novelty in general (amplitude, phase and shift based loss components have been extensively studied in time series forecasting literature), missing or incomplete ablation study leaving questions of sensitivity of the proposed loss metric to hyperparameter value choices unclear, some assumptions being made without sufficient justification, and somewhat unclear motivation (reviewers suspect that only a subset of applications of time series forecasting may benefit from or truly focus on high fidelity reconstruction of the shape of the forecast waveforms, but the authors do not clarify this cornerstone aspect of the practical utility of their method). In summary, this work is interesting and has good potential. But at this point, its limitations outweigh its advantages, and as such it ranks slightly below the acceptance threshold for ICLR.  
justification_for_why_not_higher_score: Limitations of the work as currently presented, highlighted by the reviewers, prevent this paper from being a clear accept for ICLR 2023. 
justification_for_why_not_lower_score: n/a
Decision: Reject