FreDF: Learning to Forecast in the Frequency Domain

Authors: Hao Wang, Lichen Pan, Yuan Shen, Zhichao Chen, Degui Yang, Yifei Yang, Sen Zhang, Xinggao Liu, Haoxuan Li, Dacheng Tao
Primary area: learning on time series and dynamical systems
Keywords: time series, long-term forecast

Abstract:
Time series modeling presents unique challenges due to autocorrelation in both historical data and future sequences. While current research predominantly addresses autocorrelation within historical data, the correlations among future labels are often overlooked. Specifically, modern forecasting models primarily adhere to the Direct Forecast (DF) paradigm, generating multi-step forecasts independently and disregarding label correlations over time. In this work, we demonstrate that the learning objective of DF is biased in the presence of label correlation. To address this issue, we propose the Frequency-enhanced Direct Forecast (FreDF), which mitigates label correlation by learning to forecast in the frequency domain, thereby reducing estimation bias. Our experiments show that FreDF significantly outperforms existing state-of-the-art methods and is compatible with a variety of forecast models. Code is available at https://github.com/Master-PLC/FreDF.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730448575188

thread_type:
official_review

<thread_object>
creator: reviewer_6t6t

<content>
summary:
This paper proposes to measure the distance between time series prediction outputs and target signals in the frequency domain. 
As claimed, this allows to capture label autocorrelation in time series prediction tasks. Speciafically, in practice, distances in both time and frequency domains are utilized for training in a combined manner.

strengths:
The presentation of the paper is clear and the idea is straightforward.

weaknesses:
The proposed approach lacks novelty. Measuring time series structure (e.g., autocorrelation) in the frequency domain has been well-explored. Several prior works have investigated capturing time series characteristics or enhancing robustness in loss functions, including:
  1. Incorporating Fourier transformation in loss functions [1,2].
  1. Employing DTW-based loss to keep shape information of time series [3,4].
   1. Utilizing multiresolution trends during training [5,6].

These approaches are all relevant to this study. However, this paper does not adequately investigate, discuss, or compare these related methods.

[1] Henning Lange, et al. From Fourier to Koopman: Spectral Methods for Long-term Time Series Prediction. JMLR 2021.

[2] Xinyu Yuan and Yan Qiao. Diffusion-TS: Interpretable Diffusion for General Time Series Generation. In ICLR 2024

[3] Vincent Le Guen and Nicolas Thome. Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models. In NeurIPS 2019.

[4] Vincent Le Guen and Nicolas Thome. Probabilistic time series forecasting with shape and temporal diversity. In NeurIPS 2020.

[5] Shiyu Wang, et al. TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. In ICLR 2024.

[6] Amin Shabani, et al. Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting. In ICLR 2023.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730506744597

thread_type:
official_review

<thread_object>
creator: reviewer_wm7u

<content>
summary:
The paper presents a new Frequency-enhanced Direct Forecast method for time series predictions that aims to improve the predictions generated by direct forecast (DF) models by training the models using a combination of standard MSE errors and errors defined in the FFT transformed space. The new approach is motivated by the fact that the MSE errors for direct forecast (DF) models are biased and do not properly account for correlations in predicted sequences, while errors on the FFT on the predicted sequences may be more robust to such dependencies.  The experiments on multiple datasets and multiple baseline DF models demonstrate the improved performance of the new method over baselines.

strengths:
Originality: The idea of augmenting an objective function with a loss in alternate bases that is more robust to value dependences in prediction sequences, in this case Fourier bases, is novel and interesting.

Significance: The extension applies to a subclass of direct forecasting models that predict components of future sequences independently, ignoring dependences that may exist among them.  This makes the method potentially applicable to a variety of SOTA time series models. 

Quality: The authors attempt to analyze the problem of using simple additive error functions for predicting sequences and its fixes via errors in the FFT transformed space both theoretically and experimentally. Extensive experimentation across multiple datasets and tasks show the merits of the proposed approach compared to SOTA baselines.

Other strengths: Exploration of properties and extension of the proposed framework and its benefits, such as, different prediction length, ablation models,  possible new transformations and errors defined on these transformations. 

Code: Authors provide the code for reproducing the experiments.

weaknesses:
Limited analysis of a bias in Theorem 1: Theorem 1 considers only univariate sequences, cross-correlation terms are not accounted for in the Bias formula. The sequence has dimensionality D.  

Notational inconsistency and formula errors in the paper: 
- The use of 'L' is overloaded, it denotes both the length and inputs of the input sequence. 
- Equation 1 has a mistake. Y and Y_hat should be compared on the same indexes. 
- Definition 3.2, uses 'j' instead of “i”  

The main results in the paper do not report performance over distinct random seeds. Given this is an experimental study, reporting the results on multiple distinct random seed could help to understand its sensitivity.

The short-term forecasting task results in the appendix report only qualitative time spans. It would be good to specify the results in terms of forecasting lengths, similar to the Table 1 for long-term forecasting.

questions:
Are there any settings where defining the errors by transferring the predictions to the frequency domain is detrimental to the prediction accuracy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730510250627

thread_type:
official_review

<thread_object>
creator: reviewer_3nnt

<content>
summary:
Considering that in current time series prediction, DF only reuses general multi task learning methods without considering the dependency relationships  between labels. Especially, even if there is no correlation between labels, the above loss function can be used to train a multitasking model. When training the model, DF attempts to minimize the error between the predicted label sequence and the true label sequence; This assumes that the label sequence is conditionally independent between different time steps, thereby ignoring the correlation between each time step within the label sequence. So the author uses fast Fourier transform in the frequency domain to transform the data from a temporal perspective to a frequency perspective, in order to suppress autocorrelation. And the theoretical derivation of using frequency domain conversion to suppress time step correlation is given. A loss function combining time domain and frequency domain is designed, and the effectiveness of the design method is demonstrated by experimental results.

strengths:
1. Solving the problem of autocorrelation in label sequences: The FreDF method proposed in this paper effectively bypasses the autocorrelation problem in label sequences by predicting in the frequency domain. This is a common but not fully addressed problem in existing direct prediction models.
2. Compatible with multiple prediction models: The FreDF method is not only suitable for existing state-of-the-art methods such as iTransformer, but can also be compatible with multiple prediction models. This compatibility makes it widely applicable in different prediction tasks.
3. Significant improvement in predictive performance: Experimental results show that the FreDF method is significantly superior to existing methods in multi-step prediction tasks. This indicates that the method has high accuracy and reliability in processing complex time series data.
4. Innovative applications of frequency domain analysis: By introducing frequency domain analysis into time series prediction, the FreDF method provides a new perspective to address autocorrelation issues in time series. This innovative application not only improves predictive performance, but also provides new directions for future research.

weaknesses:
1. Complexity of frequency domain conversion: Although frequency domain analysis can bypass the autocorrelation problem in label sequences, frequency domain conversion itself may increase computational complexity and time cost. This may become a bottleneck when dealing with large-scale datasets.
2. Model generalization ability: Although FreDF performed well in experiments, its generalization ability in different fields and datasets still needs further validation. Especially in practical applications, the model may need to be adjusted and optimized for specific tasks.
3. Dependence on frequency domain features: The FreDF method relies on the extraction and utilization of frequency domain features, which may limit its applicability in certain situations. For example, for data with unclear or difficult to extract frequency domain features, the performance of FreDF may not be as expected.

questions:
1. Computational complexity of frequency domain conversion: Will the frequency domain conversion mentioned in the paper significantly increase computational complexity? How efficient is the FreDF method when dealing with large-scale datasets?
2. Generalization ability: How does the FreDF method generalize across different domains and datasets? Has it been tested and validated in more practical application scenarios?
3. Applicability of frequency domain features: How effective is the FreDF method for data with unclear or difficult to extract frequency domain features? Are there any relevant experimental results or analysis?
4. Model interpretability: Does the FreDF method have interpretability in the frequency domain? How do users understand and interpret these predicted results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640647552

thread_type:
official_review

<thread_object>
creator: reviewer_erxy

<content>
summary:
This work introduces a new loss term for multi-step time-series forecasting that penalizes errors within a decorrelated representation space of the ground truth labels. In its current formulation, this decorrelated space is defined as the frequency representation of both labels and forecasts. Experimental results demonstrate that the proposed approach significantly enhances forecasting accuracy across various datasets and base models.

strengths:
1. The authors identify and address the bias introduced by label correlation in time-series modeling, which is a novel issue for me and holds substantial potential generality across different scenarios.
2. The method is sound, straightforward and shows very promising results. The theoretical results are relatively persuasive, demonstrating the bias caused by label correlation, and subsequently FreDF's elimination of label correlation and thereby bias.
3. The experiment is comprehensive. Extensive set of experiments showed that: the approach contributes to the state-of-the-art, different components of the loss contribute to performance, the approach is robust to hyperparameter values.

weaknesses:
1. It seems that this work aims to penalize errors in a space that decorrelates labels. Experiments mainly achieve de-correlation via FFT or FFT2. It will be beneficial to explore the efficacy of other transformations beyond the Fourier transform?
2. While the introduction discusses various forecasting models, it could benefit from a stronger focus on established forecasting paradigms (e.g., direct forecasting, iterative forecasting). The inclusion of forecast models (iTransformer, Linear) may detract from highlighting the contribution and role of FreDF that seems to be orthogonal to forecast models.
3. The source code should be refined. The current implementation comprises numerous scripts, and it is somewhat unclear how each script relates to the experiments discussed in the manuscript. Besides, the current environment setup appears to depend on unexpected repositories like `torchmetrics` and `patool`. Including a `Dockerfile` or `conda` environment file would help a lot.

questions:
1. In the experiments, how to determine the hyperparameters of FreDF and baselines?
2. According to the ablation study, the frequency term seems to be almost all that's needed. Is it correct?
3. During my attempt to reproduce the experiments, I encountered an error (`AttributeError: 'Exp_Short_Term_Forecast' object has no attribute 'seasonal_patterns'`) when running the short-term forecasting experiments. Is this error expected, or does it indicate a potential issue in the codebase?

Overall I think this work is interesting and promising. I will consider raising my score if my questions could receive positive responses, especially my major concerns (W1, Q1, Q3).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732519047869

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We are grateful to all reviewers for their helpful comments regarding our paper. We are pleased that Reviewers **ErxY, 3nNT, and WM7u** recognize the **significance of our identification and elimination of bias, find our method novel and interesting, and consider our empirical results promising**. Below is a summary of our responses:
- To Reviewer ErxY: We have updated the repository to facilitate ease of reproduction, and clarified the hyperparameter selection process.
- To Reviewer 3nNT: We have emphasized the efficiency advantages of FreDF, added new experiments using an additional dataset, and discussed the potential of FreDF regarding interpretability.
- To Reviewer WM7u: We have included experiments across different random seeds, provided an extended version of Theorem 1, corrected typographical errors.
- To Reviewer 6T6T: We have  thoroughly investigated, discussed, and compared the listed references.  Additionally, we conducted experiments to reproduce and compare **ALL** listed references over six working days, resulting in a comprehensive response. **The conclusion is that FreDF is differentiated from these works in multiple aspects and makes unique contributions, particularly in the identification and elimination of bias caused by label correlation.**

Please refer to our detailed responses to each point raised. We have also uploaded a revised manuscript with changes highlighted in blue. We hope that our revisions and clarifications satisfactorily address the concerns raised.
Thank you again for your valuable time and expertise.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper considers the problem of learning in time series in settings with label autocorrelation. The authors identify a gap in how existing methods behave in these settings. They then propose a new method that can address this gap. The paper includes theoretical analyses and validates the method on synthetic and real-world datasets. 

Overall, the feedback from reviewers was positive. On the one hand, the paper was praised for its innovative focus on label autocorrelation and rigorous theoretical analysis. Reviewers highlighted its potential impact on improving time series modeling and its strong experimental results. On the other hand, the paper lacked specific motivating examples and a discussion of practical applications where label autocorrelation is relevant. Having read the reviews, the rebuttal, and the paper, I am recommending that the paper be accepted at this time. 

In addition to the changes requested by reviewers, I recommend that the authors make two simple changes that can improve the impact:

1. **Motivating Applications**: The paper is missing some discussion of when/where we could "inherent autocorrelation in the label sequence." 

- Abstract and Intro: These sections include claims like: "Time series modeling is uniquely challenged by the presence of autocorrelation in both historical and label sequences." Both sections would be far more compelling if you could add an example of where labels exhibit autocorrelation, and some concrete detail of what can go wrong. 

- Body: The paper could use a more detailed discussion. In general, readers should know the kinds of applications where they can plausibly expect label autocorrection, or where they can ignore this possibility. Listing potential tests could help as well.

2. **Rephrase/Change Title:** I would strongly advise considering updating the title of your paper. The current title has two problems: (a) it's confusing since it reads as a list of nouns ("biases" could be read as a noun too); (b) it does not clearly describe your contribution (there are many different ways to "bias" a forecast; also, it doesn't make it clear if you are developing a method or studying the problem). Some alternatives include: "Learning [Time Series] with Label Autocorrelation", or "Learning Time Series in the Frequency Domain."