Dealing with Frequency Collapse in Time Series Embeddings by Post-Embedding reMapping

Authors: Haotian Si, Changhua Pei, Haiming Zhang, Dan Pei, Gaogang Xie, Jianhui li
Primary area: learning on time series and dynamical systems
Keywords: Time Series Forcasting, deep learning, spectral analysis

Abstract:
Transformer-based methods have made significant strides in time series forecasting tasks in recent years. However, we observe underfitting in numerous samples, e.g., pattern shifts or excessive deviation in extreme value regions when testing the transform-based model that converges on the training set. Through the proposed spectral analysis of adjacent embedding sequences, we identify a frequency collapse issue in the embedding features generated by the top layer of the transformer backbone. To address this, we propose the Post-Embedding ReMapping (PErM) strategy that improves the frequency-domain representation of embeddings using fixed non-linear functions. Both two kinds of PErM functions that we insert into the model can effectively resolve the frequency collapse issue and lead to significant improvements in prediction performance. Experimental results show that our method outperforms state-of-the-art algorithms across multiple datasets. We will release our code after the review phase.

Decision: reject
Reviewer scores: 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730261679323

thread_type:
official_review

<thread_object>
creator: reviewer_gvwq

<content>
summary:
The paper presents a novel approach for improving time series forecasting using transformer-based models, addressing a phenomenon termed "frequency collapse" where high-frequency signals critical for accurate prediction are lost in the embeddings. This underfitting problem particularly affects extreme value regions, leading to significant deviations between predicted values and actual time series patterns. To counter this, the authors introduce the Post-Embedding ReMapping (PErM) strategy, incorporating fixed non-linear remapping functions to enhance high-frequency representation in embeddings without needing additional training.

strengths:
1. Introduces the concept of frequency collapse in time series models, offering fresh insight into the limitations of transformer-based forecasting.
2. The proposed PErM layer is simple and plug-and-play, without increasing much complexity of the module.
3. Thorough comparisons with other advanced methods underscore the robustness of the PErM approach.

weaknesses:
1. The reason why the method can avoid frequency collapse is not presented clearly, especially for LPT. Also, the author could present some connections between RFF and LPT.
2. Some of the compared results are not consistent with previous research. For example, the PatchTST results on Traffic/Electricity are reported lower than the original paper[1]. The author should explain this inconsistency and make more fair comparisons.
3. Low frequency principle often leads to better generalization in deep learning [2]. And there are some methods that deliberately cutoff high-frequency information to achieve better performance [3]. The author should explain the connection and contradiction of these papers.
4. The paper's analysis and solution are limited to the patch-based methods. Whether the conclusions are also applicable to other methods is not clear. 
5. Some of the existing research has pointed out that channel independence leads to smooth prediction [4]. Therefore, the author should include a discussion on the research and explore the correlation between frequency collapse and channel independence.

[1] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayant Kalagnanam: A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR 2023

[2] Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, Zheng Ma: Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks. CoRR abs/1901.06523 (2019)  

[3] Zhijian Xu, Ailing Zeng, Qiang Xu: FITS: Modeling Time Series with 10k Parameters. ICLR 2024

[4]  Lu Han, Han-Jia Ye, De-Chuan Zhan: The Capacity and Robustness Trade-Off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting. IEEE Trans. Knowl. Data Eng. 36(11): 7129-7142 (2024)

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662084596

thread_type:
official_review

<thread_object>
creator: reviewer_nd11

<content>
summary:
This paper introduces a method to improve the performance of transformer-based time series forecasting methods by using a fixed transformation to supposedly enhance the model's ability to handle high-frequency content in input signals. However, as detailed below, there are several issues in the presentation, soundness, and evaluation of the approach.

strengths:
* Studying the inductive biases of modern sequence modeling architectures is an important topic that deserves more attention, and examining the spectrum of the extracted representations is a principled approach to this problem.
* The fact that transformer encoders might lose fine-grained information is also being explored in the literature (particularly in NLP [1]), and this aspect definitely deserves more attention in the context of time series.

weaknesses:
### Main issues

There are several issues.

* **Significance and soundness** Sections 3 and 4 present many obvious facts and report them as novel findings, offering only anecdotal evidence that can hardly support any claim. In particular, the paper presents as "novel" the idea of performing a Fourier transform on representations obtained by feeding the data to the model with a sliding window approach. The entirety of Section 3 reiterates this by showing something as obvious as a matrix with shifted indices in each column. Note also that embeddings produced by any autoregressive sequence model (e.g., from RNN or TCN architectures) preserve the temporal structure of the input; the same can be said for transformer architectures tailored to processing sequences (if the author think that is not the case, this would need to be showed). The theoretical analysis at the end of the section is, again, not particularly informative as it's a simple rehashing of linear algebra facts under overly simplistic assumptions. Section 4 merely shows some qualitative anecdotal results on a single dataset with a single model. Figure 6 once again shows something as obvious as the loss of high-frequency content after applying a low-pass filter to representations extracted by the model. I’m not disputing that transformers may suffer from oversmoothing (other recent work suggests they do, e.g., [1]), but rather that the analysis in this paper lacks depth and significance. Lastly, the adoption of random features and pre-trained LLM layers as fixed transformations should be further justified (see also comments on empirical evaluation).

* **Presentation** The related work section is incomplete, and the paper lacks context on similar studies in the literature. Frequency response analysis is a well-studied area in time series analysis and signal processing (e.g., see [2] for a classic reference) and is commonly used for designing filters and models. The paper should do a much better job positioning its results within the existing literature. Concerning recent ML research, filter analysis and frequency response have been studied, for example, in the context of structured sequence models [3]. Oversquashing in transformers has been studied in [1]. See also minor comments regarding the correctness of several statements.

* **Empirical evaluation** In many of the benchmarks used in the evaluation, a simple autoregressive linear model trained with ordinary least squares can achieve results far better than those reported in the paper, as shown by [4]. Additionally, [5] showed that in these benchmarks, even removing transformer and pre-trained blocks entirely from similar architectures achieves comparable performance to the full model, further hinting at limitations in the presented empirical analysis. The empirical analysis should rather focus on assessing the presence of the discussed phenomena across different relevant baselines and reference architectures. Further, all results should at least be reported with standard deviations concerning different model initializations. 

Given the above, I believe the paper would require a complete rework and a much more in-depth analysis. As such, I cannot recommend acceptance, but I do think this direction has potential.

### Minor comments

* Why is the positional encoding deliberately removed from the adopted model? (lines 301-304)
* Several statements lack clarity or justification. Some examples:
    - "The model’s forward process disrupts the internal temporal structure of the samples, thus the temporal information within each embedding is disordered. This means that performing spectral analysis directly on the embeddings is meaningless." Which model? What is the reference Transformer class? What are the assumptions here?
    - "[...] the sliding window sampling method allows the sample sequence to directly reflect the internal temporal structure of each sample. Thanks to this finding [...]" This is not a finding; it is a well-known basic fact.
    - "Fortunately, we notice a notable characteristic of time series tasks is the strong correlation between adjacent samples, which is different from other tasks." See the above.
    - "We further prove the absence of high-frequency signals in the embedding sequence." As motivated above, I believe this claim is not adequately justified.
    - "Since time series data inherently exhibits autocorrelation, employing positional embedding may disrupt this autocorrelation." How? Why?

#### References

[1] Barbero et al., "Transformers need glasses! Information over-squashing in language tasks" NeurIPS 2024\
[2] Strang and Nguyen, "Wavelets and filter banks" 1996\
[3] Fu et al., "Simple Hardware-Efficient Long Convolutions for Sequence Modeling" ICML 2023\
[4] Toner et al., "An Analysis of Linear Time Series Forecasting Models." ICML 2024\
[5] Tan et al. "Are language models actually useful for time series forecasting?" NeurIPS 2024

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730881389746

thread_type:
official_review

<thread_object>
creator: reviewer_us8q

<content>
summary:
This paper addresses the frequency collapse issue observed in Transformer-based time series forecasting models, where the embeddings generated by the model lack high-frequency signals, leading to underfitting and poor prediction performance. To tackle this problem, the authors propose the Post-Embedding ReMapping (PErM) strategy. PErM introduces a predefined non-linear remapping layer between the top layer of the Transformer backbone and the model's prediction head. This layer projects high-frequency features from the original signals into a new frequency domain while re-integrating information within the embeddings. Experimental results demonstrate that using various PErM functions effectively alleviates the frequency collapse phenomenon and significantly improves the final prediction performance.

strengths:
1. The paper identifies and addresses a novel issue, frequency collapse, in Transformer-based time series forecasting models. The proposed PErM strategy is an innovative solution that uses a non-linear remapping layer to enhance the frequency-domain representation of embeddings, offering a fresh perspective on improving model performance.
2. The authors provide thorough experimental validation of the PErM strategy. They conduct experiments on multiple datasets and compare their method with several state-of-the-art approaches, demonstrating the effectiveness of PErM in enhancing prediction accuracy. The experimental setup is robust and the results are compelling.
3. The paper is well-structured and clearly written.

weaknesses:
1. The authors claim that transformers can lead to a lack of high-frequency signals in the embeddings, but they do not provide a theoretical explanation for this phenomenon. A deeper theoretical analysis would help understand why transformers tend to produce embeddings with fewer high-frequency components.
2. The authors propose that using various PErM functions can effectively alleviate the frequency collapse phenomenon and support this claim with experimental results. However, they do not provide a clear and reasonable explanation for why PErM functions are effective in mitigating frequency collapse. A more detailed theoretical or empirical analysis would strengthen the understanding of the underlying mechanisms.
3. Some figures in the paper are not referenced or explained in the main text. This makes it difficult for readers to understand the purpose and meaning of these figures. Adding references to these figures and providing detailed explanations would enhance the readability and clarity of the paper.
4. The experimental section seems to lack some commonly used datasets, such as Electricity and Exchange. Including these datasets would provide a more comprehensive evaluation of the proposed method and allow for better comparison with existing state-of-the-art models.
5. From the results, it appears that PErMformer performs better in shorter horizons and on the ETTm2 dataset for longer horizons. The paper does not provide a theoretical explanation for this variability. Using the theoretical framework presented in the paper, a detailed analysis of why PErMformer excels in different horizon lengths could provide valuable insights and improve the understanding of the method's strengths and limitations.

questions:
Refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732192537455

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised paper is updated

comment:
We have submitted the revised paper. 3 Supplement Sections have been added to the Appendix (Appendix B,C,D), including supplementary information on spectral analysis,  supplementary analysis on frequency collapse, and supplementary analysis on PErM strategy. The text of updated section has been set to blue. We hope these sections can address some of your concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732498306022

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to your reply

comment:
Dear reviewers,

Thank you for your time and constructive feedback. We have provided specific responses to your comments after carefully considering your suggestions and concerns. We hope our explanations will dispel your concerns.

We would be happy to receive more valuable comments and have further discussions with you. We look forward to hearing from you soon!
</content>
</thread_object>

</thread>


Metareview: