FreRA: A Frequency-Refined Augmentation for Contrastive Learning on Time Series Classification

Authors: Tian Tian, Chunyan Miao, Hangwei Qian
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: time series classification, contrastive learning, frequency domain

Abstract:
Contrastive learning has emerged as a competent approach for unsupervised representation learning. However, the design of an optimal augmentation strategy, although crucial for contrastive learning, is less explored for time series classification tasks. Existing predefined time-domain augmentation methods are primarily adopted from vision and are not specific to time series data. Consequently, this cross-modality incompatibility may distort the global semantics of time series by introducing mismatched patterns into the data. To address this limitation, we present a novel perspective from the frequency domain and identify three advantages for downstream classification: 1) the frequency component naturally encodes global features, 2) the orthogonal nature of the Fourier basis allows easier isolation and independent modifications of critical and unimportant information, and 3) a compact set of frequency components can preserve semantic integrity. To fully utilize the three properties, we propose the lightweight yet effective Frequency-Refined Augmentation (FreRA) tailored for time series contrastive learning on classification tasks, which can be seamlessly integrated with contrastive learning frameworks in a plug-and-play manner. Specifically, FreRA automatically separates critical and unimportant frequency components. Accordingly, we propose Identity Modification and Self-adaptive Modification to protect global semantics in the critical frequency components and infuse variance to the unimportant ones respectively. 
Theoretically, we prove that FreRA generates semantic-preserving views. Empirically, we conduct extensive experiments on two benchmark datasets including UCR and UEA archives, as well as 5 large-scale datasets on diverse applications. FreRA consistently outperforms 10 leading baselines on time series classification, anomaly detection, and transfer learning tasks, demonstrating superior capabilities in contrastive representation learning and generalization in transfer learning scenarios across diverse datasets.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730103167229

thread_type:
official_review

<thread_object>
creator: reviewer_7f2v

<content>
summary:
This study introduces Frequency-Refined Augmentation (FreRA), a method designed to overcome limitations in current augmentation strategies for time series classification in contrastive learning. Unlike existing visual-based augmentations, FreRA leverages three key advantages of the frequency domain properties to better preserve the global semantics of time series data. FreRA automatically segregates these components, applying Identity Modification to preserve vital details and Self-adaptive Modification to add variance to less significant parts. Theoretical proofs and empirical evaluations confirm FreRA's superiority, showing it outperforms ten leading baselines across various time series tasks.

strengths:
1. This paper is well-written and well-organized.
2. The methodology is clear and the problem is well-motivated.
3. This is a plug-and-play method that appears to integrate seamlessly with existing contrastive learning frameworks.

weaknesses:
1. Compared to existing works, this work does not exhibit a notable advantage in classification performance.
2. I am somewhat confused about the experimental design for the transfer learning part: 1) Why was SHAR data selected instead of one of the datasets listed in Table 1 (e.g., UCIHAR) to evaluate the transfer capability of the algorithm? 2) Based on the experimental results in Table 2, the performance is lower than that reported in the reference work (Qian et al., 2022). What might be the reasons for this difference?
What did this experiment aim to prove?
3. In the ABLATION STUDIES part, the authors sequentially removed each of the three innovative method components for comparison. From the experimental results, the gains provided by the three modules seem roughly equivalent. If all three modules were removed, what would be the resulting performance? Looking at Table 1 in the paper, the performance of softCLT and FreRA appear quite similar. If FreRA were integrated into softCLT, would there be any gain in performance?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653045713

thread_type:
official_review

<thread_object>
creator: reviewer_gihe

<content>
summary:
The paper introduces a novel augmentation technique designed for time-series contrastive learning by leveraging frequency-domain properties. It utilized the idea of FFT which separates time-series data into critical and non-critical frequency components.

strengths:
The method utilizes the connection between frequency domain knowledge and semantic information to enhance the representation learning. The critical components capture global semantics essential for classification, while non-critical components are used for self-adaptive noise injection, which I found is an interesting link and the authors provide a comprehensive explanation for the motivation.
The authors provide extensive experiments and strong experiment results to demonstrate their method's effectiveness.

weaknesses:
1. At the beginning of the paper, the authors make strong assumptions that existing predefined augmentation methods are primarily adopted from vision and are not specific to time series data. There are already several methods, especially frequency-based augmentation, e.g., TF-C, method design for time series contrastive learning.
2. Since the paper mainly provides the frequency-based augmentation, the motivation study, such as Figure.1 probably should highlight more about whether current frequency-based method can capture the semantics, rather than only focus on the time-domain,

questions:
/
</content>
</thread_object>

</thread>
<thread>
creation date:
1730882819147

thread_type:
official_review

<thread_object>
creator: reviewer_qixi

<content>
summary:
This paper proposes Frequency-Refined Augmentation (FreRA), an augmentation method for time series contrastive learning on classification tasks. FreRA automatically separates critical and unimportant frequency components, and accordingly proposes Identity Modification and Self-adaptive Modification for different components. It conducts experiments on two benchmark datasets including UCR and UEA archives, as well as 5 large-scale datasets on diverse applications. FreRA outperforms 10 leading baselines on time series classification, anomaly detection, and transfer learning tasks.

strengths:
Data augmentation is an important problem for time series and contrastive learning. This paper investigates this problem and proposes a method from the frequency perspective. The proposed method seems easy to follow and implement. The experiments in this paper are extensive, including many different datasets and tasks. The proposed method outperforms most of the baselines.

weaknesses:
W1: Some important terms in this paper are not clearly defined, such as ‘semantic integrity’ and ‘critical and unimportant frequency components’. Why can we measure semantic integrity using mutual information? Is this consistent with humans’ understanding of semantics? How can we measure the importance of frequency components? What are these critical components critical for?

W2: The overall novelty of the proposed augmentation method is limited. Augmentation from the frequency components is not a new idea for time series. The main difference is a trainable vector s to control the augmentation of different components. It is unclear why s trained from Equation (7) can learn to select critical components automatically.

W3: It is unclear how FreRA achieves both semantic-preserving information and a considerable amount of variance. The authors need to clarify which designs correspond to these two sides respectively. 

W4: The self-adaptive modification seems simple and tricky. It only uses the vector s and threshold to select and scale unimportant frequency components. The motivation for scaling these components is unclear. 

W5: Compared with some SOTA baselines, such as SoftCLT and InfoTS, the advantage of FreRA is not clear, especially on UEA and UCR datasets. The ablation study is coarse, and some important variants are missing. For example, modifying all (or randomly selected) frequency components and modifying unimportant components randomly.

questions:
Some other questions:

Q1: As this paper focuses on time series classification, why is it also evaluated in anomaly detection? 

Q2: Figure 2 is hard to read. The different colors for vector blocks in FreRA are confusing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731076716099

thread_type:
official_review

<thread_object>
creator: reviewer_cxw7

<content>
summary:
The paper proposes a method, FreRA, to enhance time series classification by contrastive learning and sample augmentations. First, they considered the frequency domain of the time series. FreRA automatically separates the critical and unimportant frequency components. They proposed Identity Modification and Self-adaptive Modification to protect the global semantics in the critical frequency components and inject variance into the unimportant components, respectively. Extensive experimental results on several datasets show that FreRA outperforms existing methods in terms of accuracy.

strengths:
1. Overall, this paper is well-written and easy to follow.
2. The problem studied is significant, and exploring augmentation in time series is novel.
3. Extensive experimental results are promising.

weaknesses:
1. The importance distinction of this method is mostly for the entire time series, and it could be better to compare it with other methods and analyze the theoretical computational complexity.
2. Although frequency methods can improve efficiency, it is unclear whether such methods mainly focus on the low-frequency part and ignore the high-frequency part which is more important for time series prediction.
3. Do the authors consider the dependencies between channels, which is very significant for multivariate time series.
4. The authors claim that FreRA can be benefited by any contrastive learning framework, but only show the results of InfoNCE. What about other CL paradigms, such as SimCLR, etc.?  It could be better to present more sufficient ablation.
5. The experimental results are selected from the highest performances among 11 time-domain augmentations and 5 frequency-domain augmentations. Is this fair enough? There seems to be randomness with such selection strategy.
6. The results of the impacts of hyper-parameters could be moved to the main paper for a better organization.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732624384556

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear ACs and Reviewers,

We sincerely appreciate your time and effort in reviewing our work and providing constructive feedback. We would like to 1) express our gratitude for reviewers’ recognition of our work, and 2) highlight the major modifications made in our revised paper.

**We thank the reviewers for recognizing and appreciating the advantages of our work.**

* Investigating automatic augmentation in time series is **significant, novel and well-motivated**. [cXW7,Qixi,7F2V,GiHe]
* The proposed methodology is **clear, interesting, novel, and easy to follow and implement**.  [7F2V,GiHe,Qixi]
* The proposed FreRA is a **plug-and-play** method that can be integrated seamlessly with existing contrastive learning frameworks.[7F2V]
* **Extensive experimental results** are **promsing or strong**. [cXW7, Qixi, GiHe]
* The paper is **well-written, well-organized and easy to follow**. [cXW7,7F2V]

Besides the response to each reviewer, we would like to summarize the **major modifications made in our revised paper (highlighted in blue)**:

1. **More evaluations on different contrastive learning frameworks.** We expand the evaluation of FreRA on 3 additional contrastive learning frameworks. The results reported in Table 5 and Table 6 of Appendix A.9 demonstrate that FreRA is a **plug-and-play** method and it **consistently and effectively enhances existing contrastive learning frameworks**. [cXW7,7F2V]

2. **Visualization and analysis on the learned vector $\mathbf{s}$.** We visualize the learned vector $\mathbf{s}$ and demonstrate its ability to **capture the diverse distributions of critical semantics** across three different datasets. The visualization is provided in Figure 6, with explanations in Appendix A.9. [cXW7,Qixi]

3. **More discussion and visualization on predefined frequency-domain augmentations.**
    
    * We revise the statements regarding our motivation for investigating the frequency domain in the abstract to make it precise and accurate. Additionally, we include a detailed discussion on frequency-based augmentations and their limitations in Appendix A.2. [GiHe]
    * We include the predefined frequency-domain augmentation in Figure 1 to illustrate it also causes undermined semantics in the generated views. Corresponding explanations are updated accordingly. [GiHe]

4. **Presentation enhancements.**

    * We update the legend and caption in Figure 2 to clarify the use of colors for the vector blocks. [Qixi]
    * We move the ablation study on the impact of hyper-parameter from the appendix to the main paper for better organization. [cXW7]


Thank you once again for taking the precious time to review our work. We would be delighted to engage in further discussions if you have any remaining questions or concerns.

Best regards,

Paper 6561 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a new augmentation method named Frequency-Refined Augmentation (FreRA) for time series contrastive learning and classification. Reviewers agreed that the paper is well written and easy to follow, the method is well motivated, and the experiments are extensive. Meanwhile, reviewers pointed out that there are still some limitations regarding technical contribution, details on methodology, experiments, novelty, etc. Although some of these concerns have been addressed during the rebuttal and discussion stage, some issues still remain. For instance, the novelty of the proposed method is not significant, and the advantages of FreRA over existing work are not sufficiently justified. Overall, this is a borderline paper.