Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors present a self-supervised pre-training model for time series (TS) data by leveraging Time-Frequency Consistency (TF-C). The main idea is that time-based and frequency-based representations of the same time series sample are closer in the joint time-frequency space, and farther apart from the representations of other samples. The model has 4 components: a time encoder - to embed TS from time domain; a frequency encoder - to embed TS from frequency domain; and two cross-space projectors that map time-based and frequency-based representations to a unified time-frequency space. The model is trained using 3 types of losses: Contrastive time loss, Contrastive frequency loss, and Time-frequency consistency loss. To enable contrastive learning, the model uses augmentation banks for both time and frequency embeddings. The authors evaluate their method in two experimental scenarios, (a) pre-train on one dataset and test on one dataset (b) pre-train on one dataset and test on multiple datasets with 8 diverse datasets and compare it against 8 baselines methods to show the superiority of TF-C approach.
strengths_and_weaknesses: Strengths:

1. The paper is well written and easy to follow. I enjoyed reading it.
2. While frequency domain has been modeled in recent methods like CoST, the idea of using Time-Frequency Consistency, in addition to individual domain losses, for self-supervision is novel.
3. The method was compared to 8 baselines methods on 8 diverse datasets and showed strong empirical results. Through an ablation study of loss function components, the paper confirms that each loss term contribute towards the model performance.
4. The paper performed multiple experimental settings (one-to-one and one-to-multiple) and dataset scenarios (transfer learning among datasets with substantial discrepancy) to show the method outperforms the other baselines.

Weaknesses:
1. The authors present the proposed method as universal pre-training method for time-series, however, only a single time series task (time-series classification) is explored in the paper. Therefore, it is not clear if the proposed method performs competitively on other time series tasks.

Typos:

(a) Line 173 - shouldn’t it be D^tune instead if D^pret?
(b) Line 175 -  “D^pret and D^pret” ->  “D^pret and D^tune”
(c) Line 221 - “sale” ->   scale


questions: 1. The current embedding scheme and objective functions focus on global time series features which are suitable for the classification task. Can the proposed method be applied easily to time series forecasting task where modeling local features is relevant as well? 

2. Did you try pre-training the model on multiple datasets as well? 


limitations: See weaknesses.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a self-supervised contrastive pre-training target for time series via time-frequency consistency. Based on the assumption that time-based embedding and frequency-based embedding for a particular time series example should be close together in time-frequency space, consistency loss is introduced. Evaluation of eight real-world time series datasets under one-to-one and one-to-many transfer settings, the proposed pre-training target is proved to have better performance over existing methods.

=====================

Updates in the rebuttal period: Thanks for the authors' responses about updated clarifications and extended experiments. I have read them carefully and raised my scores.
strengths_and_weaknesses: Strength:
1. The paper is clearly written and well presented. 
2. Experiments on eight real-world datasets are comprehensive.

Weakness:
1. The novelty should be discussed a step further. This paper claims that "this is the first work that develops augmentations in frequency domain", however, this sentence is ambiguous. Does this paper first "augment time series self-supervised learning in frequency domain" or "develops frequency domain augmentations"? This should be awared because the reviewer has noticed that there is a relevant method BTSF to appear in ICML 2022 (first appeared in arxiv in 8 Feb 2022), using a close temporal-spectral fusion idea for time-series representation learning, which involves both time domain and frequency domain for contrastive learning as well (amazingly experimental SLEEPEEG, EPILEPSY, HAR datasets in this paper are also used in BTSF). Please consider rephrasing this claim based on the above fact. Besides, the differences and specific novel ideas of this paper to that of BTSF should be discussed.
2. More baselines should be compared. Many baselines listed in the related work are supposed to be compared in the experiments as well. Especially some of them show better performance than the currently selected baselines. Besides, please consider update the references, for example, [45]: arxiv -> ICLR 2022, [46]: arxiv -> AAAI 2022, [47]: arxiv -> IJCAI 2021. 
3. The experiments only involves classification tasks without regression task, forecasting task, anomally detection task. Hence, it is suggested to point out that this paper aims at this specific task in the title and introduction. 
4. Further understanding of frequency augmentation is needed. As a fundamental concept and assumption of the paper, the time-frequency consistency's underlying mechanism is not discussed in this paper. This paper verified its effectiveness, but doesn't provide further analysis about how it works. Theoretical analyses are also missing. Besides, as mentioned in section 4.2, augentation in frequency domain may change the time series significantly in time domain. Does the augmented time series still make sence? Since no case study is given, the effectiveness and logic of this operation is questionable.
5. Minors: - "For" -> "for" in the title. 
questions: 1. What are the detailed augmentation and frequency-domain transformation settings in the experiments? How are different augmentations compared in the frequency domain? 

limitations: 1. The novelty should be discussed a step further. 
2. Theoretical analysis, further experiments, or case studies about frequency domain augmentations should be added. 
3. More baselines should be compared.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new way of training self-supervised models for time series, such that time-frequency consistency is enforced. To this end, the authors add two additional loss terms to an earlier-introduced contrastive loss. The two additional terms enforce time, respectively frequency consistency, inspired by the triplet loss. Performance of the proposed model is compared to several strong baselines, and the proposed model is shown to outperform most baselines on most metrics.
strengths_and_weaknesses:  Originality: The proposed approach is simple, but elegant.

Quality: The paper is of good quality.

Clarity:  The paper is well-written and easy to follow, and the figures are also clear and informative, therewith helping the reader to gain a better understanding. 

Significance: The field of self-supervised learning has so far mostly been focused on imaging tasks, while many real life data are time series. Development in self-supervised learning methods, dedicated to time series is, therefore, highly significant. 
questions: Could the authors explain how it is possible that the random initialization method for one-to-many in sleepEEG --> Fd-B performs almost as good as the proposed model, and much better than most of the baselines on the AUPRC metric?

How sensitive is the performance to the setting of lambda?

Two additional minor things:
- Fig 1: the symbols for the embeddings of the time-encoder have an F superscript instead of a T.
- L277: "have" should be "has"

limitations: Yes, limitations were discussed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new method for self-supervised pre-training for time series datasets. These pre-trained models can then be fine tuned for classification on different downstream time series datasets.

In addition to using augmentations on time series data in a contrastive learning setting to obtain time series specific  representations , the authors also input frequency domain (FFT)  for input time series. These frequency domain inputs  are  also augmented through frequency domain perturbations. These frequency domain inputs and their corresponding perturbations are then used by a contrastive loss to learn frequency domain representations.

These time series and frequency domain representations are then projected to a  Time-freqeuncy space where the the time series representations and their corresponding frequency domain representations are encouraged to be closer than the time series representations and its corresponding augmented frequency domain representations (or frequency domain and its augmented time series representation) through a margin loss.

This methodology results in representations ,which when fine tuned on different types of time series datasets, leads to improved classification performance over other self-supervised representation learning baselines.
strengths_and_weaknesses: Pros
- The paper proposes a novel direction for self-superivison for time series datasets through both, perturbations in the frequency domain , as well as a join time-frequency representation space
- There has been relatively little work on learning self-supervised representations for time series datasets (at least as compared to image/vision/nlp domains). This work addresses this and presents a method which could be useful for the community.
- The method was extensively evaluated on different types of downstream time series classification tasks (that range from ECG, to human activity recognition and motor fault detection) and showed promising results
- The method was mostly  clearly described and the paper was was to follow.

Cons
(This isn't really  a con, 
- I had some issues with section 3, particularly the paragraph titled "Rationale for Time-Frequency Consistency" where some of the sentences were a bit confusing. E.g line 162. "the relationship between them provides an invariance" that can be exploited. I understand how the relationship between  time domain and frequency domain could be invariant , but I am having difficulty of seeing how this could be useful for self-supervised representations. I am thinking of this interns  invariant representations learned for images, where an image and its rotated form should have invariant representations, and is more intuitive to understand.
  Maybe more details on how the time-frequency relationship invariant can be useful for self-supervision could be helpful (perhaps though a simple example) 
questions: Could the authors provide more information about the permutation budget needed for frequency domain perturbations ?
 The authors mention that the perturbation budget is to ensure that the perturbations do not change the input seignifcantly but more details on this could be helpful (as  some low frequency pertbuations might change the signal more than high frequency perturbations , though this might depend on the input sequence ). 
Some details on how these perturbations affect the learned representations could be helpful for the readers.


limitations: The authors have talked about some limitations in their work (e.g how the method can't directly work for irregulary sampled data but encoder based methods could help solve this)


Also I think mentioning how different time series provide different types of representations which are more suitable for certain downstream time series classification datasets (like human activity pre train and fine tune gesture) could be helpful when discussing the limitations.
(The authors do mention in the results section, but mentioning in the limitations could be helpful ) 

Also  would this method work for really high dimensional datasets ? (Like 100 dimensional?)
review_title: null

Meta-review: metareview: The reviewers highlight the novelty and significance of the proposed approach, the extensive empirical evaluation, and the clarity of writing. Additional experimental results provided during the rebuttal cleared up some concerns around the general applicability of the approach. 
confidence: Certain
Decision: Accept