Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper presents SimPer, a simple yet effective SSL algorithm specifically designed for periodic tasks (e.g. heart rate estimation & exercise repetition counting through video data, temperature forecasting through satellite data). With the newly proposed techniques including periodicity-variant (i.e. input frequency) augmentation, periodic feature similarity, and generalized (weighted) contrastive loss, SimPer demonstrates impressive results in a large array of tasks including SSL (Table 2 to 7), fine-tuning (Table 8), low-shot learning (Fig 4), transfer learning (Table 9), zero-shot generalization (Table 10), robustness to spurious correlations (Fig 6), etc. against existing SSL algorithms (SimCLR, MoCo v2, BYOL and CVRL) and supervised training, on 6 datasets (2 synthetic) from 4 different domains.
strength_and_weaknesses: Strengths
+ (Clarity) The paper is very well organized and clearly written. The figures are also well made and very informative.
+ (Novelty) The paper and its proposed techniques are novel as far as I can tell. The topic is clearly important (for its e.g. medical & health applications) yet doesn’t seem to be as popularly studied as other ML/SSL topics. 
+ (Quality) The paper is of good quality in my opinion. The algorithm is well designed for periodic signals to maximally extract useful information (via frequency augmentation) to be better learned (via periodic feature similarity & generalized loss). The technical details are all correct as far as I can tell. The empirical evaluation is also comprehensive, covering 6 different SSL (and related) tasks against popular baselines, ~even though the selection of baselines could be further improved (see weaknesses).~
+ (Significance) The proposed techniques are simple, easy to implement and experimentally highly effective, making the algorithm (SimPer) a strong, potentially impactful baseline for future researchers & practitioners to use and/or improve upon.

Weaknesses
- ~(Evaluation, Minor) The algorithm is currently only benchmarked against 4 SSL baselines and a supervised counterpart, not SOTA algorithms on each of the datasets (e.g. the MAE for Countix appears much weaker than Table 7 of Dwibedi et al.). While SimPer is likely orthogonal to those SOTA algorithms and thus might be applied jointly, it’s still non-ideal to completely omit the SOTA baselines and the discussion on how SimPer could further boost the SOTA numbers.~
clarity,_quality,_novelty_and_reproducibility: Please see above for my evaluation of the clarity, quality and novelty of the paper, which are all strengths of the paper in my opinion.

Regarding reproducibility, the submission didn’t include anonymized source code. However, the authors did mention in the appendix their plan to release the code, and given the overall simplicity of the paper, I believe it won’t be too hard to reproduce its results even without the source code.

Below are some additional questions for the authors.
* Given that fractional (<1.0x) and non-integer (e.g. 1.5x rather than 2x, 3x, etc.) frequency scalings seem to be required for augmentation, how are they handled for non-synthetic datasets where frame rates are fixed? Is motion-compensated (rather than naive) frame interpolation used in this paper? How does the quality of frame interpolation affect the results, especially for datasets with larger motions like Countix?
* What’s the shape of the network’s output given a video of length $T$? $\mathbb{R}^T$ or $\mathbb{R}^{T \times H}$ with some hidden dimension $H$? In Sec 4.1, is FFT simply performed over the temporal dimension as 1D FFT? Why is FFT able to predict the underlying frequency given that the actual frequency numbers never seem to be used for feature learning?
* Are Fig 4(a) and 4(b) both showing results for RotatingDigits (according to Sec 4.2) or SCAMPS (according to Sec D.1)? Based on the MAE numbers, Fig 4(b) is likely showing SCAMPS. But given the visual similarity to Fig 1, is Fig 4(a) showing SCAMPS instead of RotatingDigits? Does UMAP create similar visualizations for both datasets?
summary_of_the_review: In my opinion, since the paper is strong in ~nearly~ all aspects (clarity, novelty, quality and significance, ~except for minor weakness in evaluation~) with sufficiently impressive results, I recommend its acceptance to the conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: SimPer is a self-supervised learning method for learning representations from periodic data such human physiology. In this paper the authors demonstrate that existing baselines such as SimCLR do not work well. The key idea in this paper is to revise the idea of feature similarity for temporal data. More specifically, the idea is to improve similarity of temporally close features (defined through index-shifting and reversing augmentations). A "random" feature is defined to be a negative feature. The results demonstrate that SimPer is performing better than some of the self-supervised learning methods.
strength_and_weaknesses: **Strengths**

* The method improves results over self-supervised methods such as SimCLR. 

* The paper is well written and easy to follow.

* The paper advances simple augmentation technique for temporal contrastive learning framework.

**Weaknesses**

* Although the approach is interesting, the experimental section would require additional results to benchmark thoroughly against CVLR. For example, this submission does not include results on Kinetics-400. (as opposed to a subset)

* There has already been self-supervised learning approaches in the area of human physiology. This submission does not compare with this ICCV paper. [ref 1]. This paper already explores the idea of maximal cross correlation. (MCC)

* It is not clear if the evaluation protocol is good enough for representations. Especially, for computing a single HR value over a large window. In other words, even if the representations are very bad for large portion of the video, the evaluation protocol would not be able to detect it.

* Additional comparisons with baselines: The results on known traditional methods are better on SCAMPS dataset than the proposed method. These baseline methods have not been to be included in this paper. It would be beneficial to the reader to have a comparison in the paper.

**References**

[ref 1] The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video, ICCV'21
[ref 2] SCAMPS: Synthetics for Camera Measurement of Physiological Signals, NeurIPS'22


clarity,_quality,_novelty_and_reproducibility: * The paper is well written and easy to follow

* The quality of the submission is high. The figures and tables are placed neatly to improve the readability of paper.

* The idea of using simple augmentation techniques for temporal learning is novel.

* The paper contains reasonable amount of information that could help in reproducibility.
summary_of_the_review: Comparing the strengths and weaknesses, this paper can be further improved to study why existing unsupervised methods are superior to the proposed method. Hence, in my opinion, the paper is not ready for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a contrastive learning framework for encoding periodic information from high-dimensional input sequences. The proposed method (SimPer) leverages an inductive bias present in most periodic processes to define suitable data augmentations, feature similarity measures, and loss functions. To create negative pairs, the method intervenes on a sequence by changing its frequency (by changing its speed). The proposed similarity metrics take into account temporal offsets or the power spectrum density of the signals using existing measures. The loss function is an interesting generalization of the InfoNCE loss where the loss for a pair of instances is weighted using side information on the similarity of the given pair (e.g., the absolute difference in the speedup of two sequences). A thorough empirical analysis shows that SimPer encodes periodic information well – significantly better than frame-based methods (SimCLR, MoCo, BYOL) and one approach based on sequences (CVRL). SimPer shows improved data efficiency compared to self-supervised and supervised methods, and improved accuracy for fine-tuning, transfer learning, OOD generalization, and robustness to spurious correlations compared to other self-supervised methods.
strength_and_weaknesses: **Strengths**

- The overall idea and the individual design choices are well-motivated and explained very clearly in the text and supported by suitable illustrations.
- It is transparently stated that the method makes use of an inductive bias to construct suitable augmentations and loss functions.
- The empirical analysis is very comprehensive. It clearly shows the benefits of the proposed method and provides helpful ablations.
- The writing and exposition of the paper are very clear. 


**Weaknesses**

- It is not sufficiently clear how to choose the sequence length and how important the hyperparameter might be for the encoding of periodic information. For instance, a very long sequence might contain multiple subsequences that exhibit different periodicities. I think it would be helpful to include an ablation for the sequence length.
- Do I correctly understand that the baselines SimCLR, MoCo, and BYOL use individual frames as inputs, whereas CVRL and SimPer use sequences as inputs? In that case, it would not be surprising that the frame-based approaches do not encode periodic information, which would be difficult to estimate from a single frame.
- It would be interesting to measure not only the periodic information that is encoded, but also how much other information (object class, etc; depending on the dataset) is retained in the encoding. For instance, Figure 6 shows that the spurious feature (digit appearance) is encoded relatively well for the training data, even though it is not an invariant feature that would help discriminate between positive and negative pairs.

**Minor comments and questions:**
- In Figure 4a, why do you plot fewer data points when you decrease the training dataset size? Are the learned representations not based on a test dataset of a fixed size?
- Why is the performance of the proposed method on the LST dataset not much better compared to the baselines?
- In the context of the Nyquist sampling theorem, it would be helpful if you briefly stated its results and why it is relevant.
- Did you decrease the margins for the abstract?

clarity,_quality,_novelty_and_reproducibility: The work has high quality and the writing and exposition are very clear. There are a few additional ablations that could be helpful (see weaknesses), but overall the experiments are very comprehensive and supportive of the claims made in the abstract and introduction.

I cannot fully assess the novelty of the paper regarding the task of periodicity estimation/representation. The related work section covers previous work on this problem and it is consistent with my literature scan.

There should be no major issues with respect to reproducibility, as the method is explained very well and pseudocode is provided in the appendix. It would certainly be helpful if the authors also provided the code to reproduce the experiments.
summary_of_the_review: The paper presents a simple framework that is well-motivated and quite intuitive. The paper has an empirical focus and, appropriately, provides comprehensive experiments and ablations. One exception is that the paper could provide more information regarding the choice of sequence length, which might be an important hyperparameter. Other than that, I have no major concerns and recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents an effective SSL algorithm specifically designed for periodic tasks (e.g. heart rate estimation & exercise repetition counting through video data, temperature forecasting through satellite data). The proposed method (SimPer) includes periodicity-variant augmentation, periodic feature similarity, and a generalized (weighted) contrastive loss. SimPer demonstrates impressive results in a large array of tasks. 
All reviewers as well as the AC agree that this is an outstanding contribution as it allows extending SSL techniques to a whole range of applications involving periodic/near-periodic data.

justification_for_why_not_lower_score: Given the large interest of the community in SSL techniques and the outstanding performance of the proposed approach, this work might have a large impact.
Decision: Accept: notable-top-5%