SimPer: Simple Self-Supervised Learning of Periodic Targets
Authors: Yuzhe Yang
Xin Liu
Jiang Wu
Silviu Borac
Dina Katabi
Ming-Zher Poh
Daniel McDuff
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Periodic learning
Self-supervised learning
Representation learning
Periodic targets
Periodicity

Abstract: From human physiology to environmental evolution, important processes in nature often exhibit meaningful and strong periodic or quasi-periodic changes. Due to their inherent label scarcity, learning useful representations for periodic tasks with limited or no supervision is of great benefit. Yet, existing self-supervised learning (SSL) methods overlook the intrinsic periodicity in data, and fail to learn representations that capture periodic or frequency attributes. In this paper, we present SimPer, a simple contrastive SSL regime for learning periodic information in data. To exploit the periodic inductive bias, SimPer introduces customized augmentations, feature similarity measures, and a generalized contrastive loss for learning efficient and robust periodic representations. Extensive experiments on common real-world tasks in human behavior analysis, environmental sensing, and healthcare domains verify the superior performance of SimPer compared to state-of-the-art SSL methods, highlighting its intriguing properties including better data efficiency, robustness to spurious correlations, and generalization to distribution shifts.

Published as a conference paper at ICLR 2023
SIMPER: SIMPLE SELF-SUPERVISED LEARNING OF PERIODIC TARGETS
Yuzhe Yang', Xin Liu", Jiang Silviu Borac Dina Katabi', .Ming-Zher Poh Daniel McDuff MIT CSAIL University of Washington
'Google
ABSTRACT
From human physiology to environmental evolution, evoiiottienalllooutiionn important processes in nature often exhibit meaningful and strong periodic or quasi-periodic changes. Due their inherent label scarcity, learning useful representations for periodic tasks with limited or no supervision is of great ben- efit. Yet, existing self-supe learning (SSL) methods overlook the intrinsic periodicity in data, and fail to learn representations that capture periodic or frequency attributes. In this paper, we present SimPer, simple contrastive SSI. regime for learning periodic information in data. To exploit the periodic inductive bias, SimPer introduces customized augmentations, feature similarity measures, and a generalized contrastive loss for learning efficient and robust periodic representations. Extensive experiments sis, environmental sensing, and healthcare domains verify the superior performance of SimPer compared to state-of-the-an SSL methods, highlighting its intriguing properties including bet- ter data efficiency, robustness to spurious correlations, and generalization to distribution shifts. Code and data available at: https: http//gtttt/.tt//////////t com cwm/ htte//yyyaaaryiiilll
common real-world tasks in human behavior analy-
Supervised
Self-Supervised
hes
u-
31-
344
SimCLR
CVRL
SimPer (Ours)
Figure 1: Learned representations of different methods on periodic learning dataset. RotatingDigits (details in Section 4). Existing self-supervised learning schemes fail capture the underlying periodic frequency information in contrast, SimPer learns robust periodic representations with high frequency resolution.
INTRODUCTION
Practical and important applications of machine learning in the real world, from monitoring the earth from space using satellite imagery (Espeholt et al., 2021) to detecting physiological vital signs in a human being (Luo al., 2019), often involve recovering periodic changes. In the health domain, learning from video measurement has shown to extract (quasi-)periodic vital signs including atrial fibrillation (Yan et al., 2018), sleep apnea episodes (Amelard al., 2018) and blood pressure (Luo et al., 2019). In the environmental remote sensing domain, periodic learning is often needed to enable nowcasting of euvironmental changes such as precipitation patterns land surface temper- ature (Sønderby et al., 2020). In the human behavior analysis domain, recovering the frequency of changes or the underlying temporal morphology in human motions (e.g., gait or hand motions) is crucial for those rehabilitating from surgery (Gu et al., 2019), for detecting the onset progres- sion of neurological conditions such as Parkinson's disease (Liu et al., 2022; Yang et al., 2022b).
While learning periodic targets is important, labeling such data is typically challenging and resource intensive. For example, if designing a method to measure heart rate, collecting videos with highly synchronized gold-standard signals from medical sensor is time consuming, labor intensive, and requires storing privacy sensitive bio-metric data. Fortunately, given the large amount of unlabeled data, self-supervised learning that captures the underlying periodicity in data would be promising.

---

Published as a conference paper at ICLR 2023
Yet, despite the great success of self- self-supervised learning (SSL) schemes on solving discrete classifi- cation or segmentation tasks, such as image classification (Chen et al., 2020; He et al., 2020), object detection (Xiao et al., 2021), action recognition (Qian et al., 2021), semantic labeling (Hu et al., 2021), less attention has been paid to designing algorithms that capture periodic or quasi-periodic temporal dynamics from data. Interestingly, we highlight that existing SSL methods inevitably over- look the intrinsic periodicity in data: Fig. 1 shows the UMAP (McInnes et al., 2018) visualization of learned representations on RotatingDigits, a toy periodic learning dataset that aims to recover the underlying rotation frequency of different digits (details in Section 4). As the figure shows, state-of- the-art (SOTA) SSL schemes fail to capture the underlying periodic or frequency information in the data. Such observations persist across tasks and domains as we show later in Section 4.
To fill the gap, we present SimPer, a simple self-supervised regime for learning periodic information in data. Specifically, to leverage the temporal properties of periodic targets, SimPer first introduces a temporal self-contrasttie learning framework, where positive and negative samples are obtained through periodicity-invarian and periodicity-variin variant augmentations from the same input instance. Further, we identify the problem of using conventional feature similarity measures (e.g., cos(-)) for periodic representation, and propose periodic feature similarity to explicitly define how measure similarity in the context of periodic learning. Finally, to harness the intrinsic continuity of augmented samples in the frequency domain, we design a generalized contrastive loss that extends the classic InfoNCE loss to soft regression variant that enables contrasting over continuous labels (frequency)
To support practical evaluation of SSL of periodic targets, we benchmark SimPer against SOTA SSL schemes on six diverse periodic learning datasets for common real-world tasks in human behavior analysis, environmental remote sensing, and healthcare. Rigorous experiments verify the robustness and efficiency of SimPer on learning periodic information in data. Our contributions are as follows:
We identify the limitation of current SSL methods on periodic learning tasks, and uncover intrinsic properties of learning periodic dynamics with sulf-suppriision over other mainstream tasks. We design SimPer, simple & effective SSL framework that learns periodic information in data.
We conduct extensive experiments six diverse periodic learning datasets in different domains: human behavior analysis, environmental sensing, and healthcare. Rigorous evaluations verify the superior performance of SimPer against SOTA SSL schemes.
Further analyses reveal intriguing properties of SimPer on its data efficiency, robustness to spuri- ous correlations & reduced training data, and generalization to unseen targets.
2 RELATED WORK
Periodic Tasks in Machine Learning. Learning or recovering periodic signals from high dimen- sional data is prevailing in real-world applications. Examples of periodic learning include recover- ing and magnifying physiological signals (e.g., heart rate or breathing) (Wu al., 2012), predicting weather and environmental changes (e.g.. nowcasting of precipitation or land surface temperatures) (Sønderby et al., 2020; Espeholt et al., 202 1), counting motions that are repetitious (e.g.. exercises or therapies) (Dwibedi et al., 2020; Ali et al., 2020), and analyzing human behavior (e.g., gait) (Liu et al., 2022). To date, much prior work has focused on designing customized neural architectures (Liu et al., 2020; Dwibedi et al., 2020), loss functions (Starke al., 2022), and leveraging relevant learning paradigms including transfer learning (Lu et al., 2018) and meta-learning (Liu et al., 2021) for periodic learning in a supervised manner, with high-quality labels available. In contrast these past work, we aim to learn robust & efficient periodic representations in a self-supervised manner.
Self-Supervised Learning. Learning with self-supervision has recently attracted increasing inter- ests, where early approaches mainly rely on pretext tasks, including exemplar classification (Doso- vitskly al., 2014), solving jigsaw puzzles (Noroozi & Favaro, 2016), object counting (Noroozi et al., 2017), clustering (Caron et al., 2018), and predicting image rotations (Gidaris et al., 2018). More recently, a line of work based on contrastive losses (Oord et al., 2018; Tian et al., 2019; Chen et al., 2020; He et al., 2020) shows great success in self-supervised representations, where similar embeddings are learned for different views of the same training example (positives), and dissimilar embeddings for different training examples (negatives). Successful extensions have been made to temporal learning domains including video understanding (Jenni et al., 2020) or action classification (Qian al., 2021). However, current SSL methods have limitations in learning periodic information, as the periodic inductive bias is often overlooked in method design. Our work extends existing SSL frameworks to periodic tasks, and introduces new techniques suitable for learning periodic targets.
2

---

Published as a conference paper at ICLR 2023
Periodicily Invariant
SimPer
0.8x
f
0.8x
1x
1.5x
Pariodicity Variant
Variaan
Input J
T
1x
(1x)
0.8x
1x
1. 1.5x
1x
Simiar
Moderately Similar
1.5x
Perlodic Similrity
Dissimilr
Figure 2: An overview of the SimPer framework. Input sequence is first passed through periodicity-vand vvrrinnt transformations T( T(.) to create series of speed (frequency) changed samples, where each augmented sample ex- hibits different underlying periodic signals due to the altered frequency, and can be treated as negative examples for each other. The augmented series are then passed through two sets of periodicity invariint transformations (-)) to create different invariant views (positives). All samples are then encoded in the feature space through shared encoder f-)). The SimPer loss calculated by contrasting over continuous speed (frequency) labels of different feature vectors, using customized periodic feature similarity measures.
THE SIMPER FRAMEWORK
When learning from periodic data in a self-supervised manner, fundamental question arises:
How do we design a self- supervisd task such that periodic inductive biases are exploited?
We note that periodic learning exhibits characteristics that are distinct from prevailing learning tasks. First, while most efforts exploring invariances engineer transformations in the spatial (e.g.. image recognition) or temporal (e.g.. video classification) domains, dynamics in the frequency domain are essential in periodic tasks, which has implications for how we design (in)variances. Second, unlike conventional SSL where a cosine distance is typically used for measuring feature similarity, repre- sentations learned for repetitious targets inherently possess periodicity that insensitive certain shifts (e.g., shifts in feature index), which warrants new machinery for measuring periodic similarity. Third, labels of periodic data have a natural ordinality and continuity in the frequency domain, which inspires the need for strategies beyond instance discrimination, that contrast over continuous targets.
We present SimPer (Simple SSL of Periodic Targets), unified SSL framework that addresses each of the above limitations. Specifically, SimPer first introduces a temporal self-contrastive learning scheme, where we design parriodcitt-invariien input instance to create its effective positive and negative views in the context of periodic learning (Section 3.1). Next, SimPer presents periodic featuurr similarity to explicitly define how one should measure the feature similarity when the learned representations inherently possess periodic infor- mation (Section 3.2). Finally, in order to exploit the continuous nature of augmented samples in the frequency domain, we propose generalized contrastive loss that extends the classic InfoNCE loss (Oord et al., 2018) from discrete instance discrimination to continuous contrast over frequencies, which takes into account the meaningful distance between continuous labels (Section 3.3).
and periodicity-varian
augmentations for the same
3.1 TEMPORAL
LEARNING FRAMEWORK
{xx+)) be the unlabeled training set, where x; E RRP denotes the input 8) the representation of x, where f(::8) is parameterized by a deep neural network with parameter 0. To preserve the full temporal dynamics and information, typically extracts frame-wise feature of x, i.c., z has the same length as input x.
Problem Setup. Let D
sequence. We denote as z = f(x;
As motivated, notivated,,reequecyy information is most essential when learning from periodic data. Precisely, augmentations that change the underlying frequency effectively alter the identity of the data (period- icity), and vice versa. This simple insight has implications for how we design proper (in)variances.
Periodicity- Variant Augmentations. We construct negative views of data through transformations in the frequency domain. Specifically, given input sequence x, we define periodicity-vane aug-
3

---

Published as a conference paper at ICLR 2023
mentations T, where T represents the set of transformations that change x with an arbitrary speed that is feasible under the Nyquist sampling theorem. As Fig. 2 shows, SimPer augments x by M times, obtaining series of speed (frequency) changed samples (x1, 72())),.... TM (x) }, whose relative speeds satisfy 81 < 82 < < 8M, 8; xx free();(x))- Such augmentation effectively changes the underlying periodic targets with shifted frequencies, thus creating different negative views. Therefore, although the original target frequency is unknown, we effectively devise pseudo speed (frequency) labels for unlabeled x. In practice, we limit the speed change range to be within (min Smax), ensuring the augmented sequence is longer than a fixed length in the time dimension.
Augmentations. We further define periodicity periodicity-invariant invariant augmentation o e S, where S denotes the set of transformations that do not change the identity of the original input.
Periodicity-Invarii
When the set is finite, i.e., S = (o .k]] we have (x)) Such augmentations can be used to learn invariances in the data from the perspective of periodicity,
fa (x)), Vi.j E [R:]
creating different positive views. Practically, we leverage spatial (e.g., crop & resize) and temporal
reverse, delay) augmentations to create different views of the same instance (see Fig. 2).
Temporal Self-Contrastive Learning. Unlike conven- tional contrastive SSL schemes where augmentations are exploited to produce invariances, i.c., creating different positive views of the data, SimPer introduces periodicity- variant augmentations to explicitly model what variances should be in periodic learning. Concretely, negative views are no longer from other different instances, but directly from the same instance itself, realizing a self-contrastive scheme. Table details the differences.
Table 1: Differences of view constructions.
Algoritho
Pusitives
Negalives
Coneestional
Instance: Same
Instance: Different
SSL methods
Aug: Tevariont
Aug: Inveriane
Instamce: Same
Instance: Stme
SimPer
Ang.: Ievaoat
Aug: Period Vaarinat
We highlight the benefits of using the self-contrastive framework. First, it provides arbitrarily large negative sample sizes, as long as the Nyquist sampling theorem is satisfied. This makes SimPer not dependent on the actual training set size, and enables effective contrasting even under limited data scenarios. We show in Section 4.2 that when drastically reducing the dataset size to only 5% of the total samples, SimPer still works equally well, substantially outperforming supervised counterparts. Second, our method naturally leads to hard negative samples, as periodic information directly being contrasted, while unrelated information (e.g.. frame appearance) are maximally preserved across negative samples. This makes SimPer robust to spurious correlations in data (Section 4.5).
3.2 FEATURE SIMILARITY IN THE CONTEXT OF PERIODIC LEARNING
We identify that feature similarity measures are also different in the context of periodic representa- tions. Consider sampling two short clips x1, X2 from the same input sequence, but with a frame shift 1. Assume the frequency does not change within the sequence, and its period T > t. Since the un- derlying information does not change. by definition their features should be close in the embedding space (i.e.. high feature similarity). However, due to the shift in time, when extracting frame-level feature vectors, the indexes of the feature representations (which represent different time stamps) will no longer be aligned. In this case, if directly using a cosine similarity as defined in conventional SSL literature, the similarity score would be low, despite the fact that the actual similarity is high.
Periodic Feature Similarity. To overcome this limitation, we propose to use periodic feature sim- ilarity measures in SimPer. Fig. 3 highlights the properties and differences between conventional feature similarity measures and the desired similarity measure in periodic learning. Specifically, ex- isting SSL methods adopt similarity measures that emphasize strict "closeness" between two feature vectors, and are sensitive to shifted or reversed feature indexes. In contrast, when aiming for learning periodic features, a proper periodic feature measure should retain high similarity for features with shifted (sometimes reversed) indexes, while also capturing continuous similarity change when the feature frequency varies, due to the meaningful distance in the frequency domain.
Concrete Instantiations. We provide two practical instantiations effectively capture the periodic feature similarity. Note that these instantiations can be easily extended to high-dimensional features (in addition the time dimension) by averaging across other dimensions.
Maximum cross-correlation (MXCorr) measures the maximum similarity as a function of offsets between signals (Welch, 1974), which can be efficiently computed in the frequency domain.
Normalized power spectrum density (nPSD) calculates the distance between the normalized PSD) of two feature vectors. The distance can be a cosine or L2 distance (details in Appendix D.4.3).
4

---

Published as a conference paper at ICLR 2023
(a) Conventional Feature Similarity (e.g, cosine)
(b) Periodic Feature Similarity
Anchor Feature
Anchor Feelure
High Similrity
Similanty
Same
Same
Low Simitntty
Incox Shifted
Index Shifted
Revereed
Reversed
Changed Frequency
Changed Frequency
Random
Random
Figure 3: Differences between (a) conventional feature similarity, and (b) periodic feature similarity. A proper periodic feature similarity measure should induce high similarity for features with shifted (sometimes reversed) indexes, while capturing continuous similarity change when the feature frequency varies.
3.3 GENERALIZED CONTRASTIVE Loss WITH CONTINUOUS TARGETS
Motivated by the fact that the augmented views are continuous in frequency. where the pseudo speed labels { M are known through augmentation (i.c., a view at 1.1x is more similar to the original than that at 2x), we relax and extend the original InfoNCE contrastive loss (Oord al., 2018) to a soft variant, where it generalizes from discrete instance discrimination to continuous targets.
From Discrete Instance Discrimination Continuous Contrast. The classic formulation of the InfoNCE contrastive loss for each input sample x is written as
cxp (z. 2)/v)
LIiONCE = - log Ez'EZ(()) exp(sim(z, x/)///
where 2 = (X) (X is the positive pair of x obtained through augmentations), 2 is the set of features in current batch, // is the temperature constant, and sim(-) is usually instantiated by a dot product. Such format indicates a hard classification task, where target label is 1 for positive pair and 0 for all negative pairs. However, negative pairs in SimPer inherently possess a meaningful distance, which reflected by the similarity of their relative speed (frequency). To capture this intrinsic continuity, we consider the contributions from all pairs, with each scaled by the similarity in their labels.
Generalized InfoNCE Loss. For input sample x, SimPer creates M variant views with different Given the features of two sets of invariant views [ 1= ( we have
speed labels { 1==1-
M.
I=1:
SsimPer Σ Summer Σ-
exp(wx.s)
log
exp(sim(2; 2,)/o)
Wyy == (81, 83),
M exp(w;,)
Σ 1m=1 exp(sim (2,,, 2/v/²²
(2)
where sim(-) denotes the periodic feature similarity as described previously, and simmael))) denotes the continuous label similarity measure. In practice, simaabal)) can be simply instantiated as inverse of the L1 or L2 label difference (c.g., 1/|s: //)):
Interpretation. LSimPer is a simple generalization of the InfoNCE loss from discrete instance dis- crimination (single target classification) to a weighted loss over all augmented pairs (soft regression variant), where the soft target exp( (w.)) )/ //Ex exp(w:.:) is driven by the label (speed) similarity wa, of each pair. Note that when the label becomes discrete (i.e., Wa E {0. 1)), Lsimerr degenerates to the original InfoNCE loss. We demonstrate in Appendix D.4.4 that such continuity modeling via a generalized loss helps achieve better downstream performance than simply applying InfoNCE.
EXPERIMENTS
Datasets. We perform extensive experiments six datasets that span different domains and tasks. Complete descriptions each dataset are in Appendix B, Fig. 7, and Table 11.
RotatingDigits (Synthetic Dataset) is a toy periodic learning dataset consists of rotating MNIST digits (Deng, 2012). The task is to predict the underlying digit rotation frequency.
SCAMPS (Human Physiology) (McDuff et al., 2022) consists of 2,800 synthetic videos of avatars with realistic peripheral blood flow. The task is to predict averaged heart rate from input videos.
UBFC (Human Physiology) (Bobbia et al., 2019) contains 42 videos with synchronized gold- standard contact PPG recordings. The task is predict averaged heart rate from input video clips.
5

---

Published as a conference paper at ICLR 2023
Table Feature evaluation results on RotatingDigits.
Table 3: Feature evaluation results on SCAMPS.
FFT
1-NN
FFT
L-NN
Metrics
MAE-
MAPE
MAE!
MAPE
Metries
MAE
MAPE
MAE
MAPE
SIMCLR (Chen 2020)
2.96
109.27
0.98
48.30
SIMCLR (Chen al., 2020)
27.48
38.39
34.09
40.79
MoCo v2 (He al., 2020)
2.83
90.78
0.62
32.74
MoCo v2 (He al., 2020)
28.16
40.23
35.61
42,47
BYOL (Grill et al., 2020)
2.20
78.43
0.46
22.08
BYOL (Grill al., 2020)
26.15
37.34
32.77
38.26
CVRL (Qian et al., 2021)
1.69
49.09
0.38
14.41
CVRL (Qian et al., 2021)
27.67
38.80
33,32
39.54
SIMPER
0.22
16.49
0.09
4.51
SIMPER
14.45
22.09
13.75
18.64
GAINS
+1,47
+32.60
+0.29
+9,90
GAINS
+11.70
+15.25
+19.02
+1.62
Table 4: Feature evaluation results on UBFC.
Table 5: Feature evaluation results on PURE.
FFT
I-NN
I-NN
Metries
MAE
MAPE
MAE+
MAPE
Metries
MAE
MAPE
MAE
MAPE
SIMCLR (Chen et 2020)
16.92
14.73
16.23
18.62
SIMCLR (Chen et al., 2020)
23.70
22.07
29.48
31.44
MoCo v2 (He et al., 2020)
14.64
13.17
15.12
16.56
MoCo v2 (Hc et al., 2020)
24.23
24.08
30.82
33.95
BYOL (Grill et al,, 2020)
17.86
16.90
18.13
19.34
BYOL (Grill al., 2020)
23.24
21.78
29.27
31.03
CVRL (Qian et al., 2021)
11.75
10.67
12.36
13.38
CVRL (Qian et al., 2021)
19.27
18.94
22.08
23.75
SIMPER
8.78
7.46
8.92
10.21
SIMPER
13.97
12.88
14.03
15.35
GAINS
+2.97
+3.21
+3,44
+3.17
GAINS
+5.30
+6.06
+8.05
+8.40
Table 6: Feature evaluation results Countix.
Table 7: Feature evaluation results on LST.
FFT
I-NN
Linear Prohing
Metrics
MAE
GM
MAE-
GM-
Metrics
MAE-
MAPE
SIMCLR (Chen et al., 2020)
3,90
2.26
4.43
3.19
SIMCLR (Chen al., 2020)
5.12
0.20
0.89
MoCo v2 (He al., 2020)
3.75
2.18
3,96
3.04
MoCo v2 (He al., 2020)
5.16
0.20
0.89
BYOL (Grill et al., 2020)
3.26
1.87
3.72
2.66
BYOL (Grill et ul., 2020)
5.71
0.24
0.86
CVRL (Qian et al., 2021)
2.81
1.38
3.15
2.12
CVRL (Qian et al., 2021)
4.88
0.18
0.91
SIMPER
2.06
0.98
2.76
1.84
SIMPER
4.84
0.18
0.90
GAINS
+0.75
+0.40
+0.99
+0,28
GAINS
+0.04
+0.00
-0.01
PURE (Human Physiology) (Stricker et al., 2014) contains 60 videos with synchronized gold- standard contact PPG recordings. The task is predict averaged heart rate from input video clips.
Countix (Action Counting). The Countix dataset (Dwibedi et al., 2020) a subset of the Kinetics (Kay al., 2017) dataset annotated with segments of repeated actions and corresponding counts. The task is to predict the count number given an input video.
Land Surface Temperature (LST) (Satellite Sensing). LST contains hourly land surface tempera- ture maps over the continental United States for 100 days (April 7th to July 16th, 2022). The task is predict future temperatures based on past satellite measurements.
Network Architectures. We choose a set of logical architectures from prior work for our experi- ments. On RotatingDigits and SCAMPS, we employ a simple 3D variant of the CNN architecture as in (Yang et al., 2022a). Following (Liu et al., 2020), we adopt variant of TS-CAN model for ex- periments on UBFC and PURE. Finally, on Countix and LST, we employ ResNet-3D-18 (He et al., 2016; Tran al., 2018) as our backbone network. Implementations details are in Appendix C.
Baselines. We compare SimPer to SOTA SSL methods, including SimCLR (Chen et al., 2020), MoCo v2 (He et al., 2020), BYOL (Grill et al., 2020). and CVRL (Qian et al., 2021), well supervised learning counterpart. We provide detailed descriptions in Appendix C.I.
Evaluation Metrics. To assess the prediction of continuous targets (c.g., frreuuency counts), common metrics for regression, such as the mean-average-error (MAE), cane--avrrage-pereeeteee error (MAPE), Pearson correlation (p), and error Geometrie Mean (GM) (Yang et al., 2021).
4.1
MAIN RESULTS
We report the main results in this section for all datasets. Complete training details, hyper-parameter settings, and additional results are provided in Appendix C and D.

---

Published as a conference paper at ICLR 2023
Table 8: Fine-tune evaluation results on all datasets. We first pre-train the feature encoder using different SSL.
methods, then fine-tune the whole network initialized with the pre-trained weights.
RotatingDigits
SCAMPS
UBFC
PURE
Countix
LST
Metries
MAE
MAPE
MAB
MAPE-
MAE
MAPE
MAE
MAPE!
MAE
GM
MAE
p²
SUPERVISED
0.72
28.96
3.61
5.33
5.13
4.72
4.25
4.93
1.50
0.73
1.54
0.96
SIMCLR (Chen al., 2020)
0.69
26.54
4.96
6.92
5.32
4.96
4.86
5.32
1.58
0.80
1.54
0.95
MoCo v2 (He et al., 2020)
0.64
24.73
5.33
7.24
5.05
4.64
4.97
5.60
1.54
0.79
1.53
0.95
BYOL (Grill et al., 2020)
0.39
20.91
3.49
5.27
5,51
5.07
4.28
4.97
1.47
0.71
1.62
0.92
CVRI. (Qian et al,, 2021)
0.34
18.82
5.52
7.34
5.07
4.70
4.19
4.71
1.48
0.71
1.49
0.96
SIMPER
0.20
14.33
3.27
4.89
4.24
3.97
3.89
4.01
1.33
0.59
1.47
0.96
GAINS vs. SUPERVISED
+0.52
+14.63
+0.34
+0.44
+0.89
+0.75
+0.36
+0.92
+0.17
+0.14
+0.07
+0.00
Reduced training dataset size
Maan Average Parcemage (1)
100%
50%
20%
10%
55%
0%
50%
SePa
Supervised
100%
SimCLR
SimPer
(a) Learned representations under different training dataset sizes
(b) Quantitative error changes
Figure 4: Data efficiency analysis of SimPer. (a) Learned representations of different algorithms on Rotat- Digits when training dataset size reduces from 100% to 5%. (b) The quantitative MAPE errors on SCAMPS with varying training dataset sizes. Complete quantitative results are provided in Appendix D.
Feature Evaluation. Following the literature (He et al., 2020; Chen et al., 2020), we first evaluate the representations learned by different methods. For dense prediction task (e.g., LST), we use the linear probing protocol by training linear regressor top of the fixed features. For tasks whose targets are frequency information, we directly evaluate the learned features using Fourier transform (FFT) and a nearest neighbor classifier (1-NN). Table 3, 5, show the feature evaluation results of SimPer compared to SOTA SSL methods. As the tables confirm, across different datasets with various common tasks, SimPer able learn better representations that achieve the best performance. Furthermore, in certain datasets, the relative improvements are even larger than 50%.
Fine-tuning. Practically, to harness the power pre-trained representations, fine-tuning the whole network with the encoder initialized using pre-trained weights is a widely adopted approach (He al., 2020). To evaluate whether SimPer pre-training is helpful for each downstream task, we fine-tune the whole network and compare the final performance. The details of the setup for each dataset and algorithm can be found Appendix C. As Table confirms, across different datasets, SimPer consistently outperforms all other SOTA SSL methods, and obtains better results compared to the supervised baseline. This demonstrates that SimPer is able to capture meaningful periodic information that is beneficial to the downstream tasks.
4.2 DATA EFFICIENCY
In real-world periodic learning applications, data is often prohibitively expensive obtain. To study the data efficiency of SimPer, we manually reduce the overall size of RotatingDigits, and plot the representations learned well as the final fine-tuning accuracy of different methods in Fig. 4.
As the figure confirms, when the dataset size is large (c.g., using 100% of the data), both supervised learning baseline and SimPer can learn good representations (Fig. 4(a)) and achieve low test errors
7

---

Published as a conference paper at ICLR 2023
Interpolation
Extrapolation
Label distribtion
Supervisd
SimPar
Label distrbution
Supavised
SinPur
Thain Sel
Test Set
Figure 5: Zero-shot generalization analysis. We create training sets with missing target frequencies and keep evenly distributed across the target range. Green regions indicate successful generalization with high frequency resolution. Ycllow regions indicate successful generalization but with low frequency resolution. Red regions represent failed generalization. SimPer learns robust representations that generalize to unseen targets.
(Fig. 4(b)). However, when the training dataset size becomes smaller, the learned representations using supervised learning get worse, and eventually lose the frequency information and resolution when only 5% of the data is available. Correspondingly, the final error in this extreme case also becomes much higher. In contrast, even with small number of training data, SimPer can consistently learn the periodic information and maintain high frequency resolution, with significant performance gains especially when the available data amount is small.
4.3 TRANSFER LEARNING
We evaluate whether the self-supervised representa- tions transferable across datasets. We use UBFC and PURE, which share the same prediction task. Following (Chen al., 2020), fine-tune the pre- trained model on the new dataset, and compare the performance across both SSL and supervised meth- ods. Table 9 reports the results, where in both cases, SimPer is able to achieve better final performance compared to supervised and SSL baselines, showing its ability to learn transferable periodic representations across different datasets.
Table Transfer learning results.
UBFC - PURE
PURE + UBFC
Metrics
MAR-
MAPE
MAR
MAPE
SUPERVISED
7.83
8.85
3.15
3.11
SIMCLR
7.86
3.46
3.80
SIMPER
6.46
6.98
2.76
2.38
GAINS
+1.37
+1.87
+0.39
+0.73
4.4 ZERO-SHOT GENERALIZATION UNSERN TARGETS
Given the continuous nature of the frequency domain, periodic learning tasks can (and almost cer- tainly will) have unseen frequency targets during training, which motivates the need for target (fre- quency) extrapolation and interpolation. To investigate zero-shot generalization to unseen targets, we manually create training sets that have certain missing targets (Fig. 5), while making the test sets evenly distributed across the target range. As Fig. confirms, in the interpolation case, both super- vised learning and SimPer can successfully interpolate the missing targets. However, the quality of interpolation varies: For supervised learning, the frequency resolution is low within the interpolation range, resulting in mixed representations for wide missing range. In contrast, SimPer learns better representations with higher frequency resolution, which has desirable discriminative properties.
Furthermore, in the extrapolation case, in the lower frequency range, both methods extrapolate reason- ably well, with SimPer capturing higher frequency resolution. However, when extrapolating to a higher frequency range, the supervised baseline completely fails to generalize, with learned features largely overlapping with the existing frequency targets in the training set. In contrast, SimPer able to generalize robustly even for the higher unseen frequency range, demonstrating its effectiveness of generalization to distribution shifts and unseen targets. Quantitative results in Table 10 confirm the observations.
Table 10: Mean absolute error (MAF, 4) results for zero-shot generalization analysis.
Interpolation
Extrapolation
Scen
Unseen
Scen
Unseen
SUPERVISED
0.09
0.85
0.03
1.74
SIMPER
0.05
0.07
0.02
0.02
GAINS
+0.04
+0.78
+0.01
+1.72
8

---

Published as a conference paper at ICLR 2023
SimCLR
SimPer
Colored by neal largets (Tequensy)
Colored by spurious targets (digi sppcerance)
Colcred by real langets (froquency)
Colored by sparinu targets (dit appearance)
Train Sel
Test Sel
Figure 6: Robustness to spurious correlations. We make the target frequency spuriously correlated with digit appearances in training while removing this correspondence in test set. SimPer is able to capture underlying periodic information & learn robust representations that generalize. Quantitative results are in Appendix D.3.
4.5 ROBUSTNESS TO SPURIOUS CORRELATIONS
We show that SimPer is able deal with spurious correlations that arise in data, while existing SSL methods often fail to learn generalizable features. Specifically, RotatingDigits dataset naturally has a spurious target: the digit appearance (number). We further enforce this information by coloring different digits with different colors in (Arjovsky et al., 2019). We then construct spuriously correlated training set by assigning a unique rotating fiequeecyy range to a specific digit, i.c., [0.5Hz, 1Hz] for digit 0, [1H 1.5Hz] for digit 1, etc, while removing the spurious correlations in test set.
As Fig. verifies, SimCLR is easy to learn information that is spuriously correlated the training data, but not the actual target of interest (frequency). As result, the learned representations do not generalize. In contrast, SimPer learns the underlying frequency information even in the presence of strong spurious correlations, demonstrating its ability to learn robust representations that generalize.
4.6 FURTHER ANALYSIS AND ABLATION STUDIES
Amount of labeled data for fine-tuning (Appendix D.2). We show that when the amount of labeled data is limited for fine-tuning, SimPer still substantially outperforms baselines by large margin, achieving 67% relative improvement in MAE even when the labeled data fraction only 5%.
Ablation: Frequency augmentation range (Appendix D.4.1). We study the effects of different speed (frequency) augmentation ranges when creating veriodicity-varian views (Table 15). While proper range can lead to certain gains, SimPer is reasonably robust to different choices.
Ablation: Number of augmented views (Appendix D.4.2). We investigate the influence of differ- ent number of augmented views (i.e., M) in SimPer. Interestingly, we find SimPer is surprisingly robust different M in given range (Table 16), where larger M often delivers better results.
Ablation: Choices of different similarity metries (Appendix D.4.3), We explore the effects of different periodic similarity measures in SimPer, where we show that SimPer is robust all afore- mentioned periodic similarity measures, achieving similar performances (Table 17).
Ablation: Effectiveness of generalized contrastive loss (Appendix D.4.4). We confirm the effec- tiveness of the generalized contrastive loss by showing its consistent performance gains across all six datasets, as compared to the vanilla InfoNCE loss (Table 18).
CONCLUSION
We present SimPer, simple and effective SSL framework for learning periodic information from data. SimPer develops customized periodicity-variinne and invariant augmentations, periodic feature similarity, and generalized contrastive loss to exploit periodic inductive biases. Extensive experi- ments on different datasets over various real-world applications verify the superior performance of SimPer, highlighting its intriguing properties such better efficiency, robustness & generalization.

---

Published as a conference paper at ICLR 2023
REFERENCES
Mohammad Rafayet Ali, Javier Hernandez, E Ray Dorsey, Ehsan Hoque, and Daniel McDuff. Spatio-temporal attention and magnification for classification of parkinson's disease from videos collected via the internet. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 207-214. IEEE, 2020.
Robert Amelard, Kaylen/ Pfisterer, Shubh Jagani, David Clausi, and Alexander Wong. Non- contact assessment of obstructive sleep apnea cardiovascular biomarkers using photoplethysmog- raphy imaging. In Optical Diagnostics and Sensing XVIII: Toward Point-of-Care Diagnostics, volume 10501, pp. 225 229. SPIE, 2018.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:190.002850 2019.
Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. Unsu- pervised skin tissue segmentation for remote photoplethysmography. Puttern Recognition Letters, 124:82-90, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pp. 1597-1607. PMLR, 2020.
Danish Contractor, Daniel McDuff, Julia Katherine Haines, Jenny Lee, Christopher Hines, Brent Hecht, Nicholas Vincent, and Hanlin Li. Behavioral use licensing for responsible 2022 ACM Conference on Fairness, Accountability, and Transpurency, pp. 778-788, 2022.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 19(6):1411144 2012.
Alexey Dosovitskiy, Jost Tobias Springenherg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NIPS, pp. 766-774, 2014.
Debidatta Dwibedi. Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Counting out time: Class agnostic video repetition counting in the wild. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10387-10396, 2020.
Lasse Espeholt, Shreya Agrawal, Casper Sønderby, Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Jason Hickey, Aaron Bell, and Nal Kalchbrenner. Skillful twelve hour precipitation forecasts using large context neural networks. arXiv preprint arXiv:21 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0777 2018
John Gideon and Simon Stent. The way to my heart is through contrastive learning: Remote photo- plethysmography from unlabelled video. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 3995-4004, 2021.
Bran-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, 2020.
Yiwen Gu, Shreya Pandit, Elham Saraee, Timothy Nordahl, Terry Ellis, and Margrit Betke. Home- based physical therapy with an interactive computer vision system. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Hanzhe Hu, Jinshi Cui, and Liwei Wang. Region-aware contrastive learning for semantic seg- mentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16291-16301, 2021.
Simon Jenni, Givi Meishvili, and Paolo Favaro. Video representation learning by recognizing tem- poral transformations. In European Conference on Computer Vision, pp. 425-442. Springer, 2020.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: (tti::700.06995 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4/2.6980 2014.
Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention net- works for on-device contactless vitals measurement. Advances in Neural Information Processing Systems, 11119000-1941,, 2020.
Xin Liu, Ziheng Jiang, Josh Fromm, Xuhai Xu, Shwetak Patel, and Daniel McDuff. Metaphys: few- shot adaptation for non-contact physiological measurement, In Proceedings of the conference health, inference, and learning, pp. 154-163, 2021.
Xin Liu, Brian L. Hill, Ziheng Jiang, Shwetak Patel, and Daniel McDuff. Efficientphys: Enabling simple, fast and accurate camera-based vitals measurement. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
Yingcheng Liu, Guo Zhang, Christopher G. Tarolli, Rumen Hristov, Stella Censen-Robert Emma M. Waddell, Taylor L. Myers, Meghan E. Pawlik, Julia M. Soto, Renee M. Wilson, Yuzhe Yang, Timothy Nordahl, Karlo J. Lizarraga, Jamie L. Adams, Ruth B. Schneider, Karl Kieburtz, Terry Ellis, E. Ray Dorsey, and Dina Katabi. Monitoring gait home with radio waves in parkin- 2019# disease: A marker of severity, progression, and medication response. Science Translational Medicine, 14(663)xaadc966,, 2022. doi: 10.1 1266cctranslmmed.dd666666
Erika Lu, Weidi Xie, and Andrew Zisserman. Class-agnostic counting. In Asian conference on computer vision, pp. 669-684. Springer, 2018
Hong Luo, Deye Yang, Andrew Barszezyk, Naresh Vempala, Jing Wei, Si Jia Wu, Paul Pu Zheng, Genyue Fu, Kang Lee, and Zhong-Ping Feng. Smartphone-based blood pressure measurement using transdermal optical imaging technology. Circulation: Cardiovascular Imaging, 12(8): c008857, 2019.
Daniel McDuff, Miah Wander, Xin Liu, Brian I. Hill, Javier Hernandez, Jonathan Lester, and Tadas Baltrusaitis. Scamps: Synthetics for camera measurement physiological signals. arXiv preprint 1rX1i:2206.04197, 2022.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 180.00330, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In ICCV, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 2018.
Rui Qian, Tianjian Meng, Boqing Gong, Ming Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6964 6974, 2021.
Boging
11

---

Published as a conference paper at ICLR 2023
Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek, Mostafa Dehghani, Avital Oliver, Tim Sal- imans, Shreya Agrawal, Jason Hickey, and Nal Kalchbrenner. Metnet: A neural weather model for precipitation forecasting. arXiv preprint arXiv:2003.12140, 2020.
Scbastian Starke, Ian Mason, and Taku Komura. Deepphase: periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4))1-13, 2022.
Ronny Stricker, Steffen Müller, and Horst-Michael Gross. Non-contact video-based pulse rate mea- surement on mobile service robot. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pp. 1056-1062. IEEE, 2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv:1906.05845 2019.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Hao Wang, Euijoon Ahn, and Jinman Kim. Self-supervised representation learning framework for remote physiological measurement using spatiotemporal augmentation loss. AAAI, 2022.
Lloyd Welch. Lower bounds on the maximum cross correlation signals (corresp.). IEEE Trans- actions Information theory, 20(3):397-399 1974.
Erroll Wood, Tadas Baltrušaitis,, Charlie Hewitt, Sebastian Dziadzio, Thomas Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild using synthetic data alone. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 1681-3691, 2021.
Hao-Ya Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William Freeman. Eulerian video magnification for revealing subtle changes in the world. ACM transactions graphics (TOG), 1-): 2012.
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10539-10548, 2021.
Bryan P Yan, William HS Lai, Christy KY Chan, Stephen Chun-Hin Chan, Lok-Hei Chan, Ka-Ming Lam, Ho- Wang Lau, Chak-Ming Ng, Lok- Yin Tai, Kin-Wa Yip, et al. Contact-free screening of atrial fibrillation by smartphone using facial pulsatile photopltthysmorrphhi signals. Journal of the American Heart Association, 7())e800558, 2018.
Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, and Dina Katabi. Delving into deep imbal- anced regression. In International Conference Machine Learning (ICML), 2021.
Yuzhe Yang, Hao Wang, and Dina Katabi. On multi-domain long-tailed recognition, imbalanced domain generalization and beyond. In European Conference on Computer Vision (ECCV), 2022a.
Yuzhe Yang, Yuan Yuan, Guo Zhang, Hao Wang, Long-Cog Chen, Yingcheng Liu, Christopher G Tarolli, Daniel Crepeau, Jan Bukartyk, Mithri Junna, al. Artificial intelligence-enabled detec- tion and assessment of parkinson's 's disease using nocturnal breathing signals. Nature Medicine, pp. 1-9, 2022b.
12

---

Published as a conference paper at ICLR 2023
A
PSEUDO CODE FOR SIMPER
We provide the pseudo code of SimPer in Algorithm 1.
Algorithm Simple Self-Supervised Learning of Periodic Targets (SimPer)
Input: Unlabeled training set = {(x;)] total training epochs E, periodicity- augme- ntations T(-) periodicity-inven innvariitt augmentations a(-), number of variant views M, encoder f for 0 to do repeat
Sample mini-batch batch {(x())] from D
for = 1 to n (in parallel) do
{x ( Jin + T (x()
// M variant views for 1-th sample // two sets of invariant views
{x MM {x, 1M + {xx (x)
(2, MEi (2/00 Ji=l + f (xx JM
(x (1) 1
Calculate (1)) end for
'SimPer for 1-th sample using {2, Ji=l {2, (1)
1M
based on Eqn. (2)
Calculate LsimPer using 1 Ei=1
(L)
PemPer and do one training step
until iterate over all training samples at current epoch end for
B DATASET DETAILS
In this section, provide the detailed information of the six datasets used in experiments. Fig. 7 shows examples of each dataset, and Table 11 provides the statistics of each dataset.
(Syntheric Dataset). We create RotatingDigits, synthetic periodic learning dataset of rotating MNIST digits (Deng, 2012), where samples are created with the original digits rotating on plain background at rotational frequencies between 0.5Hz and 5Hz. The training set consists of 1, 000 rotating video clips (100 samples per digit number), each sample with a frame length of 150 and a sampling rate of 30Hz. The test consists of 2, 000 rotating video clips (200 samples per digit number).
RotatingDigits
SCAMPS (Hummn Physiology). The SCAMPS dataset (McDuff et al., 2022) contains 2, 800 syn- thetic videos of avatars with realistic peripheral blood flow and breathing. The faces are synthe- sized using blendshape-based rig with 667 vertices and 7, 414 polygons and the identity basis is learned from a set of high-quality facial scans. These texture maps were sampled from 511 facial seans of subjects. The distribution of gender, age and ethnicity of the subjects who provided the facial scans be found in (Wood et al., 2021). Blood flow is simulated by adjusting properties of the physically-be based shading material', We randomly divide the whole dataset into training (2, 000 samples), validation (400 samples), and test (400 samples) set. Each video clip has frame length of 600 and a sampling rate of 30Hz.
UBFC (Human Physiology). The UBFC dataset (Bobbia al., 2019) contains a total of 42 videos from 42 subjects. The videos were recorded using a Logitech C920 HD Pro at 30Hz. A pulse oximeter was used to obtain the gold-standard PPG data (30Hz). The raw resolution is 640 480 and videos are recorded in a uncompressed 8-bit RGB format. We postprocess the videos by cropping the face region and resizing them 36 36. We manually divide each video into non-overlapping chunks (Liu et al., 2020) with window size 180 frames (6 seconds). The resulting number of training and test samples are 518 and 106, respectively.
PURE (Human Physiology). The PURE dataset (Stricker et al., 2014) includes 60 videos from 10 subjects (8 male, female). The subjects were asked to seat front the camera at an average distance of 1.1 meters and lit from the front with ambient natural light through window. Each subject was then instructed to perform six tasks with varying levels of head motion such as slow/fast translation between camera plane and head motion as well as small/medium head rotations. Gold-
'httpa://ww blender .org/
13

---

Published as a conference paper at ICLR 2023
saa0
(a) RitatingDigits
(b) SCAMPS (McDuff et al., 2022)
t2 April 2022
13 2022
14th April 2022
15th 2022
(c) Land Surface Temperature (LST)
(d) UBFC (Bobbia et al., 2019)
(e) PURE (Stricker et al., 2014)
(f) Countix (Dwibedi et al., 2020)
Figure 7: Examples of sequences from the datasets used in our experiments.
standard measurements were collected with pulse oximeter 60Hz. The raw video resolution is 640 x 480. We postprocess the videos by cropping the face region and resizing them to 36 x 36, and downsample the ground-trutt truth PPG signal to 30Hz from 60Hz. We manually divide each video into non-overlapping chunks (Liu et al., 2020) with a window size 180 frames (6 seconds). The resulting number of training and test samples are 1, 028 and 226, respectively.
14

---

Published as a conference paper at ICLR 2023
Table 11: Detailed statistics of the datasets used in our experiments.
Targets
Sampling freq.
Frame length
Training set
Val. set
Test set
RotatingDigits
Rotation frequency
30Hz
150
1,000
2,000
SCAMPS (McDuff' al., 2022)
Heart rate
30Hz
600
2,000
400
UBFC (Bobbia al., 2019)
Heart
30Hz
180
518
106
(Stricker al., 2014)
rate
180
1,028
Countix (Dwibedi al., 2020)
Action counts
20~30Hz
200
1,712
457
Temperature
Hourly
100
276
Countix (Action Counting). The Countix dataset (Dwibedi et al., 2020) subset of the Kinet- ies (Kay et al., 2017) dataset annotated with segments of repeated actions and corresponding counts. The creators crowdsourced the labels for repetition segments and counts for the selected classes. We further filter out videos that have frame length shorter than 200, and make all videos have fixed length of 200 frames. The resulting dataset has 1, 712 training samples, 457 validation samples, and 963 test samples, with resolution of 96 96.
Land Surface Temperature (LST) (Satellite Sensing). Land surface temperature is an indicator of the Earth surface energy budget and is widely required in applications of hydrology, meteorology of fundamental importance to the net radiation budget at the Earth's surface and for monitoring the state of crops and vegetation, the greenhouse effect and the energy flux between the atmosphere and earth surface. We created snapshot of data from the NOAA GOES-16 Level LST product comprising of hourly land surfaces temperature outputs over the continental United States (CONUS). The LST measurements are sampled hourly over a 100 day period leading to 400 LST maps at a resolution of 1, 500 x 2, 500. As the spatial resolution is high, we divide each map into four quarters Northh-est US, North-East US, South-Wess US, and South-Ease window size of 100 frames with step size of 24 (a day). The target signal is the temperature time series of the future 100 frames. The resulting dataset has 276 training samples and 92 test samples, with spatial resolution of 100 100.
and climatology.
well
important indicator of both
US). We create each input sample using
C EXPERIMENTAL SETTINGS
C.1 COMPETING ALGORITHMM
We employ the following state-of-the-art SSL algorithms for comparisons.
SimCLR (Chen et al., 2020). SimCLR learns feature representations by contrasting images with data augmentation. The positive pairs are constructed by sampling two images with different aug- mentations on one instance. The negative pairs are sampled from two different images.
MoCo (He al., 2020). MoCo learns feature representations by building large dictionaries along with a contrastive loss. MoCo maintains the dictionary as queue of data samples by enqueuing the current mini-batch and dequeuing the oldest mini-batch. The keys are encoded by a slowly progressing encoder with momentum moving average and the query encoder.
BYOL (Grill et al., 2020). BYOL leverages two neural networks learn the feature representations: the online and target networks. The online network has encoder, a projector, and a predictor while the target network shares the same architecture with different set of weights. The online network is trained by the regression targets provided by the target network.
CVRL (Qian et al., 2021) 1). Contrastive Video Representation Learning (CVRL) is self-supervised learning framework that learns spatial-temporal features representations from unlabelled videos. CVRL generates positive pairs by adding temporally consistent spatial augmentation on one videos clip and generate negative pairs by sampling different video clips. The goal of contrastive loss to minimize the embedding distance from the positive augmented video clips but maximize the distance from negative video clips.
15

---

Published as a conference paper at ICLR 2023
C.2 IMPLEMENTATION DETAILS
We describe the implementation details in this section. We first introduce parameters that are fixed to be the same across all methods, then detail the specific parameters for each dataset.
For all SSL methods, we follow the literature (Chen al., 2020; He al., 2020) and apply the same standard data augmentations in contrastive learning. For spatial augmentations, we employ random_ resiie random br brightssss, random random-gaussi an an_ blur, random grayscale, and For temporal augmentations, we mainly employ reverre and random_delay (with shorter clip subsampling (Qian et al., 2021)). Unless specified, all augmenta- tion hyper-parameters follow the original setup of each method.
RotatingDigits. On RotatingDigits, we adopt the network architecture as a simple 3D variant of the MNIST CNN used in (Yang et al., 2022a; Guirajani & Lopez-Paz, 2021). In the supervised setting, we train all models for 20 epochs using the Adam optimizer (Kingma Ba, 2014), with an initial learning rate of 10 and then decayed by 0.1 at the 12-th and 16-th epoch, respectively. We fix the batch size as 64 and use the checkpoint at the last epoch as the final model for evaluation. In the self-supervised setting, we train all models for 60 epochs, which ensures convergence for all tested algorithms. We again employ the Adam optimizer and decay the learning rating at the 40-th and 50-th epoch, respectively. Other training papeer-rramtterss remain unchanged.
SCAMPS. Similar to RotatingDigits, we employ the same 3D CNN architecture for all the SCAMPS experiments. In the supervised setting, we train all of the models for 30 epochs using the Adam optimizer, with an initial learning rate of 10-3 and then decayed by 0.1 at the 20-th and 25-th epoch, respectively. We fix the batch size as 32 and use the last checkpoint for final evaluation. In the self-supervised setting, we follow the same training regime in RotatingDigits as described in the previous section.
UBFC & PURE. Following (Liu et al., 2020; 2021), we use the temporal shift convolution attention network (TS-CAN) our backbone model. To adapt TS-CAN on SimPer, we remove the attention branch and make a variant of TS-CAN which only requires 3-channel as the input instead of 6- channel. In the supervised setting, we use the Adam optimizer, learning rate of 10-3 3 and train the network for total of 10 epochs. On the inner-dataset evaluation (i.e., test and validation are the same), we use last epoch from the training of 80% for the dataset and evaluate the pre-trained model on the last 20% dataset. On the cross-dataset evaluation, we use 80% of the dataset for training and 20% for checkpoint selection then evaluate the pre-trained model on a different dataset. In the self-supervised setting, all other parameters remain unchanged except that we train for 60 epochs to ensure the SSL. loss converges for all algorithms.
Countix. We use ResNet-3D- 18 (Tran et al., 2018; He et al., 2016) architecture for all Countix experiments, which is widely used for video-based vision tasks. In the supervised setting, we train all models for 90 epochs using the Adam optimizer with initial learning rate of 10-3 and then decayed by 0.1 at the 60-th and 80-th epoch. We fix the batch size as 32 for all experiments. In the self-supervised setting, we train all models for 200 epochs. and leave other parameters unchanged.
LST. Similar to Countix, we use the ResNet-3D-18 (Tran et al., 2018) network architecture for LST experiments. In the supervised setting, we train all models for 30 epochs using the Adam optimizer with a learning rate of 10 and a batch size of 16. In the self-supervised setting, we train all models for 60 epochs while having other hyper- ppper-parameters the same for all methods.
C.3 EVALUATION METRICS
We describe in detail all the evaluation metrics we used in our experiments.
MAE. The mean absolute error (MAE) is defined as N. Σ/ ly: til, which represents the averaged absolute difference between the ground truth and predicted values over all samples.
MAPE. The mean absolute percentage error (MAPE) is defined as 1
I -== which assesses
the averaged relative differences between the ground truth and predicted values over all samples.
GM. We use error Geometric Mean (GM) as another evaluation metric (Yang et al., 2021). GM is defined as (I ex)+,, where e, A |y Øil represents the J, error of each sample. GM aims to
16

---

Published as a conference paper at ICLR 2023
characterize the fairesss (uniformity) of model predictions using the geometric mean instead of the arithmetic mean over the prediction errors.
Pearson correlation We employ Pearson correlation for performance evaluation LST, where Pearson correlation evaluates the linear relationship between predictions and corresponding ground truth values.
D ADDITIONAL RESULTS AND ANALYSIS
D.I DATA EFFICIENCY T. REDUCED TRAINING DATA
We provide quantitative results to verify the data efficiency of SimPer in the presence of reduced training data. Specifically, we use SCAMPS dataset, and vary the training dataset size from 100% to only 5%. and use for both pre-training and fine-tuning. We show the final performance in Table 12, where SimPer is able achieve consistent performance gains compared baselines when the dataset size varies. Furthermore, the gains are more significant when the dataset size is smaller (e.g., 5%), demonstrating that SimPer is particularly robust to reduced training data.
Table 12: Data efficiency w.rt. reduced training data. We vary the training dataset size of SCAMPS (size fixed for both pre-training and fine-tuning tine-tuning), and show the final fine-tuning performance of different methods.
Dataset size
100%
50%
20%
10%
5%
Metries
MAE
MAPE
MAE-
MAPE
MAE
MAPE
MAE+
MAPE
MAE
MAPE
SUPERVISED
3.61
5.33
3.85
5.60
4.57
7.16
7.13
10.08
12.24
15.42
SIMCLR (Chen al., 2020)
4.96
6.92
6.55
9.39
6.01
9.25
7.63
10.19
13.75
15.72
CVRL (Qian al., 2021)
5.52
7.34
3.66
5.64
4.86
7.77
9.45
14.11
15.91
SIMPER
3.27
4.89
3.38
5.24
5.67
7.06
4.75
7.64
GAINS Vs. SUPERVISED
+0.34
+0.44
+0.47
+0.36
+0,64
+1.49
+2.48
+3.02
+7.49
+7.78
D.2 AMOUNT OF LABELED DATA FOR FINE-TUNING
We investigate the impact of the amount of labeled data for fine-tuning. Specifically, we the whole training set of SCAMPS as the unlabeled dataset, and vary the labeled data fraction for fine- tuning. As Table 13 confirms. when the amount labeled data limited for fine-tuning, SimPer still substantially outperforms baselines by a large margin, achieving 67% relative improvement in MAE even when the labeled data fraction is only 5%. The results again demonstrate that SimPer data efficient in terms of the amount of labeled data available.
Table 13: Data efficiency w.r.t. amount of labeled data for fine-tuning, We use all data from SCAMPS as unlabeled training set for self-supervised pre-taining. and vary size of labeled data for fine-tuning.
Labeled data fraction
100%
50%
20%
10%
5%
Metrics
MAE
MAPE
MAE-
MAPE
MAE4
MAPE
MAE
MAPE
MAE4
MAPE
SUPERVISED
3.61
5.33
3.85
5.60
4.57
7.16
7.13
10.08
12.24
15.42
SIMCLR (Chen et al., 2020)
4.96
6.92
4.92
7.09
5.57
8.46
7.82
10.53
13.21
15.64
CVRL (Qian al., 2021)
5.52
7.34
3.79
5.83
4.83
7.71
6.82
9.06
12.18
13.25
SIMPER
3.27
4.89
5.13
3.58
5.44
3.98
5.81
4.02
6.27
GAINS vs. SUPERVISED
+0.34
+0.44
+0.53
+0.47
+0.99
+1.72
+3.15
+4,27
+8.22
+9.15
D.3 ROBUSTNESS TO SPURIOUS CORRELATIONS
We provide detailed quantitative results for the spurious correlations experiment in Section 4.5.
Recall that SimCLR is easy learn information that is spuriously correlated in the training data,
17

---

Published as a conference paper at ICLR 2023
Table 14: Feature evaluation results on RotatingDigits with spurious correlations in training data. Quan- titative results addition to Fig. further verify that state-of-the-art SSI. methods (e.g., SimCIR) are vulner- able to spurious correlations, and could casily learn information that is irrelevant to periodicity; In contrast, SimPer learns desirable periodic representations that are robust to spurious correlations.
FFT
1-NN
Metrics
MAE
MAPE
MAE
MAPE
SIMCLR (Chen al., 2020)
3.06
125.48
1.49
80.28
SIMPER
0.36
15.04
0.78
27.03
GAINS
+2.70
110.44
+0.71
+53.25
and the learned representations do not generalize. Table 14 further confirms the observation, where SimCLR achieves bad feature evaluation results with large MAE & MAPE errors.
In contrast, SimPer is able to learn the underlying frequency information even in the presence of strong spurious correlations, obtaining substantially smaller errors compared to SimCLR. The sults demonstrate that SimPer is robust to spurious correlations, and can learn robust representations that generalize.
D.4 ABLATION STUDIES FOR SIMPER
In this section, we perform extensive ablation studies on SimPer to investigate the effect of different stability.
design choices as well as its
D.4.1 RANGE OF PERIODICITY-VARII FREQUENCY AUGMENTATION
We study the effect of using different ranges of the variant speed augmentations in SimPer. We use the SCAMPS dataset, and vary the speed range during SimPer pre-training. As Table 15 reports, using different speed ranges does not change the downstream performance by much, where all the results outperform the supervised baseline by notable margin.
Table 15:
Ablation study on range speed (frequency) augmentation. Default settings used the main experiments for SimPer are marked gray.
SPEED RANGE
[0.5, 1.5]
[0.8,
1.8]
[0.5, 2]
[0.5,3]
Supervised
MAPE
4.97
4.92
4.89
4.98
5.33
D.4.2 NUMBER OF PERIODICITY- VARIANT AUGMENTED VIEWS
We study the effect of different number of periodicity-varian variant augmented views M on SimPer. We again employ the SCAMPS dataset, and vary the number of augmented views as M {3, 5, 10, 20). Table 16 shows the results, where we can observe clear trend of decreased error rates when in- creasing M. Yet, when M 2 5, the benefits moderate M might be enough for the task. In the experiments of all tested datasets, balance the efficiency while maintaining the contrastive ability, we set M = 10 by default.
increasing M gradually diminish, indicating that a
Table 16: Ablation study on the number of periodicity-varinnt augmented views. Default settings used in the main experiments for SimPer are marked gray
NUM. VIEWS
3
20
Supervised
M
5.12
4.96
4.89
4.87
5.33
18

---

Published as a conference paper at ICLR 2023
Table 17: Ablation study on the choices different periodic similarity measures. Default settings used in the main experiments for SimPer are marked in gray..
SIMILARITY METRICS
MXCor
nPSD (cos(-))
nPSD (I2)
Supervised
MAPE
4.89
4.88
4.92
5.33
D.4.3 CHOICES OF DIFFERENT SIMILARITY METRICS
We investigate the impact of different choices of periodic similarity measures introduced in Section 3.2. Specifically, we study three concrete instantiations of periodic similarity measures: MXCorr, nPSD (cos())). and nPSD (T-2). As Table 17 shows, SimPer is robust to all aforementioned periodic similarity measures, achieving similar downstream performances. The results also demonstrate the effectiveness of the proposed similarity measures in periodic learning.
D.4.4 EFFECTIVENESS OF THE GENERALIZED CONTRASTIVE Loss
We assess the effectiveness of the generalized contrastive loss, as compared to the classic InfoNCE contrastive loss. Table 18 highlights the results over all six datasets, where consistent gains can be obtained when using the generalized contrastive loss in SimPer formulation.
Table 18: Ablation study on effectiveness of using generalized contrastive loss SimPer. We show the feature evaluation results (FFT, MAE*) with and without generalized contrastive loss across different datasets. Note that generalized contrastive loss with no continuity considered degenerates to InfoNCE (Oord et al., 2018). 8).
RotatingDigits
SCAMPS
UBFC
PURE
Countix
LST
SimPer (InfoNCE)
0.23
18.27
9.53
15.74
2.42
4.84
SimPer (Generalized)
0.22
14.45
8.78
13.97
2.06
4.84
Gains
+0.01
+3.82
+0.75
+1.77
+0.36
+0.00
D.4.5 CHOICES OF DIFFERENT INPUT SEQUENCE LENGTHS
Finally, we investigate the effect of different sequence lengths on the final performance in periodic learning. To make the observations more general and comprehensive, we choose three datasets from different domains (i.e., RotatingDigits, SCAMPS, and LST) to study the effect of sequence length. We fix all the experimental setups the same as Appendix B & C, and only vary the frame/sequence lengths with different yet reasonable choices for each dataset.
As highlighted from Table 19, the results illustrate the following interesting observations:
For "clean" periodic learning datasets with the periodic targets being the only dominating signal (i.c., RotatingDigits), using different frame lengths do not inherently change the final result.
For dataset with relatively high SNR (i.e., LST), SimPer is also robust to different frame lengths. The supervised results however worse with shorter clips, which could be attributed to the fact that less information used in the input.
Interestingly, for datasets where other periodic signals might exist (i.e., SCAMPS), using shorter (but with reasonable length) videos seems to slightly improve the performance of SimPer. We hypothesize that for a complex task such as video-based human physiological measurement, some videos may contain multiple periodic processes (e.g.. PPG, breathing, blinking, etc.). A smaller frame length may not be enough to capture some of the "slow" periodic processes (c.g., breathing), thus the features learned by SimPer can become more representative for PPG heart beats estimation. Nevertheless, the differences between various choices are still small, indicating that SimPer is pretty robust different frame lengths.
19

---

Published as a conference paper at ICLR 2023
Table 19: Ablation study on the input sequence lengths. We show the fine-tune evaluation results (MAE*)
using different yet reasonable sequence lengths across various datasets.
RotatingDigits
SCAMPS
LST
Frames
150
120
90
450
300
100
SUPERVISED
0.72
0.71
0.72
3.61
3.57
3.63
1.54
1.56
1.61
SIMPER
0.20
0.19
0.20
3.27
3.11
3.12
1.47
1.47
1.48
GAINS
+0.52
+0.52
+0,52
+0.34
+0,46
+0.51
+0.07
+0.09
+0.13
D.5 COMPARISONS AND COMPATIBILITY WITH SOTA SUPERVISED LEARNING METHODS
As motivated in the main paper, for each specific periodic learning application, supervised learning methods (Dwibedi et al., 2020; McDuff et al., 2022; Liu et al., 2023) have achieved remarkably good results via incorporating certain domain knowledge tailored for specific task. Therefore, provide additional results and comparisons using SOTA algorithms on each of the tested dataset. In the following, show existing SOTA baselines and demonstrate that SimPer could further boost the performance when jointly applied.
Table 20: Compatibility of SimPer with SOTA supervised techniques across different datasets. SOTA
refers to RepNet (Dwibedi et al., 2020) on Countix, and refers to EfficientPhys (Liu et al., 2023) on SCAMPS, UBFC & PURE. SimPer delivers robust performance and complements the performance of SOTA models.
Countix
SCAMPS
UBFC
PURE
Metries
MAE!
GM
MAE!
MAPE
MAE!
MAPE
MAE
MAPE
SOTA
1.03
0.41
2.42
4.10
4.14
3.79
2.87
2.89
SIMCLR
2.56
4.17
3.25
SIMPER SOTA
0.72
0.22
1.96
3.45
3.27
3.06
2.29
2.21
GAINS
+0.29
+0.19
+0.46
+0.65
+0.87
+0.73
+0.58
+0.68
Countix. In the video repetition counting domain, RepNet (Dwibedi et al., 2020), novel neural network architecture that composed of ResNet-50 encoder and a Transformer based predictor, proposed to achieve advanced results for repetitious counting in the wild. We veriffy the compatibility of SimPer with RepNet by changing the encoder on Countix to RepNet, and compare with the vanilla supervised training well as SimCLR. To ensure a fair and comparable setting, train RepNet from scratch instead using ImageNet pre-trained ResNet-50 backbones as the original paper (Dwibedi al., 2020).
SCAMPS, UBFC & PURE. video-bases human physiological sensing domain (i.e., SCAMPS, UBFC, and PURE), the main advances in the field have stemmed from better backbone architec- tures and network components (Liu et al., 2020; 2023; Gideon & Stent, 2021). In the main pa- per, for SCAMPS, since datasets UBFC and PURE, we used a more advanced backbone model (Liu et al., 2020). To further demonstrate that SimPer can improve upon SOTA methods, we employ a recent architecture, called EfficientPhys (Liu et al., 2023), which is specialized for learning physiology from videos.
synthetic dataset, we employed a simple 3D ConvNet: as for real
As confirmed in Table 20, when jointly applied with SOTA models, SimPer can further boost the performance and consistently achieves the best results regardless of datasets and tasks. In contrast, SimCLR not able improve upon SOTA supervised learning techniques. The results indicate that SimPer is orthogonal to SOTA models for learning periodic targets.
D.6 COMPARISONS TO SSL METHODS IN HUMAN PHYSIOLOGICAL MEASUREMENT
In video-based human physiological measurement domain, recent works (Wang al., 2022; Gideon & Stent, 2021) have proposed to leverage contrastive SSL for better learned features and downstream performance in the corresponding application (e.g., heart rate estimation). They studied specific SSL
20

---

Published as a conference paper at ICLR 2023
Table 21: Comparisons between SimPer and additional SSL baselines human physiological ment datasets. Compared customized SSI. algorithms in the specific domain, SimPer still delivers robust performance and consistently achieves the best results.
SCAMPS
UBFC
PURE
Metries
MAR
MAPF
MAF
MAPF-
MAF
MAPE
Without face saliency module:
(Gideon & Stent, 2021)
3.53
5.26
4.98
4.61
4.18
4.70
(Wang et al., 2022)
3.71
5.54
5.07
4.88
4.32
4.95
SIMPER
3.27
4.89
4.24
3.97
3.89
With face saliency module:
(Gidcon & Stent, 2021)
3.51
5.15
4.88
4.29
4.03
4.28
(Wang et al., 2022)
3.61
5.40
5,02
4.86
4.07
SIMPER
2.94
4.35
4.01
3.68
3.47
3.76
Comparisons between SimPer and additional SSL baselines on general periodic Icarning datasets other than human physiological measurement ones. When extending to general periodic learning tasks, SSL baselines tailored for human physiological measurement (Wang al., 2022; Gideon & Stent, 2021) no longer provide benefits, and sometimes perform even worse than vanilla supervised learning. contrast, SimPer consistently and substantially exhibits strengths general periodic learning across all domains.
Table 22:
RiggtsngDigits
Countix
LST
Metries
MAE
MAPE
MAE
GM-
MAE
SUPERVISED
0.72
28.96
1.50
0.73
1.54
0.96
(Gideon & Stent, 2021)
0.70
28.03
1.58
0.81
1.62
0,92
(Wang et al., 2022)
0.77
29.44
1.68
0.94
1.64
0.89
SIMPER
0.20
14.33
1.33
0.59
1.47
0.96
methods tailored for video-based human physiological measurement, and result, many of the proposed techniques therein only apply to that specific domain (e.g., the face detector, the saliency sampler, and the strong assumptions that are derived from the application context, ef. Table I in (Gideon & Stent, 2021)). Nevertheless, it is possible to extend the SSL objectives therein to other general periodic learning domains. this section, we provide additional experimental results and further discussions, which distinguish SimPer from these prior works.
Comparisons on the human physiological measurement task. We first compare SimPer against the aforementioned SSL methods (Gideon & Stent, 2021; Wang et al., 2022) on the human phys- iological measurement task. To provide fair comparison, we fix all methods to use a simple 3D ConvNet backbone (Gulrajani & Lopez-Paz, 2021) SCAMPS, and TS-CAN backbone (Liu al., 2020) on UBFC and PURE as stated Appendix C. As Table 21 demonstrates, SimPer out- performs these SSL baselines across all tested human physiology datasets by notable margin. We break the results out confirm that they hold regardless of whether we include the customized face saliency module (Gideon & Stent, 2021) not.
Comparisons on other periodic learning tasks. We further extend the comparisons to other gen- eral periodic learning tasks. We directly apply the SSL objectives in (Gideon & Stent, 2021; Wang et al., 2022) to other domains and datasets involving periodic learning, and show the corresponding results in Table 22. The table clearly shows that the SSL objectives the referenced papers do provide a benefit in other periodic learning domains, and sometimes perform even worse than the vanilla supervised baseline. The above results further emphasize the significance of SimPer, which consistently and substantially exhibits strengths in general periodic learning across all domains.
D.7 VISUALIZAATION OF LEARNED FEATURES
Since representations learned in periodic data naturally preserves the periodicity information, we can directly plot the leamedd 1-D features for visualization. Fig. 8 shows the learned feature comparison
21

---

Published as a conference paper at ICLR 2023
Targets
SimCLR
CVRL
SimPer
Target Frequency:
MMM
Target Frequency: 0.871
Angle
Target Frequency: 1.554 Hz
Angle
JM
Target Frequency: 2.541 Hz
Anglo
mmmmmmmmm
Target Frequency: 2. 2.958 Hz
Angle
Target Frequency: 3,638 Hz
Target Frequency: 4.161 Hz
Target Frequency: 1.738
Angle
30
100
150
510
1000
150
50
150
50
100
150
Foature Indax
Feature india
Feature Index
Feature index
Figure 8: Visualization of learned periodic representations. We directly plot the 1-D feature vector of data in the test set of RotatingDigits with different underlying target frequencies (left), via different self- supervsee learning methods (right). Existing SSL solutions (Qian et al., 2021; Chen al., 2020) fail to learn meaningful periodic representations, whereas SimPer able to capture the underlying periodicity information.
between SimCLR, CVRL, and SimPer, together with the underlying periodic information (rotation angle & frequency) in RotatingDigits. As the figure verifies, SimPer consistently learns the pe- riodic information with different frequency targets, delivering meaningful periodic representations that are robust and interpretable. In contrast, existing SSL methods cannot capture the underlying periodicity, and fail to learn useful representations for periodic learning tasks.
E BROADER IMPACTS AND LIMITATIONS
Limitations. There some limitations to approach in its current form. The SimPer features leart in some cases were not highly effective without certain fine-tuning on a downstream task. This may be explained by the fact that some videos may contain multiple periodic processes (e.g..
22

---

Published as a conference paper at ICLR 2023
Pulse/PPG, breathing, blinking, etc.). A pure SSL approach will learn features related all these periodic signals, but not information that specific to any one. One practical solution for this limitation could be incorporating the frequency priors of the targets of interest. Precisely, filter out unrelated frequencies during SimPer pre-training to force the network to learn features that constrained within certain frequency range. We leave this part future work.
Broader Impacts. While our methods are generic to tasks that involve learning periodic signals, we have selected some specific tasks which demonstrate their efficacy more coneretely. The mea- surement of health information from videos has tremendous potential for positive impact, helping lower the barrier access to frequent measurement and reduce the discomfort or inconvenience caused by wearable devices. However, there is the potential for negative applications of such tech- nology. Whether by negligence, or bad intention, unobtrusive measurement could be used to mea- sure information covertly and without the consent of user. Such application would be unethical and would also violate laws in many parts of the world measures applied to traditional medical sensing are also applied to video-based methods. We will be releasing code for our approach under a Responsible AI License (RAIL) (Contractor et al., 2022) to help practically mitigate unintended negative behavioral uses of the technology while still making the code available.
important that the same stringent
23


Reviews:
[{"review_id": "mxu_k5y3d2", "reviewer": "Reviewer_DG5c", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "SimPer is a self-supervised learning method for learning representations from periodic data such human physiology. In this paper the authors demonstrate that existing baselines such as SimCLR do not work well. The key idea in this paper is to revise the idea of feature similarity for temporal data. More specifically, the idea is to improve similarity of temporally close features (defined through index-shifting and reversing augmentations). A \"random\" feature is defined to be a negative feature. The results demonstrate that SimPer is performing better than some of the self-supervised learning methods.", "summary_of_the_review": "Comparing the strengths and weaknesses, this paper can be further improved to study why existing unsupervised methods are superior to the proposed method. Hence, in my opinion, the paper is not ready for acceptance.", "strength_and_weaknesses": "**Strengths**\n\n* The method improves results over self-supervised methods such as SimCLR. \n\n* The paper is well written and easy to follow.\n\n* The paper advances simple augmentation technique for temporal contrastive learning framework.\n\n**Weaknesses**\n\n* Although the approach is interesting, the experimental section would require additional results to benchmark thoroughly against CVLR. For example, this submission does not include results on Kinetics-400. (as opposed to a subset)\n\n* There has already been self-supervised learning approaches in the area of human physiology. This submission does not compare with this ICCV paper. [ref 1]. This paper already explores the idea of maximal cross correlation. (MCC)\n\n* It is not clear if the evaluation protocol is good enough for representations. Especially, for computing a single HR value over a large window. In other words, even if the representations are very bad for large portion of the video, the evaluation protocol would not be able to detect it.\n\n* Additional comparisons with baselines: The results on known traditional methods are better on SCAMPS dataset than the proposed method. These baseline methods have not been to be included in this paper. It would be beneficial to the reader to have a comparison in the paper.\n\n**References**\n\n[ref 1] The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video, ICCV'21\n[ref 2] SCAMPS: Synthetics for Camera Measurement of Physiological Signals, NeurIPS'22\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_0md3jBaKf", "reviewer": "Reviewer_pX9g", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents SimPer, a simple yet effective SSL algorithm specifically designed for periodic tasks (e.g. heart rate estimation & exercise repetition counting through video data, temperature forecasting through satellite data). With the newly proposed techniques including periodicity-variant (i.e. input frequency) augmentation, periodic feature similarity, and generalized (weighted) contrastive loss, SimPer demonstrates impressive results in a large array of tasks including SSL (Table 2 to 7), fine-tuning (Table 8), low-shot learning (Fig 4), transfer learning (Table 9), zero-shot generalization (Table 10), robustness to spurious correlations (Fig 6), etc. against existing SSL algorithms (SimCLR, MoCo v2, BYOL and CVRL) and supervised training, on 6 datasets (2 synthetic) from 4 different domains.", "summary_of_the_review": "In my opinion, since the paper is strong in ~nearly~ all aspects (clarity, novelty, quality and significance, ~except for minor weakness in evaluation~) with sufficiently impressive results, I recommend its acceptance to the conference.", "strength_and_weaknesses": "Strengths\n+ (Clarity) The paper is very well organized and clearly written. The figures are also well made and very informative.\n+ (Novelty) The paper and its proposed techniques are novel as far as I can tell. The topic is clearly important (for its e.g. medical & health applications) yet doesn’t seem to be as popularly studied as other ML/SSL topics. \n+ (Quality) The paper is of good quality in my opinion. The algorithm is well designed for periodic signals to maximally extract useful information (via frequency augmentation) to be better learned (via periodic feature similarity & generalized loss). The technical details are all correct as far as I can tell. The empirical evaluation is also comprehensive, covering 6 different SSL (and related) tasks against popular baselines, ~even though the selection of baselines could be further improved (see weaknesses).~\n+ (Significance) The proposed techniques are simple, easy to implement and experimentally highly effective, making the algorithm (SimPer) a strong, potentially impactful baseline for future researchers & practitioners to use and/or improve upon.\n\nWeaknesses\n- ~(Evaluation, Minor) The algorithm is currently only benchmarked against 4 SSL baselines and a supervised counterpart, not SOTA algorithms on each of the datasets (e.g. the MAE for Countix appears much weaker than Table 7 of Dwibedi et al.). While SimPer is likely orthogonal to those SOTA algorithms and thus might be applied jointly, it’s still non-ideal to completely omit the SOTA baselines and the discussion on how SimPer could further boost the SOTA numbers.~", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-uBvd5KXlV", "reviewer": "Reviewer_y54W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a contrastive learning framework for encoding periodic information from high-dimensional input sequences. The proposed method (SimPer) leverages an inductive bias present in most periodic processes to define suitable data augmentations, feature similarity measures, and loss functions. To create negative pairs, the method intervenes on a sequence by changing its frequency (by changing its speed). The proposed similarity metrics take into account temporal offsets or the power spectrum density of the signals using existing measures. The loss function is an interesting generalization of the InfoNCE loss where the loss for a pair of instances is weighted using side information on the similarity of the given pair (e.g., the absolute difference in the speedup of two sequences). A thorough empirical analysis shows that SimPer encodes periodic information well – significantly better than frame-based methods (SimCLR, MoCo, BYOL) and one approach based on sequences (CVRL). SimPer shows improved data efficiency compared to self-supervised and supervised methods, and improved accuracy for fine-tuning, transfer learning, OOD generalization, and robustness to spurious correlations compared to other self-supervised methods.", "summary_of_the_review": "The paper presents a simple framework that is well-motivated and quite intuitive. The paper has an empirical focus and, appropriately, provides comprehensive experiments and ablations. One exception is that the paper could provide more information regarding the choice of sequence length, which might be an important hyperparameter. Other than that, I have no major concerns and recommend acceptance.", "strength_and_weaknesses": "**Strengths**\n\n- The overall idea and the individual design choices are well-motivated and explained very clearly in the text and supported by suitable illustrations.\n- It is transparently stated that the method makes use of an inductive bias to construct suitable augmentations and loss functions.\n- The empirical analysis is very comprehensive. It clearly shows the benefits of the proposed method and provides helpful ablations.\n- The writing and exposition of the paper are very clear. \n\n\n**Weaknesses**\n\n- It is not sufficiently clear how to choose the sequence length and how important the hyperparameter might be for the encoding of periodic information. For instance, a very long sequence might contain multiple subsequences that exhibit different periodicities. I think it would be helpful to include an ablation for the sequence length.\n- Do I correctly understand that the baselines SimCLR, MoCo, and BYOL use individual frames as inputs, whereas CVRL and SimPer use sequences as inputs? In that case, it would not be surprising that the frame-based approaches do not encode periodic information, which would be difficult to estimate from a single frame.\n- It would be interesting to measure not only the periodic information that is encoded, but also how much other information (object class, etc; depending on the dataset) is retained in the encoding. For instance, Figure 6 shows that the spurious feature (digit appearance) is encoded relatively well for the training data, even though it is not an invariant feature that would help discriminate between positive and negative pairs.\n\n**Minor comments and questions:**\n- In Figure 4a, why do you plot fewer data points when you decrease the training dataset size? Are the learned representations not based on a test dataset of a fixed size?\n- Why is the performance of the proposed method on the LST dataset not much better compared to the baselines?\n- In the context of the Nyquist sampling theorem, it would be helpful if you briefly stated its results and why it is relevant.\n- Did you decrease the margins for the abstract?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%