Leaves: Learning Views for Time-Series Data in Contrastive Learning
Authors: Han Yu
Huiyuan Yang
Akane Sano
Venue: Submitted to ICLR 2023 (2023)
Keywords: Contrastive Learning
Data Augmentation
Learning Views
Time-Series Data
Adversarial Learning

Abstract: Contrastive learning, a self-supervised learning method that can learn representations from unlabeled data, has been developed promisingly. Many methods of contrastive learning depend on data augmentation techniques, which generate different views from the original signal. However, tuning policies and hyper-parameters for more effective data augmentation methods in contrastive learning is often time and resource-consuming. Researchers have designed approaches to automatically generate new views for some input signals, especially on the image data. But the view-learning method is not well developed for time-series data. In this work, we propose a simple but effective module for automating view generation for time-series data in contrastive learning, named learning views for time-series data (LEAVES). The proposed module learns the hyper-parameters for augmentations using adversarial training in contrastive learning. We validate the effectiveness of the proposed method using multiple time-series datasets. The experiments demonstrate that the proposed method is more effective in finding reasonable views and performs downstream tasks better than the baselines, including manually tuned augmentation-based contrastive learning methods and SOTA methods.

Under review as a conference paper at ICLR 2023
LEAVES: LEARNING VIEWS FOR TIME-SERIES DATA IN CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Contrastive learning, self- supervupereieed learning method that can learn representa- tions from unlabeled data, has been developed promisingly. Many methods of con- trastive learning depend on data augmentation techniques, which generate differ- ent views from the original signal. However, tuning policies and hyper-parameters for more effecctive data augmentation methods in contrastive learning is often time and resource-consuming Researchers have designed approaches to automatically generate views for some input signals, especially on the image data. But the view-learning method is not well developed for time-series data. In this work, we propose simple but effective module for automating view generation for time-series data in contrastive learning, named learning views for time-series data (LEAVES). The proposed module learns the hyper-parameters for augmentations using adversarial training in contrastive learning. We validate the effectiveness of the proposed method using multiple time-series datasets. The experiments demon- strate that the proposed method is more effective in finding reasonable views and performs downstream tasks better than the baselines, including manually tuned augmentatiion-bsse contrastive learning methods and SOTA methods.
INTRODUCTION
Contrastive learning has been widely applied to improve the robustness of the model for various downstream tasks such as images (Chen et al., 22020; Grill et al. 2020) Wang & Qil 2022) and time- series data (Mohsenvand et al. (020) & Strodthofi] 2022). Among the developed contrastive learning methods, data augmentation plays an essential role generating different corrupted trans- formations, as views of the original input for the pretext task. For example, [Chnn et al.) (2020) proposed a SimCLR method to maximize the agreements of augmented views from the same sam- ple pre-train the model, which significantly outperformed the previous state-of-the-art method in image classification with far less labeled data. However, the data augmentation methods selection usually empirical, and tuning set of optimized data augmentation methods can cost thousands of GPU hours even with the automating searching algorithms (Cubuk et al. 2019). Therefore, it dataset.
remains an open question how effectively generate views for
Instead of using artificially generated views, researchers have been putting efforts into training deep learning methods generate optimized views for the input samples (Tamkin al. 2020 Rusak et al. 2020). These methods generate reasonably-corrupted views for the image datasets and result satisfactory results. For example, Tamkin al. (2020) proposed the Viewraker, an adversar- ially trained convolutional module in contrastive learning, to generate augmentation for images. Nevertheless, the aforementioned method, such as the ViewMaker, might not be acclimatized when forthrightly utilized on the time-series data. The main challenge is that, for the time-series signal, we need to not only disturb the magnitudes (spatial) but also distort the temporal dimension Um et al. (017): Mchari & Strodthoff (2022). While the image-based methods can only disturb the spatial domain by adding reasonable noise to the input data.
In this work, propose LEAVES, which is lightweight module for learning views on time-series data in contrastive learning. The LEAVES is optimized adversarially against the contrastive loss to generate challenging views for the encoder in learning representations. In addition, to introduce smooth temporal perturbations to the generated views, we propose a differentiable data augmenta- tion technique for time-series data, named TimeDistort. Figure [shows the examples of the gener-

---

Under review as a conference paper at ICLR 2023
Original ViewMaker
Original LEAVES
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
100
200
300
400
500
0
100
200
300
400
500
(a)
(b)
Figure 1: Visualization of the views learned using (a) the ViewMaker and (b) the proposed LEAVES. Compared to the ViewMaker, LEAVES introduces temporal distortion and the augmented view is more faithful.
ated views of electrocardiogram (ECG) from the ViewMaker (Tamkin et al. 2020) and our method. We can find that temporal location is perturbed in Fig (a), and the flat region (T-P interval as the ECG fiducial) of the original ECG signal was completely distorted. Compared to the View- Maker, the proposed LEAVES can distort the original signal in both spatial and temporal domains, more importantly, reducing the risk of losing intact information due to excessive perturbation in time-series data. Our experiments and analysis showed that the proposed LEAVES (1) outperforms the baselines, including SimCLR and the SOTA methods, and (2) generates more reasonable views compared to the SOTA methods in time-series data.
RELATED WORKS
2.1
DUGMENTATION-BASE CONTRASTIVE LEARNING
Among the contrastive learning algorithms proposed in various areas, the data augmentation meth- ods usually play essential roles in generating views from the original input form contrastive pairs. Many contrastive learning frameworks have been recently developed based on the image transfor-
mation computer vision (He al.
2020 Chen et al. 2020 Grill al. 2020 Chen He [021]]
Tamkin et al.] 2020 Zbontar et al. 2021 Wang & Qi 2022 Zhang & Ma 2022). For example,
Chen et al.: (2020) proposed a SimCLR framework that maximizes the agreement between two views
transformed from the same image. B BOL Grril et al. 2020) encourages two networks, including target network and online network, to interact and learn from each other based on two augmented views of an image. Zbootar et al.) (2021) proposed Barlow Twin framework that applied two cor- rupted views from an image with a reeunddany--eduuctio objective function to avoid trivial constant solutions in contrastive learning.
Other than the applications in the computer vision area, contrastive learning algorithms have also been applied to time-series data (Gopal et al., 2021 Mehari & Strodthofi] 2022 Wickstrøm et al.) 2022]. For example, Gopal et al. (2021) proposed a clinical domain- knowledge-basse augmen- tation ECG data and generated views from ECG from contrastive learning. Also, Meharr & Strodthoff (2022) applied well-evaluated methods such as SimCLR, BYOL, and CPC Oord et al. (2018) time-series ECG data for clinical downstream tasks. Wickstrøm et al. (2022) generated contrastive views by applying the MixUp augmentation (Zhang et al. (017) in the time-series data. The aforementioned research has achieved promising results by leveraging unlabeled data, however, the empirically augmented views might not be optimal, especially for the datasets that are relatively new or less popular, as exploring appropriate sets of augmentations is expensive.
2.2 AUTOMATIC AUGMENTATION
Rather than setting augmentation methods with empirical settings, researchers proposed multiple methods of optimizing the appropriate augmentation strategies (Cubuk et al. 2019 Ho et al. 2019 Lim et al.) 2019; Li et al. 2020 Cubuk et al.) 2020 Liu et al. 2021 l). For example, AutoAug- ment (Cubuk et al. 2019) was designed as a reinforcement learning-basee augmentation policies, including the possibility and order of using different augmentation methods. DADA (Li et al. (020)) is a gradicnt- based optimization strategy for finding the augmentation policy with the highest probabilities after training, which significantly reduces the search time compared to
algorithm to search for
2

---

Under review as a conference paper at ICLR 2023
algorithms such as AutoAugment. These searching methods have been proven to have high perfor- mance, however, they usually require heavy computational complexity in thoroughly searching the augmentation space and finding optimized policies.
Instead of searching for augmentations from the policy space, researchers also developed learning views, which can be understood generating data transformation by neural network rather than using manually tuned augmentations (Tian et al. 2020 Rusak et al. [0200] Tamkin et al. (020). For example. Rusak et al. (2020) applied CNN structure to generate noise based on the input data and trained the perturbation generator in an adversarial manner against the supervised loss. Similarly, Tamkin et al. (2020) proposed ResNett-based ViewMaker module generate views for data for the contrastive learning framework. The training of ViewMaker was also adversarial by maximizing the contrastive loss against the objective of the representation encoder. Nevertheless, these methods lack consideration of temporal perturbation when used in time-series datasets. Therefore, in this work, we design our proposed LEAVES module generate both the magnitude and temporal perturbations in sequences.
METHODS
Contrastive learning is a self-supervised learning approach, which encourages the representations of transformations of the same inputs to be similar and learns the differences from pairs of different samples. In this work, we adopt a simple and well-proven contrastive learning method, the SimCLR (Chen al. 2020 Figure shows shows the overview of the pre-training architecture. We first introduce a differentiable LEAVES module, which can generate more challenging but still faithful views of an input. The LEAVES module is then plugged into the SimCLR framework generate different views for contrastive learning. The LEAVES is trained with the encoder in an adversarial manner.
Contrastive Loss
min
f0
fo
TimeDis
TM
MagW
Scale
Maxinizee
Jitter
LEAVES
LEAVES
(a) Differentiable LEAVES module
(b) LEAVES SimCLR
Figure 2: LEAVES designed as a plug-in module to generate augmented views in time-series data for contrastive learning. The framework trains the LEAVES and the representation encoder in an adversarial manner: LEAVES minimizes the agreements between the pairs of representations, whereas the encoder is optimized to maximize the agreement.
3.1 LEAVES
We propose LEAVES module, which is a lightweight component that can be casily plugged into the contrastive learning system. This module consists of series of differentiable data augmentation methods, including Jitter J,, Scale Ts. magnitude warping (MagW) W) Tasw, permutation (Perm) Tr, and a newly proposed method named time distortion (TimeDis) TTD- simplify the expression, use @ to indicate the in-series operation. For example, T7 TTp represents that the input data has been first transformed by adding jittering noise, then been permuted. Thus, the proposed module generates view X by
X XXTT(a) Ts(as) @ Tww((am) @ TTD(ar) Tr(ar)
3

---

Under review as a conference paper at ICLR 2023
1.00
Orgina Intering
Orig nal
Scalin
Origiaal Hagiy
0.25
Origina
Original TimeDtu
Original Timal
Narm
0.75
0.25
Figure 3: Examples of different data augmentation methods for time-series data
probabiity magnitude
Orig Inal TimeDis
0.5
GMM
Figure 4: Example of the time distortion augmentation with 3 Gaussian components. Correspond- ing locations with higher probabilities in the generated GMM up-sampled in the affine space. Conversely, locations with smaller probabilities are down-sampled in the transformation.
where a represents the hyper-parameters of the data augmentation method that control the intensity of corruption on the original sample. For instance, σ., represents the standard deviation value for generating the jittering noise. The target learning parameters of this module are the o of augmenta- tion methods. By learning these parameters, the module learns the strategy of generating views by combining multiple augmentation methods. We do not deliberately tune the order of augmentations applied to X in equation [T] the hyper-parameters and views of augmentations are independent, e.g., the applied Scale operation would not rely on the view generated by Jitter.
3.1.1 DIFFERENTIABLE DATA AUGMENTATIONS FOR TIME-SERIES DATA
Several widely used data augmentation methods are selected for LEAVES. For example, Jitter, Scale, and MagW W perturb the magnitude of the original signal; whereas time warping (TimeW) (Um al., 2017) and Perm corrupt the temporal location. A detailed description of the augmentation methods can be found in Appendix A
To optimize the hyper-parametent in these augmentation algorithms, the gradients are needed to be propagated back to these parameters during the training process. However, these augmentation methods are based on non-differentiable operations such drawing random values, indexing, etc. Therefore, we applied reparameterization tricks (Jang et al., 2016 Maddison al.) (016) to make those procedures differentiable, except the TimeW W method, cult to retrieve gradients. Thus, we propose the TimeDis augmentation method as an alternative to smoothly distort temporal information (see details in the next subsection). Figure shows the exam- ples of the six augmentation methods on time-series sample. To ensure the generated corruptions reasonable, we constrain the noise with up-hound v the maximum values of values in magnitude-basee methods Jitter, Scale, MagW- W, and K as the maximum segments in Perm.
the indexing operation makes it diffi-
TimeDis. This method relies on smooth probability distribution generate the probability of the location to be sampled in the original signal. We utilize a reparameterized Gaussian mixture model with M Gaussian components as Σ. duNNu, to generate the location indexes À E NXCxL from -1 to 1. Figure shows an example of TimeDis. Among the generated location indexes, -1 corresponds to the first time step (position 1) in the original signal, whereas I corresponds to the last use À to affine the original signal X the view X, which results
time step (position L). Then, that for the locations with dense indexes in X, the intervals among samples become looser; whereas for these locations with sparse indexes in X, the corresponding intervals would be tighter.
4

---

Under review as a conference paper at ICLR 2023
3.2 ADVERSARIAL TRAINING
In representation learning, we define the representation extracted by the encoder 2. Given N pairs of representations extracted by the encoder in the SimCIR framework as (z,, {i.j} E [1, N, the loss function of maximizing the agreements between pairs of views can be defined as:
1 2N
L =
((2k - 1, 1,2 f(2k) 2k 1)],
exp(s(z1, (xp(s(2,2,)/7)) 2T)/T)
4 = - -log
(3)
Ex=1 Li4i exp(s (s 2x)
where 8(2,, 2j) represents the cosine similarity between 2; and 2j. 1 k is an indicator function that equals to iff 1: + i, and denotes temperature parameter that fixed 0.05 in this study.
As shown in Figure 2 the LEAVES and the encoder are optimized in opposite directions: the target objective of the encoder is minimize L, whereas the LEAVES module desires to maximize L. By leveraging the adversarial training manner, we design the LEAVES module to distort the original signal as challenging possible while the encoder pairs. Under this scenario, the encoder would be robust by training against the most corrupted views. After training the SimCLR framework, the learned model weights in the encoder structure are used to initialize the model weights for supervised learning in downstream tasks.
still extract intact information from the view
EVALUATION
To evaluate the proposed method, we conduct experiments on 4 public times-series datasets, includ- ing Apnea-ECG (Penzel et al. (2000). Sleep-EDFF (Kemp et al. (018). PTB-XI. (Wagner et al.) 2020), and PAMAP2 (Reiss & Stricker 2012) for applications rhythmia, and human activities, respectively. The detailed information of the used datasets can be found in Appendix B For each the datasets, we pre-train the proposed module and the encoder and fine-tune the encoder for the downstream task. Three baselines implemented for comparison, in- cluding (1) supervised ResNet- 18, (2) SimCLR with random augmentations (see fine- tuning details in 5.2 and (3) reproduced ViewMaker network that intakes one-dimension time-series inputs.
detecting apnea, sleep stage, ar-
4.1
SLEEP APNEA DETECTION WITH SINGLE-LEAD ECG
The Apnea-ECG dataset (Penzel et al. 2000 studies the relationship between human sleep ap- nea symptoms and heart activities (monitored by ECG), which can be accessed through Phys- Gonee(Goldderger et al., 2000). Following the settings in the original release Penzel al. (2000) we used a one-minute scale of 100Hz ECGs to detect the binary labels of whether apean oecurs, with 17233 and 17010 samples in the training and testing sets, respectively. In the contrastive pre- training stage, we set the noise threshold 5 to be 0.05 for Jitter, Scale, MagW, W, M as 12 for TimeDis, and K as 5 for Perm. We used 100 epochs with learning rate of 1e-3 to pre-train the encoder and 30 epochs with learning rate of 1e-3 to fine-tune for the downstream task.
Table D shows the evaluation results for detection sleep apnea. Following the SOTAs on the same dataset, we used sensitivity (Sen.) and specificity (Spec.), which assess the ability of the model to diagnose apnea for patients, as metrics measure the model performances. When comparing the proposed model with the baseline models, we observed that LEAVES performed better than the baselines in both metrics of Sen. and Spec. And both SimCLR and ViewMaker outperformed the supervised baseline, with SimCLR showing slightly higher performances than the Vieer Compared to the SOTAs, we found the proposed method showed a competitive Sen. score but rel- atively low Spec. score. This can be caused by different settings, such filtering noisy samples and data pre- proppcesssing procedures. The baseline supervised structure is similar to in (Chang et al. 2020); however, our resulted Spec. in supervised baseline was lower than SOTAs.
5

---

Table the performances of classifying sleep stages using EEG signal. From the table, we can results with Under review as a conference paper at ICLR 2023
Evaluation results in detecting sleep apnea on the Apnea-ECG dataset. The bold values represent the best performance among the baseline models and the proposed methods.
Table 1:
Methods
Sen.
Spec.
Multi-Scale ResNet (Fang et al. (022)
0.841
0.871
ID CNN (Chang et al. (2020)
0.811
0,920
SOTAs
CNN + Decision Fusion (Singh & Majumder 2019)
0.900
0.838
Auto-Encoder HMM (Li et al. 2018
0.889
0.821
Supervised
0.843
0.788
Baselines
SimCLR
0.856
0.803
ViewMaker
0.851
0.795
Ours
LEAVES
0.865
0.821
Table 2: Evaluation results in classifying sleep stages using EEG on the Sleep-EDF dataset. The
bold values represent the best performance among the baseline models and the proposed methods.
Methods
Acc.
F1
U-Time (1D CNN-basedl (Perslev et al.
2019)
0.764
CNN + RNN (Mousavi al. 2019
84.3
0.796
SOTAs
U-Sleep (ID CNN-based) (Perslev et al. 2021
0,790
ID CNN + HMM (Li et al. 2018
84.0
0.78
Spectrogram + Transformer Phan et al. (0022)
84.9
0.789
Supervised
83.6
0.769
Baselines
SimCLR
84.8
0.783
ViewMaker
84.4
0.777
Ours
LEAVES
85.3
0.790
4.2
SLEEP STAGE CLASSIFICATION WITH EEG
Eleetrrenncephalograpyy (EEG) is an essential signal to monitor human brain activities. We tested our methods on the Sleep-EDF (cxpanded) (Kemp et al. 2018) dataset, which contains whole-night sleep recordings of 100Hz Fpz-Cz EEG signal. Following Supratak Guo (2020), we extracted 42308 30-second samples that were annotated in 5 sleep stages. In the contrastive pre-training stage, we set the noise threshold 8 to be 0.05 for Jitter, Scale, Mag W, M 10 for TimeDis, and K as 5 for Perm. We used 100 epochs with a learning rate of 1e-3 to pre-train the encoder and 30 epochs with a learning rate of le-3 to fine-tune for the downstream task. To compare the model performances with SOTAs, we use accuracy and macro fl fl-score as the metrics for evaluation.
learn that the proposed methods outperformed the baselines. Also, when comparing competitive. SOTAs, we found the achieved performances in both accuracy and macro fl-score were However, need to admit that the different experimental settings were used in the SOTAs and our work. There were several factors existed that might trigger discrepancies in performances. For example, the pre-processing procedures were uniform; also, the split train/test datasets were universal, i.e., 10 or 20-fold cross-validations with different settings were widely applied in SOTAs, whereas we split validation sets according the subject IDs.
4.3 HUMAN ACTIVITIES DETECTION WITH IMU AND HEART RATE
Human activities can be detected using wearable device data. The PAMAP2 RReiss & Stricker 2012) studies the relationship between human activities and data collected from wearable sensors of 3 inertial measurement units (IMU) and a heart rate monitor. We used 100-Hz IMU data and the upsampled heart rate data in the experiments. Following Moya Rueda et al. (2018); Tamkin ctt al. (2000), 12 of the total 18 different physical activities are used in the experiments. In the contrastive pre-training stage,
set the noise threshold ó to be 0.05 for Jitter, Scale, MagW, M as 7 for TimeDis, and K as 5 for Perm. We used 100 epochs with a learning rate of 1e-3 to pre-train the encoder and 20 epochs with learning rate of 1e-3 to fine-tune for the downstream task. To compare the model performances with SOTAs, we used accuracy and macro fl-score as the metrics for evaluation.

---

Under review as a conference paper at ICLR 2023
Evaluation results in detecting human activities on PAMAP2 dataset. The bold values represent the best performance among the baseline models and the proposed methods.
Table 3:
Methods
Acc.
F1
TD CNN (Moya Rueda al., 2018)
93.7
0.937
ViewMaker (Spectrogram + CNN) (Tamkin et al. 2020)
91.3
SOTAs
Late-Fusion ID CNN (Mekruksavanichh Jitpattankul 2021
85.5
Multi-Input DNN-GRU/Dua et al. (2021)
95.3
0.952
(subject-dependent) ResNet + BILSTM (Li & Wang 2022
97.2
0.974
Supervised
89.9
0.896
Baselines
SimCLR
92.2
0.925
ViewMaker
91.5
0.913
Ours
LEAVES
93.3
0.934
Table 4: Evaluation results in detecting cardiac arrhythmia on PTB-XL dataset. The bold values represent the best performance among the baseline models and the proposed methods.
Methods
AUC
Acc.
1D CNN (Smigiel et al.
2021)
0.910
76.5
Siwn Transformer [ii et al.
2021
0.905
Multi-Leadd-rrte
Branchh Fusion CNN (Zhang al. 2021
0.943
SOTAs
CPC (Mehari & Strodthoff, 2022
0.941
SimCLR (Mehari & Strodthoff] 2022)
0.926
BYOL (Mehar & Strodthofi (2022)
0.926
Supervised
0.924
77.3
Baselines
SimCLR
0.932
79.5
ViewMaker
0.926
78.0
Ours
LEAVES
0.931
79.3
3 shows the performances in classifying human activities with PAMAP2 dataset. Our proposed method outperformed the baselines and showed competitive results as SOTAs that shared the same train/test settings (Moya Rueda al., 2018 Tamkin et al. 2020). The study conducted by Li & Wang (2022) used a 70/30% train/test split strategy in a subject-dependent setting and achieved the highest performance among all studies. In addition, Table shows the comparison of the results reported in the original work (Tamkin et al., (2020) and the 1D version we reproduced. The model accuracy from the original work, which converted time-series data into spectrograms and leveraged 2D ResNet, is very similar to the accuracy from our implemented ID ResNet version.
4.4 ARRHYTHMIA CLASSIFICATION WITH 12-LEAD ECG
Cardiac arrhythmia is one of the main causes of cardiovascular diseases, thus detecting arrhythmia has important clinical prospects. The PTB-XL (Wggeee et al. 2020) dataset is a large dataset with contains 21,837 12-lead and 10-second ECG in 100 Hz with arrhythmia labels in classes. We follow the recommended splits of training and testing sets in the original work (Wagner et al. 2020 In the contrastive pre-training stage, we set the noise threshold 8 be 0.05 for Jitter, Scale, Mag W. M as 6 for TimeDis, and K as 5 for Perm. We used 100 epochs with a learning rate of 1e-3 to pre-train the encoder and 30 epochs with learning late of le-3 to fine-tune for the downstream task. To compare the model performances with SOTAs, we use AUC and accuracy the metrics for evaluation.
Table 4 shows the results of classifying arrhythmia with ECG sequences. We found the proposed method outperformed the supervised and ViewMaker baselines; whereas the SimCLR bascline with random augmentations performed slightly higher than the proposed method. When comparing method to the SOTAs, we found that our method showed higher AUC than some supervised methods such as (Smigiel et al. 2021 Li et al. 2021). Interestingly, when comparing our results in an ECG- centered benchmark self supervisdd learning work, it shows that our results were slightly higher than their implemented SimCLR, which might also indicate the effectiveness of the proposed LEAVES compared to manually tuned augmentations (Mehari & Strodthoff 2022.

---

Under review as a conference paper at ICLR 2023
Table 5: The quality check results on the learned views in the Apnea-ECG dataset
Excellent
Barely Acceptable
Unacceptable
Original Data
4.4%
92.4%
3.2%
ViewMaker
0.1%
50.4%
49.5%
(Ours) LEAVES
0.4%
96.0%
3.6%
Table 6: The results table of tuning augmentation methods in the SimCLR framework. The perfor- mances are listed in the metrics of accuracy/macro f'l-score
T(0.01)
T(0.02)
T(0.03)
T(0.04)
T(0.05)
Apnea-FCCe FCG
16.1/0.752
76,0/0. 746
78.8/0.775
76.510.759
78,5/0.771
Sleep-EDFE
82.5/0.771
83.2/0.775
84.0/0.778
84.8/0.783
84.5/0.780
PTB-XL
76.7/0.663
78.7/0.670
79.5/0.676
77.6/0.671
75.1/0.649
PAMAP2
89.0/0.897
90.5/0.901
91.1/0.912
92.2/0.925
91.3/0.918
5 DISCUSSION
This section introduces the ablation studies we conducted, including applying the ViewMaker frame- work in time-series data and fine-tuning the baselines SimCLR algorithm with different augmenta- tion
Also, we further introduce the proposed method's learning views and the complexity of the LEAVES module.
5.1 ABLATION STUDY: VIEW TIME-SERIES DATA
As stated in the previous section, our work is inspired by the ViewMaker (Tamkin et al.) 2020) We tested the ViewMaker framework, and improvements were observed as shown in the evalua- tion section. However, time-series data. Figure shows an example of its limitation temporal distortion and preserving intact information as mentioned in section [ To further examine the faithfulness of the generated views, we utilized an ECG quality check method (Zhao & Zhang 2018) using the NeuroKit package (Makowski et al. (021). Table 5 shows the quality check results on Apnea-ECG dataset. We can observe that the ViewMakers method perturb almost half of the ECGs into "Unacceptable", rep- resenting the signals that are hardly recognized as ECG signals. These limitations of ViewMaker when applied to the time-series dataset motivated us to develop the proposed method in this work.
also observed limitations of the ViewMaker while applying it to the
5.2 ABLATION STUDY: FINE-TUNE THE SIMCLR BASELINE
As mentioned in the previous sections, finding optimal data augmentation methods in contrastive learning is challenging as the search space of the augmentation methods is usually massive. In this work, to train a strong SimCLR baseline, we tuned the intensity of the augmentation methods for the baseline SimCLR mentioned in section 4 To simplify the experiments, dard deviation a of Jitter, Scale, MagW, TimeW to be the same value, increasing from 0.01 to 0.05 by 0.01, and fixed the maximum permutation segment 5. For example, T(0.01) represents that the a values of Jitter, Scale, MagW, TimeW are 0.01 and K = 5 for permutation. We listed the tuning performances with metrics of accuracy and macro fl -coore Table 6 With different hyper- parameters that affect the intensity of the augmentation methods, we observed different evaluation results. On some datasets, such as PAMAP2, we observed close performance while using different hyperprameters. However, the model performances seemed to be more hyper-parameter-see parameeer-sensitivee on the PTB-XL dataset. For example, while using the a values of 0.05, we observed a performance drop by big margin compared to the SimCLR with 0.03. This illustrated that finding reasonable augmentations makes contributions to contrastive learning, while on the other hand, inappropriate transformations may hurt the model performances. And our proposed method can find the appropri- ate augmentation for time series data without spending time in augmentation searching, which could be meaningful to researchers using new not commonly studied time-series datasets.
controlled the stan-
8

---

Under review as a conference paper at ICLR 2023
Jitter STD
Scalle STD
MagW
Perm Max Segments
Aphees-5 S0re:500E
0.028
ECG
Aprea ECG
Sleep EDFE
Seep-EDR
PMt-l PAMAR2
0.024
PAMAR
PTi-aL
0.045
PTD-0
PMMP2
4.5
PAMP2
PTD-0
0.010
0.022
0.020
0.02
0.030
3.-
0.025
9.015
29
40
60
00
100
20
40
80
1.00
20
40
60
80
100
0
20
40
89
160
Figure 5: Visualization of the scalar hyper-parameters learned through LEAVES
5.3
LEARNING HYPER-PARAMETER FOR AUGMENTATIONS
Given that we proposed differentiable augmentation- based approach, can speculate that the hyper-parameters controlling augmentation would change with the training process as the model learns. Figure[5[shows the scalar Syper-pprameee and Perm. We can observe that the a values for Jitter and Mag W were keeping increasing on all four dataset; whereas the value for Scale showed decreasing trend. We found the maximum segment K in Perm showed an increasing trend on PTB-XL but decreased on PAMAP2 dataset. Although was not conducted in this work, this phenomenon shows the possibility that our approach may help find appropriate views for supervised learning on the different datasets as well. A potential application to combine our proposed module with supervised learning framework by adversarial training, such as the framework by Rusak et al. (2020).
change for augmentations of Jitter, Scale, Mag W,
5.4 TIME & SPACE COMPLEXITIES
Since the targeted optimizing weights of LEAVES are the hyper-paramette of the augmentation methods, the proposed method shows advantages in terms of model space complexity compared to the previous SOTAS such as the ViewMaker. For example, there are 580K parameters to learn in our re-produced ID ViewMaker structure; whereas LEAVES optimizes 20 parameters to generate views. On the other hand, the latency of introducing LEAVES in SimCLR can be negligible. Under the training environment of AWS p3. 2xlagge (dual NVIDIA V100 GPUs) with the batch size N of 128 for 100 training epochs on Sleep-EDFF dataset, the baseline SimCLR takes 578.0 seconds per epoch average; whereas the SimCLR with LEAVES takes 390.8 seconds per epoch. Because augmentations are programmed as part of the model in LEAVES, which leverages GPU to accelerate computations, thus resulting in even shorter training time than the baseline SimCLR.
CONCLUSION
In this work, we introduced a simple but effective LEAVES module learn augmentations for time- series data in contrastive learning. With an adversarial training manner, the proposed method op- timizes the sypar-parameters for data augmentation methods in contrastive learning. The proposed method was evaluated on datasets, and the results showed performance improvements from the baselines. Particularly, without hyper-paramettr tuning. the proposed LEAVES outperformed the SimCLR baseline on 3 out of 4 downstream applications. We also demonstrated the advantages of using our method in terms of preserving intact information in augmented views, especially on ECG time-series data, compared to the SOTA work. In the future, we will introduce more augmentation methods to LEAVES to improve the variability of the module and explore its potential in tuning aug- mentations in supervised learning. Also, we will apply data. Further, investigating the interpretability of LEAVES is also an interesting direction for us to better understand the data augmentation policies in contrastive learning.
method to a broader range of time-series
ACKNOWLEDGMENTS
Use unnumbered third-level headings for the acknowledgmentss All acknowledgments, including those to funding agencies, go at the end of the paper.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we described the implementation details of the model and conducted evaluation experiments on publicly accessible datasets. We followed the procedures in the previous studies to process the data and stated the pre- prepprocsssing details in Appendix. In addition, we uploaded our example codes as part of the supplemental materials at this stage. Upon acceptance of the manuscript, we will upload our work to a GitHub repository.
ETHICS STATEMENT
We conducted experiments and evaluated our proposed methods on public time-series datasets, in- cluding the data collected from human subject studies. The proposed LEAVES has shown its ability to help improve the contrastive learning model performance in clinical applications, such detect- ing sleep apnea, arrhythmia, etc., by automatically tuning parameters for data augmentations without any requirements in domain knowledge. In addition, the proposed method can learn and generate different augmentations for samples, which can be considered an advantage in diversifying training sets in clinical applications and improving model generalizability, especially when researchers have limited but valuable data collected from study participants.
However, similarly to all other data-driven solutions in clinical studies, there exist risks in our method of making biased predictions clinical applications. For example, the generated distor- tion on clinical time-series samples may bias the intact information for certain clinical usage. For example, the temporal location changes in ECG may distort information such as the heart rate of patients. Moreover, other factors, such as poor data quality, heterogeneity among different study participants, and annotation biases, may lead inaccurate predictions. Thus, researchers should be aware of the limitations of our approach.
Moreover, all datasets used in this work are anonymized. The experiments were conducted using the datasets from PhysioNet, following the PhysioNet privacy policy [link] strictly. We ensure the datasets are not associated with any identifiable personal information from participants.
REFERENCES
Hung-Yo Chang, Cheng-Ye Yu Yeh, Chung-Te Lee, and Chun-Cheng Lin. A sleep apnea detection system based on one-dimensional deep convolution neural network model using single-lead electrocardiogram. Sensors, 20(15):4157, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Lee. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Nidhi Dua, Shiva Nand Singh, and Vijay Bhaskar Semwal. Multi-input enn-gru based human activity recognition using wearable sensors. Computing, 103(7):1461-1478, 2021.
Hengyang Fang, Changhua Lu, Feng Hong, Weiwei Jiang, and Tao Wang. Sleep apnea detection based on multi-scale residual network. Life, 12(1):119, 2022.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23)e211--222 2000.
10

---

Under review as a conference paper at ICLR 2023
Bryan Gopal, Ryan Han, Gautham Raghupathi, Andrew Ng. Geoff Tison, and Pranav Rajpurkar. 3kg: Contrastive learning of 12-lead electrocardiograms using pawpooooicall--nnppieee augmen- tations. In Machine Learning for Health, pp. 156-167. PMLR. 2021.
Bean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraard Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervvsed learning. Advances in neural information processing systems, 33:21271-212844 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Daniel Ho, Eric I Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pp. 2731-2741. PMLR, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterization with gummel-sooftmax arXiv preprint arXiv:1611.011 2016.
Bob Kemp, A Zwinderman, B Tuk, H Kamphuisen, and J Oberyé. Sleep-edf database expanded. 2018.
Kunyang Li, Weifeng Pan, Yifan Li, Qing Jiang, and Guanzheng Liu. A method to detect sleep apnea based deep neural network and hidden markov model using single-lead ecg signal. Neu- rocomputing, 294:94-101 2018.
Xiaoyu Li, Chen Li, Yuhua Wei, Yuyao Sun, Jishang Wei, Xiang Li, and Buyue Qian. Bat: Beat- aligned transformer for electrocardiogram classification. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 920-329. IEEE, 2021.
Yong Li and Luping Wang. Human activity recognition based on residual network and bilstm. Sensors, 22(2):635, 2022.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Dada: differentiable automatic data augmentation. arXiv preprint arXiv:2003.03788. 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment.
Advances in Neural Information Processing Systems, 32, 2019.
Aoming Liu, Zehao Huang, Zhiwu Huang, and Naiyan Wang. Direct differentiable augmenta- tion search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12219-12228, 2021.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.0077 2016.
Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. Neurokit2: A python toolbox for neurophysiolog- ical signal processing. Behavior Research Methods, Feb 2021. ISSN 1554-3528. doi: 10.3758/ URL http::::/doo...ooo.. 37 785 000.333/6//11200000 0001://....131113222-00000001111
Temesgen Mehari and Nils Strodthoff. Self-supervised representation learning from 12-lead ecg data. Computers in Biology and Medicine, 141:105114, 2022.
Sakorn Mekruksavanich and Anuchit Jitpattanakul. Deep convolutional neural network with mns for complex activity recognition using wrist-worn wearable sensor data. Electronics, 10(14):1685, 2021.
Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. Contrastive representation learning for electroenccehallogamm classification. In Machine Learning for Health, pp. 238-253. PMLR, 2020.
11

---

Under review as a conference paper at ICLR 2023
Sajad Mousavi, Fatemeh Afghah, and U Rajendra Acharya. Sleepeegnet: Automated sleep stage scoring with sequence to sequence deep learning approach. PloS one, 14(5)e0216456, 2019.
Fernando Moya Rueda, René Grzeszick, Gernot A Fink, Sascha Feldhorst, and Michael Ten Hom- pel. Convolutional neural networks for human activity recognition using body-worn sensors. In Informatics, volume 5, pp. 26. Multidisciplinary Digital Publishing Institute, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Thomas Penzel, George B Moody, Roger G Mark, Ary L Goldberger, and J Hermann Peter. The apnea-ecg database. In Computers Cardiology 2000. Vol. 27 (Cat. 00CH37163), pp. 255-258. IEEE, 2000.
Mathias Perslev, Michael Jensen, Sune Darkner, Poul Jørgen Jennum, and Christian Igel. U-time: A fully convolutional network for time series segmentation applied to sleep staging. Advances in Neural Information Processing Systems, 32, 2019.
Mathias Perslev, Sune Darkner, Lykke Kempfnne, Miki Nikolic, Poul Jørgen Jennum, and Christian lgel. U-sleep: resilient high-frequency sleep staging. NPJ digital medicine, 4(1):1-12, 2021.
Huy Phan, Kaare Mikkelsen, Oliver Y Chén, Philipp Koch, Alfred Mertins, and Maarten De Vos. Sleeptransformer: Automatic sleep staging with interpretability and uncertainty quantification. IEEE Transactions on Biomedical Engineering. 69(8):2456-266) 2022.
Attila Reiss and Didier Stricker. Introducing new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers, pp. 108-109. IEEE, 2012.
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolfb, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against im-age cor- ruptions by playing the game of noise. 2020.
Sinam Ajitkumar Singh and Swanirbhar Majumder. A novel approach osa detection using single- lead eeg scalogram based on deep neural network. Journal of Mechanics in Medicine and Biology, 2019.
Sandra Šmigiel, Krzysztof Palczyáski, and Damian Ledziński. Eeg signal classification using deep learning techniques based on the ptb-xl dataset. Entropy, 23(9): 121, 2021.
Akara Supratak and Yike Guo. Tinysleepnet: An efficient deep learning model for sleep stage scoring based on raw single-channel eeg. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 641-644. IEEE, 2020.
Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks: Learning views for unsuper- vised representation learning. arXiv preprint arXiv:2010.07432, 2020.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020.
Terry Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. Data augmentation of wearable sensor data for parkinson's 's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international conference on multimodal interaction, pp. 216-22260 116-220,220 2017.
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima Lunze, Wojeiech Samek, and Tobias Schaeffter. Ptb-xl, large publicly available electrocardiography dataset. Scientific data, 7(1):-115, 2020.
Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-supervised representation learning for time series. Pattern Recogni- fion Letters, 155:54-61, 2022.
12

---

Under review as a conference paper at ICLR 2023
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprini 17Xiv:1710.09412. 2017.
Jing Zhang, Deng Liang, Aiping Liu, Min Gao, Xiang Chen, Xu Zhang, and Xun Chen. Mibf-net: multi-lead- healbillad-brancch fusion network for multi-class arrhythmia classification using 12-lead eeg. IEEE Journal of Translational Engineering in Health and Medicine, 9:1-11, 2021.
Junbo Zhang and Kaisheng Ma. Rethinking the augmentation module in contrastive learning: Learn- ing bierarchical augmentation invariance with expanded views. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 16650-16659, 2022.
Zhidong Zhao and Yefei Zhang. Sqi quality evaluation mechanism of single-lead eeg signal based on simple heuristic fusion and fuzzy comprehensive evaluation. Frontiers in physiology, 9:727, 2018.
13

---

Under review as a conference paper at ICLR 2023
A EXTRA INFORMATION OF DATA AUGMENTATIONS
A.1 DATA AUGMENTATION FOR TIME SERIES DATA
Jittering is method of adding randomly generated noise to the original signal, and the generated noise obeys the Normal distribution:
X - X X EJ ((
(4)
where EJ E R RNxXLL represents the sampled noises. dr is the hyper-parameter that needed to be determined.
Scaling is to change amplitude of the signal by multiply the signal with random factors by each signal channel of the data:
X ..., &c] = 1 csæ2, ..., Eaaz,, es ((1
(5)
where Es E RNC represents the sampled factors. ag is the determined.
that needed to be
MagW (Magnitude Warping) Um et al. (2017). This method utilized randomly generated smooth curve to corrupt the magnitude of the original signal. For each data channel, we sample k nodes from (11 MM) which results knot E Nx Then, we interpolate knot evenly with linear function and form εM. The transformed view can be considered as:
x= X + EM
(6)
In this work, we fix to be 8 and only focus on the hyper-parameter d..
TimeW (Time Warping) Um et al. (017)). TimeW distorts the time intervals between time steps to corrupt the temporal locations of the original signal. Similarly as MagW, this method locates k nodes by (11,77)) and interpolate the data based randomly generated smooth curve. Same as our implementation of Mag W, we linearly interpolate k nodes to define knot E IR NxCxk as the curve for T.meW. We fix i to and focus on the hyper-parameter OT.
Permutation Um et al. (2017). Permutation is also method for distorting the temporal locations of the original data. The permutation evenly splits the original signal into K segments and shuffles the order of the segments:
X shuffle segment,.... segmmet,,)) A E K]
where K is the hyper-parameter to be determined.
A.2 REPARAMETERIIZTTOON
The aforementioned augmentation methods involve non-differentiabl operations such as sampling values S from certain distributions. For example, the Jitter method samples noise from normal distribution of (0, a.), which breaks the computational graph of the network. To propagate the gradients back to the a, we applied reparameterization tricks sample values for the augmentation operations. A randomly generated matrix e with values from 0 to is utilized.
Normal Distribution The reparameterization the Normal distribution can be considered as the following formula:
5= μ +
Where μ and a are the mean and standard deviation values for the Normal distribution.
Uniform Distribution The reparameterization of the Uniform distribution can be represented
5 = I (h I) c
Where and h represents the low and high bound of the Uniform distribution.
Bernoulli Distribution We used the RelaxedBemnouli distribution (Maddison al. 2016) to approximate the Bernoulli Distribution:
S log(p) + log(c) log(1 - c)
(10)
14

---

Under review as a conference paper at ICLR 2023
p control the probability of drawing the positive samples, and t [0. +oc) represents the tempera- ture. which controls the degree of approximation. When equals 0, the distribution is equivalent to the Bernoulli distribution; whereas if approaches +oo, the resulting samples approach the constant value 0.5. In this work, we fixed to be 0.01.
B DATASETS
B.1
PTB-XL:
The PTB-X XI Wagner al. (020) dataset is large dataset containing 21,837 clinical 12-lead ECG records from 18,885 patients of 10 second length, where 52% are male and 48% are female with ages range from rates: 100Hz and 500 Hz, available in the dataset. The raw ECG data are annotated by two car- diologists into five major categories, including normal ECG (NORM), myocardial infarction (MI), ST/T Change (STTC), Conduction Disturbance (CD) and Hypertrophy (HYP). The dataset contains a comprehensive collection of many different co-occurring pathologies and a large proportion of healthy control samples. To ensure fair comparison of machine learning algorithms trained on the dataset, we follow the recommended splits training and test sets.
95 years (median 62 and interquantile range of 22). There are two sampling
B.2 APNEA-ECG
The Apnea-ECG Penzel al. (2000) dataset studies the relationship between human sleep apnea symptoms and the heart activities (monitored by ECG). This database can be accessed through Physione(Gooldbegger et al. (2000). This dataset contains 70 records with a sampling rate of 100 Hz, from where 35 records were divided into training, and the other 35 were divided into the test set. The duration of the records varies from slightly less than 7 hours to nearly 10 hours. The labels were the annotation of each minute of each recording indicating the presence or absence of sleep apnea. Thus, we split the ECG recording into each minute, which was a total of 6000 data points for each separation. We extracted 17233 samples for the training set and 17010 samples for the test set. And the ratio of non-apnea and apnea samples in the training set was 61.49% to 38.51%.
B.3 SLEEP-EDFE
The Sleep-EDF (expanded) Kemp al. (2018) dataset contains whole-night sleep recordings from 822 subjects with physiological signals and sleep stages that were annotated manually by well- trained technicians. In this dataset, the physiological signals, including Fpz-Cz/Pz-O7 ECG, EOG, and chin EMG, were sample in 100 Hz. To model the relationship between the sleep patterns and physiological data, we split the whole-night recordings into 30-second Fpz-Cz ECG segments as in Supratak & Guo (2020), which resulted in total of 42308 ECG and sleep pattern pairs. We divided 25% of the samples into test set according to the order the subject IDs.
B.4 PAMAP2
The PAMAP2 Reiss & Stricker (2012) physical activity monitoring dataset consists data of 18 differ- ent physical activities, including houschold activities (sitting, walking, standing, vacuum cleaning, ironing, etc) and variety of exercise activities (Nordic walking, playing soccer, rope jumping, etc), performed by 9 participants wearing three inertial measurement units (IMU) and heart rate monitor. Accelerometer, gyroscope, magnetometer and temperature data are recorded from the IMUs placed on three different locations (1 IMU over the wrist on the dominant arm, IMU on the chest and IMU on the dominant side's s ankle) with sampling frequency 100Hz. Heart rate data are recorded from the heart rate monitor with sampling frequency 9Hz. The resulting dataset has 52 dimensions (3 x 17 (IMU) + (heart rate) = 52). Following the same setting as used in Moya Rueda (0188) Tamkin et al. (2020), We linearly interpolate the missing data (upsampling the sampling frequency from 9Hz to 100Hz. for heart rate), then take random 10s windows from subject record- ings with an overlap of 7s, using the same train/validation/ees splits. As mentioned in Moya Ruedal et al. (2018): TTmmkin et al. (2020), 12 of the total 18 different physical activities are used in the experiments.
15


Reviews:
[{"review_id": "obvJhdFhmFb", "reviewer": "Reviewer_KPs5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The papers approaches the problem of contrastive learning of time series representations by learning parameters for time series augmentations. This allows good \"views\" of the time series to be directly incorporated into SimCLR, leading to reasonable performance gains against the baselines.", "summary_of_the_review": "While I find this work well-written, and appreciate the extensive experiments, I also note the two main flaws I have listed above. Both those positive and negative points justify, in my opinion, the score I have given.", "strength_and_weaknesses": "Strenghts:\n- The paper is easy to follow and the idea is clearly presented and explained.\n- A diverse set of experiments are done to defend the claims of the paper. The model performs reasonably well against the proposed baselines.\n\nWeaknesses:\n- Differentiable augmentations are not a novel concept, and the adversarial framework to learn them is quite reminiscent of AdCo [1].\n- One setup missing is the forecasting experiments in e.g. TS2Vec. This method could be directly applied to it, and results on a set of orthogonal tasks would be beneficial. More importantly, TS2Vec, CoST, etc. missing as (strong) baselines: comparing to them would strenghten the experiments section.\n\n\nReferences:\n[1] AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fUTTYrWD65", "reviewer": "Reviewer_MoJJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The papers proposes to learn the view generation component for contrastive learning for time series data.\nThe alogithm is simple and includes usage of 5 common augmentations techniques and one new one (TimeDis) applied sequentialy to generate a view.\nThe loss function is similar to SimCLR and is maximized wrt parameters of LEAVES in an adversarial game.\nThe authors demonstrate improvement for the considered four medical datasets.", "summary_of_the_review": "The approach seems interesting and simple, inheriting these qualities from a similar approach from CV ViewMaker.\nWhile the obtained approach and results deserve attention, they provide a limited contribution to the community.", "strength_and_weaknesses": "Strengths:\n- The name LEAVES is really good\n- Results for three of four considered datasets are better than baseline results\n- The number of hyperparameters to learn for LEAVES is small\n- The idea of using the reparametrization trick is nice (while natural)\n\nWeakness, methods:\n- According to the presented approach, we should select maximum distortion all the time. So, it is not clear, why this effect is not observed in practice, making the solution trivial\n- The comparison and overall presentation of the method can benefit from looking on additional relevant papers that follow simiar ideas [3, 4] \n\nWeakness, experiments:\n- It seems, that to maximize the loss, we should distort the views with maximum values of sigma. It is true for most of the hyperparameters. Can you provide performance of LEAVES for hyperparameters that correspond to maximum distortion?\n- What are the architectures for the considered datasets? Why they are selected in such way?\n- Recent works on learning time series representations list particular important augmentations as well as ways to learn reporesentations to solve donwstream tasks [1, 2]. Why this work doesn't compare to those approaches and uses datasets different from that mentioned in these papers?\n- It seems, that non-contrastive approaches are superior to SimCLR even for time series data [6]. The paper would benefit from adding this trainable view generator to e.g. BYOL similar to [3]\n- Broader selection of the datasets for study would help to understand, what augmentations are useful in what settings. For example, it seems, that some of them can harm performance for periodic time series\n- The ablation study doesn't demonstrate that all used augmentations are essential to the quality of the considered approach. In particular, does TimeDis contributes to the overall quality of the approach? Now it is not clear\n\nTechnical and presentation remarks:\n- The exact training procedure is missing. Do we train the view generator and the encoder simultaneously? What other strategies are possible? Can other approaches improve the result?\n- Figure 5: what is at the axes? Iterations or Epochs for x? Hyperparameter value for y?\n- As the focus of the new method is performance, it makes sense to unite Table 1 - Table 4 in single Table in the main text, providing full version in appendix. It will help to save the space in the main text for desired technical details, that are missing now.\n\nLiterature:\n1. Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, Bixiong Xu. TS2Vec: Towards Universal Representation of Time Series. AAAI. 2022\n2. Eldele E., Ragab M., Chen Z., Wu M., Kwoh C. K., Li X., Guan C. Time-Series Representation Learning via Temporal and Contextual Contrasting. IJCAI. 2021. \n3. Shi, Yuge, et al. \"Adversarial masking for self-supervised learning.\" ICML. 2022.\n4. Reed, Colorado J., et al. \"Selfaugment: Automatic augmentation policies for self-supervised learning.\" CVPR. 2021.\n5. Koyama, Masanori, et al. \"Contrastive Representation Learning with Trainable Augmentation Channel.\" arXiv preprint arXiv:2111.07679 (2021).\n6. Marusov, Alexander, Valerii Baianov, and Alexey Zaytsev. \"Non-contrastive approaches to similarity learning: positive examples are all you need.\" arXiv preprint arXiv:2209.14750 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CYXXTp9gfI", "reviewer": "Reviewer_9y2Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose LEAVES, which is a lightweight module for learning views on time-series\ndata in contrastive learning. The LEAVES is optimized adversarially against the contrastive loss\nto generate challenging views for the encoder in learning representations.\n\n", "summary_of_the_review": "In this paper, the author introduced a simple but effective LEAVES module to learn augmentations for timeseries data in contrastive learning. With an adversarial training manner, the proposed method optimizes the hyper-parameters for data augmentation methods in contrastive learning. The idea is OK but novelty is not very obvious.", "strength_and_weaknesses": "Strength:\nThis paper proposed AutoAugment in their methods, which is quite useful for contrastive learning.\n\nWeakness:\nTheir experiment just show a sole example on some medical case. It does not show a good contribution to the whole fields. Also their novelty is very empirical-based, I am not sure if this is suitable for this conference. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "05CcjWX60dE", "reviewer": "Reviewer_iTL4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of automatically generating augmentations for time-series data in the contrastive learning framework. It proposes LEAVES, a module that learns the hyper-parameters of the time-series data augmentations. The module is trained adversarially, i.e. maximizing the contrastive loss while the encoder network is minimizing the contrastive loss. Experiments on four public time-series datasets demonstrate the efficacy of the proposed method.", "summary_of_the_review": "This paper addresses an interesting and important question - how to automatically generate good time-series data augmentations for contrastive learning. The empirical results of the proposed method are good, but the proposed method is not designed delicately and important details and experiments are missing. Therefore, the paper needs more work before it is ready to be published in my opinion, but I am happy to raise the score if all my concerns are addressed. ", "strength_and_weaknesses": "Pros:\n- Interesting topic. Contrastive learning has been shown to be very powerful, but it requires well-designed data augmentations, which has not been well studied for time-series data (compared to that for image). A good automatic time-series data augmentor should be beneficial for a great many real-world applications. \n- Good results. This paper conducts experiments on four real-world time-series datasets, and the proposed method outperforms the supervised baseline and ViewMaker (an automatic domain-agnostic data augmentor) consistently. It is also very helpful to have SOTA methods listed in the tables. \n- Nice ablation studies. The ablation studies show that the downstream performance for the four real-world tasks will be influenced by the choices of the hyper-parameters of data augmentations by a non-negligible margin, and the previous method, i.e. ViewMaker, is not able to generate satisfiable augmentations for time series data. Those underscore the need of designing a better automatic time-series data augmentor.\n\nCons:\n- This paper only uses SimCLR as the contrastive learning framework to demonstrate everything. However, several contrastive learning frameworks (e.g. BYOL,  MoCo v2/v3) have been proposed after SimCLR and have been shown to outperform SimCLR empirically. How to adapt the automatic augmentation module into those frameworks (that should be easy for the proposed method) should be described, and whether the module can improve the performance of those frameworks empirically should be verified. \n- The novelty of the method is a little bit low. The adversarial framework is the same as ViewMaker, with the only difference being the types of hyperparameters to be learned. Moreover, the proposed method can only learn the hyperparameters of augmentations that are already well-designed, and it can not generate new augmentations that have not been discovered yet. Therefore, using such a module still requires researchers to manually design augmentation types first. \n- The organization of the paper can be improved. The definition and at least of a short explanation of the data augmentation used in the proposed module, i.e. Permutation, TimeDis, MagW, Scale and Jitter, should be included in the method section. Only from Figure 3, it is very unclear what they are. Moreover, the variables to be learned should be highlighted clearly, and why they are learnable (differentiable) should also be explained. \n\nQuestion:\n- What is the difference between MagW and Jittering? Isn’t MagW included in Jittering?\n- In Equation (1), what is \\sigma_P? Isn’t the variable to be learned for Permutation the number of segments K?\n- Why are Permutation and TimeDis differentiable?\n- How do you know when to stop during the training process? As shown in Figure 5, some of the parameters, like Perm Max Segments on Sleep-EDFE and on Apnea-ECG, are not converging eventually. \n- Are the hyper-parameter values learned by LEAVES meaningful? Have you tried to re-train the encoder network using the (fixed) hyper-parameters learned by the adversarial training process? Will that deliver better or worse results compared to training the encoder and the hyper-parameters together?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject