Finding Order in Chaos: A Novel Data Augmentation Method for Time Series in Contrastive Learning
Authors: Berken Utku Demirel
Christian Holz
Venue: NeurIPS 2023 poster (2023)
Keywords: Contrastive learning
Time-series
Augmentation

Abstract: The success of contrastive learning is well known to be dependent on data augmentation.
Although the degree of data augmentations has been well controlled by utilizing pre-defined techniques in some domains like vision, time-series data augmentation is less explored and remains a challenging problem due to the complexity of the data generation mechanism, such as the intricate mechanism involved in the cardiovascular system.
Moreover, there is no widely recognized and general time-series augmentation method that can be applied across different tasks.
In this paper, we propose a novel data augmentation method for time-series tasks that aims to connect intra-class samples together, and thereby find order in the latent space.
Our method builds upon the well-known data augmentation technique of mixup by incorporating a novel approach that accounts for the non-stationary nature of time-series data.
Also, by controlling the degree of chaos created by data augmentation, our method leads to improved feature representations and performance on downstream tasks.
We evaluate our proposed method on three time-series tasks, including heart rate estimation, human activity recognition, and cardiovascular disease detection. 
Extensive experiments against the state-of-the-art methods show that the proposed method outperforms prior works on optimal data generation and known data augmentation techniques in three tasks, reflecting the effectiveness of the presented method. 
The source code is available at double-blind policy.

Finding Order in Chaos: A Novel Data Augmentation Method for Time Series in Contrastive Learning
Berken Utku Demirel
Christian Holz
Department of Computer Science ETH Zürich, Switzerland berken weeedeee@eiiieeeeeeeeeeeeeeeeeee ethz. ch
Department of Computer Science
ETH Zürrch, Switzerland christian. Auristian.hozz0nnt holz@inf eet.zinf.e...........mmm... ethz. ch
Abstract
The success of contrastive learning is well known to be dependent data aug- mentation. Although the degree of data augmentations has been well controlled by utilizzigg pre-defined techniques in some domains like vision, time-series data augmentation less explored and remains a challenging problem due to the com- plexity of the data generation mechanism, such as the intricate mechanism involved in the cardiovascular system. Moreover, there no widely recognized and gen- eral time-series augmentation method that can be applied across different tasks. In this paper, we propose a novel data augmentation method for quasi-periodic time-series tasks that aims connect intra-class samples together, and thereby find order in the latent space. Our method builds upon the well-known mixup technique by incorporating novel approach that accounts for the periodic na- ture of non-stationary time-series. Furthermore, by controlling the degree of chaos created by data augmentation, our method leads improved feature rep- resentations and performance on downstream tasks. We evaluate our proposed method on three time-series tasks, including heart rate estimation, human ac- tivity recognition, and cardiovascular disease detection. Extensive experiments against state-of-the-art methods show that the proposed approach outperforms prior works optimal data generation and known data augmentation techniques in the three tasks, reflecting the effectivvnesss of the presented method. Source code: bettss///gthhh www.cooo///i../e aeamm/ehhssipee wtw.m//thssppplaa/nnii.gggg Order in_Chaos
1 Introduction
Self-supervised learning methods have gained significant attention they enable the discovery of meaningful representations from raw data without explicit annotations. These self-supervised methods learn representations without labels by designing pretext tasks that transform the unsupervised representation learning problem into a supervised one such as predicting the rotation of images [[], contexts (2 3). Among these methods, contrastive learning (CL), which learns to distinguish semantically similar examples over dissimilar ones, stands out as a powerful approach self- supervised learning across various domains including computer vision ( speech recognition 7
10), and natural language processing 14
The success of contrastive learning relies on the creation of similar and dissimilar examples, which typically achieved through the use of data augmentations [15] Recently, was shown that data augmentations have role to create "chaos" between different intra-class samples such that they become more alike. For example, two different cars become very similar when they both cropped to the whecls. [[7]. However, in time-series data, creating similar samples with augmentation techniques is more challenging due to the complexity of the dynamical data generation mechanisms [8]. Moreover, research on contrastive learning for time series has demonstrated the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

absence of a unique data augmentation technique that consistently performs better than others in different tasks [19 20). Instead, the choice of augmentation depends on the contextual characteristics of the signal, such as perturbing the high-frequency content of a signal that carries characteristic information low frequencies does not generate useful data samples that are helpful for contrastive learning to learn class invariant features 22
Considering these limitations, in this work, we first propose a novel data augmentation method for time series data by performing tailored mixup while considering the phase and amplitude information as two separate features. Then, we perform specific operations for both features to generate positive samples by controlling the mixup coefficients for each feature to prevent aggressive augmentation. Specifically, our method employs a technique that controls the mixup ratio for each randomly chosen pair based on their distance in the latent space which is acquired through the use of a variational autoeneoder (VAE), whose objective is to learn disentangled representations of data without labels. To this end, subjecting to the distance constraint in the latent space, the mixup tries to connect semantically closer samples together more aggressively while preventing the excessive interpolation of dissimilar samples that are likely to belong to different classes. Therefore, the purpose of our proposed method for quasi-periodic time-series data augmentation is to find an order in "chaos" between different samples such that they become more alike by interpolating them in novel manner to prevent the loss of information. We summarize our contributions as follows:
We propose a novel mixup method for non-stationary considering phase and magnitude two separate features to generate samples that enhance intra-class similarity and help contrastive learning to learn slass-separated representations.
time-series data by
We present a novel approach for sampling mixup coefficients for each pair based on their similarity in the latent space, which is constructed without supervision while learning disentangled representations, to prevent aggressive augmentation between samples.
We show that the tailored mixup with coefficient sampling consistently improves the perfor- mance of contrastive learning in three time-series tasks compared to prior mixup techniques and proposed augmentation methods that generate optimal/hard positives or negatives.
Preliminaries
2.1 Notation
We use lowercase symbols (r) to denote scalar quantities, bold lowercase symbols (x) for vector values, and capital letters (X) for random variables. Functions with a parametric family of mappings are represented as fo(-) where 0 is the parameters. The discrete Fourier transformation of a real- valued time series sample is denoted as F()), yielding complex variable as X₃ where X C and k E [0, f2] is the frequency with the maximum value of Nyquist rate. The amplitude and phase of the F(x) are represented as A(x) and P(x). The real and imaginary parts of complex variable are shown as Re(.) and Im(.). The detailed calculations for operations are given in the Appendix A.
2.2 Setup
We follow the common CL setup as follows. Given a dataset [xx))]{{ where each x, consists of real-valued sequences with length L and C' channels. The objective is to train learner So which seeks to learn an invariant representation such that when it is fine-tuned on dataset D, = ((x,,,) with M < K and y, [I....,NV it can separate samples from different classes.
2.3 Motivation
As stated by prior works, mixup-based methods have poor performance in domains where data has non-fixed topology, such as trees, graphs, and languages (22) (23. Here, demonstrate how derived our proposed method by revealing the limitations of mixup for time series theoretically while considering the temporal dependencies and non- stationaritiess
Assumption 2.1 (SNR Matters), There exist one multiple bandlimited frequency ranges of interest f", where the information that average raw time-series data conveys about the labels (i.e., I(y; x))
is directly proportional to normalized signal power in that frequency range in Equation 1
2

---

N
2
I(y; x / 52(0) S Sz(O) where S₂( = lim
Σ n=-ル
f2
mne
N-+0 2N
Assumption 2.1 states that the information from time series depends on its signal-to-noise ratio (SNR). Prior works showed that specific frequency bands hold inherent information about the characteristics of time series, which helps classification [21 24
Assumption 2.2. The true underlying generative process f(), a given data distribution = {xx { is quasiperiodic, i.e., f(x 7) gxx (x)), where can be either fixed varied.
Assumption 2.2 posits that the observed data samples from the distribution D are generated by quasiperiodic function. This is minimal assumption since the quasiperiodicity the relaxed version of the periodic functions. In simpler terms, quasiperiodicity can be described as the observed signals exhibiting periodicity on a small scale, while being unpredictable on a larger scale. And, several prior works showed that the data generation mechanism of time- series data for several applications in the real world are quasiperiodic (0). Therefore, Assumption 2.2 is realistic.
Proposition 2.3 (Destructive Mixup). If Assumptions 2.7 and 2.2 hold, there exist Beta(a, a) or À ~ U(3, 1.0) with high values of B such that when linear mixup techniques are utilized, the lower bound of the mutual information for the augmented sample distribution decreases to zero.
0 S I(y:xt) < I(y;x"),
where x is the optimal sample,
x = xx + (1 A) and J. Sz-
S..
Proofs can be found in Appendix A This proposition indicates that although the augmented samples are primarily derived from anchor samples (x) with high ratios, the resulting instances may not contain any task-relevant information. In other words, the augmentation process can potentially discard the whole task-speeific information. This destructive behavior of mixup for quasi-periodic data can be attributed to the interference phenomenon in which two waves interact to form the resultant wave of the lower higher amplitude according to the phase difference as shown in Proposition 2.3
Method
We introduce novel approach to overcome the limitations of mixup by treating the magnitude and phase of sinusoidal signals as two distinct features with separate behaviors. Subsequently, we apply tailored mixup operations each feature, considering their specific characteristies and effects.
We perform the linear mixup for the magnitude each sinusoidal. However, for the phase, we take different approach and bring the phase components of the two coherent signals together by adding small value to the anchor's phase the direction of the other sample. The mixup operation performs the linear interpolation of features (31], however, interpolation of two complex variables can result in a complex variable whose phase and magnitude are completely different/far away from those two, ie.. mixup can be destructive extrapolation rather than the interpolation of features. Therefore, mix the phase of two sinusoidal as follows. We start by calculating the shortest phase difference between the two samples, denoted AQ, as described in Equation
[P(x)
P(X)] (mod 2T)
0 2m, if >
AO
0, otherwise
The sign of the calculated phase difference provides information about the relative phase location of the other sample, in either a clockwise or counterclockwise direction in the phasor diagram. And, the absolute value of it represents the shortest angular difference between two samples in radians. Based
We use phase in radians throughout the paper in the range (-T,, nT, T]

---

on the calculated phase difference between two samples, we perform mixup operation to generate diverse positive samples in Equation I such that phase and magnitude of augmented instances are interpolated properly according to the anchor sample x, without causing any destructive interference.
x = = (A(xt)
where A(x*) = AxA(x) + Ax) and
P(x) - AO| * (1 - -X),, if > 0
P(xt) { P((x AO| (1 Ap), otherwise
The proposed method which mixes the magnitude and phase of each frequency component with tailored operations. not only prevents destructive interference between time series, resulting in an increase in the lower bound of mutual information (as shown in Theorem 3.I). but also generates diverse augmented instances with the same two samples by using two different mixing coefficients.
Im
b)
Im
π
π
2
2
xt === (A(x)t)))
A(R)
x = ( (A(x)))
A
P(x)
P(*)
Re
Re
π
2(xx))
== (((())
7
n Pix 1
P(x)
x
K= x= (1
x
X
A(x² )= ) Axx) + A)* A(R)* (KA(()) cao (P(x) P(x))
A(x ) = A Ax) + +(1 Aa)
AA(x) sin ((x)) + (1 (3) sim( (xx))
p(x*) = P(x) A0 (1 Ar),
if
P(x")= = aretan
A4(x) cos P (1 () cos( (Px))
P(x) A9 (1 - Ar), otherwise
The phasor representation of linear mixup a), and proposed mixup b). The anchor, randomly chosen sample, and generated instances are represented as x, X, and x+ respectively.
Figure I:
(Guarantees for Mixing). Under assumptions 2.1 and 2.2 given any X E (0, 1]. the Theorem 3.1
mutual information for the augmented instance lower boundee by the sampled À and anchor
XI(y;)) < I(y:xt) < I(y;x") where x =F
(5)
We provide an intuitive demonstration in Figure 1 along with a detailed mathematical proof presented in Appendix [A] Our approach also offers increased flexibility in selecting the mixing coefficients of phase and magnitude, based on their sensitivities the mixing process as well as the augmentation degree for each randomly chosen pair. Since the degree augmentations has crucial importance for contrastive learning, there can be cases where augmentations are either too weak (intra-class features cannot be clustered together) or too strong (inter-elass features can also collapse to the same point) and lead to sub-optimal results [17]. To mitigate this issue and find order for augmentation degree, we search pairs of samples that are semantically closer, meaning they are more likely to belong to the aggressively
same class. We then perform the proposed mixup closer and diverse samples while decreasing the augmentation strength for less similar pairs. To find similar samples without labels, we train completely unsupervised 3- 3-VAE 332 that maps data points to latent space such that two random samples are semantically similar if they are close in the latent as shown in Proposition 3.2
these pairs, creating more
Proposition 3.2 (Consistency in Latent Space (3 ). Given a well-trained unconditional VAE with the encoder E() that produces distribution Pæ x the decoder D(.) that produces distribution qo (x|z) while the prior for is p(z), let 21 and Z2 be two latent vectors of two different real samples X1 and X2, i.e. E(x1) = Z1 and E(x2) = will have similar semantic label as in Equation 6
Z2. if the distance d(zı, 22) S 6, then D(z1) and D D(zz)
I(D(zz):)) - I(D(Zz):v)) Se,
(6)

---

where e stands for tolerable semantic difference, 8 is the maximun distance to maintain semantic consistency, and d(.) a distance measure such as cosine similarity between two vectors.
The above proposition with Theorem 3.1 motivates us to perform augmentation aggressively if two randomly chosen samples are semantically closer. Therefore, we sample the mixup coefficient for both phase and magnitude from a uniform distribution Ap U(3, 1.0) with low values of the distance between the latent vectors is below a threshold, otherwise, they are drawn from a truncated normal distribution As, Ap (N, o, 1.0) with high mean and low standard deviation.
4 Experiments
We conduct experiments on the proposed approach and compare it with other mixup methods or optimal/hard positive sample generation in the contrastive learning setup. During our experiments, we use SimCLR [15] framework without specialized architectures or memory bank for all baselines to have a fair comparison. Results with other CL frameworks can found in Appendix [E] Complete training details and hyper-parameter settings for datasets and baselines are provided in Appendix D
4.1 Datasets
We performed extensive experiments on eight datasets from three tasks that include activity recogni- tion from inertial measurements (IMUs), heart rate prediction from photoplethysmography (PPG), and cardiovascular disease classification from electrocardiogram (ECG). We provided short descriptions of each dataset below, and further detailed information with metries can be found in Appendix [
Activity recognition We used UCIHAR 34 HHAR 35 and USC 36 for activity recognition.
During the evaluation, assess the cross-person generalization capability of the contrastive models, iee. the model is evaluated on a previously unseen target domain. We follow the settings in GILE (37] treat each person as a single domain while the fine-tuning dataset is much smaller than the unsupervised one.
Heart rate prediction We used the IEEE Signal Processing Cup in 2015 (EEEE SPC) 38 and DaLia [39] for PPG-based heart rate prediction. The SPC provides two datasets, one smaller with lesser artifacts (referred to as SPC12) (38 and a bigger dataset with more participants including heavy motions (referred to as SPC22). In line with previous studies, we adopted the leave-one-session- (LOSO) cross-validation, which involves evaluating methods on subjects or sessions that were not used for pre-training and fine-tuning.
Cardiovascular disease (CVD) classification We conducted our experiments on China Physiologi- cal Signal Challenge 2018 (CPSC2018) [40] and Chapman University, Shaoxing People's Hospital (Chapman) ECG dataset [41]. We selected the same four specific leads as in [42] while treating each dataset as a single domain with small portion of the remainder dataset used for fine-tuning the pre-trained model. We split the dataset for fine-tuning and testing based on patients (each patient's recordings appear in only one set).
4.2 Baselines
Comparison with prior mixup techniques We evaluate the effectiveness of our proposed mixup by comparing it with other commonly used mixup methods, including Linear-Mixup (31),
Binary-
Mixup 433, Geometric-Mixup (22), Cut-Mix 44 Amplitude-Mix [45] and Spec-Mix [46 When
we compare the performance of mixup techniques. we follow the same framework with [47 where
the samples of mixture operation only happen in current batch samples. And, the mixup samples are
paired with anchors, i.c., without applying mixup second times, for contrastive pre-training.
Comparison with methods for optimal sample generation We evaluate the performance of our proposed method by comparing with other data generation methods and baselines in contrastive learning setup while considering previously known augmentation techniques. Traditional data augmentations for time-series [19], such as resampling, flipping, adding noise, etc. InfoMin which leverages adversarial training strategy to decrease the mutual information between samples
5

---

while maximizing the NCE loss 48). NNCLR [49], which uses nearest neighbors in the learned representation space as the positive samples. Positive feature extrapolation [50], which creates hard positives through feature extrapolation. GenRep which uses the latent space of a generative model to generate "views" of the same semantic content by sampling nearby latent vectors [5]] Aug. Bank [21], which proposes augmentation bank that manipulates frequency components of sample with a limited budget. STAug [52). which combines spectral and time augmentation for generating samples using the empirical mode decomposition and linear mixup. DACL (22), which creates positive samples by mixing hidden representations. IDAA [53], which is an adversarial method by modifying the data to be hard positives without distorting the key information about their original identities using a VAE. More implementation details for each baseline are given in Appendix
4.3 Implementation
We use combination of convolutional with LSTM-based network, which shows superior performance in many time-series tasks [19 54 55 as backbones for the encoder fo(.) where the projector is two fully-connected layers. We use InfoNCE the loss, which optimized using Adam with learning rate of 0.003. We train with a batch size of 256 for 120 epochs and decay the learning rate using the cosine decay schedule. After pre-training. we train a single linear layer classifier on features extracted from the frozen pre-trained network, i.e., linear evaluation, with the same hyperparameters. Reported results are mean and standard deviation values across three independent runs with different seeds on the same split. More details about the implementation, architectures, and hyperparameters with the trained VAES are given in Appendix D
5 Results and Discussion
Tables 2 and present the results of our proposed approach compared to state-of-the-art methods for optid positive sample generation in contrastive learning setup across the three tasks in eight datasets. Additionally, Figure 2 compares our approach with prior mixup methods (e.g., linear mixup, cutmix) without applying any other additional augmentation techniques. Overall, our proposed method has demonstrated superior performance compared to other methods seven datasets, with the second-best performance in the remaining dataset, and minor performance gap.
Table I: Performance Comparison of ours with prior works in Activity Recognition datasets
UCIHAR
HHAR
USC
Method
ACC1
MFIt
ACC1
MFI1
ACC1
MF1t
Supervised
DCL 37
77.63
51.27
60.35
CoDATS 56
68.22
45.69
GILE 37
88.17
55.61
Self-Sapervised Traditional Augs.
87.05 1.07
86.13
0.96
85.48
1.16
84.31 + 1.31
53.47
1.10
52.09
0.95
NNCLR 49
85.31 - 0.91
83.56 1.25
83.16 + 1.32
82.15 1.22
55.41
+ 1.43
52.64 1.37
InfoMin 48
38.07 - 8.15
30.66 9.15
31.58 - 10.2
29.72
11.1
35.89
1 14.3
37.77
IDAA [53
82.23 1 0.69
79.84 0.89
88.98
0.55 89.01
59.23
1.10
56.11
PosET 500
88.13 0.91
87.35
85.77
+ 1.11
85.90
1.200+1.20
41.37
+
STAug 52
89.83 - 0.71
87.69 + 1.03
87.73
55.61
1 1.08
56.74
Aug. Bank 21
65.27 - 1.12
71.16 1.24
67.95 + 1.45
75.13 1.32
43.28
47.31 - 4.68
GenRep [1
87.22 = 1.05
86.48
0.95
87.05
0.95
86.45 0.90
10.13
49.50
2.73
DACL
P2
73.12 + 1.23
66.28
+ 1.11
80.89 + 0.91
81.31 1 0.78
53.61 4 2.60
51.76 1 2.21
Ours
91.60 + 0.65
90.46
0.53
88.05
1.05
87.95 - 1.10
60.13
0.75
59.13 0.69
From these tables, we can see that our proposed method significantly outperforms DACL, which sug- gests creating positive sample by mixing fixed hidden representations in intermediate layer (22). by a large margin (up to 20.8% with 10.1% % on average in activity recognition). This suggests that when the representations are not yet linearly separable at the beginning of the contrastive training process, the interpolated representations using mixup may be dissimilar to the actual interpolated samples and may not capture their underlying features. One interesting result from our experiments is that IDAA [53] exhibits comparable performance to our method in some datasets, and even slightly outperforms our approach the HHAR dataset for activity recognition. Despite using distinct methods to generate positive instances, i.c., adversarial and mixup, our approach and IDAA algorithm

---

Table 2: Performance comparison of ours with prior works in Heart Rate Prediction datasets
IEEE SPC12
IEEE SPC 22
DaLia
Method
MAE,
RMSF4
MAE.
RMSF)
MAR
RMSE,
Supervised
DCL
22.02 3.89
28.44
28.10
32.45
6.58
11.30
CNN Ensemble"
8.74
8.58
Self-Supervised
Traditional Augs.
20.67
+: 1.13
26.35
4: 0.98
16.84
+: 1.10
22.23 +: 0.72
12.01 + 0.65
21.09 + 0.86
NNCLR
44
20.28
2.21
28.23
1.63
23.49
1.54
28.75
3.66
11.56 - 0.63
19.95 -
InfoMin
48
36.84
+: 5.11
29.78
+ 7.31
31.58
+ 4.72
29.72
1 4.83
45.89 - 8.71
50.77 -9.72
IDAA
19.02
0.96
27.42
1.11
15.37
1.21
22.41
11.12
20.45 0.699
PosET 150
25.60
1.93
33.80
+ 2.71
23.42
+ 1.50
31.51
31.51+3.77
3.71
35.99
3.95
39.92 3.12
STAug 5
27.44 1.93
35.63
3.10
19.86
2.11
30.70
L.554
18.70
30.81
Aug. Bank
21
27.31 + 2.17
37.9
2.96
27.84
+ 2.03
36.41
+3.98
35.87
+ 4.18
40.61
+ 3.74
GenRep
51
21.02 1,41
28.42
1.65
15.67
1.23
22.33
25.41
1.62
36.83
1.87
DACL
22
21.85
1.63
28.17
+ 1.75
14.67
1.10
20.06
1.21
18.44
1.32
25.61 1.45
Ours
16.26 0.72
22.48
+ 0.95
12.25
0.47
18.20
0.61
10.57
0.55
20.37 0.73
The entire dataset, excluding the test, is utilized with labels, while DCT,, the laheled data size matches that the CT.
share similarities in approaches for positive instance generation in CL setup. The IDAA algorithm aims to create hard positive samples that lie near class boundaries without changing the identity of the sample, while our method interpolates two samples to produce positive instance that is similar to the original sample while adding noise to the phase and amplitude in the direction of a randomly chosen sample. In other words, both approaches try to keep the sample identity intact by taking special precautions while generating new positive instances, which may explain their similar performance in our experiments.
In contrast, approaches that do not prioritize preserving sample identity while generating samples or hidden representations often demonstrate suboptimal performance on average while exhibiting increased variability across tasks.
Examples of such methods include PosET [50), which generates hard positive samples to im- prove contrastive learning by extrapolating features, STAug [52], which empirical mode decompo- sition with linear mixup technique together, and InfoMin [48). which tries minimize mutual in- formation between two instances in an adversar- ial manner. The performance comparison of prior mixup techniques and our proposed one is shown in Figure On average, our proposed method outper- forms all other mixup techniques while reducing the variance across tasks. What is interesting about this figure is that while the linear (31 and ampli- tude mixup 45 reach our method some datasets for activity recognition, the performance of the lin- car mixup decreases heavily for the other two tasks whereas the amplitude mixup gives reasonable per-
Table 3: Performance comparison between ours and prior work in CVD.
CPSC 2018
Chapman
Method
AUC1
AUCt
Supervised
CNN [57
95,80
Casual CNN 58
97.70
Self Self-Supervised
Traditional Augs.
67.86 3.41
74.69
2.04
NNCLR
49
70.06 1:06-:200 2.05
77.19
2.41
InfoMin
48
64.48 1.48:6..11
56.34 + 9.12
IDAA
53
80.90
93.63
+ 0.91
PosET 50
2.12
78.27
2.34
STAug 52
74.15 1.15
93.88
1 0.87
Aug. Bank 21
81.78 1.24
94.75 + 0.90
GenRep
51
52.49
3.43
86.72
+ 1.13
DACL
82,38 0.84
92.28
+ 0.97
Ours
85.30
0.45
95.90 + 0.82
formance. This empirical outcome supports our initial theorem about the destructive effect of mixup, which suggests linear mixup other derivatives can discard the whole task-specific information in the generated positive sample for quasi muasi-priiodic signals even though the mixing coefficient is sampled from distribution such that the generated samples much closer to the anchor.
5.1 Ablation Studies
Here, we present comprehensive examination of our proposed method and the effect of its com- ponents on the performance. Mainly, we investigate the effect of the proposed mixup by applying the instance selection algorithm to the linear mixup (w/o Prop. Mixup). Then, we perform our proposed mixup with the constant X and Ap coefficients without investigating latent space distances between pairs (w/o Aug. Degree). Tables tables shows the performance when the proposed mixup method not applied while choosing mixup coefficients according to the distances in the latent space for the lincar mixup. The last row illustrates
and summarize the results. The second row in the
7

---

a)
b)
95
D'Ours Lineer a Binary Cut Spec Amp a Geo
40
OOurs Lilinerr uBiiaryy Cut Spec nAmp
O i Linwer e BBinury Cut Sppee ElAmp
Accuracy.ln in Highor is better
35
(Mon Absolute
AUC
78
Lower is better.
Higher ts better.
25
2000
15
USC 10
UCHAR
HHAR
IEEE SPC12
IEEE SPCz2
DaLia
2018
Chapman
Figure 2: The comparison of mixup methods where the error bars represent the deviation across random seeds (explicit numbers are given in Appendix E). a) shows the performance in activity recognition. b) is for heart rate prediction, and finally c) shows the CVD classification. For the last two tasks, we excluded Geomix its performance is extremely poor and distorts the y-axis scale.
the performance change resulting from randomly sampling mixup coefficients without considering any relationship between the selected pair while applying tailored mixup for phase and magnitude.
Table 4: Ablation on proposed mixup with coefficient selection in Activity Recognition datasets
Method
UCIHAR
HHAR ACC1
USC
ACC1
MFI1
MI-1
ACC1
MP
Ours w/o Prop. Mixup
91.60 1 0.65
90.46 1 0.53
88.05 1 1.05
87.95 1 1.10
60.13 1 0.75
59.13 0.69
83.09 (-8.51)
81.65 (-8.81)
85.89 (-2.16)
86.01
(-1.44)
45.10 (-15.03)
43.64 (-15.49)
w/o Aug. Degree
80.86 (-10.74)
80.18 (-10.26)
87.53 (-0.95)
87.75 (-0.20)
57.00 (-3.13)
54.75 (-4.38)
The results obtained from the ablation study support the previous claims and outcomes. For example, when the linear mixup is applied instead of the proposed mixup technique for heart rate prediction (Table 55] w/o Prop. Mixup), the performance decrease is significant compared to the case when coefficients are sampled without considering the distance in the latent space (Table [S] w/o Aug. Degree). This observation indicates that as the periodicity in data increases, linear mixup can lead to significant destructive interferences, whereas our method effectively prevents such issues.
Table 5: Ablation on proposed mixup with coefficient selection in Heart Rate Prediction datasets.
IEEE SPC12
IEEE SPC 22
DaLia MAFL
Method
MAE]
RMSE,
MAF!
RMSFL
RMSFL
Ours
16.26 + 0.72
22.48 0.95
12.25 + 0.47
18.20 0.61
10.57 + 0.55
20.37 + 0.73
w/o Prop. Mixup
20.45 (+4.19)
28.51 (+6.03)
15.29 (+3.04)
24.08 (+5.88)
24.11 (+13,54)
35.45 18)
w/o Aug. Degree
19.30 (+3.04)
24.84 (+2.36)
16.01 (+3.76)
21.21 (+3.19)
1+0.0))
20.13 (-0.24)
While our mixup technique consistently en- hances performance across datasets, we ob- serve a decline when the mixing coefficients are sampled based on the distance in the latent space for two datasets. Also, the performance increase gained by sampling coefficients based on distance is relatively low compared to the proposed mixup. Several factors can explain this observation. First, the VAE might not be well trained due to the limited size of data in each class, i.e., the assumption in Proposition 3.2 does not hold. This can lead to inconsistencies in the semantic similarity of the latent space such that two close samples in the latent space might have different labels. Second, if the number of classes increases for a downstream task, the probability of sampling intra-class samples in a batch will decrease, leading to lack of performance improvement. Therefore, in future investigations, it might be beneficial to use a different distance metric for quasi-periodic time-series data such that can scale with the number of classes while considering the lack of big datasets.
Table 6: Ablation on proposed mixup with coeffi- cient selection in CVD.
CPSC 2018
Chapman
Method
AUC1
AUC
Ours w/o Prop. Mixup w/o Aug. Degree
85.30 + 0.45
95.90 + 0.82
81.20 (-4.10)
86.30 (-9.60)
80.67 (-4.63)
95.98 (+0.08)
More ablation studies about the sensitivity of mixing coefficients and performance in different self-supervised learning frameworks, like BYOL. [59 can be found in Appendix F. and E.2 And, investigations regarding whether we still need known data augmentations are given Appendix F. Examples that visually demonstrate the negative effects of linear mixup and our proposed mixup
8

---

technique to prevent this problem can be found in Appendix []] Comparative results regarding the performance of the tailored mixup in the supervised learning paradigm are given in Appendix G
6 Related Work
The goal of contrastive learning is to contrast positive with negative pairs [60] In other words, the embedding space is governed by two forces, the attraction of positive pairs and the repellence of negatives, actualized through the contrastive loss [61]. Since label information is unavailable during the training, positive pairs are generated using augmentation techniques on a single sample, while negative pairs are randomly sampled from the entire dataset. Therefore, the choice or generation of positive and negative samples plays a pivotal role in the success of contrastive learning 62 and both approaches, generation/selection of positive/negative pairs, have been investigated thoroughly in the literature [66/7]], we limit our discussion about prior works related to data augmentation techniques that create optimal or hard samples without labels.
Adversarial based approaches A growing body of literature has investigated generating samples
by using adversarial training for both positives and negatives 4X] 72 73 A seminal work about the importance of augmentations in CL. InfoMin, presented an adversarial training strategy where players try to minimize and maximize the mutual information using the NCE loss [48] CLAE, one of the first works that leveraged the adversarial approach, shows that adversarial training generates challenging positive and hard negative pairs (72). Another recent study proposed adversarial approach that generates hard samples while retaining the original sample identity by leveraging the dentity-disentangle feature of VAEs [53]. However, adversarial augmentations may change the original sample identity due to excessive perturbations and it is infeasible to tune the attack strength for every sample to preserve the identity. In other words, these approaches do not consider the sample-specifte features and use a constant perturbation coefficient for all samples whereas our proposed method considers the similarity between pairs and tunes the mixing coefficients accordingly.
Mixup based approaches Mixup-based methods have been recently explored in contrastive learn- ing [1] 47 [75) According to recent theoretical work (22), mixup has implicit data-adaptive regularization effects that promote generalization better than adding Gaussian noise, which is a commonly used augmentation strategy in both time-series and vision data 776. Although, mixup- based approaches have shown success in different problems [79 80), such as domain adaptation, creating samples using mixup in the input space is infeasible in domains where data has non fixefixed topology, such as sequences, trees, and graphs (22). Therefore, recent works suggest mixing hidden representations of samples, similar to Manifold Mixup [1]. However, this method claims that mixing fixed-length hidden representation via an intermediate layer "z = az + (1 a z can be interpreted as adding noise to a given sample in the direction of another. However, it is an overly optimistic claim because early during training, where in most cases there is usually no linear separability among the representations, this synthesis may result in hidden representations that are completely different and far away from the samples [71 [8).. Therefore, in this work, we take a different approach and modify the mixup method considering its limitations for quasi-periodic non-stationary time-series data. Also, unlike most existing methods that aim to generate hard
that are close to class boundaries-usim using adversarial approaches [53 48 72 or feature extrapolation (50) 77] our method seeks to connect semantically closer samples together using interpolation in a tailored way.
7 Conclusion
In this paper, we first demonstrate the destructive effect of linear mixup for quasi-periodic time- series data, then introduce a novel tailored mixup method to generate positive samples for the contrastive learning formulation while preventing this destructive effect and interpolating the samples appropriately. Theoretically, we show that our proposed method guarantees the interpolation of pairs without causing any loss of information while generating a diverse set of samples. Empirically, our method outperforms the prior approaches in three real-world tasks. By conducting experiments on contrastive and supervised learning settings, we show that our approach agnostic to the choice of learning paradigm. Thus, it holds the potential for utilization in generating augmented data for different learning paradigms as well. We believe that the method proposed in this paper has the potential to significantly improve learning solutions for a diverse range of time series tasks.

---

References
[1] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[2] Carl Doersch, Abhinav Kumar Gupta, and Alexci A. Efros. Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference on Computer Vision (ICCV), pages 1422-1430, 2015.
[3] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2536-2544.
IEEE Computer Society, 2016.
[4] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06)), 2:1735-1742, 2006.
[5] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances Neural Infor- mation Processing Systems, volume 27. Curran Associates, Inc., 2014.
[6] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[7] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. Wav2vec 2.0: A framework for self-supervised learning speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 20, Red Hook, NY.
USA, 2020. Curran Associates Inc.
[8] shuang ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Contrastive learning of global and local video representations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7025-7040. Curran Associates, Inc., 2021.
[9] Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, and Xuedong Huang. Unispeech: Unified speech representation learning with labeled and unlabeled data. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10937-10947. PMLR, 18-24 Jul 2021.
[10] Yu-An Chung. Yu Zhang, Wei Han, Chung -Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-training. 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244-2544 250, 2021.
[11] Hao Fu, Shaojun Zhou, Qihong Yang, Junjie Tang, Guiquan Liu, Kaikui Liu, and Xiaolong Li. Lre-bert: reprssttcnresnnttation contrastive knowledge distillation for natural language understanding. Proceedings of the AAAI Conference Artificial Intelligence, 35(14): 35(14):22800 18330 12838, May 2021.
[12] Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li. Ji Zhang, Haiqing Chen, and Yin Zhang Kace: Generating knowledge aware contrastive explanations for natural language inference. In Anmual Meeting of the Association for Computational Linguistics, 2021.
[13] X. Y. Shen, Ying Sun, Yao zhong Zhang, and Mani Najmabadi. Semi-supervised intent discovery with contrastive learning. Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, 2021.
10

---

[14] Yang Liu and Maosong Sun. Contrastive unsupervised word alignment with non-local features. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Feb. 2015.
[15] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning. ICML'20. JMLR.org. 2020.
[16] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, C. Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In International Joint Conference on Artificial Intelligence, 2021.
[17] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
[18] Mike West, Raquel Prado, and Andrew D. Krystal. Evaluation and comparison of eeg traces: Latent structure in nonstationary time series. Journal of the American Statistical Association, 9A(44))375-387, 1999.
[19] Hangwei Qian, Tian Tian, and Chunyan Miao. What makes good contrastive learning on small-scale wearable-based tasks? In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '22, page 3761-3771, New York, NY, USA, 2022. Association for Computing Machinery.
[20] Qingsong Wen, Liang Sun, Fan Yang. Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, International Joint Conferences on Artificial Intelligence Organization, 8 2021. Survey Track.
pages
[21] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In Proceedings of Neural Information Processing Systems, NeurlPS, 2022.
[22] Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards domain- agnostic contrastive learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10541. PMLR, 18-24 Jul 2021.
[23] Soyoung Yoon, Gyuwan Kim, and Kyumin Park. Ssmix: Saliency-basee span mixup for text classification. ArXiv, abs/2106.08066 2021.
[24] Martin UlIrich, Arne Küderle, Julius Hannink, Silvia Del Din, Heiko Gaßner, Franz Marxreiter, Jochen Klucken, Bjoern M. Eskofier, and Felix Kluge. Detection of gait from continuous inertial sensor data using harmonic frequencies. IEEE Journal of Biomedical and Health Informatics, 24(7):1869-185 1878, 2020.
[25] Hong Luo, Deye Yang, Andrew Barszezyk, Naresh Vempala, Jing Wei, Si Jia Wu, Paul Pu Zheng, Genyue Fu, Kang Lee, and Zhong- Ping Feng. Smartphone-basee blood pressure measurement using transdermal optical imaging technology. Circulation: Cardiovascular Imaging, 12080220 2019.
[26] Bryan P. Yan, William H. S. Lai, Christy K. Y. Chan, Stephen Chun-Hin Chan, Lok-Hei Chan, Ka-Ming Lam, Ho-Wang Lau, Chak-Ming Ng. Lok-Yin Tai, Kin-Wai Yip, Olivia T. L. To, Ben Freedman, Yukkee C. Poh, and Ming Zher Poh. Contact-free screening of atrial fibrillation by a smartphone using facial pulsatile photoplethyymooraphii signals. Journal of the American Heart Association. (8))0008588, 2018
[27] Saman Noorzadeh, Mohammad Niknazar, Bertrand Rivet, Julie Fontecave-Jallon, Pierre-Yver Yves Guméry, and Christian Jutten. Modeling quasi-periodic signals by a non-parametric model: Application on fetal eeg extraction. In 20/4 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 1889-1892, 2014.

---

[28] A Koulali and P J Clarke. Modelling quasi-periodic signals in geodetic time-series using Gaussian processes. Geophysical Journal International. 226(3):175--1714, 04 2021.
[29] M. Lemay, L. Dang, and J.M. Vesin. Quasi-periodic atrial activity components in the eog used to discriminate between paroxysmal and chronic atrial fibrillation. In 2008 Computers in Cardiology, pages 821-88 2224,22,222 2008.
[30] A Garfinkel, P S Chen, D Walter, H S Karagueuzian, B Kogan, S Evans, M Karpoukhin, C Hwang, T Uchida, M Gotoh, O Nwasokwa, Sager, and N Weiss. Quasiperiodicity and chaos in cardiac fibrillation. The Journal of Clinical Investigation, 99(2):305 --1400 1 1997.
[31] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, 2018.
[32] Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual con- cepts with a constrained variational framework. In International Conference on Learning Representations, 2016.
[33] Yingi Li, Hong Chang, Bingpeng MA, Shiguang Shan, and Xilin Chen. Optimal positive gener- ation via latent transformation for contrastive learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 18327-18342. Curran Associates, Inc., 2022.
[34] D. Anguita, Alessandro Ghio, L. Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. Human activity recognition on smartphones using a multiclass hardware-friende support vector machine. In International Workshop on Ambient Assisted Living and Home Care, 2012.
[35] Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjær- gaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. SenSys '1, page 127-140, New York, NY, USA, 2015. Association for Computing Machinery.
[36] Mi Zhang and Alexander A. Sawchuk. Use-had: A daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, UbiComp '2, page 1036-1043, New York, NY, USA, 2012. Association for Computing Machinery.
[37] Hangwei Qian, Sinno Jialin Pan, and Chunyan Miao. Latent independent excitation for general- izable sensor-based cross- pross- activity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13):1 19211-199991199999 May 2021.
[38] Zhilin Zhang, Zhouyue Pi, and Benyuan Liu. Troika: A general framework for heart rate monitoring using wrist-type photoplethysmographh signals during intensive physical exercise. JEEE Transactions on Biomedical Engineering, 62( 2015.
[39] Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Lacrhoven. Deep ppg: Large-scale heart rate estimation with convolutional neural networks. Sensors, 19(14), 2019.
[40] Eddie Y. K. Ng, Feifei Liu, Chengyu Liu, Lina Zhao, X. Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, and Jianqing Li. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics, 2018
[41] Jianwei Zheng, Jianming Zhang, Sidy Danioko, Hai Yao, Hangyuan Guo, and Cyril Rakovski. A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients. Scientific Data 7(1):48, February 2020.
[42] Erick A Perez Alday, Annie Gu, Amit Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifei Liu, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, Qiao Li, Ashish Sharma, Gari D Clifford, and Matthew Reyna. Classification of 12-lead eegs: the physionet/computing in cardiology challenge 2020. Physiological Measurement, 41(12): 124003, dec 2020.
12

---

[43] Christopher Beckham, Sina Honari, Vikas Verma, Alex M Lamb, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Chris Pal. On adversarial mixup resynthesis. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[44] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022-6031, 2019.
[45] Q. Xu, R. Zhang, Y. Zhang. Y. Wang, and Q- Tian. A fourier-based framework for domain generalization. In 202/ IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 14378-14387 Los Alamitos, CA, USA, jun 2021. IEEE Computer Society.
[46] Gwantae Kim, David K. Han, and Hanseok Ko. Specmix A mixed sample data augmentation method for training frequeee domain features. In Interspeech, 2021.
[47] Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. 2022.
[48] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[49] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In 2021 IEEF/CVF International Conference on Computer Vision (ICCV), pages 9568-9577, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society.
[50] Rui Zhu, Bingehen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10286- 10295, 2021.
[51] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[52] Xiyuan Zhang, Ranak Roy Chowdhury, Jingho Shang, Rajesh Gupta, and Dezhi Hong. Towards diverse and coherent augmentation for time-series forecasting. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[53] Kaiwen Yang, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. dientity-dieentangled adversarial augmentation for self-supervised learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 25364-25381. PMLR, 17-23 Jul 2022.
[54] Dwaipayan Biswas, Luke Everson, Muqing Liu, Madhuri Panwar, Bram-Ernst Verhoef, Shrishail Patki, Chris H. Kim, Amit Acharyya, Chris Van Hoof, Mario Konijnenburg, and Nick Van Helleputte. Cornet: Deep learning framework for ppg-based heart rate estimation and biometric identification in ambulant environment. IEEE Transactions on Biomedical Circuits and Systems, 13(2):282-25 2212222912222 2019.
[55] Akara Supratak, Hao Dong. Chao Wu, and Yike Guo. Deepsleepnet: a model for automatic sleep stage scoring based on raw single-channel eeg. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 25(1 ():11))9998--)) 20088,20088,000 Nov 2017.
[56] Garrett Wilson, Janardhan Rao Doppa, and Diane J. Cook. Multi-source deep domain adaptation with weak supervision for time-series sensor data. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &; Data Mining, KDD '20, page 1778- New York. NY, USA, 2020. Association for Computing Machinery.
13

---

[57] Dani Kiyasseh, Tingting Zhu, and David A. Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning. 2020.
[58] Crystal T. Wei, Ming-En Hsieh, Chien-Liang Liu, and Vincent S. Tseng. Contrastive heartbeats: Contrastive learning for self-supervised ecg representation and phenotyping. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1130, 2022.
[59] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020.
Curran Associates Inc.
[60] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[61] Tri Huynh, Simon Kornblith, Matthew R. Walter, Michael Maire, and Maryam Khademi. Boosting contrastive sulf--uper learning with false negative cancellation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2785 2795, January 2022.
[62] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning, 2019.
[63] Anastasiaa Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[64] Hong Xuan, Abby Stylianou, Xiaotong Liu, and Robert Pless. Hard negative examples are hard, but useful. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV, page 126-142, Berlin, Heidelberg, 2020. Apringer-Verlaee
[65] songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang. and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 27356-27368. Curran Associates, Inc., 2021.
[66] Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance classification for unsupervised visual feature learning. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 Red Hook, NY, USA, 2020. Curran Associates Inc.
[67] Ching- Cho Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8765-
8775. Curran Associates, Inc., 2020.
[68] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondfej Chum. Mining on manifolds: Metric learning without labels. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7642-7651, 2018.
[69] Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual information in contrastive learning for visual representations, 2020.
[70] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815-823, 2015.
14

---

[71] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21798-2 21 2080, Curran Associates, Inc., 2020.
[72] Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17081-17093. Curran Associates, Inc., 2020.
[73] Xiao Wang, Yuhang Huang, Dan Zeng, and Guo-Jun Qi. Caco: Both positive and negative sam- contrastive learning. IEEE Transactions
ples are directly learnable via cooperative-ad on Pattern Analysis and Machine Intelligence, pages 1-12, 2023.
[74] Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint
2020.
[75] Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICIR,, 2021.
[76] Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, and Daniel McDuff Simper: Simple self-supervised learning of periodic targets. In The Eleventh International Conference on Learning Representations, 2023.
[77] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2352-2359 2021.
[78] Pengxiang Shi, Wenwen Ye, and Zheng Qin. Self-supervised pre-training for time series classification. In 2021 International Joint Conference on Neural Networks (IJCNN), 2021.
[79] Ronghang Zhu, Ronghang Zhu, Xiang Yu, and Sheng Li. Progressive mix-up for few-shot supervised multi-source domain transfer. In ICLR, 2023.
[80] JangHyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. In International Conference on Learning Representa- tions, 2021.
[81] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6438 6447, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
[82]
Cheung and Dit- Yan Yeung. (MODALS): Modality-agnostee automated data aug- mentation in the latent space. In International Conference Learning Representations, 2021.
[83] Leandro Giacomini Rocha, Dwaipayan Biswas, Bram-Ernst Verhoef, Sergio Bampi, Chris Van Hoof, Mario Konijnenburg, Marian Verhelst, and Nick Van Helleputte. Binary cornet: Accelerator for hr estimation from wrist-ppg. IEEE Transactions on Biomedical Circuits and Systems, 14(4):715-726, 2020.
[84] Sayeed Shafayet Chowdhury, Rakib Hyder, Md. Samzid Bin Hafiz, and Mohammad Ariful Haque Real-time robust heart rate estimation from wrist-type ppg signals using multiple reference adaptive noise cancellation. IEEE Journal of Biomedical and Health Informatics, 22(2):450-459, 2018
[85] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus -uc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[86] Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa- Besaa-Binnchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
15

---

Appendix
A Proof
In this section, we present complete proofs of our theoretical study, starting with notations.
A.1 Notations
Fourier transform of real-valued sample with a finite duration is obtained as in Equation 7
X4 F(x)
Σ
X j2mkn
The amplitude and phase for each frequency are calculated from the Fourier transform as follows.
A(x) v Re((X))2 + Imx), P(x) = (Imman)(Im(XX))) Re(Xx)),
(8)
where arctan is 2-argument arctangent which is the angle measure in radians. The phasor, in Figure [ of a sample is represented as in Equation 9
X = F(x) A(x)e
(9)
A.2 Proof for Proposition 2.3
Proposition A.I (Destructive Mixup). If Assumptions 2.7 and 2.2 hold, there exist X Betafa, a) or U 1.0) with high values of 3 such that when linear mixup techniques are utilized, the lower bound of the mutual information for the augmented sample distribution decreases to zero.
0 I<y:x=) I(y;x") where
xt Ax (1 A)X and J, [s- /- S.-(F)
(10)
Proof.
x Ax + (1 A)
(11)
From the linearity of Fourier transformation and ignoring k in X₃. for the sake of easiness.
x = XX + A)- x = X X X)
(12)
(13)
Let X jjmo Xwg, where Φy and wE are random phase and frequency modulators for each frequency, sampled from distributions o; Tb wh Ω.
x jwoB Xwrk + A(X juudo Xwn)
(14)
X = x |A+e jWOn de jwm wx]
(15)
x = X A (1 (1 A)e j.wb wK]
(16)
X+ X X + (1 - Acoss (WDD)) sin (wo))))
(17)
16

---

From the periodiiii, assume that the frequency ranges of interest (f i.e. k*) are overlapped for both samples while the sampled random modulators have the following relationship.
and [w> (mod 2x),
(18)
where an odd multiple of m. Equation 17 can be simplified as follows.
Xx. Xx +++ Aoss (wi) )]
(19)
x =0 0
(20)
8
8
X
Xn f2akk
en -22kkkkk n
(21)
2
S2+(f* lim
Σ
InE j2=ff
(22)
-00 2N
n=-N
From Assumption 2.1
I(y:xt) x J. S21 (f) L (f) 5r
(23)
I(y;xt) I[y;x"))
(24)
We use Euler's s formula to expand Equation 16 to 17 While we use the frequency bins (k) and frequency values in Hz (f) interchangeably for Equations 21 and 22
Although the above proof is to show the resulting instances may not contain any televatlevant information, can also be demonstrated that the augmentation process can potentially discard the partial task-specific information (not whole) if Φb, and wP are close to indicated relationships.
17

---

A.3 Proof for Theorem 3.1
Theorem A.2 (Guarantees for Mixing), Under assumptions 2.1 and222 given any X E (0, 1). the mutual information for the augmented instance lower bounded by the sampled À and anchor x.
XI(y; x) S I(y;x*) < x(y:x") where =F= (((x)))((xt))
(25)
Proof.
x+ F (A(xx) (A(xt)<(((tt where
A(xt) = \x + (1- (1=)x) and
(26)
P(x+) ( PP(x) |Q) * (1- Ap),
if AO > 0 P(x) + |o) (1 Ap), otherwise
X = A(x*)e
(27)
X+ = ((xx))))
(28)
X+ = A(x*)
where |x =
x j2mkn
(29)
72
x+ = XA(x)
+ (1 A()
(30)
x+ = X
Xx
j2mkn j2nkon
+ A)
Σ
e j2zkn
(31)
n=-oc
e j2mkn
+ +(1- A)
Jtekkn
2
e,e
f2akn
(32)
J Szi Szi /
Sz(f)
(33)
Using the _ Sz1 (f) = S00 Sf) (i.e., both samples are normalized to have the same power) and
Assumption
I(y:xt) x J Sat(1)/ [ Sot(f) 1.
(34)
XI(y;x) I(y;xt) I(y:x*)
(35)
Proof is completed with Equation 35 by combining equations
33 and 34
Although this proof ignores the effect of phase mixing on the mutual information with the assump- tion 2.1 it is known that phase components carry semantically important features [45]. Therefore, it necessary to note that the objective of this proof is to demonstrate that by applying mixup separately to the phase and amplitude components, we can avoid destructive interference.
18

---

B Datasets
In this section, we give details about the datasets that are used during our experiments.
B.1 Human Activity Recognition
Human activity recognition using smartphones dataset (UCIHAR) 34 collected by 30 subjects within age range of 16 to 48 performing six daily living activities with waist-mounted smartphone. Six activities include walking, sitting, lying, standing. walking upstairs, and walking downstairs. Data is captured by 3-axial linear acceleration and 3-axial angular velocity
UCIHAR
constant
rate of 50 Hz. We used the pre-processing technique the same as in 37 19 such that the input contains nine channels with 128 features (it is sampled in sliding window of 2.56 seconds and 50% overlap, resulting in 128 features for each window). Windows are normalized to zero mean and unit standard deviation before feeding to models. Also, we follow the same experimental setup with prior works as follows. The experiments are conducted with a leave-one-domain-on strategy, where one of the domains is chosen be the unseen target [T9]. The contrastive pre-training is conducted with all subjects without any label information except the target one. Training of the linear layer, which added to the frozen trained encoder, only performed with the first five subjects of UCIHAR after excluding the target subject. In other words, if the target subject is O, the subjects from 1 to 29 are used train the encoder without any label information. Then, subjects from I are used to train the linear layer. And, evaluation is performed for subject O. This is performed for the first five subjects with three random seeds and the mean value reported.
HHAR Heterogeneity Dataset for Human Activity Recognition (HHAR) is collected by nine subjects within an age range of 25 to 30 performing six daily living activities with eight differ- ent Allthougg HHAR includes data from smartwatches as well, we use data from smartphones-that were kept in a tight pouch and carried by the users around their waists (35). Subjects then perform 6 activities: 'bike', 'sit', 'stairs down', 'stairs up', 'stand', and 'walk'. Due to variant sampling frequencies of smart devices used in HHAR dataset, we downsample the readings to 50 Hz and apply 100 (two seconds) and 50 as sliding window length with step size, the windows are normalized to zero mean with unit standard deviation. We used the first four subjects (i.e., a, b, c, d) as source domains.
USC USC human activity dataset (USC-HAD) is composed of 14 subjects male, 7 female, aged 21 to 49 with mean of executing 12 activities with a sensor on the front right hip. The data dimension six (3-axis accelerometer, 3-axis gyroscope) and the sample rate is 100 Hz. 12 activities include walking forward, walking left, walking right, walking upstairs, walking downstairs, running forward, jumping up, sitting, standing, sleeping, elevator up, and elevator down. We used the pre-processing technique with a smaller window size such that the input contains six channels with 100 features (it is sampled a sliding window of I second and 50% overlap, resulting in 100 features for each window). The same normalization is also applied to windows before feeding to models. We used the same setup with UCIHAR while source subjects are chosen as the last four this time.
B.2 Heart Rate Prediction
IEEE SPC This competition provided a training dataset of 12 subjects (SPC12) and test dataset of 10 subjects [39] The IEEE SPC dataset overall has 22 recordings of 22 subjects, ages ranging from 18 to 58 performing three different activities [83] Each recording has sampled data from three accelerometer signals and two PPG signals along with the sampled ECG data and the sampling frequency is 125 Hz. All these recordings were recorded from the wearable device placed on the wrist of each individual. All recordings were captured with a 2-channel pulse oximeter with green LEDs, tri-axial accelerometer, and chest ECG for the ground-truth HR estimation. During our experiments, we used PPG channels. We choose the first five subjects of SPC12 as source domains similar to activity recognition setup while the last six subjects of SPC22 are used for source domains to prevent overlapping subjects with SPC12.
Dalía PPG dataset for motion compensation and heart rate estimation in Daily Life Activities (Dal.ia) was recorded from 15 subjects (8 females, 7 males, mean age of 30.6), where each recording was approximately two hours long. PPG signals were recorded while subjects went through different
19

---

daily life activities, for instance sitting, walking, driving, cycling, working, and so on. PPG signals were recorded sampling rate 64 Hz. The first five subjects are used as source domains.
All PPG datasets are standardized as follows. Initially, a fourth-order Butterworth bandpass filter with frequency range of 0.5 Hz is applied PPG signals. Subsequently, sliding window of 8 seconds with 2-second shifts is employed for segmentation, followed by z-score normalization of each segment. Lastly, the signal is resampled frequency of 25 Hz for each segment.
B.3
Cardiovascular disease (CVD) classification
CPSC China Physiological Signal Challenge 2018 (CPSC2018), held during the 7th International Conference on Biomedical Engineering and Biotechnology in Nanjing, China. This dataset consists of 6,877 (male: 3,699; female: 3,178) and 12 lead ECG recordings lasting from seconds to 60 seconds with Hz.. We use the original labelling 440 with one normal and eight abnormal types as follows: atrial fibrillation, first-degree atrioventricular block, left bundle branch block, right bundle branch block, premature atrial contraction, premature ventricular contraction. ST-segment depression, Se--eetmett elevated. We resampled recordings 100 Hz. and excluded recordings of less than 10 seconds.
Chapman Chapman University. Shaoxing People's ' Hospital (Chapman) ECG dataset which pro- vides 12-lead ECG with 10 seconds of sampling rate of 500 Hz. The recordings are downsampled to 100 Hz, resulting in each ECG frame consisting of 1000 samples. The labeling setup follows the same approach as in 41 with four classes: atrial fibrillation, GSVT, sudden bradycardia, and sinus rhythm. The ECG frames are normalized to have a mean of 0 and scaled to have standard deviation of We split the dataset to 80-20% for training and testing as suggested [4]].
We choose leads I, II, III, and V2 during our experiments for both ECG datasets. We followed a similar setup with prior works [57] and considered each dataset as a single domain different from previous tasks. The fine-tuning of the linear layer, which is added to the frozen pre-trained encoder, performed with 80% of the same domain.
B.4 Metrics
We used the common evaluation metrics in the literature for each task. Specifically, we used accuracy (Acc) and F1 score for activity recognition [19] mean absolute error (MAE), and root mean square
error (RMSE) for heart rate prediction 33 84 and the area under the ROC curve (AUC) for
cardiovascular disease classification (57
In this section, we explain how to calculate each metric for different time-series tasks. For activity recognition, the accuracy metric is computed by dividing the sum of true positives and true negatives by the total number of samples where a window has single label. The MFI score is calculated as harmonic mean of the precision and recall where metrics are obtained globally by counting the total true positives, false negatives, and false positives similar to [19]
For heart rate prediction, the Mean Absolute Error (MAE) and Root-Mean Mean--Suuree Error (RMSE) are calculated using the following equation:
1 MAE
HRRmdde((k) HRref(k)
(36)
K
K k==
(HRmodel(k) HRref(k))
RMSE
(37)
K
where K. represents the total number of segments. The variables HRmodel(k) and HRref (k) denote the output of the model and reference heart rate values in beats-per-minute for the 4th segment, respectively. This performance metric is commonly used in PPG-based heart rate estimation studies [39]. The estimated heart rate values (HRmodellk)) are obtained using our model, while the reference heart rate values (K)r are directly taken from datasets.
20

---

The AUC score for CVD classification is calculated using the one-vs-one scheme where the average AUC is computed for all possible pairwise combinations classes for both datasets.
C
Baselines
C.1 Prior Mixup Techniques
In this section, we give a detailed explanation of each mixup technique we compare our proposed method.
Lin We apply linear mixup in Equation 38 to generate positive samples, if x has more than one channel, mixup is applied independently for each of them.
x+ Ax
X)
(38)
BinaryMix We implement the binary mixup 43 by swapping the elements of x with the elements of another randomly chosen sample x as shown below.
x m Cx (1
m x,
(39)
where m a binary mask sampled from Bernouill((p) with high values, and @ stands for Hadamard product.
GeometricMix In Geometric Mixup, we create positive sample corresponding a sample by taking its weighted-geometrr mean with another randomly chosen sample x same as [22 as shown below.
x+ =x + (1-
(40)
CutMix Cutmix is implemented similarly to Binarymix. However, instead of changing each sample point with a probability, we cut a continuous portion using a rectangle mask M from a signal x and replace it with the same portion of another randomly chosen one ă. The starting point of the mask is uniformly sampled while its length is sampled from lower values such that the augmented sample is more similar to the anchor. If the signal has multiple channels, this process is applied to all channels in the same section.
x+ Mox+ (1-M) OX, and M rect 6
(41)
where b and are the starting point and length of the rectangle wave, respectively.
AmplitudeMix AmplitudeMix is introduced for domain adaptation problems by mixing the ampli- tude information of images without mixing the phase of two samples [45]. In our setup, perform amplitude mixing on the time series data across all channels while keeping the phase component unchanged. In other words, perform the following operations.
x+ x== = (A(x <P(xt)) (Axx))((ttxt A(xt) Ax) + + (1- -A)A(x)) and P(xt) P(x)
where
(42)
SpecMix We implement the SpecMix by applying CutMix to the spectrogram of time-series where the spectrogram is calculated using the short-time Fourier transform as follows.
20
x
x, mlle 12xkn
(43)
21

---

where g/n mR is an analysis window of length M with hop length of R over the signal and calculating the discrete Fourier transform (DFT) of each segment of windowed data. The length of the Fourier transform is set to the sample size of the input time series while the hop and window parameters are set to the quarter of the length.
C.2 Prior Methods for Sample Generation
In this section, we give a detailed explanation of prior methods for data generation methods.
Traditional Augmentations We apply two separate data augmentation to the anchor for creating two instances, and the encoders are trained to maximize agreement using the contrastive loss in [TS] We search mainly for augmentations that are known in state-of-the-ee works [19]. The detailed augmentations are given in Table 22
InfoMin We train a model go (-), which is restricted to sample-wise 1 x convolutions and ReLU activations same in [48), to decrease the mutual information between two instances. In the original paper, the input sample is split into two instances (X1 and X2:3) and then adversarial training is performed. As do not have RGB channels for time-series data, added Gaussian noise to the signal for creating other instances and then perform adversarial training.
We follow a similar setup to SimCLR by applying two separate data augmentations, then we use nearest neighbors in the learned representation space as the positive in contrastive losses [49
NNCLR
PosET
We perform the dimension level mixing with extrapolation of positive features as follows:
zt =o + (1 2) ④ &
(44)
where is Hadamard product, and x Beta(a, We add I to sampled À for extrapolation as in (50).
GenRep In the original implementation of GenRep, the authors use implicit generative models (IGMs) such as BigBiGAN (85] that are trained with millions of images to create the anchor and positive instance by sampling nearby latent vectors. However, as the number of samples for training is limited in time series and there is a well-trained generator for different time-serics tasks, we use our trained VAE for sampling nearby latent vectors positives. Mainly, we sample an anchor from real data, feed it to the encoder, add a Gaussian noise sampled from a truncated normal distribution, and use the output of the decoder for the positive sample with the anchor.
STAug The Spectral and Time Augmentation (STAug) method specifically proposed for the time-series forecasting task, where the authors apply the empirical mode decomposition to decompose time series into multiple subcomponents, then reassemble these subcomponents with random weights to generate new synthetic series. Finally, in the time domain, the method uses the linear mixup to generate samples from the reassembled components. Although, the mixing coefficient sampled from beta distribution in the original implementation, we observe significant performance decreases when the same distribution with parameters used in our experiments, possibly due to the generated samples being far away from the anchor. We, therefore, investigate the case when the mixing coefficient is sampled from uniform distribution with high values, e.g., same as our method. Since there is no
Augmentation Bank The augmentation bank that perturbs frequency components of a time-series signal is proposed in [21] where the authors use it for unsupervised domain adaptation with different framework than SimCLR, namely time-frequenny consistency (TF-C). As it is novel data augmentation technique, we have implemented the frequency augmentation bank as baseline while using the SimCLR framework for a fair comparison with other methods. The authors also employed a collection of time-based augmentations for the time-domain contrastive encoder. Nonetheless, since these augmentations have already been studied in previous CL setups, we chose to exclusively utilize the frequency augmentation bank. In the paper, the authors mentioned using a small budget with low-frequency perturbations results in a performance increase, thus we chose the budget with single frequency while choosing the 0.5 with the same settings in the paper.
22

---

DACL
We perform the mixup for hidden representations, i.e., before applying projection-heaee as follows.
v+ Av + (1-
(45)
where is the fixed-length hidden representations of samples while À is sampled from uniform distribution with high values.
We follow the original implementation of authors with their proposed VAE architecture while optimizing the adversarial strength for each time-series task. We apply the FGSM adversarial attack the same in the original implementation (53) by perturhing the encoded representation of sample while adding noises along the gradient sign's g direction of the loss.
IDAA
One setup difference between this section and the previous mixup methods is that when compare our work with PosET, GenRep, DACL, and IDAA, we apply the best traditional data augmentation techniques, which are used for SimCLR implementation, mechanisms. The reason for this approach that the original implementations of certain works indicate that the proposed methods achieve optimal results when used in conjunction with known augmentations, where our observations align with these findings.
the specific positive data generation
The detailed hyperparameters for each baseline with the corresponding time series tasks are given in the following section.
D Implementation Details
D.1 Parameters for mixing
In this section, we provide the parameters that are used during our experiments. To determine the optimal parameters of the baselines for each task, conduct grid search. This search is performed on a small validation set taken from the largest dataset of the respective tasks, which are USC, Dalia and Chapman. We believe that this approach ensures fairness and produces more realistic results, as dataset-specific optimizations can lead to overfitting of parameters, particularly in smaller and less diverse datasets.
Table 7: Parameters for baselines
Method
Activity Recogaition
Heort rate Prediction
CVD Classifiution
Linear Mixup
A U(0.9, 1)
À ~ U(0.9,
A U(0085,11
Rinary Mixup
U(0.8,
m U(0.9, 1)
U10.9,
Geometric Mixup
~~ (10.9.
A (70.9,
U(0.9.1)
8 ().
d ((O
n- U(0, 1)
CulMix
UO0.1, 0.4)
0.3)
(10.1,
AmplitudeMi
As ~ (7..9.
U(0.9.
A U(0.8, I)
6~ b-0,
6- (7(0,))
6~ Ur(O, I)
SpecMix
e~ U.4.
a~ ((0.1,0.3)
a ((0.1,
PosFT
(2)o
Bata(2,2
Beta(2,2)
GenRep
A (.0
A (0,00.55,1.00
>~ (O 0.2.
DACL
À ~ U)..9.
A ~ U((099.1)
~ U(0.85, 1)
IDAA
0.1.
0.15
As (7 (.0.9.
U...7 1), Ap (7009. I) e 0.8. Aa, Ar (11.0011099)
A U[0.7 (700. U.7. Ar ~ (1.0110
Ours
0.T. (00.0000
D.2
Baseline Encoder Architecture
For the baseline encoder model, we adopt the Deep as in [9 where the architecture has 4 convolutional layers with 5 x 1 size of 64 kernels while ReLU is followed each convolution. After the convolutions, the tensor passed through dropout layer with a dropout rate of 0.5 to prevent overfitting. Then, the output of dropout is fed into the 2-layer LSTM with 128 units. After training
23

---

the baseline encoder, we attach linear layer and freeze the previous layers for fine-tuning, This
architecture is widely used for the datasets we used during our experiments [37 83 19 adopt the same network across tasks.
we therefore
D.3
VAE Models
We the total correlation variational autoencoder (3-TCVAE) 86 calculate the distance between two encoded samples in the latent space. We train the model for 100 epochs with a learning rate of 1e while setting the batch size 2048, The latent dimensions and the 3 values are set to 10 and 5, respectively. Below, we present the tables providing detailed information about the architectures of the encoder and decoder for datasets. The output of convolutional layers fed the batch normalization before the activation layer is applied. For tasks Heart rate Prediction and CVD Classification, we use task-specific encoder and decoder as the number channels and input size for datasets in each task are the same. However, two different networks, one for UCIHAR and one for others, are designed for Activity Recognition due to different number of input channels.
Table Encoder Network for UCIHAR in Activity Recognition
Encoder
Layer Name Output size
# of kemelss
Keree size
Stride
Activation
Input
Nx1x128x9
Convolution
Nx32x60x7
32
9x3
ReLU
Convolution
Nx32x27x5
32
7x3
2x1
ReLU
Convolution
Nx64x8x3
64
5x3
3x1
ReLU
Convolution
Nx128x2x1
5x3
ReLU
Convolution
Nx512x1x1
512
2x1
1x1
ReLUU
Convolution
Nx20x1x1
10
1xl
1xl
Table Decoder Network for UCIHAR in Activity Recognition
Layer Name
Output size
of kernels
Kernel size
Stride
Activation
Inpur
Nx1x
Transposed Convolation
Nx512x2x9
512
2x9
1xl
RcLU
Transposed Convolution
Nx 9
128
6x1
ReL.U
Transposed
Nx64x1629
ReLU
Transposed Convolution
Nx32x32x9
2x1
ReLU
Transposed Convolation
Nx32x64x9
32
2x1
ReLU
Transposed Convolution
Nx1x12889
4x1
2x1
Table 10: Encoder Network for USC and HHAR in Activity Recognition
Encoder
Layer Name
Output size of kernels
Keree size
Stride
Activation
Input
Nx1x100x6
Convolution
Nx32x46x5
32
9x2
2x1
ReLU
Convolution
Nx32x20x4
32
9x2
ReLU
Convolution
Nx64x8x3
64
5x2
2x1
ReLU
Convolution
Nx 128xx22x
128
5x2
2x1
ReLU
Convolution
Nx512x1x1
512
2x2
ReLU
Convolution
Nx20x1x1
10
1x1
1xl
24

---

Table 11: Decoder Network for USC and HHAR in Activity Recognition
Decoder
Layer Name
Output size
kernels
Kernel size
Stride
Activation
Input
Nx1x1011
Transposed Convolution
Nx512x2x6
512
2x6
Ixl
ReLUU
Transposed Convolation
Nx128x6x6
128
6x1
2x1
ReLU
Transposed
Nx64x12x6
2x1
ReLU
Transposed Convolution
Nx32x25x6
2x1
ReLU
Transposed Convolation
Nx32x50w6
32
2x1
ReLU
Transposed Convolution Nx1x100x6
4x1
2x1
Table 12: Encoder Network for Heart rate Prediction
Encoder
Layer Name
Output size
# of kernels
Kernel size
Stride
Activation
Input
1x1x200x1
Convolution
Nx32x94x1
32
13x1
2x1
ReLU
Convolution
Nx32x43x1
32
9x1
2x1
ReLU
Convolution
Nx64x18x1
64
9x1
ReLU
Convolution
Nx128x6x1
128
7x1
2x1
ReLU
Convolution
Nx512x1x1
512
5x1
2x1
ReLU
Convolution
Nx20x1x1
20
2x1
Table 13: Decoder Network for Heart rate Prediction
Decuder
Layer Name
Output size
# of kernels
Kernel size
Stride
Activation
Input
Nx1x10s1
Transposed Convolution
Nx512x66
6x1
1xl
ReLU
Transposed Convolution
Nx128x12x1
ReLU
Transposed Convolution
5x1
2x1
ReLU
Transposed Comvolution
Nx32x50x
4x1
2x1
ReLL
Transposed Convolution
Nx32x100xx
4x1
2x1
ReL.I.
Transposed Convolution
Nx1x200k]
4x1
2x1
Table 14: Encoder Network for CVD Classification
Encoder
Layer Name
Output size
of kemels
Kernel size
Stride
Activation
Input
Nx1x1000x4
Convolution
Nx32x330x3
32
12x2
3x1
ReLU
Convolution
Nx32x107x2
10x2
3x1
ReLU
Convolution
Nx64x34x1
64
8x2
3x1
ReLU
Convolution
Nx128x9x1
128
3x1
ReLU
Convolution
Nx512xIx1
512
7x1
3x1
ReLU
Convolution
Nx20x1x1
20
1x1
1x1
Table 15: Decoder Network for CVD Classification
Decoder
Layer Name
Ouiput size
# of kernels
Kernel size
Stride
Activation
Input
Nx1x10<1
Transposed Convolution
Nx512x4k4
512
6x1
1x1
ReLU
Transposed Convolution
Nx128x12x4
128
4x1
2x1
ReLU
Transposed Convolution
Nx64x366x4
5x1
2x1
ReL.I.
Transposed Convolution
Nx32x109<4
ReLU
Transposed Convolution
Nx32x331x4
4x1
2x1
ReLU
Transposed Convolution
NxIx100lx4
4x1
2x1
25

---

a)
-B-UCIAAR
-4-18IAR
-Q-USC
Meeen Absolite Error 6)
-Q-1EEE SPC12 -Q- SPC22 -0- -O-Dnia
C)
Chapman
95
80
90
75
70
18
85
60
65
80
50
A Ap
02
-0.2
The experiment regarding the effect of phase mixup coefficients in eight datasets. a) shows the performance in activity recognition, b) is for heart rate prediction using PPG, and finally shows the cardiovascular disease classification
Figure 3:
a)
100
-B-UCHAR
Mean Absolute Error
& t
300
80
22
80
20
48
70
85
24
-01 AM
-0.1 A
AA
Figure 4: The experiment regarding the effect of amplitude mixup coefficients in eight datasets. a) shows the performance in activity recognition, b) for heart rate prediction using PPG, and finally shows the cardiovascular disease classification
E
Additonal Results
E.1
The effect and robustness of mixing coefficients
In this section, our experiments focus on observing the impact of diverse range mixing coefficients for both phase and amplitude components. We decrease the lower threshold of distributions for sampling the mixing coefficient by 0.1 and 0.2. For example, normally the phase mixup coefficient
for Activity Recognition is sampled from truncated normal Xp ~ (1, 1, 0.9) and uniform AP ~ U (0.9, 1). We decrease the low threshold value from 0.9 to 0.8 and 0.7 and report the results for both phase and amplitude. The results are reported in Figures and for eight datasets.
From Figures 3 and 4 it can be inferred that the phase component is more sensitive to the changes. In other words, a significant decrease in performance is observed when the mixing coefficients for the phase are sampled from lower values whereas this effect is not much as severe for the amplitude coefficient, indicating that the amplitude of frequencies is more robust to changes compared to phase.
E.2 The performance in other frameworks
In this section, investigate the effect of data augmentations in three different unsupervised learning frameworks which are SimCLR (15,, BYOL (59] and TS-TCC [16]. For BYOL, the hidden size of the projector is set 128, the exponential moving average parameter the λ and X coefficients of temporal and contextual contrasting losses are set to 1, the same as in the original implementation. In TCC, the authors proposed use weak (jitter and scale) and strong (permutation and jitter) augmentation together, which is shown as TS-TCC + Traditional Augs the tables. During our experiments, we followed the original implementation of TS-TCC with their proposed architecture and applied additional augmentations after the strong one without changing the original contrastive learning framework. We set the scaling ratio to 2 and 10 for permutation (splitting the signal into random number of segments with maximum of 10 and randomly shuffling them). These parameters for augmentation strengths are set to the same values as in the original implementation.
to 0.996. For TS-TCC.
26

---

Table 16: Performance comparison of our method in different CL frameworks for Activity Recognition
UCIHAR
HHAR ACC?
USC
Method
ACCT
MF11
MFI1
ACC1
MF1
SimCI. Traditional Augs.
87.05
86.13
85.48
84.31
53.47
1.10
0.95
SimCLR + Aug. Bank
65.27 + 1.12
71.16 = 1.24
67.95
: 1.45
75.13
1.32
43. 28
+ 4.37
47.31 = 4.68
SimCLR DACL
73.12
1.23
66.28
80.89
81.31
0.78
53.61
+ 2.60
51.76
2.21
SimCLR + Ours
91.60 0.65
90.46
: 0.53
88.05
+ 1.05
87.95
+ 1.10
60.13
+ 0.75
$9.13
= 0.69
BYOL Traditional Augs.
83.41 0.95
82.13
1.12
86.41
+ 0.97
86.31
+ 1.10
58.34
1.15
55,04
1.15
BYOL Aug. Bank
73.71 1: 0.74
69.80
= 1.10
84.60
+ 0.93
84.65
+ 1.03
52.00
1.21
49.14
= 1.18
BYOL DACL
73.86
1.12
70.46
= 1.24
82.76
= 1.04
84,89
+ 0.93
47.14
= 2.08
45.34
= 2,98
BYOL Ours
87.01 + 1.10
84,92
1.13
90.31
1.16
90,45
+ 1.31
56.87
0.91
55,01
= 0.95
TS- TS-TCC+ Traditional Augs.
90.95 +: 0.87
90.30
+ 0.64
35.57
+ 1.43
40.13
+ 1.67
39.76
1.61
43.12
1.10
TS-TCC+ Aug. Bank
76.78
1.32
1.78
1.48
TS-TCC DACL
73.86
1.12
70.46
1.24
33,89
# 1.87
37
+ 1.39
36.
1.36
40.18
= 1.45
TS-TCC Ours
91.86
0.97
91.92
38.45
1.12
43.52
1.33
42.61
192
45.06
1.11
Table 17: Performance comparison of our method in different CL frameworks for Heart Rate
Prediction
Method
IEEE SPC12
SPC22
Dal
MAE]
RMSE]
MAEI
RMSE,
MAEI
RMSEL
SimCLR + Traditional Augs.
20.67 + 1.13
26.35
0.98
16.84
+ 1.10
22.23
+ 0.72
12.01
+ 0.65
21.09 0.86
SimCLR R + Aug. Bank
27.31
2.17
37.93
+: 2.96
27.84
+ 2.03
36.41
+ 3.98
35.87
+ 4.18
40.61 = 3.74
SimCLR DACL
21.85
1.63
28.17
1.75
14.67
1.10
20.06
1.21
18.44
1.32
25.61
1.45
SimCLR + Ours
16.26
1: 0.72
22.48
1 0.95
12.25
1: 0.47
18.20
1: 0.61
10.57
1 0.55
20.37 = 0.73
BYOL Traditional Augs.
20.68
0.98
27.11 0.85
21.16
- 1.10
26.83
1.05
12.03
0.75
20.77
= 0.83
BYOL+ Aug. Bank
29.13
0.91
BYOL+ + DACL
26.45 1: 1.23
33.50
1.32
21.29
+ 1.13
27.34
1 1.33
15.11 0.93
23.21 = 0.83
BYOL Ours
19.85 0.88
26.10
0.94
22.08
+ 1.24
28.20
+ 1.13
11.45 0.63
20.38 0.80
ES-TCC+ Traditional Augs.
0.92
16.10
16.18
0.95
TS- TCC
Aug. Bank
11.44
1: 1.01
17.06
+ 0.94
13.79
+ 1.21
22,41
+ 1.08
17.28
+ 1.12
25 0.98
TS-TCC DACL
11.00
1: 1.16
18.26
1.200
15.25
+ 1.26
24.40
+ 1.10
16.27 - 1.16
24.28
0.97
TS-TCC
10.82
0.65
0.73
13.63
15.90
Tables 17 and 18 compares the performance of three data augmentation techniques, traditional time-series augmentations, DACL and our proposed method, in contrastive learning frameworks of BYOL, SimCLR, and TS-TCC.
Table 18: Performance comparison of our method in different CL frameworks for CVD classification
CPSC 2018
Chapman AUC1
Method
AUC
SimCLR + Traditional Augs.
67.86 = 3.41
74.69 + 2.04
SimCLR + Aug. Bank
81.78 1.24
SimCLR + DACL
82.38 0.84
9 0.97
SimCLR Ours
85.30 = 0.45
9
BYOL Traditional Augs
75.41 = 1.34
85.63 1.43
BYOL + Aug. Bank
83.51 1: 1.12
91.03 1 1.18
BYOL + DACL
77.61 - 1.16
81.62
+ 1.24
BYOL + Ours
83.25 1.03
+
TS-TCC Traditional Augs
87.07 1.10
92.03
1,17
TS-TCC Aug. Bank
86.67 1.04
92.15
1 1.02
TS-TCC DACI.
87.63 + 0.83
92.21
+ 0.86
TS-TCC Ours
88.05 0.37
92.11
1 0.75
The results show that the BYOL more robust to the choice of augmentations than SimCLR, which is also indicated in the original paper [59]. Also, another important outcome of this ablation experiment that when the TS-TCC framework used for datasets HHAR and USC, the performance decreases compared to other datasets. A possible explanation for this decrease in the TS-TCC might be the hyper-paramesn of the augmentations that are used in the paper. The authors change the strength of the permutation window from dataset to dataset. In our experiments, we used the same hyperparameter for all activity recognition datasets, which can explain the outcome. This ablation experiment also shows that the degree of traditional augmentations is important for contrastive learning to learn class invariant representations.
27

---

E.3 Do we still need data augmentations?
In this section, we conduct experiments to observe the performance methods without additional augmentations. During our experiments, we searched for the best traditional augmentation technique for each method in a given task. We searched over common time series augmentation methods in literature (Table and applied them with baselines. Specifically, we apply Resample for Activity Recognition, Permutation with Noise for Heart rate Prediction and Noise with Scaling for CVD Classification. We have observed that these augmentations yield the best results for all baselines when applied prior to the proposed techniques. However, for GenRep, we found that applying the augmentations after generating instances results in better performance, similar to the original work [51]. We, therefore, apply these specified augmentations for each baseline and report the corresponding results.
Different from other baselines, we observed performance increases for few datasets when GenRep is applied without any augmentations. This phenomenon can be attributed to the generation of low-quality and less realistic positive samples, where additional augmentations lead alterations in semantic information, due to less number of samples during training VAE models. However, in the end, we observe that applying additional augmentations always increases the performance on average for all baselines in each task.
Table 19: Performance comparison of methods without Augs. in Activity Recognition datasets
UCIHAR
HHAR
USC
Method
ACC1
MF1
ACC1
MF11
ACC1
MFI
IDAA 53
82.23 = 0.69
79.84 + 0.89
88.98 = 0.62
89.01 + 0.55
59.23 + 1.10
56.11 + 1.54
Aug.
64.42 (-17.81)
65.17 (-14.67)
86.44 (-2.54)
86.31 (-2.70)
35.22 (-24.01)
33.62 (-22.59)
GenRep [5
87.22 - 1.05
86.48 + 0.95
87.05 + 0.95
86.45 + 0.90
50.13 + 2.85
2.85
49.50 + 2.73
Aug.
88.01 (+0.79)
(+1.64)
86.51 (-0.54)
86.33 (-0.22)
48.31 (-1.82)
47.33 (-2.17)
DACL 22
73.12 = 1.23
66.28 + 1.11
80.89 + 0.91
81.31 +: 0.78
53.61 + 2.60
51.76 + 2.21
w/o Aug.
45.17 (-27.95)
44.84 (-21.44)
56.70 (-24.19)
56.55 (-25.76)
27.12 (-26.49)
26.99 (-24.77)
Ours
91.60 = 0.65
90.46 1: 0.53
88.05 + 1.05
87,95 +: 1.10
60.13 0.75
59.13 1 0.69
w/o Aug.
84.04 (-5.56)
83.34 (-7.12)
86.70 (-1.35)
86.72 (-123)
45.55 (-14.58)
44.94 (-14.19)
Table 20: Performance comparison of methods without Augs. in Heart Rate Prediction datasets
IEEE SPC12
IEEE SPC22
DaLia
Method
MАEІ
RMSEJ
MΑE!
RMSEL
MAEJ
RMSEJ
IDAA 53
19.02 + 0.96
27.42 1.11
15.37 1.21
22.41 1 1.42
11.12 + 0.64
20.45 1: 0.69
w/o Aug.
20.19 (+1.17)
28.51 (+1.09)
16.34 (+0.97)
(43.34)
16.01 (+4.89)
25.62 (+5.17)
GenRep [1
21.02 1.41
28.42 1.65
15.67 1.23
22.33
25.41 + 1.62
36.83 +: 1.87
w/o Aug.
20.51 (-0.51)
28.35 (-0.07)
23.07 (+7.40)
33.20
20.03 (-5.38)
31.01 (-5.82)
DACL [2
21.85
28.17 1.75
14.67 1.10
1.20
18.44 1.32
25.61
w/o Aug.
22.75 (+0.90)
29.90 (+1.73)
20.88 (+6.21)
(+2.70)))70)
28.24 (+9.45)
37.33 (+11.72)
Ours
16.26 0.72
22.48 0.95
12.25 + 0.47
18.20 0.61
10.57 0.55
20.37 + 0.73
w/o Aug.
19.41 (+3.15)
26.23 (+3.75)
16.41 (+4.16)
25.71 (47.51)
16.73 (+6.16)
27.43 (+7.06)
28

---

Table 21: Performance comparison of methods without Augs. in CVD classification datasets
CPSC 2018
Chapman AUC1
Method
AUC1
IDAA 53
80.90 0.73
93.63 + 0.91
w/o Aug.
79.00 (-1.90)
92.37 (-1.26)
GenRep [51
52.49 = 3.43
86.72 + 1.13
w/o Aug.
45.17 (-7.32)
84.51 (-2.21)
DACL [22
82.38 0.84
92.28 = 0.97
w/o Aug.
73.00 (-9.38)
75.10 (-17.18)
Ours
85.30 + 0.45
95.90 + 0.82
Aug.
79.67 (-5.63)
93.48 (-2.42)
Table 22: Common time series augmentations
Domain
Augmentation
Details
Noise
Add Gaussian noise sampled from normal distribution. \/0, 0.4)
Time
Scale
Amplify channels by random distortion sampled from normal distribution (2, (2,111))
Shuffle
Randomly permutt the channels the sample. (Not available for Heart rate Prediction)
Negate
Multiply the value of the signal by factor -1
Split signals into no more than segments, then permute the segments
combine them into the original shape
Interpolate time
original sampling
Resample
and randomly down-sample initial dimensions
Rotate the 3-axial (x, and z) readings of each IMU sensor by random degree, which follows uniform around random axis in the 3D space. (Only applied for Activity Recognition) Flip the time series in time
Rotation
Time Flip
channels, ie., XAsyIt] x] x|-
Out
Randomly chose section zemo out Combination Permutation and Noise Combination of Noise and Scaling
Permatation Noise
Noise Scale
Highpass
Apply highpass filter in the frequency domain reserve high-frequency components
Frequency
Lowpass
Apply lowpass filter in the frequency domaii to reserve low frequecyy components
Phase shift
Shift the phuse of time-series data with a randomly generalized number
Frequency
Add Gaussian noise, sampled from normal
((O0 0.5), frequency spectrum
29

---

E.4 The effect of interpolating phase components
Here, we investigate the effect phase interpolation of two samples on the CL performance. In our proposed method, we bring the phase components of the two coherent signals together by adding small value the anchor's phase the direction of the other sample. In this section, we apply the opposite case of our proposed method and increase the gap of phase difference between the anchor and randomly chosen sample. However, we mix their amplitudes according to our proposed method to only observe the phase effect. In other words, we perform the mixup as in Equation 46) Note that the phase mixing Equation (46 differs from the proposed method only by the sign change.
x+ = A(xt) = )x)(x) + +1 Ax) and p P(x) A0 Xp)
= F (A(xt)
where
(46)
Also, It is important to note that we sample the mixing coefficients for both amplitude and phase from same distributions the proposed method to have fair comparison. Tables 23 24
Table 23: in SimCLR and BYOL frameworks for Activity Recognition
Performance comparison of our method and its ablation regarding the phase interpolation
UCIHAR
HHAR
USC
Method
ACC1
MF11
ACC
MFL1
ACC1
MF1
SimCLR Traditional Augs.
87.05
86.13
85.48
1.16
84.31
53.47 + 1.10
0.95
SimCLR + Phase Gap
79.62 1 1.10
80.57 : 1.03
86.55 : 0.83
86.68 1 0.71
53.61
2.60
51.76
= 2.21
SimCLR Ours
91.60 0.65
90.46
0.53
88.05 +: 1.05
87.95
1.10
60.13
0.75
59.13
0.69
BYOL. Traditional Augs.
83.41 + 0.95
82.13
1.12
86.41
+ 0.97
86.31
+ 1.10
58.34
1.15
55.04
1.15
BYOL Phase Gap
78.66 0.63
75.45
= 1.02
85.82
+ 0.91
85.16
0.92
56.14
0.67
56.20
0.75
BYOL Ours
87.01 + 1.10
84.92
90.31
+ 1.16
90.45
+
56.87 1 0.91
55.01 = 0.95
Table 24; Performance comparison of our method and its ablation regarding the phase interpolation SimCLR and BYOL frameworks for Heart Rate Prediction
Method
IEEE SPC12
IEEE SPC22
DaLia MAEJ
MAR]
RMSEL
MAE]
RMSE
RMSEJ
SimCLR + Truditional Augs.
20.67 + 1.13
26.35 : 0.98
16.84 + 1.10
22.23 + 0.72
12.01 + 0.65
21.09 = 0.86
SimCLR Phase Gap
18.90
25.29
1.56
14.60
1.03
19.84
1.15
17.57
4 1.13
27.72
1.35
SimCI.R RR + Ours
16.26
0.72
22.48
0.95
12.25
+ 0.47
18.20
0.61
10.57
+ 0.55
20.37
0.73
BYOL Traditional Augs.
20.68
1: 0.98
27.11
0.85
21.16
+ 1.10
26.83
+ 1.05
12.03
20.77
= 0.83
BYOL Phase Gap
25.93
0.96
32.68
0.90
21.87 1.03
29.13
1.03
17.46
27.24 0.83
BYOL Ours
19.85
0.88
26.10
0.94
22.08
1.24
28.20
1.13
11.45 +: 0.63
20.38 0.80
Table 25: Performance comparison of our method and its ablation regarding the phase interpolation SimCLR and BYOL frameworks for CVD classification
CPSC 2018
Chapman AUC1
Method
AUC
SimCLR + Traditional Augs.
67.86
3.41
74.69 + 2,04
SimCLR Phase Gap
77.45
1.10
91.95 0.91
SimCLR Ours
85.30
0.45
95.90
0.82
BYOI. + Traditional Augs
75.41 + 1.34
85.63
+ 1,43
BYOL- + Phase Gap
83.11 + 1.03
91.02
1.11
BYOL + Ours
83.25
1.03
91.23
1.15
30

---

E,5 The comparison of Mixup methods
In this section, we give detailed comparison of prior mixup methods with ours below tables, which are the explicit numbers for Figure 2 Our method demonstrates superior performance compared previous mixup techniques
out of 14 metrics, indicating its effectiveness. Additionally, the Amplitude Mixup technique, which yields comparable results in two datasets, further supports our claim regarding the destructive effect of simultaneously mixing phase and magnitude for time series. The relatively lower performance of Amplitude Mixup for some datasets can be explained by its limited diversity in generating positive samples since this technique has no solution for mixing the phase of samples in randomly chosen pairs. In other words, the phase of the augmented instance is the same as the anchor Amplitude Mix, the diversity of generated positive samples is less compared to other techniques.
Table 26: Performance comparison of ours with prior mixups in Activity Recognition datasets
UCIHAR
HHAR ACC1
USC
Method
ACC
MF11
MF1
ACC1
MF11
Geo
36.31 1 10.15
33.21
1 12,25
33.16 1661100 8.32
31.15 1 9.25
24.85
9.43
21.64 1 8,94
Amp
8 0.89
80.78 + 0.78
87.85 + 0.83
85.53 + 1.10
41.29
0.56
39.77
1.03
Spec
4
38.34 1.::-.95 1.95
56.73
2:3::2000 2.01
53.54
1:4:::: 1.98
2.55 23.45
21.30
2.41
Cut
50.21 1.34
48.23
1.23
57.71 1.12
53.87
+ 1.09
25.63
23.41
13.11:1. 3.11
Binary
74.13
71.31
77.12
7
42.21
0.97
41.53
1.10
Linear
1.23+22.11
80.25 + 1.93
80.11 + 2.05
81.31 1.73
40.15
39.71
1.14
Ours
84.30 0.73
83.23 + 0.58
84.51 + 1.10
83.98 + 1.03
45.36 + 0.97
43.14 + 0.81
Table 27: Performance comparison of ours with prior mixups in Heart Rate Prediction datasets
IEEE SPC12
IEEE SPC 22
DaLia MAET
Method
MAE]
RMSE!
MAE,
RMSET
RMSEL
Geo
32.65 1 7.25
48.90 1 9.87
37.15
6.74
36.32 1 6.21
38.45 1 7.31
41.32 1 6.21
Amp
23.01 1 0.95
30.10 1 1.04
18.07
1.13
23.13
1 1.43
19.05
1.63
30.41
1.65
Spec
24.09 4.10
38.41 +398
24.41
4.10
29.93
4.10
26.71
4.34
35.31
3,93
Cut
24.98 3.93
35.67
4.15
21.77
4.45
28.43
3.97
31.75
4.10
43.56
= 3.88
Binary
32.23 = 1.67
40.21
1.98
22.55
= 1.87
28.78
2.10
19.71
= 2.15
28.83
2.45
Linear
24.31 + 1.54
31.29
1299111 1.75
18.52
1.43
22.54
= 1.49
24.16
= 1.89
32.46
1.97
Ours
21.13 1 0.89
28.21
1.15
16.17
1 0.85
21.13 1 1.05
16.64
1.20
28.43
1.43
Table 28: Performance comparison of ours with prior mixups in CVD classification datasets
CPSC 2018
Chapman AUCt
Method
AUCt
Geo
45,65
6.43
61.32 5.79
Amp
84.10
1.05
89.83
1.12
Spec
69.26
3.10
70.48
+ 3.05
Cut
72.20
2.98
79.23
2.75
Binary
80.53
1.62
82.56
1.45
Linear
78.02
1.43
90.21
1.15
Ours
1.10 83.79
93.85
1.05
31

---

Illustrative Examples
In this section, we show examples of the destructive behavior of linear mixup and how our proposed mixup technique solves this problem. Figure we show two PPG waveforms that are obtained from IEEE SPC15 with the same label i.e., the same heart rate value. Also, we give the corresponding frequency domain transformations of these two waveforms Figure where the frequency axis converted to heart rate in beats-per-mene per-minute i.c., Hz corresponds to 60 bpm.
a)
b)
Anchor
Anchor
Sample
0.
Sample &
(Miltto o W 1000 M
Normalized Magnitude M
Normalized Power (dB)
Two samples with same label where the anchor has a severe motion artifact at H2.
93
01
4
60
100
140
180
180
200
Time (s)
Heart rate (bpm)
Figure a) The waveforms of anchor and random sample, b) The frequency domain (A(x)) representation of two samples.
When the linear mixup applied as in Equation 47 with À of 0.9, the resulting waveform is anticipated to contain heart rate information to an extent similar to both the anchor and the sample.
Ax (1 A)
(47)
However, when there is phase difference greater than T/2 between these two samples in the frequencies where the task spec-fce information is carried, the linear mixup destroys the information.
a)
b)
Anchor X
Lincar mixup xt
0.5
Sample
Linear mixup x"
Normalized Power (gB)
Normalized Magnituce
The linear mixup method generates augmented sample where the information the critical frequency band decreased significantly compared to both samples.
83
100
120
150
180
Time
Heart rate (bpm)
Figure 6: a) The waveform of anchor and augmented sample with linear mixup. b) The frequency domain (A(x)) representations of samples where the augmented waveform has lost all the information in the critical frequency band, i.e., the task-specific information is lost.
32

---

Figures and demonstrate the destructive behavior of linear mixup instead of feature interpolation. The linear mixup technique destroys the seek-specific information even though the two samples have the same labels and the mixup ratio is relatively high.
As proposed mixup prevents this problem and interpolates between features of samples, the information is not lost but rather enhanced as both samples have the same label, shown in Figure 7
a)
b)
Anchor X
Anchor
Proposed mixup x"
0.5
Sample
2 Normalized Magnitug (M)
Proposed mixup
Normalized Power (dB)
procosed moxup technique interpolates the features without causing any information the criticat frequency band
2.
10
00
100
120
340
150
130
200
Time (5)
Heart rate (bpm)
Figure 7: a) The waveform of anchor and augmented sample with proposed mixup technique, b) The frequency domain (A(x)) representations of samples where the augmented waveform carries the information in the critical frequency band an interpolation of two samples.
As can seen From figures 6 and 7 our proposed mixup technique not only prevents information loss due to linear mixup but also generates interpolated sample.
33

---

G Performance in Supervised Learning Paradigm
We also conduct experiments in the supervised learning paradigm with our proposed mixup method to see its effectiveness in different learning paradigms. We compare the performance of our method with prior mixup techniques. During the experiments, we follow the original implementation where the mixup is applied to the same minibatch after random shuffling. In the seminal work of mixup [11, the authors stated that interpolating only between inputs with equal labels does not lead to performance gains. Therefore, we only perform the tailored mixup without implementing any VAEs. We implement the tailored mixup for the supervised learning as in Equation 48) Although the phase equations can be simplified by omitting the absolute value, leave similar to Equation 4 for consistency.
x = F (A(xx
/P(xt)) where A(x+) A +
(x)) and if ÁO > 0 and X 2 0.5
P(x) |0]] # - Ap),
P(x) |0 - AP);
if A0 I 0 and X 2 0.5
P(x+)
(48)
P(X) - |40| * - (1-Ap), if AO > 0 and X < <.5
P(X) A0 (1 Ar);
if AO S 0 and X < 0.5
y+ = AAyx + +1 XA)UR,
where the coefficient for the AA is chosen from a beta distribution with a 0.4] within the same range of the original implementation (3I] The mixing for the phase is constrained to our original implementation with uniform Ap (0.9, 1). We searched for the best value for each time-series task and augmentation method. Unlike linear mixup and our proposed approach, for cutmix, we followed the recommendation from the original paper and searched the value close 1.
Table 29: Performance comparison in Activity Recognition within supervised learning scheme
UCIHAR
HHAR ACC1
USC
Method
ACC1
MF11
MF11
ACC1
MFI1
W/o Augs.
65.66 + 0.23
61.21
+ 0.15
91.58 + 0.07
91.64 0.11
71.93 = 0.54
68.43 + 0.78
Linear Mix
77.06
0.18
73.21
0.17
93.64 0.17
93.67 06674:00 0.08
74.45 = 0.28
71.93 + 0.43
Amp Mix
70.96 + 0.19
67.14
+ 0.33
92.50
0.15
92.54
74.02 - 0.19
71.90 + 0.26
Binary Mix
69.01 + 0.36
71.63
+ 0.11
92.36
+ 0.19
92.42
72.81
+ 0.15
70.98
+ 0.35
CutMix
67.14 1
63.31
0.48
90.37
1 0.43
90.36
$7.89 - 0.34
61.45
1 0.57
Ours
81.60 + 0.15
79.35
+ 0.13
94.02
- 0.05
94.00 - 0.06
74.85
0.19
72.45
+ 0.34
Table 30: Performance comparison in Heart Rate Prediction within supervised learning scheme
IEEE SPC12
IEEE SPC 22
Dal.ia
Method
MАE!
RMSE,
MAET
RMSEL
MAE]
RMSEJ
W/o Augs.
20.01 + 0.03
27.16 + 0.05
20.29 + 0.87
26.60 + 1.13
6.58 + 0.10
11.30 0.58
Linear Mix
20.07 1 0.09
26.93 0.10
19.98
1 0.12
24.90
0.51
6.97
0.14
12.07 1 0.51
Amp Mix
20.14
0.07
26.98
0.07
19.61
+ 0.07
24.11
11.20 1
16.07 0.43
Binary Mix
21.05 :
27.02
2 0.08
19.62 - 0.10
25.23
0.13
7.35
12.17
0.53
CutMix
20.12 0.06
0.11
19.64
24.18 0.20
10.78 - 1.23
14.40 1 1.43
Ours
19.97 + 0.05
26.98 - 0.10
19.45 1 0.12
24.35 0.18
6.49 0.08
11.69 1 0.10
Table 31: Performance comparison in CVD classification within supervised learning scheme
CPSC 2018
Chapman
Method
AUCt
AUCt
W/o Augs.
82.01 + 0.51
92.27 + 0.35
Linear Mix
80.29 + 0.93
93.02 + 0.33
Amp Mix
80.01 + 0.36
89.11 0.27
Binary Mix
78.10 0.98
80.31
+ 0.36
CutMix
80.75 - 0.78
89.17 0.58
Ours
83.75 + 0.32
95.26 + 0.24
34


Reviews:
[{"review_id": "4CROpamoJ5", "reviewer": "Reviewer_JK7u", "summary": "This paper explores data augmentation for contrastive learning that better aligns with the nature of time series. Specifically, they tailor the mix-up method separately to amplitude and phase in the frequency domain. A theoretical analysis illustrates how the method enhances task-specific information, contrasting it to the linear mix-up method which potentially discards task-specific information. Experiments on time-series from three domains also demonstrated the advantages compared with previous mix-up and contrastive learning methods. ", "questions": "See weakness above. ", "limitations": "The paper did not explicitly discuss their limitations. Some comments and questions regarding limitations of the work are discussed in the weakness and question parts above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper studies an important problem of designing augmentation for contrastive learning that adapts to the non-stationarity of time series. \n\n2. The paper presents theoretical analysis of the benefits of proposed mix-up method compared with vanilla linear mix-up method.\n\n3. The paper shows better performance on time-series datasets of multiple domains. ", "weaknesses": "**Lack of baseline comparisons**: The paper compares with a list of mix-up methods and contrastive learning methods, and the authors also incorporate time-series augmentation methods into the framework in appendix. However, it would also be interesting to have (1) some direct comparisons with existing time-series augmentation methods, e.g., [1, 2] (both have mix-up augmentation in the time or frequency domain). (2) Contrastive learning methods dedicated for time-series domain [3, 4, 5, 6] (I am curious why [4] is not one of the baselines in experiments?)\n\n[1] FrAug: Frequency Domain Augmentation for Time Series Forecasting \n\n[2] Towards Diverse and Coherent Augmentation for Time-Series Forecasting\n\n[3] CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting\n\n[4] Self-supervised contrastive pre-training for time series via time-frequency consistency\n\n[5] Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding\n\n[6] TS2Vec: Towards Universal Representation of Time Series \n\n\n**Experimental analysis**: Why linear mix-up method performs well in activity recognition task? Is there any difference in the setting of activity recognition compared with the other two tasks?\n\n**Writing**: \n\n1. Consider defining notations of x, \\tilde{x}, x^* before or in Proposition 2.3. \n\n2. The title metions \"order\" and \"chaos\"; hence, a detailed explanation of these terms, as well as how the proposed method comprehensively accounts for the \"non-stationarity\" of time series, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback about our paper on data augmentation for contrastive learning in the context of quasi-period time series data. We also thank you for valuing the importance of the problem. We would like to address certain points raised in your review. \n\nWe are well aware of all the works you cited in your review, except the [2]. \n\nWe apologize for missing reference [2], which emerged close to our submission. This paper indeed proposed a data augmentation method for time series but the overall focus was for forecasting without considering the quasi-periodic nature of signals. We applied their proposed method.But,as they emphasize different frequency components via random weights, its performance is quite low compared to ours (as can be seen from the submitted single-page pdf). We will include them in our revisited version.\n\nSecondly, we haven't explicitly compared our proposed augmentation with [1] in the original script because the work presented in [1] is for time-series forecasting and the augmentations are tailored for this purpose (not for quasi-periodic signals, similar to [2]) where comparison will be unfair. Even in the augmentation, there are look-back window x and target horizon y. How can you think integrating these augmentations to our problems make sense? For example, what is the look-back window and target horizon for ECG signals in cardiovascular disease detection?\n\nIf you look at the baselines we used, they all claim their proposed methods can be applied to other domains (e.g.,DACL). However,as a reply to your question, we applied the data augmentations in FrAug (with original parameters) which are mainly frequency masking and mixing, you can find the results in the pdf. The performance of this method is low since sometimes the key freq. components are masked or mixed as the characteristic of quasi-periodic signals is different.\n\nThirdly, regarding the other cited works, we've already applied the augmentations mentioned there in our experiments. If you look at the augmentations used in these works, you can see we already used and pointed out them in Appendix. \n\n\n[3]CoST:Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting\n\nScaling,Shifting,Jittering,\n\nAll of them are used and given in the submitted manuscript.\n\n[4]Self-supervised contrastive pre-training for time series via time-frequency consistency\n\nIn this work, the author's main contribution is not a data augmentation method but a framework for domain adaptation.That was the main reason, we did not explicitly compare this as a baseline, which will not be a fair comparison. In other words, this work is not a baseline for us. We propose a data augmentation method for unsupervised learning, not a framework.\n \nThe authors propose to perturb only the magnitude of the frequency domain without changing the phase while the perturbation is performed by sampling from Gaussian. In the paper, the authors showed that perturbation of high and low frequency affects performance differently depending on the application. Therefore, we applied this to the whole band during our submission. As per your request, we also applied to the low band separately as in the original paper. In the end, the performance decreased severely.\n\n[5]Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding.\n\nThere is no data augmentation used or proposed in this paper. Again, we propose a data augmentation method for quasi-periodic nonstationary signals. This work is not even a baseline for our work.\n\n[6]TS2Vec:Towards Universal Representation of Time Series\n\nTimestamp Masking, Random Cropping\n\nWe both applied these augmentation techniques as baselines (random zero out and permute)\n\n\nQ)Experimental analysis:Why linear mix-up method performs well in activity recognition task?Is there any difference in the setting of activity recognition compared with the other two tasks?\n\n\nThe answer to this question outlines the main contribution of this paper. The mixup method demonstrates a destructive impact in cases where it is applied to two sinusoids with the same frequencies yet different phases. The PPG (for heart rate) and ECG (for cardiovascular disease) signals exhibit a higher degree of quasi-periodicity, leading to diminished performance when used with linear mixup. These two signals display a greater level of periodicity as they stem from the human body. In contrast, in the realm of activity recognition, the extent of periodicity varies with different activities—like sitting (characterized by lower periodicity) and walking (exhibiting higher periodicity).\n\nOur proposed method addresses this issue and leads to a substantial enhancement in performance. Hope, the answer to this question clarifies the contribution of our presented work.\n\nRegarding your claim of lack of baseline comparisons, in our work, we used 8 datasets in 3 domains with 14 baselines including automatic data augmentation techniques (DACL, GenRep) and 6 different mixup methods in 2 contrastive learning frameworks. Our theoretically well-grounded mix-up overperformed all of these in 7 datasets from 3 domains while ranking second in one dataset. Your claim of lack of baseline is not realistic at all. Although we agree that using different contrastive frameworks can be more interesting, our experiments including ablation studies are comprehensive and robust invalidating any notion of inadequate baseline comparisons. Additionally, we include one more contrastive learning framework that is designed for time series where we observe the same behavior.\n\nHope, the reviewer would be convinced by the additional results and understand the difference between a lack of baseline and more interesting experiments. We believe our work has enough baseline, evaluating with 2 more contrastive learning frameworks does not add anything significant to the already proved contribution that is substantiated both theoretically and empirically.\n"}, {"review_id": "8N5qgeDOHU", "reviewer": "Reviewer_hEg3", "summary": "This paper is about a novel data augmentation method that can be used in contrastive learning for time-series tasks and aims to connect intra-class samples together, and find order in the latent space. The proposed method builds upon the mixup data augmentation technique by controlling the degree of chaos created by data augmentation. More specifically, the augmentation method considers the phase and the amplitude information as two separate features and then generates positive samples by controlling the mixup coefficients for each feature for each randomly chosen pair. This process helps to generate features that enhance intra-class similarity and help contrastive learning to learn class separated representations. The proposed method is evaluated using three time-series tasks: heart rate estimation, human activity recognition and cardiovascular disease detection and against state of the art comparison methods.", "questions": "My main concerns are regarding the readability of the paper and improvements in the structure of the paper, and the experimental setup that is missing time-series contrastive learning models. The results and the discussion will be more convincing and the performance difference more important if the author include the results of some other methods, like [22] and [80]. A comparison of the methodology and the concepts of the proposed method with [22] and [80] is needed as these paper are around the same ideas. What is different in the contributions of the proposed method?", "limitations": "There are no limitations or negative societal impact from this work. The datasets used in the paper are all publicly available.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is about an interesting topic, contrastive learning in time-series.\n- The proposed methodology is based on the idea of controlling the degree of chaos created by data augmentation methods. These ideas can be applied in the augmentation part of contrastive learning for other domain as well.\n- The proposed methodology shows improvement in the performance over ten comparison methods on three tasks and eight datasets. Some time-series contrastive learning methods are missing from the comparison though.", "weaknesses": "- The writing in the paper can be improved. The Method and Results and Discussion sections need some organization, adding subsections will help with the structure.\n- The proposed method has incremental novelty. This method builds on top of known components. The writing needs improvements to make the contributions of this paper more clear. \n- The experimental setup also seems to be missing some important state of the art. No time-series specific contrastive learning method is included in the experiments for comparison.\n- The proposed methodology seems to have similarities with TS-TCC [80] and TFC [22] (the reference numbers align with the ones in the main paper) and since these are state of the art in contrastive learning for time-series, they should be part of the comparison methods. \n- A comparison of the methodology of the proposed method with [22] and [80] is needed as these paper are around the same ideas. What is different in the contributions of the proposed method?\n- Dataset statistics are missing from the Results and Discussion section. Adding them in a table helps with the understanding of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments. And, we would like to clarify some misunderstandings and points arising from your review of our paper.\n\nRegarding the claim of your incremental novelty, it's important to note that there currently exists no prior study that has specifically addressed the destructive effects of the linear mixup when applied to periodic time series data, nor has any previous work proposed a corrective methodology in this context. While linear mixup has demonstrated efficacy within image-related domains, its applicability to time series analysis remains constrained due to the inherent periodic nature of time series data. Our research also highlights that increased signal periodicity shows a significant decline in mixup performance. Given this distinct gap in the existing literature, we argue that our work should not be characterized as incremental in nature.\n\nWe believe that our work contributes several insights, including those of technical nature. In summary,\n\nTo the best of our knowledge, our work is the first to \n\n\n* present the destructive behavior of linear mixup for quasi-periodic time-series data both empirically and theoretically with mathematical proofs.\n* propose a novel mixup approach for time-series data while preventing information loss of prior works on mixup\n* take a novel approach for sampling mixup coefficients for each pair based on their similarity in the latent space, which is constructed without supervision while learning disentangled representations, to prevent aggressive augmentation between inter-class samples.\n* demonstrate that our approach significantly outperforms well-known mixup methods with state-of-the-art data augmentation techniques in 8 datasets while comparing with 14 baselines. \n\n\nAlthough we will more strongly emphasize our contributions in the revised version of the paper, we kindly wish to draw the reviewer's focus to the non-incremental nature of the paper's contributions.\n\n\n\nRegarding your concern for [80] and [22], these papers proposed an unsupervised time-series representation learning framework. Therefore, they are similar to SimCLR and BYOL, but not our presented method.\n\nIn this paper, we propose a data augmentation strategy to improve the performance of unsupervised contrastive learning frameworks by using a tailored mixup method for quasi-periodic signals. The mixup is well-appreciated in the vision community as it offers an easy and effective data augmentation strategy, even so, many works proposed different variations of mixup such as cutmix, binarmix, geomix. In this paper, we showed that the original mixup and variations do not perform well with the quasi-periodic signals and proposed a tailored theoretically proven mixup method and showed the improvement.\n\n\nTo explain further, you can use our proposed method with any unsupervised contrastive framework such as SimCLR, BYOL, NNCLR, MoCo, or [80] and [22] as a data augmentation technique. Therefore in Appendix E, we showed how our proposed data augmentation increases the performance of BYOL and SimCLR compared to the traditional augmentation used in time series such as jittering. However, you cannot use [80] and [22] together with SimCLR or BYOL. They constitute a replacement for those self-supervised learning methods. Therefore, we think our experimental setup with the additional experiments has no missing parts and clearly indicates the contribution of the paper.\n\nMoreover, [22] proposed a pre-training strategy while considering the time-frequency consistency (TF-C) —embedding a time-based neighborhood of an example close to its frequency-based neighborhood, which they mainly used for domain adaptation whereas, in our paper, we showed the destructive feature of mixup for quasi-periodic signals and fix this drawback to increase the performance for unsupervised contrastive learning.\n\nSimilarly, in [80]  authors propose a framework to learn time-series representation from unlabeled data. Their framework requires weak and strong data augmentations where they implemented weak augmentation as a jitter-and-scale and strong augmentation as a permutation-and-jitter. We both compared the performance of those augmentations with ours in two different contrastive learning frameworks and showed the improvement. \n\n\nResponding to your question, we have also introduced another contrastive framework explicitly tailored for time series data. In the submitted pdf, we demonstrate the performance enhancement achieved through the integration of our method with this additional framework. Therefore, we can confidently say that our proposed method exhibits substantial distinctions from [80] and [22] in terms of its contributions. Our work is not a time-series contrastive learning method where we should compare with other frameworks as opposed to your claim. Furthermore, it's worth noting that we have conducted comparisons with all previously published known data augmentation methods in 8 datasets.\n\n\nRegarding the dataset statics, we already mentioned some statistics about datasets in Appendix B and we will certainly address this suggestion by incorporating dataset statistics in a table.\n\n\nWe apologize for any confusion caused by the misinterpretation. We believe that these clarifications with additional results will now help address any misconceptions and make the contribution of our paper clear to the field.\n\nThank you for your time and consideration again.\n"}, {"review_id": "CUy062m1S6", "reviewer": "Reviewer_JMVG", "summary": "This paper proposes a novel mixup method for non-stationary time-series data to generate positive samples for the contrastive learning formulation. The proposed method mixes the magnitude and phase of each frequency component of two samples that are close in the latent space of a variational autoencoder. The authors prove that the mixup process in the frequency domain does not cause the loss of information while it generates diverse samples. The paper provides an extensive empirical evaluation to show that the proposed method learns better representations than existing contrastive learning baselines on eight time series datasets from three tasks.", "questions": "1. How do you select $\\beta$? Would a very large (close to 1) $\\beta$ limit the diversity of the augmented samples?\n\n2. What does the instance selection (line 268) mean exactly?", "limitations": "Yes", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper studies an important and under-explored problem. Although contrastive learning is proven to work well on images, its performance on time series is yet limited due to the lack of good time series augmentations. While many existing work tries to find the good time series augmentations through extensive empirical studies, this paper proposes a theoretically well-grounded mix-up method for time series augmentation. Moreover, the empirical evaluation is also extensive and convincing.", "weaknesses": "1. The hyperlinks of the citations and the references are missed. \n\n2. I don't find where $x^*$ is defined.\n\n3. It would be great if there is an algorithm box explaining the whole process of drawing samples, computing the degree of the augmentation, performing the mixup, and then training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review of our work. We appreciate the feedback and are glad to hear that our work has been received positively.\n\n\nRegarding the weaknesses,\n\nthank you for bringing hyperlinks to our attention. We will fix this issue in the revised version of the manuscript. Your feedback is greatly appreciated.\n\n\nWe appreciate your observation regarding the absence of the definition for \"x*.\" In prior works, namely DACL and GenRep, this term was denoted as the optimum generated or augmented sample. This contextual reference guided our usage of the term before formally defining it. However, we understand the importance of clarity and will provide a precise definition for \"x*\" in the revised manuscript. We thank you for your valuable suggestion.\n\nWe shared the same thought in the submission process but we were keen to add another figure to the manuscript as we have a limited space. However, we appreciate your input and will incorporate a new figure to provide a clear and comprehensive overview of the process in the revisited manuscript or Appendix. Thank you for your thoughtful suggestion.\n\nRegarding the questions, we have replied to each of them below. \n\nQ1) In previous mixup methods normally $\\beta$ is chosen with high values such as 0.8 or 0.9. However, the major drawback of this is that when you increase the $\\beta$ to closer to 1, the mixup samples are getting closer to the anchor sample, and the diversity of them decreases significantly as you expected. Therefore, we followed a similar procedure with previous works and performed a grid search amongst values 0.7 to 0.9 and chose the best values. We also observed that the $\\beta$ values are quite flexible for the magnitude mixing while for the phase it should be more closer to 0.8-0.9. We believe that this can be attributed to the influence of phase values on the semantic characteristics of the signal. Altering these phase values exerts a potent impact on the inherent features conveyed by the signals.\n\nQ2) We thank the reviewer for this careful revision. It is our mistake. It should be the mixup coefficient selection according to phase and magnitude rather than the instance selection as we indeed choose mixup coefficients rather than instances.\n\nBy addressing your questions, we hope to provide clarity and resolve any uncertainties you may have. We genuinely appreciate your thorough review and the insights you've shared. Thank you once again for your careful attention to detail during the revision process and appreciation of our work.\n"}, {"review_id": "WZ51vLtXUJ", "reviewer": "Reviewer_CSD8", "summary": "This paper presents a new time series augmentation method which adopts Mixup on the amplitude and phase, respectively, after transforming the time series in the frequency domain, thus avoiding the destructive extrapolation from linear Mixup.", "questions": "1. The ablation experiments considering amplitude or phase alone for mixup need to be performed consistently with the parameter settings of the original method.\n2. Please give practical examples of how Mixup may corrupt the frequency domain information used to discriminate the generated samples.\n3. It is not clear how the so-called \"control the degree of chaos\" proposed by the authors is done.\n4. For data augmentation of time series, why does this paper choose to perform augmentation in the frequency domain instead of the time domain? In general, it is difficult to use only frequency domain information for classification over the time domain for time series classification tasks [1]. In contrast, the combination of time and frequency domain information can effectively improve the model's classification performance [2].\n5. What are the advantages of performing data augmentation in the frequency domain instead of doing data augmentation in the time domain of time series data?\n6. Why are eight datasets chosen for experiments in this paper? Its inconsistency with existing benchmark datasets for time series prediction [3,4,5], classification [6,7] and anomaly detection [8] tasks. In addition, the comparison methods in this paper do not include benchmark methods in the time series domain. For example, benchmark methods for time series prediction include Informer [3], CoST [4], and FEDFormer [5], etc., time series classification methods include OS-CNN [6] and DSN [7], etc., and time series anomaly detection includes Anomaly Transformer [8], etc.\n\n[1] Cross reconstruction transformer for self-supervised time series representation learning. arXiv, 2022.\n[2] Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency. NeurIPS, 2022.\n[3] Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. AAAI, 2021.\n[4] CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting. ICLR, 2022.\n[5] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. ICML, 2022.\n[6] Omni-Scale CNNs: a simple and effective kernel size configuration for time series classification. ICLR, 2022.\n[7] Dynamic Sparse Network for Time Series Classification: Learning What to \"See\". NeurIPS, 2022.\n[8] Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy. ICLR, 2022.\n", "limitations": "The authors did not mention the limitations. I do not have any comments on this point.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose a novel mixup method in the frequency domain for time series, avoiding the destructive extrapolation resulting from linear Mixup.\n\n2. The theoretical proof is complete and has some reference value.\n", "weaknesses": "1. It is unclear whether it works when considering amplitude or phase alone for Mixup. The relevant ablation experiments must be performed consistently with the parameter settings of the original method.\n\n2. Lack of convincing examples. For example, to illustrate the problem, perform Mixup on two samples where the frequency domain information in both samples that are strongly correlated with I(x,y) exactly cancels each other and shows whether the newly generated samples have specific frequency domain information that can be identified.\n\n3. The paper's content does not explain how to \"Find Order in Chaos\" mentioned in the title and how the \"chaos\" is reflected. And I'm confused about how the so-called \"control the degree of chaos\" proposed by the authors is done.\n\n4. Assumption 2.1 is too idealistic. The label information of time series is not only related to the time domain but also the frequency domain, and focusing on the frequency domain information only does not achieve the classification well.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review. We appreciate the feedback and the recognition of the importance of our work. We replied to each concern as follows.\n\n1)Regarding the ablation experiments focusing on amplitude or phase alone for the mixup. We'd like to clarify that we conducted a parameter search to determine the optimal settings for all mixup methods. This approach is commonly used as a general practice to ensure the best performance of the baselines in different domains.\nHowever, it's important to note that the original mixup method was initially designed for images with a distinct distribution compared to time series. As such, directly employing it for comparison would be inequitable. Even for that, the parameters for the mixup methods are very close to the original ones in the literature within a range of 0.8 to 0.9.\n\n2)Linear mixup has the potential to directly distort information across both time and frequency domains, rather than being limited solely to the frequency, as demonstrated by the mathematical proof we provided.\nAs a practical example: Assume you've two signals with the same frequency (PPG or ECG with the same heart rate), if the phase difference between these two signals is more than $\\pi/2$, there will be a destructive interference and the magnitude of the waves will decrease depending on the phase difference and magnitude of them.\n\nWe believe that it is important to mention the mixup is nothing more than adding two signals together with different ratios. If you apply this to quasi-periodic signals, your waves can destroy each other. This is quite common in nature as well from ocean waves to lights where the most famous example is the double-slit experiment [1-2] which is considered evidence for the probabilistic nature of quantum mechanics.\n\n[1]Young,Thomas (1804). \"The Bakerian lecture. Experiments and calculation relative to physical optics\". Philosophical Transactions of the Royal Society of London.\n[2]Kipnis,Naum S. (1991). History of the Principle of Interference of Light. Springer\n\nThe empirical evidence of this destructive mixup can also be seen from the question of reviewer JK7u “Why linear mix-up method performs well in activity recognition task?”\n\nThe other two tasks involve strong periodicity as they originate from the human cardiovascular system, causing linear mixup to perform much worse compared to the less periodic data from activity recognition.\n\n3)The recent paper “Chaos is a ladder” [3] ([18] in the manuscript) showed that the data augmentations create chaos where contrastive learning climbs that ladder. Also, they showed that a good data augmentation should create samples that are similar to those in intra-class.“For example, two different cars become very similar when they are both cropped to the wheels”.\nIn this work, we proposed to control the augmentation degree by controlling the mixup coefficients while looking at the semantic similarity of the samples in the latent domain in an unsupervised way. For example, if the two samples are close to each other in the latent domain, the coefficients are more aggressive to make them closer. Although, we know this approach has some limitations, we mentioned its performance improvement and limitation in Section 5.1 Ablation studies.\n\n[3]Yifei Wang et al. Chaos is a ladder:  A new theoretical understanding of contrastive learning via augmentation overlap. ICLR 2022\n\n4-5)We think these two questions align at the same point and there is a misconception we would like to clarify.\nIn this paper, we have introduced a novel approach to overcome the limitations of mixup by shifting the mixing process from the time domain to the frequency. We did this because we want to get rid of the destructive mixup, which is due to the phase difference between the same frequencies. We had to split these two pieces of information for a wave that motivates us to shift the frequency domain. Otherwise, our augmentation method has a direct correspondence within the time domain too. \n\nWe believe that the time or frequency domain has no specific advantage over it in data augmentation for contrastive learning. It is just a matter of how you can keep the task-related information while increasing the diversity of samples, which was shown by InfoMin [53,in script], a seminal work in augmentations for contrastive learning.\n\n6)The selection of 8 datasets for experiments was driven by the objective of our study, which is to investigate the mixup with quasi-periodic signals and their related applications. While benchmark methods like Informer, CoST, and FEDFormer are valuable in their respective domains, they are not employed as benchmarks within the datasets we used. If you look at the previous works with the same datasets, you can see we used the common benchmarks [4-7].\n\n[4]Hangwei Qian et al. Latent independent excitation for generalizable sensor-based cross-person activity recognition. AAAI 2021\n\n[5]Garrett Wilson et al. Multi-source deep domain adaptation with weak supervision for time-series sensor data. ACM SIGKDD 2020\n\n[6]Dwaipayan Biswas et al. Cornet: Deep learning framework for ppg-based heart rate estimation and biometric identification in the ambulant environment. IEEE Transactions on Biomedical Circuits and Systems, 2019\n\n[7]Francisco Javier Ordóñez et al. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 2016.\n\nIf we use the models you suggested for these tasks, a similar question \"Why did you use different models for these datasets?\" can be asked again.\n\nRegarding assumption 2.1, the intention behind is not to claim that label information is exclusively encoded within the time or frequency domain. Rather, it asserts that the signal-to-noise ratio contains informative characteristics of a wave.\n\nAgain, we thank the reviewer for this careful review and appreciation of our work. Hopefully, these answers have clarified any questions you may have had."}]

Author rebuttal:
We would like to thank all the reviewers for taking the time to read our paper and provide feedback.

In response to the reviewers' comments, we included two more data augmentations for comparison. We also applied our proposed data augmentation method to one more contrastive learning framework, which is specifically proposed for time-series data, in addition to SimCLR and BYOL. 

Following the submission of our responses, we would greatly appreciate receiving additional feedback from the reviewers, particularly from Reviewer JK7u and hEg3 (the first two reviewers). Their reviews appear to misinterpret the paper, viewing it more as a novel unsupervised contrastive learning framework rather than recognizing it as a novel data augmentation method for quasi-periodic signals, which is evident from their request for a direct comparison (baseline) with unsupervised learning methods.

Even if the reviewers maintain their initial assessments, we value the opportunity to gain additional clarity on your feedback.

If you believe there are areas where our responses did not fully address your concerns or if there are aspects that could be improved to merit a higher score, we would greatly appreciate your insights on those specific points. Your feedback is crucial in helping us enhance our work moving forward.

Thank you once again for your review!

Meta-review: This paper introduces a well-motivated data augmentation strategy for continuous time series data. A data modality often understudied by mainstream ML conferences, while being one of the most prevalent types of data sources in the real world. This work is relevant for whole fields focusing on modeling or predicting based on biosignals.

The paper is a great example of how new data augmentation strategies should be introduced, instead of just trying out a random strategy, the authors thoroughly motivate and validate their proposed method and then show how it leads to meaningful improvements on a wide range of tasks.

There are two remaining concerns of borderline reject reviewers which I will address below:

1. STAug baseline missing

This indeed seems to be missing from the author response pdf, please make sure to include these numbers in the final version of the manuscript. This oversight does not diminish the careful motivation and analysis of the method.

2. Request to compare to more contrastive learning methods. 

It is sufficient to focus on the most important well-established methods combined with the propose augmentation strategy. The empirical validation in this paper is extensive and I consider comparing to even more baselines out of scope for this work. The authors argued extensively to substantiate why comparing to more self-supervised learning methods will not benefit the current paper and I agree with them.
Decision: Accept (poster)