Title: Finding Order in Chaos: A Novel Data Augmentation Method for Time Series in Contrastive Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Contrastive learning

Time-series

Augmentation
Abstract: The success of contrastive learning is well known to be dependent on data augmentation.
Although the degree of data augmentations has been well controlled by utilizing pre-defined techniques in some domains like vision, time-series data augmentation is less explored and remains a challenging problem due to the complexity of the data generation mechanism, such as the intricate mechanism involved in the cardiovascular system.
Moreover, there is no widely recognized and general time-series augmentation method that can be applied across different tasks.
In this paper, we propose a novel data augmentation method for time-series tasks that aims to connect intra-class samples together, and thereby find order in the latent space.
Our method builds upon the well-known data augmentation technique of mixup by incorporating a novel approach that accounts for the non-stationary nature of time-series data.
Also, by controlling the degree of chaos created by data augmentation, our method leads to improved feature representations and performance on downstream tasks.
We evaluate our proposed method on three time-series tasks, including heart rate estimation, human activity recognition, and cardiovascular disease detection. 
Extensive experiments against the state-of-the-art methods show that the proposed method outperforms prior works on optimal data generation and known data augmentation techniques in three tasks, reflecting the effectiveness of the presented method. 
The source code is available at double-blind policy.

Finding Order in Chaos: A Novel Data Augmentation Method for Time Series in Contrastive Learning

Abstract

The success of contrastive learning is well known to be dependent on data augmentation. Although the degree of data augmentations has been well controlled by utilizing pre-defined techniques in some domains like vision, time-series data augmentation is less explored and remains a challenging problem due to the complexity of the data generation mechanism, such as the intricate mechanism involved in the cardiovascular system. Moreover, there is no widely recognized and general time-series augmentation method that can be applied across different tasks. In this paper, we propose a novel data augmentation method for quasi-periodic time-series tasks that aims to connect intra-class samples together, and thereby find order in the latent space. Our method builds upon the well-known mixup technique by incorporating a novel approach that accounts for the periodic nature of non-stationary time-series. Furthermore, by controlling the degree of chaos created by data augmentation, our method leads to improved feature representations and performance on downstream tasks. We evaluate our proposed method on three time-series tasks, including heart rate estimation, human activity recognition, and cardiovascular disease detection. Extensive experiments against state-of-the-art methods show that the proposed approach outperforms prior works on optimal data generation and known data augmentation techniques in the three tasks, reflecting the effectiveness of the presented method. Source code: https://github.com/eth-siplab/Finding_Order_in_Chaos .

1 Introduction

Self-supervised learning methods have gained significant attention as they enable the discovery of meaningful representations from raw data without explicit annotations. These self-supervised methods learn representations without labels by designing pretext tasks that transform the unsupervised representation learning problem into a supervised one such as predicting the rotation of images [1], or contexts [2, 3]. Among these methods, contrastive learning (CL), which learns to distinguish semantically similar examples over dissimilar ones, stands out as a powerful approach in selfsupervised learning across various domains including computer vision [4-6], speech recognition [710], and natural language processing [11-14].

The success of contrastive learning relies on the creation of similar and dissimilar examples, which is typically achieved through the use of data augmentations [15, 16]. Recently, it was shown that data augmentations have a role to create a ' chaos ' between different intra-class samples such that they become more alike. For example, two different cars become very similar when they are both cropped to the wheels. [17]. However, in time-series data, creating similar samples with augmentation techniques is more challenging due to the complexity of the dynamical data generation mechanisms [18]. Moreover, research on contrastive learning for time series has demonstrated the

absence of a unique data augmentation technique that consistently performs better than others in different tasks [19, 20]. Instead, the choice of augmentation depends on the contextual characteristics of the signal, such as perturbing the high-frequency content of a signal that carries characteristic information in low frequencies does not generate useful data samples that are helpful for contrastive learning to learn class invariant features [21].

Considering these limitations, in this work, we first propose a novel data augmentation method for time series data by performing a tailored mixup while considering the phase and amplitude information as two separate features. Then, we perform specific operations for both features to generate positive samples by controlling the mixup coefficients for each feature to prevent aggressive augmentation. Specifically, our method employs a technique that controls the mixup ratio for each randomly chosen pair based on their distance in the latent space which is acquired through the use of a variational autoencoder (V AE), whose objective is to learn disentangled representations of data without labels. To this end, subjecting to the distance constraint in the latent space, the mixup tries to connect semantically closer samples together more aggressively while preventing the excessive interpolation of dissimilar samples that are likely to belong to different classes. Therefore, the purpose of our proposed method for quasi-periodic time-series data augmentation is to find an order in 'chaos' between different samples such that they become more alike by interpolating them in a novel manner to prevent the loss of information. We summarize our contributions as follows:

We propose a novel mixup method for non-stationary quasi-periodic time-series data by considering phase and magnitude as two separate features to generate samples that enhance intra-class similarity and help contrastive learning to learn class-separated representations.

We present a novel approach for sampling mixup coefficients for each pair based on their similarity in the latent space, which is constructed without supervision while learning disentangled representations, to prevent aggressive augmentation between samples.

We show that the tailored mixup with coefficient sampling consistently improves the performance of contrastive learning in three time-series tasks compared to prior mixup techniques and proposed augmentation methods that generate optimal/hard positives or negatives.

2 Preliminaries

2.1 Notation

We use lowercase symbols ( x ) to denote scalar quantities, bold lowercase symbols ( x ) for vector values, and capital letters ( X ) for random variables. Functions with a parametric family of mappings are represented as f θ ( . ) where θ is the parameters. The discrete Fourier transformation of a realvalued time series sample is denoted as F ( . ) , yielding a complex variable as X k where X ∈ C and k ∈ [0 , f s / 2] is the frequency with the maximum value of Nyquist rate. The amplitude and phase of the F ( x ) are represented as A ( x ) and P ( x ) . The real and imaginary parts of a complex variable are shown as Re( . ) and Im( . ) . The detailed calculations for operations are given in the Appendix A.1.

2.2 Setup

Wefollow the common CL setup as follows. Given a dataset D = { ( x i ) } K i =1 where each x i consists of real-valued sequences with length L and C channels. The objective is to train a learner f θ which seeks to learn an invariant representation such that when it is fine-tuned on a dataset D l = { ( x i , y i ) } i M =1 with M ≪ K and y i ∈ { 1 , . . . , N } , it can separate samples from different classes.

2.3 Motivation

As stated by prior works, mixup-based methods have poor performance in domains where data has a non-fixed topology, such as trees, graphs, and languages [22, 23]. Here, we demonstrate how we derived our proposed method by revealing the limitations of mixup for time series theoretically while considering the temporal dependencies and non-stationarities.

Assumption 2.1 (SNR Matters) . There exist one or multiple bandlimited frequency ranges of interest f ∗ , where the information that average raw time-series data conveys about the labels (i.e., I ( y ; x ) ) is directly proportional to normalized signal power in that frequency range as in Equation 1.

Assumption 2.1 states that the information from a time series depends on its signal-to-noise ratio (SNR). Prior works showed that specific frequency bands hold inherent information about the characteristics of time series, which helps classification [21, 24].

Assumption 2.2. The true underlying generative process f ( . ) , for a given data distribution D = { x k } K k =1 , is quasiperiodic, i.e., f ( x + τ ) = g ( x , f ( x )) , where τ can be either fixed or varied.

Assumption 2.2 posits that the observed data samples from the distribution D are generated by a quasiperiodic function. This is a minimal assumption since the quasiperiodicity is the relaxed version of the periodic functions. In simpler terms, quasiperiodicity can be described as the observed signals exhibiting periodicity on a small scale, while being unpredictable on a larger scale. And, several prior works showed that the data generation mechanism of time-series data for several applications in the real world are quasiperiodic [25-30]. Therefore, Assumption 2.2 is realistic.

Proposition 2.3 (Destructive Mixup) . If Assumptions 2.1 and 2.2 hold, there exist λ ∼ Beta ( α, α ) or λ ∼ U ( β, 1 . 0) with high values of β such that when linear mixup techniques are utilized, the lower bound of the mutual information for the augmented sample distribution decreases to zero.

Proofs can be found in Appendix A. This proposition indicates that although the augmented samples are primarily derived from anchor samples ( x ) with high ratios, the resulting instances may not contain any task-relevant information. In other words, the augmentation process can potentially discard the whole task-specific information. This destructive behavior of mixup for quasi-periodic data can be attributed to the interference phenomenon in which two waves interact to form the resultant wave of the lower or higher amplitude according to the phase difference as shown in Proposition 2.3.

3 Method

We introduce a novel approach to overcome the limitations of mixup by treating the magnitude and phase of sinusoidal signals as two distinct features with separate behaviors. Subsequently, we apply tailored mixup operations to each feature, considering their specific characteristics and effects.

We perform the linear mixup for the magnitude of each sinusoidal. However, for the phase, we take a different approach and bring the phase components of the two coherent signals together by adding a small value to the anchor's phase in the direction of the other sample. The mixup operation performs the linear interpolation of features [31], however, interpolation of two complex variables can result in a complex variable whose phase and magnitude are completely different/far away from those two, i.e., mixup can be destructive extrapolation rather than the interpolation of features. Therefore, we mix the phase of two sinusoidal as follows. We start by calculating the shortest phase difference between the two samples, denoted as ∆Θ , as described in Equation 3 1 .

The sign of the calculated phase difference provides information about the relative phase location of the other sample, in either a clockwise or counterclockwise direction in the phasor diagram. And, the absolute value of it represents the shortest angular difference between two samples in radians. Based

on the calculated phase difference between two samples, we perform mixup operation to generate diverse positive samples as in Equation 4 such that phase and magnitude of augmented instances are interpolated properly according to the anchor sample x , without causing any destructive interference.

The proposed method which mixes the magnitude and phase of each frequency component with tailored operations, not only prevents destructive interference between time series, resulting in an increase in the lower bound of mutual information (as shown in Theorem 3.1), but also generates diverse augmented instances with the same two samples by using two different mixing coefficients.

a)

b)

Theorem 3.1 (Guarantees for Mixing) . Under assumptions 2.1 and 2.2, given any λ ∈ (0 , 1] , the mutual information for the augmented instance lower bounded by the sampled λ and anchor x .

We provide an intuitive demonstration in Figure 1, along with a detailed mathematical proof presented in Appendix A. Our approach also offers increased flexibility in selecting the mixing coefficients of phase and magnitude, based on their sensitivities to the mixing process as well as the augmentation degree for each randomly chosen pair. Since the degree of augmentations has crucial importance for contrastive learning, there can be cases where augmentations are either too weak (intra-class features cannot be clustered together) or too strong (inter-class features can also collapse to the same point) and lead to sub-optimal results [17]. To mitigate this issue and find an order for augmentation degree, we search pairs of samples that are semantically closer, meaning they are more likely to belong to the same class. We then perform the proposed mixup more aggressively on these pairs, creating more closer and diverse samples while decreasing the augmentation strength for less similar pairs. To find similar samples without labels, we train a completely unsupervised β -VAE [32] that maps data points to a latent space such that two random samples are semantically similar if they are close in the latent as shown in Proposition 3.2.

Proposition 3.2 (Consistency in Latent Space [33]) . Given a well-trained unconditional VAE with the encoder E ( . ) that produces distribution p E ( z | x ) , the decoder D ( . ) that produces distribution q D ( x | z ) while the prior for z is p ( z ) , let z 1 and z 2 be two latent vectors of two different real samples x 1 and x 2 , i.e., E ( x 1 ) = z 1 and E ( x 2 ) = z 2 . if the distance d ( z 1 , z 2 ) ≤ δ , then D ( z 1 ) and D ( z 2 ) will have a similar semantic label as in Equation 6.

where ϵ stands for tolerable semantic difference, δ is the maximum distance to maintain semantic consistency, and d ( . ) is a distance measure such as cosine similarity between two vectors.

The above proposition with Theorem 3.1 motivates us to perform augmentation aggressively if two randomly chosen samples are semantically closer. Therefore, we sample the mixup coefficient for both phase and magnitude from a uniform distribution λ A , λ P ∼ U ( β, 1 . 0) with low values of β if the distance between the latent vectors is below a threshold, otherwise, they are drawn from a truncated normal distribution λ A , λ P ∼ N t ( µ, σ, 1 . 0) with a high mean and low standard deviation.

4 Experiments

We conduct experiments on the proposed approach and compare it with other mixup methods or optimal/hard positive sample generation in the contrastive learning setup. During our experiments, we use SimCLR [15] framework without specialized architectures or a memory bank for all baselines to have a fair comparison. Results with other CL frameworks can be found in Appendix E. Complete training details and hyper-parameter settings for datasets and baselines are provided in Appendix D.

4.1 Datasets

We performed extensive experiments on eight datasets from three tasks that include activity recognition from inertial measurements (IMUs), heart rate prediction from photoplethysmography (PPG), and cardiovascular disease classification from electrocardiogram (ECG). We provided short descriptions of each dataset below, and further detailed information with metrics can be found in Appendix B.

Activity recognition We used UCIHAR [34], HHAR [35], and USC [36] for activity recognition. During the evaluation, we assess the cross-person generalization capability of the contrastive models, i.e., the model is evaluated on a previously unseen target domain. We follow the settings in GILE [37] to treat each person as a single domain while the fine-tuning dataset is much smaller than the unsupervised one.

Heart rate prediction We used the IEEE Signal Processing Cup in 2015 (IEEE SPC) [38], and DaLia [39] for PPG-based heart rate prediction. The SPC provides two datasets, one smaller with lesser artifacts (referred to as SPC12) [38] and a bigger dataset with more participants including heavy motions (referred to as SPC22). In line with previous studies, we adopted the leave-one-session-out (LOSO) cross-validation, which involves evaluating methods on subjects or sessions that were not used for pre-training and fine-tuning.

Cardiovascular disease (CVD) classification We conducted our experiments on China Physiological Signal Challenge 2018 (CPSC2018) [40] and Chapman University, Shaoxing People's Hospital (Chapman) ECG dataset [41]. We selected the same four specific leads as in [42] while treating each dataset as a single domain with a small portion of the remainder dataset used for fine-tuning the pre-trained model. We split the dataset for fine-tuning and testing based on patients (each patient's recordings appear in only one set).

4.2 Baselines

Comparison with prior mixup techniques We evaluate the effectiveness of our proposed mixup by comparing it with other commonly used mixup methods, including Linear-Mixup [31], BinaryMixup [43], Geometric-Mixup [22], Cut-Mix [44], Amplitude-Mix [45] and Spec-Mix [46]. When we compare the performance of mixup techniques, we follow the same framework with [47] where the samples of mixture operation only happen in current batch samples. And, the mixup samples are paired with anchors, i.e., without applying mixup second times, for contrastive pre-training.

Comparison with methods for optimal sample generation We evaluate the performance of our proposed method by comparing it with other data generation methods and baselines in contrastive learning setup while considering previously known augmentation techniques. Traditional data augmentations for time-series [19], such as resampling, flipping, adding noise, etc. InfoMin which leverages an adversarial training strategy to decrease the mutual information between samples

while maximizing the NCE loss [48]. NNCLR [49], which uses nearest neighbors in the learned representation space as the positive samples. Positive feature extrapolation [50], which creates hard positives through feature extrapolation. GenRep which uses the latent space of a generative model to generate 'views' of the same semantic content by sampling nearby latent vectors [51]. Aug. Bank [21], which proposes an augmentation bank that manipulates frequency components of a sample with a limited budget. STAug [52], which combines spectral and time augmentation for generating samples using the empirical mode decomposition and linear mixup. DACL [22], which creates positive samples by mixing hidden representations. IDAA [53], which is an adversarial method by modifying the data to be hard positives without distorting the key information about their original identities using a V AE. More implementation details for each baseline are given in Appendix C.

4.3 Implementation

Weuse a combination of convolutional with LSTM-based network, which shows superior performance in many time-series tasks [19, 54, 55], as backbones for the encoder f θ ( . ) where the projector is two fully-connected layers. We use InfoNCE as the loss, which is optimized using Adam with a learning rate of 0 . 003 . We train with a batch size of 256 for 120 epochs and decay the learning rate using the cosine decay schedule. After pre-training, we train a single linear layer classifier on features extracted from the frozen pre-trained network, i.e., linear evaluation, with the same hyperparameters. Reported results are mean and standard deviation values across three independent runs with different seeds on the same split. More details about the implementation, architectures, and hyperparameters with the trained V AEs are given in Appendix D.

5 Results and Discussion

Tables 1, 2, and 3 present the results of our proposed approach compared to state-of-the-art methods for optimal/hard positive sample generation in contrastive learning setup across the three tasks in eight datasets. Additionally, Figure 2 compares our approach with prior mixup methods (e.g., linear mixup, cutmix) without applying any other additional augmentation techniques. Overall, our proposed method has demonstrated superior performance compared to other methods in seven datasets, with the second-best performance in the remaining dataset, and a minor performance gap.

From these tables, we can see that our proposed method significantly outperforms DACL, which suggests creating a positive sample by mixing fixed hidden representations in an intermediate layer [22], by a large margin (up to 20.8% with a 10.1% on average in activity recognition). This suggests that when the representations are not yet linearly separable at the beginning of the contrastive training process, the interpolated representations using mixup may be dissimilar to the actual interpolated samples and may not capture their underlying features. One interesting result from our experiments is that IDAA [53] exhibits comparable performance to our method in some datasets, and even slightly outperforms our approach in the HHAR dataset for activity recognition. Despite using distinct methods to generate positive instances, i.e., adversarial and mixup, our approach and IDAA algorithm

* The entire dataset, excluding the test, is utilized with labels, while in DCL, the labeled data size matches that of the CL

share similarities in approaches for positive instance generation in CL setup. The IDAA algorithm aims to create hard positive samples that lie near class boundaries without changing the identity of the sample, while our method interpolates two samples to produce a positive instance that is similar to the original sample while adding noise to the phase and amplitude in the direction of a randomly chosen sample. In other words, both approaches try to keep the sample identity intact by taking special precautions while generating new positive instances, which may explain their similar performance in our experiments.

In contrast, approaches that do not prioritize preserving sample identity while generating samples or hidden representations often demonstrate suboptimal performance on average while exhibiting increased variability across tasks.

Examples of such methods include PosET [50], which generates hard positive samples to improve contrastive learning by extrapolating features, STAug [52], which uses empirical mode decomposition with linear mixup technique together, and InfoMin [48], which tries to minimize mutual information between two instances in an adversarial manner. The performance comparison of prior mixup techniques and our proposed one is shown in Figure 2. On average, our proposed method outperforms all other mixup techniques while reducing the variance across tasks. What is interesting about this figure is that while the linear [31] and amplitude mixup [45] reach our method in some datasets for activity recognition , the performance of the linear mixup decreases heavily for the other two tasks whereas the amplitude mixup gives reasonable per-

formance. This empirical outcome supports our initial theorem about the destructive effect of mixup, which suggests linear mixup or other derivatives can discard the whole task-specific information in the generated positive sample for quasi-periodic signals even though the mixing coefficient is sampled from a distribution such that the generated samples are much closer to the anchor.

5.1 Ablation Studies

Here, we present a comprehensive examination of our proposed method and the effect of its components on the performance. Mainly, we investigate the effect of the proposed mixup by applying the instance selection algorithm to the linear mixup (w/o Prop. Mixup). Then, we perform our proposed mixup with the constant λ A and λ P coefficients without investigating latent space distances between pairs (w/o Aug. Degree). Tables 4, 5 and 6 summarize the results. The second row in the tables shows the performance when the proposed mixup method is not applied while choosing mixup coefficients according to the distances in the latent space for the linear mixup. The last row illustrates

10

15

20

25

30

35

40

IEEE SPC12

IEEE SPC22

Ours

Linear

Binary

Cut

Spec

15

35

55

75

95

UCIHAR

HHAR

USC

Ours

Linear

Binary

Cut

Spec

Geo

65

70

75

80

85

90

95

100

CPSCP 2018

Chapman

Ours

Linear

Binary

Cut

Spec

a)

b)

c)

Accuracy in  %

Higher is better.

Mean Absolute Error

Lower is better.

AUC in %

Higher is better.

the performance change resulting from randomly sampling mixup coefficients without considering any relationship between the selected pair while applying tailored mixup for phase and magnitude.

The results obtained from the ablation study support the previous claims and outcomes. For example, when the linear mixup is applied instead of the proposed mixup technique for heart rate prediction (Table 5, w/o Prop. Mixup), the performance decrease is significant compared to the case when coefficients are sampled without considering the distance in the latent space (Table 5, w/o Aug. Degree). This observation indicates that as the periodicity in data increases, linear mixup can lead to significant destructive interferences, whereas our method effectively prevents such issues.

While our mixup technique consistently enhances performance across datasets, we observe a decline when the mixing coefficients are sampled based on the distance in the latent space for two datasets. Also, the performance increase gained by sampling coefficients based on distance is relatively low compared to the proposed mixup. Several factors can explain this observation. First, the V AE might not be

well trained due to the limited size of data in each class, i.e., the assumption in Proposition 3.2 does not hold. This can lead to inconsistencies in the semantic similarity of the latent space such that two close samples in the latent space might have different labels. Second, if the number of classes increases for a downstream task, the probability of sampling intra-class samples in a batch will decrease, leading to a lack of performance improvement. Therefore, in future investigations, it might be beneficial to use a different distance metric for quasi-periodic time-series data such that it can scale with the number of classes while considering the lack of big datasets.

More ablation studies about the sensitivity of mixing coefficients and performance in different self-supervised learning frameworks, like BYOL [59] can be found in Appendix E.1 and E.2. And, investigations regarding whether we still need known data augmentations are given in Appendix E.3. Examples that visually demonstrate the negative effects of linear mixup and our proposed mixup

technique to prevent this problem can be found in Appendix F. Comparative results regarding the performance of the tailored mixup in the supervised learning paradigm are given in Appendix G.

6 Related Work

The goal of contrastive learning is to contrast positive with negative pairs [60]. In other words, the embedding space is governed by two forces, the attraction of positive pairs and the repellence of negatives, actualized through the contrastive loss [61]. Since label information is unavailable during the training, positive pairs are generated using augmentation techniques on a single sample, while negative pairs are randomly sampled from the entire dataset. Therefore, the choice or generation of positive and negative samples plays a pivotal role in the success of contrastive learning [62-65] and both approaches, generation/selection of positive/negative pairs, have been investigated thoroughly in the literature [66-71], we limit our discussion about prior works related to data augmentation techniques that create optimal or hard samples without labels.

Adversarial based approaches A growing body of literature has investigated generating samples by using adversarial training for both positives and negatives [48, 72, 73]. A seminal work about the importance of augmentations in CL, InfoMin, presented an adversarial training strategy where players try to minimize and maximize the mutual information using the NCE loss [48]. CLAE, one of the first works that leveraged the adversarial approach, shows that adversarial training generates challenging positive and hard negative pairs [72]. Another recent study proposed an adversarial approach that generates hard samples while retaining the original sample identity by leveraging the identity-disentangled feature of V AEs [53]. However, adversarial augmentations may change the original sample identity due to excessive perturbations and it is infeasible to tune the attack strength for every sample to preserve the identity. In other words, these approaches do not consider the sample-specific features and use a constant perturbation coefficient for all samples whereas our proposed method considers the similarity between pairs and tunes the mixing coefficients accordingly.

Mixup based approaches Mixup-based methods have been recently explored in contrastive learning [22, 71, 47, 74, 75]. According to a recent theoretical work [22], mixup has implicit data-adaptive regularization effects that promote generalization better than adding Gaussian noise, which is a commonly used augmentation strategy in both time-series and vision data [76-78]. Although, mixupbased approaches have shown success in different problems [79, 80], such as domain adaptation, creating samples using mixup in the input space is infeasible in domains where data has a non-fixed topology, such as sequences, trees, and graphs [22]. Therefore, recent works suggest mixing hidden representations of samples, similar to Manifold Mixup [81]. However, this method claims that mixing fixed-length hidden representation via an intermediate layer " z = α z +(1 -α ) ˜ z " can be interpreted as adding noise to a given sample in the direction of another. However, it is an overly optimistic claim because early during training, where in most cases there is usually no linear separability among the representations, this synthesis may result in hidden representations that are completely different and far away from the samples [71, 82]. Therefore, in this work, we take a different approach and modify the mixup method considering its limitations for quasi-periodic non-stationary time-series data. Also, unlike most existing methods that aim to generate hard samples-samples that are close to class boundaries-using adversarial approaches [53, 48, 72] or feature extrapolation [50, 71], our method seeks to connect semantically closer samples together using interpolation in a tailored way.

7 Conclusion

In this paper, we first demonstrate the destructive effect of linear mixup for quasi-periodic timeseries data, then introduce a novel tailored mixup method to generate positive samples for the contrastive learning formulation while preventing this destructive effect and interpolating the samples appropriately. Theoretically, we show that our proposed method guarantees the interpolation of pairs without causing any loss of information while generating a diverse set of samples. Empirically, our method outperforms the prior approaches in three real-world tasks. By conducting experiments on contrastive and supervised learning settings, we show that our approach is agnostic to the choice of learning paradigm. Thus, it holds the potential for utilization in generating augmented data for different learning paradigms as well. We believe that the method proposed in this paper has the potential to significantly improve learning solutions for a diverse range of time series tasks.

References

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Carl Doersch, Abhinav Kumar Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference on Computer Vision (ICCV) , pages 1422-1430, 2015.

Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 2536-2544. IEEE Computer Society, 2016.

Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , 2:1735-1742, 2006.

Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. Wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

shuang ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Contrastive learning of global and local video representations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 7025-7040. Curran Associates, Inc., 2021.

Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, and Xuedong Huang. Unispeech: Unified speech representation learning with labeled and unlabeled data. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 10937-10947. PMLR, 18-24 Jul 2021.

Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-bert: Combining contrastive learning and masked language modeling for selfsupervised speech pre-training. 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pages 244-250, 2021.

Hao Fu, Shaojun Zhou, Qihong Yang, Junjie Tang, Guiquan Liu, Kaikui Liu, and Xiaolong Li. Lrc-bert: Latent-representation contrastive knowledge distillation for natural language understanding. Proceedings of the AAAI Conference on Artificial Intelligence , 35(14):1283012838, May 2021.

Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li, Ji Zhang, Haiqing Chen, and Yin Zhang. Kace: Generating knowledge aware contrastive explanations for natural language inference. In Annual Meeting of the Association for Computational Linguistics , 2021.

X. Y. Shen, Ying Sun, Yao zhong Zhang, and Mani Najmabadi. Semi-supervised intent discovery with contrastive learning. Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI , 2021.

Yang Liu and Maosong Sun. Contrastive unsupervised word alignment with non-local features. Proceedings of the AAAI Conference on Artificial Intelligence , 29(1), Feb. 2015.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, C. Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In International Joint Conference on Artificial Intelligence , 2021.

Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022.

Mike West, Raquel Prado, and Andrew D. Krystal. Evaluation and comparison of eeg traces: Latent structure in nonstationary time series. Journal of the American Statistical Association , 94(446):375-387, 1999.

Hangwei Qian, Tian Tian, and Chunyan Miao. What makes good contrastive learning on small-scale wearable-based tasks? In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '22, page 3761-3771, New York, NY, USA, 2022. Association for Computing Machinery.

Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 4653-4660. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Survey Track.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In Proceedings of Neural Information Processing Systems, NeurIPS , 2022.

Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards domainagnostic contrastive learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 10530-10541. PMLR, 18-24 Jul 2021.

Soyoung Yoon, Gyuwan Kim, and Kyumin Park. Ssmix: Saliency-based span mixup for text classification. ArXiv , abs/2106.08062, 2021.

Martin Ullrich, Arne Küderle, Julius Hannink, Silvia Del Din, Heiko Gaßner, Franz Marxreiter, Jochen Klucken, Bjoern M. Eskofier, and Felix Kluge. Detection of gait from continuous inertial sensor data using harmonic frequencies. IEEE Journal of Biomedical and Health Informatics , 24(7):1869-1878, 2020.

Hong Luo, Deye Yang, Andrew Barszczyk, Naresh Vempala, Jing Wei, Si Jia Wu, Paul Pu Zheng, Genyue Fu, Kang Lee, and Zhong-Ping Feng. Smartphone-based blood pressure measurement using transdermal optical imaging technology. Circulation: Cardiovascular Imaging , 12(8):e008857, 2019.

Bryan P. Yan, William H. S. Lai, Christy K. Y. Chan, Stephen Chun-Hin Chan, Lok-Hei Chan, Ka-Ming Lam, Ho-Wang Lau, Chak-Ming Ng, Lok-Yin Tai, Kin-Wai Yip, Olivia T. L. To, Ben Freedman, Yukkee C. Poh, and Ming-Zher Poh. Contact-free screening of atrial fibrillation by a smartphone using facial pulsatile photoplethysmographic signals. Journal of the American Heart Association , 7(8):e008585, 2018.

Saman Noorzadeh, Mohammad Niknazar, Bertrand Rivet, Julie Fontecave-Jallon, Pierre-Yves Guméry, and Christian Jutten. Modeling quasi-periodic signals by a non-parametric model: Application on fetal ecg extraction. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society , pages 1889-1892, 2014.

A Koulali and P J Clarke. Modelling quasi-periodic signals in geodetic time-series using Gaussian processes. Geophysical Journal International , 226(3):1705-1714, 04 2021.

M. Lemay, L. Dang, and J.M. Vesin. Quasi-periodic atrial activity components in the ecg used to discriminate between paroxysmal and chronic atrial fibrillation. In 2008 Computers in Cardiology , pages 821-824, 2008.

A Garfinkel, P S Chen, D O Walter, H S Karagueuzian, B Kogan, S J Evans, M Karpoukhin, C Hwang, T Uchida, M Gotoh, O Nwasokwa, P Sager, and J N Weiss. Quasiperiodicity and chaos in cardiac fibrillation. The Journal of Clinical Investigation , 99(2):305-314, 1 1997.

Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018 , 2018.

Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations , 2016.

Yinqi Li, Hong Chang, Bingpeng MA, Shiguang Shan, and Xilin Chen. Optimal positive generation via latent transformation for contrastive learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 18327-18342. Curran Associates, Inc., 2022.

D. Anguita, Alessandro Ghio, L. Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In International Workshop on Ambient Assisted Living and Home Care , 2012.

Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. SenSys '15, page 127-140, New York, NY, USA, 2015. Association for Computing Machinery.

Mi Zhang and Alexander A. Sawchuk. Usc-had: A daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing , UbiComp '12, page 1036-1043, New York, NY, USA, 2012. Association for Computing Machinery.

Hangwei Qian, Sinno Jialin Pan, and Chunyan Miao. Latent independent excitation for generalizable sensor-based cross-person activity recognition. Proceedings of the AAAI Conference on Artificial Intelligence , 35(13):11921-11929, May 2021.

Zhilin Zhang, Zhouyue Pi, and Benyuan Liu. Troika: A general framework for heart rate monitoring using wrist-type photoplethysmographic signals during intensive physical exercise. IEEE Transactions on Biomedical Engineering , 62(2):522-531, 2015.

Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. Deep ppg: Large-scale heart rate estimation with convolutional neural networks. Sensors , 19(14), 2019.

Eddie Y. K. Ng, Feifei Liu, Chengyu Liu, Lina Zhao, X. Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, and Jianqing Li. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics , 2018.

Jianwei Zheng, Jianming Zhang, Sidy Danioko, Hai Yao, Hangyuan Guo, and Cyril Rakovski. A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients. Scientific Data , 7(1):48, February 2020.

Erick A Perez Alday, Annie Gu, Amit J Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifei Liu, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, Qiao Li, Ashish Sharma, Gari D Clifford, and Matthew A Reyna. Classification of 12-lead ecgs: the physionet/computing in cardiology challenge 2020. Physiological Measurement , 41(12):124003, dec 2020.

Christopher Beckham, Sina Honari, Vikas Verma, Alex M Lamb, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Chris Pal. On adversarial mixup resynthesis. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6022-6031, 2019.

Q. Xu, R. Zhang, Y. Zhang, Y. Wang, and Q. Tian. A fourier-based framework for domain generalization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14378-14387, Los Alamitos, CA, USA, jun 2021. IEEE Computer Society.

Gwantae Kim, David K. Han, and Hanseok Ko. Specmix : A mixed sample data augmentation method for training withtime-frequency domain features. In Interspeech , 2021.

Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. 2022.

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 9568-9577, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society.

Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10286-10295, 2021.

Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Xiyuan Zhang, Ranak Roy Chowdhury, Jingbo Shang, Rajesh Gupta, and Dezhi Hong. Towards diverse and coherent augmentation for time-series forecasting. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023.

Kaiwen Yang, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Identity-disentangled adversarial augmentation for self-supervised learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 25364-25381. PMLR, 17-23 Jul 2022.

Dwaipayan Biswas, Luke Everson, Muqing Liu, Madhuri Panwar, Bram-Ernst Verhoef, Shrishail Patki, Chris H. Kim, Amit Acharyya, Chris Van Hoof, Mario Konijnenburg, and Nick Van Helleputte. Cornet: Deep learning framework for ppg-based heart rate estimation and biometric identification in ambulant environment. IEEE Transactions on Biomedical Circuits and Systems , 13(2):282-291, 2019.

Akara Supratak, Hao Dong, Chao Wu, and Yike Guo. Deepsleepnet: a model for automatic sleep stage scoring based on raw single-channel eeg. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 25(11):1998-2008, Nov 2017.

Garrett Wilson, Janardhan Rao Doppa, and Diane J. Cook. Multi-source deep domain adaptation with weak supervision for time-series sensor data. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &; Data Mining , KDD '20, page 1768-1778, New York, NY, USA, 2020. Association for Computing Machinery.

Dani Kiyasseh, Tingting Zhu, and David A. Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning , 2020.

Crystal T. Wei, Ming-En Hsieh, Chien-Liang Liu, and Vincent S. Tseng. Contrastive heartbeats: Contrastive learning for self-supervised ecg representation and phenotyping. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1126-1130, 2022.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Tri Huynh, Simon Kornblith, Matthew R. Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2785-2795, January 2022.

Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning , 2019.

Anastasiia Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Hong Xuan, Abby Stylianou, Xiaotong Liu, and Robert Pless. Hard negative examples are hard, but useful. In Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV , page 126-142, Berlin, Heidelberg, 2020. Springer-Verlag.

Songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 27356-27368. Curran Associates, Inc., 2021.

Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance classification for unsupervised visual feature learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 87658775. Curran Associates, Inc., 2020.

Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondˇ rej Chum. Mining on manifolds: Metric learning without labels. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7642-7651, 2018.

Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual information in contrastive learning for visual representations, 2020.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 815-823, 2015.

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21798-21809. Curran Associates, Inc., 2020.

Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 17081-17093. Curran Associates, Inc., 2020.

Xiao Wang, Yuhang Huang, Dan Zeng, and Guo-Jun Qi. Caco: Both positive and negative samples are directly learnable via cooperative-adversarial contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-12, 2023.

Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se-Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint arXiv:2010.06300 , 2020.

Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR , 2021.

Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, and Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. In The Eleventh International Conference on Learning Representations , 2023.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 2352-2359, 2021.

Pengxiang Shi, Wenwen Ye, and Zheng Qin. Self-supervised pre-training for time series classification. In 2021 International Joint Conference on Neural Networks (IJCNN) , 2021.

Ronghang Zhu, Ronghang Zhu, Xiang Yu, and Sheng Li. Progressive mix-up for few-shot supervised multi-source domain transfer. In ICLR , 2023.

JangHyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. In International Conference on Learning Representations , 2021.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David LopezPaz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 6438-6447, Long Beach, California, USA, 09-15 Jun 2019. PMLR.

Tsz-Him Cheung and Dit-Yan Yeung. {MODALS}: Modality-agnostic automated data augmentation in the latent space. In International Conference on Learning Representations , 2021.

Leandro Giacomini Rocha, Dwaipayan Biswas, Bram-Ernst Verhoef, Sergio Bampi, Chris Van Hoof, Mario Konijnenburg, Marian Verhelst, and Nick Van Helleputte. Binary cornet: Accelerator for hr estimation from wrist-ppg. IEEE Transactions on Biomedical Circuits and Systems , 14(4):715-726, 2020.

Sayeed Shafayet Chowdhury, Rakib Hyder, Md. Samzid Bin Hafiz, and Mohammad Ariful Haque. Real-time robust heart rate estimation from wrist-type ppg signals using multiple reference adaptive noise cancellation. IEEE Journal of Biomedical and Health Informatics , 22(2):450-459, 2018.

Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Appendix

A Proof

In this section, we present complete proofs of our theoretical study, starting with notations.

A.1 Notations

Fourier transform of a real-valued sample with a finite duration is obtained as in Equation 7.

The amplitude and phase for each frequency are calculated from the Fourier transform as follows.

where arctan is a 2-argument arctangent which is the angle measure in radians. The phasor, as in Figure 1, of a sample is represented as in Equation 9.

A.2 Proof for Proposition 2.3

Proposition A.1 (Destructive Mixup) . If Assumptions 2.1 and 2.2 hold, there exist λ ∼ Beta ( α, α ) or λ ∼ U ( β, 1 . 0) with high values of β such that when linear mixup techniques are utilized, the lower bound of the mutual information for the augmented sample distribution decreases to zero.

Proof.

From the linearity of Fourier transformation and ignoring k in X k for the sake of easiness.

Let ˜ X = e -jω ϕ k X ω k , where ϕ k and ω k are random phase and frequency modulators for each frequency, sampled from distributions ϕ k ∼ Φ , ω k ∼ Ω .

From the quasi-periodicity, assume that the frequency ranges of interest ( f ∗ , i.e., k ∗ ) are overlapped for both samples while the sampled random modulators have the following relationship.

where θ is an odd multiple of π . Equation 17 can be simplified as follows.

From Assumption 2.1,

We use Euler's formula to expand Equation 16 to 17. While we use the frequency bins ( k ) and frequency values in Hz ( f ) interchangeably for Equations 21 and 22.

Although the above proof is to show the resulting instances may not contain any task-relevant information, it can also be demonstrated that the augmentation process can potentially discard the partial task-specific information (not whole) if ϕ k and ω k are close to indicated relationships.

A.3 Proof for Theorem 3.1

Theorem A.2 (Guarantees for Mixing) . Under assumptions 2.1 and 2.2, given any λ ∈ (0 , 1] , the mutual information for the augmented instance lower bounded by the sampled λ and anchor x .

Proof.

Using the ∫ ∞ -∞ S x + ( f ) = ∫ ∞ ∞ S ˜ x ( f ) (i.e., both samples are normalized to have the same power) and Assumption 2.1,

Proof is completed with Equation 35 by combining equations 33 and 34.

Although this proof ignores the effect of phase mixing on the mutual information with the assumption 2.1, it is known that phase components carry semantically important features [45]. Therefore, it is necessary to note that the objective of this proof is to demonstrate that by applying mixup separately to the phase and amplitude components, we can avoid destructive interference.

B Datasets

In this section, we give details about the datasets that are used during our experiments.

B.1 Human Activity Recognition

UCIHAR Human activity recognition using smartphones dataset (UCIHAR) [34] is collected by 30 subjects within an age range of 16 to 48 performing six daily living activities with a waist-mounted smartphone. Six activities include walking, sitting, lying, standing, walking upstairs, and walking downstairs. Data is captured by 3-axial linear acceleration and 3-axial angular velocity at a constant rate of 50 Hz. We used the pre-processing technique the same as in [37, 19] such that the input contains nine channels with 128 features (it is sampled in sliding window of 2.56 seconds and 50% overlap, resulting in 128 features for each window). Windows are normalized to zero mean and unit standard deviation before feeding to models. Also, we follow the same experimental setup with prior works as follows. The experiments are conducted with a leave-one-domain-out strategy, where one of the domains is chosen to be the unseen target [19]. The contrastive pre-training is conducted with all subjects without any label information except the target one. Training of the linear layer, which is added to the frozen trained encoder, is only performed with the first five subjects of UCIHAR after excluding the target subject. In other words, if the target subject is 0, the subjects from 1 to 29 are used to train the encoder without any label information. Then, subjects from 1 to 4 are used to train the linear layer. And, evaluation is performed for subject 0. This is performed for the first five subjects with three random seeds and the mean value is reported.

HHAR Heterogeneity Dataset for Human Activity Recognition (HHAR) is collected by nine subjects within an age range of 25 to 30 performing six daily living activities with eight different smartphones-Although HHAR includes data from smartwatches as well, we use data from smartphones-that were kept in a tight pouch and carried by the users around their waists [35]. Subjects then perform 6 activities: 'bike', 'sit', 'stairs down', 'stairs up', 'stand', and 'walk'. Due to variant sampling frequencies of smart devices used in HHAR dataset, we downsample the readings to 50 Hz and apply 100 (two seconds) and 50 as sliding window length with step size, the windows are normalized to zero mean with unit standard deviation. We used the first four subjects (i.e., a, b, c, d) as source domains.

USC USC human activity dataset (USC-HAD) is composed of 14 subjects (7 male, 7 female, aged 21 to 49 with a mean of 30.1) executing 12 activities with a sensor on the front right hip. The data dimension is six (3-axis accelerometer, 3-axis gyroscope) and the sample rate is 100 Hz. 12 activities include walking forward, walking left, walking right, walking upstairs, walking downstairs, running forward, jumping up, sitting, standing, sleeping, elevator up, and elevator down. We used the pre-processing technique with a smaller window size such that the input contains six channels with 100 features (it is sampled in a sliding window of 1 second and 50% overlap, resulting in 100 features for each window). The same normalization is also applied to windows before feeding to models. We used the same setup with UCIHAR while source subjects are chosen as the last four this time.

B.2 Heart Rate Prediction

IEEE SPC This competition provided a training dataset of 12 subjects (SPC12) and a test dataset of 10 subjects [39]. The IEEE SPC dataset overall has 22 recordings of 22 subjects, ages ranging from 18 to 58 performing three different activities [83]. Each recording has sampled data from three accelerometer signals and two PPG signals along with the sampled ECG data and the sampling frequency is 125 Hz. All these recordings were recorded from the wearable device placed on the wrist of each individual. All recordings were captured with a 2-channel pulse oximeter with green LEDs, a tri-axial accelerometer, and a chest ECG for the ground-truth HR estimation. During our experiments, we used PPG channels. We choose the first five subjects of SPC12 as source domains similar to activity recognition setup while the last six subjects of SPC22 are used for source domains to prevent overlapping subjects with SPC12.

Dalia PPG dataset for motion compensation and heart rate estimation in Daily Life Activities (DaLia) was recorded from 15 subjects (8 females, 7 males, mean age of 30 . 6 ), where each recording was approximately two hours long. PPG signals were recorded while subjects went through different

daily life activities, for instance sitting, walking, driving, cycling, working, and so on. PPG signals were recorded at a sampling rate of 64 Hz. The first five subjects are used as source domains.

All PPG datasets are standardized as follows. Initially, a fourth-order Butterworth bandpass filter with a frequency range of 0.5-4 Hz is applied to PPG signals. Subsequently, a sliding window of 8 seconds with 2-second shifts is employed for segmentation, followed by z-score normalization of each segment. Lastly, the signal is resampled to a frequency of 25 Hz for each segment.

B.3 Cardiovascular disease (CVD) classification

CPSC China Physiological Signal Challenge 2018 (CPSC2018), held during the 7th International Conference on Biomedical Engineering and Biotechnology in Nanjing, China. This dataset consists of 6,877 (male: 3,699; female: 3,178) and 12 lead ECG recordings lasting from 6 seconds to 60 seconds with 500 Hz. We use the original labelling [40] with one normal and eight abnormal types as follows: atrial fibrillation, first-degree atrioventricular block, left bundle branch block, right bundle branch block, premature atrial contraction, premature ventricular contraction, ST-segment depression, ST-segment elevated. We resampled recordings to 100 Hz and excluded recordings of less than 10 seconds.

Chapman Chapman University, Shaoxing People's Hospital (Chapman) ECG dataset which provides 12-lead ECG with 10 seconds of a sampling rate of 500 Hz. The recordings are downsampled to 100 Hz, resulting in each ECG frame consisting of 1000 samples. The labeling setup follows the same approach as in [41] with four classes: atrial fibrillation, GSVT, sudden bradycardia, and sinus rhythm. The ECG frames are normalized to have a mean of 0 and scaled to have a standard deviation of 1. We split the dataset to 80-20% for training and testing as suggested in [41].

We choose leads I, II, III, and V2 during our experiments for both ECG datasets. We followed a similar setup with prior works [57] and considered each dataset as a single domain different from previous tasks. The fine-tuning of the linear layer, which is added to the frozen pre-trained encoder, is performed with 80% of the same domain.

B.4 Metrics

We used the common evaluation metrics in the literature for each task. Specifically, we used accuracy (Acc) and F1 score for activity recognition [19], mean absolute error (MAE), and root mean square error (RMSE) for heart rate prediction [39, 84], and the area under the ROC curve (AUC) for cardiovascular disease classification [57].

In this section, we explain how to calculate each metric for different time-series tasks. For activity recognition, the accuracy metric is computed by dividing the sum of true positives and true negatives by the total number of samples where a window has a single label. The MF1 score is calculated as a harmonic mean of the precision and recall where metrics are obtained globally by counting the total true positives, false negatives, and false positives similar to [19].

For heart rate prediction, the Mean Absolute Error (MAE) and Root-Mean-Square Error (RMSE) are calculated using the following equation:

where K represents the total number of segments. The variables HR model ( k ) and HR ref ( k ) denote the output of the model and reference heart rate values in beats-per-minute for the k th segment, respectively. This performance metric is commonly used in PPG-based heart rate estimation studies [39]. The estimated heart rate values (HR model ( k ) ) are obtained using our model, while the reference heart rate values (HR ref ( k ) ) are directly taken from datasets.

The AUC score for CVD classification is calculated using the one-vs-one scheme where the average AUC is computed for all possible pairwise combinations of classes for both datasets.

C Baselines

C.1 Prior Mixup Techniques

In this section, we give a detailed explanation of each mixup technique we compare our proposed method.

LinearMix We apply linear mixup as in Equation 38 to generate positive samples, if x has more than one channel, mixup is applied independently for each of them.

BinaryMix We implement the binary mixup [43] by swapping the elements of x with the elements of another randomly chosen sample ˜ x as shown below.

where m is a binary mask sampled from a Bernouilli( ρ ) with high values, and ⊙ stands for Hadamard product.

GeometricMix In Geometric Mixup, we create a positive sample corresponding to a sample x by taking its weighted-geometric mean with another randomly chosen sample ˜ x same as [22] as shown below.

CutMix Cutmix is implemented similarly to Binarymix. However, instead of changing each sample point with a probability, we cut a continuous portion using a rectangle mask M from a signal x and replace it with the same portion of another randomly chosen one ˜ x . The starting point of the mask is uniformly sampled while its length is sampled from lower values such that the augmented sample is more similar to the anchor. If the signal has multiple channels, this process is applied to all channels in the same section.

where b and a are the starting point and length of the rectangle wave, respectively.

AmplitudeMix AmplitudeMix is introduced for domain adaptation problems by mixing the amplitude information of images without mixing the phase of two samples [45]. In our setup, we perform amplitude mixing on the time series data across all channels while keeping the phase component unchanged. In other words, we perform the following operations.

SpecMix We implement the SpecMix by applying CutMix to the spectrogram of time-series where the spectrogram is calculated using the short-time Fourier transform as follows.

where g [ n -mR ] is an analysis window of length M with hop length of R over the signal and calculating the discrete Fourier transform (DFT) of each segment of windowed data. The length of the Fourier transform is set to the sample size of the input time series while the hop and window parameters are set to the quarter of the length.

C.2 Prior Methods for Sample Generation

In this section, we give a detailed explanation of prior methods for data generation methods.

Traditional Augmentations We apply two separate data augmentation to the anchor for creating two instances, and the encoders are trained to maximize agreement using the contrastive loss in [15]. We search mainly for augmentations that are known in state-of-the-art works [19]. The detailed augmentations are given in Table 22.

InfoMin We train a model g θ ( . ) , which is restricted to sample-wise 1 × 1 convolutions and ReLU activations same as in [48], to decrease the mutual information between two instances. In the original paper, the input sample is split into two instances ( X 1 and X 2:3 ) and then adversarial training is performed. As we do not have RGB channels for time-series data, we added Gaussian noise to the signal for creating other instances and then perform adversarial training.

NNCLR We follow a similar setup to SimCLR by applying two separate data augmentations, then we use nearest neighbors in the learned representation space as the positive in contrastive losses [49].

PosET We perform the dimension level mixing with extrapolation of positive features as follows:

where ⊙ is Hadamard product, and λ ∼ Beta ( α, α ) . We add 1 to sampled λ for extrapolation as in [50].

GenRep In the original implementation of GenRep, the authors use implicit generative models (IGMs) such as BigBiGAN [85] that are trained with millions of images to create the anchor and positive instance by sampling nearby latent vectors. However, as the number of samples for training is limited in time series and there is a well-trained generator for different time-series tasks, we use our trained VAE for sampling nearby latent vectors as positives. Mainly, we sample an anchor from real data, feed it to the encoder, add a Gaussian noise sampled from a truncated normal distribution, and use the output of the decoder for the positive sample with the anchor.

STAug The Spectral and Time Augmentation (STAug) method is specifically proposed for the time-series forecasting task, where the authors apply the empirical mode decomposition to decompose time series into multiple subcomponents, then reassemble these subcomponents with random weights to generate new synthetic series. Finally, in the time domain, the method uses the linear mixup to generate samples from the reassembled components. Although, the mixing coefficient sampled from a beta distribution in the original implementation, we observe significant performance decreases when the same distribution with parameters is used in our experiments, possibly due to the generated samples being far away from the anchor. We, therefore, investigate the case when the mixing coefficient is sampled from uniform distribution with high values, e.g., same as our method. Since there is no

Augmentation Bank The augmentation bank that perturbs frequency components of a time-series signal is proposed in [21] where the authors use it for unsupervised domain adaptation with a different framework than SimCLR, namely time-frequency consistency (TF-C). As it is a novel data augmentation technique, we have implemented the frequency augmentation bank as a baseline while using the SimCLR framework for a fair comparison with other methods. The authors also employed a collection of time-based augmentations for the time-domain contrastive encoder. Nonetheless, since these augmentations have already been studied in previous CL setups, we chose to exclusively utilize the frequency augmentation bank. In the paper, the authors mentioned using a small budget with low-frequency perturbations results in a performance increase, thus we chose the budget with a single frequency while choosing the α = 0 . 5 with the same settings in the paper.

DACL We perform the mixup for hidden representations, i.e., before applying projection-head, as follows.

where v is the fixed-length hidden representations of samples while λ is sampled from uniform distribution with high values.

IDAA We follow the original implementation of authors with their proposed VAE architecture while optimizing the adversarial strength for each time-series task. We apply the FGSM adversarial attack the same as in the original implementation [53] by perturbing the encoded representation of a sample while adding noises along the gradient sign's direction of the loss.

One setup difference between this section and the previous mixup methods is that when we compare our work with PosET, GenRep, DACL, and IDAA, we apply the best traditional data augmentation techniques, which are used for SimCLR implementation, to the specific positive data generation mechanisms. The reason for this approach is that the original implementations of certain works indicate that the proposed methods achieve optimal results when used in conjunction with known augmentations, where our observations align with these findings.

The detailed hyperparameters for each baseline with the corresponding time series tasks are given in the following section.

D Implementation Details

D.1 Parameters for mixing

In this section, we provide the parameters that are used during our experiments. To determine the optimal parameters of the baselines for each task, we conduct a grid search. This search is performed on a small validation set taken from the largest dataset of the respective tasks, which are USC, Dalia and Chapman. We believe that this approach ensures fairness and produces more realistic results, as dataset-specific optimizations can lead to overfitting of parameters, particularly in smaller and less diverse datasets.

D.2 Baseline Encoder Architecture

For the baseline encoder model, we adopt the DeepConvLSTM as in [19] where the architecture has 4 convolutional layers with 5 × 1 size of 64 kernels while ReLU is followed each convolution. After the convolutions, the tensor is passed through a dropout layer with a dropout rate of 0.5 to prevent overfitting. Then, the output of dropout is fed into the 2-layer LSTM with 128 units. After training

the baseline encoder, we attach a linear layer and freeze the previous layers for fine-tuning. This architecture is widely used for the datasets we used during our experiments [37, 83, 19], we therefore adopt the same network across tasks.

D.3 VAE Models

We use the total correlation variational autoencoder ( β -TCVAE) [86] to calculate the distance between two encoded samples in the latent space. We train the model for 100 epochs with a learning rate of 1 e -3 while setting the batch size to 2048. The latent dimensions and the β values are set to 10 and 5, respectively. Below, we present the tables providing detailed information about the architectures of the encoder and decoder for datasets. The output of convolutional layers is fed to the batch normalization before the activation layer is applied. For tasks Heart rate Prediction and CVD Classification , we use task-specific encoder and decoder as the number of channels and input size for datasets in each task are the same. However, two different networks, one for UCIHAR and one for others, are designed for the Activity Recognition due to different number of input channels.

45

50

55

60

65

70

75

80

85

90

95

-0.2

-0.1

0

Δ

λ

P

UCIHAR

HHAR

USC

70

75

80

85

90

95

100

-0.2

-0.1

0

AUC

Δ

λ

P

CPSC

Chapman

a)

b)

c)

10

12

14

16

18

20

22

24

26

-0.2

-0.1

0

Mean Absolute Error

Δ

λ

P

IEEE SPC12

IEEE SPC22

50

60

70

80

90

100

-0.2

-0.1

0

Δ

λ

A

UCIHAR

HHAR

USC

75

80

85

90

95

100

-0.2

-0.1

0

AUC

Δ

λ

A

CPSC

Chapman

a)

b)

c)

10

12

14

16

18

20

22

24

26

-0.2

-0.1

0

Mean Absolute Error

Δ

λ

A

IEEE SPC12

IEEE SPC22

E Additonal Results

E.1 The effect and robustness of mixing coefficients

In this section, our experiments focus on observing the impact of a diverse range of mixing coefficients for both phase and amplitude components. We decrease the lower threshold of distributions for sampling the mixing coefficient by 0 . 1 and 0 . 2 . For example, normally the phase mixup coefficient for Activity Recognition is sampled from truncated normal λ P ∼ N t (1 , 0 . 1 , 0 . 9) and uniform λ P ∼ U (0 . 9 , 1) . We decrease the low threshold value from 0 . 9 to 0 . 8 and 0 . 7 and report the results for both phase and amplitude. The results are reported in Figures 3 and 4 for eight datasets.

From Figures 3 and 4, it can be inferred that the phase component is more sensitive to the changes. In other words, a significant decrease in performance is observed when the mixing coefficients for the phase are sampled from lower values whereas this effect is not as much as severe for the amplitude coefficient, indicating that the amplitude of frequencies is more robust to changes compared to phase.

E.2 The performance in other frameworks

In this section, we investigate the effect of data augmentations in three different unsupervised learning frameworks which are SimCLR [15], BYOL [59] and TS-TCC [16]. For BYOL, the hidden size of the projector is set to 128, the exponential moving average parameter is set to 0.996. For TS-TCC, the λ 1 and λ 2 coefficients of temporal and contextual contrasting losses are set to 1, the same as in the original implementation. In TS-TCC, the authors proposed to use a weak (jitter and scale) and strong (permutation and jitter) augmentation together, which is shown as TS-TCC + Traditional Augs in the tables. During our experiments, we followed the original implementation of TS-TCC with their proposed architecture and applied additional augmentations after the strong one without changing the original contrastive learning framework. We set the scaling ratio to 2 and 10 for permutation (splitting the signal into a random number of segments with a maximum of 10 and randomly shuffling them). These parameters for augmentation strengths are set to the same values as in the original implementation.

Tables 16 17 and 18 compares the performance of three data augmentation techniques, traditional time-series augmentations, DACL and our proposed method, in contrastive learning frameworks of BYOL, SimCLR, and TS-TCC.

The results show that the BYOL is more robust to the choice of augmentations than SimCLR, which is also indicated in the original paper [59]. Also, another important outcome of this ablation experiment is that when the TS-TCC framework is used for datasets HHAR and USC, the performance decreases compared to other datasets. A possible explanation for this decrease in the TS-TCC might be the hyper-parameters of the augmentations that are used in the paper. The authors change the strength of the permutation window from dataset to dataset. In our experiments, we used the same hyperparameter for all activity recognition datasets, which can explain the outcome. This ablation experiment also shows that the degree of traditional augmentations is important for contrastive learning to learn class invariant representations.

E.3 Do we still need data augmentations?

In this section, we conduct experiments to observe the performance of methods without additional augmentations. During our experiments, we searched for the best traditional augmentation technique for each method in a given task. We searched over common time series augmentation methods in literature (Table 22), and applied them with baselines. Specifically, we apply Resample for Activity Recognition , Permutation with Noise for Heart rate Prediction and Noise with Scaling for CVD Classification . We have observed that these augmentations yield the best results for all baselines when applied prior to the proposed techniques. However, for GenRep, we found that applying the augmentations after generating instances results in better performance, similar to the original work [51]. We, therefore, apply these specified augmentations for each baseline and report the corresponding results.

Different from other baselines, we observed performance increases for a few datasets when GenRep is applied without any augmentations. This phenomenon can be attributed to the generation of low-quality and less realistic positive samples, where additional augmentations lead to alterations in semantic information, due to less number of samples during training VAE models. However, in the end, we observe that applying additional augmentations always increases the performance on average for all baselines in each task.

E.4 The effect of interpolating phase components

Here, we investigate the effect of phase interpolation of two samples on the CL performance. In our proposed method, we bring the phase components of the two coherent signals together by adding a small value to the anchor's phase in the direction of the other sample. In this section, we apply the opposite case of our proposed method and increase the gap of phase difference between the anchor and randomly chosen sample. However, we mix their amplitudes according to our proposed method to only observe the phase effect. In other words, we perform the mixup as in Equation 46. Note that the phase mixing in Equation 46 differs from the proposed method only by the sign change.

Also, It is important to note that we sample the mixing coefficients for both amplitude and phase from the same distributions in the proposed method to have a fair comparison. Tables 23 24 25.

E.5 The comparison of Mixup methods

In this section, we give a detailed comparison of prior mixup methods with ours below tables, which are the explicit numbers for Figure 2. Our method demonstrates superior performance compared to previous mixup techniques in 11 out of 14 metrics, indicating its effectiveness. Additionally, the Amplitude Mixup technique, which yields comparable results in two datasets, further supports our claim regarding the destructive effect of simultaneously mixing phase and magnitude for time series. The relatively lower performance of Amplitude Mixup for some datasets can be explained by its limited diversity in generating positive samples since this technique has no solution for mixing the phase of samples in randomly chosen pairs. In other words, as the phase of the augmented instance is the same as the anchor in Amplitude Mix, the diversity of generated positive samples is less compared to other techniques.

F Illustrative Examples

In this section, we show examples of the destructive behavior of linear mixup and how our proposed mixup technique solves this problem. In Figure 5 a) , we show two PPG waveforms that are obtained from IEEE SPC15 with the same label i.e., the same heart rate value. Also, we give the corresponding frequency domain transformations of these two waveforms in Figure 5 b) where the frequency axis is converted to heart rate in beats-per-minute i.e., 1 Hz corresponds to 60 bpm.

40

60

80

100

120

140

160

180

200

0

0.1

0.2

0.3

0.4

0.5

Anchor

Sample

Normalized Power (dB)

a)

b)

1

2

3

4

5

6

7

8

-2

-1

0

1

2

Anchor

Sample

Normalized Magnitude (V)

Time (s)

Heart rate (bpm)

Two samples with the same label

where the anchor has a severe

motion artifact at 2-3 Hz.

0

When the linear mixup is applied as in Equation 47 with a λ of 0.9, the resulting waveform is anticipated to contain heart rate information to an extent similar to both the anchor and the sample.

However, when there is a phase difference greater than π/ 2 between these two samples in the frequencies where the task-specific information is carried, the linear mixup destroys the information.

40

60

80

100

120

140

160

180

200

0

0.1

0.2

0.3

0.4

0.5

Anchor

Sample

Linear mixup

The linear mixup method

generates an augmented

sample where the information

in the critical frequency band

decreased significantly

compared to both samples.

Normalized Power (dB)

Heart rate (bpm)

a)

b)

1

2

3

4

5

6

7

8

-2

-1

0

1

2

Anchor

Linear mixup

Normalized Magnitude (V)

Time (s)

0

Figures 5 and 6 demonstrate the destructive behavior of linear mixup instead of feature interpolation. The linear mixup technique destroys the task-specific information even though the two samples have the same labels and the mixup ratio is relatively high.

As our proposed mixup prevents this problem and interpolates between features of two samples, the information is not lost but rather enhanced as both samples have the same label, shown in Figure 7.

1

2

3

4

5

6

7

8

-2

-1

0

1

2

0

40

60

80

100

120

140

160

180

200

0

0.1

0.2

0.3

0.4

0.5

Normalized Power (dB)

Heart rate (bpm)

Normalized Magnitude (V)

Time (s)

Anchor

Sample

Proposed mixup

Anchor

Proposed mixup

a)

b)

The proposed mixup technique

interpolates the features without

causing any information loss in

the critical frequency band.

As can be seen From figures 6 and 7, our proposed mixup technique not only prevents information loss due to linear mixup but also generates an interpolated sample.

G Performance in Supervised Learning Paradigm

We also conduct experiments in the supervised learning paradigm with our proposed mixup method to see its effectiveness in different learning paradigms. We compare the performance of our method with prior mixup techniques. During the experiments, we follow the original implementation where the mixup is applied to the same minibatch after random shuffling. In the seminal work of mixup [31], the authors stated that interpolating only between inputs with equal labels does not lead to performance gains. Therefore, we only perform the tailored mixup without implementing any V AEs. We implement the tailored mixup for the supervised learning as in Equation 48. Although the phase equations can be simplified by omitting the absolute value, we leave it similar to Equation 4 for consistency.

where the coefficient for the λ A is chosen from a beta distribution with α ∈ [0 . 1 , 0 . 4] within the same range of the original implementation [31]. The mixing for the phase is constrained to our original implementation with a uniform λ P ∼ U (0 . 9 , 1) . We searched for the best α value for each time-series task and augmentation method. Unlike linear mixup and our proposed approach, for cutmix, we followed the recommendation from the original paper and searched the α value close to 1.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new time series augmentation method which adopts Mixup on the amplitude and phase, respectively, after transforming the time series in the frequency domain, thus avoiding the destructive extrapolation from linear Mixup.\n\nSTRENGTHS:\n1. The authors propose a novel mixup method in the frequency domain for time series, avoiding the destructive extrapolation resulting from linear Mixup.\n\n2. The theoretical proof is complete and has some reference value.\n\nWEAKNESSES:\n1. It is unclear whether it works when considering amplitude or phase alone for Mixup. The relevant ablation experiments must be performed consistently with the parameter settings of the original method.\n\n2. Lack of convincing examples. For example, to illustrate the problem, perform Mixup on two samples where the frequency domain information in both samples that are strongly correlated with I(x,y) exactly cancels each other and shows whether the newly generated samples have specific frequency domain information that can be identified.\n\n3. The paper's content does not explain how to \"Find Order in Chaos\" mentioned in the title and how the \"chaos\" is reflected. And I'm confused about how the so-called \"control the degree of chaos\" proposed by the authors is done.\n\n4. Assumption 2.1 is too idealistic. The label information of time series is not only related to the time domain but also the frequency domain, and focusing on the frequency domain information only does not achieve the classification well.\n\nQUESTIONS:\n1. The ablation experiments considering amplitude or phase alone for mixup need to be performed consistently with the parameter settings of the original method.\n2. Please give practical examples of how Mixup may corrupt the frequency domain information used to discriminate the generated samples.\n3. It is not clear how the so-called \"control the degree of chaos\" proposed by the authors is done.\n4. For data augmentation of time series, why does this paper choose to perform augmentation in the frequency domain instead of the time domain? In general, it is difficult to use only frequency domain information for classification over the time domain for time series classification tasks [1]. In contrast, the combination of time and frequency domain information can effectively improve the model's classification performance [2].\n5. What are the advantages of performing data augmentation in the frequency domain instead of doing data augmentation in the time domain of time series data?\n6. Why are eight datasets chosen for experiments in this paper? Its inconsistency with existing benchmark datasets for time series prediction [3,4,5], classification [6,7] and anomaly detection [8] tasks. In addition, the comparison methods in this paper do not include benchmark methods in the time series domain. For example, benchmark methods for time series prediction include Informer [3], CoST [4], and FEDFormer [5], etc., time series classification methods include OS-CNN [6] and DSN [7], etc., and time series anomaly detection includes Anomaly Transformer [8], etc.\n\n[1] Cross reconstruction transformer for self-supervised time series representation learning. arXiv, 2022.\n[2] Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency. NeurIPS, 2022.\n[3] Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. AAAI, 2021.\n[4] CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting. ICLR, 2022.\n[5] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. ICML, 2022.\n[6] Omni-Scale CNNs: a simple and effective kernel size configuration for time series classification. ICLR, 2022.\n[7] Dynamic Sparse Network for Time Series Classification: Learning What to \"See\". NeurIPS, 2022.\n[8] Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy. ICLR, 2022.\n\nLIMITATIONS:\nThe authors did not mention the limitations. I do not have any comments on this point.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a novel mixup method for non-stationary time-series data to generate positive samples for the contrastive learning formulation. The proposed method mixes the magnitude and phase of each frequency component of two samples that are close in the latent space of a variational autoencoder. The authors prove that the mixup process in the frequency domain does not cause the loss of information while it generates diverse samples. The paper provides an extensive empirical evaluation to show that the proposed method learns better representations than existing contrastive learning baselines on eight time series datasets from three tasks.

STRENGTHS:
The paper studies an important and under-explored problem. Although contrastive learning is proven to work well on images, its performance on time series is yet limited due to the lack of good time series augmentations. While many existing work tries to find the good time series augmentations through extensive empirical studies, this paper proposes a theoretically well-grounded mix-up method for time series augmentation. Moreover, the empirical evaluation is also extensive and convincing.

WEAKNESSES:
1. The hyperlinks of the citations and the references are missed. 

2. I don't find where $x^*$ is defined.

3. It would be great if there is an algorithm box explaining the whole process of drawing samples, computing the degree of the augmentation, performing the mixup, and then training.

QUESTIONS:
1. How do you select $\beta$? Would a very large (close to 1) $\beta$ limit the diversity of the augmented samples?

2. What does the instance selection (line 268) mean exactly?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about a novel data augmentation method that can be used in contrastive learning for time-series tasks and aims to connect intra-class samples together, and find order in the latent space. The proposed method builds upon the mixup data augmentation technique by controlling the degree of chaos created by data augmentation. More specifically, the augmentation method considers the phase and the amplitude information as two separate features and then generates positive samples by controlling the mixup coefficients for each feature for each randomly chosen pair. This process helps to generate features that enhance intra-class similarity and help contrastive learning to learn class separated representations. The proposed method is evaluated using three time-series tasks: heart rate estimation, human activity recognition and cardiovascular disease detection and against state of the art comparison methods.\n\nSTRENGTHS:\n- The paper is about an interesting topic, contrastive learning in time-series.\n- The proposed methodology is based on the idea of controlling the degree of chaos created by data augmentation methods. These ideas can be applied in the augmentation part of contrastive learning for other domain as well.\n- The proposed methodology shows improvement in the performance over ten comparison methods on three tasks and eight datasets. Some time-series contrastive learning methods are missing from the comparison though.\n\nWEAKNESSES:\n- The writing in the paper can be improved. The Method and Results and Discussion sections need some organization, adding subsections will help with the structure.\n- The proposed method has incremental novelty. This method builds on top of known components. The writing needs improvements to make the contributions of this paper more clear. \n- The experimental setup also seems to be missing some important state of the art. No time-series specific contrastive learning method is included in the experiments for comparison.\n- The proposed methodology seems to have similarities with TS-TCC [80] and TFC [22] (the reference numbers align with the ones in the main paper) and since these are state of the art in contrastive learning for time-series, they should be part of the comparison methods. \n- A comparison of the methodology of the proposed method with [22] and [80] is needed as these paper are around the same ideas. What is different in the contributions of the proposed method?\n- Dataset statistics are missing from the Results and Discussion section. Adding them in a table helps with the understanding of the results.\n\nQUESTIONS:\nMy main concerns are regarding the readability of the paper and improvements in the structure of the paper, and the experimental setup that is missing time-series contrastive learning models. The results and the discussion will be more convincing and the performance difference more important if the author include the results of some other methods, like [22] and [80]. A comparison of the methodology and the concepts of the proposed method with [22] and [80] is needed as these paper are around the same ideas. What is different in the contributions of the proposed method?\n\nLIMITATIONS:\nThere are no limitations or negative societal impact from this work. The datasets used in the paper are all publicly available.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores data augmentation for contrastive learning that better aligns with the nature of time series. Specifically, they tailor the mix-up method separately to amplitude and phase in the frequency domain. A theoretical analysis illustrates how the method enhances task-specific information, contrasting it to the linear mix-up method which potentially discards task-specific information. Experiments on time-series from three domains also demonstrated the advantages compared with previous mix-up and contrastive learning methods.\n\nSTRENGTHS:\n1. This paper studies an important problem of designing augmentation for contrastive learning that adapts to the non-stationarity of time series. \n\n2. The paper presents theoretical analysis of the benefits of proposed mix-up method compared with vanilla linear mix-up method.\n\n3. The paper shows better performance on time-series datasets of multiple domains.\n\nWEAKNESSES:\n**Lack of baseline comparisons**: The paper compares with a list of mix-up methods and contrastive learning methods, and the authors also incorporate time-series augmentation methods into the framework in appendix. However, it would also be interesting to have (1) some direct comparisons with existing time-series augmentation methods, e.g., [1, 2] (both have mix-up augmentation in the time or frequency domain). (2) Contrastive learning methods dedicated for time-series domain [3, 4, 5, 6] (I am curious why [4] is not one of the baselines in experiments?)\n\n[1] FrAug: Frequency Domain Augmentation for Time Series Forecasting \n\n[2] Towards Diverse and Coherent Augmentation for Time-Series Forecasting\n\n[3] CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting\n\n[4] Self-supervised contrastive pre-training for time series via time-frequency consistency\n\n[5] Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding\n\n[6] TS2Vec: Towards Universal Representation of Time Series \n\n\n**Experimental analysis**: Why linear mix-up method performs well in activity recognition task? Is there any difference in the setting of activity recognition compared with the other two tasks?\n\n**Writing**: \n\n1. Consider defining notations of x, \\tilde{x}, x^* before or in Proposition 2.3. \n\n2. The title metions \"order\" and \"chaos\"; hence, a detailed explanation of these terms, as well as how the proposed method comprehensively accounts for the \"non-stationarity\" of time series, would be beneficial.\n\nQUESTIONS:\nSee weakness above.\n\nLIMITATIONS:\nThe paper did not explicitly discuss their limitations. Some comments and questions regarding limitations of the work are discussed in the weakness and question parts above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}