SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Berken Utku Demirel
ETH Zürich, Switzerland
berken.demirel@inf.ethz.ch
Christian Holz
christian HOLz@inf.ethz.ch
The success of contrastive learning is well known to be dependent on data augmentation. Although the degree of data augmentations has been well controlled by utilizing pre-defined techniques in some domains like vision, time-series data augmentation is less explored and remains a challenging problem due to the complexity of the data generation mechanism, such as the intricate mechanism involved in the cardiovascular system. Moreover, there is no widely recognized and general time-series augmentation method that can be applied across different tasks. In this paper, we propose a novel data augmentation method for quasi-periodic time-series tasks that aims to connect intra-class samples together, and thereby find order in the latent space. Our method builds upon the well-known mixup technique by incorporating a novel approach that accounts for the periodic nature of non-stationary time-series. Furthermore, by controlling the degree of chaos created by data augmentation, our method leads to improved feature representations and performance on downstream tasks. We evaluate our proposed method on three time-series tasks, including heart rate estimation, human activity recognition, and cardiovascular disease detection. Extensive experiments against state-of-the-art methods show that the proposed approach outperforms prior works on optimal data generation and known data augmentation techniques in the three tasks, reflecting the effectiveness of the presented method. Source code: https://github.com/eth-siplab/Finding_Order_in_Chaos.
Self-supervised learning methods have gained significant attention as they enable the discovery of meaningful representations from raw data without explicit annotations. These self-supervised methods learn representations without labels by designing pretext tasks that transform the unsupervised representation learning problem into a supervised one such as predicting the rotation of images [1], or contexts [2, 3]. Among these methods, contrastive learning (CL), which learns to distinguish semantically similar examples over dissimilar ones, stands out as a powerful approach in self-supervised learning across various domains including computer vision [4-6], speech recognition [7-10], and natural language processing [11-14].
The success of contrastive learning relies on the creation of similar and dissimilar examples, which is typically achieved through the use of data augmentations [15, 16]. Recently, it was shown that data augmentations have a role to create a "chaos" between different intra-class samples such that they become more alike. For example, two different cars become very similar when they are both cropped to the wheels. [17]. However, in time-series data, creating similar samples with augmentation techniques is more challenging due to the complexity of the dynamical data generation mechanisms [18]. Moreover, research on contrastive learning for time series has demonstrated the
absence of a unique data augmentation technique that consistently performs better than others in different tasks [19, 20]. Instead, the choice of augmentation depends on the contextual characteristics of the signal, such as perturbing the high-frequency content of a signal that carries characteristic information in low frequencies does not generate useful data samples that are helpful for contrastive learning to learn class invariant features [21].
Considering these limitations, in this work, we first propose a novel data augmentation method for time series data by performing a tailored mixup while considering the phase and amplitude information as two separate features. Then, we perform specific operations for both features to generate positive samples by controlling the mixup coefficients for each feature to prevent aggressive augmentation. Specifically, our method employs a technique that controls the mixup ratio for each randomly chosen pair based on their distance in the latent space which is acquired through the use of a variational autoencoder (VAE), whose objective is to learn disentangled representations of data without labels. To this end, subjecting to the distance constraint in the latent space, the mixup tries to connect semantically closer samples together more aggressively while preventing the excessive interpolation of dissimilar samples that are likely to belong to different classes. Therefore, the purpose of our proposed method for quasi-periodic time-series data augmentation is to find an order in "chaos" between different samples such that they become more alike by interpolating them in a novel manner to prevent the loss of information. We summarize our contributions as follows:
- We propose a novel mixup method for non-stationary quasi-periodic time-series data by considering phase and magnitude as two separate features to generate samples that enhance intra-class similarity and help contrastive learning to learn class-separated representations.
- We present a novel approach for sampling mixup coefficients for each pair based on their similarity in the latent space, which is constructed without supervision while learning disentangled representations, to prevent aggressive augmentation between samples.
- We show that the tailored mixup with coefficient sampling consistently improves the performance of contrastive learning in three time-series tasks compared to prior mixup techniques and proposed augmentation methods that generate optimal/hard positives or negatives.
We use lowercase symbols  $(x)$  to denote scalar quantities, bold lowercase symbols  $(\mathbf{x})$  for vector values, and capital letters  $(X)$  for random variables. Functions with a parametric family of mappings are represented as  $f_{\theta}(.)$  where  $\theta$  is the parameters. The discrete Fourier transformation of a real-valued time series sample is denoted as  $\mathcal{F}(.)$ , yielding a complex variable as  $\mathrm{X_k}$  where  $\mathrm{X} \in \mathbb{C}$  and  $k \in [0, f_s / 2]$  is the frequency with the maximum value of Nyquist rate. The amplitude and phase of the  $\mathcal{F}(\mathbf{x})$  are represented as  $A(\mathbf{x})$  and  $P(\mathbf{x})$ . The real and imaginary parts of a complex variable are shown as  $\operatorname{Re}(.)$  and  $\operatorname{Im}(.)$ . The detailed calculations for operations are given in the Appendix A.1.
2.2 Setup
We follow the common CL setup as follows. Given a dataset  $\mathcal{D} = \{(\mathbf{x}_i)\}_{i=1}^K$  where each  $\mathbf{x}_i$  consists of real-valued sequences with length  $L$  and  $C$  channels. The objective is to train a learner  $f_\theta$  which seeks to learn an invariant representation such that when it is fine-tuned on a dataset  $\mathcal{D}_l = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^M$  with  $M \ll K$  and  $\mathbf{y}_i \in \{1, \dots, N\}$ , it can separate samples from different classes.
2.3 Motivation
As stated by prior works, mixup-based methods have poor performance in domains where data has a non-fixed topology, such as trees, graphs, and languages [22, 23]. Here, we demonstrate how we derived our proposed method by revealing the limitations of mixup for time series theoretically while considering the temporal dependencies and non-stationarities.
Assumption 2.1 (SNR Matters). There exist one or multiple bandlimited frequency ranges of interest  $f^{*}$ , where the information that average raw time-series data conveys about the labels (i.e.,  $\mathcal{I}(\mathbf{y};\mathbf{x})$ ) is directly proportional to normalized signal power in that frequency range as in Equation 1.
$$
\mathcal {I} (\mathbf {y}; \mathbf {x}) \propto \int_ {f ^ {*}} S _ {x} (f) / \int_ {- \infty} ^ {\infty} S _ {x} (f) \text {w h e r e} S _ {x} (f) = \lim  _ {N \rightarrow \infty} \frac {1}{2 N} \left| \sum_ {n = - N} ^ {N} x _ {n} e ^ {- j 2 \pi f n} \right| ^ {2} \tag {1}
$$
Assumption 2.1 states that the information from a time series depends on its signal-to-noise ratio (SNR). Prior works showed that specific frequency bands hold inherent information about the characteristics of time series, which helps classification [21, 24].
Assumption 2.2. The true underlying generative process  $f(\cdot)$ , for a given data distribution  $\mathcal{D} = \{\mathbf{x}_k\}_{k=1}^K$ , is quasiperiodic, i.e.,  $f(\mathbf{x} + \tau) = g(\mathbf{x}, f(\mathbf{x}))$ , where  $\tau$  can be either fixed or varied.
Assumption 2.2 posits that the observed data samples from the distribution  $\mathcal{D}$  are generated by a quasiperiodic function. This is a minimal assumption since the quasiperiodicity is the relaxed version of the periodic functions. In simpler terms, quasiperiodicity can be described as the observed signals exhibiting periodicity on a small scale, while being unpredictable on a larger scale. And, several prior works showed that the data generation mechanism of time-series data for several applications in the real world are quasiperiodic [25-30]. Therefore, Assumption 2.2 is realistic.
Proposition 2.3 (Destructive Mixup). If Assumptions 2.1 and 2.2 hold, there exist  $\lambda \sim \text{Beta}(\alpha, \alpha)$  or  $\lambda \sim U(\beta, 1.0)$  with high values of  $\beta$  such that when linear mixup techniques are utilized, the lower bound of the mutual information for the augmented sample distribution decreases to zero.
$$
0 \leq \mathcal {I} (\mathbf {y}; \mathbf {x} ^ {+}) <   \mathcal {I} (\mathbf {y}; \mathbf {x} ^ {*}), \quad \text {w h e r e} \mathbf {x} ^ {*} \text {i s t h e o p t i m a l s a m p l e},
$$
$$
\mathbf {x} ^ {+} = \lambda \mathbf {x} + (1 - \lambda) \tilde {\mathbf {x}} \text {a n d} \int_ {f ^ {*}} S _ {x ^ {*}} (f) = \int_ {- \infty} ^ {\infty} S _ {x ^ {*}} (f) \tag {2}
$$
Proofs can be found in Appendix A. This proposition indicates that although the augmented samples are primarily derived from anchor samples  $(\mathbf{x})$  with high ratios, the resulting instances may not contain any task-relevant information. In other words, the augmentation process can potentially discard the whole task-specific information. This destructive behavior of mixup for quasi-periodic data can be attributed to the interference phenomenon in which two waves interact to form the resultant wave of the lower or higher amplitude according to the phase difference as shown in Proposition 2.3.
We introduce a novel approach to overcome the limitations of mixup by treating the magnitude and phase of sinusoidal signals as two distinct features with separate behaviors. Subsequently, we apply tailored mixup operations to each feature, considering their specific characteristics and effects.
We perform the linear mixup for the magnitude of each sinusoidal. However, for the phase, we take a different approach and bring the phase components of the two coherent signals together by adding a small value to the anchor's phase in the direction of the other sample. The mixup operation performs the linear interpolation of features [31], however, interpolation of two complex variables can result in a complex variable whose phase and magnitude are completely different/far away from those two, i.e., mixup can be destructive extrapolation rather than the interpolation of features. Therefore, we mix the phase of two sinusoidal as follows. We start by calculating the shortest phase difference between the two samples, denoted as  $\Delta \Theta$ , as described in Equation  $3^{1}$ .
$$
\theta \equiv [ P (\mathbf {x}) - P (\tilde {\mathbf {x}}) ] (\mathrm {m o d} 2 \pi)
$$
$$
\Delta \Theta = \left\{ \begin{array}{l l} \theta - 2 \pi , & \text {i f} \theta > \pi \\ \theta , & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
The sign of the calculated phase difference provides information about the relative phase location of the other sample, in either a clockwise or counterclockwise direction in the phasor diagram. And, the absolute value of it represents the shortest angular difference between two samples in radians. Based
<sup>1</sup>We use phase in radians throughout the paper in the range  $(-\pi, \pi]$
on the calculated phase difference between two samples, we perform mixup operation to generate diverse positive samples as in Equation 4 such that phase and magnitude of augmented instances are interpolated properly according to the anchor sample  $\mathbf{x}$ , without causing any destructive interference.
$$
\mathbf {x} ^ {+} = \mathcal {F} ^ {- 1} (A (\mathbf {x} ^ {+}) \angle P (\mathbf {x} ^ {+})) \quad \text {w h e r e} A (\mathbf {x} ^ {+}) = \lambda_ {A} A (\mathbf {x}) + (1 - \lambda_ {A}) A (\tilde {\mathbf {x}}) \text {a n d}
$$
$$
P \left(\mathbf {x} ^ {+}\right) = \left\{ \begin{array}{l l} P (\mathbf {x}) - | \Delta \Theta | * (1 - \lambda_ {P}), & \text {i f} \Delta \Theta > 0 \\ P (\mathbf {x}) + | \Delta \Theta | * (1 - \lambda_ {P}), & \text {o t h e r w i s e} \end{array} \right. \tag {4}
$$
The proposed method which mixes the magnitude and phase of each frequency component with tailored operations, not only prevents destructive interference between time series, resulting in an increase in the lower bound of mutual information (as shown in Theorem 3.1), but also generates diverse augmented instances with the same two samples by using two different mixing coefficients.
Figure 1: The phasor representation of linear mixup a), and proposed mixup b). The anchor, randomly chosen sample, and generated instances are represented as  $\mathbf{x}$ ,  $\bar{\mathbf{x}}$ , and  $\mathbf{x}^{+}$ , respectively.
Theorem 3.1 (Guarantees for Mixing). Under assumptions 2.1 and 2.2, given any  $\lambda \in (0,1]$ , the mutual information for the augmented instance lower bounded by the sampled  $\lambda$  and anchor  $\mathbf{x}$
$$
\lambda \mathcal {I} (\mathbf {y}; \mathbf {x}) \leq \mathcal {I} (\mathbf {y}; \mathbf {x} ^ {+}) <   \mathcal {I} (\mathbf {y}; \mathbf {x} ^ {*}) \text {w h e r e} \mathbf {x} ^ {+} = \mathcal {F} ^ {- 1} (A (\mathbf {x} ^ {+}) \angle P (\mathbf {x} ^ {+})) \tag {5}
$$
We provide an intuitive demonstration in Figure 1, along with a detailed mathematical proof presented in Appendix A. Our approach also offers increased flexibility in selecting the mixing coefficients of phase and magnitude, based on their sensitivities to the mixing process as well as the augmentation degree for each randomly chosen pair. Since the degree of augmentations has crucial importance for contrastive learning, there can be cases where augmentations are either too weak (intra-class features cannot be clustered together) or too strong (inter-class features can also collapse to the same point) and lead to sub-optimal results [17]. To mitigate this issue and find an order for augmentation degree, we search pairs of samples that are semantically closer, meaning they are more likely to belong to the same class. We then perform the proposed mixup more aggressively on these pairs, creating more closer and diverse samples while decreasing the augmentation strength for less similar pairs. To find similar samples without labels, we train a completely unsupervised  $\beta$ -VAE [32] that maps data points to a latent space such that two random samples are semantically similar if they are close in the latent as shown in Proposition 3.2.
Proposition 3.2 (Consistency in Latent Space [33]). Given a well-trained unconditional VAE with the encoder  $E(\cdot)$  that produces distribution  $p_E(\mathbf{z}|\mathbf{x})$ , the decoder  $D(\cdot)$  that produces distribution  $q_D(\mathbf{x}|\mathbf{z})$  while the prior for  $\mathbf{z}$  is  $p(\mathbf{z})$ , let  $\mathbf{z}_1$  and  $\mathbf{z}_2$  be two latent vectors of two different real samples  $\mathbf{x}_1$  and  $\mathbf{x}_2$ , i.e.,  $E(\mathbf{x}_1) = \mathbf{z}_1$  and  $E(\mathbf{x}_2) = \mathbf{z}_2$ . If the distance  $d(\mathbf{z}_1, \mathbf{z}_2) \leq \delta$ , then  $D(\mathbf{z}_1)$  and  $D(\mathbf{z}_2)$  will have a similar semantic label as in Equation 6.
$$
\left| \mathcal {I} \left(D \left(\mathbf {z} _ {\mathbf {1}}\right); \mathbf {y}\right) - \mathcal {I} \left(D \left(\mathbf {z} _ {\mathbf {2}}\right); \mathbf {y}\right) \right| \leq \epsilon , \tag {6}
$$
where  $\epsilon$  stands for tolerable semantic difference,  $\delta$  is the maximum distance to maintain semantic consistency, and  $d(.)^{\prime}$  is a distance measure such as cosine similarity between two vectors.
The above proposition with Theorem 3.1 motivates us to perform augmentation aggressively if two randomly chosen samples are semantically closer. Therefore, we sample the mixup coefficient for both phase and magnitude from a uniform distribution  $\lambda_{A},\lambda_{P}\sim U(\beta ,1.0)$  with low values of  $\beta$  if the distance between the latent vectors is below a threshold, otherwise, they are drawn from a truncated normal distribution  $\lambda_{A},\lambda_{P}\sim \mathcal{N}^{t}(\mu ,\sigma ,1.0)$  with a high mean and low standard deviation.
We conduct experiments on the proposed approach and compare it with other mixup methods or optimal/hard positive sample generation in the contrastive learning setup. During our experiments, we use SimCLR [15] framework without specialized architectures or a memory bank for all baselines to have a fair comparison. Results with other CL frameworks can be found in Appendix E. Complete training details and hyper-parameter settings for datasets and baselines are provided in Appendix D.
We performed extensive experiments on eight datasets from three tasks that include activity recognition from inertial measurements (IMUs), heart rate prediction from photoplethysmography (PPG), and cardiovascular disease classification from electrocardiogram (ECG). We provided short descriptions of each dataset below, and further detailed information with metrics can be found in Appendix B.
Activity recognition We used UCIHAR [34], HHAR [35], and USC [36] for activity recognition. During the evaluation, we assess the cross-person generalization capability of the contrastive models, i.e., the model is evaluated on a previously unseen target domain. We follow the settings in GILE [37] to treat each person as a single domain while the fine-tuning dataset is much smaller than the unsupervised one.
Heart rate prediction We used the IEEE Signal Processing Cup in 2015 (IEEE SPC) [38], and DaLia [39] for PPG-based heart rate prediction. The SPC provides two datasets, one smaller with lesser artifacts (referred to as SPC12) [38] and a bigger dataset with more participants including heavy motions (referred to as SPC22). In line with previous studies, we adopted the leave-one-session-out (LOSO) cross-validation, which involves evaluating methods on subjects or sessions that were not used for pre-training and fine-tuning.
Cardiovascular disease (CVD) classification We conducted our experiments on China Physiological Signal Challenge 2018 (CPSC2018) [40] and Chapman University, Shaoxing People's Hospital (Chapman) ECG dataset [41]. We selected the same four specific leads as in [42] while treating each dataset as a single domain with a small portion of the remainder dataset used for fine-tuning the pre-trained model. We split the dataset for fine-tuning and testing based on patients (each patient's recordings appear in only one set).
Comparison with prior mixup techniques We evaluate the effectiveness of our proposed mixup by comparing it with other commonly used mixup methods, including Linear-Mixup [31], Binary-Mixup [43], Geometric-Mixup [22], Cut-Mix [44], Amplitude-Mix [45] and Spec-Mix [46]. When we compare the performance of mixup techniques, we follow the same framework with [47] where the samples of mixture operation only happen in current batch samples. And, the mixup samples are paired with anchors, i.e., without applying mixup second times, for contrastive pre-training.
Comparison with methods for optimal sample generation We evaluate the performance of our proposed method by comparing it with other data generation methods and baselines in contrastive learning setup while considering previously known augmentation techniques. Traditional data augmentations for time-series [19], such as resampling, flipping, adding noise, etc. InfoMin which leverages an adversarial training strategy to decrease the mutual information between samples
while maximizing the NCE loss [48]. NNCLR [49], which uses nearest neighbors in the learned representation space as the positive samples. Positive feature extrapolation [50], which creates hard positives through feature extrapolation. GenRep which uses the latent space of a generative model to generate "views" of the same semantic content by sampling nearby latent vectors [51]. Aug. Bank [21], which proposes an augmentation bank that manipulates frequency components of a sample with a limited budget. STAug [52], which combines spectral and time augmentation for generating samples using the empirical mode decomposition and linear mixup. DACL [22], which creates positive samples by mixing hidden representations. IDAA [53], which is an adversarial method by modifying the data to be hard positives without distorting the key information about their original identities using a VAE. More implementation details for each baseline are given in Appendix C.
4.3 Implementation
We use a combination of convolutional with LSTM-based network, which shows superior performance in many time-series tasks [19, 54, 55], as backbones for the encoder  $f_{\theta}(.)$  where the projector is two fully-connected layers. We use InfoNCE as the loss, which is optimized using Adam with a learning rate of 0.003. We train with a batch size of 256 for 120 epochs and decay the learning rate using the cosine decay schedule. After pre-training, we train a single linear layer classifier on features extracted from the frozen pre-trained network, i.e., linear evaluation, with the same hyperparameters. Reported results are mean and standard deviation values across three independent runs with different seeds on the same split. More details about the implementation, architectures, and hyperparameters with the trained VAEs are given in Appendix D.
Tables 1, 2, and 3 present the results of our proposed approach compared to state-of-the-art methods for optimal/hard positive sample generation in contrastive learning setup across the three tasks in eight datasets. Additionally, Figure 2 compares our approach with prior mixup methods (e.g., linear mixup, cutmix) without applying any other additional augmentation techniques. Overall, our proposed method has demonstrated superior performance compared to other methods in seven datasets, with the second-best performance in the remaining dataset, and a minor performance gap.
Table 1: Performance Comparison of ours with prior works in Activity Recognition datasets
Method	UCIHAR	HHAR	USC
ACC↑	MF1↑	ACC↑	MF1↑	ACC↑	MF1↑
Supervised
DCL [37]	77.63	-	51.27	-	60.35	-
CoDATS [56]	68.22	-	45.69	-	-	-
GILE [37]	88.17	-	55.61	-	-	-
Self-Supervised
Traditional Augs.	87.05 ± 1.07	86.13 ± 0.96	85.48 ± 1.16	84.31 ± 1.31	53.47 ± 1.10	52.09 ± 0.95
NNCLR [49]	85.31 ± 0.91	83.56 ± 1.25	83.16 ± 1.32	82.15 ± 1.25	55.41 ± 1.43	52.64 ± 1.37
InfoMin [48]	38.07 ± 8.15	30.66 ± 9.15	31.58 ± 10.2	29.72 ± 11.1	35.89 ± 14.3	37.77 ± 9.12
IDAA [53]	82.23 ± 0.69	79.84 ± 0.89	88.98 ± 0.62	89.01 ± 0.55	59.23 ± 1.10	56.11 ± 1.54
PosET [50]	88.13 ± 0.91	87.35 ± 0.96	85.77 ± 1.11	85.90 ± 1.20	41.37 ± 5.63	39.43 ± 5.72
STAug [52]	89.83 ± 0.71	88.91 ± 0.62	87.69 ± 1.03	87.73 ± 0.93	55.61 ± 1.08	56.74 ± 1.21
Aug. Bank [21]	65.27 ± 1.12	71.16 ± 1.24	67.95 ± 1.45	75.13 ± 1.32	43.28 ± 4.37	47.31 ± 4.68
GenRep [51]	87.22 ± 1.05	86.48 ± 0.95	87.05 ± 0.95	86.45 ± 0.90	50.13 ± 2.85	49.50 ± 2.73
DACL [22]	73.12 ± 1.23	66.28 ± 1.11	80.89 ± 0.91	81.31 ± 0.78	53.61 ± 2.60	51.76 ± 2.21
Ours	91.60 ± 0.65	90.46 ± 0.53	88.05 ± 1.05	87.95 ± 1.10	60.13 ± 0.75	59.13 ± 0.69