SlowGuess/NeurIPS_2023_OCR huggingface dataset:
HHAR Heterogeneity Dataset for Human Activity Recognition (HHAR) is collected by nine subjects within an age range of 25 to 30 performing six daily living activities with eight different smartphones—Although HHAR includes data from smartwatches as well, we use data from smartphones—that were kept in a tight pouch and carried by the users around their waists [35]. Subjects then perform 6 activities: 'bike', 'sit', 'stairs down', 'stairs up', 'stand', and 'walk'. Due to variant sampling frequencies of smart devices used in HHAR dataset, we downsample the readings to  $50\mathrm{Hz}$  and apply 100 (two seconds) and 50 as sliding window length with step size, the windows are normalized to zero mean with unit standard deviation. We used the first four subjects (i.e., a, b, c, d) as source domains.
USC USC human activity dataset (USC-HAD) is composed of 14 subjects (7 male, 7 female, aged 21 to 49 with a mean of 30.1) executing 12 activities with a sensor on the front right hip. The data dimension is six (3-axis accelerometer, 3-axis gyroscope) and the sample rate is  $100\mathrm{Hz}$ . 12 activities include walking forward, walking left, walking right, walking upstairs, walking downstairs, running forward, jumping up, sitting, standing, sleeping, elevator up, and elevator down. We used the pre-processing technique with a smaller window size such that the input contains six channels with 100 features (it is sampled in a sliding window of 1 second and  $50\%$  overlap, resulting in 100 features for each window). The same normalization is also applied to windows before feeding to models. We used the same setup with UCIHAR while source subjects are chosen as the last four this time.
B.2 Heart Rate Prediction
IEEE SPC This competition provided a training dataset of 12 subjects (SPC12) and a test dataset of 10 subjects [39]. The IEEE SPC dataset overall has 22 recordings of 22 subjects, ages ranging from 18 to 58 performing three different activities [83]. Each recording has sampled data from three accelerometer signals and two PPG signals along with the sampled ECG data and the sampling frequency is  $125\mathrm{Hz}$ . All these recordings were recorded from the wearable device placed on the wrist of each individual. All recordings were captured with a 2-channel pulse oximeter with green LEDs, a tri-axial accelerometer, and a chest ECG for the ground-truth HR estimation. During our experiments, we used PPG channels. We choose the first five subjects of SPC12 as source domains similar to activity recognition setup while the last six subjects of SPC22 are used for source domains to prevent overlapping subjects with SPC12.
Dalia PPG dataset for motion compensation and heart rate estimation in Daily Life Activities (DaLia) was recorded from 15 subjects (8 females, 7 males, mean age of 30.6), where each recording was approximately two hours long. PPG signals were recorded while subjects went through different
daily life activities, for instance sitting, walking, driving, cycling, working, and so on. PPG signals were recorded at a sampling rate of  $64\mathrm{Hz}$ . The first five subjects are used as source domains.
All PPG datasets are standardized as follows. Initially, a fourth-order Butterworth bandpass filter with a frequency range of  $0.5 - 4\mathrm{Hz}$  is applied to PPG signals. Subsequently, a sliding window of 8 seconds with 2-second shifts is employed for segmentation, followed by z-score normalization of each segment. Lastly, the signal is resampled to a frequency of  $25\mathrm{Hz}$  for each segment.
B.3 Cardiovascular disease (CVD) classification
CPSC China Physiological Signal Challenge 2018 (CPSC2018), held during the 7th International Conference on Biomedical Engineering and Biotechnology in Nanjing, China. This dataset consists of 6,877 (male: 3,699; female: 3,178) and 12 lead ECG recordings lasting from 6 seconds to 60 seconds with  $500\mathrm{Hz}$ . We use the original labelling [40] with one normal and eight abnormal types as follows: atrial fibrillation, first-degree atrioventricular block, left bundle branch block, right bundle branch block, premature atrial contraction, premature ventricular contraction, ST-segment depression, ST-segment elevated. We resampled recordings to  $100\mathrm{Hz}$  and excluded recordings of less than 10 seconds.
Chapman Chapman University, Shaoxing People's Hospital (Chapman) ECG dataset which provides 12-lead ECG with 10 seconds of a sampling rate of  $500\mathrm{Hz}$ . The recordings are downsampled to  $100\mathrm{Hz}$ , resulting in each ECG frame consisting of 1000 samples. The labeling setup follows the same approach as in [41] with four classes: atrial fibrillation, GSVT, sudden bradycardia, and sinus rhythm. The ECG frames are normalized to have a mean of 0 and scaled to have a standard deviation of 1. We split the dataset to  $80 - 20\%$  for training and testing as suggested in [41].
We choose leads I, II, III, and V2 during our experiments for both ECG datasets. We followed a similar setup with prior works [57] and considered each dataset as a single domain different from previous tasks. The fine-tuning of the linear layer, which is added to the frozen pre-trained encoder, is performed with  $80\%$  of the same domain.
B.4 Metrics
We used the common evaluation metrics in the literature for each task. Specifically, we used accuracy (Acc) and F1 score for activity recognition [19], mean absolute error (MAE), and root mean square error (RMSE) for heart rate prediction [39, 84], and the area under the ROC curve (AUC) for cardiovascular disease classification [57].
In this section, we explain how to calculate each metric for different time-series tasks. For activity recognition, the accuracy metric is computed by dividing the sum of true positives and true negatives by the total number of samples where a window has a single label. The MF1 score is calculated as a harmonic mean of the precision and recall where metrics are obtained globally by counting the total true positives, false negatives, and false positives similar to [19].
For heart rate prediction, the Mean Absolute Error (MAE) and Root-Mean-Square Error (RMSE) are calculated using the following equation:
$$
\mathrm {M A E} = \frac {1}{K} \sum_ {k = 1} ^ {K} | \mathrm {H R m o d e l} (k) - \mathrm {H R r e f} (k) | \tag {36}
$$
$$
\mathrm {R M S E} = \sqrt {\frac {\sum_ {k = 1} ^ {K} (\mathrm {H R m o d e l} (k) - \mathrm {H R r e f} (k)) ^ {2}}{K}}, \tag {37}
$$
where  $\mathbf{K}$  represents the total number of segments. The variables  $\mathrm{HRmodel}(k)$  and  $\mathrm{HRref}(k)$  denote the output of the model and reference heart rate values in beats-per-minute for the  $k^{th}$  segment, respectively. This performance metric is commonly used in PPG-based heart rate estimation studies [39]. The estimated heart rate values  $(\mathrm{HRmodel}(k))$  are obtained using our model, while the reference heart rate values  $(\mathrm{HRref}(k))$  are directly taken from datasets.
The AUC score for CVD classification is calculated using the one-vs-one scheme where the average AUC is computed for all possible pairwise combinations of classes for both datasets.
C Baselines
C.1 Prior Mixup Techniques
In this section, we give a detailed explanation of each mixup technique we compare our proposed method.
LinearMix We apply linear mixup as in Equation 38 to generate positive samples, if  $\mathbf{x}$  has more than one channel, mixup is applied independently for each of them.
$$
\mathbf {x} ^ {+} = \lambda \mathbf {x} + (1 - \lambda) \tilde {\mathbf {x}} \tag {38}
$$
BinaryMix We implement the binary mixup [43] by swapping the elements of  $\mathbf{x}$  with the elements of another randomly chosen sample  $\tilde{\mathbf{x}}$  as shown below.
$$
\mathbf {x} ^ {+} = \mathbf {m} \odot \mathbf {x} + (1 - \mathbf {m}) \odot \tilde {\mathbf {x}}, \tag {39}
$$
where  $\mathbf{m}$  is a binary mask sampled from a Bernoulli  $(\rho)$  with high values, and  $\odot$  stands for Hadamard product.
GeometricMix In Geometric Mixup, we create a positive sample corresponding to a sample  $\mathbf{x}$  by taking its weighted-geometric mean with another randomly chosen sample  $\tilde{\mathbf{x}}$  same as [22] as shown below.
$$
\mathbf {x} ^ {+} = \mathbf {x} ^ {\lambda} + \tilde {\mathbf {x}} ^ {(1 - \lambda)} \tag {40}
$$
CutMix Cutmix is implemented similarly to Binarymix. However, instead of changing each sample point with a probability, we cut a continuous portion using a rectangle mask  $\mathbf{M}$  from a signal  $\mathbf{x}$  and replace it with the same portion of another randomly chosen one  $\tilde{\mathbf{x}}$ . The starting point of the mask is uniformly sampled while its length is sampled from lower values such that the augmented sample is more similar to the anchor. If the signal has multiple channels, this process is applied to all channels in the same section.
$$
\mathbf {x} ^ {+} = \mathbf {M} \odot \mathbf {x} + (1 - \mathbf {M}) \odot \tilde {\mathbf {x}}, \quad \text {a n d} \quad \mathbf {M} = \operatorname {r e c t} \left(\frac {b}{a}\right), \tag {41}
$$
where  $b$  and  $a$  are the starting point and length of the rectangle wave, respectively.
AmplitudeMix AmplitudeMix is introduced for domain adaptation problems by mixing the amplitude information of images without mixing the phase of two samples [45]. In our setup, we perform amplitude mixing on the time series data across all channels while keeping the phase component unchanged. In other words, we perform the following operations.
$$
A \left(\mathbf {x} ^ {+}\right) = \lambda_ {A} A (\mathbf {x}) + (1 - \lambda_ {A}) A (\tilde {\mathbf {x}}) \quad \text {a n d} \quad P \left(\mathbf {x} ^ {+}\right) = P (\mathbf {x}) \tag {42}
$$
SpecMix We implement the SpecMix by applying CutMix to the spectrogram of time-series where the spectrogram is calculated using the short-time Fourier transform as follows.
$$
\mathrm {X} _ {k} ^ {+} = \sum_ {n = - \infty} ^ {\infty} \mathbf {x} _ {n} g [ n - m R ] e ^ {- j 2 \pi k n}, \tag {43}
$$
where  $g[n - mR]$  is an analysis window of length M with hop length of R over the signal and calculating the discrete Fourier transform (DFT) of each segment of windowed data. The length of the Fourier transform is set to the sample size of the input time series while the hop and window parameters are set to the quarter of the length.
C.2 Prior Methods for Sample Generation
In this section, we give a detailed explanation of prior methods for data generation methods.
Traditional Augmentations We apply two separate data augmentation to the anchor for creating two instances, and the encoders are trained to maximize agreement using the contrastive loss in [15]. We search mainly for augmentations that are known in state-of-the-art works [19]. The detailed augmentations are given in Table 22.
InfoMin We train a model  $g_{\theta}(.)$ , which is restricted to sample-wise  $1 \times 1$  convolutions and ReLU activations same as in [48], to decrease the mutual information between two instances. In the original paper, the input sample is split into two instances  $(X_{1}$  and  $X_{2:3}$ ) and then adversarial training is performed. As we do not have RGB channels for time-series data, we added Gaussian noise to the signal for creating other instances and then perform adversarial training.
NNCLR We follow a similar setup to SimCLR by applying two separate data augmentations, then we use nearest neighbors in the learned representation space as the positive in contrastive losses [49].
PosET We perform the dimension level mixing with extrapolation of positive features as follows:
$$
\mathbf {z} ^ {+} = \lambda \odot \mathbf {z} + (1 - \lambda) \odot \tilde {\mathbf {z}}, \tag {44}
$$
where  $\odot$  is Hadamard product, and  $\lambda \sim Beta(\alpha, \alpha)$ . We add 1 to sampled  $\lambda$  for extrapolation as in [50].
GenRep In the original implementation of GenRep, the authors use implicit generative models (IGMs) such as BigBiGAN [85] that are trained with millions of images to create the anchor and positive instance by sampling nearby latent vectors. However, as the number of samples for training is limited in time series and there is a well-trained generator for different time-series tasks, we use our trained VAE for sampling nearby latent vectors as positives. Mainly, we sample an anchor from real data, feed it to the encoder, add a Gaussian noise sampled from a truncated normal distribution, and use the output of the decoder for the positive sample with the anchor.
STAug The Spectral and Time Augmentation (STAug) method is specifically proposed for the time-series forecasting task, where the authors apply the empirical mode decomposition to decompose time series into multiple subcomponents, then reassemble these subcomponents with random weights to generate new synthetic series. Finally, in the time domain, the method uses the linear mixup to generate samples from the reassembled components. Although, the mixing coefficient sampled from a beta distribution in the original implementation, we observe significant performance decreases when the same distribution with parameters is used in our experiments, possibly due to the generated samples being far away from the anchor. We, therefore, investigate the case when the mixing coefficient is sampled from uniform distribution with high values, e.g., same as our method. Since there is no
Augmentation Bank The augmentation bank that perturbs frequency components of a time-series signal is proposed in [21] where the authors use it for unsupervised domain adaptation with a different framework than SimCLR, namely time-frequency consistency (TF-C). As it is a novel data augmentation technique, we have implemented the frequency augmentation bank as a baseline while using the SimCLR framework for a fair comparison with other methods. The authors also employed a collection of time-based augmentations for the time-domain contrastive encoder. Nonetheless, since these augmentations have already been studied in previous CL setups, we chose to exclusively utilize the frequency augmentation bank. In the paper, the authors mentioned using a small budget with low-frequency perturbations results in a performance increase, thus we chose the budget with a single frequency while choosing the  $\alpha = 0.5$  with the same settings in the paper.
DACL We perform the mixup for hidden representations, i.e., before applying projection-head, as follows.
$$
\mathbf {v} ^ {+} = \lambda \mathbf {v} + (1 - \lambda) \tilde {\mathbf {v}}, \tag {45}
$$
where  $\mathbf{v}$  is the fixed-length hidden representations of samples while  $\lambda$  is sampled from uniform distribution with high values.
IDAA We follow the original implementation of authors with their proposed VAE architecture while optimizing the adversarial strength for each time-series task. We apply the FGSM adversarial attack the same as in the original implementation [53] by perturbing the encoded representation of a sample while adding noises along the gradient sign's direction of the loss.
One setup difference between this section and the previous mixup methods is that when we compare our work with PosET, GenRep, DAGL, and IDAA, we apply the best traditional data augmentation techniques, which are used for SimCLR implementation, to the specific positive data generation mechanisms. The reason for this approach is that the original implementations of certain works indicate that the proposed methods achieve optimal results when used in conjunction with known augmentations, where our observations align with these findings.
The detailed hyperparameters for each baseline with the corresponding time series tasks are given in the following section.
D.1 Parameters for mixing
In this section, we provide the parameters that are used during our experiments. To determine the optimal parameters of the baselines for each task, we conduct a grid search. This search is performed on a small validation set taken from the largest dataset of the respective tasks, which are USC, Dalia and Chapman. We believe that this approach ensures fairness and produces more realistic results, as dataset-specific optimizations can lead to overfitting of parameters, particularly in smaller and less diverse datasets.
Table 7: Parameters for baselines
Method	Activity Recognition	Heart rate Prediction	CVD Classification
Linear Mixup	λ ~ U(0.9,1)	λ ~ U(0.9,1)	λ ~ U(0.85,1)
Binary Mixup	m ~ U(0.8,1)	m ~ U(0.9,1)	m ~ U(0.9,1)
Geometric Mixup	λ ~ U(0.9,1)	λ ~ U(0.9,1)	λ ~ U(0.9,1)
CutMix	b ~ U(0,1)	b ~ U(0,1)	b ~ U(0,1)
a ~ U(0.1,0.4)	a ~ U(0.1,0.3)	a ~ U(0.1,0.3)
AmplitudeMix	λA ~ U(0.9,1)	λA ~ U(0.9,1)	λA ~ U(0.8,1)
SpecMix	b ~ U(0,1)	b ~ U(0,1)	b ~ U(0,1)
a ~ U(0.1,0.4)	a ~ U(0.1,0.3)	a ~ U(0.1,0.3)
PosET	λ ~ Beta(2,2)	λ ~ Beta(2,2)	λ ~ Beta(2,2)
GenRep	λ ~ Nt(0,0.2,1.0)	λ ~ Nt(0,0.25,1.0)	λ ~ Nt(0,0.2,1.0)
DACL	λ ~ U(0.9,1)	λ ~ U(0.9,1)	λ ~ U(0.85,1)
IDAA	δ = 0.1	δ = 0.15	δ = 0.2
Ours	λA ~ U(0.7,1), λP ~ U(0.9,1)	λA ~ U(0.7,1), λP ~ U(0.9,1)	λA ~ U(0.7,1), λP ~ U(0.9,1)
ε = 0.7, λA, λP ~ Nt(0.9,0.1,0.9)	ε = 0.8, λA, λP ~ Nt(1,0.1,0.9)	ε = 0.7, λA, λP ~ Nt(1,0.1,0.9)
D.2 Baseline Encoder Architecture
For the baseline encoder model, we adopt the DeepConvLSTM as in [19] where the architecture has 4 convolutional layers with  $5 \times 1$  size of 64 kernels while ReLU is followed each convolution. After the convolutions, the tensor is passed through a dropout layer with a dropout rate of 0.5 to prevent overfitting. Then, the output of dropout is fed into the 2-layer LSTM with 128 units. After training
the baseline encoder, we attach a linear layer and freeze the previous layers for fine-tuning. This architecture is widely used for the datasets we used during our experiments [37, 83, 19], we therefore adopt the same network across tasks.
D.3 VAE Models
We use the total correlation variational autoencoder  $(\beta$ -TCVAE) [86] to calculate the distance between two encoded samples in the latent space. We train the model for 100 epochs with a learning rate of  $1e - 3$  while setting the batch size to 2048. The latent dimensions and the  $\beta$  values are set to 10 and 5, respectively. Below, we present the tables providing detailed information about the architectures of the encoder and decoder for datasets. The output of convolutional layers is fed to the batch normalization before the activation layer is applied. For tasks Heart rate Prediction and CVD Classification, we use task-specific encoder and decoder as the number of channels and input size for datasets in each task are the same. However, two different networks, one for UCIHAR and one for others, are designed for the Activity Recognition due to different number of input channels.
Table 8: Encoder Network for UCIHAR in Activity Recognition
Encoder
Layer Name	Output size	# of kernels	Kernel size	Stride	Activation
Input	Nx1x128x9
Convolution	Nx32x60x7	32	9x3	2x1	ReLU
Convolution	Nx32x27x5	32	7x3	2x1	ReLU
Convolution	Nx64x8x3	64	5x3	3x1	ReLU
Convolution	Nx128x2x1	128	5x3	2x1	ReLU
Convolution	Nx512x1x1	512	2x1	1x1	ReLU
Convolution	Nx20x1x1	10	1x1	1x1
Table 9: Decoder Network for UCIHAR in Activity Recognition
Decoder
Layer Name	Output size	# of kernels	Kernel size	Stride	Activation
Input	Nx1x10x1
Transposed Convolution	Nx512x2x9	512	2x9	1x1	ReLU
Transposed Convolution	Nx128x8x9	128	4x1	6x1	ReLU
Transposed Convolution	Nx64x16x9	64	4x1	2x1	ReLU
Transposed Convolution	Nx32x32x9	32	4x1	2x1	ReLU
Transposed Convolution	Nx32x64x9	32	4x1	2x1	ReLU
Transposed Convolution	Nx1x128x9	1	4x1	2x1
Table 10: Encoder Network for USC and HHAR in Activity Recognition
Encoder
Layer Name	Output size	# of kernels	Kernel size	Stride	Activation
Input	Nx1x100x6
Convolution	Nx32x46x5	32	9x2	2x1	ReLU
Convolution	Nx32x20x4	32	9x2	2x1	ReLU
Convolution	Nx64x8x3	64	5x2	2x1	ReLU
Convolution	Nx128x2x2	128	5x2	2x1	ReLU
Convolution	Nx512x1x1	512	2x2	1x1	ReLU
Convolution	Nx20x1x1	10	1x1	1x1
Table 11: Decoder Network for USC and HHAR in Activity Recognition
Decoder
Layer Name	Output size	# of kernels	Kernel size	Stride	Activation
Input	Nx1x10x1
Transposed Convolution	Nx512x2x6	512	2x6	1x1	ReLU
Transposed Convolution	Nx128x6x6	128	6x1	2x1	ReLU
Transposed Convolution	Nx64x12x6	64	4x1	2x1	ReLU
Transposed Convolution	Nx32x25x6	32	5x1	2x1	ReLU
Transposed Convolution	Nx32x50x6	32	4x1	2x1	ReLU
Transposed Convolution	Nx1x100x6	1	4x1	2x1
Table 12: Encoder Network for Heart rate Prediction
Encoder
Layer Name	Output size	# of kernels	Kernel size	Stride	Activation
Input	Nx1x200x1
Convolution	Nx32x94x1	32	13x1	2x1	ReLU
Convolution	Nx32x43x1	32	9x1	2x1	ReLU
Convolution	Nx64x18x1	64	9x1	2x1	ReLU
Convolution	Nx128x6x1	128	7x1	2x1	ReLU
Convolution	Nx512x1x1	512	5x1	2x1	ReLU
Convolution	Nx20x1x1	20	2x1	1x1
Table 13: Decoder Network for Heart rate Prediction
Decoder
Layer Name	Output size	# of kernels	Kernel size	Stride	Activation
Input	Nx1x10x1
Transposed Convolution	Nx512x6x1	512	6x1	1x1	ReLU
Transposed Convolution	Nx128x12x1	128	4x1	2x1	ReLU
Transposed Convolution	Nx64x25x1	64	5x1	2x1	ReLU
Transposed Convolution	Nx32x50x1	32	4x1	2x1	ReLU
Transposed Convolution	Nx32x100x1	32	4x1	2x1	ReLU
Transposed Convolution	Nx1x200x1	1	4x1	2x1
Table 14: Encoder Network for CVD Classification
Encoder
Layer Name	Output size	# of kernels	Kernel size	Stride	Activation
Input	Nx1x1000x4
Convolution	Nx32x330x3	32	12x2	3x1	ReLU
Convolution	Nx32x107x2	32	10x2	3x1	ReLU
Convolution	Nx64x34x1	64	8x2	3x1	ReLU
Convolution	Nx128x9x1	128	8x1	3x1	ReLU
Convolution	Nx512x1x1	512	7x1	3x1	ReLU
Convolution	Nx20x1x1	20	1x1	1x1
Table 15: Decoder Network for CVD Classification
Decoder
Layer Name	Output size	# of kernels	Kernel size	Stride	Activation
Input	Nx1x10x1
Transposed Convolution	Nx512x4x4	512	6x1	1x1	ReLU
Transposed Convolution	Nx128x12x4	128	4x1	2x1	ReLU
Transposed Convolution	Nx64x36x4	64	5x1	2x1	ReLU
Transposed Convolution	Nx32x109x4	32	4x1	2x1	ReLU
Transposed Convolution	Nx32x331x4	32	4x1	2x1	ReLU
Transposed Convolution	Nx1x1000x4	1	4x1	2x1
Figure 3: The experiment regarding the effect of phase mixup coefficients in eight datasets. a) shows the performance in activity recognition, b) is for heart rate prediction using PPG, and finally c) shows the cardiovascular disease classification
Figure 4: The experiment regarding the effect of amplitude mixup coefficients in eight datasets. a) shows the performance in activity recognition, b) is for heart rate prediction using PPG, and finally c) shows the cardiovascular disease classification
E.1 The effect and robustness of mixing coefficients
In this section, our experiments focus on observing the impact of a diverse range of mixing coefficients for both phase and amplitude components. We decrease the lower threshold of distributions for sampling the mixing coefficient by 0.1 and 0.2. For example, normally the phase mixup coefficient for Activity Recognition is sampled from truncated normal  $\lambda_{P} \sim \mathcal{N}^{t}(1,0.1,0.9)$  and uniform  $\lambda_{P} \sim U(0.9,1)$ . We decrease the low threshold value from 0.9 to 0.8 and 0.7 and report the results for both phase and amplitude. The results are reported in Figures 3 and 4 for eight datasets.
From Figures 3 and 4, it can be inferred that the phase component is more sensitive to the changes. In other words, a significant decrease in performance is observed when the mixing coefficients for the phase are sampled from lower values whereas this effect is not as much as severe for the amplitude coefficient, indicating that the amplitude of frequencies is more robust to changes compared to phase.
E.2 The performance in other frameworks