SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Acoustic-seismic identification Data Set (ACIDS): ACIDS is an ideal dataset for developing and training acoustic/seismic classification/ID algorithms. The data was collected by 2 co-located acoustic/seismic sensor systems. There are over 270 data runs (single target only) from 9 different types of ground vehicles in 3 different environmental conditions. The ground vehicles were traveling at constant speeds from one direction toward the sensor systems passing the closest point of approach (CPA) and then away from the sensor systems. The microphone data is low-pass filtered at  $400\mathrm{Hz}$  via a 6th-order filter to prevent spectral aliasing and high-pass filtered at  $25\mathrm{Hz}$  via a 1st-order filter to reduce wind noise. The data is digitized by a 16-bit A/D at the rate of  $1025\mathrm{Hz}$ . The CPA to the sensor systems varied from  $25\mathrm{m}$  to  $100\mathrm{m}$ . The speed varied from  $5\mathrm{km/hr}$  to  $40\mathrm{km/hr}$  depending upon the particular run, the vehicle, and the environmental condition. We randomly partition the runs into training, validation, and testing datasets with a ratio of 8:1:1. It is more challenging than MOD since domain shift caused by vehicle speed, distance, or terrain between training and testing can be included. No information related to the target types is revealed except the numerical labels.
RealWorld-HAR [18]: This is a public dataset using the accelerometer, gyroscope, magnetometer, and light signals to recognize 8 common human activities (climbing stairs down and up, jumping, lying, standing, sitting, running/jogging, and walking) from 15 subjects. Only the data collected from "waist" is used in our experiments. The sampling rate of all selected sensors is  $100\mathrm{Hz}$ . We use the
<sup>4</sup>The work was deemed Not Human Subjects Research (NHSR) because the purpose of the experiment was to test the performance of an AI algorithm in the presence of noise, as opposed to collecting data about humans. The humans who assisted with the experiment, in essence, acted as "lab technicians" who operate machinery for experimental purposes.
leave-one-out evaluation strategy where 10 random subjects are used for training, 2 subjects are used for validation, and 3 subjects are used for testing.
Physical Activity Monitoring dataset (PAMAP2) [16]: This dataset contains data of 18 different physical activities (e.g., walking, cycling, playing soccer, etc) performed by 9 subjects using inertial measurement units (IMUs) that are put at the chest, wrist (of dominant arm), and dominant side's ankle respectively. Only data collected from the "wrist" is used in our experiment. Each IMU records readings from a 3-axis accelerometer, gyroscope, and magnetometer. The sampling rates of all sensors are  $100\mathrm{Hz}$ . We use the leave-one-out evaluation strategy where 7 random subjects are used for training, and 2 subjects are used for testing.
B Data Preprocessing
In our data preprocessing, we first divide the time-series data into equal-length data samples and further segment each sample into overlapped/non-overlapped intervals. The signals within each interval are processed by the Fourier transform to obtain the spectrum. In this way, both the time-domain information and frequency-domain patterns are preserved. The generated time-frequency spectrogram is further fed into the backbone feature encoders. We define a set of data augmentations in both the time domain before the Fourier transform and the frequency domain after the Fourier transform. For each sample, only one random augmentation from either the time domain or the frequency domain is selected and applied. To further increase the randomness of data augmentations in multimodal applications, we let each modality have a probability of 0.5 to be processed by the selected random augmentation.
B.1 Data Augmentations
We follow the common practices in [22, 7, 10, 19] to define the augmentations used in the time domain and frequency domain respectively.
B.1.1 Time-Domain Augmentations
Here we list the used time-domain augmentations.
- Scaling: We multiply the input signals with values sampled from a Gaussian distribution.
- Permutation: Given intervals within a sample, we randomly permute the order of the intervals.
- Negation: The signal values are multiplied by a factor of -1.
- Time Warp: Randomly stretching/distorting the time locations of the signal values based on a smooth random curve.
- Magnitude Warp: The magnitude of each time series is multiplied by a curve created by cubicspline with a set number of knots at random magnitudes.
- Horizontal Flip: The entire time series of the sample is flipped in the time direction.
- Jitter: We add random Gaussian noise to signals.
- Channel Shuffle: We randomly shuffle the channels of multi-variate time-series data (e.g., X, Y, Z dimensions of three-axis accelerometer input).
- Time Masking: We randomly mask a portion of the time intervals within a sample window with 0.
B.1.2 Frequency-Domain Augmentations
Here we list the used frequency-domain augmentations.
- Phase Shift: Given the complex frequency spectrum, we add a random value between  $-\pi$  to  $\pi$  to their phase values.
- Frequency Masking: We randomly mask a portion of frequency ranges with 0.
Supervised: We train the whole model including the encoder and linear classifier in a fully supervised manner using all available labels.
SimCLR [1] is a simple yet powerful contrastive learning framework proposed for vision tasks. For this work, we randomly formulate batches. During pretraining, we apply random augmentations to generate two different views of each sample, with a contrastive objective of bringing different transformations (augmentations) of the same samples closer while repelling the representations of different samples. The framework optimizes the parameters of the underlying backbone model by minimizing the NT-Xent loss [1]. Similar to [1], we take different samples from the same minibatch as the negative samples. That is, different views of the same sample are considered positive pairs, while views generated from different samples are considered negative pairs.
MoCoV3 [2] is a SOTA contrastive learning framework for Vision Transformers (ViT). It leverages a query encoder  $f_{q}$  and a key momentum encoder  $f_{k}$  on two stochastically augmented views of a sample to output a query vector  $q$  and a key vector  $k$ . It uses random batch sampling and learns by maximizing the agreement between the positive encoded query and an encoded key pair. In the latest version of MoCo (V3), for a given query  $q$ , the positive key  $k^{+}$  is encoded from the same sample as  $q$ , while the negative labels  $k^{-}$  are encoded keys of other samples within the same mini-batch. Both encoders have a similar structure including a backbone plus a projection head, and the query encoder  $f_{q}$  has an additional projection head at the end. The key momentum encoder  $f_{k}$  is slowly updated by a query momentum with the query encoder  $f_{q}$ .
CMC [20] is a contrastive learning framework focusing on learning from multiview observations. It learns meaningful data representations by contrasting the encoded features from different modalities. To achieve this, it maximizes the agreement between the synchronized representations of different modalities. For each randomly sampled batch with a random augmentation, the backbone model extracts vector representations of each modality. Then, for each pair of modalities, we maximize the similarity between modality representations of the same samples and regard mismatched modality representations from different samples as negative pairs. We sum up the losses for all pairs of modalities to optimize the backbone parameters. For downstream tasks, a linear classification layer is applied on top of concatenated modality representations.
MAE [6] is a self-supervised learning approach based on the auto-encoding paradigm. It incorporates the Transformer architecture and achieves SOTA performance on multiple vision tasks. Unlike contrastive learning, MAE does not depend heavily on random augmentations. During the pretraining, we randomly mask a significant portion (i.e.,  $75\%$ ) of each modality input. Instead of dropping the masked patches as in the original MAE paper, we replace them with 0 values to ensure consistent dimensions for the Swin-Transformer and DeepSense operations. A separate encoder and decoder are used for each modality. Before encoding, the modality spectrogram is first projected into fixed-size (e.g., 2x2) patches through a convolutional layer, on top of which the modality embeddings are extracted by the modality encoder. Between independent modality encoding and decoding, we first apply multiple fully-connected layers to the concatenated modality features for modality information fusion and then use separate MLP projection layers to get the projected modality embeddings before decoding. This step is created to enable interactions between modalities. Finally, the modality decoder reconstructs the modality input from the projected modality embeddings. The overall objective is to minimize the mean squared error (MSE) between the original modality patches and the reconstructed modality patches on the masked locations. During the inference, the modality decoders are dropped and only modality encoders are used to extract the latent representations from unmasked modality input. In the end, a linear classification layer is applied to the concatenated modality embeddings to serve the downstream task.
Cosmo [14] focuses on contrastive fusion learning from multimodal time-series data to extract modality-consistent information. Cosmo applies separate modality encoders to extract the embedding vector of each modality from the randomly sampled mini-batches. After encoding, each modality embedding is mapped to a hypersphere through an MLP projector and a normalization layer. Then, Cosmo applies a fusion-based feature augmentation to generate  $P$  randomly combined features by multiplying the modality embeddings with  $P$  normalized random weight vectors. When calculating contrastive loss, these  $P$  fusion-based augmented features are considered as positive pairs, while features generated through the same approach but from different samples are treated as negative pairs.
Cocoa [3] extends the self-supervised learning of multimodal sensing data by exploring both the cross-modal correlation and intra-modal separation. Similar to other modality-level contrastive frameworks, Cocoa applies a separate backbone encoder to extract the latent embedding of each modality from the randomly sampled and augmented mini-batch. Cocoa has two losses: Cross-modality correlation loss and discriminator loss. Cross-modality correlation loss maximizes the consistency between different modality embeddings corresponding to the same sample by defining them as hard positive pairs. On the contrary, discriminator loss tends to minimize the agreement within a modality, by separating modality embeddings of irrelevant samples within the mini-batch from each other.
GMC [15] introduces a multimodal contrastive loss function that encourages the geometric alignment of different modality embeddings. Similar to other multimodal contrastive frameworks, samples are randomly batched and augmented. GMC consists of modality-specific encoders and a joint encoder that simultaneously takes all modality data as input. An additional linear layer is used to map the joint embedding to the same space as individual modality embeddings. Then, a shared projection head is then employed to project both the modality embeddings and the joint embeddings before calculating the contrastive loss. To align the local views (i.e., individual modality embeddings) with the global view (i.e., joint embeddings) in a context-aware manner, GMC minimizes a multimodal contrastive NT-Xent loss by defining the modality-specific embeddings and joint embeddings of the same samples as positive pairs, while treating local-global embedding pairs from different samples as negative pairs.
MTSS [17] is a predictive self-supervised learning framework by exploiting the distinguishability among different data transformations. It uses random augmentation ID prediction as the pretext task during the pretraining. Specifically, MTSS first formulates random batches and applies random augmentation to either time or frequency domain. Each modality is augmented with the selected random augmentation with a probability of  $50\%$ . Then, individual modality encoders extract modality embeddings from their input, followed by modality fusion to compute the overall sample embeddings. Different from contrastive frameworks, a shallow classifier is included to classify "which random augmentation is applied to the input". A cross-entropy loss is calculated between the predicted augmentation ID and the actual augmentation ID as the pertaining objective. For downstream tasks, only the backbone sample encoder (including the modality encoder and modality fusion layers) is used to extract the sample embeddings, along with a linear classification layer appended at the end of the sample encoder.
TS2Vec [24] proposes to learn representations of time series by simultaneously performing temporal contrastive tasks and instance contrastive tasks at multiple granularities (i.e., lengths of sample windows). Instead of creating random batch samples, TS2Vec involves randomly sampled sequences in each batch, with each sequence containing temporally close samples. TS2Vec employs a hierarchical contrasting method to learn representations at multiple sample window granularities. It always regards the same sample under different augmentations and sequence contexts as the positive pairs, while in the instance contrastive task, different samples from separate sequences are regarded as negative pairs, and in the temporal contrastive task, different samples within the same sequence are regarded as negative pairs. At each sample window level, TS2Vec computes both the temporal contrastive loss and instance discrimination loss.
TNC [21] learns time series representations with a debiased contrastive objective to distinguish samples within the temporal neighborhood from temporally distant samples. It utilizes a backbone encoder to extract the feature representations from the time series data in a randomly sampled sequence batch. For each sample, TNC identifies a group of samples with similar timestamps as neighboring samples and a group of distant samples as non- neighboring samples. In this paper, we consider samples within the same sequence as the neighboring samples and samples from different sequences as non- neighboring samples. A discriminator is used to learn the time series distribution by predicting the probability of each sample and its neighboring/non- neighboring samples being in the same window. The objective is to maximize the similarity of neighboring samples while pushing the similarity of non- neighboring samples to zero.
TS-TCC [4] learns robust representation by performing cross-view predictions and contrasting both temporal and contextual information. It randomly groups multiple sequences into a mini-batch. It first generates two views through random augmentations on each sample. For each view, it extracts context vectors of each timestamp from all sample representations up to this timestamp within the sequence with an autoregressive model and then uses the context vectors from one view to predict
Table 6: DeepSense Configurations.
Dataset	MOD	ACIDS	RealWorld-HAR	PAMAP2
Dropout Ratio	0.2	0.2	0.2	0.2
Mod Conv Kernel	aud: [1, 5], sei: [1,3]	[1,4]	[1, 3]	[1, 5]
Mod Conv Channel	128	128	128	64
Mod Conv Layers	5	6	6	4
Recurrent Dim	256	128	256	64
Recurrent Layers	2	2	2	2
FC Dim	512	256	256	128
the future timesteps of the other view. In the temporal contrastive task, given cross-view predicted representations at a future timestamp, it regards the true future representation at that timestamp from the same sequence as the positive pair and regards samples at that timestamp from other sequences as negative pairs. In the contextual contrastive task, TS-TCC calculates NT-Xent loss by considering different augmentations of the same sample as positive pairs and considering different samples within the same mini-batch as negative pairs.
D Backbone Models
We tested with two different backbone encoders in this paper: DeepSense and Swin-Transformer (SW-T for short). Both models process the spectrogram of each input sensing modality separately, before the information fusion between the sensing modalities. For each backbone model, the configuration is tuned to achieve the best-supervised model accuracy.
DeepSense [23]: It is a state-of-the-art neural network model for time-series sensing data processing. Given the time-frequency spectrogram of each sensing modality, it first uses stacked convolutional layers to extract localized modality features within each time interval. Then, modality information fusion is performed by taking the mean of flattened modality features. Finally, the features across time intervals are aggregated through recurrent layers (e.g., Gated Recurrent Unit (GRU)). For learning frameworks that operate on modality-level features (i.e., FOCAL, CMC, Cosmo, Cocoa, and MAE), we skip the mean fusion among modalities and use individual recurrent layers for each modality, before calculating the pretrain loss.
Swin-Transformer (SW-T) [11]: It is a state-of-the-art Transformer model for processing image data. We adapt it to process the time-frequency spectrogram input. Similar to convolution operations, it adaptively allocates attention within subframe windows of input with hierarchical resolutions. The modality input is first partitioned into patches with a convolutional layer. Then, it gradually extracts features from local and shifted windows with multiple blocks. The shift window operation is introduced to break the boundary of partitioned windows and increase the perception area of each window. Each block consists of multiple self-attention layers. The patch resolution of the feature map is halved at the end of each block by merging neighboring patches while the channel number is doubled, such that the receptive field increases as going into deeper layers while the number of patches within each window is fixed. A separate SW-T encoder is used to extract features from each modality input, after which a stack of self-attention layers is appended for information fusion from multiple modalities. Similarly, for learning frameworks that operate on modality-level features, we skip the attention-based fusion blocks and directly calculate pretrain losses on top of modality features.
E Training Configurations
In this section, we detail the training strategies used in this paper, which are summarized in Table 8. For each framework, the same configuration is mostly shared between different backbone encoders with few exceptions.
During the pertaining, we use the AdamW [12] optimizer with the cosine schedules [13]. The start learning rate is tuned accordingly for each framework according to their convergence situation. We did observe Cosmo [14] is hard to converge in some cases thus we have to reduce its start learning rate. The used batch size is 256, where 64 short sequences of 4 samples are randomly selected in each batch. The constitution of sequences is determined at the initialization and does not change over
Table 7: Swin-Transformer Configurations.
Dataset	MOD	ACIDS	RealWorld-HAR	PAMAP2
Dropout Ratio	0.2	0.2	0.2	0.2
Patch Size	aud: [1, 40], sei: [1,1]	[1, 8]	[1, 2]	[1, 2]
Window Size	[3, 3]	[2,4]	[3, 3]	[3, 5]
Mod Feature Block Num	[2, 2, 4]	[2, 2, 4]	[2, 2, 2]	[2, 2, 2]
Mod Feature Block Channels	[64, 128, 256]	[64, 128, 256]	[32, 64, 128]	[32, 64, 128]
Head Num	4	4	4	4
Mod Fusion Channel	256	256	128	128
Mod Fusion Head Num	4	4	4	4
Mod Fusion Block	2	2	2	2
FC Dim	512	512	256	128
Table 8: Training configurations. (We use LR for Learning Rate)
Dataset	MOD	ACIDS	RealWorld-HAR	PAMAP2
Temperature	0.07	0.2	0.07	0.07
Batch Size	256	256	256	256
Sequence Length	4	4	4	4
Pretrain Optimizer	AdamW	AdamW	AdamW	AdamW
Pretrain Max LR	Default: 1e-4Cosmo, TNC, GMC, TS2Vec, TSTCC: 1e-5	Default: 1e-4Cosmo: 1e-5	Default: 1e-4CMC, GMC: 5e-4Cosmo: 1e-5	Default: 1e-4CMC, GMC: 5e-4Cosmo: 1e-5
Pretrain Min LR	1e-07	1e-07	1e-07	1e-07
Pretrain Scheduler	Cosine	Cosine	Cosine	Cosine
Pretrain Epochs	6000	3000	1000	1000
Pretrain Weight Decay	0.05	0.05	0.05	0.05
Finetune Optimizer	Adam	Adam	Adam	Adam
Finetune Start LR	0.001	0.0003	0..001	0.001
Finetune Scheduler	step	step	step	step
Finetune LR Decay	0.2	0.2	0.2	0.2
Finetune LR Period	50	50	50	50
Finetune Epochs	200	200	200	200
training epochs. The temperature is tuned to achieve the best linear classification performance after the finetuning. A weight decay of 0.05 is used as the training regularization.
During the finetuning, we use the Adam [9] optimizer with the step scheduler. Essentially, the learning rate decays by 0.2 at the end of each period. By default, finetuning runs for 200 epochs in total, and each period is 50 epochs. Besides, the weight decay parameter is separately tuned for each framework for the best balance between training fit and validation fit.
The models are trained on a lab workstation with AMD Threadripper PRO 3000WX Processor of 64 cores and NVIDIA RTX 3090 GPUs. The implementation is based on PyTorch 1.14, and the pretraining on a single GPU spans between 3 hours to 4 days among different datasets and backbone encoders.
F Additional Evaluation Results
In this section, we report additional evaluation results and analyses that are not included in the main paper.
F.1 Finetuning: Complete Linear Classification Results
Setup: For each dataset, we apply two backbone encoders (DeepSense and SW-T), and finetune the linear classifier with three different ratios of available labels (100%, 10%, and 1%). For label ratios 10% and 1%, we take 5 random portions of labels for finetuning in each training framework and report the mean and standard deviation among the runs with all testing data. The best result under each configuration is highlighted with the **bold** text. Besides, we also train a supervised model for each configuration as a reference to the self-supervised frameworks.
Analysis: Table 9, Table 10, Table 11, and Table 12 summarize the complete linear finetuning results on MOD, ACIDS, RealWorld-HAR, and PAMAP2 datasets, respectively.