SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(both in  $1025\mathrm{Hz}$ ). It includes data on 9 types of ground vehicles in 3 different terrains. (3) RealWorld-HAR [46] is a public dataset using accelerometer, gyroscope, magnetometer, and light signals (all in  $50\mathrm{Hz}$ ) to recognize 8 common human activities. (4) PAMAP2 [41] is another public dataset using accelerometer, gyroscope, and magnetometer signals (all in  $100\mathrm{Hz}$ ) to recognize 18 different physical activities. The length of the samples and the intervals, as well as the time overlap ratios between intervals within samples of each dataset, as listed in Table 1, are configured to achieve the best-supervised classification performance.
Data Augmentations: Candidate augmentations are defined in both the time domain before STFT and the frequency domain after STFT, where only one out of them is randomly selected in each forward pass. Time-domain augmentations include scaling, permutation, negation, time warp, magnitude warp, horizontal flip, jitter, channel shuffle, and time masking; frequency-domain augmentations include phase shift and frequency masking. Details can be found in Appendix B.
Baselines: We consider 12 baselines in total, including a supervised benchmark, three representative self-supervised learning frameworks from vision tasks that perform instance discrimination (SimCLR [3], MoCoV3 [5], and MAE [13]), four modality-matching contrastive frameworks for multimodal input (CMC [48], Cosmo [35], Cocoa [6], GMC [37]), three SOTA contrastive frameworks for time series (TS2Vec [63], TNC [49], TS-TCC [8]), and one predictive baseline (MTSS [42]). Their detailed introductions can be found in Appendix C. All compared frameworks use the same encoder structures (except for minor differences in module orders). The evaluation metrics are accuracy and (macro) F1 score.
Backbone Encoders: We apply two different modality encoders in this paper. (1) DeepSense [59] uses convolutional layers to extract localized feature patterns within each time interval, and then uses recurrent layers to aggregate information across all intervals within a sample window. (2) SwinTransformer (SW-T) [26] uses stacked Transformer blocks to extract local information from shifted windows in a hierarchical manner. Their details and configurations can be found in Appendix D.
Figure 4: Finetuning results with KNN classifiers  $(\mathrm{K} = 5)$ , with SW-T encoder.
Both models are implemented in PyTorch 1.14 and optimized with an AdamW [28] optimizer under cosine learning rate schedules [27]. Without special specifications, we report the results on SW-T backbone. Detailed training configurations are introduced in Appendix E.
Finetune Classifiers: We evaluate with two lightweight classifiers during the finetune stage. Linear probing adds a linear layer on top of the pretrained sample features (or concatenated modality features) with the encoder fixed during fine-tuning. KNN classifier directly uses the voting of 5 nearest neighbors (based on the Euclidean distances between samples) in the finetuning training samples to generate the predicted labels for the testing samples.
4.2 Finetune Results
Linear Probing: Table 2 presents the finetune results using a linear classifier (The complete results can be found in Appendix F). FOCAL consistently achieves the best accuracy and F1 score across different datasets and backbone encoders. It outperforms CMC by  $4.48\%$  to  $18.01\%$  in accuracy, demonstrating the importance of extracting private modality information in addition to the shared information. SimCLR and MoCo achieved suboptimal performance on HAR datasets (i.e., RealWorld-HAR and PAMAP2) which have more than two modalities, showing that instance discrimination alone as the pretext task can not learn comprehensive feature patterns from multiple modalities. GMC, as the SOTA framework for multi-modal time series, beats the contrastive frameworks for single-modal time series (TS2Vec, TNC, and TS-TCC) in most cases, but is secondary to FOCAL. On MOD dataset, FOCAL achieves higher accuracy and F1 score than the supervised model by taking advantage of massive unlabeled samples.
Besides, we plot the finetune results on the MOD dataset under three different label ratios (100%, 10%, and 1%) in Figure 3. FOCAL overall achieves better label efficiency during the finetuning. It attains 3.51% relative improvement with 100% labels and 10.56% relative improvement with 1% labels, compared to the best-performing baseline. Among the baselines, TNC performs best with 1% label but does not generalize well when more labels are available (e.g., 100%).
Figure 3: Linear probing under different label ratios on MOD with SW-T encoder.
KNN Evaluation: The evaluation results with a KNN
classifier  $(\mathrm{K} = 5)$  using all available labels<sup>3</sup> are plotted in Figure 4 (See Appendix F for complete results). KNN operates in a non-parametric way and can be expressive if the learned latent space aligns well with the underlying semantics. For multi-modal contrastive frameworks (i.e., FOCAL, CMC, Cosmo, Cocoa, GMC), we concatenate the modality features as the sample feature. FOCAL consistently performs better than the baselines across all datasets, proving its representational power in the non-parametric classification setting.
4.3 Additional Downstream Tasks
**Clustering:** In multi-modal learning, the clusterability of the learned modality representations is preferable. Meaningful representation clusters should be coherent with the underlying semantic labels. For each modality, we first use K-means clustering with pretrained modality embeddings to get the cluster labels and measure the consistency between the cluster labels and the ground-truth category labels, with two common clustering metrics adjusted rand score (ARI) [45] and normalized mutual information score (NMI) [1]. The results for five multi-modal frameworks are presented in Figure 5.
3 In MOD, 7,335 samples out of the pretraining set are labeled. All other datasets are fully labeled.
Figure 5: Clustering evaluation results. We use SW-T as the backbone encoder. The mean and standard deviation among sensory modalities are reported. Higher scores are better.
(a) FOCAL
(b) CMC
(c) GMC
Figure 6: t-SNE visualization of the concatenated modality features (DeepSense encoder, MOD dataset). Different colors represent different object classes.
(d) Cocoa
Table 3: Ablation Results with SW-T Encoder and Linear Classifier.
Metrics	MOD	ACIDS	RealWorld-HAR	PAMAP2
Acc	F1	Acc	F1	Acc	F1	Acc	F1
FOCAL-noPrivate	0.9296	0.9284	0.7981	0.7100	0.8869	0.8768	0.7938	0.7787
FOCAL-noOrth	0.9705	0.9692	0.9311	0.8261	0.9186	0.9257	0.8371	0.8233
FOCAL-wDistInd	0.5773	0.5502	0.4926	0.4157	0.9099	0.9084	0.6518	0.5503
FOCAL-noTemp	0.9671	0.9659	0.9456	0.8014	0.9361	0.9425	0.8367	0.8255
FOCAL-wTempCon	0.9363	0.9359	0.9287	0.7587	0.8793	0.8842	0.8391	0.8242
FOCAL	0.9805	0.9800	0.9489	0.8262	0.9451	0.9503	0.8580	0.8401
We separately evaluate the clusterability of each modality and report the mean and standard deviation among modalities. FOCAL is superior in learning representations better aligned with the target labels. Although CMC performs closely to FOCAL on MOD and RealWorld-HAR datasets, it performs poorly on ACIDS and PAMAP2 datasets. Besides, we qualitatively visualize the concatenated sample embeddings of the multi-modal contrastive frameworks after t-SNE [51] dimension reduction on MOD dataset in Figure 6. We can see that FOCAL achieved better separation among different classes than the compared baselines.
Distance and Speed Classification: In the MOD dataset, we separately collect new data samples recording the distance and speed of the vehicles. Data in this experiment is collected in a different environment with different vehicles from the pretraining data, thus accounting for potential domain shifts. The same pretrained models in previous experiments are used, and the results are summarized in Figure 7. The instance discrimination frameworks (i.e., SimCLR and MoCo) are relatively more resilient than the modality
matching frameworks (i.e., CMC, Cocoa, Cosmo, and GMC), while FOCAL fills this gap by capturing the transformation consistency information in private modality spaces.
Figure 7: Additional downstream tasks on MOD dataset.
Table 4: Benefits of Temporal Constraints to SOTA baselines on ACIDS
Metrics	SimCLR	MoCo	CMC	Cocoa	GMC
Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
wTemp	0.7461	0.6938	0.7836	0.6618	0.8690	0.7090	0.8543	0.7665	0.9347	0.8109
Vanilla	0.7438	0.6101	0.7717	0.6205	0.8443	0.7244	0.6644	0.5359	0.9096	0.7929
Table 5: Benefits of Temporal Constraints to SOTA baselines on PAMAP2
Metrics	SimCLR	MoCo	CMC	Cocoa	GMC
Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
wTemp	0.7129	0.6884	0.7800	0.7602	0.7804	0.7583	0.8442	0.8146	0.8253	0.8114
Vanilla	0.6802	0.6583	0.7559	0.7387	0.7906	0.7706	0.7603	0.7187	0.8119	0.7860
(a) Lambda private
(b) Lambda orthogonal
(c) Lambda temporal
(d) Margin
Setup: Here we compare with five variants of FOCAL, including FOCAL-noPrivate (no private space), FOCAL-noOrth (no orthogonality constraint), FOCAL-wDistInd (replace orthogonality with distributional independence), FOCAL-noTemp (no temporal structural constraint), FOCALwTempCon (replace temporal structural constraint with the temporal contrastive task).
Figure 9: Loss weights sensitivity test.
(a) MOD
Figure 8: Convergence curves.
(b) PAMAP2
Analysis: The ablation study results are summarized
in Table 3. First, the accuracy decreases by  $5.20\% - 5.90\%$  after removing the private space and the related contrastive task, since only the shared information among modalities is leveraged. Second, on top of the private space task, both the orthogonality constraint and the temporal structural constraint further improve the accuracy by  $1.39\% - 2.99\%$ , which proves they both contribute positively to FOCAL. Third, replacing the geometrical orthogonality constraint with the distributional independence constraint causes significant degradation and may even cause the model training to collapse. Similarly, in our experiments, conducting temporal contrastive tasks leads to a noticeable accuracy decrease in three out of four datasets. To further demonstrate the contribution of our temporal structural constraint in accelerating the convergence during the pretraining, we use KNN classifier to periodically evaluate the quality of the learned representations. Concatenated modality embeddings are used as sample representations. We visualize the achieved KNN accuracy curves on MOD and PAMAP2 with and without the temporal constraint in Figure 8. The temporal constraint clearly improves the semantical structure of the learned embeddings in early epochs of pretraining, by rejecting obviously counter-intuitive parameter values that violate the constraint.
4.5 General Applicability of the Temporal Constraint
To validate the general applicability of the proposed temporal constraint, we apply it to multiple contrastive learning baselines (i.e., SimCLR, MoCo, CMC, Cocoa, and GMC). Table 4 and 5 summarize the results on ACIDS and PAMAP2 respectively, and we have observed noticeable performance improvement in most cases (up to  $18.99\%$  on ACIDS and up to  $8.39\%$  on PAMAP2). It demonstrates that the temporal constraint can be used as a plugin to enhance existing contrastive learning frameworks for time-series data.
4.6 Sensitivity Test on Loss Weights
We also perform a sensitivity test on the loss weight values and plot the performance of FOCAL against different hyperparameters in Figure 9. We observe that FOCAL is generally robust against the
hyperparameter selections, with less than  $2\%$  accuracy fluctuations in all cases. For this reason, we did not perform a comprehensive hyperparameter search in our experiment. Besides, combining our observations in the ablation study that the private space task, orthogonality constraint, and temporal constraint all contribute positively to the performance of FOCAL, we conclude that the competition between learning objectives does not happen in FOCAL.
Self-Supervised learning for multimodal data. Self-supervised learning from multimodal data has been extensively studied in the literature, including contrastive learning [48, 35, 6, 37, 38, 25, 32, 2], masked autoencoders (MAE) [13, 11, 19], and variational autoencoder (VAE) [18, 50, 21], surpassing conventional generation-based semi-supervised learning approaches [61]. As a leading paradigm in learning transferable and discriminative features [66, 24, 57] with a variety of successful applications [16, 29, 40, 30, 47], we mainly consider contrastive learning in this paper. On one hand, as we show in the experimental results, normal contrastive frameworks based on instance discriminations (SimCLR [20], MoCO [5], BYOL [12], SimSiam [4]) may lead to suboptimal results without accommodating the multimodal properties. On the other hand, existing contrastive learning frameworks for multimodal data [38, 48, 35, 6, 37] mostly focus on the consistency between modalities but ignore the information heterogeneity when they are collaboratively utilized in sensing tasks. CLIP [38], as a representative multimodal contrastive framework, mainly addresses the importance of using natural language to augment the visual models through learning their pairings. Similarly, both CMC [48] and GMC [37] highlight the information matching between modality embedding to another modality embedding or the joint embedding. Instead, in FOCAL, we simultaneously consider modality consistency and discriminative modality-exclusive information by designing corresponding contrastive tasks and imposing the information independence constraint. Besides, most existing multimodal contrastive learning frameworks are limited to vision and language modalities [62, 22, 47, 64, 31], while the domain knowledge might not be directly applicable to sensory modalities with time-series signals in IoT applications.
Contrastive learning for time series. There has been increasing interest in developing contrastive learning frameworks for time-series data [10, 63, 44, 7, 49, 22, 65, 36]. TS2Vec [63], TFC [65], TNC [49], and TS-TCC [7] were based on the time-series properties but did not consider the multimodal collaboration properties. Besides, TFC [65] and RF-URL [44] were designed from the consistency between time and frequency representations, or different time-frequency representations, while restricting the backbone encoder structures. Cosmo [35] and Cocoa [6] are the most recent attempts at contrastive learning from multi-modal time series, but they were not able to sufficiently utilize the complementary information from sensory modalities. CoST [54] proposed to separate the seasonal-trend representations for time series forecasting. Different from the existing works, the objective of FOCAL is to maximally extract complementary and discriminative features from multimodal sensory modalities, to facilitate the downstream recognition tasks. CLUDA [36] worked on the unsupervised domain adaptation (UDA) problem for time series, which is not directly comparable to our solution. In FOCAL, no expert features are assumed to be available as in ExpCLR [34].
We proposed a novel contrastive learning framework, FOCAL, for self-supervised learning from multimodal time-series sensing signals. FOCAL encodes each modality input into a factorized orthogonal latent space including shared features and private features. We learn each part by applying different contrastive learning objectives addressing the modality consistency in the shared space and the transformation invariance in the private space. Besides, we design a lightweight temporal structural constraint as an information regularization during the pretraining. Extensive evaluations on four multimodal sensing datasets with two encoder networks and two lightweight classifiers, demonstrate the superiority of FOCAL over 11 SOTA baselines, under different levels of supervision. Our future work would focus on extracting domain-invariant features in multi-vantage sensing applications to make the pretrained model resilient against task-unrelated environmental factors (e.g., terrain, wind, sensor-facing directions).
Research reported in this paper was sponsored in part by the Army Research Laboratory under Cooperative Agreement W911NF-17-20196, NSF CNS 20-38817, National Natural Science Foundation of China (No. 62332014), DARPA award HR001121C0165, DARPA award HR00112290105, DoD Basic Research Office award HQ00342110002, and the Boeing Company. Shengzhong Liu is also sponsored by the "Double-First Class" Startup Research Funding of Shanghai Jiao Tong University. The views and conclusions contained in this document are those of the author(s) and should not be interpreted as representing the official policies of the CCDC Army Research Laboratory, or the US government. The US government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation hereon.
[1] Alessia Amelio and Clara Pizzuti. Is normalized mutual information a fair measure for comparing community detection methods? In IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 2015.
[2] Razvan Brinzea, Bulat Khaertdinov, and Stylianos Asteriadis. Contrastive learning with cross-modal knowledge mining for multimodal human activity recognition. In International Joint Conference on Neural Networks (IJCNN), 2022.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.
[4] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[5] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision (CVPR), 2021.
[6] Shohreh Deldari, Hao Xue, Aaqib Saeed, Daniel V. Smith, and Flora D. Salim. Cocoa: Cross modality contrastive learning for sensor data. ACM Interact. Mob. Wearable Ubiquitous Technol. (IMWUT), 6(3), 2022.
[7] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
[8] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. arXiv preprint arXiv:2208.06616, 2022.
[9] Shengyu Feng, Baoyu Jing, Yada Zhu, and Hanghang Tong. Adversarial graph contrastive learning with information regularization. In International World Wide Web Conference (WWW), 2022.
[10] Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[11] Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022.
[12] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[62] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[63] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAI Conference on Artificial Intelligence (AAAI), 2022.
[64] Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich, and Dietrich Klakow. MCSE: Multimodal contrastive learning of sentence embeddings. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2022.
[65] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[66] Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In International Conference on Learning Representations (ICLR), 2021.
The appendix of this paper is structured as follows.
Section A introduces the evaluated datasets and their statistics.
- Section B introduces the preprocessing procedure in our experiments.
- Section C details the compared baselines in our experiments.
- Section D introduces the used backbone model structure and their related configurations.
- Section E summarizes the training strategies we applied in this paper.
- Section F presents more comprehensive evaluation results.
- Section G lists the existing limitations of the work and potential solutions for future extensions.
The basic statistics for each dataset are summarized in Table 1.
Moving Object Detection (MOD): This is a self-collected dataset using sensor nodes consisting of a RaspberryShake 4D (from https://raspberryshake.org/) and a microphone array to collect the vibration signals caused by nearby moving vehicles. The data was collected from two different sites, where one was a former State park repurposed for research purposes, while the other was a large college parking lot. The RaspberryShake featured a geophone designed to measure seismic vibrations due to remote earthquakes. It was found to be much more sensitive to vibrations introduced by nearby moving objects than, say, accelerometers on a smartphone. In this dataset, we introduced each of seven different targets alternately in the vicinity of the sensor nodes: A Polaris off-road vehicle (from https://ranger.polaris.com/), a Chevrolet Silverado, a Warthog all-terrain Unmanned Ground Vehicle (from https://clearpathrobotics.com/), a Motorcycle, a Tesla, a Mustang, and a dismount human. Each target moved around at a different speed, while our sensors collected the corresponding seismic and acoustic signals. Only one target is considered during our experiments. The sampling rate for the seismic signal was  $100\mathrm{Hz}$  and the acoustic signal was collected under  $16000\mathrm{Hz}$  (which was downsampled to  $8000\mathrm{Hz}$  in the preprocessing). For each target, the collection lasted between 40 minutes to 1 hour. The training, validation, and testing datasets are randomly partitioned with a ratio of 8:1:1 at the sample level. (See IRB note.) We do plan to release this dataset for public usage after the paper anonymization period.