SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Analysis: We observe a significant drop in performance on most of the self-supervised learning frameworks on speed classification. When using SW-T as the encoder, FOCAL still dominates the performance over other baselines, exceeding the strongest baseline by  $6.07\%$  accuracy and  $10.32\%$  F1 score. When using DeepSense as the encoder, FOCAL also achieves comparable high performance as the current baselines. The advantage of FOCAL persists in the correlated accuracy metric where the physical correlations among classes are counted. Considering the heterogeneous finetune tasks, the potential domain shift, and the leading performance, we conclude that FOCAL is promising in learning fundamental feature patterns from multi-modal sensing data that could serve an extensive set of downstream tasks.
F.5 Ablation Study Results
Steup: We first briefly introduce the compared variants of FOCAL in our ablation study. In these variants, they are set up in the same way as FOCAL except for the places we explain below.
- FOCAL-noPrivate: We remove the private modality space and its related contrastive task but only apply the cross-modal matching task.
- FOCAL-noOrth: We keep the private modality space, but do not enforce the orthogonality constraint between the shared feature and private feature of the same modality, and the private features between pairs of modalities.
- FOCAL-wDistInd: We replace the geometrical orthogonality constraint with statistical independence between modality embedding distributions. Specifically, we follow the approach proposed in [8] to disentangle the distribution of latent subspaces, which minimizes the mutual information between shared-private spaces of the same modality and private-private spaces between two modalities. Given two embedding distributions, it minimizes the KL divergence between their joint distribution and the product of two marginal distributions. Following the density-r ratio trick, we train a classifier consisting of several fully-connected layers to discriminate samples from the originally matched pairs of embeddings and the randomly selected embedding pairs, which has been shown to approximate the density ratio needed to estimate the KL divergence within sample batches. Similar to GAN [5], we train the discriminator alternatively with modality encoders until convergence.
- FOCAL-noTemp: We remove the temporal structural constraint proposed in FOCAL.
- FOCAL-wTempCon: We replace the temporal structural constraint with a temporal contrastive task. Given a modality, we regard close sample pairs within a short sequence as positive samples and regard distant sample pairs from different short sequences as negative samples, and conduct discrimination between positive samples and negative samples.
Table 16: Ablation Results with DeepSense Encoder and Linear Classifier
Metrics	MOD	ACIDS	RealWorld-HAR	PAMAP2
Acc	F1	Acc	F1	Acc	F1	Acc	F1
FOCAL-noPrivate	0.939	0.938	0.8803	0.7229	0.8742	0.843	0.8146	0.8017
FOCAL-noOrth	0.9691	0.9688	0.9068	0.8218	0.9061	0.8967	0.828	0.7957
FOCAL-wDistInd	0.9223	0.9223	0.9493	0.8347	0.9438	0.9287	0.7921	0.7344
FOCAL-noTemp	0.9557	0.9551	0.9461	0.872	0.9319	0.9237	0.8414	0.8162
FOCAL-wTempCon	0.9564	0.956	0.9255	0.8124	0.9353	0.9141	0.8497	0.8131
FOCAL	0.9732	0.9729	0.9516	0.8580	0.9382	0.9290	0.8588	0.8463
Analysis: The complete ablation results on DeepSense encoder are presented in Table 16. Similar to our observations with SW-T encoder, all of the three components introduced in FOCAL (private space, orthogonality constraint, and temporal constraint) contribute positively to the downstream performance. However, we do find the orthogonality constraint and the temporal constraint play a more important role in the performance improvement with the DeepSense encoder than that with SW-T encoder on ACIDS, RealWorld-HAR, and PAMAP2 datasets. Besides, it is noticeable that distributional independence contributes positively to FOCAL on ACIDS and RealWorld-HAR datasets but contributes negatively to FOCAL on MOD and PAMAP2 datasets. We leave it as future work to investigate more into the role of distributional independence in factorizing the latent space within the multimodal contrastive learning paradigm.
G Limitations and Potential Extensions
Assumption on Modality Synchronization: We assume the signals simultaneously arrived at all sensory modalities such that the information at different modalities is synchronized. However, in some scenarios, different signals propagate at significantly different speeds. For instance, light travels much faster than sound. The shared modality embeddings can not be directly matched for the same samples without signal synchronizations between the modalities.
Computational Complexity of Pretraining Loss: In the current design, we take all pairs of modalities to compute their shared space consistency loss and private space orthogonality loss, which leads to  $O(K^2)$  complexity to the number of modalities  $K$ . On one hand, we assume the modality number is limited to a handful count in most sensing applications; on the hand, we leave it as one of our future work to reduce the computational complexity in pretraining loss calculation.
Dependency on Data Augmentations: Our current contrastive learning paradigm is still not fully self-supervised, because we need to design a set of transformations (i.e., data augmentations) for the private modality feature learning. However, different from image data, designing proper label-invariant data augmentations for time-series data can be challenging in some applications, especially when we do not have knowledge about the potential downstream tasks. One potential solution is to integrate the masked reconstruction learning paradigm into the framework, such that data augmentations can be avoided or less depended on.
Multi-Device Collaboration: This paper focused on multi-modal collaborative sensing settings while multi-device collaboration is not fully considered. The general design of contrastive learning in factorized latent space is extensible to the multi-device setting, but more designs need to be introduced to further address the heterogeneity contained in different vantage points and the scalability issues related to the number of participating sensor nodes in large-scale distributed sensing scenarios.
Resiliency Against Domain Shift: Although FOCAL improves the downstream performance of contrastive learning from multimodal sensing signals, it still exhibits relatively low accuracy in speed classification when data is collected from a different environment. There are multiple environmental factors that can lead to such degradations, including terrain, wind, sensor facing directions. We hope to integrate domain-invariant considerations into the learning objective in the future such that apparently task-unrelated information is decoupled and removed from the pretrained embedding space, and the model resiliency can be significantly enhanced.
[18] Timo Sztyler and Heiner Stuckenschmidt. On-body localization of wearable devices: An investigation of position-aware activity recognition. In IEEE International Conference on Pervasive Computing and Communications (PerCom), 2016.
[19] Chi Ian Tang, Ignacio Perez-Pozuelo, Dimitris Spathis, and Cecilia Mascolo. Exploring contrastive learning in human activity recognition for healthcare. arXiv preprint arXiv:2011.11542, 2020.
[20] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision (ECCV), 2020.
[21] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations (ICLR), 2021.
[22] Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulic. Data augmentation of wearable sensor data for parkinson's disease monitoring using convolutional neural networks. In ACM international conference on multimodal interaction (ICMI), 2017.
[23] Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In International Conference on World Wide Web (WWW), 2017.
[24] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAI Conference on Artificial Intelligence (AAAI), 2022.