Title: FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Space
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multimodal Time Series; Contrastive Learning; Factorized Latent Space
Abstract: This paper proposes a novel contrastive learning framework, called FOCAL, for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. Existing multimodal contrastive frameworks mostly rely on the shared information between sensory modalities, but do not explicitly consider the exclusive modality information that could be critical to understanding the underlying sensing physics. Besides, contrastive frameworks for time series have not handled the temporal information locality appropriately. FOCAL solves these challenges by making the following contributions: First, given multimodal time series, it encodes each modality into a factorized latent space consisting of shared features and private features that are orthogonal to each other. The shared space emphasizes feature patterns consistent across sensory modalities through a modal-matching objective. In contrast, the private space extracts modality-exclusive information through a transformation-invariant objective. Second, we propose a temporal structural constraint for modality features, such that the average distance between temporally neighboring samples is no larger than that of temporally distant samples. Extensive evaluations are performed on four multimodal sensing datasets with two backbone encoders and two classifiers to demonstrate the superiority of FOCAL. It consistently outperforms the state-of-the-art baselines in downstream tasks with a clear margin, under different ratios of available labels. The code and self-collected dataset are available at https://github.com/tomoyoshki/focal.

FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Space

Abstract

This paper proposes a novel contrastive learning framework, called FOCAL, for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. Existing multimodal contrastive frameworks mostly rely on the shared information between sensory modalities, but do not explicitly consider the exclusive modality information that could be critical to understanding the underlying sensing physics. Besides, contrastive frameworks for time series have not handled the temporal information locality appropriately. FOCAL solves these challenges by making the following contributions: First, given multimodal time series, it encodes each modality into a factorized latent space consisting of shared features and private features that are orthogonal to each other. The shared space emphasizes feature patterns consistent across sensory modalities through a modal-matching objective. In contrast, the private space extracts modality-exclusive information through a transformation-invariant objective. Second, we propose a temporal structural constraint for modality features, such that the average distance between temporally neighboring samples is no larger than that of temporally distant samples. Extensive evaluations are performed on four multimodal sensing datasets with two backbone encoders and two classifiers to demonstrate the superiority of FOCAL. It consistently outperforms the state-of-the-art baselines in downstream tasks with a clear margin, under different ratios of available labels. The code and self-collected dataset are available at https://github.com/tomoyoshki/focal.

1 Introduction

As a representative self-supervised learning (SSL) paradigm, contrastive learning (CL) has achieved unprecedented success in vision tasks [3, 14, 4, 12, 55] and are increasingly leveraged in learning from time series [10, 7, 63, 65, 36, 58]. However, many IoT applications [39, 56, 33] rely on heterogeneous sensory modalities to collaboratively perceive the physical surroundings and lead to a more complicated learning space. This paper aims to build a contrastive learning framework that maximally extracts complementary information from multimodal time-series sensing signals [23, 60],. The key challenge is to define appropriate similarity/distance measures ( i.e. , who should be close to whom) in the joint multimodal embedding space that facilitates the downstream tasks [53, 57].

Existing contrastive frameworks for time series either ignore the heterogeneity among sensory modalities and are limited to instance-level discrimination [22, 17], or are designed to extract only shared information across sensory modalities [48, 6, 35]. For instance, as a representative framework, CMC [48] builds on the hypothesis that only information shared across views corresponds to the 37th Conference on Neural Information Processing Systems (NeurIPS 2023).

Mod-1

Mod-2

Seq-

…

Seq-

𝑖′

Seq-

𝑖′

Mod-2

Mod-1

Private

Feature

Factorized

Orthogonal

Sample-

𝑖

Aug-1

Private Feature

Sample-

𝑖

Aug-2

Private Feature

Sample-

𝑖′

Aug-1

Private Feature

Sample-

𝑖;

Aug-2

Private Feature

Time

Private

Contrastive

Loss

Orthogonality Loss

Positive Pair

Negative Pair

Loss

Samples within A Sequence

Different Sequences

Different Modalities

...

...

Seq-

𝑖

Seq-

𝑖

...

...

Seq-

…

Sample-

𝑖

Mod-1

Shared Feature

Sample-

𝑖

Mod-2

Shared Feature

Sample-

𝑖′

Mod-1

Shared Feature

Sample-

𝑖′

Mod-2

Shared Feature

Shared

Contrastive

Loss

Temporal

Locality

Loss

Distance Matrix

Multimodal Time Series

Seq-

𝑖

Seq...

Seq-

𝑖′

1

2

MLP Projector

Time-Freq Encoder

Rand

Aug-1

Rand

Aug-2

3

4

actual physical target. We argue that the strength of multimodal sensing lies in that the collaborating modalities not only share information but also enhance each other by providing complementary information exclusive to the modalities. The pretraining objective in contrastive learning should be calibrated to extract all semantically meaningful and generalizable patterns.

In addition, we observe that existing contrastive frameworks for time series do not handle the temporal information locality in a proper way. Temporal contrastive works ( e.g. , TNC [49]) force the temporally close samples to be positive pairs of similarity 1 and force the temporally distant samples to be negative pairs of similarity 0, which may contradict long-term seasonality patterns. For example, a circling motorcycle may cause periodical vibration patterns to nearby microphone arrays, violating the above strict contrastive objective. Alternatively, TFC [65] optimizes the consistency between time-domain and frequency-domain representations, preventing neither encoder from simultaneously extracting features from the time-frequency spectrogram, which are known to achieve superior performance in learning from sensing signals [59].

To overcome these limitations, we propose a novel contrastive framework, FOCAL, for self-supervised representation learning from multimodal time-series sensing data. An overview is presented in Figure 1. Motivated by [43], we encode the input ( i.e. , fixedlength window of sensor readings) of each modality into a factorized orthogonal latent space, composed of a shared space and a private space. The shared features and private features of the same modality, as well as the private features between different modalities, are mutually orthogonal to emphasize their semantical independence. The shared space is designed

CMC

Our Approach

𝑀𝑜𝑑

!

𝑀𝑜𝑑

"

𝑀𝑜𝑑

!

𝑀𝑜𝑑

"

Shared

Shared

Private

Private

Private

Private

to capture information consistency across modalities, while the private space captures modalityexclusive but transformation-invariant information. As we show in Figure 2, FOCAL outperforms CMC [48] by further emphasizing modality-exclusive sectors of target information. Besides, we define a temporal structural constraint to the distance measure of modality embeddings through a loose ranking constraint between temporally 'close sample pairs' and 'distant sample pairs'. During the pertaining, we use a sampling strategy that randomly selects multiple short sequences with fixed length ( e.g. , 4 samples per sequence) to constitute training batches. Instead of performing temporal contrastive tasks as [49], we confine the average intra-sequence distance between temporally neighboring samples ( i.e. , 3 in Figure 1) to be no larger than the average inter-sequence distances between temporally distant samples ( i.e. , 4 in Figure 1). It simultaneously addresses overall temporal information locality and tolerates occasional violations of locality caused by periodicity, and it turns out to accelerate the pretraining convergence in practice.

On the one hand, we justify the feature orthogonality constraints from two aspects. First, the shared features and private features of a modality should be orthogonal, such that the private space can avoid reusing the shared information but instead exploit modality-exclusive discriminative information. Second, the private features of different modalities should be orthogonal to each other. Otherwise, the overlapped semantics should be included in their shared space. In summary, the orthogonality

constraint is imposed to refine the heterogeneous modality-exclusive information that is undervalued in existing cross-modal contrastive tasks.

On the other hand, the temporal information locality within time series is defined through coarsegrained distance orders for two reasons. First, we do not always regard temporally close samples as positive pairs (with similarity 1), such that fine-grained differences between neighboring samples can be included. Second, we only enforce the temporal constraint at the statistical average scale by comparing the average intra-sequence distances to the average inter-sequence distances to fit potential exceptions caused by long-term seasonal signal patterns and significantly reduce the computational complexity because we avoid traversing the ranking losses within all sample triplets [15].

In summary, we define the following four perspectives of pretraining objectives in FOCAL.

Modality Consistency in Shared Space: In the shared space, we push the features of different modalities of the same sample to be similar to each other compared to randomly mismatched modality features from two temporally distant samples.

Transformation Consistency in Private space: In the private space, we push the modality features under two random augmentations to be similar to each other compared to modality features from two samples.

Orthogonality Constraint: We enforce the shared feature and private feature of the same modality, as well as private features of different modalities, to be orthogonal to each other.

Temporal Locality Constraint: We restrict the average distance of samples within a short time sequence to be no larger than the average distance between two random sequences.

We extensively evaluate FOCAL against eleven state-of-the-art baselines on four multimodal sensing datasets with two different backbone encoders (DeepSense [59] and Swin-Transformer [26]). The finetuning performance is evaluated with two light-weight classifiers ( i.e. , linear and KNN classifier). It consistently achieves higher accuracy and F1 scores than the baselines. We also break down the contribution of individual components through step-by-step ablation studies.

2 Problem Formulation

Assume we have a set of P sensory modalities M = { M 1 , M 2 , . . . , M P } , and a large set of N unlabeled pretraining data from all sensory modalities X = { x 1 , x 2 , . . . , x N } . Each sample is a fixed-length window of signals from all sensory modalities. For each sample x i , we use x ij to denote the input from sensory modality M j . The original input of each sensory modality is a multivariate time series, and we apply Short-Time Fourier Transform (STFT) as the preprocessing procedure to extract the sample time-frequency representation. As we introduce in Appendix B, the processed modality input is x ij ∈ R C × I × S , where C denotes the number of input channels, I denotes the number of time intervals within a sample window, and S denotes the spectrum length after applying the Fourier transform to each interval 1 . We have a set of backbone encoders E = { E 1 , E 2 , . . . , E P } such that the encoder E i of modality M i can encode the modality input x ij into an embedding vector h ij = E j ( x ij )) ∈ R K , where K is the unified dimension of modality embedding vectors. We use ⟨ h ij , h i ′ j ′ ⟩ to denote the inner product between two embedding vectors h ij and h i ′ j ′ . The encoders extract both the time and frequency patterns from the preprocessed input. We also have a small set of N ′ supervised samples for finetuning X s = { x s 1 , x s 2 , . . . , x s N ′ } , where each sample x s i is associated with a label y s i . Please note X s is not necessarily a subset of the pretraining set X due to potential domain adaptations, i.e. , X s ̸⊆ X . The objective of self-supervised pretraining is to use the unlabeled dataset X to optimize the parameters of modality encoders E such that the model accuracy finetuned on the supervised dataset X s is maximized.

3 FOCAL Framework

In this section, we start with an overview of the framework and then separately introduce the two main functional components: (1) contrastive learning in factorized orthogonal latent space, and (2) temporal structural constraint.

3.1 Overview

The key questions to answer in designing a contrastive learning framework include the selection of positive/negative pairs and the contrastive loss functions . Compared to image input about which we have almost no knowledge without human annotations, the meta-level information of multimodal timeseries input, i.e. , cross-modal correspondence and temporal information locality , can be effectively leveraged during the pretraining to shape the learned joint embedding space. As we show in Figure 1, FOCAL groups multiple randomly sampled fixed-length ( i.e. , L samples) short sequences of samples into a batch B , with cardinality |B| = B , and only conducts contrastive comparisons within the batch without any memory banks. Each modality input x ij goes through two randomly selected augmentations to generate two augmented versions, and each augmented input is separately encoded by the modality encoder E j to get two versions of modality embeddings h ij and ˜ h ij . The encoder network structures are not this paper's original contribution, so we leave them in Appendix D.

As the output of modality encoding, each modality embedding h ij is projected through a non-linear multilayer perceptron (MLP) projector into two orthogonal embeddings, a shared embedding h shared ij and a private embedding h private ij . The two embeddings share all encoder layers except the MLP projector. Separate contrastive learning tasks are applied to each embedding to capture different aspects of information. At the same time, shared-private and private-private modality features of the same sample are mutually orthogonal. In addition, we apply a temporal structural constraint between intra-sequence distances and inter-sequence distances to both projected modality embeddings.

3.2 Multimodal Contrastive Learning in Factorized Orthogonal Space

In multimodal collaborative sensing, information from different sensory modalities is not fully overlapped, thus extracting modality-exclusive discriminative information can reinforce the shared information during the downstream task finetuning. Projecting each modality embedding into separate shared space and private space while applying the orthogonality constraints avoids the shared information being reused in the private space optimization.

Contrastive Task for Shared Space: We use the shared feature space to learn modality consistency information. Specifically, we only consider samples within a batch B but across different short sequences, and assume the same random augmentation is applied. We iterate over all pairs of modalities M j and M j ′ , regarding different modality embeddings of the same sample ( x shared ij , x shared ij ′ )

( e.g. , 1 in Figure 1) as positive pairs, and regard two random modality embeddings from different samples ( x shared ij , x shared i ′ j ′ ) as negative pairs. We calculate the following InfoNCE loss [48],

̸

where τ is a temperature parameter that controls the penalties on hard negative samples [52].

Contrastive Task for Private Space: We use the private feature space to learn modality-exclusive information that is useful in discriminating different sequences within a batch B , through capturing transformation consistency information. Specifically, for each modality M j , we consider the encoded embeddings of two randomly augmented versions of the same samples as positive pairs ( e.g. , 2 in Figure 1), i.e. , ( h private ij , ˜ h private ij ), where we use ˜ h private ij to denote a differently augmented variant. The remaining 2B-2 modality embeddings in the batch are considered negative pairs to h private ij . We use the NT-Xent [3] loss for the private space contrastive task,

̸

Orthogonality Constraint: To enforce the orthogonality constraint between the shared feature and private feature of the same modality, as well as the private features between different modalities, such that they can capture independent semantic information in the factorized space, we apply a cosine

embedding loss that directly minimizes their angular similarities,

3.3 Temporal Structural Constraint

Appropriately shaping the temporal information locality in latent modality embedding space is challenging. First, simply considering temporally close samples as positive pairs and pushing their semantical similarities to 1 as [63, 49] can be problematic because they ignore the continuously evolving factors ( e.g. , distance) that make differences between temporally neighboring samples. Second, it is also impractical to predefine a fixed similarity curve with respect to the time difference between two samples, which is highly context-dependent and can not be known in advance.

Considering the complexity of temporal information correlations, we stop defining positive versus negative pairs in the temporal dimension. Instead, we only apply a loose ranking loss to specify the relationships of distances at the coarse-grained sequence level (intra-sequence distance vs. intersequence distance) as an information regularization [9]. Given a sequence, we restrict the average distances between samples within the sequence ( e.g. , 3 in Figure 1) to be no larger than their average distance to samples from other sequences ( e.g. , 4 in Figure 1). To accommodate occasional violations of temporal locality caused by long time periodicity (such that temporally distant samples could be more similar than neighboring samples), and further reduce the computational complexity caused by multiplicative traverse of sample triplets, we restrict the average intra-sequence distances to be no larger than average inter-sequence distances.

We calculate the distance in the Euclidean space, to facilitate the downstream Euclidean classifiers. Given the sample-level distance matrix D ∈ R BL × BL , we first compute the sequence-level mean distance matrix ¯ D ∈ R B × B through aggregating sample-level distances 2 , such that the average distance between sequence s and sequence s ′ is ¯ D ss ′ = 1 /L 2 ∑ i ∈ s,i ′ ∈ s ′ D pq , then the temporal locality loss is defined by,

̸

where the margin is the predefined degree of separation. It is worth noting that the temporal structural constraint is applied to holistic modality embeddings, including both the shared and private parts.

3.4 Overall Training Objective

In summary, FOCAL pretraining simultaneously considers latent correlations between (1) synchronized embeddings of different modalities; (2) differently augmented modality embeddings; and (3) temporally neighboring sample embeddings. It minimizes the following overall loss,

where λ p , λ o , and λ t are hyperparameters that control the weights of each loss component.

4 Evaluation

In this section, we evaluate FOCAL by comparing it with 11 popular SOTA self-supervised learning frameworks on four different datasets. We start with the experimental setups and then introduce the main evaluation results, followed by comprehensive ablation studies.

4.1 Experimental Setup

Datasets: (1) MOD is a self-collected dataset using acoustic (8000Hz) and seismic (100Hz) signals to classify moving vehicle types. It includes 6 different vehicle types and 1 class of human walking. (2) ACIDS is an ideal dataset for vehicle classification using acoustic signals and seismic signals

̸

(both in 1025Hz). It includes data on 9 types of ground vehicles in 3 different terrains. (3) RealWorldHAR [46] is a public dataset using accelerometer, gyroscope, magnetometer, and light signals (all in 50Hz) to recognize 8 common human activities. (4) PAMAP2 [41] is another public dataset using accelerometer, gyroscope, and magnetometer signals (all in 100Hz) to recognize 18 different physical activities. The length of the samples and the intervals, as well as the time overlap ratios between intervals within samples of each dataset, as listed in Table 1, are configured to achieve the best-supervised classification performance.

Data Augmentations: Candidate augmentations are defined in both the time domain before STFT and the frequency domain after STFT, where only one out of them is randomly selected in each forward pass. Time-domain augmentations include scaling, permutation, negation, time warp, magnitude warp, horizontal flip, jitter, channel shuffle, and time masking; frequency-domain augmentations include phase shift and frequency masking. Details can be found in Appendix B.

Baselines: We consider 12 baselines in total, including a supervised benchmark, three representative self-supervised learning frameworks from vision tasks that perform instance discrimination (SimCLR [3], MoCoV3 [5], and MAE [13]), four modality-matching contrastive frameworks for multimodal input (CMC [48], Cosmo [35], Cocoa [6], GMC [37]), three SOTA contrastive frameworks for time series (TS2Vec [63], TNC [49], TS-TCC [8]), and one predictive baseline (MTSS [42]). Their detailed introductions can be found in Appendix C. All compared frameworks use the same encoder structures (except for minor differences in module orders). The evaluation metrics are accuracy and (macro) F1 score.

Backbone Encoders: We apply two different modality encoders in this paper. (1) DeepSense [59] uses convolutional layers to extract localized feature patterns within each time interval, and then uses recurrent layers to aggregate information across all intervals within a sample window. (2) SwinTransformer (SW-T) [26] uses stacked Transformer blocks to extract local information from shifted windows in a hierarchical manner. Their details and configurations can be found in Appendix D.

MOD

ACIDS

RealWorld-HAR

PAMAP2

0.2

0.4

0.6

0.8

1.0

MOD

ACIDS

RealWorld-HAR

PAMAP2

0.2

0.4

0.6

0.8

1.0

SimCLR

MoCo

CMC

MAE

Cosmo

Cocoa

MTSS

TS2Vec

GMC

TNC

TS-TCC

FOCAL

Both models are implemented in PyTorch 1.14 and optimized with an AdamW [28] optimizer under cosine learning rate schedules [27]. Without special specifications, we report the results on SW-T backbone. Detailed training configurations are introduced in Appendix E.

Finetune Classifiers: We evaluate with two lightweight classifiers during the finetune stage. Linear probling adds a linear layer on top of the pretrained sample features (or concatenated modality features) with the encoder fixed during fine-tuning. KNN classifier directly uses the voting of 5 nearest neighbors (based on the Euclidean distances between samples) in the finetuning training samples to generate the predicted labels for the testing samples.

4.2 Finetune Results

Linear Probing: Table 2 presents the finetune results using a linear classifier (The complete results can be found in Appendix F). FOCAL consistently achieves the best accuracy and F1 score across different datasets and backbone encoders. It outperforms CMC by 4.48% to 18.01% in accuracy, demonstrating the importance of extracting private modality information in addition to the shared information. SimCLR and MoCo achieved suboptimal performance on HAR datasets ( i.e. , RealWorldHAR and PAMAP2) which have more than two modalities, showing that instance discrimination alone as the pretext task can not learn comprehensive feature patterns from multiple modalities. GMC, as the SOTA framework for multi-modal time series, beats the contrastive frameworks for single-modal time series (TS2Vec, TNC, and TS-TCC) in most cases, but is secondary to FOCAL. On MODdataset, FOCAL achieves higher accuracy and F1 score than the supervised model by taking advantage of massive unlabeled samples.

Besides, we plot the finetune results on the MOD dataset under three different label ratios (100%, 10%, and 1%) in Figure 3. FOCAL overall achieves better label efficiency during the finetuning. It attains 3.51% relative improvement with 100% labels and 10.56% relative improvement with 1% labels, compared to the best-performing baseline. Among the baselines, TNC performs best with 1% label but does not generalize well when more labels are available ( e.g. , 100%).

100% Labels

10% Labels

1% Labels

0.2

0.4

0.6

0.8

1.0

SimCLR

MoCo

CMC

MAE

Cosmo

Cocoa

MTSS

TS2Vec

GMC

TNC

TS-TCC

FOCAL

KNN Evaluation: The evaluation results with a KNN classifier (K=5) using all available labels 3 are plotted in Figure 4 (See Appendix F for complete results). KNN operates in a non-parametric way and can be expressive if the learned latent space aligns well with the underlying semantics. For multi-modal contrastive frameworks ( i.e. , FOCAL, CMC, Cosmo, Cocoa, GMC), we concatenate the modality features as the sample feature. FOCAL consistently performs better than the baselines across all datasets, proving its representational power in the non-parametric classification setting.

4.3 Additional Downstream Tasks

Clustering: In multi-modal learning, the clusterability of the learned modality representations is preferable. Meaningful representation clusters should be coherent with the underlying semantic labels. For each modality, we first use K-means clustering with pretrained modality embeddings to get the cluster labels and measure the consistency between the cluster labels and the ground-truth category labels, with two common clustering metrics adjusted rand score (ARI) [45] and normalized mutual information score (NMI) [1]. The results for five multi-modal frameworks are presented in Figure 5.

MOD

ACIDS

RealWorld-HAR

PAMAP2

Dataset

0.0

0.2

0.4

0.6

0.8

ARI

CMC

Cocoa

Cosmo

FOCAL

GMC

MOD

ACIDS

RealWorld-HAR

PAMAP2

Dataset

0.0

0.2

0.4

0.6

0.8

NMI

CMC

Cocoa

Cosmo

FOCAL

GMC

(a) FOCAL

(b) CMC

(c) GMC

(d) Cocoa

We separately evaluate the clusterability of each modality and report the mean and standard deviation among modalities. FOCAL is superior in learning representations better aligned with the target labels. Although CMC performs closely to FOCAL on MOD and RealWorld-HAR datasets, it performs poorly on ACIDS and PAMAP2 datasets. Besides, we qualitatively visualize the concatenated sample embeddings of the multi-modal contrastive frameworks after t-SNE [51] dimension reduction on MODdataset in Figure 6. We can see that FOCAL achieved better separation among different classes than the compared baselines.

Distance and Speed Classification: In the MOD dataset, we separately collect new data samples recording the distance and speed of the vehicles. Data in this experiment is collected in a different environment with different vehicles from the pretraining data, thus accounting for potential domain shifts. The same pretrained models in previous experiments are used, and the results are summarized in Figure 7. The instance discrimination frameworks ( i.e. , SimCLR and MoCo) are relatively more resilient than the modality

Distance Classification

Speed Classification

0.2

0.4

0.6

0.8

1.0

SimCLR

MoCo

CMC

MAE

Cosmo

Cocoa

MTSS

TS2Vec

GMC

TNC

TS-TCC

FOCAL

matching frameworks ( i.e. , CMC, Cocoa, Cosmo, and GMC), while FOCAL fills this gap by capturing the transformation consistency information in private modality spaces.

4.4 Ablation Studies

0.1

1

3

10

0.80

0.85

0.90

0.95

1.00

(a) Lambda private

0.1

1

3

10

0.80

0.85

0.90

0.95

1.00

(b) Lambda orthogonal

0.1

1

3

5

10

0.80

0.85

0.90

0.95

1.00

(c) Lambda temporal

0.1

1

3

5

0.80

0.85

0.90

0.95

1.00

(d) Margin

ACIDS

PAMAP2

Setup: Here we compare with five variants of FOCAL, including FOCAL-noPrivate (no private space), FOCAL-noOrth (no orthogonality constraint), FOCAL-wDistInd (replace orthogonality with distributional independence), FOCAL-noTemp (no temporal structural constraint), FOCALwTempCon (replace temporal structural constraint with the temporal contrastive task).

Analysis: The ablation study results are summarized in Table 3. First, the accuracy decreases by 5.20%-5.90% after removing the private space and the related contrastive task, since only the shared information among modalities is leveraged. Second, on top of the private space task, both the orthogonality constraint and the temporal structural constraint further improve the accuracy by 1.39%-2.99%, which proves they both contribute positively to FOCAL. Third, replacing the geometrical orthogonality constraint with the distributional independence constraint causes significant degradation and may even cause the model training to collapse. Similarly, in our experiments, conducting temporal contrastive tasks leads to a noticeable accuracy decrease in three out of four datasets. To further demonstrate the contribution of our temporal structural constraint in accelerating the convergence during the pretraining, we use KNN classifier to periodically evaluate the quality of the learned representations. Concatenated modality embeddings are used as sample representations. We visualize the achieved KNN accuracy curves on MOD and PAMAP2 with and without the temporal constraint in Figure 8. The temporal constraint clearly improves the semantical structure of the learned embeddings in early epochs of pretraining, by rejecting obviously counter-intuitive parameter values that violate the constraint.

4.5 General Applicability of the Temporal Constraint

To validate the general applicability of the proposed temporal constraint, we apply it to multiple contrastive learning baselines ( i.e. , SimCLR, MoCo, CMC, Cocoa, and GMC). Table 4 and 5 summarize the results on ACIDS and PAMAP2 respectively, and we have observed noticeable performance improvement in most cases (up to 18.99% on ACIDS and up to 8.39% on PAMAP2). It demonstrates that the temporal constraint can be used as a plugin to enhance existing contrastive learning frameworks for time-series data.

4.6 Sensitivity Test on Loss Weights

We also perform a sensitivity test on the loss weight values and plot the performance of FOCAL against different hyperparameters in Figure 9. We observe that FOCAL is generally robust against the

10 1

10 2

10 3

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

FOCAL

FOCAL-noTemp

(a) MOD

10 1

10 2

10 3

Epoch

0.5

0.6

0.7

0.8

0.9

1.0

FOCAL

FOCAL-noTemp

(b) PAMAP2

hyperparameter selections, with less than 2% accuracy fluctuations in all cases. For this reason, we did not perform a comprehensive hyperparameter search in our experiment. Besides, combining our observations in the ablation study that the private space task, orthogonality constraint, and temporal constraint all contribute positively to the performance of FOCAL, we conclude that the competition between learning objectives does not happen in FOCAL.

5 Related Works

Self-Supervised learning for multimodal data. Self-supervised learning from multimodal data has been extensively studied in the literature, including contrastive learning [48, 35, 6, 37, 38, 25, 32, 2], masked autoencoders (MAE) [13, 11, 19], and variational autoencoder (VAE) [18, 50, 21], surpassing conventional generation-based semi-supervised learning approaches [61]. As a leading paradigm in learning transferable and discriminative features [66, 24, 57] with a variety of successful applications [16, 29, 40, 30, 47], we mainly consider contrastive learning in this paper. On one hand, as we show in the experimental results, normal contrastive frameworks based on instance discriminations (SimCLR [20], MoCO [5], BYOL [12], SimSiam [4]) may lead to suboptimal results without accommodating the multimodal properties. On the other hand, existing contrastive learning frameworks for multimodal data [38, 48, 35, 6, 37] mostly focus on the consistency between modalities but ignore the information heterogeneity when they are collaboratively utilized in sensing tasks. CLIP [38], as a representative multimodal contrastive framework, mainly addresses the importance of using natural language to augment the visual models through learning their pairings. Similarly, both CMC [48] and GMC [37] highlight the information matching between modality embedding to another modality embedding or the joint embedding. Instead, in FOCAL, we simultaneously consider modality consistency and discriminative modality-exclusive information by designing corresponding contrastive tasks and imposing the information independence constraint. Besides, most existing multimodal contrastive learning frameworks are limited to vision and language modalities [62, 22, 47, 64, 31], while the domain knowledge might not be directly applicable to sensory modalities with time-series signals in IoT applications.

Contrastive learning for time series. There has been increasing interest in developing contrastive learning frameworks for time-series data [10, 63, 44, 7, 49, 22, 65, 36]. TS2Vec [63], TFC [65], TNC [49], and TS-TCC [7] were based on the time-series properties but did not consider the multimodal collaboration properties. Besides, TFC [65] and RF-URL [44] were designed from the consistency between time and frequency representations, or different time-frequency representations, while restricting the backbone encoder structures. Cosmo [35] and Cocoa [6] are the most recent attempts at contrastive learning from multi-modal time series, but they were not able to sufficiently utilize the complementary information from sensory modalities. CoST [54] proposed to separate the seasonal-trend representations for time series forecasting. Different from the existing works, the objective of FOCAL is to maximally extract complementary and discriminative features from multimodal sensory modalities, to facilitate the downstream recognition tasks. CLUDA [36] worked on the unsupervised domain adaptation (UDA) problem for time series, which is not directly comparable to our solution. In FOCAL, no expert features are assumed to be available as in ExpCLR [34].

6 Conclusion

We proposed a novel contrastive learning framework, FOCAL, for self-supervised learning from multimodal time-series sensing signals. FOCAL encodes each modality input into a factorized orthogonal latent space including shared features and private features. We learn each part by applying different contrastive learning objectives addressing the modality consistency in the shared space and the transformation invariance in the private space. Besides, we design a lightweight temporal structural constraint as an information regularization during the pretraining. Extensive evaluations on four multimodal sensing datasets with two encoder networks and two lightweight classifiers, demonstrate the superiority of FOCAL over 11 SOTA baselines, under different levels of supervision. Our future work would focus on extracting domain-invariant features in multi-vantage sensing applications to make the pretrained model resilient against task-unrelated environmental factors ( e.g. , terrain, wind, sensor-facing directions).

Acknowledgments and Disclosure of Funding

Research reported in this paper was sponsored in part by the Army Research Laboratory under Cooperative Agreement W911NF-17-20196, NSF CNS 20-38817, National Natural Science Foundation of China (No. 62332014), DARPA award HR001121C0165, DARPA award HR00112290105, DoD Basic Research Office award HQ00342110002, and the Boeing Company. Shengzhong Liu is also sponsored by the 'Double-First Class' Startup Research Funding of Shanghai Jiao Tong University. The views and conclusions contained in this document are those of the author(s) and should not be interpreted as representing the official policies of the CCDC Army Research Laboratory, or the US government. The US government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation hereon.

References

Alessia Amelio and Clara Pizzuti. Is normalized mutual information a fair measure for comparing community detection methods? In IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) , 2015.

Razvan Brinzea, Bulat Khaertdinov, and Stylianos Asteriadis. Contrastive learning with crossmodal knowledge mining for multimodal human activity recognition. In International Joint Conference on Neural Networks (IJCNN) , 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML) , 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision (CVPR) , 2021.

Shohreh Deldari, Hao Xue, Aaqib Saeed, Daniel V. Smith, and Flora D. Salim. Cocoa: Cross modality contrastive learning for sensor data. ACM Interact. Mob. Wearable Ubiquitous Technol. (IMWUT) , 6(3), 2022.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI) , 2021.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. arXiv preprint arXiv:2208.06616 , 2022.

Shengyu Feng, Baoyu Jing, Yada Zhu, and Hanghang Tong. Adversarial graph contrastive learning with information regularization. In International World Wide Web Conference (WWW) , 2022.

Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML , 2022.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

David T Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi. Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives. In AAAI Conference on Artificial Intelligence (AAAI) , 2022.

Umangi Jain, Alex Wilson, and Varun Gulshan. Multimodal contrastive learning for remote sensing tasks. In Self-Supervised Learning - Theory and Practice, NeurIPS Workshop , 2022.

Bulat Khaertdinov, Esam Ghaleb, and Stylianos Asteriadis. Contrastive self-supervised learning for sensor-based human activity recognition. In IEEE International Joint Conference on Biometrics (IJCB) . IEEE, 2021.

Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. Mvae: Multimodal variational autoencoder for fake news detection. In International World Wide Web Conference (WWW) , pages 2915-2921, 2019.

Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. Masked vision and language modeling for multi-modal representation learning. In International Conference on Learning Representations (ICLR) , 2023.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Oliver Limoyo, Trevor Ablett, and Jonathan Kelly. Learning sequential latent variable models from multimodal time series data. In Ivan Petrovic, Emanuele Menegatti, and Ivan Markovi´ c, editors, Intelligent Autonomous Systems 17 , 2023.

Dongxin Liu, Tianshi Wang, Shengzhong Liu, Ruijie Wang, Shuochao Yao, and Tarek Abdelzaher. Contrastive self-supervised representation learning for sensing signals from the time-frequency perspective. In IEEE International Conference on Computer Communications and Networks (ICCCN) , 2021.

Shengzhong Liu, Shuochao Yao, Jinyang Li, Dongxin Liu, Tianshi Wang, Huajie Shao, and Tarek Abdelzaher. Giobalfusion: A global attentional deep learning framework for multisensor information fusion. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) , 4(1):1-27, 2020.

Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering (TKDE) , 35(1):857-876, 2021.

Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi. Contrastive multimodal fusion with tupleinfonce. In IEEE/CVF International Conference on Computer Vision (CVPR) , 2021.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision (CVPR) , 2021.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR) , 2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR) , 2019.

Shuang Ma, Sai Vemprala, Wenshan Wang, Jayesh K. Gupta, Yale Song, Daniel McDufft, and Ashish Kapoor. Compass: Contrastive multimodal pretraining for autonomous systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022.

Hiroshi Morioka and Aapo Hyvarinen. Connectivity-contrastive learning: Combining causal discovery and representation learning for multimodal data. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2023.

Basil Mustafa, Carlos Riquelme Ruiz, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2023.

Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. Moddrop: adaptive multimodal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 38(8):1692-1706, 2015.

Manuel T Nonnenmacher, Lukas Oldenburg, Ingo Steinwart, and David Reeb. Utilizing expert features for contrastive learning of time-series representations. In International Conference on Machine Learning (ICML) , 2022.

Xiaomin Ouyang, Xian Shuai, Jiayu Zhou, Ivy Wang Shi, Zhiyuan Xie, Guoliang Xing, and Jianwei Huang. Cosmo: Contrastive fusion learning with small data for multimodal human activity recognition. In International Conference on Mobile Computing And Networking (MobiCom) , 2022.

Yilmazcan Ozyurt, Stefan Feuerriegel, and Ce Zhang. Contrastive learning for unsupervised domain adaptation of time series. In International Conference on Learning Representations (ICLR) , 2023.

Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Geometric multimodal contrastive representation learning. In International Conference on Machine Learning (ICML) , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , 2021.

Valentin Radu, Nicholas D Lane, Sourav Bhattacharya, Cecilia Mascolo, Mahesh K Marina, and Fahim Kawsar. Towards multimodal deep learning for activity recognition on mobile devices. In ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) , 2016.

Aniruddh Raghu, Payal Chandak, Ridwan Alam, John Guttag, and Collin Stultz. Contrastive pre-training for multimodal medical time series. In NeurIPS Workshop on Learning from Time Series for Health , 2022.

Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In International Symposium on Wearable Computers (ISWC) , 2012.

Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien. Multi-task self-supervised learning for human activity detection. ACM Interact. Mob. Wearable Ubiquitous Technol. (IMWUT) , 3(2), 2019.

Mathieu Salzmann, Carl Henrik Ek, Raquel Urtasun, and Trevor Darrell. Factorized orthogonal latent spaces. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2010.

Ruiyuan Song, Dongheng Zhang, Zhi Wu, Cong Yu, Chunyang Xie, Shuai Yang, Yang Hu, and Yan Chen. Rf-url: unsupervised representation learning for rf sensing. In International Conference on Mobile Computing And Networking (MobiCom) , 2022.

Douglas Steinley. Properties of the hubert-arable adjusted rand index. Psychological methods , 9(3):386, 2004.

Timo Sztyler and Heiner Stuckenschmidt. On-body localization of wearable devices: An investigation of position-aware activity recognition. In IEEE International Conference on Pervasive Computing and Communications (PerCom) , 2016.

Aiham Taleb, Matthias Kirchler, Remo Monti, and Christoph Lippert. Contig: Self-supervised multimodal contrastive learning for medical imaging with genetics. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision (ECCV) , 2020.

Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations (ICLR) , 2021.

Xinming Tu, Zhi-Jie Cao, Sara Mostafavi, Ge Gao, et al. Cross-linked unified embedding for cross-modality representation learning. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research (JMLR) , 9(11), 2008.

Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning (ICML) , 2020.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In International Conference on Learning Representations (ICLR) , 2022.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Han Xu, Zheng Yang, Zimu Zhou, Longfei Shangguan, Ke Yi, and Yunhao Liu. Indoor localization via multi-modal sensing on smartphones. In ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) , 2016.

Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In International Conference on Machine Learning (ICML) , pages 24851-24871, 2022.

Xinyu Yang, Zhenguo Zhang, and Rongyi Cui. Timeclr: A self-supervised contrastive learning framework for univariate time series representation. Knowledge-Based Systems , 245:108606, 2022.

Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In International Conference on World Wide Web (WWW) , 2017.

Shuochao Yao, Yiran Zhao, Huajie Shao, Dongxin Liu, Shengzhong Liu, Yifan Hao, Ailing Piao, Shaohan Hu, Su Lu, and Tarek F Abdelzaher. Sadeepsense: Self-attention deep learning framework for heterogeneous on-device sensors in internet of things applications. In IEEE International Conference on Computer Communications (INFOCOM) , 2019.

Shuochao Yao, Yiran Zhao, Huajie Shao, Chao Zhang, Aston Zhang, Shaohan Hu, Dongxin Liu, Shengzhong Liu, Lu Su, and Tarek Abdelzaher. Sensegan: Enabling deep learning for internet of things with a semi-supervised framework. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) , 2(3):1-21, 2018.

Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAI Conference on Artificial Intelligence (AAAI) , 2022.

Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich, and Dietrich Klakow. MCSE: Multimodal contrastive learning of sentence embeddings. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , 2022.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In International Conference on Learning Representations (ICLR) , 2021.

Appendix

The appendix of this paper is structured as follows.

Section A introduces the evaluated datasets and their statistics.

Section B introduces the preprocessing procedure in our experiments.

Section C details the compared baselines in our experiments.

Section D introduces the used backbone model structure and their related configurations.

Section E summarizes the training strategies we applied in this paper.

Section F presents more comprehensive evaluation results.

Section G lists the existing limitations of the work and potential solutions for future extensions.

A Datasets

The basic statistics for each dataset are summarized in Table 1.

Moving Object Detection (MOD): This is a self-collected dataset using sensor nodes consisting of a RaspberryShake 4D (from https://raspberryshake.org/ ) and a microphone array to collect the vibration signals caused by nearby moving vehicles. The data was collected from two different sites, where one was a former State park repurposed for research purposes, while the other was a large college parking lot. The RaspberryShake featured a geophone designed to measure seismic vibrations due to remote earthquakes. It was found to be much more sensitive to vibrations introduced by nearby moving objects than, say, accelerometers on a smartphone. In this dataset, we introduced each of seven different targets alternately in the vicinity of the sensor nodes: A Polaris off-road vehicle (from https://ranger.polaris.com/ ), a Chevrolet Silverado, a Warthog all-terrain Unmanned Ground Vehicle (from https://clearpathrobotics.com/ ), a Motorcycle, a Tesla, a Mustang, and a dismount human. Each target moved around at a different speed, while our sensors collected the corresponding seismic and acoustic signals. Only one target is considered during our experiments. The sampling rate for the seismic signal was 100Hz and the acoustic signal was collected under 16000Hz (which was downsampled to 8000Hz in the preprocessing). For each target, the collection lasted between 40 minutes to 1 hour. The training, validation, and testing datasets are randomly partitioned with a ratio of 8:1:1 at the sample level. (See IRB note. 4 ) We do plan to release this dataset for public usage after the paper anonymization period.

Acoustic-seismic identification Data Set (ACIDS): ACIDS is an ideal dataset for developing and training acoustic/seismic classification/ID algorithms. The data was collected by 2 co-located acoustic/seismic sensor systems. There are over 270 data runs (single target only) from 9 different types of ground vehicles in 3 different environmental conditions. The ground vehicles were traveling at constant speeds from one direction toward the sensor systems passing the closest point of approach (CPA) and then away from the sensor systems. The microphone data is low-pass filtered at 400 Hz via a 6th-order filter to prevent spectral aliasing and high-pass filtered at 25 Hz via a 1st-order filter to reduce wind noise. The data is digitized by a 16-bit A/D at the rate of 1025 Hz. The CPA to the sensor systems varied from 25m to 100m. The speed varied from 5km/hr to 40km/hr depending upon the particular run, the vehicle, and the environmental condition. We randomly partition the runs into training, validation, and testing datasets with a ratio of 8:1:1. It is more challenging than MOD since domain shift caused by vehicle speed, distance, or terrain between training and testing can be included. No information related to the target types is revealed except the numerical labels.

RealWorld-HAR [18] : This is a public dataset using the accelerometer, gyroscope, magnetometer, and light signals to recognize 8 common human activities (climbing stairs down and up, jumping, lying, standing, sitting, running/jogging, and walking) from 15 subjects. Only the data collected from "waist" is used in our experiments. The sampling rate of all selected sensors is 100Hz. We use the

leave-one-out evaluation strategy where 10 random subjects are used for training, 2 subjects are used for validation, and 3 subjects are used for testing.

Physical Activity Monitoring dataset (PAMAP2) [16] : This dataset contains data of 18 different physical activities (e.g., walking, cycling, playing soccer, etc) performed by 9 subjects using inertial measurement units (IMUs) that are put at the chest, wrist (of dominant arm), and dominant side's ankle respectively. Only data collected from the "wrist" is used in our experiment. Each IMU records readings from a 3-axis accelerometer, gyroscope, and magnetometer. The sampling rates of all sensors are 100Hz. We use the leave-one-out evaluation strategy where 7 random subjects are used for training, and 2 subjects are used for testing.

B Data Preprocessing

In our data preprocessing, we first divide the time-series data into equal-length data samples and further segment each sample into overlapped/non-overlapped intervals. The signals within each interval are processed by the Fourier transform to obtain the spectrum. In this way, both the timedomain information and frequency-domain patterns are preserved. The generated time-frequency spectrogram is further fed into the backbone feature encoders. We define a set of data augmentations in both the time domain before the Fourier transform and the frequency domain after the Fourier transform. For each sample, only one random augmentation from either the time domain or the frequency domain is selected and applied. To further increase the randomness of data augmentations in multimodal applications, we let each modality have a probability of 0.5 to be processed by the selected random augmentation.

B.1 Data Augmentations

We follow the common practices in [22, 7, 10, 19] to define the augmentations used in the time domain and frequency domain respectively.

B.1.1 Time-Domain Augmentations

Here we list the used time-domain augmentations.

Scaling: We multiply the input signals with values sampled from a Gaussian distribution.

Permutation: Given intervals within a sample, we randomly permute the order of the intervals.

Negation: The signal values are multiplied by a factor of -1.

Time Warp: Randomly stretching/distorting the time locations of the signal values based on a smooth random curve.

Magnitude Warp: The magnitude of each time series is multiplied by a curve created by cubicspline with a set number of knots at random magnitudes.

Horizontal Flip: The entire time series of the sample is flipped in the time direction.

Jitter: We add random Gaussian noise to signals.

Channel Shuffle: We randomly shuffle the channels of multi-variate time-series data ( e.g. , X, Y, Z dimensions of three-axis accelerometer input).

Time Masking: We randomly mask a portion of the time intervals within a sample window with 0.

B.1.2 Frequency-Domain Augmentations

Here we list the used frequency-domain augmentations.

Phase Shift: Given the complex frequency spectrum, we add a random value between -π to π to their phase values.

Frequency Masking: We randomly mask a portion of frequency ranges with 0.

C Baselines

Supervised: We train the whole model including the encoder and linear classifier in a fully supervised manner using all available labels.

SimCLR [1] is a simple yet powerful contrastive learning framework proposed for vision tasks. For this work, we randomly formulate batches. During pretraining, we apply random augmentations to generate two different views of each sample, with a contrastive objective of bringing different transformations (augmentations) of the same samples closer while repelling the representations of different samples. The framework optimizes the parameters of the underlying backbone model by minimizing the NT-Xent loss [1]. Similar to [1], we take different samples from the same minibatch as the negative samples. That is, different views of the same sample are considered positive pairs, while views generated from different samples are considered negative pairs.

MoCoV3 [2] is a SOTA contrastive learning framework for Vision Transformers (ViT). It leverages a query encoder f q and a key momentum encoder f k on two stochastically augmented views of a sample to output a query vector q and a key vector k . It uses random batch sampling and learns by maximizing the agreement between the positive encoded query and an encoded key pair. In the latest version of MoCo (V3), for a given query q , the positive key k + is encoded from the same sample as q , while the negative labels k -are encoded keys of other samples within the same mini-batch. Both encoders have a similar structure including a backbone plus a projection head, and the query encoder f q has an additional projection head at the end. The key momentum encoder f k is slowly updated by a query momentum with the query encoder f q .

CMC [20] is a contrastive learning framework focusing on learning from multiview observations. It learns meaningful data representations by contrasting the encoded features from different modalities. To achieve this, it maximizes the agreement between the synchronized representations of different modalities. For each randomly sampled batch with a random augmentation, the backbone model extracts vector representations of each modality. Then, for each pair of modalities, we maximize the similarity between modality representations of the same samples and regard mismatched modality representations from different samples as negative pairs. We sum up the losses for all pairs of modalities to optimize the backbone parameters. For downstream tasks, a linear classification layer is applied on top of concatenated modality representations.

MAE [6] is a self-supervised learning approach based on the auto-encoding paradigm. It incorporates the Transformer architecture and achieves SOTA performance on multiple vision tasks. Unlike contrastive learning, MAE does not depend heavily on random augmentations. During the pretraining, we randomly mask a significant portion ( i.e. , 75%) of each modality input. Instead of dropping the masked patches as in the original MAE paper, we replace them with 0 values to ensure consistent dimensions for the Swin-Transformer and DeepSense operations. A separate encoder and decoder are used for each modality. Before encoding, the modality spectrogram is first projected into fixed-size ( e.g. , 2x2) patches through a convolutional layer, on top of which the modality embeddings are extracted by the modality encoder. Between independent modality encoding and decoding, we first apply multiple fully-connected layers to the concatenated modality features for modality information fusion and then use separate MLP projection layers to get the projected modality embeddings before decoding. This step is created to enable interactions between modalities. Finally, the modality decoder reconstructs the modality input from the projected modality embeddings. The overall objective is to minimize the mean squared error (MSE) between the original modality patches and the reconstructed modality patches on the masked locations. During the inference, the modality decoders are dropped and only modality encoders are used to extract the latent representations from unmasked modality input. In the end, a linear classification layer is applied to the concatenated modality embeddings to serve the downstream task.

Cosmo [14] focuses on contrastive fusion learning from multimodal time-series data to extract modality-consistent information. Cosmo applies separate modality encoders to extract the embedding vector of each modality from the randomly sampled mini-batches. After encoding, each modality embedding is mapped to a hypersphere through an MLP projector and a normalization layer. Then, Cosmo applies a fusion-based feature augmentation to generate P randomly combined features by multiplying the modality embeddings with P normalized random weight vectors. When calculating contrastive loss, these P fusion-based augmented features are considered as positive pairs, while features generated through the same approach but from different samples are treated as negative pairs.

Cocoa [3] extends the self-supervised learning of multimodal sensing data by exploring both the crossmodal correlation and intra-modal separation. Similar to other modality-level contrastive frameworks, Cocoa applies a separate backbone encoder to extract the latent embedding of each modality from the randomly sampled and augmented mini-batch. Cocoa has two losses: Cross-modality correlation loss and discriminator loss. Cross-modality correlation loss maximizes the consistency between different modality embeddings corresponding to the same sample by defining them as hard positive pairs. On the contrary, discriminator loss tends to minimize the agreement within a modality, by separating modality embeddings of irrelevant samples within the mini-batch from each other.

GMC [15] introduces a multimodal contrastive loss function that encourages the geometric alignment of different modality embeddings. Similar to other multimodal contrastive frameworks, samples are randomly batched and augmented. GMC consists of modality-specific encoders and a joint encoder that simultaneously takes all modality data as input. An additional linear layer is used to map the joint embedding to the same space as individual modality embeddings. Then, a shared projection head is then employed to project both the modality embeddings and the joint embeddings before calculating the contrastive loss. To align the local views ( i.e. , individual modality embeddings) with the global view ( i.e. , joint embeddings) in a context-aware manner, GMC minimizes a multimodal contrastive NT-Xent loss by defining the modality-specific embeddings and joint embeddings of the same samples as positive pairs, while treating local-global embedding pairs from different samples as negative pairs.

MTSS [17] is a predictive self-supervised learning framework by exploiting the distinguishability among different data transformations. It uses random augmentation ID prediction as the pretext task during the pretraining. Specifically, MTSS first formulates random batches and applies random augmentation to either time or frequency domain. Each modality is augmented with the selected random augmentation with a probability of 50%. Then, individual modality encoders extract modality embeddings from their input, followed by modality fusion to compute the overall sample embeddings. Different from contrastive frameworks, a shallow classifier is included to classify 'which random augmentation is applied to the input'. A cross-entropy loss is calculated between the predicted augmentation ID and the actual augmentation ID as the pertaining objective. For downstream tasks, only the backbone sample encoder (including the modality encoder and modality fusion layers) is used to extract the sample embeddings, along with a linear classification layer appended at the end of the sample encoder.

TS2Vec [24] proposes to learn representations of time series by simultaneously performing temporal contrastive tasks and instance contrastive tasks at multiple granularities ( i.e. , lengths of sample windows). Instead of creating random batch samples, TS2Vec involves randomly sampled sequences in each batch, with each sequence containing temporally close samples. TS2Vec employs a hierarchical contrasting method to learn representations at multiple sample window granularities. It always regards the same sample under different augmentations and sequence contexts as the positive pairs, while in the instance contrastive task, different samples from separate sequences are regarded as negative pairs, and in the temporal contrastive task, different samples within the same sequence are regarded as negative pairs. At each sample window level, TS2Vec computes both the temporal contrastive loss and instance discrimination loss.

TNC [21] learns time series representations with a debiased contrastive objective to distinguish samples within the temporal neighborhood from temporally distant samples. It utilizes a backbone encoder to extract the feature representations from the time series data in a randomly sampled sequence batch. For each sample, TNC identifies a group of samples with similar timestamps as neighboring samples and a group of distant samples as non-neighboring samples. In this paper, we consider samples within the same sequence as the neighboring samples and samples from different sequences as non-neighboring samples. A discriminator is used to learn the time series distribution by predicting the probability of each sample and its neighboring/non-neighboring samples being in the same window. The objective is to maximize the similarity of neighboring samples while pushing the similarity of non-neighboring samples to zero.

TS-TCC [4] learns robust representation by performing cross-view predictions and contrasting both temporal and contextual information. It randomly groups multiple sequences into a mini-batch. It first generates two views through random augmentations on each sample. For each view, it extracts context vectors of each timestamp from all sample representations up to this timestamp within the sequence with an autoregressive model and then uses the context vectors from one view to predict

the future timesteps of the other view. In the temporal contrastive task, given cross-view predicted representations at a future timestamp, it regards the true future representation at that timestamp from the same sequence as the positive pair and regards samples at that timestamp from other sequences as negative pairs. In the contextual contrastive task, TS-TCC calculates NT-Xent loss by considering different augmentations of the same sample as positive pairs and considering different samples within the same mini-batch as negative pairs.

D Backbone Models

We tested with two different backbone encoders in this paper: DeepSense and Swin-Transformer (SWT for short). Both models process the spectrogram of each input sensing modality separately, before the information fusion between the sensing modalities. For each backbone model, the configuration is tuned to achieve the best-supervised model accuracy.

DeepSense [23] : It is a state-of-the-art neural network model for time-series sensing data processing. Given the time-frequency spectrogram of each sensing modality, it first uses stacked convolutional layers to extract localized modality features within each time interval. Then, modality information fusion is performed by taking the mean of flattened modality features. Finally, the features across time intervals are aggregated through recurrent layers ( e.g. , Gated Recurrent Unit (GRU)). For learning frameworks that operate on modality-level features ( i.e. , FOCAL, CMC, Cosmo, Cocoa, and MAE), we skip the mean fusion among modalities and use individual recurrent layers for each modality, before calculating the pretrain loss.

Swin-Transformer (SW-T) [11] : It is a state-of-the-art Transformer model for processing image data. We adapt it to process the time-frequency spectrogram input. Similar to convolution operations, it adaptively allocates attention within subframe windows of input with hierarchical resolutions. The modality input is first partitioned into patches with a convolutional layer. Then, it gradually extracts features from local and shifted windows with multiple blocks. The shift window operation is introduced to break the boundary of partitioned windows and increase the perception area of each window. Each block consists of multiple self-attention layers. The patch resolution of the feature map is halved at the end of each block by merging neighboring patches while the channel number is doubled, such that the receptive field increases as going into deeper layers while the number of patches within each window is fixed. A separate SW-T encoder is used to extract features from each modality input, after which a stack of self-attention layers is appended for information fusion from multiple modalities. Similarly, for learning frameworks that operate on modality-level features, we skip the attention-based fusion blocks and directly calculate pretrain losses on top of modality features.

E Training Configurations

In this section, we detail the training strategies used in this paper, which are summarized in Table 8. For each framework, the same configuration is mostly shared between different backbone encoders with few exceptions.

During the pertaining, we use the AdamW [12] optimizer with the cosine schedules [13]. The start learning rate is tuned accordingly for each framework according to their convergence situation. We did observe Cosmo [14] is hard to converge in some cases thus we have to reduce its start learning rate. The used batch size is 256, where 64 short sequences of 4 samples are randomly selected in each batch. The constitution of sequences is determined at the initialization and does not change over

training epochs. The temperature is tuned to achieve the best linear classification performance after the finetuning. A weight decay of 0.05 is used as the training regularization.

During the finetuning, we use the Adam [9] optimizer with the step scheduler. Essentially, the learning rate decays by 0.2 at the end of each period. By default, finetuning runs for 200 epochs in total, and each period is 50 epochs. Besides, the weight decay parameter is separately tuned for each framework for the best balance between training fit and validation fit.

The models are trained on a lab workstation with AMD Threadripper PRO 3000WX Processor of 64 cores and NVIDIA RTX 3090 GPUs. The implementation is based on PyTorch 1.14, and the pretraining on a single GPU spans between 3 hours to 4 days among different datasets and backbone encoders.

F Additional Evaluation Results

In this section, we report additional evaluation results and analyses that are not included in the main paper.

F.1 Finetuning: Complete Linear Classification Results

Setup: For each dataset, we apply two backbone encoders (DeepSense and SW-T), and finetune the linear classifier with three different ratios of available labels (100%, 10%, and 1%). For label ratios 10% and 1%, we take 5 random portions of labels for finetuning in each training framework and report the mean and standard deviation among the runs with all testing data. The best result under each configuration is highlighted with the bold text. Besides, we also train a supervised model for each configuration as a reference to the self-supervised frameworks.

Analysis: Table 9, Table 10, Table 11, and Table 12 summarize the complete linear finetuning results on MOD, ACIDS, RealWorld-HAR, and PAMAP2 datasets, respectively.

First, FOCAL consistently demonstrates significant improvements in both accuracy and F1 score across all label ratios compared to other self-supervised learning baselines on the ACIDS, RealWorldHAR, and PAMAP2 datasets. In the case of the MOD dataset under 1% labels, FOCAL achieves similar accuracy to TNC with the DeepSense encoder but beats TNC by 10.56% with the SW-T encoder. These results underline the superior performance of FOCAL in multimodal time series sensing data and emphasize the importance of the underlying relationship between the shared and private modality features through time.

Second, the performance improvements persist across backbone encoders and different label ratios, proving the advantage of FOCAL in improving the label efficiency during downstream finetuning. Although there are a few cases where some baselines perform close to FOCAL ( e.g. , TNC with DeepSense encoder on MOD dataset under 1% labels), such comparability does not persist across encoders.

Third, FOCAL shows comparable performance to the supervised model when all available labels ( i.e. , 100%) are used in the training. However, when fewer labels are available, FOCAL shows a larger advantage over the supervised oracle, demonstrating its capability to better leverage the limited available labels in adapting to downstream tasks. On average, FOCAL surpasses the supervised model by 1.37% with 100% labels, 15.04% with 10% labels, and 68.39% with 1% labels. By learning semantically meaningful multimodal representations from the massive unlabeled inputs during the pretraining phase, FOCAL can effectively utilize limited data labels during the finetuning process. This is especially reflected in the MOD results, where we have around 6 times more data in pretraining than the finetuning and achieve 3.49% and 9.58% improvement over the supervised model.

Fourth, between the backbone encoders, we found FOCAL brings more relative performance improvement to SW-T than DeepSense compared to their supervised versions. With FOCAL training, SW-T beats DeepSense in two out of four datasets ( i.e. , MODand RealWorld-HAR), while DeepSense is always the better encoder architecture with supervised training. Besides, the performance improvement on SW-T is more significant when the number of available labels is low during the finetuning ( i.e. , 10% and 1%) since larger performance gaps are observed between FOCAL and supervised models.

F.2 Finetuning: Complete KNN Classification Results

Setup: In addition to linear probing, we further evaluate the self-supervised frameworks on four datasets using the K-Nearest-Neighbors (KNN, K=5) classifier without introducing new parameters. This evaluation method allows us to examine the quality of learned representations without new

training steps. We first construct a KNN estimator using the encoded sample features and corresponding labels from finetuning data. For multi-modal frameworks, we directly concatenate modality embeddings as the sample-level representations. Subsequently, the estimator predicts the test labels according to the labels of neighboring samples in the supervised set X s and computes the testing accuracy accordingly.

Analysis: The complete evaluation results with the KNN classifier are reported in Table 13. FOCAL consistently surpasses the performance of other self-supervised learning baselines in most cases. The KNN evaluation results are mostly consistent with the linear classification results, but there are

Dataset

MOD

ACIDS

RealWorld-HAR

PAMAP2

Encoder

Framework

ARI

NMI

ARI

NMI

ARI

NMI

ARI

NMI

DeepSense

CMC

0.3936 ± 0.0125

0.5224 ± 0.0206

0.2926 ± 0.0156

0.5833 ± 0.0051

0.2187 ± 0.1094

0.4354 ± 0.1713

0.3024 ± 0.0118

0.5063 ± 0.0120

Cosmo

0.1384 ± 0.0540

0.2552 ± 0.0803

0.5217 ± 0.0074

0.6416 ± 0.0184

0.4231 ± 0.2726

0.5318 ± 0.2564

0.3583 ± 0.0781

0.5212 ± 0.0671

Cocoa

0.3502 ± 0.0184

0.4444 ± 0.0135

0.5453 ± 0.0229

0.6767 ± 0.0184

0.3385 ± 0.1826

0.4792 ± 0.1940

0.3493 ± 0.0230

0.5091 ± 0.0184

GMC

0.1982 ± 0.0674

0.3925 ± 0.0416

0.2490 ± 0.0403

0.5296 ± 0.0150

0.3433 ± 0.1836

0.4794 ± 0.1978

0.3078 ± 0.0194

0.5092 ± 0.0221

FOCAL

0.3929 ± 0.0222

0.5067 ± 0.0226

0.5723 ± 0.0440

0.7213 ± 0.0432

0.4400 ± 0.2465

0.5545 ± 0.2437

0.4759 ± 0.0695

0.6037 ± 0.0558

SW-T

CMC

0.4314 ± 0.2716

0.5413 ± 0.2612

0.3604 ± 0.0119

0.5881 ± 0.0009

0.4014 ± 0.0528

0.5275 ± 0.0532

0.3718 ± 0.0480

0.5562 ± 0.0401

Cosmo

0.2865 ± 0.1521

0.4140 ± 0.1946

0.4436 ± 0.0145

0.5469 ± 0.0015

0.0029 ± 0.0020

0.0107 ± 0.0025

0.2425 ± 0.0301

0.3604 ± 0.0347

Cocoa

0.4281 ± 0.2314

0.5308 ± 0.2405

0.4363 ± 0.0020

0.6824 ± 0.0261

0.2487 ± 0.0053

0.3897 ± 0.0024

0.3658 ± 0.0540

0.5330 ± 0.0472

GMC

0.3973 ± 0.2177

0.4940 ± 0.2184

0.2055 ± 0.0029

0.4971 ± 0.0066

0.3050 ± 0.0076

0.4342 ± 0.0052

0.2794 ± 0.0206

0.5044 ± 0.0329

FOCAL

0.4660 ± 0.2737

0.5693 ± 0.2579

0.6050 ± 0.1027

0.7389 ± 0.0774

0.4319 ± 0.0851

0.5462 ± 0.0717

0.4785 ± 0.0914

0.6130 ± 0.0730

also a few exceptions. With the SW-T encoder, FOCAL exceeds the best baseline by an average of 4.85%. When using DeepSense as the encoder, FOCAL outperforms the most competitive contrastive framework baseline by 1.18% across all datasets. In the RealWorld-HAR dataset, DeepSense with MAE achieves higher accuracy than FOCAL, but it fails in the linear classification scenario and fails to generalize to other datasets and backbone encoders. In comparison to other contrastive learning baselines, FOCAL still demonstrates its superiority in KNN classification. Between the two encoders on FOCAL, SW-T outperforms DeepSense in three out of four datasets, which further shows the benefits FOCAL brings to SW-T training.

F.3 Complete Clustering Results

Setup: We further evaluate the clustering performance of FOCAL with other multimodal selfsupervised learning baselines, including CMC, Cosmo, Cocoa, and GMC. We apply K-means clustering to the encoded embeddings from each framework, by setting the number of clusters equal to the number of unique classes in the testing dataset. As mentioned before, the preferred cluster structure by the SSL frameworks should align well with the underlying ground-truth labels in addition to presenting clear separation among the clusters. Following this objective, we quantitatively assess the clustering performance by independently calculating the Adjusted Rand Index (ARI) and the Normalized Mutual Information (NMI) of each modality to provide an accurate comparison of the alignment between the pretrained clusters and ground-truth classes. ARI evaluates the similarity between the clustering assignments generated by the K-means clusters and the label distribution of the test data. With a value range of -1 to 1, ARI indicates a high degree of agreement between the two clusterings when close to 1, random agreement when close to zero, and a clustering performance worse than random when approaching -1. NMI serves as an external metric for measuring the clustering quality. A score close to 1 indicates a perfect correlation between the clusterings, and a score of 0 demonstrates no mutual information between the clusters. Lastly, we performed t-SNE to qualitatively visualize the sample embeddings after concatenating the modality embeddings.

Analysis: In Table 14, we present the clustering results with the average and standard deviation of ARI and NMI across all modalities. As the results show, FOCAL consistently achieves the highest or similar ARI scores in comparison to other multimodal contrastive frameworks. When using SW-T as the encoder, FOCAL outperforms the strongest baseline by an average ARI margin of 8.33% and an average NMI margin of 4%. With DeepSense as the encoder, FOCAL surpasses the best baseline by an average ARI margin of 4.61% and an average NMI margin of 3.35%. Although CMC exhibits comparable performance for the MOD dataset when using DeepSense as an encoder, FOCAL with DeepSense exceeds CMC by an average of 16.8% and 8.47% in ARI and NMI across the four datasets. These results confirm our claim that FOCAL produces higher quality modality representations compared to the baseline multi-modal contrastive frameworks. We also found the general ARI and NMI values are relatively low because there could be multiple perspectives affecting the cluster structures that lead to complicated underlying semantics while we only evaluate one perspective among them.

Figures 10 and 11 represent the t-SNE visualizations of the encoded sample embeddings by FOCAL. Wecan observe a clear separation between individual clusters on MOD, ACIDS, and RealWorld-HAR, indicating that FOCAL effectively captures the distinct characteristics of each class. However, for the PAMAP2 dataset, we notice various overlaps between different embeddings. This observation suggests that the underlying structure of the PAMAP2 dataset is more challenging to differentiate compared to other datasets, potentially due to similarities among a large number of classes with 18 different physical activities. This discovery is also consistent with our linear probing results,

(a) MOD

(b) ACIDS

(c) RealWorld-HAR

(d) PAMAP2

(a) MOD

(b) ACIDS

(c) RealWorld-HAR

(d) PAMAP2

(a) FOCAL

(b) CMC

(c) GMC

(d) Cocoa

which perform slightly worse on the PAMAP2 dataset. Besides, we provide the t-SNE visualization comparison between FOCAL, CMC, GMC, and Cocoa with Swin-Transformer encoder in Figure 12.

F.4 Complete Additional Downstream Task Results

Setup: We collected additional data samples for the MOD dataset and finetuned our pretrained models from previous experiments. Specifically, we evaluated our pretrained models by finetuning the classifier layer on two downstream tasks, distance classification, and speed classification tasks, with data obtained from different environments and new types of vehicles. These alterations in the data lead to domain adaptation, referring to changes in the data's distribution. For speed classification, the classifier predicts the speed of the moving object between 5, 10, 15, and 20 mph. For distance classification, the classifier outputs whether the detected object is close, near, or far away.

Three metrics are evaluated in this experiment. In addition to the normal accuracy and (macro) F1 score, we also define a new metric called correlated accuracy . It considers the semantical distances between different classes and assigns different penalties to different misclassifications cases. Intuitively, for a sample with ground truth speed 5, a misclassification of speed 20 should be assigned more penalty than a misclassification of speed 10. Given a sample label pair ( x s i , y s i ) , the predicted label y i , and the number of classes C , we define the maximum class distance as max( i, C -i -1) ,

then the correlated accuracy is calculated by

where the penalty of misclassification is linearly interpolated according to the distance of the predicted label and the ground truth label, divided by the maximum distance to this class. The value range of the correlated accuracy is still [0 , 1] , where 0 means the worst and 1 means the best.

Analysis: We observe a significant drop in performance on most of the self-supervised learning frameworks on speed classification. When using SW-T as the encoder, FOCAL still dominates the performance over other baselines, exceeding the strongest baseline by 6.07% accuracy and 10.32 % F1 score. When using DeepSense as the encoder, FOCAL also achieves comparable high performance as the current baselines. The advantage of FOCAL persists in the correlated accuracy metric where the physical correlations among classes are counted. Considering the heterogeneous finetune tasks, the potential domain shift, and the leading performance, we conclude that FOCAL is promising in learning fundamental feature patterns from multi-modal sensing data that could serve an extensive set of downstream tasks.

F.5 Ablation Study Results

Steup : We first briefly introduce the compared variants of FOCAL in our ablation study. In these variants, they are set up in the same way as FOCAL except for the places we explain below.

FOCAL-noPrivate: We remove the private modality space and its related contrastive task but only apply the cross-modal matching task.

FOCAL-noOrth: We keep the private modality space, but do not enforce the orthogonality constraint between the shared feature and private feature of the same modality, and the private features between pairs of modalities.

FOCAL-wDistInd: We replace the geometrical orthogonality constraint with statistical independence between modality embedding distributions. Specifically, we follow the approach proposed in [8] to disentangle the distribution of latent subspaces, which minimizes the mutual information between shared-private spaces of the same modality and privateprivate spaces between two modalities. Given two embedding distributions, it minimizes the KL divergence between their joint distribution and the product of two marginal distributions. Following the density-r atio trick, we train a classifier consisting of several fully-connected layers to discriminate samples from the originally matched pairs of embeddings and the randomly selected embedding pairs, which has been shown to approximate the density ratio needed to estimate the KL divergence within sample batches. Similar to GAN [5], we train the discriminator alternatively with modality encoders until convergence.

FOCAL-noTemp: We remove the temporal structural constraint proposed in FOCAL.

FOCAL-wTempCon: We replace the temporal structural constraint with a temporal contrastive task. Given a modality, we regard close sample pairs within a short sequence as positive samples and regard distant sample pairs from different short sequences as negative samples, and conduct discrimination between positive samples and negative samples.

Analysis: The complete ablation results on DeepSense encoder are presented in Table 16. Similar to our observations with SW-T encoder, all of the three components introduced in FOCAL (private space, orthogonality constraint, and temporal constraint) contribute positively to the downstream performance. However, we do find the orthogonality constraint and the temporal constraint play a more important role in the performance improvement with the DeepSense encoder than that with SW-T encoder on ACIDS, RealWorld-HAR, and PAMAP2 datasets. Besides, it is noticeable that distributional independence contributes positively to FOCAL on ACIDS and RealWorld-HAR datasets but contributes negatively to FOCAL on MOD and PAMAP2 datasets. We leave it as future work to investigate more into the role of distributional independence in factorizing the latent space within the multimodal contrastive learning paradigm.

G Limitations and Potential Extensions

Assumption on Modality Synchronization: We assume the signals simultaneously arrived at all sensory modalities such that the information at different modalities is synchronized. However, in some scenarios, different signals propagate at significantly different speeds. For instance, light travels much faster than sound. The shared modality embeddings can not be directly matched for the same samples without signal synchronizations between the modalities.

Computational Complexity of Pretraining Loss: In the current design, we take all pairs of modalities to compute their shared space consistency loss and private space orthogonality loss, which leads to O ( K 2 ) complexity to the number of modalities K . On one hand, we assume the modality number is limited to a handful count in most sensing applications; on the hand, we leave it as one of our future work to reduce the computational complexity in pretraining loss calculation.

Dependency on Data Augmentations: Our current contrastive learning paradigm is still not fully self-supervised, because we need to design a set of transformations ( i.e. , data augmentations) for the private modality feature learning. However, different from image data, designing proper labelinvariant data augmentations for time-series data can be challenging in some applications, especially when we do not have knowledge about the potential downstream tasks. One potential solution is to integrate the masked reconstruction learning paradigm into the framework, such that data augmentations can be avoided or less depended on.

Multi-Device Collaboration: This paper focused on multi-modal collaborative sensing settings while multi-device collaboration is not fully considered. The general design of contrastive learning in factorized latent space is extensible to the multi-device setting, but more designs need to be introduced to further address the heterogeneity contained in different vantage points and the scalability issues related to the number of participating sensor nodes in large-scale distributed sensing scenarios.

Resiliency Against Domain Shift: Although FOCAL improves the downstream performance of contrastive learning from multimodal sensing signals, it still exhibits relatively low accuracy in speed classification when data is collected from a different environment. There are multiple environmental factors that can lead to such degradations, including terrain, wind, sensor facing directions. We hope to integrate domain-invariant considerations into the learning objective in the future such that apparently task-unrelated information is decoupled and removed from the pretrained embedding space, and the model resiliency can be significantly enhanced.

References

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML) , 2020.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision (CVPR) , 2021.

Shohreh Deldari, Hao Xue, Aaqib Saeed, Daniel V. Smith, and Flora D. Salim. Cocoa: Cross modality contrastive learning for sensor data. ACM Interact. Mob. Wearable Ubiquitous Technol. (IMWUT) , 6(3), 2022.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. arXiv preprint arXiv:2208.06616 , 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS) , 2014.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Bulat Khaertdinov, Esam Ghaleb, and Stylianos Asteriadis. Contrastive self-supervised learning for sensor-based human activity recognition. In IEEE International Joint Conference on Biometrics (IJCB) . IEEE, 2021.

Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning (ICML) , 2018.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Dongxin Liu, Tianshi Wang, Shengzhong Liu, Ruijie Wang, Shuochao Yao, and Tarek Abdelzaher. Contrastive self-supervised representation learning for sensing signals from the time-frequency perspective. In IEEE International Conference on Computer Communications and Networks (ICCCN) , 2021.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision (CVPR) , 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR) , 2017.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR) , 2017.

Xiaomin Ouyang, Xian Shuai, Jiayu Zhou, Ivy Wang Shi, Zhiyuan Xie, Guoliang Xing, and Jianwei Huang. Cosmo: Contrastive fusion learning with small data for multimodal human activity recognition. In International Conference on Mobile Computing And Networking (MobiCom) , 2022.

Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Geometric multimodal contrastive representation learning. In International Conference on Machine Learning (ICML) , 2022.

Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In International Symposium on Wearable Computers (ISWC) , 2012.

Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien. Multi-task self-supervised learning for human activity detection. ACM Interact. Mob. Wearable Ubiquitous Technol. (IMWUT) , 3(2), 2019.

Timo Sztyler and Heiner Stuckenschmidt. On-body localization of wearable devices: An investigation of position-aware activity recognition. In IEEE International Conference on Pervasive Computing and Communications (PerCom) , 2016.

Chi Ian Tang, Ignacio Perez-Pozuelo, Dimitris Spathis, and Cecilia Mascolo. Exploring contrastive learning in human activity recognition for healthcare. arXiv preprint arXiv:2011.11542 , 2020.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision (ECCV) , 2020.

Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations (ICLR) , 2021.

Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kuli´ c. Data augmentation of wearable sensor data for parkinson's disease monitoring using convolutional neural networks. In ACM international conference on multimodal interaction (ICMI) , 2017.

Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In International Conference on World Wide Web (WWW) , 2017.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAI Conference on Artificial Intelligence (AAAI) , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a self-supervised contrastive learning framework for extracting comprehensive features from multimodal time-series sensing signals. The framework refines contrastive learning by modeling shared and private features and exploiting orthogonality constraints. Moreover, a Temporal Structural Constraint is proposed to make up for the locality of processing the temporal information locality appropriately in contrastive learning. Finally, to demonstrate the effectiveness of the proposed FOCAL, the authors conduct extensive experiments on multiple multimodal sensing datasets.\n\nSTRENGTHS:\n1. Extensive experiments are conducted on four benchmarks: MOD, ACIDS, RealWorld-HAR and PAMAP.\n2. This work is relatively novel and technically sound.\n3. The description and explanation of motivation are clear.\n4. It seems to be a pioneer work to consider and solve temporal information locality.\n5. The results are promising.\n\nWEAKNESSES:\n1. In the experiments, what is the intuition of the hyperparameter setting of Eq.5? Sensitivity analysis appears to be lacking.\n2. Whether the proposed temporal structural constraint is also applicable to other comparative learning frameworks to solve the problem of temporal information locality.\n3. From the ablation results, the private space seems to be important for contrastive learning, somewhat similar to intra-modal contrastive loss. This work is based on strong orthogonality constraints, but in fact, the performance of the FOCAL-noOrth setting has not dropped too much. I am worried whether the orthogonality assumption is too strong.\n4. Eq.4 is essentially a triplet contrastive loss, are there possible alternatives? Moreover, the value of the “margin ” does not seem to be mentioned in the paper.\n5. I think Figure 6 should be compared with the visualization results of the best baseline (or some variants of FOCAL), so as to reflect the differences or superiority. like Supervised vs. MoCo vs. FOCAL? Otherwise, it feels like visualization for visualization's sake.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors provide a comprehensive discussion about limitations and potential extensions in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multimodal contrastive learning framework (FOCAL) for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. In which, FOCAL first decouples each modality into two subspaces, i.e., shared and private spaces, and uses simple soft orthogonal loss as an objective. Then, the temporal structural constraint for modality features is proposed to enhance feature temporally. The authors use four multimodal sensing datasets with two backbone encoders and two classifiers to conduct many experiments.\n\nSTRENGTHS:\n1. The motivation and contribution of the paper are clearly described.\n2. Many experiments are presented in the manuscript.\n\nWEAKNESSES:\n1.\tThe biggest concern is the effectiveness of multimodal feature factorization. From the line 161, this work uses the simple soft orthogonal constraint to decouple or factorize each modality feature. However, the single orthogonal constraint is too simple and may not decouple multimodal features well. In particular, the information is easily leaked between modalities, and many prior works have tried to solve this problem such as [1-3]. In addition, there is no visualization result for both subspaces in the experimental part, so it is difficult to determine whether the factorization is effectiveness.\n\n[1] Li Y, Wang Y, Cui Z. Decoupled Multimodal Distilling for Emotion Recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 6631-6640.\n\n[2] Ouyang J, Adeli E, Pohl K M, et al. Representation disentanglement for multi-modal brain MRI analysis[C]//Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings 27. Springer International Publishing, 2021: 321-333.\n\n[3] Yang D, Huang S, Kuang H, et al. Disentangled representation learning for multimodal emotion recognition[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 1642-1651.\n\n2.\tIn the line 29, the word “ignore” may not be as appropriate. For example, the CLIP contrastive frameworks does not emphasize shared or private spaces, but we cannot assume that this work ignores heterogeneity, only that they do not explicitly consider. Therefore, it might be better to replace \"ignore\" with \"explicitly consider\".\n3.\tIn the experiments, why not consider some more common multimodal scenarios, such as the MOSI [4] and MOSEI [5] datasets, which contain the three most common modalities in the real world, i.e., Language, Visual and Acoustics, and whose features are packaged as time-series-based, in line with the settings in this paper.\n\n[4] Zadeh A, Zellers R, Pincus E, et al. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[J]. arXiv preprint arXiv:1606.06259, 2016.\n\n[5] Zadeh A A B, Liang P P, Poria S, et al. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018: 2236-2246.\n\n4.\tThe public datasets lack the necessary references, please check carefully. (Line 199-206)\n\nQUESTIONS:\nWill the MOD dataset be open-released?\n\nLIMITATIONS:\nNo limitations are provided in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper proposes FOCAL, a contrastive learning method for multimodal time-series signals. The main idea is to first encode each modality into a factorized orthogonal space, and design four pretraining objectives that enforce modality consistency, transformation consistency, orthogonality constraint and temporal locality constraint. Experimental results demonstrate the efficacy of FOCAL.

STRENGTHS:
+ The two motivations for designing the contrastive objectives make sense to me --- modality-private features can contribute in contrastive learning, and the temporal structural constraint also seems valid. 

+ The authors conduct an extensive evaluation of FOCAL --- it is compared with many contrastive learning baselines and evaluated on four multimodal datasets. The improvement achieved by FOCAL is large. 

+ The ablation study is great and provides a good analysis on the four pretraining objectives. 

+ The paper is clearly presented and easy to follow.

WEAKNESSES:
While I generally agree with the motivations and the proposed four objectives, Eq (5) is a combination of four objectives, and it seems hard to balance the four terms. I wonder if some of the training objectives may compete during training, and how do the authors set the three hyper-parameters $\lambda_p$, $\lambda_o$ and $\lambda_t$? Does it require some manual hyper-parameter tuning? Jointly optimizing the four objectives does not look like an easy task.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes the limitations have been well discussed in Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FOCAL, a new contrastive learning framework for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. Unlike previous frameworks that focused solely on shared information between sensory modalities, FOCAL also factors in exclusive modality information crucial for understanding physical semantics. It also properly handles temporal information locality, which previous time series contrastive frameworks have not done. FOCAL creates a factorized latent space of shared and private (modality-exclusive) features from each modality and applies a temporal structural constraint on these features. Through extensive testing on four multimodal sensing datasets, FOCAL consistently outperforms the state-of-the-art baselines in downstream tasks, demonstrating its superior performance.\n\nSTRENGTHS:\n- This paper is excellently written, with clear logic and motivation that is easy to understand. \n- The results presented are impressive and Figure 1 does a fantastic job in aiding the understanding of the paper. \n-  The idea of modeling modality-shared and specific information is very great. \nGreat work!\n\nWEAKNESSES:\nI couldn't find any significant weaknesses in this paper. However, I should note that I'm not an expert in this topic. While I am familiar with multimodality, and contrastive, my understanding of time series analysis is limited\n\nQUESTIONS:\n- Is there any hyperparameter search performed for λp, λo, and λt which control the weights of each loss component? Additionally, was there any hyperparameter search carried out for other methods?\"\n- \"Why does the FOCAL-wDistInd perform worse than the noOrth model? Does the replacement of orthogonality with distributional independence contribute to this difference?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
The proposed FOCAL is a constrasitive learning method or time-series signals. The major contributions are:
1. For multi-modality samples, FOCAL learns shared feature that are similar between modalities, and a private features that are similar intra-modality but different across modalities.
2. The shared and private features are orthoigonal, so that they focus on different aspects.
3. The Temporal Locality Constraint which restricts the average distance within a short sequence to be less than the average distance between two random sequences. The "average" operation allows some samples to be similar even though they are from distant samples in time-domain.

Extensive experiments are conducted with 4 datasets and 12 baselines, which gives consistent SOTA performance on classification and clustering tasks.

STRENGTHS:
1. The idea of private feature is novel, and effective according to the ablation study.
2. Although the temporal structural constraint is a simple modification, it provides noticable performance increase.
3. The proposed method is simple and easy to follow. Detailed network design, data processing, training configurations are provided in Appendix.

WEAKNESSES:
1. The proposed contribution is mostly the loss functions, e.g., private and shared feature constrative loss, temporal locality loss, etc. I believe these loss can be applied to existing methods with simple modification. It would be very strong to show that the proposed loss serves as a plugin that enhance other contrastive learning methods.

QUESTIONS:
See weakness.

LIMITATIONS:
The paper discussed little on the techinical limitation and future directions. It would be great the discuss more on failure cases.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27