SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shengzhong Liu*, Tomoyoshi Kimura†, Dongxin Liu‡, Ruijie Wang†, Jinyang Li†, Suhas Diggavi§, Mani Srivastava§, Tarek Abdelzaher†
*Shanghai Jiao Tong University, †University of Illinois at Urbana-Champaign
$^\ddagger$ Meta,  $^\S$ Univeristy of California, Los Angeles
shengzhong@sjtu.edu.cn, {tkimura4, ruijiew2, jinyang7, zaher}@illinois.edu
dxliu@meta.com, {suhas, mbs}@ee.ucla.edu
This paper proposes a novel contrastive learning framework, called FOCAL, for extracting comprehensive features from multimodal time-series sensing signals through self-supervised training. Existing multimodal contrastive frameworks mostly rely on the shared information between sensory modalities, but do not explicitly consider the exclusive modality information that could be critical to understanding the underlying sensing physics. Besides, contrastive frameworks for time series have not handled the temporal information locality appropriately. FOCAL solves these challenges by making the following contributions: First, given multimodal time series, it encodes each modality into a factorized latent space consisting of shared features and private features that are orthogonal to each other. The shared space emphasizes feature patterns consistent across sensory modalities through a modal-matching objective. In contrast, the private space extracts modality-exclusive information through a transformation-invariant objective. Second, we propose a temporal structural constraint for modality features, such that the average distance between temporally neighboring samples is no larger than that of temporally distant samples. Extensive evaluations are performed on four multimodal sensing datasets with two backbone encoders and two classifiers to demonstrate the superiority of FOCAL. It consistently outperforms the state-of-the-art baselines in downstream tasks with a clear margin, under different ratios of available labels. The code and self-collected dataset are available at https://github.com/tomoyoshki/focal.
As a representative self-supervised learning (SSL) paradigm, contrastive learning (CL) has achieved unprecedented success in vision tasks [3, 14, 4, 12, 55] and are increasingly leveraged in learning from time series [10, 7, 63, 65, 36, 58]. However, many IoT applications [39, 56, 33] rely on heterogeneous sensory modalities to collaboratively perceive the physical surroundings and lead to a more complicated learning space. This paper aims to build a contrastive learning framework that maximally extracts complementary information from multimodal time-series sensing signals [23, 60]. The key challenge is to define appropriate similarity/distance measures (i.e., who should be close to whom) in the joint multimodal embedding space that facilitates the downstream tasks [53, 57].
Existing contrastive frameworks for time series either ignore the heterogeneity among sensory modalities and are limited to instance-level discrimination [22, 17], or are designed to extract only shared information across sensory modalities [48, 6, 35]. For instance, as a representative framework, CMC [48] builds on the hypothesis that only information shared across views corresponds to the 37th Conference on Neural Information Processing Systems (NeurIPS 2023).
Figure 1: Overview of the FOCAL framework. Best viewed in color.
actual physical target. We argue that the strength of multimodal sensing lies in that the collaborating modalities not only share information but also enhance each other by providing complementary information exclusive to the modalities. The pretraining objective in contrastive learning should be calibrated to extract all semantically meaningful and generalizable patterns.
In addition, we observe that existing contrastive frameworks for time series do not handle the temporal information locality in a proper way. Temporal contrastive works (e.g., TNC [49]) force the temporally close samples to be positive pairs of similarity 1 and force the temporally distant samples to be negative pairs of similarity 0, which may contradict long-term seasonality patterns. For example, a circling motorcycle may cause periodical vibration patterns to nearby microphone arrays, violating the above strict contrastive objective. Alternatively, TFC [65] optimizes the consistency between time-domain and frequency-domain representations, preventing neither encoder from simultaneously extracting features from the time-frequency spectrogram, which are known to achieve superior performance in learning from sensing signals [59].
To overcome these limitations, we propose a novel contrastive framework, FOCAL, for self-supervised representation learning from multimodal time-series sensing data. An overview is presented in Figure 1. Motivated by [43], we encode the input (i.e., fixed-length window of sensor readings) of each modality into a factorized orthogonal latent space, composed of a shared space and a private space. The shared features and private features of the same modality, as well as the private features between different modalities, are mutually orthogonal to emphasize their semantical independence. The shared space is designed
Figure 2: Information diagram between CMC and the proposed FOCAL. Figure adapted from [48]. Blue color denotes used information sectors.
to capture information consistency across modalities, while the private space captures modality-exclusive but transformation-invariant information. As we show in Figure 2, FOCAL outperforms CMC [48] by further emphasizing modality-exclusive sectors of target information. Besides, we define a temporal structural constraint to the distance measure of modality embeddings through a loose ranking constraint between temporally "close sample pairs" and "distant sample pairs". During the pertaining, we use a sampling strategy that randomly selects multiple short sequences with fixed length (e.g., 4 samples per sequence) to constitute training batches. Instead of performing temporal contrastive tasks as [49], we confine the average intra-sequence distance between temporally neighboring samples (i.e., 3 in Figure 1) to be no larger than the average inter-sequence distances between temporally distant samples (i.e., 4 in Figure 1). It simultaneously addresses overall temporal information locality and tolerates occasional violations of locality caused by periodicity, and it turns out to accelerate the pretraining convergence in practice.
On the one hand, we justify the feature orthogonality constraints from two aspects. First, the shared features and private features of a modality should be orthogonal, such that the private space can avoid reusing the shared information but instead exploit modality-exclusive discriminative information. Second, the private features of different modalities should be orthogonal to each other. Otherwise, the overlapped semantics should be included in their shared space. In summary, the orthogonality
constraint is imposed to refine the heterogeneous modality-exclusive information that is undervalued in existing cross-modal contrastive tasks.
On the other hand, the temporal information locality within time series is defined through coarse-grained distance orders for two reasons. First, we do not always regard temporally close samples as positive pairs (with similarity 1), such that fine-grained differences between neighboring samples can be included. Second, we only enforce the temporal constraint at the statistical average scale by comparing the average intra-sequence distances to the average inter-sequence distances to fit potential exceptions caused by long-term seasonal signal patterns and significantly reduce the computational complexity because we avoid traversing the ranking losses within all sample triplets [15].
In summary, we define the following four perspectives of pretraining objectives in FOCAL.
- Modality Consistency in Shared Space: In the shared space, we push the features of different modalities of the same sample to be similar to each other compared to randomly mismatched modality features from two temporally distant samples.
- Transformation Consistency in Private space: In the private space, we push the modality features under two random augmentations to be similar to each other compared to modality features from two samples.
- Orthogonality Constraint: We enforce the shared feature and private feature of the same modality, as well as private features of different modalities, to be orthogonal to each other.
- Temporal Locality Constraint: We restrict the average distance of samples within a short time sequence to be no larger than the average distance between two random sequences.
We extensively evaluate FOCAL against eleven state-of-the-art baselines on four multimodal sensing datasets with two different backbone encoders (DeepSense [59] and Swin-Transformer [26]). The finetuning performance is evaluated with two light-weight classifiers (i.e., linear and KNN classifier). It consistently achieves higher accuracy and F1 scores than the baselines. We also break down the contribution of individual components through step-by-step ablation studies.
Assume we have a set of  $P$  sensory modalities  $\mathcal{M} = \{M_1, M_2, \ldots, M_P\}$ , and a large set of  $N$  unlabeled pretraining data from all sensory modalities  $\mathcal{X} = \{\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_N\}$ . Each sample is a fixed-length window of signals from all sensory modalities. For each sample  $\mathbf{x}_i$ , we use  $\mathbf{x}_{ij}$  to denote the input from sensory modality  $M_j$ . The original input of each sensory modality is a multivariate time series, and we apply Short-Time Fourier Transform (STFT) as the preprocessing procedure to extract the sample time-frequency representation. As we introduce in Appendix B, the processed modality input is  $\mathbf{x}_{ij} \in \mathbb{R}^{C \times I \times S}$ , where  $C$  denotes the number of input channels,  $I$  denotes the number of time intervals within a sample window, and  $S$  denotes the spectrum length after applying the Fourier transform to each interval<sup>1</sup>. We have a set of backbone encoders  $\mathcal{E} = \{E_1, E_2, \ldots, E_P\}$  such that the encoder  $E_i$  of modality  $M_i$  can encode the modality input  $\mathbf{x}_{ij}$  into an embedding vector  $\mathbf{h}_{ij} = E_j(\mathbf{x}_{ij}) \in \mathbb{R}^K$ , where  $K$  is the unified dimension of modality embedding vectors. We use  $\langle \mathbf{h}_{ij}, \mathbf{h}_{i'j'} \rangle$  to denote the inner product between two embedding vectors  $\mathbf{h}_{ij}$  and  $\mathbf{h}_{i'j'}$ . The encoders extract both the time and frequency patterns from the preprocessed input. We also have a small set of  $N'$  supervised samples for finetuning  $\mathcal{X}^s = \{\mathbf{x}_1^s, \mathbf{x}_2^s, \ldots, \mathbf{x}_{N'}^s\}$ , where each sample  $\mathbf{x}_i^s$  is associated with a label  $y_i^s$ . Please note  $\mathcal{X}^s$  is not necessarily a subset of the pretraining set  $\mathcal{X}$  due to potential domain adaptations, i.e.,  $\mathcal{X}^s \nsubseteq \mathcal{X}$ . The objective of self-supervised pretraining is to use the unlabeled dataset  $\mathcal{X}$  to optimize the parameters of modality encoders  $\mathcal{E}$  such that the model accuracy finetuned on the supervised dataset  $\mathcal{X}^s$  is maximized.
3 FOCAL Framework
In this section, we start with an overview of the framework and then separately introduce the two main functional components: (1) contrastive learning in factorized orthogonal latent space, and (2) temporal structural constraint.
For notational simplicity, we do not differentiate the sampling rates of sensory modalities, but we can handle the case since each modality has a separate feature encoder.
The key questions to answer in designing a contrastive learning framework include the selection of positive/negative pairs and the contrastive loss functions. Compared to image input about which we have almost no knowledge without human annotations, the meta-level information of multimodal time-series input, i.e., cross-modal correspondence and temporal information locality, can be effectively leveraged during the pretraining to shape the learned joint embedding space. As we show in Figure 1, FOCAL groups multiple randomly sampled fixed-length (i.e.,  $L$  samples) short sequences of samples into a batch  $\mathcal{B}$ , with cardinality  $|\mathcal{B}| = B$ , and only conducts contrastive comparisons within the batch without any memory banks. Each modality input  $\mathbf{x}_{ij}$  goes through two randomly selected augmentations to generate two augmented versions, and each augmented input is separately encoded by the modality encoder  $E_{j}$  to get two versions of modality embeddings  $\mathbf{h}_{ij}$  and  $\tilde{\mathbf{h}}_{ij}$ . The encoder network structures are not this paper's original contribution, so we leave them in Appendix D.
As the output of modality encoding, each modality embedding  $\mathbf{h}_{ij}$  is projected through a non-linear multilayer perceptron (MLP) projector into two orthogonal embeddings, a shared embedding  $\mathbf{h}_{ij}^{shared}$  and a private embedding  $\mathbf{h}_{ij}^{private}$ . The two embeddings share all encoder layers except the MLP projector. Separate contrastive learning tasks are applied to each embedding to capture different aspects of information. At the same time, shared-private and private-private modality features of the same sample are mutually orthogonal. In addition, we apply a temporal structural constraint between intra-sequence distances and inter-sequence distances to both projected modality embeddings.
3.2 Multimodal Contrastive Learning in Factorized Orthogonal Space
In multimodal collaborative sensing, information from different sensory modalities is not fully overlapped, thus extracting modality-exclusive discriminative information can reinforce the shared information during the downstream task finetuning. Projecting each modality embedding into separate shared space and private space while applying the orthogonality constraints avoids the shared information being reused in the private space optimization.
Contrastive Task for Shared Space: We use the shared feature space to learn modality consistency information. Specifically, we only consider samples within a batch  $\mathcal{B}$  but across different short sequences, and assume the same random augmentation is applied. We iterate over all pairs of modalities  $M_{j}$  and  $M_{j'}$ , regarding different modality embeddings of the same sample ( $\mathbf{x}_{ij}^{shared}, \mathbf{x}_{ij'}^{shared}$ ) (e.g.,  $\mathbf{\Phi}$  in Figure 1) as positive pairs, and regard two random modality embeddings from different samples ( $\mathbf{x}_{ij}^{shared}, \mathbf{x}_{i'j'}^{shared}$ ) as negative pairs. We calculate the following InfoNCE loss [48],
$$
\mathcal {L} _ {\text {s h a r e d}} = - \sum_ {i} \sum_ {M _ {j}, M _ {j ^ {\prime}} \in \mathcal {M}, j \neq j ^ {\prime}} \log \frac {\exp \left(\left\langle \mathbf {h} _ {i j} ^ {\text {s h a r e d}} , \mathbf {h} _ {i j ^ {\prime}} ^ {\text {s h a r e d}} \right\rangle / \tau\right)}{\sum_ {i ^ {\prime} \in \mathcal {B}} \exp \left(\left\langle \mathbf {h} _ {i j} ^ {\text {s h a r e d}} , \mathbf {h} _ {i ^ {\prime} j ^ {\prime}} ^ {\text {s h a r e d}} \right\rangle / \tau\right)}, \tag {1}
$$
where  $\tau$  is a temperature parameter that controls the penalties on hard negative samples [52].
Contrastive Task for Private Space: We use the private feature space to learn modality-exclusive information that is useful in discriminating different sequences within a batch  $\mathcal{B}$ , through capturing transformation consistency information. Specifically, for each modality  $M_{j}$ , we consider the encoded embeddings of two randomly augmented versions of the same samples as positive pairs (e.g., 2 in Figure 1), i.e.,  $(\mathbf{h}_{ij}^{private}, \tilde{\mathbf{h}}_{ij}^{private})$ , where we use  $\tilde{\mathbf{h}}_{ij}^{private}$  to denote a differently augmented variant. The remaining 2B-2 modality embeddings in the batch are considered negative pairs to  $\mathbf{h}_{ij}^{private}$ . We use the NT-Xent [3] loss for the private space contrastive task,
$$
\mathcal {L} _ {\text {p r i v a t e}} = - \sum_ {i} \sum_ {M _ {j} \in \mathcal {M}} \log \frac {\exp \left(\left\langle \mathbf {h} _ {i j} ^ {\text {p r i v a t e}}, \tilde {\mathbf {h}} _ {i j} ^ {\text {p r i v a t e}} \right\rangle / \tau\right)}{\sum_ {i ^ {\prime} \in \mathcal {B} , i ^ {\prime} \neq i} \exp \left(\left\langle \mathbf {h} _ {i j} ^ {\text {p r i v a t e}}, \tilde {\mathbf {h}} _ {i ^ {\prime} j} ^ {\text {p r i v a t e}} \right\rangle / \tau\right) + \sum_ {i ^ {\prime} \in \mathcal {B}} \exp \left(\left\langle \mathbf {h} _ {i j} ^ {\text {p r i v a t e}}, \tilde {\mathbf {h}} _ {i ^ {\prime} j} ^ {\text {p r i v a t e}} \right\rangle / \tau\right)}.
$$
Orthogonality Constraint: To enforce the orthogonality constraint between the shared feature and private feature of the same modality, as well as the private features between different modalities, such that they can capture independent semantic information in the factorized space, we apply a cosine
embedding loss that directly minimizes their angular similarities,
$$
\mathcal {L} _ {\text {o r t h o g o n a l}} = \sum_ {i} \sum_ {M _ {j} \in \mathcal {M}} \left\langle \mathbf {h} _ {i j} ^ {\text {s h a r e d}}, \mathbf {h} _ {i j} ^ {\text {p r i v a t e}} \right\rangle + \sum_ {i} \sum_ {M _ {j}, M _ {j} ^ {\prime} \in \mathcal {M}, j \neq j ^ {\prime}} \left\langle \mathbf {h} _ {i j} ^ {\text {p r i v a t e}}, \mathbf {h} _ {i j ^ {\prime}} ^ {\text {p r i v a t e}} \right\rangle . \tag {3}
$$
3.3 Temporal Structural Constraint
Appropriately shaping the temporal information locality in latent modality embedding space is challenging. First, simply considering temporally close samples as positive pairs and pushing their semantical similarities to 1 as [63, 49] can be problematic because they ignore the continuously evolving factors (e.g., distance) that make differences between temporally neighboring samples. Second, it is also impractical to predefine a fixed similarity curve with respect to the time difference between two samples, which is highly context-dependent and can not be known in advance.
Considering the complexity of temporal information correlations, we stop defining positive versus negative pairs in the temporal dimension. Instead, we only apply a loose ranking loss to specify the relationships of distances at the coarse-grained sequence level (intra-sequence distance vs. inter-sequence distance) as an information regularization [9]. Given a sequence, we restrict the average distances between samples within the sequence (e.g., 3 in Figure 1) to be no larger than their average distance to samples from other sequences (e.g., 4 in Figure 1). To accommodate occasional violations of temporal locality caused by long time periodicity (such that temporally distant samples could be more similar than neighboring samples), and further reduce the computational complexity caused by multiplicative traverse of sample triplets, we restrict the average intra-sequence distances to be no larger than average inter-sequence distances.
We calculate the distance in the Euclidean space, to facilitate the downstream Euclidean classifiers. Given the sample-level distance matrix  $\mathbf{D} \in \mathbb{R}^{BL \times BL}$ , we first compute the sequence-level mean distance matrix  $\bar{\mathbf{D}} \in \mathbb{R}^{B \times B}$  through aggregating sample-level distances $^2$ , such that the average distance between sequence  $s$  and sequence  $s'$  is  $\bar{D}_{ss'} = 1 / L^2 \sum_{i \in s, i' \in s'} D_{pq}$ , then the temporal locality loss is defined by,
$$
\mathcal {L} _ {\text {t e m p o r a l}} = \sum_ {s} \sum_ {s ^ {\prime} \neq s} \max  \left(\bar {D} _ {s s} - \bar {D} _ {s s ^ {\prime}} + \operatorname {m a r g i n}, 0\right), \tag {4}
$$
where the margin is the predefined degree of separation. It is worth noting that the temporal structural constraint is applied to holistic modality embeddings, including both the shared and private parts.
3.4 Overall Training Objective
In summary, FOCAL pretraining simultaneously considers latent correlations between (1) synchronized embeddings of different modalities; (2) differently augmented modality embeddings; and (3) temporally neighboring sample embeddings. It minimizes the following overall loss,
$$
\mathcal {L} = \mathcal {L} _ {\text {s h a r e d}} + \lambda_ {p} \cdot \mathcal {L} _ {\text {p r i v a t e}} + \lambda_ {o} \cdot \mathcal {L} _ {\text {o r t h o g o n a l}} + \lambda_ {t} \cdot \mathcal {L} _ {\text {t e m p o r a l}}, \tag {5}
$$
where  $\lambda_{p},\lambda_{o}$  , and  $\lambda_{t}$  are hyperparameters that control the weights of each loss component.
In this section, we evaluate FOCAL by comparing it with 11 popular SOTA self-supervised learning frameworks on four different datasets. We start with the experimental setups and then introduce the main evaluation results, followed by comprehensive ablation studies.
Datasets: (1) MOD is a self-collected dataset using acoustic  $(8000\mathrm{Hz})$  and seismic  $(100\mathrm{Hz})$  signals to classify moving vehicle types. It includes 6 different vehicle types and 1 class of human walking. (2) ACIDS is an ideal dataset for vehicle classification using acoustic signals and seismic signals
2The sample-level diagonal elements that always have distance 0 are skipped during the aggregation.
Table 1: Statistical Summaries of Evaluated Datasets.
Dataset	Classes	Modalities (Freq)	Sample Length	Interval (Overlap)	#Samples	#Labels
MOD	7	acoustic (8000Hz), seismic (100Hz)	2 sec	0.2 sec (0%)	39,609	7,335
ACIDS	9	acoustic, seismic (both 1025Hz)	1 sec	0.25 sec (50%)	27,597	27,597
RealWorld-HAR	8	acc, gyro, mag, lig (all 50Hz)	5 sec	1 sec (50%)	12,887	12,887
PAMAP2	18	acc, gyr, mag (all 100Hz)	2 sec	0.4 sec (50%)	9,611	9,611
Table 2: Finetune Results with Linear Classifier
Dataset	MOD	ACIDS	RealWorld-HAR	PAMAP2
Encoder	Framework	Acc	F1	Acc	F1	Acc	F1	Acc	F1
DeepSense	Supervised	0.9404	0.9399	0.9566	0.8407	0.9348	0.9388	0.8849	0.8761
SimCLR	0.8855	0.8855	0.7438	0.6101	0.7138	0.6841	0.6802	0.6583
MoCo	0.8808	0.8812	0.7717	0.6205	0.7859	0.7708	0.7559	0.7387
CMC	0.9196	0.9186	0.8443	0.7244	0.7975	0.8116	0.7906	0.7706
MAE	0.5981	0.5993	0.6644	0.5618	0.7565	0.7515	0.7114	0.6158
Cosmo	0.8989	0.8998	0.8511	0.6929	0.8956	0.8888	0.8356	0.8135
Cocoa	0.8774	0.8764	0.6644	0.5359	0.8465	0.8488	0.7603	0.7187
MTSS	0.4153	0.3582	0.4352	0.2441	0.2989	0.1405	0.3541	0.1795
TS2Vec	0.7669	0.7648	0.5224	0.3587	0.6595	0.5984	0.5729	0.4715
GMC	0.9257	0.9267	0.9096	0.7929	0.8869	0.8948	0.8119	0.7860
TNC	0.9518	0.9528	0.8237	0.6936	0.8892	0.8971	0.8387	0.8143
TS-TCC	0.8707	0.8735	0.7667	0.6164	0.8073	0.8010	0.7776	0.7250
FOCAL	0.9732	0.9729	0.9516	0.8580	0.9382	0.9290	0.8588	0.8463
SW-T	Supervised	0.8948	0.8931	0.9137	0.7770	0.9313	0.9278	0.8612	0.8384
SimCLR	0.9250	0.9247	0.9128	0.8144	0.7046	0.7220	0.7705	0.7424
MoCo	0.9390	0.9384	0.9174	0.8100	0.7813	0.8024	0.7717	0.7313
CMC	0.9129	0.9105	0.8128	0.6857	0.8840	0.8955	0.8080	0.7901
MAE	0.7803	0.7772	0.8516	0.7023	0.8829	0.8813	0.7910	0.7606
Cosmo	0.3429	0.3378	0.7110	0.6086	0.8604	0.8169	0.7741	0.7366
Cocoa	0.7040	0.7038	0.7096	0.5794	0.8892	0.8861	0.7689	0.7317
MTSS	0.4206	0.4163	0.3429	0.2250	0.5136	0.4370	0.2847	0.1714
TS2Vec	0.7254	0.7174	0.7183	0.5748	0.6151	0.5955	0.6195	0.5426
GMC	0.8640	0.8611	0.9402	0.7766	0.9319	0.9379	0.8312	0.8083
TNC	0.8533	0.8539	0.8352	0.7372	0.8817	0.8784	0.8013	0.7506
TS-TCC	0.8734	0.8735	0.9041	0.7547	0.8731	0.8454	0.7997	0.7260
FOCAL	0.9805	0.9800	0.9489	0.8262	0.9451	0.9503	0.8580	0.8401