SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 9: Fintuning Experiments with Linear Classifier on MOD dataset.
Encoder	Framework	Label Ratio: 1.0	Label Ratio: 0.1	Label Ratio: 0.01
Acc	F1	Acc	F1	Acc	F1
DeepSense	Supervised	0.9404	0.9399	0.6821 ± 0.0442	0.6810 ± 0.0475	0.3567 ± 0.0450	0.3366 ± 0.0365
SimCLR	0.8855	0.8855	0.8186 ± 0.0055	0.8162 ± 0.0058	0.5934 ± 0.0319	0.5808 ± 0.0337
MoCo	0.8808	0.8812	0.7819 ± 0.0078	0.7763 ± 0.0089	0.5038 ± 0.0377	0.4794 ± 0.0509
CMC	0.9196	0.9186	0.8938 ± 0.0055	0.8920 ± 0.0056	0.7645 ± 0.0131	0.7459 ± 0.0224
MAE	0.5981	0.5993	0.4963 ± 0.0083	0.4985 ± 0.0041	0.3586 ± 0.0347	0.3292 ± 0.0497
Cosmo	0.8989	0.8998	0.8505 ± 0.0066	0.8519 ± 0.0061	0.7025 ± 0.0169	0.7025 ± 0.0171
Cocoa	0.8774	0.8764	0.8397 ± 0.0058	0.8378 ± 0.0055	0.7181 ± 0.0198	0.6998 ± 0.0226
MTSS	0.4153	0.3582	0.3863 ± 0.0058	0.3139 ± 0.0081	0.3140 ± 0.0084	0.2527 ± 0.0198
TS2Vec	0.7669	0.7648	0.7018 ± 0.0066	0.6980 ± 0.0070	0.5319 ± 0.0199	0.5150 ± 0.0230
GMC	0.9257	0.9267	0.8812 ± 0.0061	0.8820 ± 0.0069	0.7198 ± 0.0097	0.6983 ± 0.0204
TNC	0.9518	0.9528	0.9437 ± 0.0055	0.9446 ± 0.0054	0.8616 ± 0.0330	0.8469 ± 0.0620
TSTCC	0.8707	0.8735	0.8295 ± 0.0034	0.8319 ± 0.0036	0.6080 ± 0.0321	0.5753 ± 0.0553
FOCAL	0.9732	0.9729	0.9485 ± 0.0038	0.9480 ± 0.0039	0.8567 ± 0.0151	0.8544 ± 0.0173
SW-T	Supervised	0.8948	0.8931	0.5555 ± 0.0164	0.5450 ± 0.0197	0.2028 ± 0.0111	0.1638 ± 0.0196
SimCLR	0.9250	0.9247	0.8891 ± 0.0040	0.8888 ± 0.0042	0.7523 ± 0.0368	0.7443 ± 0.0442
MoCo	0.9390	0.9384	0.9073 ± 0.0032	0.9073 ± 0.0032	0.7482 ± 0.0228	0.7409 ± 0.0269
CMC	0.9129	0.9105	0.8691 ± 0.0067	0.8661 ± 0.0067	0.6994 ± 0.0157	0.6835 ± 0.0191
MAE	0.7803	0.7772	0.6561 ± 0.0119	0.6480 ± 0.0120	0.3764 ± 0.0200	0.3544 ± 0.0297
Cosmo	0.3429	0.3378	0.2122 ± 0.0087	0.1989 ± 0.0071	0.1753 ± 0.0152	0.1346 ± 0.0138
Cocoa	0.7040	0.7038	0.6869 ± 0.0145	0.6833 ± 0.0177	0.6122 ± 0.0162	0.5955 ± 0.0300
MTSS	0.4206	0.4163	0.3799 ± 0.0087	0.3700 ± 0.0081	0.3113 ± 0.0259	0.2964 ± 0.0191
TS2Vec	0.7254	0.7174	0.6522 ± 0.0086	0.6434 ± 0.0099	0.4750 ± 0.0225	0.4477 ± 0.0355
GMC	0.8640	0.8611	0.7712 ± 0.0049	0.7685 ± 0.0053	0.5191 ± 0.0209	0.4959 ± 0.0348
TNC	0.8533	0.8539	0.8436 ± 0.0068	0.8443 ± 0.0070	0.7996 ± 0.0331	0.7935 ± 0.0419
TSTCC	0.8734	0.8735	0.8564 ± 0.0040	0.8558 ± 0.0038	0.7473 ± 0.0220	0.7322 ± 0.0470
FOCAL	0.9805	0.9800	0.9593 ± 0.0025	0.9584 ± 0.0024	0.8840 ± 0.0299	0.8776 ± 0.0389
First, FOCAL consistently demonstrates significant improvements in both accuracy and F1 score across all label ratios compared to other self-supervised learning baselines on the ACIDS, RealWorld-HAR, and PAMAP2 datasets. In the case of the MOD dataset under  $1\%$  labels, FOCAL achieves similar accuracy to TNC with the DeepSense encoder but beats TNC by  $10.56\%$  with the SW-T encoder. These results underline the superior performance of FOCAL in multimodal time series sensing data and emphasize the importance of the underlying relationship between the shared and private modality features through time.
Second, the performance improvements persist across backbone encoders and different label ratios, proving the advantage of FOCAL in improving the label efficiency during downstream finetuning. Although there are a few cases where some baselines perform close to FOCAL (e.g., TNC with DeepSense encoder on MOD dataset under  $1\%$  labels), such comparability does not persist across encoders.
Third, FOCAL shows comparable performance to the supervised model when all available labels (i.e.,  $100\%$ ) are used in the training. However, when fewer labels are available, FOCAL shows a larger advantage over the supervised oracle, demonstrating its capability to better leverage the limited available labels in adapting to downstream tasks. On average, FOCAL surpasses the supervised model by  $1.37\%$  with  $100\%$  labels,  $15.04\%$  with  $10\%$  labels, and  $68.39\%$  with  $1\%$  labels. By learning semantically meaningful multimodal representations from the massive unlabeled inputs during the pretraining phase, FOCAL can effectively utilize limited data labels during the finetuning process. This is especially reflected in the MOD results, where we have around 6 times more data in pretraining than the finetuning and achieve  $3.49\%$  and  $9.58\%$  improvement over the supervised model.
Fourth, between the backbone encoders, we found FOCAL brings more relative performance improvement to SW-T than DeepSense compared to their supervised versions. With FOCAL training, SW-T beats DeepSense in two out of four datasets (i.e., MOD and RealWorld-HAR), while DeepSense is always the better encoder architecture with supervised training. Besides, the performance improvement on SW-T is more significant when the number of available labels is low during the finetuning (i.e.,  $10\%$  and  $1\%$ ) since larger performance gaps are observed between FOCAL and supervised models.
Table 10: Fintuning Experiments with Linear Classifier on ACIDS dataset.
Encoder	Framework	Label Ratio: 1.0	Label Ratio: 0.1	Label Ratio: 0.01
Acc	F1	Acc	F1	Acc	F1
DeepSense	Supervised	0.9566	0.8407	0.9379 ± 0.0158	0.8006 ± 0.0316	0.7567 ± 0.0335	0.5754 ± 0.0406
SimCLR	0.7438	0.6101	0.7111 ± 0.0157	0.5773 ± 0.0166	0.6166 ± 0.0206	0.4392 ± 0.0430
MoCo	0.7717	0.6205	0.7433 ± 0.0269	0.5833 ± 0.0243	0.6637 ± 0.0414	0.4827 ± 0.0470
CMC	0.8443	0.7244	0.7370 ± 0.0126	0.6139 ± 0.0180	0.6313 ± 0.0633	0.4726 ± 0.0786
MAE	0.6644	0.5618	0.5862 ± 0.0024	0.4479 ± 0.0062	0.4901 ± 0.0309	0.2825 ± 0.0293
Cosmo	0.8511	0.6929	0.8532 ± 0.0176	0.7083 ± 0.0199	0.7288 ± 0.0231	0.5571 ± 0.0447
Cocoa	0.6644	0.5359	0.6174 ± 0.0106	0.4605 ± 0.0219	0.5617 ± 0.0223	0.3811 ± 0.0289
MTSS	0.4352	0.2441	0.4247 ± 0.0341	0.2130 ± 0.0385	0.4280 ± 0.0274	0.1879 ± 0.0333
TS2Vec	0.5224	0.3587	0.5299 ± 0.0121	0.3554 ± 0.0113	0.5341 ± 0.0363	0.3516 ± 0.0366
GMC	0.9096	0.7929	0.8890 ± 0.0090	0.7681 ± 0.0178	0.7156 ± 0.0603	0.5573 ± 0.0693
TNC	0.8237	0.6936	0.8063 ± 0.0156	0.6635 ± 0.0370	0.7428 ± 0.0419	0.5760 ± 0.0576
TSTCC	0.7667	0.6164	0.7655 ± 0.0094	0.6127 ± 0.0083	0.6697 ± 0.0354	0.4846 ± 0.0368
FOCAL	0.9516	0.8580	0.9253 ± 0.0143	0.8007 ± 0.0199	0.7829 ± 0.0448	0.5940 ± 0.0514
SW-T	Supervised	0.9137	0.7770	0.7310 ± 0.0224	0.5532 ± 0.0158	0.2666 ± 0.0319	0.1531 ± 0.0398
SimCLR	0.9128	0.8144	0.8882 ± 0.0154	0.7751 ± 0.0161	0.7580 ± 0.0380	0.6030 ± 0.0565
MoCo	0.9174	0.8100	0.9069 ± 0.0111	0.7841 ± 0.0192	0.7990 ± 0.0299	0.6235 ± 0.0408
CMC	0.8128	0.6857	0.7985 ± 0.0129	0.6700 ± 0.0170	0.6583 ± 0.0401	0.4990 ± 0.0422
MAE	0.8516	0.7023	0.7916 ± 0.0066	0.6344 ± 0.0088	0.4751 ± 0.0631	0.3440 ± 0.0317
Cosmo	0.7110	0.6086	0.6722 ± 0.0102	0.5279 ± 0.0067	0.5419 ± 0.0235	0.3710 ± 0.0114
Cocoa	0.7096	0.5794	0.6711 ± 0.0117	0.5324 ± 0.0127	0.6262 ± 0.0282	0.4585 ± 0.0212
MTSS	0.3429	0.2250	0.2878 ± 0.0292	0.1782 ± 0.0113	0.2946 ± 0.0499	0.1564 ± 0.0142
TS2Vec	0.7183	0.5748	0.6756 ± 0.0124	0.5003 ± 0.0119	0.5801 ± 0.0194	0.3837 ± 0.0153
GMC	0.9402	0.7766	0.9014 ± 0.0116	0.7278 ± 0.0148	0.7089 ± 0.0426	0.5250 ± 0.0401
TNC	0.8352	0.7372	0.8158 ± 0.0135	0.7051 ± 0.0176	0.6827 ± 0.0469	0.5424 ± 0.0500
TSTCC	0.9041	0.7547	0.9009 ± 0.0062	0.7449 ± 0.0202	0.7656 ± 0.0378	0.5806 ± 0.0223
FOCAL	0.9489	0.8262	0.9400 ± 0.0081	0.7975 ± 0.0199	0.8669 ± 0.0287	0.6844 ± 0.0372
Table 11: Fintuning Experiments with Linear Classifier on RealWorld-HAR dataset.
Encoder	Framework	Label Ratio: 1.0	Label Ratio: 0.1	Label Ratio: 0.01
Acc	F1	Acc	F1	Acc	F1
DeepSense	Supervised	0.9348	0.9388	0.9256 ± 0.0056	0.9233 ± 0.0104	0.7305 ± 0.0270	0.6158 ± 0.0341
SimCLR	0.7138	0.6841	0.6597 ± 0.0182	0.6126 ± 0.0198	0.5334 ± 0.0566	0.4271 ± 0.0518
MoCo	0.7859	0.7708	0.7454 ± 0.0206	0.6687 ± 0.0340	0.5110 ± 0.0409	0.4018 ± 0.0552
CMC	0.7975	0.8116	0.7482 ± 0.0328	0.7590 ± 0.0282	0.5169 ± 0.0314	0.4716 ± 0.0455
MAE	0.7565	0.7515	0.7206 ± 0.0181	0.7056 ± 0.0175	0.5556 ± 0.0527	0.4593 ± 0.0541
Cosmo	0.8956	0.8888	0.8814 ± 0.0123	0.8626 ± 0.0338	0.8434 ± 0.0376	0.7775 ± 0.0801
Cocoa	0.8465	0.8488	0.8492 ± 0.0070	0.8211 ± 0.0068	0.7155 ± 0.0397	0.6381 ± 0.0324
MTSS	0.2989	0.1405	0.1905 ± 0.0503	0.0692 ± 0.0328	0.1698 ± 0.0365	0.0600 ± 0.0355
TS2Vec	0.6595	0.5984	0.6419 ± 0.0189	0.5721 ± 0.0154	0.6147 ± 0.0456	0.5197 ± 0.0241
GMC	0.8869	0.8948	0.8872 ± 0.0172	0.8842 ± 0.0124	0.7954 ± 0.0367	0.7620 ± 0.0442
TNC	0.8892	0.8971	0.8712 ± 0.0238	0.8629 ± 0.0260	0.7991 ± 0.0390	0.7337 ± 0.0229
TSTCC	0.8073	0.8010	0.7892 ± 0.0146	0.7625 ± 0.0223	0.7213 ± 0.0320	0.6181 ± 0.0352
FOCAL	0.9382	0.9290	0.9335 ± 0.0053	0.9224 ± 0.0075	0.8518 ± 0.0274	0.7933 ± 0.0436
SW-T	Supervised	0.9313	0.9278	0.7264 ± 0.0411	0.6090 ± 0.0447	0.4541 ± 0.0694	0.2771 ± 0.0798
SimCLR	0.7046	0.7220	0.6717 ± 0.0062	0.6892 ± 0.0081	0.4867 ± 0.0431	0.4267 ± 0.0674
MoCo	0.7813	0.8024	0.7324 ± 0.0096	0.7425 ± 0.0173	0.5541 ± 0.0462	0.4823 ± 0.0391
CMC	0.8840	0.8955	0.8352 ± 0.0154	0.8424 ± 0.0156	0.5602 ± 0.0411	0.5245 ± 0.0549
MAE	0.8829	0.8813	0.7873 ± 0.0100	0.7224 ± 0.0314	0.5602 ± 0.0275	0.4699 ± 0.0205
Cosmo	0.8604	0.8169	0.7710 ± 0.0134	0.6899 ± 0.0178	0.6089 ± 0.0256	0.5230 ± 0.0395
Cocoa	0.8892	0.8861	0.8609 ± 0.0110	0.8501 ± 0.0143	0.7430 ± 0.0321	0.6657 ± 0.0432
MTSS	0.5136	0.4370	0.4359 ± 0.0281	0.3690 ± 0.0303	0.3547 ± 0.0156	0.2792 ± 0.0202
TS2Vec	0.6151	0.5955	0.6074 ± 0.0202	0.5540 ± 0.0201	0.5667 ± 0.0451	0.4876 ± 0.0464
GMC	0.9319	0.9379	0.9081 ± 0.0108	0.9115 ± 0.0092	0.7925 ± 0.0426	0.7453 ± 0.0581
TNC	0.8817	0.8784	0.8635 ± 0.0109	0.8525 ± 0.0100	0.8061 ± 0.0215	0.7494 ± 0.0452
TSTCC	0.8731	0.8454	0.8606 ± 0.0114	0.8070 ± 0.0233	0.7374 ± 0.0434	0.6685 ± 0.0642
FOCAL	0.9452	0.9492	0.9370 ± 0.0069	0.9421 ± 0.0060	0.8301 ± 0.0428	0.7519 ± 0.0578
F.2 Finetuning: Complete KNN Classification Results
Setup: In addition to linear probing, we further evaluate the self-supervised frameworks on four datasets using the K-Nearest-Neighbors (KNN,  $\mathrm{K} = 5$ ) classifier without introducing new parameters. This evaluation method allows us to examine the quality of learned representations without new
Table 12: Fintuning Experiments with Linear Classifier on PAMAP2 dataset.
Encoder	Framework	Label Ratio: 1.0	Label Ratio: 0.1	Label Ratio: 0.01
Acc	F1	Acc	F1	Acc	F1
DeepSense	Supervised	0.8849	0.8761	0.8080 ± 0.0071	0.7649 ± 0.0275	0.6539 ± 0.0303	0.5695 ± 0.0726
SimCLR	0.6802	0.6583	0.6132 ± 0.0174	0.5606 ± 0.0247	0.4352 ± 0.0340	0.3305 ± 0.0197
MoCo	0.7559	0.7387	0.6325 ± 0.0177	0.5601 ± 0.0401	0.3872 ± 0.0301	0.2873 ± 0.0274
CMC	0.7906	0.7706	0.6687 ± 0.0263	0.5653 ± 0.0602	0.2724 ± 0.0287	0.1676 ± 0.0248
MAE	0.7114	0.6158	0.5769 ± 0.0222	0.4514 ± 0.0239	0.2734 ± 0.0192	0.1096 ± 0.0198
Cosmo	0.8356	0.8135	0.7790 ± 0.0220	0.7427 ± 0.0341	0.6782 ± 0.0226	0.5740 ± 0.0293
Cocoa	0.7603	0.7187	0.7132 ± 0.0105	0.6432 ± 0.0082	0.5922 ± 0.0234	0.5293 ± 0.0232
MTSS	0.3541	0.1795	0.2891 ± 0.0416	0.1169 ± 0.0378	0.1857 ± 0.0546	0.0710 ± 0.0406
TS2Vec	0.5729	0.4715	0.5416 ± 0.0171	0.4433 ± 0.0177	0.4399 ± 0.0341	0.3335 ± 0.0445
GMC	0.8119	0.7860	0.7528 ± 0.0097	0.6975 ± 0.0207	0.5837 ± 0.0367	0.4899 ± 0.0510
TNC	0.8387	0.8143	0.8287 ± 0.0022	0.8068 ± 0.0059	0.7365 ± 0.0414	0.6469 ± 0.0682
TSTCC	0.7776	0.7250	0.7489 ± 0.0105	0.6401 ± 0.0201	0.5348 ± 0.0782	0.4368 ± 0.0852
FOCAL	0.8604	0.8463	0.8373 ± 0.0041	0.8175 ± 0.0074	0.7521 ± 0.0151	0.6900 ± 0.0325
SW-T	Supervised	0.8612	0.8384	0.7295 ± 0.0135	0.6434 ± 0.0230	0.4048 ± 0.0337	0.3159 ± 0.0271
SimCLR	0.7705	0.7424	0.7307 ± 0.0060	0.6871 ± 0.0103	0.5416 ± 0.0441	0.4708 ± 0.0627
MoCo	0.7717	0.7313	0.7112 ± 0.0203	0.6356 ± 0.0331	0.4774 ± 0.0220	0.3740 ± 0.0301
CMC	0.8080	0.7901	0.6864 ± 0.0259	0.4590 ± 0.0131	0.1852 ± 0.0221	0.1283 ± 0.0127
MAE	0.7910	0.7606	0.6655 ± 0.0067	0.6028 ± 0.0129	0.3603 ± 0.0416	0.2866 ± 0.0402
Cosmo	0.7741	0.7366	0.6702 ± 0.0051	0.5958 ± 0.0107	0.4555 ± 0.0381	0.3870 ± 0.0297
Cocoa	0.7689	0.7317	0.7461 ± 0.0047	0.7048 ± 0.0115	0.6594 ± 0.0228	0.5973 ± 0.0243
MTSS	0.2847	0.1714	0.2558 ± 0.0109	0.1585 ± 0.0097	0.2133 ± 0.0164	0.1265 ± 0.0215
TS2Vec	0.6195	0.5426	0.6001 ± 0.0133	0.5249 ± 0.0154	0.5051 ± 0.0402	0.4123 ± 0.0374
GMC	0.8312	0.8083	0.7686 ± 0.0118	0.7297 ± 0.0140	0.5704 ± 0.0409	0.4965 ± 0.0426
TNC	0.8013	0.7506	0.7921 ± 0.0083	0.7380 ± 0.0144	0.7222 ± 0.0305	0.6378 ± 0.0488
TSTCC	0.7997	0.7260	0.7800 ± 0.0094	0.6890 ± 0.0148	0.6438 ± 0.0569	0.5566 ± 0.0509
FOCAL	0.8442	0.8287	0.8179 ± 0.0117	0.7856 ± 0.0177	0.7371 ± 0.0332	0.6630 ± 0.0410
Table 13: Complete KNN Results
Encoders	Framework	MOD	ACIDS	RealWorld-HAR	PAMAP2
Acc	F1	Acc	F1	Acc	F1	Acc	F1
DeepSense	SimCLR	0.8238	0.8240	0.7402	0.5637	0.6584	0.6234	0.6451	0.6114
MoCo	0.8446	0.8444	0.7735	0.5957	0.7496	0.7134	0.6924	0.6766
CMC	0.9002	0.8989	0.7584	0.6516	0.5216	0.5868	0.8032	0.7938
MAE	0.6470	0.6451	0.7457	0.5610	0.8794	0.8817	0.6857	0.6427
Cosmo	0.8379	0.8387	0.7986	0.6284	0.8102	0.7817	0.8005	0.7743
Cocoa	0.7910	0.7877	0.6758	0.4966	0.7778	0.7459	0.7129	0.6974
MTSS	0.3443	0.3249	0.4333	0.2417	0.5101	0.4384	0.3931	0.3379
TS2Vec	0.6966	0.6875	0.5726	0.3602	0.6480	0.5832	0.5639	0.5180
GMC	0.8533	0.8526	0.7411	0.6210	0.7415	0.7560	0.7843	0.7543
TNC	0.9498	0.9508	0.7813	0.6203	0.7882	0.7565	0.7993	0.7653
TSTCC	0.8607	0.8615	0.8192	0.6443	0.7686	0.7658	0.8032	0.7896
FOCAL	0.9551	0.9544	0.9247	0.7938	0.8205	0.8254	0.8482	0.8378
SW-T	SimCLR	0.9022	0.9021	0.8553	0.7086	0.6532	0.6767	0.7441	0.7178
MoCo	0.9344	0.9343	0.8311	0.6943	0.7103	0.7303	0.7082	0.6678
CMC	0.8305	0.8261	0.7187	0.6355	0.5701	0.6007	0.7709	0.7694
MAE	0.3389	0.3104	0.5945	0.4194	0.6428	0.6080	0.5517	0.4969
Cosmo	0.2786	0.2621	0.5790	0.4573	0.7086	0.6389	0.6672	0.5874
Cocoa	0.5941	0.5793	0.5311	0.4261	0.7421	0.7496	0.7188	0.7070
MTSS	0.3423	0.3376	0.3151	0.1890	0.4882	0.4431	0.2007	0.1649
TS2Vec	0.5847	0.5718	0.6050	0.4144	0.5580	0.5335	0.5623	0.5040
GMC	0.5318	0.5180	0.7589	0.6150	0.7380	0.7455	0.7567	0.7401
TNC	0.8265	0.8263	0.7795	0.6725	0.8009	0.7817	0.7674	0.7189
TSTCC	0.8607	0.8613	0.8356	0.6700	0.7582	0.7512	0.7780	0.7369
FOCAL	0.9665	0.9664	0.8826	0.7643	0.8586	0.8665	0.8549	0.8484
training steps. We first construct a KNN estimator using the encoded sample features and corresponding labels from finetuning data. For multi-modal frameworks, we directly concatenate modality embeddings as the sample-level representations. Subsequently, the estimator predicts the test labels according to the labels of neighboring samples in the supervised set  $\mathcal{X}^s$  and computes the testing accuracy accordingly.
Analysis: The complete evaluation results with the KNN classifier are reported in Table 13. FOCAL consistently surpasses the performance of other self-supervised learning baselines in most cases. The KNN evaluation results are mostly consistent with the linear classification results, but there are
Table 14: Clustering Evaluation
Dataset	MOD	ACIDS	RealWorld-HAR	PAMAP2
Encoder	Framework	ARI	NMI	ARI	NMI	ARI	NMI	ARI	NMI
DeepSense	CMC	0.3936 ± 0.0125	0.5224 ± 0.0206	0.2926 ± 0.0156	0.5833 ± 0.0051	0.2187 ± 0.1094	0.4354 ± 0.1713	0.3024 ± 0.0118	0.5063 ± 0.0120
Cosmo	0.1384 ± 0.0540	0.2552 ± 0.0803	0.5217 ± 0.0074	0.6416 ± 0.0184	0.4231 ± 0.2726	0.5318 ± 0.2564	0.3583 ± 0.0781	0.5212 ± 0.0671
Cocoa	0.3502 ± 0.0184	0.4444 ± 0.0135	0.5453 ± 0.0229	0.6767 ± 0.0184	0.3385 ± 0.1826	0.4792 ± 0.1940	0.3493 ± 0.0230	0.5091 ± 0.0184
GMC	0.1982 ± 0.0674	0.3925 ± 0.0416	0.2490 ± 0.0403	0.5296 ± 0.0150	0.3433 ± 0.1836	0.4794 ± 0.1978	0.3078 ± 0.0194	0.5092 ± 0.0221
FOCAL	0.3929 ± 0.0222	0.5067 ± 0.0226	0.5723 ± 0.0440	0.7213 ± 0.0432	0.4400 ± 0.2465	0.5545 ± 0.2437	0.4759 ± 0.0695	0.6037 ± 0.0558
SW-T	CMC	0.4314 ± 0.2716	0.5413 ± 0.2612	0.3604 ± 0.0119	0.5881 ± 0.0009	0.4014 ± 0.0528	0.5275 ± 0.0532	0.3718 ± 0.0480	0.5562 ± 0.0401
Cosmo	0.2865 ± 0.1521	0.4140 ± 0.1946	0.4436 ± 0.0145	0.5469 ± 0.0015	0.0029 ± 0.0020	0.0107 ± 0.0025	0.2425 ± 0.0301	0.3604 ± 0.0347
Cocoa	0.4281 ± 0.2314	0.5308 ± 0.2405	0.4363 ± 0.0020	0.6824 ± 0.0261	0.2487 ± 0.0053	0.3897 ± 0.0024	0.3658 ± 0.0540	0.5330 ± 0.0472
GMC	0.3973 ± 0.2177	0.4940 ± 0.2184	0.2055 ± 0.0029	0.4971 ± 0.0066	0.3050 ± 0.0076	0.4342 ± 0.0052	0.2794 ± 0.0206	0.5044 ± 0.0329
FOCAL	0.4660 ± 0.2737	0.5693 ± 0.2579	0.6050 ± 0.1027	0.7389 ± 0.0774	0.4319 ± 0.0851	0.5462 ± 0.0717	0.4785 ± 0.0914	0.6130 ± 0.0730
also a few exceptions. With the SW-T encoder, FOCAL exceeds the best baseline by an average of  $4.85\%$ . When using DeepSense as the encoder, FOCAL outperforms the most competitive contrastive framework baseline by  $1.18\%$  across all datasets. In the RealWorld-HAR dataset, DeepSense with MAE achieves higher accuracy than FOCAL, but it fails in the linear classification scenario and fails to generalize to other datasets and backbone encoders. In comparison to other contrastive learning baselines, FOCAL still demonstrates its superiority in KNN classification. Between the two encoders on FOCAL, SW-T outperforms DeepSense in three out of four datasets, which further shows the benefits FOCAL brings to SW-T training.
F.3 Complete Clustering Results
Setup: We further evaluate the clustering performance of FOCAL with other multimodal self-supervised learning baselines, including CMC, Cosmo, Cocoa, and GMC. We apply K-means clustering to the encoded embeddings from each framework, by setting the number of clusters equal to the number of unique classes in the testing dataset. As mentioned before, the preferred cluster structure by the SSL frameworks should align well with the underlying ground-truth labels in addition to presenting clear separation among the clusters. Following this objective, we quantitatively assess the clustering performance by independently calculating the Adjusted Rand Index (ARI) and the Normalized Mutual Information (NMI) of each modality to provide an accurate comparison of the alignment between the pretrained clusters and ground-truth classes. ARI evaluates the similarity between the clustering assignments generated by the K-means clusters and the label distribution of the test data. With a value range of -1 to 1, ARI indicates a high degree of agreement between the two clusterings when close to 1, random agreement when close to zero, and a clustering performance worse than random when approaching -1. NMI serves as an external metric for measuring the clustering quality. A score close to 1 indicates a perfect correlation between the clusterings, and a score of 0 demonstrates no mutual information between the clusters. Lastly, we performed t-SNE to qualitatively visualize the sample embeddings after concatenating the modality embeddings.
Analysis: In Table 14, we present the clustering results with the average and standard deviation of ARI and NMI across all modalities. As the results show, FOCAL consistently achieves the highest or similar ARI scores in comparison to other multimodal contrastive frameworks. When using SW-T as the encoder, FOCAL outperforms the strongest baseline by an average ARI margin of  $8.33\%$  and an average NMI margin of  $4\%$ . With DeepSense as the encoder, FOCAL surpasses the best baseline by an average ARI margin of  $4.61\%$  and an average NMI margin of  $3.35\%$ . Although CMC exhibits comparable performance for the MOD dataset when using DeepSense as an encoder, FOCAL with DeepSense exceeds CMC by an average of  $16.8\%$  and  $8.47\%$  in ARI and NMI across the four datasets. These results confirm our claim that FOCAL produces higher quality modality representations compared to the baseline multi-modal contrastive frameworks. We also found the general ARI and NMI values are relatively low because there could be multiple perspectives affecting the cluster structures that lead to complicated underlying semantics while we only evaluate one perspective among them.
Figures 10 and 11 represent the t-SNE visualizations of the encoded sample embeddings by FOCAL. We can observe a clear separation between individual clusters on MOD, ACIDS, and RealWorld-HAR, indicating that FOCAL effectively captures the distinct characteristics of each class. However, for the PAMAP2 dataset, we notice various overlaps between different embeddings. This observation suggests that the underlying structure of the PAMAP2 dataset is more challenging to differentiate compared to other datasets, potentially due to similarities among a large number of classes with 18 different physical activities. This discovery is also consistent with our linear probing results,
(a) MOD
(b) ACIDS
(c) RealWorld-HAR
(d) PAMAP2
Figure 10: t-SNE visualization of the concatenated modality features in FOCAL with SW-T encoder.
(a) MOD
Figure 11: t-SNE visualization of the concatenated modality features in FOCAL with DeepSense encoder.
(a)FOCAL
Figure 12: t-SNE visualization of the concatenated modality features with SW-T encoder (MOD dataset).
(c) GMC
which perform slightly worse on the PAMAP2 dataset. Besides, we provide the t-SNE visualization comparison between FOCAL, CMC, GMC, and Cocoa with Swin-Transformer encoder in Figure 12.
F.4 Complete Additional Downstream Task Results
Setup: We collected additional data samples for the MOD dataset and finetuned our pretrained models from previous experiments. Specifically, we evaluated our pretrained models by finetuning the classifier layer on two downstream tasks, distance classification, and speed classification tasks, with data obtained from different environments and new types of vehicles. These alterations in the data lead to domain adaptation, referring to changes in the data's distribution. For speed classification, the classifier predicts the speed of the moving object between 5, 10, 15, and  $20\mathrm{mph}$ . For distance classification, the classifier outputs whether the detected object is close, near, or far away.
Three metrics are evaluated in this experiment. In addition to the normal accuracy and (macro) F1 score, we also define a new metric called correlated accuracy. It considers the semantical distances between different classes and assigns different penalties to different misclassifications cases. Intuitively, for a sample with ground truth speed 5, a misclassification of speed 20 should be assigned more penalty than a misclassification of speed 10. Given a sample label pair  $(\mathbf{x}_i^s,y_i^s)$ , the predicted label  $y_{i}$ , and the number of classes  $C$ , we define the maximum class distance as  $\max (i,C - i - 1)$
Table 15: Linear Finetune Results with Extended Tasks on MOD
Task	Distance Classification	Speed Classification
Encoder	SW-T	DeepSense	SW-T	DeepSense
Framework	Acc	F1	Corr Acc	Acc	F1	Corr Acc	Acc	F1	Corr Acc	Acc	F1	Corr Acc
SimCLR	0.9090	0.8694	0.9545	0.8787	0.8057	0.9242	0.5511	0.5514	0.7524	0.5596	0.5438	0.7751
MoCo	0.9090	0.8694	0.9545	0.8484	0.7374	0.9091	0.6108	0.6105	0.7879	0.5767	0.5655	0.7794
CMC	0.8180	0.7507	0.8636	0.9393	0.8181	0.9697	0.5170	0.5175	0.7268	0.6022	0.6016	0.7850
MAE	0.7272	0.4917	0.8333	0.7272	0.4969	0.8030	0.4545	0.4383	0.6932	0.4034	0.3929	0.6506
Cosmo	0.6363	0.2592	0.8182	0.9393	0.8730	0.9545	0.2926	0.2779	0.5459	0.5681	0.5566	0.7737
Cocoa	0.8181	0.6898	0.8939	0.8181	0.6966	0.8333	0.4005	0.3618	0.6851	0.5625	0.5580	0.7628
MTSS	0.7272	0.4832	0.8030	0.8787	0.6180	0.9394	0.3522	0.2711	0.6544	0.4005	0.3482	0.6856
TS2Vec	0.6969	0.5869	0.7879	0.9090	0.8469	0.9242	0.4517	0.4473	0.6799	0.5198	0.5073	0.7476
GMC	0.8181	0.7450	0.8788	0.8484	0.7956	0.8788	0.4460	0.4405	0.6856	0.6250	0.6232	0.7917
TNC	0.8484	0.8015	0.8788	0.8787	0.8169	0.9242	0.4375	0.4322	0.6643	0.6108	0.6077	0.7841
TS-TCC	0.7878	0.6575	0.8939	0.8484	0.7312	0.9242	0.5284	0.5230	0.7311	0.5255	0.5138	0.7486
FOCAL	0.9697	0.9726	0.9848	0.9393	0.8985	0.9697	0.6960	0.6920	0.8329	0.6647	0.6682	0.8234
then the correlated accuracy is calculated by
$$
\operatorname {c o r r} _ {-} \operatorname {a c c} = \frac {1}{N ^ {\prime}} \sum_ {i} \left(1 - \frac {\left| y _ {i} - y _ {i} ^ {s} \right|}{\max  (i , C - i - 1)}\right), \tag {6}
$$
where the penalty of misclassification is linearly interpolated according to the distance of the predicted label and the ground truth label, divided by the maximum distance to this class. The value range of the correlated accuracy is still  $[0, 1]$ , where 0 means the worst and 1 means the best.