SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(1) Unified encoder. We attempt to unify the encoder for these pre-training methods. Specifically, we adopt the vanilla Transformer [39] with channel independent [26] for forecasting to accomplish cross-domain transfer between datasets with different variate numbers. As for the classification, we use 1D-ResNet [13] as the
encoder following [57]. Besides, we do a comprehensive hyperparameter search for all baselines. For the Transformer encoder, we vary the number of Transformer layers in  $\{1,2,3,4\}$ , select the model dimension from  $\{16,32,64,128,256\}$ , and the attention head from  $\{4,8,16,32\}$ . For the 1D-ResNet, we search the number of 1D-ResNet layers from  $\{1,2,3,4\}$ , the kernel size from  $\{3,5,8\}$  respectively. Additionally, for the masked modeling methods TST [56], Ti-MAE [21], we also searched the masked ratio  $r = \{0.125,0.25,0.5,0.75\}$  for better performance.
(2) Official implementation. We also implement the baselines following the corresponding official codes, including encoder, hyperparameters, etc. The comparisons are included in Section E of this supplementary material. We directly report the results from their original papers for the same set. For mismatched settings, the results are from our implementation.
Finally, for baselines Ti-MAE [21], TST [56], CoST [46], and TS2Vec [55], we report the results based on the unified encoder in the main text. But for baselines LaST [42] and TF-C [57], we report the results of the official code implementation or their original paper, which are limited by their model structures. As a result, the performances of all baselines with unified encoder (that we reported in the main text) generally surpass their official implementation and results reported in their own paper. Table 8 shows more details. Full experimental results are in Section E.
A.3 Pre-training and Fine-tuning Configuration
We built two types of pre-training and fine-tuning scenarios, in-domain and cross-domain, based on the benchmarks of forecasting and classification tasks to compare the effectiveness of our method and other time series pre-training methods.
We pre-train a model on one subset for forecasting tasks and fine-tune it to the same dataset to build seven in-domain transfer evaluation scenarios. In cross-domain evaluation, we pre-train a model on one specific dataset and use other datasets for fine-tuning. Based on the above settings, we constructed fifteen in- and cross-domain pre-training and fine-tuning experiments, covering the same dataset with the same sampled frequency, different datasets with the same sampled frequency, and different datasets with different sampled frequencies.
We use the same dataset, Epilepsy, to construct the in-domain setting for classification tasks. For the cross-domain setting, we pre-train a model for classification tasks on a univariate time series dataset SleepEEG with the most complex temporal dynamics and the most samples. And then fine-tune the model separately on Epilepsy, FD-B, Gesture, and EMG. Furthermore, we constructed four cross-domain evaluation scenarios by pre-training from SleepEEG and fine-tuning to Epilepsy, FD-B, Gesture, and EMG because of fewer commonalities and the enormous gap among these datasets. Table 9 shows detailed pre-training and fine-tuning settings.
Table 9: Pre-training and fine-tuning scenarios for time series forecasting (Fore.) and classification (Class.) tasks, including the same and different datasets and in- and cross-domain settings.
Tasks	Evaluation	Scenarios	Characteristic
Fore.	In-domain	ETTh1 → ETTh1ETTh2 → ETTh2ETTm1 → ETTm1ETTm2 → ETTm2Weather → WeatherElectricity → ElectricityTraffic → Traffic	The same dataset with the same frequency
Cross-domain	ETTh2 → ETTh1ETTm2 → ETTm1	Different datasets with the same frequency.
{ETTm1, ETTm2, Weather} → ETTh1{ETTh1, ETTm2, Weather} → ETTm1	Different datasets with different frequencies.
Class.	In-domain	Epilepsy → Epilepsy	The same dataset with the same frequency.
Cross-domain	SleepEEG → {Epilepsy, FD-B, Gesture, EMG}	Different datasets with different frequencies.
A.4 Model and Training Configuration
Following the previous convention, we choose the encoder part of Transformer [39] with channel independent as the feature extractor for forecasting tasks. For the classification tasks, we adopt 1D-ResNet [13] as the encoder following [57]. In the pre-training stages, we pre-train the model with different learning rates and batch sizes according to the pre-train datasets. Then we fine-tune it to downstream forecasting and classification tasks supervised by L2 and Cross-Entropy losses, respectively. The configuration details are in Table 10.
Table 10: Model and training configuration in Forecasting (Fore.) and Classification (Class.) tasks.
Tasks	Encoder	Pre-training	Fine-tuning
e_layers	d_model	learning rate	batch size	epochs	learning rate	loss function	batch size	epochs
Fore.	2	16	1e-3	32	50	1e-4	L2	{16,32}	10
Class.	3	128	1e-4	128	10	1e-4	Cross-Entropy	32	300
B Hyperparameter Sensitivity
We verify the hyperparameter sensitivity of SimMTM on ETTh1 in Table 11, including masked ratio (r), the number of masked series (M), temperature  $(\tau)$ , masked function (Mask), encoder depth  $(e_{\mathrm{layers}})$ , and the hidden dimension  $(d_{\mathrm{model}})$ . Lower MSE and MAE represent better performance.
As shown in Table 11 (a) and 11 (b), we can observe the effect of the method is closely related to the trade-off of the masked ratio and the number of masked series. Hence, a reasonable balance between the two kinds of parameters is critical. For the temperature hyperparameter of softmax normalization  $(\tau)$ , we use an appropriately small  $\tau$  that leads to higher differences and diversity of masked sequences. For the masked methods, we chose two masked methods for verification: masking following random distribution and masking following geometric distribution [56]. The results show that the method based on geometric masking is better than random masking modeling. Besides, we can find that 2 encoder layers are enough for reconstruction tasks. Note our method SimMTM consistently performs better than training from scratch under various hyperparameter changes.
Table 11: Hyperparameter sensitivity experiments on ETTh1 for the in-domain setting. The entries marked in bold are the same which specify the default settings. This table format follows [11].
(a) Masked ratio	(b) Masked numbers	(c) Temperature
Ratio	MSE	MAE	Numbers	MSE	MAE	Value	MSE	MAE
12.5%	0.429	0.440	1	0.429	0.437	0.02	0.409	0.428
25%	0.427	0.434	2	0.416	0.429	0.2	0.409	0.429
50%	0.409	0.428	3	0.409	0.428	2	0.416	0.428
75%	0.422	0.434	4	0.419	0.431
(d) Masked function	(e) Encoder depth	(f) Hidden layer dimension
Type	MSE	MAE	Layers	MSE	MAE	Dim	MSE	MAE
Random	0.409	0.431	1	0.420	0.426	16	0.409	0.428
Geometric	0.409	0.428	2	0.409	0.428	32	0.420	0.432
3	0.421	0.430	64	0.422	0.434
4	0.426	0.436	128	0.428	0.444
C Ablations on Aggregation Setting
SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold. We explored two types of aggregation settings.
(1) Positive Samples Aggregation (PSA): only aggregate multiple positive neighbors (the masked series of the same sample) to reconstruct masked time points.
(2) Positive and Negative Samples Aggregation (PNSA): aggregate both positive and negative neighbors (the masked series of all samples) to reconstruct masked time points.
As shown in Table 12, although PSA made good progress compared to training from scratch (Random Init.), PNSA is consistently better than SimMTM PSA in all ablation settings. In masked time-series modeling, masking can be viewed as adding noise to the original data, and masked modeling is to project masked data from the neighborhood back to the original manifold. We use positive and negative masked time series as the reconstruction candidates to drive the model to select the positive samples adaptively, which can make the model learn the structure of the manifold better. Therefore, as stated in the Method Section of the main text, we choose positive and negative sample aggregation (PNSA) as the standard aggregation setting of SimMTM.
Table 12: Ablations on aggregation setting in forecasting (MSE) and classification (Acc) tasks for in- and cross-domain settings. A smaller MSE or a higher Accuracy indicates a better result  $(\uparrow)$ .
Tasks	Evaluation	Scenarios	Aggregation	Metric
Forecasting	In-domain	ETTh1 → ETTh1	Random init.	0.431
SimMTM (PSA)	0.420 ↑
SimMTM (PNSA)	0.409 ↑
Cross-domain	ETTh2 → ETTh1	Random init.	0.431
SimMTM (PSA)	0.426 ↑
SimMTM (PNSA)	0.415 ↑
Classification	In-domain	Epilepsy → Epilepsy	Random init.	89.83
SimMTM (PSA)	92.56 ↑
SimMTM (PNSA)	94.75 ↑
Cross-domain	SleepEEG → EMG	Random init.	77.80
SimMTM (PSA)	87.80 ↑
SimMTM (PNSA)	97.56 ↑
D Comparison of Masked Modeling
To investigate the reconstruction process of different masked modeling methods, we plot both original and reconstructed time series from TST and SimMTM in Figure 6, where TST [56] follows the canonical masked modeling paradigm and learns to predict removed time points based on the remaining time points. In Figure 6, we can find that direct reconstruction is too difficult in time series, even for the  $12.5\%$  masking ratio. As for the  $75\%$  masking ratio, TST degenerates more seriously. Because of this poor reconstruction effect, direct reconstruction is difficult to provide reliable guidance to model pre-training. In contrast, our proposed SimMTM can precisely reconstruct the original time series, benefiting the representation learning. These results also support our design in neighborhood reconstruction.
Figure 6: Comparison of the canonical masked modeling paradigm TST and neighborhood aggregation masked modeling SimMTM in reconstructing time series. All the cases are shown from ETTh1.
E Full Results
Due to the limited length of the text, we summarize all the experiments in the main text into two parts: the main experiment and the analytical experiment. We categorize and index them in Tabel 13, 14.
Table 13: The main results of pre-training and fine-tuning scenarios for time series forecasting and classification tasks, including the same and different encoder for in- and cross-domain settings.
Tasks	Evaluation	Encoder	Labels Name
Forecasting	In-domain	The model utilized in the original papers	Table 15
Transformer with channel independent	Table 16
Cross-domain	The model utilized in the original papers	Table 17
Transformer with channel independent	Table 18
Classification	In-domain	The model utilized in the original papers	Table 23
1D-ResNet	Table 24
Cross-domain	The model utilized in the original papers	Table 23
1D-ResNet	Table 24
Table 14: The model analysis results of pre-training and fine-tuning scenarios for time series forecasting and classification tasks with the unified encoder for in- and cross-domain settings.
Tasks	Evaluation	Analysis	Labels Name
Forecasting	In-domain	Ablation study	Table 19
Model generality	Table 21
Cross-domain	Ablation study	Table 20
Limited data	Table 22
Classification	In-domain	Ablation study	Table 25
Cross-domain	Ablation study	Table 25
SimMTM is inspired by the manifold perspective of masked modeling. Although we have provided relatively comprehensive results to verify the model's effectiveness, the model performance still needs theoretical guarantees. In fact, the most high-impact works in the self-supervised pre-training community are without theoretical analysis, such as BERT [7], GPT-3 [10], MAE [11] and SimMIM [50]. Thus, we would like to leave this problem as a future work.
The masking ratio of masked modeling methods is an essential hyper-parameter. Although we have provided a chosen principle to masking ratio  $r$  and the number of masked time series  $M$  as  $M \propto r$  in the main text, we still need to tune these two hyperparameters for different datasets to achieve the best performance. Notably, previous methods also chose the masking ratio solely based on the empirical results [7, 11]. Thus, despite there exist limitations of SimMTM in choosing hyperparameters, the principle of  $M \propto r$  can somewhat ease this problem. And the chosen strategy of the masking ratio can also be a potential topic in masked modeling [44].
G Social Impacts
This paper presents SimMTM as a new masked modeling method for time series. SimMTM achieves state-of-the-art in two mainstream time series analysis tasks, which can be a good supplement for the self-supervised pre-training community. We will also publish the codebase of time-series pre-training to facilitate future research.
This paper only focuses on the algorithm design. Using all the codes and datasets strictly follows the corresponding licenses (Appendix A.1). There is no potential ethical risk or negative social impact.
Table 19: Full ablation studies for the in-domain setting of forecasting. The standard deviations of SimMTM are within 0.005 for MSE and within 0.004 for MAE.
Input-336Metric	Supervised	W/o Lreconstruction	W/o Lconstraint	SimMTM
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	96	0.380	0.412	0.377	0.408	0.381	0.409	0.379	0.407
192	0.416	0.434	0.419	0.443	0.409	0.443	0.412	0.424
336	0.448	0.458	0.423	0.434	0.432	0.444	0.421	0.431
720	0.481	0.487	0.437	0.454	0.447	0.454	0.424	0.449
Avg	0.431	0.448	0.414	0.435	0.417	0.438	0.409	0.428
ETTh2	96	0.325	0.374	0.288	0.344	0.312	0.365	0.293	0.347
192	0.400	0.424	0.356	0.391	0.389	0.418	0.355	0.386
336	0.405	0.433	0.368	0.406	0.396	0.432	0.370	0.401
720	0.451	0.475	0.409	0.432	0.448	0.479	0.395	0.427
Avg	0.395	0.427	0.355	0.393	0.386	0.424	0.353	0.390
ETTm1	96	0.295	0.346	0.291	0.343	0.282	0.337	0.288	0.348
192	0.333	0.374	0.330	0.390	0.324	0.388	0.327	0.373
336	0.370	0.398	0.369	0.399	0.366	0.397	0.363	0.395
720	0.427	0.431	0.417	0.429	0.424	0.435	0.412	0.424
Avg	0.356	0.387	0.352	0.390	0.349	0.389	0.348	0.385
ETTm2	96	0.175	0.268	0.174	0.265	0.170	0.261	0.172	0.261
192	0.240	0.312	0.232	0.303	0.244	0.320	0.223	0.300
336	0.298	0.351	0.313	0.365	0.279	0.334	0.282	0.331
720	0.403	0.413	0.376	0.451	0.376	0.378	0.374	0.388
Avg	0.279	0.336	0.274	0.346	0.267	0.323	0.263	0.320
Weather	96	0.166	0.216	0.164	0.209	0.160	0.212	0.158	0.211
192	0.208	0.254	0.203	0.258	0.203	0.251	0.199	0.249
336	0.257	0.290	0.244	0.289	0.253	0.290	0.246	0.286
720	0.326	0.338	0.322	0.343	0.325	0.340	0.317	0.337
Avg	0.239	0.275	0.233	0.275	0.235	0.273	0.230	0.271
Electricity	96	0.190	0.279	0.177	0.270	0.134	0.220	0.133	0.223
192	0.195	0.285	0.184	0.279	0.163	0.274	0.147	0.237
336	0.211	0.301	0.202	0.300	0.223	0.311	0.166	0.265
720	0.253	0.333	0.250	0.337	0.241	0.321	0.203	0.297
Avg	0.212	0.300	0.203	0.397	0.190	0.282	0.162	0.256
Traffic	96	0.471	0.309	0.366	0.257	0.457	0.301	0.368	0.262
192	0.475	0.308	0.373	0.266	0.468	0.325	0.373	0.251
336	0.490	0.315	0.401	0.249	0.487	0.302	0.395	0.254
720	0.524	0.332	0.472	0.312	0.485	0.315	0.432	0.290
Avg	0.490	0.316	0.403	0.271	0.474	0.311	0.392	0.264
Table 20: Full ablation studies on transfer to ETTh1 and ETTm1 for the cross-domain setting of forecasting. The standard deviations of SimMTM are within 0.005 for MSE and within 0.004 for MAE.
Input-336Metric	Supervised	W/o Lreconstruction	W/o Lconstraint	SimMTM
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh2↓ETTh1	96	0.380	0.412	0.377	0.400	0.402	0.411	0.372	0.401
192	0.416	0.434	0.417	0.424	0.417	0.420	0.414	0.425
336	0.448	0.458	0.437	0.439	0.437	0.435	0.429	0.436
720	0.481	0.487	0.448	0.463	0.456	0.467	0.446	0.458
Avg	0.431	0.448	0.420	0.432	0.423	0.430	0.415	0.430
ETTm1↓ETTh1	96	0.380	0.412	0.382	0.397	0.375	0.399	0.367	0.398
192	0.416	0.434	0.418	0.418	0.413	0.422	0.396	0.421
336	0.448	0.458	0.437	0.434	0.434	0.438	0.471	0.437
720	0.481	0.487	0.459	0.469	0.467	0.475	0.454	0.463
Avg	0.431	0.448	0.424	0.430	0.422	0.434	0.422	0.430
ETTm2↓ETTh1	96	0.380	0.412	0.388	0.418	0.384	0.415	0.388	0.421
192	0.416	0.434	0.429	0.444	0.423	0.439	0.419	0.423
336	0.448	0.458	0.467	0.472	0.458	0.465	0.435	0.444
720	0.481	0.487	0.521	0.507	0.501	0.497	0.468	0.474
Avg	0.431	0.448	0.451	0.460	0.441	0.454	0.428	0.441
Weather↓ETTh1	96	0.380	0.412	0.385	0.400	0.394	0.406	0.477	0.444
192	0.416	0.434	0.417	0.429	0.425	0.424	0.454	0.522
336	0.448	0.458	0.434	0.434	0.441	0.439	0.424	0.434
720	0.481	0.487	0.444	0.464	0.446	0.468	0.468	0.469
Avg	0.431	0.448	0.420	0.432	0.427	0.434	0.456	0.467
ETTh1↓ETTh1	96	0.295	0.346	0.286	0.341	0.290	0.346	0.290	0.348
192	0.333	0.374	0.322	0.362	0.353	0.388	0.327	0.372
336	0.370	0.398	0.362	0.418	0.362	0.412	0.357	0.392
720	0.427	0.431	0.417	0.431	0.422	0.432	0.409	0.423
Avg	0.356	0.387	0.347	0.388	0.357	0.395	0.346	0.384
ETTh2↓ETTh1	96	0.295	0.346	0.299	0.348	0.301	0.352	0.322	0.347
192	0.333	0.374	0.324	0.366	0.332	0.359	0.332	0.372
336	0.370	0.398	0.374	0.401	0.389	0.382	0.394	0.391
720	0.427	0.431	0.415	0.419	0.421	0.442	0.411	0.424
Avg	0.356	0.387	0.353	0.386	0.361	0.384	0.365	0.384
ETTm2↓ETTh1	96	0.295	0.346	0.299	0.351	0.285	0.336	0.297	0.348
192	0.333	0.374	0.334	0.372	0.343	0.366	0.332	0.370
336	0.370	0.398	0.362	0.388	0.360	0.399	0.364	0.393
720	0.427	0.431	0.417	0.431	0.422	0.432	0.410	0.421
Avg	0.356	0.387	0.353	0.386	0.353	0.383	0.351	0.383
Weather↓ETTh1	96	0.295	0.346	0.322	0.361	0.309	0.354	0.294	0.354
192	0.333	0.374	0.344	0.378	0.343	0.365	0.318	0.355
336	0.370	0.398	0.371	0.399	0.401	0.411	0.361	0.397
720	0.427	0.431	0.426	0.422	0.425	0.427	0.427	0.426
Avg	0.356	0.387	0.366	0.390	0.370	0.389	0.350	0.383
Table 21: Full results for applying SimMTM to four advanced time series forecasting models under the in-domain setting. The gray mark represents negative transfer  $(\downarrow)$ .
Models	Transformer [39]	Autoformer [47]	Ns Transformer [24]	PatchTST [26]
Random init. +SimMTM	Random init. +SimMTM	Random init. +SimMTM	Random init. +Sub-serie Masking +SimMTM
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	96	0.847	0.731	0.775	0.691	0.536	0.548	0.526	0.536	0.513	0.491	0.490	0.489	0.375	0.399	0.366	0.398	0.373	0.399
192	1.084	0.841	0.918	0.763	0.543	0.551	0.523	0.548	0.534	0.504	0.517	0.499	0.414	0.421	0.431	0.443	0.406	0.428
336	1.350	0.956	1.079	0.845	0.615	0.592	0.595	0.591	0.588	0.535	0.552	0.520	0.431	0.436	0.450↓	0.456↓	0.422	0.431
720	1.069	0.817	0.935	0.761	0.599	0.600	0.600	0.597	0.643	0.616	0.614	0.598	0.449	0.466	0.472↓	0.484↓	0.436	0.452
Avg	1.088	0.836	0.927	0.761	0.573	0.573	0.561	0.568	0.570	0.537	0.543	0.527	0.417	0.431	0.430↓	0.445↓	0.409	0.428
ETTh2	96	2.029	1.150	1.879	1.104	0.492	0.517	0.488	0.514	0.476	0.458	0.445	0.448	0.274	0.336	0.284↓	0.343↓	0.274	0.337
192	6.785	2.099	5.054	1.771	0.556	0.551	0.547	0.549	0.512	0.493	0.482	0.502	0.339	0.379	0.355↓	0.387↓	0.339	0.377
336	4.568	1.711	4.242	1.658	0.572	0.578	0.563	0.570	0.552	0.551	0.512	0.537	0.331	0.380	0.379↓	0.411↓	0.327	0.381
720	3.030	1.486	2.815	1.413	0.580	0.588	0.575	0.588	0.562	0.560	0.531	0.568	0.379	0.422	0.400↓	0.435↓	0.375	0.423
Avg	4.103	1.612	3.498	1.487	0.550	0.559	0.543	0.555	0.526	0.516	0.493	0.514	0.331	0.379	0.355↓	0.394↓	0.329	0.379
ETTm1	96	0.562	0.520	0.513	0.497	0.523	0.488	0.482	0.465	0.386	0.398	0.340	0.376	0.290	0.342	0.289↓	0.344↓	0.288	0.343
192	0.810	0.668	0.686	0.606	0.543	0.498	0.499	0.476	0.459	0.444	0.423	0.445	0.332	0.369	0.323	0.368	0.329	0.367
336	1.096	0.814	1.003	0.760	0.675	0.551	0.601	0.524	0.495	0.464	0.423	0.459	0.366	0.392	0.353	0.387	0.361	0.387
720	1.136	0.813	1.032	0.790	0.720	0.528	0.629	0.555	0.585	0.516	0.539	0.499	0.420	0.424	0.398	0.416	0.413	0.417
Avg	0.901	0.704	0.809	0.663	0.615	0.528	0.553	0.505	0.481	0.456	0.431	0.445	0.352	0.382	0.341	0.379	0.348	0.378
ETTm2	96	0.508	0.539	0.336	0.425	0.255	0.339	0.255	0.340	0.192	0.274	0.188	0.277	0.165	0.255	0.166↓	0.256↓	0.163	0.253
192	0.972	0.721	0.713	0.610	0.281	0.340	0.276	0.332	0.280	0.339	0.277	0.336	0.220	0.292	0.221↓	0.295↓	0.219	0.292
336	1.419	0.897	1.517	0.942	0.339	0.372	0.309	0.359	0.334	0.361	0.325	0.355	0.278	0.329	0.278	0.333↓	0.275	0.328
720	3.598	1.445	2.720	1.254	0.422	0.419	0.420	0.410	0.417	0.413	0.414	0.412	0.367	0.385	0.365	0.388↓	0.359	0.381
Avg	1.624	0.901	1.322	0.808	0.324	0.368	0.315	0.360	0.306	0.347	0.301	0.345	0.258	0.317	0.258	0.318↓	0.254	0.313
Table 22: Full results for fine-tuning to limited data scenarios. We fine-tune the model pre-trained from ETTh2 to ETTh1 with different data proportions  $\{10\%, 25\%, 50\%, 75\%, 100\}$ .
Models	SimMTM	Random init.	Ti-MAE[21]	TST[56]	LaST[42]	TF-C[57]	CoST[46]	TS2Vec[55]
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh2↓ETTh1	10%	0.591	0.523	0.653	0.558	0.660	0.517	0.783	0.588	0.645	0.507	0.799	0.783	0.784	0.604	0.655	0.550
25%	0.535	0.490	0.632	0.502	0.594	0.518	0.641	0.578	0.610	0.611	0.736	0.725	0.624	0.539	0.632	0.543
50%	0.491	0.473	0.512	0.479	0.550	0.504	0.525	0.509	0.540	0.513	0.731	0.704	0.540	0.499	0.599	0.526
75%	0.466	0.458	0.499	0.488	0.475	0.465	0.516	0.488	0.479	0.470	0.697	0.689	0.494	0.475	0.577	0.534
100%	0.415	0.430	0.431	0.448	0.466	0.456	0.469	0.459	0.443	0.471	0.635	0.634	0.428	0.433	0.517	0.486
Table 23: In- and cross-domain settings of classification, where all the baselines are based on the encoder utilized in their original papers. For in-domain setting, we pre-train and fine-tune on the same dataset: Epilepsy. For cross-domain setting, we pre-train the model on SleepEEG and then fine-tune it on different datasets: Epilepsy, FD-B, Gesture, and EMG.
Scenarios	Models	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)	Avg (%)
In-Domain	Epilepsy	Random init.	89.83	92.13	74.47	79.59	84.00
TS2vec [55]	92.17	93.84	81.19	85.71	88.23
CoST[46]	88.07	91.58	66.05	69.11	78.70
LaST [42]	92.11	93.12	81.47	85.74	88.11
TST [56]	80.21	40.11	50.00	44.51	53.71
Ti-MAE [21]	90.09	93.90	77.24	78.21	84.86
TF-C [57]	93.96	94.87	85.82	89.46	91.03
SimMTM	94.75	95.60	89.93	91.41	92.92
Cross-Domain	SleepEEG	Random init.	89.83	92.13	74.47	79.59	84.00
TS2vec [55]	93.95	90.59	90.39	90.45	91.35
CoST[46]	88.40	88.20	72.34	76.88	81.45
LaST [42]	86.46	90.77	66.35	70.67	78.56
TST [56]	80.21	40.11	50.00	44.51	53.71
Ti-MAE [21]	89.71	72.36	67.47	68.55	74.52
TF-C [57]	94.95	94.56	89.08	91.49	92.52
SimMTM	95.49	93.36	92.28	92.81	93.49
SleepEEG	Random init.	47.36	48.29	52.35	49.11	49.28
TS2vec [55]	47.90	43.39	48.42	43.89	45.90
CoST[46]	47.06	38.79	38.42	34.79	39.76
LaST [42]	46.67	43.90	47.71	45.17	45.86
TST [56]	46.40	41.58	45.50	41.34	43.71
Ti-MAE [21]	60.88	66.98	68.94	66.56	65.84
TF-C [57]	69.38	75.59	72.02	74.87	72.97
SimMTM	69.40	74.18	76.41	75.11	73.78
SleepEEG	Random init.	42.19	47.51	49.63	48.86	47.05
TS2vec [55]	69.17	65.45	68.54	65.70	67.22
CoST[46]	68.33	65.30	68.33	66.42	67.09
LaST [42]	64.17	70.36	64.17	58.76	64.37
TST [56]	69.17	66.60	69.17	66.01	67.74
Ti-MAE [21]	71.88	70.35	76.75	68.37	71.84
TF-C [57]	76.42	77.31	74.29	75.72	75.94
SimMTM	80.00	79.03	80.00	78.67	79.43
SleepEEG	Random init.	77.80	59.09	66.67	62.38	66.49
TS2vec [55]	78.54	80.40	67.85	67.66	73.61
CoST[46]	53.65	49.07	42.10	35.27	45.02
LaST [42]	66.34	79.34	63.33	72.55	70.39
TST [56]	78.34	77.11	80.30	68.89	76.16
Ti-MAE [21]	69.99	70.25	63.44	70.89	68.64
TF-C [57]	81.71	72.65	81.59	76.83	78.20
SimMTM	97.56	98.33	98.04	98.14	98.02