SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling
Authors: Jiaxiang Dong
Haixu Wu
Haoran Zhang
Li Zhang
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Time-series analysis
pre-training
masked time-series modeling
TL;DR: We present a new masked modeling method for time series as SimMTM. Unlike the previous, it reconstructs the original time series with multiple series. SimMTM achieves state-of-the-art fine-tuning performance in both forecasting and classification.

Abstract: Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings.

SimMTT: A Simple Pre-Training Framework for Masked Time-Series Modeling
Jiaxiang Dong, Haixu Wu," Haoran Zhang, Li Zhang, Jianmin Wang, Mingsheng Long School of Software, BNRist, Tsinghua University, China
{djx200, 22j==2,,-hr2200 200aa000mm0mmass..... teinghua.nhh...... edu. en, suhaizuf8@gmail. com
(lizhang, imwang, mww.anhhnnngtttingnaaaa edu. .n
Abstract
Time series analysis is widely used in extensive areas. Recently, reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted im- mense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but comple- mentary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked model- ing. Experimentally, SimMTMM achieves state-of-the-an fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings. Code is available at
1
Introduction
Time series analysis has attached immense importance in extensive real-world applications, such as financial analysis, energy planning and etc [T]. Vast amounts of time series are incrementally collected from 1oT and wearable devices. However, the semantic information of time series is mainly buried camman-nndisceribbee temporal variations, making it difficult to annotate. Recently, self-supervised pre-training has been widely explored [ (55. which benefits deep models from pretext knowledge learned over large-scale unlabeled data and further promotes the performance of various downstream tasks. Mainly, achieved great success in many areas, such as masked language modeling (MLM) [1] 32 3 10)
well-recognized pre- mre-trannnn paradigm, masked modeling has
and masked image modeling (MIM) [I] 50 [20]. This paper extends pre-training methods time series, especially masked time-series modeling (MTM).
The canonical technique of masked modeling is to optimize the model by learning reconstruct the masked content based on the unmasked part 77]. However, unlike images and natural languages whose patches words contain much even redundant semantic information, valuable semantic information of time series is mainly included in the temporal variations, such as the trend, periodicity, and peak valley, which correspond to unique weather processes, abnormal faults, etc. in the real world. Therefore, directly masking a portion of time points will seriously ruin
observe that the
"Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Drrect Recoraraction
OrgaallTne Serics RecorninateeTerc. Tare
No ahborhood Auy aptles
Origial Time Senis
Mulpple Masked Series
Reconntrustd Time Sories
Modee
-1
-0
giodl Series
Origina Series
-33
.124
Manifold
Manifold
(a) Canonical Masked Modeling
(b) SimMTTM
Figure 1: Comparison between (a) canonical masked modeling and (b) SimMTM in both manifold
perspective and reconstruction performance. The showcase is to recover 50% masked time series.
the temporal variations of the original time series, which makes the reconstruction task too difficult to guide representation learning of time series. (Figure I.
view the randomly masked series as the "neighbor" of the original time series outside the manifold and the reconstruction process is project the masked series back the manifold of original series. However, as we analyzed above, direct reconstruction may fail since the essential temporal variations are ruined by random masking. Inspired by the manifold perspective, reconstruction convention of masked modeling and propose a natural idea of reconstructing the original data from its multiple "neighbors", i.e. multiple masked series. Although the temporal variations of the original time series have been partially dropped in each randomly masked series, the multiple randomly masked series will complement each other, making the reconstruction process much more accessible than directly reconstructing the original series from single masked series. This process will also pre-train the model uncover the local structure of the time series manifold implicitly, thereby benefiting masked modeling and representation learning [35] 4T..
According to the analysis in stacked denoising autoencoders [40], as shown in Figure !
go beyond the straightforward
Based the above insights, we propose the SimMTM simple but effective pre-training framework for time series in this paper. Instead of directly reconstructing the masked time points from unmasked parts, SimMTM recovers the original time series from multiple randomly masked time series. Technically, SimMTM presents a neighborhood aggregation design for reconstruction, which is to aggregate the point-wise representations of time series based similarities learned in the series-wise representation space. In addition to the reconstruction loss, a constraint loss is presented to guide the ceries-wise representation learning based on the neighborhood assumption of the time series manifold. Empowering by above designs, SimMTM achieves consistent ofatc-of-thh-arr in various time series analysis tasks when tunint the pre-trained model into downstream tasks, covering both the low-level forecasting and high-level classification tasks, even if there is a clear domain shift between pre-training and fine-tuning datasets. Overall, our contributions are summarized as follows:
Inspired by the manifold perspective of masking, propose a new task for masked time- series modeling, which is to reconstruct the original time series on the manifold based on multiple masked series outside the manifold.
Technically, we present SimMTM as a simple but effective pre- pre-training framework, which aggregates point-wise representations for reconstruction based on the similarities learned in series-wise representation space.
SimMTM consistently achieves the state-of-the-arr fine-tuning performance in typical time series analysis tasks, including low-level forecasting and high-level classification, covering both in- and cross-domain settings.
2 Related Work
2.1 Self-supervised Pre-training
Self supervsee pre-training is an important research topic for learning generalizable and shared knowledge from large-scale data and further benefiting downstream tasks [15]. Originally, this topic has been widely explored in computer vision and natural language processing. Elaborative manually- designed superiidd tasks are presented, which can be roughly categorized into contrastive learning [12 5 and masked modeling [1]. Recently, following the well-established contrastive learning and masked modeling paradigms, some self-supervised pre-training methods for time series
have been proposed 99] 22 34 33, 36 54 [22 6).

---

Contrastive learning. The critical insight of contrastive learning is to optimize the representation space based on the manually designed positive and negative pairs, where representations of positive pairs are optimized to be close to each other. In contrast, negative ones tend to be far apart [48] 14).
The canonical design presented in SimCLR [37 views different augmentations of the same sample as positive pairs and augmentations among different samples as negative pairs.
Recendly in time series pre-training, many designs of positive and negative pairs have been proposed by utilizing the invariant properties of time series, Concretely, to make the representation learning seamlessly related temporal variations, TimCLR [53] adopts the DTW 25 to generate phase- shift and syplitude-changee augmentations, which is more suitable for time series context. TS2 [55] splits multiple time series into several patches and further defines the contrastive loss in both instance- wise and patch-wise aspects. TS-TCC [8] presents a new temporal contrastive learning task as making the augmentations predict each other's future. Mixing-up 455 exploits a data augmentation scheme in which new samples are generated by mixing two data samples. LaST [42 aims to disentangle the seasonal-trend representations in the latent space based on variational inference. Afterward, CoST [46] employs contrastive losses in both time and frequency domain to learn discriminative seasonal and trend representations. Besides, TF-C [57] proposes a novel fimefreeuency consistency architecture and optimizes time-based and frequency-basee representations of the same example to be close to each other. Note that contrastive learning mainly focuses on the high-level information [49], and the series-wise patch-wisg wise representations inherently mismatch the low-level tasks, such as time series forecasting. Thus, in this paper, we focus on the masked modeling paradigm.
Masked modeling. The masked modeling paradigm optimizes the model by learning to reconstruct the masked content from the unmasked part. This paradigm has been widely explored in computer vision and natural language processing. which is to predict the masked words of sentence [7]
and masked patches of an image [ 50] respectively. As for the time series analysis, TST [56] follows the canonical masked modeling paradigm and learns predict removed time points based on the remaining time points. Next, PatchTST (26) proposes to predict masked subseries-level patches to capture the local semantic information and reduce memory usage. Ti-MAE (2] uses mask modeling as an auxiliary task to boost the forecasting and classification performances of advanced Transformer-based methods. However, as we stated before, directly masking time series will ruin the essential temporal variations, making the reconstruction too difficult to guide the representation learning. Unlike the direct reconstruction in previous works, SimMTM presents a new masked modeling task, which is reconstructing the original time series from multiple randomly masked series.
2.2 Understanding Masked Modeling
Masked modeling has been explored in stacked denoising autoencoders [40. where masking is viewed as adding noise to the original data and the masked modeling is to project masked data from the neighborhood back to the original manifold, namely denoising. It has recently been widely used in pre-training, which can learn valuable low-level information from data unsupervisedly [49]. Inspired by the manifold perspective, we go beyond the classical denoising process and project the masked data back to the manifold by aggregating multiple masked time series within the neighborhood.
3 SimMTM
As aforementioned, to tackle the problem that randomly masking time series will ruin the essential temporal variation information, SimMTM proposes to reconstruct the original time series from multiple masked time series. To implement this. SimMTM first learns similarities among multiple time series in the series-wise representation space and then aggregates the point-wise representations of these time series based on pre-learned ceries-wise similarities. Next, we will detail the techniques in both model architecture and pre-training protocol aspects.
3.1 Overall Architecture
The reconstruction process of SimMTM involves the following four modules: masking, representation learning, stries-wise similarity learning and point-wise reconstruction.
3

---

Self ratessperviiee Pre-training
Representstles Orighet Senis
Masked
Decoder
wan Simbaity - wareewss Age spation
For fan each other
Sories
Similurty
( 11-1
Encoder
Point-iie App sextion
Projetor
Poirt-vise Representatio
(x.) U (xt
Origiaa Masked Sansa
y Viy
Decoder
M
Reconomunted Original Series
[ (M,
Figure 2: Architecture of SimMTM, which reconstructs the original time series by adaptive aggregat- ing multiple masked time series based series-wing similarities learned contrastively from data.
Masking. Given {x: mini-batch of N time series samples, where x; RLxC contains L time points and C' observed variates, we can easily generate set of masked series for each sample x; by randomly masking a portion of time points along the temporal dimension, formalizing by:
{ M
Mask, (x;);
where [0. denotes the masked portion. M is hyperparameter for the number of masked time series. x/ E R represents the j-th masked time series of x,, where the values of masked time points are replaced by zeros. With above process, we can obtain batch of augmented time series.
For clarity, we present all the (N(M + 1)) input series in a set as X' U ((x)) (
Representation learning. After the encoder and projector layer, we can obtain the point-wise representations Z and series-wise representations S, which can be formalized as:
2 z U (z (Z)
Ex)c s=
u = Projector (2), (2) 1=1
1m1
where 2,, 7 RLxdtt and 5 1xddmm Encoder(-) denotes the model encoder, which can project the input data into deep representations and will transferred to downstream tasks during the fine-tuning process. In this paper, we implement the encoder well-acknowledged Transformer 33 and ResNet [3]. As for the Projector(.)). we employ a simple MLP layer along the temporal dimension obtain series-wise representations. More details be found in Section Technically, the encoder is applied input series separately, namely U (Encoder (x;) and so does for projector. Here we adopt the set-style formalization for conciseness.
{Encoder (Encoder(x))))
Series-wise similarity learning. Note that directly averaging multiple masked time series will result in the over-smoothing problem (40), impeding the representation learning. Thus, to precisely reconstruct the original time series, we attempt to utilize the similarities among scries-wise represen- tations S for weighted aggregation, namely exploiting the local structure of the time series manifold. For simplification, we formalize the calculation of series-wiss similarities as follows:
uv
R Sim
IR DXD D N(M 1), Ru,v
U.
|u ||
where R is the matrix of pair-wise similarities for (N(M + 1)) input samples in series-wise repre- sentation space, which are measured by the cosine distance. Ru, is the calculated similarity between series-wise representations
E S.
Point-wise aggregation. As shown in Figure 2 based on the learned series-wing similarities, the aggregation process for the i-th original time series is:
exp (Rp /T)
2,
Σ Σ ES)(S))))) exp exp(Ra,s*) ('E
z'
where z' represents the corresponding point- wise representation of s', i.c. z' Projector(s').. 2, R Lixdaa is the reconstructed point-wisg wint representation. denotes the temperature hyperparameter

---

of softmax normalization for ceries-wise similarities. is notable that as described in Eq. 4 for
each time series Xi, the reconstruction is not only based its own masked series (x M f=1- We also introduce other series representations 8\ (((s) into aggregation, which requires the model to suppress interference of less-related noise series and precisely learn similar representations for both masked and the original series, namely guiding the model to learn the manifold structure better. After decoder, we can obtain the reconstructed original time series, namely
& ) = Decoder( ( (()))));
(5)
where & B is the reconstruction x,. Decoder(-) is instantiated simple MLP layer along the channel dimension following [50]
3.2 Self-supervised Pre-training
Following the masked modeling paradigm, SimMTM is supervised by reconstruction loss:
Creconstraction
Σ x -
(6)
Note that the reconstruction process directly based on the series-wise similarities, while it is hard to guarantee the model captures the precise similarities without explicit constraints in the series-wise representation space. Thus, to avoid trivial aggregation, we also utilize the neighborhood assumption of the time series manifold to calibrate the structure of criesswise representation space S. For clarity, we formalize the neighborhood assumption by defining positive and negative pairs as follows:
Positive pairs: (s;, st.s)) s E 1
1==
(7)
Negative pairs: (s,, S, ), s E {$k} U 1
where s," and s; mean the elements that are assumed as close to and far away from s; respectively. Eq. (7) indicates that the original time series and its masked series will present close representations and be far away from the representations from other series in S. For each veries-wiss representation s E S, we define the set of its assumed close series as S S. Note that to avoid the dominating representation, we assume that S+. With the above formalization, can define manifold constraint to series-wise representation space as:
exp(Ra,x /T) E exp(RRg,s'/5
counstraint =-
E SES
log
(8)
s'ES!
which can optimize the learned series-wise representations to satisfy the neighhorhood assumption in Eq. 7 better. Finally, the overall optimization process of SimMTM can be represented as follows:
min Creconstruction consstraint
(9)
where O denotes the set of all parameters of the deep architecture. To trade off the two parts in Eq. ( we adopt the tuning strategy presented by [17]. which can adjust the hyperparameters adaptively according to the homoscedastic uncertainty of each loss.
4 Experiments
To fully evaluate SimMTM, we conduct experiments on two typical time series analysis tasks: forecasting and classification, covering low-level and high-level representation learning. Further, we present the fine-tuning performance for each task under in- and cross-domain settings.
Benchmarks. We summarize the experiment benchmarks in Table comprising twelve datasets that cover two mainstream time series analysis tasks: time series forecasting and classification. Concretely, we have followed the standard experimental setups in Autoformer [47] for the forecasting tasks and the experiment settings proposed by TF-C [57] for classification.
5

---

Implementations. We compare SimMTM with six competitive state-of-the-ar baseline methods, including the contrastive learning methods: TF-C [57]. CoST [46), Ve2 [55], LaST 42 the masked modeling method: Ti- MAE
Table 1: Summary of experiment benchmarks.
Tasks
Datasets
Semantic
ETTh
Electricity
21 TST [56]. TF-C [57] and Ti-
ETTml.ETTm2
Electricity
221 are the previous best pre-training methods. We experiment on both in- and cross- domain settings. For the in-domain setting, pre-train and fine-tune the model using the same dataset. As for the cross- cross-domain setting, we pre- train the model on a certain dataset and fine-tune the encoder to different datasets. More details are provided in Appendix A
MAE
Forccasting
Weather
Weather
Electricity
Electricity
Traffic
Transportation
SleepEEG
EEG
Epilepsy
EEG
Classification
FD-B
Faulty Detection
Gesture
Hand Movement
Unified encoder. To make fair compari- son, we attempt to unify encoder for these pre-
EMG
Muscle Responses
training methods. Concretely, we adopt vamilla Transformer as the unified encoder for forecasting. The channel-independee design allows models to accomplish cross-domain transfer between datasets with different variate numbers, As for the classification, we use NeRResNet [3] as the shared encoder following [57]. Notably, for LaST 42 and TFC [57]. since their designs are closely related to model structures, we directly report results from their papers or reproduce with official codes. For other baselines and SimMTM, the results in the main text are from the unified encoder. The results from their official papers are also compared in Appendix A.2 For all baselines, the results with a unified encoder generally surpass results reported by themselves ves.
39 with the channel independence
4.1
Main results
We summarize the results in forecasting and classification of in- and cross-domain settings in Figure In all these settings, SimMTM outperforms other baselines significantly. It is also notable that although the masking-based method Ti-MAE (21) achieves good performance in the forecasting task (x-axis of Figure 3). it fails in the classification task (y-axis). Besides, contrastive-based methods fail in low-level forecasting tasks. These results indicate that previous methods cannot simultaneously cover high-level and level tasks, highlighting the advantages of SimMTM in task generality.
In-Domain
Cross-Domaie
100
95
MAE
CoST
Classifiaation (Spllopy to Epilspoy (Aoc)
*
(StesEEG to EMG (Aco)
95
LaST
SimMTM
20
90
85
80
30-
75
70-
TST
Random Init.
LaST
SimMTM
65
70
60
0.52
Forecasting (ETTH) ETTH (MSE)
Forecasting ThT to (MSB)
Figure 3: Performance comparison of time series pre-training methods forecasting (MSE 1) and classification (Acc 1) tasks, including both in-domain (left) and cross-domain (right) settings.
4.2 Forecasting
In-domain. As shown in Table empowering by SimMTM pre-training, the model performance promoted significantly (SimMTM vs. Random init.). Besides, SimMTM consistently outperforms other pre-training methods. On the average of all benchmarks, SimMTM achieves 8.3% MSE reduc- tion and 4.3% MAE reduction compared the advanced masked modeling baseline Ti-MAE 21).

---

Table 2: In-domain setting of forecasting the future O time points based on the past 336 time points.
All results are averaged from 4 different choices of E {96, 192, 336, 720}. A smaller MSE or MAE indicates better prediction. See Appendix B for full results.
Models
SimMTM Random init. TI-MAE 21
TST 56
LaST 42
TF-C
S7
CoST 46
Ve2
35
Metric
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTh1
0.4499 0.428 0.431 0.448 0.423 0.446 0.466 0.462 0.474 0.461 0.527 0.513 0.485 0.472 0.446 0.456
ETTh2
0.353
0.390 0.395 0.427 0.380 0.386 0.404 0.421 0.449 0.459 0.692 0.724 0.399 0.427 0.417 0.468
ETTml
0.348
0.385 0.356 0.387 0.366 0.391 0.373 0.389 0.398 0.398 0,496 0.474 0.356 0.385 0.699 0.557
ETTm2
0.263
0.320 0.279 0.336 0.267 0.325
0.297 0.347 0.265 0.327 0.465 0.562 0.314 0.365 0.326 0.361
Weather
0.230 0.271 0.239 0.275 0.234 0.265 0.239 0.276 0.232 0.261 0.286 0.349 0.324 0.329 0.233 0.267
Electricity
0.162 0.256 0.212 0.200
0.296 0.209 0.289 0.186 0.274 0.363 0.432 0.215 0,295 0.213 0.293
Traffic
0.392 0.264 0.490 0.416 0.310 0.586 0.362 0.713 0.397 0.717 0.456 0.435 0,362 0.470 0.350
Avg
0.308 0.331 0.343 0.356 0.346 0.368 0.364 0.388 0.368 0.507 0.501 0.361 0.376 0.401 0.393
Table 3:
Cross-domain setting of forecasting the future O time points based on the past 336 time points. All results are averaged from different choices of O E {96, 192, 336, 720]. lower MSE or MAE indicates a better prediction. Full results in Appendix
Models
SimMTM Ti-MAE 21
TST 56
LaST 42
TF-C 57
CoST 46
TS2Vec
55
Metric
MSE
MAE MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTH2 + ETTh1
0.415 0.430 0.466 0.456 0.469 0.459 0.443 0.471 0.635 0.634 0.428 0.433 0.517 0.486
ETTml ETThl
0.422 0.430 0.495 0.469 0.475 0.463 0.426 0.441 0.700 0.702 0.620 0.541 0.484 0.482
ETTm2 - ETThl
0.428 0.441 0.464 0.456 0.453 0.450 0.503 0.507 1.091 0.814 0.598 0.548 0.616 0.550
Weather - BTTh1
0.456 0.467 0.462 0.464 0.465 0.456
0.518 0.487 0.463 0.460
ETTh1 ETTml
0.346 0.384 0.360 0.390 0.373 0.393 0.353 0.390 0.746 0.652 0.370 0.393 0.699 0.557
ETTh2 ETTml
0.365
0.384 0.383 0.402 0.391 0.409 0.475 0.489 0.750 0.654 0.363 0.387 0.694 0.557
FTTm2 - E
0.351
0.383 0.390
0.410 0.382 0.402 0.414 0.464 0.758 0.699 0.385 0.412 0.423 0.420
Weather + ETTml
0.350 0.383 0.411 0.423 0.368 0.392
0.382 0.403 0.382 0.395
Avg
0.392 0.413 0.429 0.434 0.422 0.428 0.436 0.460 0.780 0.693 0.458 0.451 0.535 0.488
14.7% MSE reduction and 12.0% MAE reduction compared to the contrastive baseline CoST 46 also notable that both Ti-MAE (2] and TST [56) outperform all the contrastive- baased baselines. This indicates that masked modeling based point-wise reconstruction suits the forecasting task better than the series-wise contrastive pre-training.
Cross-domain. As demonstrated in Table we present multiple scenarios to verify the effectiveness under the cross-domain setting, where SimMTM consistently outperforms other baselines. Note that the channel-inden -eneeee-indeeendeent encoder enables the comparing baselines to be capable of transferring pre-trained models between datasets with different variate numbers: Weather + (ETThI, ETTm] J. But for LaST and TF-C with model-specific designs, they cannot be applied to these scenarios. While negative migration has been observed some cross-domain scenarios, such Weather ETTh] and ETTh2 ETTml, SimMTM is still significantly overall superior to other baselines.
7

---

Table 4: In- and cross cross-domain settings of classification. For the in-domain setting, we pre-train and fine-tune the model on the same dataset Epilepsy. For the cross-domain setting, we pre-train a model on SleepÉEG and fine-tune it to multiple target datasets: Epilepsy, FD-B, Gesture, EMG. Accuracy (%) score are recorded. Full results are included Appendix
Models
SimMTM Random init. Ti-MAE
TST
LaST
TF-C
57
CoST
46
Tec
55
Epilepsy - Epilepsy
94.75
89.83
80.34
80.89
92.11
93.96
92.35
92.33
SleepEEGG Epilepsy
95.49
89.83
73.45
82.89
86.46
94.95
93.66
94.46
SleepEEG FD-B
69.40
47.36
67.98
65.57
46.67
69.38
54.82
60.74
SleepEEG - Gesture
80.00
42.19
75.54
75.12
64.17
76.42
73.33
73.33
SleepEEG -> EMG
97.56
77.80
63.52
75.89
66.34
81.74
73.17
80.92
Avg
87.44
69.40
72.17
76.07
71.15
83.29
77.47
80.36
In-Domain - ETThl to ETThl
Cross-Domain ETTh2 to ETThl
In Domaiinn- Epilepsy to Epilepsy
Cross- Domain SleepEEG to EMG
041
Classficatian (Accumcc)
Clazsfication (Accumsy)
Forecaztnn (MS5
6:1
Forecazen (MSE)
54.7
6.03
6:47
8434
DUST
6.41
SA4D
Figure 4: Ablations of SimMTM on the reconstruction loss (Crec) and constraint loss (Ceum) in time series forecasting (left part) and classification (right part) tasks under both in-domain and cross-domain settings. More ablations are included in Appendix
4.3 Classification
In-domain. We investigate the in-domain pre-training effect on the classification tasks Table Note that different from forecasting. the classification task requires the model learn high-level time series representations. From Table 4 we can find that the contrastive pre-training baselines achieve competitive performances. In contrast, the masking-based model Ti-MAE (21 and TST (56) perform poorly, and TST even exhibits negative transfer phenomenon compared to the random initialization, indicating that contrastive learning is generally more suitable for classification tasks. Surprisingly, although SimMTM follows the masked modeling paradigm, with our reconstruction task, can still achieve the best performance in the classification task. This is benefited from the neighborhood aggregation from multiple masked series, which enables the model to exploit the local structure of the time series manifold.
Cross-domain. We experiment with four cross- coms-d transfer scenarios in Table SleepEEG - (Epilepsy, FD-B, Gesture, EMG), where the target datasets are distinct from the pre-training dataset. Due to the large gap between pre-training and fine-tuning datasets, the baselines perform poorly in most cases, while SimMTM still surpasses other baselines and the random initialization significantly. Especially for SleepEEG + EMG, SimMTM remarkably surpasses previous state-of-the-as TF-C (Accuracy: 97.56% vs. 81.74%). These results demonstrate that SimMTM can precisely capture valuable knowledge from pre-training datasets and uniformly benefit extensive downstream datasets.
4.4 Model Analysis
Ablations. As shown in Figure 4 we provide ablations to two parts of the training loss SimMTM. It is observed that both Erecontruction and Leonstrait are essential to the final performance. Especially, for SleepEEG - EMG experiment, SimMTM surpasses the random initialization remarkably, where reconstruction and constraint losses provide 9.7% and 16.0% absolute improvement respectively. Besides, we can also find that in comparison to frecomstruction Lconsraint provides more contributions to the final results. This comes from design that the constraint loss uncovers proper time series

---

Table 5: Representation analysis for different methods in classification and forecasting tasks. For each model, we calculate the Centered Kernel Alignment (CKA) similarity (%) [88] between repre- sentations from the first and the last encoder layers to measure the representation-leaming property of deep models. Since the bottom layer representations usually contain low-level or detailed informa- tion, a smaller CKA similarity means the top layer includes different information from the bottom layer and indicates the model tends to learn high-level representations or more abstract information. For comparison, also calculate the AAKA between pre-trained and fine-ture tee-tuned models, where smaller value indicates a smaller representation gap between pre-training and fine-tuning, and the representations have stronger universality and portability.
Tasks
Ti-MAE 21
TST [56]
LaST 42
TF-C [
CoST [46]
TS2vec [
SimMTM
Pre-training
84.12
54.98
82.01
85.78
60.74
70.01
33.87
Classification
Fine-tuning
87.26
55,80
79.56
86.30
62.24
69.79
32.84
CCK
3.14
0.82
2.45
1.53
1.50
0.22
1.04
Pre-training 83.60
99.76
75.20
59.35
87.09
70.20
97.79
Forecasting
Fine-tuning
91.24
94.92
79.25
60.60
77.38
83.73
97.89
ACKAl
7.64
4.84
4.05
1.25
9.70
13.53
0.11
Sum| ACkA|
10.78
5.66
6.50
2.77
11.20
13.75
1.15
Table Performance by applying SimMTM to four advanced time series forecasting models under the in-domain setting. "+ Sub-series Masking" refers to the sub-series masked modeling pre-training method proposed by PatchTST (26) itself. MSE and MAE are averaged from all prediction lengths in (9,,,92.33)..20) The detailed results for each forecasting horizon are in Appendix
Dataset
ETTh1
ETTh2
ETTml
ETTm2
Model
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Transformer
39
1.088
0,836
4.103
1.612
0.901
0.704
1.624
0,901
+ SimMTM
0.927
0.761
3,498
1.487
0.809
0.663
1.322
0.808
Autoformer 47
0.573
0.573
0.550
0.559
0.615
0.528
0.324
0.368
+ -SmMTMM
0.561
0.568
0.543
0.555
0.553
0.505
0.315
0.360
NS Transformer
2
0.570
0.537
0.526
0.516
0.481
0.456
0.306
0.347
+ SimMTM
0.543
0.527
0.493
0.514
0.431
0.455
0,301
0,345
PatchTST (26]
0.417
0,431
0.331
0.379
0.352
0,382
0.258
0,317
Sub-series Masking
0.430]
0,445
0.355
0.3941
0.341
0.379
0.258
0.3181
+ SimMTM
0.409
0,428
0.329
0.379
0.348
0.378
0.254
0.313
manifold helpful for reconstruction from multiple masked series, without which the neighborhood aggregation will degenerate to the trivial average.
Representation analysis. To illustrate the advantages of SimMTM intuitively, provide representation analysis in T where we can find the following observations. Firstly, we can find that the CKA value of M in the classification task is clearly smaller than the values in the forecasting task, where the former is a high-level task and the latter requires low-level representations. These results demonstrate that SimMTM can learn adaptive representations for different tasks, which can be benefited from our design in the pre-training loss. Concretely, the temporal variations of classification pre-training datasets are much more diverse than the forecasting datasets. Thus, the Cconstraint will be easier for optimization in classification, deriving smaller CKA value. Secondly, from |.CKA|. is observed that the models pre- preitrained from SimMTM present a smaller gap with respect to the final fine-tuned model in representation learning properties, which is why SimMTM can consistently improve downstream tasks.
Model generality. From Table 6 we can find that as general time series pre-training framework, SimMTM can consistently improve the forecasting performance of diverse advanced base models, even for the state-of-the-art time series forecasting model PatchTST (26). This generality also

---

133N6c
TRCC
TST
75%
0.440
0.428
0.422
0.422
TIMAA
C:ST
0.70
Masked Ratio (r)
0.65
Foreazting
0.60
50%
0.429
0.416
0.409
0.419
0.425
0.417
0.430
0.45
0.40
Data Proportion (%)
Numbers of Masked Series (M)
Figure Model analysis. Left part for fine-tuning ETTh2 pre-trained model ETTh1 with limited data, where a smaller MSE indicates better performance. Right part presents the MSE performance of SimMTM in the ETTh1 "input-336-pmt 333--reddcc-96 in-domain setting with different masked ratio and numbers of masked series M, where darker red means better performance.
indicates that can further improve the model's performance by employing advanced base models as encoders. also notable that different from the negative transfer phenomenon caused by the canonical sub-series masked modeling used in the PatchTST paper [26], the consistency improvement of SimMTM further verifies the effectiveness of our design.
Fine-tuning to limited data scenarios. One essential application of pre-training models is to provide prior knowledge for downstream tasks, especially for limited data scenarios, which is critical the fast-adaption of deep models. Thus, to verify the effectiveness of SimMTM and other pre- training methods in data-limited scenarios, we pre-train model on ETTh2 and fine-tune it to ETTh1 with different choices for the remaining proportions of training data. All results are presented in Figure 5 We can find that SimMTM achieves significant performance gains in different data proportions compared to other time series pre-training methods. Specifically, for the 10% data fine- tuning setting, SimMTM significantly outperforms the advanced masking based method Ti-MAE [2]) (MSE: 0.591 vs. 0.660). Compared with the contrastive-but -ased method TF-C [57] SimMTM also achieves 26.0% MSE reduction. These results further verify that SimMTM can effectively capture valuable knowledge from datasets and boost the final performance, even in limited data scenarios.
Masking strategy. Note that the difficulty of reconstructing the original time series increases along with the increase of the masked ratio, but decreases when the number of neighbor masked series increases. We explore the potential relationship between the masked ratio and the number of masked series used for reconstruction, namely r and M in Eq. [ respectively. The experimental results in obtain better results. Experimentally,
Figure show show masking ratio as 50% and adopt three masked series for reconstruction throughout this paper. In addtion,
that we need to set M
choose the
find that only using worse than the settings with larger M, where the latter enables the model to discover the relations between input series and its neighbors. See Figure for intuitive understanding. These results further highlight the advantage of our design in manifold learning.
masked series (M = 1) for pre-training generally performs
Conclusion
This paper presents SimMTM, simple pre-training framework for masked time-series modeling, Going beyond the previous convention in reconstructing the original time series from unmasked time points, SimMTM proposes a new masked modeling task as reconstructing the original series from its multiple neighbor masked series. Concretely, SimMTM aggregates the point-wise representations based on the series-war wise similarities, which are carefully constrained by the neighborhood assumption on the time series manifold. Experimentally, SimMTM can furthest bridge the gap between pre- trained and fine-tuned models, thereby achieving consistent state-of-the-art in distinct forecasting and classification tasks compared to the most advanced time series pre-training methods, covering both in-domain and cross-domain settings. the future, we will further extend SimMTM to large-scale and diverse pre- pre-training datasets in pursuing the foundation model for time series analysis.
10

---

Acknowledgments
This work was supported by the National Key Research and Development Plan (2021 FFB17152000) National Natural Science Foundation of China (62022050 and 62021002), and Beijing Nova Program (Z201 1000066000)))
References
[1] Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian Elger. Indications of nonlinear deterministic and inite-dimensional structures in time series of brain electrical activity: Dependencc on recording region and brain state. Physical Review E, 2001.
[2] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. JCML, 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. In NeurlPS, 2020.
[4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
[5] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In /CML, 2020.
[6] Mingyue Cheng, Qi Liu, Zhiding Liu, Hao Zhang, Rujiao Zhang, and Enhong Chen, Timemae: Self- supervised representations of time series with decoupled masked autoencoders. 7-KDE, 2023.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, 2018.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In I/CAI, 2021.
[9] Yean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In NeurlPS, 2019.
[10] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In IJCNLP, 2020.
[11] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[12] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[13] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[14] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh. Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning. Technologies, 2020.
[15] Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transfcrability in deep learning: A survey. arXiv preprint XXix:2220.00866 2022.
[16] Bob Kemp, Aeilko II Zwinderman, Bert Tuk. Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of sleep-dependent neuronal feedback loop: the slow-wave microcontinuity the eeg. TBME, 2000.
[17] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty weigh losses for scene geometry and semantics. In CVPR, 2018.
[18] Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity neural network representations revisited. In JCML, 2019.
[19] Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in clectromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM, 2016.
11

---

[20] Yanghao Li. Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In CVPR. 2023.
[21] Zhe Li, Zhongwen Rao, Lujia Pan. Pengyun Wang. and Zenglin Xu. Ti-mae: Self-supervised masked time series autoencoders. arXiv preprint arXiv:2301.066 08871, 2023.
[22] Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. uwave: Accelerometer-based personalized gesture recognition and its applications. In PMC, 2009.
[23] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. TKDE, 2021.
[24] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the Stationarity in time series forecasting, In NewrIPS, 2022.
[25] Abdullah Al Mueen and Eamonn J. Keogh. Extracting optimal performance from dynamic time warping. KDD, 2016.
[26] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.
[27] Manuel T Nonnenmacher, Lukas Oldenburg, Ingo Steinwart, and David Reeb. Utilizing expert features for contrastive learning of time-series representations. In ICML, 2022.
[28] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.
Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[29] PeMS. Traffic Dataset. httt:://pmma dot. ca..ov/
[30] Toolio PhysioBank. Physionet: components of new research resource for complex physiologic signals. Circulation, 2000.
[31] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer lcarning with a unified text-to-text transformer. JMLR, 2020.
[33] Quentin Rebjock, Baris Kurt, Tim Januschowski, and Laurent Callot. Online false discovery rate control for anomaly detection in time series. In NeurlPS, 2021.
[34] Pritam Sarkar and Ali Etemad. Self-supervised learning for ocg-based emotion recognition. In ICASSP. 2020.
[35] Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.
[36] Fan-Keng Sun, Chris Lang, and Duane Boning. Adjusting for autocorrelated errors in neural networks for time series. In NewrIPS, 2021.
[37] Chi Ian Tang, Ignacio Sarez-Pozuelo, Dimitris Spathis, and Cecilia Mascolo. Exploring contrastive learning in human activity recognition for healthcare. In NeurlPS, 2020.
[38] UCL. UCI Electricity Load Time Series Dataset. Tlectricity.oaaaiiggrram2211111.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrlPS, 2017.
[40] Pascal Vincent, H. Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. JMLR, 2010.
[41] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In ICML, 2020.
12

---

[42] Zhiyuan Wang, Xovee Xu. Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. In NewrIPS, 2022.
[43]
Wetterstation. Weather Dataset. https::/www.bge. btt. jena mpg. de/uetter
[44] Alexander Weltig, Tianyu Gao. Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? In EACL, 2023.
[45] Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen, Mixing up contrastive learning: Seff-superviied representation learning for time series. PRI,, 2022.
[46] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In ICLR, 2022.
[47] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-corrclation for long-term series forecasting. In NeurIPS, 2021.
[48] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
[49] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In CVPR, 2023.
[50] Zhenda Xie, Zheng Zhang. Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
[51] Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long, Anomaly transformer: Time series anomaly detection with association discrepancy. In ICLR, 2021.
[52] Ling Yang and Shenda Hong. Unsupervised time-serics representation learning with iterative bilinear temporal-spectral fusion. In ICML, 2022.
[53] Xinyu Yang, Zhenguo Zhang, and Rongyi Cui. Timecl:: A self-supervised contrastive learning framework for univariate time series representation. KBS, 2022.
[54] Hugo Yèehe, Gideon Dresdner, Francesco Locatello, Matthias Hüser, and Gunnar Rätsch. Neighborhood contrastive learning applied to online patient monitoring. In ICML, 2021.
[55] Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards Universal Representation of Time Series. In AAAI, 2022.
[56] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-basee framework for multivariate time series representation learning. In SIGKDD, 2021.
[57] Xiang Zhang. Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitikk. Self-supervised contrastive pre-training for time series via time frequenyy consistency. In NeurIPS, 2022.
[58] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
13

---

A Implementation Details
All the experiments repeated five times, implemented in PyTorch 28 and conducted NVIDIA A A SXM4 40GB GPU. implement the baselines based on their official implementation and follow the configuration from their original papers. For the metrics, we adopt the mean square error (MSE) and mean absolute emorr (MAE) for the time series forecasting, As for the classification, accuracy, precision. recall, Fl score, and their average value recorded.
A.1 Dataset Description
We conduct experiments to evaluate the effect of our method under in-domain and cross-domain settings on twelve real-world datasets for two typical time series analysis tasks: forecasting and classification, covering diverse application scenarios (electricity system, neurological healthcare, human activity recognition, mechanical fault delection, and physical status monitoring). different types of signals (ECG, EMG, acceleration, vibration, power load, weather, and transoirtation), multivariate channel dimensions (from to 862), varying times series lengths (from 96 to 5120) and large span sampling ratio (from 100 Hz to 4000 Hz). The detailed descriptions of these datasets summarized in Table
Table 7: Dataset descriptions. Samples are organized in
Tasks
Datasets
Channels
Length
Samples
Classes
Information
Frequency
ETTHLETT12
(96.192.336.720)
8545/2881/2881
Electricity
Hour
Forecasting
ETTmL.ETIm
7
(96.192.336.720)
34465/11521/11522
Electricity
15 Mins
Weather
19,,192,336,720 36792/5271/10540
Weather
Mins
Electricity
321
(96,192.336,720)
18317/2633/5261
Electricity
Hour
Traffic
862
(96.192.336.720
12185/1757/3509
Transportation
Hour
SlecpEEG
371005/-/-
EEG
100 Hz
Classification
Epilepsy
1
178
60/20/1 1420
2
EEG
174 Hz
FD-B
1
5120
60/21/135599
Faulty Detection
64K Hz
Gesture
3
315
320/120/120
Hand Movement
100 Hz
EMG
1500
122/41/41
Muscle responses
4K Hz
(1) ETT (4 subsets)
contains time series oil temperature and power load collected by electricity transformers from July 2016 to July 2018. ETT is a group of four subsets with different recorded frequencies: ETTh1/ETTh2 are recorded every hour, and ETTmm/ETTmn are recorded every 15 minutes.
58
(2) WEATHER 4g includes meteorological time series with 21 weather indicators collected every 10 minutes from the Weather Station of the Max Planck Biogeochemistry Institute in 2020.
(3) ELECTRICITY 38 records the hourly electricity consumption of 321 clients from 2012 2014. Values are in kW of cach 15 min. time labels report Portuguese hour. However, all days present 96 measures x4). For every year in March, time change day (which has only 23 hours), values between 1:00 am and 2:00 am are zero for all points. For every year in October, time change day (which has 25 hours), the values between 1:00 am and 2:00 am aggregated consumption of two hours.
(4) TRAFFIC 22] encompasses the hourly measures of road occupancy rates obtained from 862 sensors situated in the San Francisco Bay area freeways. These measurements were carried out between January 2015 and December 2016.
(5) SLEEPEEG [6]] contains whole- whole-night sleeping clectroencephaalggaapy (EEG) recordings from 82 healthy subjects. We follow the same data preprocessing approach as (57) and get 371,055 univariate brainwaves. Each brainwave is sampled at frequency of 100 Hz and associated with one of five sleeping stages: Wake, Non-rapid eye movement (3 sub-states), and Rapid Eye Movement.
14

---

(6) EPILEPSY [1 monitors the brain activities of 500 subjects with single-channel EEG sensor. Every subject recorded for 23.6 seconds of brain activities. The dataset is sampled at 178 Hz and contains 11,500 samples in four classes (eyes open, eyes closed, EEG measured the healthy brain region, and EEG measured the tumor region) of the original five categories each sample are classified as positive, and the remaining classes are used as negative.
total. We follow procedure described by [57]. The
(7) FD-B [19 is generated by clectromechanical drive systems. It monitors the condition of rolling bearings and detects their failures based on the monitoring conditions, which include speed, load torque, and radial force. Concretely, FD-B has 13,640 samples in total. Each recording is sampled 64k Hz with 3-class labels: undamaged, inner damaged, and outer damaged.
(8) GESTURE [2 are collected from hand gestures based on the paths hand movement recorded by an accelerometer. The eight gestures are hand swiping left, right, up, and down, hand waving in a counterclockwise or clockwise circle, hand waving in square, and waving right arrow, respectively. This dataset contains 440 examples balanced classification labels that be used, and each sample includes eight different kinds of gesture categories.
(9) EMG [30 sampled with 4K Hz and consists of 163 single-channel EMG recordings from the anterior tih- ialis musele of three healthy volunteers suffering from ncuropathy and myopathy. Each patient is a classification category, so each sample is associated with one of threc classes.
A.2 Baselines Implementation
We compare SimMTM against six state-of-the-art baselines. To make fair and comprehensive comparison, tried two baseline implementation approaches for forecasting and classification tasks: the unified encoder and reproduced with their official implementation encoder. Notably, LaST 42 and TF-C [57] are closely related to model structures. We directly report results from their papers or reproduce codes with official implementation.
Table 8: Baselines implementation details.
Baselines
Task
Encoder
Performance Comparison
Report
Channel-indepemddnt Transformer
better
Main text
Forecasting
Official implementation
Section
Ti-MAE 21
ID-ResNet
better
Main text
Classification
Official implementation
Section E
Channel-independdet Transformer
better
Main text
Forecasting
Official implementation
Section E
TST 56
ID- ID-ResNet
better
Main text
Classification
Official implementation
Section E
Forecasting
Official implementation
Main text
LaST 4
Classification
Official implementation
Main text
Forecasting
Official implementation
Main
TF-C 57
Classification
Official implementation
Main text
Channel-inde Transformer
better
Main text
Forecasting
Official implementation
Section E
CoST
ID-ResNet
better
Main text
Classification
Official implementation
Scction
Channel-indeeeendet Transformer
better
Main text
Forecasting
Official implementation
Section F
TS2Vec (55)
ID-ResNet
better
Main text
Classification
Official implementation
Section F
(1) Unified encoder. We attempt to unify the encoder for these pre-training methods. Specifically, we adopt the
vanilla Transformer [39] with channel independent (26 for forecasting accomplish cross doss-n transfer
between datasets with different variate numbers. As for the classification, use ID-ResNet [13 the
15

---

encoder following [57]. Besides, we do comprehensive hyperparameter search for all baselines. For the Transformer encoder, vary the number of Transformer layers in (1, 4), select the model dimension from {16, 32, 64, 128, 256). and the attention head from (4,8, 16, 32). For the ID-ResNet, we search the number of ID-ResNet layers from { [1, 2, 3, 4), the kernel size from {3, 5, 8) respectively. Additionally, for the masked modeling methods TST [56]. Ti-MAE 221, we also searched the masked ratio = {0.125, 0.25, 0.5, 0.75 for better performance.
(2) Official implementation. We also implement the baselines following the corresponding official codes, including encoder, hyperparameters, etc. The comparisons included in Section [ of this supplementary material. We directly report the results from their original papers for same set, For mismatched settings, the results are from our implementation.
Finally, for baselines Ti-MAE (2T). TST [36. CoST 46. and TS2 Vec [55] we report the results based on the unified encoder in the main text. But for baselines LaST [42] and TF-C [57]. we report the results of the official code implementation their original paper, which are limited by their model structures. As result, the performances of all baselines with unified encoder (that we reported in the main text) generally surpass their
official implementation and results reported in their own paper. TableBssoows more details. Full experimental results are in Scction H
A.3 Pre-training and Fine-tuning Configuration
We built two types of pre-training and fine-tuning scenarios, in-domain and cross-domain, based on the bench- marks of forecasting and classification tasks to compare the effectiveness of our method and other time series pre-training methods.
We pre-train model on subset for forecasting tasks and fine-tune it to the same dataset to build seven in-domain transfer evaluation scenarios. In cross-domali evaluation, we pre-train a model on one specific dataset and use other datasets for tine-tuning. Based on the above settings, we constructed fifteen in- and cross doosann pre-training and fine-tuning experiments, covering the same dataset with the same sampled frequency, different datasets with the same sampled frequency, and different datasets with different sampled frequencies.
We use the same dataset, Epilepsy, to construct the in-domain setting for classification tasks. For the cross- domain setting, we pre-train model for classification tasks on univariate time series dataset SleepEEG with the most complex temporal dynamics and the most samples. And then fine-tune model separately on Epilepsy. FD-B, Gesture, and EMG. Furthermore, we constructed four cross-domain evaluation scenarios by pre- pre-training from SIcepEEG and fine-tuning to Epilepsy, FD-B, Gesture, and EMG because of fewer commonalities and the enormous gap among these datasets. Table 9 shows detailed pre-training and fine-tuning settings.
Table 9: Pre-training and fine-tuning scenarios for time series forecasting (Fore.) and classification (Class.) tasks, including the same and different datasets and in- and cross-domain settings.
Tasks
Evaluation
Scenarios
Characteristic
ETTh1 ETTHl
ETTh2 ETTh2
ETTml FTTml
In-domain
ETTm2 -> ETTm2
The same dataset with the same frequency
Weather - Weather
Forc.
Electricity - Electricity
Traffic - Traffic
ETTh2 > ETThl
Different datuscts with the same frequency.
ETTm2 >> ETTml
Cross-domain
(ETTml, ETTm2, Weather) - ETThl
Different datasets with different frequencies.
(ETTh), ETTh2, Weather) -> ETTml
In-domain
Epilepsy >> Epilepsy
The same dataset with the same frequency.
Class.
Cross-domain
SleepEEG + (Fpileesyy FD-B, B, Gesture, EMG) Different datasets with different frequencies.
16

---

A.4 Model and Training Configuration
Following the previous convention, we choose the encoder part of Transformer [39] with channel independent as the feature extractor for forecasting tasks. For the classification tasks, we adopt ID-ResNet (13] as the encoder following (57). In the pre-training stages, pre-train the model with different learning rates and batch sizes according to the pre-train datasets. Then we fine-tune downstream forecasting and classification tasks supervised by L2 and Cross- Cooss-E losses, respectively. The configuration details are Table
Table 10: Model and training configuration in Forecasting (Fore.) and Classification (Class.) tasks.
Encoder
Pre-training
Fine-tuning
Tasks
Clayers dmodel
learning
batch size
epochs
learning
loss function
batch size
epachs
Fore. 16
1e-3
50
le-4
L2
[1633
10
Class, 128
le-4
128
10
le-4
Cross Cross-Entropy Entropy
32
300
B Hyperparameter Sensitivity
We verify the hyperparameter sensitivity of SimMTM on ETTht in Table 11 including masked ratio (r), the number of masked series (M), temperature (7), masked function (Mask), encoder depth (Clayers), ), and the hidden dimension (dmodda)). Lower MSE and MAE represent better performance.
As shown in Table II (a) and (b),
observe the effect of the method is closely related to the trade-off' of the masked ratio and the number masked series. Hence, a reasonable balance between the two kinds of parameters critical. For the temperature hyperparameter softmax normalization (T), we use an appropriately small r that lcads to higher differences and diversity of masked sequences. For the masked methods, we chose two masked methods for verification: masking following random distribution and masking following geometric distribution [56]. The results show that the method based geometric masking better than random masking modeling. Besides, we can find that 2 encoder layers are enough for reconstruction tasks. Note our method SimMTM consistently performs better than training from scratch under various hyperparameter changes.
Table 11: Hyperparameter sensitivity experiments on ETTh1 for the in-domain setting. The entries
marked in bold are the same which specify the default settings. This table format follows
(a) Masked ratio
(b) Masked numbers
Temperature
Ratio
MSE
MAE
Numbers
MSE
MAE
Value
MSE
MAE
12.5%
0.429
0.440
0.429
0.437
0.02
0.409
0.428
25%
0.427
0.434
0.416
0.429
0.2
0.409
0.429
50%
0.409
0.428
3
0.409
0.428
2
0.416
0.428
75%
0.422
0.434
4
0.419
0.431
(d) Masked function
(e) Encoder depth
(f) Hidden layer dimension
Type
MSE
MAE
Layers
MSE
MAE
Dim
MSE
MAE
Random
0.409
0.431
1
0.420
0.426
16
0.409
0.428
Geometric
0.409
0.428
2
0.409
0.428
32
0.420
0.432
3
0.421
0.430
64
0.422
0.434
0.426
0.436
128
0.428
0.444
17

---

C Ablations on Aggregation Setting
SimMTM proposes to recover masked time points by the weighted ageregation of multiple neighbors outside the manifold. We explored two types of aggregation settings.
(1) Positive Samples Aggregation (PSA): only aggregate multiple positive neighbors (the masked series of the same sample) reconstruct masked time points.
(2) Positive and Negative Samples Aggregation (PNSA): aggregate both positive and negative neighbors (the masked scries all samples) to reconstruct masked time points.
As shown in Table T2 although PSA made good progress compared to training from scratch (Random Init.), PNSA is consistenily better than SimMTM PSA in all ablation settings. In masked time-series modeling, masking can be viewed as adding noise to the original data, and masked modeling is to project masked data from the neighborhood back the original manifold. We positive and negative masked time series the reconstruction candidates drive the model select the positive samples adaptively, which can make the model learn the structure of the manifold better. Therefore, as stated in the Method Section of the main text, we choose positive and negative sample aggregation (PNSA) as the standard aggregation setting of SimMTM.
Table 12: Ablations on aggregation setting in forecasting (MSE) and classification (Acc) tasks for in- and cross-domain settings. A smaller MSE higher Accuracy indicates better result (f).
Tasks
Evaluation
Scenarios
Aggregation
Metric
Random init.
0.431
In-domain
ETTh ETThl
SimMTM (PSA)
0.420 f
SimMTM (PNSA)
0.409
Forecasting
Random init.
0.431
Cross-domain
ETTh2 ETThl
SimMTM (PSA)
0.426 f
SimMTM (PNSA)
0.415
Random init.
89.83
In-domain
Epilepsy + Epilepsy
SimMTM (PSA)
92.56
SimMTM (PNSA)
94.75
Classification
Random init.
77.80
Cross-domain
SleepEEG + EMG
SimMTM (PSA)
87.80
SimMTM (PNSA)
97.56
D Comparison of Masked Modeling
To investigate the reconstruction process of different masked modeling methods, we plot both original and reconstructed time series from TST and SimMTM in Figure where TST (56) follows the canonical masked modeling paradigm and learns predict removed time points based the remaining time points. In Figure 6 find that direct reconstruction is too difficult time series, 75% masking ratio, TST degenerates more seriously. Because of this poor reconstruction effect, direct reconstruction is difficult to provide reliable guidance to model pre- preitrainnn. In contrast, our proposed SimMTM can precisely reconstruct the original time series, benefiting the representation learning. These results also support design in neighborhood reconstruction.
for the 12.5% masking ratio. As for
18

---

TST Masked 12.5%
TST
Masked 25%5
TST
Masked 50%
TST Masked 75%
Origial Time
Origiaal
-0.-
-124
-18-
SmMMTM (Ours) Masked 12.5%
SmMTIM (Ours) Masked 25%
SIMMTM (Curs)
Masked 50%
SIMMTM (Ours) Masked 75%
OriginaTinna Serie
-10-
-10x
-14
Figure 6: Comparison of the canonical masked modeling paradigm TST and neighborhood aggrega- tion masked modeling SimMTM in reconstructing time series. All the cases are shown from ETThI.
E
Full Results
Due to limited length of text, we summarize experiments the main text into two parts: the main experiment and the analytical experiment. We categorize and index them in Tabel 13 14
Table 13: The main results of pre-training and fine-tuning scenarios for time series forecasting and classification tasks, including the same and different encoder for in- and cross-domain settings.
Tasks
Evaluation
Encoder
Tabels Name
The model utilized in the original papers
Table [5
In-domain
Forecasting
Transformer with channel independent
Table 16
The model utilized in the original papers
Table 17
Cross-domain
Transformer with channel independent
Table 18
The model utilized in the original papers
Table 27
In-domain
Classification
ID-ResNet
Table 24
The model utilized in the original papers
Table 23
Cross-domain
1D-ResNet
Table 24
Table 14: The model analysis results of pre-training and fine-tuning scenarios for time series forecasting and classification tasks with the unified encoder for in- and cross-domain settings.
Tasks
Evaluation
Analysis
Tabels Name
Ablation study
Table 19
In-domain
Forecasting
Model generality
Table 21
Ablation study
Table 20
Cross-domain
Limited data
Table 22
In-domain
Ablation study
Table 25
Classification
Cross-domain
Ablation study
Table 25
19

---

Limitations
SimMTM is inspired by the manifold perspective of masked modeling, Although we have provided relatively comprehensive results verify the model's effectivvness, the model performance still needs theoretical guaran- toes. In fact, the high-impact works in the self-supervised pre-training community are without theoretical analysis, such as BERT 7 GPT-3 (1O), MAE [1] and SimMIM 50 Thus, we would like to leave this problem as a future work.
The masking ratio of masked modeling methods is an essential hyper-parameter. Although we have provided a chosen principle to masking ratio and the number of masked time series M as M x in the main text, we still need tune these hyperparameters for different datasets achieve the best performance. Notably, previous methods also chose the masking ratio solely based the empirical results [ Thus, despite there exist limitations of SimMTM in choosing hyperparameters, the principle of M x can somewhat case this problem. And the chosen strategy of the masking ratio can also be potential topic in masked modeling [44].
G Social Impacts
This paper presents SimMTM as a new masked modeling method for time series. SimMTM achieves state-of- the-art in two mainstream time series analysis tasks, which can be good supplement for the self-supervised pre-training community. We will also publish the codebase of time-series pre-training to facilitate future research.
This paper only focuses the algorithm design. Using all the codes and datascts strictly follows the correspond- ing licenses (Appendix A.I. There is no potential ethical risk or negative social impact.
20

---

Table 15: Complete results of long-term forecasting tasks for the in-domain setting of forecasting the future O E (96, 192, 336, 720) time points based on the past 336 time points. All the results of baselines are based on the encoder utilized in their original papers. The standard deviations of SimMTM are within 0.005 for MSE and within 0.004 for MAE.
Models
SimMTM Random init. Ti-MAE (21
TST [56
LaST [42
TF-C S7
CoST
[46]
Vec
55
Metric
MSE
MAE MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96 0.379 0,407 0.380 0.412 0.708
0.570
0.503 0.527 0.399 0.412 0,463 0.406 0.514 0.512 0.709 0,650
192
0.412 0.424 0.416 0.434
0.725
0.587
0.601 0.552 0,484 0.468 0.531 0.540 0.655
0.590
0.927 0.757
336
0.421
0,431
0.448
0.458
0.713
0,589
0.625
0,541 0,580
0.533
0,535
0.545 0,790
0.666
0.986 0.811
720
0.424
0.449 0.481
0.487
0.736
0.618
0.768
0.628 0.432
0.432
0.577
0.562 0.880
0.739
0.967 0.790
Avg
0.409
0.428 0.431
0,448
0.721
0.591
0.624
0.562 0.474
0.461 0.527
0.513
0.710
0.627
0.897 0.752
96
0.293
0.347 0.325 0.374
0.443
0.465
0.335
0.392 0.331 0.390
0.463 0.521 0.465
0.482
0.506
0.477
ETTH 112
192
0.355
0.386
0.400
0.424
0,533
0.516
0.444
0.441 0.451
0.452
0,525 0.561
0.671
0.599
0.567
0.547
336
0.370
0.401 0.405
0,433
0,445
0.472
0.455
0.494
0,460
0.478
0,850 0.883
0.848
0,776
0.694
0,628
720
0.395
0.427 0.451
0.475
0.507
0.498
0.481
0.504
0.552 0.509 0.930 0.932
0.871
0.811
0.728
0.838
Avg
0.353
0.390
0.395
0.427
0.482
0.488
0.429
0.458
0.449
0.457
0.692
0.724
0.714
0.667
0.624
0.623
96
0.288
0.348 0.295 0.346
0,647
0.497
0.454
0.456
0,316
0.355
0.419
0.401
0.376
0,420
0.563
0,551
mml
192
0.327
0.373 0.333 0.374
0.597
0,508
0.471
0.490
0.349
0.366
0,471
0.438
0.420
0.451
0.599
0.558
336
0.363
0.395
0.370
0.398
0.699
0.525
0.457
0.451
0,429
0.407
0,540
0.509
0.482
0.494
0.685
0.594
720
0.412
0.424
0.427
0.431
0.786
0.596
0.594
0.488
0.496
0.464
0,552 0.548
0.628
0.578
0.831
0.698
Avg
0.348
0.385
0.356
0.387
0.682
0,532
0.494
0.471
0.398
0.398
0,496
0.474
0.477
0,486
0.670
0.600
96
0..12
0.261 0.175 0.268
0.304
0.357
0,363
0.301 0.163
0.255
0,401 0.477 0.276
0,384
0.448
0,482
ETTm2
192
0,223
0.300
0.240
0.312 0.334
0.387
0.342
0.364
0,239
0.303
0.422 0.490 0.500
0.532
0.545
0.536
336
0.282
0.331
0.298
0.351
0.420
0.441
0.414
0.361
0.259
0.366
0,513 0.508 0.680
0.695
0.681
0.744
720
0.374
0.388 0.403
0.413
0.508
0.481
0.580
0.456
0.397
0.382 0.523 0.772 0.925
0.914
0.691
0.837
Avg
0,263
0.320 0.279
0.336
0.392
0.417
0.425
0.371 0.265
0.327
0.465 0.562 0.595
0.631
0.591
0.650
96
0.158
0.211 0.166 0,216
0.216
0,280
0,292
0.370 0.153 0.211 0.215 0.296
0.327
0.359
0.433
0.462
Weather
192
0.199
0.249 0.208 0,254
0.303
0.335
0.410
0.473 0.207 0,250
0.267 0.345 0.390
0,422
0.508
0,518
336
0.246
0.286
0.257
0.290
0.351
0.358
0.434
0.427 0,249
0.264
0.299
0.360
0.477
0.446
0.545
0.549
720
0.317
0.337 0.326 0.338
0,425
0.399
0.539 0.523 0.319 0.320 0.361 0.395
0.551
0.586
0.576
0.572
Avg
0,230
0.271 0.239
0.275
0.324
0.343
0.419
0.448 0.232 0.261
0.286
0.349
0.436
0.453
0.516 0.525
96
0.133
0,223 0.190 0.279
0.399
0.412
0.292 0.370 0.166 0.254 0.366 0.436 0.230
0.353
0.322
0.401
Electricity
192
0.147
0.237 0.195 0.285 0,400
0.460
0.270
0.373 0.178 0.278 0,366 0.433 0.253
0.371
0.343
0.416
336
0.166
0.265
0.211 0.301
0,564
0.573
0.334
0.323 0.186 0.275 0,358
0.428
0.197
0.287
0.362
0.435
720
0.203
0.297
0.253 0,333 0.880
0.770
0,344
0.346 0.213 0.288 0.363 0.431 0.230
0,328
0.388
0.456
Avg
0.162
0.256 0.212 0,300
0.561
0.554
0,310
0.353 0.186 0.274 0.363 0.432
0.228
0.335
0.354
0.427
96
0.368
0.262 0.471 0.309
0.431
0.482
0.559
0.454 0.706 0.385 0.613 0.340
0.751
0.431
0.321
0.367
Traffic
192
0.373
0.251
0.475 0,308 0.491
0,346
0,583
0.493 0.709
0.388 0.619 0.516 0.751
0.424
0.476
0.367
336
0.395
0,254
0.490
0.315
0,502
0,384
0.637
0.469 0.714 0.394 0.785
0.497 0.761
0,425
0.499
0.376
720
0.432
0.290
0.524
0.332
0.533
0.543
0.663
0.594 0.723 0.421 0.850 0.472 0.780
0.433
0.563
0.390
Avg
0.392
0.264
0.490
0.316
0.489
0.399
0.611
0.503
0.713
0.397
0.717 0.456
0.761
0,428
0.501
0.375
21

---

Table 16: Complete results of long-term forecasting tasks for the in-domain setting of forecasting the future E {96, 192, 336, 720} time points based on the past 336 time points. All the results of baseline are based on the unified
Transformer encoder. The standard deviations of SimMTM are within 0.005 for MSE and within 0.004 for MAE.
Models
SimMTM Random init. Ti-MAE [2
TST [56]
LaST 42
TF-C [57
CoST [46
VS2
SS
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE MAE
MSE
MAE
MSE
MAE
96
0.379 0.407 0.380 0.412
0.356
0.420
0.401
0.425
0.422
0.436
0.392
0.420
E17H ETTh)
192
0.412
0.424 0.416 0.434
0.421
0.434
0.427
0.432
0.520
0.487
0.445
0.452
336
0.421
0.431
0.448
0.458
0.447
0.446
0.519
0.487
0.472
0.462
0.453
0.455
720
0.424
0.449 0.481
0.487
0.469
0.482
0.515
0.504
0.525
0.501
0.495
0.496
Avg
0.449
0.428
0.431
0.448
0.423
0.446
0.466 0.462
0.485
0.472
U.446
0,456
96
0,293
0.347 0.325
0.374
0.339
0.378
0.322 0.358
0.321
0.374
0.365
0.509
192
0.355
0.386 0,400
0.424
0.380
0.402
0.448 0.435
0.380
0.403
0.396
0.422
336
0,370
0.401
0,405
0.433
0.388
0.323
0.420 0.440
0,430
0,451
0.399
0.436
720
0.395
0.427 0.451
0.475
0.414
0.442
0.424 0.452
0.466
0,480
0.508
0.503
Avg
0.353
0.390 0.395
0.427
0.380
0.386
0.404 0.421
0.399
0.427
0.417
0.468
96
0,288
0.348 0.295
0.346
0.305
0.351
0.310
0.348
0.291
0.343
0.681
0.689
192
0.327
0.373 0.333
0.374
0.343
0.374
0.362 0.380
0.330
0.370
0.689
0.551
336
0.363
0.395
0.370
0.398
0.387
0.407
0.389 0.402
0.382
0.401
0.704
0.559
720
0.412
0.424
0.427
0.431
0.428
0.432
0.433 0.427
0.422
0,425
0.721
0.571
Avg
00.488
0.385
0.356
0.387
0.366
0.391
0.373 0.389
0.356 0,385
0.699
0.557
96
0.172
0.261 0.175
0,268
0.174
0.258
0.215 0.296
0.242 0.333
0.224
0.303
ET1m2
192
0.223
0.300
0.240
0.312
0.257
0.303
0.259 0.323
0.283 0.345 0.273
0.331
336
0.282
0.331
0.298
0.351
0.277
0.333
0.319 0.364
0.303
0.349
0.399
0.402
720
0.374
0.388 0.403
0.413
0.360
0.404
0.395 0.405
0.431 0.431
0.406
0.408
Avg
(0.263
0.320 0.279
0.336
0.267
0.325
0.297 0.347
0.314 0.365
0.326
0.361
96
0.158
0.211 0.166 0.216
0.153
0.196
0.162 0.214
0,216 0,280
0.154
0.205
Weather
192
0,199
0.249 0.208
0.254
0.214
0.253
0.203 0.252
0.303
0.335
0,200
0.243
336
0.246
0.286
0.257 0.290
0.243
0.272
0.260
0.297
0.351
0.358
0.252
0.286
720
0.317
0.337 0.326
0.338
0.324
0.349
0.330 0.342
0.425
0.343
0.324
0.335
Avg
(0.230
0.271 0.239
0.275
0.234
0.265
0.239 0.276
0.324 0.329
0.233
0.267
96
0,133
0.223 0.190
0.279
0.163
0.255
0.186 0.268
0.197
0.277
0.195
0.275
Electricity
192
0.147
0.237 0.195 0.285
0.194
0.288
0.193 0.276
0.197
0.279
0.195
0.277
336
0.166
0.265
0.211
0.301
0.201
0.298
0.206 0.289
0.211
0.295
0.210
0.294
720
0.203
0.297 0.253
0.333
0.263
0.343
0.250 0.324
0.255
0,330
0.252
0.327
Avg
0.162
0.256 0.212
0,300
0,205
0.296
0.209 0.289
0.215
0.295
0.213
0.293
96
0.368
0.262 0.471 0.309
0.448
0.298
0.595
0.360
0.378
0.365
0.480
0.357
Tralicc
192
0.373
0.251
0.475
0,308
0.445
0,301
0.576 0.353
0,371
0.352
0.439
0.336
336
0.395
0.254
0,490
0.315
0.492
0.320
0.569
0.362
0.467
0,354
0.460
0.344
720
0.432
0.290
0.524
0.332
0.514
0.321
0.603
0.372
0.525
0.378
0.499
0.364
Avg
(0.392
0.264
0.490
0.316
0.475
0.310
0.586 0.362
0.435 0,362
0.470
0.350
22

---

Table 17: Complete results of long- forecasting tasks for the cross-domain setting of forecasting the future O E {96, 192, 336, 720] time points based on the past 336 time points. All the results of baselines are based on the encoder utilized in their original papers. The standard deviations of SiMMTM are within 0.005 for MSE and within 0.004 for MAE
Models
SinMTM
Random init.
T-MAE
21
TST
56
LaST
42
TF-C
57
CoST
46
TS2Vec
SS
Metric
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.372
0.401
0.380
0.412
0.703
0,562
0.653
0.468
0.362
0.420
0.596
0.569
0.378
0.421
0.849
0.694
ETTh2
192
0.414
0.425
0.416
0.434
0.715
0.567
0.658
0.502
0.426
0.478
0.614
0.621
0.424
0.451
0.909
0.738
336
0.429
0.436
0.448
0,458
0.733
0.579
0.631
0.561
0.522
0.509
0.694
0.664
0.651
0.582
1.082
0.775
ETTh!
720
0.446
0.458
0.481
0.487
0,762
0.622
0.638
0.608
0,460
0.478
0.635
0.683
0.883
0.701
0.934
0.769
Avgl
0.415
0.430
0.431
0.448
0.728
0.583
0.645
0.535
0.443
0.471
0.635
0.634
0.584
0.539
0.944
0.744
0.367
0.398
0.380
0.412
0.715
0.581
0.627
0.477
0.360
0.374
0.666
0.647
0.423
0.450
0.991
0.765
ETTml
192
0.396
0.421
0.416
0.434
0.729
0.587
0.628
0.500
0.381
0.371
0.672
0.653
0.641
0.578
0.829
0.699
336
0.471
0.437
0.448
0.458
0.712
0,583
0.683
0.554
0.472
0.531
0.626
0.711
0.863
0.694
0.971
0.787
FTTh
720
0.454
0.463
0.481
0.487
0.747
0.627
0.642
0.600
0.490
0.488
0.835
0.797
1.071
0.805
1.037
0.820
Avel
0.422
0.430
0.431
0.448
0.726
0.595
0.645
0.533
0,426
0.441
0.700
0.702
0.750
0.632
0.957
0.768
96
0,388
0.421
0.380
0.412
0.699
0.566
0.559
0.489
0.428
0.454
0.968
0.738
0.377
0.419
0.783
0.669
ETTm2
192
0.419
0.423
0.416
0.434
0.722
0.573
0.600
0.579
0.427
0.497
1,080
0.801
0.422
0,450
0.828
0.691
336
0.435
0.444
0.448
0.458
0.714
0.569
0.677
0.572
0.528
0.540
1.091
0.824
0.648
0.990
0.762
E
720
0.468
0.474
0.481
0.487
0.760
0.611
0.694
0.664
0.527
0.537
1.226
0.893
0.880
0.699
0.985
0.783
Avg
0.428
0.441
0.431
0.448
0.724
0.580
0.632
0.576
0.503
0.507
1.091
0.814
0.582
0.537
0.896
0.726
96
0.477
0.444
0.380
0.412
Weather
192
0.454
0.522
0.416
0.434
336
0.424
0.434
0.448
0.458
ETThl
720
0.468
0.469
0.481
0.487
Avg
0.456
0.467
0.431
0.448
96
0.290
0.348
0.295
0.346
0.667
0.521
0.425
0.381
0.295
0.387
0.672
0.600
0.248
0.332
0.605
0.561
ETTbl
192
0.327
0.372
0.333
0,374
0.561
0.479
0.495
0.478
0.335
0.379
0.721
0.639
0.336
0.391
0.615
0.561
336
0.357
0,392
0.370
0.398
0.690
0.533
0,456
0.441
0.379
0.363
0.755
0.664
0.381
0.421
0,763
0.677
ETTml
720
0.409
0.423
0.427
0.431
0.744
0.583
0.554
0.477
0.403
0.431
0.837
0.705
0.469
0,482
0.805
0.664
Avgl
0.346
0.384
0.356
0.387
0.666
0.529
0.482
0.444
0.353
0.390
0.746
0.652
0.359
0.407
0.697
0.616
96
0.322
0,347
0.295
0,346
0.658
0.505
0.449
0.343
0,314
0.396
0.677
0.603
0.253
0,342
0.466
0.480
ETTh2
192
0.332
0.372
0.333
0.374
0.594
0.511
0.407
0.587
0.545
0.718
0.638
0.367
0.392
0.557
0.532
336
0.394
0.370
0.398
0.732
0.532
0.407
0.519
0.631
0.584
0.755
0.663
0.388
0.646
ETTml
720
0.411
0.424
0.427
0.431
0.768
0.592
0.557
0.523
0,368
0.429
0.848
0.712
0.498
0,488
0.752
0.638
Ave|
0.365
0.384
0.356
0.387
0.688
0.535
0.472
0.448
0.475
0.489
0.750
0.654
0.377
0.413
0.606
0.556
0.297
0.348
0.295
0.346
0.647
0.497
0.471
0.422
0.304
0.388
0.610
0.577
0.239
0.331
0.586
0.515
ETTm2
192
0.332
0.370
0.333
0.374
0.597
0.508
0.495
0.442
0.429
0.494
0.725
0.657
0.339
0.371
0.624
0.562
336
0.364
0.393
0.370
0.398
0.700
0.525
0.455
0.424
0.499
0.523
0.768
0.684
0.371
0.421
1.035
0.806
ETTml
720
0.410
0.421
0.427
0.431
0.786
0.596
0.498
0.532
0.422
0.450
0.927
0.759
0.467
0.481
0.780
0.669
Avg|
0.351
0.383
0.356
0,387
0.682
0,531
0.480
0.455
0.414
0.464
0.758
0.669
0.354
0.401
0.756
0.638
96
0.304
0.354
0.295
0.346
Weather
192
0.338
0,375
0.333
0.374
336
0.371
0.397
0.370
0.398
ETTml
720
0.417
0.426
0.427
0.431
Avg
0.358
0.388
0.356
0.387
23

---

Table 18: Complete results long-term forecasting tasks for the in-domain setting. All the results of baseline based the unified channel-independee Transformer encoder. The past sequence length is set as 336. The unified channel -indeeeddd transformer model can perform the transfer experiment between datasets with different variables. The standard deviations of SimMTM are within 0.005 for MSE and within 0.004 for MAE.
Models
SimMTM Random init, Ti-MAE (1
TST
LaST
TF-C
CoST 46
Vcc
SS
Metric
MSE
MAE MSE MAE MSE
MAE
MSE
MAE MSE
MAE MSE MAE
MSE
MAE
MSE
MAE
96 0.372 0.401 0,380 0.412 0.399
0.424
0.401 0.425
0.376 0.362 U.436 0.430
ET
192
0.414 0,425 0.416 0.434
0,454
0.440
0.531 0,484
0.376 0.362 0.455 0.440
336
0.429 0.436 0.448
0.458
0.497
0.469
0.474 0.459
0.444 0.444 0.689 0.584
ETThl
720 0.446 0.458
0.481
0.487
0.515
0.492
0.471 0.469
0.517 0.510 0.489 0.490
Avg
0.415 0.430 0.431
0.448
0.466
0.456
0.469 0.459
0.428 0.433 0.517 0.486
96 0.367 0.398 0.380 0.412 0,400
0.418
0.443 0,440
0.465 0.456 0.413 0.443
ETTml
192
0.396 0.421 0.416 0.434 0.434
0.445
0.471 0.455
0.722 0.588 0.459 0.465
336 0.471 0.437 0.448 0.458
0,510
0.467
0.462 0.455
0.712 0.586 0.614 0.554
ETThl
720 0.454 0.463 0,481 0.487
0,636
0.544
0.525 0.503
0.581 0.533 0,450 0.464
Avg
0.422 0.430 0.431
0.448
0.495
0.469
0.475 0.463
0.620 0.541 0.484 0.482
96 0.388 0.421 0.380
0.412 0.433
0.431
0.389 0.413
0.403 0.426 0.483 0.480
ETTm2
192 0.419 0.423 0.416 0.434
0.474
0.458
0.463 0.452
0.457 0.468 0.579 0.537
336
0,435 0,444
0.448
0.458
0.515
0.448
0.492 0.465
0.794 0,682 0.673 0.563
ETThl
720 0.468 0.474
0.481
0.487
0.496
0.488
0.468 0.468
0.739 0.617 0.729 0.620
Avg 0.428 0.441 0.431
0.448
0.464
0.456
0.453 0.450
0.598 0.548 0.616 0.550
96 0.477 0.444
0,380
0.412
0.397
0.440
0.428 0.429
0.421 0,410 0.393 0.410
Weather
192 0.454 0.522 0.416
0.434
0.458
0.466
0.461
0.451
0.539 0.503 0.440 0.437
336 0.424 0.434 0.448
0.458
0.479
0.463 0.456
0.568 0.514 0.450 0.451
ETTh1
720 0.468 0.469
0.481
0.487
0.515
0.492
0.507 0.489
0,544 0,522 0.567 0.541
Avg 0.456 0.467 0.431
0.462
0.464
0.465 0.456
0.518 0.487 0.463 0.460
96 0.290 0.348 0.295 0.346
0.311
0.355
0.315 0.354
0.308 0.355 0.681 0.545
TTThl
192 0.327 0.372 0.333 0.374 0.337
0.372
0.365 0,391
0.357 0.390 0.689 0.551
336 0.357 0.392 0.370 0.398
0,372
0.398
0.384 0,400
0.396 0.402 0.705 0.560
ETTml
720 0.409 0.423 0.427 0.431 0,422
0.433
0.428 0.426
0.419 0.423 0.722 0.571
Avg 0.346 0.384 0.356
0.387
0.360
0.390
0.373 0.393
0.370 0.393 0.699 0.557
96 0.322 0.347 0.295 0.346 0.323
0.362
0.338 0,383
0.322 0.351 0.679 0.546
ETTh2
192 0.332 0.372 0.333 0.374
0.370
0.395
0.394 0.408
0.331 0.373 0.673 0.551
336 0.394 0.391 0.370
0.398
0.397
0.413
0.401 0.412
0.382 0.397 0.703 0.557
ETTml
720 0.411 0.424
0.427
0.431
0.442
0.439
0.434 0.432
0.417 0.428 0.722 0.573
Avg 0.365 0.384 0.356
0.387
0.383
0.402
0.391 0.409
0.363 0.387 0.694 0.557
96 0.297 0.348
0.295 0.346
0.333
0.378
0.327 0.364
0.320 0.364 0.422 0.434
ETTm2
192 0.332 0.370 0.333 0.374
0.381
0.398
0.362 0.389
0.367 0.386 0.387 0.371
336 0.364 0.393 0.370 0.398
0.394
0.413
0.401 0.418
0.374 0.394 0.402 0.444
ETTml
720 0.410 0.421 0.427 0.431 0.455
0.453
0.437 0.437
0.479 0.503 0.481 0.432
Avg 0.351 0.383 0.356
0.387 0.390
0.410
0.382 0.402
0.385 0.412 0.423 0.420
96 0.294 0.354
0.295 0.346
0.338
0.380
0.324 0,366
0.324 0.360 0.329 0.359
Weather
192 0.318 0.355 0.333 0.374 0.473
0.457
0.349 0,377
0,359 0.387 0.392 0,392
336 0.361 0.397
0.370
0.398
0.402
0.415
0.378 0.398
0.395 0.399 0.372 0.400
ETTml
720 0.427 0.426 0.427 0.431 0.432
0.438
0.422 0.427
0.450 0.467 0.434 0.429
Avg 0.350 0.383 0.356
0.387
0.411
0.423
0.368 0.392
0.382 0.403 0.382 0.395
24

---

Table 19: Full ablation studies for the in-domain setting of forecasting. The standard deviations of SimMTM are within 0.005 for MSE and within 0.004 for MAE.
Input-336
Supervised
Wio Ereannstmccion
W/o Cosnsraiit
SimMTM
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.380
0.412
0.377
0.408
0.381
0.409
0.379
0.407
ETThl
192
0.416
0.434
0.419
0.443
0.409
0.443
0.412
0.424
336
0.448
0.458
0.423
0.434
0.432
0.444
0.421
0.431
720
0.481
0.487
0.437
0.454
0.447
0.454
0.424
0.449
Avg
0,431
0.448
0.414
0.435
0.417
0.438
0,409
0.428
0.325
0.374
0,288
0.344
0.312
0.365
0.293
0.347
ETTh2
192
0.400
0.424
0.356
0.391
0.389
0.418
0.355
0.386
336
0.405
0.433
0.368
0.406
0,396
0.432
0.370
0,401
720
0,451
0.475
0.409
0,432
0.448
0.479
0,395
0.427
Avg
0.395
0.427
0.355
0.393
0.386
0.424
0.353
0.390
96
0.295
0.346
0.291
0.343
0.282
0.337
0.288
0.348
ETTml
192
0.333
0.374
0.330
0.390
0.324
0.388
0.327
0.373
336
0.370
0.398
0.369
0.399
0.366
0.397
0.363
0.395
720
0.427
0.431
0.417
0.429
0.424
0.435
0,412
0.424
Avg
0,356
0.387
0.352
0.390
0.349
0,389
0.348
0.385
96
0.175
0.268
0.174
0.265
0.170
0.261
0.172
0.261
ETTm2
192
0.240
0.312
0.232
0.303
0.244
0.320
0.223
0.300
336
0.298
0.351
0.313
0.365
0.279
0.334
0.282
0.331
720
0,403
0.413
0.376
0.451
0.376
0.378
0.374
0.388
Avg
0.279
0.336
0.274
0.346
0.267
0.323
0.263
0.320
96
0.166
0.216
0.164
0.209
0.160
0.212
0.158
0.211
Weather
192
0.208
0.254
0.203
0.258
0.203
0.251
0.199
0.249
336
0.257
0.290
0.244
0.289
0.253
0.290
0.246
0.286
720
0.326
0.338
0.322
0.343
0.325
0.340
0.317
0.337
Avg
0.239
0.275
0,233
0.275
0.235
0.273
0.230
0.271
96
0.190
0.279
0.177
0.270
0.134
0.220
0.133
0.223
Electricity
192
0.195
0.28S
0.184
0.279
0.163
0.274
0.147
0.237
336
0.211
0.301
0.202
0.300
0.223
0.311
0.166
0.265
720
0.253
0.333
0.250
0.337
0.241
0.321
0.203
0.297
Avg
0.212
0.300
0.203
0.397
0.190
0.282
0.162
0.256
96
0.471
0.309
0.366
0.257
0.457
0.301
0.368
0.262
Traffic
192
0.475
0.308
0.373
0.266
0.468
0.325
0.373
0.251
336
0.490
0.315
0.401
0.249
0.487
0.302
0.395
0.254
720
0.524
0.332
0.472
0.312
0.485
0.315
0.432
0.290
Avg
0.490
0.316
0.403
0.271
0.474
0.311
0.392
0.264
25

---

Table 20: Full ablation studies on transfer ETThl and ETTml for the cross-domain setting of
forecasting. The standard deviations of SimMTM are within 0.005 for MSE and within 0.004 for
MAE.
Input-336
Supervised
W/o Lesconstuction
Wo Leonstraise
SimMTM
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.380
0.412
0.377
0.400
0.402
0.411
0.372
0.401
ET
0.416
0.434
0.417
0.424
0.417
0.420
0.414
0.425
336
0.448
0.458
0.437
0.439
0.437
0.435
0.429
0.436
ETThl
720
0.481
0.487
0.448
0.463
0.456
0.467
0.446
0.458
Avg
0.431
0.448
0.420
0.432
0.423
0.430
0.415
0.430
96
0.380
0.412
0.382
0.397
0.375
0.399
0.367
0.398
ETTml
192
0.416
0.434
0.418
0,418
0.413
0.422
0.396
0.421
336
0.448
0.458
0.437
0.434
0.434
0.438
0,471
0.437
ETThl
720
0.481
0.487
0.459
0.469
0.467
0.475
0.454
0.463
Avg
0.431
0.448
0.424
0.430
0.422
0.434
0.422
0.430
96
0.380
0,412
0.388
0.418
0.384
0.415
0.388
0.421
ETTm2
192
0.416
0.434
0.429
0.444
0.423
0.439
0.419
0.423
336
0.448
0,458
0.467
0.472
0.458
0.465
0.435
0.444
ETThi
720
0.481
0.487
0.521
0.507
0.501
0.497
0.468
0.474
Avg
0.431
0.448
0.451
0.460
0.441
0.454
0.428
0.441
96
0,380
0,412
0.385
0.400
0.394
0.406
0.477
0.444
Weather
192
0.416
0.434
0.417
0.429
0.425
0.424
0.454
0.522
336
0.448
0.458
0.434
0.434
0.441
0.439
0.424
0.434
ETTh1
720
0,481
0.487
0.444
0.464
0.446
0.468
0.468
0.469
Avg
0.431
0.448
0.420
0.432
0.427
0.434
0.456
0.467
96
0.295
0.346
0,286
0.341
0.290
0.346
0.290
0.348
ETTh1
192
0.333
0.374
0.322
0.362
0.353
0.388
0.327
0.372
0.370
0.398
0.362
0.418
0.362
0,412
0.357
0.392
ETTml
720
0.427
0.431
0.417
0.431
0.422
0.432
0.409
0.423
Avg
0,356
0.387
0.347
0,388
0.357
0.395
0.346
0.384
0.295
0.346
0.299
0.348
0.301
0.352
0.322
0.347
ETTh2
0,333
0.374
0.324
0.366
0.332
0.359
0.332
0.372
336
0.370
0.398
0.374
0.401
0.389
0,382
0.394
0.391
ETTml
720
0.427
0.431
0.415
0.419
0.421
0.442
0.411
0.424
Avg
0.356
0,387
0.353
0,386
0,361
0.384
0.365
0,384
96
0.295
0.346
0.299
0.351
0.285
0.336
0.297
0.348
ETTm2
192
0.333
0.374
0.334
0.372
0.343
0.366
0.332
0.370
336
0.370
0.398
0.362
0.388
0.360
0.399
0.364
0.393
ETTml
720
0.427
0.431
0.417
0.431
0.422
0,432
0.410
0.421
Avg
0.356
0.387
0.353
0.386
0.353
0,383
0.351
0,383
96
0.295
0.346
0.322
0.361
0.309
0.354
0.294
0.354
Weather
192
0,333
0.374
0.344
0.378
0.343
0.365
0.318
0.355
336
0,370
0.398
0.371
0.399
0.401
0.411
0.361
0.397
ETTml
720
0.427
0.431
0.426
0.422
0.425
0.427
0.427
0.426
Avg
0.356
0.387
0.366
0.390
0.370
0.389
0.350
0.383
26

---

Table 21: Full results for applying SimMTM to four advanced time series forecasting models under the in-domain setting. The gray mark represents negative transfer (4).
Transfomer
39
Autoformer
+7
Ns Trunsformer
24
PatcTTST
26
Models
RRndomm init.
SimMIM Random init. +SmMTMM Random init. +SimMTM Random init. +Sub-seric Masking +SimMTM
Metric
MSE
MAE MSE MAE MSE MAE MSE
MAE MSE MAE MSE MAE
MSE MAE
MSE
MAE
MSE
MAE
96
0.847 0.731 0.775 0.691 0.536 0.548 0.526 0.536 0.513 0.491 0,490 0,489 0.375 0.399
0.366
0.398
0.373 0.399
192
1.084
0.841 0.918 0.763 0.543 0.551 0.523 0.548 0.534
0.504
0.517 0.499
0.414 0.421
0.431
0.443
0.406 0.428
336
1,350
0.956
1,079 0.845 0.615 0,592 0,595 0.591 0.588
0.535
0,552
0.520
0.431 0.436
0,450,
0.456]
0,422 0.431
720
1.069
0.817
0.935
0,761 0.599 0.600
0.600
0.597 0.643
0,616
0.614
0.598
0.449
0.466
0.472.
0.484]
0.436 0.452
Avg
1.088
0.836
0.927
0.761 0.573 0.573 0.561 0.568
0.570
0.537 0.543
0.527 0.417 0.431
0.430.
0.445L
0.409 0.428
96
2.029
1.150
1.879
1.104
0.492 0.517 0.488
0.514 0.476 0.458
0.445
0.448
0.274 0.336
0.284.
0,343)
0.274 0.337
192
6.785
2.099
5.054
1,771
0.556 0.551 0.547 0.549 0.512 0.493 0.482 0.502 0.339 0.379 0.355,
0,387L
0.339
0.377
336
4.568
1,711
4.242
1.658
0.572 0,578
0.563
0.570
0.552 0.551
0,512
0.537 0.331
0,380 0.379L
0.4111
0.327
0.381
720
3,030
1.466
2.815
1.413
0.580
0.588
0.575
0.588 0.562 0.560
0.531 0.568
0.379
0.422
0.400.
0.435L
0.375 0.423
Avg
4.103
1.612
3.498
1.487
0.550
0.559
0.543
0.555
0.526
0.516
0.493
0.514
0.331
0.379
0.355
0.394J
0.329
0.379
96
0.562
0.520
0.513
0.497 0.523 0,488
0.482
0.465
0.386
0.398
0,340
0.376
0.290
0.342
0.289.
0,3441
0,288
0.343
192
0.810
0.668
0,686 0.606 0.543 0.498 0.499
0.476
0.459
0.444
0.445
0.332 0.369
0.323
0.368
0.329
0.367
336
1.096
0.814
1.003
0.760
0.675
0.551
0.601
0.524
0.495
0.464
0.423
0.459
0.366
0.392
0.353
0.387
0.361
0.387
720
1.136
0.813
1 1.032 0,790 0.720 0.528
0.629
0.555
0.585
0,516
0.539
0.499
0.420
0.424
0.398
0,416
0,413
0.417
Avg
0..901
0.704 0.809 0.663 0.615
0.528
0.553
0.505
0.481
0,456
0.431
0.445
0.352
0,382
0.341
0.379
0.348 0.378
96
0.508
0.539
0.336
0.425 0.255 0.339
0.255
0.340
0.192
0.274
0.188
0.277
0.165
0.255
0.166.
0.256]
0.163
0.253
E17mm
192
0.972
0.721
0.713
0.610 0.281 0.340 0,276 0.332
0.280 0.339
0,277
0.336
0.220
0.292
0.221
0.295]
0.219 0.292
336
1.419
0.897
1.517
0,942 0.339 0.372 0.309 0.359 0.334
0.361 0,325
0.355
0.278
0.329
0.278
0,333
0.275 0.328
720
3.598
1.445
2.720
1,254 0.422
0.419
0.420
0.410
0.417
0.413
0.414
0.412
0.367
0.385
0.365
0.388]
0.359
0.381
Avg|
1.624
0.901
1.322
0,808
0.324
0.368
0.315 0.360
0.306
0.347
0.301
0.345
0.258
0.317
0.258
0.318)
0.254 0.313
Table 22: Full results for fine-tuning to limited data scenarios. We fine-tune the model pre-trained
from ETTh2 to ETThl with different data proportions {10%, 25% 50%, 75%, 100%}.
Models
SimMTM Random init. Ti-MAF 21
TST 56
L.ST 42
TF-C 57
CoST(46)
TS2Vec
55
Metric
MSE MAE
MSE
MAE
MSE
MAE
MSE MAE
MSE
MAE MSE MAE MSE
MAE MAE MSE
10%
0.591 0.523 0.653 0.558
0.660 0.517 0.783 0.588 0.645 0.507 0.799 0.783 0.784 0.604 0.655 0.550
ETTh2
25%
0.535
0.490 0.632 0.502
0.594
0.518
0.641 0.578 0.610 0.611 0.736 0.725 0.624 0.539 0.632 0.543
50%
0.491
0.473 0,512 0.479
0.550
0.504
0.525 0.509 0.540 0,513 0.731 0.704 0.540 0.499 0.599 0.526
ETThl
75%
0.466
0.458
0..99
0.488
0.475
0.465
0.516 0.488 0.479 0.470 0.697 0.689 0.494 0.475 0.577 0.534
100%
0.415
0.430
0.431
0.448
0.466
0.456
0.469
0.459 0.443 0.471 0.635 0.634 0.428 0.433 0.517 0.486
27

---

Table 23: In- and cross domain settings of classification, where all the baselines are based the encoder utilized their original papers. For in-domain setting, we pre-train and fine-tune on the same dataset: Epilepsy. For cross cross-domain setting, pre-train the model on SleepEEG and then fine-tune on different datasets: Epilepsy, FD-B, Gesture, and EMG.
Scenarios
Models
Accuracy (%)
Precision (%)
Recall (%)
F1 (%)
Avg (%)
Random init.
89.83
92.13
74.47
79.59
84.00
In-Domain
TS2vec
55
92.17
93.84
81.19
85.71
88.23
Epilepsy
CoST
88.07
91.58
66.05
78.70
LaST 42
92.11
93.12
81.47
85.74
88.
Epilepsy
TST [66
80.21
40.11
50.00
44.51
53.71
Ti-MAE
90.09
93.90
77.24
78.21
84.86
TF-C 57
93.96
94.87
85.82
89.46
91.03
SimMTM
94.75
95.60
89.93
91.41
92.92
Random init.
89.83
92.13
74.47
79.59
84.00
TS2vec 55
93.95
90.59
90.39
90.45
91.35
SleepEEG
CoST 46
88.40
88.20
72.34
76.88
81.45
LaST 42
86.46
90.77
66.35
70.67
78.56
Epilepsy
TST 56
80.21
40.11
50.00
53.71
Ti-MAE 21
89.71
72.36
67.47
68.55
74.52
TF-C 57
94.95
94.56
89.08
91.49
92.52
SimMTM
95.49
93.36
92.28
92.81
93.49
Random init.
47.36
48.29
52.35
49.11
49.28
TS2vec
47.90
43.39
48.42
43.89
45.90
SleepEEG
CoST 46
47.06
38.79
38.42
34.79
39.76
LaST 42
46.67
43.90
47.71
45.17
45.86
FD-B
TST 56
46.40
41.58
45.50
41.34
43.71
Ti-MAE
60.88
66.98
68.94
66.56
65.84
Cross- Domman
TF-C 57
69.38
75.59
72.02
74.87
72.97
SimMTM
69.40
74.18
76.41
75.11
73.78
Random init.
42.19
47.51
49.63
48.86
47.05
TS2vec 55
69.17
65.45
68.54
65.70
67.22
SleepEEG
CoST 46
68.33
65.30
68.33
66.42
67.09
LaST 42
64.17
70.36
64.17
58.76
64.37
Gesture
TST 56
69.17
66.60
69.17
66.01
67.74
Ti-MAE 21
71.88
70.35
76.75
68.37
71.84
TF-C 57
76.42
77.31
74.29
75.72
75.94
SimMTM
80.00
79.03
80.00
78.67
79.43
Random init.
77.80
59.09
66.67
62.38
66.49
TS2vec 551
78.54
80.40
67.85
67.66
73.61
SleepEEG + EMG
CoST 46
53.65
49.07
42.10
35.27
45.02
LaST 42
66.34
79.34
63.33
72.55
70.39
TST [66]
78.34
77.11
80.30
68.89
76.16
Ti-MAE
69.99
70.25
63.44
70.89
68.64
TF-C 57
81.71
72.65
81.59
76.83
78.20
SimMTM
97.56
98.33
98.04
98.14
98.02
28

---

Table 24: In- and cross-domain settings of classification based on the unified 1-D ResNet encoder. For in-domain setting, we pre-train and fine-tune on the same dataset: Epilepsy. For cross-domain setting, pre-train SleepEERG and fine-tune different domain datasets: Epilepsy, FD-B, Gesture, and EMG.
Scenarios
Models
Accuracy (%)
Precision (%)
Recall (%)
Fl (%)
Avg (%)
Random init.
89.83
92.13
74.47
79.59
84.00
in-Domain
TS2vec 55
92.33
94.53
81.11
86.33
88.58
Epilepsy
CoST 46
92.35
94.73
81.16
85.92
88.54
LaST 42
Epilepsy
TST 56
80.89
90.38
51.73
48.01
67.75
Ti-MAE
80.34
90.16
50.33
45.20
66.51
TF-C 57
93.96
94.87
85.82
89.46
91.03
SimMTM
94.75
95.60
89.93
91.41
92.92
Random init.
89.83
92.13
74.47
79.59
84.00
TS2vec 55
94.46
91.99
90.28
91.10
91.95
SleepEEG
CoST 46
93.66
91.39
88.08
89.60
90.68
LaST 42
Epilepsy
TST [66]
82.89
86.15
79.02
80.44
82.13
Ti-MAE
73.45
72.56
65.34
77.20
72.14 14
TF-C S7
94.95
94.56
89.08
91.49
92.52
SimMTM
95.49
93.36
92.28
92.81
93.49
Random init.
47.36
48.29
52.35
49.11
49.28
TS2vee
60.74
59.60
64.27
61.07
61.42
SleepEEG
CoST
54.82
51.92
63.30
54,34
56.09
LaST 42
FD-B
TST 56
65.57
70.05
67.57
64.41
66.90
Ti-MAE 21
67.98
62.83
64.45
63.36
64.66
Cross- Domain
TF-C 57
69.38
75.59
72.02
74.87
72.97
SimMTM
69.40
74.18
76.41
75.11
73.78
Random Init.
42.19
47.51
49.63
48.86
47.05
TS2vec 5S
73.33
70.88
73.33
71.56
72.27
SleepEEG
CoST 46
73.33
74.37
73.33
71.16
73.04
LaST 42
Gesture
TST 56
75.
76.05
67.74
73.24
73.04
Ti-MAE 21
75.54
69.32
72.42
69.32
71.65
TF-C 57
76.42
77.31
74.29
75.72
75.94
SimMTM
80.00
79.03
80.00
78.67
79.43
Random init.
77.80
59.09
66.67
62.38
66.49
TS2vec (55
80.92
69.63
67.65
67.90
71.52
SIeepEEG
CoST 46
73.17
70.47
69.84
70.00
70.87
LaST 42
EMG
TST 56
75.89
74.67
80.66
78.48
77.43
Ti-MAE
63.52
67.77
70.55
58.32
65.04
TF-C S7
81.71
72.65
81.59
76.83
78.20
SimMTM
97.56
98.33
98.04
98.14
98.02
29

---

Table 25: Full ablation studies for in-domain and cross-domain settings of classification. Under the Avg metric, the standard deviations of SimMTM are within 0.2% for Epilepsy, within 0.5% for FD-B, within 0.6% for Gesture, and within 0.1% for EMG.
Scenarios
Accuracy (%)
Precision (%)
Recall (%)
Fl (%)
Avg (%)
Random init.
89.83
92.13
74.47
79.59
84.00
Epilepsy + Epilepsy
W/o Crecomsttutioon
93.80
96.11
86.11
89.45
91.37
W/o Leonstraint
90.99
92.81
79.86
84.13
86.95
SimMTM
94.75
95.60
89.93
91.41
92.92
Random init.
89.83
92.13
74.47
79.59
84.00
SleepEEG + Epilepsy
W/o Levemmtruction
94.54
93.87
88.46
90.84
91.93
W/o Leanetaint
91.73
90.57
82.21
85.53
87.51
SimMTM
95.49
93.36
92.28
92.81
93.49
Random init.
47.36
48.29
52.35
49.11
49.28
SleepEEG + FD-B
W/o Lreconetructoon
66.11
67.97
74.70
70.01
69.70 70
W/o Leanstriint
53.71
69.48
62.67
50.86
59.18
SimMTM
69.40
74.18
76.41
75.11
73.78
Random init.
42.19
47.51
49.63
48.86
47.05
SleepEEG + Gesture
W/o Fraconstruction
78.50
79.01
78.50
77.17
78.30
Wlo Leonstrint
76.67
74.91
76.67
74.80
75.76
SimMTM
80.00
79.03
80.00
78.67
79.43
Random init.
77.80
59.09
66.67
62.38
66.49
SleepEEG 4 EMG
Wlo freconstuction
90.24
94.20
78.04
81.53
86.00
W/o Leonstraint
85.37
89.97
69.62
70.74
78.93
SimMTM
97.56
98.33
98.04
98.14
98.02
30


Reviews:
[{"review_id": "lgEI9vtKIJ", "reviewer": "Reviewer_xbHA", "summary": "This paper proposes a simple pre-training framework for masked time-series modeling. Instead of reconstructing origin data directly, which is unsuitable for time series, this paper recovers masked time points by the weighted aggregation of multiple neighbors outside the manifold. The reviewer appreciates the novelty method proposed, but is still concerned about some results.", "questions": "In terms of experimental results, why Random init. outperforms TF-C in Table 2? Does SimMTM use a stronger baseline? Is it an unfair comparison? Maybe TF-C or other methods will outperform SimMTM with the same baseline. It seems that the model learns directly aggregate information from different masked series, which learns more temporal information than other semantic information.\n", "limitations": "The authors should discuss more implications about using large-scale pre-training in forecasting in terms of fairness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1) The method is of novelty. The multi-level aggregation and representation method is appealing.\n2) Experimentally, both high-/low-level tasks benefit from this method.", "weaknesses": "1. The experimental results need more justification (see Questions). \n2. The relationship with manifold learning needs more description. What is the progress in this aspect?\n3. The masked parts in Figure 1 and Figure 1 in Supplement material should be annotated and more results should be listed. A well-trained network usually predicts low-frequency results. Why the result of TST in the right-upper part in Figure 1 in Supplement material contains too many high-frequency parts?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank Reviewer xbHA for providing an insightful review.\n\n#### **About the Weaknesses**\n\n**Q1**: The relationship with manifold learning needs more description. What is the progress in this aspect?\n\nMasked modeling is a mainstream paradigm in self-supervised pre-training. However, the temporal information of the time series is significantly distorted as the masked ratio increases, making the reconstruction task too difficult to guide representation learning. SimMTM firstly introduces the core idea of neighborhood aggregation from manifold learning into self-supervised time series representation learning by reconstructing the original time series using multiple neighborhood masked series.\n\nNumerous experimental results further demonstrate the importance of introducing the idea of neighborhood aggregation from manifold learning into self-supervised representation learning of time series.\n- SimMTM achieves 8.9% average MSE reduction (0.335→0.305) and 4.6% average MAE reduction (0.445→0.329) compared to the advanced masked modeling baseline Ti-MAE (2023) in in-domain forecasting benchmarks ($\\underline{\\text{Table 3 in the main text}}$), which reconstructs the original time series based on standard masked modeling.\n- The averaged accuracy of SimMTM remarkably surpasses previous state-of-the-art TF-C (2022) (87.44% vs. 83.29%), as shown in $\\underline{\\text{Table 4 in the main text}}$.\n\n**Q2**: Why the results of TST contains too many high-frequency parts?\n\nMasking will make the input series temporally irregular, bringing noise to the reconstruction, thereby leading to high-frequency results. Following your suggestion, we have included more showcases in $\\underline{\\text{Figure 1 in the global response PDF}}$.\n\nSimMTM reconstructs the original time series by aggregating multiple neighboring masked series. The information from multiple neighboring masked time series is aggregated reasonably and complements each other. Multiple neighboring aggregating brings a more stable reconstruction effect, which is more beneficial for the representation learning of time series.\n\n#### **About the Questions**\n\n**Q1**: Why Random init. outperforms TF-C?\n\nWe have mentioned the particularity of TF-C and LaST in $\\underline{\\text{lines 207-209 in the main text}}$. TF-C is closely related to its model structure. Thus we do not unify the Encoder of TF-C in the original submission.\n\nIn addition, TF-C primarily focuses on time series classification tasks, not forecasting tasks. Specifically, TF-C, which is based on time and frequency series-wise representation contrastive learning, is more suitable for series-wise classification tasks than point-wise forecasting tasks.\n\nTo further validate, we attempted to replace the dual-tower TF-C (CNN) with TF-C (Transformer). We show the averaged MSE/MAE from 4 different forecasting lengths {96,192,336,720} based on the past 336 time points. Results showed that TF-C (Transformer) outperformed TF-C (CNN) in forecasting tasks but still significantly lagged behind SimMTM. Note that even if we change each part of it the same as SimMTM (Transformer), its unique dual-tower design cannot be strictly consistent with SimMTM.\n\n- Forecasting tasks in the in-domain setting\n    |Average MSE/MAE|TF-C (CNN)|TF-C (Transformer)|SimMTM\n    |:-:|:-:|:-:|:-:|\n    |ETTh1|0.637/0.638|0.492/0.481|0.404/0.428\n    |ETTh2|0.398/0.398|0.401/0.437|0.348/0.391\n    |ETTm1|0.744/0.652|0.466/0.445|0.340/0.379\n    |ETTm2|1.755/0.947|0.295/0.341|0.260/0.318\n\n**Q2**: Does SimMTM use a stronger baseline? Is it an unfair comparison?\n\nEnsuring complete consistency in the backbone used by all baselines is often unrealizable due to the high dependence between model architecture and the pre-training method in this domain. Existing work also commonly faces this problem [1,2,3].\n\nWe have made a great effort to ensure a fair comparison. For all baselines and all tasks, we have conducted both experiments: using a unified backbone and the specific backbone proposed in their original papers. The comprehensive experiments and descriptions have been presented in $\\underline{\\text{lines 203-211 in the main text}}$, $\\underline{\\text{Table 7, 8 in Supplementary Materials}}$.\n\n[1] Yue, et al., TS2Vec: towards universal representation of time series. AAAI, 2022.  \n[2] Zhang, et al., Self-supervised contrastive pre-training for time series via time-frequency consistency. NeurIPS, 2022.  \n[3] Nie, et al., A time series is worth 64 words: long-term forecasting with transformers. ICLR, 2023.  \n\n**Q3**: The model learns more temporal information than other semantic information.\n\n As stated in $\\underline{\\text{lines 5-6 in abstract}}$, the semantic information of time series is mainly concentrated in temporal variations. Thus, the detailed semantic information can be learned by aggravating multiple time series for reconstruction. \n\n In addition, the aggregation process also relies on the similarity among series-wise representations from multiple time series, which makes the model also learn the global semantic information.\n\n#### **About the Limitations**\n\n**Q1**: The authors should discuss more implications about using large-scale pre-training in forecasting in terms of fairness.\n\nWe pay great attention to the issue of fair comparison. \n- We conduct two types of experiments for all baselines and all tasks: using a unified backbone and the specific backbone proposed in their original papers.\n- We strictly separate training, validation, and testing for all data to prevent data leakage issues."}, {"review_id": "Y5tq0LXpje", "reviewer": "Reviewer_ZNHw", "summary": "The authors propose SimMTM, a novel representation learning method for time-series data based on random masking. They experimentally verify the effectiveness of the proposed method from both in-domain and cross-domain perspectives. Comprehensive comparative experiments with existing methods are conducted, demonstrating the effectiveness of the proposed approach.", "questions": "- What is the definition of $\\operatorname{Encoder}(\\mathcal{X})$? Typically, a function $f \\colon X \\rightarrow Y$ can use all the information in $X$ to calculate $Y$, but it seems that the Encoder function defined here applies the same function in parallel to some dimension. However, it is not clear from the text which dimension this parallel processing applies to.\nFor example, there would be at least two possibilities: $\\operatorname{Encoder}(\\mathcal{X})$ is kind of a “syntax suger” for either $\\bigcup\\_i \\operatorname{Encoder}( \\\\{x\\_i\\\\} \\cup \\\\{\\bar{x}\\_i^j \\\\}\\_{j} )$ or $\\bigcup\\_i ( \\\\{ \\operatorname{Encoder}(x\\_i) \\\\} \\cup \\\\{ \\operatorname{Encoder}(\\bar{x}\\_i^j) \\\\}\\_{j} )$. The same can be said about the definitions of the Projector and Decoder.\n- Although I understood the training pipeline of the proposed method, I had questions regarding its behavior at inference time. Does the proposed method compute internal representations from the true sequence and $M$ randomly masked sequences at inference time, similar to training time? Or does it only use the true sequence without random masking at inference time? Additionally, how is training conducted during the fine-tuning stage in the pipeline?\n- The authors mention the analysis of a quantity called the CKA value in line 262 of Section 4.4, but it is unclear what this quantity actually is. While there is an explanation of Pre-training/Fine-tuning CKA in the caption of Table 5, if the CKA value mentioned in the main text refers to these, its meaning should be clearly explained in the text as well. Moreover, I did not understand why a small $| \\Delta_{\\mathrm{CKA}} |$ leads to “acquiring adaptive representations for different tasks” (l.264). In principle, it should be possible to drastically change the internal representation by fine-tuning while maintaining the degree of change in the internal representation along the evolution of the layers. Suppose $x_\\mathrm{first}$ and $x_\\mathrm{last}$ be the representation in the first and the last layer at the pre-training phase, respectively. We can consider one idealistic situation that the representations after fine-tuning are given as $y_{\\mathrm{first}} = x_{\\mathrm{first}} + \\delta_{\\mathrm{task}}$ and $y_{\\mathrm{last}} = x_{\\mathrm{last}} + \\delta_{\\mathrm{task}}$ with a task-dependent fixed vector $\\delta$. In such a situation, the values of Pre-training/Fine-tuning CKA would be nearly the same, and the difference in CKA values would go to zero by construction. However, the internal representations drastically change depending on the task choice. In this context, I believe there is a logical gap in the authors' argument.\n- In masked modeling-based representation learning, the proposed method uses the true sequence without masking, along with the input. This setup is generally considered to make it more challenging to avoid trivial local solutions that merely output the input as is. Can it be said that such a phenomenon does not occur? Moreover, if a trivial local solution can be avoided, what factors of the proposed method enable this?\n- As a naive control condition for the idea of using multiple different random masks, it is possible to consider the variant of using only one random mask that reduces the masked portion $r$, or takes the position of the mask to have a specific rules that is not completely iid. Can the proposed method be claimed to be effective against such variants? In particular, the model structure of using multiple random masks is expected to have at least some overhead in terms of training/inference computation time compared to such simple control conditions.", "limitations": "The limitations of this paper are adequately discussed in the appendix. Furthermore, the effectiveness of the proposed method can be confirmed from various perspectives through exhaustive comparative experiments. However, as mentioned earlier, there are major concerns primarily about the presentation of the paper. These are points that affect the reproducibility of the results and the authors' claims, so it would be necessary to resolve all problems of unclear logical progression for this paper to be accepted.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors have conducted extensive comparative experiments with existing methods and quantitatively demonstrated the effectiveness of the proposed method. In addition to in-domain generalization, they also focus on cross-domain generalization. This verification is indispensable for validating the effectiveness of deep representation learning assuming large-scale open data.", "weaknesses": "**Presentation quality**\n\nThe current presentation style of this paper is confusing. Particularly, there is a lack of consistency in the representation of formulas and symbols, making it very difficult to grasp the overall logic of the paper. Some specific examples include:\n\n1. In Equation 3, the symbol $\\times$ is used to denote both the Cartesian product and scalar multiplication when defining the domain of $R$.\n2. The transposition notation in Equation 3 is counterintuitive as $\\mathbf{s}$ is defined as a row vector.\n3. Equation 7 is overly verbose and exacerbates the difficulty in reading, as it does not contribute to the subsequent. It would be sufficient to simply declare that only the situation where $i = k$ is treated as positive pairs.\n4. While $\\mathbf{s}$ is defined as a single vector, $\\mathbf{s}^+$ is defined as a set of vectors.\n\nSome other points that I could not make from the paper are discussed below in the Questions section.\n\n**Comparison with prior literature**\n\nThe comparison with prior literature is insufficient. For example, references [1, 2] can be cited as representative examples of representation learning methods for deep neural networks under time-series data. In particular, a model that generalized the method in [2] as a generalized contrastive learning method not limited to the time-series domain was proposed later. Both [2] and [3] have superior theoretical guarantees for identifiability from the perspective of nonlinear ICA. The authors should clearly indicate the proposed model's advantages over these approaches.\n\n**References**\n\n1. Hyvarinen, A., & Morioka, H. (2016). Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. *Advances in neural information processing systems*, *29*.\n2. Oord, A. V. D., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. *arXiv preprint arXiv:1807.03748*.\n3. Hyvarinen, A., Sasaki, H., & Turner, R. (2019, April). Nonlinear ICA using auxiliary variables and generalized contrastive learning. In *The 22nd International Conference on Artificial Intelligence and Statistics* (pp. 859-868). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer ZNHw for the detailed and insightful suggestions.\n\n#### **About Presentation Quality**\n\n(1) We will rephrase $Eq. (3)$ as $\\mathbf{R}=\\text{Sim}(\\mathcal{S})\\in\\mathbb{R}^{D\\times D},D=N(M+1),$\n\n(2) $\\mathbf{R}_{\\text{u},\\text{v}}$ compute the cosine distance via transposition in $Eq. (3)$, resulting in a value that represents the similarity between vectors $\\text{u}$ and $\\text{v}$.\n\n(3) We will remove $Eq. (7)$ in the main text.\n\n(4) We will modify $\\text{s}^+$ to $\\mathcal{S}^+$ as a positive series set.\n\n#### **About Comparison with Prior Literature**\n\n**Q1**: The comparison with prior literature is insufficient.\n\nWe have compared SimMTM with six competitive state-of-the-art baselines in the main text, which are representative and recently published in the time series domain, including Ti-MAE(2023), TST(2021), LaST(2022), TF-C(2022), etc.\n\nAs per reviewer's request, we have reimplemented and compared the generalized methods you suggested in $\\underline{\\text{Table~1, 2 in the global response PDF}}$. SimMTM performs best and achieves 15.5%, 11.5%, and 10.6% average MSE reduction compared to TCL(2016), CPC(2018), and PCL(2019) in forecasting tasks, improves 3.33% average accuracy compared to PCL(2019) in classification tasks.\n\n#### **About the Questions**\n\n**Q1**: What is the definition of $\\text{Encoder}(\\mathcal{X})$ and how to apply it?\n\nIn the previous formulation, we organized time series and its masked series along with batch dimension, which is a conventional usage in pre-training. Thus, as you pointed out, $\\text{Encoder}(\\mathcal{X})=\\bigcup_i(\\{\\text{Encoder}(\\text{x}\\_i)\\}\\cup\\{\\text{Encoder}(\\overline{\\text{x}}\\_i^j)\\_j\\})$, which means $\\text{Encoder}$ will process the input series separately.\n\n**Q2**: Understanding the training pipeline of SimMTM.\n\nSimMTM follows the standard pre-training and fine-tuning paradigm, including three stages: pre-training, fine-tuning, and inference. SimMTM mainly focuses on modeling time series representations in pre-training. Notably, masking is only applied in pre-training, not fine-tuning and inference.\n- In fine-tuning, the pre-trained Encoder is retained while Projector and Decoder are removed. Different task heads are appended to the pre-trained Encoder and fine-tune all parameters for ten epochs to validate the performance. Note that we organize data into batches for fine-tuning without masking.\n- In inference, the model's parameters remain unchanged, and the fine-tuned model is directly used for inference. A single series without masking will be input to Encoder with the specific head.\n\n**Q3**: What is $\\text{CKA}$ and why a small pre-training/fine-tuning $|\\Delta_\\text{CKA}|$ acquire adaptive representations for different tasks?\n\n(1) $\\text{CKA}$ can be used to measure the representation-learning property of deep models.\n\nCentered Kernel Alignment ($\\text{CKA}$) is a statistical measure between representations that identify correspondences in models with different initializations. We can use it to measure the representation-learning property of deep models by calculating $\\text{CKA}$ between a model's first and last layer representations. A larger $\\text{CKA}$ indicates the model tends to learn high-level representations, which is widely used in previous works [1,2,3].\n\n(2) $|\\Delta_\\text{CKA}|$ can measure the difference in representation-learning property between pre-training and fine-tuning models.\n\nSuppose both pre-training and fine-tuning models hold the same high-level or low-level representation-learning property, these two models will present the close $\\text{CKA}$ values, corresponding to smaller $|\\Delta_\\text{CKA}|$. Thus, we use the $|\\Delta_\\text{CKA}|$ to measure the gap between pre-training and fine-tuning models. In other words, **we do not attempt to use $|\\Delta_\\text{CKA}|$ to measure the change of representations but use it to quantify the change of model representation-learning property**.\n\n[1] Xie,et al, Revealing the Dark Secrets of Masked Image Modeling. CVPR, 2022.  \n[2] Kornblith,et al, Similarity of Neural Network Representations Revisited. ICML, 2019.  \n[3] Wu,et al, TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. ICLR, 2023.  \n\n**Q4**: Why the usage of true series in SimMTM will not lead to trivial local solutions?\n\nNote that SimMTM reconstructed point-wise series representations based on series-wise representation similarity. Although the true series-wise representation $\\text{s}\\_i$ undergo similarity calculation in $Eq. (2)$, we only use the point-wise representations of $\\text{z}^\\prime$ for aggregation (corresponding to $\\text{s}^\\prime \\in \\mathcal{S}\\backslash\\{\\text{s}\\_i\\}$). It implies that the reconstruction of SimMTM solely relies on the series-wise similarity values among the true and other series, disregarding the specific point-wise values of the true series. It is not simply reconstructing the original series with the true series, so trivial local solutions will not occur.\n\n**Q5**: Try variants of masking rules.\n\nThe variant you mentioned, only using one masked series with a lower masked ratio can perform well but cannot beat multiple masked series settings. (As shown in $\\underline{\\text{Fig. 5 in the main text}}$). Further, we have analyzed the relationship between masked ratio and masked numbers in Fig. 5, where a reasonable balance between masked ratio and masked numbers is critical.\n\nAs you requested, we also compared different masked rules for ETTh1, including:\n- Fixed position: Masking tail or head\n- Random position: Masking large, small segment or points\n\nThe experiments are in $\\underline{\\text{Table~3 in the global response PDF}}$. Results show random masking is superior to fixed masking (Small Segment > MaskedTail > Masked Head). The difference is insignificant among random masking.\n\nAs mentioned in **Q2**, masked modeling involves only pre-training, so there is no computational overhead problem in fine-tuning and inference."}, {"review_id": "FspvppcCkR", "reviewer": "Reviewer_L1Yx", "summary": "Pre-training models have been using self-supervised learning in various fields (NLP, Vision). However, masked modeling, a representative method of self-supervised learning, was difficult to apply to time-series tasks. Randomly masked data is difficult to show good performance because semantic information of time series including temporal changes is broken. Thus, SimMTM recovers the masked point in time through weighted aggregation of multiple neighbors outside the manifold. This not only restores simple, but also reconstructs complementarily, leading to good performance. SimMTM performs forecasting and classification (in-domain, cross-domain) on various datasets and shows good performance.", "questions": "Regarding weakness, I don't know what $z'$ means in Eq.4 and line 154. No explanation of $s'$ in \"$z'$ represents the corresponding point-wise representation of $s'$\". Does $s'$ mean series-wise representations $S$?", "limitations": "In fact, there is no limit to this work", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. They explained the problem of masked modeling in time-series data well from the time-series point of view.\n2. Also, to apply masked modeling to time-series data, the characteristics of time-series were well utilized. In other words, masked modeling was not simply applied to time-series, but the characteristics of time-series data were well understood and applied accordingly.\n3. Experiments were conducted on various tasks of the pre-training model, and remarkable performance improvements were achieved.", "weaknesses": "It seems that there is a part that does not match the notation in the formula and Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank Reviewer L1Yx for providing an insightful review.\n\n#### **About the Weaknesses**\n\n**Q1**: A part that does not match the notation in the formula and Fig. 2.\n\nThank you for your careful reading. To facilitate summation operations and distinguish different sample representations, we have introduced new symbols, $\\mathbf{z}^{\\prime}$ and $\\mathbf{s}^{\\prime}$. \n\n- $\\mathbf{z}^{\\prime}$ represents the point-wise representation of time series.\n- $\\mathbf{s}^{\\prime}$ represents the series-wise representation of time series.\n\n#### **About the Questions**\n\n**Q1**: The explanation of $\\mathbf{z}^{\\prime}$ and $\\mathbf{s}^{\\prime}$.\n\nThank you for the constructive reviews, your understanding is correct. We will modify the description and $Eq. (4)$ in the original text as follows:\n\n- Rephrase $\\underline{\\text{lines 153-154 in the main text}}$ as:\n\n     where $\\mathbf{s}^{\\prime}$ represents series-wise representation, while $\\mathbf{z}^\\prime$ represents the corresponding point-wise representation, where $\\mathbf{s}^{\\prime} = \\operatorname{Projector}(\\mathbf{z}^{\\prime})$ and $\\widehat{\\mathbf{z}}\\_{i}\\in\\mathbb{R}^{L\\times d_{\\text{model}}}$ is the extracted point-wise representation.\n\n- Modify $\\underline{\\text{Eq. (4) in the main text}}$ to:\n$$\n\\begin{equation}\n\\begin{split}\n    & \\mathbf{s}^{\\prime} = \\text{Projector}(\\mathbf{z}^{\\prime}), \\newline\n    & \\widehat{\\mathbf{z}}\\_{i} = \\sum\\_{\\mathbf{s}^{\\prime} \\in \\mathcal{S}\\backslash\\\\{\\mathbf{s}\\_{i}\\\\}}\\frac{\\text{exp}({\\mathbf{R}\\_{\\mathbf{s}\\_{i},{\\mathbf{s}^\\prime}}/\\tau})}{\\sum\\_{{\\mathbf{s}^\\prime}^{\\prime} \\in \\mathcal{S}\\backslash\\\\{\\mathbf{s}\\_{i}\\\\}} \\text{exp}({\\mathbf{R}\\_{\\mathbf{s}\\_{i},{\\mathbf{s}^\\prime}^\\prime}/\\tau})}\\mathbf{z}^\\prime\n\\end{split}\n\\end{equation}\n$$"}, {"review_id": "AVB14fZAvh", "reviewer": "Reviewer_5Qee", "summary": "This paper presents SimMTM, a simple pre-training framework for masked time-series modeling. The core idea is to train the model to reconstruct the time series by aggregating information from multiple masked series rather than one single masked series. While the temporal pattern is destroyed in a single masked series, multiple masked series contain complementary information, making the reconstruction process more accessible. The aggregation of point-wise representations is weighted by their series-wise similarities. The proposed loss adds the reconstruction loss with a contrastive loss on the series representations. The author provides extensive empirical studies on both time series classification and time series forecasting tasks to evaluate the learned representations.", "questions": "The paper says, \"For each time series, the reconstruction is not only based on its own masked series.\" I wonder how much does the method benefit from aggregating representations from other time series?\n\n", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is sound and well-motivated. Reconstructing the original time series from multiple randomly masked series is novel, in my opinion. The write-up is easy to follow. The overview figure is also illustrative and helps the understanding. The empirical evaluation is comprehensive and convincing.", "weaknesses": "1. In Eq 4, the sum over the variable $s'$ is confusing without carefully reading the text below it. In my opinion, it would be more clear to have the sum over $z'$ and have $s' = Projector(z')$ on the right.\n\n2. Eq 7 is not precise. I understand that the authors mean the intra-similarity of samples should be maximized. However,  the sets before and after $~$ are the same, and therefore they are already \"close\". Please considering revise eq 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 5Qee for providing a meaningful review and insightful suggestions.\n\n#### **About the Weaknesses**\n\n**Q1**: The confusing of $Eq. (4)$.\n\nThank you for your suggestion. Your understanding is correct. We will provide a clearer description and modify $Eq. (4)$ as follows:\n\n- Rephrase $\\underline{\\text{lines 153-154 in the main text}}$ as:\n\n     where $\\mathbf{s}^{\\prime}$ represents series-wise representation, while $\\mathbf{z}^\\prime$ represents the corresponding point-wise representation, where $\\mathbf{s}^{\\prime} = \\operatorname{Projector}(\\mathbf{z}^{\\prime})$ and $\\widehat{\\mathbf{z}}\\_{i}\\in\\mathbb{R}^{L\\times d_{\\text{model}}}$ is the extracted point-wise representation.\n\n- Modify $\\underline{\\text{Eq. (4) in the main text}}$ to:\n$$\n\\begin{equation}\n\\begin{split}\n    & \\mathbf{s}^{\\prime} = \\text{Projector}(\\mathbf{z}^{\\prime}), \\newline\n    & \\widehat{\\mathbf{z}}\\_{i} = \\sum\\_{\\mathbf{s}^{\\prime} \\in \\mathcal{S}\\backslash\\\\{\\mathbf{s}\\_{i}\\\\}}\\frac{\\text{exp}({\\mathbf{R}\\_{\\mathbf{s}\\_{i},{\\mathbf{s}^\\prime}}/\\tau})}{\\sum\\_{{\\mathbf{s}^\\prime}^{\\prime} \\in \\mathcal{S}\\backslash\\\\{\\mathbf{s}\\_{i}\\\\}} \\text{exp}({\\mathbf{R}\\_{\\mathbf{s}\\_{i},{\\mathbf{s}^\\prime}^\\prime}/\\tau})}\\mathbf{z}^\\prime\n\\end{split}\n\\end{equation}\n$$\n\n**Q2**: $Eq. (7)$ is not precise.\n\nThanks for your suggestion, and we will rewrite $Eq. (7)$ to better represent positive pairs and negative pairs as follows:\n\n$$\n\\begin{equation}\n\\begin{split}\n    & \\text{Positive pairs:}~~\\left(\\mathbf{s}\\_{i}, \\mathbf{s}\\_{i}^{+} \\right), \\mathbf{s}\\_{i}^{+} \\in \\{\\overline{\\mathbf{s}}\\_{i}^{j}\\}\\_{j=1}^{M}, \\newline\n    & \\text{Negative pairs:}~\\left(\\mathbf{s}\\_{i}, \\mathbf{s}\\_{i}^{-} \\right), \n    \\mathbf{s}\\_{i}^{-} \\in \\{\\mathbf{s}\\_{k}\\} \\cup \\{\\overline{\\mathbf{s}}\\_{k}^{j}\\}\\_{j=1}^{M}, i\\neq k\n\\end{split}\n\\end{equation}\n$$\n\n#### **About the Questions**\n\n**Q1**: What is the benefit of aggregating representations from other time series?\n\nAs we stated in $\\underline{\\text{lines 155-159 in the main text}}$, the aggregation of representations from other time series requires the model to suppress the interference of less-related noise series and precisely learn similar representations for both the masked and the original series, namely guiding the model to learn the manifold structure better.\n\nTo further verify the benefit, we also conducted a comparison experiment as follows, where \"Own Masked Series\" represents aggregating representations from their own masked series, and \"All Masked Series\" means aggregating representations from their own and other masked series. We present the averaged MSE/MAE from 4 different forecasting horizons {96,192,336,720\\} based on the past 336 time points for the in-domain forecasting tasks. We also record the Accuracy (%) score for classification tasks as follows:\n\n- Forecasting Tasks\n    | Average MSE/MAE|   Own Masked Series | All Masked Series (Ours) |\n    | :-------------:| :------------: | :------------: | \n    | ETTh1          |   0.407/0.430  |   0.404/0.428  |          \n    | ETTh2          |   0.348/0.393  |   0.348/0.391  |       \n    | ETTm1          |   0.348/0.380  |   0.340/0.379  |       \n    | ETTm2          |   0.263/0.318  |   0.260/0.318  |     \n    | Avg            |   0.342/0.380  |   **0.338**/**0.379**  |\n\n- Classification Tasks\n    |    Accuracy (%)   |   Own Masked Series | All Masked Series (Ours) |\n    | :------------------:| :-----------------: | :----------------------: | \n    | Epilepsy → Epilepsy |   94.18             |             94.75        |\n    | SleepEEG → Epilepsy |   95.49             |             95.49        |       \n    | SleepEEG → FD-B     |   67.01             |             69.40        |       \n    | SleepEEG → Gesture  |   77.08             |             80.00        |       \n    | SleepEEG → EMG      |   92.24             |             97.56        |       \n    | Avg                 |   85.20             |           **87.44**      |\n\nThe experimental results show that \"All Masked Series\" performs consistently better than \"Own Masked Series\" in forecasting and classification tasks, which proves that using both own and other less-related noise series for reconstruction is meaningful."}]

Author rebuttal:
## Summary of Revisions and Global Response

We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for improving our paper further.

Since the standard masked modeling seriously ruins vital temporal variations of time series, this paper presents SimMTM, a simple pre-training framework for masked time series modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. **SimMTM achieves state-of-the-art fine-tuning performance compared to 6 advanced baselines in 12 well-established benchmarks in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings.**

The reviewers generally held positive opinions of our paper, in that the proposed method is "**novel**", "**a novel representation learning method**", "**sound and well-motivated**", this paper has "**comprehensive comparative experiments**", and it archives "**remarkable performance improvements**", and the empirical evaluation is "**comprehensive and convincing**".

The reviewers also raised insightful and constructive concerns. We have made every effort to address all the concerns by providing sufficient evidence and requested results. Here is the summary of the major revisions:

- **Clarify the unified backbone setting (Reviewer xbHA)**: For all baselines and all tasks, we have conducted two types of experiments in the original submission, including unified backbone and the specific backbone proposed in their original papers. Furthermore, we perform experiments by keeping each part of the dual-tower TF-C consistent to SimMTM. All results show SimMTM produce the best performance.

- **Resolve the writing issues (Reviewer ZNHw, L1Yx, 5Qee)**: We have rewritten or modified some formulas and corrected some writing errors to improve presentation quality.
  
- **Analyze the relationship of masked ratios and numbers (Reviewer ZNHw)**: We've proven that one masked series with a lower masked ratio can perform well, and we have further analyzed the relationship between masked ratio and masked numbers. Results demonstrate that the design of multiple masked series aggregation in SimMTM is critical, and a reasonable balance between masked ratio and masked numbers shows better performance.

- **Add comparison in different masking rules (Reviewer ZNHw)**: We further explored the effect of different masking rules, including fixed position (tail or head) and random position (large, small segment or points) masking. Results show random masking performs better than fixed masking, and the difference is insignificant among different random masking.

- **Analysis the effects of different reconstruction candidates (Reviewer 5Qee)**: We have conducted a comparison experiment by aggregating from own or all masked series in the reconstruction process. Results show using both own and other less-related noise series for reconstruction performs better.

- **Add baselines (Reviewer ZNHw)**: By comparing SimMTM and three generalized representation learning methods TCL(2016), CPC(2018), and PCL(2019) in forecasting and classification tasks, we further demonstrate that SimMTM achieves state-of-the-art performance.

The valuable suggestions from reviewers are very helpful for us to improve our paper. We'd be very happy to answer any further questions. 

Looking forward to the reviewer's feedback.

#### **The mentioned Tables and Figures are included in the following PDE file.**

- **Figure 1**: More showcases for Reviewer xbHA.
- **Tabel 1**: New baselines in forecasting for Reviewer ZNHw.
- **Tabel 2**: New baselines in classification for Reviewer ZNHw.
- **Tabel 3**: Different masking rules for Reviewer ZNHw.

Meta-review: Post rebuttal, all of reviewers lean toward acceptance. The AC checked all the materials and concurs that the paper has done a valuable exploration of the "masked-modeling" idea in time-series data, addressing issues encountered by the simple baseline and providing sufficient experimental validations. Thus the work should be accepted. Please incorporate necessary changes in the final version.
Decision: Accept (spotlight)