Learning to Embed Time Series Patches Independently
Authors: Seunghan Lee
Taeyoung Park
Kibok Lee
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised learning
masked time-series modeling
contrastive learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose patch-independent strategies for self-supervised representation learning via masked time series modeling.

Abstract: Masked time series modeling has recently gained much attention as a self-supervised representation learning strategy for time series.
Inspired by masked image modeling in computer vision, recent works first patchify and partially mask out time series, and then train Transformers to capture the dependencies between patches by predicting masked patches from unmasked patches.
However, we argue that capturing such patch dependencies might not be an optimal strategy for time series representation learning;
rather, learning to embed patches independently results in better time series representations.
Specifically, we propose to use 1) the simple patch reconstruction task, which autoencode each patch without looking at other patches, and 2) the simple patch-wise MLP that embeds each patch independently.
In addition, we introduce complementary contrastive learning to hierarchically capture adjacent time series information efficiently.
Our proposed method improves time series forecasting and classification performance compared to state-of-the-art Transformer-based models, while it is more efficient in terms of the number of parameters and training time.
Code is available at this repository: https://github.com/seunghan96/pits.

Published as a conference paper at ICLR 2024
LEARNING TO EMBED TIME SERIES PATCHES INDEPENDENTLY
Seunghan Lee, Taeyoung Park, Kibok Lee Department of Statistics and Data Science, Yonsei University (seunghan:611, tpark, kibok][yonsesd aac.kaa.k.. kr
ABSTRACT
Masked time series modeling has recently gained much attention self- supervised representation learning strategy for time series. Inspired by masked image modeling in computer vision, recent works first patchify and partially mask out time series, and then train Transformers to capture the dependencies between patches by predicting masked patches from unmasked patches. However, we argue that capturing such patch dependencies might not be an optimal strategy for time series representation learning; rather, learning embed patches independently results in better time series representations. Specifically, we propose to use 1) the simple patch reconstruction task, which autoencode each patch without looking at other patches, and 2) the simple patch-wist MLP that embeds each patch in- dependently. In addition, we introduce complementary contrastive learning to hierarchically capture adjacent time series information efficiently. Our proposed method improves time series forecasting and classification performance compared to state-of-the-an Transformer- based models, while it is more efficient in terms of the number of parameters and training/inference time. Code is available at this h////gttutuh thub. com/ (eeunghan9//iite
repository: ht tps::: / /gg
INTRODUCTION
Time series (TS) data finds application in a range of downstream tasks, including forecasting, classification, and anomaly detection. Deep learning has shown its superior performance in TS analysis, where learning good representations is crucial to the success of deep learning, and self- supervised learning has emerged as promising strategy for hamessing unlabeled data effectively. Notably, contrastive learning (CL. and masked modeling (MM) have demonstrated impressive performance in TS analysis as well as other domains such as natural language processing (Devlin et al., 2018; Brown et al., 2020) and computer vision (Chen et al., 2020; Dosovitskiy et al., 2021).
Masked time series modeling (MTM) task partially masks out TS and predicts the masked parts from the unmasked parts using encoders capturing dependencies among the patches, such as Transformers (Zerveas et al., 2021; Nie et al., 2023). However, we argue that learning such dependencies among patches, e.g.. predicting the unmasked parts based on the masked parts and utilizing architectures capturing dependencies among the patches, might not be necessary for representation learning.
To this end, we introduce the concept of patch independence which does not consider the interaction between TS patches when embedding them. This concept is realized through two key aspects: 1) the pretraining task and 2) the model architecture. Firstly, we propose patch reconstruction task
20-
V000000000
so-
Shound Truts
that reconstructs the unmasked patches, unlike the conventional MM that predicts the masked ones. We refer to these tasks as the mathh-independent (PI) task and the patch-dependen (PD) task, respectively, as the former does not require information about other patches to reconstruct each patch, while the latter does. Figure! illustrates a toy example of TS forecasting. While the Transformer pretrained on the PD task (Nie et al., 2023) fails to predict test data under distribution shift, the one pretrained on the PI task is robust it. Secondly, employ the simple PI architecture (e.g., MLP), exhibiting better efficiency and performance than the conventional PD architecture (e.g., Transformer).
Train
Test
650
120
alo
Figure 1: Pl.s. PD.
In this paper, we propose Patch Independence for Time Series (PITS), which utilizes unmasked patch reconstruction the PI pretraining task and MLP the PI architecture. On top of that, we introduce complementary CL to efficiently capture adjacent time series information, where CL is performed using two augmented views of original samples that are masked in complementary ways.

---

Published as a conference paper at ICLR 2024
CL for TS'
TST (KDI 2021)
TS2Nec (AAAI 2022)
FEDRomna (CML 2022)
DLiscar (ANAI 2023)
PushIST (CLE 2025)
TimeMAE (aAii 2023)
SmnMTM (NeurIPS 2025)
PTTS (Ous))
CL
Pretraining
method
/
No (Sup)
Dawnerrcam task
Forecasting
Clesificatio
Loss (NeurIPS 2019), Self-Time (arXwv 2020), TS ID (JICNN 2021), TS TSC (IICAI 2021), TNC (ICLR 2021), Mixing up (PR Letters
2022), TF-C (NeurIPS 2022), TimeCLR (KBS 2022), CA TCC (TPAMI 2023).
Table 1: Comparison table of SOTA methods in TS.
We conduct extensive experiments on various tasks, demonstrating that our proposed method out- performs the state-of-the ofrt----t-e--tttt (SOTA) performance in both forecasting and classification tasks, under both standard and transfer learning settings. The main contributions are summarized as follows:
We argue that learning embed time series patches independently is superior to learning them dependently for TS representation learning, in terms of both performance and efficiency. To achieve patch independence. we propose PITS, which incorporates two major modifications on the MTM: 1) to make the task patch-inder autch-independent, reconstructing the unmasked patches instead of predicting the masked ones, and 2) make the encoder patch-independent, eliminating the attention mechanism while retaining MLP to ignore correlation between the patches during encoding.
We introduce complementary contrastive learning to hierarchically capture adjacent TS information efficiently, where positive pairs are made by complementary random masking.
We present extensive experiments for both low-level forecasting and high-level classification, demonstrating that method improves SOTA performance various downstream tasks. Also, we discover that PI tasks outperforms PD tasks in managing distribution shifts, and that PI architecture is more interpretable and robust to patch size compared PD architecture.
RELATED WORKS
Self-supervised learning. In recent years, self-supervised learning (SSL) has gained attention for learning powerful representations from unlabeled data across various domains. The success of SSL comes from active research on pretext tasks that predict a certain aspect of data without supervision. Next token prediction (Brown et al., 2020) and masked token prediction (Devlin et al., 2018) commonly used in natural language processing, and jigsaw puzzles (Noroozi & Favaro, 2016) and rotation prediction (Gidaris & Komodakis, 2018) are commonly used in computer vision.
Recently, contrastive learning (CL) (Hadsell et al., 2006) has emerged as effective pretext task. The key principle of CL to maximize similarities between positive pairs while minimizing similarities between negative pairs (Gao et al., 2021; Chen al., 2020; Yue et al., 2022). Another promising technique is masked modeling (MM), which trains the models to reconstruct masked patches based on the unmasked part. For instance, in natural language processing, models predict masked words within sentence (Devlin et al., 2018), while in computer vision, they predict masked patches in images (Baevski al., 2022; He al., 2022; Xie al., 2022) within their respective domains.
Masked time series modeling. Besides CL, MM has gained attention pretext task for SSL in TS. This task involves masking a portion of the TS and predicting the missing values, known as masked time series modeling (MTM). While CL has shown impressive performance in high-level classification tasks, MM has excelled in low-level forecasting tasks (Yue et al., 2022; Nie et al., 2023). TST (Zerveas et al., 2021) applies the MM paradigm to TS, aiming to reconstruct masked timestamps. PatchTST (Nie et al., 2023) focuses predicting masked subseries-level patches to capture local semantic information and reduce memory usage. SimMTM (Dong et al., 2023) reconstructs the original TS from multiple masked TS. TimeMAE (Cheng et al., 2023) trains transformer-basee encoder using two pretext tasks, masked codeword classification and masked representation regression. Table I compares various methods in TS including two criterions: pretraining methods and downstream tasks, where No (Sup.) in Pretraining method indicates supervised learning method that does not employ pretraining.
in terms of
Different from recent MTM works, we propose to reconstruct unmasked patches through autoencod- ing. A primary concern on autoencoding is the trivial solution of identity mapping, such that the dimension of hidden layers should be smaller than the input. To alleviate this, we introduce dropout after intermediate fully-comn cullecconeetted (FC) layers, which similar to the case stacked denoising autoencoders (Liang & Liu, 2015), where the ablation study be found in Figure 4.
2

---

Published as a conference paper at ICLR 2024
(a)
architecture
architecture
Lypoon
Cou
22
MiL-Missr
Transformer
TRULU
1RALU
Leccoal Recorstrution
Eey, Contrastive Lova
4PH.41 1:
PPatcching
@i 2.MPP
Patching
WaMi/emii 3. MLP-Mixer
+ Patching
Wachicrriee Transformer
+ Parching
Putching
Neoking
K-N-
X(1
Linaarisyarr
Liineer
Figure 2: Patch-independent strategy of PITS. (a) illustrates the pretraining tasks and encoder architectures in terms of PI and PD. (b) demonstrates the proposed PITS, which utilizes PI task with architecture. TS is divided into patches and augmented with complementary masking. Rep- resentations from the 1st and 2nd layers of MLP is used for CL and the reconstruction, respectively.
Combination of CL and MM. There have been recent efforts to combine CL and MM for represen- tation learning (Jiang et al., 2023; Yi et al., 2023; Huang et al., 2022; Gong et al., 2023; Dong et al., 2023). Among these works, SimMTM (Dong et al., 2023) addresses an MM task with a regularizer in its objective function in the form of a contrastive loss. However, differs from our work in that focuses CL between TS, while our proposed CL operates with patches within single TS.
Complementary masking. SdAE (Chen et al., 2022) employs a student branch for information reconstruction and a teacher branch to generate latent representations of masked tokens, utilizing a complementary multi-fold masking strategy maintain relevant mutual information between the branches. TSCAE (Ye et al., 2023) addresses the gap between upstream and downstream mismatches in the pretraining model based MM by introducing complementary masks for teacher-student networks, and CFM (Liao et al., 2022) introduces a trainable complementary masking strategy for feature selection. Our proposed complementary masking strategy differs in that it is not designed for a distillation model, and our masks are not Iearnable but randomly generated.
Linear models for time series forecasting, Transformer (Vaswani et al., 2017) is a popular sequence modeling architecture that has prompted surge in Transformer-based solutions for time series analy- sis (Wen et al., 2022). Transformers derive their primary strength from the multi-head self-attention mechanism, excelling extracting semantic correlations within extensive sequences. Nevertheless, recent work by Zeng et al. (2023) shows that simple linear models can still extract such information captured by Transformer- baasedd methods. Motivated by this work, we propose to use a simple MLP architecture that does not encode interaction between time series patches.
3 METHODS
(H.em) We address the task of learning an embedding function So xx - (f.c.n) for a TS patch where
Pp {f2l (i.c,a) are the number of TS, number of channels in single TS, and number of patches in single channel of a single TS. The input and the output dimension, which are the patch size and patch embedding dimension, are denoted as P and D, respectively, i.c., æp (1,0,)) E R² and z (f.o,)) e R... Our goal is to learn fo extracting representations that perform well on various downstream tasks.
f. z {2(1))))) }, and 1..... B, c L...., C, I. Here, B, C, N
Channel independence & Patch independence. We the channel independence architecture for our method, where all channels share the same model weights and embedded independently, i.c, fe is independent c. This has shown robust prediction to the distribution shift compared to channel-dependen approaches (Han et al., 2023). Also, we propose to use the PI architecture, where all patches share the same model weights and embedded independently. i.e, fo is independent to n. We illustrate four different PI/PD architectures in Figure 2(a), where we use MLP for our proposed PITS, due to its efficiency and performance, as demonstrated in Table and Table 7, respectively.
3.1 PATCH-INDEPPEDDET TASK: PATCH RECONSTRUCTION
Unlike the conventional MM task (i.e., PD task) that predicts masked patches using unmasked ones, propose the patch reconstruction task (i.e., PI task) that autoencodes each patch without looking at the other patches, as depicted in Figure 2(a). Hence, while the original PD task requires capturing patch dependencies, proposed task does not. A patchified univariate TS can be reconstructed in
3

---

Published as a conference paper at ICLR 2024
two different ways': 1) reconstruction at once by FC layer processing the concatenation of patch
@p ((.0))) Wiconnat (2(6e.) where Wi E R-N-PxN-D and 2) patch-wise
representations: concat
reconstruction by FC layer processing each patch representation: Zp xlien) = (i(ln))) where
W E RPxD_ Similar to Nie et al. (2023), employ the patch-wise reconstruction which yields better performance across experiments.
3.2 PATCI--NDEPPENDEN ARCIITECTURE: MLP
While MTM has been usually studied with Transformers for capturing dependencies between patches, argue that learning to embed patches independently is better. Following this idea, we propose the simple PI architecture, so that the encoder solely focuses on extracting patch-wise representations. Figure 2(a) shows the examples of PI/PD pretraining tasks and encoder architectures. For PI architectures, Linear consists of a single FC layer model and MLP consists of a two-layer MLP with ReLU. For PD architectures, MLP-Mixer* (Tolstikhin al., 2021: Chen et al., 2023) consists of single FC layer for time-mixing (N-dim) followed by a two-layer MLP for patch-mixing (D-dim), and Transformer consists of a self-attention layer followed by a two-layer MLP, following Nie et al. (2023). The comparison of the efficiency between MLP and Transformer in terms of the number of parameters and training/inference time is provided in Table 13.
3.3 COMPLEMENTARY CONTRASTIVE LEARNING
To further boost performance of learned representations, we propose complementary CL to hierar- chically capture adjacent TS information. CL requires two views to generate positive pairs, and we achieve this by a complementary masking strategy: for a TS æ and a mask m with the same length, consider m and (1 m)
two views, where @ is the element-wise multiplication and we use 50% masking ratio for experiments. Note that the purpose of masking is to generate two views for CL; it does not affect the proposed PI task, and does not require an additional forward pass when using the proposed PI architectures, such that the additional computational cost is negligible.
Figure illustrates an example of complementary CL, where we perform CL hierarchically (Yue
Patch Represeetattona::
Coarse
et al., 2022) by max-pooling on the patch repre-
Max Post
sentations along the temporal axis, and compute
and aggregate losses computed at each level. Then.
the model learns find missing patch information in one view, by contrasting the similarity with an- other view and the others, so that the model can capture adjacent TS information hierarchically.
1
5
7
8
View
View 2
Figure 3: Complementary contrastive learning.
3.4 OBJECTIVE FUNCTION
As illustrated in Figure 2(b), we perform CL at the first layer and reconstruction by an additional projection head on top of the second layer, based on the ablation study in Table 9. To distinguish them, denote representations obtained from the two layers in MLP 21 and 22, respectively. Reconstruction loss. As discussed in Section 3.1, we feed z2 into the patch-wise linear projection head to get a reconstructed result: Tp = z2. Then, the reconstruction loss can be written as:
Crecom
ΣΣΣ
B
C
N
m
(l.n.) o ( (1.c.)) - - a((ham)) )1,
||0 m (hemm)) o
x
(f.c.) - (1...))
22p
|?
i=1 n=
C
N
((.,.)) mn
- .)ffien)
(1)
1=1
(t.c.)) is masked, and 1 otherwise. As derived in Eq. 1, the where m (1,..)) O if the first view Φp reconstruction task not affected by complementary masking, i.e., reconstructing the unmasked patches of the two views is the same as reconstructing patches without complementary masking.
Biases are omitted for conciseness. While TSMixer is a variation of Mixe-Mixer proposed for TS concurrent to work, we found that TSMixer does not perform well with SSL,, so we our own variation of MI.P-Mixer here.
4

---

Published as a conference paper at ICLR 2024
Contrastive loss. Inspired by the cross-entropy loss-like formulation of the contrastive loss in Lee et al. (2021), we establish a softmax probability for the relative similarity among all the similarities considered when computing temporal contrastive loss. For conciseness, let 2 ((...) 21
((e.22N)
and ø1 ((ccn+N)) be the two views the patch embedding x (l..,)) Then, the softmax probability for a pair of patch indices (n, rt') is defined as:
exp(z1 (f)e.)) 2 (honn)
c,,
E L exp(z) ((..)) 21 =(hea)))
where we use the dot product as the similarity measure Then, the total contrastive loss can be written as:
B C 2N LEL 2BCN ΣΣΣ
(n,
where we compute the hierarchical losses by max-pooling x (Lenn)'s along with the dimension repeatedly with the following substitutions until N
(k...)) < MaxPool((2(he222 2 ((,,222)) N [N/2]-
The final loss of PITS is the sum of the reconstruction loss and hierarchical contrastive loss:
Lreeon + La.
Instance normalization. To mitigate the problem of distribution shift between the training and testing data, normalize each TS with zero mean and unit standard deviation (Kim al., 2021). Specifically, we normalize each TS before patching and mean and deviation are added back to the predicted output.
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Tasks and evaluation metries. We demonstrate the effectiveness of the proposed PITS on two downstream tasks: time series forecasting (TSF) and classification (TSC) tasks. For evaluation, we mainly follow the standard SSL framework that pretrains and fine-tunes the model on the same dataset, but we also consider in no-mimain and cross-domain transfer learning settings in some experiments. As evaluation metrics, we use the mean squared error (MSE) and mean absolute error (MAE) for TSF. and accuracy, precision, recall, and the Fj score for TSC.
4.2 TIME SERIES FORECASTING
Datasets and baseline methods. For forecasting tasks, we experiment seven datasets, including four ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2), Weather, Traffic, and Electricity (Wu et al., 2021), with prediction horizon of / {96, 192, 336, 720}. J. For the baseline methods, we consider Transformer-based models, including PatchTST (Nie et al., 2023), SimMTM (Dong et al., 2023), FEDformer (Zhou al., 2022), and Autoformer (Wu al., 2021), and linear/MLP models, including DLinear (Zeng et al., 2023) and TSMixer (Chen et al., 2023). We also compare PITS and PatchTST without self-supervised pretraining 3. which essentially compares PI and PD architectures only. We follow the experimental setups and baseline results from PatchTST, SimMTM, and TSMixer. For all hyperparameter tuning, we utilize a separate validation dataset, following the standard protocol of splitting all datasets into training, validation, and test sets in chronological order with ratio of 6:2:2 for the ETT datasets and 7:1:2 for the other datasets (Wu et al., 2021).
Standard setting. Table shows the comprehensive results on the multivariate TSF task, demon- strating that our proposed PITS is competitive to PatchTST in both settings, which is the SOTA Transformer- based method, while PITS is much more efficient than PatchTST. SimMTM is con- current work showing similar performance to ours in SSL while significantly worse in supervised
For PITS and PatchTST supervised leaming, patches overlapped following Nie al. (2023).
5

---

Published as a conference paper at ICLR 2024
Mediit
Seperssed
PITS
PITS w/ CL
PILSTST
SaIMTM
PITS
PaxhIST
SuNMTN
DLince:
TSMisei
FEDHones
MSS
MAB
50
8.307
0.303
8.367
0.303
0.379
0
0.367
0,402
0.369
0.33
0.315
0.39
0.350
0.41
0.555
0.359
0.311
0.392
0.376
0.45
0.435
0.64
8:155
0.438
0.455
0.415
0.489
0.436
0.443
0.50
0.44
0.46
0.48
130
9/425
0.453
0444
0.439
0.465
0.474
0.530
2.55
0.556
0.465
0.449
2,855
0.451
0.459
0.472
0.400
0.63
0,472
0.499
0.492
0.315
0.517
0.329
0.377
0.381
0
03366
0.215
0L330
0.379
0.00
0.43
0.885
0:413
0.594
01385
0.017
0.46
0.46
0454
136
0.356
0.397
0.01
11.00
0480
0.67
03663
0,011
1111
0.394
01331
0.30
0.005
0.111
0.68
0%9
0.006
0.00
0.67
0/47
750
0.261
0.425
0.11
1.%
0/11
0.4
0.906
0.911
10
0.09
0.379
0,422
0.551
0/01
0.65
0.551
0.415
11570
0/11
0.40
0/451
0190
0.244
16384
03911
11.51
0.51
0,389
0.541
00255
0.3%
12300
0354
01396
0:36
0.29
0.20
0,350
6:35
0,500
0,50
01922
192
0.328
0.313
03319
11511
111
0.599
11113
0.50
1:300
0.39
0335
0.39
03335
0.74
0.58
0.57
11380
6383
6415
0.34
0.9
14308
0.663
0.2
0229
15889
0.385
0.366
0391
11310)
0:38
0.30
0.39
0.382
0391
11425
8330
0.82
0:49
0.91
0.163
0.200
0.100
0.353
0057
0.5
0.166
0227
0255
0.16
0.175
6,25
0.667
0.260
0.103
0.353
0.00
0.71
0.205
0.29
03:1
9.16
0318
0.5%6
0.363
0.31
0332
0.381
730
0,337
0.335
0318
0336
0
1131
1341
0.000
0411
0.99
B/411
0330
0.133
0300
0.030
0/14
Weather
0.50
0.81
0.38
6381
0.306
6.354
0.663
8.191
0,343
6.5
0359
0.315
0.50
136
8:344
0.500
0111
6.50
033
1111
070
11
0.50
6.00
036
0.19
0.21
0.300
6338
0357
0.311
0391
0.314
0.330
0.39
0.288
029
0.356
0:31
0.323
0.362
0.33
0.336
030
0,40
0.436
0.255
0.734
0.393
0.25
IL368
13592
IL367
0.251
0.991
0.358
0.400
0:382
0.58
0.55
0.350
0,591
0.571
193
0.38
11371
03345
11.771
0.71
0:11
11000
0250
IL385
0.291
01575
0.30
11425
8:35
039
11364
CA10
1:300
8631
01302
0.250
0,08
0.396
0:365
6.315
0.46
0.3%
0.13
0.75
0,395
0.130
0.225
01131
0.224
8.126
0.211
11213
0.22
01:21
0.226
1L130
0:222
0.110
0,279
0.140
0.231
0.31
I.229
0.186
0,340
0.135
0.13
51145
01:45
0.237
01:46
0:340
1915
0.153
01360
0131
01666
0.082
0.901
0.161
6.313
0.714
0.39
750
0.194
0.387
0.801
01.40)
0.00
0 7900
01700
0.70
0.089
0.241
0.706
0
01255
0.111
0.313
0.301
0.190
0.388
0.731
0.368
0.716
0
Avorigo
0.337
0.394
0.314
0.306
0.311
0.304
0.202
0.327
0.355
0.111
0.313
We used the official code to replicate the results.
SimMTM is a concurrent work to ours.
Table 2: Results of multivariate TSF. We compare both the supervised and self-supervised versions of PatchTST and our method. The best results are in bold and the second best are underlined.
Mesic: MSE
FF
PITS
L.
Sug.
FF
PuchTST
LP
Sup:
Target
14434
PRS
8.405
11423
PackIST
D/064
SIeATIM
D411
TirMAA
8.728
11543
TST
0.445
LIST
01033
11:54
ETTE.
0.401
0.403
0.424
0.434
0.417
0.351
BTT22
0,334
0.337
0.373
0,364
6.331
Avtras
0.315
0,33%
0.385
0,400
0.385
8.76
0.363
0,429
0.561
0.483
Weother
6TTm2
0.344
0:225
0.23
0.344
0,247
0.225
0.34
1.76
0,264
0,233
0,230
0,258
0.35
0.55
0.433
18.83
0:421
0.776
0.438
0396
6.2%
D408:
0776
01413
0,775
0.300
0.582
11377
Traffic
0.403
0.405
0.401
0.00
0.424
0.366
15731
0.353
0355
11:48
0,300
0.355
0.146
11359
Elecoiiity
0.157
0.161
0.160
0.159
0.168
0.162
Wodlat
ETT
0.496
8.43
0,03
0.56
Asarage
0.301
0.506
0.304
0.314
0.500
0.507
Aorige
6.79
0.02
0.366
0.97
0.94
Table 3: PITS vs. PatchTST.
Table 4: Results of TSF with transfer learning.
learning. Table 3 compares PITS and PatchTST under three different scenarios: fine-tuning (FT), linear probing (LP), and supervised learning without supf-supe pretraining (Sup.), where we present the average MSE across four horizons. As shown in Table 3, PITS outperforms PatchTST for all scenarios on average.
Transfer learning. In in-domain transfer, we experiment datasets with the same frequency for the source and target datasets, whereas in cross-domain transfer, datasets with different frequencies are utilized for the source and target datasets. Table shows the results of the average MSE across four horizons, which demonstrates that our proposed PITS surpasses the SOTA methods in most cases.
4.3 TIME SERIES CLASSIFICATION
Datasets and baseline methods. For classification tasks, we use five datasets, SleepEEG (Kemp et al., 2000), Epilepsy (Andrzejak et al., 2001), FD-B (Lessmeier et al., 2016), Gesture (Liu et al., 2009), and EMG (Goldberger et al., 2000). For the baseline methods, we employ TS-SD (Shi et al., 2021), TS2 (Yue et al., 2022), CoST (Woo et al., 2022), LaST (Wang et al., 2022), Mixing-Up (Wickstrøm et al., 2022), TS-TCC (Eldele et al., 2021), TF-C (Zhang et al., 2022), TST (Zerveas et al., 2021), TimeMAE (Cheng al., 2023) and SimMTM (Dong et al., 2023).
Standard setting. Table 5 demonstrates that our proposed PITS outperforms all SOTA methods in all metries on the SleepEEG dataset. This contrasts with the results prior works that CL is superior to MTM for classification tasks (Yue et al., 2022): while prior MTM methods such as TST and TimeMAE shows relatively low performance compared to CL methods such as TS2Vec and TF-C', the proposed PITS outperforms CL methods, even without complementary CL.
Transfer learning, For transfer learning, we conduct experiments in both in-domain and cross- transfer settings, using SleepHEG as the source dataset for both settings. For in-domain transfer, we use target datasets from the same domain as the source dataset, which share the characteristic of
An exception is SimMTM (Dong et al., 2023), which is officially published at the time of submission.

---

Published as a conference paper at ICLR 2024
ACC.
PRE.
REC.
TS2Vcc
92.17
93.84
$1.19
85.71
COST
LAST
92.11
88.07
91.58
93.12
66.05
81.47
69.11
85.74
TEAD
01.37
13:18
BBC
16:31
70.00
15
35 09
ACC:
31:15
FNC.
06.54
97/0
68:22
ASC
66.66
PMC.
06.01
K25.
94.00
63:16
1915
15.85
15.35
21
TFC
93.96
94.87
89.46
56.81
18:30
12:34
1830
81:00
18.35
18.43
$6.00
88.13
00.00
06.35
09-62
53.00
-0011
42:00
15.3 0/22
TST
80.21
40.11
50.00
44.51
Monpsh
01.21
43:11
2610
66.11
81:01
7144
76.13
68:33
07:10
08.33
04377
11.34
1006
25.55
13 41
TimeMAE
80.34
90.16
50.33
45.20
SimMTM
94.75
95.60
$9.93
91.41
TEC
05.05
51.55
1816
91 11
68 11
11.08
73
14.07
76.13
76.36
75:30
6171
73.65
11.00
PITS w/o CL
95.22
95.35
95 27
95.30 30
TreMAR
86.11
1.
81
06
14
11
45.00
78-16
76.25
2130
68
68.05
71.24
61.6
50.3
PITS
95.67
95.63
95.67
95.64
95.71
95.70
18.55 15.66
03.58
00.3
02.90
1080
1869
1958
Table 5: Results of TSC.
Table 6: Results of TSC with transfer learning.
Pl acrhitecture
PD architecture
Linear
MI.P
MI.P-Mixer
Transformer
Task
PD
Pt
Gain(S)
PD
Pl
Gain(
PD
Pl
Gain(%)
PD
Pl
Gain(S))
ETTa1
0.408
0.408
+0.0
0.418
0.407
+2.6
0.420
0.409
+2.6
0.425
0.415
+2.4
ETTh2
0.343
0.338
0.361
0.334
+7.5
0.365
+6.6
+3.1
ETTml
0.359
0,358
+0.2
0.356
0.355
+0.3
0.354
0.352
+0.66
0,350
0,350
+0.0
FTTm2
0,254
0.243
0.258
0.253
0.259
0.253
+2.3
+6.6
Average
0.342
0.340
+0.3
0.348
0.337
+3.2
0.350
0.339
+3.1
0.351
0.341
+2.8
Table 7: Effectiveness of PI strategies. Pretraining with the PI task consistently outperforms the PD task across all architectures. The results are reported as the average across four horizons.
being EEG datasets, while we use target datasets from the different domain for cross-domain transfer. Table 6 demonstrates that our PITS outperforms SOTA methods in all scenarios. In particular, the performance gain is significant in the challenging cross-domain transfer learning setting, implying that PITS would be more practical in real-world applications under domain shifts.
4.4 ABLATION STUDY
Effect of PI/PD tasks/architectures. To assess the effect of our proposed Pl pretraining task and Pl encoder architecture, we conduct an ablation study Table using a common input horizon of 512 and patch size of 12. Recall that the PD task predicts masked patches using unmasked patches while the PI task autoencodes patches, and the PD architectures include interaction among patches using either the fulle-connected layer (MLP-Mixer) while the PI architectures (Linear, MLP) do not. As shown in Table 7, PI pretraining results in better TSF performance than PD pretraining regardless of the choice of the architecture. Also, PI architectures exhibit competitive performance compared to PD architectures, while PI architectures more lightweight and efficient demonstrated in Table 13. Among them, MIP shows the best performance while keeping efficiency, so we use MLP as the architecture of PITS throughout all experiments.
the self-attention module (Transformer),
Hidden dimension and dropout. The PI task may raise a concern on the trivial solution: when the hidden dimension D is larger than the input dimension P. the identity mapping perfectly reconstructs the input. This can be addressed by introducing dropout, where we add a dropout layer before the linear projection head. Figure 4 displays the average MSE on four ETT datasets across four horizons under various hidden dimensions common input horizon of 512, without dropout or with the dropout rate of 0.2. Note that for this experiment, the input dimension (patch size) is 12, and trivial solution can occur if D 2 12. The results confirm that using dropout is necessary to learn high Figure 4: MSE by D and dropout.
0.348
0.346
Aug. MSE
wio Dropout
wi Dropout
0.342
0.340
in MLP with
12 16
32
128
Hidden dimension
dimensional representations, leading to better performance. Based this result, we tune D {32, 64, 128) throughout experiments, while performance is consistent with D values in the range. An ablation study with different dropout rates can be found in Appendix K.
Performance of various pretrain tasks. In addition to the 1) PD task of reconstructing the masked patches (Xn)) and 2) Pl task of autoencoding the unmasked patches (X.). we also employ two other basic tasks for comparison: 3) predicting X, from pero-filled patches (0) and 4) autoencoding O.
Table 8 displays the average MSE on four ETT datasets across four horizons with a common input horizon of 512, highlighting that the model pretrained with the PD task performs even worse than
7

---

Published as a conference paper at ICLR 2024
22
=2
Task
Trans- former
Layer
P1
CL
Layer
CL
96
0.371
0.367
0.369
Input
Output
w/o CL
CL
ETTa1
0.720
0.407
0.117
0.442
0.401
192
0.396
0.401
0.403
X,
X.
0.341
0.338
0.330
ET
0.396
0.334
0.366
0.371
0.334
336
0.411
0.415
0.428
X,
Xm
0.351
0,348
0.364
ETTml
0.711
0.357
0.356
0.358
0.342
720
0.448
0.425
0.460
0
X,,
0.342
0.348
0.348
ETTm2
0.381
0.253
0.254
0.343
Avg.
0.552
0.338
0.348
0.359
0.130
Avg.
0.407
0.401 0.415
Table 8: Pretraining tasks.
Table 9: Effect of CL..
Table 10: Representation for downstream tasks.
Encoder Architecture Transformer
Pl task
ETTh1
ETTh2
ETTml
ETTm2
Avg.
Linear MLP
Transformer
0.425
0.353
0.350
0.274
0.351
0.425
0.408
0.418
w/o CL
0.407
0.334
0.357
0.253
0.338
2) task -> Pl task 0.408
MLP
w/ non-hier. CL
0.405
0.333
0.353
0,252
0.336
0.415
0.407
w/ hier. CL
0.401
0.334
0.342
0.244
0.330
3) + Complementary CL
0.401
Table 11: Hierarchical design of complementary CL.
Table 12: PatchTST-PPITS
the two basic tasks with 0 inputs. This emphasizes the ineffectiveness of the PD task and the effectiveness of the proposed PI task.
Which representation to use for downstream tasks? In SSL., the boundary of the encoder and the task-specific projection head is often unclear. To determine the location to extract representation for downstream tasks, we conduct experiments using representations from intermediate layers in MLP: 1) from the first layer. 2) Z2 from the second layer, and 3) z from the additional projection layer attached on top of the second layer. Table 10 displays the MSE of ETTh across four horizons, indicating that the second layer 22 yields the best results.
Location of complementary CL. To assess the effect of complementary CL together with PI reconstruction, we conduct an ablation study on the choice pretext tasks and their location the MLP encoder: the contrastive and/or reconstruction loss is computed on the first or second layer, or neither. Table 9 displays the average MSE on four ETT datasets across four horizons. We observe that the PI reconstruction task is essential, and CL is effective when it is considered in the first layer.
Hierarchical design of complementary CL. The proposed complementary CL is structured hierar- chically to capture both coarse and fine- fineigrained information in time series. To evaluate the effect of this hierarchical design, consider three different options: 1) without CL, 2) with non-hierarchical CL, and 3) with hicrarchical CL. Table 11 presents the average MSE on four ETT datasets across four horizons, highlighting the performance gain by the hierarchical design.
Comparison with PStchTST. PITS can be derived from PatchTST, by changing the pretraining task and encoder architecture. Table 12 shows how each modification contributes the performance improvement the ETThl dataset. Note that we apply mask ratio of 50% to PStchTST, which does not affect the performance (marked with ").
ANALYSIS
PI task is more robust to distribution shift than PD task. To assess the robustness of pretraining tasks to distribution shifts, which are commonly observed in real-world datasets (Han et al., 2023), we generate 98 toy examples exhibiting varying degrees of distribution shift, as depicted in the left panel of Figure 5. The degree of shift is characterized by changes in slope and amplitude. The right panel of Figure 5 visualizes the performance gap between the models trained with the PD and PI tasks, where the horizontal and vertical axis correspond to the slope and amplitude differences between training and test phases, respectively. The result indicates that the model trained with the PI task exhibits overall better robustness to distribution shifts as the MSE difference is non-negative in all regime and the gap increases as the shift becomes more severe, particularly when the slope is flipped or amplitude is increased.
MLP is more robust to patch size than Transformer. To assess the robustness of encoder architec- tures patch size, compare MLP and Transformer using ETTh1 with different patch sizes. Figure illustrates the results, indicating that MLP is more robust for both the PI and PD tasks, resulting in consistently better forecasting performance across various patch sizes.
8

---

Published as a conference paper at ICLR 2024
Scenarios Distribution Shift Seasonailty)
MSE difference (PD-PI)
Dataset: ETTh1
325
0.455-
MLP mus
0440
75
Ouetraution
Tlass 1D loo4
W
14
0.420
25
803
-30
Train
Test
-75
100
200
240
282
4
8
12
Patch Size
16
24
32
Time Irilex
Figure 5: PI vs. PD tasks under distribution shifts.
Figure 6: MSE by patch size.
90
TS with different trend & seasonality
[ETTm1]
patchee 12 langth
Patch
MLP (PI arch)
Patch 42
Transformer (PD arch)
Time Step
Prodobet WETTE
Figure 7: Downstream task weight W E H
Figure 8: t-SNE visualization.
MLP is more interpretable than Transformer. While PI architectures process each patch inde- pendently, PD architectures share information from all patches, leading to information leaks among patches. This makes MLP more interpretable than Transformer, of the linear layer additionally introduced and Iearned for the downstream task shows each patch's 'ss contribution to predictions. Figure illustrates the seasonality of ETTml and the downstream weight matrix trained on ETTml for both architectures. While the weight matrix of the linear layer on top of Transformer is mostly uniform, that of MLP reveals seasonal patterns and emphasizes recent information, highlighting that MLP captures the seasonality better than Transformer.
visualizing the weight matrix
Efficiency analysis. To demonstrate the efficiency of the PI architecture, we compare PatchTST and PITS in terms of the number of parameters and train- ing/inference time on ETTm2. As shown in Table 13, PITS outperforms PatchTST with significantly fewer parameters and faster training and inference, where we pretrain for 100 epochs and perform inference with the entire test dataset. The comparison of the efficiency between self-supervised and supervised settings is pro- vided Appendix! J.
Self-supervisel settings
PaichTST
PITS
CL.
W/CL
w/ hier. CL
Number of params
406,028
5,772
Pretrain time
27
Inference time (sec)
Avg. MSE
0.274
0.253
0,252
0.244
Table 13: Time/parameter efficiency.
t-SNE visualization. To evaluate the quality of representations obtained from the PI and PD tasks, we utilize t-SNE (Van der Maaten & Hinton, 2008) for visualization. For this analysis, create toy examples with classes of its own trend and seasonality patterns, as shown in Figure 8. The results demonstrate that representations learned from the PI task better distinguishes between classes.
CONCLUSION
This paper revisits masked modeling in time series analysis, focusing on two key aspects: 1) the pretraining task and the model architecture. In contrast to previous works that primarily emphasize dependencies between TS patches, we advocate a patch-independent approach on two fronts: 1) by introducing a patch reconstruction task and 2) employing patch-wise MLP. Our results demonstrate that the proposed PI approach is more robust to distribution shifts and patch size compared to the PD approach, resulting in superior performance while more efficient both forecasting and classification tasks. We hope that work sheds light the effectiveness of self- self-supervised learning through simple pretraining tasks and model architectures in various domains, and provides a strong baseline to future works time series analysis.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
The proposed self-supervised learning algorithm, employing patch-independent strategies in terms of pretraining tasks and model architecture, holds the potential to have a significant impact in the field of representation learning for time series, especially in scenarios where annotation is searce or not available. This algorithm can be effectively applied in various real-world settings, encompassing both forecasting and classification tasks, even in situations where distribution shifts are severe. Furthermore, we foresee that the concept of utilizing lightweight architectures will serve source of inspiration for future endeavors across domains where substantial computational resources are not readily accessible.
Nevertheless, as is the case with any algorithm, ethical considerations come to the forefront. One notable ethical concern relates to the possibility of the algorithm perpetuating biases inherent in the pretraining datasets. It is necessary assess and mitigate potential biases within the pretraining dataset before deploying the algorithm real-world applications. To ensure the responsible utilization of the algorithm, we committed to providing the source code which will promote transparency and reproducibility, enabling fellow researchers to scrutinize and rectify potential biases and guard against any misuse.
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (2020R1 100R1220 A2CIA01005949
RS- RSS-203-000117005 the MSnT(Ministry of Science and ICT), Korea, under the ICAN(ICT Challenge and Advanced Network of HRD) support program (RS-2023-00259934) supervised by the IITP(Inttitue for Infor- mation & Communications Technology Planning Evaluation), the Yonsei University Research Fund (2023-22-0071), and the Son Jiho Research Grant of Yonsei University (2023-22-0006).
0022114410333
REFERENCES
Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian Elger. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64 (6):061907, 2001.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: general framework for self- self-supervised learning in speech, vision and language. In ICML, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. Tsmixer: An all-mip architecture for time series forecasting. TMLR, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In ECCV, pp. 108-124. Springer, 2022.
Mingyue Cheng, Qi Liu, Zhiding Liu, Hao Zhang, Rujiao Zhang, and Enhong Chen. Timemae: Self-supervised representations of time series with decoupled masked autoencoders. arXiv preprint 00Xi::2303300320, 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI, 2018.
Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jiammin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. In NeurlPS, 2023.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, In IJCAI, 2021.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee- Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for eeii-supervised time-series classification. TPAMI, 2023.
Haoyi Fan, Fengbin Zhang, and Yue Gao. Self-supervised time series representation learning by inter-intra relational reasoning. arXiv preprint arXiv:201 131448, 2020.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In NeurIPS, 2019.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence
embeddings. In EMNLP, 2021.
Spyros Gidaris and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung- -ang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals. Circulation, 101(23):e11-=222., 2000.
Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder. In ICLR, 2023.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Lu Han, Han-Jia Ye, and De-Chuan Zhan. The capacity and robustness trade-off: Revisiting the chan- nel independent strategy for multivariate time series forecasting. arXiv preprint arXiv:2304.05200, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Zhicheng Huang. Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. arXiv preprint arXiv:2207. 13532, 2022.
Ziyu Jiang, Yinpeng Chen, Mengchen Liu, Dongdong Chen, Xiyang Dai, Lu Yuan, Zicheng Liu, and Zhangyang Wang. Layer grafted pre-training: Bridging contrastive learning and masked image modeling for label-efficient representations. ICLR, 2023.
Bob Kemp, Aeilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. IEEE Transactions on Biomedical Engineering. 47(9): (899):185-11944 1194, 2000.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In ICLR, 2021.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference, volume 3. PHM Society, 2016.
11

---

Published as a conference paper at ICLR 2024
Jianglin Liang and Ruifang Liu. Stacked denoising autoencoder and dropout together to prevent overfitting in deep neural network. In CISP, 2015.
Yiwen Liao, Jochen Rivoir, Raphaël Latty, and Bin Yang. Deep feature selection using novel complementary feature mask. arXiv preprint AxKiv:2209.1288.0 2022.
Jun Liu, Lin Zhong, Jehan Wickramasuriya, and Vijay Vasudevan. Uwave: accelerometer- based personalized gesture recognition and its applications. Pervasive and Mobile Computing, 5(6): 657-675, 2009.
Yushan Nie, Nam H Nguyen, Pattarawat Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, 2016.
Pengxiang Shi, Wenwen Ye, and Zheng Qin. Self-supervised pre-training for time series classification. In IJCNN, 2021.
Iya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. NeurIPS, 2021.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In ICLR, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11). 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trent representations for time series forecasting. In NeurlPS, 2022.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07122 2022.
Kristoffer Wiekstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self- Sele-super representation learning for time series. Pattern Recognition Letters, 155:54 61, 2022.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-tres representations for time series forecasting. In ICLR, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with correlatooooiono for long-term series forecasting. In NeurIPS, 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
Xinyu Yang, Zhenguo Zhang, and Rongyi Cui. Timeclr: A self-supervised contrastive learning framework for univariate time series representation. Knowledge-Band -ased Systems,
2022.
Shaoxiong Ye, Jing Huang, and Lifu Zhu. Complementary mask self-supervised pre-training based on teacher-student network. In ACCTCS, pp. 199-206. IEEE, 2023.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang. Dian Li. Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. In ICLR, 2023.
Zhanwei Yue, Yiqun Wang, Jinghua Duan, Tao Yang, Chen Huang, Yunhai Tong, and Bo Xu. Ts2vec: Towards universal representation of time series. AAAI 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, 2023.
12

---

Published as a conference paper at ICLR 2024
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer- based framework for multivariate time series representation learning. In SIGKDD, 2021.
Xiaotian Zhang, Zeyu Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In NeurlPS, 2022.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, 2022.
13

---

Published as a conference paper at ICLR 2024
A DATASET DESCRIPTION
A.1 TIME SERIES FORECASTING
For time series forecasting, we assess the effectiveness of our proposed PITS using seven datasets, including four ETT datasets (ETTh1, ETTh2, ETTml, ETTm2), Weather, Traffic, and Electricity. These datasets have been widely employed for benchmarking and are publicly accessible (Wu et al., 2021). The statisties of these datasets summarized in Table A.1.
Datasets
ETThl
ETTh2
ETTml
ETTm2
Weather
Traffic
Electricity
Features
7
7
862 17544
321
Timesteps
17420
17420
69680
69680
52696
26304
Table A.1: Statistics of datasets for forecasting.
A.2 TIME SERIES CLASSIFICATION
For time series classification, we use five datasets of different characteristics, as described in Table A.2. Note that both SleepEEG and Epilepsy datasets belong to the same domain, characterized by being EEG datasets. For transfer learning tasks, we define them as being part of the same domain.
Dataset
# Samples
# Channels
# Classes
Length
Freq (Hz)
SleepEEG
371,055
1
5
200
100
Epilepsy
60 /0/ / 11,420
1
2
178
174
FD-B
60 21 / 13,559
1
3
5,120
64,000
Gesture
320 120 120
3
8
315
100
EMG
122 / 41 /4
1
3
1,500
4,000
Table A.2: Statistics of datasets for classification.
B EXPERIMENTAL SETTINGS
We follow the standard practice of splitting all datasets into training, validation, and test sets in chronological order (Wu et al., 2021). The splitting ratios were set at 6:2:2 for the ETT dataset and 7:1:2 for the other datasets. It is important to note that benefit from minimal hyperparameters due to our use of a simple architecture. We conduct hyperparameter search for three key parameters using the predefined validation dataset: the hidden dimension of the MI.P (D (32, 64, 128)), patch size (P E {12, 18, 24)), and input horizon (I. E 336, 512, 768). For self-supervised learning, we utilize a shared pretrained weight for all prediction horizons, making more efficient compared supervised learning in the long term.
In both self-supervised pretraining and supervised learning, we utilize an epoch size of 100. During fine-tuning in self-supend self-supervised learning, we apply linear probing for either 10 or 20 epochs, depending on the dataset, to update the model head. Subsequently, we perform end-to-end fine-tumin line-tuning of the entire network for twice the epoch duration of linear probing, following the approach outlined in PatchTST (Nie et al., 2023). The dropout ratio for the fully connected layer preceding the prediction head is set to 0.2.
14

---

Published as a conference paper at ICLR 2024
C HYPERPARAMETERS
C.I TIME SERIES FORECASTING
C.I.1 SELF-SUPERVISED LEARNING
Architecture
Epocls
Dataset
Dimension (D)
Patch size (P)
Number of patches (N)
Pretrain
Fine-tuning
Linear-probing
ETTh1
128
12
ETTh2
42
ETTml
100
ETTm2
20
Weather
128
24
30
32
Traffic
20
256
Electricity
32
30
C.1.2
SUPERVISED LEARNING
Architecture
Dataset
Epochs
Dimension (D)
Patch size (P)
Number of patches (N)
ETTh1
256
42
ETTh2
64
28
ETTml
24
100
ETTm2
128
Weather
42
Traffic
64
Electricity
32
C.1.3 TRANSFER LEARNING
Dataset
Epochs
Source
Target
Fine-tuning
Linear-probing
ETTh2
ETTh1
5
ETTm2
ETTml
20
10
ETTm2
ETThí
10
ETTh2
ETTml
5
ETTh2
ETTml
5
ETTml
ETThl
5
20
ETTh1
ETTml
5
Weather
ETThl
10
Weather ETTml
5
15

---

Published as a conference paper at ICLR 2024
C.2
TIME SERIES CLASSIFICATION
C.2.1 TRANSFER LEARNING
Datasct
Architecture
Epochs
Source
Target
Dimension (D)
Patch size (P)
Number of patches (N)
Aggregate
Pretrain
Fine-tuning
Epilepsy
64
8
22
max
20
150
SR
FD-B
128
8
22
avg
60
2000
Gesture
128
16
concat
20
100
EMG
max
100
3000
D TIME SERIES FORECASTING
To demonstrate the effecctiveness of PITS compared to other SOTA self-supervised methods, compare PITS with methods including PatchTST (Nie et al., 2023), SimMTM (Dong et al., 2023), TimeMAE (Cheng et al., 2023), TST (Zerveas al., 2021) MTM methods, and TF-C (Zhang et al., 2022), CoST (Woo et al., 2022), Ve2 (Yue et al., 2022) as CL methods. The results presented in Table D.I showcase the superior performance of PITS over these methods in multivariate time series forecasting task.
Models
Self supervised
PTT
WT.
PaccTTV'
SimMTM!
Tim/MAR
TNV
1:ST
TT-4"
C:ST
TNNNe
Merro
MSI
MAI:
MSB:
MAB
MSE
MAB
MSSE
MAB
MSE:
MAR
MSU
MAN
MSS
MAB
MSI:
MAE
MSS
MAB
MSE
MAG
0.413
0.428
0.403
0.425
0.551
0.635
0.500
0.757
336
0.630
0.435
0.466
0.155
0.630
0,713
01559
0.530
0.533
0.535
0.545
0.790
0.066
0996
720
0.425
0.452
0.444
0.45
0.465
0,474
0.430
0.43
0.736
0.618
0.768
0.628
0.432
0.42
0.577
0.562
0.680
0.799
0.957
0.790
06
1.260
0,333
0.269
0.34
0.305
0.351
0.288
0347
0.443
0.665
0.335
0.392
0.331
0.500
0.463
0.511
0.665
0.422
0.566
0.477
192
0.329
0.571
0.333
11315
0.361
0.392
0.46
0385
0.53
0536
0.484
0.441
0.451
0.531
0.555
11581
0.671
0.599
0.577
0.577
336
0.363
0.27
0.363
0-445
0.45
0.850
0.885
0.548
0.776
0.994
0.625
720
0.19
0.466
0.266
0.409
0,930
11232
0.771
0.728
ET1o2 6TTal
192
96
1.331
0.373
0.354
0.338
0.300
131
033
13
0.35
131
0.59
6369
0.30
0.900
0.906
0.411
0
0.340
0.566
0.355
0.411
0.001
0238
0.430
0.576
0.451
0.420
0.500
0.555
0.558
0.551
336
0.359
0.385
03655
030
0364
0.300
300
0335
0.659
0.525
0.57
0,511
0.429
0.407
0.540
0.509
0.482
0.494
0.655
0.594
720
113000
0.414
10430
181
0.00
03415
0411
0.76
03590
0.544
07418
0.456
0.464
0.353
11.48
0.676
0.575
0331
0.895
96
1165
0,200
0,100
0255
0,00
0:355
0.314
0.401
0.477
0.276
0:764
0.44
192
0113
1270
0232
030
022
0.423
0
0,500
0.545
0.536
310
0.39
1335
0365
0.300
0.45]
0.513
11.0%
0.837
Weatee
96
0.151
0300
01.11
0205
6.145
6.1
0111
0302
0.316
0.250
0.292
0.370
0.153
0.211
0.215
0.256
0.327
0.350
0.43
0.462
152
11995
0201
(220)
1247
0:38
1225
035
0.30
0.315
0,110
0,473
0.307
0.250
0.267
11545
0
0.422
0.56
0.515
336
0260
11245
1233
0225
0.26
0383
0:351
0356
0.444
0,427
0.349
0.364
0.299
0.360
0.477
0.446
0.545
0.549
720
01116
0,308
11111
1.300
0.30
0.30
0.300
003
0.425
0.999
0,539
0.323
0319
0.330
0.361
11395
0.551
0.396
0.576
0.573
Truffie
0.70
0.385
0,613
0.424
0.367
316
0.408
0217
0.90
0.29
0.355
0.254
0.394
0.765
01897
0.761
0.425
0.899
0,376
730
0.445
0.10
0.290
0.780
0.433
0155
0.900
Dlesmaty
96
8.130
0.255
0.131
0.26
0.126
0.221
0.133
0.21
0.399
0412
0.292
0.370
0.166
0.544
0.366
0.436
0.230
0.353
0.322
0.401
192
R.144
0.240
0.145
02200
0.145
0.258
0.147
0:237
0.400
01650
0,270
0.373
0.178
0.278
0.366
0.435
0.253
0.371
0.343
0.416
316
IL1681
0.256
101
01226
0.164
0.256
0.100
0.765
0.364
01775
0.334
0.323
116
0.375
0.154
11878
0.197
0.387
0.932
0.635
720
0.201
0.200
0.200
0.200
0.203
0.297
0.550
0.288
0,363
0.431
0.230
0.328
0.358
0.456
0.314
0.353
0.306
0.365
0.501
0.560
0.318
0.593
0.565
We used the official code to replicate the results.
SimMTM is concurrent work to ours.
Table D.1: Results of multivariate TSF with self-supervised methods. The best results are in bold and the second best are underlined.
16

---

Published as a conference paper at ICLR 2024
E TRANSFER LEARNING
For time series forecasting under transfer learning. we consider both in-domain and cross-domain transfer learning settings, where we consider datasets with same frequency as in-domain. We perform transfer learning in both in-domain and cross- doms-n using five datasets: four ETT datasests and Weather. The full results are described in Table E. where missing values are not reported in literature.
TeIREST
100
638
0210
6390
8.63
14.9
1.01
6455
043
0014
9414
3445
8.67
8.60
E.05
6-13
E314
E331
666
0461
088
3227
0.56
8.38
146
6-47
3.46
0.50
195
8.45
E45
680
653
60M
178
1.07
6.43
0911
3200
135
1.43
101
0331
0.70
3336
1:99
1340
8.59
101
039
038
0.55
8.70
0.87
1.50
1965
8:43
(409
6-04
633
6311
0634
050
1283
1.51
1-01
088
6415
0358
147
9450
3-31
0.99
1465
8.52
630
679
076
9M6
1.25
041
1:45
C39
953
103
0,21
8.43
6.40
3447
130
6:05
0.23
0414
053
0731
1400
8.5
6-60
181
046
083
060
5,80
8.30
1.61
1-41
6.62
0.03
646
023
3476
0.99
1.64
1.66
L.37
1221
080
638
600
CTTR
1333
0376
3136
9.36
8.333
1001
6.44
0375
0354
6386
037)
036
1718
0.56
1.51
1471
8-45
L-07
6545
631
0358
030
035
055
02215
1-34
6712
0F3
Croor Reaaa
STTbal
100
1431
0441
3480
17:31
3417
1.01
6:43
6411
048
6463
0553
1.70
3/01
6729
1.87
1431
L.11
6311
635
064
637
083
066
1321
0961
8.36
1.41
1.50
0.57
194
1.40
1.40
070
1m
083
1.25
030
ETI
1339
0.776
1136
0.56
1131
1.51
6:37
039
0398
038
1377
3370
0.51
1-30
1.65
8.06
135
635
(30
041
036
mTbal
0216
3434
145
646
042
3421
146
8-07
1.45
E37
EN8
0MH
Weaker
100
0,303
3,61
3.30
1485
1.00
1.45
0.5
6422
0311
0.7
0451
3355
150
135
8431
3136
1.50
1.6
6313
0374
035
0388
3387
2230
25.5
Table E.1: Results of multivariate TS forecasting with transfer learning. We conduct experiments under two settings: (1) in-domain and (2) cross-domain transfer. The best results are in bold and the second best are underlined.
17

---

Published as a conference paper at ICLR 2024
F COMPARISON WITH PATCHTST
We compare our proposed method with PatchTST three versions: 1) fine-tuning (FT), linear probing (LP), and supervised learning (SL). The results are described in Table F.I, which demonstrates that our proposed method outperforms PatchTST in every version in most of the datasets.
PITS
PatchTST
Models
FT
LP
SL
FT
LP
SL
Metric
MSE
MAF
MSR
MAE
MSE
MAE
MSE
MAE
MSF
MAE
MSE
MAE
96
0.367
0.393
0.366
0.392
0.369
0.397
0.379
0.408
0.382
0.410
0.375
0.399
192
0.401
0.416
0.398
0.414
0.403
0.416
0.414
0.428
0.433
0.441
0.414
0.421
336
0.415
0.428
0.419
0.427
0.409
0.426
0.435
0.446
0.439
0.446
0.431
0.436
720
0.425
0.452
0.430
0.454
0.456
0.465
0.468
0.474
0.482
0.482
0.449
0.466
avg
0.401
0,421
0.403
0.422
0.409
0,426
0.424
0.439
0.434
0.445
0.417
0.431
96
0.269
0.333
0.269
0,333
0.281
0.343
0.306
0.351
0.299
0.350
0.274
0.336
ETTh2
192
0.329
0.371
0.331
0.373
0.345
0,383
0,361
0.392
0.363
0.394
0,339
0.379
336
0.356
0.397
0.352
0.395
0.334
0.389
0.405
0.427
0,386
0,417
0.331
0,380
720
0.383
0.425
0.383
0.425
0.389
0.430
0,419
0.446
0.409
0,440
0.379
0.422
avg
0.334
0.382
0.335
0.381
0.337
0.386
0.373
0.404
0.364
0,400
0.331
0.379
96
0.294
0.354
0.307
0.349
0.296
0.346
0.294
0.345
0.296
0.349
0,290
0.342
ETTml
192
0.321
0.373
0.337
0,368
0.330
0.369
0.327
0.369
0.333
0.370
0.332
0.369
336
0.359
0.388
0.365
0.389
0.360
0.388
0.364
0.390
0.368
0.390
0,366
0.392
720
0.396
0.414
0.415
0.412
0.416
0.421
0.409
0.415
0.422
0.418
0.420
0.424
avg
0.342
0.381
0.356
0.379
0.351
0
0.349
0,380
0.355
0,382
0.352
0.382
96
0.165
0.260
0.160
0.252
0.163
0.255
0.167
0.256
0.168
0.257
0.165
0.255
ET7m2
192
0.213
0.291
0,214
0,289
0.215
0,293
0.232
0.302
0.231
0.302
0.220
0.292
336
0.263
0.325
0.263
0,324
0.266
0.329
0.291
0.342
0.290
0.341
0,278
0.329
720
0.337
0.373
0.342
0.376
0.342
0,380
0.368
0.390
0.366
0,387
0.367
0.385
avg
0.244
0.310
0,244
0.310
0,247
0,314
0.264
0.323
0.264
0.322
0,258
0.315
96
0.151
0.201
0.167
0.222
0.154
0.202
0.146
0.194
0.160
0.211
0.152
0.199
Weather
192
0.195
0.242
0.211
0.259
0.191
0,242
0.192
0.238
0.248
0.197
0.243
336
0.244
0,280
0.256
0.293
0.245
0,280
0.245
0,280
0.251
0.285
0,249
0.283
720
0.314
0.330
0.319
0.338
0.309
0.330
0.320
0.336
0.319
0.334
0,320
0.335
avg
0.225
0.262
0,239
0.278
0.225
0.263
0.226
0,262
0.233
0,269
0,230
0.265
96
0.372
0.258
0.384
0.266
0,375
0,264
0.393
0.275
0.399
0,294
0.367
0.251
Traffic
192
0.396
0.271
0.395
0.270
0.389
0.270
0.376
0.254
0.412
0.298
0.385
0.259
336
0.411
0.280
0.409
0.276
0,401
0.277
0.384
0.259
0.425
0.306
0,398
0.265
720
0.436
0.290
0.438
0.295
0.437
0,294
0.446
0.306
0.460
0.323
0.434
0,287
avg
0.403
0.271
0.406
0.267
0,401
0.276
0.400
0.274
0.424
0.305
0.396
0.266
Electricity
96
0.130
0.225
0.132
0.228
0.132
0.228
0.126
0.221
0,138
0.237
0,130
0.222
192
0.144
0.240
0.147
0.242
0.147
0.242
0.145
0.238
0.156
0.252
0.148
0.240
336
0.160
0.256
0.163
0.258
0.162
0.261
0.164
0.256
0.265
0.167
0.261
720
0.197
0,290
0.201
0.290
0.199
0.290
0.200
0.290
0.208
0.297
0,202
0.291
avg
0.157
0.253
0.161
0.254
0.160
0.256
0.159
0.251
0.168
0.263
0.162
0.254
Average
0.301
0.327
0.306
0.328
0.304
0.329
0.314
0.333
0.320
0.341
0.307
0.327
Table F.1: PITS vs. PatchTST in multivariate time series forecasting.
18

---

Published as a conference paper at ICLR 2024
G EFFECTIVENESS OF PI TASK AND CONTRASTIVE LEARNING
To assess the effectiveness the proposed patch reconstruction task and complementary contrastive learning, we conduct ablation studies in both time series forecasting and time series classification.
G.1 TIME SERIES FORECASTING
To examine the effect of PI task and CL forecasting, conduct experiment using four ETT datasets. The results in Table G.1 demonstrate that performing CL with the representation obtained from the first layer and PI with the one from the second layer gives the best performance.
Layer
PI
CL
I Layer
CL
Pl
CL+PI
CL.
Pl
96
0.715
0.367
0.372
0.381
0.367
ETTh1
192
0.720
0.400
0.409
0.416
0.401
336
0.719
0.426
0.422
0.462
0.415
720
0.727
0.443
0.465
0.509
0.425
avg
0.720
0.409
0.417
0.442
0.401
96
0.373
0.270
0.307
0.303
0.269
ETTh2
192
0.384
0.331
0.362
0.373
0.329
336
0.386
0.361
0.387
0.391
0.356
720
0.432
0.384
0.408
0,416
0.383
avg
0.394
0.336
0.366
0.371
0.334
96
0.693
0.305
0.302
0.300
0.294
ETTml
192
0.702
0.335
0.337
0.336
0.321
336
0.716
0.366
0.365
0.369
0.359
720
0.731
0.413
0.413
0.426
0.396
avg
0.711
0.355
0.356
0.358
0.342
96
0.346
0.160
0.167
0.171
0.165
ETTm2
192
0.368
0.215
0.225
0.235
0.213
336
0.397
0.266
0.274
0,278
0.263
720
0.424
0.346
0.351
0.376
0.337
avg
0.381
0.247
0.254
0.265
0.244
Total avg
0.552
0.337
0.348
0.359
0.330
Table G.1: Effect of PI task and CL on time series forecasting.
G.2 TIME SERIES CLASSIFICATION
To evaluate the impact of employing CL and PI on classification, we conducted an experiment using the Epilepsy dataset. The results presented in Table G.2 demonstrate that long PI task is employed, the performance is robust to the design choices.
Layer
Pl
CL.
Layer
CL
PI
CL+PI
CL
PI
SleepEEG
ACC.
91.61
95.27
95.67
95.67
95.67
PRE.
92.11
95.35
95.63
95.70
95.63
REC..
91.61
95.27
95.66
95.66
95.67
Fl.. 91.79
95.30
95.68
95.68
95.64
Table G.2: Effect of PI task and CL on time series classification.
19

---

Published as a conference paper at ICLR 2024
H EFFECTIVENESS OF PI STRATEGIES
In this experiment, we investigate the impact of our proposed PI strategies from two perspectives: 1) the pretraining task and 2) the encoder architecture. The results, shown in Table H.l, encompass four ETT datasets with four different forecasting horizons with a common input horizon of 512. These results demonstrate that the PI task consistently outperforms the conventional PD task across all considered architectures.
Architecture
PI
PD
Linear
MLP
MLPMixer
Transtormer
Task
PD
Pl
PD
PD
Pl
PD
Pl
96
0.366
0.365
0.375
0.366
0.378
0.368
0.371
0.372
ETThl
192
0.398
0.398
0.407
0.397
0,414
0.399
0.410
0.404
336
0,423
0.424
0.427
0.427
0.422
0,427
0.443
0.434
720
0.444
0.444
0.463
0.440
0.465
0.440
0.475
0.452
avg
0.408
0.408
0.418
0.407
0.420
0.409
0.425
0.415
96
0.272
0.270
0.290
0.270
0.301
0.276
0.283
0.271
ETT12
192
0.332
0,333
0.361
0.329
0.353
0.334
0.351
0,332
336
0.370
0.364
0.373
0.353
0.394
0.363
0.378
0,369
720
0.396
0.385
0.418
0.384
0.411
0.389
0.400
0.395
avg
0.343
0.338
0.361
0.334
0.365
0.341
0.353
0.342
96
0.304
0,304
0.298
0.302
0.294
0.296
0.294
0.297
ETTml
192
0.337
0.338
0.341
0.337
0.332
0.334
0.335
0.336
336
0,370
0,368
0.368
0.363
0.364
0.363
0.365
0,359
720
0.423
0.423
0,416
0.420
0.418
0,416
0.405
0.403
avg
0.359
0.358
0.356
0.355
0.354
0.352
0,350
0.350
96
0.163
0.163
0.169
0.164
0.170
0.164
0.172
0.172
ETTm2
192
0.219
0.218
0.224
0.218
0.226
0.218
0.240
0,221
336
0.272
0.271
0.275
0.271
0.276
0.272
0.300
0.274
720
0,362
0.361
0.363
0.359
0.361
0.359
0,383
0.356
avg
0.254
0.253
0.258
0.253
0.259
0.253
0.274
0.256
Total avg
0.341
0.339
0.348
0.337
0.350
0.339
0.351
0.341
Table H.1: Effectiveness of PI tasks and PI architectures.
I ROBUSTNESS TO PATCH SIZE
To evaluate the robustness of encoder architectures to patch size, we compare MLP and Transformer with different patch sizes with ETTh2 and ETTm2 with common input horizon of 512. The left and the right panel of Figure 1.1 illustrate the average MSE of four horizons of ETTh2 and ETTm2,
respectively.
Dataset: ETTh2
Dataset: ETTm2
MIP Pl
0.280
MLP task
Avg. MSE (4 harizons)
0.37
MLP PD task
Trans + H task
Avg. MSE (4 horizonc)
0.275
PO
0.270
Toon FD task
0.36
0.265
0.35
0.260
0.34
0.255
12
12
16
2/1
32
12
12
16
24
32
Patch Size
Patch Size
Figure I.l: Robustness of PI task to patch size.
20

---

Published as a conference paper at ICLR 2024
J EFFICIENCY OF PITS IN SELF-SUPERVISED AND SUPERVISED SETTINGS
We compare the efficiency of PITS between self supervised and supervised settings on the ETTm2 dataset. We calculate the pretraining time and fine-tuning time of PITS under the self-supervised setting, well as the training time under the supervised setting. Table J.1 presents the results, with the time required for fine-tuning (in the self-supervised setting) and supervised training across four different horizons {96, 192, 336, 720). We used an epoch size of for both pretraining in self-supervised settings and training in supervised settings. For fine-tuning, we trained linear head for 10 epochs, followed by end-to-end fine-tuning the entire network for an additional 20 epochs, following PatchTST. For self- supervieee learning, utilize shared pretrained weight for all prediction horizons, enhancing efficiency over the long-term setting compared to supervised learning. Given that pretraining is done before training downstream tasks, fine-tuning the pretrained model more efficient than training from scratch, while providing better performance.
PITS
Self-supervised (w/ hier. CL)
Supervised
Pretrain
Fine-tune
Train
Horizon
96
192
336
720
96
192
336
720
Time (min)
16
1.2
1.4
1.5
1.6
4.2
4.3
5.3
6.9
Avg. MSE
0.244
0.255
Table J.1: Comparison of training time under self-supervised and supervised settings.
K PERFORMANCE BY DROPOUT RATE
Figure K.1 displays the average MSE across four horizons, and Table K. lists all the MSE values for four ETT datasets trained with MIP of D = 32 at various dropout rates with a common input horizon of 512. These results emphasize the importance of incorporating dropout during the pretraining phase of the reconstruction task, as it helps prevent trivial solutions when the hidden dimension is greater than the input dimension.
Dropout Rate
0.348
Dropout rate
ETThl
ETTh2
ETTml
ETTm2
Avg.
0.345
0.0
0.418
0.359
0.359
0.257
0.348
0.1
0.410
0.334
0.358
0.253
0.339
0.2
0.407
0.334
0.357
0.253
0.338
0.338
0.338
0.338
0.338
0.3
0.407
0.333
0.357
0.253
0.338
0.4
0.407
0.334
0.356
0.253
0.338
0.335
0.5
0.406
0.335
0.356
0.253
0.337
0.0
0.1
0.2
0.3
0.4
0.5
Table K.1: MSE by dropout.
Figure K.I: Avg. MSE by dropout.
L PERFORMANCE OF VARIOUS PRETRAIN TASKS
To see if the conventional PD task of reconstructing the masked patches (X,)) with the unmasked patches (X,,) appropriate TS representation learning, we employ two other simple pretraining tasks of 1) predicting X,, with zero-value patches (0) and 2) reconstructing O with themselves. Table L.1 presents the results for four ETT datasets with a common input horizon of 512 across three different architectures: Transformer, MLP without CL, and MLP with CL. These results underscore that models pretarined with PD task performs even worse than the two basic pretraining tasks with velue patch inputs, highlighting the ineffectiveness of the PI task and emphasizing the importance of the proposed Pl task.
21

---

Published as a conference paper at ICLR 2024
TranSoemmrr
Input
ETTml
ETTm2
ETTB2
ETTaat
0.415
0.355
0.253
0.331
0.341
0.329
0.425
0.376
0.362
0.261
0.364
0.410
0.354
0.256
0.256
0.356
0.353
0,353
Table L.1: Performance of various pretraining tasks.
M STATISTICS OF RESULTS OVER MULTIPLE RUNS
To see if the performance of PITS is consistent, show the statistics of results with five different random seeds. We compute the mean and standard deviation of both MSE and MAE, as shown in Table M. The results indicate that the performance of PITS is consistent for both under self- supervised and supervised settings.
Models
Self-supervised
Supervised
Metric
MSE
MAE
MSE
MAE
96
00.03
0.393
0.369=0.0011
0.397 10.0017
192
0.401 20.01=0.0008 0005
0.416 0008
0.403
0.41640.000 0000
336
0.415
0.428 0010
0.409=0.0002
0.426 00011
720
0.425 1000077
0.452 10.0045
0.446_00.0010
0.465
96
0.269 2.2299000003 0013
0.333 0004
0.281=000009
0.343 0033
ETTh2
192
0.329 10.0007
0.371 1.000115
0.345_..o0m
0.383 1000000
336
0.356 000001
0.397 0.00010 0010
0..04_00.0019
0.389 1000017
720
0.383
0.425 0005
0.389
0.38990.00033 0.0003
0.430
96
0.294_0.000z 0022
0.354
0.296 200011
0.346-+0.0005
ETTml
192
0.321 0.321 0099
0.373
-..300000009
0.369
336
0.359 0.359__o.000 0000029
0.383
0.360
1..6001.00005
0.388
720
0.396_0.0081
0.41440.0060
0.416_0.0009
0.421 10.0010
96
0. 0.165=0.0017
0. 0.260
0.163
0. 0.255 0.0001
ETTm2
192
0.213 10.0009
0.291 10.0011
0.215_1.0005
0.293 0.293-10.004
336
0.0002 0.263
0.325 1322510.000
0.266
0.32910.0013
720
0..07=0.0013
0.373+0.0000
0.342=0.0002
0.380
Weather
96
20.015
0.201 10.007
0.104
0.20240.0000
192
0.195
0.242 0009
0.191 0.191_0.00115
0.242
336
0.244
0.280400.0017
0.245=0.0009
0.280
720
0.314_0.0016
0.330
0.309
0.33040.0006
96
0.372 0..04
0.2584n.0033
0.305_0.0003
0.26441.0002
Traffic
192
0.396 10.0001
0.271 10.0002
0.389
0.389_o..0002
0.270
336
0.411 0.411 20.0041
0.280 0030
0.401 =0.000
0.277 +0.000 0001
720
0.436
0. 0.290 0057
0.437=0.0m
0.2 0.29440.00 0004
Electricity
96
0.130
0.0003
0.225 10.0003
0.132 111322..oom 0000010
0.228 0.228 10.0011
192
0. 0.144=o.. 0008
0.240 0007
0.147 = 0.0000
0.242
336
0.160 11.60=0.000 0005
0.256 0006
0.162 0000000000 0008
0.26140.000
720
0.194
0.287440.00 0002
0.199
0. 0.299440.0022
Table M.I: Results of PITS on multivariate TSF over five runs.
22


Reviews:
[{"review_id": "QbWF4ZRR8W", "reviewer": "Reviewer_ZBCM", "summary": "Authors propose a mix of contrastive learning and masked modelling on time-series data, and a patch-independent approach to reconstruction is shown to outperform patch dependant (vanilla mask modelling) approach. Their PI approach auto encodes unmanned patches. Authors evaluate on forecasting and classification of time-series data.", "questions": "Figure 5 is not as clear what it is conveying? What is Fig. 5 Left showing in terms of colours? Fig. 5 Right it is a bit confusing since it is PD-PI and x & y axis represent the slop and amplitude of diff between training and test phases, It is not clear to me what is exactly being changed to have the distribution shift in the time-series data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Extensive experiments comparing to other methods and ablating different components. \n- Incremental but shown to improve approach with simple mechanisms.\n- Interesting analysis on distribution shift", "weaknesses": "- Novelty is weak, their contribution auto encoding is quite established before masked modelling literature and one of the early approaches to representation learning. It is more that exploring this within time-series data which seems to be there contribution. Also mixing of CL and masked modelling has been explored in other methods but not exactly similar to their approach.\n\n\n- Missing reference to previous method that Combine CL and MAE but in a different context and different method: Gong, Yuan, et al. \"Contrastive audio-visual masked autoencoder.\" arXiv preprint arXiv:2210.07839 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xcTp1qS2Ft", "reviewer": "Reviewer_DsGC", "summary": "This paper tackles the task of time series forecasting using masked modeling. The authors propose a patch reconstruction task to facilitate training. The patches are treated independently in the patch reconstruction task. A complimentary contrastive learning, where two views with a 50% masking ratio are used for CL, is utilized. Experiments on the common forecasting benchmark with 7 datasets, and the classification benchmark with 5 datasets, show the efficacy of the proposed method.", "questions": "1. What is the input horizon (look-back window/length of historic sequence) for the TSF task (Table 3)? This detail is missing.\n2. Are the numbers in Table 3 averaged over multiple runs (with different random seeds) or are they single runs only?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow. The method seems straightforward to implement.\n2. The experiments are thorough. The proposed method is evaluated on two tasks with a total of 12 datasets. Also, the transfer learning setting is explored. The key PI vs. PD task is analyzed through quantitative and qualitative experiments.", "weaknesses": "1. The training and inference efficiency analyses are brief or missing. It would be useful to see whether the patch-independent design can also bring benefits to inference time. The model requires contrastive learning and reconstruction loss, which might drastically increase training time compared to other supervised learning methods. Therefore it would be useful to see it compared to supervised learning methods as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Az1yZ7TmBv", "reviewer": "Reviewer_aEzH", "summary": "The paper proposes a new approach called PITS (Patch Independence for Time Series) for self-supervised representation learning of time series data. The authors argue that previous methods that capture patch dependencies may not be optimal and instead propose learning to embed patches independently. They introduce the \"Patch-Independence\"-based MTM (Masked Time Series Modeling) method and complementary contrastive learning. They demonstrate that using an MLP as the PI architecture in MTM can independently reconstruct unmasked patches while disregarding interactions between patches, thereby improving efficiency while reducing the parameter count. Simultaneously, it has been shown that introducing complementary contrastive learning effectively captures information from adjacent time series. The model has been tested on two downstream tasks: time series forecasting and classification, demonstrating significant advantages in terms of performance, architectural interpretability, and robustness.", "questions": "1.\tCould you provide a more detailed explanation for why the PI architecture is superior to the PD architecture? The results of the comparative experiments are quite significant, but we believe that adding some qualitative analysis would make the argument more convincing.\n2.\tWould it be possible to offer a more comprehensive explanation of how the proposed method is implemented, including details on hyperparameter configurations? We would appreciate seeing more comparative experimental results to further validate the rationale for these selections.\n3.\tHave you conducted comparisons between the proposed method and alternative baselines, such as autoencoders or other self-supervised learning approaches tailored for time series data?\n4.\tHave you taken into account the limitations of the proposed model? Could you elaborate on potential challenges or specific scenarios in which the method might exhibit suboptimal performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Originality: The paper introduces a novel approach to self-supervised time series representation learning, with a focus on patch independence. This represents a unique perspective challenging the traditional methods of capturing patch dependencies. Additionally, it is the first model to integrate MTM and CL, showcasing the innovation of the implementation while improving model performance.\n2) Quality: The paper is well-written and provides a comprehensive analysis of the proposed method. The experimental design is thorough, including comparisons with state-of-the-art methods, and it demonstrates strong performance in two downstream tasks. Furthermore, the completeness of the ablation experiments illustrates attention to various model details.\n3) Clarity: The paper is clear and well-organized. The authors provide a clear motivation for their work, explain the proposed method in detail, and present the experimental results in a clear and concise manner.\n4) Significance: The proposed method achieves superior performance compared to existing approaches in time series forecasting and classification tasks. The efficiency of the method in terms of parameters and training time is also highlighted, which makes it more practical for real-world applications.", "weaknesses": "A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence: if you believe experiments are insufficient, explain why and exactly what is missing, etc.\n1.\tThe representation in Figure 1(a) should be clearer and correspond one-to-one with the description in Section 3.2. For example, the inconsistency between \"MLPmixer\" in the figure and \"MLP-Mixer\" in the text should be rectified, and the explanations for both in the PI architecture need to be more explicit. In general, a more comprehensive explanation for this figure is needed, and citation markers may be introduced as necessary.\n2.\tThe two layers described in Figure 1(b) should be clearly labeled, corresponding to one for the CL task and one for the PI task, as indicated in the caption.\n3.\tThe notation \"No(SL)\" in Table 1 should be explained in the description, specifically in Section 2.\n4.\tOne of the contributions is the use of self-supervised mask reconstruction, but for the TSF task, only two baseline models using self-supervised mask reconstruction are provided. The evaluation section could benefit from more comparisons with other baselines that use self-supervised mask reconstruction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4704N4dGvh", "reviewer": "Reviewer_oFTu", "summary": "The key idea of the paper \"Learning to Embed Time Series Patches Independently\" is to propose a new approach for time series representation learning by focusing on embedding patches independently rather than capturing patch dependencies. The authors introduce a patch reconstruction task that autoencodes each patch without considering other patches and a patch-wise MLP architecture for embedding. They also incorporate complementary contrastive learning to efficiently capture adjacent time series information. Experimental results demonstrate that their proposed method outperforms state-of-the-art Transformer-based models in time series forecasting and classification tasks while being more efficient in terms of parameters and training time.", "questions": "Please refer to the weakness part. Further, in Table 2, does 46 mean 46 hour of training time?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) This paper is easy to follow.\n2) The paper introduces the concept of patch independence for time series representation learning, arguing that learning to embed patches independently leads to better representations. This novel approach challenges the conventional strategy of capturing patch dependencies and proposes a patch reconstruction task and a patch-wise MLP architecture.\n3) The paper presents extensive experiments on various tasks, including low-level and high-level tasks, demonstrating the superiority of the proposed method compared to state-of-the-art  models. The experiments are conducted under both standard and transfer learning settings, further validating the effectiveness of the approach (a significant performance gain is reported).", "weaknesses": "1) Despite the author's cautionary language, using only 'argue' rather than 'claim', 'learning to embed time series patches independently is superior to learning them dependently for TS representation learning' still sounds a bit lax to me. Especially considering the lack of substantial experimental evidence to support such an argument. (Comparison among representative methods cannot support the argument ‘A is superior to B when A,B refer to a set of methods’)\n\n2) Given that you’ve mentioned the complementary masked strategy for CL and claim it as a ‘main contribution’, it should be investigated in related work. (Such strategy has already been proposed earlier, though in other domain. Such as Ref 1,2,3 provided below) \n\n[1] SdAE: Self-distillated Masked Autoencoder\n\n[2] Complementary Mask Self-Supervised Pre-training Based on Teacher-Student Network\n\n[3] Deep Feature Selection Using a Novel Complementary Feature Mask\n\n3) Lack of clarity in exposition (for certain sentences): The paper should strive to provide clear and concise explanations. For example, “Secondly, we propose to utilize the simple PI model architecture (e.g., MLP), as opposed to the conventional PD architecture (e.g., Transformer), which is not only more efficient but also performs better.”  What does which refer to in this sentence?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)