A Time Series is Worth 64 Words:  Long-term Forecasting with Transformers
Authors: Yuqi Nie
Nam H Nguyen
Phanwadee Sinthong
Jayant Kalagnanam
Venue: ICLR 2023 poster (2023)
Keywords: time series
transformer
forecasting
channel-independence
self-supervised learning
representation learning

Abstract: We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-training performed on one dataset to other datasets also produces SOTA forecasting accuracy.

Published as a conference paper at ICLR 2023
A TIME SERIES IS WORTH 64 WORDS:
LONG- FORECASTING WITH TRANSFORMERS
Yuqi ; Nam H. Nguyen' ; Phanwadee Sinthong", Jayant Kalagnanam"
Princeton University IBBM Research
ymie@princeton.ee edu, nnguyenêus, ibm. com, bayant@us com com
Gift. Sift.niinthngeeibm.cc
ABSTRACT
We propose efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. is based on key components: (i) segmentation of time series into subseries-leve patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains single univariate time series that shares the same embed- ding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information retained the embedding: computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independen patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that SOTA Transformer-based models. We also apply our model to self-supervised pre- training tasks and attain excellent fine-tuning performance, which outperforms supervised training large datasets. Transferring of masked pre- pre-trained repre- sentation on one dataset to others also produces SOTA forecasting accuracy.
INTRODUCTION
Forecasting is one of the most important tasks in time series analysis. With the rapid growth of deep learning models, the number of research works has increased significantly on this topic (Bryan & Stefan 2021 Torres et al. 2021 Lara-Beniter et al. 2021). Deep models have shown excellent performance not only on forecasting tasks, but also on representation learning where abstract rep- resentation can be extracted and transferred to various downstream tasks such as classification and anomaly detection to attain state-of-the-ar performance.
Among deep learning models, Transformer has achieved great success various application fields such as natural language processing (NLP) (Kalyan et al. 202I). computer vision (CV) (Khan al., 202I), speech (Karita et al. 2019), and more recently time series (Wen et al. 2022 benefiting from its attention mechanism which can automatically learn the connections between elements in a se- quence, thus becomes ideal for sequential modeling tasks. Informer (Zhou al. (021). Autoformer (Wu al. 2021), and FEDformer (Zhou et al. 2022) are among the best variants of the Transformer model successfully applying to time series data. Unfortunately, regardless of the complicated design of Transformer based models, it is shown in the recent paper (Zeng et al. (022)) that very simple linear model can outperform all of the previous models on variety of common benchmarks and challenges the usefulness of Transformer for time series forecasting. In this paper, we attempt to an- swer this question by proposing a -indeeeeeeedeeeeeenneee patch time series Transformer (PatchTST) model that contains two key designs:
Patching. Time series forecasting aims understand the correlation between data in each dif- ferent time steps. However, single time step does not have semantic meaning like a word in sentence, thus extracting local semantic information essential in analyzing their connections. Most of the previous works only use point-wise input tokens, just handcrafted information
"qqual contribution.

---

Published as a conference paper at ICLR 2023
Models
L
N
patch
methed
MSE
Running time (s) with & 338i
Channel- independent
380
96
down-sampled
0.447
Datasct
w. patch
w.o. patch
Gain
PatchTST
336
-12
0.367
Traflic
464
10040
22
336
self-supervised
0,349
Fllctricity
300
5730
x 19
Weather
156
680
x
Channel-miaee FliDFormerr
336
336
0.597
innar
336
336
Table 1: A case study of multivariate time series forecasting on Traffic dataset. The prediction hori- zon is 96. Results with different look-back window L and number of input tokens N are reported. The best result in bold and the second best is underlined. Down-sampled means sampling every 4 step and adding the last value. All the results are from supervised training except the best result which uses self-supervised learning.
from series. In contrast, we enhance the locality and capture comprehensive semantic informa- tion that is not available in point-level by aggregating time steps into subseries-level patches.
Shaamel-independennce A multivariate time series is a multi-channell signal, and each Trans- former input token be represented by data from either single channel or multiple channels. Depending on the design of input tokens, different variants of the Transformer architecture have been proposed. tor of all time series features and projects it to the embedding space to mix information. On the other hand, channel-independence means that each input token only contains information from single channel. This was proven to work well with CNN (Zheng et al. 2014) and linear models (Zng et al. 2022). but hasn't 't been applied to Transformer-bast models yet.
Sttannel-mixing refers the latter case where the input token takes the vec-
We offer snapshot of our key results in Table by doing case study on Traffic dataset, which consists of 862 time series. Our model has several advantages:
1. Reduction on time and space complexity: The original Transformer has O(N2) complexity on both time and space, where N is the number of input tokens. Without pre-processing N will have the same value as input sequence length L. which becomes a primary bottleneck of computation time and memory in practice. By applying patching, we can reduce N by factor of the stride: N Rs L/S, thus reducing the complexity quadratically. I Illustraee the usefulness of patching. By setting patch length P 16 and stride S 8 with L 336, the training time is significantly reduced as much as 22 time on large datasets.
Capability of learning from longer look-back window: Table [ shows that by increasing look- back window L from 96 to 336, MSE can be reduced from 0.518 to 0.397. However, simply extending L comes at the cost of larger memory and computational usage. Since time series often carries heavily temporal redundant information, some previous work tried to ignore parts of data points by using downsampling or a carefully designed sparse connection of attention (Li al. 2019) and the model still yields sufficient information to forecast well. We study the case when L = 380 and the time series is sampled every 4 steps with the last point added to sequence, resulting in the number of input tokens being N 96. The model achieves better MSE score (0.447) than using the data sequence containing the most recent 96 time steps (0.518), indicating that longer look-back window conveys more important information even with the same number of input tokens. This leads us to think of a question: is there a way to avoid throwing values while maintaining long look-back window? Patching is a good answer to it. It can group local time steps that may contain similar values while at the same time enables the model to reduce the input token length for computational benefit. As evident in Table MSE score is further reduced from 0.397 to 0.367 with patching when L = 336.
3. Capability of representation learning: With the emergence of powerful self-supervised learn- ing techniques, sophisticated models with multiple non-linear layers of abstraction are re- quired to capture abstract representation of the data. Simple models like linear ones (Zeng) ett al., 2022) may not be preferred for that task due to its limited expressibility. With our PStchTST model, not only confirm that Transformer is actually effective for time series forecasting, but also demonstrate the representation capability that can further enhance the forecasting performance. Our PST has achieved the best MSE (0.349) in Table 1
2

---

Published as a conference paper at ICLR 2023
We introduce our approach in more detail and conduct extensive experiments in the following sec- tions to conclusively prove our claims. We not only demonstrate the model effectiveness with super- vised forecasting results and ablation studies, but also achieves SOTA selff-uuprrised representation learning and transfer learning performance.
2 RELATED WORK
Patch Transformer-base Models. Transformer (Vaswani et al. 2017) has demonstrated a sig- nificant potential different data modalities. Among all applications, patching is essential part when local semantic information is important. In NLP, BERT (Devlin et al. 2018) considers subword-bas based tokenization (Schuster & Nakajima 2012) instead of performing character-based tokenization. In CV, Vision Transformer (ViT) (Dosovitskiy et al., 2021) is a milestone work that splits image into 16x 16 patches before feeding into the Transformer model. The following influ- ential works such as BEiT (Bao et al. 2022) and masked autoencoders (He et al. (02T) are all using patches as input. Similarly, in speech researchers are using convolutions extract information in sub-sequence levels from raw audio input (Baevski et al. 2020 Hsu et al. (02T).
Transformer- based Long-term Time Series Forecasting. There large body of work that tries to apply Transformer models to forecast long-term time series in recent years. We here summarize some of them. LogTrans (Li et al.) (2019) uses convolutional self-attention layers with LogSparse design to capture local information and reduce the space complexity. Informer (Zhou et al. 2021) proposes a ProhSparse self-attention with distilling techniques to extract the most important keys efficiently. Autoformer (Wu et al.) (02I)) borrows the ideas of decomposition and auto-correlation from traditional time series analysis methods. FEDformer (Zhou et al. 2022) uses Fourier enhanced structure to get linear complexity. Pyraformer (Liu et al. 2022) applies pyramidal attention module with scter and intra-scale scale connections which also get a linear complexity.
Most of these models focus designing novel mechanisms to reduce the complexity of original attention mechanism, thus achieving better performance on forecasting, especially when the pre- diction length is long. However, most of the models use point-wise attention, which ignores the importance of patches. LogTrans (Li et al.) 2019) avoids a point-wise dot product between the key and query, but its value is still based on a single time step. Autoformer (Wu al., [202]] uses auto- correlation to get patch level connections, but it is a handcrafted design which doesn"t 't include all the semantic information within a patch. Triformer (Cirstea al., 2022) proposes patch attention, but the purpose is to reduce complexity by using a pseudo timestamp the query within a patch, thus neither treats patch as a input unit, nor reveals the semantic importance behind it.
Time Series Representation Learning, Besides supervised learning, self-supervised learning is also an important research topic since has shown the potential to learn useful representations for downstream tasks. There are many non- mon-Transffrrmr--bea
models proposed in recent years to learn representations in time series (Franceschi et al. 2019; Tonekaboni et al., 2021 Yang & Hong 2022] Yue et al. (002)). Meanwhile, Transformer is known to be an ideal candidate towards foundation models (Bommasani et al., 202T) and learning universal representations. However, although people have made attempts on Transformer-bate based models like time series Transformer (TST) (Zerveas et al. 2021) and TS-TCC (Eldele et al. 2021), the potential is still not fully realized yet.
3 PROPOSED METHOD
3.1 MODEL STRUCTURE
We consider the following problem: given a collection of multivariate time series samples with look- back window L (æ1,..., Z.) where each æ; time step t is a vector of dimension M, would like to forecast T future values (æ.+1,..., ). Our PStchTST is illustrated in Figure where the model makes use of the vanilla Transformer encoder as its core architecture.
Forward Process. We denote a i-th univariate series of length L starting at time index as (i)
(t) where i L,.., M. The input (æ1,.., is split to M univariate series ar ERILL where each of them is fed independently into the Transformer backbone according to
L (f(() ..., .,,z%
3

---

Published as a conference paper at ICLR 2023
Trainfformen Backbone
Channd-
Concatenate
x M2M
60
2 RRAT, L...M
(a) PatchTST Model Overview
Reconstructed Masked Patches
40 a
Output Untvariate Series
Flatten Linear Head
Linear Layer
Transtormer Encoder
Transformer Encoder
Aik &
Nerr
Makillad
Projection Position Embedding
Projection Position Embedding
<0 NWV
Instance Norm Patching
Instance Norm Patching
2
(b) Transformer Backbone (Supervised)
(c) Transformer Backbone (Self-supervised)
PatchTST architecture. (a) Multivariate time series data is divided into different chan- nels. They share the same Transformer backbone, but the forward processes are independent. (b) Each channel univariate series is passed through instance normalization operator and segmented into patches. These patches are used as Transformer input tokens. (c) Masked self-supervised represen- tation learning with PatchTST where patches are randomly selected and set to zero. The model will reconstruct the masked patches.
Figure 1:
our channel-independenee setting. Then the Transformer backbone will provide prediction results (+() (8()) (6 #1T) 1xT accordingly.
Patching. Each input univariate time series (1)) is first divided into patches which can either overlapped or ovn-overlapped. Denote the patch length P and the stride the non overlapping region between two consecutive patches as S, then the patching process will generate the a sequence
of patches x. E BPNN where N is the number of patches, N |
+ 2. Here, we pad S repeated numbers of the last value JT E E to the end of the original sequence before patching.
With the use of patches, the number of input tokens can reduce from L approximately L/S. This implies the memory usage and computational complexity of the attention map are quadrati- cally decreased by factor of S. Thus constrained on the training time and GPU memory, patch design can allow the model to see the longer historical sequence, which can significantly improve the forecasting performance, as demonstrated in Table
Transformer Encoder. We use vanilla Transformer encoder that maps the observed signals to the latent representations. The patches are mapped the Transformer latent space of dimension D via
trainable linear projection W, E DxP, and a learnable additive position encoding W pos E R D
is applied to monitor the temporal order of patches: x W Wp
(i)
E DxN Then each head h = 1,.,,, H
W poss where x d
denote the input that will be fed into Transformer encoder in Figure
in multi-head attention will transform them into query matrices
Q
(a (1) W. h key matrices
K(O (o) W. and value matrices V (3) (ar()) where W WK DDde and
4

---

Published as a conference paper at ICLR 2023
W RDDD After that a scaled production is used for getting attention output 0, E <RD 2h (T VV( (0)
( )T===
Attention((2 K 1/ Softmax
vdl
The multi-head attention block also includes BatchNorm layers and a feed forward network with residual connections as shown in Figure [T
Afterwards it generates the representation denoted as
20)) R DxN Finally flatten layer with linear head is used to obtain the prediction result 2(+)
L+1 L+T) RIxT
Loss Function. We choose to use the MSE loss measure the discrepancy between the prediction and the ground truth. The loss in each channel is gathered and averaged over M time series to get
the overall objective loss: L E. NT Σ || [+1:L++ #++1:4+7
Instance Normalization. This technique has recently been proposed to help mitigating the distri- bution shift effect between the training and testing data (Ulyanov et al.) [016)] Kim et al., (022). It simply normalizes each time series instance x+)) with zero mean and unit standard deviation. In
essence, we normalize each a before patching and the mean and deviation are added back to the output prediction.
3.2 REPRESENTATION LEARNING
Self-supervised representation learning has become popular approach extract high level abstract representation from unlabelled data. In this section, we apply PatchTST to obtain useful represen- tation of the multivariate time series. We will show that the learnt representation can be effectively transferred to forecasting tasks. Among popular methods to learn representation via self-supervise pre-training, masked autoencoder has been applied successfully NLP (Devlin et al. 2018) and CV (He et al., 2021) domains. This technique is conceptually simple: portion of input sequence is intentionally removed at random and the model is trained to recover the missing contents.
Masked encoder has been recently employed in time series and delivered notable performance on classification and regression tasks (Zerveas al. (02T). The authors proposed to apply the multi- variate time series to Transformer, where each input token is a vector æ; consisting of time series values at time step i-th. Masking is placed randomly within each time series and across different series. However, there are two potential issues with this setting: First, masking is applied at the level of single time steps. The masked values at the current time step can be easily inferred by interpo- lating with the immediate proceeding of the entire sequence, which deviates from our goal of learning important abstract representation of the whole signal. /Zerveas et al. (2021) proposed complex randomization strategies resolve the problem in which groups of time series with different sizes are randomly masked.
succeeding time values without high level understanding
Second, the design of the output layer for forecasting task can be troublesome. Given the repre- sentation vectors & E IR corresponding to all L time steps, mapping these vectors to the output containing M variables each with prediction horizon T via linear map requires a parameter matrix W of dimension (L- D) x (M T). This matrix can be particularly oversized if either one or all of these four values are large. This may cause overfitting when the number of downstream training samples is scarce.
Our proposed PatchTST can naturally overcome the aforementioned issues. As shown Figure I we use the same Transformer encoder as the supervised settings. The prediction head is removed and a D x P linear layer attached. As opposed to supervised model where patches can be overlapped, divide each input sequence into regular non-overlapping patches. It is for convenience to ensure observed patches do not contain information of the masked ones. We then select a subset of the patch indices uniformly at random and mask the patches according to these selected indices with zero values. The model is trained with MSE loss to reconstruct the masked patches.
We emphasize that each time series will have its own latent representation that are cross-learned via a shared weight mechanism. This design can allow the pre-training data to contain different number of time series than the downstream data, which may not be feasible by other approaches.
Zerveas al. 2021) has shown that Batch outperforms LayerNorm in time series Transformer.
5

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 LONG-TERM TIME SERIES FORECASTING
We evaluate the performance of our proposed PatchTST on 8 popular datasets, including Weather, Traffic, Electricity, ILI and 4 ETT datasets (ETThI, ETTh2, ETTml, ETTm2). These datasets have been extensively utilized for benchmarking and publicly available on (Wu et al., (021) The statistics of those datasets are summarized in [2 We would like to highlight several large datasets: Weather, Traffic, and Electricity. They have many more number of time series, thus the results would be more stable and less susceptible to overfitting than other smaller datasets.
Datasets.
Datasets
Weather
Traffic
Electricity
ILI
ETTh1
ETTh2
ETTml
ETTm2
Features
862
321
7
7
Timesteps
52696
17544
26304
966
17420
17420
69680
69680
Table 2: Statistics of popular datasets for benchmark.
Baselines and Experimental Settings. We choose the SOTA Transformer-based models, includ- ing FEDformer (Zhou et al., 2022), Autoformer (Wu et al. 2021), Informer (Zhou et al. 2021) Pyraformer (Liu et al. 2022). LogTran eLi al. 2019), and recent non-Transformer--aased model
DLinear (Zeg et al. (022) as our baselines. All of the models are following the same experimental setup with prediction length T {24, 36, 48, 60} for ILI dataset and T E (96, 192, 336, 720} for other datasets as in the original papers. We collect baseline results from Zeng et al. (022) with the default look-back window L = 96 for Transformer-basee
models, and 336 for DLinear, But in order to avoid under-estimating the baselines, we also run FEDformer, Autoformer and Informer for six different look-back window L E (24, 48, 96, 192, 336, 720). and always choose the best results to create strong baselines. More details about the baselines could be found in Appendix A.1.2 We calculate the MSE and MAE of multivariate time series forecasting as metrics.
Model Variants. We propose two versions of PatchTST in Table 3 PatchTST/64 implies the number of input patches is 64, which uses the look-back window L = 512. PatchTST/42 means the number of input patches is 42, which has the default look-back window L = 336. Both of them use patch length P = 16 and stride S = 8. Thus, we could use PatchTST/42 as a fair comparison models, and P explore even better results on larger datasets. More experimental details are provided in Appendix[A.. A.I
to DLinear and other Transformer-baatt
Results. Table 3 shows the multivariate long-term forecasting results. Overall, our model outper- form all baseline methods. Quantitatively, compared with the best results that Transformer-basee models can offer, PatchTST/64 achieves an overall 21.0% reduction on MSE and 16. 7% reduction on MAE, while PStchTST/42 attains an overall 20. 20.2% reduction on MSE and 16.46 66.4% reduction on MAE. Compared with the DLinear model, PatchTST can still outperform it in general, especially on large datasets (Weather, Traffic, Electricity) and ILI dataset. We also experiment with univariate datasets where the results are provided in Appendix A.3
4.2 REPRESENTATION LEARNING
In this section, conduct experiments with masked self-supervised learning where we the patches to be non-overlapped. Otherwise stated, across all representation learning experiments the input sequence length is chosen to be 512 and patch size is set to 12, which results 42 patches. We consider high masking ratio where 40% of the patches are masked with zero values. We first apply self-supervised pre-training on the datasets mentioned in Section 4. for 100 epochs. Once the pre-trained model on each dataset is available, we perform supervised training to evaluate the (a) linear probing and (b) end-to-end fine-tuning. With (a), we only train the model head for 20 epochs while freezing the rest of the network; With (b), we apply linear probing for 10 epochs to update the model head and then end-to-end fine-tuning the entire network for 20 epochs. It was proven that a two-step strategy with linear probing followed by fine-tuning can outperform only doing fine-tuning directly (Kumar et al., 2022). We select a few representative results on below, and full benchmark can be found in Appendix A.5
learned representation with two options:

---

Published as a conference paper at ICLR 2023
Modell
PatchTST164
PatchTST:42
DI
FEDiormer
Autoffomer
Informer
Pynalommer
LogTriins
Metric
MSE
MAB
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAP
MSE
MAE
MSE
MAE
MSE
MAB
Weaber
0.149
0.198
0.152
0.199
0.176
0.237
0.238
0.314
0.249
0.329
0.354
0.405
0.896
0.556
0:458
0.190
192
0.194
0.241
0.197
0.245
0.220
0.282
0.275
0.529
0.325
0.370
0.419
0.434
0.622
0.624
0.588
0.589
336
0.245
0.282
0.249
0.283
0.265
0.319
0.339
0.577
0.351
0.391
0.583
0.543
0.739
0.753
0.797
0.652
0.314
0.334
0.320
0.335
0.323
0.362
0.389
0.409
0.415
0.426
0.916
0.705
1.004
0.924
0.869
0.675
96
0.360
0,249
0 0.367
0.251
0,110
0 0.282
0.576
0.559
0.597
0.371
0.733
0.410
2.085
0.468
0.654
0.384
Electriityy Traflic
192
0.379
0.256
0.385
0.259
0.425
0.287
0.610
0.580
0.607
0. 0.382
0.777
0.435
0.867
0.467
0.685
0,300
336
0.392
0.264
0.396
0.265
0.436
0.296
0.608
0.375
0.623
0.387
0.776
0.434
0.869
0.469
0.734
0.408
720
0.432
0,286
0.134
0.287
0:166
0.315
0.621
0.575
0.639
0.395
0.827
0.166
0.881
0.173
0.717
0.396
96
0.129
0.222
0.150
0.222
0.140
0.257
0.186
0.502
0.196
0.313
0,304
0.393
0.386
0.449
0.258
0.357
0.147
0.240
0.148
0.153
0.249
0.197
0.211
0.324
0.327
0.417
0,386
0.M3
0.266
0,368
336
0.163
0.259
0.167
0.261
0.169
0.267
0,213
0.528
0.214
0.327
0.333
0,442
0.378
0:443
0,250
0.380
720
0.197
0.290
0.202
0.291
0,203
0.301
0,233
0.344
0.236
0.342
0.351
0.427
0.376
0.445
0,283
0.376
1,319
0.754
1.522
0.814
2.215
1.081
2.624
1.095
2.906
1.182
1.657
1.119
1,120
2.012
1.150
1.111
36
1.579
0.870
1.430
0.834
1:963
0.963
2.516
1.021
2.585
1.038
4.650
1,463
7.394
2,031
4,799
1,467
48
1.553
0.815
1,673
0.854
2.150
1.004
2,505
1.041
3.024
1.145
5.004
1.542
7.551
2.057
4,800
1.468
1.470
0.788
1.529
0.862
2.368
1.096
2.742
1.122
2.761
1.114
5.071
1.543
7.662
2.100
5,278
1,560
96
0.370
0,400
0. 0.375
0.399
0.375
0.399
0.376
0,415
0,435
0,446
0.941
0.764
0.664
0.612
0,878
0,740
192
0.413
0.429
0.414
0.421
0.405
0.416
0.425
0.446
0.456
0.457
1.007
0.786
0,790
0.681
1.037
0.824
0.440
0.431
0.439
0.443
0,444
0.462
0.486
0.487
1,038
0.784
0.891
0.738
1,238
0,902
720
0.447
0.468
0.149
0.466
0.172
0.490
0.169
0.192
0.515
0.517
1,141
0.857
0.963
0.782
1.135
0.852
96
0.274
0.337
0.274
0.336
0.289
0.353
0.332
0.574
0.332
0.368
1.549
0.952
0.645
0.597
2.116
1.197
0.341
0.382
0.339
0.379
0.383
0.418
0.407
0,446
0.426
0.434
3.792
1.542
0.788
0.683
4.315
1.635
336
0.329
0.381
0.331
0.380
0:418
0:165
0.400
0:147
0.177
0.179
4.215
1.612
0.907
0.747
1.121
1.601
720
0.379
0.422
0.379
0.422
0.605
0.551
0,412
0.469
0,453
0,490
3.656
1.619
0.963
0.788
3,188
1.540
0.293
0.346
0.290
0.342
0.299
0.343
0.326
0.390
0.510
0.492
0.626
0,560
0.543
0.510
0,000
0.546
0.333
0.370
0.332
0.369
0.335
0.365
0.365
0.415
0.514
0.495
0.725
0.619
0.557
0.537
0.837
0.700
336
0. 1,369
0.392
0.366
0.302
0.369
0.386
0.392
0.425
0,510
0.492
1,0055
0.741
0,754
0.655
1.124
0,832
720
0.416
0.420
0.420
0.424
0,425
0.421
0,446
0,458
0.527
0.493
1.133
0.845
0.908
0.724
1,153
0,830
E77m2
0, 0,166
0.256
0.165
0.255
0.167
0.260
0.180
0.271
0.205
0.293
0.355
0,462
0.435
0.507
0.768
0.642
192
0.223
0.296
0.220
0.292
0:224
0.303
0.252
0.318
0,278
0.336
0.595
0.586
0,730
0.673
0.989
0.757
0.329
0.278
0.329
0.281
0,342
0.324
0.364
0.343
0.379
1,270
0.871
1,201
0.845
1,334
0.872
720
0.362
0,385
0.367
0.385
0.397
0.421
0.410
0.420
0.414
0.419
3,001
1.267
3.625
1.451
3,048
1,328
Table 3: Multivariate long-term forecasting results with supervised PatchTST. We use prediction
lengths T E {24, 36, 48, 60} for ILI dataset and T E {96, 192, 336, 720} for the others. The best results are in bold and the second best are underlined.
PatGhTST
Models
DLincar
FEDfomerr
Autoformer
Informer
Fine tuning
Lin. Prob.
Sup.
Metric
MSE
MAF
MSE
MAB
MAE
MSE
MAF
MSE.
MAF
MSE
MAP.
Flectricity Traffic Weather
965
0.144
0.193
0.158
0.209
0.152
0.199
0.176
0.237
0,238
0.314
0.249
0.329
0.354
0.405
192
0.190
0.236
0.243
0.220
0.282
0.275
0.370
0.419
0.434
336
0.244
0.280
0.251
0.285
0.249
0.283
0.265
0.319
0.339
0.377
0.351
0.391
0,583
0,543
720
0.320
0.335
0.336
0.362
0.389
0.409
0.415
0.426
0.916
0.705
96
0.352
0.244
0.399
0.294
0.367
0.251
0.410
0.282
0.376
0.359
0.397
0,371
0,733
0.410
192
0.371
0.253
0.412
0.298
0.385
0.259
0.423
0.287
0.610
0.380
0.607
0.382
0.777
0.435
336
0,381
0.257
0.425
0.306
0.398
0.265
0.436
0.296
0.608
0.375
0.623
0.387
0.776
0.434
720
0.425
0.282
0.460
0.323
0.434
0.287
0.466
0.315
0.621
0.375
0.639
0.395
0.827
0.466
96
0.126
0.221
0.138
0.237
0.130
0.223
0.140
0.237
0.186
0.302
0.196
0.313
0.304
0.393
192
0.145
0.238
0,156
0.252
0.148
0.240
0,153
0.249
0.197
0.311
0.211
0.324
0.327
0.417
336
0.164
0.256
0.170
0.265
0.167
0.261
0.199
0.267
0.213
0.328
0.214
0.327
0.333
0.422
720
0.193
0,291
0.208
0.297
0.202
0
0.203
0.301
0.233
0.344
0.236
0.342
0.351
0.427
Table 4: Multivariate long-term term forecasting results with self-super PatchTST, We use prediction lengths T E {96,, 192, 336, 720}. The best results are in bold and the second best are underlined.
Comparison with Supervised Methods. Table 4 compares the performance of PatchTST (with fine-tuning, linear probing, and supervising from scratch) with other supervised method. As shown in the table, on large datasets our pre- pre-training procedure contributes a clear improvement compared to supervised training from scratch. By just fine-tuning the model head (linear probing), the fore- casting performance is already comparable with training the entire network from scratch and better than DLinear. The best results are observed with end-to-end finc-tuning. Self-supervised PatchTST significantly outperforms other Transformer-basee models on all the datasets.
Transfer Learning. We test the capability of transfering the pre-trained model to downstream tasks. In particular, we pre-train the model on Electricity dataset and fine-tune on other datasets. We observe from Table 5 that overall the fine-tuning MSE is lightly worse than pre-training and fine- tuning on the same dataset, which is reasonable. The fine-tuning performance is also worse than supervised training in some cases. However, the forecasting performance is still better than other
7

---

Published as a conference paper at ICLR 2023
Models
PaRETST
DLincar
FEDfomer
Autoformer
Informer
Fine-tuning
Lin. Prob.
Sup.
Metric
MSE
MĀE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Tmffic Weather
96
0.145
0.195
0.163
0.152
0.199
0.176
0.237
0.238
0.314
0.249
0.329
0.354
0.405
192
0.193
0.243
0.205
0.252
0.197
0.243
0.220
0.282
0.275
0.329
0.325
0.370
0.419
0.434
336
0.244
0.280
0.253
0.289
0.249
0.283
0.265
0.319
0.339
0.377
0.351
0.391
0.583
0.543
720
0.321
0.337
0.320
0.336
0.320
0.335
0.323
0.362
0.389
0.409
0.415
0.426
0.916
0.705
96
0.388
0.273
0.400
0.288
0.367
0.251
0,410
0.282
0.576
0.359
0.597
0.371
0.733
0,410
192
0,400
0.277
0.412
0.293
0.385
0.259
0.423
0.287
0.610
0.380
0.607
0.382
0.777
0.435
336
0.408
0. 0.280
0.425
0.307
0.398
0.265
0.436
0.296
0.608
0.375
0.623
0.387
0.776
0,434
720
0.447
0.310
0.457
0.317
0.434
0.287
0.466
0.315
0.621
0.375
0.639
0.395
0.827
0.466
Table 5: Transfer learning task: PatchTST is pre-trained on Electricity dataset and the model is transferred other datasets. The best results are in bold and the second best are underlined.
PatcTST
Models
IMP.
Transferred
Self supervised
BTSF
TS2Vec
TNC
TCC
Metrics
MSF
MSR
MAF
MSF
MAR
MSF
MAF
MSF
MAF
MSE
MAF
MSE
MAF
24
42.3%
0.312
0.362
0.322
0.369
0.541
0.519
0.599
0.534
0.632
0.596
0.653
0.610
48
0.378
0.354
0.613
0.524
0.555
0.705
0.688
0.693
168
34.5%
0.424
0.437
0.419
0.424
0.640
0,532
0.755
0.636
1.097
0.993
1.129
1.044
336
48.5%
0.472
0.472
0.445
0.446
0.864
0.689
0,907
0.717
1.454
0.919
1.492
1.076
720
48.8%
0.508
0.507
0.487
0.478
0.993
0.712
1,048
0.790
1.604
1.118
1.603
1.206
Table 6: Representation learning methods comparison. Column name transferred implies pre- training PatchTST on Traffic dataset and transferring the representation to ETTh1, while self- supervised implies both pre-training and linear probing on ETTh1. The best and second best results are in bold and underlined. IMP. denotes the improvement on best results of PatchTST compared to that of baselines, which is in the range of 34.5% 48.8% on various prediction lengths.
models. Note that opposed to supervised PatchTST where the entire model is trained for each prediction horizon, here we only retrain the linear head or the entire model for much fewer epochs, which results in significant computational time reduction.
Comparison with Other Self-supervised Methods. We compare our self-supervised model with BTSF YYang& & Hong. 2022). TS2Vec (uue et al., 2022). TNC (Tonekaboni al., (02T), and TS- TCC (Eldele et al.] (021)) which are among the state-of-the-an contrastive learning representation methods for time series f We test the forecasting performance on ETTh1 dataset, where we only apply linear probing after the learned representation is obtained (only fine-tune the last linear layer) to make the comparison fair. Results from Table 6 strongly indicates the superior performance of PatchTST, both from vre-trainin on its own ETTh data (self-supervised columns) or pre-training on Traffic (transferred columns).
4.3 ABLATION STUDY
Patching and -ineeeeeendee We study the effects of patching and hhunnel-indeeendenee in Table We include FEDformer as the SOTA benchmark for Transformer- based model. By com- paring results with and without the design of patching / channel--nndepender observe that both of them are important factors in improving the forecasting performance. The moti- vation of patching is natural; furthermore this technique improves the running time and memory con- sumption as shown in Table duue to shorter Transformer sequence input. Shannel-indeeendeeii the other hand, may not be as intuitive as patching is in terms of the technical advantages. Therefore, more preferable
accordingly, one can
on
we provide an in-depth analysis on the key factors that make in Appendix A.7 More ablation study results are available in Appendix A.4
Varying Look-back Window. In principle, longer look-back window increases the receptive field, which will potentially improves the forecasting performance. However, as argued in (Zeng et al.) 2022]. this phenomenon hasn't been observed in most of the Transformer- based models. We also demonstrate in Figure 2 that in most cases, these Transformer- from longer look-back window L, which indicates their ineffectiveness capturing temporal in- formation. In contrast, our PatchTST consistently reduces the MSE scores as the receptive field increases, which confirms our model's capability to learn from longer look-back window.
baselines have not benefited
We cite results of TS2Vec from Yue et al. 2022 and (BTSE)TCCTSS-CCC from Yang & Hong 2022
8

---

Published as a conference paper at ICLR 2023
PatchTST
Models
FEDiormer
P+CI
CI
P
Original
0.152
0.199
0.164
0.213
0.168
0.225
0.177
0.236
0.238
0.314
Wealher
192
0.197
0.243
0.205
0.250
0.213
0.262
0.221
0.270
0.275
0.329
336
0.249
0.283
0.255
0.289
0.266
0.300
0.271
0.306
0.339
0.377
96
0.367
0.251
0.397
0.271
0.595
0.376
0.576
0.359
Flectricity Tralicc
192
0.385
0.259
0.411
0.2776
0.612
0.387
0.610
0,380
336
0.398
0.423
0.282
0.651
0.391
0.608
0,375
0.309
0.621
0.375
96
0.130
0.222
0.136
0.231
0.196
0.307
0.208
0.318
0.186
0.302
192
0.148
0.240
0.164
0.263
0.215
0.323
0.197
0.311
720
0.202
0.291
0.219
0.312
0.244
0.345
0.233
0.344
Table Ablation study of patching and channel-ndeependence PatchTST. 4 cases are included: (a) both patching and shannel-indeeendence are included in model (P+CI): (b) only channel- independence (CI); (c) only patching (P); (d) neither of them is included (Original TST model). PatchTST means supervised PatchTST/42. '-' in table means the model runs out of GPU memory (NVIDIA A40 48GB) even with batch size 1. The best results are in bold.
Electricity
T
0.8
0.5
1.0
0.6
MSe
0.4
0.5
0.8
0.3
0.3
0.2
0.4
336
720
Electricity T-720
Traffic T-720
Weather T-720
1.1
0.9
0.8
1.2
0.7
0.8
0.6
1.0-
0.5
0.8
0.5
0.6
0.2
0.4
48
192
336
720
24
48
96
192
336
720
24
96
192
336
720
Informer
Transformer
FEDfermer
Autoformer
- PatchTST
Forecasting performance (MSE) with varying look-back windows 3 large datasets: Electricity, Traffic, and Weather. The look-back windows are selected to 24, 48, 96, 192, 336, 720, and the prediction horizons are T 96, 720. We use supervised PatchTST/42 and other open-source Transformer based baselines for this experiment.
Figure 2:
5 CONCLUSION AND FUTURE WORK
This paper proposes an effective design Transformer- models for time series forecasting tasks by introducing two key components: patching and channel-independenn structure. Compared to the previous works, could capture local semantie information and benefit from longer look-back windows. We not only show that model outperforms other baselines in supervised learning, but also prove its promising capability in self-supervised representation learning and transfer learning.
Our model exhibits the potential to be the based model for future work of Transformer-baseee fore- casting and be building block for time series foundation models. Patching is simple but proven to be an effective operator that can be transferred easily to other models. Channel-indepeneeee -independence, the other hand, can be further exploited to incorporate the correlation between different channels. would be important future step to model the cross-channel dependencies properly.

---

Published as a conference paper at ICLR 2023
REFERENCES
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances Neural Information Processing Systems, 33:12449-123 -2460, 2020.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: /803.0127, 2018.
Hangbo Bao. Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image trans-
formers. In International Conference on Learning Representations, 2022. URL https: //open ew. net. d
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2 07250, 2021.
G. E. P. Box and Gwilym M. Jenkins. Time series analysis, forecasting and control. 1970.
Lim Bryan and Zohren Stefan. Time-series forecasting with deep learning: a survey. Phil. Trans. R. Soc. A, 2021. doi: 10. 211199///ta.22200022000
Defu Cao, Yujing Wang. Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bix- iong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time- series forecasting. Advances neural information processing systems, 33:17766-17778 2020.
Weiqi Chen, Wenwei Wang, Bingging Peng, Qingsong Wen, Tian Zhou, and Liang Sun. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting, In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 146-156, 2022.
Yuzhou Chen, Ignacio Segovia, and Yulia R Gel. Z-genets: time zigzags at graph convolutional networks for time series forecasting. In International Conference on Machine Learning, pp- 1684-1694. PMLR, 2021.
Rabrin-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting. In Proceedings of the Thirty-Fins First International Joint Conference Artificial In- telligence, IJCAI-22, pp. 1994-2001, 2022. doi: 10.24963/iccai.20027777 URL https: www.org//10 211.3/49933113 2022/272222277
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/18 1s000000,,485, 2018. URL
http::/arxiv.. wwtp://///////..oooo//.o 04805
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL. https: //openreview.
d=Y cbF cbFdNTTy
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pp. 1352-2355, 2021.
Eugene F Fama. Efficient capital markets: A review of theory and empirical work. The journal of Finance, 25(2):33-4477, 1970.
Jean- Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems, volume 32, 2019. URL
http:///wccceciao.....i.ioe..ooa neurips wttpriis..cpppppppeeeeee httpp//ww...cc.pppp///////////1. 33cccaee222ee255222eleaaalae3333
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked au- toencoders are scalable vision learners. CoRR, abs/2111.0653 2021. URL https: //axxvv. org/abs/ 211l/111111 06377
Dan Hendrycks and Kevin Gimpel. Gaussian N.KV::6606.04415. 2016.
error
linear units (gelus). arXiv preprint
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput., 1997.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Katikapalli Subramanyam Kalyan, Ajit Rajasckharan, and Sivanesan Sangeetha. Ammus: A sur- vey of transformer-based pretrained models in natural language processing. arXiv preprint arXiv:2/08.05542, 2021.
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto. Xiaofei Wang, et al. A compar- ative study on transformer vs mn in speech applications. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 449 456. IEEE, 2019.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Computing Surveys (CSUR). 2021.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference Learning Representations, 2022. URL https: http::tttt//snnoe //tperev//ppeneviee. JRLhttps:
net/ het/EEorumm111=GG0000000000111 cGDAkOolCOp)
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of--distribttin. In International Con- ference on Learning Representations, 2022. URL https: https::/ hatps/////ppeeeviieeeeee net www.:ewweetfffrmmm//////m forum?
Ld==YYne2XSJJW
Pedro Lara-Benítez, Manuel Carranza-Garcia and José C Riquelme. An experimental review on deep learning architectures for time series forecasting. International Journal of Neural Systems, 31(03):2130011 2021.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang. and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. In Advances in Neural Information Processing Systems, vol- ume 32, 2019. URL https:: http://pccccccceeeeeeeeeeeaeeapp
37770065cc02234422222222. 9d8 Pap.ooaappppp.p.. -dff
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complcxity pyramidal attention for long-range time series modeling and fore- casting. In International Conference on Learning Representationss 2022.
Barbara Rossi. Exchange rate predictability. Journal of Economic Literature, 51(4), De- cember 2013. doi: 10. 10.112///e......1000 1063. URL https: /wwww aww.eb. www.:a/awweb.orgarrtteeee www.wwb.rrr.//tttlllllsss.. id=10. 1el.l2257555 10157//el.... 1d=10.1257/jel.5.11 5.....0633
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181- 1191, 2020.
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5152, 2012. doi: N0.1199ICASSP22122229977
11

---

Published as a conference paper at ICLR 2023
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Repre- sentations, 2021.
JCuCN
José F Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martinez-Alvarez, and Alicia Troncoso. Deep learning for time series forecasting: a survey. Big Data, 91))3-211, 2021.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, volume 30, 2017. URL httppss///pprcrcdnnggg.g... neurips. /c/pppper22017//1////eeeeee 7deel9 hag.aa/e225.77aaa1ooocooaeaaaaaa
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma. Junchi Yan, and Liang Sun. Transformers in time series: A survey, arXiv preprint arXivv2202.07125, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition transform- ers with Corr-Coorllttion for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilin- ear temporal-spectral fusion. In ICMI,, 2022.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205. 200.:2205.3304 13504, 2022.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eick- hoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2114-2124, 2021.
Yi Zheng, Qi Liu, Enhong Chen, Yong Ge, and Leon Zhao. Time series classification using multi- channels deep convolutional neural networks. In International conference web-age information management, pp. 298-310. Springer, 2014.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty- Fifth AAAI Conference on Artificial Intelligence, volume 35, pp. 1 1100-11115, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning, 2022.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 EXPERIMENTAL DETAILS
A.1.1 DATASETS
We use 8 popular multivariate datasets provided in (Wu et al. 2021) for forecasting and representa- tion learning. W collects meteorological indicators in Germany, such humidity and air temperature. Traffi TraffePdatasel dataset records the road occupancy rates from different sensors on San Francisco freeways. Electricity is a dataset that describes 321 customers' hourly electricity con- sumption. ILA dataset collects the number of patients and influenza-like illness ratio in weekly frequency. ETTV (Electricity Transformer Temperature) datasets are collected from two different electric transformers labeled with and 2, and each of them contains 2 different resolutions (15 minutes and hour) denoted with m and Thus, in total we have ETT datasets: ETTml, ETTm2, ETThI, and E77h2.
There is an additional Exchange-rate dataset mentioned in the original paper, which is the daily exchange-rate of eight different countries. However, financial datasets generally have different prop- erties compared to time series datasets in other fields, for example the predictability. It well known that if market is efficient, the best prediction for #; will be just - (Famal 1970). Rossi (2013) argues that the toughest benchmark for exchange-rate forecasting is a random walk without drift. Also, Zeng et al. (2022) shows that by simply repeating the last value in the look-back window, the MSE loss on exchange-rate dataset can outperform be comparable to the best results. Therefore, we are prudent containing it into our benchmark.
A.1.2 DETAILS OF BASELINE SETTINGS
The default look-back windows for different baseline models could be different. For Transformer- based models, the default book-back window is L 96; and for DLinear, the default look-back win- dow is L = 336. The reason of this difference is that Transformer-basee baselines are easy to overfit when look-back window is long while DLinear tend to underfit. However, this can possibly lead to an under-estimation of the baselines. To address this issue, we re-run FEDformer, Autoformer and Informer by ourselves for six different look- window L E [24, 48, 96, 192, 336, 720). And for each forecasting task (aka each different prediction length on each dataset), we choose the best one from those six results. Thus it could be strong baseline.
The ILI dataset is much smaller than the other datasets, so different set of parameters is applied (the default look-back windows of Transformer- based models and DLinear are L 36 and 104 respectively; we run FEDformer, Autoformer and Informer for six different look-back window (24, 36, 48, 60, 104, 144) and choose the best results).
A.1.3 BASELINES FROM TRADITIONAL MODELS
Time series has been an ancient field of study, with many traditional models developed, for exam- ple the famous ARIMA model (Box & Jenkins 1970). With the bloom of deep learning commu- nity, many new models were proposed for sequence modeling and time series forecasting before Transformer appears, such LSTM (Hochreiter & Schmidhuber] 1997). TCN (Bai et al. 2018 and DeepAR (Salinas et al. 2020). However, they are demonstrated to be not effective as Transformer- based models long-term forecasting tasks (Zhou et al., 2021 Wu et al. 2021), thus we don't include them in our baselines.
http:://wwwwbb.-jaa..p.....//.
"hups://pems.dot wwt.:aaa..do....o..oooo.oo
'httpss//archive.i www.iiieeiiiiiiiiiii...mm wtt....ctt/////aatsse//eeeiriiii htt.://ttuiuuuueeeee/eeeeeeeeeem 12014
"hurs://giss.de" http:/gg......googgg///////////e
13

---

Published as a conference paper at ICLR 2023
A.1.4 MODEL PARAMETERS
By default, PStchTST contains encoder layers with head number H 16 and dimension of latent space D = 128. The feed forward network in Transformer encoder block consists of 2 linear layers with GELU (Hendrycks & Gimpel 2016) activation function: one projecting the hidden representation D = 128 to new dimension F = 256, and another layer that project it back to D 128. For very small datasets (II.I, ETTh1, ETTh2), reduced size of parameters is used (H = 4, 16 and F = 128) to mitigate the possible overfitting. Dropout with probability 0.2 is applied in the encoders for all experiments. The code will be publicly available.
A.1.5 IMPLEMENTATION DETAILS
Although PatchT'T processes channels in parallel which has make multiple copies of the Trans- former's weights, the computation can be implemented efficiently and does not require any special operator. The batch of samples of IR MMxx with size B M operator to generate 4D tensor of size x M x x which represents a batch of ( E R in M series. By reshaping the tensor to form 3D one of size (B M) x P N, this batch can be consumed by any standard Transformer implementation.
passed through the patching
We further argue that our proposed PatchTST contains additional benefits: The components in Trans- former backbone module shown in Figure can differ across different input series, for instance the embedding layers and head layers. Note that if the embedding layers designed differently for each group of time series, the reshaping step will be applied after embedding. Besides, the number of variables the multivariate time series during the training may need match the number of series for testing. This is especially beneficial for self-supervised pre-training where the pre-training data can have different number of variables from the fine-tuning data.
A.2 VISUALIZATION
We visualize the long-term forecasting results of supervised PatchTST/42 and other baselines in Figure Here, we predict 192 steps ahead on Weather and Eletricity datasets and steps ahead on ILI dataset. PatchTST provides the best forecasting both in terms of scale and bias.
DRIPAN
FEDbrrerr
10-
05
10C
130
200
100
:30
200
10
20
30
44)
50
50
Weacher
Electricity
Figure 3: Visualization of 192-step forecasting on { Weather, Traffic} datasets with the look-back
window L 336 and 60-step forecasting on ILI dataset with L 104. PatchTST (in red) can capture the trend and closest to the ground truth (in blue).
A.3 UNIVARIATE FORECASTING
Table g summaries the results of univariate forecasting on ETT datasets. There is target feature "oil temperature" within those datasets, which is the univariate series that we are trying to forecast. We cite the baseline results from (Zeng et al. 2022).
A.4 MORE RESULTS ON ABLATION STUDY
A.4.1 VARYING PATCH LENGTH
This experiment studies the effect of patch lengths to the forecasting performance. We fix the look- back window to be 336 and vary the patch lengths P [4, 8, 16, 24, 32, 40). The stride length is set
14

---

Published as a conference paper at ICLR 2023
Models
P1ccETST764
PachTTST42
DLingar
PEDIormer
Autelormer
Informer
Log Trans
MSE
MAE
MAE
MSE
MSE
MAE
MSE
96
0.059
0.189
0.055
0.179
0.056
0.130
0.079
0.215
0.071
0.206
0.193
0.377
0.283
0.468
192
0.074
0.071
0.205
0.071
0.204
0.245
0.262
0.217
0.395
0.234
336
0.076
0.220
0.081
0.225
0.098
0.244
0.119
0.270
0.107
0.258
0.202
0.38
0.386
0.546
720
0.087
0.236
0.087
0.232
0.189
0.359
0.142
0.299
0.126
0.283
0.183
0.355
0.475
0.629
ETTm22 ETTml B1712
96
0.131
0.284
0.129
0.282
0.131
0.279
0.128
0.271
0.153
0.306
0.213
0.373
0.217
0.379
192
0.171
0.329
0.168
0.328
0.176
0.329
0.185
0.330
0.204
0.351
0.227
0.387
0.281
0.429
336
0.171
0.336
0.185
0.351
0.209
0.367
0.231
0.378
0.246
0.389
0.242
0.401
0.293
0.437
720
0.223
0.380
0.224
0.383
0.276
0.426
0.278
0.420
0.268
0.409
0.291
0.439
0.218
0.387
96
0.0266
0.123
0.026
0.121
D.028
0.123
0.033
0.140
0,056
0.183
0.109
0.277
0.04
0.171
192
0.040
0.151
0.039
0.150
0.045
0.156
0.058
0.186
0
0.216
0.151
0.310
0.157
0.317
336
0.053
0.174
0.053
0.173
0.061
0.182
0.084
0.231
0.076
0.218
0.427
0.591
0.289
0.459
720
0.073
0.206
0.074
0.207
0.080
0.210
0.102
0.250
0.110
0.267
0.438
0.586
0.430
0.579
0.065
0.065
0.186
0.063
0.183
0.067
0.198
0,065
0.189
0.088
0.225
0.075
192
0.093
0.231
0.094
0.231
0.092
0.227
0.102
0.245
0.118
0.256
0.1
0.283
0.129
0.275
336
0.121
0.266
0.265
0.119
0.261
0.130
0.279
0.154
0.305
0.180
0.336
0.154
0.302
720
0.172
0.322
0.171
0.322
0.175
0.320
0.178
0.325
0.182
0.335
0.300
0.435
0.160
0.321
Table 8: Univariate long-term forecasting results with supervised PSt ETT datasets used with prediction lengths 7' E (96, 192, 336, 720}. The best results are in bold.
the same as patch length, meaning overlapping between patches. The model is trained to predict 96 steps. One observation from Figure is that MSE scores don't vary significantly with different choices of P. which indicate the robustness of our model against the patch length hyperparameter. Overall, PatchTST benefits from increased patch length, not only in forecasting performance but also in the computational reduction. The ideal patch length may depend on the dataset, but P between {8, 16} seems to be general good numbers.
0.156
Weather
0.135
Electricity
0.47
Traffic
0.155
0.46
0.134
0.45-
0.154
0.44
0.153
MSE
0.133
0.43
0.152
0.132
0.42
0.151
0.41
0.150
0.131
0.40-
0.39
0.149
32
0.130
4
12
16
24
32
40
4
8
12
16
24
32
40
Figure 4: MSE scores with varying patch lengths P [2,4,8, 12, 16, 24, 32, 40] where the look- back window 336 and the prediction length is 96.
A.4.2 VARYING LOOK-BACK WINDOW
Here provide full benchmark of quantitative results in Table varying look-back window in supervised PatchTST/42 regarding Figure 2 in the main text. Generally speaking, our model gains performance improvement with increasing look-back window, which show the effectiveness of our model in learning information from longer receptive field.
A.4.3 PATCHING AND CHANNEL-
Implementation Details. For ablation study on patching and channel independence section 4.3 run difffrent variants of PatchTST:
Both patching and chamnel-independdnce are included in model (P+CI): this the full PatchTST model that we have proposed in paper.
Only chameel-independdence (CI): we simply set both patch length P and stride to be 1 to avoid patching and only keep -haanee-indeeendence.
instead of reshaping the 4D tensor from B x M N to (B-M) x X N, we reshape it to B x (M- PP) x for paancllmixing with patching.
Only patching (P): referring to the implementation in A.1.5
15

---

Published as a conference paper at ICLR 2023
L
24(24)
48(36)
96(48)
192(60)
336(100)
72011445
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MSE
MAE
Weather
96
0.222
0.246
0.212
0.243
0.178
0.219
0.160
0.204
0.152
0.199
0.147
0.198
192
0.265
0.279
0.254
0.277
0.224
0.259
0.204
0.245
0.197
0.243
0.240
336
0.325
0.322
0,310
0.316
0.278
0.298
0.257
0.285
0.249
0.283
0.242
0,282
720
0.404
0.374
0.385
0.365
0.350
0.346
0.329
0.358
0.520
0.335
0.304
0.328
11.1 Electricity Traffic
96
0.766
0.419
0.671
0.381
0.477
0.305
0.401
0.267
0.367
0.251
0.365
0.251
192
0.725
0.398
0.616
0.356
0.471
0.299
0,406
0.268
0.385
0.259
0.382
0.258
336
0.752
0.410
0.635
0.364
0.485
0.305
0.421
0.277
0.398
0.265
0.398
0.267
720
0.786
0.427
0.673
0.383
0.518
0.325
0.452
0.297
0.434
0.287
0.436
0.289
96
0.268
0,316
0.225
0.293
0.174
0.259
0.138
0.230
0.130
0.222
0.130
0.224
192
0,259
0.316
0.217
0.291
0.178
0.265
0.149
0.243
0.148
0.240
0,147
0.241
336
0.283
0.335
0,238
0.309
0.196
0.282
0.169
0,262
0.167
0.261
0.163
0.259
720
0.321
0.365
0.278
0,342
0.237
0.316
0.211
0.299
0.202
0.291
0.197
0.290
24
3.0662
1.118
1,610
0.803
1:28
0.704
1.300
0,700
1,222
0.814
1,470
0.793
36
2.732
1.071
1 1.262
0.731
1.251
0.752
1.367
0.776
1.430
0.834
1.518
0.856
48
3.059
1.117
1,991
0,845
1,901
0.879
1.690
0.812
L,673
0.854
1.834
0.921
60
2,610
1.057
1.702
0.829
1.61
0.844
1.526
0.795
1.529
0.862
1,656
0.885
L2Tm2 177Tll ETTh2 ETTT
0.464
0,445
0,410
0,393
0.408
0.382
0.401
0.376
0,408
192
0,521
0.474
0.469
0.448
0.445
0.414
0.428
0.425
0.414
0.421
0.413
0.431
3336
0.570
0.498
0.516
0.451
0.451
0,436
0.454
720
0.575
0,522
0,509
0.487
0.480
0.471
0.452
0.459
0.449
0.466
0.458
0.477
0.333
0
0.307
0.285
0,340
0.279
0.341
192
0.422
0.409
0.397
0.399
0.377
0.393
0.356
0,386
0.339
0.379
0.349
0.386
0,442
0.432
0,412
0,420
0.381
0.409
0.350
0.395
0.331
0,380
0.409
720
0.462
0.453
0.434
0.441
0.412
0.433
0.395
0.427
0.379
0.422
0.394
0.434
0.624
0.481
0,424
0,403
0.321
0.360
0.291
0.340
0.290
0.342
0.295
0.348
192
0.671
0.507
0.468
0.429
0.362
0.384
0.328
0.365
0.332
0.369
0.334
0.373
336
0.714
0.533
0.501
0,453
0.392
0.402
0.365
0.389
0.366
0.392
0.361
0.393
720
0.744
0.554
0,553
0.484
0,450
0.435
0.422
0.423
0.420
0.424
0.416
0.419
96
0.212
0.290
0.189
0.272
0.178
0.260
0.169
0.254
0.165
0.255
0.162
0.254
192
0.282
0.334
0,260
0.317
0.249
0.307
0.230
0.294
0.220
0.292
0.216
0.293
336
0.354
0.376
0.328
0.359
0.313
0.346
0,280
0.329
0.278
0.329
0.269
0.329
720
0.458
0.433
0,429
0.415
0.400
0.398
0.378
0,386
0.367
0.385
0.350
0,380
Table 9: Multivariate long-term forecasting results with varying look-back window L in supervised PatchTST/42.
Neither patching nor channel-independence is included (Original), which is just the original TST model Zerveas et al. (02TT)
Note that the default number of maximum epochs for Electricity and Traffic datasets are reduced from 100 to 20 for ablation experiments due to the huge space and time complexity of the original time series Transformer and channel independent model (no patching).
Full Benchmark of Ablation Study. The full benchmark is shown in Table 10, which is a com- pleted version of Table [ in the main text. We can observe that patching together with channel- independence achieves the best results from the table, especially larger datasets (Weather, Traf- fic, and Electricity) where the models are less susceptible to overfitting and thus the results would be more convincing. As one can see the improvement is robust both patching and channel- independence. It is interesting to see that the improvement on ILI dataset is significant as well.
A.4.4 INSTANCE NORMALIZATION
Normalization is a technology used in many time series model to improve the forecasting perfor- mance (Kim et al. 2022 Chen et al. 2022 Zeng et al.] 2022). In this experiment, we perform analysis on the effect of the instance normalization in our model. We train two models PStchTST/64 and PatchTST/42 with and without using instance normalization and observe the evaluated scores. As indicated in Table [I instance normalization improves the forecasting performance slightly on two models. However, even without instance normalization operator, PatchTST still outperforms other Transformer methods on most of the datasets. This is to confirm that the improvement mainly comes from patching and channel independence designs.
A.5
MORE RESULTS ON SELF-SUPERVISED REPRESENTATION LEARNING
A.5.1 FULL BENCHMARK OF MULTIVARIATE FORECASTING
In this section we provide a full benchmark of multivariate forecasting results with self-supervised PSt in Table 12 which is an extended version of Table 4
16

---

Published as a conference paper at ICLR 2023
PaccTTST
Models
FEDformer
P+CI
Original
96
0.152
0.199
0.164
0.213
0.168
0.225
0.177
0.236
0.238
0.314
Wealtee
192
0.197
0.243
0.205
0.250
0.213
0.262
0.221
0.270
0.275
0.329
0.255
0.289
0.266
0.300
0.339
0.377
720
0.320
0.336
0.327
0.343
0.351
0.359
0.340
0.353
0,389
0.409
96
0.367
0,251
0.397
0.271
0.595
0.376
0.576
0.359
11J Electricity Trafiic
192
0.385
0.259
0.411
0.276
0.612
0.387
0,610
0,380
0.375
720
0.434
0,287
0.457
0.309
0.621
0.375
96
0.130
0.222
0.136
0.231
0.196
0.307
0.205
0.318
0,186
0.302
192
0.148
0.240
0.164
0.263
0,215
0.323
0.197
0.311
0.262
0.228
0.213
0.328
720
0.202
0,291
0.219
0.312
0.244
0.345
0,233
0.344
24
1,522
0.814
2.111
1.048
2.157
0.964
2.737
1.081
2,624
1.095
1.430
2,000
1.002
2.564
1.058
2.126
2.516
1.021
1.673
0.854
2.167
1.029
2.348
1,022
2.178
0.971
2.505
1.041
60
1.529
0.862
2.075
1.021
2,486
1.065
2.354
1.026
2,742
1.122
0.375
0.399
0.365
0.395
0.416
0.438
0.455
0.459
0.376
0.415
B1Th2 ETThl
0.446
336
0.431
0,436
0.430
0.433
0.484
0,480
0.514
0.503
0.444
0.462
720
0.449
0.466
0.449
0.454
0,500
0.494
0.531
0.520
0.469
0.492
0.274
0.336
0.277
0.337
0,334
0.388
0.348
0.394
0.332
0.374
0.446
336
0.331
0,380
0.333
0.383
0.361
0.414
0.369
0.419
0,400
0.447
720
0.379
0.422
0.379
0.420
0.423
0.448
0.433
0.458
0.412
0.469
0.300
0.354
0.326
0.370
0.326
0.390
ETTm2 ETTml
192
0.332
0.369
0.333
0.374
0.391
0.405
0.373
0.398
0.365
0.415
336
0.366
0.392
0.369
0.397
0.427
0.425
0.415
0.421
0.392
0.425
720
0.420
0.424
0.413
0.423
0.481
0.457
0.480
0.459
0.446
0.458
0.259
0.195
192
0,220
0,292
0.223
0.295
0.259
0.314
0.265
0.328
0,252
0.318
336
0.278
0.329
0.279
0.330
0.297
0.345
0.323
0.365
0,324
0.364
720
0.367
0.385
0.370
0.387
0.400
0.404
0.469
0.444
0.410
0.420
Table 10: Ablation study of patching (P) and chamnel-independenee (CI) in PatchTST/42. A full
benchmark regarding Table 7 The best results are in bold. "-' in table means the model runs out of GPU memory (NVIDIA A40 48GB) even with batch size
17

---

Published as a conference paper at ICLR 2023
Models
PiRchTSST/644 (+in)
PatchTST244 (-in)
(44))
PachTSTT/42 (-in)
FEDiormer
Ausoliemmer
Infommer
Metric
MSE
MAP
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
0.149
0.198
0.161
0.152
0.199
0.210
0.238
0.314
0.249
0.529
0.354
0:405
Wenlher
192
0.194
0.241
0.201
0.254
0.192
0.243
0.199
0.250
0.275
0.329
0.325
0.370
0,499
0.434
336
0.245
0.282
0.253
0.298
0.219
0.283
0.248
0.291
0.339
0.377
0.351
0.391
0.583
0.513
0.314
0.334
0.323
0.320
0.335
0.342
0.389
0.499
0415
0426
0916
0.705
96
0.360
11249
0.413
0.295
0.367
0.251
0.425
0.299
0.576
0.359
0.597
0.371
0.733
0.410
Electrictt Traflic
192
0.379
0.256
0.425
0.302
0.385
0.259
0.439
0.302
0.610
0.380
0.607
0.382
0.777
0.435
336
0.392
0.264
0.435
0.307
0.398
0.265
0.456
0.316
0.608
0.375
0.623
0.387
0.776
0.434
720
0,432
0,286
0.473
0.321
0.434
0.287
0.488
0.333
0,621
0.375
0.639
0.395
0.827
0.166
96
0.129
0.222
0.133
0.230
0.130
0.222
0.131
0.226
0.186
0.302
0.196
0.313
0.304
0.393
0.147
0.240
0.148
0.344
0.148
0.240
0.150
0.344
0.197
0.311
0.211
0.334
0.327
0.417
336
0,163
0.259
0.164
0.262
0.167
0.261
0.168
0.267
0.213
0.328
0.214
0.327
0.333
0.422
720
0.197
0.290
0.196
0.291
0.202
0.291
0.201
0.298
0.233
0.344
0.236
0.342
0.351
0.427
1,319
0.754
3.563
1.317
1.522
0.811
3.489
1.345
2.624
1,095
2.906
1.182
4.657
1.419
36
1,579
0.870
3.426
1.205
1.430
0.834
4.629
1.550
2,516
1.021
2,585
1.038
4.650
1.463
1.553
00:155
4.309
1.449
1.673
0.854
3.746
1.383
2,506
1.041
3.024
1.145
5.004
1.542
1,470
0.788
4.065
1.402
1.529
0.862
5.174
1.622
2,742
1.122
2.761
1.114
5,071
1.543
96
0.370
0:400
0.385
0.410
0.375
0.399
0.388
0:412
0,376
0.415
0.435
0.446
0.941
0.769
192
0.413
0.429
0.417
0.432
0.414
0.421
0.43
0.438
0.423
0.446
0.466
0457
1.017
0.786
336
0.422
0.440
0.439
0.449
0,431
0.436
0.454
0.458
0,444
0,462
0,486
0.487
1.038
0.784
720
0.447
0.468
0.478
0.494
0.449
0.466
0.494
0.497
0.469
0.492
0.515
0.517
1.144
0.857
0.274
0.337
0.299
0.359
0.274
0.336
0.374
0.332
0.374
0.332
0.368
1.549
0.952
192
0.341
0.382
0.354
0.401
0.339
0,379
0,402
0,422
0,407
0,446
0,126
0.434
3.792
1.512
336
0.329
0. 1.384
0.374
0.420
0.331
0.380
0.448
0.465
0.400
0.447
01477
0.49
4.215
1.642
0.379
0,422
0.479
0.492
0.379
0.422
0.588
0,412
0,469
0,433
0,490
3,656
1.619
96
0.293
0.346
0.308
0.358
11,290
0.342
013008
0.358
0,326
0,390
0,510
0,492
01626
0.500
192
0.333
0.370
0.335
0.375
0,332
0.369
0.356
0.390
0.365
0.415
0.514
0.495
0.725
0.619
336
0.369
0.392
0.362
0.392
0,366
0,392
0.411
0,392
0.125
0.510
0.192
1.005
0.711
720
0.416
11.40
0.432
0.429
(1.420)
0:424
0.430
0.439
0.446
0,458
0,527
0.493
1,133
0.845
BTTm2
96
0.166
0.256
0.172
0.258
0.165
0.255
0.167
0.257
0.180
0.271
0.205
0.293
0.355
0.402
0.223
0.296
0.245
0.220
0.292
0.226
0.303
0.252
0.318
0.278
0.336
0.595
0.5%6
336
0.274
0.329
0.306
0.346
0,278
0.329
0.301
0.348
0.324
0.364
0.343
0.379
1.270
0.871
720
0.362
0.385
0.391
0.404
0.367
0.385
0.392
0.407
0.410
0.420
0/114
0.119
3.001
1.267
Table 11: Multivariate long-term forecasting results of supervised PatchTST with instance normal- ization (+in) without instance normalization (-in). The best results are in bold and the second best are underlined. Although the models perform slightly better with instance normalization, compared to other Transformer models, the proposed approach achieve significantly better forecasting on most of the datasets even without instance normalization.
A.5.2 FULL BENCHMARK OF TRANSFER LEARNING
In this section we provide Table 1 which contains the results of pre-training on Electricity dataset then transferred to other 6 datasets. Except Traffic data, the number of time series employed in the pre-training is much larger than the number of series during fine-tuning. It is a full version with respect to Table 5 in the main text and more cogently proves the capability to do transfer learning using our Patch' TST model.
A.6 ROBUSTNESS ANALYSIS
A. RESULTS WITH DIFFERENT RANDOM SEEDS
The results reported in the main text and appendix above are run with the fixed random seed 2021. To examine the robustness of our results, we train the supervised PatchTST model with 5 different random seeds: 2019, 2020, 2021, 2022, 2023 and calculate the MSE and MAE scores with each selected seed. The mean and standard derivation of the results are reported in Table [4] It is clear that the variances are considerably small which indicates the robustness against choice of random seeds of our model.
We also validate the self-supervised PatchTST model on different runs. We pre-train the model once and fine-tune the model 5 times with different random batch selections. The mean and standard derivation across different runs are also provided in Table [4] We also observe that the variance is insignificant especially on large datasets while higher variance can be seen on smaller datasets.
18

---

Published as a conference paper at ICLR 2023
Models
PaREETST
DLinear
FEDformer
Autoformer
Informer
Finc-tuning
Lin. Prob.
Sup.
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
Weather
0.144
0.193
0.158
0.209
0.152
0.199
0.176
0.237
0.238
0.314
0.249
0.329
0.354
0,405
192
0.190
0.236
0.203
0.249
0.197
0.243
0.220
0.282
0.275
0.329
0.325
0.370
0.49
0.434
336
0.244
0.280
0.251
0.285
0.249
0.283
0.265
0.319
0.339
0.377
0.351
0.391
0.583
0.543
720
0.320
0.335
0.321
0.336
0.320
0.335
0.323
0. 0.362
0.389
0.409
0,415
0.426
0.916
0.705
0.352
0.244
0.399
0.294
0.367
0.251
0.410
0.282
0.576
0.359
0.597
0.371
0.733
0.410
ETTT22 ET/ml ET ElTbl Electricity Trffic
192
0.371
0.253
0.412
0.298
0.385
0.259
0.423
0.287
0.610
0.380
0.607
0.382
0.777
0.435
336
0.381
0.257
0.425
0.306
0.398
0.265
0.436
0.296
0.608
0.375
0.623
0.387
0.776
0.434
720
0.425
0.282
0.460
0.323
0.434
0.287
0,466
0.315
0.621
0.375
0.639
0.395
0.827
0.466
96
0.126
0.221
0.138
0.237
0.130
0.222
0.140
0.237
0.186
0,302
0.19%
U.313
0.304
0.393
192
0.145
0.238
0.156
0.252
0.148
0.240
0.153
0.249
0.197
0.311
0.211
0.324
0.327
0.417
336
0,164
0.256
0.170
0,265
0.167
0.261
0.169
0.267
0.213
0.328
0.214
0.327
0,333
0,422
720
0.193
0.291
0.208
0.297
0.202
0.291
0.203
0.501
0.235
0.344
0.236
0.342
0,351
0.427
96
0.366
0.399
0.371
0.400
0.375
0.399
0.375
0.399
0.376
0.415
0.435
0.446
0.941
0.769
192
0,431
0.443
0.411
0,428
0,414
0.421
0.405
0,416
0.423
0.446
0.456
0,457
1,007
0.786
336
0.450
0.456
0.445
0.446
0.431
0.436
0.439
0.443
0.444
0.462
D 0.486
0.487
1.038
0.784
0.472
0.484
0.487
0.478
0.449
0.466
0.472
0.490
0.469
0.492
0.515
0.517
1.144
0.857
96
0.284
0.3433
0.285
0.344
0.274
0.336
0.289
0.353
0.332
0.374
0.332
0.368
1.549
0.952
0.355
0.387
0.356
0.387
0.339
0.379
0.383
0.418
0.446
0.426
0.434
3.792
1.542
336
0.379
0.411
0.377
0.410
0.331
0.380
0.448
0.465
0.400
0.447
0.477
0.479
4.215
1.642
720
0.435
0.395
0.379
0.422
0.605
0.469
0.453
0.490
3.656
96
0.289
0.344
0.292
0.348
0.290
0.342
0.299
0.343
0.326
0.390
0.510
0.492
0.626
0.560
0.323
0.368
0.329
0.369
0.332
0.369
0.335
0.365
0.365
0.415
0.514
0,495
0.725
0.619
336
0.353
0.387
0.364
0.391
0.366
0.392
0,369
0.386
0.392
0.425
0,510
0.492
1.005
0.741
720
0.398
0,416
0.415
0.419
0.420
0.424
0.425
0.421
0.446
0.458
0.527
0.493
1.133
0.845
96
0.166
0.256
0.167
0.257
0.165
0.255
0.7
0.260
0.180
0.271
0.205
0.293
0.355
0.462
192
0.221
0.295
0.229
0.300
0.220
0.292
0.224
0.303
0.252
0.318
0.278
0.336
0.595
0,586
336
0.278
0.333
0.289
0.343
0.278
0.329
0.281
0. 0.342
0.324
0.364
0.343
0.379
1.270
0.871
720
0.365
0.388
0.363
0.386
0.367
0.385
0.397
0.421
0.410
0.420
0.414
0.419
3,001
1,267
Table 12: Multivariate long-term forecasting results with self-supervised PatchTST. An full bench- mark regarding Table 4 The best results are in bold.
STT
Models
tune-tuning
Lin. Prob.
Sup.
inear
FFDfomer
Autoformer
Informer
Metric
MSE
MAE
MIE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Weather
96
0.145
0.195
0.163
0.216
0.152
0.199
0.176
0.237
0.238
0.314
0.249
0.329
0.354
0.405
192
0.193
0.243
0.205
0.252
0.197
0.243
0.220
0.275
0.329
0.325
0.370
0.434
336
0,244
0.280
0.253
0.289
0,249
0,283
0.265
0.319
0.339
0.377
0.351
0.391
0.583
0,543
720
0.521
0.337
0.320
0.320
0.335
0.323
0.362
0.409
0.415
0.426
0.705
ETTm2 ETTml E77h2 ETThl Traffic
96
0.388
0.273
0.400
0.288
0.367
0,251
0.410
0.282
0.576
0.359
0.597
0.371
0.733
0,410
192
0.400
0.277
0.412
0.293
0.385
0.259
0.423
0.287
0.610
0.380
0.607
0.382
0.777
0.435
336
0.408
0.280
0.425
0.307
0.39%
0.265
0.436
0.296
0.608
0.375
0.623
0.387
0.776
0.434
720
0.447
0.310
0.457
0.317
0.434
0.287
0.466
0.315
0.621
0.375
0.639
0.395
0.827
0.466
96
0.368
0.398
0.372
0.402
0.375
0.399
0.375
0.399
0.376
0,155
0.435
0,446
0.541
0.769
192
0.425
0.439
0.411
0.428
0.414
0.421
0.405
0.416
0.423
0.446
0.456
0.457
1.007
0.786
336
0.470
0.471
0.442
0.454
0.431
0.436
0.439
0.443
0.444
0.462
0.486
0.487
1.038
0.784
720
0.472
0.484
0.497
0,501
0.449
0.466
0.472
0.490
0.469
0.492
0.515
U.517
1.144
0.857
96
0.285
0.345
0.280
0.341
0.274
0.334
0.289
0.355
0.332
0.374
0.332
0.368
1,349
0.952
192
0.350
0.388
0.350
0.387
0.339
0.379
0.383
0.418
0.407
0.446
0.426
0.434
3.792
1.542
336
0.378
0.410
0.373
0,410
0.331
0,380
0.448
0.465
0,400
0.447
0.477
0.479
4.215
1.642
720
0.401
0.438
0.398
0.436
0.379
0.422
0.605
0.551
0.412
0.469
0.453
0.490
3.656
1.619
96
0.288
0.345
0.291
0.346
0.290
0,342
0.299
0.345
0.326
0.390
0,510
0.492
0.626
0.560
192
0.330
0.372
0.335
0.373
0.332
0.369
0.335
0.365
0.365
0.415
0.514
0.495
0.725
0.619
336
0.359
0.392
0.365
0.391
0.366
0.392
0.369
0.392
0.425
0.510
0.492
1.005
0.741
720
0.406
0.421
0.423
0.424
0.420
0.424
0.425
0.421
0.446
0.458
0.527
0.493
1.133
0.845
96
0.164
0.256
0.166
0.257
0.165
0,155
0.167
0.260
0.180
0.271
0.205
0.293
0.155
0.462
192
0.223
0.296
0.221
0.295
0.220
0,292
0.224
0.303
0.252
0.318
0,278
0.336
0.595
0.5%6
336
0.277
0.332
0.277
0.332
0.278
0.329
0.281
0.342
0.324
0.364
0.343
0.379
1,270
0.871
720
0.365
0.387
0.368
0.389
0.367
0.385
0.397
0.421
0.410
0.420
0.414
0.419
3.001
1.267
Table 13: Transfer learning task: PatchTST is pre-trained on Electricity dataset and the model is
transferred to other datasets. A full benchmark regarding Table 13
The best results are in bold.
A.6.2 RESULTS WITH DIFFERENT MODEL PARAMETERS
To see whether PatchTST is sensitive to the choice of Transformer settings, we perform another ex- periments with varying model parameters. We vary the number of Transformer layers L = {3, 4, and select the model dimension D {128, 256} while the inner-layer of the feed forward network is F = 2D. In total, there are 6 different sets of model hyper-parameter to examine. Figure 5 shows the MSE scores of these combinations different datasets. Except ILI dataset reveals high variance with different hyper-parameters.
settings, other datasets are robust to the choice of model
19

---

Published as a conference paper at ICLR 2023
1.
STtchTST/42 supervised
PatchTST/42 self- supervised
Metric
MSE
MAE
MSE
MAE
0.152540.00224
0.2002=00.0025
0.1450=0.000
0.1937=000001
Weather
192
0.1975+0.0015
0,2434+0.0010
0.089300.0003
0.2364=0.0006
336
0.2494a0.0012
0.2041
0.2413=0.0003
0.2774=0.0005
720
0.319440.0002
0.3352=0.0003
0.3156=0.00020
0.331620.0016
0.366920.0000
0.250Mi00.0007
0.3528:000022
0.2443:000001
E7Thh2 ETThl Electricity Trallic
192
0,3858+0.000 0.000
0.2586=0.0004
0.3729=0.0013
0.2531=0,0009
336
0.399420.0011
0.2672a0.0016
0.3846=000000
0.258820.0011
720
0,438340.000
0.2913+0.0104
0.4241=0.0007
0.2816=0.0010
0.130420.0000
0,223420.0006
0.1256=00.007
0.2210=0,0003
192
0. 0.1482+0.0002
0,2403=0.0002
0.1451=0.0002
0.2
336
0.165900.000
0.259620.0000
0.1624=0.0010
0.257620.0009
720
0.2019+0.0000
0.2017#000006
0.1990=0.0002
0.2916=0.0002
96
0.3752a0.0000
0,399920.0004
0.37000:0000035
0,4001
192
0.4127=0.0012
0.4207
0.4146=0.0012
0.4287=0.0013
336
0.4278::000033
0.4334:0.00028
0.4285=0.0018
0.4402:0.0017
0.4462+0.00035
0.4637s0.0027
0.4670=0.0052
0.4768=0:0033
96
0.2749=0.0005
0,3363:2000006
0,286920.003
0.343920.0016
192
0.3385=0.0010
0.3789200.0014
0.3523=0.004B
0.3855=0.0027
336
0.3288=0.0010
0.3823=0.0007
0.3779=0.0057
0.4112=0.003
0.3784=0.0010
0,4212:0.0009
0.3993=0.0054
0.438520.0038
96
0,2893+0.0009
0:341500.0007
0.2876=0.0012
0.3427=0.0011
ETTm2 ET1ml
192
0.3316:40.0000 0.1695=0.0007
0.3296:0000066
0.168820.0016
336
0.3661#0.0022
0.3914#0.0012
0.35K3=0.0015
0.3879=000006
720
0.4200a0000056
0.424340.0033
0.4094000044
0.4193=00,0033
96
0.1647=000011
0,2538=0.0010
0.1637=0.0000
0.2537=00.0024
192
0,222340.0018
0,293620.0014
0.2175=0.0011
0.290820.0013
0.277540.0020
0.3297.0.0010
0.2706:0000066
0.3260:0.0016
720
0.3648:0.0022
0,383320.0010
0.353920,0023
0.379920.0024
Table 14: Multivariate long-term forecasting results with different random seeds in supervised and self-supervised PatchT The best results in bold.
etth1
etth2
weather
illness
0.386
0.3
0.15
2.3
0.378
0.29
0.145
0.37
0.14
0.28
0.362
0.135
0.354
0.27
2.0
1
2
3
4
6
1
2
3
4
6
1
3
4
5
6
3
4
5
6
Parameter Combination ettml
Parameter Combination ettm2
Parameter Combination traffic
Parameter Combination electricity
0.17
0.29
0.13
0.165
MSE
0.16
0.38
0.125
0.28
0.155
0.37
0.12
0.27
0.15
0.115
4
1
1
1 3 6
Parameter Combination
Parameter Combination
Parameter Combination
Parameter Combination
Figure 5: MSE scores with varying model parameters. Each bar indicates the MSE score of parameter combination. The combinations (L, D) (3, 128). (3, 256). (4, 128), (4, 256), (5, 128), (5, 256) are orderly labeled from I to 6 in the figure. The model is run with supervised PStchTST/42 forecast 96 steps. For Traffic and Electricity datasets, we reduce the maximum number of epochs to 50 to save computational time.
A.7 CHANNEL-INDEPENDENCE ANALYSIS
Intuitively, channel-mixing models should outperform the channel-indeppendet ones since they have more flexibility to explore the cross-channel information, while with channel-independent the cor- relation is indirectly learnt via weight sharing. However, this is contrast to what we have observed. In Section A.7.I we will provide in-depth analysis on why fhannel-independeer casting performance than channel mixing, and in Section A.7.2 we show that channel- indeleenddneeeece is general technique that can be used not only for TST but also for the other models.
has better fore-
20

---

Published as a conference paper at ICLR 2023
A.7.1 CHANNEL-INDEPENDENN VS CHANNEL- MIXING
We find 3 key factors that makes channel-independen models more preferable:
Since each time series is passed through the Transformer separately, generates its own attention maps. That means different series can learn different attention patterns for their prediction, as shown in Figure 6 In contrast, with the channel mixing approach, all the series share the same attention patterns, which may be harmful if the underlying multivariate time series carries series of different behaviors. Figure f reveals an interesting observation that the prediction of unrelated time series relies on different attention patterns while similar series can produce similar maps (e.g. series 11, 25, and 81 contain similar patterns while they are different from others). We suspect this adaptability is one of the main reasons why PStchTST performs much better forecasting than Informer and other channel-mixing models.
Adaptability:
Channel-mixing models may need more training data to match the performance of the channel- independent ones. The flexibility of learning cross-channel correlations could be double- edged sword, because it may need much more data to learn the information from different channels and different time steps jointly and appropriately, while channel- -enenellnddeeeeentt models only focus on learning information along the time axis. We examine this assumption by exper- iments where we train the models with varying training data size and and the result is shown on left panel of Figure 7 It is clear that channel-independen models converges faster against the size of training data. As what we have observed in the figure and Table 2, the size those widely used time series datasets may not be large enough for channel-mixin models to obtain similar performances in supervised learning.
Channel-independent models are less likely to overfit data during training. We record the MSE loss on test data and plot on the right panel of Figure 7 Channel-mixing models show over- fitting after a few initial epochs, while channel-independen models continue optimizing the loss with more training epochs. The best trained models are determined by validation data, which are approximately the lowest points in the test loss curves. It is clear that the forecasting performance of channel-independen models are better.
Furthermore, we would like to comment on a few additional technical advantages of channel- independence: (1) Possibility of learning spatial correlations across series: Although we haven't 't focused on this research in our paper, the hhannel-nddependene design can be naturally extended to learn cross-channel relationships by using methods like graph neural networks (Cao et al.) 2020 Chen et al. 202T). (2) Multi-task learning where different loss types can be imposed on different time series where the same underlying Transformer model is shared. (3) More robust to noise: If noise is dominant in one or several series, this noise will be projected to other series in the embed- ding space if we mix channels. Channel independence can mitigate this problem by only retaining the noise in these noisy channels. We can further alleviate the noise by introducing smaller weights the objective losses that associate with noisy channels.
A.7.2 PERFORMANCE OF CHANNEL-INDEPENDEEN ON OTHER MODELS
To show that channel-independenee is general technique that can be applied to the other models, we apply it to Informer (Zhou et al. 2021 Autoformer (Wu et al. 2021), and FEDformer (Zhou al. 2022). The results are shown in Table 15 The channel-independenl technique can improve the forecasting performance of those models generally. Although they are still not able to outperform PStchTST which is based on vanilla attention mechanism, we believe that more performance boost and computational reduction can be obtained with more advanced attention designs incorporating the channel-independence architecture.
21

---

Published as a conference paper at ICLR 2023
Models
PaccTTST/42
Informer
Informer-CI Cl
Autoformer
Autoformer CI
FEDromeer
FEDiomeer-C CT
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Weather
96
0.152
0.199
0.300
0.384
0.174
0.232
0.266
0.336
0.227
0,289
0.217
0.296
0.214
0.278
192
0.197
0.243
0.598
0.544
0.214
0.270
0,307
0.367
0,269
0,318
0,276
0.336
0.258
0.322
336
0.249
0.283
0.578
0.523
0.266
0.310
0.359
0.395
0.315
0.344
0.339
0.380
0.302
0.336
720
0.320
0.335
1.059
0.741
0.327
0.356
0,419
0.428
0.384
0.389
0.403
0.428
0.374
0.369
0.251
0.705
0.402
0.613
0.388
Traffic
0.259
0.720
0.407
0.616
0.382
0.604
0,373
336
0.398
0.265
0.777
0.420
0.750
0.421
0.622
0.337
0.621
0.383
720
0.434
0.2687
0,864
0.472
0.660
0.408
0.626
0.382
1L1 Electricity
96
0.130
0.222
0.274
0.368
0,203
0.299
0.201
0.317
0.193
0.308
192
0.148
0.240
0.296
0.386
0.221
0.316
0.222
0.334
0,201
0,315
336
0.167
0.261
0.300
0.394
0.241
0.337
0.231
0.338
0.214
0.329
720
0.202
0.291
0.373
0.439
0.314
0.391
0,254
0.361
0.246
0.355
24
1.522
0.814
5.764
1.677
5.514
1.629
3.483
1.287
4,210
1.500
3.228
1.260
3,280
1,264
1.430
0.834
4.755
5.515
1.628
3.103
1.148
2,809
2.679
1.080
2,862
1.126
48
1.673
0.854
4.763
1.469
5,263
1.574
2.669
1,085
3,218
1.267
2.622
1.078
2,834
1.150
60
1.529
0.862
5,264
1,564
5,330
1.602
2,770
1.125
3.627
1.396
2.857
1.157
3,115
1,240
96
0.375
0.399
0.865
0.713
0.590
0,517
0.449
0.459
0,414
0.421
0.376
0.419
0.387
0.407
ETT12 ET1h
192
0.414
0.421
1.008
0.792
0.677
0.566
0.500
0.482
0.453
0.448
0.420
0.448
0.439
0.438
336
0.431
0.436
1.107
0.809
0.710
0.600
0.521
0.496
0.496
0.468
0.459
0.465
0.479
0.455
720
0.449
0.466
1.181
0.865
0.777
0.660
0,514
0.512
0.662
0.568
0.506
0.507
0.485
0.478
96
0.274
0.336
3.755
1.525
0.390
0.410
0.358
0.397
0.337
0.373
0.346
0.388
0.297
0.348
192
0.339
0.379
5.602
1.931
0.456
0.452
0.409
0.419
0.380
0.482
0.486
0.432
0.443
720
0.379
0.422
3.647
1.625
0.843
0.661
0.515
0.511
0.443
0.463
0.463
0.474
0.422
0.444
96
0.290
0.342
0.672
0.571
0,383
0.414
0.505
0.475
0.455
0.441
0.379
0.419
0.408
0.413
ETTml
192
0.332
0.369
0.795
0.669
0.420
0.434
0.583
0.496
0.598
0.512
0.426
0.441
0.445
0.432
336
0.366
0.392
1,212
0.871
0.465
01.667
0.621
0.537
0,566
0.504
0.445
0.459
0.476
0.452
720
0.420
0.424
1.166
0.823
0.529
0.502
0.671
0.561
0.680
0.557
0.543
0.490
0.533
0.481
96
0.165
0.255
0.365
0.453
0.208
0.298
0.255
0.339
0,218
0.308
0.203
0.287
0.198
0.284
ETTm2
192
0.220
0.292
0.533
0.563
0.274
0.345
0.281
0.340
0.281
0.339
0.269
0.328
0.259
0.320
336
0.329
1.363
0.887
0.339
0.372
0.336
0.370
0.366
720
0.385
1.338
0.433
0.432
0.428
0.418
Table 15:
Channel-independence for other models. CI denote channel-independenne. Basclines
without CI are cited from Zeng et al. (2022). The better results between CI and non-CI on versions are in bold. PatchTST/42 is placed the left for easy reference to other Cl-based models. '-' denotes
running out of GPU memory even with batch size 1, or exceeding the maximum running time (12
hours).
22

---

Published as a conference paper at ICLR 2023
attention map
forecast vs. actual
atterdion map
Forecast actubl
0.5-
1000
11
0.0
W
-1.5-
-2.0
130
attention
actusi
attentton
Porecast
1.-
1.5
1.5-
0.5
sen1e 02
-0.5
-1.0
3.5
-1.5
20
40
1.00
200
250
20
150
150
200
map
map
1.5
1.0
13
22
-0.5
-10-
02-
-1.5
20
40
50
140
150
200
252
20
40
100
150
200
attertion mep
Forecest VS. actuell
attersion map
forecest.oo Vs. actue
3.0
0.5
22190
-0.5
-1.0
-1.0
20
40
1.50
200
26
1.50
1.50
250
attertion map
Forred andll vS.. actiuall
atternion map
For sereecco aesi voa. act.uai
-0.5
-2.9
attention map
forecast uS. actual
attercton
Forocast actusl
1.0
1.5
0.5
3.0
0.0
23-
0.5
-0.5
12-
-100
20
40
200
120
attercion
1.-
1.-
14h62 37
20-
20-
0.5
30-
0.5
-0.5
-15-
20
40
50
50
100
130
200
200
20
40
50
120 forecast
150
200
250
map
forocast
map
1.0
20-
32
23
50
02
20
40
65
50
120
150
200
252
20
40
150
150
200
250
actual
predict
Figure 6: Attention maps and the forecasting of few time series from Electricity dataset with supervised PTtc4 Attention map is calculated by averaging the attention matrices over all the heads and across all the layers. For each time series, show the attention map and the prediction in orange. The blue curves are the actual data. The curves before the back lines are the actual input data. Channel-independencc design allow each series learn its attention map for forecasting in which the pattern can be more similar for more correlated series and different otherwise.
23

---

Published as a conference paper at ICLR 2023
chanrel-ireeeendence
Seennel-iddeeeddeeee chenmel-mixing
chanrel-mixing
0.24
0.22
0.20
0.20
0.18
0.18
0.16
0.0
0.2
0.4
0.5
0.8
1.0
10
train size
Figure 7: Chamncl-independence vs channel-mixing on Weather dataset. The base model is PatchTST/42, and the prediction length is 96. We plot the mean values and error bars with differ- ent random seeds: {2019, 2020, 2021, 2022, 2023]. Left Panel: Test loss vs train size. Here, train size denotes the fraction of the training data that is used to learn the model from scratch. Channel- independence contributes to a quicker convergence as more training data is available. Right Panel: Test loss vs epochs. Here, we use full train data and plot first epochs. Channel-mixing model quickly overfits the data.
24


Reviews:
[{"review_id": "f6dxw7p61go", "reviewer": "Reviewer_zBqz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a transformer architecture that forecast multivariate time series channel-wise. In particular, the model employs patching by grouping values in temporal neighborhoods, so that the locality is enhanced and efficiency is improved. The model can also be trained in a self-supervised way by masking random patches, and it is shown such self-supervised pre-training can lead to superior performance with proper fine-tuning.", "summary_of_the_review": "Overall I think the current version of this paper does not meet the bar of ICLR due to limited novelty and insufficient clarity.", "strength_and_weaknesses": "Strength\n1. The essential idea of the paper is well motivated and presented.\n2. Extensive experiments are done to compare the proposed model with a variety of baselines.\n\nWeakness:\n1. The proposed patching is not novel as LogTrans (Li et al. 2019) has already proposed to use convolution before transformer to enhance locality. The linear processing in PatchTST with fixed path size and stride is not different from 1D convolution, even though a larger grid of configuration is investigated in the experiments.\n2. The idea of masked modeling in representation learning is also marginally novel. The motivation of such a strategy is vague in certain parts. For example, in section 3.2, paragraph 2, _Since time series is often temporally redundant, the masked values at the current time step can be easily referred from neighboring values, which makes the reconstruction process trivial and thus the representation may not carry important abstract information_, the triviality here is not clear: in which case (\"redundancy\") the current time step can be too \"easily\" to refer? Masked modeling itself aims to refer the masked token/patch from contextual objects, why is it deemed trivial in time series? Moreover, in paragraph 3, the prediction in multivariate time series can also employ certain degree of weight sharing to avoid $(L\\cdot D) \\times (M\\cdot T)$ parameter matrices.\n3. Detailed questions: \n+ in supervised learning, the model is said to have a flatten layer with linear head to get univariate time series. When the patches have overlapping, how is the prediction within the overlapping determined?\n+ how is the linear probing and end-to-end fine-tuning done after representation learning? Is linear probing simply reconstructing masked patches again but not update the backbone layers except the final linear head? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YrNMB3hCNL", "reviewer": "Reviewer_2c5h", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an efficient design of Transformer-based models (PatchTST) for multivariate time series forecasting and self-supervised representation learning. It segments time series into patches following similar philosophy of ViT and assume channels are independent. Extensive experiments on 8 eight datasets show that it can outperform state-of-the-art models while requires less compute.", "summary_of_the_review": "\n\n I update my score to 8 since the authors' response resolve concerns and provide source code on Github.\n\n\\------------------------------------------------------------------------------\n\nThe idea of utilizing patch to deal with time series is interesting and have good empirical results. However, channel independence is hardly novel. In addition, I have concerns regarding experimental settings and am not sure if baselines are under-estimated and results statistically significant.", "strength_and_weaknesses": "Strength\n\n*  The idea of utilizing patch like ViT to reduce sequence length and better capture locality is quite simple and effective\n* Extensive experiments on multiple datasets show that it can outperform recently state-of-the-art methods in different forecasting horizons and ablation study regarding patches and channel independence is conducted across multiple datasets.\n*  This paper is well-organized and easy to understand\n\nWeaknesses\n\n* Channel independence is hardly novel as previous work, e.g. DeepAR [1] and LogTrans, also have this assumption. These two work model multivariate time series by sharing backbone parameters (LSTM/Transformer) of different channels during training, which can reduce parameter numbers and overfitting issue. \n\n* In experimental setups, \"For Transformer-based models, the default lookback window is L = 96\". However, this look-back window length can be quite short and may not be ideal especially when forecasting window is long, e.g. 336. In figure 2 (Electricity-T=720), best look-back window for Autoformer is  192/336 rather than 96. This setting make me have some concerns that baselines may be under-estimated.\n\n* Self-supervised methods are only a little better and standard supervised fine-tuning. I am wondering if authors can have multiple runs and report mean and standard deviation so that readers may know how significant it is.\n\n[1] Flunkert, Valentin et al. “DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks.” ArXiv abs/1704.04110 (2020): n. pag.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VOIvLJmX0Q", "reviewer": "Reviewer_FEPh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies an efficient way to use the transformer-based model in time-series forecasting tasks. The proposed model, PatchTST, first folds the sequence into several patches which significantly reduces the total sequence length and then splits the multi-channel forecasting signal into independent tasks. In numerical tests on several real datasets, the proposed model reaches the SOTA performance. The authors also test their model on self-supervised learning and verify a promising sign that the transformer-based models could be fit on time-series pretraining tasks.", "summary_of_the_review": "This paper proposes an efficient transformer-based model in time-series forecasting tasks and strong numerical performance is presented. My major concern is the technical novelty as the proposed model is mainly a direct application of standard VIT.\n\n\nMinor issues:\n\n1. The \"64 words\" in the title is a little bit misleading. In the famous VIT paper \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\", the patch size is 16 x 16. However, the patch size/length in the current paper is around 8-16 which doesn't match 64. I suggest authors modify the title.\n\n2. More ablation studies. In the current presentation, the ablation without instance norm is not included. In the literature, using the instance norm or the equivalent RevIN in [1] could significantly improve the numerical performance. In order to further highlight the effectiveness of the patch and the channel independence ideas, I'm wondering if adding one more ablation study related to the instance norm will be helpful.\n\n3. Robustness test. In time-series forecasting tasks, usually, the results are very sensitive to the model/training parameters.  How about the proposed model? Can we also include a section to discuss the robustness of the proposed model?\n\n\n[1] Kim, T., Kim, J., Tae, Y., Park, C., Choi, J. H., & Choo, J. (2021, September). Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations.\n\n\n", "strength_and_weaknesses": "Strength:\n1. Good numerical performance forecasting, reaching the SOTA results on several real datasets.\n\nWeakness:\n1. Based on the current presentation, the proposed model is analogous to the standard vision transformer (VIT). The technical novelty is limited.\n2. The pretraining datasets are relatively small compared to datasets in CV/NLP fields and the domains of the datasets merely share the mutual underlying knowledge. Combining with marignal improvements reported in Table 4 and Table 5, I tends to believe that the pretraining stage isn't meaningful to the considered datasets.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4X6HH0VGYp", "reviewer": "Reviewer_3Fzy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. There are two main components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. The experiment results show the effectiveness of their method.", "summary_of_the_review": "The paper proposes two main technics for time series tasks. Even though the experiment results are better than baselines, the improvement is marginal and there miss some deep analysis. Thus, I vote for marginally below the acceptance threshold.", "strength_and_weaknesses": "Strength: Multivariate time series forecasting and self-supervised representation learning are important tasks for time series. The proposed methods have some improvements on both tasks.\n\nWeaknesses: I have the following questions.\n(i) Segmentation of time series is a naive extension from computer vision (MAE). Thus, it lacks novelty in my perspective.\n(ii) In my view, channel-independence is an orthogonal technic compared with other methods. In other words, the authors can perform channel-independence on other baselines. Can the authors show the performances with channel-independence in Table 3 and Table 4?\n(iii) Can you provide some analysis for channel-independence? Since different channels have correlations compared with others, why channel-indpendence can improve performance?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster