One Fits All: Power General Time Series Analysis by Pretrained LM
Authors: Tian Zhou
Peisong Niu
Xue Wang
Liang Sun
Rong Jin
Venue: NeurIPS 2023 spotlight (2023)
Keywords: general time series analysis
time series forecasting
cross modality knowledge transfer; pretrained language model;
TL;DR: One Fits All: Power General Time Series Analysis by Pretrained LM, a work that synthesis practical performance and theoretical analysis

Abstract: Although we have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where a unified model can be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre-trained model for time series analysis is the lack of a large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks, as illustrated in Figure1. We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and a crucial step towards understanding the universality of a pre-trained transformer. 
The code is publicly available at https://anonymous.4open.science/r/Pretrained-LM-for-TSForcasting-C561.

One Fits All: Power General Time Series Analysis by Pretrained LM
Tian Zhou' Peisong Niu' Xue Wang" Liang Sun Rong Jin
(tian. zt, nwwpiisong....... nps, xue. w, were.w,liannn llang....... s sun,jinrong jr 0al mallibaba-inne inc. com
Abstract
Although have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where unified model be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre- pre-trained model for time series analysis is the lack of large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to comparable or sate-offthhe-att performance in all main time series analysis tasks, as illustrated Figure [ We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and crucial step towards understanding the universality of a pre-trained transformer. The code is publicly available at https://gi gitggttthuuu.... www.tauu..oomm.aaaaa COw.DAMO-DI-MMM Ltts//OeeFFFt__111_
Introduction
Time series analysis is a fundamental problem Hyndman & Athanasopoulos (2021) that has played an important role in many real-world applications Wen al. (2022), such as retail sales forecasting Böse et al. (2117; Courty & Li [999). imputation of missing data for economic time series Friedman (1962) anomaly detection for industrial maintenance Gao al. (2020), and classification of time series from various domain [smail Fawaz et al. (2019). Numerous statistical and machine learning methods have been developed for time series analysis in the past. Inspired by its great success in natural language processing and computer vision Vaswani et al. (2017); [Dvlin et al. (2019);
Dosovitskiy et al. (202T): Rao al. (202T). transformer has been introduced to various time series tasks with promising results Wen et al. (2023). especially for time series forecasting Lim et al. (202))
Zhou et 2002 202 Wu et al. (2021): Nie et al. (2022).
We have recently witnessed the rapid development of foundation models in NLP. The key idea is to pre-train a large language model from billions of tokens facilitate model training for downstream tasks, particularly when we have a few, sometimes even zero, labeled instances. Another advantage of
Equal contribution Corresponding authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

I Long-term Forecasting
8.0
Short-lerm
Zero-shot
Forecasting
BGPF2-hachhn
DPoxhTST
nETS@mmr
ai LightS
14.00
DIN incer
Hun
Astulomme
8:33
Dielame
Few-shot 8.30
Imputation
ua
Classification
Anomaly Detection
Figure 1: Model performance comparison on various tasks.
foundation models that they provide a unified framework for handling diverse tasks, which contrasts conventional wisdom where each task requires a specially designed algorithm. However, so far, little progress has been made to exploit pre-trained foundation models for time series analysis. One main challenge is the lack of the large amount of data to train foundation model for time series analysis. The largest data sets for time series analysis is less than 10GB Godahewa et al. (202T), which much smaller than that for NLP. To address this challenge, we propose to leverage pre-trained language models for general time series analysis. Our approach provides a unified framework for diverse time series tasks, such as classification, anomaly detection, forecasting, and few-shot or zero-shot learning, As shown in Figure T using the same backbone, our approach performs either on-par or better than the state-of oftte--f--he-rr methods for all main time series analysis tasks. Besides extensive empirical studies, we also investigate why a transformer model pre-trained from the language domain can be adapted to time series analysis with almost no change. Our analysis indicates that the self attentonn modules in the pre-trained transformer acquire the ability to perform certain non-data-dependan operations through training. These operations are closely linked principal component analysis over the input patterns. We believe it is this generic function performed by the self-attention module that allows trained transformer models to be so-called universal compute engine Lu al. (2022) or general computation calculator Giannou et al. (003). We support our claims by conducting an empirical investigation of the resemblance in model behaviors when self-attention is substituted with PCA, and by providing a theoretical analysis of their correlation.
Here summarize our key contributions follows:
We propose unified framework that uses a frozen pre-trained language model to achieve a SOTA or comparable performance in all major types of time series analysis tasks supported by thorough and extensive experiments, including time series classification, short/long-term forecasting, imputation, anomaly detection, few-shot and gero-sample forecasting.
2. We found, both theoretically and empirically, that self-attention performs a function similar to PCA, which helps explain the universality of transformer models.
3. We demonstrate the universality of our approach by exploring a pre-trained transformer model from another backbond model (BERT) or modality (computer vision) to power the time series forecasting.
The remainder of this paper is structured as follows. Section 2 briefly summarizes the related work. Section 3 presents the proposed detailed model structure. In Section 4, we conduct thorough and extensive evaluation of the performance of cross-modality time series analysis using our proposed method in seven main time series analysis tasks compared to various SOTA baseline models. Section 5

---

presents various ablation studies, and Section 6 demonstrates the universality of our proposed method using pre-trained models with another structure or pre-traines from another modality. In Section we provide theoretical explanation of the connection between self-attention and PCA. Finally, in Section 8, we discuss our results and future directions. Due to space limit, more extensive discussion of related work, experimental results, and theoretical analysis are provided in the Appendix.
2 Related Work
In this section, we provide short reviews of literature in the areas of time series amalysis, in in-modality transfer learning, and cross-modality knowledge transfer learning. We postpone the discussion of works for end-to-end time series analysis to Appendix B due to the limited space.
In-modality Transfer Learning through pre-trained models In recent years, large number of research works have verified the effectiveness the pre-trained model from NLP, CV to Vision- and-Language (VL). Latest studies for NLP focus learning contextual word embeddings for downstream tasks. With the increase of computing power, the very deep transformer models have shown powerful representation ability in various language tasks. Among them, BERT Devlin et al.) (2019) uses transformer encoders and employs masked language modeling task that aims to recover the random masked tokens within a text. OpenAl proposed GPT Radford & Narasimhan (2018) that trains transformer decoders on a large language corpus and then fine-tunes on task-specific data. GPT2 Radford al. (2019) is trained on larger datasets with much more parameters and can be transferred to various downstream tasks. Since transformer models can adapt to various inputs, the idea of pre-training can also be well adapted visual tasks. DEiT Touvron et (202I) proposed a ceacher-studens strategy for transformers with convolution neural networks (CNNs) as the teacher model and achieves competitive performance. BEiT Bao al.) (2022) converts images as visual tokens and successfully uses the BERT model in CV. However, because of the insufficient training sample, there little research on pre-trained models on general time series analysis that cover all major tasks like CV or NLP domain.
B
Cross-modalins knowledge transfer Since transformers can handle different modal tasks through tokenizing the inputs to embeddings, it is also an interesting topic whether the transformers have universal representation ability and can be used for transferring between various domains. The VI. pre-trained model VLMo Bao et al. (202T) proposed a stagewise pre-training strategy that utilizes frozen attention blocks pre-trained by image-only data to train the language expert. One of the most related works which transfer knowledge from pre-trained language model to other domains is Lu et al. (2022), which studies the strong performance a frozen pre-trained language model (LM) compared to end-to-end transformer alternative learned from other domains' data. Another relative work for knowledge transfer to the time series is the Voice2series Yang et al. (2021), which leverages a pre-trained speech processing model for time series classification and achieves superior performance. To the best of our knowledge, no previous research has investigated cross- cross-modality knowledge transfer for the time series forecasting task, let alone general time series analysis.
3
Methodology
3.1
Model Structure
The architecture we employ is depicted in Figure We utilize parameters from NLP pretrained transformer models for time series analysis, with a focus on the GPT2 model Radford et al. (2019). We also experiment with other models, such as BERT Devlin et al. (2019) and BEiT Bao et al. (0022. to further demonstrate that the universal performance of cross-domain knowledge transfer exists in a wide range of pre-trained models.
Frozen Pretrained Block Our architecture retains the positional embedding layers and self-attention blocks from the pre-traine treired models. As self-attention layers and FFN (Feedforward Neural Networks) contain the majority of learned knowledge from pre-trained language models, we opt to freeze the self-attention blocks while fine-tuning.
Positional Embeddings and Layer Normalization To enhance downstream tasks with minimal effort, we fine-tune the positional embeddings and layer normalization layer, which is considered
3

---

Output Linear Layer
Layer
Protrained Params
Frozen Transformer Blocks
Fead Forward
Positional
Embeddings
Add Layer Norm
Input Embedding
Mut--Haad Attentinn
Instance Norm Patching
Fine-tune
Figure 2: Model architecture. Pre-trained parameters transferred to the time series forecasting tasks. Self-attention and Feedforward layers the transformer blocks are frozen while only the embedding layer, normalization layers, and output layer require training.
standard practice[uu al. (2022); Houlsby al. (2019). As result, retrain these components during fine-tuning.
Input Embedding Given our goal of applying the NLP pre-trained model to various tasks and new modality, we must redesign and train the input embedding layer. This layer is responsible for projecting the time-series data to the required dimensions of the specific pre-trained model. To accomplish this, we use linear probing, which also reduces the number parameters required for training.
Normalization Data normalization is crucial for pre-trained models across various modalities. In addition to the layer norm utilized in pre-trained LM, we also incorporate simple data normalization block, reverse instance norm Kim et al.) (2022), further facilitate knowledge transfer. This normalization block simply normalizes the input time series using mean and variance, and then adds them back to the output.
Patching To extract local semantic information, we utilize patching Nie et al. (2022 by aggregating adjacent time steps form a single patch-based token. Patching enables a significant increase in the input historical time horizon while maintaining the same token length and reducing infor- mation redundancy for transformer models. our architecture, we apply patching after instance normalization.
4
Main Time Series Analysis Tasks
Our proposed method excels in various downstream time series analysis tasks through fine-tuning. To demonstrate the effectiveness our approach, we conduct extensive experiments on major types of downstream tasks, including time series classification, anomaly detection, imputation, short/long-tee -ert/ forecasting and few-shot/zero-she forecasting. To ensure a fair comparison, we use berkkbbe FPT and adhere to the experimental settings of TimesNet Wu et al. (2023). Due to the space limit, only the summarized results are presented below except zero-shot forecasting. Full experimental
results of the other six downstream tasks can be found in Appendix D.3 respectively.
D.2
D.7
H.6
H.9
Baselines We select representative baselines and cite their results from Wu et al. 2023), which includes the most recent and quite extensive empirical studies of time series. The baselines in- clude CNN- based models: TimesNet Wu et al. (2023); (2022) and DLinear Zeng et al.) T based models: Reformer Kitaev al. (2020).
models: LightTS Zhang et al.
Transformer-
Informer Zhou et al. (2021). Autoformer Wu et al. (2021). FEDformer Zhou et al. (2022) Non- stationary Transformer Liu al. (2022), ETSformer Woo et al. (002)). PatchTST |ie al. 2022).
Besides, N-HiTS Challu et al. (2022) and N-BEATS Oreshkin et al. (2019) are used for short-term
forecasting. Anomaly Transformer Xu al. (202T) used for anomaly detection. XGBoost Chen
Guestrin 2016), Rocket Dempster et al. (2020) LSTNet Lai et al. (0118) LSSL Gu et al. (202

---

Pyraformer Liu et al. (2021), TCN Franceschi et al. (2019) and Flowformer Huang et al. (2022 are
used for classihcation.
4.1 Main Results
Overall, as shown in [ GPT2-backbone FPT outperforms other models in most tasks, including long/short-term forecasting, classification, anomaly detection, imputation, and Now-shot/zero-shot forecasting. This confirms that time series tasks can also take advantage of cross-modality transferred knowledge. In the following,
GPT2(K) represent GPT2-backbone with first K Layers.
4.2 Imputation
Setups We conduct experiments on six popular real-world datasets, including 4 ETT datasets Zhou et al. (2021) (ETTh1, ETTh2, ETTm1, ETTm2), Electricity and Weathef] where the data-missing is common. Following the settings of TimesNet, different random mask ratios (1 12.5%, 25%, 37.5%, 50%) 1) of time points are selected for the evaluation various proportions of missing data.
Results The results are shown Table that GPT2(3) FPT achieves the best performance on most datasets. Particularly, compared to the previous SOTA TimesNet, GPT2(3) FPT yields a relative 11.5% MSE reduction on ETTh ,,nd a 4.1 % MSE reduction on average on six benchmark datasets. It verifies that the proposed method can also effectively mine temporal patterns of incomplete time series.
Table I: Imputation task. We randomly mask (12.5%, 25% 37.5%. 50%) time points of 91-9999-ee length time series. The results averaged from 4 different mask ratios. Black: best, Red: second best. Appendix H.8 shows the full results.
Methods
GPT2(3)
TimesNet
PatcTTT MSE
ETSformer MSE
LighTS MSE
DLincar MSE
FEDiormer
Stationary
Autoformer MSE
Informee MSE
Reformer
MSE
MAE
MSE MAE
MAB
MAE
MAE
MAE
MSE MAE
MSE
MAE
MAE
MAE
MSE
MAE
[005]
0.071
0.166
ETTm2
0,021
0
11022
0.088
0.029
0.102
0.208
0.327
0.046
0.151
0.06
0.208
0.101
0.215
0.026 0.099
0.099
0.105
0.156
0.292
0.157
0.280
ETTh1
0.069
0,173
0.078
0.187
0.115
0.234
0.202
0.329
0.284
0.373
0.201
0.306
0.117
0.246
0.094
0.201
0.103
0.214
0.161
0.279
0.122
0.245
ETTb2
0,048
0.141
0,049
0.146
0.068
0.163
0.367
0.436
0.119
0.250
0.142
0.259
0.163
0.279
0.053
0.152
0055
0.156
0.337
0.452
0.234
0.352
8.090)
0.207
B.092
0.210
0.131
0.262
0.132
0.260
0.130
0.259
0.100
0.218
0.222
0.200
0.313
Weather
0.031
0.056
0.030
0.054
0.034
0.055
0.076
0.171
0.055
0.117
0.052
0.110
0..099
0.203
0.032
0.059
0.031
0.057
0.045
0.104
0.038
0.087
Averago
0,047
0,127
0,049
0,132
0.060
0,111
0.197
0.309
0.123
0.228
0.119
0.221
0.112
0.229
0,056
0,142
0.14200.06
0,151
0.165
0,273
0.134
0.240
4.3 Time Series Classification
Setups To evaluate the model's capacity for high-level representation learning, we employ sequence-level clas- sification. Specifically, we follow the same setting TimesNet: For classification, 10 multivariate UEA clas- sification datasets |Bagnall et al. (2018) are selected for evaluation, including gesture, action, audio recognition medical diagnosis and other practical tasks.
1.0
1500
Results As shown in Figure GPT2(6) FPT achieves an average accuracy of 74.00%. %. surpassing all baselines including TimesNet (73.60%). Specifically, compared recent published patch-transfore www.--ornnnr----re based models Nie et al.) (2022). GPT2(6) FPT surpasses it by a large margin 9.0% which shows the prior NI.P transfer knowledge can indeed help in time series representation.
Figure 3: Model comparison in classi- fication. The results are averaged from 10 subsets of UEA. Appendix shol the full results.
4.4 Time Series Anomaly Detection
Setups Detecting anomalies in time series is vital in industrial applications, ranging from health mon- itoring to space & earth exploration. We compare models on five commonly used datasets, including
201 12014
5

---

SMD et al. (2019), Hundman et al. (2018). SMAPHundman et al. (2018), Mathur & Tippenhauer (2016) and PSMAbdulall et al. 2021). To perform fair comparison, only the classical reconstruction error is used for all baseline models to the make the setting the same as TimesNet.
Results Table 2 demonstrates that GPT2(6) FPT also achieves the best performance with the averaged Fl-score 86.72% surpassing previous SOTA method TimesNet by 1.7%. Thus, in addition to its proficiency in representing complete sequences for classification purposes, GPT2(6) FPT is capable of detecting infrequent anomalies within time series.
Table 2: Anomaly detection task. We calculate the Fi-score (as %) for each dataset. *, in the Transformers indicates the name of *former. Black: hest, Red: second best. Appendix H.7 shows the full results.
Methods Ours
GPT2(6)
TimesNal PatchTS.
ETS.
FED.
LighTT
DLincar Statiosary
Auto.
Pyra
Anomaly."
LogTran.
Trans.
SMD
86.89
84.61
84.62
83.13
85.08
82.53
77.10
84.72
85.11
83.04
85.49
81.65
75.32
76.21
79.56
78.57
77.50
78.68
SMAP
72.88
69.39
68.82
69.50
70.76
69.21
69.26
71.09
71.12
71.09
71.18
69.92
70.40
69.97
69.70
SWaT
94.23
93.02
85.72
84.91
93.19
93.33
87.52
79.88
92.74
91.78
83.10
81.43
82.80
80.52
80.37
97.13
91.76
97.23
97.15
93.55
76.07
Average 86.72
85.24
82,79
82,87 84.97
84,23
82.46
82.08
84.26
82.57
80.50
78.83
77,31
76.60
76.88
reprodace results TimesNet by https///ghhubb.ccc/ com//umm//////eeemmmmm Time Series Library We replace the joint criterion in Anomaly Transformer with reconstruction emoor for fäir comparison.
4.5 Long-term Forecasting
Setups Eight popular real-worlds benchmark datasets Wu et al. (2023). including Weather, Traffic Electricity, ILIi and 4 ETT datasets (ETTh1, ETTh2, ETTml, ETTm2), are used for long-term forecasting evaluation. Additional information regarding the discussion on the input length setting can be found in the appendix [H.10
Results As shown in Table 3 GPT2(6) FPT achieves comparable performance with PatchTST and outperforms other baselines. Specifically, compared with recent published SOTA method TimesNet, G6T2(6) FPT yields relative 9.3% average MSE reduction.
Table I Long-term forecasting task. All the results are averaged from 4 different prediction lengths, that is (24, 36, 48, 60) for ILI and [96, 192, 336, 720} for the others. Black: best, Red: second best. Appendix D333
shows
the full results.
Methods
MSE
GPT2(6)
MAE
TimesNat MSE
MAE
MSE
ETSformer
MAE
LightTS MSE
MAE
Dlincer MSE
MAE
FEDiomee MSE
MAE
PacchTST MSE
MAE
Siaiionry MSE
MAE
Autoformer MSE
MAE
Inforner MSE
MAE
MSE
Reformer
MAE
Weather
0.237
0.270
0.259
0..88
0.271
0.334
0.261
0.312
0.249
0.300
0.309
0.360
0.225
0.264
0.288
0.314
103338
0.382
0.634
0.548
0.803
0.66
ETTb1
0.427
0.426
0.423
0.437
0.440
0.460
0.413
0.570
0.795
0.915
ETT22
0.346
0.384
0.414
0.427
0.499
0.452
0.602
0.543
0.431
0.447
0447
0.449
0.330
0.379
0.526
0.516
0.400
0.459
4.431
1.729
6.736
2.191
ETTml
0.352
0
0.400
0.406
0.429
0.425
0.435
0.437
0.357
0.378
0.448
0:452
0.351
0.387
0
0.256
0.588
0.517
0.961
0.734
0.799
0.671
ETTm2
0.266
0.326
0.291
0.333
0.293
0.342
0.409
0.436
0.267
0.334
0.305
0.349
0.255
0,315
0.306
0.347
0.327
0.371
1.410
0.810
1.479
0.915
1.925
0
2,139
0.931
2.97
7.3K7
2.03
2.160
1.841
1.144
1.443
0.7%8
2.07
0.914
1,161
5.137
1.544
4.724
1,425
ECI.
0.167
0,263
0.192
0.295
0.208
0.323
0.229
0.329
0.166
0.263
0.214
0.327
0.161
0.253
0.193
0.296
0.227
0.338
0.311
0.397
0.338
0.422
Traffic
0.414
0.294
0.620
0.336
0.621
0.396
0.622
0.392
0.434
0.295
0.610
0.376
0.390
0,264
0.624
0.340
0.628
0.379
0.764
0.416
0.741
0.422
Avenge
0.516
0.407
0.596
0.662
0.473
1,303
0.562
0.436
0.701
0.489
0.446
0.386
0.633
0.757
0.511
1.836
0.871
2.081
0.954
4.6 Short-term Forecasting
Setups To fully evaluate different algorithms in forecasting tasks, we also conduct short- forecasting (with relatively short forecasting horizon) experiments on M4 Makridakis al. (2018). contains marketing data of various frequencies.
Results The results in Table show 4sow that the performance of GPT2-backbone (6) FPT is superior to advanced Transformer-based and MLP-based models, and comparable to TimesNet and N-BEATS.
4.7
Few-shot Forecasting
The large language model (LLM) has demonstrated remarkable performance in both few-shot and zero-shot learning settings Brown et al. 20200 OpenAl (003)) It can be argued that shw-shot and
http://pems..ot.ca.goo https://g...dddoo///gg////ttttve

---

Table 4: Short-term forecasting task on M4. The prediction lengths are in [6, 48] and results are weighted averaged from several datasets under different sample intervals. Black: best, Red: second best. Appendix H.9 shows the full results.
Methods
GPT26)
TimesNet
PatcTTT
N-IETS
N-BEATS
ETSformer
LighITS
DLingar
FEDiormer
Stationzyy
Autoformer
Informer
Reformer
SMAPE
11,991
11,829
12,059
11,927
11,851
14.718
13.525
13,639
12,840
12.780
12,909
14.096
18,200
MASE
1.613
1.599
2,408
2.111
2,095
1:201
1.956
2.718
4.223
OWA
D.861
0.851
0.869
0.861
0.855
1.172
1.511
1,051
0.18
0.930
0.939
1.230
1.775
zero-shot learning also represent the ultimate tasks for universal time series forecasting model. To extensively evaluate the representation power of the GPT2(6) for time series analysis, we conduct experiments under few-shot and zero-shot learning settings.
Similar to traditional experimental settings, each time series is split into three parts: training data, validation data, and test data. For few-shot learning, only a certain percentage (10%, 5%) timesteps of training data are used.
The results of 10% few-shot learning are shown in Table 5 Compared to TimesNet, DLinear, PatchTST and other methods, GPT2(6) FPT achieves the best performance. Traditionally, CNN- based and single MLP-based models are considered more data-efficient for training and suitable for few-shot learning methods. In comparison convolution-based TimesNet and MLP-based DLinear models, GPT2(6) FPT demonstrates a relative average MSE reduction of 33.3% and 13.5 % respectively. We add comparison with traditional algorithms (ETS, ARIMA, NaiveDrift) in the Appendix D.5 as well, and GTP2(6)FPT also surpass all those traditional methods.
Table 5: Few-shot learning task 10% data. All the results averaged from 4 different prediction lengths (196, 192, 336, 720)). Black: best, Red: second best. Appendix D.2 shows the detailed results of 10% and 5%
data.
Methods
MSE
GPT2
MSE
TimesNet
MAE
MSB
Di incer
MAE
MSE
FEDiormer
MSE
PStchTST
MSE
Asteriemez
MSE
Staionary
MAE
MSE
ETS/ommer
MAB
MSE
LighTT
MAB
MSC
Informer
MAB
MSE
Reformer
MAB
Weather
0.238
(1:279
0.301
003118
(0.545
LTTh]
0.590
0.369
0.638
0.633
0,701
0.914
1.179
1,375
KTTh2
0.397
0.479
0.608
01466
0.475
0.415
0.488
0.461
0399
2.655
3.871
3,485
1,455
ETTml
0464
0.441
0.66
0411
0.429
0.721 0.685
0.501
0.466
0.802
0.628
0.797 0.577
0.979 0.14
0.970
0.704
1.192
0.820
1.425
0.856
ETTm2
0.293 0.335
0.319 0.383
0.316
0.368
0.463
0.888
0.296
0.343
1,341 0.90
0.332 0.366
0.447 0.987
0.967
0.755
3,369
1,439
3,977
1.596
BCT.
0.176
0.269
0.323 0.392
01880
0.280
0.346
0.428
0.180
0.269
0.481
0.478
0.443
0.479
0.659
0.617
0.441
0.488
1.194
0.800
0.365
0.768
Trafiic
0.440
0.369
0.961
0.535
0.496
0.371 0.663
0.425
0.430
0.305
0.749
0.466
1.453
0.815
1.913
0.936
1.247
0.684
1.534
0,811
|,550
0.821
Averaga
0.371
0.367
(0.556
0.458
0.429
0.409
0.511
0.422
0.385
0.376
0.676
0.687
0.559
[0.67
0,522
0.912
0.665
1.137
0.712
1,850
0.967
1,888
0.974
4.8 Zero-shot forecasting
This task is used to evaluate the cross datasets adaption ability of our proposed algorithm, i.e. how well model is able to perform on dataset A (without any training data from A) when it is trained from dataset B.
The results are summarized in Table 6 The GPT2(6) FPT model consistently outperforms all recent ofate-of-the-art transformer and ALP-based time series forecasting methods. Compared to recently published state-of-the -frtthr-rrt MLP-based method Dlinear, convolution- method Timesnet, and munsformer-basee method Patchtst, 6PT demonstrates a relative average metric reduction of 13.1 %, 13.6% % and 7.3% %. respectively. Also, the proposed method is comparable to N-BEATS without any meta-learning design and outperforms N-BEATS in the ELECTR dataset. We attribute this to the knowledge transfer capability from the FPT model.
5 Ablations
In this section, conduct several ablations on model selection and effectiveness of pre-training The detailed results are shown in Appendix H We introduce several variants, GPT2(0) FPT, GPT2(6) without freezing and GPT2(6) without pre-training
Model Selection We separately analyze the number of GPT2 layers and the fine-tuning parameters selection. The results in Appendix H show that GPT2 with 6-layers is sound choice compared to full or few layers and partially freezing can avoid catastrophic forgetting, enabling fine-tuning without overfitting.
7

---

Table 6: Zero-shot learning results. Dptaset-specifie metrics aggregated over each dataset. A lower value indicates better performance. The source dataset of M3. Tourism, Electricity are M4. For M4, the source data for N- N-BEATS is FRED, and M3 for other models. Black: best, Red: second best, Violet: third best. Appendix D.7 shows full results.
Methods Metric
M4 sMAPE
M3 SMAPE
TOURISM MAPE
ELECTR ND X 100
Average
N-BEATS
11.70
17.8
15.19
DLinear
28.51
17.6
18.86
TimesNet
13.55
14.17
28.
19.3
18.96
PatchTST
13.22
13.06
27.10
17.3
17.67
ETSformer
27.74
16.03
180.40
44.2
67.09
LighTTS
13.62
17.90
66.99
19.6
29.52
Stationary
13.32
15.29
43.75
22.0
23.59
FEDfomer
15.04
13.53
31.55
18.4
19.63
Autoformer
40.39
33,9
27.54
Informer
19.04
15.82
35.82
21.2
22.97
Reformer
14.09
13.37
25.48
21.6
18.63
GPT2(6)
16.38
Effectiveness of Pre-training The results are shown Table 7 GPT2(6) FPT outperforms both GPT2(0) FPT and S0P12-random-inittalizee suggesting that GPT2 with pre-training parameters can achieve improvement on times series tasks. Besides, GPT2(6) FPT performs better than GPT2- unfrozen, demonstrating that partially freezing also helps. Also, results in Appendix H.2 show that random initialized GPT2(6) with freezing performs poorly and the pre-trained knowledge is instrumental for time series tasks.
Table Ablation study 10% data. All the results averaged from 4 different prediction lengths. No Freeze represents (6T2( without freezing, No Pretrain represents GPT2(6) without pre-training. Black: hest.
GPT2(6)
GPT2(0)
Freeze MSE
No Pretrain
Methods
MSE
MAE
MSE
MAE
MAE
MSE
MAE
Weather
0.237
0.270
0.263
0.297
0.273
0.302
0.277
0.305
ETThl
0.427
0.426
0.874
0.647
0.753
0.596
1.326
0.743
ETTh2
0.346
0.394
0.666
0.559
0.447
0.451
0.502
0.479
6 Exploring Transfer Learning from others: The Unexceptional Nature of GPT2-based-FPT
We also present experiments pretrained BEiT-backbone FPT model Bao et al. (2022) to illustrate the generality of pre-trained models for cross-domain knowledge transferring. The results in Table demonstrate that the ability of knowledge transfer is not exclusive to GPT2-based pre-trained language models. Subsequently, theoretical analysis will shed light on the universality of this phenomenon.
FPT Devlin et al. (2019) model and the image-
Table 8: Results of frozen pretrained transformer variants 5% ETTh2 and ETTm2. All the results are averaged
from different prediction lengths. Black: best. Appendix H.5 shows the full results.
GPT2(6)
BERT(6)
BEIT(6)
DLinear MSE
PatchTST MSE
FEDfomer MSE
Autoformer
Methods
MSE
MAE
MSE
MAE
MSE
MAE
MAE
MAE
MAE
MSE
MAE
ETTh2
0.400
0.433
0.452
0.451
0.459
0.454
0.827
0.615
0.439
0.448
0.441
0.457
0.470
0.489
ETTm2
0.346
0.318
0.357
0.315
0.357
0.426
0.314
0.381
0.404
7 Training/Inferencing Cost
Analysis of computational cost is helpful for investigating the practicality of the L.LM- model. The results can be found in table 9 Each bascline model comes in two variants, featuring model

---

Table 9: Training parameters and Training/Inference Cost Comparison
Model
Training Params Training Params Percentages
Training Time for 1 step(s)
Inference Time for Batch(s)
FEDformer-32
44k
100
0.889
0.170
TimesNett322
2M
100
0.747
0.302
PatchTST-32
543K
100
0.043
0.022
FEDiormer-766 768
33M
100
0.208
0.056
TimexNet-768
42M
100
5.723
2.162
PatchTST-768
20M
100
0.457
0.123
GPT-2(3)-768
4M
6.12
0.093
0.032
GPT-2(6)-768
4M
4.6
0.104
0.054
hidden dimensions of 32 and 768, which align with GPT-2's specifications. Furthermore, the majority of the baseline models consist of three layers. We assessed the computational cost using batch from ETTh2 (with a batch size of 128) on a 32G V100 GPU.
The results indicate that GPT-2(3) has substantially enhanced time efficiency and reduced parameter quantity compared to baselines with the same model dimension. This was a surprise since we initially anticipated that this large language model might be slower. However, surmise that the efficient optimization of huggingface's GPT model implementation primarily accounts for such significant improvement in time costs. Furthermore, GPT-2(3) and GPT-2(6) demonstrate mere 6.12% and 4.60% proportion of learnable parameters among the overall parameter size, respectively.
8 Towards Understanding the Universality of Transformer: Connecting Self-Attention with PCA
The observation, i.e. to modify its model, makes us believe that the underlying model is doing something very generic and independent from texts despite it being trained from text data. Our analysis aims to show that part of this generic function can be related to PCA, as minimizing the gradient with respect to the self-attention layer seems do something similar PCA. In this section, we take the first step towards revealing the generality of self-attention by connecting the self-attention with principal component analysis (PCA). Moreover, when coming the question of why fine-tuning is restricted to the embedding layer and layer norm, following our hypothesis that the pre- pre-trained LM as a whole performs something generic, partially fine-tuning any of its components may break the generic function and lead to relatively poor performance for time series analysis.
directly use trained LM for time series forecasting without having
For each layer, calculate and perform statistical analysis of the pairwise token similarity values. Specifically, we denote each output feature map with shape of (b, n, d), where b is the batch size, the number of tokens, and d the dimension of each token feature. We calculate the cosine similarity, and the resulting pairwise similarity matrix of shape (b, n2). Next we count the number of occurrences of similarity values within each interval as a simple statistical analysis.
Our analysis is motivated by the observation that the within-layer token similarity increases with deeper layers in transformer. We report the layer-wise average token cosine similarity ETTh2 dataset in Figure 4 (a. c), where we mix weights from pre-trained LM with weights randomly sampled from Gaussian distribution. Here we summarize our observations: a) in randomly initialed GPT2 (6) model, the token similarity is low among all layers (0.1 - 0.2); b) when gradually switched to the pretrained GPT2 model, the token similarity significantly increases in the deep layers and eventually reaches more than 0.9 in the last layer. One potential explanation for the increasing token similarity that all the token vectors are projected into the low-dimensional top eigenvector space of input patterns. To verify this idea, we further conduct experiments where we replace the self self-attention module with PCA and find token similarity patterns remain unchanged according to Figure 44 (b). which further justifies the potential connection between PCA and self-attention.
To build the theoretical connection between PCA and self-attention, ent structure of self-attention. Let X (#1,-.., ..n) E ND be the input pattern, and let f(X) = (Xi, ...., T NND + ND be the function for self-attention, i.c., S(X) = 2oftmax(XXAX where WOW E DDD
first analyze the gradi-

---

00000
Biplac Rade PM
63 C4 31
137%
Rankes Karia
Rondin Ratto
(4) Pas fommance: Diflereat Replace Ratios
01100
056
080
ma
Janden Baal 4P%
Randen Jane 10%
Baskees Ruaio
(b) Token Similarty: PCA as Amention
(3) Tokm Similaity: Different Replace Ratios
Figure 4: (a, <) The performance and token similarity within samples with respect to each layer with diffcrent random mixed ratio. Pre-trained parameters are mixed with random initial parameters according to certain proportions. (b) Token similarity within samples when replacing the attention with PCA.
0f((x) N day
Lemma 8.1. Let the Jacobian =
represent the gradient f(X) w.r.t the
-ij=1
input pattern, then we have |J|2 S
A/
Σ 1=
(P.. + 22) 2 En= P.. + A
where
A Al EM Pt. Bj E 2
+ 1 2
Σn=
|x,1? and P.. Σm exp(a, Any)
exp(x Ang))
This lemma reveals an important gradient structure of J. The proof of essentially follows the analysis in Kim et al. (202I), and we include it in Appendix G for completeness.
Using the gradient structure revealed Lemma 8.1 we can connect self- self-attention with PCA. In order to minimize the norm of gradient [J/2, we essentially need to make Σ 12, Ey= Py
small. When A is small and all the input patterns are centered at (i.e. ΣN De = 0), we have EN |z, - X P... ~ Em Jx; X x
The theorem below shows that A minimizing the objective Σ |xg X T XAx,I² contains the largest eigenvectors of X X where is the rank of A. Theorem 1. Let We and WK be matrices of size D x m. eigenvalues of X X ranked in descending order, and let 02 TR. 1,..., D be the corresponding
Let \1 2 > An be the
eigenvectors. The optimal solution A that minimizes Σ I - Ax,, 2 given by
E X, U,
The proof of Theorem can be found in Appendix G Following Theorem I through the training of pushing gradient to zero, self-attention learns to perform function closely related PCA.
9
Conclusions
In this paper, we developed a foundation model for time series analysis, based on pre-trained model from NLPP CV, that (a) facilitate the model training for downstream tasks, and (b) provide unified framework for diverse time series analysis tasks. Our empirical studies show that the proposed method performs on par better than the state-of-the-arr approaches on almost all time series tasks. We also examine the universality of transformer by connecting self-attention with PCA, an important step towards understanding how generative models work in practice. On the other hand, we do recognize some limitations of our work: the zero-shot performance of our approach is still behind N-beat on several datasets, and our analysis of the generality of transformer is still the early stage. Moving forward, we plan to improve the performance of our approach by exploiting the parameter efficient fine-tuning approaches which usually introduce additional structures into the pre-trained model for better adaption. To better understand the universality of transformer, we also plan to examine it from the viewpoint of n-gram language model, an approach that is taken by Flhage et al. (2021); Olsson al. 2022). In Appendix F
we include our initial analysis along this direction.
10

---

Acknowledgement
We would like express sincere gratitude to Ziqing Ma, Qingsong Wen, Mengni Ye, and Tao Yao for their valuable suggestions and proofreading assistance throughout the development of this paper. Their insightful feedback and attention detail greatly improved the quality and clarity of our work.
References
Abdulaal, A., Liu, Z.,, and Lancewicki, Practical approach asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery data mining, pp. 2485-2494, 2021.
Bagnall, A., Dau, H. A., Lines, J.. Flynn, M., Large. J., Bostrom, A., Southam. P., and Keogh, E. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.0000 2018.
Bao, H., Wang, W. Dong, L., Liu, Q. Mohammed, O. K., Aggarwal, K., Som, S., and Wei, F. arXiv preprint
Vlmo: Unified vision-language pre-training with mixture-of-moddaty-e-re arXiv:2111.0233 2021.
Bao, H., Dong, L., Piao, S., and Wei, BEit: BERT pre-training of image transformers. In International Conference Learning Representations, 2022.
Böse, J.-H., Flunkert, V., Gasthaus, J., Januschowski,, T., Lange, D., Salinas, D., Schelter, S., Seeger, M., and Wang, Y. Probabilistic demand forecasting at scale. Proceedings of the VLDB Endowment, 16944-706 2017.
Box, G. E. and Jenkins, G. M. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):91-19,, 1968.
Box, G. E. and Pierce, D. A. Distribution of residual autocorrelations in autoregressive-interraed moving average time series models. Journal of the American statistical Association, 65(332): 1509-1526, 1970.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T. J., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E,, Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Challu, C.. Olivares, K. G., Oreshkin, B. N., Garza, F., Mergenthaler, M., and Dubrawski, A. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201. 12886, 2022.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. KDD '1., 2016.
Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 141..3:412.35555 2014.
Courty, P. and Li, H. Timing of seasonal sales. The Journal of Business, 72(4):545-572, 1999.
Dempster, A., Petitjean, F., and Webb, G. I. ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5): 1454 1495, 2020.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA, June 2-7, 2019, pp. 4171-4186, 2019.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 16 words: Transformers for image recognition at scale. 9th International Conference on Learning Representations (ICLR), Austria, May 3-7, 2021, 2021.

---

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A.. Conerly, T., DasSarma, N., Drain, D., Ganguli, D.,, Saatfield-Dodde Z., Hernandez, D., Jones, A.. Kernion, J., Lovitt, L.., Ndousse, K., Amodei, D., Brown, TT, Clark, J., Kaplan, J., McCandlish, S., and Olah, C. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transfomme-cirriiii http://iissss.po//oo////oaoo/nee
Franceschi, J.-Y.. Dieuleveut, A., and Jaggi. M. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
Friedman, M. The interpolation of time series by related series. J. Amer. Statist. Assoc, 1962.
Gao, J., Song, X., Wen, Q. Wang, P., Sun, L., and Xu, H. RobusTTAD: Robust time series anomaly detection via decomposition and convolutional neural networks. KDD Workshop on Mining and Learning from Time Series KDDDMMileTS 20), 2020.
Giannou, A., Rajput, S., Sohn, J.-y., Lee, K., Lee, J. D., and Papailiopoulos, D. Looped Transformers as Programmable Computers. arXiv e-prints, art. arXiv:2301.13196 January 2023. doi: 10.48550/ arXiv.2301.13196
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., and Montero-Manso, P. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.003880 2021.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 1997.
Houlsby, N., Giurgiu, A., Jastrrebbski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International Conference Machine Learning, pp. 2790-2799. PMLR, 2019.
Huang, Z. Shi, X., Zhang, C., Wang, Q., Cheung, K. C., Qin, H., Dai, J., and Li, H. Flowformer: A transformer architecture for optical flow. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pp. 668-685. Springer, 2022.
Hundman, K., Constantinou, V., Laporte, C., Colwell, I., and Soderstrom, T. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp. 387-395, 2018.
Hyndman, R. and Athanasopoulos, G. Forecasting: Principles and Practice. OTexts, Australia, 3rd
edition, 2021.
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, I.., and Muller, P.-A. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 33(4):917-963, 2019,
Kim, H., Papamakarios, G., and Mnih, A. The lipschitz constant of self-attention. In International Conference on Machine Learning, pp. 5562-556556655 5-711 PMLR, 2021.
Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, J. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference Learning Representations, 2022.
Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
Lacoste-Julien, S., Schmidt, M., and Bach, F. A simpler approach obtaining an o (1/t) convergence rate for the projected stochastic subgradient method. arXiv preprint arXiv:1212.2002 2012.
Lai, G., Chang, W.-C., Yang, Y., and Liu, H. Modeling long-and short-term temporal patterns with deep neural networks. The s1 international ACM SIGIR conference on research development in information retrieval, pp. 95-104, 2018.
12

---

Lim, B., Ank, S. Ó., Loeff, N., and Pfister, T. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting. 2021.
Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., and Dustdar, S. Pyraformer: Low-complexity pyra- midal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021.
Liu, Y., Wu, H., Wang. J., and Long, M. Non-stationary transformers: Exploring the stationarity in time series forecasting. In Advances in Neural Information Processing Systems, 2022.
Lu, K., Grover, A., Abbeel, P., and Mordatch, I. Frozen pretrained transformers as universal computation engines. Proceedings of the AAAI Conference Artificial Intelligence, 36(7); 7628-7636, Jun. 2022.
Makridakis, S., Spiliotis, E.,, and Assimakopoulos, V. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802 2018.
Mathur, A. and Tippenhauer, N. O. Swat: water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physices systems for smart water networks (CySWater), pp. 31-36. IEEF, 2016.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnamam, J. A time series worth 64 words: Long-term forecasting with transformers. ArXiv, abs/2211.14777 2022.
Olsson, C.,, Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kermion, J., Lovitt, L.. Ndousse. K., Amodei, D... Brown, T., Clark. J., Kaplan, J.. McCandlish, S., and Olah, C. In-context learning and induction heads. Transformer Circuits Thread, 2022.
learminn
aunducionnheadd/nddee html.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv: 10437, 2019.
Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Meta-learning framework with applica- tions to zero-shot time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, number 10, pp. 9242 -2500, 2021.
Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. 2019.
Rao, Y., Zhao, W., Zhu, Z., Lu, J., and Zhou, J. Global filter networks for image classification. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., and Pei, D. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2828-2837, 2019.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Vardi, G., Ychudai, G., and Shamir, O. On the optimal memorization power of relu neural networks. arXiv preprint 10.0v:210000318/) 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser Lukasz, and Polosukhin, I. Attention is all you need. arXiv preprint arXiv: 03762, 2017.
Wang, P., Wang. X., Wang, F., Lin, M., Chang, S., Li, H., and Jin, R. Kvt: k-nn attention for boosting vision transformers. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 285-302. Springer, 2022.
13

---

Wang. T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. ArXiv, abs/2005.10242, 2020.
Wen, Q. Yang, L., Zhou, T., and Sun, L. Robust time series analysis and applications: An industrial perspective. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4836 4837- 2022.
Wen, Q. Zhou, T., Zhang, C., Chen, W., Ma, Z.,, Yan, J., and Sun, L. Transformers in time series: A survey. In International Joint Conference Artificial Intelligence(I/CAA),, 2023.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C.. Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URLhttps: www. aclweb. org/ http//anhhooo/22299
emn emnlp-demos.6
Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, S. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022.
Wu, H., Xu, J., Wang. J., and Long, M. Autoformer: Decomposition transformers with auto- correlation for long-term series forecasting. In Advances in Neural Information Processing Systems (NeurlPS), pp. 101-112, 2021.
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023.
Xu, J., Wu, H., Wang, J., and Long, M. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:21 2021.
Yang, C.-H. H., Tsai, Y.-Y,, and Chen, P.-Y. Voice2series: Reprogramming acoustic models for time
series classification. In International Conference on Machine Learning, pp. 11808- 11819, 2021.
Yun, C., Chang, Y.-.., Bhojanapalli, S., Rawat, A. S., Reddi, S., and Kumar, S. O (n) connections are expressive enough: Universal approximability of sparse transformers. Advances in Neural Information Processing Systems, 13:13783-13794, 2020.
Zeng. A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? 2023.
Zhang, T., Zhang, Y., Cao, W., Bian, J., Yi, X., Zheng, S., and Li, J. Less more: Fast mul- tivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186, 2022.
Zhou, H., Zhang, S., Peng. J., Zhang, S., Li, J.. Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
Zhou, T., Ma, Z., Wen, Q. Wang, X., Sun, L., and Jin, R. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference Machine Learning (ICMI. 2022), 2022.
14

---

A Visualization
In order to clarify the representation ability more clearly, Figure 5 provides providss showeases of imputation. long-term forecasting and few-shot forecasting. Especially for few-shot learning, GPT2(6) can accurately forecast, while TimesNet and DLinear fail in this task.
(a) Imputation on ETTh1 12.5%
11
Anl MAY
Arn Net
80
cc
-10
GPT2(3)
TimesNet
DLinear
(b) Long-term forecasting ETTml
GPT2(6)
TimesNet
DLinear
(c) Few-shot forecasting on ETTh2
-25
:28
-24
Im
-20
-20
GPT2(6)
TimesNet
DL Diinear
Figure Visualization of imputation, long-term forecasting and few-shot forecasting.
B
Related Works
We have presented novel general time series analysis model in this paper, and to the best of our knowledge, there has been limited work on similar comprehensive methods for time series analysis. The most closely related field is time series forecasting, where transformer models have gained widespread popularity. Therefore, our focus this related work will primarily be on introducing the end-to-end time series forecasting method.
Time series forecasting models can be roughly divided into three categories, ranging from the classic ARIMA models to the most recent transformer models. The first generation of well-discussed models can be dated back to auto-regressive family, such as ARIMA Box & Jenkins (1968); [ Box & Pierce (1970) that follows the Markov process and recursively execute sequential forecasting. However, is limited to stationary sequences while most time series is non-stationary. Additionally, with the bloom of deep neural networks, recurrent neural networks (RNNs), such LSTM Hoccreiter SSchiihhuber (1997) and GRU Chung et al. (2014). were designed for sequential tasks. Yet the recurrent model is inefficient for training and long-term dependencies are still under resolved.
Recently, transformer models have achieve great progress in NLP Vaswani et (201 Devlin et al. (2019): : Radford et al. (2019) and CV Dosovitskiy et al. (2021): Bao et al. (2022) tasks. Also, large amount of transformer models are proposed to apply to time series forecasting Wen al. (2023). In the following, briefly introduce several representative algorithms. Informer Zhou et al. (202
15

---

proposes probability sparse attention mechanism to deal with long-term dependencies. Autoformer Wu et al. (02T) introduces a decomposition transformer architecture and replaces the attention module with an Auto-Coreelatio mechanism. FEDformer ZZhou (Zhou et al. (2022) uses Fourier enhanced structure to improve computational efficiency and achieves linear complexity. Similar to patching in ViT et al. (02T). PatchTST Nie et al. (2022) employs segmentation of time series that divide a sequence into patches to increase input length and reduce information redundancy. Besides, simple MLP-basee model DLinear Zeng al. (2023) outperforms most transformer models and it validates channel-inde -innell-nddeeenence works well in time series forecasting. Recently, TimesNet Wu et al. (2023) has treated time series 2D signal and utilized convolution- basidd inception net backbone to function as comprehensive time series analysis model. This work is closely related to our tasks in this paper.
C Dataset Details
In this section, we separately summarize dataset details long/short-term forecasting and few-shot/zero- shot forecasting.
Datasets of Long-term Forecasting and Few-shot Learning The details of datasets are shown as follows: 1) ETT datasets Zhou et al. (202T) contain electricity load of various resolutions (ETTh & ETTm) from two electricity stations. 2) Weather contains 21 meteorological indicators of Germany within 1 year; 3) Illness contains the influenza-like illness patients in the United States; 4) Electricity dataset contains the electricity consumption; 5) Traffic dataset contains the occupation rate freeway
system across the State of California. Table I0 summarizes details of feature statistics.
Similar to PatchTST Nie al. 2022), Exchange not contained. Zeng et al. (2023) shows that simply repeating the last value in the look-back window can outperform or be comparable to the best results. Also, ILI not used for few-shot learning for the limited quantity that hard to follow the definition of few-shot.
Table 10: Dataset details of few-shot learning.
Dataset
Length
Dimension
Frequency
ETTh
17420
hour
ETTm
69680
15 min
Weather
52696
22
10 min
ILI
966
7
7 days
Electricity
26304
321
1 hour
Traffic 17544
862
hour
Datasets of Short-term Forecasting and Zero-shot Learning The details of short-term forecasting and zero-shot learning datasets shown follows: 1) M4 is large and diverse dataset that contains time series of various frequencies and fields, including business, financial and economic forecasting:
2) M3 smaller than M4, but also contains time series from diverse domains and frequencies; 3) TOURISM is the dataset of tourism activities with different frequencies and contains a much higher fraction of erratic series compared with M4; 4) ELECTR represents the electricity usage monitoring of 370 customers over three years. Table o summarizes details of the datasets and zero-shot mapping between source and target.
D Experimental Details
All the deep learning networks are implemented in PyTorch and trained on NVIDIA V100 32GB GPUs. We use the pre-trained models from Wolf et al.) (2020) for experiments. For few-shot learning. an early stopping counter is employed to stop the training process after three epochs if no loss degradation on the valid set is observed. Plus, we convert the multivariate data into univariate data. Specifically, we treat each feature of the sequence as a single time series. This is mainly for memory efficiency after patching of GPT2(6) and previous works, DI ineer and PST have proved the effectiveness of channel-independennc.
16

---

Table 11: Datasets and mapping details of zero-shot learning.
Dataset Length
Mapping
Horizon
M4
M3
M3 Yearly
645
Yearly
M3 Quarterly
756
8
Quarterly
M3 Monthly
1428
18
Monthly
M3 Others
174
8
Monthly
M4 Yearly
23000
6
Yearly
M4 Quarterly
24000
8
Quarterly
M4 Monthly
48000
Monthly
M4 Weekly
359
13
Monthly
M4 Daily
4227
14
Monthly
M4 Hourly
414
48
Monthly
TOURISM Yearly
518
4
Yearly
Yearly
TOURISM Quarterly
427
8
Quarterly
Quarterly
TOURISM Monthly
366
24
Monthly
Monthly
ELECTR
1311
168
Hourly
Monthly
D.I
Accuracy Metrics
For long-term/ssort-term forecasting and few-shot forecasting, and mean absolute error (MAE) as metrics, For zero-shot learning, mean absolute percentage error (MAPE) used for TOURISM; symmetric MAPE (sMAPE) is used for M3 and M4; normalized deviation (ND) is used for ELECTR. All experiments are repeated 3 times and the mean of the metrics used in the final results.
mean square error (MSE)
D.2 Detailed Definition and Results for Few-shot and Long-term Forecasting
Task Definition Since Zeng et al. (2023) and et al. (2022) have verified that bhannel-independenee works well for time series datasets, treat each multivariate series multiple independent univariate series. Similar to traditional experimental settings, each time series is split into three parts: training data, validation data, and test data. For the few-shot forecasting task, only certain percentage (5%, 10%) timesteps of training data are used, and the other two parts remain unchanged. The evaluation metrics remain the same for classic multivariate time series forecasting. We repeat this experiment 3 times and report the average metrics in the following experiments.
Detail Experiment Tables for Few-shot Time-Series Forecasting in Table
12 and Table
17

---

Table 12: Few-shot learning results 5% data. We use prediction length O E (96, 192,336, 720]. A lower MSE indicates better performance, and the best results are highlighted in bold. means that 5% time series is not sufficient to constitute training set.
Methods
GPT2(6)
GPT2(0)
DLinear
PoschTST
TimesNet
FEDforer
Ausoformer
Stationary
ETSformer
LighrTS
Informer
Reformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE|
MSE
MAE
MSE
MAE
MSE
MAB
| MSB MAE
MSB
MAE
MSE
MAE|
MSB
MAE|
MSE
MAE
MSE
MAE
E77h2 E7Th1 Weather
192
96
0.115 0,230
0.227
0.276
0.191
0.244
0.289
0.243
0.184
0.228
0.283
0.242
0.171
0.230
0.224
0.277
0.207
0.272
0.307
0.253
0.265
0.229
0.317
0.309
0.278 0.333 0.290 0.307
0,227 0.299 0.215 0.252
0.218 0.295
0.218
0.24
0.331
0.230
0.274
0.323
0.285
0.620
0.497
0.545
0.497
00406
0.446
0.435
0.450
0.286
0.322
0.303
0.313
0.328
0.353
0.392
0.351
0.393
0.353
0.39%
0.355
0.649
0.547
0.465
0.459
720
0.366
0.379
0.391
0.393
0.364
0.388
0.384
0.387
0.400
0.385
0.391
0.394
0.387
0.389
0.452
0.407
0461
0.461
0.401
0.418
0.570
0.522
0.471
0.468
Avg.
0.263
0.301
0.282
0.314
0.263
0.308
0.269
0.303
0.298
0.318
0.309
0.353
0.310
0.353
0.327
0.328
0.333
0.371
0.305
0.345
0.584
0.527
0.447
0.453
[0.543
0.506
0.593
0.529
0.91
0.812
L.198
0.795
192
0.748
0.580
1.220
0.778
0.720
0.601
0.711
0.570
0.910
0.665
0.652
0.563
0.725
0.602
0.913
0.615
1.221
0.853
1.525
0.93
1.219
0.828
1.273
0.853
336
0.595 0.754
1.852
0.965
0.984
0,727
0.816
0.619
0.945
0.653
0.751
0.594
0.761
0.624
0.935
0.644
1.179
0,832
1.347
0.87
1.202
0.811
1.254
0,857
720
Avg.
0.560
1,299
0.611
0.691
0.617
0.658
0,562
0.722
0.598
0.616
1.189
1,451
1,225
0.817
1,241
0.835
96
00.376
0.421
0.551
0.507
0.442
0.456
0.401
0.421
0.409
0.420
0,300
0.424
[0.428
0.468
0.408
0.423
0.6778
0.619
2,022
1.006
3.837
1.508
3,753
1,518
192
0.418
0.441
0.765
0.610
0.617
0.547
0.457
0.455
0.483
0,464
0.457
D.465
01,496
0.504
0.497
0.468
0.845
0.647
3,534
1.348
3,975
1,933
3,516
1,473
0,408
0.767
1.434
0.409
0,449
0.477
D.483
01.48
0.727
1.451
3,956
3,312
1.427
720
Avg.
0.
0.433
10.694
0.577
0.827
0.615
0.439
0.448
0..66
0-444
0.441
0:447
0.47
0.489
0.470
0.457
0.800
0.681
3.206
1.268
3.922
1.653
3,527
1.472
96
00.386
0.405
00.522
0.512
0.332
0.374
0.399
0.414
0.606
0.518
0.628
0.544
0.726
0.578
0.823
0.587
L.03]
0.747
1.048
0,733
1.130
0.775
1.234
0.798
ETTm2 grFml
0.440
0.438
0.632
0,441
0.136
0.681
0.539
0.606
0.566
0.750
0.591
0,766
0.756
0.788
1.287
0.839
336
0,485
0.459
0.767
0.58-1
0,102
0.116
0,499
0.467
0.786
0.597
0.807
0.628
00851
0.659
0.870
0.603
1.138
0.787
1.147
0.775
1.198
0.809
1.288
0.812
720
0.577
0.499
1.334
0.742
0.511
0.489
0.767
0.587
0.796
0.593
0.822
0.633
0.857
0.655
0.893
0.611
1.245
0.831
1,200
0.799
1.175
0.794
1.247
0.828
Arg.
0:412
0.150
0.828
0.593
0.400
0.417
0.526
0,476
0.717
0.561
0.730
0.592
0.796
0.620
0.857
0.598
L.125
0.782
1.123
0,765
1,163
0.791
1.261
0.826
96
00:199
0.280
0.282
0.547
0.236
0.326
0.206
0.288
0.220
0.299
0.222
0:320
00.292
0.322
0.238
0.116
0.4004
0.485
1.108
0.772
3.599
1.47%
3,883
1,545
192
0.256
0.316
0.346
0.383
0. 0.306
0.373
0.264
0.324
0
0.361
0. 0.394
0.361
0.291
0.357
0.298
0.349
0.479
0.521
1.317
0.850
3.578
1.475
3.553
1.484
336
0.318
0.355
0:429
0,437
0.
13423
0.334
0.367
0.338
0.366
0.378
0.427
0.477
0.517
0.353
0.380
0.552
0.555
1.415
0,879
3.561
1.473
3.446
1,460
0.466)
0.43
0.674
0.444
0.432
0.509
01,65
0.523
0.510
0.555
0.538
0,445
0.984
3.896
1.555
3.445
1.460
Avg.
0.308
0.346
0.452
0.431
0.399
0.426
0.314
0.352
0.344
0.372
0.381
0.404
00.38%
0.433
0.341
0.372
0.534
0.547
1,415
0.871
3.658
1.489
3.58]
1.487
96
[0.143
0.241
0.117
0.246
0.150
0.251
0.145
0.244
0.315
0.389
0.235
0.322
0.2997
0.367
0.484
0.518
0.697
0.638
0.6639
0.609
1.265
0.919
1.414
0.855
ECL
0.159
0.255
0.163
0.260
0.163
0.263
0,163
0.260
0,318
0.396
0.247
0.341
0.308
0.375
0.501
0.531
0.718
0.648
0.772
0.678
1.298
0.939
1.240
0.919
0.179
0.182
0.183
0.340
0,415
0.367
0.356
0.334
0.758
0.901
1.302
0.942
1.253
0.921
720
0.233
0.325
0.239
0.329
0.219
0.311
0.233
0.323
0.635
0.613
0.318
0.394
0.426
0.466
0.952
0.786
1,028
0.788
1.200
0.871
1.259
0.919
1.249
0.921
Avg.
0.178
0.273
0.278 0,182
0.176
0.275
0.181
0.277
0.402
0.453
0.266
0.353
0.346
0.404
0.627
0.603
0.500
0.685
0.878
0.725
1,281
0.929
1.289
0.904
Tim Fe
0.419
0.298
0.404
0.854
0.492
0.670
0.121
0.795
0.855
0.821
1.586
0.841
192
0/434
0.305
0.179
0.352
0.447
0.315
0.112
0.294
0.891
0.517
0.653
0.105
0.8837
0.503
1,509
0.838
1,856
0.928
1,688
0.848
1.596
0.831
1.602
0.811
336
0.449
0.315
0.477
0.345
0.478
0,333
0.439
0.310
0.853
0.471
0.707
0:445
0.:667
0.523
1.602
0.860
3,060
0.999
1.826
0.903
1.621
0.841
1.668
0.868
720
Avg.
0.434
0.305
0.317
0.867
0.193
0.676
0,123
0.833
0.502
1.526
0.795
1,591
0.832
L618 0.851
Average
(0.377 0.375
(0:775
0.465
0.4441 0.413
0..92
0,383
0.552
0.464
0.483
0.445
|0.337
0.480
0.697
0.537
|0.909
0.675
1.341
0.789
1.878
0.994
1.819 0.966
18

---

Table 13: Few-shot learning results 10% data. We use prediction length O E (96, 192, 336, 720). A lower MSE indicates better performance, and the best results are highlighted in bold. means that 10% time series is not sufficient to constitute training set.
Methods
GPT2(6)
GPT2(0)
DLincar
PaccTTST
TimesNet
FEDGormer
Autoformer
Stationary
ETSformer
LighTTS
Informer
Reformer
Merric
MSF
MAF
MSF
MAF
MSF
MAR]
MSE
MAF
MSF
MAE]
MSR
MAF
MSR
MAF
MSE
MAF
MSE
MAF]
MSE
MAF
MSE
MAP
ETTM2 B7Th Wexther
192
96
0.163
0.210
0.254
0.215
00119
0.243
0.284
0.240
0.117
0.215
0.221
0.263
0.165
0.210
0.215
0.257
0.184
0.245
0.263
0.230
0.250
0.188
0.304
0.253
0.270
0.221 0.297
0.322
0.269
0.192 0.234
0.295
0.199
0.279
0.272
0.332
0.217
0.259
0.269
0.304
0.552
0.374
0,478
0.401
00.33
0.522
0.462
0.380
0.256
0.292
0.270
0.321
0.312
0.346
0.351
0.370
0.724
0.541
0.715
0.535
720
0.321
0.339
0.348
0.358
0.320
0.316
0.332
0.346
0.381
0.371
0.387
0.393
0.390
0.396
0.111
0:405
0.137
0.118
0.577
0.382
0.739
0.558
0.61]
0.500
Avg.
0.238
0.275
0.263
0.297
0.241
0.283
0.242
0.279
0.279
0.301
0.284
0.324
0.300
0.342
0.318
0.323
0.318
0.360
0.289
0.322
0.597
0.495
0.546
0.469
01.558
0.456
0,861
0.628
0.512
0.499
0.552
1.179
0.792
1.184
0.790
192
0.570
0.516
0.700
0.587
0.538
0.598
0.524
0.797
0.593
0.624
0:555
0.722
0.598
0.915
0.629
1.155
0.854
1.199
0.806
1.295
0,850
336
0.608
0.535
0.801
0.635
0.721
0.622
0.657
0.550
0.941
0.648
0.091
0.574
0.750
0.619
0.939
0.644
1.179
0.822
1.347
0.870
1.202
0.811
1.294
0.854
720
0.725
0 0.591
1.385
0.831
0.986
0.743
0.762
0.610
0.877
0.641
0.728
0.614
0.721
0.616
0.887
0.645
1.273
0.874
1.534
0.947
1.217
0.825
1.223
0,838
Avg.
0.590
0.639
0.561
0.7702
0.833
96
00.331
0.374
0.539
0.495
0.357
0.411
0.353
0.389
0.378
0.409
0.382
0.416
0,413
0.451
0.389
0.411
0.678
0.619
2.022
1.006
3.837
1.508
3.788
1.533
192
0.402
0.411
0.675
0.555
0.569
0.511
0.401
0.414
0.490
0.467
0.478
0:474
0.477
0.477
0.473
0.455
0.785
0.666
2.529
1.104
3,856
1.513
3,552
1,483
336
0.406
0.435
0.718
0.580
0.671
0.572
0.426
0.441
0.537
0,494
0.504
0.501
0.547
0.543
0.507
0.480
0.839
0.694
2.453
1.122
3.952
1.526
3.395
1.526
0:449
0.732
0,477
0,450
0.510
0.491
0.499
0.509
0.516
0.477
3.816
1.407
3.812
1.503
3.205
1,401
Avg.
0.397
0.421
[.666
0.559
0.605
0.538
0.415
0.431
0.479
0.465
0.406
0.475
0..488
0.499
0,462
0,455
0.894
0.713
2,655
1.100
3.872
1.513
3.485
1,496
m2 ml
96
0..990
0,429
0.404
0.425
00610
1.666
0.508
0.352
0.392
0.410
0.419
0.583
0.501
0.528
0.617
0.578
0.518
0.546
0.774
0.754
0.614
0.592
0.761
0.568
0.911
0.688
0.921
0.682
1.162
0.795
0.785
L.42
0.862
0.841
336
0.469
0.439
0.895
0.615
0.419
0.134
0.876
0.454
0.725
0.568
0.998
0.775
0,869
0.677
0.803
0.587
0.991
0.719
0.998
0.716
1.227
0.908
1.150
0.866
720
0.569
0.498
0.916
0.646
0.490
0.477
0.681
0.556
0.769
0.549
0.693
0.579
0.810
0.630
0.844
0.581
1.062
0.747
1.007
0.719
1.207
0.797
1.366
0.850
Avg.
0.464
0.441
(0,772
0.577
0.411
01.429
0.501
0.466
0.677
0.537
0.722
0.605
01.802
0,628
0,797
0.578
0,980
0.714
0.971
0.705
1,192
0.821
1.476
0,856
96
00.188
0.191
0.212
0.285
00:291
0.399
[0.33]
0.813
3.203
1.407
4.195
1.628
192
0.251
0.309
0.353
0.384
0.278
0.345
0.252
0.317
0.270
0.323
0.307
0.379
0.694
0u691
0.291
0.543
0.400
0.464
1.008
0.768
3.112
1.387
4.042
1.601
336
0.307
0.346
0:420
0.422
0.338
0.385
0.306
0.353
0.323
0.353
0.543
0.559
2,408
1.407
0.348
0.376
0.469
0.498
1.031
0.775
3,255
1.421
3,963
1,585
01446
0.614
1:913
1.166
3:711
1,532
Avg.
00.293
0.335
0.4402
0.410
0.316
0.368
0.296
0.345
0.320
0.353
0.463
0:488
1.342
0,930
0.332
0.366
0.447
0.487
0.987
0.756
3.370
3.97%
1.587
96
(0.139 0.237
0.112
0.240
0.155
0.253
0.1-10
0.238
0.299
0.373
0.251
0.323
0.261
0,345
0.420
0.466
0.599
0.587
0.3550
0:425
1,259
0.919
0.999
0.781
BCL
192
0.156
0.252
0.158
0.254
0.164
0. 16264
0.160
0.255
0.305
0.379
0.261
0.356
0.338
0.405
0.411
0.459
0.620
0.598
0.576
0.448
1,160
0.873
0.938
0,753
336
0.175
0.175
0.180
0.319
0.391
0.360
0.445
0.410
0.474
1.157
0.872
0.925
0.745
720
0.233
0.317
0.230
0.315
0.223
0.321
0,241
0.325
0.369
0.126
0.530
0.585
0.715
0.685
0.510
0.521
0.757
0.661
0.611
0.597
1.203
0.898
1.001
0.790
Avg.
0.176
0.269
0.176
0.270
0.180
0,280
0.180
0.273
0.323
0.392
0.346
0.427
0.431
0.478
0.444
0.480
0.660
0.617
0.441
0.489
1.195
0.891
0.965
0.768
Tenf Fice
01,414
0.297
0.289
0.719
0.416
0.639
0,400
0.672
0.405
1,412
1.643
0.636
1.557
0,821
1.527
0.815
0.476
0.301
0.363
0.305
0,415
0.296
0.748
0.228
0.637
D.416
0..22
0.424
1,419
0.816
1 1.44
0854
1,207
0.661
0.765
1.538
0.817
336
0.434
0.30%
0.488
0.365
0.449
0.313
0.426
0.304
0.853
0.471
0.655
0.427
0.749
0.454
1.443
0.815
1,711
0.878
1.334
0.713
1.521
0.812
1,550
0.819
720
0.487
0.337
0.537
0.386
0.484
0.336
0.474
0.331
1,485
0.825
0.722
0:456
01:447
0.499
1.539
0.837
3660
1.157
1.292
0.726
1.605
0.846
1,588
0,833
Avg.
0.951
0.535
0.663
0.425
0.774
1.551
0.821
Average 00.371
0.367
0.521
0.447
0.413
0.401
0.385
0.376
0.556
0.458
0.511
0.172
0.687
0.559
0.674
0.522
0.912
0.665
1.137
0.712
1.850
0.967
L.888
0.974
19

---

D.3 Long-term Time-series Forecasting
Here we investigate whether our architecture performs consistently well with more training data. Thus, we follow the classical experiment settings Nie et al. (2022) and conduct experiments on full data. The results are shown in Table [4] Overall, 66 FPT achieves comparable performance to PatchTST,, Dlinear and outperforms other baselines by large margin. Compared with the second best transformer-based baseline method FEDformer, G6T FPT yields an overall 18.7% relatively MSE reduction. It verifies the effectiveness of NLP pretrained model in time series forecasting, not limited to the few-shot setting.
Detail Experiment Table for Long-term Time-Series Forecasting in table 14
Table 14: Full results on full data. We use prediction length O E {96, 192, 336, ]20] for ILI and O {24, 36, 48, 60) for others. A lower MSE indicates better performance. Black: best, Red: second best.
Methods
GPT2(6)
GPT2(0)
DLincar
PatchTST
TimesNot
FEDiommer
Autoformac
Siitionay
ETSTermer
LighTT
Informer
Reformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE|
MSE
MAE
MSE
MAE
MSE
MAB]
MSE
MAE
MSE
MAE
MSE
MAE]
MSB
MAE
MSE
MAE
MSE
MAE
E77M Weather
00.162
0.212
0.181
00.176
0.237
0.149 0.198
0.172
0.266
0.173
0.223
0.119
0.281
0.242
0.300
[0,689
192
0.204
0.248
0.222
0.266
0.220
0.282
0.194
0.241
0.219 0.261
0.276
0.336
0.307
0.367
0.245
0.285
0.237
0.312
0.227
0.267
0.598
0.544
0.752
0.638
336
0.251
0.286
0.270
0.299
0.265
0.319
0,215
0.282
0.280
0.306
0.359
0,380
0.359
0.395
0.321
0.338
0.298
0.353
0.282
0.334
0,578
0.523
0.639
0.596
720
0.326
0.337
0.338
0.345
0.333
0.362
0.314
0.334
0.365
0.359
0.403
0.428
0.419
0.428
0.414
0.410
0352
0.288
0.352
0.386
1,599
0.741
1.130
0.792
Avg
0.233
0.270
0.252
0.285
0.248
0.300
0.225
0.264
0.259
0.360
0.338
0.362
0.288
0.314
0271
0.334
0.261
0.312
0.634
0.656
96
00.376
0.307
D.422
0.428
0.375
D. 0.399)
03370
0.399
D.384
0.402
0.336
0.419
0.449
0.459
0.513
0-491
0.494
0.479
00.424
0.432
0.865
0.713
00.37
0.837
0.72X
192
0:416
0.118
0.166
0.150
0.105
0:116
0:413
0:421
0.136
0.429
04420
0:118
0.500
0.482
0.534
0.504
0.538
0.501
0:175
0.462
1,008
0.792
0,923
0.766
0,442
0.435
0.488
0.439
0:443
0,422
0.4%
0.499
0.588
0.535
0.574
0.531
0.518
0.488
1.107
0,835
730
01447
0.456
0.485
0.47%
0472
0.490
0:447
0.466
0.521
0,500
0.507
0,514
0.512
0.643
0.616
0.562
0:535
01,447
0.533
1.181
0.865
1.257
0.889
Avg
01:477
0.426
0.465
0.155
0.412
0.437
0.413
0.430
0.458
0.450
0.440
0,160
0.496
0.487
0,570
0.537
0.542
0,510
0:491
0.179
1,000
0.795
1.029
0.805
96
00.285
0.342
10.318
0.368
0.2289
0.353
0..274
0.336
0.340
0.374
0.355
0.397
0.346
0,388
0.476
0.458
(0:340
0.391
0.3397
0.437
3.758
1,525
2.626
1.317
ET71Q
192
0.354
0.383
0.383
0.418
0.339
0.402
0,452
0.493
0.430
0.439
0.520
5.602
1,911
11.12
2.979
336
0.373
0.407
0.406
0.427
0.448
0.465
0.339
0.380
0.452
0,452
0.496
0.487
0.482
0.4%6
0.552
0.551
0.485
0.479
0.626
0.559
4.721
1.835
9.323
2.769
720
0.406
0.441
0:420
0.446
0.605
0.551
0.379
0.422
0.462
0.468
0.463
0.474
0.515
0.511
0,562
0.560
0.500
0.497
0.863
0.672
3.647
1.625
3.874
1.697
Arg
0.354
0.394
0.381
0.412
0.431
0.446
0.330
0.379
0.414
0.427
0.437
0.449
0.450
0.459
0.526
0.516
0.439
0.452
0.602
0.543
4.431
1.729
6,736
2.191
96
00.292
0
0.330
0.372
10:299
0.343
(0.290
0.342
0.338
0.375
0.379
0.419
0.500
0.475
0.386
0.398
(0:375
0.39%
01.744
0.400
0.672
0.571
10.338
0.573
577m2 ETTml
192
0.332
0.372
0.371
0.391
0.335
0.365
0.332
0.569
0.374
0.387
0.126
0.111
0.553
0.496
0.159
0.461
0.108
0.110
0.400
0.107
0.795
0.669
0.658
0.592
336
0.366
0. 0.394
0.398
0.409
0.369
0.386
0.366
0. 0.392
0.410
0.411
0.445
0.459
0.621
0.537
0.495
0.464
0.435
0,428
0.43K
0.438
1,212
0.871
0.898
0.721
0,417
0.425
0.421
0.416
0.420
0,450
0.516
0.499
0.527
1,166
0.841
Avg
00.352
0.383
0.388
0.403
0.357
0.378
0.351
0.380
0.100
0.106
0.448
0.152
0.588
0.517
0.181
0.156
0.119
0.125
00:35
0.437
0.961
0.734
0.799
0.671
96
[0.173
0.262
0.112
0
10.167
0.269
0.165
0.255
0.187
0.267
0.201
0.287
0.255
0.339
0.192
0.274
00.889
0.290
0.2099
0.308
0.365
0.453
0.658
0.619
0.229
0.301
0.245
0.317
0.224
0.303
0.220
0.292
0.249
0.309
0.269
0.328
0.281
0.340
0.280
0.339
0.253
0.319
0.311
0.382
0.533
0.563
1.078
0.827
336
0.286
0.341
0.302
0,352
0.281
0.342
0.274
0.329
0.321
0.351
0.325
0.366
0.339
0.372
0.334
0.361
00311
0:357
01442
0
1,363
0.887
1.549
0.972
720
0.578
0.401
0.399
0.408
0.397
0.421
0.362
0.385
0.408
0.403
0.421
0.415
0.433
0.432
0.417
0.43
0.414
0.413
0.675
0.587
3.379
1.358
2.631
1.242
Avg
0.266
0.326
0.284
0.339
0.267
0.333
0.255
0,315
0.291
0.333
0.305
0.349
0.327
0.371
0.306
0.347
0.293
0.342
0.409
0.436
1.410
0.810
1.479
0.915
24
2,063
2.723
1.099
2:215
1.081
1.319
0.754
2.317
1,260
3.483
1.287
2.294
0.945
2.527
1.020
8.313
2. 2.144
5.764
1.677
1.868
0.892
2.027
0.966
1.963
0.963
1.430
0.834
1,972
0.920
2.679
1.080
3,103
1.148
1.825
0.848
2.615
1.007
6.631
1.902
4.755
1,667
1.783
1.448
11.1
1.790
0.884
2.206
1.022
2. 2,1
1.021
1.553
0.815
2.238
0.910
2.622
1.078
2.669
1.085
2,010
0.900
1.359
0.972
7.299
1 1.962
1,763
1.469
1,832
1.165
1.979
0.957
1.976
0.983
26338
1.096
1.470
0.788
2.027
0.928
2:857
1.157
2.770
1.125
2.178
0.963
2.487
1.016
7.283
1.985
5,264
1.564
4.852
1.483
Arg
1.955
2.233
1.017
2.
LO41
1.433
2.139
0.931
2.547
1.144
3.006
2.077
0.914
2.497
1.004
7.382
2.003
5.137
1.544
1.724
96
00.199
0.23%
0.
0.234
0. 140
0.237
0.119
0.222
0.168
0.272
0.199
0.308
0.200
0.317
0.169
0.273
0.188
0. 0.304
00.207
0. 0.307
0.274
0.368
0.312
0.402
ECL
192
0.153
0.251
0.152
0.247
0.153
0.249
0.157
0.240
0.184
0.289
0.201
0.315
0.222
0.334
0.182
0.286
0.199
0.315
0.213
0.316
0.296
0.386
0.348
0.433
0.169
0.266
0.169
0.267
0.163
0.231
0.304
0.212
0.329
(1.330
0.333
0.206
0.297
0.207
0.295
0.203
0.301
0.197
0.290
0.220
0.520
0.246
0.254
0.361
0.222
0.321
0.233
0.345
01.268
0.360
0.173
0,439
0.340
0.420
Avg
0.167
0.263
00.166
0.259
0.166
0,263
0.161
0.252
0,192
0.295
0.214
0.327
0.227
0.338
0.193
0.296
0.208
0.323
0.2229
0.329
0,311
0.397
0.338
0.412
Tre fee
96
00.388
0.407
0.290
0.282
0.390
0.272
0.410
0.423
0.287
0.282
0.360
0.379
0.249
0.256
0.593
0.617
0.336
0.321
005577
0.373
0.366
0.613
0,388
0.613
0.612
0.340
0.338
006007
0.621
0.392
0.399
0.615
0.601
0.391
0.352
0.719
0.391
10,732
0.423
336
0:412
0.294
0.113
0.280
0.136
0.296
0,392
0.261
0.629
0.336
0.621
0.383
0.622
0.337
0.618
0.328
0.622
0.396
0.613
0.386
0.777
0.120
0.742
0.120
720
0.450
0,312
0.447
0.298
0.466
0.315
0.432
0.286
0.640
0.350
0.626
0.382
0.660
0.408
0.653
0.355
0.632
0.396
0.658
0.407
0.864
0.472
0.755
0.423
Avg
0.414
0.294
0.413
0,281
0.433
0.295
0.390
0.263
0.620
0.336
0.610
0.376
0.628
0.379
0.624
0.340
0.621
0.396
0.622
0.392
0,764
0.416
0,741
0.422
Average
00.516
0.407
0.573
0.0.431
00.622
0.436
(1.446
0.386
0.596
0,433
0.701
0.489
0.757
0.511
0.633
0.46
00062
0.473
1.303
0.616
1.836
0.871
2.081
0.954
20

---

D.4 Mean and STD for Few-shot Learning
Table [3 lists both mean and STD for GPT2(6). DLinear and PatchTST with runs on 5% ETTh2 and ETTm2. The results show a small variance in performance of GPT2(6) that represents the stability of GPT2(6).
Table 15: A subset of results showing both Mean and STD on 5% datasets.
Methods Metric
GPT2-backbonede Layers) MAR
ETTm2 ETTR2
96
0.376
0.0072
0.421 + 0.0054
192
0.418 1: 0.0013
0.441 : 0.0014
336
0.439 0.0002
720
96
0.199 1 0.0040
0.280 1: 0.0042
192
0.0017
336
0,318
+ 0.0046
0.353 + 0.0032
720
0,460
0.0132
0.436 0.0066
D.5 Comparison with Traditional Methods on Few-shot Learning
Since deep learning methods are more advantageous than traditional methods when applied to large datasets. For few-shot learning, traditional methods should also consider. The results are shown in Table 16 that GPT2(6) also achieves best performance.
Table 16: Comparison with traditional methods.
Methods Metric
GPT2(6) 5%
GPT2(6)
10%
ETS
ARIMA
NaiveDrift
MSE
MAE
MSE
MAE
MSF
MAE
MSF
MAF
MSE
MAF
#27 12
96
0.421
0,331
0.374
2,944
0.742
0.481
0.443
0,764
0.561
192
0.418
0.441
0.402
0.411
10.226
1,212
0.585
0.495
1.560
0,785
96
0.386
0.405
0.390
0.404
52.237
2.689
0.693
0.547
1.539
0,913
192
0.440
0.438
0.429
0.423
186.445
4.654
0.710
0.557
2.869
1.215
D.6 Baselines with Instance Normalization
Instance normalization Kim et al. (2022) is a plug-in for time series for distribution shift. Most baselines, such as Autoformer and FEDformer not equipped with instance normalization. Thus, for fair comparison, we add the experiment, as in Table [7] for basclines w/o instance normalization and GPT(6) can also perform superior.
Table 17: Comparison on 5% data. Autoformer and FEDformer are equiped with instance normaliza- tion.
Methods Metric
GPTZ26)
PatchTST MSE
DLincar
Autoformer MSE
Autoformen(Revin) MAR
FeDlormer MSE
(EDformen(Revin)
MSE
MAR
MAE
MAR
MAF
0.199
0.280
0.206
0.288
0.236
0.326
0,232
0.322
0.224
0.300
0.229
0.320
0.223
0.296
192
0.256
0.316
0.264
0.324
0.306
0.373
0.291
0.357
0.296
0.343
0.294
0.361
0.288
0.336
D.7 Detailed Definition and Results of Zero-shot Learning
Task Definition Each experiment contains two distinct datasets, source, and target datasets. The source dataset used train the model and then forecasts without fine-tuning in the target dataset. The target dataset is split into von-overlapping historical and test sequences. We use the historical sequence as input to the model, and the obtained output used to calculate errors with the test sequences. Besides meta-learning-basee models like N-BEATS, evaluated models' parameters are not allowed any adjustment using the forecasting phase. Also, same as Oreshkin et al. (202T). each data set adopts a specific metric (M4: MAAEE; M3: sMAPE; TOURISM: MAPE; ELECTR: ND)
21

---

Detailed Results Here, we list detailed performance of zero-shot learning in Table 18| Table 19 and Table 20 For each dataset, we separately list the performance models under diverse frequency. Compared to the most recent published method Dinnerr, GPT2(6) performs superior most situations. Also, GPT2(6) does not use any information from the test data, but achieves comparable performance of meta-leaning based N-BEATS.
Table 18: Zero-shot performance on M4 (sMAPE).
Yearly (23k)
Quarterly (24k)
Monthly (48k)
Others (5k)
Average
(100k)
N-BEATS-FR
13.267
9.596
12.676
4.696
11.675
DLinear-M3
14.193
18.856
14,765
9.194
15,337
TimesNet-M3
15.655
11.877
16.165
6.863
14,553
PatchTST-M3
13,966
10.929
14.664
7.087
13.228
ET
27.846
36.134
25.114
12.338
27.748
LightTS-M3
13.787
11.289
15.181
9.117
13.623
Stationary-Mr M3
14.988
11.686
16.098
6.977
14.327
FEDformer- M3
13.887
11.513
18.154
7.529
15.047
Autoformer- M3
14.552
17.341
25,063
9.666
20.022
InformerM33
18.542
16.907
23.454
7.348
19.047
ReformerM3 15.652
11.051
15.604
7.001
14,092
-PT
13.740
10.787
14.630
7.081
13.125
Table 19: Zero-shot performance on M3 (sMAPE).
Yearly (645)
Quarterly (756)
Monthly (1428)
Others (174)
Average (3003)
N-BEATS-M4
15.07
9.07
13.19
4.29
12.38
N-BEATS-FR
16.43
9.05
13.30
4.51
12.61
DLinear-M4
17.43
9.74
15.65
6.81
14.03
M4
18.75
12.26
14.01
6.88
14.17
PatchTST-M4
15.99
9.62
14.71
9.44
13.39
ETSformer-Ms
20.56
11.65
16.97
10.57
16.03
LightTS-M4
15.63
9.40
24.60
8.28
17.90
Stationary-Me M4
17.05
12.56
16.82
8.13
15.29
FEDformer- M4
16.00
9.48
15.12
8.94
13.53
Autoformer-M M4
16.
13.92
16,91
14.68
15.87
Informer-M M4
19.70
13.00
15.91
13.03
15.82
Reformer-M4
16.03
9.76
14.80
7.53
13.37
GP
16.42
10.13
14.10
4.81
13.06
E Proof
In our numerical experiments, we obtain two interesting observations. First, the token similarity within a sample is larger in pretrained LM. We report the layer-wise average token cosine similarity in ETTh2 experiment Figure In particular, Figure 7 (a) shows that fine-tuned random initialed GPT2(6) model, the token similarity is around 0.1-0.2 among different layers. When switching to the frozen pre-trained GPT2-F FPT model, the token similarity significantly increases in the deep layers and eventually reaches more than 0.9 in the last layer. The ETTh2 dataset contains high volatility hourly information related to the electricity transformer temperature. similarity implies the eigh-frequency noise in the data is eased and only low-frequency information will reserved. In other words, after going through the pretrained GPT2-FPT model, the signal-noise ratio is enhanced. We use the following theorem to characterize this behavior.
this situation, higher token
22

---

Table 20: Zero-shot performance Tourism (MAPE).
Yearly (518)
Quarterly (427)
Monthly (366)
Average (1311)
N-BEATS M4
23.57
14.66
19.32
18.82
N-BEATS-FR
23.43
14.45
20.47
19.46
DLinear-Mi M4
39.59
18.30
24.76
28.51
TimesNet-Me 35.59
19.22
30.54
28.84
PatchTST-Mr
33.23
19.27
27.57
27.10
ETSfomcr-MM
391.60
35.56
50.47
180.40
LightTS M4
138.22
16.28
25.34
66.99
Stationary-M4
35,42
35.15
65.58
43.75
FEDformer-M4 43.41
19.88
28.39
31.55
Autoformer-Me
51.19
34.95
31.47
40.39
Informer M4
41.16
30.98
33,92
35,82
Reformer-M M4
33.86
16.85
23.71
25.48
GPT2(6)-MB
27.17
16.21
21.92
22.14
E.1 Theorem E.1
Theorem E.1 (informal). We consider the self-attention for 1-th query token. Let's assume the input token I; are bounded with mean μ for i 1,,,.... n. Under mild conditions, with high probability, the output value token Vi converges to μW, on the order of (n21//2), where W, is the parameter matrix to compute the value token.
The Theorem E.1 describes the self-attention structure can efficiently make output value token Vi converge its mean value w... In the time series forecasting task, each token represents several adjacent points in time series. When the time series has some periodical or translation invariant structures, by comparing given token with other tokens, one could have a higher chance to figure out those invariant structures. This phenomenon especially important in few-shot forecasting tasks. Without enough token noise distillation ability, the model will more likely tend to overfit due to insufficient training data.
We denote H; as i-th element of vector æ. Wy as the element at i-th row and j-th column of matrix W, and Wj as the j-th row of matrix W. Moreover, denote æ; as the i-th patch (token) of the inputs with x; = X,.
Before given the formal statement the Theorem E.1 we first show the assumptions.
1. The token x; is the sub-suusee gaussian random vector with mean μ4, and variance (o2/d)I for 1,2....,
2. μ follows a discrete distribution with finite values μ V. Morcover, there exist < M1, 0 1/2 14 such that a) ||p Vi, and b) μ, W. WT kH; [M2, va] for all i and TI U2 for all μ4 # μ4 V.
μ.. W. W k
3. Wv and Wa WT are element-wise bounded with vs and Pe respectively, that is, ( V/ and (Wo WT (iz) I Vo, for all i. from to
In the above assumptions, ensure that for given query patch, the difference between the clustering center and noises are large enough to distinguished.
Let patch æ; be oi-subgaussian random variable with mean ir and all putches follow the same clustering center of query 1. Per Assumptions aforementioned, when Vd 2 3(2(6, d) + 22 + vs), then with probability 1 56, we have
Theorem E.2 (formal statement Theorem E.
23

---

Ei=1 exp
Va W W 2:
D:
2
2d 6
HiWy
S 4exp
log
دド=1 exp
(2a W W kT))
vd
dn
1/2 14 + (d),d)
+7 exp
-1
|| Wro
Vd
where (b, = 20v1ve log (1) + 20 ve log ($).
Proof.
See the proof of Lemma 2 in Wang el al. 2022 with h1 = k- = n.
E.2 Theorem
We first give the formal statement of Theorem E.4
Theorem E.3 (formal statement Theorem E.4). Let g: E R2 and yi E R be the feuture
map vector and forecasting largets for the sample i = 1,2,..., N respectively, and we assume
i= ? a for some > O. We want learn matrix W E RRXT from the following N E
optimization problem:
N yy: y:
W = arg min
Σ
W
(I)
2N
we apply stochastic gradient descent with diminishing step sizes th + at step t, we will need t = O steps to reach
1
W,9.
yol2
W
-11l2 Se,
2N
2N
where W* is the optimal solution and W, is the step's solution and Ô we suppress the logarithmic dependence.
Proof. As we assume N. i1=1 9:9; > ol, the hessian of optimization problem in is also
positive definite, which equivalent to the optimization problem in (I) is strongly convex with parameter proportional to σ. Then via standard stochastic gradient decent analysis (e.g., section 3.1 Lacoste-Julien al. (2012)) we obtain:
1 2N
logt at
Σ
W - y:112
Σ
WW'g g:
-v113 s 50
0 (3)
2N
f=1
Therefore, to reach optimization gap, we just need to set -(--11-1))
The second observation is that for the pretrained GPT2-FPT model, the last transformer layer's outputs, i... feature maps, are spread widely throughout the feature space. We report the t-SNE visualization of the feature maps for GPT2-FPT and an end-to-end model PatchTST in Figure In Figure (a) and (b), we color the samples chunked from the one single time series into the same color and the same configuration of the T-SNE is applied. One may observe that the feature maps of GPT2-FPT has less concentration compared to PSt It implies the GPT2-FPT's feature maps corresponding different samples more distinctive which eventually facilitates the learning ability of the last MLP layer. Researchers Wang & Isola (220)) have found that contrastive learning- -based representation learning may result in uniform distribution of training data, and such behavior plays an important role in its good downstream task performance. We use the following theorem to justify
Theorem E.4 (informal). Let gi and be the feature map vector and forecasting targets for the sample i = 1,2, N respectively, and we assume &. E. g.g. 2 I for some > O. Under mild conditions, if we train an MLP layer that maps feature maps to forecusting targets via the stochastic gradient descent, the total step to reach some optimization tolerance is on the order of O(o
24

---

Replacz
Replace
2.00
0200
Raplica
Replac: Ratio 40%
Replace
Replase
Figure 6: The performance and token similarity within samples with respect to each layer with different random replace ratios. Pretrained parameters are replaced by random initial parameters according to certain proportions.
790
1751
2700
000
0.9
580
01512
48
0ypr
2000
1000
2500
03
3.
05
02
00
Random Initializaaionn
(b) Pretrained Parametions
PCA as Atentinn
Figure The token similarity within samples with respect each layer. (a) GPT2-moPretrain-moe (b) GPT2-Pretraiie -nrde:: (c) Pretrained attention is replaced by PCA.
The Theorem E,4 considers the covariate matrix of feature maps being positive definite that indicates the set of all feature maps {gi} } spans the whole feature spaces, and the higher spread level gives a larger or. In this case, if we only want to learn an MLP layer, the problem reduces to well-conditioned least-squared regression problem. Then the fast convergence rate is achieved.
Efficiently learning the last MLP layer plays a very important role in time series forecasting and substantially impact the prediction performance. In /Zeng al. (2023), the authors show that learning single MLP layer can also bring very promising performance. In few-shot forecasting, the pre-trained GPT2 model may still preserve highly diverse feature maps than end-to-end type models and eventually leads to fast learning speed on the last MLP layer.
Another possible benefit of wide spared feature maps enhancing the model memorization ability when using a multi-layer decoder structure. In the literature on network memorization ability (e.g., Vardi al. (2021): Yun al. (2020)), the deep learning model tends to have better memorization ability when feature maps are well separated. In forecasting tasks, capturing extreme or rare behavior very important. The pretrained GPT gains more capacity in the decoder to correctly forecast. uncommon time series.
F
N-gram Explanation for Universality
Why does the proposed pretrained siseainned-fozze-mmodee work so effectively? We have achieved state-of- the-art performance in time series analysis using language model that is mostly trained on natural language data. The answer lies in the universality of the frozen structure, which includes attention layers and Feed Forward layers. We can represent images and time series forecasting tasks as an n-gram estimation problem, akin to text analysis. by employing patching approach. This method treats subsequences of time series or image patches as individual tokens. Central to sequential prediction is the n-order Markov process, and simple way to capture the n-order Markov process n-gram language model. To predict next token Wig we need to compute y Ww [W1,- wn-1). 1), which be further computed as pl p(wgw1 ... wn-1)/p(ww, ... wx-1). Hence, the core of n-gram
25

---

65
34
09
Seg arc0 .arg?
(5) GPI2-backbome
PatchTTT
(c) Token similarity
Figure 8: The t-SNE visualization of sample feature maps for (a) GPT-backbone, (b) end- mod-to-end-- PatehTST-model. (c) The token similarity within samples within different continuous sequence lengths.
language model is to estimate the probability of observing a sequence of tokens. When n is large, most of n token sequences will not be observed from data, leading to the sparse data problem, a common challenge faced by n-gram language model. As a result, a large body of research in n-gram language model is focused on how to effectively estimate probability of having n-token sequences even when they are NOT observed from data. We hypothesize that the transformer model pretrained by GPT-2 essentially allows us to estimate p(wpW11 ... wn-1) from observations of significantly shorter token sequences. In this section, we will show that the function of estimating probabilities of longer sequences from observation of shorter sequences is universal and is independent from domain as long as data exhibit skew distribution (e-g.. follows a power law). We note that our work is closely related to the discussion presented in Elhage et al. (2021); ) Olsson et al. (0022). where the authors also connect the function of transformer to compute of n-grams. We however note that our key result is to show the universality in computing probability of longer sequences from observations of shorter sequences, which can't be found in any existing studies. Although the discussion restricted to discrete tokens, it should be generalized to continuous signals signals into a finite number of discrete tokens, similar to what BEiT Bao et al. (2022) did.
can always quantize continuous
To gain a better understanding, let's start by examining a "zero-layer Transformer model. This model operates by taking token, embedding it, and transforming it back to produce logits that predict the subsequent token. Because it cannot transfer information from other tokens, it relies solely on the current token to predict the next one. Consequently, the optimal behavior of this model is closely resemble the bigram log-likelihood.
Then we move on the so-called "attention-only" transformer, which doesn't 't have MLP layers. As discussed in recent work Elhage et al. (202I), one-layer attention-only Transformers can be comprehended as a combination of a bigram model and multiple "skip-trigram" models (impacting the probabilities of sequences "A.. BC"). This can be intuitively understood as each attention head having the ability to selectively attend from the current token ("B") to a previous token ("A") and transfer relevant information to fine-tune the probability of potential subsequent tokens ("C"). Olsson al. al..((2.0)))) (2022) further discusses a multi-layer transformer can do more complex n- n-gram estimation using an induction heads mechanism. To be more precise, induction heads employ a straightforward principle: the '[A][B] ... [A] + [B]' rule, which elevates the likelihood of generating the subsequent token 'B' given the current token "' if there is fuzzy match of the AB bigram in the historical context. This rule seems to largely decouple and B, which means they do not memorize a fixed table of n-gram statistics. The rule [A][B] ... [A] + [B] applies regardless of what A and B are, which can abstract to new patterns.
Building upon these discussions, we are now prepared to substantiate the following argument: For sequential data following a power law, there is a potentially universal solution to the final estimation of n-gram probabilities. That's 's the reason behind the universality of pretrained LM's performance cross- domain tasks. For simplicity, we assume that n so large that we are unable to observe any occurrence of n-gram from data, and we only observe the occurrence of n' hr-grmms with n' n.. We denote by si the ith unique n-gram, and by the notation sy E s. if n' -gram si appears
26

---

in s. the ith n-gram. Let m. be the number of unique n n-grams. According to the maximum entropy model, our estimation of n n-gram probabilities can be cast into the following optimization problem:
min Ei p(s?) log((s)) S. I. Ding'e Esp p(s,) - o(sg') where p(s))) represents the proba- bility of observing pattern s) from the data and j E [m n' E [n-11] 1].
For each constraint for P(s})) ) we introduce Lagrangian dual variable Az', and rewrite the optimiza- tion problem as follows:
E exp (Eiws)eeg'ey Eop 15 -E Ejn D('))
miny log
where n-gram probability p(53) is given as p(87) 2(A) exp [ Ei Eap X3') and 2(2) =
E exp (n (i)ay' Exp X
In the case that all n-grams follow power law, for each n'' E [n mn-18 1], we divide n' r'-gram into two groups: the group Var includes the high frequency n' n'-gram and the group Un including the low frequency of n' -r-g.am For simplicity, we assume that the probability for all the high frequency n'-grams are roughly an' E [0, 1] and the probability for all the low frequency n' -rags are roughly 3n E [0, 1]. By assuming that all the patterns in Vn,' and Un share similar appearance frequency, simplify the optimization problem by only introducing two dual variables for each n'-gram, i.e. Ag for high-frequency patterns and Ag for low- fow-frequency patterns as follow Using these notations, we have the optimization problem simplified as
minx
log(> (E exp(E 1n- Ejseg' Eay No 1(s E Vnr)
+25 I(55' E (nx
-E (xr In' + 2b' hnd
where In' = Enger.. ū(55) and hw = Eoyeuu, eu,, D(s)))
Furthermore, let o/' be the probability to observe high frequency n'-gram appearing in any n-gram, and g be the probability observe low frequency n' n'-gram appearing in any n-gram, have Σ exp Ejag' Eop No (ss'' E Var) + NE I(s)' u.:)
my IT==1 (1+ of exp(A: () + q. exp(> )) + (Vm,).
By skipping the term O (m/m) further simplify the optimization problem as
min
En log
1 + ohd exp(X:)) - + Σn log [ [1+ q. exp(>r') - >² ha
which is equivalent
Xg. miny log + oh exp(A))
>Aghn
-
As illustrated by the above analysis, dual variables
X min log
1 + o8 exp(A)) Ahn-
X²" and Ag' will only depend on statistics q. %%6 In' and hng They are independent from the detailed statistics 7(5))) and how each n'-gram appears in different n-gram. Thus, this simple analysis does indicate, to some degree, that the solution obtained from the maximum entropy model can be universal, as long as n-grams follow skewed distributions like power law.
We informally demonstrate that transformer models utilize attention mechanisms to perform a sophisticated form of n-gram estimation, and the generation rule for such n-gram distributions could be universal. This is how universality is achieved in our proposed cross-domain knowledge transfer. However, we currently lack concrete metric to evaluate the performance of knowledge transfer between different domains, which requires further investigation. Nonetheless, in our experimental study, demonstrate that transformer model (beit) Bao et al. 2022) trained on images can perform well on cross-domain time series forecasting tasks.
27

---

G
Connection between self-attention and Principle component analysis
Understand the Gradient Structure of Self-Attention
Let X = (I) ( E R be the input pattern, and let f(X) = ((x()),..,,x()))))
RNDD + R be the function for self-attention, i.e.
fX) = (xltmax(XAXx]]
where WoWk RDDD, Let the Jacobian J (fx()
represent the gradient f(X)
i-j= with respect to input pattern. The lemma below shows an important structure of J.
dxy
2
Lemma G.I.
[ 1 |Al2 Σ
(Pe, + 2)) 24 - E3 Pa
+ A
where
A = 2 Σ N Pig Zzj - N
-E PAATT
Al
N Σ
2j=1
?
and
Pu. Ek=1 exp(x, Araa)
exp(2 Axe)
Proof. According to the analysis from the work, we have the gradient Jig 0f((X)
is given by
Jı, = Pi + + T (x Ao,, + F) where Q' = diag(P,,)) - P..
Here Pa: E
R represents the i-th row of matrix P. We thus have
[// s ΣN ExaiMal
s E Pag + E q'x////b
Em IX (X bblAle
s N N+ +M E Pulks E Pigzs
s N W+ EN, Eje Pig |rg -
Ein Bazi 1// Σn
Pig |s] (xy xTr..)l
s EM (Pi. Jn X p... N
Ala
Σ
Pu lay
X xTR.l" 14 Σ mn
As indicated by Lemma 1, one of the key components in the upper bound of Jacobian is |x, Σj=1 ?.. Thus, through the optimization, we like to reduce the size of the gradient and therefore may prefer to reduce the quantity to Σ | Σド- P. Hence, it will be interesting understand the choice of W and W that leads to the minimization of ΣN Jx; E P
ie.. the following optimization problem to control the size of A. Connection between Self-Attention and Principal Component Analysis
min |AleSA
Σ I Σ Pajar where is introduced
Let consider the optimization problem in proximate Pi.
GG) when is small, we can ap- -x1/NN. We have By assuming that all the input patterns are
Ptj R N
+ - Any
Define
N EN Ir, - X PT. = = and Σn X XAx ((I-XTIX xXXA)xX x) The theorem below shows that A minimizing the objective MN [r X X Axx contains the largest m eigenvectors of X X where m the rank of A.
-
zero centralized, we have
Theorem 2. Let We and WK be matrices of size D x m. Let \ 2 A2 2 2 Ap be the eigenvalues of XT X ranked in descending order, and let v, E BD i 1....,D be the corre- sponding eigenvectors. The optimal solution A that minimizes EN px XT Axx is given by
A 1 x c.u.
Proof. Since Wo, Wx E Dxm where < D, we know that A is matrix of rank m. Hence, N Ek=m+h d We also know that by choosing A as
we know
min N N
jx XAx >
A = -i=l D.U
we have
28

---

XT XT (1-1_ 20 ) xxxx x) = =E A
Hence, the solution for minimizing Σ> - X X Ax;/² is essential a weighted combination of top eigenvectors of XTX. Since small gradient will prefer small quantity of Σ bs Ax,,? 3 by minimizing through the self-attention layer, essentially choose weight matrix Wo and Wx to be aligned with the principal directions of X X.
H Experiment Analysis and Other Key Results
H.1 Experiment analysis of GPT2-FPT model
In this section, conduct experiments to analyze whether the self-attention frozen pre-trained model improves performance compared with overall fine-tuning and random initialization. Firstly, we compare GPT2(66 FPT with the same model without freezing (No Freeze) and random initial model (No Pre-train). For the end-to-end paradigm No Pre-train GPT2-backbone (6 Layers), we directly train all parameters of the model. We summarize the results in Table 21 and Table 22 Then we analyze the performance of various layers to clarify our selection of GPT2(6) FPT.
Table 21: Model analysis results 5% data. We prediction length O (96, 396, 720] for IIJI and O E {24,36, 36, 48, 60} for others.
Methods
GPT2(6)
No Freeze
No Pretrain
Metric
MSE
MAE
MSE
MAE
MSE
MAB
m2 ETTwl ETT12 ETT41 Weather
96
0.175
0.230
0.185
0.229
0.199
0.254
192
0.227
0.276
0.275
0.300
0.262
0.302
336
0.286
0.331
0.326
0.345
720
0.366
0.379
0.380
0.388
0.405
0.396
0.543
0.564
0.882
0.643
192
0,748
0.580
0.907
0.632
1,389
0,817
336
0.754
0.595
0
0.655
2.968
1,149
720
96
0.376
0.421
0.440
0.449
0.465
0.457
192
0.418
0.441
0.503
0.478
0.614
0.536
336
0,408
0.439
0.691
0.572
0.596
0.529
720
96
0.386
0.405
0.429
0.452
0.394
0,410
192
0.440
0.438
0.496
0.470
0.432
0.432
336
0.485
0.459
0.535
0.489
0.491
0.464
720
0.557
0.592
0.564
96
0.199
0.280
0,217
0.293
0.301
0.353
0.321
336
0.318
0.353
0.331
0.368
0.371
0.398
720
0.460
0.439
0.460
0.436
0.659
0.528
Fine-tune More Parameters Compared with fine-tuning all parameters, self-attention frozen pre- trained model GPT2(6) FPT achieves better performance on most datasets and yields an overall 12.7% % relative MSE reduction on 5% data and 11.5% relative MSE reduction 10% data. It verifies that frozen pre-trained attention layers are effective for time series forecasting.
Parameters Initialization Compared with the random initial model, self-attention frozen pre-trained model GPT2(6) FPT achieves better performance on most datasets and yields an overall 21.2% % relative MSE reduction on 5% data and 14.3% % relative MSE reduction on 10% data. It again suggests that a model pre-trained on cross-domain data can achieve significant performance improvement in time series forecasting.
The Number of GPT2 Layers For most transformer-based methods in time-series forecasting Wu et al. (2021); : Nie et al.] (2022), no more than 3 encoder layers are included. However, most pre- trained models with at least 12 layers may suffer from overfitting in time series forecasting. To better balance performance and computational efficiency, we test using various numbers of layers on ETTh2. Additionally, we train a completely random initialized non non-pretrained GPT2 as a comparison. The results are shown in Figure for both 5% and 10% data, the pre-trained model unable to do well with few layers but significantly outperforms non-pre non-pre-trained GPT2 with more attention blocks transferred from NLP. It indicates that pre-trained attention layers produce
Zhou et al.] (2022); y
29

---

Table 22: No Pretrain and No Freeze results on 10% data. (96, 192, 336, 720} for ILI and O E {24, 36, 48, 60} for others.
We use prediction length O
GPT2(6)
No Freeze
Merric
MSF
MAF.
MSE
MAE
MSF
MAR
ETTx2 ETTmi ET ETTh1 Weather
96
0.163
0.215
0.168
0.221
0.175
0.229
192
0.210
0.254
0.238
0.286
0.244
0.287
0.256
0.292
0.318
0.301
720
0.321
0.339
0.398
0.383
0.390
0.378
96
0.458
0.532
0.680
0.560
192
0.570
0.516
0,713
0.579
0.738
0.602
0.586
720
0.725
0.591
0,945
0.688
2.994
1.169
0.394
192
0.402
0.411
0,464
0.455
0.482
0.466
336
0.406
0.433
0.420
0,439
0.540
0.496
720
0.449
0.464
0.535
0.515
0.564
0.519
0.390
0,404
0.429
0,430
0.385
0.401
192
0.429
0.423
0.463
0.446
0.426
0.421
336
0.469
0.439
0.530
0.470
0.506
0.455
0.591
0.576
96
0.188
0.269
0,243
0.311
0.244
0.315
192
0.251
0.307
0.352
0.318
0.363
336
0.307
0.346
0.337
0.364
0.409
0.412
0.426
0.440
0.473
great benefit in time series forecasting. Also, the pre-trained model achieves better performance between and layers. Thus GPT2 with 6 layers is chosen as our default architecture.
5% ETTh2
10% ETTh2
ETTh2 96, GPT2
ETTh2 96 GPT2
ETTh2 192 GPT2
192 GPT2
ETTh2 96 GP_GPT2 napretrain
0.8
ETTh2, 96 266_GP22_noppeeee nopretrann
ETTh2 192 GPT2 nopretrain
ETTh2 192 GPT2 nopretrain
0.7
MSE
0.5
0.4
0.4
Number of Lavers
Number of Layers
Figure 9: Comparison of pre-trained and non-pre-trained GPT2 with various layers on ETTh2. Color represents various prediction length O {96, 192) and line style means different models.
H.2 No Pre-training but Freezing
For comprehensively ablation pre-training and freezing strategies, also add experiment for random initialized GPT2(6) with freezing. The results in Table 23 shows that only input and output modules not work and pre-trained knowledge play importance part in time series tasks.
Table 23: Ablation on random initialized model with freezing.
Methods
GPT2(6)
No Freeze
No Pretrain
No Pretrain Freeze
MSE
0.376
0.41
0.480
0.459
0.465
0.457
0.540
0.497
192
0.488
0.441
0.503
0.478
0.614
0.536
0.721
0.580
30

---

H.3 Fine-Tuning Parameters Selection
In this section, we conduct ablation experiments study which parameters are important to fine-tune. Since the input embedding and output layers are randomly initialized for adapting to new domain, they must be trained. Then, we study adding layer normalization and positional embeddings to the list of tunettunng parameters. Table showsw the results that re-train parameters of layer normalization and positional embeddings can bring certain benefits, especially in longer prediction lengths. Thus, we follow the standard practice to re-train positional embeddings and layer normalization.
Table 24: Ablation by fixing positional embeddings or layer normalization on 5% ETTml and ETTm2. Parameters of GPT2(6) are successively added to the list of fine-tumed parameters.
Methods
Input Ourput
LN
POS
Metric
MSE
MSE
MSE
ETTm2 ETTml
0.395
0.392
0.409
0.386
0.405
192
0.444
0.438
01436
0.435
0.440
0.438
0.510
0.495
720
0.607
0,517
0.564
0,503
0.557
0.499
96
0.198
0.282
0.198
0.279
0.199
0.280
0.261
0.324
0.263
0.325
0.256
0.316
336
0.336
0,377
0.322
0,356
0.318
0.353
720
0.473
0.444
0.457
0.435
0.460
0.436
H.4 Analysis of Data Volume
Results of few-shot learning show that GPT2(6) FPT shows SOTA performance in few-shot learning tasks in which the model trained on 5% data and 10% data. Plus, it has comparable performance with the SOTA baselines PatchTST and Dlinear on full sample forecasting setting as well. This phenomenon raises a question that how performance changes with an increase data sample size. Thus, we conduct experiments on various percentages P E (5%, 10%, 20%. 50%, 80%, 100%) of ETTh2. Figure 10 shows that the performance improvement for GPT2(6) FPT is almost flattened. These results illustrate that such cross-domain FPT model is extremely efficient in few-shot time series forecasting and only requires few fine-tuning samples to reach SOTA performance. For more complete data, end-to-end training models start to catch up, but still, GPT2(6) FPT model can be comparable those SOTA end-to-ent training algorithms.
Various Percentage of ETTh2
GPT2-backboned6 Layeri_ETTh2_66
0.60
DLinear
Sairear_ETTh2..5 ETTh2_96
PatchTST ETTh2
0.55
GPT2-backbone66 Layeri_ 192
DLinear 192
PatchTST_ETIE EETh2_ 192
0.50
0.45
0.35
0.30
20%
50%
Percentage of Data
Figure 10: Results various percentages of ETTh2. Line color represents different models and line style means various prediction lengths O E {96, 192].
31

---

H.S Knowledge transfer with other Pre-trained Transformer Models
We investigate how other pre-trained transformer models perform and whether other domains can also help. Another NLP pre-trained model BeRTiD et al. (2019) and the CV pre-trained model BEiT Bao et al. (002)) are trained on 5% ETTh2 and 5% ET ETTm2. Similar to GPT2, we only reserve layers and freeze attention blocks. Our results shown in Table 25 that (ERT( FPT and (6)T( FPT are comparable to PatchTST and remarkably surpass other baselines. We come to the conclusion that the universality of our proposed architecture holds actoss other pre-trained-transfomoe transfomerr models. Moreover, the domain of successful knowledge transfer in time series forecasting is not limited to natural language. Knowledge from the CV domain can also help, supported by BEiT's experimental results.
Table 25: Results of frozen pretrained transformer variants on 5% ETTh2 and ETTm2. We use prediction length O {96, 192, 336, 720}. A lower MSE indicates better performance. Black: best, Red: second best, Violet: third best. '-' means that 5% time series is not sufficient to constitute a training set.
Methools
Metric
96
192
F1
336
720
96
192
FTTTi2
336
720
GPT33/hachhoeet Layers)
MSE
0.376
0.421
0.408
0.199
0.256
0.318
0.460
MAE
0.419
0.441
0.439
0.280
0,316
0.353
0.436
BERT-backlondd6 Layers)
MSE
0,480
0.222
11.188
0,465
0.300
0.335
0.367
0.428
BEIT-backhomlf6 Layers)
MAE
MSE
0.405
0.418
0.446
0.448
0.524
0.500
0.200
0.291
0.326
0.272
0.362
0.331
0.433
0.452
Deines et al 2023
MSE
00:42
0.617
1.424
10236
0.306
0.380
0.674
MAE
01:456
0.542
0.849
0.326
0.373
0.423
0.583
PatchT'S
MSE
0.401
0.452
0.464
0.206
0.264
0.334
0.454
0.421
0.288
0.324
0.367 0.483
FED Zhou el.
2022
MAE
MSE
0.390
0.424
0.465
0:157
0.483
0.477
0.320
0.299
0,361
0,290
0.427
0.378
0.510
0.523
[02232
Autoforme Wa al.
2021
MAE
0.468
0.504
0.496
0.322
0.357
0.517 0.538
H.6 Full Results of Classification
Table 26: Full results for the classification task. *, in the Transformers indicates the name of *former.
Mithods
Classical mathidk
XGBoess
Rocket
LSTNee
RNN
LSSL.
TCN
Trans:
Rs.
In
Pyra.
Transformers
Auto:
Station
FED.
ETS.
Flow:
DL innner
MLP
LighTSS
TimeoNer
(PT22)5
Elhanol Elhano Lonentration
13.7
<5.2
3015
31.1
25.3
12.7
11.9
31.5
10.3
31.6
32.7
31.2
28.1
33.3
32.6
25.7
25.7
11.2
Deteriitttonn
63.3
64.7
05.7
66.7
52.8
67.3
68.6
67.0
65.7
68.4
68.0
66.0
66.3
67.6
68.0
67.55
68.6
69.2
Thndwriing
15.8
58.8
25.8
24.6
53.3
32.0
27.4
32.8
29.4
36.7
31.6
26.0
32.5
33.8
27.0
26.1
32.1
32.7
Heartbeat
75.2
77.6
77.2
98.0
SotRegulaiioSCPP
84.6
92.5
39.8
93.2
SelRegilatimSSPP
48.9
52.8
52.2
55.6
56.7
53.3
53.3
57.3
56.4
55.0
56.
$1.1
$7.3
59.4
SpokenAnibieiigla
09.6
71.2
10000
100.0
95.5
98.4
97.0
100.0
99.6
1000
1000
1000
1000
95.8
81.4
10000
99.0
99.2
IWWneffestrel shoaay
75.9
94.4
87.8
85.9
88.4
85.6
85.6
85.6
$3.4
65.9
87.5
85.3
85.0
85.6
82.1
80.3
85.3
88.
Average
66.0
725
713
703
20.3
71.
71.
72.1
703
71.1
72.7
703
71.0
750
07.5
7044
73.6
74,0
H.7
Full Results of Anomaly Detection
H.8
Full Results of Imputation
H.9
Full Results of Short-term Forecasting
H.10
Input Length Setting Discussion
The consideration of input length is of great importance. is widely believed that longer input lengths have the potential to generate superior results. However, in practice, certain algorithms might fall short in effectively utilizing long input signals due to overfitting issues either. Here, we conducted long-term forecasting experiments by comparing our approach with the best reported values from different baseline papers. This was done to avoid any biases stemming from selective tuning of bascline parameters. While some may argue in favor of fairer comparison using fixed input length,
32

---

Table 27: Full results for the anomaly detection.
Methods
SM
MSI. R
SMAP
SWAIT
PSM
Avg Fl
Metrics
R
F1
P
F1
P
R
F1
R
F1
R
F1
%
(PT(6)
88.89
84.95
86.89
$2.00
82.91
82.45
90.60
60.95
72.88
92.20
96.34
94.23
98.62
95.68
97.13
86.72
TimesNee"
90.14
97.34
PatchIST
87.26
92.14
84.62
70.96
78.70
90.64
80.94
35.72
93.47
96.08
ETSGemer
87.44
79.23
83:13
$5.15
85.03
92:25
$5.75
69.50
90.02
80.
84.91
99.31
85.28
91.76
82.87
FEDformer
87.95
82.39
85.08
77.14
90.07
78.57
90.57
58.10
70.76
90.17
96.42
93.19
97.31
97.16
97:23
84.97
LighTTS
87.10
78.42
82.53
82.40
75.78
78.95
92. 5s
55.27
69.21
91.9
94.72
93.33
98.37
95.97
97.35
84.23
DLinear
83.62
71.52
77.10
64.34
85.43
84.85
92.
55.41
69.26
80.91
95. 35.30
$7.52
98.28
89.26
93.55
82.46
Stationayy
89.33
81.21
34.62
08.55
89.14
77.50
89.37
59.00
71.09
68.03
96.75
79.83
97.82
96.76
97.29
82.08
Autoformer
88.06
82.35
85.11
77.27
80.92
79.05
90.40
58.62
71.13
89.85
95.81
92.74
99.08
88.15
93.29
84.26
Pyraformer
85.61
50.61
83.04
83.81
85,91
$4.86
92.54
57.71
71,09
$7,92
96.00
91.78
71,67
96.02
82.08
62.57
Transformer"
85.49
91.85
97.32
94.72
77.23
$1.62
91.77
34,95
90.11
57.12
69,92
70.29
96.75
81.43
96,33
77.30
75.33
Reformer
82.58
69.24
75.32
85.51
90.91
57.44
70.40
72,50
96.53
32.80
59.93
73.61
Log Transfommer
83.46
70.13
76.21
73.05
87.37
79.57
89. 15
57.50
1.599
69.97
68.67
97.32
80.52
63,06
98.00
76.74
76.60
Transtormer 83.58
76.13
79.56
71.57
87.37
78.08
89.37
57.12
09.70
04.84
96.53
80.37
62.75
96.36
76.07
70.88
We rependuce the results TimessNet by http://gituuh.con/uune Time Series Librrry
WW replace the joint critarion Anomily Transormer with muvestraciin for far comparison.
Table 28: Full results for the imputation task.
Methods
GPT2(3)
TincsNet
PaecTTT MSE
FTSmom
LighhT
Dlincar
FRDfomer
Stationary
Antoformer
Informer
Reformer
Muskc
Ratio
MSI
MAE
MSE
MAE
MAE
MSE
MAE
MSC MAB
MSC
MAB
MSC
MAB
MSC
MAB
MSE
MAB
MSC
MAB
MSG
MAB
12.5%
(U917
0J885
0.023
0.101
[.04] 0.130
0.096
0.220
0.093
0.206
0.0R0
0.193
0.052
0.166
0.032
0.119
0.046
0.144
0.063
0.180
0.042
0.146
ECL BTT1 BT7N 877 877
25%
0.022
0.096 0.023
0.101
0.064
0.135
0.096
0.229
0.093
0.206
0.080
0.193
0.052
0.166
0.032
0.119
0.016
0.144
0.063
0.150
0.012
0.146
37.5%
0.029
0.111 0.029
0.111 0J549
0.143
0.133
0.271
0.113
0.251
0.130
0.219
0.309
0.191
0:339
0.131
0,057
0.351
0.079
0.200
0.063
0.382
50%
(.040
0.128
0.036
0.124
0,55
0.151
0.186
0.323
0.134
0.255
0.132
0.248
0.089
0.218
0.047
0.145
0.067
0.174
0.001
0.218
0.083
0.308
Aog
0.028
0.105
0.027
0.107
0.047
0.160
0.120
0.253
0.104
0.218
0,093
0.206
0.062
0.177
0,036
0.126
0,051
0.150
0.158
0,055
0.166
12.5%
0.917
0.238
0,080
0.086
0.101
0.272
0.262
37.5%
(.022
0JW7
0.091
0.104
0.356
0.159
0.222
0.130
0:231
0.108
0.108
0.293
0.300
50%
0.025
0.095
0.066
0,08
0.034
0.110
0.323
0.421
0.080
0.174
0.247
0.156
0.276
0.030
0.108
0.035
0.119
0.200
0.333
0.309
Aug
(.021
0.084
0.022
0.088
0.029
0.102
0.208
0.327
0.046
0,151
0.096
0.208
0.101
0.215
0:026
0.059
0.029
0.108
0.156
0.292
0.157
0.280
12.5%
(UM43
0.140
0.067
0.159
0.093
0.201
0.126
0.263
0.240
0.345
0.151
0.267
0.077
0.100
0.060
0.165
0.074
0.182
0.114
0.234
0.074
0.194
25%
0.054
0.156
0.069
0.178
0,107
0.217
0.169
0.304
0.265
0,364
0.180
0.292
0.105
0.236
0.050
0.189
0.050
0.203
0.140
0.262
0.302
0.227
37.5%
0U072
0.180
0.084
0.196
0.120
0.250
0.220
0.347
0.296
0,382
0.215
0.318
0.134
0.255
0.1002
0.212
0.10)
0.222
0.114
0.253
0.335
01361
50%
0.107
0,216
0.102
0.215
0,141
0.248
0.293
0.402
0.334
0.404
0.257
0.347
0,165
0.299
0.133
0.240
0,137
0.248
0.215
0.325
0.179
0.298
Aug
0.201
0.214
0.279
0.245
12.5%
0.280
0,044
0.135
0,140
0.340
0.246
0.247
0.049
0.147
0.149
0.444
0.331
37.5%
0.061
0.147
0,522
0.131
0.067
0.166
0.465
0.126
0.257
0:276
0.187
0.304
0.056
0.15%
0.163
0.462
0.370
50%
0,069
0.158
0.060
0,162
0.073
0.174
0.602
0,572
0.136
0.268
0.1R1
0.299
0,230
0.341
0.065
0.170
0.068
0.173
0.369
0.472
0.316
0.419
Aug
0.948
0.141
0.049
0.146
0.065
0.163
0.367
0.436
0.119
0.250
0.142
0.259
0.163
0.279
0.003
0.152
0.055
0.156
0.335
0.451
0.234
0.352
12.5%
0.060
0.194
0.085
0.202
0.055
0. 1600
0. 196
0.321
10.002
0.229
0.093
0.214
0.107
0.237
0.009
0.210
0.069
0.210
0.218
0.336
0.190
0.308
25%
0J887
0.205
0.089
0.206
0.065
0.175
0.207
0.332
0.121
0.252
0.115
0.247
0.120
0.251
0.097
0.214
0.096
0.220
0.219
0.326
0.197
0.312
37.5%
01004
0.211
0.014
0.215
0.676
0.189
0.219
0.344
0.141
0.273
0.144
0.276
0.136
0.266
0.100
0.230
0.104
0.225
0:222
0.328
0.303
0.315
50%
0.101
0.220
0,100
0.221
0.091
0.206
0.235
0.357
0,160
0.293
0,175
0.305
0.158
0.284
0.108
0.228
0,113
0.239
0.228
0.331
0,210
0.319
Aug
0.225
0.313
Weatur
12.5%
0,028
0,052
0.031
0.14]
0.101
0.111
0.10
0.162
0.056
0.047
0,054
0.100
0:003
0.076
0.082
97.5%
0.033
0.060
0.857
0,855
0.058
0.180
0.058
0.121
0.057
0.117
0.107
0.229
0.033
0.062
0.030
0.000
0.111
0.040
0.001
50%
0.037
0.065
0.034
0.062
0.038
0.063
0.102
0.207
0.065
0.133
0.066
0.134
0.183
0.312
0.037
0.068
0.037
0.067
0.083
0.114
0.016
0.089
Avg
(U031
0.056
8.030
0.054
0.063
0. 144
0.076
0.171
0.055
0.117
0.052
0.110
0.099
0.203
0.0022
0.059
0.151
0.057
0:045
0.104
0.008
0.067
we are starting to shift our focus towards pursuing more accurate algorithms that possess enhanced capabilities for handling longer inputs. Instead of restricting the input length a fixed small value. is pragmatic to tune both the input length and model parameters based on performance, as it is often the primary concern in practical usage. Exploring the utilization of extremely long inputs, such as Chatgpt or LLM, is among our future research directions.
33

---

Table 29: Full results short-term forecasting.
Methods
GPT2(6)
TimesNet
PaccTST
N-HTS
N-BEATS
ETSkemer
LighTTS
DLinear
FEDlormer
Stationay
Autofomer
Informer
Reformet
Youly
13,418
MASE
3.015
2.9%6
3.019
3.045
3.043
4.487
3.109
4,283
3.048
3.078
3.134
3.418
3,800
1.088
Monthig Quarteriy
SMAPE
10.177
10,100
10,202
10.124
13.376
11,364
12.145
10.792
10,958
11.338
11,360
13,313
MASE
L.94
1.182
1.33
1.194
1.169
1.966
1.328
1.520
1,283
1.325
1.365
1,401
1,775
OWA
0,98
11890
0.921
0.899
0,886
1.502
1,000
1,106
0.988
0.981
1.012
1,027
1,252
SMAPE
12894
12,670
12.959
12.791
12.677
14.588
14,014
13.514
14,260
13.9177
13,958
14.062
20. 128
1.053
OWA
0.897
0.878
0.905
0.899
0.880
1.149
0.981
0.956
1.012
0.998
1.002
1,024
1.927
Orkera
SMAPE
4.900
4.891
4.952
5.061
4.925
7.267
15.880
6.709
4.954
6.302
5.485
24,460
32.491
MASE
3,228
3.302
3.347
3.216
3.391
5,240
11,434
4,953
3.264
4,064
3.865
20,960
33.355
Anerage
13.525
12.780
MASE
1.600
1.5K5
1.623
1.613
1.599
2.408
2.111
2.095
1.701
1.756
1.771
2.718
4,223
OWA
0
0.851
0.869
0.861
0.855
1.172
1.051
1.051
0.918
0.930
0.939
1,230
1,775
34


Reviews:
[{"review_id": "Dw5SBnjlm0", "reviewer": "Reviewer_qAFk", "summary": "The paper proposes to use pretrained Transformer for time series analysis. \nThey freeze the attention and FFN layers but fine-tune the position embeddings and add-norm modules to adapt the model to a given task. \nThis results demonstrate that this method leads to strong performances in time series analysis tasks. \nThe authors proved a theorem to explain why the text-based pretrained model could generalize well to time series analysis, by drawing insights from PCA.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents very interesting theoretical and empirical findings. Their extensive experiments show that a pretrained text-based transformer model could be adapted to time series analysis and achieve strong performances. I think the findings are worth presenting to the ML community. \n\nTheir theoretical analysis explains this by connecting self-attention mechanism with PCA, for which I have concerns/questions as expressed in Weakness. \n\nThe paper is overall clearly written.", "weaknesses": "I didn't give a high soundness score because the paper hasn't well justified the choice of tuning add-norm layers and positional embeddings. Line-117 claims that it \"is considered a standard practice\" but I am afraid it is not true. \nParameter-efficient adaptation has been a hot topic and there have been various kinds of methods, such as adding feed-forward adapters (Houlsby et al. ICML 2019) or attentional adapters (Zhao et al. EMNLP 2022), tuning bias terms (Zaken et al. ACL 2022), LoRA (Hu et al. ICLR 2022), and prefix tuning (Li and Liang ACL 2021, Qin and Eisner NAACL 2021). \nThere hasn't been a \"standard\". \n\nI think the paper can be improved with experimental analysis on different kinds of LM adaptation methods, which will complement the current results. \n\nMore important, the choice of tuning parameters is related to the theorem in the paper: the PCA insights are drawn for the self-attention mechanism, making me wonder: does the choice of tuning parameters lead to the analysis of the self-attention? Or does the analysis of self-attention lead to the choice of tuning parameters? Or neither? Will results with any other adaptation methods give you different kinds of interpretation? \n\nThe mentioned references are: \n\nhttps://arxiv.org/abs/1902.00751 (already cited)\n\nhttps://arxiv.org/abs/2211.01979 \n\nhttps://arxiv.org/abs/2106.10199\n\nhttps://openreview.net/forum?id=nZeVKeeFYf9\n\nhttps://arxiv.org/abs/2101.00190\n\nhttps://aclanthology.org/2021.naacl-main.410/\n\nAnother line of work that this paper should discuss is the NLP literature that also finds the \"per-layer high cosine\" phenomenon. \n\nThis phenomenon is first discussed for skip-gram embeddings: https://aclanthology.org/D17-1308.pdf \n\nThen it is also found in Transformers: https://arxiv.org/pdf/1909.00512.pdf \n\nThere is also argument about why cosine metric may not mean that much: https://aclanthology.org/2022.acl-short.45/\n\nThe above is to only name a few and there are other papers that one can find by tracing their citation relations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer qAFk for the favorable evaluation of our work's theoretical and empirical contributions. It's particularly encouraging to note your belief that the findings merit presentation to the ML community. Such endorsements validate and inspire our continued dedication to this research. We deeply appreciate your detailed and perceptive feedback, and we are committed to addressing your concerns.\n\n***Q1 for Reviewer qAFk. the choice of tuning add-norm layers and positional embeddings and following PEFT studies***\n\nWe agree with the Reviewer that efforts like tuning normalization layers, position embedding, and parameter-efficient fine tuning can further improve performance. \nIn fact, we have conducted experiments with PEFT and observed the improvement. However, we chose not to include these findings in our current work in order to highlight the core contribution of the cross-module knowledge transfer to time series. We focus on demonstrating that pre-trained language models can yield strong performance in general time series analysis with minimal finetuning, with cross-module knowledge transfer as our central theme. To support this, we have conducted extensive experiments with frozen FFN and attention layers, using pre-trained parameters mixed with random values to show that the pre-trained-transformer block is vital. While our initial experiment with PEFT shows encouraging results, it introduces new learnable parameters and layers that do not fully align with our main argument, and thus will be left for the future examination.\n\n | Weather                | 96 (MSE) | 192 (MSE) | 336 (MSE) | 720 (MSE) |\n | ---------------------- | -------- | --------- | --------- | --------- |\n | GPT-2 (6)              | 0.162    | 0.204     | 0.254     | 0.326     |\n | PatchTST               | 0.149    | 0.194     | 0.245     | 0.314     |\n | GPT-2 (6) + Adapter    | 0.147    | 0.197     | 0.243     | 0.313     |\n | GPT-2 (6) + NewAdapter | 0.143    | 0.188     | 0.239     | 0.310     |\n\n\n***Q2 for Reviewer qAFk. The PCA insights are drawn for the self-attention mechanism, making me wonder: does the choice of tuning parameters lead to the analysis of the self-attention? Or does the analysis of self-attention lead to the choice of tuning parameters? Or neither? Will results with any other adaptation methods give you different kinds of interpretation?***\n\nOn the high level, we aim to test with minimal tuning whether NLP pre-trained model parameters can be transferred to time series analysis, which was initially inspired by our empirical studies and further supported our analysis that self-attention can essentially deliver the general purposed function of PCA. \n\n***Q3 for Reviewer qAFk. Another line of work that this paper should discuss is the NLP literature that also finds the \"per-layer high cosine\" phenomenon.***\n\nThank you for the invaluable suggestion! We will incorporate the literatures on \"per-layer high cosine\" in the revised version."}, {"review_id": "7W9kFpjWRP", "reviewer": "Reviewer_GLop", "summary": "\nThe paper presents a unified framework for time-series tasks using pre-trained language models. The author make a united model through a fine-tuning approach that focuses on fine-tuning specific parts of the pre-trained language model, rather than the entire set of parameters. As a result of this adaptation, the proposed fine-tuned model achieves comparable performance to previous methods across various datasets. Experimental results also demonstrate the versatility of the fine-tuned model by successfully transferring knowledge from pre-trained datasets, such as images and text. \n\n", "questions": "* Why did the proposed approach fail to demonstrate good performance in the zero-shot task, unlike in the other tasks?", "limitations": "yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed framework surpasses existing methods.\n* The paper is easily comprehensible and straightforward.\n* The evaluation of various time-series tasks highlights the advantages of the proposed approach in numerous cases.\n", "weaknesses": "* Novelty: The proposed fine-tuning approach appears to be more of an incremental improvement. Fine-tuning pre-trained language models is not entirely new. \n* Concerns regarding practicality: Does the architecture based on pre-trained language models require more computational resources compared to other models? It would be helpful to analyze the computation cost for inference in fair comparison settings, such as using the same number of parameters, to demonstrate the effectiveness of the proposed approach.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate Reviewer GLop's positive acknowledgment of our methodology's efficacy and numerical performance. We are especially grateful for the detailed and insightful feedback provided. Rest assured, we are dedicated to addressing your concerns and enhancing our work.\n\n***Q1 for Reviwer GLop. Novelty: The proposed fine-tuning approach appears to be more of an incremental improvement. Fine-tuning pre-trained language models is not entirely new.***\n\nEvidently, fine-tuning pre-trained language models for in-modality transfer learning is not a new concept and has been widely adopted in various domains. However, the key novelty of our work is cross-modality knowledge transfer showing that a pre-trained language model learned from texts can be successfully used for time series analysis with most of its parameters frozen. Compared to texts, time series data is more noisy and diverse, and many time series data is application dependent, which makes the cross-modality transferring learning challenging. In this paper, we support this claim both by empirical studies and the analysis of self-attention. We believe that such a transfer is \"mildly shocking\", as Reviewer kdgy highlighted, as there has been no previous work demonstrating that such a transfer is possible, let alone achieving state-of-the-art performance in all downstream time-series analysis tasks. Furthermore, current studies are limited to a relatively simple LM (GPT-2), we envision that a more complicated LM, such as LLAMA, can lead to more improvements in time series analysis.\n\n***Q2 for Reviwer GLop. Concerns regarding practicality: Does the architecture based on pre-trained language models require more computational resources compared to other models? It would be helpful to analyze the computation cost for inference in fair comparison settings, such as using the same number of parameters, to demonstrate the effectiveness of the proposed approach.***\n\nWe highly agree that analysis of computational cost is helpful for investigating the practicality of the LLM-based model. The results can be found in the table below. Each baseline model comes in two variants, featuring model hidden dimensions of 32 and 768, which align with GPT-2's specifications. Furthermore, the majority of the baseline models consist of three layers. We assessed the computational cost using a batch from ETTh2 (with a batch size of 128) on a 32G V100 GPU. \n\nThe results indicate that GPT-2(3) has substantially enhanced time efficiency and reduced parameter quantity compared to baselines with the same model dimension. This was a surprise since we initially anticipated that this large language model might be slower. However, we surmise that the efficient optimization of huggingface's GPT model implementation primarily accounts for such a significant improvement in time costs. Furthermore, GPT-2(3) and GPT-2(6) demonstrate a mere 6.12\\% and 4.60\\% proportion of learnable parameters among the overall parameter size, respectively.\n\n | Model         | Training Params | Training Params Percentage(%) | Training Time for 1 Batch (s) | Inference Time for 1 Batch (s) |\n | ------------- | --------------- | ----------------------------- | ----------------------------- | ------------------------------ |\n | FEDformer-32  | 437,319         | 100                           | 0.889                         | 0.17                           |\n | TimesNet-32   | 1,905,015       | 100                           | 0.747                         | 0.302                          |\n | PatchTST-32   | 543,232         | 100                           | 0.043                         | 0.022                          |\n | ------------- | --------------- | ----------------------------- | ----------------------------- | ------------------------------ |\n | FEDformer-768 | 33,105,415      | 100                           | 0.208                         | 0.056                          |\n | TimesNet-768  | 42,358,519      | 100                           | 5.723                         | 2.162                          |\n | PatchTST-768  | 19,677,024      | 100                           | 0.457                         | 0.123                          |\n | GPT-2(3)-768  | 3,906,912       | 6.12                          | 0.093                         | 0.032                          |\n | ------------- | --------------- | ----------------------------- | ----------------------------- | ------------------------------ |\n | GPT-2(6)-768  | 3,916,128       | 4.60                          | 0.104                         | 0.054                          |\n\n***Q3 for Reviwer GLop. Why did the proposed approach fail to demonstrate good performance in the zero-shot task, unlike in the other tasks?***\n\nFor the zero-shot tasks, our goal is to verify the representation power of LLMs for time series analysis, and thus focusing on the comparison with a few recently proposed algorithms, such as DLinear, PatchTST and TimesNet. Our empirical studies show that GPT-2(6) yields similar performance as the above state-of-the-art methods designed for time series analysis. However, none of these methods is specially designed for zero-shot learning. In contrast, N-BEATS, as noted in [1,2], has an unique model design (e.g. backcasting and ensemble learning) that enables domain adaption without having to modify its weights, making it particularly suitable for zero-shot learning. That explains why N-BEATS clearly outperforms other competitors in zero-shot learning. \n\n[1] Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M., TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis, ICLR, 2023.\n\n[2] Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y, N-beats: Neural basis expansion analysis for interpretable time series forecasting, arXiv:1905.10437, 2019."}, {"review_id": "qZbcGLL0a7", "reviewer": "Reviewer_kdgy", "summary": "The authors discuss a method by which a deep transformer model pre-trained on an NLP or CV task can be adapted to a wide variety of applications involving classification or prediction of scalar time series data. This includes an embedding scheme to represent the time series input in the same embedding space as expected by the transformer layers and a fine-tuning approach whereby the multi-head attention and feedforward components of the transformer blocks are frozen, but the embedding and layer norm components are trained. Using the GPT-2 backbone, they apply this method to classification, anomaly detection, imputation, and various forecasting tasks, showing generally good performance compared to many competitors. They", "questions": "Could the authors comment on the discrepancy between Informer performance reported in their Table 13 vs. in [2]? What could explain the substantial difference in metrics on the same task? I certainly don’t mean to claim that [2] must be right, but the size of the difference is enough to affect the interpretation of the results, so it merits some discussion. [Addressed in discussion period]\n\nCould the authors share their perspective on *why* their FPT approach seems to work so well, and what are the main drivers of this result? \n\nWhat unique insights for time series representation are provided by the PCA analysis in Section 7? \n\nWhy not report the results in Figure 1 as a table? They would be much easier to read and compare.", "limitations": "There is no direct potential for negative social impact. Potential limitations for the method and results are either discussed or covered above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem of finding a broadly performant time series model is interesting and challenging, particularly in light of the diverse architectures proposed and mixed results obtained by deep neural networks for time series applications. The idea of simply using a pre-trained language model with minimal adaptation is surprising and worth discussion in the scientific community. The empirical analysis attempts to cover a very broad range of tasks and datasets, which is an appropriately high bar for a paper that proposes a “one fits all” approach to time series analysis.", "weaknesses": "* The time series applications I know best here are anomaly detection and long-range forecasting, and for both of these settings there are some issues with the reported results. \n     - For anomaly detection, there is significant information missing. How are binary anomaly decisions made from the reconstruction error? How are precision and recall computed when anomalies are given by windows rather than points (e.g. SMAP, MSL)? These decisions can have a strong influence on the reported results.\n     - While comparisons are extensive for anomaly detection, they do not seem to actually capture the state of the art. For example, the OmniAnomaly method [1], which also naturally generalizes to multivariate time series, significantly outperforms the GPT2(6) model in F1 score on all datasets for which both are evaluated (SMD, MSL, SMAP).\n    - The long-range forecasting results are likewise missing comparisons to methods that significantly outperform all reported scores. For example, the S4 model [2] reports lower MSE and MAE on Weather, ETTh1, ETTh2, and ETTm1 (see Table 13, [2]).\n    - Moreover, the results reported for Informer in [2] are significantly better than what is reported in Table 13 of the present paper, beating GPT2(6) on ETTh1 and ETTh2. It is unclear what the source of this discrepancy is, but it is large enough to be relevant when drawing conclusions across methods.\n\n* The idea that a lightly-adapted NLP or CV transformer backbone is a highly competitive, highly general time series model is mildly shocking, at least to a time series practitioner. This paper largely misses the opportunity to explore *why* this might be the case, and *what* are the main drivers of this result. For example, do the pre-trained weights actually matter, or is it the architecture? Is fine-tuning required, and if so, why is fine-tuning only the embeddings and layer norm components sufficient? As it stands, the paper communicates an interesting empirical result without providing much insight as to its explanation.\n\n* The specific adaptations of the FPT approach - i.e. the time series encoder and fine-tuning protocol - are both inadequately described. The encoder operations should be described in mathematical detail. The algorithmic details of fine-tuning should at least be provided in the supplement.\n\n* The PCA analysis seems largely unrelated to the preceding work in the paper. It seems to be a general observation about self-attention, without any particular connection to the specific time series applications considered. Moreover, PCA itself is certainly not a one-size-fits-all solution to time series modeling, so it cannot be the basis for a convincing explanation of the model’s success.\n\n[1] Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., & Pei, D. (2019). Robust anomaly detection for multivariate time series through stochastic recurrent neural network. KDD.\n\n[2] Gu, A., Goel, K., & Re, C. (2021). Efficiently modeling long sequences with structured state spaces. ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer kdgy for the positive evaluation of our work's intriguing and challenging contributions. We're deeply heartened by the sentiment that \"The idea of simply using a pre-trained language model with minimal adaptation is surprising and worth discussion in the scientific community.\" Such feedback not only affirms our efforts but also motivates us to further our research in this domain. Thanks for your detailed and insightful comments to help us improve our work. We hope your concerns will be addressed.\n\n***Q1:Anomaly detection.binary anomaly decisions, precision and recall computed method***\n\nTo make a fair comparison, we mainly follow the binary anomaly decision methods from TimesNet[2] and using the same window size as 100 throughout the main text. Note that TimesNet[2] have performed an extensive comparison of different SOTA anomaly detection methods in the literature in this setting, which makes our empiricial studies comparable to a wide range of literatures. \nMore specifically, we focus on unsupervised time series anomaly detection. Experimentally, each dataset\nincludes training, validation and testing subsets. Anomalies are only labeled in the testing subset.\nWe select the hyper-parameters following the Gap Statistic method (Tibshirani et al., 2001) in K-Means as described below:\n\n• After the training phase, we apply the model to the validation subset (without label) and obtain the anomaly scores(reconstruction loss) of all time points.\n\n• We count the frequency of the anomaly scores in the validation subset. It is observed that the distribution of anomaly scores is separated into two clusters. We find that the cluster with a larger anomaly score contains r time points. And for our model, r is closed to 0.1\\%, 0.5\\%, 1\\% for SWaT, SMD and other datasets, respectively .\n\nNote that, directly setting the $\\delta$ is also feasible. we can fix the $\\delta$ as 0.1 for the SMD, MSL and SWaT datasets, 0.01 for the SMAP and PSM datasets, which yield a quite close performance to setting r.\n\nWe understand that more complicated strategies on those issues can further improve the detection accuracy. We did not explore those choices as it departs from the main theme of this work, i.e. pointing out the possibility of leveraging frozen LM for various time series analysis tasks.  \n\n***Q2:Anomaly detection. The state of the art problem. the OmniAnomaly method [1], which also naturally generalizes to multivariate time series, significantly outperforms the GPT2(6) model in F1 score on all datasets for which both are evaluated (SMD, MSL, SMAP).***\n\nOur approach is based on Anomaly Transformer [4] by substituting the joint criterion with the reconstruction error. \n\nIn the upcoming table, it is evident that integrating an additional association discrepancy via a learnable Gaussian kernel significantly bolsters the Anomaly Transformer's performance. Yet, this introduces a quandary, as the Gaussian kernel doesn't impact the primary forecasting output or the reconstruction loss. To genuinely assess the inherent capabilities of each backbone algorithm—primarily forecasting methods suited for broad time series analysis—it appears more reasonable to focus exclusively on the reconstruction error. Gven the results reported in previous study [4], which indicated that the Anomaly Transformer outperforms OmniAnomaly, we chose to exclude OmniAnomaly from our comparison. Furthermore, as per Table 1 in reference [4], OmniAnomaly exhibits a comparatively weaker performance than GPT2(6), primarily on SMD, SWaT, and PSM datasets.\n\n | Methods | GPT2(6) | TimesNet | Anomaly.* | Anomaly | OmniAnomaly |\n | --------| -------- | --------- | --------- | --------- |--------- |\n | SMD  | 86.89    | 84.61     | 85.49     | 92.33     | 85.22     |\n | MSL  | 82.45    | 81.84     | 83.31     | 93.59     | 87.67      |\n | SMAP | 72.88    | 69.39    | 71.18     | 96.69     | 86.92     |\n | SWaT | 94.23   | 93.02     | 83.10    | 94.07     | 82.83     |\n | PSM  | 97.13    | 97.34     | 79.40     | 97.89     | 80.83     |\n | Average | 86.72    | 85.24     | 80.50     | 94.91     | 84.69     |\n* We replace the joint criterion in Anomaly Transformer (2021) with reconstruction error for a fair comparison.\n\n***Q3: Long-term forecasting. S4 comparison and discrepancy***\n\nFirst, the discrepancy in the reported results for Informer between [2] and our work is due to the fact that results in [2] are on univariate forecasting, whereas our study focused on multivariate forecasting. It is well recognized in previous studies [3, 5] that multi-variate forecasting is considerably more challenging than univariate forecasting as it aims to make predictions for multiple channels by a single model. Thus, most recent studies of time series analysis, e.g. [3, 5], shift to multi-variate forecasting. Without exception, our study follows the same trend, and in fact, most of the results for baseline methods come from [3]. This also explains the concern from the reviewer that in [2] S4 reports lower MSE and MAE for several datasets -- again, these numbers are based on univariate forecasting, not mult-variate forecasting. Our empirical studies did reveal significantly better results for univariate forecasting, which will be included in the appendix for the final version. The performance of S4 in univariate forecasting is markedly inferior to that of FedFormer, AutoFormer, and FiLM, as detailed in Table 10 of the FiLM appendix. \n\n***Q4,5,6:In the global response***\n\n[3] Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M., TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis, ICLR, 2023.\n\n[4] Xu, J., Wu, H., Wang, J., and Long, M, Anomaly transformer: Time series anomaly detection with association discrepancy, ICLR, 2022.\n\n[5] Zhang, T., Zhang, Y., Cao, W., Bian, J., Yi, X., Zheng, S., and Li, J., Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures"}, {"review_id": "GtobxUerjP", "reviewer": "Reviewer_idRZ", "summary": "This paper shows that fine-tuning language models along with some task specific layers, e.g. input and position embedding and lay norms, can achieve comparable and sate-of-the-art performance on various time series tasks, including forecasting, imputation, anomaly detection and classification tasks. ", "questions": "See Weaknesses.\n\nMissing references:\n\n(1) LogTransformer was proposed in [1]. Results of LogTransformer was in this paper but [1] was not cited.\n(2) Using pre-trained transformers to model time series is not a new idea. [2] uses pre-train LMs to conduct time series forecasting and [3] uses Vision Transformer to model irregularly sampled time series. However, this paper does not discussion this two work and add them as baselines in appropriate tasks.\n\n[1] Li et al. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. NeurIPS 2019.\n[2] Xue and Salim. PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting. \n[3] Li et al. Time Series as Images: Vision Transformer for Irregularly Sampled Time Series. ", "limitations": "See Weaknesses and Questions section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The method for time series analysis is simple and can work for different time series analysis tasks.\n\n(2) Results are comparable or state-of-the-art in different benchmark datasets and time series analysis tasks.\n", "weaknesses": "(1) This work mainly conduct experiments on simple benchmark datasets. Many real-word datasets have complicated dynamics and large noise and the prior knowledge within pre-train models do not necessarily align with such dynamics. If the success of using pre-train models are due to the prior knowledge pre-train models own, what are results if the time series data does not align with such prior knowledge? This paper seems not to have deep discussion regarding this matter.\n(2) It is unclear why GPT-2 (6) perform better than the full-layer fine-tuning baseline. Deep layers often tend to learn global information within the data. It is not clear why authors use GPT-2 (6). Authors lack of deep discussion regarding this matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer dRZ for the favorable evaluation of our methodology's simplicity and effectiveness, as well as its state-of-the-art performance across diverse time series tasks. We deeply appreciate your detailed and perceptive feedback. Rest assured, we are committed to addressing and resolving your concerns.\n\n***Q1 for Reviewer idRZ. This work mainly conduct experiments on simple benchmark datasets. Many real-word datasets have complicated dynamics and large noise and the prior knowledge within pre-train models do not necessarily align with such dynamics. If the success of using pre-train models are due to the prior knowledge pre-train models own, what are results if the time series data does not align with such prior knowledge? This paper seems not to have deep discussion regarding this matter.***\n\nWe concur wholeheartedly with the reviewer that the absence of a comprehensive and large-scale benchmark dataset—akin to ImageNet in computer vision—is a substantial challenge for the whole community. We eagerly anticipate future advancements that will offer a more extensive and diverse benchmark suite for researchers.\n\nHowever, we argue that we have included most complicated benchmark datasets for time series forecasting in our study, e.g. the M4 dataset with 100,000 time series that served as the foundation for the fourth Makridakis forecasting competition—a competition widely recognized as one of the most representative in time series forecasting. The benchmark datasets used in our study have also been used by 400 papers in the last two years, with 20 papers from premier conferences, according to Semantic Scholar analysis. Although it is impossible for these datasets to showcase all the challenges faced by real applications of time series analysis, they do highlight a wide range of key challenges from real applications. \n\nWe further argue that both LM and time series analysis is based on the idea of auto-regression. It is this high-level similarity that inspires us to explore frozen LM for time series analysis. When encountering a mismatch in prior assumptions between model and datasets, appropriate fine-tuning technology can always be used to fill out the gap. \n\n***Q2 for Reviewer idRZ. It is unclear why GPT-2 (6) perform better than the full-layer fine-tuning baseline. Deep layers often tend to learn global information within the data. It is not clear why authors use GPT-2 (6). Authors lack of deep discussion regarding this matter.***\n\nOne reason for GPT-2(6) to perform better than full layer fine-tuning baseline is the rank collapsing property of transformer. In other words, as we go into deeper layers, we tend to see that more and more tokens exhibit similar vectors. Since most time series analyses depend on a relatively shorter history compared to texts, it further amplifies the impact of the rank-collapsing property. As a result, important detailed information may be removed from the outputs from the deep layers, limiting the prediction accuracy. \n\n***Q3 for Reviewer idRZ. LogTransformer was proposed in [1]. Results of LogTransformer was in this paper but [1] was not cited.***\n\nThanks for the reminder, we will add the reference of LogTransformer. \n\n***Q4 for Reviewer idRZ. Using pre-trained transformers to model time series is not a new idea. [2] uses pre-train LMs to conduct time series forecasting and [3] uses Vision Transformer to model irregularly sampled time series. However, this paper does not discussion this two work and add them as baselines in appropriate tasks.***\n\nAlthough on the very high level, using pre-trained transformer for time series analysis is not completely new, our study is clearly new and novel: we show it is possible to directly use a frozen LM to achieve the state-of-the-art performance for time series analysis. Compared to texts, time series data is more noisy and diverse, and many time series data is application dependent, which makes the cross-modality transferring learning challenging. We support this claim by both extensive empirical studies and theoretical analysis. \n\nBoth papers the reviewer mentioned are interesting, and we intend to incorporate them into the related works section. However, we would like to clarify that both works are completely different and do not explore the exact same problem as us. Specifically, [2] directly uses pre-trained LMs for time series forecasting through prompt engineering, its performance for multi-step forecasting is not close to the state-of-the-art. And its emphasis is the prompt engineering. In addition, the prompt-based forecasting is limited and cannot be applied in many real-world applications. [3] leverages the VIT structure for time series analysis, and does not address the main theme of this work, i.e. cross modality transferring -- directly use frozen LM for time series analysis. \n"}, {"review_id": "ERaHp68t22", "reviewer": "Reviewer_qj2Q", "summary": "In this paper, the authors suggest leveraging pre-trained large language models, specifically utilizing a frozen Transformer backbone, for comprehensive time series analysis. The unique method involves fine-tuning only the input embeddings of time series data and the output layers. The authors substantiate their proposal with large-scale experiments across multiple benchmarks, demonstrating that their approach consistently outperforms various established baseline methods in diverse time series analysis tasks.", "questions": "N/A", "limitations": "N/A", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. The paper's proposed method impresses with its simplicity, efficacy, and clear motivation. Despite not introducing any complex modules or additional hyperparameters to the existing model, it consistently outperforms more intricate techniques.\n\n2. The experiments undertaken within the paper are robust and comprehensive. The authors have tested the proposed method across a broad spectrum of time series analysis aspects, such as long-term and short-term forecasting, and abnormality detection. In each scenario, the method demonstrates superior performance when compared with multiple strong baselines.\n\n3. The insightful correlation drawn between self-attention and Principal Component Analysis (PCA) elucidates why the proposed method performs remarkably well. This strengthens the credibility of the results, making them more convincing.", "weaknesses": "1. The paper neglects to detail the computational cost differences between the proposed method and alternative approaches. Given that a pre-trained GPT-2 backbone is potentially substantial in size, the computational expense of the presented method might significantly outweigh that of other baseline methods.\n\n2. Although the proposed method seems compatible with various autoregressive Transformers, the experiments rely exclusively on GPT-2. It would be beneficial to extend the experimental scope to encompass additional models such as DeBERTa, GPT-J, OPT, among others.\n\n3. The paper's overall presentation requires improvement. Tables appear congested, and font sizes within figures may be too small for legible printing. Additionally, there is an inconsistent distribution of space on some pages, and many section titles are overly lengthy. This compromises readability and detracts from the overall impact of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to express our sincere gratitude to Reviewer qj2Q for the positive assessment of our work. Your endorsement is both affirming and motivating, and it strengthens our resolve to continue our research in this field. We concur with Reviewer qj2Q's observation that the presentation of our paper could be enhanced. Specifically, we acknowledge that the tables appear cluttered and that the font sizes may be too small. This issue arose as we endeavored to comprehensively address a wide range of topics within the confines of this nine-page paper. Our goal was to deliver an exhaustive numerical analysis and provide a clear exposition of our knowledge transfer approach.\n\nWe recognize that wading through all the supporting details can be challenging for reviewers, and we lament that space limitations forced us to omit some essential information in our supplementary material. It's noteworthy that our full paper, inclusive of all supporting information, spans over 30 pages. Nevertheless, we value your feedback and will make every effort to incorporate your suggestions in our revisions.\n\n***Q1 for Reviewer qj2Q. The paper neglects to detail the computational cost differences between the proposed method and alternative approaches. Given that a pre-trained GPT-2 backbone is potentially substantial in size, the computational expense of the presented method might significantly outweigh that of other baseline methods.***\n\nThank you for pointing that out. We concur that evaluating the computational cost is crucial, especially for large models like the one under consideration. The subsequent table presents the results.\n\nEach baseline model is offered in two configurations, with model dimensions of 32 and 768 (analogous to GPT-2). Additionally, each baseline model consists of three layers. We assessed the computational expenses using a single batch of ETTh2 (with a batch size of 128) on a solitary V100 GPU. The results indicate that GPT-2(3) offers a marked enhancement in both time efficiency and parameter count relative to the baselines with equivalent model dimensions. This substantial uptick in time efficiency can be primarily attributed to the proficient optimization techniques employed by huggingface. Furthermore, the training parameters constitute only 6.12\\% for GPT-2(3) and 4.60\\% for GPT-2(6).\n\n | Model         | Training Params | Training Params Percentage(%) | Training Time for 1 Batch (s) | Inference Time for 1 Batch (s) |\n | ------------- | --------------- | ----------------------------- | ----------------------------- | ------------------------------ |\n | FEDformer-32  | 437,319         | 100                           | 0.889                         | 0.17                           |\n | TimesNet-32   | 1,905,015       | 100                           | 0.747                         | 0.302                          |\n | PatchTST-32   | 543,232         | 100                           | 0.043                         | 0.022                          |\n | ------------- | --------------- | ----------------------------- | ----------------------------- | ------------------------------ |\n | FEDformer-768 | 33,105,415      | 100                           | 0.208                         | 0.056                          |\n | TimesNet-768  | 42,358,519      | 100                           | 5.723                         | 2.162                          |\n | PatchTST-768  | 19,677,024      | 100                           | 0.457                         | 0.123                          |\n | GPT-2(3)-768  | 3,906,912       | 6.12                          | 0.093                         | 0.032                          |\n | ------------- | --------------- | ----------------------------- | ----------------------------- | ------------------------------ |\n | GPT-2(6)-768  | 3,916,128       | 4.60                          | 0.104                         | 0.054                          |\n\n***Q2 for Reviewer qj2Q. Although the proposed method seems compatible with various autoregressive Transformers, the experiments rely exclusively on GPT-2. It would be beneficial to extend the experimental scope to encompass additional models such as DeBERTa, GPT-J, OPT, among others.***\n\nWe wholeheartedly concur that broadening our experimental range to include more models would augment the depth of our study. However, given the significant effort and resources required to conduct a wide array of experiments, we primarily relied on GPT-2 for our core investigations. Nevertheless, we have also utilized the CV pre-trained model BEiT, and the NLP pre-trained model BERT. Both were trained on 5\\% of ETTh2 and 5\\% of ETTm2, underscoring that the capability for knowledge transfer isn't exclusive to GPT-2. Detailed results related to these models are available in Appendix H.5. As we move forward, our research will delve into the performance implications of transferring pre-trained models from diverse modalities to time series analysis.\n\n\n***Q3 for Reviewer qj2Q. The paper's overall presentation requires improvement. Tables appear congested, and font sizes within figures may be too small for legible printing. Additionally, there is an inconsistent distribution of space on some pages, and many section titles are overly lengthy. This compromises readability and detracts from the overall impact of the findings.***\n\nThank you for your feedback. Owing to page constraints, we had to condense our figures and tables, inadvertently affecting their readability. In subsequent versions, we will refine the paper's layout, ensuring legible font sizes and optimal paragraph spacing.\n"}]

Author rebuttal:
We thank the Reviewers for the insightful comments and detailed feedback. We were delighted that reviewers find our paper has the following advantages: 

**Innovative Findings**: Using pretrained transformers for time series.(qAFk, kdgy, qj2Q) 

**Clear Writing:** Easily comprehensible content.(qAFk, GLop, qj2Q)

**Robust Analysis**: Wide-ranging experiments across tasks/datasets.(qAFk, GLop, kdgy, qj2Q)

**Simplicity & Top-tier Results**: Outperforms complex techniques.(GLop, idRZ, qj2Q)

**Attention & PCA Insight**: Explains model's success.(qAFk, qj2Q)


We've addressed certain concerns in individual rebuttals. However, owing to space limitations and the extent of questions from Reviewer kdgy, we've moved some responses to this global reply section.

***Q4 for Reviewer kdgy. The specific adaptations of the FPT approach - i.e. the time series encoder and fine-tuning protocol - are both inadequately described. The encoder operations should be described in mathematical detail. The algorithmic details of fine-tuning should at least be provided in the supplement.***

We use the GPT-2 model [6] as our time series encoder and the detailed mathematical detail and architecture can be found in [6, 7]. Thus, with the page limit, we cannot go into further detail on this in the paper. In the fine-tuning stage, we freeze certain parameters of GPT-2 by modifying the $requires grad$ to false in PyTorch as shown in our provided code. We will provide the pseudocode of GTP-2 method in the appendix for the final version.

***Q5 for Reviewer kdgy. Could the authors comment on the discrepancy between Informer performance reported in their Table 13 vs. in [2]? What could explain the substantial difference in metrics on the same task? I certainly don’t mean to claim that [2] must be right, but the size of the difference is enough to affect the interpretation of the results, so it merits some discussion.***

The tasks referenced in Table 13 and [2] are distinct. [2] exclusively reports results for univariate forecasting, whereas our experiments focus on multivariate long-term forecasting, leading to a noticeable disparity in performance. Our experiment settings and baseline citations are consistent with previous works, such as [3]. We will provide clearer details regarding our experimental procedures in the revised version.

***Q6 for Reviewer kdgy. This paper largely misses the opportunity to explore why this might be the case, and what are the main drivers of this result. For example, do the pre-trained weights actually matter, or is it the architecture? Is fine-tuning required, and if so, why is fine-tuning only the embeddings and layer norm components sufficient? As it stands, the paper communicates an interesting empirical result without providing much insight as to its explanation. Could the authors share their perspective on why their FPT approach seems to work so well, and what are the main drivers of this result? What unique insights for time series representation are provided by the PCA analysis in Section 7?***

We have run extensive experiments for various finetuning and we did not include them in the paper due to the space limitation. Our empirical studies show that the weights learned by LM are essential to the success of cross modality transferring. In fact, full fine tuning of all the weights leads to significant degradation in performance -- that is the reason why we freeze all the weights for FFN and attention layers and only tune embedding and layer norm. To explain why FPT works so well for time series analysis, we try to understand the function of self-attention layers and find that it may be closely related to PCA, a general-purpose function independent from any domain. This domain-independent property delivered by attention layers makes us believe in the possibility of using a frozen LM for time series. 

***Q7 for Reviewer kdgy. Why not report the results in Figure 1 as a table? They would be much easier to read and compare.***

Thank you for your suggestion. Due to page constraints, we consolidated the classification task results into a figure, which admittedly affected readability. We plan to reorganize the layout and include a table displaying the results either in the main body or the appendix.

Moreover, our approach mirrored that of the baseline. Since the TimesNET paper presents classification outcomes in a comparable format, we felt it prudent to maintain this consistency to facilitate easier comparison.

 |          | XGBoost | Rocket | LSTNet | LSSL | TCN  | DLinear | LightTS | TimesNet |
 | -------- | ------- | ------ | ------ | ---- | ---- | ------- | ------- | -------- |
 | Accuracy | 66.0    | 72.5   | 71.8   | 70.9 | 70.3 | 67.5    | 70.4    | 73.6     |

 |          | Transformer | Reformer | Informer | Pyraf. | Autof. | Stationf. | FEDf. | ETSf. | Flowf. | GPT2(6) |
 | -------- | ----------- | -------- | -------- | ---------- | ---------- | ------------- | --------- | --------- | ---------- | ------- |
 | Accuracy | 71.9        | 71.5     | 72.1     | 70.8       | 71.1       | 72.7          | 70.7      | 71.0      | 73.0       | 74.0    |


[2] Gu, A., Goel, K., & Re, C. (2021). Efficiently modeling long sequences with structured state spaces. ICLR.

[3] Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M., TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis, ICLR, 2023.

[6] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I, Language models are unsupervised multitask learners, 2019.

[7] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser Lukasz, and Polosukhin, I, Attention is all you need, arXiv:1706.03762, 2017

Meta-review: Authors show that a pre-trained language model can be a good starting point for time series modeling, which seems strange when you think about it. They show results on several different datasets and achieve compelling results on them. They provide some understanding of why pertaining would help, through the relationship of self-attention to PCA. The reviews were generally quite positive. Several concerns raised by the reviewers seem to have been addressed by the authors, which resulted in two of them unweighting the original score.

Perhaps a couple of things that could have been done better were:
1. More in depth of exploration why pertaining with LM's would help on sequence modeling.
2. Looking into other ways of fine-tuning the pretrained models.

Nevertheless, the paper makes an interesting observational contribution, and nice results, as it stands. I think the community will find the results interesting.
Decision: Accept (spotlight)