Title: One Fits All: Power General Time Series Analysis by Pretrained LM
Venue: NeurIPS 2023
Accepted: yes
Keywords: general time series analysis

time series forecasting

cross modality knowledge transfer; pretrained language model;
Abstract: Although we have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where a unified model can be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre-trained model for time series analysis is the lack of a large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks, as illustrated in Figure1. We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and a crucial step towards understanding the universality of a pre-trained transformer. 
The code is publicly available at https://anonymous.4open.science/r/Pretrained-LM-for-TSForcasting-C561.

One Fits All: Power General Time Series Analysis by Pretrained LM

Abstract

Although we have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where a unified model can be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre-trained model for time series analysis is the lack of a large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks, as illustrated in Figure 1. We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and a crucial step towards understanding the universality of a pre-trained transformer. The code is publicly available at https://github.com/DAMO-DI-ML/One_Fits_All .

1 Introduction

Time series analysis is a fundamental problem Hyndman & Athanasopoulos (2021) that has played an important role in many real-world applications Wen et al. (2022), such as retail sales forecasting Böse et al. (2017); Courty & Li (1999) , imputation of missing data for economic time series Friedman (1962) anomaly detection for industrial maintenance Gao et al. (2020), and classification of time series from various domain Ismail Fawaz et al. (2019). Numerous statistical and machine learning methods have been developed for time series analysis in the past. Inspired by its great success in natural language processing and computer vision Vaswani et al. (2017); Devlin et al. (2019); Dosovitskiy et al. (2021); Rao et al. (2021), transformer has been introduced to various time series tasks with promising results Wen et al. (2023), especially for time series forecasting Lim et al. (2021); Zhou et al. (2022, 2021); Wu et al. (2021); Nie et al. (2022).

We have recently witnessed the rapid development of foundation models in NLP. The key idea is to pre-train a large language model from billions of tokens to facilitate model training for downstream tasks, particularly when we have a few, sometimes even zero, labeled instances. Another advantage of

GPT2-backbone

TimesNet

PatchTST

ETSformer

LightTS

DLinear

Stationary

FEDformer

Autoformer

Informer

Reformer

Long-term

Forecasting

Short-term

Forecasting

Anomaly Detection

Imputation

Classification

74.00

Zero-shot

Few-shot

87.00

0.04

0.30

16.00

11.50

0.40

20.50

0.22

76.00

65.00

2.10

34.00

2.20

1.40

16.50

0.14

80.00

69.00

1.30

26.00

foundation models is that they provide a unified framework for handling diverse tasks, which contrasts conventional wisdom where each task requires a specially designed algorithm. However, so far, little progress has been made to exploit pre-trained or foundation models for time series analysis. One main challenge is the lack of the large amount of data to train a foundation model for time series analysis. The largest data sets for time series analysis is less than 10GB Godahewa et al. (2021), which is much smaller than that for NLP. To address this challenge, we propose to leverage pre-trained language models for general time series analysis. Our approach provides a unified framework for diverse time series tasks, such as classification, anomaly detection, forecasting, and few-shot or zero-shot learning. As shown in Figure 1, using the same backbone, our approach performs either on-par or better than the state-of-the-art methods for all main time series analysis tasks. Besides extensive empirical studies, we also investigate why a transformer model pre-trained from the language domain can be adapted to time series analysis with almost no change. Our analysis indicates that the self-attention modules in the pre-trained transformer acquire the ability to perform certain non-data-dependent operations through training. These operations are closely linked to principal component analysis over the input patterns. We believe it is this generic function performed by the self-attention module that allows trained transformer models to be so-called universal compute engine Lu et al. (2022) or general computation calculator Giannou et al. (2023). We support our claims by conducting an empirical investigation of the resemblance in model behaviors when self-attention is substituted with PCA, and by providing a theoretical analysis of their correlation.

Here we summarize our key contributions as follows:

We propose a unified framework that uses a frozen pre-trained language model to achieve a SOTA or comparable performance in all major types of time series analysis tasks supported by thorough and extensive experiments, including time series classification, short/long-term forecasting, imputation, anomaly detection, few-shot and zero-sample forecasting.

We found, both theoretically and empirically, that self-attention performs a function similar to PCA, which helps explain the universality of transformer models.

We demonstrate the universality of our approach by exploring a pre-trained transformer model from another backbond model (BERT) or modality (computer vision) to power the time series forecasting.

The remainder of this paper is structured as follows. Section 2 briefly summarizes the related work. Section 3 presents the proposed detailed model structure. In Section 4, we conduct a thorough and extensive evaluation of the performance of cross-modality time series analysis using our proposed method in seven main time series analysis tasks compared to various SOTA baseline models. Section 5

presents various ablation studies, and Section 6 demonstrates the universality of our proposed method using pre-trained models with another structure or pre-trained from another modality. In Section 7, we provide a theoretical explanation of the connection between self-attention and PCA. Finally, in Section 8, we discuss our results and future directions. Due to space limit, more extensive discussion of related work, experimental results, and theoretical analysis are provided in the Appendix.

2 Related Work

In this section, we provide short reviews of literature in the areas of time series analysis, in-modality transfer learning, and cross-modality knowledge transfer learning. We postpone the discussion of works for end-to-end time series analysis to Appendix B, due to the limited space.

In-modality Transfer Learning through pre-trained models In recent years, a large number of research works have verified the effectiveness of the pre-trained model from NLP, CV to Visionand-Language (VL). Latest studies for NLP focus on learning contextual word embeddings for downstream tasks. With the increase of computing power, the very deep transformer models have shown powerful representation ability in various language tasks. Among them, BERT Devlin et al. (2019) uses transformer encoders and employs masked language modeling task that aims to recover the random masked tokens within a text. OpenAI proposed GPT Radford & Narasimhan (2018) that trains transformer decoders on a large language corpus and then fine-tunes on task-specific data. GPT2 Radford et al. (2019) is trained on larger datasets with much more parameters and can be transferred to various downstream tasks. Since transformer models can adapt to various inputs, the idea of pre-training can also be well adapted to visual tasks. DEiT Touvron et al. (2021) proposed a teacher-student strategy for transformers with convolution neural networks (CNNs) as the teacher model and achieves competitive performance. BEiT Bao et al. (2022) converts images as visual tokens and successfully uses the BERT model in CV. However, because of the insufficient training sample , there is little research on pre-trained models on general time series analysis that cover all major tasks like CV or NLP domain.

Cross-modality knowledge transfer Since transformers can handle different modal tasks through tokenizing the inputs to embeddings, it is also an interesting topic whether the transformers have universal representation ability and can be used for transferring between various domains. The VL pre-trained model VLMo Bao et al. (2021) proposed a stagewise pre-training strategy that utilizes frozen attention blocks pre-trained by image-only data to train the language expert. One of the most related works which transfer knowledge from a pre-trained language model to other domains is Lu et al. (2022), which studies the strong performance of a frozen pre-trained language model (LM) compared to an end-to-end transformer alternative learned from other domains' data. Another relative work for knowledge transfer to the time series is the Voice2series Yang et al. (2021), which leverages a pre-trained speech processing model for time series classification and achieves superior performance. To the best of our knowledge, no previous research has investigated cross-modality knowledge transfer for the time series forecasting task, let alone general time series analysis.

3 Methodology

3.1 Model Structure

The architecture we employ is depicted in Figure 2. We utilize parameters from NLP pretrained transformer models for time series analysis, with a focus on the GPT2 model Radford et al. (2019). We also experiment with other models, such as BERT Devlin et al. (2019) and BEiT Bao et al. (2022), to further demonstrate that the universal performance of cross-domain knowledge transfer exists in a wide range of pre-trained models.

Frozen Pretrained Block Our architecture retains the positional embedding layers and self-attention blocks from the pre-trained models. As self-attention layers and FFN (Feedforward Neural Networks) contain the majority of learned knowledge from pre-trained language models, we opt to freeze the self-attention blocks while fine-tuning.

Positional Embeddings and Layer Normalization To enhance downstream tasks with minimal effort, we fine-tune the positional embeddings and layer normalization layer, which is considered a

Instance Norm + Patching

Positional

Embeddings

Input Embedding

Output Linear Layer

Frozen

Transformer Blocks

Multi-Head Attention

Add & Layer Norm

Feed Forward

Add & Layer Norm

❄

"

"

❄

"

×"

Pretrained

Params

Fine-tune

!

Frozen

❄

standard practiceLu et al. (2022); Houlsby et al. (2019). As a result, we retrain these components during fine-tuning.

Input Embedding Given our goal of applying the NLP pre-trained model to various tasks and a new modality, we must redesign and train the input embedding layer. This layer is responsible for projecting the time-series data to the required dimensions of the specific pre-trained model. To accomplish this, we use linear probing, which also reduces the number of parameters required for training.

Normalization Data normalization is crucial for pre-trained models across various modalities. In addition to the layer norm utilized in pre-trained LM, we also incorporate a simple data normalization block, reverse instance norm Kim et al. (2022), to further facilitate knowledge transfer. This normalization block simply normalizes the input time series using mean and variance, and then adds them back to the output.

Patching To extract local semantic information, we utilize patching Nie et al. (2022) by aggregating adjacent time steps to form a single patch-based token. Patching enables a significant increase in the input historical time horizon while maintaining the same token length and reducing information redundancy for transformer models. In our architecture, we apply patching after instance normalization.

4 Main Time Series Analysis Tasks

Our proposed method excels in various downstream time series analysis tasks through fine-tuning. To demonstrate the effectiveness of our approach, we conduct extensive experiments on major types of downstream tasks, including time series classification, anomaly detection, imputation, short/long-term forecasting and few-shot/zero-shot forecasting. To ensure a fair comparison, we use GPT2-backbone FPT and adhere to the experimental settings of TimesNet Wu et al. (2023). Due to the space limit, only the summarized results are presented below except zero-shot forecasting. Full experimental results of the other six downstream tasks can be found in Appendix D.3, D.2, D.7, H.6, H.7, H.8, H.9 respectively.

Baselines We select representative baselines and cite their results from Wu et al. (2023), which includes the most recent and quite extensive empirical studies of time series. The baselines include CNN-based models: TimesNet Wu et al. (2023); MLP-based models: LightTS Zhang et al. (2022) and DLinear Zeng et al. (2023); Transformer-based models: Reformer Kitaev et al. (2020), Informer Zhou et al. (2021), Autoformer Wu et al. (2021), FEDformer Zhou et al. (2022), Nonstationary Transformer Liu et al. (2022), ETSformer Woo et al. (2022), PatchTST Nie et al. (2022). Besides, N-HiTS Challu et al. (2022) and N-BEATS Oreshkin et al. (2019) are used for short-term forecasting. Anomaly Transformer Xu et al. (2021) is used for anomaly detection. XGBoost Chen & Guestrin (2016), Rocket Dempster et al. (2020), LSTNet Lai et al. (2018), LSSL Gu et al. (2021),

Pyraformer Liu et al. (2021), TCN Franceschi et al. (2019) and Flowformer Huang et al. (2022) are used for classification.

4.1 Main Results

Overall, as shown in Figure 1, GPT2-backbone FPT outperforms other models in most tasks, including long/short-term forecasting, classification, anomaly detection, imputation, and fow-shot/zero-short forecasting. This confirms that time series tasks can also take advantage of cross-modality transferred knowledge. In the following, we use GPT2(K) to represent GPT2-backbone with first K Layers.

4.2 Imputation

Setups We conduct experiments on six popular real-world datasets, including 4 ETT datasets Zhou et al. (2021) (ETTh1, ETTh2, ETTm1, ETTm2), Electricity and Weather, where the data-missing is common. Following the settings of TimesNet, different random mask ratios ({12.5%, 25%, 37.5%, 50%}) of time points are selected for the evaluation on various proportions of missing data.

Results The results are shown in Table 1 that GPT2(3) FPT achieves the best performance on most datasets. Particularly, compared to the previous SOTA TimesNet, GPT2(3) FPT yields a relative 11.5% MSE reduction on ETTh1,and a 4.1% MSE reduction on average on six benchmark datasets. It verifies that the proposed method can also effectively mine temporal patterns of incomplete time series.

4.3 Time Series Classification

Setups To evaluate the model's capacity for high-level representation learning, we employ sequence-level classification. Specifically, we follow the same setting as TimesNet: For classification, 10 multivariate UEA classification datasets Bagnall et al. (2018) are selected for evaluation, including gesture, action, audio recognition medical diagnosis and other practical tasks.

Results As shown in Figure 3, GPT2(6) FPT achieves an average accuracy of 74.00%, surpassing all baselines including TimesNet (73.60%). Specifically, compared to recent published patch-transformer-based models Nie et al. (2022) , GPT2(6) FPT surpasses it by a large margin 9.0% which shows the prior NLP transfer knowledge can indeed help in time series representation.

4.4 Time Series Anomaly Detection

Setups Detecting anomalies in time series is vital in industrial applications, ranging from health monitoring to space & earth exploration. We compare models on five commonly used datasets, including

Average Accuracy (%)

XGBoost

DLinear

PatchTST

TCN

LightTS.

FEDformer

Pyraformer

LSSL

ETSformer

Autoformer

Reformer

LSTNet

Transformer

Informer

Rocket

Stationary

Flowformer

TimesNet

GPT2(6)

66.00

67.50

67.90

70.30

70.40

70.70

70.80

70.90

71.00

71.10

71.50

71.80

71.90

72.10

72.50

72.70

73.00

73.60

74.00

SMDSu et al. (2019), MSLHundman et al. (2018), SMAPHundman et al. (2018), SWaTMathur & Tippenhauer (2016) and PSMAbdulaal et al. (2021). To perform a fair comparison, only the classical reconstruction error is used for all baseline models to the make the setting the same as TimesNet.

Results Table 2 demonstrates that GPT2(6) FPT also achieves the best performance with the averaged F1-score 86.72% , surpassing previous SOTA method TimesNet by 1.7% . Thus, in addition to its proficiency in representing complete sequences for classification purposes, GPT2(6) FPT is capable of detecting infrequent anomalies within time series.

Table 2: Anomaly detection task. We calculate the F1-score (as %) for each dataset. ∗ . in the Transformers indicates the name of ∗ former. Black : best, Red : second best. Appendix H.7 shows the full results.

We reproduce the results of TimesNet by https://github.com/thuml/Time-Series-Library.

4.5 Long-term Forecasting

Setups Eight popular real-world benchmark datasets Wu et al. (2023), including Weather, Traffic , Electricity, ILI , and 4 ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2), are used for long-term forecasting evaluation. Additional information regarding the discussion on the input length setting can be found in the appendix H.10.

Results As shown in Table 3, GPT2(6) FPT achieves comparable performance with PatchTST and outperforms other baselines. Specifically, compared with recent published SOTA method TimesNet, GPT2(6) FPT yields a relative 9.3% average MSE reduction.

4.6 Short-term Forecasting

Setups To fully evaluate different algorithms in forecasting tasks, we also conduct short-term forecasting (with relatively short forecasting horizon) experiments on M4 Makridakis et al. (2018), contains marketing data of various frequencies.

Results The results in Table 4 show that the performance of GPT2-backbone (6) FPT is superior to advanced Transformer-based and MLP-based models, and comparable to TimesNet and N-BEATS.

4.7 Few-shot Forecasting

The large language model (LLM) has demonstrated remarkable performance in both few-shot and zero-shot learning settings Brown et al. (2020); OpenAI (2023). It can be argued that few-shot and

zero-shot learning also represent the ultimate tasks for a universal time series forecasting model. To extensively evaluate the representation power of the GPT2(6) for time series analysis, we conduct experiments under few-shot and zero-shot learning settings.

Similar to traditional experimental settings, each time series is split into three parts: training data, validation data, and test data. For few-shot learning, only a certain percentage (10%, 5%) timesteps of training data are used.

The results of 10% few-shot learning are shown in Table 5. Compared to TimesNet, DLinear, PatchTST and other methods, GPT2(6) FPT achieves the best performance. Traditionally, CNNbased and single MLP-based models are considered more data-efficient for training and suitable for few-shot learning methods. In comparison to convolution-based TimesNet and MLP-based DLinear models, GPT2(6) FPT demonstrates a relative average MSE reduction of 33.3% and 13.5% respectively. We add a comparison with traditional algorithms (ETS, ARIMA, NaiveDrift) in the Appendix D.5 as well, and GTP2(6)FPT also surpass all those traditional methods.

4.8 Zero-shot forecasting

This task is used to evaluate the cross datasets adaption ability of our proposed algorithm, i.e. how well a model is able to perform on dataset A (without any training data from A ) when it is trained from dataset B .

The results are summarized in Table 6. The GPT2(6) FPT model consistently outperforms all recent state-of-the-art transformer and MLP-based time series forecasting methods. Compared to recently published state-of-the-art MLP-based method Dlinear, convolution-based method Timesnet, and transformer-based method Patchtst, GPT2(6)FPT demonstrates a relative average metric reduction of 13.1% , 13.6% and 7.3% , respectively. Also, the proposed method is comparable to N-BEATS without any meta-learning design and outperforms N-BEATS in the ELECTR dataset. We attribute this to the knowledge transfer capability from the FPT model.

5 Ablations

In this section, we conduct several ablations on model selection and effectiveness of pre-training. The detailed results are shown in Appendix H. We introduce several variants, GPT2(0) FPT, GPT2(6) without freezing and GPT2(6) without pre-training.

Model Selection We separately analyze the number of GPT2 layers and the fine-tuning parameters selection. The results in Appendix H show that GPT2 with 6-layers is a sound choice compared to full or few layers and partially freezing can avoid catastrophic forgetting, enabling fine-tuning without overfitting.

Effectiveness of Pre-training The results are shown in Table 7, GPT2(6) FPT outperforms both GPT2(0) FPT and GPT2-random-initialized, suggesting that GPT2 with pre-training parameters can achieve improvement on times series tasks. Besides, GPT2(6) FPT performs better than GPT2unfrozen, demonstrating that partially freezing also helps. Also, results in Appendix H.2 show that random initialized GPT2(6) with freezing performs poorly and the pre-trained knowledge is instrumental for time series tasks.

6 Exploring Transfer Learning from others: The Unexceptional Nature of GPT2-based-FPT

We also present experiments on BERT-backbond FPT Devlin et al. (2019) model and the imagepretrained BEiT-backbone FPT model Bao et al. (2022) to illustrate the generality of pre-trained models for cross-domain knowledge transferring. The results in Table 8 demonstrate that the ability of knowledge transfer is not exclusive to GPT2-based pre-trained language models. Subsequently, our theoretical analysis will shed light on the universality of this phenomenon.

7 Training/Inferencing Cost

Analysis of computational cost is helpful for investigating the practicality of the LLM-based model. The results can be found in table 9. Each baseline model comes in two variants, featuring model

hidden dimensions of 32 and 768, which align with GPT-2's specifications. Furthermore, the majority of the baseline models consist of three layers. We assessed the computational cost using a batch from ETTh2 (with a batch size of 128) on a 32G V100 GPU.

The results indicate that GPT-2(3) has substantially enhanced time efficiency and reduced parameter quantity compared to baselines with the same model dimension. This was a surprise since we initially anticipated that this large language model might be slower. However, we surmise that the efficient optimization of huggingface's GPT model implementation primarily accounts for such a significant improvement in time costs. Furthermore, GPT-2(3) and GPT-2(6) demonstrate a mere 6.12% and 4.60% proportion of learnable parameters among the overall parameter size, respectively.

8 Towards Understanding the Universality of Transformer: Connecting Self-Attention with PCA

The observation, i.e. we can directly use a trained LM for time series forecasting without having to modify its model, makes us believe that the underlying model is doing something very generic and independent from texts despite it being trained from text data. Our analysis aims to show that part of this generic function can be related to PCA, as minimizing the gradient with respect to the self-attention layer seems to do something similar to PCA. In this section, we take the first step towards revealing the generality of self-attention by connecting the self-attention with principal component analysis (PCA). Moreover, when coming the question of why fine-tuning is restricted to the embedding layer and layer norm, following our hypothesis that the pre-trained LM as a whole performs something generic, partially fine-tuning any of its components may break the generic function and lead to relatively poor performance for time series analysis.

For each layer, we calculate and perform statistical analysis of the pairwise token similarity values. Specifically, we denote each output feature map with shape of ( b, n, d ) , where b is the batch size, n is the number of tokens, and d is the dimension of each token feature. We calculate the cosine similarity, and the resulting pairwise similarity matrix of shape ( b, n, n ) . Next we count the number of occurrences of similarity values within each interval as a simple statistical analysis.

Our analysis is motivated by the observation that the within-layer token similarity increases with deeper layers in transformer. We report the layer-wise average token cosine similarity on ETTh2 dataset in Figure 4 (a, c), where we mix weights from pre-trained LM with weights randomly sampled from Gaussian distribution. Here we summarize our observations: a) in a randomly initialed GPT2 (6) model, the token similarity is low among all layers ( 0 . 1 -0 . 2 ); b) when gradually switched to the pretrained GPT2 model, the token similarity significantly increases in the deep layers and eventually reaches more than 0.9 in the last layer. One potential explanation for the increasing token similarity is that all the token vectors are projected into the low-dimensional top eigenvector space of input patterns. To verify this idea, we further conduct experiments where we replace the self-attention module with PCA and find token similarity patterns remain unchanged according to Figure 4 (b), which further justifies the potential connection between PCA and self-attention.

To build the theoretical connection between PCA and self-attention, we first analyze the gradient structure of self-attention. Let X = ( x 1 , . . . , x N ) ⊤ ∈ R N × D be the input pattern, and let f ( X ) = ( f 1 ( X ) , . . . , f N ( x )) ⊤ : R N × D ↦→ R N × D be the function for self-attention, i.e., f i ( X ) = softmax ( XAX ⊤ ) X where A = W Q W ⊤ K ∈ R D × D .

(b) Token Similarity: PCA as Attention

(c) Token Similarity: Different Replace Ratios

Replace Ratio (%)

(a) Performance: Different Replace Ratios

(3W / 3H) a

0.50 -

0.48

0.46

0.44

0.42

0.52

MSE

MAE

qune

300

400

800

700

500

600

layer_4

layer_2

layer_3

layer_5

layer_6

layer_1

layer_0

qune)

4000

3000

2000

layer_5

layer_3

layer_2

layer_1

layer_0

layer_6

layer_4

10000

12500

17500

15000

7500

口

layer_5

layer_4

layer_6

layer_3

layer_2

layer_1

layer_0

0.40

200

1000

5000

100

2500

20

40

60

80

100

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0

0.2

0.8

0.2

80

1.0

17500

15000

12500

10000

20000

7500

layer_6

layer_5

layer_4

layer_0

layer_3

layer_2

layer_1

10000

17500

15000

12500

7500

二

layer_6

layer_5

layer_3

layer_o

layer_4

layer_2

layer_1

17500

15000

12500

7500

layer_4

layer_5

layer_6

layer_3

layer_2

layer_1

layer_0

17500

15000

12500

7500

layer_4

layer_5

layer_0

layer_2

layer_1

layer_6

layer_3

5000

5000

5000

5000

2500

2500

2500

2500

0

0.2

0.4

0.6

0.8

1.0

0.2

0.8

1.0

Z0

0.8

1.0

Z0

0.8

1.0

̸

This lemma reveals an important gradient structure of J . The proof of essentially follows the analysis in Kim et al. (2021), and we include it in Appendix G for completeness.

Using the gradient structure revealed in Lemma 8.1, we can connect self-attention with PCA. In order to minimize the norm of gradient | J | 2 , we essentially need to make ∑ N i =1 | x i -∑ N j =1 P i,j x j | 2 small. When A is small and all the input patterns are centered at 0 (i.e. ∑ N i =1 x i = 0 ), we have ∑ N i =1 | x i -X ⊤ P i, : | 2 ≈ ∑ N i =1 | x i -X ⊤ XAx i | 2 .

The theorem below shows that A minimizing the objective ∑ N i =1 | x i -X ⊤ XAx i | 2 contains the largest m eigenvectors of X ⊤ X where m is the rank of A .

Theorem 1 . Let W Q and W K be matrices of size D × m . Let λ 1 ≥ λ 2 ≥ ... ≥ λ D be the eigenvalues of X ⊤ X ranked in descending order, and let v i ∈ R D , i = 1 , . . . , D be the corresponding eigenvectors. The optimal solution A ∗ that minimizes ∑ N i =1 | x i -X ⊤ XAx i | 2 is given by A = ∑ m i =1 1 λ i v i v ⊤ i .

The proof of Theorem 1 can be found in Appendix G. Following Theorem 1, through the training of pushing gradient to zero, self-attention learns to perform a function closely related to PCA.

9 Conclusions

In this paper, we developed a foundation model for time series analysis, based on pre-trained model from NLP or CV, that can (a) facilitate the model training for downstream tasks, and (b) provide unified framework for diverse time series analysis tasks. Our empirical studies show that the proposed method performs on par or better than the state-of-the-art approaches on almost all time series tasks. We also examine the universality of transformer by connecting self-attention with PCA, an important step towards understanding how generative models work in practice. On the other hand, we do recognize some limitations of our work: the zero-shot performance of our approach is still behind N-beat on several datasets, and our analysis of the generality of transformer is still in the early stage. Moving forward, we plan to improve the performance of our approach by exploiting the parameter efficient fine-tuning approaches which usually introduce additional structures into the pre-trained model for better adaption. To better understand the universality of transformer, we also plan to examine it from the viewpoint of n-gram language model, an approach that is taken by Elhage et al. (2021); Olsson et al. (2022). In Appendix F, we include our initial analysis along this direction.

References

Abdulaal, A., Liu, Z., and Lancewicki, T. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pp. 2485-2494, 2021.

Bagnall, A., Dau, H. A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., and Keogh, E. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., and Wei, F. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358 , 2021.

Bao, H., Dong, L., Piao, S., and Wei, F. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations , 2022.

Böse, J.-H., Flunkert, V., Gasthaus, J., Januschowski, T., Lange, D., Salinas, D., Schelter, S., Seeger, M., and Wang, Y. Probabilistic demand forecasting at scale. Proceedings of the VLDB Endowment , 10(12):1694-1705, 2017.

Box, G. E. and Jenkins, G. M. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 17(2):91-109, 1968.

Box, G. E. and Pierce, D. A. Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. Journal of the American statistical Association , 65(332): 1509-1526, 1970.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T. J., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. ArXiv , abs/2005.14165, 2020.

Challu, C., Olivares, K. G., Oreshkin, B. N., Garza, F., Mergenthaler, M., and Dubrawski, A. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 , 2022.

Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. KDD '16, 2016.

Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Courty, P. and Li, H. Timing of seasonal sales. The Journal of Business , 72(4):545-572, 1999.

Dempster, A., Petitjean, F., and Webb, G. I. ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery , 34(5): 1454-1495, 2020.

Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA, June 2-7, 2019 , pp. 4171-4186, 2019.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations (ICLR), Austria, May 3-7, 2021 , 2021.

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Franceschi, J.-Y., Dieuleveut, A., and Jaggi, M. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems , 32, 2019.

Friedman, M. The interpolation of time series by related series. J. Amer. Statist. Assoc , 1962.

Gao, J., Song, X., Wen, Q., Wang, P., Sun, L., and Xu, H. RobustTAD: Robust time series anomaly detection via decomposition and convolutional neural networks. KDD Workshop on Mining and Learning from Time Series (KDD-MileTS'20) , 2020.

Giannou, A., Rajput, S., Sohn, J.-y., Lee, K., Lee, J. D., and Papailiopoulos, D. Looped Transformers as Programmable Computers. arXiv e-prints , art. arXiv:2301.13196, January 2023. doi: 10.48550/ arXiv.2301.13196.

Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., and Montero-Manso, P. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks , 2021.

Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021.

Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pp. 2790-2799. PMLR, 2019.

Huang, Z., Shi, X., Zhang, C., Wang, Q., Cheung, K. C., Qin, H., Dai, J., and Li, H. Flowformer: A transformer architecture for optical flow. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pp. 668-685. Springer, 2022.

Hundman, K., Constantinou, V., Laporte, C., Colwell, I., and Soderstrom, T. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pp. 387-395, 2018.

Hyndman, R. and Athanasopoulos, G. Forecasting: Principles and Practice . OTexts, Australia, 3rd edition, 2021.

Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., and Muller, P.-A. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery , 33(4):917-963, 2019.

Kim, H., Papamakarios, G., and Mnih, A. The lipschitz constant of self-attention. In International Conference on Machine Learning , pp. 5562-5571. PMLR, 2021.

Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, J. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations , 2022.

Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Lacoste-Julien, S., Schmidt, M., and Bach, F. A simpler approach to obtaining an o (1/t) convergence rate for the projected stochastic subgradient method. arXiv preprint arXiv:1212.2002 , 2012.

Lai, G., Chang, W.-C., Yang, Y., and Liu, H. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pp. 95-104, 2018.

Lim, B., Arık, S. Ö., Loeff, N., and Pfister, T. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting , 2021.

Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., and Dustdar, S. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations , 2021.

Liu, Y., Wu, H., Wang, J., and Long, M. Non-stationary transformers: Exploring the stationarity in time series forecasting. In Advances in Neural Information Processing Systems , 2022.

Lu, K., Grover, A., Abbeel, P., and Mordatch, I. Frozen pretrained transformers as universal computation engines. Proceedings of the AAAI Conference on Artificial Intelligence , 36(7): 7628-7636, Jun. 2022.

Makridakis, S., Spiliotis, E., and Assimakopoulos, V. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting , 34(4):802-808, 2018.

Mathur, A. P. and Tippenhauer, N. O. Swat: A water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater) , pp. 31-36. IEEE, 2016.

Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. ArXiv , abs/2211.14730, 2022.

Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learning and induction heads. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.

OpenAI. Gpt-4 technical report. ArXiv , abs/2303.08774, 2023.

Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 , 2019.

Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y. Meta-learning framework with applications to zero-shot time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence , number 10, pp. 9242-9250, 2021.

Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.

Rao, Y., Zhao, W., Zhu, Z., Lu, J., and Zhou, J. Global filter networks for image classification. Advances in Neural Information Processing Systems (NeurIPS) , 34, 2021.

Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., and Pei, D. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pp. 2828-2837, 2019.

Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , pp. 10347-10357. PMLR, 2021.

Vardi, G., Yehudai, G., and Shamir, O. On the optimal memorization power of relu neural networks. arXiv preprint arXiv:2110.03187 , 2021.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser Lukasz, and Polosukhin, I. Attention is all you need. arXiv preprint arXiv:1706.03762 , 2017.

Wang, P., Wang, X., Wang, F., Lin, M., Chang, S., Li, H., and Jin, R. Kvt: k-nn attention for boosting vision transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV , pp. 285-302. Springer, 2022.

Wang, T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. ArXiv , abs/2005.10242, 2020.

Wen, Q., Yang, L., Zhou, T., and Sun, L. Robust time series analysis and applications: An industrial perspective. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pp. 4836-4837, 2022.

Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., and Sun, L. Transformers in time series: A survey. In International Joint Conference on Artificial Intelligence(IJCAI) , 2023.

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-of-theart natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020. emnlp-demos.6 .

Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, S. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381 , 2022.

Wu, H., Xu, J., Wang, J., and Long, M. Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting. In Advances in Neural Information Processing Systems (NeurIPS) , pp. 101-112, 2021.

Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=ju_Uqw384Oq .

Xu, J., Wu, H., Wang, J., and Long, M. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:2110.02642 , 2021.

Yang, C.-H. H., Tsai, Y.-Y., and Chen, P.-Y. Voice2series: Reprogramming acoustic models for time series classification. In International Conference on Machine Learning , pp. 11808-11819, 2021.

Yun, C., Chang, Y.-W., Bhojanapalli, S., Rawat, A. S., Reddi, S., and Kumar, S. O (n) connections are expressive enough: Universal approximability of sparse transformers. Advances in Neural Information Processing Systems , 33:13783-13794, 2020.

Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? 2023.

Zhang, T., Zhang, Y., Cao, W., Bian, J., Yi, X., Zheng, S., and Li, J. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 , 2022.

Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI , 2021.

Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022) , 2022.

A Visualization

In order to clarify the representation ability more clearly, Figure 5 provides showcases of imputation, long-term forecasting and few-shot forecasting. Especially for few-shot learning, GPT2(6) can accurately forecast, while TimesNet and DLinear fail in this task.

(a) Imputation on ETTh1 12.5%

GPT2(3)

TimesNet

DLinear

(b) Long-term forecasting on ETTm1

GPT2(6)

TimesNet

DLinear

(c) Few-shot forecasting on ETTh2

GPT2(6)

TimesNet

DLinear

2.0

GPT2(3)

GroundTruth

2.0

GroundTruth

TimesNet

2.0

DLinear

GroundTruth

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

0.5

0.5

0.5

1.0

1.0

1.0

0

20

40

60

08

0

20

40

60

08

0

20

40

60

80

1

1

1

0

0

0

1

-1

1

-2

2

-2

-3

3

-3

GroundTruth

-4

GroundTruth

-4

GroundTruth

GPT2(6)

TimesNet

DLinear

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

0.0

GroundTruth

GroundTruth

GroundTruth

GPT2(6)

0.5

TimesNet

0.5

DLinear

0.5

1.0

1.0

1.0

1.5

1.5

1.5

2.0

2.0

2.0

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

B Related Works

We have presented a novel general time series analysis model in this paper, and to the best of our knowledge, there has been limited work on similar comprehensive methods for time series analysis. The most closely related field is time series forecasting, where transformer models have gained widespread popularity. Therefore, our focus in this related work will primarily be on introducing the end-to-end time series forecasting method.

Time series forecasting models can be roughly divided into three categories, ranging from the classic ARIMA models to the most recent transformer models. The first generation of well-discussed models can be dated back to auto-regressive family, such as ARIMA Box & Jenkins (1968); Box & Pierce (1970) that follows the Markov process and recursively execute sequential forecasting. However, it is limited to stationary sequences while most time series is non-stationary. Additionally, with the bloom of deep neural networks, recurrent neural networks (RNNs), such as LSTM Hochreiter &Schmidhuber (1997) and GRU Chung et al. (2014), were designed for sequential tasks. Yet the recurrent model is inefficient for training and long-term dependencies are still under resolved.

Recently, transformer models have achieve great progress in NLP Vaswani et al. (2017); Devlin et al. (2019); Radford et al. (2019) and CV Dosovitskiy et al. (2021); Bao et al. (2022) tasks. Also, a large amount of transformer models are proposed to apply to time series forecasting Wen et al. (2023). In the following, we briefly introduce several representative algorithms. Informer Zhou et al. (2021)

proposes a probability sparse attention mechanism to deal with long-term dependencies. Autoformer Wu et al. (2021) introduces a decomposition transformer architecture and replaces the attention module with an Auto-Correlation mechanism. FEDformer Zhou et al. (2022) uses Fourier enhanced structure to improve computational efficiency and achieves linear complexity. Similar to patching in ViT Dosovitskiy et al. (2021), PatchTST Nie et al. (2022) employs segmentation of time series that divide a sequence into patches to increase input length and reduce information redundancy. Besides, a simple MLP-based model DLinear Zeng et al. (2023) outperforms most transformer models and it validates channel-independence works well in time series forecasting. Recently, TimesNet Wu et al. (2023) has treated time series as a 2D signal and utilized a convolution-based inception net backbone to function as a comprehensive time series analysis model. This work is closely related to our tasks in this paper.

C Dataset Details

In this section, we separately summarize dataset details long/short-term forecasting and few-shot/zeroshot forecasting.

Datasets of Long-term Forecasting and Few-shot Learning The details of datasets are shown as follows: 1) ETT datasets Zhou et al. (2021) contain electricity load of various resolutions (ETTh & ETTm) from two electricity stations. 2) Weather contains 21 meteorological indicators of Germany within 1 year; 3) Illness contains the influenza-like illness patients in the United States; 4) Electricity dataset contains the electricity consumption; 5) Traffic dataset contains the occupation rate of freeway system across the State of California. Table 10 summarizes details of feature statistics.

Similar to PatchTST Nie et al. (2022), Exchange is not contained. Zeng et al. (2023) shows that simply repeating the last value in the look-back window can outperform or be comparable to the best results. Also, ILI is not used for few-shot learning for the limited quantity that is hard to follow the definition of few-shot.

Datasets of Short-term Forecasting and Zero-shot Learning The details of short-term forecasting and zero-shot learning datasets are shown as follows: 1) M4 is a large and diverse dataset that contains time series of various frequencies and fields, including business, financial and economic forecasting; 2) M3 is smaller than M4, but also contains time series from diverse domains and frequencies; 3) TOURISM is the dataset of tourism activities with different frequencies and contains a much higher fraction of erratic series compared with M4; 4) ELECTR represents the electricity usage monitoring of 370 customers over three years. Table 6 summarizes details of the datasets and zero-shot mapping between source and target.

D Experimental Details

All the deep learning networks are implemented in PyTorch and trained on NVIDIA V100 32GB GPUs. We use the pre-trained models from Wolf et al. (2020) for experiments. For few-shot learning, an early stopping counter is employed to stop the training process after three epochs if no loss degradation on the valid set is observed. Plus, we convert the multivariate data into univariate data. Specifically, we treat each feature of the sequence as a single time series. This is mainly for memory efficiency after patching of GPT2(6) and previous works, DLinear and PatchTST, have proved the effectiveness of channel-independence.

D.1 Accuracy Metrics

For long-term/short-term forecasting and few-shot forecasting, we use mean square error (MSE) and mean absolute error (MAE) as metrics. For zero-shot learning, mean absolute percentage error (MAPE) is used for TOURISM; symmetric MAPE (sMAPE) is used for M3 and M4; normalized deviation (ND) is used for ELECTR. All experiments are repeated 3 times and the mean of the metrics is used in the final results.

D.2 Detailed Definition and Results for Few-shot and Long-term Forecasting

Task Definition Since Zeng et al. (2023) and Nie et al. (2022) have verified that channel-independence works well for time series datasets, we treat each multivariate series as multiple independent univariate series. Similar to traditional experimental settings, each time series is split into three parts: training data, validation data, and test data. For the few-shot forecasting task, only a certain percentage ( 5% , 10% ) timesteps of training data are used, and the other two parts remain unchanged. The evaluation metrics remain the same as for classic multivariate time series forecasting. We repeat this experiment 3 times and report the average metrics in the following experiments.

Detail Experiment Tables for Few-shot Time-Series Forecasting in Table 12 and Table 13

D.3 Long-term Time-series Forecasting

Here we investigate whether our architecture performs consistently well with more training data. Thus, we follow the classical experiment settings of Nie et al. (2022) and conduct experiments on full data. The results are shown in Table 14. Overall, GPT2(6) FPT achieves comparable performance to PatchTST, Dlinear and outperforms other baselines by a large margin. Compared with the second best transformer-based baseline method FEDformer, GPT2(6) FPT yields an overall 18.7% relatively MSE reduction. It verifies the effectiveness of NLP pretrained model in time series forecasting, not limited to the few-shot setting.

Detail Experiment Table for Long-term Time-Series Forecasting in table 14

Table 14: Full results on full data. We use prediction length O ∈ { 96 , 192 , 336 , 720 } for ILI and O ∈ { 24 , 36 , 48 , 60 } for others. A lower MSE indicates better performance. Black : best, Red : second best.

D.4 Mean and STD for Few-shot Learning

Table 15 lists both mean and STD for GPT2(6), DLinear and PatchTST with 3 runs on 5% ETTh2 and ETTm2. The results show a small variance in performance of GPT2(6) that represents the stability of GPT2(6).

D.5 Comparison with Traditional Methods on Few-shot Learning

Since deep learning methods are more advantageous than traditional methods when applied to large datasets. For few-shot learning, traditional methods should also consider. The results are shown in Table 16 that GPT2(6) also achieves best performance.

D.6 Baselines with Instance Normalization

Instance normalization Kim et al. (2022) is a plug-in for time series for distribution shift. Most baselines, such as Autoformer and FEDformer are not equipped with instance normalization. Thus, for a fair comparison, we add the experiment, as in Table 17, for baselines w/o instance normalization and GPT(6) can also perform superior.

D.7 Detailed Definition and Results of Zero-shot Learning

Task Definition Each experiment contains two distinct datasets, source, and target datasets. The source dataset is used to train the model and then forecasts without fine-tuning in the target dataset. The target dataset is split into non-overlapping historical and test sequences. We use the historical sequence as input to the model, and the obtained output is used to calculate errors with the test sequences. Besides meta-learning-based models like N-BEATS, evaluated models' parameters are not allowed any adjustment using the forecasting phase. Also, same as Oreshkin et al. (2021), each data set adopts a specific metric (M4: sMAPE; M3: sMAPE; TOURISM: MAPE; ELECTR: ND)

Detailed Results Here, we list detailed performance of zero-shot learning in Table 18, Table 19 and Table 20. For each dataset, we separately list the performance of models under diverse frequency. Compared to the most recent published method DLinear, GPT2(6) performs superior in most situations. Also, GPT2(6) does not use any information from the test data, but achieves a comparable performance of meta-leaning based N-BEATS.

E Proof

In our numerical experiments, we obtain two interesting observations. First, the token similarity within a sample is larger in pretrained LM. We report the layer-wise average token cosine similarity in ETTh2 experiment in Figure 7. In particular, Figure 7 (a) shows that in a fine-tuned random initialed GPT2(6) model, the token similarity is around 0.1-0.2 among different layers. When switching to the frozen pre-trained GPT2-FPT model, the token similarity significantly increases in the deep layers and eventually reaches more than 0.9 in the last layer. The ETTh2 dataset contains high volatility hourly information related to the electricity transformer temperature. In this situation, higher token similarity implies the high-frequency noise in the data is eased and only low-frequency information will be reserved. In other words, after going through the pretrained GPT2-FPT model, the signal-noise ratio is enhanced. We use the following theorem to characterize this behavior.

E.1 Theorem E.1

Theorem E.1 (informal) . We consider the self-attention for l -th query token. Let's assume the input token x i are bounded with mean µ for i = 1 , 2 , ..., n . Under mild conditions, with high probability, the output value token V l converges to µ W v on the order of O ( n -1 / 2 ) , where W v is the parameter matrix to compute the value token.

The Theorem E.1 describes the self-attention structure can efficiently make output value token V l converge its mean value µ W v . In the time series forecasting task, each token represents several adjacent points in a time series. When the time series has some periodical or translation invariant structures, by comparing a given token with other tokens, one could have a higher chance to figure out those invariant structures. This phenomenon is especially important in few-shot forecasting tasks. Without enough token noise distillation ability, the model will more likely tend to overfit due to insufficient training data.

We denote x i as i -th element of vector x , W ij as the element at i -th row and j -th column of matrix W , and W j as the j -th row of matrix W . Moreover, we denote x i as the i -th patch (token) of the inputs with x i = X i .

Before given the formal statement of the Theorem E.1, we first show the assumptions.

The token x i is the sub-gaussian random vector with mean µ i and variance ( σ 2 /d ) I for i = 1 , 2 , ..., n .

̸

µ follows a discrete distribution with finite values µ ∈ V . Moreover, there exist 0 < ν 1 , 0 < ν 2 < ν 4 such that a) ∥ µ i ∥ = ν 1 , and b) µ i W Q W T K µ i ∈ [ ν 2 , ν 4 ] for all i and | µ i W Q W ⊤ K µ ⊤ j | ≤ ν 2 for all µ i = µ j ∈ V .

W V and W Q W ⊤ K are element-wise bounded with ν 5 and ν 6 respectively, that is, | W ( ij ) V | ≤ ν 5 and | ( W Q W ⊤ K ) ( ij ) | ≤ ν 6 , for all i, j from 1 to d .

In the above assumptions, we ensure that for a given query patch, the difference between the clustering center and noises are large enough to be distinguished.

Theorem E.2 (formal statement of Theorem E.1) . Let patch x i be σ 2 -subgaussian random variable with mean µ i and all n patches follow the same clustering center of query l . Per Assumptions aforementioned, when √ d ≥ 3( ψ ( δ, d ) + ν 2 + ν 4 ) , then with probability 1 -5 δ , we have

Proof. See the proof of Lemma 2 in Wang et al. (2022) with k 1 = k = n . ■

E.2 Theorem E.4

We first give the formal statement of Theorem E.4.

Theorem E.3 (formal statement of Theorem E.4) . Let g i ∈ ❘ d and y i ∈ ❘ T be the feature map vector and forecasting targets for the sample i = 1 , 2 , ..., N respectively, and we assume 1 N ∑ N i =1 g i g ⊤ i ⪰ σI for some σ > 0 . We want to learn a matrix W ∈ ❘ d × T from the following optimization problem:

If we apply stochastic gradient descent with diminishing step sizes η t = 1 σt at step t , we will need t = ˜ O ( ϵ -1 σ -1 ) steps to reach

where W ∗ is the optimal solution and W j is the j step's solution and ˜ O we suppress the logarithmic dependence.

Proof. As we assume 1 N ∑ T i =1 g i g ⊤ i ⪰ σI , the hessian of optimization problem in (1) is also positive definite, which is equivalent to the optimization problem in (1) is strongly convex with parameter proportional to σ . Then via standard stochastic gradient decent analysis (e.g., section 3.1 in Lacoste-Julien et al. (2012)), we obtain:

Therefore, to reach ϵ optimization gap, we just need to set t = ˜ O ( σ -1 ϵ -1 ) . ■

The second observation is that for the pretrained GPT2-FPT model, the last transformer layer's outputs, i.e., feature maps, are spread widely throughout the feature space. We report the t-SNE visualization of the feature maps for GPT2-FPT and an end-to-end model PatchTST in Figure 8. In Figure 8 (a) and (b), we color the samples chunked from the one single time series into the same color and the same configuration of the T-SNE is applied. One may observe that the feature maps of GPT2-FPT has less concentration compared to PatchTST. It implies the GPT2-FPT's feature maps corresponding to different samples are more distinctive which eventually facilitates the learning ability of the last MLP layer. Researchers Wang & Isola (2020) have found that contrastive learning-based representation learning may result in a uniform distribution of training data, and such behavior plays an important role in its good downstream task performance. We use the following theorem to justify it.

Theorem E.4 (informal) . Let g i and y i be the feature map vector and forecasting targets for the sample i = 1 , 2 , ..., N respectively, and we assume 1 N ∑ N i =1 g i g ⊤ i ⪰ σI for some σ > 0 . Under mild conditions, if we train an MLP layer that maps feature maps to forecasting targets via the stochastic gradient descent, the total step to reach some optimization tolerance is on the order of O ( σ -1 ) .

Replace Ratio 100%

Replace Ratio 80%

Replace Ratio 60%

Replace Ratio 40%

Replace Ratio 20%

Replace Ratio 0%

Replace Ratio (%)

0.525

0.550

MSE

MAE

600

700

500

layer_3

layer_2

layer_4

layer_5

layer_6

3500

3000

2500

layer_4

layer_3

layer_2

layer_5

layer_6

17500

15000

12500

口

layer_4

layer_3

layer_2

1layer_6

1layer_5

0.500

8

yun

400

300

layer_1

layer_0

2000

1500

layer_0

layer_1

10000

7500

layer_0

layer_1

200

1000

5000

Performance (MSE / MAE)

100

500

2500

0.475

0.1

0.5

0.6

0.7

0.8

0.9

0

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

1.0

0.450

0.425

0.400

10000

17500

15000

12500

layer_6

1layer_5

layer_4

sake)

layer_2

layer_1

layer_0

10000

15000

12500

17500

layer_s

layer_4

layer_0

layer_6

layer_1

layer_3

layer_2

16000

14000

12000

10000

8000

layer_5

layer_4

I

layer_6

layer_3

layer_2

layer_1

layer_0

7500

7500

6000

5000

5000

4000

0.375

2500

2500

2000

20

40

60

08

100

0.2

0.4

9'0

8'0

1.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

(a) Random Initialization

(b) Pretrained Parameters

(c) PCA as Attention

Count

700

600

500

400

layer_6

layer_5

layer_4

layer_3

layer_2

layer_1

layer_0

Count

15000

12500

10000

17500

layer_6

layer_5

layer_3

layer_2

layer_0

layer_4

layer_1

Count

20000

17500

15000

12500

10000

layer_5

layer_6

layer_4

layer_3

layer_2

layer_1

layer_0

300

7500

7500

200

5000

5000

100

2500

2500

0

0.1

0.2

E0

0.4

0.5

0.6

0.7

0.8

0.9

0

0.2

0.4

0.6

0.8

1.0

0

0.2

0.4

0.6

0.8

1.0

The Theorem E.4 considers the covariate matrix of feature maps being positive definite that indicates the set of all feature maps { g i } spans the whole feature spaces, and the higher spread level gives a larger σ . In this case, if we only want to learn an MLP layer, the problem reduces to a well-conditioned least-squared regression problem. Then the fast convergence rate is achieved.

Efficiently learning the last MLP layer plays a very important role in time series forecasting and can substantially impact the prediction performance. In Zeng et al. (2023), the authors show that learning a single MLP layer can also bring very promising performance. In few-shot forecasting, the pre-trained GPT2 model may still preserve highly diverse feature maps than end-to-end type models and eventually leads to fast learning speed on the last MLP layer.

Another possible benefit of wide spared feature maps is enhancing the model memorization ability when using a multi-layer decoder structure. In the literature on network memorization ability (e.g., Vardi et al. (2021); Yun et al. (2020)), the deep learning model tends to have better memorization ability when feature maps are well separated. In forecasting tasks, capturing extreme or rare behavior is very important. The pretrained GPT gains more capacity in the decoder to correctly forecast uncommon time series.

F N-gram Explanation for Universality

Why does the proposed pretrained-frozen-model work so effectively? We have achieved state-ofthe-art performance in time series analysis using a language model that is mostly trained on natural language data. The answer lies in the universality of the frozen structure, which includes attention layers and Feed Forward layers. We can represent images and time series forecasting tasks as an n-gram estimation problem, akin to text analysis, by employing a patching approach. This method treats subsequences of time series or image patches as individual tokens. Central to sequential prediction is the n -order Markov process, and a simple way to capture the n -order Markov process is n -gram language model. To predict next token w 0 , we need to compute p ( w 0 | w 1 , . . . , w n -1 ) , which can be further computed as p ( w 0 w 1 . . . w n -1 ) /p ( w 1 . . . w n -1 ) . Hence, the core of n -gram

(a) GPT2-backbone

(b) PatchTST

(c) Token similarity

1.0

GPT2-backbone(6 Layers)

0.9

PatchTST

0.8

0.8

0.6

0.6

F8°0

Average Similarity

0.4

0.4

0.7

0.2

0.2

0.0

0.0

0.6

1.0

1.0

80

80

0.5

00

0.6

00

0.6

0.2

0.4

0.2

0.4

0.4

0.4

0.6

0.2

0.4

0.6

0.2

0

100

200

300

400

500

600

700

80

10

0.0

80

1.0

0.0

Sequence Length

language model is to estimate the probability of observing a sequence of n tokens. When n is large, most of n token sequences will not be observed from data, leading to the sparse data problem, a common challenge faced by n -gram language model. As a result, a large body of research in n -gram language model is focused on how to effectively estimate probability of having n -token sequences even when they are NOT observed from data. We hypothesize that the transformer model pretrained by GPT-2 essentially allows us to estimate p ( w 0 w 1 . . . w n -1 ) from observations of significantly shorter token sequences. In this section, we will show that the function of estimating probabilities of longer sequences from observation of shorter sequences is universal and is independent from domain as long as data exhibit a skew distribution (e.g., follows a power law). We note that our work is closely related to the discussion presented in Elhage et al. (2021); Olsson et al. (2022), where the authors also connect the function of transformer to compute of n -grams. We however note that our key result is to show the universality in computing probability of longer sequences from observations of shorter sequences, which can't be found in any existing studies. Although the discussion is restricted to discrete tokens, it should be generalized to continuous signals as we can always quantize continuous signals into a finite number of discrete tokens, similar to what BEiT Bao et al. (2022) did.

To gain a better understanding, let's start by examining a "zero-layer" Transformer model. This model operates by taking a token, embedding it, and transforming it back to produce logits that predict the subsequent token. Because it cannot transfer information from other tokens, it relies solely on the current token to predict the next one. Consequently, the optimal behavior of this model is to closely resemble the bigram log-likelihood.

Then we move on to the so-called "attention-only" transformer, which doesn't have MLP layers. As discussed in a recent work Elhage et al. (2021), one-layer attention-only Transformers can be comprehended as a combination of a bigram model and multiple "skip-trigram" models (impacting the probabilities of sequences "A. . . BC"). This can be intuitively understood as each attention head having the ability to selectively attend from the current token ("B") to a previous token ("A") and transfer relevant information to fine-tune the probability of potential subsequent tokens ("C"). Olsson et al. (2022) further discusses a multi-layer transformer can do more complex n-gram estimation using an induction heads mechanism. To be more precise, induction heads employ a straightforward principle: the '[A][B] ... [A] → [B]' rule, which elevates the likelihood of generating the subsequent token 'B' given the current token 'A' if there is a fuzzy match of the AB bigram in the historical context. This rule seems to largely decouple A and B, which means they do not memorize a fixed table of n-gram statistics. The rule [A][B] . . . [A] → [B] applies regardless of what A and B are, which can abstract to new patterns.

Building upon these discussions, we are now prepared to substantiate the following argument: For sequential data following a power law, there is a potentially universal solution to the final estimation of n-gram probabilities . That's the reason behind the universality of pretrained LM's performance in cross-domain tasks. For simplicity, we assume that n is so large that we are unable to observe any occurrence of n -gram from data, and we only observe the occurrence of n ′ -grams with n ′ < n . We denote by s n i the i th unique n -gram, and by the notation s n ′ j ∈ s n i if n ′ -gram s n ′ j appears

in s n i , the i th n -gram. Let m n be the number of unique n -grams. According to the maximum entropy model, our estimation of n-gram probabilities can be cast into the following optimization problem:

min ∑ m n i =1 p ( s n i ) log p ( s n i ) s. t. ∑ i : s n ′ j ∈ s n i p ( s n i ) = ̂ p ( s n ′ j ) where ̂ p ( s n ′ j ) represents the probability of observing pattern s n ′ j from the data and j ∈ [ m n ′ ] , n ′ ∈ [ n -1] .

For each constraint for ̂ p ( s n ′ j ) , we introduce a Lagrangian dual variable λ n ′ j , and rewrite the optimization problem as follows:

In the case that all n-grams follow a power law, for each n ′ ∈ [ n -1] , we divide n ′ -gram into two groups: the group V n ′ includes the high frequency n ′ -gram and the group U n ′ including the low frequency of n ′ -gram. For simplicity, we assume that the probability for all the high frequency n ′ -grams are roughly α n ′ ∈ [0 , 1] and the probability for all the low frequency n ′ -grams are roughly β n ′ ∈ [0 , 1] . By assuming that all the patterns in V n ′ and U n ′ share similar appearance frequency, we simplify the optimization problem by only introducing two dual variables for each n ′ -gram, i.e. λ n ′ a for high-frequency patterns and λ n ′ b for low-frequency patterns as follow Using these notations, we have the optimization problem simplified as

Furthermore, let q n ′ a be the probability to observe a high frequency n ′ -gram appearing in any n -gram, and q n ′ b be the probability to observe a low frequency n ′ -gram appearing in any n -gram, we have

By skipping the term O ( √ m n ) , we further simplify the optimization problem as

which is equivalent to

λ n ′ a and λ n ′ b will only depend on statistics q n ′ a , q n ′ b , g n ′ and h n ′ . They are independent from the detailed statistics ̂ p ( s n ′ j ) and how each n ′ -gram appears in different n -gram. Thus, this simple analysis does indicate, to some degree, that the solution obtained from the maximum entropy model can be universal, as long as n -grams follow skewed distributions like power law.

We informally demonstrate that transformer models utilize attention mechanisms to perform a sophisticated form of n-gram estimation, and the generation rule for such n-gram distributions could be universal. This is how universality is achieved in our proposed cross-domain knowledge transfer. However, we currently lack a concrete metric to evaluate the performance of knowledge transfer between different domains, which requires further investigation. Nonetheless, in our experimental study, we demonstrate that a transformer model (beit) Bao et al. (2022) trained on images can perform well on cross-domain time series forecasting tasks.

G Connection between self-attention and Principle component analysis

Understand the Gradient Structure of Self-Attention

Let X = ( x 1 , . . . , x N ) ⊤ ∈ R N × D be the input pattern, and let f ( X ) = ( f 1 ( X ) , . . . , f N ( x )) ⊤ : R N × D ↦→ R N × D be the function for self-attention, i.e.

where A = W Q W ⊤ K ∈ R D × D . Let the Jacobian J = [ ∂f i ( X ) ∂x j ] N i,j =1 represent the gradient f ( X ) with respect to input pattern. The lemma below shows an important structure of J .

̸

Proof. According to the analysis from the work, we have the gradient J i,j = ∂f i ( X ) x j is given by J i,j = P i,j I + X ⊤ Q i ( XAδ i,j + E j,i XA ⊤ ) where Q i = diag ( P i, : ) -P i, : P ⊤ i, : Here P i, : ∈ R N + represents the i -th row of matrix P . We thus have

̸

As indicated by Lemma 1, one of the key components in the upper bound of Jacobian is | x i -∑ N j =1 P i,j x j | 2 . Thus, through the optimization, we like to reduce the size of the gradient and therefore may prefer to reduce the quantity to ∑ N i =1 | x i -∑ N j =1 P i,j x j | 2 . Hence, it will be interesting to understand the choice of W Q and W K that leads to the minimization of ∑ N i =1 | x i -∑ N j =1 P i,j x j | 2 , i.e. the following optimization problem min | A | F ≤ ρ ∑ N i =1 ∣ ∣ ∣ x i -∑ N j =1 P i,j x j ∣ ∣ ∣ 2 where ρ is introduced to control the size of A .

Connection between Self-Attention and Principal Component Analysis

Let consider the optimization problem in (G) when ρ is small, we can approximate P i,j as P i,j ≈ 1 N + 1 N x ⊤ i Ax j Define ¯ x = X ⊤ 1 /N . We have ∑ N i =1 | x i -X ⊤ P i, : | 2 = ∑ N i =1 ∣ ∣ x i -¯ x -X ⊤ XAx i ∣ ∣ 2 By assuming that all the input patterns are zero centralized, we have ¯ x = 0 and ∑ N i =1 | x i -X ⊤ XAx i | 2 = tr ( ( I -X ⊤ XA ) 2 X ⊤ X ) The theorem below shows that A minimizing the objective ∑ N i =1 | x i -X ⊤ XAx i | 2 contains the largest m eigenvectors of X ⊤ X where m is the rank of A .

Theorem 2 . Let W Q and W K be matrices of size D × m . Let λ 1 ≥ λ 2 ≥ ... ≥ λ D be the eigenvalues of X ⊤ X ranked in descending order, and let v i ∈ R D , i = 1 , . . . , D be the corresponding eigenvectors. The optimal solution A ∗ that minimizes ∑ N i =1 | x i -X ⊤ XAx i | 2 is given by A = ∑ m i =1 1 λ i v i v ⊤ i

Proof. Since W Q , W K ∈ R D × m where m < D , we know that A is a matrix of rank m . Hence, we know min A ∑ N i =1 | x i -X ⊤ XAx i | 2 ≥ ∑ N k = m +1 λ k We also know that by choosing A as A = ∑ m i =1 1 λ i v i v ⊤ i we have

Hence, the solution A for minimizing ∑ N i =1 | x i -X ⊤ XAx i | 2 is essential a weighted combination of top eigenvectors of X ⊤ X . Since a small gradient will prefer a small quantity of ∑ N i =1 | x i -X ⊤ XAx i | 2 , by minimizing through the self-attention layer, we essentially choose weight matrix W Q and W K to be aligned with the principal directions of X ⊤ X . ■

H Experiment Analysis and Other Key Results

H.1 Experiment analysis of GPT2-FPT model

In this section, we conduct experiments to analyze whether the self-attention frozen pre-trained model improves performance compared with overall fine-tuning and random initialization. Firstly, we compare GPT2(6) FPT with the same model without freezing (No Freeze) and random initial model (No Pre-train). For the end-to-end paradigm No Pre-train GPT2-backbone (6 Layers), we directly train all parameters of the model. We summarize the results in Table 21 and Table 22. Then we analyze the performance of various layers to clarify our selection of GPT2(6) FPT.

Fine-tune More Parameters Compared with fine-tuning all parameters, self-attention frozen pretrained model GPT2(6) FPT achieves better performance on most datasets and yields an overall 12.7% relative MSE reduction on 5% data and 11.5% relative MSE reduction on 10% data. It verifies that frozen pre-trained attention layers are effective for time series forecasting.

Parameters Initialization Compared with the random initial model, self-attention frozen pre-trained model GPT2(6) FPT achieves better performance on most datasets and yields an overall 21.2% relative MSE reduction on 5% data and 14.3% relative MSE reduction on 10% data. It again suggests that a model pre-trained on cross-domain data can achieve significant performance improvement in time series forecasting.

The Number of GPT2 Layers For most transformer-based methods in time-series forecasting Zhou et al. (2022); Wu et al. (2021); Nie et al. (2022), no more than 3 encoder layers are included. However, most pre-trained models with at least 12 layers may suffer from overfitting in time series forecasting. To better balance performance and computational efficiency, we test using various numbers of layers on ETTh2. Additionally, we train a completely random initialized non-pretrained GPT2 as a comparison. The results are shown in Figure 9, for both 5% and 10% data, the pre-trained model is unable to do well with few layers but significantly outperforms non-pre-trained GPT2 with more attention blocks transferred from NLP. It indicates that pre-trained attention layers produce

a great benefit in time series forecasting. Also, the pre-trained model achieves better performance between 3 and 9 layers. Thus GPT2 with 6 layers is chosen as our default architecture.

5%ETTh2

10% ETTh2

0.9

0.9

ETTh2_96_GPT2

ETTh2_96_GPT2

ETTh2_192_GPT2

ETTh2_192_GPT2

0.8

ETTh2_96_GPT2_nopretrain

0.8

ETTh2_96_GPT2_nopretrain

ETTh2_192_GPT2_nopretrain

ETTh2_192_GPT2_nopretrain

0.7

0.7

MSE

0.6

0.6

0.5

0.5

0.4

0.4

0

1

3

6

9

12

0

1

3

6

9

12

Number of Layers

NumberofLayers

H.2 No Pre-training but Freezing

For comprehensively ablation on pre-training and freezing strategies, we also add experiment for random initialized GPT2(6) with freezing. The results in Table 23 shows that only input and output modules can not work and pre-trained knowledge play an importance part in time series tasks.

H.3 Fine-Tuning Parameters Selection

In this section, we conduct ablation experiments to study which parameters are important to fine-tune. Since the input embedding and output layers are randomly initialized for adapting to a new domain, they must be trained. Then, we study adding layer normalization and positional embeddings to the list of fine-tuning parameters. Table 24 shows the results that re-train parameters of layer normalization and positional embeddings can bring certain benefits, especially in longer prediction lengths. Thus, we follow the standard practice to re-train positional embeddings and layer normalization.

H.4 Analysis of Data Volume

Results of few-shot learning show that GPT2(6) FPT shows SOTA performance in few-shot learning tasks in which the model is trained on 5% data and 10% data. Plus, it has comparable performance with the SOTA baselines PatchTST and Dlinear on full sample forecasting setting as well. This phenomenon raises a question that how performance changes with an increase in data sample size. Thus, we conduct experiments on various percentages P ∈ { 5% , 10% , 20% , 50% , 80% , 100% } of ETTh2. Figure 10 shows that the performance improvement for GPT2(6) FPT is almost flattened. These results illustrate that such a cross-domain FPT model is extremely efficient in few-shot time series forecasting and only requires a few fine-tuning samples to reach a SOTA performance. For more complete data, end-to-end training models start to catch up, but still, a GPT2(6) FPT model can be comparable to those SOTA end-to-end training algorithms.

H.5 Knowledge transfer with other Pre-trained Transformer Models

We investigate how other pre-trained transformer models perform and whether other domains can also help. Another NLP pre-trained model BERT Devlin et al. (2019) and the CV pre-trained model BEiT Bao et al. (2022) are trained on 5% ETTh2 and 5% ETTm2. Similar to GPT2, we only reserve 6 layers and freeze attention blocks. Our results are shown in Table 25 that BERT(6) FPT and BEiT(6) FPT are comparable to PatchTST and remarkably surpass other baselines. We come to the conclusion that the universality of our proposed architecture holds across other pre-trained-transformer models. Moreover, the domain of successful knowledge transfer in time series forecasting is not limited to natural language. Knowledge from the CV domain can also help, supported by BEiT's experimental results.

H.6 Full Results of Classification

H.7 Full Results of Anomaly Detection

H.8 Full Results of Imputation

H.9 Full Results of Short-term Forecasting

H.10 Input Length Setting Discussion

The consideration of input length is of great importance. It is widely believed that longer input lengths have the potential to generate superior results. However, in practice, certain algorithms might fall short in effectively utilizing long input signals due to overfitting issues either. Here, we conducted long-term forecasting experiments by comparing our approach with the best reported values from different baseline papers. This was done to avoid any biases stemming from selective tuning of baseline parameters. While some may argue in favor of a fairer comparison using a fixed input length,

* We reproduce the results of TimesNet by https://github.com/thuml/Time-Series-Library.

** We replace the joint criterion in Anomaly Transformer with reconstruction error for fair comparison.

we are starting to shift our focus towards pursuing more accurate algorithms that possess enhanced capabilities for handling longer inputs. Instead of restricting the input length to a fixed small value, it is pragmatic to tune both the input length and model parameters based on performance, as it is often the primary concern in practical usage. Exploring the utilization of extremely long inputs, such as Chatgpt or LLM, is among our future research directions.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors suggest leveraging pre-trained large language models, specifically utilizing a frozen Transformer backbone, for comprehensive time series analysis. The unique method involves fine-tuning only the input embeddings of time series data and the output layers. The authors substantiate their proposal with large-scale experiments across multiple benchmarks, demonstrating that their approach consistently outperforms various established baseline methods in diverse time series analysis tasks.\n\nSTRENGTHS:\n1. The paper's proposed method impresses with its simplicity, efficacy, and clear motivation. Despite not introducing any complex modules or additional hyperparameters to the existing model, it consistently outperforms more intricate techniques.\n\n2. The experiments undertaken within the paper are robust and comprehensive. The authors have tested the proposed method across a broad spectrum of time series analysis aspects, such as long-term and short-term forecasting, and abnormality detection. In each scenario, the method demonstrates superior performance when compared with multiple strong baselines.\n\n3. The insightful correlation drawn between self-attention and Principal Component Analysis (PCA) elucidates why the proposed method performs remarkably well. This strengthens the credibility of the results, making them more convincing.\n\nWEAKNESSES:\n1. The paper neglects to detail the computational cost differences between the proposed method and alternative approaches. Given that a pre-trained GPT-2 backbone is potentially substantial in size, the computational expense of the presented method might significantly outweigh that of other baseline methods.\n\n2. Although the proposed method seems compatible with various autoregressive Transformers, the experiments rely exclusively on GPT-2. It would be beneficial to extend the experimental scope to encompass additional models such as DeBERTa, GPT-J, OPT, among others.\n\n3. The paper's overall presentation requires improvement. Tables appear congested, and font sizes within figures may be too small for legible printing. Additionally, there is an inconsistent distribution of space on some pages, and many section titles are overly lengthy. This compromises readability and detracts from the overall impact of the findings.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that fine-tuning language models along with some task specific layers, e.g. input and position embedding and lay norms, can achieve comparable and sate-of-the-art performance on various time series tasks, including forecasting, imputation, anomaly detection and classification tasks.\n\nSTRENGTHS:\n(1) The method for time series analysis is simple and can work for different time series analysis tasks.\n\n(2) Results are comparable or state-of-the-art in different benchmark datasets and time series analysis tasks.\n\nWEAKNESSES:\n(1) This work mainly conduct experiments on simple benchmark datasets. Many real-word datasets have complicated dynamics and large noise and the prior knowledge within pre-train models do not necessarily align with such dynamics. If the success of using pre-train models are due to the prior knowledge pre-train models own, what are results if the time series data does not align with such prior knowledge? This paper seems not to have deep discussion regarding this matter.\n(2) It is unclear why GPT-2 (6) perform better than the full-layer fine-tuning baseline. Deep layers often tend to learn global information within the data. It is not clear why authors use GPT-2 (6). Authors lack of deep discussion regarding this matter.\n\nQUESTIONS:\nSee Weaknesses.\n\nMissing references:\n\n(1) LogTransformer was proposed in [1]. Results of LogTransformer was in this paper but [1] was not cited.\n(2) Using pre-trained transformers to model time series is not a new idea. [2] uses pre-train LMs to conduct time series forecasting and [3] uses Vision Transformer to model irregularly sampled time series. However, this paper does not discussion this two work and add them as baselines in appropriate tasks.\n\n[1] Li et al. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. NeurIPS 2019.\n[2] Xue and Salim. PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting. \n[3] Li et al. Time Series as Images: Vision Transformer for Irregularly Sampled Time Series.\n\nLIMITATIONS:\nSee Weaknesses and Questions section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors discuss a method by which a deep transformer model pre-trained on an NLP or CV task can be adapted to a wide variety of applications involving classification or prediction of scalar time series data. This includes an embedding scheme to represent the time series input in the same embedding space as expected by the transformer layers and a fine-tuning approach whereby the multi-head attention and feedforward components of the transformer blocks are frozen, but the embedding and layer norm components are trained. Using the GPT-2 backbone, they apply this method to classification, anomaly detection, imputation, and various forecasting tasks, showing generally good performance compared to many competitors. They\n\nSTRENGTHS:\nThe problem of finding a broadly performant time series model is interesting and challenging, particularly in light of the diverse architectures proposed and mixed results obtained by deep neural networks for time series applications. The idea of simply using a pre-trained language model with minimal adaptation is surprising and worth discussion in the scientific community. The empirical analysis attempts to cover a very broad range of tasks and datasets, which is an appropriately high bar for a paper that proposes a “one fits all” approach to time series analysis.\n\nWEAKNESSES:\n* The time series applications I know best here are anomaly detection and long-range forecasting, and for both of these settings there are some issues with the reported results. \n     - For anomaly detection, there is significant information missing. How are binary anomaly decisions made from the reconstruction error? How are precision and recall computed when anomalies are given by windows rather than points (e.g. SMAP, MSL)? These decisions can have a strong influence on the reported results.\n     - While comparisons are extensive for anomaly detection, they do not seem to actually capture the state of the art. For example, the OmniAnomaly method [1], which also naturally generalizes to multivariate time series, significantly outperforms the GPT2(6) model in F1 score on all datasets for which both are evaluated (SMD, MSL, SMAP).\n    - The long-range forecasting results are likewise missing comparisons to methods that significantly outperform all reported scores. For example, the S4 model [2] reports lower MSE and MAE on Weather, ETTh1, ETTh2, and ETTm1 (see Table 13, [2]).\n    - Moreover, the results reported for Informer in [2] are significantly better than what is reported in Table 13 of the present paper, beating GPT2(6) on ETTh1 and ETTh2. It is unclear what the source of this discrepancy is, but it is large enough to be relevant when drawing conclusions across methods.\n\n* The idea that a lightly-adapted NLP or CV transformer backbone is a highly competitive, highly general time series model is mildly shocking, at least to a time series practitioner. This paper largely misses the opportunity to explore *why* this might be the case, and *what* are the main drivers of this result. For example, do the pre-trained weights actually matter, or is it the architecture? Is fine-tuning required, and if so, why is fine-tuning only the embeddings and layer norm components sufficient? As it stands, the paper communicates an interesting empirical result without providing much insight as to its explanation.\n\n* The specific adaptations of the FPT approach - i.e. the time series encoder and fine-tuning protocol - are both inadequately described. The encoder operations should be described in mathematical detail. The algorithmic details of fine-tuning should at least be provided in the supplement.\n\n* The PCA analysis seems largely unrelated to the preceding work in the paper. It seems to be a general observation about self-attention, without any particular connection to the specific time series applications considered. Moreover, PCA itself is certainly not a one-size-fits-all solution to time series modeling, so it cannot be the basis for a convincing explanation of the model’s success.\n\n[1] Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., & Pei, D. (2019). Robust anomaly detection for multivariate time series through stochastic recurrent neural network. KDD.\n\n[2] Gu, A., Goel, K., & Re, C. (2021). Efficiently modeling long sequences with structured state spaces. ICLR.\n\nQUESTIONS:\nCould the authors comment on the discrepancy between Informer performance reported in their Table 13 vs. in [2]? What could explain the substantial difference in metrics on the same task? I certainly don’t mean to claim that [2] must be right, but the size of the difference is enough to affect the interpretation of the results, so it merits some discussion. [Addressed in discussion period]\n\nCould the authors share their perspective on *why* their FPT approach seems to work so well, and what are the main drivers of this result? \n\nWhat unique insights for time series representation are provided by the PCA analysis in Section 7? \n\nWhy not report the results in Figure 1 as a table? They would be much easier to read and compare.\n\nLIMITATIONS:\nThere is no direct potential for negative social impact. Potential limitations for the method and results are either discussed or covered above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper presents a unified framework for time-series tasks using pre-trained language models. The author make a united model through a fine-tuning approach that focuses on fine-tuning specific parts of the pre-trained language model, rather than the entire set of parameters. As a result of this adaptation, the proposed fine-tuned model achieves comparable performance to previous methods across various datasets. Experimental results also demonstrate the versatility of the fine-tuned model by successfully transferring knowledge from pre-trained datasets, such as images and text.

STRENGTHS:
* The proposed framework surpasses existing methods.
* The paper is easily comprehensible and straightforward.
* The evaluation of various time-series tasks highlights the advantages of the proposed approach in numerous cases.

WEAKNESSES:
* Novelty: The proposed fine-tuning approach appears to be more of an incremental improvement. Fine-tuning pre-trained language models is not entirely new. 
* Concerns regarding practicality: Does the architecture based on pre-trained language models require more computational resources compared to other models? It would be helpful to analyze the computation cost for inference in fair comparison settings, such as using the same number of parameters, to demonstrate the effectiveness of the proposed approach.

QUESTIONS:
* Why did the proposed approach fail to demonstrate good performance in the zero-shot task, unlike in the other tasks?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to use pretrained Transformer for time series analysis. \nThey freeze the attention and FFN layers but fine-tune the position embeddings and add-norm modules to adapt the model to a given task. \nThis results demonstrate that this method leads to strong performances in time series analysis tasks. \nThe authors proved a theorem to explain why the text-based pretrained model could generalize well to time series analysis, by drawing insights from PCA.\n\nSTRENGTHS:\nThe paper presents very interesting theoretical and empirical findings. Their extensive experiments show that a pretrained text-based transformer model could be adapted to time series analysis and achieve strong performances. I think the findings are worth presenting to the ML community. \n\nTheir theoretical analysis explains this by connecting self-attention mechanism with PCA, for which I have concerns/questions as expressed in Weakness. \n\nThe paper is overall clearly written.\n\nWEAKNESSES:\nI didn't give a high soundness score because the paper hasn't well justified the choice of tuning add-norm layers and positional embeddings. Line-117 claims that it \"is considered a standard practice\" but I am afraid it is not true. \nParameter-efficient adaptation has been a hot topic and there have been various kinds of methods, such as adding feed-forward adapters (Houlsby et al. ICML 2019) or attentional adapters (Zhao et al. EMNLP 2022), tuning bias terms (Zaken et al. ACL 2022), LoRA (Hu et al. ICLR 2022), and prefix tuning (Li and Liang ACL 2021, Qin and Eisner NAACL 2021). \nThere hasn't been a \"standard\". \n\nI think the paper can be improved with experimental analysis on different kinds of LM adaptation methods, which will complement the current results. \n\nMore important, the choice of tuning parameters is related to the theorem in the paper: the PCA insights are drawn for the self-attention mechanism, making me wonder: does the choice of tuning parameters lead to the analysis of the self-attention? Or does the analysis of self-attention lead to the choice of tuning parameters? Or neither? Will results with any other adaptation methods give you different kinds of interpretation? \n\nThe mentioned references are: \n\nhttps://arxiv.org/abs/1902.00751 (already cited)\n\nhttps://arxiv.org/abs/2211.01979 \n\nhttps://arxiv.org/abs/2106.10199\n\nhttps://openreview.net/forum?id=nZeVKeeFYf9\n\nhttps://arxiv.org/abs/2101.00190\n\nhttps://aclanthology.org/2021.naacl-main.410/\n\nAnother line of work that this paper should discuss is the NLP literature that also finds the \"per-layer high cosine\" phenomenon. \n\nThis phenomenon is first discussed for skip-gram embeddings: https://aclanthology.org/D17-1308.pdf \n\nThen it is also found in Transformers: https://arxiv.org/pdf/1909.00512.pdf \n\nThere is also argument about why cosine metric may not mean that much: https://aclanthology.org/2022.acl-short.45/\n\nThe above is to only name a few and there are other papers that one can find by tracing their citation relations.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}