TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting
Authors: Defu Cao
Furong Jia
Sercan O Arik
Tomas Pfister
Yixiang Zheng
Wen Ye
Yan Liu
Venue: ICLR 2024 poster (2024)
Keywords: Forecasting; Time Series; Large Language Model
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This work propose a generative pre-trained transformer (GPT) for time series forecasting.

Abstract: The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the-art results, the best-performing architectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demonstrated impressive performance via training one general-purpose model across various textual datasets. It is intriguing to explore whether GPT-type architectures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components; and (ii) introducing the design of prompts to facilitate distribution adaptation in different types of time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over state-of-the-art methods on zero shot setting for a number of time series benchmark datasets. This performance gain is observed not only in scenarios involving previously unseen datasets but also in scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute a foundational model-building framework.

Published as a conference paper at ICLR 2024
TEMPO: PROMPT-BASED GENERATIVE PRE-TRAINED TRANSFORMER FOR TIME SERIES FORECASTING
Defu Cao', Furong Jia', Sercan Ö. Arik,, Tomas Pfister", Yixiang Zheng', Wen Ye', Yan Liu
1 University of Southern California
2 Google Cloud Al Research
fucac, florajia, yixlangzheng, yewen, yanliu.esJeusc.ed (soarik, spfister/8googio.o
ABSTRACT
The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the results, the best-performing ar- chitectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demon- strated impressive performance via training one general-purposs. model across various textual datasets. It is intriguing to explore whether GPT-type architec- tures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components: and (ii) introducing the design of prompts facilitate distribution adaptation in different types of time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over -tate-of-the-art methods on zero shot setting for a number of time series benchmark datasets. This performance gain is observed not only in scenarios involving previously unseen datasets but also scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute foundational model-building framework.
INTRODUCTION
Time series forecasting, i.e., predicting future data based on historical observations, has broad real- world upplications, such as health, transportation, finance and so In the past decade, numerous deep neural network architectures have been applied to time series modeling, including convolutional neural networks (CNN) (Bai et al., 2018), recurrent neural networks (RNN) (Siami-Namini et al., 2018), graph neural networks (GNN) (Li et al., 2018; Cao et al., 2021), and Transformers (Liu et al., 2021; Zhou et al., 2021: Wu et al., 2023; Zhou al., 2022; Woo et al., 2022; Kitaev et al., 2020; Nie et al., 2023), leading to state-of-the-arts results. While achieving strong prediction performance, some of the previous works on time series mostly benefit from the advance in sequence modeling (from RNN and GNN, to transformers) that captures temporal dependencies but have not fully capitalized on the benefits of intricate patterns within time series data, such as seasonality, trend, and residual. These components are the key differentiating factors of time series from classical sequence data architectures might
(Fildes et al., 1991). As result, recent studies suggest that deep learning- not be robust as previously thought and might even be outperformed by shallow neural networks or even linear models on some benchmarks (Zeng et al., 2023; Zhang et al., 2022b; Wu et al., 2023; Ekambaram et al., 2023; Fan et al., 2022). Despite the notable success of deep learning forecasters, the vast majority of them still follow a conventional training mechanism, training and predicting using the same datasets.
Meanwhile, the rise of foundation models in natural language processing (NLP) and computer vision (CV), such as LLaMA (Touvron et al., 2023), CLIP (Radford et al., 2021) and ChatGPT, marks

---

Published as a conference paper at ICLR 2024
major milestones effective representation learning. It is extremely intriguing to explore a pre- trained path for foundation time series models with vast amounts of data, facilitating performance improvement in downstream tasks. Some recent works shed light into the possibility of building general transformers for time series (Zhou et al., 2023; Sun et al., 2023; Goswami et al., 2024; Das al., 2023b; Rasul al., 2023). However, the theoretical and practical understanding of such models has not reached the consensus observed in other domains where generative models have been widely acknowledged (Garza & Seerenttaler-Canseee 2023). In addition, prompting techniques in LLM (such as InstructGPT (Ouyang et al., 2022)) provide way to leverage the model's existing representations during pre-training instead of requiring learning from scratch. However, existing backbone structures and prompt techniques in language models do not fully capture the evolution of temporal patterns as N-BEATS (Oreshkin et al., 2019) and AutoFormer (Wu et al., 2021), which are fundamental for time series modeling.
In this paper, we make an attempt to address the timely challenges of adapting large pre-trained models for time series forecasting tasks and developing prompt-based generative pre-training transformer for time series, namely TEMPO. TEMPO consists of two key analytical components for effective time series representation learning: and seasonality, and the other concentrates on obtaining more universal and transferrable insights from the inherent properties of data through a prompt-based approach. Specifically, TEMPO firstly decomposes time series input into three additive components, i.e., trend, seasonality, and residuals via locally weighted scatterplot smoothing (Cleveland al., 1990). Each of these temporal inputs is subsequently mapped to its corresponding hidden space to construct the time series input embedding of the generative pre-trained transformer (GPT). We conduct a formal analysis, bridging the time series domain with the frequency domain, to highlight the necessity of decomposing such components for time series analysis. In addition, we theoretically reveal that the attention mechanism hard to achieve the decomposition automatically. Second, TEMPO utilizes a soft prompt to efficiently tune the GPT (Radford et al., 2019) for forecasting tasks by guiding the reuse of a collection of learnable continuous vector representations that encode temporal knowledge of trend and seasonality. In addition, we leverage the three key additive components of time series data-trente trend, scasonality, and residuals- to provide input components (Hastie, 2017). Experiment results on zero shot setting and multimodal setting of TEMPO pave the path to foundational models for time series. Besides, we demonstrate the stable predictive power of our model on unseen samples with textual information on two multimodal datasets including TETS (Text for Time Series) dataset, which is first introduced in this work to foster further research topics of pre-trained time series models.
focuses modeling specific time series patterns, such trends
interpretable framework for comprehending the interactions among
In summary, the main contributions of our paper include: (1) We introduce an interpretable prompt- maning-based generative transformer, TEMPO, for time series representation learning. It further drives a paradigm shift in time series forecasting from conventional deep learning methods to pre-trained foundational models. (2) We adapt pre-trained models for time series by focusing on two fundamental inductive biases: First, we utilize decomposed trend, seasonality, and residual information. Second, we explore the soft prompt strategies to accommodate time series data's 's dynamic nature. (3) Through extensive experimentation on benchmark datasets and two multimodal datasets, our model demonstrates superior performance. Notably, our robust results towards highlights the potential of foundational models in the realm of time series forecasting.
RELATED WORKS
Pre-trained Large Language Models for Time Series. The recent development of Large Language Models (LLMs) has opened up new possibilities for time-series modeling. LLMs, such as T5 (Raffel et al., 2020). GPT (Radford et al., 2018), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), LLaMA (Touvron et al., 2023), have demonstrated a strong ability to understand complex dependencies of heterogeneous textual data and provide reasonable generations. Recently, there is growing interest in applying language models to time series tasks (Jin et al., 2024a; Gruver et al., 2024). For example, Xue & Salim naively convert time series data to text sequence inputs and achieves encouraging results. Sun et al. propose text prototype aligend embedding to enable LLMs handle time series data. In addition. Yu et al. present an innovative approach towards leveraging LLMs for explainable financial time series forecasting. The works in (Zhou et al., 2023) and (Chang et al., 2023) are the most relevant ones our work, they both introduce approaches for
2

---

Published as a conference paper at ICLR 2024
A
Thand
Attention
Protrally Hlcck
Seesorel
WV
Layer
Brakkel
Pry- Pre-talved Parameterss from Damerative Pro-
Nat--tad-hterris
Hidden States
Prompt
Noomalkee
fimbeiding
Thind
Pandiing
Transformer
2
(ee-Normalizee
Pronpt
MUUN Seasomet
X
De Ce--ormalize
Prediction
oine servic irput
Patching 15
xi M.
%
Frebedding
Prompt
De-Hormalize
Hormalked
Embeedinng
%
Unear
The architecture of proposed TEMPO-GPP The trend Xr, scasonal X and residual X components are treated as different semantic inductive biases to feed into the pre-trained transformer.
Figure 1:
time-serics analysis by strategically leveraging and ine-tuning LLMs. However, these studies directly employ time series data to construct embeddings, without adequately capturing the inherent and unqiue characteristics of time series data which is challenging to decouple such information within the LLMs (Shin al., 2020). In addition, there is still very limited work LLM for multimodal data with time series. METS (Li et al., 2023) is one of the early works pursuing this direction. While the experiment results are encouraging, is difficult extend METS to other modalities since the embedding alignment between time series and texts are specific. Please refer to the suvery papers (Jin et al., 2023; 2024b) for further references time series meeting LLMs.
Prompt tuning. Prompt tuning is an efficient, low-cost way of adapting a pre-trained foundation model new downstream tasks which has been adapted downstream tasks across various domains. In NLP domain, soft prompts with trainable representation are used through prompt-tuning (Lester al., 2021) or prefix-tur tatfi--tuinngg (Li & Liang, 2021). Prompting techniques have also been extended CV tasks like object detection(Li et al., 2022) and image captioning (Zhang al., 2022a), etc and other domains such as misinformation (Zhang al., 2024). Multimodal works, such as CLIP (Radford al., 2021), use textual prompts to perform image classification and achieve SOTA performance. In addition, L2P (Wang et al., 2022b) demonstrates the potential learnable prompts stored in a shared pool enable continual learning without rehearsal buffer, and Dualprompt (Wang et al., 2022a) introduces a dual-space prompt architecture, maintaining separate prompt encodings for general knowledge and expert information, etc. Our research builds upon these concepts by exploring the use of prompt design from indicative bias specifically for temporal reasoning and knowledge sharing across time series forecasting problems.
METHODOLOGY
In our work, we adopt a hybrid approach that incorporates the robustness of statistical time series analysis with the adaptability of data-driven methods. As shown in Figure 1, we propose novel integration of seasonal and trend decomposition from STL (Cleveland et al., 1990) into the pre-trained transformers. This strategy allows us to exploit the unique strengths of both statistical and machine learning methods, enhancing our model's capacity to handle time series data efficiently. Moreover, a semi-soft prompting approach is introduced to enhance the adaptability of pre-trained models for handling time series data. This innovative approach enables the models to merge their extensive learned knowledge with the unique requirements intrinsic to time series analysis.
3.1 PROBLEM DEFINITION
Given observed values of previous K timestamps, the task of multivariate time-series forecasting aims predict the values for the next H timestamps. That is,
X
F(x)
V
(1)
3

---

Published as a conference paper at ICLR 2024
where .i., 11- 1 is the vector of -tepp estimation from timestamp of channel i corresponding to the i-th feature. Given the historical values x x.-K x;-,,, 1- parameter & and prompt V In anticipation of the foundational model's strong generalization capabillities across unseen datasets, we default to a zero-shot learning configuration in the absence of specific indications. This approach entails that the model not privy to the target dataset's history value and horizon value during the training process.
can be inferred by model F with
3.2 TIME SERIES INPUT REPRESENTATION
For time series data, representing the complex input by decomposing it into meaningful components, such as trend and season components, can help extract information optimally. In this paper, given the input X E _ where is the feature (channel) size and L it the length of the time series, the additive STL decomposition (Cleveland et al., 1990) can be represented as:
X = X + X +
(2)
Here, i is the channel index (corresponding to a certain covariate) for multivariate time series input, and the trend Xr E I J. Σ} X captures the underlying long-term pattern in the data, where m = 2k + and k is the averaging step size. The seasonal component Xs R encapsulates the repeating short-term cycles, which can be estimated after removing the trend component. The residual component Xa Rx represents the remainder of the data after the trend and seasonality have been extracted. Note that, in practice, it is suggested to leverage as much information as possible to achieve a more precise decomposition. However, in consideration of computational efficiency, we opt not to use the STL decomposition on the largest possible data window on each instance. Instead, we perform local decomposition within each instance using a fixed window size. Inspired by N-BEATs (Oreshkin et al., 2019), we introduce learnable parameters for estimating the various local decomposition components. Same for the others. This principle applies to other components of the model as well. In Appendix G, we establish connection between time series forecasting and frequency domain prediction, where our findings indicate that decomposition significantly simplifies the prediction process. Note that such decomposition is of more importance in current transformer-hhe based methods as the attention mechanism, in theory, may not disentangle the disorthogonal trend and season signals automatically:
Theorem 3.1 Suppose that we have time series signal X = XTe + + X t E tal. Let E (ei, e2...., denote a set of orthogonal bases. Let Es C E denote the subset of E on which Xst has non-zero eigenvalues and E, E denote the subset of E on which XT: has non-zero eigenvalues. If Stt and XTt are not orthogonal, i.e. Eim1 Xi,X's + O, then Er n Es 4 0. i.e. E can not disentangle the signals onto two disjoint sets of buses.
The proof can be found in Appendix G. Theorem 3.1 states that if trend and seasonal components of a time series are non-orthogonal, they cannot be fully disentangled and separated by any set of orthogonal bases. According (Zhou al., 2023), the self-attention layer naturally learns an orthogonal transformation, akin to PCA's decomposition into orthogonal principal components. Thus, applying attention directly to a raw time series would be ineffective at disentangling non-orthogonal trend and seasonal components. For the remainder of the methodology section, we will utilize the trend component Xy as the exemplary case. We first apply reverse instance normalization (Kim et al., 2022) on each global component and local input respectively to facilitate knowledge transfer and minimize losses introduced by distribution shifts. That is, for each sample IT; from X -th channel
of time t, f2 = 7T' (Tr E, Vrrre//Vaan [ære] + cr + Br, where E, [zre] and Var [=/v]] are the instance-specific mean and standard deviation; Yr and B3r are trainable affine parameter vectors for trend component. In addition, we implement a mean square error (MSE) reconstruction loss function to ensure that the local decomposition aligns with the global STL decomposition observed in the training data. The decomposition loss function, denoted as T. fx 0r) X². where fr is the function with learnable variables Or for mapping local dccomposition to be aligned with the global decomposition after normalization X... Then, following (Nie et al., 2023), we combine time-series patching with temporal encoding to extract local semantics by aggregating adjacent time steps into tokens, significantly increasing the historical horizon while reducing redundancy. Specifically, we get the patched token for the i-th normalized trend component for fr (X,, 0r) with
4

---

Published as a conference paper at ICLR 2024
Pf E RRLPN,, where Lp is the patch length, N the stride. We get patched tokens Ps and PR in the same way. Then, we feed the patched time series tokens to the embedding layer f to get the representation Pj = f(1%) E PRxx for the language model architecture to transfer its language capabilities to the novel sequential modality effectively, where Lg is the embedding size.
[1-=,p) [T. kpp
+ is the number of patches and is
8
3.3 PROMPT DESIGN
Prompting techniques have demonstrated remarkable effectiveness across a wide range of applications by leveraging the power of task task-specific knowledge encoded within carefully crafted prompts. This success can be attributed to the prompts' ability to provide a structured framework that aligns the model's outputs with the desired objectives, resulting in enhanced accuracy, coherence, and overall quality of the generated content. Previous works mostly focus on utilizing fixed prompt to boost the pre-trained models' performance through fine-tuning (Brown et al., 2020). In pursuit of leveraging the rich semantic information encapsulated within various time series components, our research introduces a semi-soft prompting strategy. This approach involves the generation of distinct prompts corresponding to each primary time series component: trend, seasonality, and residuals. 'Predict the future time step given the [trend, season, residual]' serves as the template from which we derive our component-specifee prompts. These are subsequently concatenated with the relevant component data, thereby enabling a more refined modeling approach that acknowledges the multifaceted nature of time series data. Specifically, commence by translating the trend-specine prompts into the word embedding space, followed by a linear transformation to derive the learnable trend prompt vector Vi- This so-called 'semi-soft' prompt design thus strikes balance between the interpretability and initial guidance of a 'hard' prompt and the adaptability of a 'soft' prompt. The combined embedding of this prompt with the time series representation encapsulated by:
FT [V:; Pr]
Here, ø7 denotes the aggregation of embeddings along the temporal axis. This concatenation procedure is mirrored for the seasonality and residual components, yielding &, and æn respectively, This framework allows for an instance be associated with specific prompts as the inductive bias, jointly encoding critical information relevant to the forecasting task, such as recurring patterns, overarching trends, and inherent seasonality effects. It is of note that our prompt design maintains a high degree of adaptability, ensuring compatibility with a broad spectrum of time series analyses. In particular, similar with (Wang et al., 2022a), we introduce prompt pool as an extension of our design of soft prompt in Appendix D, aimed at accommodating the characteristically non-stationary nature of real-world time series data and the associated distributional shifts (Huang et al., 2020; Fan et al., 2023). This adaptability underscores the potential of our prompting strategy to evolve in congruence with the complexities presented by diverse time series datasets.
3.4 GENERATIVE PRE-TRAING TRANSFORMER ARCHITECTURE
We use the decodeerbbsee generative pre-trained transformer (GPT) as the backbone to build the basis for the time-series representations. To utilize the decomposed semantic information in data-efficient way, we choose to concatenate the prompt and different components together and put them into the GPT block. Specifically, the input of our time series embedding can be formulated as: x = æT + & + æR. where @ corresponds to concatenate operation and x. can be treated as different sentences. Note that, another alternative way to build separate GPT blocks handle different types of time series components. Inside the GPT block, we adopt the strategy used in (Zhou et al., 2023) and opt to update the gradients of the position embedding layer and layer normalization layers. In addition, we employ LORA (Low-Rank Adaptation) (Hu et al., 2021) to adapt to varying time series distributions efficiently as it performs adaptation with significantly fewer parameters.
The overall forecasting result should be additive combination of the individual component pre- dictions. Finally, the outputs of n features from the GPT block can be split into Zr. Zs, ZR E p (output corresponding to trend, seasonality, and residual) based on their positions in the input order. Each Z component is then fed into fully connected layers to generate predictions Y. E x where LH is the prediction length. The forecast results can be formulated as follows: Y = Yr + Yss + Yzz. After that, de-normalize Y according to the corresponding statistics used in
5

---

Published as a conference paper at ICLR 2024
+ E. [=]]. By recombining these additive elements, our approach aims to reconstruct the full temporal trajectory most representative of the underlying dynamics across varied timescales captured by the decomposed input representation.
the normalization step: Y
/ Var [x]] \===
te-
In order to achieve interpretability, we explore both linear and nonlinear interactions among trend, seasonal, and residual components in their contribution to the final output. Therefore we construct an interpretable generalized additive model (GAM) (Hastie, 2017) based on GPT's output to learn how the three components interact with each other, which is: g Fg + Fx,) E, Fz, (4z, ) where Fg is a normalizing constant, the footnote corresponds to the trend, season, and residual component. {I,} } is of set of multiple interact components. Then, we can calculate the first-order sensitivity index (Sobol', 1990) or SHAP (SHapley Additive exPlanations) value (Lundberg & Lee, 2017) to measure the sensitivity of each component.
EXPERIMENTS
Our experiments conducted using widely-reoognized time series benchmark datasets, such as those detailed in (Zhou et al., 2021), alongside the GDELT dataset (Jia et al., 2024) and our proposed TETS dataset. These comprehensive datasets encompass diverse array of domains, including, but not limited to, electricity (ETThI, ETTh2, ETTml, ETTm2, Electricity), traffic (Traffic), climate (Weather), news (GDELT), and finance (TETS), with data sampling frequencies ranging from minutes, hours to days and quarters. The inclusion of such varied datasets ensures a thorough evaluation of our experimental setups across multiple dimensions of time series data. Due the absence of a standard test split for zero-shot comparison, we adopt a uniform training methodology to ensure fair performance assessment across datasets unseen during model training. Specifically, to advance the paradigm of foundation models within the domain of transfer learning, we investigate a zero-shot setting for our experiments, which is the 'many-to-one' scenario: training on multiple source datasets followed by zero-shot forecasting on distinct, unseen target dataset. For instance, when evaluating performance on a 'weather' dataset, our model is pre- pre-trained on diverse datasets including 'ETTm1, ETTm2, ETTh1, ETTh2, Electricity, and Traffic' without exposure to the target weather data. This mann-to-one' approach differs fundamentally from 'one-to-one" or 'one-to-many" configurations (Zhang et al., 2022c) by using diverse pre-training datasets from varied domains, like traffic and weather data. This diversity, while rich, introduces complexity, as the model must identify patterns across potentially misaligned samples, complicating learning compared to models trained and tested on in distribution datasets.
We use GPT-2 (Radford et al., 2019) as our backbone to build TEMPO as shown in Figure 1. To comprehensively demonstrate the performance of our model, we compare TEMPO with the following baselines over long-term forecasting and short-term forecasting: (1) The pre-trained LLM-based based models, including Bert (Devlin et al., 2019), GPT2 (Radford et al., 2019; Zhou et al., 2023), T5 (Raffel et al., 2020), and L.M (Touvron al., 2023). (2) The Transformer-basee PatchTST (Nic et al., 2023), FEDformer (Zhou et al., 2022), ETSformer (Woo et al., 2022) and Informer (Zhou et al., 2021). (3) The variant of Linear-based models, DLinear (Zeng et al., 2023) model. (4) General 2D- variaaiio model, TimesNet (Wu et al., 2023). Following traditional forecasting works, we report the Mean Squared (MSE) and Mean Absolute Error (MAE) results in this section. Please refer to the Appendix B and F for the detailed experiment setting and baselines.
models, including the
4.1 ZERO SHOT LONG-TERM FORECASTING RESULTS
Table presents the performance of multiple time series forecasting models on MSE and MAE metrics across different prediction lengths under the 'many-to-one' setting, with lower scores indicating more accurate forecasts. Our proposed model, TEMPO, surpassed existing baselines on average over all prediction horizons across all datasets, highlighting the broad applicability of TEMPO. Our model achieves the highest average performance scores. Specifically, it improves the weather and ETTm] datasets by around 6.5% and 19.1%, respectively in MAE compared to the previous state-of-the th-thh--n art model, PStchTST. It also secures the lowest error rates across numerous individual datasct- prreicctttriiiiiinnn length configurations. Compared to other pre- pre-trained models for forecasting, TEMPO consistently delivers the best results across different time series datasets. These results suggest that incorporating
TTMPO's source code he found https: //gittub.b.. com www.c//mm/cccoo recca-reeeaccc /

---

Published as a conference paper at ICLR 2024
Table 1: Transfer learning of long-term forecasting results on time series benchmark datasets. We use prediction length O E {96, 192, 336, 720). A lower MSE indicates better performance. Hereafter, for the tables, the best results are marked in bold and the second optimal in underlined, respectively with MSE/MAE.
ECL
Trallie
Weather
Buml
Eumz
Eubl
Euh2
Horizon
Model
MSE/MAE
MSE/MAE
MSE/MAB
MSE/MAE
MSE/MAE
MSE/MAE
MSE/MAE
TEMPO
0.178/0.276
0,476/0.43
0.211/0.254
0.43800.22
0.18550.267
0.400N0.406
0.301/0.353
GPT2
0.193/02888
0,522/0.380
0.226/0.274
0.486/0.438
0.193/0.273
0.400M0.416
0.320/0,363
T5
0.1850.2282 0.30
0.217/0.271
0.529/0.464
0.196N0.268
0.400/0.409
0.32610,366
PatchTST
0.489/0.546
100200.611
0,247/0.301
0.733/0.554
0.273/0.345
0,57/0.518
0.379/0.412
96
Timesnet
0.293/0.369
0,585/0.401
0,247/0.295
0.518/0.470
0.202/0.296 290
0.407/0.423
0.315/0.362
FEDrormer
0.3000.399
0.835/0.564
0.292/0.346
0.698/0.553
0. 0.665/0.634
0.509/0.502
0.385/0.426
FTSfomer
0. 0.707/0.638
1,419/0.995
0.453/0.416
1.117/06678
0.35M0-404
0.469/0.457
0.405/14228
Informer
0,512/0,531
1,400008300
0.837/0.711
0,880/06657
D. 0.263/0.360
0.642/0.562
0,704/0.65
DLinear
0. 0.19/20.292
0.609/0.424
0.212/0.275
0.624/0.522
0,26400332
0.414/0.421
0.334X1.389
TEMPO
0.198/0.293
0.496/0.355
0.254/0.298
0.461/0.432
0.243/0.304
0.426/0.421
0.355/0.389
GPT2
0.207/03300
0.533/03887
0.273/03112
0.516/0.461
0.254N0.312
0.441/0.433
0.381/0.402
0.205/0.302
0.524/0.174
0,277/0.321
4.523/0.454
0.246/0.
0.428/0.426
0.413/0.410
PatchTST 0.465/0.535
0,992/0.633
0,277/0.324
0.739/0.563
0.299N0.355
0.580M0.528
0.387//.417
192
Timesnet 0,283/0.366
0.64/0.431
0,316/0.342
0,55X).490
0.261/0.318
0.439M0.439
0. 0.394/0.406
FEDformer
0,390/0068
0.8690.5579
0,372/0 426
0.819/0.608
0.358/0.416
0.68310.596
0.921/0.748 748
ETSformer
0.721/0.645
0,995/0.658
0,545/0.466
1.598/0.800
0,390N0.416
0.548/0.503
0.476/0.468
Informer
0.625/0.619
0.872/0.506
0,431/00455
1,461/0.892
0.494N0.516
0.798/0.632
0.455/0.883
DLinnar
0.204/0.300
0,595/0.412
0.259/0.30 308
0. 0.599/0.511
0,292/0.365 365
0.439/0.437
0. 1/3.1/00415
TEMPO
0.209/0.349
0.503/0.356
0,292/0.332
0.515/0.467
0.309/0.345
0.441/0.430
0.379/0.008
GPT2
0.231/0.324 0,566/0 421
0.44 10.379
0,571 402
0.315/0.35
0.449/0.444 444
0. 0.394/0.416
T5
0.229/0.321 0.550/0.391
0.330/0.330
0,572/0.500 504
0.316/0.346
0.442/0.438
0.416/0.427
PatchTST 0.531/0.569
0.987/0.626
0.317/0.347
0.755/0.576
0.342/0.382
0.677/0.573
0.386/1.225
336
Timesnet
0.733/0663
1.609/0.864
0.359/0.372
0.638/0.558 532
0.38/0.392
0.555/0.500 503
0. 38.4//14413
DEDrormer
0,317/0.06
1.006/0.640
0.639/0.600
0.785/0.624
0.372/0.424
0.582/0.542
-/5.755
ETStomer
0,86200707
0.940/0.211
0.487/0.444
1.154/0.682
0.409/0.428
0.728/0.585
0.446/0.451
Informer
1,222/0.863
0.978/0.507
0,370/0.412
0.949/0.631
0.788/0.622
1.125/0.810
1.389/0.888
DLincar
0,231/03325
0.624/0.477
0,304/0.342
0.622/0.534
0.361/0.411
0.463/0.464
0.471/0.482
TEMPO
0,279/0.355
0,5389.376
0.370/0.379
0,591/0.509
0,386/0,395
0.443/0.451
0.409/0.4400
GPT2
0.262/0.347 0.396/0.999
0.484/0.422
0.646/0.54
0.394/0.397
0.445/0.454
0.434//04448
T5
0.266/0.351
0,578/0.004
0.528/0.451
0. 0.694/0.568
D. 0.394/0.397
0.443/0.458
8.425/0.440
PatchTST 0,475/0.532
1.152/0.706 706
0.375/0.388
0,739/0.57
0.421/0
0.540/0.521
0,425/0.448
720
Timesnet 1.16600859
1.974/0.971
0.423/0.405
0.723H0.577
0.399/0.409
0.438/0.461
0.394K0.431
FEDformer
0.423/0488
0.965/0.652
0.409/0.425
0.816/0.614
0.455/0.462
0.688/06618
0.427X1.452
FTSfomer
0.666/0.440
0,798/0.518
0.592/0.506
1.038/0.665
0.444/0.438
0.61510.561
0.446/14666
Informer
0.881/0.778
1.532/0.800
1.133/0.842
0,779/0.616
1.075/0.725
0.836/06887
1.33011866
DLinear
0,259/03352
0.623/0.42
0.363/0.389
0.639/0.559
0.515/0.490
0.467/1.481
0.639X1,559
TEMPO
0.216/0.368
0.503/0.358
0.282/0.316
0.501/0.458
0,280N0.322
0.428/0.427
0.361/0.398
GFT2
0,223/03115
0.554/03997
0.356/0.347
0.555/0.485
0.289/0.333
0.436/0.436
0. 0.382/1.407
0,221/03314
0,5400.384
0.338/0.343
0,58/0.498
0,287/0.329
0,428/0.433
0. 0,396/0.411
PatchTST
0.49/0.545
1.0390.652
0.304/0.340
0.741/0.566
0.334N0.376
0.592/0.535
0. 0.394/0.425
Avg
Timesnet
0.619/0.557
1,202/0.667
0.336/0.354
0.607/0.517
0.311/0.352
0.460N0.457
0.372/0.403
FEDformer
0.358/0.439
0.919/0.609
0,428/0.449
0. 0,780X0.000
0.46310.484
0.616/0.565
-f1 845
ETSfomer
0.75000.664
1.0380.648
0,519/0.488
1.227/0.707
0.399/0.422
0.590N0.227
0.443/1.533
Informer
0.81000668
1.1960.661
0.693/0.605
1.01 7/17/99699
0.655/0.556
0.850/0.673
0.970m.812
Diiinc
0.222/03117
0.613/
0. 0.284/0.329
0.621/0.531
0. 0.358/0.405
0.446/0.551
0.456/0.461
LLM with the well-designed prompt and implementing time series decomposition can contribute significantly to enhancing the accuracy and stability of zero-shot time series forecasting.
4.2 SHORT-TERM FORECASTING WITH CONTEXTUAL INFORMATION
Dataset and metries. In this section, we introduce TETS, new benchmark dataset built upon S&P 500 dataset combining contextual information and time series, to the community. Following (Cao et al., 2023), we choose the symmetric mean absolute percentage error (SMAPE) our metric in this section. Moreover, the GDELT is also used to verify the effectiveness the our proposed method. Please refer to Appendix B.2 and Appendix B.3 for the detailed dataset setting of TETS and GDELT;
7

---

Published as a conference paper at ICLR 2024
Table 2: SMAPE results of EBITDA from TETS and GDELT. The result of EBITDA includes
outliers removed where SMAPE exceeds 0.8/0.9. The best results are marked in bold and the second optimal in underlined respectively with 0.8 & 0.9. (Sectors: CC: Consumer Cyclical: CD: Consumer Defensive; Ind: Industrials; RE: Real Estate; Events: 11: Disapprove; 17: Coerce; 19:Fight.)
EBITDA Dataset
Sectors
TEMPO
LLaMA
GPT2
Informer
PatchTST
Reformer
DLincar
CC
32.27/33.48
33,13/34.31
33,77/35.37
33.42/35.33
32.65/33.81
41,12/43.17
41.44/43.18
37.23/39.00
33.53/35.65
CD
25.9/26.25
26.34/26.62
26.86/27.15
27.34/28.3
26.44/26.79
35.65/36.08
31.6/31.98
29.93/30.36
27.01/28.00
Ind
26.7/27.42
27.17/27.98
27.9/28.63
27.89/28.95
27.3/28.12
34.83/35.87
33.84/34.87
30.23/31.20 28
27.59/28. 84
RE
29.46/30.11
29.63/30.48
30.62/31.21
30.62/31.66
30,1/30.64
36.4/37.22
37.63/38.31
31.23/31.69
29.95/30.92
GDELT Dataset
11 38.77
40.23
39.03
38.89
39.04
42.00
40.45
46.72
40.14
41.02
42.50
41.20
41.10
41.24
44.44
42.72
48.08
42.45
19
44.03
45.49
44.17
44.09
44.29
47.45
45.49
48.30
45.40
Appendix H for the proposed pipeline of collecting TETS dataset with both time series and textual information.
Contextual Information. In order to incorporate the contextual information into our proposed TEMPO, we leverage the built-in tokenization capabilities of the generative pre-trained transformer to derive embeddings of input text. Then, we utilize these text embeddings corresponding to each time series instance, T'ert, to construct soft prompts with learnable parameters and concatenate them at the beginning of the input embedding, that is, x Teat @ FT @ Is A æn- Where the x. for EBITDA is conducted with semi-soft prompt. This method is not strictly confined to our proposed model but can be feasibly applied in similar works enhance their capability of handling and benefiting from contextual information. Comparisons with other design strategies of contextual information are provided in the Appendix D.4 for further reference.
Results. From the transfer learning perspective, we choose to report the setting of 'many-to-many which means we train a model using in-domain sectors data and directly do the zero-shot test on all cross-domain sectors. The SMAPE results of using different baseline models and our model on the TETS dataset and GDELT dataset are listed Table 2 which is also zero-shot setting as data samples from those sectors are not seen during the training stage. Examining the results across all sectors, our proposed model, which combines time series data with supplementary summary (contextual) data, outperforms all the baseline methods in cross-domain sectors. Besides, we observe that transformer- based architectures training from scratch, specifically tailored for time series analysis malysis-such such as PatchTST, Informer, and Reformer (Kitaev et al., 2020)-tend to underperform in comparison to transformers pre-trained on linguistic datasets. This performance discrepancy indicates that the parameter initialization derived from pre-trained language models confers superior starting point for model optimization. Consequently, these pre-trained models exhibit enhanced capabilities and adaptability within zero-shot learning contexts. Furthermore, in instances where the time series data exhibits a strong correlation to other modalities, such as textual information, devising an effective strategy amalgamate these distinct modalities could lead enhanced performance gains.
ANALYSIS
5.1
ABLATION STUDY
The provided ablation study. Table 3, offers critical insights into the impact of the prompt and decomposition components on the performance of our model. In this table, the MSE and MAE on various datasets are reported for four scenarios: the original model configuration ("TEMPO'); the model without the prompt design and without decomposition, which is the setting of "w/o Dec"; the model without prompt design ('w/o Pro') and the model without the decomposition loss alignment (' ('w/o Dec Loss'). Averagely, the exclusion of the prompt component leads to a deterioration in the model's 's predictive accuracy, indicating the prompt can be important factor in enhancing the model's 's overall performance. The omission of decomposition loss typically results in a decline in model performance. Decomposition loss facilitates the use of richer historical dataset, which
8

---

Published as a conference paper at ICLR 2024
TEMPO
Dee
Pio
wio
MSE/MAE MSEMAE
MSE/MAE
MSE/MAE
96
0.17800226
0.19500.244
0.185/0228
D.191KL298
192
D. 0.19800243
0,210/0.30 301
0.196/0.201
0.205/0 5015
BCL
0.237M0.326
0.22//0.318
720
0.279003355 0.221/003551
0.269/0.355 359
0.262X1.333
0.216/0338
0,228/0.319
0.219033133
0,225/0.522
96
0:438/0.424
0,51660
0.452/0:431
0.428/0.425
0.461/0.432
0.518/0.462
0.42/0:45
0.494K0.663
0.51540.467
0.622/0.5155
0.519/0.174
0.51100412
720
0.591/0.509
0.644h0.50
0.582/0.11
0.594K0.521
Avg
0.501/0.458
0.575/0.48)
0.506/0..00
0.515/0.475
Figure 2:
The SHAP values of decomposed com- ponents of TEMPO for ETTml.
Table 3: Ablation study TEMPO.
enhances the quality of individual decomposition components. This improvement component quality is important for the model's forecasting accuracy. Note that employing the prompt design isolation, without the support of decomposition, can detrimentally impact the backbone model's performance in most cases. This can be due to the difficulties in effectively prompting time series data from its raw form with limited semantic information. These findings underscore the essential nature of both prompt and decomposition elements in achieving robust forecasting capabilities under the sero-shot setting.
5.2 INTERPRETING MODEL PREDICTIONS
SHAP (SHapley Additive exaanttionns) values serve as comprehensive measure of feature impor- tance, quantifying the average contribution of each feature to the prediction output across all possible feature combinations. As shown in Figure 2, when applied our seasonal and trend decomposition, the SHAP values from the generalized additive model (GAM) suggest dominant influence of the seasonal component on the model's predictions, implying a significant dependency of the model on the overall recurring patterns within the data. While the directional shifts of ETTml dataset's contribution is relatively stable. The escalating values in the 'Error' column, which denote the discrepancy between the model's predictions and the ground truth, indicate potential decline in the model's accuracy as the prediction length increases which is indeed observed in most experi- ments run. In this context, the STL decomposition proves invaluable as enables us to identify and quantify the individual contributions of each component to the overall predictions, as demonstrated by the SHAP values. This detailed understanding can yield critical insights in how the pre-trained transformer is interpreting and leveraging the decomposing pre-processing step, thereby providing robust foundation for model optimization and enhancement. SHAP values for weather dataset can be found at Figure 14.
6 CONCLUSION
This paper proposes a soft prompt based generative transformer, TEMPO, which achieves state-of-the- art performance in zero-shot time series forecasting. We introduce the novel integration of prompts and seasonal trend decomposition together within pre-trained Transformer- maaseddeer-baeedd backbone to allow the model to focus on appropriately utilizing knowledge from different temporal semantics components. Moreover, we demonstrate the effectiveness of TEMPO with multimodel input, effectively leveraging contextual information in time series forecasting. Lastly, with extensive experiments, we highlight the superiority of TEMPO in accuracy, and generalizability. One potential limitation worth further investigation is that superior LLMs with better numerical reasoning capabilities might yield better results. In addition, the encouraging results of TEMPO the zero-shot experiments shed light into effective foundational models for time series.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is partially supported by the NSF Award #2125142 and NSF Award #2226087. The funding from these sources has been cornerstone in enabling to bring our project to fruition. We would like to extend our thanks to Yizhou Zhang. James Enouen, Qiang Huang, Chuizheng Meng, and Hao Niu for their invaluable discussions and insights in shaping the direction and execution of our work. We are also deeply grateful to the anonymous reviewers for their rigorous review process. Their detailed comments and constructive suggestions have significantly contributed the improvement of this paper. The time and effort they invested in providing feedback have been invaluable and have greatly assisted in refining our work.
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in neural information processing systems, 14162, 2020.
Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Conguri Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong. et al. Spectral temporal graph neural network for multivariate time-series forecasting. Advances in neural information processing systems, 2021.
Defu Cao, Yixiang Zheng. Parisa Hassanzadeh, Simran Lamba, Xiaomo Liu, and Yan Liu. Large scale financial time series forecasting with multi-faceted model. In Proceedings of the Fourth ACM International Conference on Al in Finance, pp. 472 480, 2023.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 6989-6997, 2023.
Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308. 08466, 2023.
Robert B Cleveland, William Cleveland, Jean McRae, and Irma Terpenning. Stl: A seasonal-tren decomposition. J. Off. Stat, 6(1):3-73, 1990.
Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series series dense encoder. arXiv preprint arXi::2344.08424 2023a.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310. 10688, 2023b.
Jacob Devlin, Ming-Wai Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA, June 2-7, 2019, pp. 4171 4171-4186, 2019.
Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mip-mixer model for multivariate time series forecasting. In Proceed- ings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 23, pp. 459-469 New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 041145/358030553599553 3580305 3599533.
URL /ttts:////oi.ooo/00015555
Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie-Yan Liu. DEPTS: Deep expansion learning for periodic time series forecasting. In International Conference on Learning Representations, 2022.
10

---

Published as a conference paper at ICLR 2024
Wei Fan, Pengyang Wang. Dongkun Wang. Dongjie Wang, Yuanchun Zhou, and Yanjie Fu. Dish-ts: a general paradigm for alleviating distribution shift in time series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 7522-7529, 2023.
Robert Fildes, Andrew Harvey, Mike West, and Jeff Harrison. Forecasting, structural time series models and the kalman filter. The Journal of the Operational Research Society, 42:1031, 1991. doi: 10.2307/2583225.
Azul Garza and Max Mergenthaler-Cansee Timegpt-1. arXiv preprint arXiv:2310.0358) 2023.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03883 2024.
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.
Trevor Hastie. Generalized additive models. In Statistical models in S, pp. 249-307. Routledge, 2017.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Causal discovery from beterogencousshoostaaiinaar data. The Journal of Machine Learning Research, 21(1):3482-3534, 2020.
Rob J Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast package for r. Journal of statistical software, 27:1-22, 2008.
Furong Jia, Kevin Wang, Yixiang Zheng, Defu Cao, and Yan Liu. Gpt4mts: Prompt-based large language model for multimodal time-series forecasting. In The 14th Symposium on Educational Advances in Artificial Intelligence (EAAI-24), 2024.
Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and survey and outlook. arXiv preprint 21X11:2310.10196. 2023.
data: A
Ming Jin, Shiyu Wang. Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https::////.......t....tt /wppnreeiieeeeeeeeeeeeeeeeeeeeee New..wwww. net. Forum? d=Unb3cVFta..
Ming Jin, Yifan Zhang, Wei Chen, Kexin Zhang, Yuxuan Liang, Bin Yang, Jindong Wang, Shirui Pan, and Qingsong Wen. Position paper: What can large language models tell us about time series analysis. arXiv preprint vrXiv:2402.027133 2024b.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference Learning Representations, 2022.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference Learning Representations (ICLR), Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale parameter-efficient prompt tuning. arXiv preprint urXiv:2 (04.0099,0 2021.
Jun Li, Che Liu, Sibo Cheng, Rossella Areucci, and Shenda Hong. Frozen language model helps ecg zero-shot learning, 2023.
11

---

Published as a conference paper at ICLR 2024
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq benq-Neng Hwang, et al. Grounded language-ima image pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10965-10975, 2022.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 1rXiv:210100// 2021.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR *18), 2018.
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35:5816-5828, 2022.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li. Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference learning representations, 2021.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers effective for time series forecasting. arXiv preprint arXiv:23 2023.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Yuqi Nie, Nam H. Nguyen. Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations (ICLR '23), 2023.
OpenAl. Gpt-4 technical report, 2023.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 2019.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models follow instructions with human feedback. ArXiv, 0bs/22033021553 2022. URL https: / /api..pp.../// semant iescholcccccrrrr CoggCorppsIDD 24 646999909.
Antony Papadimitriou, Urjitkumar Patel, Lisa Kim, Grace Bang, Azadeh Nematzadeh, and Xiaomo Liu. A mactee- approach to large scale financial forecasting, In Proceedings of the First ACM International Conference on Al in Finance, pp. 1-8, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2185555885-555, 2020.
12

---

Published as a conference paper at ICLR 2024
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Bilos, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schnei- der, et al. lla-llama: Towards foundation models for time series forecasting. arXiv preprint 1rX.v22310.08278, 2023.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 4222 4235, 2020.
Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. A comparison of arima and Istm in forecasting time series. In 2018 17th IEEE international conference on machine learning and applications (ICMLA), pp. 1394-1401. IEEE, 2018.
ya Meerovich Sobol'. On sensitivity estimation for nonlinear mathematical models. Matematich- eskoe modelirovanie, 2(1):112-118 1990.
Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. Test: Text prototype aligned embedding to activate Ilm's 's ability for time series. arXiv preprint arXiv:2308. 08241, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, 139977, 2023. URL https: //ap:///pp.. /api.pp..../ semanticscholat org/
Corpus ID :
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.0138/ 2022.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXi::2402.02592, 2024.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- with auto-correlation for long- sengtterm series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), pp. 101-112, 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023. URL https: /opp//evieiieeeeee net/ forum?
/
id=jj_Uqq388 40q.
Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 2023.
Xinli Yu, Zheng Chen, Yuan Ling, Shujing Dong. Zongyi Liu, and Yanbin Lu. Temporal data meets llm llm-explainaale financial time series forecasting. arXiv preprint 2rXiv:2306.11025, 2023.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effeetive for time series forecasting? In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022a.
13

---

Published as a conference paper at ICLR 2024
Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriente oriented mlp structures. arXiv preprint S211::2207.011880 2022b.
Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems, 35:3988-4003 2022c.
Yizhou Zhang, Lun Du, Defu Cao, Qiang Fu, and Yan Liu. Guiding large language models with divide-and-conquer program for discerning problem solving. arXiv preprint arXiv:2402.05339, 2024.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference Learning Representutions, 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of AAAI, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. Proc. 39th International Conference Machine Learning (ICML 2022), 2022.
Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 2023,
14

---

Published as a conference paper at ICLR 2024
A SHOWCASES
A.1 COMPARE WITH GPT4TS
In Figure 3, 4, 5, 6, 7, we plot the comparison of the predicted value from our model and GPT4TS model given look-bace back window. As shown in the datasets, truth, which is also shown through our superior performance over other models in table 1. We select time series with different characteristics under different prediction lengths O {96, 192]: time series with high variability (Figure 5 a), periodic (Figure 3 a, Figure 3 b, a, 4 b), non-periodic with change in trend (Figure a, Figure 6 b)
able to predict close to the ground
GroundThuth
GroundTruth
Prediction
GPT41S Prediction
Ours Prediction
Prediction
20
80
25
50
75
100
125
(a) prediction length O = 96
(b) prediction length O = 192
Figure 3: Visualization of long-term forecasting results. Compared between our model TEMPO and GPT4TS on ETThl dataset
0.25
0.0
0.00
-0.
0.50
-1.0
Ground Iruth
GroundTruth
Prediction
GPT4TS Prediction Prediction
Durs Prediction
20
25
100
150
175
(a) prediction length = 96
(b) prediction length O = 192
Figure 4: Visualization of long- forecasting results. Compared between our model TEMPO and GPT4TS on ETTh2 dataset
A.2 COMPARE WITH TIMEGPT
also compare our results with TimeGPT (Garza & Mergenthaler-Cansen 2023), which is capable of generating accurate predictions for a diverse range of datasets not seen during training, demonstrating superior performance in zero-shot inference compared to traditional statistical, machine learning, and deep learning methods. Access to TimeGPT-1 (Beta) is provided through Python SDK and a REST API. This accessibility allows us to explore TimeGPT's forecasting capabilities on datasets. As shown in Figure and Figure 9, despite its design for various downstream tasks,
15

---

Published as a conference paper at ICLR 2024
-2
33
Groundiruth
GroundTruth GPT4TS Predictien
GPT4TS Prediction
Ours
Ours Prodiction
20
40
80
25
50
100
prediction length O 96
(h) prediction length O = 192
Figure 5: Visualization of long-term forecasting results. Compared between model TEMPO and GPT4TS on ETTml dataset
GPT4TS Irediction Prediction
0.4
022
0.6
0.4
-10
-1.2
GroundTuuth Ours Prediction
-12
-1.4
0
20
40
60
80
(a) prediction length = 96
(b) prediction length O = 192
Figure 6: Visualization of long-term forecasting results. Compared between our model TEMPO and GPT4TS on ETTm2 dataset
GroundTPuth
GPT4T5 Prediction
1.0
AMA MM
-0.500
-0.525
0.4
Groundiruth Ours Prediction
-0.30
-0.575
20
40
60
80
100
125
prediction length O 96
(b) prediction length O = 192
Figure 7: Visualization of long-term forecasting results on weather dataset. Compared between our model TEMPO and GPT4TS on weather dataset
16

---

Published as a conference paper at ICLR 2024
Sounchun SPEATS Frectsioo
man
Granthen
au mnip
azs
exe8 souo
Pooe spupp
0009
(a) prediction length
96
prediction length = 192
Figure 8: Visualization long-tery term forecasting results on ETTml dataset. Compared between our model TEMPO and TimeGPT on weather dataset
12
Tradit Poeese
ni-tem ma
Fack e7.00
11-18 stoe
narm
marzt
mene
the 10
22:0
prediction length
96
(b) prediction length
192
Figure 9: Visualization long-term forecasting results on ETTm2 dataset. Compared between our model TEMPO and TimeGPT on weather dataset
is important to note that TimeGPT may not perform well in long-term forecasting scenarios. In contrast, our proposed model excels zero-shot settings, including long-temm forecasting, illustrating the need for foundation models that can adapt to both the breadth of time series applications and the depth of forecasting horizons.
B EXPERIMENT SETTING
B. TOWARDS FOUNDATION MODEL EXPERIMENTS DETAILS
It has been well-established that -hannel-independence works well for time series datasets, we treat each multivariate time series as multiple in- dependent univariate time series. We use popular time series benchmark datasets (Zhou al., 2021): ETTml, ETTm2, ETTh1, ETTh2, Weather, Electricity, Traffic, ILI and exchnge. 1) ETTm1, ETTm2, ETTh1, ETTh2 contain electricity load from two elec- tricity stations at 15 minutes level and hourly level. 2) Weather dataset contains 21 metcorological indicators Germany within I year; 3) Electricity dataset contains electricity consumption; 4) Traffic dataset contains the occupation rate of the freeway system across the State of California. The lookback window L is following (Zhou et al., 2023), and the prediction length O is set to {96. 192, 336. 720). In this experiment part, experiments were conducted using single NVIDIA A100 GPU, with a batch size set to 256, and focused long-term forecasting by employing Mean Squared Error (MSE) loss function. To
Table 4: Dataset details of benchmark dataset.
Dataset
Length
Covariates
Sampling Period
ETTh
17420
hour
ETTm
69680
7
15 min
Weather
52696
22
10 min
Electricity
26304
321
I hour hour
Traffic 17544
862
17

---

Published as a conference paper at ICLR 2024
ensure the reliability of our results, we performed three iterative loops and calculated the average of the outcomes. Our exploration covered [3, 6] GPT layers and tested various weights, [0.001, 0.01, and 1], for the MSE loss function applied to the reconstructed components of the time series. We have documented the optimal results obtained from this search. comprehensive analysis of the impact that the number of GPT layers has the performance will be addressed in future research.
Towards Foundation Model's Zero Shot Setting For each prediction length, we train a model on a mixture of training data from different domains and test the model on the target unseen domain's ' data. We construct the combined training dataset by pooling the training data and fullyshuffline them. To prevent undue bias and ensure fair representation of data from each domain in the combined training data, we select an equal number of training examples from each domain's s training data. We noted that the number of training samples that ETTh1 and ETTh2 has is on much smaller magnitude compared to the other three training datasets (ETTml, Weather, Electricity), so selecting the minimum number of training samples among all other training datasets would result in too much data loss from ETTmI, Weather, and Electricity, etc. Therefore, we included all training examples from ETThl and ETTh2 in the combined training dataset. Similar to traditional experimental settings, each time series (ETTh1, ETTh2, ETTm1, Weather, Electricity, ETTm2, Traffic) is split into three parts: training data, validation data, and test data following in 7:1:2 ratio in (Zhou et al., 2022), and we only merge the training and validation data. For ETTml, ETTm2, Weather and Electricity data, the number of examples sampled to be pooled into the combined training dataset is chosen be the minimum number of training examples among these training datasets.
B.2 PROPOSED TETS DATASET SETTING
Prompt:
Suppose you are living (Year: 2000), can you help me summarize the news and reports in (Year: 2000Y's (quarter: second quarter) for (company name: Company A), which an (company sector: Technology) company. Please directly give me the answer limited to 2 sentences without apology.
GPT Response:
Description:
In the second quarter of 2000, Company A reported a net profit of $233 million, up from $123 million in the same quarter of the previous year, driven by strong sales of its X computers and Products Y. However, the company's stock price dropped after warning that its below expectations due to slower sales.
profits would be
Figure 10: Example for designing prompts using OPENAI ChatGPT-3.5 API.
Data Collection Our time series data for financial analysis and forecasting are derived primarily from the financial statements of companies including balance sheets, income statements, and cash flow statements. Specifically, we utilize data from the 500 largest U.S. companies across 11 sectors as listed in the Standard & Poor's 500 Index (S&P 500), which we divide into two parts: the first seven sectors for training and evaluation, and the remaining four for zero-shot forecasting tasks to test the model's 's ability to predict in unseen domains. While collecting corresponding contextual information from the abundance of digital news sources challenging. OpenAl's ChatGPT API offers a solution to gather and condense relevant news efficiently. By inputting key details into the API and limiting the response to 10 tokens, as shown in Figure 10, we can swiftly extract pertinent contextual data to improve our analysis. Please refer to Section H for further details of creating TETS dataset.
Prediction objective The primary objective of our experiment is to forecast the Earnings Before Interest, Taxes, Depreciation and Amortization(EBITDA) for companies listed in S&P500, and our data range from 2000 to 2022. Following the multivariate time series framework presented in (Papadimitriou et al., 2020), we select foundational financial metrics from the income statements as input features: cost of goods sold (COGS), selling, general and administrative expenses (SG&A). RD expenses (RD EXP). EBITDA, and Revenue. Comparing with other metrics, the selected metrics contain information more relevant our prediction objective. For Large Language based models,
18

---

Published as a conference paper at ICLR 2024
including model TEMPO, GPT4TS, and T5, apply channel-independeer strategy to perform univariate time series forecasting tasks. All five features are used for training (predicting its future value based its past value), while only EBITDA is accessible during the training stage. Other models follow the multivariate time series forecasting setting, treating the five features as multivariate input and predicting the target, EBITDA, both in the training and testing stages.
We predict quarterly EBITDA based on the past 20 quarters' data. This predicted value is then used forecast the next quarter's s EBITDA, iteratively four times, leading yearly prediction. order to measure the accuracy of these predictions based on the cumulative yearly value (sum of 4 quarters), we employ the symmetric mean absolute percentage error (SMAPE) as the evaluation metric as well as the forecasting loss function in this experimental part.
Data Split For companies under each sector, we employ the windowing method to generate cohesive training and testing instances. Under the Channel-independee setting where separate each feature to obtain univariate time series, we get 80,600 samples from the seven in-domain sectors, and 9,199 samples from the four zero-shot sectors(also known cross-domain sectors), five as much as we get in the channel dependent setting. The sectors splitting is elaborated in H. In our experiments shown in table 2, We use 70% of in-domain data for training, 10% of in-domain data for evaluation, and all zero-shot data for unseen testing.
Symmetric Mean Absolute Percentage Error In reality, the magnitude financial metrics can vary significantly among different companies. So, choose the symmetric mean absolute percentage error (SMAPE), a percentage- based accuracy measure, as our evaluation metric. For EBITDA, there many negative results that may influence the final SMAPE. We use the form of SMAPE-Abs SMAPE:
200%
F. Arl + [A,
AbSSMAPE
Y1
Here, F. represents the true value, A, represents the predicted value our system, and n represents the total time steps need to forecast.
SMAPE can be particularly sensitive to outliers. Specifically, when the true data and prediction have opposite signs, the resulting error may be up 200%, seriously distorting the final results. Following the approach in (Papadimitriou et al., 2020), we filter out data points at the 80% and 90% thresholds and find most of the outliers related to significant financial shifts due mergers & acquisitions (M&A).
B.3 GDELT DATASET SETTING
We utilized the GDELT dataset (Jia et al., 2024), which focuses on predicting the respective mentions and mentions in the news media. We utilized the data collected from the 55 regions under the US and the national data for the US and divided the 10 event root types in the dataset into unseen and seen sets, as demonstrated in Table We focused on predicting the three key variables NumMentions, NumArticles. NumSources related to the particular event type within a given timeframe and geographical region. We apply chaanel-independeme. time series forecasting tsks for all baseline models and our model. All three features are used for training and evaluation (predicting its future value based on past value).
strategy perform univariate
We predict the future days based on the past 15 days' data directly. In other to measure the accuracy of the predicitions, we use mean square error (MSE) and mean absolute error (MAE). For each region, we employ the windowing method to generate cohesive training and testing instances for each event root type. Under our Shaanee--nddeendeeee seven in-domain sectors (seen sectors) for training, and 76,048 samples for evaluating under the three zero-shot sectors (unseen sectors). In our experiments, we use 70% in-domain data for training, 10% for evaluation and all zero-shot data for unseen testing.
setting, we get 122,008 samples from the
19

---

Published as a conference paper at ICLR 2024
Event Number
Event Type Name
01
Make Public Statement
02
Appeal
03
Express Intent to Cooperate
Training Event
04
Consult
05
Engage in Diplomatic Cooperation
07
Provide Aid
08
Yield
Disapprove
Test Event
17
Coerce
19
Fight
Table 5: Event number to event type Name table
C FURTHER RESULTS
C.I. SELE-SUPERVISED REPRESENTATION LEARNING
Our proposed model architecture can designed to support self-supervised learning and thus further embrace foundation models for time series. Following (Nie et al., 2023), we mask random subset of patches by replacing them with zeros, where the patches are divided into non-overlapping patches for simplicity and to avoid masked patches influencing predictions. The prediction head is removed and replaced with a linear layer to reconstruct the masked patches. The model is trained to minimize the MSE between the predicted and true masked patches. To handle multivariate time series with varying numbers of features, we apply channel independence (Zeng et al., 2023) to model each time series independently.
With the strong performance TEMPO showed under the experiment 'many-tooone" zero-shot setting, from the perspective of self-supervised cross-domain foundational model, we further investigate using a TEMPO model trained on datasets across domains can still achieve comparable performance on unseen domains. Here. we still use the 'many-to-one' setting but the model trained in self- supervised manner. Specifically, we first use all other domain's 's data to train representation model then only use 5% data of the training data to fine turn the total model with the prediction layer as a forecasting downstream task. Table 6 provides comprehensive comparison of our model against other baseline models on three multivariate time series datasets that are unseen by the models during training. namely electricity and traffic and weather. All these selected 3 datasets are entirely dissimilar to any data the model has encountered before. TEMPO outperforms baseline models, achieving the lowest MSE and MAE in most cases. Note that TEMPO's average MSE and MAE is 7.3% and 4.6% less than the best-performing baseline model (GPT2) for the weather dataset, respectively. This finding shed light on the strong generalizability of TEMPO and indicated its potential of serving foundational time series forecasting model, maintaining robust performance for unseen domains.
C.2 COMPARING WITH FULL-SHOT STATE-OF-TIEE-ARTS BASELINES
Towards foundation model training differs significantly from the one-to-one/many scenarios, where pre-training involves a homogenous dataset, often with consistent season patterns, sampling rates, and temporal scales. This homogeneity facilitates pattern learning transferable to fine-tuned datasets. In contrast, towards foundation model training involves pre-training on highly diverse datasets, such as merging traffic and weather data, which may hinder the model's ability to discern underlying patterns. In Table 7, we provide further results on ETThl and ETTh2 datasets, demonstrating that the performance of TEMPO (zero-shot setting) surpasses that of state-of-the-art models specifically designed for these target datasets with full-shot settings. The results in Table 7 are obtained from (Liu et al., 2023). including but not limited to iTransformer(Liu al., 2023), Crossformer (Zhang & Yan, 2022), TiDE (Das et al., 2023a) and SCINet (Liu et al., 2022), which are also reported in our contemporaneous work, MOIRAI (Woo et al., 2024).
20

---

Published as a conference paper at ICLR 2024
Table Self-supervised representation learning results are fine-tuned on 5% in-domain datasets. We use prediction length O E (96, 192, 336, 720). A lower MSE indicates better performance, and the best results are bold.
TEMPO
GPT4TS
TS4TS
Ben4TS
PatchTST
MSFMMARE
MSE/MAE
MSFMMAE
MSF/MAF
MSFMMAFE
0.19/0.29
0,202/0.301
0.199/0.293
0.202/0.298
0.21/0.308
0.211/0.31
0.217/0.313
0.238/0.337
0.227/0.321
0.223/0.312
BCL
0.229/0.32
0.258/03533
0.273/0.364
0.256/0.345
0,282/0.357
720
0,375/0.444
0.43/0.475
0.455/0.49
0,442/0.479
0.606/0.561
0.251/0.342
0.277/0.361
0.291/0.371
0,282/0.361
0.33/0.385
0.56/0.411
0.607/0.417
0.543/04088
0.591/0.423
0.577/0.403
0.575/0.419
0.603/0.421
0.594/0.431
3/0.432
0.596/0.411
Traffic
0.597/0.433
0.63/0.435
0.659/0.458
0.639/0.445
0.665/0.454
720
0.65/0.452
0.643/0.439
0.69/0.49
0.744/0.496
0,802/0.501
Avg
0.595/0.429
1/62120
0.622/0.447
0.647/0.449
0.66/0.442
0.217/0.268
0.288/0.31
0,252/0.288
0.237/0.288
0.249/0.288
0.265/0.302
0.305/0.331
0.322/0.336
0.29 1/..333
0.272/0.314
Weather
336
0.322/0.342
0.338/0.353
0.346/0.188
0.335/0.354
0,311/0.341
720
0.41/0.3977
0.381/0.377
0.444/0.42
0.466/0.436
0.385/0.386
Avg
0.304/0.327
0.328/0.343
0.341/0.351
0,332/0.35
0.305/0.331
Table 7: Results of long term forecasting experiments ETTh1 and ETTh2. The best results are marked in bold and the second optimal underlined, respectively with MSE/MAF. Note that the TEMPO is under zero-shot setting and other models are under full-shot setting.
TEMPO
iTansformer
TimesNet
PachTST
Crossformer
TDDE
DLineur
SCINet
FEDiormer
MSE/MAE
MSE/MAE
MSE/MAE
MSEMAA
MSE/MAE
MSE/MAE
MSE/MAE
96
0,400/0.406
0.386/.405
0.384M0402
0.1440.419
0.4
0,479N1464
0.386/0.00
0.654N0.599
0.376/0.419
192
0.426/0.421
0.441/0.436
0.436/0429
0.4600.445
0,47100,474
0,52510.492
0,43700.432
0,719/06331
0.420/04448
E1
0.441/0.430
0.487/0.458
0.491/1469
0.501/0.666
0.5700.546
0.565M.515
0.778X0659
0.45900465
720
0.443/0.451
0.503/0.491
0.521/0.500
48500N
0.653/1.21
0.594K1.558
0,519NI.166
0.83611699
0,506/0.5U7
Avg.
0.428/0.427
0.454/14447
454
4.46990.54
0.529/0.522
0.54100.007
0.456/0.452
0.747/0.647
0.440/0.460
96
0.301/0.351
0.297/01149
0.34000377
0.30220.348
0.74580.584
0.400014440
0,333/1.387
0,707/0621
0.358/.3997
BT7h2
192
0.355/0.389
0. 0.3000.400
0.4020.444
000
0.871/0.666
0,528/0.509
0,477/0.476
0.86000689
0.429/0.439
336
0.379/0.408
0.428/1.432
0.452/0.511
0.42601433
1.043/).31
0.64300.51
0.594/0.54
1.0000.744
0,496/0.487
720
0.409/0.440
0.427/0-445
0.462/0.677
0.43100.446
1.1040,763
0,874X0.679
0.831N06657
1.249/0888
0.463/0.474
Avg.
0.361/00.988
1.33004407
0.41401427
0.942/1684
10.550
0.559X0.515
0.954X0.723
0.437/0.449
21

---

Published as a conference paper at ICLR 2024
Table Compare the results with ARIMA.
ECL
Traffic
Weather
Bttm2
TEMPO
ARIMA
TEMPO
ARIMA
TEMPO
ARIMA
TEMPO
ARIMA
MSE/MAR
MSE/MAE
MSE/MAE
MSE/MAR
MSE/MAR
MSE/MAE
MSE/MAR
MSE/MAE
96
0.178/0.276
1.220/0.814
0.476/0.343
1,99770.924
0.211/0.254
0,217/0.258
0.185/0.267
0.225/0.301
192
0.198/0.293
1.264/0.842
0,496/0.355
2.044/0.944
0.254/0.298
0.263/0.299
0.243/0.304
0.298/0.345
336
0.209/0.309
1.31 1/0.866
0.503/0.356
2.096/0.960
0.292/0.332
0,330/0.377
0,309/0.345
0.370/0.386
720
0.279/0.355
1.364/0.891
0,538/0.376
2.138/0.971
0.393/0.387
0.425/0.405
0.386/0.395
0.478/0.445
Avg.
0.216/0.308
1.290/0.853
0.503/0.357
2.069/0.950
0.287/0.318
0.309/0.327
0.280/0.328
0.343/0.369
C.3
COMPARING WITH ARIMA
As a pioneering foundation model, TEMPO is engineered to forecast future values directly, eliminating the necessity for retraining with each new data instance. Its underlying framework captures intricate temporal patterns, granting the versatility to generalize across various time series. In this study, we compare TEMPO's 's forecasting prowess with that of the ARIMA model (Hyndman & Khandakar, 2008), which is renowned for its capacity to make accurate predictions within a specific time series once the initial model parameters have been set. While ARIMA models excel in continuing predictions within the series they are configured for, they do not inherently possess the faculty to forecast across disparate time series without recalibration. We obtain the ARIMA's forecasting results from (Challu et al., 2023). As shown in Table 8, the results highlight the superior adaptability of our 'towards foundation model' TEMPO which retains its predictive accuracy even when applied to time series beyond its training scope, thereby illustrating the feasibility of more universal and resilient forecasting methodologies.
D FURTHER ANALYSIS
D.1 DESIGN OF PROMPT POOL
In this section, we propose another potential prompt design for addressing non-stationary nature of real-world time series data with distributional shifts (Huang et al., 2020). Specifically, we introduce a shared pool of prompts stored as distinct key-value pairs. Ideally, we want the model to leverage related past experiences, where similar input time series tend to retrieve the same group of prompts from the pool (Wang et al., 2022b). This would allow the model to selectively recall the most representative prompts at the level of individual time series instance input. In addition, this approach can enhance the modeling efficiency and predictive performance, the model would be better equipped to recognize and apply learned patterns across diverse datasets via shared representation pool. Prompts in the pool could encode temporal dependencies, trends, seasonality effects relevant to different time periods. Specifically, the pool of prompt key-value pairs is defined as:
(5) where M is length of prompt pool, Vm E RL,xLE is single prompt with token length I. and the same embedding size LE as Pjj and km KK- {km)== with the shape of R. The score-matching process can be formulated with the score-matching function 7 (Pt, = Pj. k../ |mmll Il. where 7 RLR RL - R. The model is trained an end-to-end way to optimize predictions with the prompts. The query Pj- that used to retrieve the top-K corresponding value comes from the patched time series input. Therefore, similar time series can be assigned to similar prompts. Denoting {s, {s,} subset of K. indices for the selected top-K prompts, our input embedding of trend is as follows:
VK- [(k,, Vi), (k2, V2) (kM, Var)),
DT [V., Voe:
(6)
where we concatenate all the tokens along the temporal length dimension, so as Dg. Eg. Each instance be assigned to multiple prompts, which can jointly encode knowledge pertinent to the forecasting task- such as periodic patterns exhibited by the time series, prevailing trends, or seasonality effects.
22

---

Published as a conference paper at ICLR 2024
Table 9: Compare the different prompt designs on the ETTm2 dataset.
Semi-soft
Soft
Hard
Pool
Pool mask all
mse/mae
mse/mae
mse/mae
mse/mae
mse/mae
96
0.182/0.263
0.19//0.271
0.182/0.267
0.185/0.267
0.1952/0.274
192
0.243/0.304
0.252/0.307
0.243/0.308 302
0.242/0.304
0.2739/0.324
336
0.309/0.344
0.306/0.348
0.299/0.340
0.289/0.336
0.3131/0.354
720
0.384/0.392
0.386/0.394
0.380/0.392
0.373/00386
0.3794/0.390
Avg.
0.280/0.326
0.283/0.330
0.276/0.325
0.273/0.323
0.290/0.335
D.2 RESULTS ON DIFFERENT PROMPT DESIGN
In this section, we examine the impact of various prompt designs on model performance. We utilize the 'semi-soft' prompt outlined in Section 3.3, where the prompt vectors initialized semi-softly; the soft prompt, which entails the random initialization of vectors of identical dimensions to the 'semi-soft' prompt; and the hard prompt, which is semantically meaningful and remains fixed post- tokenization. Additionally, we explore the prompt pool, as described in Section D.I, and employ similar leave-one-our approach mask all prompts within the pool to investigate its effectiveness.
The findings, presented in Table 9, reveal that, in the ETTm2 dataset, the prompt pool outperforms the 'semi-soft' prompt three out of four scenarios, underscoring the potential of prompts to enhance model capacity and adaptability to shifts in data distribution. Furthermore, we observe that prompts with explicit semantic content (Semi-soft and Hard) surpass the performance of simple soft prompts. This suggests that incorporating semantic information discrete indicators within pre-trained model can more effectively orchestrate domain knowledge. This understanding informs the design of prompts for efficient interaction with language models, especially in applications where precision and relevance of the output are crucial.
D.3 ANALYSIS ON PROMPT POOL
Here a summary of how the prompts are initialized and trained in our work:
Initialization: The prompt embeddings in the pool are randomly initialized from normal distribution, as is standard practice for trainable parameters in neural networks.
Training: The prompts' value and all other model parameters are trained in an end-to-end manner to optimize the forecasting objective. This allows the prompts to be continuously updated to encode relevant temporal knowledge.
The number of prompts and embedding dimensions are treated as hyperparameters and tuned for good performance. Different pool settings, including pool size, top number, and prompt length, will lead different results. To explore this, we conduct total of 27 experiments, setting distinct values for each of the settings: (1) pool size 10, 20, and 30. (2) top & numbers of 1, 2, and 3. (3) prompt lengths of 1, 2, and 3. We choose the combination with the best results for TEMPO settings. For the long-term and short-term forecasting experiments, we choose a pool size with = 30 and K== and prompt length is 3. Detailed design analysis provides insights into prompt similarity and selection. Note that, the prompt pool's 's key in (Wang et al., 2022b) is trainable which allows us to maintain consistent and distinct characteristics of time series data for analysis. Our work offers an initial exploration into compttbassed tuning for time series forecasting, but substantial room remains for advancing prompt pool design.
D.3.1 PROMPT SELECTION DISTRIBUTION
To elucidate the mechanics behind prompt selection, we have visualized the distribution histograms for chosen prompts corresponding to the trend, seasonal, and residual elements of the ETTm2 dataset Figure 1.. In our experimental framework, each data point is permitted to select multiple prompts-we with three prompts being chosen per component. Consequently, the frequency is deter- mined by the number of times a particular prompt is selected across the dataset. The histograms reveal pronounced discrepancies in prompt preferences between periodic and seasonal components.
23

---

Published as a conference paper at ICLR 2024
0.9
Season
0.8
0.5
0.2
0.1
Prompt
Figure 11: Prompt Distribution for prompt pool selection.
For instance, within the ETTm2 dataset, prompts 11, 20, and 24 are predominantly selected for capturing trends, whereas prompts 10, and 29 are primarily chosen for seasonal fluctuations. This observation substantiates the premise that the prompt pool is adept at furnishing specialized prompts tailored to discrete characteristics of time series data.
D.4 ANALYSIS ON DESIGNS OF INJECTING CONTEXTUAL INFORMATION
Table 10: SMAPE results of GEBDIT dataset with different textual information injection.
TP
TP
Sum
SumP
SumP @
SMAPE
38.77
38.77
38.75
38.90
38.91
38.82
39.04
38.79
17
41.02
41.03
40,95
41.05
41.24
41.08
41.38
41.08
44.03
44.02
44.06
44.10
44.41
44.19
44.73
44.24
In this section, we investigate the influence of various configurations of textual injection and original prompt design from multi-modality perspective. As depicted in Table 10, eight distinct prompt designs were formulated. 'Sum' denotes the utilization of direct summary of textual data prompt, while 'SumP' signifies the use of textual information as a query within the prompt pool. The symbols '+' and "0" represent summation and concatenation operations, respectively. 'TP' stands for 'time series prompt pool,' and 'Semi' indicates 'semi-soft prompt' where we manually design the prompt, with trainable parameters, referred to as "Predict the future time step given the {time series data type)" )" for 3 different time series (Trend, Season, Residual) after decomposition. Each design choice exerts distinct impact on the performance metrics. The direct incorporation of textual information along with the prompt pool yields the most optimal and near-optimal outcomes. In future work, we aim to delve deeper into the analysis of multimodal solution design strategies for time series forecasting.
D.5 HIDDEN REPRESENTATION
Figure 12 demonstrates the difference between the representation of the output hidden space from the pre-trained langauge model. While the representation of time series learned from GPT4TS is centered as a whole, the representation of the decomposed component from TEMPO implies a certain soft boundary between the three components. This is demonstration of how TEMPO is able to learn the representation trend, seasonality, residual parts respectively, which contributes to the superior performance of our model TEMPO.
24

---

Published as a conference paper at ICLR 2024
(a) TEMPO-ETTH1
(b) GPT4TS-ETTh
(c) TEMPO-ETTh2
(d) GPT4TS-ETT12
(e) TEMPO-ETTIE ETTml
(l) GPT4TS-ETTI
TEMPO-ETTm2
(h) GPT4TS-ETTm2
Figure 12: Comparison of GPT4TS representation with TEMPO representation for prediction length 96 using TSNE. Trend red, seasonality blue, residual in green.
25

---

Published as a conference paper at ICLR 2024
D.6 MODEL TRAINING TIME COMPARISON
Figure 13 illustrates the training time of other baseline models in comparison to our model TEMPO. To ensure fairness, calculated the percentage of runtime for models operating identical machines and utilizing equivalent computational resources. Each model's training time is presented as a ratio relative TEMPO's training time. A value less than indicates that the model trains faster than TEMPO, while value greater than 1 suggests the opposite. We use horizontal bars to visually represent each model's relative training time, with the bars extending to the left or right of the central vertical line based on whether they are faster or slower than our model TEMPO, respectively.
Visual Comparison on relative training time of other models and our proposed model TEMPO under channel independent setting.
Figure 13:
BERT
0.32
LLAMA
4.62
TimesNet
5.70
DLinear
0.02
LightTS
0.03
PatchTST
0.20
FEDformer
3.45
ETSformer
0.77
T5
0.43
GPT4TS
0.24
TEMPO
1.00
5
THE ROLES OF GENERALIZED ADDITIVE MODELS (GAM) AND SHAPLEY ADDITIVE EXPLANATIONS (SHAP)
In our paper. GAM and SHAP serve as instrumental tools, not only for affirming anticipated findings but also for yielding deeper insights and explanations into the inner workings of intricate models.
Role of GAM: GAM inherently models the effects of different features as additive compo- nents. This characteristic of GAM provides intrinsic interpretability to TEMPO. It's not merely tool for confirming the absence of patterns in residuals; it also helps us understand how each feature contributes to the final prediction.
Role of SHAP: SHAP helps in attributing feature effects post-hoc explain the predictions made by complex models, which may otherwise be opaque.
The utility of GAM and SHAP in our analysis can be detailed as follows:
Confirmation of Assumptions: the analyses quantitatively confirm assumptions about model behavior with data-driven evidence, rather than just intuition. This substantiation increases the trust and transparency in the model's predictions;
Detecting Unexpected Behaviors: the component attribution could reveal unexpected be- haviors if present. For example, residual impact being higher than expected could indicate overfitting noise.
Providing Nuanced Insights: SHAP provides nuance beyond high-level expectations, like showing the increasing error of seasonal components in longer forecasts.
26

---

Published as a conference paper at ICLR 2024
1.0
Companent Pacermge
0.4
Trend Seasonality
720
Prediction Langtt)
Figure 14: The SHAP (SHapley Additive exPlanations) values of decomposed components of TEMPO for weather dataset.
In our paper, we use the ETTml and weather datasets detailed examples. The full results used to calculate the SHAP value can be found at Table 11. In datasets exhibiting strong seasonality, the seasonal component may display much larger variations than the residual component. Conversely, in datasets with minimal seasonality, the variations between these two components should be more comparable. We can calculate the strength of seasonality via:
Var (Rs)
= max 0, Var (S.) + Var (R.)
When we compare the seasonality strengths of different datasets, we find that ETTml (as shown in Figure 2, with seasonality strength of 0.99) constitutes strongly seasonal data, whereas the weather dataset (depicted in Figure 14 with a seasonality strength of 0.476) exhibits less seasonality and more pronounced trend. These findings align with the conclusions drawn from the SHAP values. The performance degradation of ETTm1, when the prediction length is increased, can be primarily attributed to inaccuracies in the prediction of seasonal terms. In summary, SHAP provides pivotal descriptive power for model transparency, moving beyond intuition. The ability to discern how much and where components contribute enables targeted improvements. These insights can guide us in better leveraging inductive bias to enhance both efficiency and effectiveness in the era of pre-training models. One of the interesting future works is that we can adaptively and selectively optimize specific components based on the GAM structure and SHAP scores during the training process. This approach would allow us to focus our computational resources and efforts on the most influential components, thereby improving the overall effectiveness of the model.
F BASELINE MODEL EXPLANATIONS
We demonstrate the baseline models we compared with in our experiments in the following:
DLinear (Zeng et al., 2023): DLinear combines decomposition scheme from Autoformer and FEDformer with linear layers to predict time series data by modeling trend and seasonal components separately and summing their features for enhanced performance in trend-rich datasets.
PatchTST (Nie et al., 2023): PatchTST is Transformer-bases model for multivariate time series forecasting that segments data into subseries patches and uses channel-independen design to efficiently reduce computational costs while enhancing long-term prediction accuracy.
27

---

Published as a conference paper at ICLR 2024
Table 11: SHAP original values for each component
trend
SCuson
residual
ScasOn
residual
emply
complete set
MSE/MAE
MSE/MAE
MSE/MAE
MSE/MAE
MSE/MAE
MSE/MAE
MSE/MAE
MSE/MAE
0.437/0.432
0.670/0.226
0,457/0.441
0.663/0.541
0,472/0.455
0.68000.535
L.14/0.790
0,438/0.422
192
0,466/0.447
0.646/0.518
0.488/0.455
0.682/0.529
0.483/0.455
0.66640.526
1.101/0.789
0.461/0.432
336
0.505/0.466
0.672/0.530
0.526/0.476
0.680/0.531
0.52410.475
0.707/0.543
1.102/0.790
0.515/0:467
0.579/0.507
0.678/0.549
0.586/0.508
0.684/0.548
0.592/0.509
0.709/0.558
1.105/0.794
0.497/0.463
0.666/0.531
0.514/0.470
0.677/0.537
0.518/0.474
0.691/0.540
1.103/0791
0.501/0.458
0.213/0.267
0.202/0.261
0.205/0.264
0.223/0.289
0.234/0.293
0.220/0.284
0.637/0.608
0.21
192
0.266/0.317
0.251/0.297
0,256/0.306
0.254/0.304
0.290M0.335
0.262/0.316
0.638/0.608
0.254/0.296
Weather
336
0,317/0.356
0,290/0.333
0.295/0.331
0.293/0.331
0.328/0.357
0.31330.356
0.64010.609
0.292/0.332
720
0.402/0.401
0.371/0.383
0.377/0.380
0.364/0.378
0.389/0.393
0.38540.398
0.638/0.610
0.370/0.379
Avg
0.3000.335
0.279/0.318
0.283/0.320
0.283/0.325
0.310/0.345
0.295/0.339
0.638/0.609
0.282/0.3116
FEDformer (Zhou al., 2022): FEDformer combines seasonal-tres decomposition with Transformers for time series forecasting, leveraging frequency insights for efficiency and accuracy, outperforming state-of-the-at methods.
Informer (Zhou al., 2021): Informer is transformer- model optimized for long sequence time-series forecasting, leveraging ProbSparse self-attention for efficiency, self- attention distilling for handling long inputs, and a generative decoder for rapid predictions.
ETSformer (Woo et al., 2022): ETSformer is a novel Transformer architecture for time- series forecasting that integrates exponential smoothing principles, replacing traditional self-attention with exponential smoothing attention and frequency attention, enhance accuracy, efficiency, and interpretability.
TimesNet (Wu et al., 2023): TimesNet transforms ID time series into 2D tensors capturing intra- and inter-period variations and uses TimesB lock with inception block to extract complex temporal patterns, excelling in multiple time series tasks.
GPT-2 (Radford et al., 2019): GPT-2 a decoder-based language model developed by OpenAl, designed to generate coherent and diverse textual content from given prompt. In our work, we use the GPT-2 with 6 layers as the backbone, which adapted from GPT4TS (Zhou et al., 2023).
BERT (Devlin et al., 2019): BERT (Bidirectional Encoder Representations from Trans- formers) is an encoder-based deep learning model utilizing the Transformer architecture designed by Google to understand the context of words in sentence by analyzing text bi-directionally.
T5 (Raffel et al., 2020): T5 (Text-to-Text Transfer Transformer) a state-of-ther neural network model with encoder-decoder based architecture designed by Google that converts every language problem into text-to-text format.
LLaMA (Touvron al., 2023): LLaMA (Large Langauge Model Meta AI) is collection of state-of-the-art foundation language models ranging from 7B to 65B parameters delivering exceptional performance, while significantly reducing the needed computational power and resources. In our work, we use the first 6 layers of 7B LLaMA.
G THEORICAL ANALYSIS
G.1 PROOF OF THEOREM 3.1
Theorem G.1 Suppose that we have time series signal Y(t) = S()) + T(I) + R(t), E [t,, tn). where S(f) is the seasonal signal (periodical), T(t) is the trend signal (non-periodical) and R(t) is the residual signal. Let E = {e1, e,,..., Cn denote a set of orthogonal bases. Let Es C E denote the subset of E which S(t) has non-zero eigenvalues and ET C E denote the subset of E on which T(t) has non-zero eigenvalues. If S() and T(t) are not orthogonal, i.e. Σ ((t,)T(tt) # O, then Er (1 Es + 0, i.e. E can not disentangle the two signals onto two disjoint sets of bases.
28

---

Published as a conference paper at ICLR 2024
Proof 1
We decompose S(t) and [t)) onto E and acquire that S(t) = E a.e, and T(t) = EE b
Then it is obvious that e, E Eg > a, + and e E ET = bi # Now. let us consider the inner product of S(t) and T(t):
S(L)T()) ((t)(((())))))
acee) ② b.e,) Eb,e,
Eabjere i
(8)
Note that the components found by PCA is set of orthogonal basis. Thus, for any i + j. we have ecej = O. Thus, we have:
Σ S(1)T(()) = S(1) T(I) = =E ases) ② bsei) = E aabillea
Note that Ex=1 ((4)T(())) = O. Thus, there must be al least one such that a; + 0 Dnd by O. Thus, E Es and e, E Er, in other words, Er n Es * 0.
The above theorem proves that if T(t) and S(t) are not orthogonal, then there does not exist set of orthogonal bases that disentangle S(t) and T(t) onto two disjoint sets of bases. Note that is common that periodical signal not orthogonal with non-periodical signal. This is because the spectrum of a periodical signal is discrete and the spectrum of a periodical signal is continuous. Thus, it is very likely that there exist overlaps on those non- non-zero frequencies of the periodical signal. Note that PCA also aims at learning a set of orthogonal bases on the data. We can quickly acquire a corollary that PCA can not disentangle the two signals into two disjoint sets of bases. Based on (Zhou et al., 2023)'s Theorem we can reveal that self-attention in pre-trained large models learns to perform a function closely related to PCA. Therefore, the self-attention mechanism cannot automatically decompose the time series into its trend and seasonal components unless we manually perform this operation.
G.2 INTERPRETING MODEL PREDICTIONS FROM FREQUENCY DOMAIN
In addition to Section 5.2, which gives an experimental perspective on why decomposition can aid forecasting results, we provide theoretical analysis from the spectral domain. Specifically, time series signals can be represented Forecasting challenging because real-world series comprises convoluted mixtures of variations with overlapping periodicities. However, by shifting our view to the frequency domain, identify distinct components via STL decomposition containing isolated frequencies that stand out clearly from the rest of the spectrum. This separation of dominant periodic patterns is crucial because forecasting future values equates to predicting how these underlying frequencies evolve over time:
combination of different frequencies in the spectral domain.
Proposition G.2 (Equivalence of time domain forecasting and frequency domain forecasting ) Assume 2o. ø1, IN-1 and iq ø1..., -1, N are the input and output sequences of the frequency model. Then, 2N transferred from the frequency domain is the predicted value at timestamp N.
Given input sequence {a. 0, 1,..., N - 1}, where N is the number of discrete timestamps, in the time domain, the Discrete Fourier Transform (DFT, F)) and inverse Discrete Fourier Transform (iDFT, /') operation to obtain the frequency domain can be defined as:
1 N
F(u)
Σ
f(z)e == 0, O.
(10)
2=0
N-1
f(x) = Σ
F(uje
Zzur
x = O..,....
(11)
According to Proposition G.2, assuming that the next value of F(u), can be predicted as FNN.. other unknown variables in the time and frequency domains, including the (N + 1)th discrete sample f(N) and the DFT's result F'(u), u 0,1,2,..., N are determined by the given F'(N).
29

---

Published as a conference paper at ICLR 2024
Proof 2 Let
N-1
=
Σ
f(z)(
N+T
(12)
N N+1
N-1
B
f(Z)E
12 N+1
(13)
N
2=0
then we have:
S(N) (N + ()(FF(N) - B)e
N+1
(14)
F'(u) A+ (F' - B) For 2,,, 1, the value of F'(u)) F(u) can be represented as:
(15)
F A
N + f(N)e
M N+1
(16)
For u N, the value of (NN) can be represented as
12p22 F(N) B S(N)e
1
NFr
(17)
Given FNN,,, we can inference (u) by:
F'(u)= - (F((((N) B)e
1
, ,0.0,1,2...,
(18)
and f(N) by:
(N - B)e NII
(19)
Thus, the only variable that needs to be predicted is (NN)..
This proposition reveals that it is easy to predict patterns in the frequency domain, we can more easily predict the time series' future values. Forecasting equates to predicting the evolution of the underlying frequencies that make up the time series signal. STL decomposition significantly aids this task by separating components with distinct dominant periodic patterns. With STL, each component presents far fewer intertwining periodic influences to disentangle, which notably simplifies the prediction problem. For instance, the trend component may exhibit lone annual cycle that clearly dominates its spectrum. A targeted predictive model focusing solely on accurately estimating the progression of this isolated frequency can generate accurate forecasts. Likewise, the seasonal element neatly isolates recurring daily or weekly frequencies. Models tailored specifically for these known periodicities allow for highly predictable extrapolations. In contrast, directly modeling the raw data's 's condensed spectrum with numerous blended periodic components yiclds unsatisfactory approximations. The overlapping frequencies are difficult distinguish and predict independently.
Conceptualizing forecasting through a frequency domain lens reveals how STL decomposes complex spectral mixtures into distinguishable frequency- based sub-problems. This allows implementation optimized predictive strategies to uncover patterns in each component for markedly improved time series predictions. essence, STL. facilitates accurate future predictions by disentangling the spectral content into simpler predictable forms.
H DETAIL OF THE TETS DATASET
Time series data Analyzing and forecasting company's future profitability and viability essential for its development and investment strategies. Financial assessment and prediction are data-driven, mostly relying the combination of diverse data types including company reports, etc. In this project, our primary sources are the company's 's financial statements: balanced sheet, income statements, and cash flow statements.
The Standard Poor's 500 Index (S&P 500) represents a stock market index that measures the stock performance of the 500 largest companies in the U.S. sectors in the S&P50 are included
30

---

Published as a conference paper at ICLR 2024
in our dataset: Basic Materials (21 companies), Communication Services (26 companies). Energy (22 companies), Financial Services (69 companies), Healthcare (65 companies), Technology (71 companies), Utilities (30 companies), Consumer Cyclical (58 companies), Consumer Defensive (36 companies), Industrials (73 companies), Real Estate (32 companies). In terms of dataset division, we separate the sectors in our dataset to achieve both in-domain task setting and zero-shot task setting. The first seven sectors are treated as training and evaluation sectors, while the last four sectors are reserved as unseen sectors for zero-shot forecasting task.
To address missing numerical information for companies in the S&P 500 that lack data prior to 2010, we apply linear interpolation after experimenting with various methods. Linear interpolation is a technique that estimates a value within range using two known end-point values. For missing values in research and development expenses, we adopted zero-filling strategy. This because null entries in these statements typically indicate that the company did not make any investment in that area.
Contextual data collection This rise of Large- pre-trained models (LLMs) in the field of Natural Langauge Processing has provided new possibilities for their application in time seris analysis. LLMS have proven useful for analyzing and learning complicated relationships and making inferences across different time series sequences. However, most existing approaches primarily convert time series data to direct input into LLMs, overlooking the fact that the LLMs are pre-trained specifically for natural language and thus neglecting the incorporation of contextual data.
Further, the information contained in time series data is limited, especially in the financial field. Time series data in the financial field, such as company statements, primarily reflect the financial numeric changes based on the company's 's historical strategy and broader macroeconomic shifts. These data contain the company's internal historical information. However, the broader market environment, referred to as external information, also plays an important role in the company's future development. For example, medicine and healthcare companies experienced steady growth before the outbreak of COVID-19, 19. But between 2019 and 2020, after the outbreak of the pandemic, the financial statements of such companies were impacted significantly. As a result, we recognize the value of integrating news and reports external data sources to complement internal information contained in time series data. The information contained in the external data mainly includes 3 parts: (i). Policy shifts across regions (ii). Significant events occurring globally (iii.. Public reaction to companies' products. Together, these elements provide supplementary information missing in time series data (internal data), therefore enhancing our forecasting capabilities.
Extracting contextual data, such as news and reports, from varied sources presents a significant challenge. In today's ' digital age, numerous news websites and apps deliver wide range of world news, spanning from influential news affecting entire industries to trivial minor reports. Thus, it is crucial filter and summarize the information, distinguishing between pivotal and less significant news. Fortunately, the recently released ChatGPT API by Open Al offers the capability of collecting and summarizing news and reports for a specified duration.
Through consolidating all relevant details query, quarter, yearly context, company information, and specific requirements into user message and setting a cap at 1 tokens for response, we can efficiently obtain the desired contextual information from ChatGPT API. For illustration, Figure 10 displays an example from company A, showcasing designed prompts and corresponding responses from ChatGPT 3.5. If the contextual information can not be generated, the API often returns messages with keywords such as 'unfortumately" and 'sorry'. We detect and replace them with the term 'None', representing neutral contextual information. Additionally, Figure 15 and 17 provide illustration of our dataset, encompassing both time series data and the corresponding contextual texts. A detailed view of the contextual texts can be seen in Figure 16 and 18.
31

---

Published as a conference paper at ICLR 2024
EBITDA Company Technology
LARE: 2006 12.51
EBITDA
SunIN iN fiss, spatta of 2003.
valee 1523000200
dEC 2008 01
Tampony A stn.
$78500038.0
volue e500C000 D
Compaty brthe-acledd 2005.
peotccot 537 FITE
fot you st aoeen ppen pede pour Avo
port
wor aae
Date
Figure 15: EBITDA for Company A with contextual information
Company reported net profit of $32 million, its highest third third-quarter profit in four years, and released its new Product M.
2002 third quarter
In the second quarter of 2005, Company A's profits rose 425%, with Product sales accounting for most of the increase. The company also announced plans to start using technique computers.
2
2005 second quarter
fourth quarter 2007, Company A announced record-breaking of million Product S, and also launched their revamped line of Product N.
2007 fourth quarter
In the first quarter of 2009, Company A reported 1% decline In sales and a 17% drop profits compared to downtum as a contributing factor. The company also announced the release of the U technique software and the new Product IS.
period the previous year, citing the global economic
2008 first quarter
Figure 16: Example of generated contextual information for Company A marked in Figure 15
FRITDA Company
valce 836000000.0 g Company
reported
dale 2008-29-30 summary Company
reported $0.55
EBITDA
GatE 164505mA D summary Compony cartings o...
value. 415770000.0
summaryy 2012 Compuny 5 reported
wor prrecciio pesaaci pau sore
Date
Figure 17: EBITDA for Company with contextual informatino
32

---

Published as a conference paper at ICLR 2024
Company reported fourth-quarter earnings of $189 million, supported by continued growth in its electronics unit. The company also announced plans to acquire the company for $143 million
2006 fourth quarter
Company reported net income of $0.95 share for Q4 of 2008, down from $1.21 per share in the same quarter of the previous year. The company also experienced a decrease sales to economic recession.
2
2008 fourth quarter
2012's third quarter, Company B reported weaker-than-expesie earnings due decline in its business, but still projected higher sales and profits for the year. The company also announced plans to expand its production facilities D Russia.
3 2012 third quarter
In 4Q 2012, Company reported a net income of $138.7 million, down from $289.3 million same quarter of 2011; the company's revenue also decreased by 6.2% to $2.56 billion.
2012 fourth quarter
Figure 18: Example of generated contextual information for Company marked in Figure 17
Table 12: Table of Main Notation on TEMPO
Notation
Description
channel prediction at time step
x;
tt channel look back window/historical values time step
model parameter
V
prompt value from prompt pool
X
input data which be decomposed into Xr X X
Xrt, Xst, XRa
trend, season, residual component set in time
x/e
tt channel tt timestep of 2
X
predict value of trend component
P
patch of input data m th key in prompt pool
km
Vm
m th value in prompt pool
V
prompt pool
AC
hyperparameter, number of prompts to choose
M
hyperparameter, length of prompt pool
z
GPT output for (trend, seasonal, residual)
Lg
prediction length
Lx
embedding vector length
Y.
final predict value before de- de-normalization final predict value
33


Reviews:
[{"review_id": "tk1gvink0K", "reviewer": "Reviewer_2NZn", "summary": "This paper proposes a new interpretable solution, TEMPO, for time series forecasting using the power of pre-trained generative models. Specifically, this work first addresses that the semantic information inside time series is important, for example, the authors utilize the trend, seasonality, and residual information to build the tokens of the pre-trained generative models. In addition, this paper proposes a prompt pool to memory the historical patterns of different time series. The reasonable design and results indicate that TEMPO paves the path of further exploring the pre-training models’ power for time series problems.", "questions": "1. Do you have any insights on the prompt pool’s marginal improvement, which is somehow against the intuition;\n2. The description of TETS dataset is suggested to include into  the main paper;\n3. Can you provide the reason on why only decoder-based GPT is included in the experiments?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper’s writing is clear and easy to follow. For example, the methodology part gives a clear description on how to build the time series input representation and the design of the prompt pool. The experiments on long-term time series forecasting, short-term forecasting and towards foundation model’s training are well organized to prove the model’s power from different aspects.\n2. The proposed solution is well motivated: the motivation of decomposition is supported by both empirically and theoretically and the introduce of retrieval-based prompt selection can help the large pre-trained model handle complex non-stationary time series data with distribution shifts.\n3. Utilizing the pre-trained transformer backbone, TEMPO give a state-of-the-art results on the popular time series research dataset.", "weaknesses": "1. The prompt pool’s improvement is limited: the prompt pool is supposed to have more contribution to the accuracy as the intuition is clear and convincing.\n2. The collection of TETS dataset is not clear: a clear but simple discription in the main paper is necessary.\n3. It seems only decoder-based pretrain model is considered in this paper. The encoder based backbone (like Bert) and encoder-decoder based backbone (like T5) is also recommend in this stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I4gu39uxqe", "reviewer": "Reviewer_W4kZ", "summary": "Authors propose TEMPO leveraging a pre-trained language model for time-series forecasting tasks. The two main components of the proposed approach: the decomposition of time series into trend, seasonality, and residuals, as well as the prompt learning, effectively increase the forecasting performance. The improvement is significant. Also, the paper demonstrates the ability of the proposed method to be trained with few-shots and to be adapted to unseen datasets.", "questions": "1) In the few-shot learning setting, do you need to finetune the model on other time-series datasets first?\n2) What is the computation complexity of the proposed method, and how it is comparable to other methods?\n3) Based on the ablation study in Table 9, it seems that w/o prompts have less effect on the final performance. Sometimes without prompts, the performance can be even better. Could you explain this?\n4) How the prompts are initialized and trained.\n5) In Table 1, why don’t you include the traffic forecasting results?\n6) How generalizable can the model be for the unseen domain transfer? For example, what if you choose weather and traffic as the training domain then apply the model to EETm1 and EETm2", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well-written.\n2) The improvement of forecasting performance over benchmark methods is significant and consistent across datasets.", "weaknesses": "1) The idea of decomposition of trend, seasonality, and residuals is not that novel and has been used for time-series forecasting.\n2) The theorem 3.1 does not directly prove the point “more importance in current transformer-based methods as the attention mechanism, in theory, may not disentangle the disorthogonal trend and season signals automatically\".\n3) The result shown in Figure 2 seems to be obvious since the trend is easier to learn and may take a large portion of the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G9AdPXMHXn", "reviewer": "Reviewer_izvx", "summary": "The paper proposes a way to leverage pre-trained language models for time-series forecasting. Their method is based on two key ideas: (1) decomposing time-series into trend, season and residual components can aid time-series forecasting, and (2) prompting large language models based on a shared pool of prompts.", "questions": "1. Theorem 3.1 -- I do not fully understand the implications of Theorem 3.1 and how that affects the design choices of the authors. What do we learn from the theorem?\n2. Model design: The authors mention they use temporal encoding from PatchTST but do not provide any definition or references for it. As far as I am aware, PatchTST does not use temporal encoding. That is, if my understanding of temporal encoding is accurate. Moreover, work prior to PatchTST, either Informer or Autoformer use temporal encoding instead.\n3. Experimentation: How are LLM-based models like BERT, T5, LLaMA used for forecasting? What is the input to these models? What is the output? \n4. Experimentation: What is the context length for the long-horizon forecasting task?\n5. Experimentation: How are 5/10% of the data points used? How do you define a data point in a long-horizon forecasting time-series? Also 5-10% seems like a lot of data, given that most of the time-series have multiple channels, and the model processes them independently.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Writing: The paper is well-written for the most part. \n2. Interpretability: The authors aim to shed some light on the time-series predictions made by the model. \n3. Modeling time-series and text together: I really liked this key insight of the paper. I think it is under-explored and valuable. \n4. Theory: The authors provide some theoretical insight into the design decisions behind their model.", "weaknesses": "Claims:\n1. While achieving strong prediction performance, the previous works on timeseries mostly benefit ... that captures temporal dependencies but overlooks a series of intricate patterns within timeseries data, such as seasonality, trend, and residual.\" -- I do not agree with the authors, multiple recent approaches using deep learning for time-series forecasting have decomposed model inputs into trend and seasonal components. See N-BEATS, N-HITS (stacked MLP based models), AutoFormer, as an example.\n2. Theorem 3.1 -- I do not fully understand the implications of Theorem 3.1 and how that affects the design choices of the authors.\n3. Prompt pool captures seasonal and trend components: The authors provide an example of 3 time-series from 1 dataset to demonstrate that the prompts capture season and trend components, but I am not sure this is sufficient evidence. It would be interesting to look at the distribution of prompts for multiple (or all) time-series in one or more datasets, as time-series are clustered based on their trend and/or seasonality components. I believe this would give a more, dataset level evidence for the authors' claims.\n4. Interpretability: I am not sure how the GAM and SHAP provide interpretability, beyond confirming what is expected from these models, i.e. the residuals do not have any pattern. \n\nExperimentation: \n1. \"Large-scale experiments/benchmarks\": The authors omit several benchmarks, and therefore I would argue that the experiments are not large-scale. For e.g., for long horizon datasets, the authors do not use the Influenza-like Illnesses and Exchange-rate datasets which PatchTST and TimesNet, and other recent studies. Secondly, there are multiple short-horizon benchmarks, like M3 or the M4 datasets, and the much larger and comprehensive Monash Forecasting archive, yet the authors do not confirm their methods on these datasets. \n2. Multiple methods are omitted from the evaluation, for e.g. statistical methods such as Auto-ARIMA, Auto-THETA, Auto-ETS etc., and deep learning methods such as N-HITS and N-BEATS. Also the authors cite PromptCast but do not compare their method to this particular baseline.\n3. The value of prompt pool-- The authors demonstrate in Table 9 that the prompt pool helps model prediction. How would they explain the methods without prompt pooling doing better on some datasets for some forecasting horizons?\n\nClarity:\n1. Insufficient details in model design and experimentation. See Questions.\n\nMinor: \n1. Typos: inclduing, outpemforms ... etc. \n2. References: I would encourage the authors to find references to accepted papers, instead of citing their ArXiv versions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)