AutoTimes: Autoregressive Time Series Forecasters via Large Language Models
Authors: Yong Liu
Guo Qin
Xiangdong Huang
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2024 poster (2024)
Keywords: Time Series Forecasting
Large Language Models
Primary area: deep_learning_architectures
TL;DR: We propose AutoTimes to repurpose large language models as autoregressive time series forecasters, which exhibits zero-shot, in-context learning and multimodal utilizability with state-of-the-art performance and efficiency.

Abstract: Foundation models of time series have not been fully developed due to the limited availability of time series corpora and the underexploration of scalable pre-training. Based on the similar sequential formulation of time series and natural language, increasing research demonstrates the feasibility of leveraging large language models (LLM) for time series. Nevertheless, the inherent autoregressive property and decoder-only architecture of LLMs have not been fully considered, resulting in insufficient utilization of LLM abilities. To fully revitalize the general-purpose token transition and multi-step generation capability of large language models, we propose AutoTimes to repurpose LLMs as autoregressive time series forecasters, which projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. Compatible with any decoder-only LLMs, the consequent forecaster exhibits the flexibility of the lookback length and scalability with larger LLMs. Further, we formulate time series as prompts, extending the context for prediction beyond the lookback window, termed in-context forecasting. By introducing LLM-embedded textual timestamps, AutoTimes can utilize chronological information to align multivariate time series. Empirically, AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5\times$ training/inference speedup compared to advanced LLM-based forecasters. Code is available at this repository: https://github.com/thuml/AutoTimes.

AutoTimes: Autoregressive Time Series Forecasters via Large Language Models
Yong Liu, Guo Qin; Xiangdong Huang, Jianmin Wang, Mingsheng Long
School of Software, BNRist, Tsinghua University, China
[liuyong21, 2anguo244-0mille tsinghua..eeee. cn,
(huangxdong, jaanonngggjamaaaae immangg (tsngggghhn)ttingguaaa edu.
Abstract
Foundation models of time series have not been fully developed due to the limited availability of time series corpora and the underexploration of scalable pre-training. Based on the similar sequential formulation of time series and natural language, increasing research demonstrates the feasibility of leveraging large language mod- els (LLM) for time series. Nevertheless, the inherent autoregressive property and decoder-only architecture of LLMs have not been fully considered, resulting in insufficient utilization of LLM abilities. To fully revitalize the general-purpent token transition and multi-step generation capability of large language models, we propose Auto Times to repurpose LLMs as Autoregressive Time series forecasters, which projects time series into the embedding space of language tokens and autore- gressively generates future predictions with arbitrary lengths. Compatible with any decoder-only LLMs, the consequent forecaster exhibits the flexibility of the look- back length and scalability with larger LLMs. Further, we formulate time series as prompts, extending the context for prediction beyond the lookback window, termed in-context forecasting. By introducing I.L.Membedded textual timestamps, Auto- Times can utilize chronological information align multivariate time series. Em- pirically, Auto achieves state-of-te wate-of-theeare the-art with 0.1% trainable parameters and over 5x training/inference speedup compared to advanced LLM-based forecasters. Code is available this repository: https: // //g gitgtu......... com/ wtwwm//uutooiiiiii
Introduction
Time series forecasting is of crucial demand real-world applications, covering various domains including climate, economics, energy, operations, etc. [22] 43). The growing challenges of general- purpose forecasting, where one model is versatile to handle variable-length scenarios [24 41
and the
prediction is necessarily instructed by auxiliary information in other modalities [40
44 underscore
the demand for foundation models [3] of time series, which are aimed to exhibit enhanced capabilities,
including multi-step generation, zero-shot generalization 49 13 in-context learning and multimodal
utilization [15], thereby expanding the scope of time series forecasting to a wider range of situations.
Nevertheless, the development of time series foundation models has been hampered by the limited availability of large-scale pre-training datasets and the technical uncertainty of scalable backbones.
In contrast, rapid progress is witnessed in large language models (LLM), facilitated by extensive text corpora [50], available pre-trained models (36), and well-established adaptation techniques (14). Notably, language and time series share basic commonalities in sequence modeling and generation by learned token transitions, presenting opportunities to adopt off-the-shelf LLMs for time series.
'qqual Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Time Series
(a) Forecasting Approach
(b) Prompting Mechanism
LLM
Neso Tokoen Prediction
Relevant Series
Flatien Project
Language as Prompls (Previous)
Lookback Series
LLM
LLM
LLM
Predicted Series
Natural Language
Cuncat
Non-Auforegressive
Autoregressive (Ours)
Time Series as Prompts (Ours)
Figure 1: (a) Prevalent LLM4TS methods non-autoregressivell generate predictions with the globally flattened representation of lookback series, while large language models inherently predict the next tokens by autoregression [47]. (b) Previous methods adopt language prompts that may lead to the modality disparity, while we find time series can be self-prompted, termed in-context forecasting.
Despite recent studies on large language models for time series (LLM4TS) achieving performance breakthroughs in current forecasting benchmarks [15], the mechanism by which LLMs are aligned to the time series modality still remains obscure. The pilot work, FPT 49 leverages LLMs as generic sequential representation extractors for time series, influencing subsequent LLM4TS methodologies. As depicted in [ the non-autoregressive approach, where time series are segmented into tokens, flattens and projects all lookback tokens for the prediction in single step. However, it causes inconsistencies in both model structure and generative approach of LLMs: decoder-only models for autoregressive generation converted to encoder-only and non-autin aat-rrtrorrrssveeeeccooeeeeeeeee forerrergrresiveeoorccsssssss forecasters.
Given that prior studies 9 38 reveal that generalization performance of LLMs is largely derived
from the decoder-only structure trained autoregressively. talents of LLMs may not be fully exhibited. It is also supported by the recent rethinking of previous LLM4TS methods (35], which generally lack the maintenance of autoregression, the essential characteristic of both large language models and statistical forecasters [5 39). Therefore, autoregressive LLM4TS methods are underexplored, which can potentially unlock multi-step generation like LLMs, presenting one model for arbitrary lengths.
Motivated by the reflections, we propose Auto Times adapt L.MSs as time series forecasters, which retrieves the consistency of autoregression with revitalized LLM capabilities to produce foundation models for time series forecasting. Technically, we independently embed time series segments into the latent space of language models by the consistent training objective: next token prediction (2] To fully leverage the inherent token transitions of LLMs and reduce the training cost, we freeze the LLM and establish token embedding and projection for time series, which only account for up to 0.1% total parameters. The consequent forecaster adopts autoregressive inference like LLMs, which no longer constrained to specific (ookback/forecast lengths. Going beyond conventional time series forecasting, we propose in-context forecasting as shown in Figure [ where time series can be self-prompted by relevant contexts. We further adopt L.L.-embedded timestamps as the position embedding to utilize chronological information and align multiple variates. Our contributions are summarized as follows:
By refining the inconsistency of non- non-autoregressvve LLM4TS methods, we propose to inherit the autoregressive property of LLMs, which frees our method from training respectively on the lookback length and allows arbitrary-Iength predictions with chronological awareness.
We present AutoTimes, a simple but effective approach to acquire LLM-based forecasters by lightweight adaptation, which utilizes the inherent token transition as the future extrapolation time series. Further, we propose in-context forecasting, which renovates the conventional paradigm by introducing relevant time series prompts to enhance forecasting.
Compared with state-of-the-ant methods, repurposed forecaster achieves superior perfor- mance while saving over 80% training and inference time, and further exhibits zero-shot generalizability, in-context forecasting, and scaling behavior empowered by LLMs.
2 Related Work
2.1 Autoregressive Models
Autoregression is an essential concept of both language modeling and time series forecasting. De- spite prevalent deep forecasters [10 26 42, 48) adopt non-autoregressive approach without the

---

requirement of iterative forecasting. autoregression, the absent exploration in deep forecasters, serves as the fundamental principle of statistical methods, which enables variable-length predictions. The most well-known model, ARIMA 4 is developed by incorporating differencing AR and MA models, which are both autoregressive models with learned time-invariant transition from the past to the future. Incorporated with decomposition and pre-defined transitions, exponential smoothing [39 and state space models (SSM) [12 23 also take the same autoregressive formulation.
Autoregressive language models 2 are trained with fine-grained supervision, where the gen- erated token of each position is independently supervised. Consequently, they are not constrained by specific input/output lengths and excel at multi-step generation. Furthermore, existing LLMs are inherently autoregressive models [47), which demonstrate advanced abilities that are not present in small models, such as the generalization [38], scalability [6], and task generality [i [2]]. Therefore, it is imperative to adapt off-the-shelf L.L.s as autoregressive forecasters, which keeps the consistency to fully revitalize the model capacity and ceneral-purposs token transitions.
2.2 Large Language Models for Time Series
With the immense advancement of large language model infrastructure, LLM4TS methods have been experiencing significant development in recent years. PrompiCast [44] reformulates time series as text pairs and accomplishes forecasting a series as numerical tokens, demonstrating the zero-shot generalizability in time series forecasting. FPT [49] fine-tunes parameters of the LLM to adapt it general representation extractor serving for multiple time series analysis tasks. UniTTime [21) adapts a language model across diverse time series for unified forecaster of multiple domains. Based on thriving prompting techniques, deft language prompts [1 21 and soft prompting 7 for time series are further investigated.
task. LLMTime [13] regards time
LLM4TS methods have achieved performance breakthroughs in time series forecasting, but the cost of training and inference can sometimes be resource-consumine due to the immensity of LLMs. Recent revisiting of LLM4TS methods has revealed the inefficacy of LLMs adapted in the non-autoregressive approach [35.. By contrast, Time frozen LLMs, transfers the general- purpose token transition, and introduces minimal parameters to realize autoregressive next token prediction, thereby achieving better model efficiency and consistent utilization large models. We further provide Table that categorizes prevalent LLM4TS methods by several essential aspects.
2.3 Multimodal Language Models
Multimodal models have been well-developed upon LLMs, among which vision language models (VLM) have experienced rapid growth [7). The booming pre-trained vision backbones [[] (0) together with the instruction tuning paradigm, has revealed the potential of LLMs for vision tasks, where visual tokens and language tokens are concatenated as the input of the LLM [0). Inspired by this, previous LLM4TS methods utilize instructive language tokens as prefix-prompts for time Unlike previous works, our proposed method regards time series itself as the instructive prompt. avoids the modality gap caused by concatenating time series and language tokens directly. We incorporate chronological information, the textual timestamp of time series, such that the language model can effectively perceive date and periodicity as the position embedding, and align simultaneous events from different time series 22 for multivariate forecasting.
series analysis [1] 34 44
Table 1: Comparison of LLM4TS methods: Autoregressive categories
forecasters by whether to conduct autoregression. Freeze LLM enables quick adaptation, which would otherwise require significant resources for fine-tuning, Multimodal refers to the utilization of information from other modalities. Prior to AutoTimes, none of the LLM4TS methods achieved all three.
Method
AutoTimes TimeLLM [1S] UniTime 21 FPT 49
LLMTame
13
TEST 34
TEMPO ?
PrompiCast [4
Autoregressive
/
Freeze LLM
X X
/
/
Multimodal
X
3 Method
The proposed AutoTimes adapts large language models multivariate time series forecasting. Given lookback observations X1.L {x1,...,x., E LxC with L time steps and C' variates, the objective
3

---

Model Perspective
Token Perspective
Language Transitions
the
quick
brown
fox
jumps
over
the
lazy
dog
OOO
Repurpose
Token-wise Aligument
Forecaster
Time
o0o
Transitions
Figure 2: An example to illustrate how AutoTimes adapts language models for time series forecasting.
predict the future F time steps F = I XL+ Rx Besides, the textual timestamp a (e.g. 2016/07 /05 00:00:00), as the most common covariate, is adopted for prediction, which is aligned with time points x, E I time I. The task is train LLM-bbased forecaster that is able to predict with the (varying) lookback length L for the (arbitrary) forecast length F as:
(XILL,LLLF)) + XL+1.L+..
(I)
3.1 Modality Alignment
Time series token To empower the forecaster with the capability to predict time series for arbitrary lengths, we repurpose autoregressive LLMs as time series forecasters as depicted in Figure 2 Prior to this, define time series token as the consecutive and non-overlapping segment of single variate. It is regarded as the common token of the LLM-based forecaster, which encompasses series variations and mitigates excessively long autoregression. To focus modeling temporal variations, our forecaster predicts each variate independently. Beyond Channel Independence (26] that implicitly captures the multivariate correlation [22] by shared parameters, TutoTimes converts timestamps into position embeddings and explicitly aligns simultaneous segment tokens, which detailed in the next paragraph. Therefore, simplify x, the time point of specific variate I; R. Given single-variate time series of context length NS, the i-th segment of length S is denoted as:
S; (E6--1)5t).....2 (is) 1 i= i.
Considering the general-purposs token transition, we freeze the parameters of large language models. To realize the token-wirg wise alignment between time series tokens and language tokens, we establish RegmeentEmeedddngg(( R + D that independently embeds segments into the latent space:
SE;
Segment Embedding(s.), N,
where D is consistent with the dimension of the LLM.
Position embedding Timestamp, an essential covariate indicating the chronological information, generally utilized as extra embedding in previous deep forecasters 43 48 However, increasing models [10, 26 45) have discarded the embedding and found the performance will not be greatly affected, implying the improper encoding of timestamps. In contrast, textual timestamps have been demonstrated as an enhancement in LLM4TS methods, which are always formulated into prefix- prompts 15 21. Nevertheless, it also leads to excessive context length, impeding LLMs from paying sufficient attention to time series tokens and inducing time-consuming feed-forwarding Inspired by the functionality of position embedding, which incorporates information about the relative or absolute position the tokens [37]. We adopt -emeeddddddd timestamps as position embeddings to utilize temporal information and align simultaneous events (segments) from different varieties.
Technically, we formulate the starting and end timestamps of corresponding segments by the template demonstrated in Figure Experimentally, we observe that the simple template without deft design can consistently boost the forecasting performance in Appendix [D.3] aiding the LLM-based forecaster to comprehend the date and align different variates based on Channel Independence. Since all the previous language tokens are visible to the special ending token <EOS> of sentence, we adopt the embedding of <BOS> TE; E IR D the position embedding from textual timestamps:
TE-- SelectLast [LLM[innslamppTeepaaeeff
Notably, TE, pre-computed by L.LMs such that runtime forwarding for language tokens is not required during training. Given that the latent space of the LLM locates both time series tokens and

---

Ground Truth
Next Token Prediction
Large Language Model
Template
MSE
MSE
MSE
This
Fmberddings of Texts
from 201 6/71 00:00:00 to 2016/7/1 23:00:00
J
Segment Projection
Timestamp
Time Series
(Pre-computed)
2016//11 00:00:00
E
Large Language Model
2016/7/1 2330000
Contest Length
20167/2 0000000
Position Embedding
TR.
SE
TE
SE
TBc
SR
201672 23:00:00
Segment Embedding
201677 00:00:00
Time Series Segments
Segpeent Length
2016/7/1
201 6/7/2
201 6/777
Figure 3: Overview of AutoTTmme:: (1) time series and corresponding timestamps segmented; (2) textual timestamps are converted into the position embeddings by the L.L;; (3) time series segments embedded and projected by next token prediction, where intermediate layers of LLM are frozen.
language tokens, the position embedding can be integrated with the corresponding time span without increasing the context length. Concretely, the token embedding E, E TR. D obtained by:
E, = SE, TE,
(5)
3.2
Next Token Prediction
As shown in Figure 3 prevalent LLMs 6 36 are endowed with the capability predicting the next token s; based on the preceding tokens Szj. We reutilize LLMs in fully consistent approach and generate prediction of arbitrary lengths iteratively. Given time series of context length NS, the input series is segmented and embedded into N token embeddings (EI..... E ). The training objective independently generate the next tokens .......,,,+.. We feed the token embeddings E, into the intermediate layers of the LLM, which inherently parameterize token transitions:
[E Æx+1} } ((LM. ,.,)).,,,,)))
(6)
We adopt Projection :) R + R to independently projects embeddings to segments:
$; Segment (rojection(EE). i = 2...., N 1.
Finally, each predicted segment is supervised by the losen-wise ground truth to optimize the parame- ters of embedding and projection layers, which are simply implemented by multi-layer perceptrons:
LMSF
NS Σ js, & i ..
(8)
Notably, the context length NS decided during training. representing the maximum input length during inference. Therefore, one consequent forecaster is suitable for different input lengths like the LLM, validated in Appendix [D.4] Moreover, AutoTimes can generate predictions arbitrary lengths by iterative multi-step forecasting, proven overcome error accumulation better than state-of-the-an forecasters in Section 4.1 since autoregressive LLMs inherently excel at multi-step generation:
F
8 LLMFForecaster(scc:)) 1==...... L''
Instead of respectively training models different lookback/forecast lengths, AutoTimes handles all the scenarios by one model. Surprisingly, with consistency autoregression, it also inherits notable generalizability and scaling behavior of LLMs, which is demonstrated in Sections 4,2 and
5

---

3.3 In-Context Forecasting
Large language models are capable of generating expected outputs based on provided task demonstra- tions from downstream datasets without gradient updating, known as the in-context learning ability. The task demonstrations are generally constituted by paired questions and answers [47]. Formally, the context C (g(x(1) )..)).... ((x("), .y(m)) represents set of demonstrations with m pairs, where g(-) is the template that transforms each question and answer into natural language.
In terms of time series forecasting, we propose constitute the pair by bookback-forecat windows, which are exactly represented as successive time points from earlier historical observations. Hence, we time series in target datasets prompts, extending the context for prediction beyond consecutive lookback series. We denote the extended context as C, which contains m time series prompts tsp
C = [tsp
Xst, earlier historical time series), J = 1..... m. 1 S L.
(10)
During training, we first obtain an LLM-based forecaster on a source dataset and select time series prompts from the downstream target dataset based on a unified strategy. During inference, we ensure all the prompts appear before the window to be predicted, such that there is data leakage from future information. As shown in Figure we concatenate time series prompts with lookback series and feed them as the context of the forecaster, termed in-context forecasting:
f (LLLL,,,LLL + &L+1LL+F
(1I)
4 Experiments
We conduct thorough evaluations of the performance of AutoTimes, including time series forecasting, zero-shot forecasting, and the proposed in-context forecasting, Additional analyses are included to evaluate the generality, scaling behavior, and adaptation cost of large language models. Detailed code implementation for reproduction is provided in our public code repository.
4.1 Time Series Forecasting
For long-term time series forecasting, we extensively include real-world datasets, including ETTh1, ECL. Traffic, Weather [43,, and Solar-Energy [22] For short-term forecasting, we adopt the well-acknowleedeed M4 competition 225 Detailed descriptions are provided in Appendix A
Benchmarks
Baselines We compare AutoTimes with state-of-the-art models, including advanced LLM4TS methods: TimeLIM (15), UniTime (21), and FPT [49]: well-acknowledged deep forecasters: i'Trans-
former (22). DLinear (45), PatchTST [26). and TimesNet [42]. For the challenging short-term
forecasting, we further include competitive baselines: Koopa (23), N-HiTS (8) and N-BEATS [28]
All baselines are officially implemented or reported. We adopt LLaMA-7B (6) as our base LLM.
Detailed implementations, error bars, and hyperparameter analysis are provided in Appendix B and
Setups For short-term forecasting, we follow the well-acknowledged TimesNet 42 which assesses
the fundamental ability of forecasters in modeling temporal variations. For long-term forecasting, establish a novel one-for-all benchmark: a single forecaster is trained on one dataset and subsequently utilized for all prediction lengths. We highlight that this approach evaluates the basic versatility as foundation models of time series, which aims to break the prevailing practice of extensive training across diverse real-world scenarios. To be specific, we evaluate all methods by rolling forecasting: model is trained with predetermined input/output lengths, and the predicted values are integrated as part of the input in subsequent iterations until reaching the desired forecast length. Therefore, the key to success in this task lies in mitigating multi-step error accumulation. Still, the conventional one-for-one approach that trains forecasters respectively on each length is also provided in Table 12
The average results are presented in Table 2 3 with the best results in bold and the second best underlined. AutoTimes consistently outperforms all counterparts of short-term forecasting in Table demonstrating the basic ability of L.LM-based forecasters to capture diverse series variations. Further, in the one-for-all long-term scenarios, AutoTimes surpasses other LLM4TS methods and deep forecasters in 80% datasets in Table 3 outperforming previous ofte-of-the-art TimeLLM by
Results

---

Table 2: Average short-term forecasting results on the M4 [25]
Full results are provided in Table
Models
AutoTTimes
TimeLLM
FPT
Koopa
N-HiTS
DLinear
PatchTST
TimesNet
FiLM
N-BEATS
Average
sMAPE
11.831
11.983
11.991
11.863
11.960
12.418
13.022
11,930
12.489
11.910
MASE
1.585
1.595
1,600
1.595
1.606
1.656
1.814
1.597
1.690
1.613
OWA
0.850
0.859
0.861
0.858
0.861
0.891
0.954
0.867
0.902
0.862
Table 3: Long-term forecasting results of one-for-all: we conduct rolling forecasting with a single model trained on each dataset and accomplish four desired forecast lengths in {96, 336, 720). AutoTimes adapt LLMs with the context length C = 672. We set the input length L = 672 and output length F 96 in other methods. All results are averaged. Full results is provided in Table I0
Models|
Auto Times
TimeLLM [5]]]]]]]] UniTime 21
FPT 49
iTrans.
22
DLinear
45
PatcTTST
26
TimesNet
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE MAE
ETThl
0,389
0.422 0.412 0.437 0.683 0.596 0.429 0,439 0.421 0.445 0.426 0.444 0.409 0.430 0,495 0.491
ECL
0.159 0.253 0.181 0,288 0.325 0.399 0.184 0.284 0.164 0.258 0.165 0.265 0.169 0.268 0.201 0.303
Weather
0.235
0.273 0.225 0.266 0.461 0.459 0.228 0.266 0.266 0.291 0.239 0.291 0.226 0.268 0.264 0.293
Traffic
0.374
0.264 0.410 0.303
0.584
0.367 0.461 0.326 0.384
0.274 0.423 0.298
0.391 0.275 0.602 0.322
Solar. 0.199 0.242 0.263 0.335 0.392 0.462 0.236 0.303 0.213 0.291 0.222 0.283 0.202 0.269 0.213 0.295
9. 9.12% in average. Compared with other forecasters trained in the one-for-one scenario in Table 12 Times still achieved state-of of-theeart performance in 70% of settings without respective training.
By diving into the proposed one-for-all and the traditional one-for-one benchmarks in Table 3 and 12 it is notable that prevalent deep forecasters, such Transformer-bssed forecasters and DLinear, can achieve competitive and even better results under rolling forecasting. Nevertheless, the performance of non- aoneauooreersssvve LLM4TS methods can degenerate a lot without respective training. Therefore, it highlights our persistent utilization of autoregression and thorough leveraging of inherent token transitions of L.Mss thereby mitigating error accumulation during multi-step rolling forecasting.
4.2 Zero-Shot Forecasting
Setups Large language models have exhibited remarkable zero-shot generalization capability (6) To verify whether our based forecaster inherits this ability, where no training sample of the target domain is available, we assess the performance of zero-shot forecasting, Concretely, we adhere the benchmark established by FPT 49], where the forecaster is initially trained source domain and subsequently evaluated on an unseen target domain. We conduct the transfer learning between the M3 and M4 competitions, both of which encompass abundant temporal variation patterns but follow different data distributions. We compare AutoTTimes with deep forecasters and FPT as the only LLM4TS method, given that only FPT has exhibited zero-shot generalization in this benchmark.
Table 4: Zero-shot forecasting results in averaged SMAPE. M4 + M3 trains forecasters on the datasets of M4 and evaluates on M3, and vice versa. Detailed results are provided in Appendix D.2
Models
AutoTTimes
FPT
DLinear
PatchTST
TimesNet NSFormer FRDFormer Informer Reformer
M4 + M3
12.75
13.06
14.03
13.06
14.17
15.29
13.53
15.82
13.37
MB + M4|
13.036
13.125
15.337
13.228
14.553
14.327
15.047
19.047
14.092
The comprehensive results of zero-shot forecasting are presented in Table 4 AutoTimes demonstrates superior performance compared to deep forecasters and FPT in both M4 - M3 and M3 M4 scenarios. is evident that LLM4TS methods generally achieve improved performance in this task due to the enhanced model capacity, leading to 15% SMAPE reduction compared with the efficient forecaster DLinear. Despite sharing the same Transformer backbone, L.M4TS methods still outperform PST due to the transferable knowledge pre-trained on large corpora of sequences. This underscores the advantage of leveraging L.MS for time series forecasting. Moreover, AutoTimes inherits general-purpose purpose token transitions, surpassing FPT without tuning intermediate LLM layers.
Results
7

---

Zero-Shot Forecasting
In-Context Forecasting
Zero shot Forecasticg
Tarpet Domain Trained on Sounce Demain
S2*
P2
PS
S3
S4'
contect Forecasting
Forecaster
Forecaster
13.51
SMAPE
13.25
S2
S3
S4'
P1
P3
P4
Si
S2
S3
S1
Time Series Prompt
Lookback
Forecast
P18
P21
PS
S2
S4
5B
Averaee
Figure 4: Demonstration of in-context forecasting and results compared with zero-shot. We uniformly select the foremost time points from the target domain as prompts and concatenate them with lookback to obtain the prediction. AutoTimes adapts LLMs on the source domain with larger context length place the additional time series prompt. Supplementary showcases are provided in Figure 12
4.3 In-Context Forecasting
Setups We conduct in-context forecasting on AutoTimes, which depicted in Figure 4 Similar to zero-shot forecasting, the task is to apply a forecaster, trained on source dataset, to an unseen target dataset. Additionally, several task demonstrations from the target domain, referred to time series prompts in Equation[IQ I0 are available during inference. Specifically, we concatenate these prompts with the lookback window form the context for prediction.
We adopt the aforementioned M4 - M3 scenario. Since the samples of the M3 dataset are univariate time series with different lengths, always predict the last F time points of each sample during inference. We set the lookback length L, = F, and thus the length of time series prompts is 2F. 1. Therefore,
We set the number of prompts the source M4 dataset with the context length of 3F, We adopt an intuitive strategy to select the prompt: uniformly adopting the first 2F time points of the time series the corresponding prompt. Supposing the lookback series starts after time / (2 F), in-context forecasting is formulated as:
initially train LLM-based forecaster
f (1212F) Fe+1:+F, 8c4 12+2F) + F+4.41.4...
(12)
To prevent data leakage of future information, too short samples are discarded to prevent the overlap between the prompt and the future prediction. The implementation details of in-context forecasting are provided in Appendix D.6. We further investiigaee different prompt retrieval strategies. Insightful results are provided to reveal the influence of using time series prompts for interactive prediction and take-away instructions prompt engineering in the time series modality.
Results The quantitative results of in-context forecasting are provided on the right of Figure 4 The results of zero-shot forecasting, where downstream demonstration is available, are compared as baseline. Benefiting from the time series prompts the target domain, our LLM-bbsed forecaster with the proposed in in-context forecasting paradigm achieves consistent promotions all MB subsets and the averaged 13.3% SMAPE reduction compared with zero-shot forecasting. In contrast to previous LLM4TS methods that rely deft language prompts, LLMs adopted by Auto Aimesimes can be instructed by time series itself with our intuitive prompting engineering. From the perspective of the forecasting paradigm, extend the prediction context beyond the lookback window. To inherit token transitions language models parameterized by intermediate layers, AutoTimes takes a crucial step by establishing a mapping between time series segments and the latent space of language tokens, which is however absent in non-autoregressive LLM4TS methods. Therefore, ensuring autoregression consistency enhances the effective utilization of LLMs as foundation models.
4.4 Method Analysis
methods focus on applying their approach to specific LLMs. We demonstrate that AutoTimes is compatible with any decoder-only L.MMs. By extensively training M.L--based forecasters by AutoTimes based on prevalent L.MMs, including GPT-2 (3I), OPT (46)
Generality Previous LLM4TS 15
49
and LLaMA (36), we present the results in Table highlighting the generality of AutoTimes.
8

---

Table 5: Averaged results of alternative language models. Full results are provided in Table 18
LLM
GPT-2 (124M)
OPT-350M
OPT-1.3B
OPT-2.7B
OPT-6.7B
LLaMA-7B
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ECL
0.173
0.266
0.168
0.263
0.164
0.258
0.164
0.258
0.162
0.256
0.159
0.253
ETTh1
0.397
0.425
0.401
0.429
0.396
0.424
0.394
0.424
0.394
0.424
0,389
0.423
Traffic
0.406
0.276
0.405
0.277
0.397
0,271
0.394
0.269
0.393
0.270
0.374
0.264
Weather
0.242
0.278
0.240
0.275
0.240
0.276
0.243
0.277
0.247
0.282
0.235
0.273
Scaling behavior Scalability is an essential characteristic that emerges from small models to large foundation models. By investigating the results presented in Table 5 we observe that the prediction accuracy of the forecaster generally improves with the increase in LLM parameters. This scaling behavior of LLM-based forecasters introduces trade-off between performance and adaptation cost. To provide comprehensive assessment, we evaluate each adapted forecaster from three perspectives: performance, training speed, and parameters, presented in Figure ]
We observe that the largest LLaMA-7B consistently delivers optimal forecasting performance. As a relatively small language model, OPT-1.3B exhibits good parameter efficiency out-of-the--be thx-bhxeoxxx forecaster.
ECL
Traffic
GPT-2
Count
GPT-2
OPT-250M
0,400
OPT-6.78
125M
OPT-1.38
MSC
OPT.250M
MSF
0.390
OPT-2.78
0.165
OPT-6.78
Params Count
OPT-1.38
OPT.2.7B
LLaMA.7B
0,380
Lo
0. 0.160-
125M
0,370
125
250
375
500
625
750
875
125
250
375
500
625
750
875
Training Time (ms/iter)
Training Time (ms/iter)
Figure 5: Efficiency comparison of alternative LLMs, evaluated by the same configuration of Table
Adaptation cost To mitigate the substantial cost of adapting large language models, AutoTimes introduces minimal parameters with all intermediate layers of L.M frozen. Additionally, seam- lessly integrate the language tokens (e.g. textual timestamps) without excessive context length and runtime overhead for training, thereby significantly reducing the adaptation cost. Figure 6 presents comprehensive efficiency analysis with advanced LLM4TS methods: FPT is applicable on GPT-2 and TimeLLM is applicable on LLaMA-7B. Not only does AutoTime achieve better results in Table 3 but its training and reasoning time is also greatly reduced, bringing over 5x speedup average. In terms of parameter efficiency, AutoTimes focuses on establishing the embedding for time series segments, which is simply implemented by the MLP (0.79.)))) (0.79M)) account for 0.1% parameters of the LLM (7 Therefore, the results affirm the effectiveness of reutilizing the inherent token transition.
AutoTimes
FPT
TimeLLM
Training Time (s/iter)
Inference Time (s/iter)
Tunable Parameters (M)
1.896
0.8
0.734
45.66
40
0.6
0.4
20
0.284
0.354
0.2-
0.133
0.163
7.01
0.035
0.022
0.44
0.79
0.0
GPT-2
LLaMA-7B
GPT-2
LLaMA-7B
GPT-2
LLaMA-7B
Figure 6: Comparison of AutoTimes and other LLM4TS methods in terms of training/inference time and tunable parameters with the same batch size (224) on the ETTh1 dataset.

---

Ablation study Recent research has raised doubts about the validity of previous LLM4TS meth- ods (35), which predominantly adopt non-autoregreesion, that is, treating the LLM as a BtRT-style pre-trained backbone and utilize a globally flattening projector on all lookback tokens. Here, we provide thorough ablation study to examine our proposed AutoTimes in Table 6 The results under- score that our method maintains the consistency of the decoder-only architecture and autoregressive inference, effectively leveraging LLMs and addressing the concerns regarding performance improve- ment and adaptation cost. Further, we provide a comparison by substituting our token-wise segment projection (consistent with LLMs) with the flatten linear head [26] (common in von-autoregressive forecasters). Results of Table 21 in the Appendix reveal that the performance of non- -utoorrtreeeesivvve generation is consistently inferior to that of our autoregressive Auto AutoTimes approach.
Table 6: We follow the protocol of LLM4TS ablation studies [35 to verify whether the LLM truly useful in our AutoTTmes: (1) w/o LLM replaces the language model entirely and passing imput tokens directly to the last layer; (2) LLM2Amt replaces the language model with a single multi- -haad attention layer; (3) LIM27rsf replaces the language model with a single transformer block.
Dataset
ETTh1
ECL
Type
AutoTTims
w/o L.M
LLMZAIt
L.LM2Tesf
Auto'Times
w/o L.M
LLMZAUn
LLM2Trsf
Metric
MSE MAE
MSE MAE MSE MAE MSE MAE
MSE
MAE
MSE MAE MSE MAE MSE MAE
Pred-96
0.360
0.400 0.365 0.399 0.383 0.404 0.377 0.401
0.129 0.225 0.171 0.263 0.156 0.255 0.162 0.263
Pred-192
0.388
0.419 0.405 0.425 0.414 0,422 0.406 0.420
0.147
0.241
0.192 0.282 0.178 0.276 0.189 0.287
Pred-336
0.401
0.429
0.429
0.441
0.431
0.432
0.421
0.431
0.162
0.258
0.216
0.304
0.198
0.295 0.216 0.309
Pred-720 0.406 0.440 0.450 0.468 0.456 0.454 0.449 0.452 0.199 0.288
0.264 0.342 0.230 0.320 0.258 0.340
LoRA adaptation By incorporating low-rank adaptation technique [4] on the intermediate LLM layers, the token transition of the large language model can be further fine-tuned to align the future extrapolation of time series. Table provides the performance comparing the incorporation of LoRA, which consistently improves the performance of the LLM- -ased forecaster adapted by Auto Times.
Table 7: Full long- term-termmnn forecasting results of AutoTimes and AutoTimes equipped with LoRA 14]
Dataset
ETTh1
ECL
Weather
Traffic
EnlaryEnergy
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
AutoTimes
0.360
0.397
0.140
0.236
0.158
0.208
0.369
0.257
0.179
0.220
Pred-96
LoRA
0.357
0.397
0.130
0.225
0.151
0.201
0.360
0.256
0.176
0.219
AutoTimes
0.391
0,419
0,159
0.253
0.207
0.254
0.394
0.268
0.198
0,236
Pred-192
+ LoRA
0.391
0.420
0.149
0.242
0.197
0.244
0,383
0.267
0.195
0.235
AutoTimes
0.408
0.432
0.177
0.270
0.262
0.298
0.413
0.278
0.213
0.252
Pred-336
LoRA
0.409
0.433
0.164
0,259
0.251
0.287
0.401
0.277
0,208
0,249
AutoTimes
0.429
0.452
0.216
0.303
0.342
0.353
0.449
0.299
0.239
0.277
Pred-720
LoRA
0.426
0.451
0.202
0.293
0.326
0.339
0.440
0.300
0.225
0.268
Conclusion
This paper aims to develop foundation models for time series forecasting. We utilize off-the-shelf LLMs as autoregressive forecasters by transferring the general- purpose and multi-step generation ability. Different from prior methods, notice prevalent non autoregressive LLM4TS methods may contradict the decoder-only structure and lead to insufficient utilization of LLMs. Experimentally. the proposed method achieves state-of-the-un performance with remarkable model efficiency. Further analysis reveals that our forecaster effectively inherits advanced capabilities such as zero-shot and in-context forecasting, and is able to utilize both instructive times series and timestamps. In the future, we will further incorporate advanced low-rank adaptation and utilize booming language backbones.
10

---

Acknowledgments
This work was supported by the Ministry of Industry and Information Technology of China.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 1-23716-23736, 2022.
[2] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
[3] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bemstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint arXiv:200..07258, 2021.
[4] George Box. Box and jenkins: time series analysis. forecasting and control. In A Very British Affair: Six Britons and the Development of Time Series Analysis During the 20th Century, pages 161-215. Springer, 2013.
[5] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are fow-shot leamerrs. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng. Wen Ye, and Yan Liu. Tempo: Prompt- based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04945 2023.
[8] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 6989-6997, 2023.
[9] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Why can gpt leam in-context? language models secretly perform gradient descent meta optimizers. arXiv preprint arXiv:2212. 10559, 2022.
[10] Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-ten Bong-term forecasting with tide: Time-series dense encoder. arXiv preprint WwXiv:2304.08424 2023.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Hcigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[12] James Durbin and Siem Jan Koopman. Time series analysis by state space methods, volume 38. OUP Oxford, 2012.
[13] Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820, 2023.
[14] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0m 106.09685 2021.
[15] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Fang Shirui Pan, et al. Time- Ilm: Time series forecasting by reprogramming large language models. aiviv preprint arXiv:2310.0172 2023.
[16] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 003611, 2020.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.680 69800, 2014.

---

[18] Guokun Lai, Wei-Cheng Chang, Yiming Yang. and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
[19] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping hanguage-imag pre- ve-trraning for unified vision-language understanding and generation. International Conference on Machine Learning, pages 12888-12900. PMIR, 2022.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.080 08485, 2023.
[21] Xu Liu, Junfeng Hu, Yuan Li. Shizhe Diao. Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. arXiv preprint arXiv:2310.0975/ 2023.
[22] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itrans- former: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.
[23] Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. arXiv preprint arXiv:2305. 18803, 2023.
[24] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Transformers for time series analysis at scale. arXiv preprint 2r0::22402.02368, 2024.
[25] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54-74, 2020.
[26] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211. 2022.
[27] R OpenAL. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.
[28] Boris N Oreshkin. Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905. 1et:::9955.0037..... 2019.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 -7633 PMLR. 2021.
[31] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[32] Colin Raffel. Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research. 21(140):1-67, 2020.
[33] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
[34] Chenxi Sun, Yaliang Li. Hongyan Li, and Shenda Hong. Test: Text prototype aligned embedding to activate I1m's ability for time series. arXiv preprini 208.0220,.., 2023.
[35] Mingtian Tan, Mike A Merrill, Vinayak Gupta. Tim Althoff, and Thomas Hartvigsen. Are language models actually useful for time series forecasting? arXiv preprint arXiv:2406.164 2024.
[36] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2302 13971, 2023.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
12

---

[38] Thomas Wang. Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung. Iz Beltagy. Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Muchine Learning, pages 22964-22984, PMLR. 2022.
[39] Peter R Winters, Forecasting sales by exponentially weighted moving averages. Management science. 6(3):324-342, 1960.
[40] Gerald Woo, Chenghao Liu, Akshat Kumar, and Doyen Sahoo. Pushing the limits of pre- pre-training for time series forecasting in the cloudops domain. arXiv preprint arXiv:2310.0066. 2023.
[41] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong. Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprini arXiv:2402. 2024.
[42] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d- ad-variation modeling for general time series analysis. arXiv preprint arXiv:2210002/88, 2022.
[43] Haixu Wu, Jiehui Xu, Jianmin Wang. and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, Advances in Neural Information Processing Systems, 34:22419- 224330, 2021.
[44] Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 2023.
[45] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[46] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 01068, 2022.
[47] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang. Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303.188 182223 2023.
[48] Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li. Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time- series forecasting. Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11101 2021.
[49] Tian Zhou, Peisong Niu, Xue Wang. Liang Sun, and Rong Jin. One all: Power general time series analysis by pretrained Im. arXiv preprint arXiv:2302. 19999, 2023.
[50] Yukun Zhu, Ryan Kiros, Rich Zemel. Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
13

---

A Dataset Descriptions
We conduct experiments to evaluate the performance of the proposed TimeTimes on seven real-world datasets spanning diverse domains: (1) ETThl 488 spans from July 2016 to July 2018 and consists of seven factors related to electricity transformers. (2) Weather [3] encompasses 21 metcorological factors collected every 10 minutes in 2020 from the Weather Station of the Max Planck Biogeochemistry Institute. (3) ECL (43 captures hourly electricity consumption data from 321 clients. (4) Traffic 443 gathers hourly road occupancy rates from 862 sensors on San Francisco Bay area freeways, covering the period from January 2015 to December 2016. (5) Solar-Energy [8] records solar power production from 137 PV plants in 2006, sampled every 10 minutes. (6) M4 is competition dataset encompassing various time series across different frequencies and domains such as business and cconomics. (7) M3, albeit smaller than M4, also contains diverse time series from various domains.
We follow the same data processing and train- validation test set split protocol used in TimesNet [43), where the train, validation, and test datasets are strictly divided according to chronological order to ensure no data leakage. As for long- term forecasting settings, fix the context length of Times and the lookback length of other compared methods as 672 in ETT, ECI., Traffic, Weather, and Solar-Energy, and the forecast length varies in (96, 192, 336, 720). For the short-term forecasting on M4 and M3 datasets, the input length is generally set to twice the output length according to the official implementation of TimesNet. The details are provided in Table g
Table 8: Detailed dataset descriptions. Dim denotes the variate number. Dataset Size denotes the total number of time points in (Train, Validation, Test) splits respectively. Forecast Length denotes the future time points to be predicted. Frequency denotes the sampling interval of time points.
Dataset
Dim
Forecast Length
Dataset Size
Frequency
Information
ETTh1
7
[96, 192, 336, 720}
(8545, 2881)
Hourly
Electricity
Weather
21
(96, 192, 336. 720)
(36792, 5271, 10540)
10min
Weather
ECL
321
[96, 192, 336, 720}
(18317, 2633, 5261)
Hourly
Electricity
Traffic
862
196, 192, 336, 720
(12185, 1757, 3509)
Hourly
Transportation
Solar-Energy
137
196, 192, 336, 720
(36601, 5161, 10417)
10min
Energy
M4-Yearly
6
(23000, 0, 23000)
Yearly
Demographic
M4-Quarterly
8
(24000, 0, 24000)
Quarterly
Finance
M4-Monthly
18
(48000, 0, 48000)
Monthly
Industry
MeeWle
1
13
(359, 0, 359)
Weekly
Macro
M4-Daily
14
(4227, 0, 4227)
Daily
Micro
M4-Hourly
48
(414, 0, 414)
Hourly
Other
M3-Ycarly
6
(645, 0, 645)
Yearly
Demographic
M3-Quarterly
8
(756, 0, 756)
Quarterly
Finance
M3-Monthly
I
18
(1428, 0, 1428)
Monthly
Industry
M3-Others
8
(174, 0, 174)
Weekly
Macro
B
Implementation Details
AutoTimes processes timestamps in textual form rather than numerical encoding, potentially enabling to handle other textual data such as news or logs. We utilize LLM to obtain embedding for the special token <EOS> to capture embedding for the entire sentence. Pseudo-code for this process is depicted in Algorithm[] []] noting that in the context of multivariate time series forecasting, timestamps timestamps can implicitly express relationships between variates even with channel independence. Further, assuming there are C variates since the number of timestamps is C of the total time point count, these embeddings can be efficiently pre- pe-computed by large language models.
It is worth shared across variates. Thus,
After obtaining embedding for the timestamps, we repurpose LLM for time series forecasting using Algorithm 2 At this stage, only the parameters of EmbedditEmeedding and SegmentProjeetioon are updated, while the
14

---

parameters of LLMs remain entirely frozen. During inference. Auto Times utilizes the last token generated as its prediction and then employs this oulput create subsequent predictions autoregressively. This approach enables Times to predict sequences of variable lengths with just one model dynamically. Such capability is crucial real-world application scenarios. The pscudo-code code in Algorithm P illustrates this process.
All the experiments are conducted using PyTorch (29] on NVIDIA A100 GPUs. We employ Adam 17 with an initial learning rate in (10-s, 5 x 10 10041 and MSE loss for model optimization. We adopt Channel Independence [26] for multivariate time series and utilize our position embeddings of timestamps explicitly align them. The batch size is chosen from {256, 1024, 2048), and we set the number of training epochs 10. As for Segment Embeddinng
we implement them by either linear layer or Results of deep forecaster are based on the benchmark provided by the TimesNet [43] repository. which is fairly built on the same configurations provided by the original paper. LLM4TS methods [15 21 49] are implemented by their official and open-source repository. Unless otherwise specified. base L.M. We also present the standard deviation AutoTTimes forccasting performance with three random seeds Table demonstrating that the performance AutoTimes stable.
SegmentProjecten
LLaMA-7B (36) the default
Algorithm 1 AutoTTmes Generate Text Embedding
Require: Input time series X(i-1)8-112
2-th token of length S Model dimension of the L.MM D
1: 8, (#((-1)5+1>>.>> .Ti
2: TE; = Lelect (ILM((Tatttalllrrrttionsss))
D TEŞ E TR.
3: Return TE;
Return textual embedding of i-th token
Algorithm 2 AutoTTmes Repurpose LLM
Require: Input time series {x1..... ((N+1)x text embeddings (TE
TEN
Token
number N
1: for in N1,
2: 8; (T(L-1)8+1 Cis)
8 E RS
3: SE; Segment (mbeediinn)
SE, RD
4: E, =SE, + TE,
E, E RD
5:
LLMLayers((E E.)))
> E E R
6: for in (2...., N .N...,,+... 1):
7: 5;
D R
8: 8N+1 = {INS+I,..., 2N8+))
9: LMSE NS
1 Σ |s: -.1l2, E No....N111
D Objective of the next token prediction
Algorithm 3 AutoTimes LLM Forecasting
Require: Input time series (.,,....,44 UN x text embeddings (TE,...., TE
Lookback
token number N1 S N
1: for i in [1,...NN]:
2: S; (I F4s)
D E R
3: SE, SegmentEEbeeddnn((s)
SE, E R
4: E, SEE; + TE;
E, E R
5: (Éz,... 1N LLMLayers ( EN, ))
É, R
6: for in ( 1}:
7: 8; Segment Projection(E:)
b 8 E RS
8: Return 8N+1
Return last token $ E R the prediction
15

---

Algorithm 4 AutoTiime- Autoregressive Generation
Require: Input time series TN,, textual embeddings (TE ..... TEN token number N2 - N1
Forecast
1: x = (INis!
2: prediction = ()
3: for in (1,..., N2 N1}:
4: y LLMMFrrecatterxx, TEM++-)
>> Details in Algorithm Concatenate for the input for next iteration Record prediction results
5: < (x,}}
6: prediction {prediction,y 9}
7: Return prediction
Return result
IR TNN x
Table 9: Performance and standard deviations of AumeT.mes. Results come from three random seeds.
Dataset
ETThl
ECL
Weather
Horizon
MSE
MAE
MSE
MAE
MSE
MAE
0.300-10.00
0.0002
0.129
0.225-0.001
0.15311.0000
0.203110000
192
0..08-00002
0.41910.001
0.147
0.201
0.201
0.250110000
336
0..01-00003
0.429-0.002
0.102-10.02
0.008
0.256:10.002
0.29310.001
720
0.406=00004
0.440
0.199=0.004
0.288=0.002
0.331
0.34540.001
Dataset
Traffic
Solar- Solar-Fnergy
Horizon
MSE
MAE
MSE
MAE
96
0.343=10.00
0.208:10.001
0.171+00.001
0.201=0.001
192
0.362-10.001
0.257:10.001
0.100:0.001
0.236=00002
0.379=0.002
0.266=0.000
0.203=0.002
0.248=0.002
720
0.413=0.003
0.284=0.002
0.222=0.002
0.262=0.003
C Hyperparameter Sensitivity
Segment Embedding and Segmentt Pronutttrrojjeeee jectiiin are uniformly implemented by MLP. The number of layers is fixed as 2 and the hidden dimension is selected from [256, 512, 1024) according the validation loss. The segment length set as S in multivariate datasets and set as the prediction length S F in M3 and M4. We verify the robustness of AutoTimes of hyperparameters as follows: the layer number and hidden and wongmenttrrojectieo
dimension of regmentEmmeeddin that Auto Times is insensitive to the configurations of embedding and projection layers. Besides, performance be improved by increasing the context length. For long prediction lengths, larger segment length is favored.
context length, and segment length. We observe
Number
Hidden Dimension
Context Longth
Segment Length
0.42
0.18
11.40
MIE
0.16
0.14
0.55
pred 96
pied 192
pred 336
prod
720
Figure 7: Hyperparameter sensitivity of Aimes. Each curve presents a specific forecast length.
16

---

D Supplementary Results
D.1 Time Series Forecasting
We compare the performance of AutoTimes with state-of-the-ar LLM4TS methods and well-acknowledged deep forecasters. Table shows detailed short-term forecast results M4. [able[TO]pa presents presensss results of the one-for-all forecasting benchmark across ETTh1, ECL, Traffic, Weather, and Solar- Energy datasets. evaluate all methods by rolling forecasting: each model is trained with input length 672 and output length 96, and the predicted values are integrated as part of the input in the next iteration until reaching the desired forecast horizon.
Furthermore, the traditional one-for me--orr-ne approach, where forecasters are trained individually for each prediction length, is also presented in Table 12 The results reproduced using their corresponding ofticial code. For the sake of rigor, we also provide our reproduced results with the officially reported results in Table [3
Additionally, we evaluate AutoTimes along with other baseline models recent henchmarks (24). Results presented in Table T5 We also look forward to evaluating on more diverse benehmarks in the future.
Table 10:
Full long-term forecasting results of one-for-allt we conduct rolling forecasting with single model trained on each dataset and accomplish four desired forecast lengths in {96, 192, 336, 720]. AutoTimme adapt LLMs with the context length C = 672. We set the input length L 672 and output length
96 in other methods, which are all implemented by their official code.
Method Times TimeLLM
[5
UniTime
21
FIT 49
iTrans.
DLinear
45
PatchTST
26 TimesNel E
Metric
MSE MAE MSE
MAE
MSE
MAE
MSE MAH
MSH
MAH
MSH
MAE
MSE
MAE
MSE MAE
96
0.360 0.400 0.380
0.412
0.386 0.409
0.377 0.404
0.387 0.418
0.369
0.400
0.374
0.401
0.452
0.463
192
0.388 0.419 0.408
0.431
0.639 0.577
0.41
0.405
0.422
0.405
0.422
0.474 0.477
336
0.401
0.429 0.425
0.443
0.680 0.814
0.440
0..40
0.550
0.445
0.423
0.435
0.493
0.489
720
0.406 0.440
0.434
0.463
0.891
0.719
0.488
0.493
0.508
0.434
0.460
0.560
0.534
Avg
0.389 0.422 0.412
0.437
0.596 0.683
0.429
0.439
0.445
0.426
0.444
0.409
0.430
0.495 0.491
96
0.129 0.225
0.137
0.244
0.171
0.266
0.1 0. 0.229
0.88
0.238
0.132
0.232
0.184 0.288
192
0.147 0.241
0.158
0.266
0.293
0.378
0.158
0.258
0.245
0.152
0.251 0.151 0.250 0.192 0.295
336
0.162
0.258
0.183
0.292
0.379
0.448
0.181 0.285
168 0.262
0.167
0.268
0.171 0.272 0.200 0.303
720
0.199 0.288
0.247
0.348
0.455
0.502
0.258
0.355
0.294
0.203
0.302
0.222
0.318
0.228 0.325
Avg
0.159 0.250 0.181
0.288
0.399 0.325
0.184 0.284
0.258
0.165
0.265
0.169
0.268 0.201 0.303
96
0.153
0.149
0.200
0.180 0.223
0.154 0.200
0.27
0.169
0.229
0.149
0.202 0.169 0.228
Weather
192
0.201
0,250
0.193
0.243
0.450 0.451
0.196
0.243
0.268
0.211
0,268
0.194 0.245 0.222 0,269
336
0.256
0.293
0.243
0.284
0.594
0.570
0.244 0.28200.290003300
0.306
0.244 0.285
0.290 0.310
0.331 0.345
0.315
0.336
0.618
0.593
0.318 0.37
0.362
0.317 0.338
0.376 0.364
Avg
0.235 0.225
0.266
0.461 0.459
0.228 0.26660..666
0.239
0.291
0.226 0.268
0.264 0.293
96
0.343
0.248
0.376
0.280
0.438
0.291
00.95 0.283
0.259
0.399
0.285
0.359
0.255
0.593 0.315
192
0.362
0.257
0.397
0.294
0.538
0.353
0.425 0.302
0.373
0.267
0.409
0,290
0.377 0.265 0.596 0.317
336
0.379
0.266 0.420
0.311
0.621
0.389
0.463 0.386 0.275
0.422
0.297
0.393 0.276
0.600 0.319
720
0.413 0.284
0.448
0.326
0.737
0.435
0.560 0.399 0.296
0.2960.461
0.461
0.319
0.436
0.305
0.619 0.335
Avg
00.374 0.764 01.410
0.303
00.584 0.367
0.441 0.326
0.274
0.423
0.298
0.391
0.275
0.602 0.322
0.171
0.221
0.224
0.289
0.223
0.274
0.6
0.261
0.183
0.265
0.1
0.258
0.168
0.237
0.180 0.272
192
0.190 0.236 0).248
0.315
0.373
0.431
0.219
0.284
(0.205
0.214
0,274
0.189
0.257
0.199 0.286
336
0.203
0.248 0.269
0.338
0.445
0.536
0.245
0.311
0.224
0.299
0.233
0.291
0.212
0.277
0.220 0.301
720
0.222
0.262
0.310
0.396
0.526
0.608
0.285
0.354
0.239
0.246
0.31
0.246
0.307
0.240
0.305
0.251 0.321
Avg
0.197 0.242 0.263
0.335
0.392
0.462
0.236
0.303
0.213
0.291
0.222
0.283
0.202
0.269
0.213 0.295
17

---

Table 11: Full results of short-term forecasting, We follow the same protocol of TimesNet
42
Method
Auto Times
TimeLLM
FPT
Koopa
N-HiTS
N-BEATS
PatchTST
TimesNet
DLincar
FILM
Yearly
SMAPE
13.319
13,419
13.531
13.352
13.371
13.866
13.517
13.394
14.012
13,466
MASE
2.993
3.005
3.015
2.997
3.025
3.006
3.031
3.004
3,071
3.059
OWA
0.784
0.789
0.793
0.786
0.790
0.802
0.795
0.787
0.815
0.797
Monthly Quarterly
SMAPE
10.101
10.110
10.177
10.159
10.454
10.689
10.847
10.101
10.758
10.074
MASE
1.182
1.178
1.194
1.189
1.219
1.294
1.315
1.183
1.306
1.163
OWA
0.890
0.889
0.897
0.895
0.919
0.957
0.972
0.890
0.905
0.881
SMAPE
12.710
12.980
12.894
12.730
12.794
13.372
14.584
12.866
13.377
12.801
MASE
0.934
0.963
0.956
0.953
0.960
1.014
1.169
0.964
1.021
0.955
OWA
0.880
0.903
0.897
0.901
0.895
0.940
1.055
0.894
0.944
0.893
Others
SMAPE
4.843
4.795
4.940
4.861
4.696
4.894
6.184
4.982
5,259
5,008
MASE
3.277
3.178
3.228
3.124
3.130
3.358
4.818
3.323
3.608
3.443
OWA
1.026
1.006
1.029
1.004
0.988
1.044
1.140
1.048
1.122
1.070
Average
SMAPE
11.831
11,983
11.991
11.863
11,960
12,418
13.022
11.930
12.489
11,910
MASE
1.585
1.595
1.600
1.595
1.606
1.656
1.814
1.597
1.690
1.613
OWA
0.850
0.859
0.861
0.858
0.861
0.891
0.954
0.867
0.902
0.862
Table 12: Long-term forecasting results of one-for-one: AutoTimes trains one LLM-based forecaster handle all prediction lengths by autoregression, whereas other models are trained respectively on each prediction length. AutosTmms adapts LLMs with the context length C = 672. The lookback
length is set as 672 others. All results are averaged. Full results provided in Table 14
Models| Times Timel
[15 UniTime 21
FPT 49
iTrans.
22
innan
35 PutchTST (6 TimesNer
4b
Metric MSE MAR MSE MAE MSR MAE MSE MAE MSF MAE MSE MAE
MSE MAE
MSE MAE
ETThl
00.389 0.422 0.409 0.432 0.438 0.445 0.426 0.438 0.438 0.450 0.423 0.437 0.413 0.431 0.458 0.450
ECL
[0.159 0.253 0.170 0.275 0.194 0.287 0.167 0.264 0.161 0.256 0.177 0.274 0.159 0.253 0.192 0.295
Weather
00.255
0.273 0.227 0.266 0.260 0.283 0.231 0.269 0.238 0.272 0.240 0.300
0.226 0.264 0.259 0.287
Traffic
[0.374
0.264 0.402 0.294 0.460
0.301 0.416 0.295 0.379 0.272 0.434 0.295
0.391 0.264 0.620 0.336
Solar.
00.197 0.242 0.234 0.293 0.254 0.291 0.229 0.296 0.202 0.269 0.217 0.278 0.189 0.257 0.200 0.268
Table 13: Results of LLM4TS methods from the original paper and reproduction by official code.
Models AutoTTmmss TimeLLM'
15
TimeLLM 15
FPT 49
FPT
49
UniTTime
21
UniTime 21
Metric
MSE MAE MSE
MAE
MSE
MAE
MSE
MAE MSE
MAE MSE
MAE MSE MAE
ETTh1
00.89 0.422 0.408
0.423
0.409
0.432 0.427 0.426 0.426 0.438 0.442 0.448 0.438 0.445
ECL
0.159
0.253 0.159
0.253
0.170
0.275
0.167 0.263 0.167 0.264 0.216 0.305 0.194 0.287
Weather
0.235
0.273 0.225
0.257
0.227
0.266
0.237
0.270 0.231 0.269 0.253
0.276 0.260 0.283
Traffic
00.374 0.264 0.388
0.264
0.402
0.294 0.414 0.294 0.416 0.295
0.460 0.301
Solar.
0.199 0.242
0.234
0.293
0.229 0.296
0.254 0.291
Methods with means results from the original paper; without means reproduction.
n indicates that results are not reported in the original paper.
18

---

Table 14: Full long-term forecasting results of one-for-one: AutoTTimes trains one L.LM-hased forecaster to handle all prediction lengths by autoregression, whereas other models are trained respectively on each prediction length. AutoTimes adapt LLMs with the context length C' = 672. The lookback length set as 672 in others, which are all implemented by their official code.
One-for-all
Trained respectively on specific lookback/predictionn length
Method
AutoTTimes
TimeLLM [5]
UniTime
21
FPT 49
iTaass
DLincar
45
PatchTST
26
TimesNet
4
Metric
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE MAE
MSE MAE
MSE MAE
96
0.360 0.400
0.380
0.412
0.386
0.409
0.377
0.404
0.386 0.405 0.375 0.399
0.370 0.399
0.384 0.402
192
0.388 0.419
0.405
0.422
0.428
0.413 0.424
0.422 0.439 0.405 0.416
0.421
0.557 0.436
336
0.401
0.429
0.422
0.433
0.464
0.436 0.444
0.444
0.457 0.439 0.443
0.422
0.436
0.491 0.469
720
0.406
0.440
0.430
0.459
0.473
0.477 0.481
0.500
0.498
0.472 0.490
0.490/0.447
0.447
0.466
0.521 0.500
Avg
0.389
0.422
0.409
0.432
0.438
0..44500.422 0.438
(0.438
0.450 0.423 0.437
0.413 0.431
0.458 0.450
96 0.129
0.225
0.137 0.244
0.171
0.266 037 0.236
0.132 0.227 0.153 0.237
0.129
0.222
0.168 0.272
192
0.147 0.241
0.162 0.271
0.1778 0.274 0.1 0.251 0.153 0.249 0.152 0.249
0.147
0.240
0.184 0,289
336
0.162 0.258
0.175 0.279
0.194
0.289 0. 169 0.267
0.167 0.262 0.169 0.267
0.163
0.259
0.898 0.300
720 0.199 0.288
0.207 0.306
0.232
0.319 0.300
0.285 0.233 0.344
0.197
0.290
0.220 0.320
Avg 0.159 0.253
0. 170 0.275
0. 0.287 0.28870.166
0. 167 0.264 00166 0.256 0.177 0.274
0
0.159
0.253
0. 0.295
0.153 0.203
0.149 0.200
0.180
0.223 0. 154 0.205
0.1
0.211 0.152
0.237
0.149
0.198
0.172 0,220
Weather
192 0.201 0.250
0.195 0.243
0.226
0.261 0.196 0.245 0.205
0.250 0.220 0.282
0.194
0.241
0.219 0.261
336 0.256 0.293
0.245 0.282
0.280
0.300
0.300
0.290
0.254
0.289 0.265 0.319
0.245
0.282
0.280 0.306
720 0.331 0.345
0.318
0.338
0.355
0.348
0.332 0.337
0.329
0.340
0.323 0.362
0.314 0.334
0.365 0.359
Avg 0.235 0.273
0.227
0.266
0.260
0.283
0.231 0.269
0.2338
0.272 0.240 0.300
0.226 0.264
0.259 0.287
96
0.343
0.248
0.373
0.280
0..48
0.291
0.339
0.283
0.351
0.257 0.410 0.282
0.360
0.249
0.593 0.321
Traffic
192
0.362
0.257
0.390
0.288 0..46
0.293
0.410
0.290
0.364
0.265 0.423 0.287 0.379 0.256
0.617 0.336
336
0.379
0.266
0.407
0.299
0.461
0.4014
0.295
0.382
0.273 0.436 0.296
0.264
0.629 0.336
720
0.413
0.284
0.438
0.310
0.410
0..94
0.318
0.318
0.445
0.311
0.420 0.292 0.466 0.315
0.432 0.286
0. 0.640 0.350
Avg
0.374
0.264
0.402 0.294
0.460
0.301
0..416 0.295
0.379
0.272 0.434 0.295
0.391 0.264
0.620 0.336
96
0.171
0.221
0.224
0.289
01.23
0.274 196 0.261
187
0.255 0.191 0.256
168 0.237
0.178 0.256
Solar- Energy
192
0.190
0.236
0.244
0.289
0.251
0.224 0.292
0.200
0.270 0.211 0.273
0.187 0.263
0.200 0.268
336
0.203 0.248
0.225
0.291
0.270
0.301
0.240 0.308 0.209 0.276 0,228 0.287
0.196 0.260
0.212 0.274
720
0.222 0.262
0.243
0.301
0.271
0.298
0.256 0.321 0.213 0.276 0,236 0.295
0.205 0.269
0.211 1 0.273
AAg 0.197 0.242
0.234
0.293
0.254
0.291
0.2 o. 0.296 0.20 0.269 0.217 0.278
189
0.257
0.200 0.268
Table 15: Forecasting results on additional benchmark datasets 24 (672-pred-96).
Models
AutoTTime
PTtchTST
iTransformer
DLinear
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Australian Electricity
0.150
0.228
0.163
0.242
0.153
0.233
0.167
0.250
Bdg-2 Panther
0.537
0.458
0.565
0.476
0.546
0.462
0.581
0.499
Oikolab Weather
0.603
0.577
0.635
0.603
0.630
0.591
0.663
0.611
19

---

D.2 Zero-Shot Forecasting
Following the zero-shot forecasting of FPT [15]. each experiment comprises the source and target datasets. We train model on the source dataset and apply the model on the target dataset for predictions directly.
Notably, the zero-shot scenarios conducted respectively the subsets (e-g. M4 Monthly M3 Monthly) the subsets are divided by the sampling frequency but follow different distributions [25].
+ M3, which means training on M4 and testing on M3, we directly utilizee same model the short-term forecasting experiments reported in Table [T] Considering different horizons in subsets, for M3 Yearly, M3 Quarterly, and M3 Monthly, directly employ models trained on corresponding subsets of M4 for testing. As for M3 Others, test using the model trained on M4 Quarterly to keep the same horizon.
For M3 - M4, similarly, for M4 Yearly, M4 Quarterly, and M4 Monthly, we directly employ models trained on corresponding subsets of M3 for testing. For the remaining subsets, M4 Weekly, M4 Daily, and M4 Hourly, we perform inference using the model trained on Monthly. Table[6sshove the detailed result.
Table 16: Results of zero-shot forecasting. We adopt the same protocol as FPT 49 M4 - M3 means training forecasters on M4 datasets and evaluating the performance M3, and vice versa. Results of compared baselines are reported from FPT [49]. Lower SMAPE indicates better performance.
Method
Times
FPT
DLinear PatchTST
TimesNet NSformer FEDformer Informer Reformer
Yearly
15.71
16.42
17.43
15.99
18.75
17.05
16.00
19.70
16.03
M4 M3
Quarterly
9.35
10.13
9.74
9.62
12.26
12.56
9.48
13.00
9.76
Monthly
14.06
14.10
15.65
14.71
14.01
16.82
15.12
15.91
14.80
Others
5.79
4.81
6.81
9.44
6.88
8.13
8.94
13.03
7.53
Average
12.75
13.06
14.03
13.39
14.17
15.29
13.53
15.82
13.37
Yearly
13.728
13.740
14.193
13.966
15.655
14.988
13.887
18.542
15.652
Quarterly
10.742
10.787
18.856
10,929
11.877
11.686
11,513
16.907
11.051
Monthly
14.558
14.630
14.765
14.664
16.165
16.098
18.154
23.454
15.604
M3
Others
6,259
7.081
9.194
7.087
6.863
6.977
7.529
7.348
7,001
Average
13.036
13.125
15.337
13.228
14.553
14.327
15.047
19.047
14.092
D.3 Method Generality
Mainstream LLMs predominantly adopt the decoder-only architecture, and AutoTimes can utilize any decoder- only L.M.. We conduct experiments on various types and sizes L.LMs, including GPT-2 31 multiple sizes
of OPT 4. and LLaMA [6). Detailed configurations and results are shown in Table 17 and 18 demonstrating general trend where performance improves as the model size increases, consistent with the scaling law [16]
Table 17: Detailed method configurations of AutoTimes for alternative language models.
LLM
GPT-2 (124M)
OPT-350M
OPT-1.3B
OPT-2.7B
OPT-6.7B
LLaMA-7B
Hidden Dim.
768
1024
2048
2560
4096
4096
Embedding
2-layer MLP
2-layer MLP
2-layer MLP 2-layer MLP 2-layer MLP
Linear
Trainable Param, (M)
0.44
0.58
1.10
1.36
2.15
0.79
D.4 Variable Lookback Length
In the conventional forecasting paradigm. deep forecasters are trained respectively on lookback/forecast lengths, limiting their applicability to single lookback length. In contrast, LLMs have the versatility to handle various input lengths. This capability is derived from Rotary Position Embedding (33 and the next toke prediction objective, where LLMs are trained with token-wise supervision in Equation 8 that the generated token cach position is supervised. While non autorerressive LLM4TS methods are typically constrained to fixed lookback setting, AutoTimes with consistent training objective has the flexibility to deal with different lookback lengths. We present the results in Figure S where adapt the LLM by AutoTimes with the context length of C 672. and evaluate the performance with different lookback lengths, which demonstrates the inherited versatility of based forecasters. Moreover, the performance is generally improving with increased available lookback observations, leading to an averaged 9.3% promotion from 384 to 672. By contrast, several works have observed that the performance of respectively trained deep forecasters does not necessarily improve with the increasing of lookback length 222 26 45).
20

---

Table 18: Full Results of alternative LLMs, which are adapted with the context length C -672.
L.M
GPT-2 (124M)
OPT-350M
OPT-1.3B
OPT-2.7B
OPT-6.7B
IJaMA-7B
Metric
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.140
0.236
0.136
0.233
0.132
0.228
0.132
0.227
0.130
0.226
0.129
0.225
192
0.159
0.253
0.154
0.249
0.150
0.245
0.149
0.244
0.148
0.242
0.147
0.241
ECL
336
0.177
0.270
0.171
0.267
0.167
0.262
0.167
0,262
0.165
0.260
0.162
0.258
720
0.216
0.303
0.211
0.301
0.206
0.296
0.207
0.297
0.204
0.295
0.199
0.288
Avg
0.173
0.266
0.168
0.263
0.164
0.258
0.164
0.258
0.162
0.256
0.159
0.253
96
0.360
0.397
0.365
0.403
0.357
0.395
0.360
0.398
0.357
0.397
0.360
0.400
FTTh
192
0.391
0.419
0.395
0.423
0.389
0.417
0.389
0.419
0.386
0.417
0.388
0.419
336
0.408
0.432
0.411
0.434
0.408
0.431
0.404
0.430
0.404
0.429
0.401
0.429
720
0.429
0.452
0.432
0.457
0.430
0.452
0.421
0.447
0.427
0.454
0.406
0.440
Avg
0.397
0.425
0.401
0.429
0.396
0.424
0.394
0.424
0.394
0.424
0.389
0.423
96
0.369
0.257
0.371
0.260
0.361
0.253
0.358
0.251
0.357
0.251
0.343
0.248
Traffic
192
0.394
0.268
0.393
0.270
0.383
0.263
0.380
0.261
0.379
0.261
0.362
0.257
336
0.413
0.278
0.411
0.279
0.402
0.273
0.399
0.271
0.398
0.272
0.379
0.266
720
0.449
0.299
0.446
0.300
0.440
0.295
0.437
0.293
0.437
0.294
0.413
0.284
Avg
0.406
0.276
0.405
0.277
0.397
0.271
0,394
0.269
0.393
0.270
0.374
0.264
96
0.158
0.208
0.157
0.208
0.1557
0.207
0.157
0.207
0.159
0.209
0.153
0.203
Weather
192
0.207
0.254
0.205
0.252
0.207
0.253
0,206
0.253
0.208
0.256
0.201
0.250
336
0.262
0.298
0.261
0.294
0.263
0.296
0.265
0.297
0.268
0.302
0.256
0.293
720
0.342
0.353
0.335
0.346
0.334
0.347
0.344
0.351
0.354
0.360
0.235
0.273
Avg
0.242
0.278
0.240
0.275
0.240
0.276
0.243
0.277
0.247
0.282
0.235
0.273
ETTs1
ECL
Traffic
Weather
0.142
0,360
0.340
0.175
0.139
0.375
0,355
0,170
MSB
0.136
MSS
MSE
0.165
0.330
0:350
0.160
0.365
0.130
0.345
0.155
0.360
672
400
677
885
672
602
Figure 8: Performance of LLM-based forecasters on the pred-96 scenario, which are adapted by AutoTimes by the context length C = 672 and directly applied on different lookback lengths.
D.5 Timestamps as Position Embeddings
We conduct an ablation study on the proposed position embeddings that integrate timestamps, prevalent textual covariate real-world applications. As shown in Figure the forecasting performance consistently promoted across all multivariate datasets and prediction lengths. The steady improvement can be altributed to the fact that timestamps demote the absolute position of time series segments on the timeline, explicitly aligning different variates in multivariate scenarios. The incrcasing promotion with longer prediction length also implics that chronological information, such date and periodicity, yields significant benefits for long-term forccasting.
ETTbl
ECL
Traffic
Weather
0.42
0.35
0.20
0.41
0.40
0.40
0.39
MSE
MIE
0.25
0.16
0.20
0.37
wil Gmodammp
6 win timotamp with linotamp
w/o timeatapp wish timestamp
wel timcetemp
with cimetiump
0.36
0.54
0.15
Figure 9: Ablation whether to utilize textual timestamps the position embedding. Results of different prediction lengths are provided, where the embedding leads to consistent performance promotion across all datasets, and the promotion can increase with longer prediction length.
21

---

D.6 In-Context Forecasting
For in-context forecasting, similar to zero-shot forecasting in Appendix D.2 train our model using the source dataset and directly evaluate it the target dataset. In this task, we first choose M4 the source dataset and M3 as the target dataset. It is important to note that the structure of the M3 and M4 datasets differs from typical datasets used for long-term forecasting. They consist of multiple univariate time sequences of different lengths. The final part of each sequence serves as the test set, while the preceding part is used for training.
Implementation In shro-shot scenarios, we use sequence of length F' preceding and consecutive to the test set input, referred the lookback window, where F' is the forecast length of each subset. During in-context forecasting, we concatenate the first 2F time points that helong the same sequence with the lookback window as input. We aim to enhance prediction performance by incorporating more contextual information. Too short sequences ( 4F) are discarded to prevent overlap between the prompt and the lookback window. For a fair comparison, both zero-shot and in-context forecasting performance are reported on the same remaining sequences. Figure 12 provides showcases of zero-shot and in-context forecasting.
Prompt engineering Regarding in-context learning. we further delve into the effect of different strategies to retrieve time series as prompts, which is provided in Table 19 P.1 and P.2 correspond to the zero-shot and in-context forecasting evaluated in Section 4.3 The prompt of P.3 contains the last 2F time points preceding the beginning of the lookback window. Another retrieval of prompts as P.4, adopts time series that come from another uncorrelated time series (out-of-scries), We can obtain the following observations:
P.I v.s. P.4 indicates that the selected prompt is not suitable, since the prompt does not come from the earlier observations the same series to be predicted. Although the context window becomes larger, the averaged performance will deteriorate because of irrelevant prompts.
P.2 and P.3 indicates that most cases, selecting the relevant 2F time series from the same series provide better contextual information.
This highlights the prompt engincering for in-context forccasting. An intuitive suggestion is to utilize consecutive, inter-periodic, and multiple prompts. To verify this idea, we analyze the periodic effect of time series prompts.
Table 19: Effects of different strategies to retrieve time series as prompts for in-context forecasting.
Context for prediction
Yearly
Quarterly
Monthly
Others
Averaged Err.
P.1: Lookback F
21.52
12.03
13.09
8.46
13.61
P.2: Prompt from first 2F + Lookback F
17.03
10.29
12.24
5,33
11.80
P.3: Prompt from last 2F Lookback F
16.30
9.59
12.09
6,24
11.48
P.4: Prompt from 2F of other series Lookback /
18.95
12.18
14.37
9.46
13.98
In previous experiments, adopt M3 and M4 datasets, which consistent with the zero-shot experiment of FPT [49]. to present the promotion of our in-context paradigm. To provide more rigorous conclusions, we extend the evaluation to widely recognized datasets. Details the experiment are as follows: By using a trained model checkpoint on a source domain (Traffic), we conduct forccasting without gradient update on target ETT datasets. We evaluate the pred-96 performance on the last variate (OT). For the zero-shot scenario, the input is length- -288 lookback series. For in-context forecasting, the input is (length-384 series prompt + length-288 lookback series). Considering the dataset periodicity, the prompt is uniformly selected as the Ahead-24 (one-day-ahead) series of the original lookback series. To eliminate the performance boost that comes from extending the input length, we also provide the results of length-677 672 lookback series in the zero-shot scenario. Moreover, we further delve into the effect of different strategies select time series prompts:
Ahead-Perione The prompt is uniformly selected as the Ahead-21 series of the original lookback series where 21 is one of the periods (daily period) of ETT.
Ahndd-Random: The prompt is randomly selected as the previous series of the original series.
Fixed Prompt: The prompt is fixed the first 384 time points in the variate-OT.
Other Variate: The prompt is uniformly selected as Ahcad-24 24 series, but comes from other variates.
Results in Table 20 demonstrate the effectiveness of using suitable time series prompts and highlight the influence
of prompt engineering. Using inter-period prompts can outperform simply extending lookback window.
The benefit of the proposed in-context forecasting to extend the input context of time series forecasting beyond a continuous lookback window. Since the essence of prompts is to incorporate useful domain -peciiii knowledge, here is one use case of in contette forecasting: Considering predicting the weather of one day. one approach is extend the lookback length from days weekends. However, it can also introduce noisy information since
22

---

Table 20: Strategies to select time series prompts based on periodicity for in-context forecasting.
Context for prediction
ETTh1-OT
ETTh2-OT
ETTm2-OT ETTml-OT
Average Ert.
P.O: Zero-Shot (Input-288)
0.0673
0.1637
0.0424
0.1669
0.1101
P.I: Zero-Shot (Input-672)
0.0657
0.1538
0.0415
0.1701
0.1078
P.2: Ahcad-Period (nput-672)
0.0645
0.1513
0.0399
0.1629
0.1047
P.3: Chead-Randor (Input-672) -672)
0.0666
0.1621
0.0407
0.1719
0,1103
P:4: Fixed Prompt (1nput-672)
0.0769
0.1859
0.0512
0.2104
0.1311
P.5: Ether-Variater (Input-672)
0.1263
0.1780
0.0852
0.2297
0.1548
non-stationary meteorological conditions can change with seasons. Another practical way is to consider how the weather changes on the same day in the last year (or years). Although the input is not continuous, the input context becomes relevant based on prior knowledge about the periodicity (yearly). Therefore, in-context forecasting makes prior knowledge incorporatable and gets performance promotion.
D.7 Ablation Study
In addition to the ablation study of whether I.MSs are useful in AutoTimes (Ta further delve into the main difference between our method and previous L.LM4TS approach and provide comprehensive ablation study. results presented in [able[[]]emoosterrr that performance of aoo-autooegressoonn projection consistently inferior to that of our autoregressive AutoTimes approach.
Table 21: Ablation study of the autoregression. FlattenHead replaces the segment-wise projection of Times by flatten and linear head [26]. which is prevalent in non-autoregressive forecasters.
Dataset
ETThl
ECL
Weather
Traffic
Type
AutoTimes
FlattenHead
AutoTTimes
FlattenHead
AutoTTims
FlattenHead
Auto Times
FlattenHead
Metric
MSF
MAE MSE MAE MSE MAE
MSE MAR
MSF
MAR
MSE MAF
MSF
MAE
MSE MAE
Pred-96
0.360 0.400 0.385
0.420
0.129 0.225 0.142 0.247
0.153 0.203 0.155 0.209
0.343 0.248 0.367 0.261
Pred-192
0.388
0.419
0.445
0.463
00.147
0.241 0.157 0.259
0.201
0.250 0.202 0.251
0.362
0.257
0.391 0.282
Pred-336
0.401
0.429
0.463
0.475
(0.162
0.258
0.201
0.311
0.256
0.293
0.257
0.286
0.379
0.266
0.404
0.287
Pred-720
0.406
0.440
0.574
0.542
(0.199
0.288
0.232 0.331
0.331
0.345
0.333
0.261
0.443
0.284
0.432 0.294
E
Showcases
To facilitate a clear comparison among various models, we present additional prediction showcases for long- term-term forecasting and short-tert term forecasting. These examples provided by the following models: TimeLLM (15). FPT (99 and PStchTST 26). Of all the models, AutoTimes delivers the most accurate future series predictions. Additionally, we provide the showcases zero-shot and in-context forecasting Figure
Time
Figure 10: Visualization of input- imput-672.0 672 17722-redct-96 results on the Traffic dataset.
23

---

Auto Times
Time LLM
FPT
PaschTS
Figure 11: Visualization of input- 36--prddcc--- 18 results on the M4 Monthly dataset.
3970
3510
3000
Prongenng
6500
2920
3600
4300
Time Serie Fronsiing
Coeer Spm
0002
Time Ser hispping
3502
Cod
4700
5002
6620
5000
5350
41000
4500
14000
Jarriia le wouttr
4250
15000
7502
14000
4500
Series Frmppiig
8000
Sar Proppting
M
6000
4000
6002
5500
2002
4000
Serke Frmpping
Papi
4000
Series Promptin
4052
Omral 9.0
3300
4690
7820
Teur Peeppit
3502
S800
5.20
Time Series Frompiing
Oren Path
10200
Time Sarten Pompting
2630
4800
Figure 12: Showcases of zero-shot and in-context forecasting, For in-context forecasting, beyond the lookback window, we uniformly adopt the first 2F time points that belong to the same sequence as the prompt and concatenate them the prediction context, which achieves more accurate prediction.
F
Broader Impact
F1
Impact on Real-world Applications
time series forecasting, which is faced with increasing challenges such This paper copes with variable-length scenarios, good generalizability with scarce samples, utilization of as the versatility to handle multimodality, and instructive downstream prompts. Since previous studies have demonstrated the feasibility of leveraging large language models for time series, we propose simple but effective approach AutoTimes obtain LLM based forecasters, which keeps the consistency of autoregression. the-art art performance on forecasting benchmarks and demonstrates remarkable adaptation speed and parameter efficiency. Besides, advanced capabilities such as multi-step generation and in-context learning are inherited by the repurposed forecaster. Therefore, the proposed method makes it promising tackle real-worl world applications,
model achieves state-of-

---

which helps our society prevent risks in advance and make better decisions with limited computational budgets. Our paper mainly focuses scientific research and has obvious negative social impact.
F.2 Impact on Future Research
In this paper, we find prevalent non-autoregreessvve LLM4TS methods have inconsistencies in the model structure and generative approach with LLMs, leading insufficient utilization of the inherent multi-step token transition. Given that the generalizability and generative ability LLMs largely derived from the autoregressive manner, the potentials LLMs may not be fully exhibited time series forecasting. Therefore, we propose to adapt LLMs by the consistent training objective, the next token prediction, and accomplish arbitrary-length forecasting by iterative generation. Beyond the conventional forecasting paradigm, the context for prediction is extended, and earlier historical time series can be utilized advantagcous prompts. The compatibility with LLMs and insights from autoregression can instructive for future LLM4TS research and the development foundation time series models.
propuse in-contexi forecasting, where
G Limitation
proposed method not supported probabilistic forecasting, since AutoTimes only establishes the mapping between time series segments to latent embeddings of the LLM, instead of discrete language tokens. Advanced low-rank adaptation is under exploration in our work, which can further align suitable token transitions as the future extrapolation of time series. More deftly designed embedding and projection layers underexplored to support more compatible tokenization for time series. Besides, it is fascinating to apply AutoTimms real-world multimodal time series datasets (such as news-stocks, logs- mesuuressur,,,nss)) which leaves our future work.
25

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section the main text, where the claims and contributions are included.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Please refer to Section Gl Appendix, where we provide several aspects of limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well spec--fffccitinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
26

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to Section B in the main text and code in our public repository, including the detailed configurations of experiments and the scripts for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Please refer to Section A of Appendix and code in our public repository, including dataset descriptions and how to access them.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (btzpe://nips.cc//ppp
Criidea/CodeSule www.euubbisiiioooooccc.o for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : //t/://nipp cc/public/ qul.dess/CoodSuubmiiessoooooo... for more details.
27

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparamcters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification: Please refer to Section C and of Appendix, where we state how hyperparameters are chosen and the detailed description of expermments.
Guidelines:
answer means that paper does not include experiments.
The experimental setting should be presented in the of the paper level of detail that
necessary appreciate the results and make sense them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Section B and Table of Appendix, where we report standard deviations of results with three random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, hootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors is verified.
For asymmetric distributions, the authors should be careful symmetric error hars would yield results that are out of range (e.g. negative error rates).
show in tables figures
error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please refer to Section of Appendix, where we provide sufficient information on the computing resource.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies httpp:///eutueeee http://uurrppaccccpppllie////eee wwtppubll../.hiaaauuiallllllllla
Answer: [Yes]
Justification:
We have reviewed and the reasearch conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e.g.. if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Section of Appendix, where we discuss societal impacts and influences on future research.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
29

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All creators of datasets are properly credited by citations in Section
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. www.coom//aaac /attttttstts has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-per re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
30

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
31


Reviews:
[{"review_id": "vGITTleSP6", "reviewer": "Reviewer_acJd", "summary": "The authors present in this paper an interesting approach where LLMs are leveraged to be fledged as time series forecasters. This proposed approach is based on freezing the LLM backbone to update a small amount of parameters to generate suitable time series embeddings which, together with time stamps as positional embeddings, provide suitable forecasts. Moreover, the authors provide an interesting notion of in-context forecasting, where the corresponding model is taught with examples how to forecast without gradient updates. Finally, the authors present numerical evaluations on diverse datasets.", "questions": "- In Section 3 the authors mention that they aim to forecast covariates as well. Is this correct? In general, one can assume that certain kind of covariates are available in the future, like timestamps or boolean variables indicating that something specific will happen in the future. But in general, I am not sure that the authors truly want to forecast covariates.", "limitations": "The authors have devoted a section describing sensible limitations of their work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors provide an interesting view and current challenge in the field of time series forecasting: how to leverage LLMs for time series forecasting with minimal effort and minimal model modification. The authors provide an approach that basically freezes the LLM backbone to only update parameters related to a suitable embedding of time series, together with the usage of timestamps prompts as positional encoders. Moreover, the emphasise that the number of updated parameters is around 0.1% of those of the LLM backbone.\n\nThe authors provide an interesting paradigm that naturally arises from LLMs: how can we do forecaster fine-tuning without gradient updates. This brings in-context forecasting as a contribution of the authors, which seems to have potentially an interesting resourceful idea for the community.", "weaknesses": "Perhaps the main weakness comes from the evaluations. The main points that I would invite the authors to aggressively address are the following:\n- the amount of baselines used for comparison is very limited. It is completely understandable that the authors miss some baselines because the field is just moving extremely quickly. Nevertheless, the authors should at least cite a relevant fraction of these recent works, for instance :\n- - Woo et al, 2024: Unified Training of Universal Time Series Forecasting Transformers\n- - Ansari et al, 2024: Chronos: Learning the Language of Time Series\n- - Dooly et al, 2023: ForecastPFN: Synthetically-Trained Zero-Shot Forecasting\n- - Goswami et al 2024: Moment: A family of open time-series foundation models\n- The amount of datasets used is limited. It is understandable that the authors provide a limited amount of datasets for evaluations, as one of the main hindrances in the field is the lack of publicly available data (at least in comparison to other fields). Yet, one can see that other contributions have shared datasets and made them freely available. See for instance:\n- - Woo et al, 2024: Unified Training of Universal Time Series Forecasting Transformers\n- - - and the corresponding HF source: https://huggingface.co/datasets/Salesforce/lotsa_data\n- Evaluations of in-context forecasting are limited. Following the previous points, I think this is one of the most exciting points of the paper and the reader would appreciate to see more systematic evaluations and explorations of this notion. Currently there is only one evaluation which involves M3 and M4 datasets, perhaps following the approach presented in One-Fits-All paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer acJd\n\nMany thanks to Reviewer acJd for providing a detailed review and recognizing our contributions.\n\n**Q1**: More baseline models for evaluation.\n\nWe acknowledge the importance of the recent works and will certainly incorporate the suggested references in our revision. As per your suggestion, we include the mentioned models based on the official code to enlarge our time series forecasting baselines. Here are the results:\n\n|ETTh1 (MSE)|AutoTimes|MOIRIA|MOMENT|Chronos|\n|-|-|-|-|-|\n|Pred-96|**0.360**|0.384|0.387|0.571|\n|Pred-192|**0.388**|0.425|0.410|0.654|\n|Pred-336|**0.401**|0.456|0.422|0.712|\n|Pred-720|**0.406**|0.470|0.454|0.774|\n\n|ECL (MSE)|AutoTimes|MOIRIA|MOMENT|Chronos|\n|-|-|-|-|-|\n|Pred-96|**0.129**|0.158|0.136|-|\n|Pred-192|**0.147**|0.174|0.152|-|\n|Pred-336|**0.162**|0.191|0.167|-|\n|Pred-720|**0.199**|0.229|0.205|-|\n\n|Traffic (MSE)|AutoTimes|MOIRIA|MOMENT|Chronos|\n|-|-|-|-|-|\n|Pred-96|**0.343**|-|0.391|0.770|\n|Pred-192|**0.362**|-|0.404|OOM|\n|Pred-336|**0.379**|-|0.414|OOM|\n|Pred-720|**0.413**|-|0.450|OOM|\n\nAs shown above, MOIRIA and Chronos follow the paradigm of **pre-training -> zero-shot forecasting** (- indicates that the test set is included in the pre-training dataset and thus not reported). MOMENT follows **pre-training -> fine-tuning on each dataset and length**. AutoTimes does not involve pre-training on time series since it adopts the pre-trained LLM and **fine-tunes it on each dataset, using one model for all prediction lengths**.\n\nIn terms of performance, AutoTimes consistently achieves the best. Still, we also appreciate the zero-shot forecasting ability of natively trained large time series models, which provide an out-of-the-box experience that is free from training/tuning the models.\n\n**Q2**: More benchmark datasets for evaluation.\n\nWe appreciate your suggested works that have contributed valuable data resources. Thus, we conduct evaluations on several datasets from [1], which come from various domains and applications.\n\n|Australian Electricity Demand (MSE)|AutoTimes|PatchTST|iTransformer|DLinear|\n|-|-|-|-|-|\n|Pred-96|**0.150**|0.163|0.153|0.167|\n|Pred-192|**0.203**|0.216|0.214|0.211|\n|Pred-336|**0.236**|0.255|0.244|0.237|\n|Pred-720|**0.264**|0.289|0.267|0.269|\n\n|Bdg-2 Panther (MSE)|AutoTimes|PatchTST|iTransformer|DLinear|\n|-|-|-|-|-|\n|Pred-96|**0.537**|0.565|0.546|0.581|\n|Pred-192|**0.663**|0.707|0.694|0.693|\n|Pred-336|**0.741**|0.807|0.774|0.781|\n|Pred-720|**0.802**|0.911|0.832|0.829|\n\n|Oikolab Weather (MSE)|AutoTimes|PatchTST|iTransformer|DLinear|\n|-|-|-|-|-|\n|Pred-96|**0.603**|0.635|0.630|0.663|\n|Pred-192|**0.643**|0.678|0.660|0.694|\n|Pred-336|**0.666**|0.685|0.677|0.711|\n|Pred-720|**0.697**|0.710|0.698|0.727|\n\nThe above results show that AutoTimes still outperforms the state-of-the-art deep models, which are good enhancements for the robustness of experiments. We will include these in the revision and make more complete evaluations of the lotsa dataset [1].\n\n**Q3**: Systematic evaluations on the proposed in-context forecasting.\n\nThanks a lot for your scientific rigor. We adopt M3 and M4 datasets, which are consistent with the zero-shot experiment of One-Fits-All paper, to present the promotion of our in-context paradigm. As per your request, we extend the evaluation to widely recognized datasets. Details of the experiment are as follows:\n\nBy using a trained model checkpoint on a source domain (Traffic), we conduct forecasting without gradient update on target ETT datasets. We evaluate the Pred-96 performance on the last variate (OT). \n\n* For the zero-shot scenario, the input is Length-288 lookback series.\n* For in-context forecasting, the input is (Length-384 series prompt + Length-288 lookback series). Considering the dataset periodicity, the prompt is uniformly selected as the Ahead-24 series of the original lookback series.\n* To eliminate the performance boost that comes from extending the input length, we also provide the results of Length-672 lookback series in the zero-shot scenario.\n\n|Dataset (MSE)|In-Context (Prompt-384 + Input-288)|Zero-Shot (Input-288)|Zero-Shot (Input-672)|\n|-|-|-|-|\n|ETTh1-OT|**0.0645**|0.0673|0.0657|\n|ETTh2-OT|**0.1513**|0.1637|0.1538|\n|ETTm1-OT|**0.0399**|0.0424|0.0415|\n|ETTm2-OT|**0.1629**|0.1669|0.1701|\n\nMoreover, we further delve into the effect of different strategies to select time series prompts:\n\n|Dataset (MSE)|Ahead-Period|Ahead-Random|Fixed Prompt|Other-Variates|Baseline (Zero-Shot)|\n|-|-|-|-|-|-|\n|ETTh1-OT|**0.0645**|0.0666|0.0769|0.1263|0.0657|\n|ETTh2-OT|**0.1513**|0.1621|0.1859|0.1780|0.1538|\n|ETTm1-OT|**0.0399**|0.0407|0.0512|0.0852|0.0415|\n|ETTm2-OT|**0.1629**|0.1719|0.2104|0.2297|0.1701|\n\n* **Ahead-Period**: The prompt is uniformly selected as the Ahead-24 series of the original lookback series where 24 is one of the periods of ETT.\n* **Ahead-Random**: The prompt is randomly selected as the previous series of the original lookback series.\n* **Fixed Prompt**: The prompt is fixed as the first piece of the series in the variate-OT.\n* **Other Variate**: The prompt is uniformly selected as the Ahead-24 series, but comes from other variate of ETT.\n\nThe above results demonstrate the effectiveness of using suitable time series prompts and highlight the influence of prompt engineering. Using in-period time series prompts can even outperform extending the lookback window. We also provide a detailed explanation in $\\underline{\\text{Q3 of Reviewer 1PWH}}$. Thus, the in-context forecasting paradigm is **meaningful for the real-world application**.\n\n**Q4**: Whether we aim to forecast covariates or not?\n\nIn Section 3, we use the timestamps as the covariate to improve forecasting, but we do not predict the covariate. As shown in $\\underline{\\text{Equation 1 of the main text}}$, we are concerned about the multivariate scenario, where each time series needs to be predicted.\n\n[1] Woo et al. Unified Training of Universal Time Series Forecasting Transformers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCRWOALnrM", "reviewer": "Reviewer_1PWH", "summary": "This paper proposes a model named AutoTimes to repurpose LLMs for time series forecasting. Different from previous methods that use flattening and linear projection to get a prediction, this model repurposes LLMs in an autoregressive way, which is closer to the pre-training process of LLMs. Specifically, the main backbone of LLMs is frozen, and new patch embedding/output layers are added like in previous works. Absolute timestamps are embedded through LLMs to serve as position embeddings. Experiments show that the proposed model achieves SOTA performance and is more efficient.", "questions": "1. Is LLM necessary for timestamp embedding? What if replacing it with ordinary nn.Embedding like Informer?\n\n2. How does the proposed model reduce the error accumulation of autoregressive models, given that there is no specifical design for this?\n\n3. Please report the number of learnable parameters in Table 5. Larger models have larger hidden states so that patch embedding/output layers for them have more learnable parameters. Therefore, it is uncertain whether the performance improvement comes from the scaling behavior of LLMs or from having more learnable parameters for tuning.", "limitations": "No potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main body of the thesis is clear and easy to understand.\n\n2. Reproposing LLMs in an autoregressive way is intuitive and more reasonable than previous linear projection ones.\n\n3. Numerous experiments were conducted to demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The most I concerned is whether this type of methods truly leverages the capabilities of the pre-trained LLMs. Please conduct the following ablation study: randomly initialize a large model, freeze its parameters, train it using the proposed method, and compare it with the pre-trained ones.\n\n2. The description of multimodal in Table 1 seems to be overselling, as the proposed model only uses texts for timestamps embedding and does not have capability to leverage natural languages.\n\n3. The in-context forecasting part is confusing: 1)What are the use cases for such a method? Table 16 shows that it's more effective to extend the lookback window. So what is the case that we can not get a longer lookback window but can get a window a long time ago? 2)Such a method can not even ensure the input series is continuous, so where does the promotion come from?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 1PWH\n\nMany thanks to Reviewer 1PWH for providing a detailed and insightful review.\n\n**Q1**: Whether AutoTimes truly leverages the capabilities of the pre-trained LLMs.\n\nWe noticed that the recent work[1] has raised questions about non-autoregressive LLM4TS methods. It is also the main claim of our paper that the inconsistent model structure and generative approach will cause insufficient utilization of LLMs for forecasting. We thoroughly conduct all types of ablations[1] (**Random Init** is the suggested ablation by the reviewer):\n\n|ETTh1 (MSE\\|MAE)| AutoTimes|Random Init|w/o LLM|LLM2Attn|LLM2Trsf|\n|-|-|-|-|-|-|\n|Pred-96|**0.360**\\|**0.400**|0.373\\|0.408|0.365\\|0.399|0.383\\|0.404|0.377\\|0.401|\n|Pred-192|**0.388**\\|**0.419**|0.394\\|0.421|0.405\\|0.425|0.414\\|0.422|0.406\\|0.420|\n|Pred-336|**0.401**\\|**0.429**|0.405\\|0.430|0.429\\|0.441|0.431\\|0.432|0.421\\|0.431|\n|Pred-720|**0.406**\\|**0.440**|0.418\\|0.447|0.450\\|0.468|0.456\\|0.454|0.449\\|0.452|\n\n|ECL (MSE\\|MAE)|AutoTimes|Random Init|w/o LLM|LLM2Attn|LLM2Trsf|\n|-|-|-|-|-|-|\n|Pred-96|**0.129**\\|**0.225**|0.148\\|0.245|0.171\\|0.263|0.156\\|0.255|0.162\\|0.263|\n|Pred-192|**0.147**\\|**0.241**|0.163\\|0.259|0.192\\|0.282|0.178\\|0.276|0.189\\|0.287|\n|Pred-336|**0.162**\\|**0.258**|0.179\\|0.274|0.216\\|0.304|0.198\\|0.295|0.216\\|0.309|\n|Pred-720|**0.199**\\|**0.288**|0.217\\|0.305|0.264\\|0.342|0.230\\|0.320|0.258\\|0.340|\n\nThe above results highlight that the autoregression way of AutoTimes can truly utilize the LLM. The core difference: instead of regarding LLMs as representation extractors in a BERT-style, we figure out the mechanism of LLM4TS that **the general-purpose token transition is transferable among time series and natural language**, such that the generation ability of LLMs can be fully revitalized.\n\n**Q2**: The description of multimodal in Table 1 seems to be overselling.\n\nThanks for your suggestion. In the initial version, we demonstrate that AutoTimes can take advantage of textual timestamps, which are the most accessible ones in real-world applications. Considering the scope of multimodal models, we will remove the point in Table 1 unless being evaluated on well-acknowledged multimodal datasets.\n\n**Q3**: The use cases for in-context forecasting and how in-context forecasting gets the promotion. \n\nThe value of the proposed in-context forecasting is to **extend the input context of time series forecasting beyond a continuous lookback window**. As the reviewer mentioned, $\\underline{\\text{Table 16}}$ shows that extending the lookback window (P.2) or using trivial prompts (P.3) respectively excel at different subsets but the overall difference is small. \n\nSince the essence of prompts is to incorporate useful domain-specific knowledge, here is one use case of in-context forecasting: Considering predicting the weather of one day, one approach is to extend the lookback length from days to weekends. However, it can also introduce noisy information since non-stationary meteorological conditions can change with seasons. Another practical way is to consider how the weather changes on the same day in the last year (or years). Although the input is not continuous, **the input context becomes more relevant based on prior knowledge** about the periodicity (yearly). Therefore, in-context forecasting makes the prior knowledge incorporatable and gets the promotion. \n\nWe also provide an exploration of prompt engineering in $\\underline{\\text{Q3 of Reviewer acJd}}$, in which the usage of discontinuous lookback windows can indeed outperform continuous lookback windows at well-acknowledged datasets.\n\n**Q4**: Ablations about the timestamp embedding.\n\nAs per your suggestion, we compare the ways of embedding timestamps in AutoTimes. Here are the results:\n\n|ETTh1 (MSE\\|MAE)|LLM Embedding|nn.Embedding|w/o Embedding|\n|-|-|-|-|\n|Pred-96|**0.360**\\|**0.400**|0.370\\|0.405|0.368\\|0.402|\n|Pred-192|**0.388**\\|**0.419**|0.396\\|0.422|0.395\\|0.421|\n|Pred-336|**0.401**\\|**0.429**|0.408\\|0.430|0.413\\|0.433|\n|Pred-720|**0.406**\\|**0.440**|0.422\\|0.448|0.439\\|0.459|\n\n|ECL (MSE\\|MAE)|LLM Embedding|nn.Embedding|w/o Embedding|\n|-|-|-|-|\n|Pred-96|**0.129**\\|**0.225**|0.132\\|0.231|0.131\\|0.227|\n|Pred-192|**0.147**\\|**0.241**|0.150\\|0.243|0.149\\|0.243|\n|Pred-336|**0.162**\\|**0.258**|0.165\\|0.260|0.166\\|0.261|\n|Pred-720|**0.199**\\|**0.288**|0.203\\|0.291|0.204\\|0.293|\n\nResults show that using timestamp embeddings from LLMs achieves better performance, which indicates better alignment with learned series embeddings in AutoTimes.\n\n**Q5**: How does the method overcome error accumulation?\n\nIt is true that there is no specific design for it in AutoTimes. Actually, the performance degradation during rolling forecast not only comes from the gap between the ground truth and prediction (error accumulation) but also comes from the dropping of lookback time series (lookback cut-off).\n\nTo be more precise, AutoTimes for one-for-all scenarios mainly copes with the second issue: predicting the next token of each position to keep our LLM-based forecaster feasible on prolonged inputs, while non-autoregressive models have a fixed input length. We will rephrase the relevant statements in our paper.\n\n**Q6**: Report the number of learnable parameters in Table 5 and confirm the scaling behavior of LLMs.\n\nThanks for your scientific rigor. We will include the following results in our revision, where a larger LLaMA-7B with fewer trainable parameters can still achieve better performance compared to OPT (1.3B~6.7B).\n\n|Datasets|GPT-2|OPT-350M|OPT-1.3B|OPT-2.7B|OPT-6.7B|LLaMA-7B|\n|-|-|-|-|-|-|-|\n|Hidden Dim.|768|1024|2048|2560|4096|4096|\n|Embedding layer|2-layer MLP|2-layer MLP|2-layer MLP|2-layer MLP|2-layer MLP|nn.Linear|\n|Trainable Param. (M)|0.44|0.58|1.10|1.36|2.15|0.79|\n|MSE (Avg)|0.397|0.401|0.396|0.394|0.394|0.389|\n\n[1] Tan et al. Are Language Models Actually Useful for Time Series Forecasting?\n\n[2] Woo et al. Unified Training of Universal Time Series Forecasting Transformers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0UKh3EvTpJ", "reviewer": "Reviewer_EAH7", "summary": "The authors present AutoTimes, a method that utilizes large language models (LLMs) for time series forecasting. One of the key underexplored research topics addressed by the authors is the lack of models and pre-training mechanisms that result in foundation models capable of handling lookback and forecasting horizons of arbitrary length. This is achieved by adapting the LLM forecasting framework to autoregressively forecast time series segments. Furthermore, the paper outlines techniques such as in-context learning to further improve prediction performance. Compared to previous works, the methodology requires only a small number of trainable parameters compared to previous LLM fine-tuning techniques. As far as I am aware, this work is the first to be capable of handling multimodal input and producing an autoregressive forecast in the domain of LLMs and time series.", "questions": "1. **Claim of missing data for foundation models:** I am aware that there is not a plethora of datasets available, but did you look into [1] (datasets) or [2] (models)?\n\n2. While I see the autoregressive part as a great contribution, I am not entirely convinced that the proposed embedding scheme, which only fine-tunes a small portion, brings a competitive improvement. There are multiple works [4, 5] that do not require any fine-tuning of the LLMs.\n\n3. L106: \"Unlike previous works, our proposed method regards time series itself as the instructive prompt. It avoids the modality gap caused by concatenating time series and language tokens directly.\" This statement gives the impression that your approach is superior. I would be interested to see how your model compares to more of these learnable-prompt methods, especially how it compares to *Test* [4] performance-wise, which appears to be superior to Time-LLM.\n\n4. How does AutoTimes handle missing data or irregular time series intervals? Your claimed improvement over [6], which does not use language directly, has the disadvantage that you cannot simply overcome NaNs in the time series directly.\n\n5. \"Unlike previous works, our proposed method regards time series itself as the instructive prompt. It avoids the modality gap caused by concatenating time series and language tokens directly.\" I am a bit confused. What kind of model checkpoint are you using? Instruction tuned? An instructive prompt is a directive where the user provides explicit instructions to guide the response. I thought you were skipping the language level. Could you outline how you embed your time series and text and how exactly this is fed into the model?\n\n---\n\n[1] Goswami et al. \"MOMENT: A Family of Open Time-series Foundation Models.\" ICML 2024.\n\n[2] Ekambaram et al. \"Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series\" arXiv:2401.03955\n\n[3] Chang et al. \"LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters\" 2024\n\n[4] Sun et al. \"TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series\" ICLR 2024\n\n[5] Wang et al. \"Xinglei Wang, Meng Fang, Zichao Zeng, and Tao Cheng. Where Would I Go Next? Large Language Models as Human Mobility Predictors\" 2023\n\n[6] Gruver, Nate, et al. “Large Language Models Are Zero-Shot Time Series Forecasters.” _Advances in Neural Information Processing Systems_, 2023", "limitations": "While the paper provides a limitations section, I don't think it is sufficient to bury it in the appendix. During my first read, I thought the authors entirely skipped the limitations until I found it in the checklist.\n\nHere are some specific points that I believe are missing or need improvement:\n\n1. Although the paper claims that the embedding and projection layers only account for 0.1% of the total parameters, the scalability of these layers for very large datasets or extremely long time series is not thoroughly discussed. An evaluation of how the method performs with significantly larger datasets would strengthen the paper.\n\n2. While the authors mention that they leave out real-world datasets for future work, I think the approach was not tested on datasets with missing values. This is especially important as their tokenization scheme seems to be based on non-overlapping patches, which are known to lose their locality assumption when missing values occur. An analysis of how the model handles missing data would provide a more comprehensive evaluation of its robustness and applicability.\n\nIn addition to that, the authors answered broader societal impacts sufficiently.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. I appreciate how the authors clearly outlined their contributions. \n- The observation that non-autoregressive methods may contradict decoder-only LLMs and the shortcomings of prior methods is well-motivated. The solution proposed in the context of LLMs and time series is novel.\n- The introduction of a One-for-all benchmark, which involves prediction horizons transfer learning instead of dataset transfer learning, is innovative.\n- The authors provide strong motivation and experimental proof to support their claims.\n- The flexibility and scalability of their method are demonstrated by successfully swapping out different LLMs with varying numbers of parameters.\n- The continuous improvement over previous SOTA methods on multiple datasets, along with their thorough ablations, strengthens their claims and illustrates the method's flexibility.", "weaknesses": "- I'm not a fan of your chosen color scheme. While I appreciate that you try to include colors for different entities, it is too vivid (jump from Figure 3 to 4 to 5), making the figures tough to read, especially when skimming through them quickly (which most first readers will do initially). It is not a major objection, but improving this aspect could enhance your manuscript.\n- Typo:\n    - L21: etc [22, 42], missing dot.\n- The paper makes certain simplified assumptions, such as treating time series segments independently for embedding. This might overlook complex inter-dependencies present in real-world time series data. Addressing these inter-dependencies could enhance the robustness and applicability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer EAH7\n\nMany thanks to Reviewer EAH7 for providing a thorough insightful review and recognizing our contributions.\n\n**Q1**: Suggestion to improve the presentation of the paper.\n\nThanks for your valuable feedback regarding the color scheme and the mentioned typo. We will use a more subdued scheme and fix the typo in the revision.\n\n**Q2**: Address inter-dependencies in real-world time series data.\n\nWe acknowledge that real-world multivariate time series often exhibit complex inter-dependencies that can significantly impact analysis. In terms of that, AutoTimes adopts Channel Independence like previous methods and further uses the timestamp as the position embedding to implicitly align different variates. \n\nAs you insightfully point out, it is necessary to explore the complex inter-dependencies, which is a hot topic in current deep time series models. It is also an essential problem for LLM4TS methods since the gap between natural language (1-D discrete token sequence) and time series (multi-dimensional continuous sequence) poses increasing challenges for LLMs to explicitly utilize the relationship between sequences.\n\nWe will explore several potential approaches: **integrating textual descriptions of variates** and **employing adaptors for variate correlating**. Your suggestion will guide us in refining our methodology.\n\n**Q3**: Claim of missing data for foundation models and explore the scalability on larger datasets.\n\nThanks for your mentioned works, we are excited to see recent works advancing the development of datasets and pre-trained large models in the field of time series. We will cite them in the related work section and polish the claim. \n\nBased on the mentioned works, we also provide an evaluation of AutoTimes on larger datasets (Time-Series Pile[1]) to address your concern about the scalability of the trainable layers:\n\n|Perfromance of Subset (MSE\\|MAE)|nn.Linear|3-layer MLP|\n|-|-|-|\n|ETTh1|0.724\\|0.586|**0.363**\\|**0.395**|\n|Weather|0.288\\|0.335|**0.166**\\|**0.211**|\n|ECL|0.856\\|0.764|**0.135**\\|**0.231**|\n|Traffic|1.393\\|0.799|**0.351**\\|**0.247**|\n\nAs we delve into layer scalability, the essence of designing the embedding scheme to address heterogeneous time series is highlighted, which provides good guidance for our future research.\n\n**Q4**: The effectiveness of fine-tuning and comparison with more LLM4TS methods.\n\nWe appreciate your concerns about the effectiveness of our fine-tuning approach, especially in light of several works that use LLMs for time series without fine-tuning. **We intend to leverage the general token transition of LLMs while tailoring it to the specific characteristics of the dataset**, which is achieved by freezing the LLM backbone and training new embedding layers only.\n\nWe provided detailed code and scripts in the $\\underline{\\text{supplementary material}}$ to ensure all the results are reproducible. Further, we compare the mentioned TEST[2], and AutoTimes achieves better performance on the majority of datasets.\n\n|Datasets (MSE\\|MAE)|AutoTimes|TEST|\n|-|-|-|\n|ETTh1|**0.389**\\|**0.422**|0.414\\|0.431|\n|Weather|0.235\\|0.273|**0.229**\\|**0.271**|\n|ECL|**0.159**\\|**0.253**|0.162\\|0.254|\n|Traffic|**0.374**\\|**0.264**|0.430\\|0.295|\n\n**Q5**: How does AutoTimes handle missing data or irregular time series intervals?\n\nThank you for your insightful question. At this stage, AutoTimes does not specifically address missing data or irregular intervals. It is the same with current works focusing on regular forecasting scenarios where time series are complete and consistently sampled.\n\nWe acknowledge that handling missing values and irregular intervals is a critical aspect of time series analysis, and we will add this as a limitation and conduct evaluations on well-acknowledged datasets in the future.\n\nAccording to your suggestions, we will also consider moving the limitations section to a more prominent position within the main body of the paper to ensure that readers can easily access and engage with this critical information.\n\n**Q6**: How does AutoTime embed and feed time series and texts?\n\nThe claim: \"our proposed method regards time series itself as the instructive prompt...\" refers to the following:\n\n* As depicted in $\\underline{\\text{Figure 1(b)}}$, previous LLM4TS methods feed (language tokens | lookback time series) to handle multimodal input.\n* As depicted in $\\underline{\\text{Figure 3}}$, AutoTimes feed (time series prompt | lookback time series) to enable in-context forecasting, where time series is self-prompted.\n* As shown in $\\underline{\\text{Equation 5}}$, AutoTimes uses textual timestamps as position embeddings and adds them with the corresponding series embedding of each token.\n\nTherefore, AutoTimes presents **prompting time series in two directions**. Horizontally, AutoTimes append the time series prompt at the front of the lookback series and regards it as the task demonstration. Vertically (series embedding + timestamp embedding), merging token-wise embeddings can use timestamps in natural language and aligns multiple variates.\n\n[1] Goswami et al. Moment: A Family of Open Time-Series Foundation Models.\n\n[2] Sun et al. TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pmmUHZ32CO", "reviewer": "Reviewer_k2uw", "summary": "This paper argues that existing LLM based time series models have not fully exploited the inherent autoregressive property and the decoder-only architecture of LLMs. To address this problem, this paper introduces a novel AutoTimes model, which exploits the autoregressive property of LLMs. Experimental results demonstrate that AutoTimes could outperform the baseline methods.", "questions": "Please see the weaknesses.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper exploits the autoregressive property of the LLMs. \n2. Comprehensive experiments are conducted to demonstrate the effectiveness of the proposed AutoTimes, including forecasting, zero-shot forecasting and in-context forecasting.\n3. This paper is well-written, and is easy-to-follow.", "weaknesses": "1. Autoregression seems a little bit trivial.\n2. It is not quite clear how much improvements can be brought by auto-regressively generating time series. Either direct comparison or theoretical analysis should be provided.\n3. It is difficult to directly draw conclusions about the scaling behaviors from Table 5. The authors draw conclusions by comparing results of OPT-x models, rather than comparing different models e.g., compare OPT with LLaMA or GPT-2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer k2uw\n\nMany thanks to Reviewer k2uw for providing a valuable review.\n\n**Q1**: Reclarify the contributions of the proposed method.\n\nThe reviewer mentioned that \"the paper exploits the autoregressive property of the LLMs\". We agree with this argument but also would like to highlight detailed contributions of our work:\n\n* We delve for **the first time into the autoregression in LLM4TS methods**. It addresses the raising concerns about the effectiveness of prevalent non-autoregressive LLM4TS methods (refer to $\\underline{\\text{Q1 of Reviewer 1PWH}}$).\n* We proposed the one-for-all benchmark that **breaks the status quo of respective training on specific lookback-forecast length** (appreciated by $\\underline{\\text{Reviewer EAH7}}$), which is an essential step toward foundation models.\n* Our method **achieves state-of-the-art forecasting performance** and requires **significantly fewer tunable parameters among advanced LLM4TS methods**. The method's efficiency is acknowledged by all other reviewers.\n* We present the concept of **in-context forecasting for the first time**, where time series prompts and prompt engineering are closely researched (refer to $\\underline{\\text{Q3 of Reviewer acJd}}$).\n* Beyond adopting LLMs on end-to-end forecasting, we facilitate the full capabilities of LLMs for time series, such as iterative multistep generation, zero-shot generalization, and scaling behavior.\n\n**Q2**: How much improvement is brought by autoregressively generating time series?\n\nTo address your concern, we provide a comprehensive ablation study: AutoTimes (FlattenHead) replaces the original segment-wise projection layer by (Flatten + linear head) of PatchTST, the prevalent module in non-autoregressive models. Here are the results:\n\n|ETTh1 (MSE\\|MAE)|AutoTimes (Original)|AutoTimes (FlattenHead)|\n|-|-|-|\n|Pred-96|**0.360**\\|**0.400**|0.385\\|0.420|\n|Pred-192|**0.388**\\|**0.419**|0.445\\|0.463|\n|Pred-336|**0.401**\\|**0.429**|0.463\\|0.475|\n|Pred-720|**0.406**\\|**0.440**|0.574\\|0.542|\n\n|ECL (MSE\\|MAE)|AutoTimes (Original)|AutoTimes (FlattenHead)|\n|-|-|-|\n|Pred-96|**0.129**\\|**0.225**|0.142\\|0.247|\n|Pred-192|**0.147**\\|**0.241**|0.157\\|0.259|\n|Pred-336|**0.162**\\|**0.258**|0.201\\|0.311|\n|Pred-720|**0.199**\\|**0.288**|0.232\\|0.331|\n\n|Weather (MSE\\|MAE)|AutoTimes (Original)|AutoTimes (FlattenHead)|\n|-|-|-|\n|Pred-96|**0.153**\\|**0.203**|0.155\\|0.209|\n|Pred-192|**0.201**\\|**0.250**|0.202\\|0.251|\n|Pred-336|**0.256**\\|**0.293**|0.257\\|0.286|\n|Pred-720|**0.331**\\|**0.345**|0.333\\|0.349|\n\n|Traffic (MSE\\|MAE)|AutoTimes (Original)|AutoTimes (FlattenHead)|\n|-|-|-|\n|Pred-96|**0.343**\\|**0.248**|0.367\\|0.261|\n|Pred-192|**0.362**\\|**0.257**|0.391\\|0.282|\n|Pred-336|**0.379**\\|**0.266**|0.404\\|0.287|\n|Pred-720|**0.413**\\|**0.284**|0.432\\|0.294|\n\nAs shown in the above tables, the performance of non-autoregressively generation is consistently worse than original AutoTimes. In addition to empirical results, we'd like to provide an analysis of the two approaches:\n\n* **One model fits all lengths:** While most deep forecasters have to be trained and applied to specific length settings, autoregressive models are more feasible in variable-length scenarios. We provide the comparison as follows:\n\n|Comparison|Non-autoregressive|Autoregressive|\n|-|-|-|\n|Training|Trained with specific lookback-forecast lengths|Trained with the context length with **each generated token being supervised**|\n|One-step Forecasting|Applicable only on fixed lookback-forecast lengths|Flexible on scenarios **less than the context length** like large language models|\n|Rolling Forecasting|Has to drop the lookback series because of the fixed input length|**Can prolong the lookback horizon** until the total length exceeds the context length|\n\n* **Fewer parameters for training:** Supposing $L$ is the lookback length, $F$ is the forecast length, $S$ is the segment (token) length and $N$ is the segment (token) number. We count the parameters for embedding and projection:\n\n * Non-autoregression: time series segment ($S$) -> representation ($D$) -> flattened ($N\\times D$)-> future time series ($F$)\n * Autoregression: time series segment ($S$) -> representation ($D$) -> next time series segment ($S$)\n\n In non-autoregressive models, all the tokens of the lookback series **are flattened and projected to future time series**, which leads to costly parameters of $ND\\times F$. Instead, the projection of AutoTimes is independently conducted on segments, leading to fewer introduced parameters $D\\times S$.\n\n* **Consistent with the utilization of LLMs:** the main claim of our paper is that non-autoregressive LLM4TS leads to inconsistencies, where inherently GPT-style models are fine-tuned in the BERT-style. Instead, we suppose the token transition of LLMs is general-purpose and find it transferable as the transition of time series segments. Consequently, the powerful generation ability of LLMs can be naturally inherited.\n\n**Q3**: Concern about the scaling behaviors of LLM-based forecasters.\n\nThank you for your feedback regarding the conclusions drawn from Table 5. It is recommended to see the scaling behavior in $\\underline{\\text{Figure 4 of the main text}}$, where the scaling behavior is not only observed in OPT-x models but also revealed among GPT-2, OPT, and LLaMA. \n\nWe further eliminate the influence of parameter count in trainable layers. As shown in the following table, a larger LLaMA-7B with fewer trainable parameters can still achieve better performance compared to OPT (1.3B~6.7B), demonstrating the performance stems from the scaling of LLMs, not simply the parameter of trainable layers.\n\n|Datasets|GPT-2|OPT-350M|OPT-1.3B|OPT-2.7B|OPT-6.7B|LLaMA-7B|\n|-|-|-|-|-|-|-|\n|Hidden Dimension|768|1024|2048|2560|4096|4096|\n|Embedding layer|2-layer MLP|2-layer MLP|2-layer MLP|2-layer MLP|2-layer MLP|nn.Linear|\n|Trainable Parameters (M)|0.44|0.58|1.10|1.36|2.15|0.79|\n|Performance (Avg. MSE)|0.397|0.401|0.396|0.394|0.394|0.389|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Summary of Rebuttal

We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve our paper further.

In this work, we proposed an effective approach (AutoTimes) to repurpose LLMs as **autoregressive forecasters**. Unlike previous works that adopt LLMs as non-autoregressive models, we maintain **consistency with the training and inference of LLMs**. AutoTimes exhibits **variable-length feasibility**, **scalability** with larger LLMs, and **utilization of textual timestamps**, achieving **state-of-the-art performance** with minimally trainable parameters. Further, **we propose in-context forecasting for the first time**, extending the conventional forecasting context to discontinuous time series windows as task demonstrations.

The reviewers generally held positive opinions of our paper, in that the proposed method is "**well-motivated**", "**intuitive**", "**novel**", and "**more reasonable than previous ones**", the paper is "**well-written**" and "**easy to follow**", in-context forecasting is "**resourceful idea for the community**", "**one-for-all benchmark is innovative**", "**numerous experiments were conducted**" and "**achieve continuous improvement over previous SOTA methods**".

The reviewers also raised insightful and constructive concerns. We made every effort to address all the concerns by providing sufficient evidence and requested results. Here is the summary of the major revisions:

* **More evaluations (Reviewer acJd, EAH7)**: We extensively include the mentioned baseline models and datasets. By making great efforts to complete the evaluations, we verify that our method still achieves the best performance and good generality on new baselines and benchmarks.
* **Systematic exploration of in-context forecasting  (Reviewer acJd, 1PWH)**: We delve into in-context forecasting, including more evaluated datasets and different strategies to retrieve time series prompts. It highlights the significance of incorporating prior knowledge (such as periodicity) into the prompt engineering of time series.
* **Ablation study (Reviewer 1PWH, k2uw**): We conduct comprehensive ablations to confirm that AutoTimes truly utilize the ability of LLMs, and highlight the improvement and significance of autoregression. We also provide ablation on alternative embeddings of timestamps and confirm the scaling behavior of our methodology.
* **Technical contributions (Reviewer k2uw)**: We highlight the contribution of introducing autoregression into LLM4TS first, which facilitates the full abilities and efficiency of LLMs. By analyzing autoregressive and non-autoregressive approaches, we illustrate the advantages in both theoretical and experimental aspects.
* **Polished writings (Reviewer EAH7)**: We summarize the revisions and future directions with helpful suggestions from the reviewers. The peculiarities and limitations of our work are clarified more clearly.

The valuable suggestions from reviewers are very helpful for us to revise the paper to a better shape. We'd be very happy to answer any further questions.

Looking forward to the reviewer's feedback.

Meta-review: This paper introduces AutoTimes, a novel approach for time series forecasting using Large Language Models (LLMs). It leverages the inherent autoregressive property and decoder-only architecture of LLMs, generating time series segments autoregressively and incorporating techniques like positional embeddings and in-context learning for improved predictions. 

The proposed method shows strong performance in experiments. The paper is well-written. 

The original reviews raised key concerns regarding the limited baselines, overselling claims, unclear benefits of in-context forecasting, and scaling behaviors. The rebuttal has included an extensive amount of new results and analysis, addressing most of the concerns. 

Overall, the strengths of the paper overweighs the weaknesses.
Decision: Accept (poster)