Title: AutoTimes: Autoregressive Time Series Forecasters via Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Time Series Forecasting

Large Language Models
Abstract: Foundation models of time series have not been fully developed due to the limited availability of time series corpora and the underexploration of scalable pre-training. Based on the similar sequential formulation of time series and natural language, increasing research demonstrates the feasibility of leveraging large language models (LLM) for time series. Nevertheless, the inherent autoregressive property and decoder-only architecture of LLMs have not been fully considered, resulting in insufficient utilization of LLM abilities. To fully revitalize the general-purpose token transition and multi-step generation capability of large language models, we propose AutoTimes to repurpose LLMs as autoregressive time series forecasters, which projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. Compatible with any decoder-only LLMs, the consequent forecaster exhibits the flexibility of the lookback length and scalability with larger LLMs. Further, we formulate time series as prompts, extending the context for prediction beyond the lookback window, termed in-context forecasting. By introducing LLM-embedded textual timestamps, AutoTimes can utilize chronological information to align multivariate time series. Empirically, AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5\times$ training/inference speedup compared to advanced LLM-based forecasters. Code is available at this repository: https://github.com/thuml/AutoTimes.

AutoTimes: Autoregressive Time Series Forecasters via Large Language Models

Abstract

Foundation models of time series have not been fully developed due to the limited availability of time series corpora and the underexploration of scalable pre-training. Based on the similar sequential formulation of time series and natural language, increasing research demonstrates the feasibility of leveraging large language models (LLM) for time series. Nevertheless, the inherent autoregressive property and decoder-only architecture of LLMs have not been fully considered, resulting in insufficient utilization of LLM abilities. To fully revitalize the general-purpose token transition and multi-step generation capability of large language models, we propose AutoTimes to repurpose LLMs as Auto regressive Time s eries forecasters, which projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. Compatible with any decoder-only LLMs, the consequent forecaster exhibits the flexibility of the lookback length and scalability with larger LLMs. Further, we formulate time series as prompts, extending the context for prediction beyond the lookback window, termed in-context forecasting . By introducing LLM-embedded textual timestamps, AutoTimes can utilize chronological information to align multivariate time series. Empirically, AutoTimes achieves state-of-the-art with 0 . 1% trainable parameters and over 5 × training/inference speedup compared to advanced LLM-based forecasters. Code is available at this repository: https://github.com/thuml/AutoTimes .

1 Introduction

Time series forecasting is of crucial demand in real-world applications, covering various domains including climate, economics, energy, operations, etc. [22, 43]. The growing challenges of generalpurpose forecasting, where one model is versatile to handle variable-length scenarios [24, 41] and the prediction is necessarily instructed by auxiliary information in other modalities [40, 44], underscore the demand for foundation models [3] of time series, which are aimed to exhibit enhanced capabilities, including multi-step generation, zero-shot generalization [49, 13], in-context learning and multimodal utilization [15], thereby expanding the scope of time series forecasting to a wider range of situations.

Nevertheless, the development of time series foundation models has been hampered by the limited availability of large-scale pre-training datasets and the technical uncertainty of scalable backbones. In contrast, rapid progress is witnessed in large language models (LLM), facilitated by extensive text corpora [50], available pre-trained models [36], and well-established adaptation techniques [14]. Notably, language and time series share basic commonalities in sequence modeling and generation by learned token transitions, presenting opportunities to adopt off-the-shelf LLMs for time series.

Autoregressive

2'

3'

4'

1

2

3

4'

1

2

3

Non-Autoregressive

1

2

3

+

Concat

(a) Forecasting Approach

(b) Prompting Mechanisms

5'

5'

4'

LLM

Next Token

Prediction

LLM

Lookback Tokens

Predicted Tokens

…

Time Series

Sequence-wise Prompting

4'

5'

6'

LLM

Language as Prompts

Autoregressive (Ours)

Time Series as Prompts (Ours)

1

2

3

+

Concat

LLM

Language as Prompts (Previous)

1

2

3

+

Concat

LLM

Non-Autoregressive

LLM

LLM

Next Token

Prediction

…

Time Series

Natural Language

Predicted Series

Lookback Series

Relevant Series

Despite recent studies on large language models for time series (LLM4TS) achieving performance breakthroughs in current forecasting benchmarks [15], the mechanism by which LLMs are aligned to the time series modality still remains obscure. The pilot work, FPT [49] leverages LLMs as generic sequential representation extractors for time series, influencing subsequent LLM4TS methodologies. As depicted in Figure 1 (a), the non-autoregressive approach, where time series are segmented into tokens, flattens and projects all lookback tokens for the prediction in a single step. However, it causes inconsistencies in both model structure and generative approach of LLMs: decoder-only models for autoregressive generation are converted to encoder-only and non-autoregressive forecasters .

Given that prior studies [9, 38] reveal that generalization performance of LLMs is largely derived from the decoder-only structure trained autoregressively, talents of LLMs may not be fully exhibited. It is also supported by the recent rethinking of previous LLM4TS methods [35], which generally lack the maintenance of autoregression, the essential characteristic of both large language models and statistical forecasters [5, 39]. Therefore, autoregressive LLM4TS methods are underexplored, which can potentially unlock multi-step generation like LLMs, presenting one model for arbitrary lengths.

Motivated by the reflections, we propose AutoTimes to adapt LLMs as time series forecasters, which retrieves the consistency of autoregression with revitalized LLM capabilities to produce foundation models for time series forecasting . Technically, we independently embed time series segments into the latent space of language models by the consistent training objective: next token prediction [2]. To fully leverage the inherent token transitions of LLMs and reduce the training cost, we freeze the LLM and establish token embedding and projection for time series, which only account for up to 0 . 1% total parameters. The consequent forecaster adopts autoregressive inference like LLMs, which is no longer constrained to specific lookback/forecast lengths. Going beyond conventional time series forecasting, we propose in-context forecasting as shown in Figure 1, where time series can be self-prompted by relevant contexts. We further adopt LLM-embedded timestamps as the position embedding to utilize chronological information and align multiple variates. Our contributions are summarized as follows:

By refining the inconsistency of non-autoregressive LLM4TS methods, we propose to inherit the autoregressive property of LLMs, which frees our method from training respectively on the lookback length and allows arbitrary-length predictions with chronological awareness.

We present AutoTimes, a simple but effective approach to acquire LLM-based forecasters by lightweight adaptation, which utilizes the inherent token transition as the future extrapolation of time series. Further, we propose in-context forecasting, which renovates the conventional paradigm by introducing relevant time series prompts to enhance forecasting.

Compared with state-of-the-art methods, our repurposed forecaster achieves superior performance while saving over 80% training and inference time, and further exhibits zero-shot generalizability, in-context forecasting, and scaling behavior empowered by LLMs.

2 Related Work

2.1 Autoregressive Models

Autoregression is an essential concept of both language modeling and time series forecasting. Despite prevalent deep forecasters [10, 26, 42, 48] adopt a non-autoregressive approach without the

requirement of iterative forecasting, autoregression, the absent exploration in deep forecasters, serves as the fundamental principle of statistical methods, which enables variable-length predictions. The most well-known model, ARIMA [4] is developed by incorporating differencing on AR and MA models, which are both autoregressive models with learned time-invariant transition from the past to the future. Incorporated with decomposition and pre-defined transitions, exponential smoothing [39] and state space models (SSM) [12, 23] also take the same autoregressive formulation.

Autoregressive language models [27, 31] are trained with fine-grained supervision, where the generated token of each position is independently supervised. Consequently, they are not constrained by specific input/output lengths and excel at multi-step generation. Furthermore, existing LLMs are inherently autoregressive models [47], which demonstrate advanced abilities that are not present in small models, such as the generalization [38], scalability [6], and task generality [31, 32]. Therefore, it is imperative to adapt off-the-shelf LLMs as autoregressive forecasters, which keeps the consistency to fully revitalize the model capacity and general-purpose token transitions.

2.2 Large Language Models for Time Series

With the immense advancement of large language model infrastructure, LLM4TS methods have been experiencing significant development in recent years. PromptCast [44] reformulates time series as text pairs and accomplishes forecasting as a sentence-to-sentence task. LLMTime [13] regards time series as numerical tokens, demonstrating the zero-shot generalizability in time series forecasting. FPT [49] fine-tunes parameters of the LLM to adapt it as a general representation extractor serving for multiple time series analysis tasks. UniTime [21] adapts a language model across diverse time series for a unified forecaster of multiple domains. Based on thriving prompting techniques, deft language prompts [15, 21] and soft prompting [7] for time series are further investigated.

LLM4TS methods have achieved performance breakthroughs in time series forecasting, but the cost of training and inference can sometimes be resource-consuming due to the immensity of LLMs. Recent revisiting of LLM4TS methods has revealed the inefficacy of LLMs adapted in the non-autoregressive approach [35]. By contrast, AutoTimes frozen LLMs, transfers the general-purpose token transition, and introduces minimal parameters to realize autoregressive next token prediction, thereby achieving better model efficiency and consistent utilization of large models. We further provide Table 1 that categorizes prevalent LLM4TS methods by several essential aspects.

2.3 Multimodal Language Models

Multimodal models have been well-developed upon LLMs, among which vision language models (VLM) have experienced rapid growth [1, 27]. The booming pre-trained vision backbones [11, 30], together with the instruction tuning paradigm, has revealed the potential of LLMs for vision tasks, where visual tokens and language tokens are concatenated as the input of the LLM [19, 20]. Inspired by this, previous LLM4TS methods utilize instructive language tokens as prefix-prompts for time series analysis [15, 34, 44]. Unlike previous works, our proposed method regards time series itself as the instructive prompt. It avoids the modality gap caused by concatenating time series and language tokens directly. We incorporate chronological information, the textual timestamp of time series, such that the language model can effectively perceive date and periodicity as the position embedding, and align simultaneous events from different time series [22] for multivariate forecasting.

3 Method

The proposed AutoTimes adapts large language models for multivariate time series forecasting. Given lookback observations x 1: L = { x 1 , . . . , x L } ∈ R L × C with L time steps and C variates, the objective

the

quick

fox

over

the

lazy

dog

Language

Transitions

Time Series

Transitions

Repurpose

LLM

Forecaster

Token Perspective

Model Perspective

Token-wise

Alignment

is to predict the future F time steps x L +1: L + F = { x L +1 , . . . , x L + F } ∈ R F × C . Besides, the textual timestamp a t (e.g. 2016 / 07 / 05 00 : 00 : 00 ), as the most common covariate, is adopted for prediction, which is aligned with time points x t ∈ R C at time t . The task is to train an LLM-based forecaster f that is able to predict with the (varying) lookback length L for the (arbitrary) forecast length F as:

3.1 Modality Alignment

Time series token To empower the forecaster with the capability to predict time series for arbitrary lengths, we repurpose autoregressive LLMs as time series forecasters as depicted in Figure 2. Prior to this, we define time series token as the consecutive and non-overlapping segment of a single variate. It is regarded as the common token of the LLM-based forecaster, which encompasses series variations and mitigates excessively long autoregression. To focus on modeling temporal variations, our forecaster predicts each variate independently. Beyond Channel Independence [26] that implicitly captures the multivariate correlation [22] by shared parameters, AutoTimes converts timestamps into position embeddings and explicitly aligns simultaneous segment tokens, which is detailed in the next paragraph. Therefore, we simplify x t as the time point of specific variate x t ∈ R . Given a single-variate time series of context length NS , the i -th segment of length S is denoted as:

Considering the general-purpose token transition, we freeze the parameters of large language models. To realize the token-wise alignment between time series tokens and language tokens, we establish SegmentEmbedding( · ) : R S ↦→ R D that independently embeds segments into the latent space:

where D is consistent with the dimension of the LLM.

Position embedding Timestamp, an essential covariate indicating the chronological information, is generally utilized as an extra embedding in previous deep forecasters [43, 48]. However, increasing models [10, 26, 45] have discarded the embedding and found the performance will not be greatly affected, implying the improper encoding of timestamps. In contrast, textual timestamps have been demonstrated as an enhancement in LLM4TS methods, which are always formulated into prefixprompts [15, 21]. Nevertheless, it also leads to excessive context length, impeding LLMs from paying sufficient attention to time series tokens and inducing time-consuming feed-forwarding. Inspired by the functionality of position embedding, which incorporates information about the relative or absolute position of the tokens [37]. We adopt LLM-embedded timestamps as position embeddings to utilize temporal information and align simultaneous events (segments) from different varieties.

Technically, we formulate the starting and end timestamps of corresponding segments by the template demonstrated in Figure 3. Experimentally, we observe that the simple template without deft design can consistently boost the forecasting performance in Appendix D.5, aiding the LLM-based forecaster to comprehend the date and align different variates based on Channel Independence. Since all the previous language tokens are visible to the special ending token <EOS> of a sentence, we adopt the embedding of <EOS> as TE i ∈ R D as the position embedding from textual timestamps:

Notably, TE i is pre-computed by LLMs such that runtime forwarding for language tokens is not required during training. Given that the latent space of the LLM locates both time series tokens and

3.2 Next Token Prediction

As shown in Figure 3, prevalent LLMs [6, 36] are endowed with the capability of predicting the next token s i based on the preceding tokens s <i . We reutilize LLMs in a fully consistent approach and generate prediction of arbitrary lengths iteratively. Given a time series of context length NS , the input series is segmented and embedded into N token embeddings { E 1 , . . . , E N } . The training objective is to independently generate the next tokens { ˆ s 2 , . . . , ˆ s N +1 } . We feed the token embeddings E i into the intermediate layers of the LLM, which inherently parameterize token transitions:

We adopt SegmentProjection( · ) : R D ↦→ R S to independently projects embeddings to segments:

Finally, each predicted segment is supervised by the token-wise ground truth to optimize the parameters of embedding and projection layers, which are simply implemented by multi-layer perceptrons:

Notably, the context length NS is decided during training, representing the maximum input length during inference. Therefore, one consequent forecaster is suitable for different input lengths like the LLM, validated in Appendix D.4. Moreover, AutoTimes can generate predictions of arbitrary lengths by iterative multi-step forecasting, proven to overcome error accumulation better than state-of-the-art forecasters in Section 4.1, since autoregressive LLMs inherently excel at multi-step generation:

Instead of respectively training models on different lookback/forecast lengths, AutoTimes handles all the scenarios by one model. Surprisingly, with the consistency of autoregression, it also inherits notable generalizability and scaling behavior of LLMs, which is demonstrated in Sections 4.2 and 4.4.

<latexi sh

1_b

64="cRJ3NX8o fF

v

z

w

MW

5y

BA

>

C0H

jV

LS

D2

r

p

Zdugk

I

U

q

O

9

T

K

+

EP

/

Y

Q

n

G

7

m

<latexi sh

1_b

64="oM0

A

y+wS

f

K/k

Oc

U3X

>

C

H

jV

L

N

FD2

r

p

Zdug

RI

W

9

m

T

n

q

8

G

Q

J

Pv

Y

5

z

E

7

B

<latexi sh

1_b

64="mQHCW7pu

+N

nBwykf

d

V

A

>

3

cj

LS

FD2

r

Ivgq

o

K9

Z

0

RT

/

P

X

M

z

Y

O

E

5

U

8

J

G

<latexi sh

1_b

64="

GKJ

Ng

H

wj rLFd72QEDT0/

>A

Cz3

c

V

S

vqk omIu

y

fUB

p

O

9

8

+

P

M

Y

5

Z

R

W

n

X

<latexi sh

1_b

64="

fM

dP7OUC0W

+F

35

n

oA

>

cjVHLS

N

D2

r

p

BIvgq

Q

m

Ek

z

R

Y

K

/J

T

yZ

9

G

u

X

w

8

<latexi sh

1_b

64="Pc

C

9UGXo/A

rMD7

m

OY0I

>

H

jV

LS

N

F

2

pf

Zdugk

w

R

W

vq

J

n

T

K

+

E

8

yQ5

z

B

3

<latexi sh

1_b

64="I3

u+vSmV

Lp

9N

A

7R

TKgE

>

C0H

cj

FD2

r

f

Zd

k

w

o

8

W

/

X

U

Q

5

z

J

G

n

Y

M

O

yB

q

P

<latexi sh

1_b

64="w75

P

p

WqygQ+

j

JN0

ozM

>A

C

3

c

VHLS

FD2

r

f

dBIv

K9

Z

mRT

nu

/

U

k

YG

X

8

O

E

<latexi sh

1_b

64="S

ZRcEgH

M/

F9

BoPDC

f

>A

z3

jV

L

N

2

r

p

du

k

I

0

7

+

W

JO

5

y

QG

8Y

T

v

X

m

q

K

n

<latexi sh

1_b

64="

z

Vp

I

f

gOmw

n

9HR0

>A

C

3

cj

LS

N

FD2

r

7dB

v

q

Q

o

+5

Ek

UP

Y

K

/J

Z

My

8

u

G

W

T

X

<latexi sh

1_b

64="

3

VrfH

Co

8

L

Knk

Z

u

XA

>

0

cj

S

N

FD2

/q

QT

I

Y

UR

p

9

m

W

P+B

v

7

w

E

z5

d

g

M

J

G

y

O

<latexi sh

1_b

64="FUL8

By

Pg

AkD

EM

wv

>

C0H

cjV

S

N

2

r

pf

Zdu

RI

7

o

qK

m

T

n

9

/

+

X

G

Q

O

J

3

Y

5

W

z

<latexi sh

1_b

64="

f

GrPYCmEwS

dW

+

k

jgc

>A

3

VHL

N

FD

2v

p

u

RI

7

o

K9

Z

0

T

yn

/

Q

X

q

M

z

O

B

8

J

U

5

<latexi sh

1_b

64="720

Q3C

F8d5

z

I

q

>

j

HLS

N

D

G

v

w

K

om

+

BW

pJO

MZ

T

g

u9

/EP

y

R

f

n

X

U

r

<latexi sh

1_b

64="

BQI

fCSO+FG

m2

yLDUR

k

>A

3

cjVH

N

vWpdug

w

7

o

5

E

Pz

Y

K

/J

T

Z

9

r

M

n

0

q

X

8

8'

This is the series

from 2016/7/2 00:00:00

to 2016/7/2 15:00:00

<EOS>

Template

Ground Truth

Segment Projection

Segment Embedding

Next Token Prediction

1

2

3

2

3

4

2'

3'

4'

as Position

Embedding

Time Series

Segments

7

8

MSE

MSE

MSE

This is the series

from 2016/7/2 00:00:00

to 2016/7/2 15:00:00

<EOS>

This is the series

to 2016/7/1 23:00:00

2016/7/7

2016/7/1

2016/7/2

……

2016/7/7  23:00:00

Timestamp

……

Time Series

……

(Pre-computed)

This is the series

from 2016/7/2 00:00:00

to 2016/7/2 15:00:00

<EOS>

This is the series

from 2016/7/2 00:00:00

to 2016/7/2 15:00:00

<EOS>

Embeddings

of Texts

s

2

SE

2

TE

2

TE

1

ˆ

S

3

ˆ

S

2

s

1

SE

1

ˆ

E

2

ˆ

E

3

s

7

TE

7

SE

7

ˆ

E

8

ˆ

S

8

2016/7/2  00:00:00

……

2016/7/2  23:00:00

……

2016/7/1  23:00:00

Context Length

Segment Length

…

…

…

…

…

…

S

language tokens, the position embedding can be integrated with the corresponding time span without increasing the context length. Concretely, the token embedding E i ∈ R D is obtained by:

3.3 In-Context Forecasting

Large language models are capable of generating expected outputs based on provided task demonstrations from downstream datasets without gradient updating, known as the in-context learning ability. The task demonstrations are generally constituted by paired questions and answers [47]. Formally, the context C = { g ( x (1) , y (1) ) , . . . , g ( x ( m ) , y ( m ) ) } represents a set of demonstrations with m pairs, where g ( · ) is the template that transforms each question and answer into natural language.

In terms of time series forecasting, we propose to constitute the pair by lookback-forecast windows, which are exactly represented as successive time points from earlier historical observations. Hence, we use time series in target datasets as prompts, extending the context for prediction beyond consecutive lookback series . We denote the extended context as C , which contains m time series prompts tsp ( j ) :

During training, we first obtain an LLM-based forecaster on a source dataset and select time series prompts from the downstream target dataset based on a unified strategy. During inference, we ensure all the prompts appear before the window to be predicted, such that there is no data leakage from future information. As shown in Figure 4, we concatenate time series prompts with lookback series and feed them as the context of the forecaster, termed in-context forecasting :

4 Experiments

We conduct thorough evaluations of the performance of AutoTimes, including time series forecasting, zero-shot forecasting, and the proposed in-context forecasting. Additional analyses are included to evaluate the generality, scaling behavior, and adaptation cost of large language models. Detailed code implementation for reproduction is provided in our public code repository.

4.1 Time Series Forecasting

Benchmarks For long-term time series forecasting, we extensively include real-world datasets, including ETTh1, ECL, Traffic, Weather [43], and Solar-Energy [22]. For short-term forecasting, we adopt the well-acknowledged M4 competition [25]. Detailed descriptions are provided in Appendix A.

Baselines We compare AutoTimes with state-of-the-art models, including advanced LLM4TS methods: TimeLLM [15], UniTime [21], and FPT [49]; well-acknowledged deep forecasters: iTransformer [22], DLinear [45], PatchTST [26], and TimesNet [42]. For the challenging short-term forecasting, we further include competitive baselines: Koopa [23], N-HiTS [8] and N-BEATS [28]. All baselines are officially implemented or reported. We adopt LLaMA-7B [36] as our base LLM. Detailed implementations, error bars, and hyperparameter analysis are provided in Appendix B and C.

Setups For short-term forecasting, we follow the well-acknowledged TimesNet [42], which assesses the fundamental ability of forecasters in modeling temporal variations. For long-term forecasting, we establish a novel one-for-all benchmark: a single forecaster is trained on one dataset and subsequently utilized for all prediction lengths. We highlight that this approach evaluates the basic versatility as foundation models of time series, which aims to break the prevailing practice of extensive training across diverse real-world scenarios. To be specific, we evaluate all methods by rolling forecasting: a model is trained with predetermined input/output lengths, and the predicted values are integrated as part of the input in subsequent iterations until reaching the desired forecast length. Therefore, the key to success in this task lies in mitigating multi-step error accumulation. Still, the conventional one-for-one approach that trains forecasters respectively on each length is also provided in Table 12.

Results The average results are presented in Table 2- 3, with the best results in bold and the second best underlined. AutoTimes consistently outperforms all counterparts of short-term forecasting in Table 2, demonstrating the basic ability of LLM-based forecasters to capture diverse series variations. Further, in the one-for-all long-term scenarios, AutoTimes surpasses other LLM4TS methods and deep forecasters in 80% datasets in Table 3, outperforming previous state-of-the-art TimeLLM by

Table 3: Long-term forecasting results of one-for-all: we conduct rolling forecasting with a single model trained on each dataset and accomplish four desired forecast lengths in { 96 , 192 , 336 , 720 } . AutoTimes adapt LLMs with the context length C = 672 . We set the input length L = 672 and output length F = 96 in other methods. All results are averaged. Full results is provided in Table 10.

9 . 12% in average. Compared with other forecasters trained in the one-for-one scenario in Table 12, AutoTimes still achieved state-of-the-art performance in 70% of settings without respective training.

By diving into the proposed one-for-all and the traditional one-for-one benchmarks in Table 3 and 12, it is notable that prevalent deep forecasters, such as Transformer-based forecasters and DLinear, can achieve competitive and even better results under rolling forecasting. Nevertheless, the performance of non-autoregressive LLM4TS methods can degenerate a lot without respective training. Therefore, it highlights our persistent utilization of autoregression and thorough leveraging of inherent token transitions of LLMs, thereby mitigating error accumulation during multi-step rolling forecasting.

4.2 Zero-Shot Forecasting

Setups Large language models have exhibited remarkable zero-shot generalization capability [6]. To verify whether our LLM-based forecaster inherits this ability, where no training sample of the target domain is available, we assess the performance of zero-shot forecasting. Concretely, we adhere to the benchmark established by FPT [49], where the forecaster is initially trained on a source domain and subsequently evaluated on an unseen target domain. We conduct the transfer learning between the M3 and M4 competitions, both of which encompass abundant temporal variation patterns but follow different data distributions. We compare AutoTimes with deep forecasters and FPT as the only LLM4TS method, given that only FPT has exhibited zero-shot generalization in this benchmark.

Results The comprehensive results of zero-shot forecasting are presented in Table 4. AutoTimes demonstrates superior performance compared to deep forecasters and FPT in both M4 → M3 and M3 → M4 scenarios. It is evident that LLM4TS methods generally achieve improved performance in this task due to the enhanced model capacity, leading to a 15% SMAPE reduction compared with the efficient forecaster DLinear. Despite sharing the same Transformer backbone, LLM4TS methods still outperform PatchTST due to the transferable knowledge pre-trained on large corpora of sequences. This underscores the advantage of leveraging LLMs for time series forecasting. Moreover, AutoTimes inherits general-purpose token transitions, surpassing FPT without tuning intermediate LLM layers.

Lookback

Zero-Shot Forecasting

S2'

S3'

S1

S2

S3

S1

S2

S3

Forecast

S5'

…

Forecaster

S5

In-Context Forecasting

S2'

S3'

S1

S2

S3

S5'

…

P1

P2

P3

P4

P2'

P3'

P4'

P5'

Forecaster

…

Time Series Prompt

P1

P2

P3

P4

Target Domain

Trained on Source Domain

4.3 In-Context Forecasting

Setups We conduct in-context forecasting on AutoTimes, which is depicted in Figure 4. Similar to zero-shot forecasting, the task is to apply a forecaster, trained on a source dataset, to an unseen target dataset. Additionally, several task demonstrations from the target domain, referred to as time series prompts in Equation 10, are available during inference. Specifically, we concatenate these prompts with the lookback window to form the context for prediction.

We adopt the aforementioned M4 → M3 scenario. Since the samples of the M3 dataset are univariate time series with different lengths, we always predict the last F time points of each sample during inference. We set the lookback length L = F , and thus the length of time series prompts is 2 F . We set the number of prompts m = 1 . Therefore, we initially train an LLM-based forecaster on the source M4 dataset with the context length of 3 F . We adopt an intuitive strategy to select the prompt: uniformly adopting the first 2 F time points of the time series as the corresponding prompt. Supposing the lookback series starts after time t ( ≥ F ) , in-context forecasting is formulated as:

To prevent data leakage of future information, too short samples are discarded to prevent the overlap between the prompt and the future prediction. The implementation details of in-context forecasting are provided in Appendix D.6. We further investigate different prompt retrieval strategies. Insightful results are provided to reveal the influence of using time series prompts for interactive prediction and take-away instructions of prompt engineering in the time series modality.

Results The quantitative results of in-context forecasting are provided on the right of Figure 4. The results of zero-shot forecasting, where no downstream demonstration is available, are compared as the baseline. Benefiting from the time series prompts of the target domain, our LLM-based forecaster with the proposed in-context forecasting paradigm achieves consistent promotions on all M3 subsets and the averaged 13 . 3% SMAPE reduction compared with zero-shot forecasting. In contrast to previous LLM4TS methods that rely on deft language prompts, LLMs adopted by AutoTimes can be instructed by time series itself with our intuitive prompting engineering. From the perspective of the forecasting paradigm, we extend the prediction context beyond the lookback window. To inherit token transitions of language models parameterized by intermediate layers, AutoTimes takes a crucial step by establishing a mapping between time series segments and the latent space of language tokens, which is however absent in non-autoregressive LLM4TS methods. Therefore, ensuring autoregression consistency enhances the effective utilization of LLMs as foundation models.

4.4 Method Analysis

Generality Previous LLM4TS [15, 49] methods focus on applying their approach to specific LLMs. We demonstrate that AutoTimes is compatible with any decoder-only LLMs. By extensively training LLM-based forecasters by AutoTimes based on prevalent LLMs, including GPT-2 [31], OPT [46], and LLaMA [36], we present the results in Table 5, highlighting the generality of AutoTimes.

Scaling behavior Scalability is an essential characteristic that emerges from small models to large foundation models. By investigating the results presented in Table 5, we observe that the prediction accuracy of the forecaster generally improves with the increase in LLM parameters. This scaling behavior of LLM-based forecasters introduces a trade-off between performance and adaptation cost. To provide a comprehensive assessment, we evaluate each adapted forecaster from three perspectives: performance, training speed, and parameters, as presented in Figure 5. We observe that the largest LLaMA-7B consistently delivers optimal forecasting performance. As a relatively small language model, OPT-1.3B exhibits good parameter efficiency as an out-of-the-box forecaster.

MSE

ECL

GPT-2

OPT-250M

OPT-1.3B OPT-2.7B

OPT-6.7B

LLaMA-7B

Params Count

125M 1B

3B

7B

0

125

250

375

500

625

750

875

Training Time (ms/iter)

0.160

0.170

0.165

LLaMA-7B

OPT-6.7B

OPT-2.7B

OPT-1.3B

OPT-250M

GPT-2

Params Count

125M 1B

3B

7B

0

125

250

375

500

625

750

875

Training Time (ms/iter)

Traffic

0.380

0.390

0.400

0.370

MSE

Adaptation cost To mitigate the substantial cost of adapting large language models, AutoTimes introduces minimal parameters with all intermediate layers of LLM frozen. Additionally, we seamlessly integrate the language tokens (e.g. textual timestamps) without excessive context length and runtime overhead for training, thereby significantly reducing the adaptation cost. Figure 6 presents a comprehensive efficiency analysis with advanced LLM4TS methods: FPT is applicable on GPT-2 and TimeLLM is applicable on LLaMA-7B. Not only does AutoTime achieve better results in Table 3, but its training and reasoning time is also greatly reduced, bringing over 5 × speedup on average. In terms of parameter efficiency, AutoTimes focuses on establishing the embedding for time series segments, which is simply implemented by the MLP ( 0 . 79 M ) account for 0 . 1% parameters of the LLM ( 7 B ). Therefore, the results affirm the effectiveness of reutilizing the inherent token transition.

Ablation study Recent research has raised doubts about the validity of previous LLM4TS methods [35], which predominantly adopt non-autoregression, that is, treating the LLM as a BERT-style pre-trained backbone and utilize a globally flattening projector on all lookback tokens. Here, we provide a thorough ablation study to examine our proposed AutoTimes in Table 6. The results underscore that our method maintains the consistency of the decoder-only architecture and autoregressive inference, effectively leveraging LLMs and addressing the concerns regarding performance improvement and adaptation cost. Further, we provide a comparison by substituting our token-wise segment projection (consistent with LLMs) with the flatten linear head [26] (common in non-autoregressive forecasters). Results of Table 21 in the Appendix reveal that the performance of non-autoregressive generation is consistently inferior to that of our autoregressive AutoTimes approach.

LoRA adaptation By incorporating low-rank adaptation technique [14] on the intermediate LLM layers, the token transition of the large language model can be further fine-tuned to align the future extrapolation of time series. Table 7 provides the performance comparing the incorporation of LoRA, which consistently improves the performance of the LLM-based forecaster adapted by AutoTimes.

5 Conclusion

This paper aims to develop foundation models for time series forecasting. We utilize off-the-shelf LLMs as autoregressive forecasters by transferring the general-purpose and multi-step generation ability. Different from prior methods, we notice prevalent non-autoregressive LLM4TS methods may contradict the decoder-only structure and lead to insufficient utilization of LLMs. Experimentally, the proposed method achieves state-of-the-art performance with remarkable model efficiency. Further analysis reveals that our forecaster effectively inherits advanced capabilities such as zero-shot and in-context forecasting, and is able to utilize both instructive times series and timestamps. In the future, we will further incorporate advanced low-rank adaptation and utilize booming language backbones.

Acknowledgments

This work was supported by the Ministry of Industry and Information Technology of China.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems , 13, 2000.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

George Box. Box and jenkins: time series analysis, forecasting and control. In A Very British Affair: Six Britons and the Development of Time Series Analysis During the 20th Century , pages 161-215. Springer, 2013.

George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control . John Wiley & Sons, 2015.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Promptbased generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948 , 2023.

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 6989-6997, 2023.

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559 , 2022.

Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprint arXiv:2304.08424 , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

James Durbin and Siem Jan Koopman. Time series analysis by state space methods , volume 38. OUP Oxford, 2012.

Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820 , 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 , 2023.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485 , 2023.

Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. arXiv preprint arXiv:2310.09751 , 2023.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 , 2023.

Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. arXiv preprint arXiv:2305.18803 , 2023.

Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Transformers for time series analysis at scale. arXiv preprint arXiv:2402.02368 , 2024.

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting , 36(1):54-74, 2020.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 , 2022.

R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article , 2:13, 2023.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 , 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. Test: Text prototype aligned embedding to activate llm's ability for time series. arXiv preprint arXiv:2308.08241 , 2023.

Mingtian Tan, Mike A Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language models actually useful for time series forecasting? arXiv preprint arXiv:2406.16964 , 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning , pages 22964-22984. PMLR, 2022.

Peter R Winters. Forecasting sales by exponentially weighted moving averages. Management science , 6(3):324-342, 1960.

Gerald Woo, Chenghao Liu, Akshat Kumar, and Doyen Sahoo. Pushing the limits of pre-training for time series forecasting in the cloudops domain. arXiv preprint arXiv:2310.05063 , 2023.

Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592 , 2024.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186 , 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021.

Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering , 2023.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 , 2023.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained lm. arXiv preprint arXiv:2302.11939 , 2023.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision , pages 19-27, 2015.

A Dataset Descriptions

We conduct experiments to evaluate the performance of the proposed AutoTimes on seven real-world datasets spanning diverse domains: (1) ETTh1 [48] spans from July 2016 to July 2018 and consists of seven factors related to electricity transformers. (2) Weather [43] encompasses 21 meteorological factors collected every 10 minutes in 2020 from the Weather Station of the Max Planck Biogeochemistry Institute. (3) ECL [43] captures hourly electricity consumption data from 321 clients. (4) Traffic [43] gathers hourly road occupancy rates from 862 sensors on San Francisco Bay area freeways, covering the period from January 2015 to December 2016. (5) Solar-Energy [18] records solar power production from 137 PV plants in 2006, sampled every 10 minutes. (6) M4 is a competition dataset encompassing various time series across different frequencies and domains such as business and economics. (7) M3, albeit smaller than M4, also contains diverse time series from various domains.

We follow the same data processing and train-validation-test set split protocol used in TimesNet [43], where the train, validation, and test datasets are strictly divided according to chronological order to ensure no data leakage. As for long-term forecasting settings, we fix the context length of AutoTimes and the lookback length of other compared methods as 672 in ETT, ECL, Traffic, Weather, and Solar-Energy, and the forecast length varies in {96, 192, 336, 720}. For the short-term forecasting on M4 and M3 datasets, the input length is generally set to twice the output length according to the official implementation of TimesNet. The details are provided in Table 8.

B Implementation Details

AutoTimes processes timestamps in textual form rather than numerical encoding, potentially enabling to handle other textual data such as news or logs. We utilize LLM to obtain embedding for the special token <EOS> to capture embedding for the entire sentence. Pseudo-code for this process is depicted in Algorithm 1. It is worth noting that in the context of multivariate time series forecasting, timestamps are shared across variates. Thus, timestamps can implicitly express relationships between variates even with channel independence. Further, assuming there are C variates since the number of timestamps is 1 C of the total time point count, these embeddings can be efficiently pre-computed by large language models.

After obtaining embedding for the timestamps, we repurpose LLM for time series forecasting using Algorithm 2. At this stage, only the parameters of SegmentEmbedding and SegmentProjection are updated, while the

parameters of LLMs remain entirely frozen. During inference, AutoTimes utilizes the last token generated as its prediction and then employs this output to create subsequent predictions autoregressively. This approach enables AutoTimes to predict sequences of variable lengths with just one model dynamically. Such capability is crucial in real-world application scenarios. The pseudo-code in Algorithm 3-4 illustrates this process.

All the experiments are conducted using PyTorch [29] on NVIDIA A100 GPUs. We employ Adam [17] with an initial learning rate in { 10 -3 , 5 × 10 -4 , 10 -4 } and MSE loss for model optimization. We adopt Channel Independence [26] for multivariate time series and utilize our position embeddings of timestamps to explicitly align them. The batch size is chosen from { 256 , 1024 , 2048 } , and we set the number of training epochs as 10 . As for SegmentEmbedding and SegmentProjection , we implement them by either a linear layer or MLP. Results of deep forecaster are based on the benchmark provided by the TimesNet [43] repository, which is fairly built on the same configurations provided by the original paper. LLM4TS methods [15, 21, 49] are implemented by their official and open-source repository. Unless otherwise specified, we use LLaMA-7B [36] as the default base LLM. We also present the standard deviation of AutoTimes forecasting performance with three random seeds in Table 9, demonstrating that the performance of AutoTimes is stable.

Algorithm 1 AutoTimes - Generate Text Embedding

Algorithm 2 AutoTimes - Repurpose LLM

Algorithm 3 AutoTimes - LLM Forecasting

Algorithm 4 AutoTimes - Autoregressive Generation Require: Input time series { x 1 , . . . , x N 1 × S } ; textual embeddings { TE 1 , . . . , TE N 2 } ▷ Forecast token number N 2 -N 1 1: x = { x 1 , . . . , x N 1 × S } 2: prediction = {} 3: for i in { 1 , . . . , N 2 -N 1 } : 4: for ˆ y = LLMForecaster( x , TE : N 1 + i -1 ) ▷ Details in Algorithm 3 5: for x ←{ x , ˆ y } ▷ Concatenate for the input for next iteration 6: for prediction ←{ prediction , ˆ y } ▷ Record prediction results 7: Return prediction ▷ Return result ∈ R ( N 2 -N 1 ) × S

C Hyperparameter Sensitivity

SegmentEmbedding and SegmentProjection are uniformly implemented by MLP. The number of layers is fixed as 2 and the hidden dimension is selected from { 256 , 512 , 1024 } according to the validation loss. The segment length is set as S = 96 in multivariate datasets and is set as the prediction length S = F in M3 and M4. We verify the robustness of AutoTimes of hyperparameters as follows: the layer number and hidden dimension of SegmentEmbedding and SegmentProjection , context length, and segment length. We observe that AutoTimes is insensitive to the configurations of embedding and projection layers. Besides, performance can be improved by increasing the context length. For long prediction lengths, a larger segment length is favored.

D Supplementary Results

D.1 Time Series Forecasting

We compare the performance of AutoTimes with state-of-the-art LLM4TS methods and well-acknowledged deep forecasters. Table 11 shows detailed short-term forecast results on M4. Table 10 presents results of the one-for-all forecasting benchmark across ETTh1, ECL, Traffic, Weather, and Solar-Energy datasets. We evaluate all methods by rolling forecasting: each model is trained with input length 672 and output length 96 , and the predicted values are integrated as part of the input in the next iteration until reaching the desired forecast horizon.

Furthermore, the traditional one-for-one approach, where forecasters are trained individually for each prediction length, is also presented in Table 12. The results are reproduced using their corresponding official code. For the sake of rigor, we also provide our reproduced results with the officially reported results in Table 13.

Additionally, we evaluate AutoTimes along with other baseline models on recent benchmarks [24]. Results are presented in Table 15. We also look forward to evaluating on more diverse benchmarks in the future.

D.2 Zero-Shot Forecasting

Following the zero-shot forecasting of FPT [15], each experiment comprises the source and target datasets. We train a model on the source dataset and apply the model on the target dataset for predictions directly.

Notably, the zero-shot scenarios are conducted respectively on the subsets (e.g. M4 Monthly → M3 Monthly) and the subsets are divided by the sampling frequency but follow different distributions [25].

For M4 → M3, which means training on M4 and testing on M3, we directly utilize the same model in the short-term forecasting experiments reported in Table 11. Considering different horizons in subsets, for M3 Yearly, M3 Quarterly, and M3 Monthly, we directly employ models trained on corresponding subsets of M4 for testing. As for M3 Others, we test using the model trained on M4 Quarterly to keep the same horizon.

For M3 → M4, similarly, for M4 Yearly, M4 Quarterly, and M4 Monthly, we directly employ models trained on corresponding subsets of M3 for testing. For the remaining subsets, M4 Weekly, M4 Daily, and M4 Hourly, we perform inference using the model trained on M3 Monthly. Table 16 shows the detailed result.

D.3 Method Generality

Mainstream LLMs predominantly adopt the decoder-only architecture, and AutoTimes can utilize any decoderonly LLM. We conduct experiments on various types and sizes of LLMs, including GPT-2 [31], multiple sizes of OPT [46], and LLaMA [36]. Detailed configurations and results are shown in Table 17 and 18, demonstrating a general trend where performance improves as the model size increases, consistent with the scaling law [16].

D.4 Variable Lookback Length

In the conventional forecasting paradigm, deep forecasters are trained respectively on lookback/forecast lengths, limiting their applicability to a single lookback length. In contrast, LLMs have the versatility to handle various input lengths. This capability is derived from Rotary Position Embedding [33] and the next toke prediction objective, where LLMs are trained with token-wise supervision in Equation 8, that is, the generated token at each position is supervised. While non-autoregressive LLM4TS methods are typically constrained to a fixed lookback setting, AutoTimes with a consistent training objective has the flexibility to deal with different lookback lengths. We present the results in Figure 8, where we adapt the LLM by AutoTimes with the context length of C = 672 , and evaluate the performance with different lookback lengths, which demonstrates the inherited versatility of LLM-based forecasters. Moreover, the performance is generally improving with increased available lookback observations, leading to an averaged 9 . 3% promotion from 384 to 672 . By contrast, several works have observed that the performance of respectively trained deep forecasters does not necessarily improve with the increasing of lookback length [22, 26, 45].

D.5 Timestamps as Position Embeddings

We conduct an ablation study on the proposed position embeddings that integrate timestamps, a prevalent textual covariate in real-world applications. As shown in Figure 9, the forecasting performance is consistently promoted across all multivariate datasets and prediction lengths. The steady improvement can be attributed to the fact that timestamps demote the absolute position of time series segments on the timeline, explicitly aligning different variates in multivariate scenarios. The increasing promotion with longer prediction length also implies that chronological information, such as date and periodicity, yields significant benefits for long-term forecasting.

D.6 In-Context Forecasting

For in-context forecasting, similar to zero-shot forecasting in Appendix D.2, we train our model using the source dataset and directly evaluate it on the target dataset. In this task, we first choose M4 as the source dataset and M3 as the target dataset. It is important to note that the structure of the M3 and M4 datasets differs from typical datasets used for long-term forecasting. They consist of multiple univariate time sequences of different lengths. The final part of each sequence serves as the test set, while the preceding part is used for training.

Implementation In zero-shot scenarios, we use a sequence of length F preceding and consecutive to the test set as input, referred to as the lookback window, where F is the forecast length of each subset. During in-context forecasting, we concatenate the first 2 F time points that belong to the same sequence with the lookback window as input. We aim to enhance prediction performance by incorporating more contextual information. Too short sequences ( ≤ 4 F ) are discarded to prevent overlap between the prompt and the lookback window. For a fair comparison, both zero-shot and in-context forecasting performance are reported on the same remaining sequences. Figure 12 provides showcases of zero-shot and in-context forecasting.

Prompt engineering Regarding in-context learning, we further delve into the effect of different strategies to retrieve time series as prompts, which is provided in Table 19. P.1 and P.2 correspond to the zero-shot and in-context forecasting evaluated in Section 4.3. The prompt of P.3 contains the last 2 F time points preceding the beginning of the lookback window. Another retrieval of prompts as P.4 , adopts time series that come from another uncorrelated time series (out-of-series). We can obtain the following observations:

P.1 v.s. P.4 indicates that the selected prompt is not suitable, since the prompt does not come from the earlier observations of the same series to be predicted. Although the context window becomes larger, the averaged performance will deteriorate because of irrelevant prompts.

P.2 and P.3 indicates that in most cases, selecting the relevant 2 F time series from the same series can provide better contextual information.

This highlights the prompt engineering for in-context forecasting. An intuitive suggestion is to utilize consecutive, inter-periodic, and multiple prompts. To verify this idea, we analyze the periodic effect of time series prompts.

In previous experiments, we adopt M3 and M4 datasets, which are consistent with the zero-shot experiment of FPT [49], to present the promotion of our in-context paradigm. To provide more rigorous conclusions, we extend the evaluation to widely recognized datasets. Details of the experiment are as follows: By using a trained model checkpoint on a source domain (Traffic), we conduct forecasting without gradient update on target ETT datasets. We evaluate the pred96 performance on the last variate (OT). For the zero-shot scenario, the input is length288 lookback series. For in-context forecasting, the input is (length384 series prompt + length288 lookback series). Considering the dataset periodicity, the prompt is uniformly selected as the Ahead-24 (one-day-ahead) series of the original lookback series. To eliminate the performance boost that comes from extending the input length, we also provide the results of length672 lookback series in the zero-shot scenario. Moreover, we further delve into the effect of different strategies to select time series prompts:

Ahead-Period : The prompt is uniformly selected as the Ahead24 series of the original lookback series where 24 is one of the periods (daily period) of ETT.

Ahead-Random : The prompt is randomly selected as the previous series of the original series.

Fixed Prompt : The prompt is fixed as the first 384 time points in the variate-OT.

Other Variate : The prompt is uniformly selected as Ahead24 series, but comes from other variates.

Results in Table 20 demonstrate the effectiveness of using suitable time series prompts and highlight the influence of prompt engineering. Using inter-period prompts can outperform simply extending lookback window.

The benefit of the proposed in-context forecasting is to extend the input context of time series forecasting beyond a continuous lookback window. Since the essence of prompts is to incorporate useful domain-specific knowledge, here is one use case of in-context forecasting: Considering predicting the weather of one day, one approach is to extend the lookback length from days to weekends. However, it can also introduce noisy information since

non-stationary meteorological conditions can change with seasons. Another practical way is to consider how the weather changes on the same day in the last year (or years). Although the input is not continuous, the input context becomes more relevant based on prior knowledge about the periodicity (yearly). Therefore, in-context forecasting makes prior knowledge incorporatable and gets performance promotion.

D.7 Ablation Study

In addition to the ablation study of whether LLMs are useful in AutoTimes (Table 6), we further delve into the main difference between our method and previous LLM4TS approach and provide a comprehensive ablation study. The results presented in Table 21 demonstrate that the performance of non-autoregression projection is consistently inferior to that of our autoregressive AutoTimes approach.

E Showcases

To facilitate a clear comparison among various models, we present additional prediction showcases for long-term forecasting and short-term forecasting. These examples are provided by the following models: TimeLLM [15], FPT [49] and PatchTST [26]. Of all the models, AutoTimes delivers the most accurate future series predictions. Additionally, we provide the showcases of zero-shot and in-context forecasting in Figure 12.

AutoTimes

Time-LLM

FPT

PatchTST

AutoTimes

Time-LLM

FPT

PatchTST

F Broader Impact

F.1 Impact on Real-world Applications

This paper copes with general-purpose time series forecasting, which is faced with increasing challenges such as the versatility to handle variable-length scenarios, good generalizability with scarce samples, utilization of multimodality, and instructive downstream prompts. Since previous studies have demonstrated the feasibility of leveraging large language models for time series, we propose a simple but effective approach as AutoTimes to obtain LLM-based forecasters, which keeps the consistency of autoregression. Our model achieves state-ofthe-art performance on forecasting benchmarks and demonstrates remarkable adaptation speed and parameter efficiency. Besides, advanced capabilities such as multi-step generation and in-context learning are inherited by the repurposed forecaster. Therefore, the proposed method makes it promising to tackle real-world applications,

which helps our society prevent risks in advance and make better decisions with limited computational budgets. Our paper mainly focuses on scientific research and has no obvious negative social impact.

F.2 Impact on Future Research

In this paper, we find prevalent non-autoregressive LLM4TS methods have inconsistencies in the model structure and generative approach with LLMs, leading to insufficient utilization of the inherent multi-step token transition. Given that the generalizability and generative ability of LLMs are largely derived from the autoregressive manner, the potentials of LLMs may not be fully exhibited in time series forecasting. Therefore, we propose to adapt LLMs by the consistent training objective, the next token prediction, and accomplish arbitrary-length forecasting by iterative generation. Beyond the conventional forecasting paradigm, we propose in-context forecasting, where the context for prediction is extended, and earlier historical time series can be utilized as advantageous prompts. The compatibility with LLMs and insights from autoregression can be instructive for future LLM4TS research and the development of foundation time series models.

G Limitation

The proposed method has not supported probabilistic forecasting, since AutoTimes only establishes the mapping between time series segments to latent embeddings of the LLM, instead of discrete language tokens. Advanced low-rank adaptation is under exploration in our work, which can further align suitable token transitions as the future extrapolation of time series. More deftly designed embedding and projection layers are underexplored to support more compatible tokenization for time series. Besides, it is fascinating to apply AutoTimes on real-world multimodal time series datasets (such as news-stocks, and logs-measurements), which leaves our future work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to Section 1 of the main text, where the claims and contributions are included.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section G of Appendix, where we provide several aspects of limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section B in the main text and code in our public repository, including the detailed configurations of experiments and the scripts for reproduction.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to Section A of Appendix and code in our public repository, including dataset descriptions and how to access them.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section C and D of Appendix, where we state how hyperparameters are chosen and the detailed description of experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to Section B and Table 9 of Appendix, where we report standard deviations of results with three random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Section B of Appendix, where we provide sufficient information on the computing resource.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed and the reasearch conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to Section F of Appendix, where we discuss societal impacts and influences on future research.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All creators of datasets are properly credited by citations in Section A.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper argues that existing LLM based time series models have not fully exploited the inherent autoregressive property and the decoder-only architecture of LLMs. To address this problem, this paper introduces a novel AutoTimes model, which exploits the autoregressive property of LLMs. Experimental results demonstrate that AutoTimes could outperform the baseline methods.

STRENGTHS:
1. This paper exploits the autoregressive property of the LLMs. 
2. Comprehensive experiments are conducted to demonstrate the effectiveness of the proposed AutoTimes, including forecasting, zero-shot forecasting and in-context forecasting.
3. This paper is well-written, and is easy-to-follow.

WEAKNESSES:
1. Autoregression seems a little bit trivial.
2. It is not quite clear how much improvements can be brought by auto-regressively generating time series. Either direct comparison or theoretical analysis should be provided.
3. It is difficult to directly draw conclusions about the scaling behaviors from Table 5. The authors draw conclusions by comparing results of OPT-x models, rather than comparing different models e.g., compare OPT with LLaMA or GPT-2.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present AutoTimes, a method that utilizes large language models (LLMs) for time series forecasting. One of the key underexplored research topics addressed by the authors is the lack of models and pre-training mechanisms that result in foundation models capable of handling lookback and forecasting horizons of arbitrary length. This is achieved by adapting the LLM forecasting framework to autoregressively forecast time series segments. Furthermore, the paper outlines techniques such as in-context learning to further improve prediction performance. Compared to previous works, the methodology requires only a small number of trainable parameters compared to previous LLM fine-tuning techniques. As far as I am aware, this work is the first to be capable of handling multimodal input and producing an autoregressive forecast in the domain of LLMs and time series.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow. I appreciate how the authors clearly outlined their contributions. \n- The observation that non-autoregressive methods may contradict decoder-only LLMs and the shortcomings of prior methods is well-motivated. The solution proposed in the context of LLMs and time series is novel.\n- The introduction of a One-for-all benchmark, which involves prediction horizons transfer learning instead of dataset transfer learning, is innovative.\n- The authors provide strong motivation and experimental proof to support their claims.\n- The flexibility and scalability of their method are demonstrated by successfully swapping out different LLMs with varying numbers of parameters.\n- The continuous improvement over previous SOTA methods on multiple datasets, along with their thorough ablations, strengthens their claims and illustrates the method's flexibility.\n\nWEAKNESSES:\n- I'm not a fan of your chosen color scheme. While I appreciate that you try to include colors for different entities, it is too vivid (jump from Figure 3 to 4 to 5), making the figures tough to read, especially when skimming through them quickly (which most first readers will do initially). It is not a major objection, but improving this aspect could enhance your manuscript.\n- Typo:\n    - L21: etc [22, 42], missing dot.\n- The paper makes certain simplified assumptions, such as treating time series segments independently for embedding. This might overlook complex inter-dependencies present in real-world time series data. Addressing these inter-dependencies could enhance the robustness and applicability of the proposed method.\n\nQUESTIONS:\n1. **Claim of missing data for foundation models:** I am aware that there is not a plethora of datasets available, but did you look into [1] (datasets) or [2] (models)?\n\n2. While I see the autoregressive part as a great contribution, I am not entirely convinced that the proposed embedding scheme, which only fine-tunes a small portion, brings a competitive improvement. There are multiple works [4, 5] that do not require any fine-tuning of the LLMs.\n\n3. L106: \"Unlike previous works, our proposed method regards time series itself as the instructive prompt. It avoids the modality gap caused by concatenating time series and language tokens directly.\" This statement gives the impression that your approach is superior. I would be interested to see how your model compares to more of these learnable-prompt methods, especially how it compares to *Test* [4] performance-wise, which appears to be superior to Time-LLM.\n\n4. How does AutoTimes handle missing data or irregular time series intervals? Your claimed improvement over [6], which does not use language directly, has the disadvantage that you cannot simply overcome NaNs in the time series directly.\n\n5. \"Unlike previous works, our proposed method regards time series itself as the instructive prompt. It avoids the modality gap caused by concatenating time series and language tokens directly.\" I am a bit confused. What kind of model checkpoint are you using? Instruction tuned? An instructive prompt is a directive where the user provides explicit instructions to guide the response. I thought you were skipping the language level. Could you outline how you embed your time series and text and how exactly this is fed into the model?\n\n---\n\n[1] Goswami et al. \"MOMENT: A Family of Open Time-series Foundation Models.\" ICML 2024.\n\n[2] Ekambaram et al. \"Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series\" arXiv:2401.03955\n\n[3] Chang et al. \"LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters\" 2024\n\n[4] Sun et al. \"TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series\" ICLR 2024\n\n[5] Wang et al. \"Xinglei Wang, Meng Fang, Zichao Zeng, and Tao Cheng. Where Would I Go Next? Large Language Models as Human Mobility Predictors\" 2023\n\n[6] Gruver, Nate, et al. “Large Language Models Are Zero-Shot Time Series Forecasters.” _Advances in Neural Information Processing Systems_, 2023\n\nLIMITATIONS:\nWhile the paper provides a limitations section, I don't think it is sufficient to bury it in the appendix. During my first read, I thought the authors entirely skipped the limitations until I found it in the checklist.\n\nHere are some specific points that I believe are missing or need improvement:\n\n1. Although the paper claims that the embedding and projection layers only account for 0.1% of the total parameters, the scalability of these layers for very large datasets or extremely long time series is not thoroughly discussed. An evaluation of how the method performs with significantly larger datasets would strengthen the paper.\n\n2. While the authors mention that they leave out real-world datasets for future work, I think the approach was not tested on datasets with missing values. This is especially important as their tokenization scheme seems to be based on non-overlapping patches, which are known to lose their locality assumption when missing values occur. An analysis of how the model handles missing data would provide a more comprehensive evaluation of its robustness and applicability.\n\nIn addition to that, the authors answered broader societal impacts sufficiently.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a model named AutoTimes to repurpose LLMs for time series forecasting. Different from previous methods that use flattening and linear projection to get a prediction, this model repurposes LLMs in an autoregressive way, which is closer to the pre-training process of LLMs. Specifically, the main backbone of LLMs is frozen, and new patch embedding/output layers are added like in previous works. Absolute timestamps are embedded through LLMs to serve as position embeddings. Experiments show that the proposed model achieves SOTA performance and is more efficient.\n\nSTRENGTHS:\n1. The main body of the thesis is clear and easy to understand.\n\n2. Reproposing LLMs in an autoregressive way is intuitive and more reasonable than previous linear projection ones.\n\n3. Numerous experiments were conducted to demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. The most I concerned is whether this type of methods truly leverages the capabilities of the pre-trained LLMs. Please conduct the following ablation study: randomly initialize a large model, freeze its parameters, train it using the proposed method, and compare it with the pre-trained ones.\n\n2. The description of multimodal in Table 1 seems to be overselling, as the proposed model only uses texts for timestamps embedding and does not have capability to leverage natural languages.\n\n3. The in-context forecasting part is confusing: 1)What are the use cases for such a method? Table 16 shows that it's more effective to extend the lookback window. So what is the case that we can not get a longer lookback window but can get a window a long time ago? 2)Such a method can not even ensure the input series is continuous, so where does the promotion come from?\n\nQUESTIONS:\n1. Is LLM necessary for timestamp embedding? What if replacing it with ordinary nn.Embedding like Informer?\n\n2. How does the proposed model reduce the error accumulation of autoregressive models, given that there is no specifical design for this?\n\n3. Please report the number of learnable parameters in Table 5. Larger models have larger hidden states so that patch embedding/output layers for them have more learnable parameters. Therefore, it is uncertain whether the performance improvement comes from the scaling behavior of LLMs or from having more learnable parameters for tuning.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present in this paper an interesting approach where LLMs are leveraged to be fledged as time series forecasters. This proposed approach is based on freezing the LLM backbone to update a small amount of parameters to generate suitable time series embeddings which, together with time stamps as positional embeddings, provide suitable forecasts. Moreover, the authors provide an interesting notion of in-context forecasting, where the corresponding model is taught with examples how to forecast without gradient updates. Finally, the authors present numerical evaluations on diverse datasets.\n\nSTRENGTHS:\nThe authors provide an interesting view and current challenge in the field of time series forecasting: how to leverage LLMs for time series forecasting with minimal effort and minimal model modification. The authors provide an approach that basically freezes the LLM backbone to only update parameters related to a suitable embedding of time series, together with the usage of timestamps prompts as positional encoders. Moreover, the emphasise that the number of updated parameters is around 0.1% of those of the LLM backbone.\n\nThe authors provide an interesting paradigm that naturally arises from LLMs: how can we do forecaster fine-tuning without gradient updates. This brings in-context forecasting as a contribution of the authors, which seems to have potentially an interesting resourceful idea for the community.\n\nWEAKNESSES:\nPerhaps the main weakness comes from the evaluations. The main points that I would invite the authors to aggressively address are the following:\n- the amount of baselines used for comparison is very limited. It is completely understandable that the authors miss some baselines because the field is just moving extremely quickly. Nevertheless, the authors should at least cite a relevant fraction of these recent works, for instance :\n- - Woo et al, 2024: Unified Training of Universal Time Series Forecasting Transformers\n- - Ansari et al, 2024: Chronos: Learning the Language of Time Series\n- - Dooly et al, 2023: ForecastPFN: Synthetically-Trained Zero-Shot Forecasting\n- - Goswami et al 2024: Moment: A family of open time-series foundation models\n- The amount of datasets used is limited. It is understandable that the authors provide a limited amount of datasets for evaluations, as one of the main hindrances in the field is the lack of publicly available data (at least in comparison to other fields). Yet, one can see that other contributions have shared datasets and made them freely available. See for instance:\n- - Woo et al, 2024: Unified Training of Universal Time Series Forecasting Transformers\n- - - and the corresponding HF source: https://huggingface.co/datasets/Salesforce/lotsa_data\n- Evaluations of in-context forecasting are limited. Following the previous points, I think this is one of the most exciting points of the paper and the reader would appreciate to see more systematic evaluations and explorations of this notion. Currently there is only one evaluation which involves M3 and M4 datasets, perhaps following the approach presented in One-Fits-All paper.\n\nQUESTIONS:\n- In Section 3 the authors mention that they aim to forecast covariates as well. Is this correct? In general, one can assume that certain kind of covariates are available in the future, like timestamps or boolean variables indicating that something specific will happen in the future. But in general, I am not sure that the authors truly want to forecast covariates.\n\nLIMITATIONS:\nThe authors have devoted a section describing sensible limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}