Timer-XL: Long-Context Transformers for Unified Time Series Forecasting
Authors: Yong Liu
Guo Qin
Xiangdong Huang
Jianmin Wang
Mingsheng Long
Venue: ICLR 2025 Poster (2025)
Keywords: Time Series Forecasting
Transformer
Foundation Model
Primary area: learning on time series and dynamical systems
TL;DR: We introduce a generative Transformer for multi-dimensional time series, which achieves SOTA performance in supervised training and zero-shot forecasting.

Abstract: We present Timer-XL, a causal Transformer for unified time series forecasting. To uniformly predict multidimensional time series, we generalize next token prediction, predominantly adopted for 1D token sequences, to multivariate next token prediction. The paradigm formulates various forecasting tasks as a long-context prediction problem. We opt for decoder-only Transformers that capture causal dependencies from varying-length contexts for unified forecasting, making predictions on non-stationary univariate time series, multivariate series with complicated dynamics and correlations, as well as covariate-informed contexts that include exogenous variables. Technically, we propose a universal TimeAttention to capture fine-grained intra- and inter-series dependencies of flattened time series tokens (patches), which is further enhanced by deft position embedding for temporal causality and variable equivalence. Timer-XL achieves state-of-the-art performance across task-specific forecasting benchmarks through a unified approach. Based on large-scale pre-training, Timer-XL achieves state-of-the-art zero-shot performance, making it a promising architecture for pre-trained time series models. Code is available at this repository: https://github.com/thuml/Timer-XL.

Published as a conference paper at ICLR 2025
TIMER-XL: LONG-CONTEXI TRANSFORMERS FOR UNIFIED TIME SERIES FORECASTING
Yong Liu,' Guo Qin; Xiangdong Huang, Jianmin Wang, Mingsheng Long School of Software, BNRist, Tsinghua University, Beijing 100084, China { uycng221, qi qinguo24] @ma t tainghua edu. (huangxdong, 3 simwang, mi eeiggsenggettsinggee en
ABSTRACT
We present Timer-XL, causal Transformer for unified time series forecasting, To uniformly predict multidimensionall time series, we generalize next token predic- tion, predominantly adopted for 1D token sequences, multivariate next token prediction. The paradigm formulates various forecasting tasks as a long-co long-context prediction problem. We opt for decoder-only Transformers that capture causal dependencies from varying-length contexts for unified forecasting, making predic- tions on non-w -stationttny univariate time series, multivariate series with complicated dynamics and correlations, as well as covariate-informed contexts that include exogenous variables. Technically, we propose a universal TimeAttention to capture fine-grained intra- and inter-series dependencies of flattened time series tokens (patches), which is further enhanced by deft position embedding for temporal causality and variable equivalence. Timer-XI XL achieves bate-of-the--ee performance across task-specific forecasting benchmarks through a unified approach. Based on large-scale pre-training, Timer-LL achieves oftte-of-theaarr zero-shot performance, making it promising architecture for pre-trained time series models. Code is available this repository: https: com hhomm Cotuum/Timees /TTmmer------ XL
INTRODUCTION
Transformers have been extensively applied to time series forecasting, becoming the backbone of task-specific models (Zhou et (201 Wu al.] (02T) and pre-trained models (Das et al. 2023).
While the majority of prior works have focused on long-term forecasting, reliable predictions are made by considering endogenous variations and exogenous correlations in the context (Box 2013).
Besides, the context length of pre-trained Transformers determines the maximum input and output length during inference. Therefore, long-context Transformers are more versatile than shorter ones, facilitating long- sengeeeueecce and high-resolution generation (Yin et al. 2023 Wang et al. 2024a)
However, existing Transformers the time series field crucially encounter the context bottleneck. As shown in Figure [ unlike Transformers for natural language and vision that learn dependencies among thousands millions tokens (Kirillov al., 2023) OpenAl 2023). time-series Transformers typically operate around limited contexts of up hundreds of time series tokens (patches) (Nie) al. 2022). For univariate forecasting, a short-context input leads to insufficient learning of global tendencies, struggling to address non-stationarity in real-world time series (Hyndman 2018). For multivariate forecasting, increasing research has demonstrated the effectiveness of explicitly capturing intra- and inter-channel dependencies (Zhang & Yan 2022 Liu et al. 2023 2024a). highlighting the practical urgency of extending the context length to encompass cnter-correlated time series.
Recently, causal Transformers characterized by the decoder-only architecture have become a pre- dominant choice of large language models (Zhao et al., (023) and garnered increasing attention in the development of large time series models (Rasul al. [2023] Ansari al.) 2024). Based on contextual flexibility and autoregressive next token prediction, one model can accommodate varying lookback and prediction lengths (Liu al. 2024b). Therefore, pre-training on longer contexts not only empowers them with the fundamental capability to incorporate more contextual information but
"Equal Contribution

---

Published as a conference paper at ICLR 2025
W
Languege Mickls
Unifled Time Series Forecasting
Vieion Mediil
(0) Univeriate
Midichs
Past Future
friend, how many apples have A: The 7
Nondatiooary
Cont-Context Transformer
Yeurly Corteal
077
(b) Mulivoriate
OrafonAll
Forecaster
J
langer
Controt
brdogesous Conted
Forgreem
Tokens of Modality
# Tokens of Context
We compare the context length (measured by token number) of Transformers in different modalities and propose Timer-XL that increases the length to thousands of patch tokens. Given the generality across contexts, Timer-XL XL is a versatile solution for various forecasting tasks.
Figure I:
also enhances the model versatility toward one-for-all foundation model. Regarding any-variate and any-length time series as one context, previous work (Liu et al. 2024a) has achieved unified modeling on flattened tokens based on noncausal Transformers. However, our empirical results (Figure reveal that encoder-only forecasters may encounter performance degradation in long-context forecasting, while decoder-only Transformers can mitigate this degradation well.
In this work, we generalize the training objective of language modeling to mullivariate next token prediction, achieving unified time series forecasting that covers tasks in Figure | (right). Based on the decoder-only architecture, we propose TimeAttention to facilitate Transformers on multidimensional time series, presenting Kronecker-basee masking mechanism to train time-series Transformers in a channel-dependen approach. With specialized position embedding for multivariate series, Tttentten- (Zaheer
tion is aware of the chronological order of time points and achieves permutation--quivalernn al. 2017) variables. We enlarge the context to thousands of patch tokens and achieve state-of- the-art on univariate, multivariate, and covariate-infone on large-scale datasets, we present Timer-XL as extra long version of pre- pre-trained time-series Trans- formers (Timer) (Liu et al. 2024c), which outperforms recent large models in zero-shot forecasting. Our contributions lie in three aspects:
forecasting benchmarks. By pre-training
We propose multivariate next token prediction and unified time series forecasting, strength- ening Transformers with enlarged contexts to make information-commlltt predictions.
We introduce Timenttention, a novel causal self-attention tailored for multidimensional time series, facilitating intra- and inter-series modeling with positional awareness and maintaining causality and scalability of Transformers.
We propose Timer-XL. a versatile Transformer for one-for-all forecasting, which mitigates performance degradation in long-context time series, achieves state-of-the-art performance in task-specific benchmarks, and presents notable zero-shot performance by pre-training.
2 RELATED WORK
Transformers Vaswani al. 2017) for time series forecasting have undergone rapid advancements. Initial Transformer-busee forecasters primarily focused on long-term forecasting (Li et al. 2019 Zhou et al. 2021 Wu al. 202T, Sun Zhang. 2024). However, the context length not growing in pace, which hinders Transformers from making information-complet predictions. Another advancement has focused on multivariate forecasting. Unlike natural language, time series are multidimensional and inherently correlated (Hyndman 2018). To learn intra- and inter-series dependencies, different tokenization of time-series Transformers has been proposed. including point-wise (Lim et al. 2021)) ). patch-wise (Nie et al., (2022), and variable- wise (Liu et al.) (003) approaches, with deftly tailored architectures (Zhang & Yan [2022; Wang et al.) 2024b). However, few works highlight that multidi- mensional time series be uniformly tackled by long-context Transformers without architectural
2

---

Published as a conference paper at ICLR 2025
modification. In this work, we leverage causal Transformers, which excel at handling long- long-context sequences, and unify time series forecasting tasks into multivariate next token prediction.
Recently, time-series Transformers have experienced the evolution from small task-specific models to pre-trained large models (Das et al. 2023; Woo et al. 2024, Ansari et al. 2024). Among them, decoder-only Transformer is predominantly adopted as the backbone of large language models (Tou- vron et al. 2023 OpenAl 2023), positioning as a scalable choice for general time series analysis (Liu et at... ( By independently predicting each token with supervision, decoder-only models are also multi-length forecasters (Liu et al. 2024b). avoiding resource-intensive training and lookback-search.
However, existing decoder-only Transformers are generally pre-trained in channel -indeeedene approach, making them inaccessible to inter-series dependencies.
Prior work has employed encoder-only Transformers capture dependencies of multivariate time series (Liu et al., 2024a). However, our empirical study found that this architecture can be incom- patible with causal forecasting, limiting the performance of Transformers. To implement next token prediction and multivariate forecasting in single Transformer, we renovate the attention module, which disentangles fine-grained token dependencies into variable dependencies and temporal causal masks, capturing intra- and inter-series dependencies with causality and scalability maintained. In Table] we list representative time-series Transformers and highlight their differences.
Table I: Comparison among representative time-series Transformers.
PatchTST
iTrans. 2023
TimeXer 2024b
UniTST 2024a
Moirai 2024
Timer
Timer-XL (Ours)
Model
2022
2024c
Intra-Series
Inter-Series
X
Causal Trm.
X
X
Pre-Trained
X
X
APPROACH
In this section, we first introduce decoder-only Transformer to illustrate the procedure of next token prediction on univariate time series. an extension, we design TimeArtention and propose Timer-XL. for unified time series forecasting. It is applicable to univariate, multivariate, and covariate-informed scenarios by generalizing the context from ID sequences to 2D time series.
3.1
TIMER
Timer (Liu et al. 2024c) is time-series Transformer trained by next token prediction (Bengio et al. 2000), which regards single-dimensional time series as non-overlapping patch tokens.
Next Token Prediction Given an univariate time series X = {x1, ... ... .Tr) of length TP, time series token defined as consccutive time points, also termed as the patch token:
x; = ( I.P] R², i = I....,T.
(I)
The training objective independently predict the next patch token maximize the likelihood:
P(X) II 1(xx (xg)),
(2)
which is realized by decoder-only architecture with the block number L and model dimension D:
h W.x; i= 1,
H' TrmBlock H -111)), 1 1- 1 1...., L.
( W' I.....T. T.
For simplicity, we omit the block index 1. Timer adopts W.. Wa E Dx that independently embed and project the token embeddings as H {h,} R TrmBlock includes feed-forward network and self-attention with the temporal causal mask T E TTX h, E R the context representation of the previous i tokens. All predicted x;41 supervised with ground truth via MSE loss.
3

---

Published as a conference paper at ICLR 2025
3.2 GENERALIZE ID SEQUENCES TO 2D TIME SERIES
For the enlarged context with the additional dimension, our proposed attention mechanism aims to (1) thoroughly capture intra- and inter-series dependencies and (2) preserve causality within the temporal dimension. Without loss of generality, we illustrate this with the case of multivariate forecasting.
Multivariate Next Token Prediction Given a multivariate time series X R with the number of variables N, the time series token Xm,; is defined as the i-th patch of the m-th variable:
{Xm, (1m1(1))441....... Xm,ip] E "R,,, m I,..., NN, i = 1...., T. The training objective is still to independently predict the next token. Unlike before, each prediction is made based on tokens of the previous time ($ i) from all N variables:
Xm,i
(4)
N
T
N
T
P(X)
II
p X0 (())
II
p (m,.++ X1,5o... XN.S))-
(5)
m==
m=1 1=1
Compared with Equation 2 the multivariate context length increases from T to NT. By contrast, the benefit is that this paradigm learns causal dependencies within each sequence while incorporating exogenous variable correlations from other sequences, making it universal forecasting paradigm that outperforms channel- -ineeedenn (Nie et al.) 2022 or variable-centric models (Liu et al. 2023).
Technically, we independently apply W. RDx each token to obtain patch-wise representation hm, E RD,, which will encompass contextual information from Ni tokens through Transformer blocks and be eventually projected by Wa E PxP into the predicted patch token Xm,i-1-
Position Embedding Position embedding has not been sufficiently explored in time-series Trans- -itiatttioonnivaiaannc self-attention, positional embedding is required to reflect the chronological order of tokens the temporal dimension. As for the variable dimension, shuffling the input order of variables should not affect anything other than the output order of variables. Formally, the processing on multiple variables should be permutation-eqquialenn (Zaheer al.) 2017).
formers. To avoid inherent
2024), a widely utilized position embed- To meet the above requirements, we adopt RoPE (Su et al.
ding on the temporal dimension. For the variable dimension, we two learnable scalars in each head to keep the crrmutation-qquivallee them together, we provide detailed ablations Section E.3 to demonstrate the effectiveness:
of variables (Woo et al. (024). Beyond simply incorporating
Amn.ij Aj h W WoRRoss) W hn.j
1(m n) + uu. 1(m # n),
(6)
where Wg. WK. W, Dxde and dę is the dimension of the query, key, and value. RØ.: dd is the rotary matrix with rotation degree t 0, 1-)) is the indicator function, and u, E R are Icarnable parameters for the token to distinguish its endogenous and exogenous time series.
TiteAttention In contrast to variable-wise (Liu et al., 2023) and non-causal patch-wise tokens (Nie al., 2022, Woo et al. 2024). our TimeAttention aims to capture causal patch- wise dependencies within and among all variables. Concretely, we sort patch tokens by flattening their 2D indices into manner, which is illustrated in the upper left of Figure 2 Note that the order of variables does not matter, since Equation guarantees their
ID indices in the temporal
We provide an intuitive example to illustrate the causal dependencies within multivariate time series: considering the 2nd token of time series A. To predict its next token, its representation h should be exactly dependent on the tokens-(1,, 2, 4, 5..). Similarly, we provide all causal dependencies of each token in Figure| 12 Based on the visualized attention mask and variable dependencies presented in Figure? 2] where all variables are inter-correlated, all token dependencies in A can be formally disentangled by the Kronecker product into (1) the adjacency matrix of the variable dependency graph C E R.N X and (2) the causal temporal mask T E R
[ i TiJ = { otherwise, Cm., = {
if variable m is dependent on n, otherwise.
Let the Kronecker product @ (RNVXN, RTXT) + NNNT take two matrices and produce a block matrix. Consequently, TiteAttention is formulated follows:
TimeAttention(F H)- Softmax [Mask(C & T) + A
HW., Mask(M) - {(0
Mf, 1,
(8)
Vdz
if if
4

---

Published as a conference paper at ICLR 2025
Time Series
Variable Dependency
Multivariate
with Covariate B
A-Tokens
Time Series
Variable
B-Tolers
Dependency
Univariate
Depandencies
Depondencies
Masked Token
Tokens
Unmasked Token
0
DDDeendencies
Illustration of Attentitn For univariate series, temporal mask T keeps the causality. Given multivariate patch tokens sorted in a temporal-first order, we adopt the variable dependencies C, an all-one matrix, as the left-operand of Kronecker product, expanding temporal mask to block matrix, which exactly reflects dependencies of multivariate next token prediction. The formulation is also generalizable to univariate and covariate-informed contexts with pre-defined variable dependency.
Figure 2:
Eventually, token representations in H (bm,i) E NTxD will be independently processed by feed-forward network and layer normalization, and fed into the next Transformer block.
Unified Time Series Forecasting In multivariate forecasting, the variable dependency forms the complete graph, presenting all-one matrix C. By generalizing TimeAttentin on multiple sequences, Transformers can leverage its fegth--lexibiiiity to encompass relevant covariates as well. In this case, Timer-XL adapted in two steps: (1) formulate the customized variable dependency as and (2) optimize the model using the supervision of target variables. An example (target-A-con A-covariito--e- BB) of Aitenttention is illustrated on the right of Figure 2 In a nutshell, we adopt position embeddings for the temporal and variable dimensions. To achieve unified time series forecasting, we flatten 2D time series into a unified context and capture fine me--grined causal token dependencies.
4 EXPERIMENTS
We conduct evaluations of Timer-XL in three aspects, including (1) supervised training as a task- specific forecaster, (2) large-scale pre-training as a zero-shot forecaster, and (3) assessing the effec- tiveness of TimeAttennion and model efficiency. Given that the long-context forecasting paradigm receives less attention in the community. which can be concealed due to the performance saturation on previous benchmarks (Makridakis et al. 20200] Wu et al.] (022), we established new long-context forecasting benchmarks. Detailed experimental configurations are provided in Appendix B
4.1 UNIVARIATE TIME SERIES FORECASTING
Setups Due to the insufficient dataset length when extending contexts in univariate datasets (Makri) dakis et al. 2220)), we adopt multivariate datasets from Liu et al. (2023). Although these datasets are originally multivariate, they aim to be predicted in a univariate approach with the implementation of channel independence. Different from the previous long-tern term forecasting setting, we focus on reliable prediction based on a long context. Therefore, we fix the prediction horizon and increase the lookback length to monthly and yearly levels. We also establish a long-context univariate benchmark based on the challenging 40-year ECMWF Reanalysis v5 dataset (Hersbach et al. 2020), where yearly contexts adopted to predict the land-surface temperature of single site (ERA5-S).
Results As shown in Figure the accuracy of univariate prediction generally be improved by extending the daily context to monthly. We draw a similar conclusion on ERAS (Table (5). where extending the context consistently helps in the specific model architecture. Notably, Timer-XI. with
5

---

Published as a conference paper at ICLR 2025
ETThi
FCL
0.42
Day
6.4
Ale
Weok
<Macch
Tiar
Traffie
PEMSO3
Raal757
1M0ch
LWWre
Figure 3: Univariate forecasting (pred-96) of well-acknowledger benchmarks under channel indepen- dence (Nie al. 2022). We increase the lookback length encompass monthly and yearly contexts.
decoder-only architecture outperforms encoder-only Transformer and linear forecaster in excessively long contexts. Further, we conduct representation analysis in Appendix E.4 revealing that Timer-Xi XL is proficient at adaptively selecting information in vast observations and thus achieves breakthrough performance. It is also noteworthy that the performance of monthly and yearly contexts improves slowly and deteriorates, which may stem from increased noise and training difficulty inherent in data, which leaves a future direction to improve the context efficiency. Table 2 provides results on ERA5-S. Timer-X XL consistently outperforms PatchTST on all sites, which can be credited to the maintenance of causality and token-wisg wise supervision in the decoder-only architecture.
Non-stationary Forecasting We delve into widespread non-stationarity in univariate tasks. is commonly tackled by normalization (Kim et al. 2021) that greatly improves Transformer performance previous benchmarks. However, we find it may be caused by the insufficient time span and training samples in these datasets. While normalization simplifies learning by aligning series with different means and variances to the same distribution, it limits the model capacity Transformers, preventing them from learning variations among windows. The by-product can be mode collapse and oversmooth predictions. Table and Table[16| we evaluate the performance on ERA5 and datasets from Wu et al.) (2022). which validates that Timer-XI XI. can achieve better results without instance normalization.
Table 2: Univariate forecasting mouut-3772-prd---a of ERA5-S, encompassing 117k time points in each station (40-years). We evaluate PStchTST and Timer-XL with and without normalization (Kim ct al.
2021 Norm. indicates using the normalization. We train one model for each site separately.
Station
Beijing
Hongkong
London
New York
Paris
Senul
Shanghai
Average
Model
MSE MAE MSE MAE] MSE MAE] MSE MAE MSE MAEJ MSE
MAE MSE MAE
MSE MAE
PatchTST
0.0791 0.221 0.189 0.327 0.277 0.415 0,186 0.334 0.266 0.407 0.0940
0.238
0.137 0.289
0.175 0.319 0.6 0.319
Norm.
0.0797 0.220 0.1 0.323
0.281
0.419 0.184 0.334 0.272 0.411
0.0914
0.233
0.136 0.287
Timer-XL.
0.0739 0.210 0.179 0.316 0.262
0.404
082
0.327 0.254 0.399
0.0901
0.229
0.134
0.282
0.168 0.310
+ Norm.
0.0742 0.210 0.183 0.317 0.278 0.418
0.181 0.330 0.264 0.407
0.0896
0.227
0.133 0.281
0.172 0.313
4.2 MULTIVARIATE TIME SERIES FORECASTING
Setups We follow iTransformer (Liu al. 2023) evaluate multivariate forecasting performance. Toward one-for-all forecaster, we evaluate performance of rolling forecast, that is, we trained one model for all prediction horizons by integrating the previous prediction into the lookback window in the next iteration. We further establish long-context multivariate forecasting benchmarks: ERAS multi-station land-surface temperature prediction (ERAS-MS), and the global temperature and wind speed forecasting challenge (GTWSF) (Wu et al. 2023 to learn complex temporal dynamics and variable correlations with sufficient training samples.
As shown in Tables 4 and Figure Timer-Xt XL achieves the best results on both previous and new benchmarks. Essentially. Transformers that explicitly capture inter-series dependencies, such as UniTST (Liu et al., (2024a) and iTranformmer reasonably achieve decent performance Table 3 Beyond (Transformer, Timer-XI. can model fine-grained patch-wise temporal dependencies. With
Results

---

Published as a conference paper at ICLR 2025
TimeAAttentinn, Timer-XL XL outperforms Timer especially on high-dimensional time series (13.2% in ECL and 6.3% in Traffic, with thousands of tokens in the context). Compared with the encoder-only UniTST, decoder-only Transformers excel at generalizing across varying prediction lengths in Table 4
Statistics-bssed
Numerical Simulation
Deep Models
Ours
9.691
9.995
MSE (Global Temp.)
MSE (Global Wind]
13.241
11,26
9.706
9,203
976
7291 7602 7,56 7,122
4.100
5-248
4124
4,614
3,009 865 3,077 3.786
Halh Woars
Proplar
Gosr
D8A3
Deep
HAEATS
SteeChay
Pyraennee
Conformer
UnTaT
Timor XL
HoleWiees
Propiet
COST
6Fg
DRA3
Deap AB
H4EATS
Conformet Pyrdorme
Uni UnIsT
TimerX1
Figure 4: Multivariate forecasting of GTWSF (2-day-pred -edd--red------ --ay), involving 3850 worldwide stations
spanning two years. Results of the baseline models are officially reported by Ding et al. 2024).
Table 3:
Multivariate forecasting (96- (06-pred-96) of trained from scratch. Results of baseline models are officially reported by Liu al. (2023
benchmarks. All models are
Timer.XI.
UNITST
iTransfommer
DLinear
PatchTST
TimesNet
Stationary
Autoformer
Models
(Ours)
2024c
20248
2023
2023
2112
2022
2022h
20217
Metric
MSS MAE MSE MAE]
MSE
MAE
MSE
MAE|
MSE
MAE
MSB MAE]
MSE
MAE] MSB MAE]
MSE MAE
ECI.
10.138
0.233
151199 0.244
[0.1 0.235
00.188 0.240 10.197
0.282
0.270 0.168 0.272 0.169 0.273 0.201 0.317
ETThl
[0,381
0.399|
(0.386
0.40| |0.385 0.402
(0.386 0,405 (0.386 0,400 0.414 0.419 0,384 0,402 0,402|0.51 0.513 0.491 (0.499 0.459
Traffic
00.387
0.260
[0.413
0.265
[0.389 0.265
(0.395
0.268
[0.650 0.396 0.462 0.295
0.593
0.321|0.612 [06622 0.338 00613 0.388
0.321
Weather
00.165
0.209
j0.176
0.215
00.15
(0.665
0.210
(0.174 0.214 10.196 0,255
0.177 0.218
0.172
0.220|0.17] 0,223 002666 0.336
Solar Energy 0.200 0.229 |0.204 0.234 |0.003 0.232 (0.203 0.237 [0.290 0.378 0.234 0,286 (0.250 0.292 0.292| 0.21. 0.249 j0.884 0.711
Table 4: Multivariate forecasting (672-pred-(99) 192, 336, 720) of well-acknowledded benchmarks. We evaluate one-for-all forecasters following Liu et al. (2024b: rolling forecasting for four forecast lengths with one model. Averaged results are reported here and full results are provided in Table
TImer-XI. (Ours)
UniTST
iTrammformer
DI DLinear
PatchTST
TimesNet
Stationary
Autoformer
Models
2024c
20248
2023
2023
2022
2022
20226
2021
Metric
MSE MAE|
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
ECL
|0.155
0.246|
1.161
0.251
(0.163
0.257
0.164 0.258
16 0.265 0.169 0.268 0.2011 0.303
J.266 0.358
0.289 0.379
ETTh1
|0.409
0.430
|p.418
0.436
[0.429 0.447
[0.421 0.445
(0.426 0.444 0.412 0.435 00.495
0.491
|0.505 0.513
|0.517 0.528
Traffic
|0.374
0.255
0.384
0.259
(0.385
0.265
[0.384 0.274
00.423
0.298 0.391 0.275 0.602
0.322
0.630
0.347 0.684 0.433
Weather
|.2240
0.273
00.23
|02232
0.270
[0.231
0.272
(0.266
0.291
(0.239
0.291
0.226
0.268
(0.264
0.293
00.308
0.329
(0.435 0.455
Solar-Energy
00.998 0.249 Jo.233
0.249
(0.241 0.275 (0.213 0.291
|0.222 0.283 0.202 0.269 00.13 0.295|0.255 0.315
0.315|0.265 (0.665 0.325
Ablation Study Patching (Nie et al. 2022) has been demonstrated as an effective tokenization approach for time series, leading to the boom of Transformers in supervised deep forecasters and large time series models. To better cope with multivariate time series forecasting, we compared typical models on real- world benchmarks address key questions: (1) whether to conduct explicit inter- series modeling or not (channel independence) and (2) whether to use decoder-only or encoder-onke Transformers. The combination presents four Transformers in Table which shows that Timer-XL combines the advantages of explicit inter-series modeling and the decoder-only architecture, which is suitable for multivariate time series forecasting with sufficient training samples.
4.3 COVARIATE- TIME SERIES FORECASTING
Setups For the covariate-infome forecasting, adopt the well-acknowledgee electricity price forecasting (EPF) task (Lago et al.) 202I). Each subset contains electricity price as the endogenous variable and two exogenous variables. Therefore, the variable dependency for Ximer-XIBi is formulated
7

---

Published as a conference paper at ICLR 2025
Table 5: Multivariate forecasting (input-3072-pred-99) of ERA5-MS (40 years and 7 stations). We fairly evaluate Transformers that adopt patched time series. CI. indicates whether the Transformer uses channel independence (Nie et al. (022). Arch. categorizes them into the encoder-only (E) and we train one model for all sites.
decoder-only (D) architectures. Different from ERA5-S in Table 2
Station
Beijing
Hongkong
London
New York
Puris
Scoul
Shanghai
Average
Model
[CL,
Arch.
MSE MAE] MSE MAB| MSE MAEJ
MAEJMSE MSSE
MAE|
MSE MAB]
MSE
MAE]
MSE MAE
MSE MAE
PatchTST Yes E [0.0815 0.222
0.222|0.110 0.190 0.326 00.255 0.414 (0.185 0.333 0.265 0.407 |0.0977 0.240
0.240|0.139 0.139 0.290 0.176 0.319
UniTST |No E [0.0753
0.213/0.179 0.318 (0.269 0.410 0.410|0.185 10.185 0.330 (0.25 0.401 |0.0901 0.230|0.135 0.284 00.170 0.312
Timer
Yes D
[0,0734
0.210/0.182 0.319 (0.268 0.407 [0.183 0.329 (0.255 0.39970.0877 0.226|0.132 0.281 00.169 0.310
Timer-XL. No
[0.0736 0.209|0.115 0.114 0.309 (0.263 0.404 0.404]0.188 10.182 0.327 00.252 0.396/0.009) 0.225 0.130 0.278 0.166 0.307
as C = [[I, 1, 1), [o, 1,0,, [0, 0, I]I. To investigate whether to learn causal or noncausal patch-wise dependencies in covariates, we implement two versions of Timer-XL: the original one with temporal causal mask T, and the noncausal one with T replaced by an all-one matrix.
Results As shown in Table Timer-XL XL outperforms of-te-of-the-an models in covariate-informed tasks. Compared with TimeXer (Wang et al. 2024b). which treats an entire covariate as a token, Timer-XI. learns fine-grained patch-wise dependencies. By the noncausal version of Timer-XI,, we surprisingly find consistent conclusions with endogenous variables: results will be better if Timer-XL learns causal dependencies within exogenous variables. It again validates that next token prediction that maintains causality has higher upper limit of performance.
Table 6: Covariate-
forecasting (168- pred-24) 24) of EPF. We implement two versions of Timer- XL: Noncausal indicates that we do not maintain the causality within covariates by replacing temporal causal mask with all-one matrix. Results of baselines are officially reported by Wang et al. 2
Timer-XI. (Ours)
Timer. XI. (Noncausal)
TimeXer 20224h
iTransformer
DLinear
PatchTST
Crossformer
TimesNet
Autoformer
Models
2023
2023
2022
[202]]
2022
[202]]
Metric
MSE
MAE
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE MAE
NP
0.234
0.262
1.237
0. 26.2661
0.238
0.268
(0.665
0.300
(0.309
0.321
|02667
0.284
1.245
0.289
(0.250
0.289
(0.402 0.398
PJM
0.089
0.187
0.092
0. 0.188
0.008
0.188
|0.097
0.197
10.108
0.215
10.006 0.209
0.198
(0.097 0.195 |0.168 0.267
BE
0.331
0.243
0..40
0.279|
0.379
0.243
|0.394
0.270
[0.463
0.313
|0.403
0.264
|0.436
0.294 (0.419 0.288
|0.500 0.333
FR
00.381
0.204]0.406
0.220
0.384
0.208
[0.439
0.233
|0.429 0.260
0.411 0.260
0.220
004400
0.216 |0.431 0.234 (0.519 0.295
DE
00.434
0.415
10.45
0.415
0.440
0.418
(0.499
0.443
|0520
0.463
|0.461 0.432
0.432|0.540
00540
0.423
|0.502
0.446
(0.674 0.544
Average
0.330
0.262 00316 0.273|
0.273/0.306 0.306 0.265 |0.335 0.289 J0.366 0.314 0.282
0.28210.362 |03322 0.284
|0.340 0.290
(0.453 0.368
4.4 PRE-TRAINED TIME-SERIES TRANSFORMERS
Setups Pre-training enriches time-series Transformers with generalizable forecasting capabilities. The outcome large time series model can cope with widespread challenges of few-shot and zero-shot forecasting. In this section, we conduct univariate pre-training on UTSD (Liu et al. 2024c) and LOTSA (Woo et al. 2024) and evaluate zero-shot performance on benchmarks from Wu et al. (2022). We further conduct large-scale multivariate pre-training on our ERAS-Large dataset, which spans 40 years and encompasses 4920 stations. Subsequently, we evaluate three types of generalization results comparing PatchTST (encoder-only Transformer) and Timer-XL (decoder-only Transformer): pre-training on 80% stations and 80% time span and then forecast on the remaining stations (variable generalization). remaining time span (temporal generalization). and remaining split of time span and stations (variable and temporal generalization). To evaluate the benefit of pre-training with longer context, we compare the zero-shot performance of Timer (202cc) and Timer-XI. XL. where the context length of pre-training is increased from 1440 to 2880.
Results We compare generalization performance on ERAS-Large in the middle of Figure 5 (a). Timer-XL XL achieves better results than PatchTST in all cases, revealing that decoder-only architecture has stronger generalization capability. Figure 5 (b) compares zero-shot performance of two pre- trained Transformers with different context lengths, where Timer-XI. outperforms previous Timer on
8

---

Published as a conference paper at ICLR 2025
all benchmark datasets, validating that long- cong-context pre-training enhances large time series models. In Table 7 we provide a comprehensive zero-shot evaluation under a comparable pre-traing vee-triniin scale and model size, where Timer-XL achieves notable performance with better sample efficiency. The versatility and scalability make it a promising backbone of foundation models.
Variables
(a) ERAS (Pred-96, 4920 Stations)
(b) TSLib (Pred-96, UTSD Pre- Pre-Trained)
One-for-All
Generalization
Titasr XL
Thow
Thear
25.29
12.36
Time
12-
12.16
3.344
Unified
Contest
Datasets
Tempori LT)
(atalle
T+V
11: Brn2 sTial ETVed
Trfiic Warder
Figure 5: Illustration of one-for-all generalization (left). Based the contextual flexibilityy Timer-XL can predict heterogeneous time series, indicating three directions of generalization shown on the left. We compare performance when generalizing across the time and variables (middle), and zero-shot results across datasets (right), emphasizing the benefit of long-contex. pre-training.
Table 7: Averaged results of zero-shot forecasting. A lower MSE or MAE indicates a better prediction. Corresponding prediction lengths include {96, 192, 336, 720). Full results of all prediction lengths provided in Table 13 1s Count represents the number of wins achieved by a model under all prediction lengths and datasets. The detailed configuration of Timer-Xi Mimer-XI.aae Base is provided in Table
Models
Timer-Xl.a. (Ours)
Time-MoEasor
Time-MoFas (2024]
Moiraisue
Moirais.
Moirni..o
TimesMM
MOMENT
Chromess..
Chromoe
2024
2024)
2024
2024
X024
2025
2024
Mctric MSE MAR
MSE
MSE
MAE
MSE MAE MSE
MAE MAE
MSE
MAE
MSE
ETTml
[0.323
0.392
(0.394
0.415
[0.376 0.405
(0.356
0.391
[0.436
4006 0.385 (1.422 a
(0.433 0.418]0.670 0.536
(0.645
0,500
(0.558
0.465
ETTm2
(0.233
0.336 (0.317 0,365
(0.316
0,361
|0.288 0.344
J0.307
0,347 |u.311 3239 w. 0,343/o 328 0.34600.316 0.365 (0.310 0,350
J0.295 0.338
ETTh]
|0,404
0.417 [0,400
0.424
[0.394
0.119
J0.412
0.426
J0.428
0.427
[0.417
0.419/0.480
0.439j0.473 0.443|0.683 0.566 (0.91 0,468
(o.588 0.466
ETTH2
[0.347
0.388
(0.366
0,404
(0.408
0,415
J0.371
0.399
[0:361
0.384j0.362 0.382/0 332/0.367
0.377jo 37770.3322
0.40|p.361
0.409) (0.405 0.410
(0.455
ECL
0,278|
10.218
10. 0,274/0
0.270]
[0.765 0.686 (0.214
0,278
J0.204 0.273
Weather
[0.256
0.294 (0.265 0.297 [0.270 0,300
10.256
0.288 10.275
0,286
00226 0.281 10110000 264
0.273
[0.294 0.326|0.292 0,315 (0.279 0.306
1" Coun|
15
10
10
10
0
*
Dataset for pre pre-training is not evaluated on corresponding models, which is denoted by dash (-).
Traflic from (PEMS) is generally used during the pre truining of large models and thus not evaluated here. Our model checkpoint is available at atthtps:/huugiingicce co/thuml/timert htttuu///iiiebuus----4- buse-84m]
4.5 MODEL ANALYSIS
Model Efficiency To evaluate the model efficiency of Timer-XI. with respect the context length, is essential to recognize the distinct characteristics of time series data compared to 1D sequences. Unlike natural language, the time series modality is characterized by the variable number N and the input length. We adopt two representative multivariate datasets with different N, and provide the memory footprint and training speed under gradually prolonged input. We evaluate typical approaches to handle multivariate series: (1) Timer-XL and Moiria that adopt channel dependence; (2) Timer that adopts channel independence. Intuitively, the complexity of the first type (NNT22) while the complexity of self-attention under channel independence is ONT22). However, results shown in Figure 6 reveal that measured overheads of Timer-XI. much less than N times of Timer.
Since the previous analysis of model efficiency on time-series Transformer predominantly focuses on self-attention on ID time series, we initially present a theoretical derivation of the computational complexity of Transformers on 2D time series, including the parameter counts, memory footprint, and FLOPs in Table g We find that other parts of Transformers, such as feed-forward network, have a matter which approach is adopted to handle multivariate time series. They also account for dominant overheads in existing benchmarks since the context length is not large enough, confirming empirical results. Further, we introduce FlashAntention (Dao et al. 2022 to
complexity of (NNT)

---

Published as a conference paper at ICLR 2025
improve the model efficiency, which is computationally equivalent and reduces the overall memory footprint of Timer-XL XL to ((TT) without affecting performance.
Weather (2) Variables)
Weather (21 Variahles)
2004
0.0k
Mennyy Usggo MM15)
Mottal
Moinl
1.500
Tunst-NI
Thining Spoaa
Timer
Timar-Xi,, with Fledsknention
Timas XI with Flocittation
1000
Titter
500
96
192
384
450
576
672
192
288
364
576
572
(321
ECL (321 Variables)
ECL (321 Variables)
15000
0.4
Mencry Usoge (8008)
Mairal
Mitri
Titter
10000
XL FlalvAuention
Timer FlabAtration
Timer
0.2
Timer
5000
0.0
Figure 6: Efficiency analysis. We compare representative time-series Transformers multivariate datasets with variable numbers ranging from ten hundred and increase the lookback length.
Learned Attention
Averaged Block
Series
10
10
Cornt
20
Attenf
Attmlli,
Conto 0.78
Sub- Jub-Block(3,
ACF Plot (Variable
60
20
000
-25
20
40 50
Figure 7: Visualization of TimeAttention. is from the first sample of a length 672 in the test split of Traffic. We visualize the last 10 variables with each contains tokens. We present correlattnn function plot. Auto-correlation can be reflected by the distribution of attention scores (bottom right). We average TimeAttention across sub-blocks, which indicates Pearson correlations (upper right).
Representation Analysis In addition to the enhanced performance, fine-grained token dependencies offer improved interpretability. We present a showcase visualization from Traffic Figure 7 is observed that sub-matrices along the diagonal generally receive greater attention, which reasonably reveals predominant dependencies within the endogenous variable. By zooming in sub-block that corresponds Variable-3, we observe that the attention distribution of the last row indicate certain strong dependencies among patch tokens. This observation is also supported by the auto-correlation function plot (ACF), which reveals auto-correlations with certain lags and thus the model pays special attention to these tokens. Furthermore, we average each sub-matrix into one scalar. The outcome matrix can also illustrate Pearson correlations presented in the raw data.
CONCLUSION AND FUTURE WORK
In this paper, we emphasize the efficacy of causal Transformers in the forecasting of long-context time series. To facilitate long-context Transformers on diverse tasks, we propose multivariate next token prediction, a novel paradigm to predict multidimensional series with covariates. We present Timer- XL enhanced by TimeAttention as an extra-long version of pre-trained time-series Transformers. simultaneously captures temporal dynamics and variable correlations by enhanced self-attention. In addition to achieving state-of-the of-te----hh-rr the performance on extensive benchmarks, we establish challenging benchmarks for long-context forecasting. By pre-training
surgescclle heterogeneous time series, Timer- XL demonstrates notable zero-shot performance as a large time-series model. In the future, we will improve computational efficiency and build large domain-specifie models with Timer-XI.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (U2342217 and 62021002), the BNRist Project, and the National Engineering Research Center for Big Data Software.
REFERENCES
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815. 2024.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
George Box. Box and jenkins: time series analysis, forecasting and control. A Very British Affair: Six Britons and the Development Time Series Analysis During the 20th Century, pp. 161-215. Springer, 2013.
Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecasting, Advances in neural information processing systems, 11117766-17777 2020.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16355 2022.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting, arXiv preprint arXiv:2310. 2023.
Xiaohan Ding, Yiyuan Zhang, Yixiao Ge, Sijie Zhao, Lin Song, Xiangyu Yue, and Ying Shan. Unireplknet: A universal perception large-kernel convnet for audio video point cloud time-series and image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5513-5524, 2024.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885, 2024.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730):1999-22)) 2020.
RJ Hyndman. Forecasting: principles and practice. OTexts, 2018.
Taesung Kim, Jinhee Kim. Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jacgul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1r12v:1412.6990 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 4015 2023.
Jesus Lago, Grzegorz Marcjasz, Bart De Schutter, and Rafat Weron. Forecasting day-ahead electricity prices: A review of tfateoofthe-an algorithms, best practices and an open-access benchmark. Applied Energy, 293: 16983, 2021.
Guokun Lai, Wei- Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 104, 2018.
11

---

Published as a conference paper at ICLR 2025
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in neural information processing systems, 32, 2019.
Bryan Lim, Sercan Ô Anı,, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting. 37(4): 1748-17644 2021.
Juncheng Liu, Chenghao Liu, Gerald Woo, Yiwei Wang, Bryan Hooi, Caiming Xiong, and Doyen Sahoo. Unitst: Effectively modeling inter-series and intra-series dependencies for multivariate time series forecasting. arXiv preprint arXiv:2406.0499) 2024a.
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35:5816 1-5528, 2022a.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li. Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference learming representations, 2021.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity time series forecasting. Advances in Neural Information Processing Systems, 35: 9881-9893, 2022b.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.
Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Autotimes: Autore- gressive time series forecasters via large language models. arXiv preprint arXiv:2402.02370. 2024b.
Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. In Forty-first International Conference Machine Learning, 2024c.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54-74, 2020.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint 1rXiv:2211. 14730, 2022.
OpenAl. Gpt-4 technical report. arxiv 2303.08774. View Article, 2:13, 2023.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv: /905. 10437, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
PEMS. Traffic Dataset. http:://pms.... dot ca. gov
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 214555585-5551, 2020.
12

---

Published as a conference paper at ICLR 2025
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Bilos, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schnei- der, et al. Lag-Ilama: Towards foundation models for time series forecasting, arXiv preprint arXiv:2310.08278, 2023.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3):
1181-1191, 2020.
Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time- moe: Billion-scale time series foundation models with mixture of experts. arXiv preprint arXiv:2409. 16040, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transforeer with rotary position embedding. Neurocomputiing, 568: 127063, 2024.
Huihui Sun and Xiaofeng Zhang. Study on coded permutation entropy of finite length gaussian white noise time series. Chinese Journal of Electronics, 33(1):185-19) 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. Beyond the limits: survey of techniques to extend the context length in large language models. arXiv preprint arXiv:2402.02244, 2024a.
Yuxuan Wang, Haixu Wu, Jiaxiang Dong. Yong Liu, Yunzhong Qiu, Haoran Zhang, Jianmin Wang, and Mingsheng Long. Timexer: Empowering transformers for time series forecasting with exogenous variables. arXiv preprint 10Xiv2240..19072, 2024b.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592. 2024.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, Advances in Neural Information Processing Systems, 02222499224300 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint 1rXiv.2210.02186, 2022.
Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with a unified deep model. Nature Machine Intelligence, (6616,02661, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dine coss-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2025
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 11106-11115, 2021.
A PROOF OF MODEL EFFICIENCY
A.1 SETUPS
Given an input univariate time series divided into tokens according to the patch size P, which is fed into the vanilla Transformer. The training objective is to predict the next token of P time points. We will generalize the derivation from ID sequences to 2D time series based on different approaches to handle multivariate data with the variable number N. We adopt the same denotations before: Transformer consists of L blocks with model dimension D. The multi-head attention mechanism has H heads, each with a dimension of dy for query, key, and value, and dę /.- The intermediate dimension of feed-forward network is set as Dr = a D. The results are summarized in Table X we provide the detailed proof in the following sections.
Table 8; Parameters count and computational complexity of Transformers for multivariate time series.
Metric
Type
Count
Complexity
FLOPss
Channel Independence
12(PDNT + L(D H)N +2+ ())
OLLDNTOD 7)) O(LDNT(D NT)
(Training Spreds
Channel Dependence
12(PDNT + L(D H) + (2++ N)LDRNT)
Encoder Only
(4+2))LD' + +1 (1+T))D
O(LD")
Parameters
Decoder-Only
(4 (4+ + +2DD
O(LD")
Memory
Self Attention
4(D+ P)N+ - (32 LDNT + 4LHN9T
O(LHN9T9)
Fomppint
FlachAttention
4(1 P)NT [32 80) I.DTT
O(LDNT)
block number of Transformers, D the dimension of embeddings (the hidden dimension FFN Dr is set o)) the head number and the dimension of query. key, and value de = D:H. The ovethead is to train multivariate time series (N-variables and TP time points) with patch token length P and context length T. Set N for training on univariate time series.
A.2 FLOPS
As preliminary, the multiplication between matrix A E Raxm and matrix C mx requires mnp multiplications and mnp additions, resulting in 2mmp floating-point operations. Given batched matrices A E IR Bxxxm and C E IR Bxxxp B times matrix multiplications will be performed. is evident that the batch size is linear multiplier. Thus, we first omit B to calculate the operations of dealing with one univariate series, and then we will reintroduce
analyze channel independence.
The computational cost of Transformers can be primarily categorized into two types: (1) multi-head attention calculation and (2) linear transformations. In contrast, the operations of layer normalization. residual connection, activation functions, and position embedding with the complexity of O(TD) are less significant. Therefore, we derive the computational complexity mainly with respect to the above two types by delving into the forwarding process of one univariate series.
Patch Embedding The tokenized time series (x:) RxP mapped into the embedding space through the patch-wise embedding W. E DxP rresuuting in DP operations.
Self-Attention The calculation of self-attention begins with the computation of query, key and value by multiplying the patch embeddings with matrices W q. Wx. W. E Dxde respectively in H heads, which incurs a computational cost of 6H = 6DPT and yields Q V xxdx, Next, the dot product QK E = is conducted in each head, leading to 2Hd = 2DT operations. Following this, the Pre-Softmax map is divided by yd and processed through Softmax, which includes exponentiation, summation, and normalization of each element, resulting in 4HT² operations. The subsequent multiplication with V incurs 2Hd = 2DT2 operations. Finally, multiple heads are concatenated and multiplied by W. E RDDD, contributing 2D2T operations.
14

---

Published as a conference paper at ICLR 2025
Feed-Forward Network It first projects the token representations into the dimension of Djf and subsequently projects it back to the dimension D, resulting in total operations of 4ax D²T.
Patch Projection For encoder-only models, all token representations are flattened and papped directly to P time points by Wa E RTDxP, In contrast, token-wise projector W E R pxP in decoder-only models independently map each token the predicted next token. In both cases, the number of operations is 2PDT, but the token-wise projector will result in a smaller parameter count.
The forwarding operations in L-layers Transformer is 4PDT + 4L(D + HYT + (8 + 4a) in sum. Considering that the majority of operations in Transformers are binary operations (e.g.. matrix multiplications), the gradients for both matrices are computed separately. As a result, the number of operations in backpropagation is the twice of forwarding. Therefore, the total operations of training Transformer on a univariate series consisting of patches, each of length P. is derived as:
f(T) 12PDT 12L + HYT + (24 + LD'T.
We plug typical hyperparameters in the current time-series Transformers and forecasting benchmarks: D 512, H 8, L 4, a 4,T 7, and P 96, obtain that:
f(T) = 249607 + 760872967 x 3.28 10 "0" +
Due to the prevalence of short contexts in the time series field, where T << D leads significant coefficient in (T), we find the primary computational burden of time-series Transformer lies in linear transformations with (T"), rather than in multi-head self-ateention with the O(T²) complexity.
For multivariate series with variables, FLOPs is influenced by the handling of multivariate data. When adopting channel independence (Timer and PatchTST), N can be regarded as the batch size B:
N f(T) = 12PDNT + ( + H) + (24 + LDx NT.
(9)
For models that capture fine mne-grained intra- and inter-series dependencies (Timer-XL and UniTST) in multivariate scries, N reflected as the enlarged number of tokens:
f(NT) 12PDNT + 12L(D + H)N (24+ 12a) NT.
(10)
Notably, FLOPSs is not entirely equivalent to actual runtime. While Fltennittntion increases the overall FLOPs due to its recomputation process, it reduces the number of memory reads and writes. Given that on GPUs, computation is significantly faster than memory access, using FlashAttention actually lead to further improvements in runtime performance.
A.3 PARAMETER COUNT
From the above analysis, we observe that the parameter count of Transformers includes the following:
Patch Embedding
W. RxP P to obtain patch embeddings.
Self-Attention
W WR, W. E Dxdx of H heads and W, E DxD for all heads.
Feed-Forward Network
Win Wimz E Dx in ford-forward network.
Layer Normalization It contains the weight W E ER and the bias b E TRP, Every Transformer block includes two normalizations after multi-head attention and ferw-forward network respectively.
Patch Projection Wa E R TDxP in flatten head and Wa E DxP in token-win wise projection.
In sum, the total count of parameters in time-series Transformers can be expressed as:
(4 2a)DD + 4LD + (1DT)PP, using flatten head, using token-wise projection. (11)
Parameter Count
(4+ (4+2a)LDD 4LD + 2PD,
15

---

Published as a conference paper at ICLR 2025
A.4 MEMORY FOOTPRINT
The memory footprint during training can be primarily categorized into three parts: activation values stored for backpropagation, model parameters, and optimizer parameters.
Regardless of other precision types (e.g., FP16), model parameters and gradients are typically stored as 32-bit doating-point numbers, with each parameter occupying bytes of memory. For time-series Transformers, memory footprint of activation values is given as follows:
Patch Embedding
Gradient computation for Wa preserves its input (x:) E RXX of 4PT bytes.
Self-Attention Gradient calculation for W WA. W, Dxd requires their inputs H E RTXD amounting to a total of 4DT bytes. The dot product for attention map also needs to store Q K, V E Rxxxxdr which collectively require total of 12DT bytes of memory. Gradient computation of W. E DDD necessitates the concatenated multi-head attention representations H E R RT which occupies 4DT bytes. If memory-efficient attention mechanisms like FlashAttention (Dao et al. 2022) not applied, the outcome QK will be stored and occupy 4HT bytes. Instead, if FlashAttention is adopted, the storage overhead can be avoided.
Fred-Forward Network ReL.U activation funetion typically employed this module. The input H E TRTxD must be retained, requiring total of 4D7 bytes. Additionally, the product Wini H also needs to be stored, amounting to 4D,I' bytes. Similarly, the output activations of ReLU, which serve as the input for subsequent linear transformations, necessitate another 4D bytes.
Layer Normalization Each block of Transformer encompasses two layer normalizations, with each normalization retaining its input, resulting in the memory requirement of 8DT bytes.
Patch Projection
To perform backpropagation for Wa E RR it is necessary to retain its input H TT resulting in total memory requirement of 4DT bytes.
The formula for the total activation values of the entire model occupying GPU memory is as follows:
[4(D + P)T + (32 8D) 4LHT2 w/o FlashAttention, 14(D + PYT + (32 + LDT,
Memory Footprint
(12)
with FlashAttention.
The derived occupancy of activation values increases proportionally with the batch size B. For multivariate series, N can be used as a multiplier in channel independence. For channel independence models, we can substitute T with NT as before. The total memory footprint is the sum of activation values and parameters of model and optimizer, which proportional to the parameter count derived in Equation| II Due to the limited model size in the time series field, the memory consumption of parameters minimal and can be considered negligible in practice. Therefore, the overall memory footprint can be predominantly determined by the occupied memory of activation values.
B EXPERIMENTAL DETAILS
B. DATASETS
We conduct experiments on well-acknowledged benchmarks to evaluate performance of the proposed Timer-XI, which includes (1) ETT (Zhou et al. (02T) contains 7 factors of electricity transformers from July 2016 to July 2018, which is recorded every hour or 15 minutes. (2) Weather (Wu et al. 2002T) includes 21 metcorological factors collected every 10 minutes from the Max Planck Biogeochemistry Institute Weather Station in 2020. (3) ECI. (Wu al. 202T) records the hourly electricity consumption data of 321 clients. (4) Traffic (Wu et al.) 2021 collects hourly road occupancy rates measured by 862 sensors on the San Francisco Bay area highways from January 2015 to December 2016. (5) Solar-Energy (Lai et al. 2018) records the solar power production of 137 PV plants in 2006, which are sampled every 10 minutes. (7) PEMS (Liu et al. 2022a) contains records from the public traffic network in California collected in 5-minute time windows. (8) EPF (Lago et al.) 2021) includes five subsets that span six years. Each contains the electricity price as the endogenous variable to be predicted and two exogenous variables of the day-d electricity markets. (9) GTWSF (Wu et al.
16

---

Published as a conference paper at ICLR 2025
2023 is dataset collected from the National Centers for Environmental Information (NCEI). This large-scale collection contains hourly averaged wind speed and temperature data from 3850 stations with different geographical scales and densities each, spanning from 2019 to 2021. (10) UTSD (Liu et al. 2024c) is a multi-domain time series dataset, which includes seven domains with a hierarchy of four volumes. We adopt the largest volume that encompasses billion time points for pre-training.
We further establish challenging forecasting benchmarks based on the ECMWF Reanalysis v5 (ERAS) dataset (Hersbach et al. 2020) prevent potential overfitting and performance saturation of deep forecasters in existing benchmarks. Concretely, ERAS is the fifth generation ECMWF atmospheric reanalysis of the global climate covering the period from January 1940 to the present, which provides hourly estimates of large number of atmospheric, land, and occanic climate variables, and includes information about uncertainties for all variables at reduced spatial and temporal resolutions. Due to its pattern sufficiency of temporal dynamics and variable correlations, we could establish practical benchmarks to thoroughly evaluate the performance for univariate and multivariate forecasting, as well as adopt it for large-scale pre-training to develop domain-specific large time series models.
Our datasets are constructed as follows:
To establish a realistic univariate forecasting benchmark, we start from the basic principle of forecastability and make the prediction sufficient lookback lengths. Instead of the short time span of training in previous benchmarks (generally no more than 2 years), we curated three-hour frequency dataset spanning 40 years (January 1979 to December 2018) from ERA5, encompassing 116880 time points. In order to prevent overfitting on a single time series, we selected worldwide stations to form seven subsets.
ERAS-S:
Each univariate series of ERA5-S provides partial observations governed by the spatio-temporal global weather system. Since discovering the global spatio-temporal correlations presents a fundamental challenge in meteorology, we convert ERA5-S into ERA5-MS by using seven subsets as a challenging multivariate forecasting benchmark. Based on the average results in Tables [ndd]] [5] we can validate the existence of multi-station correlations among selected stations, which have enhanced the average prediction accuracy.
ERAS-MS:
To explore the pure data-driven approach to build domain-specificc large time series models, we further expanded the number of stations as ERA5- a dataset that evenly covers metcorological 4920 worldwide stations and spans 40 years. We establish the dataset for pre-training, which is expected to generalize across the time (train on the past observations and generalize to the future) and across stations (train on partial stations and generalize to other unseen stations). The total number time points around half billion.
ERAS-Large:
We follow the same data processing and train- validation- split protocol used in TimesNet (Wu et al. 2022). where the train, validation, and test datasets divided according to chronological order to prevent data leakage. Detailed dataset descriptions and prediction settings are provided in Table 9
B.2 BASELINE MODELS
We aim to present Timer-XL as a foundation model for unified time series forecasting. We thoroughly include well-acknowledged and advanced models in each forecasting task. For univariate time series forecasting, we compare Timer-XL with PatchTST (Nie et al. 2022) under channel independence. For multivariate time series prediction, we report official results from Liu et al. 2023 2024b); Ding et al. (2024), including UniRepL.KNet (2024). i'ransformer (2023). Corrformer (2023), DLinear (2023). TimesNet 22022), Non-stationary Transformer (2022b). Pyraformer (202T). Autoformer (2021)
StemGNN (2020), DeepAR (020)) and N-BEATS (2019). We further reproduce the performance of related Transformers: Timer (2024c) and UniTST (2024a) based on their official repositories. For covariate-informe. time series forecasting, we report the official results of TimeXer (2024b). For zero-shot forecasting, we follow Liu al. (202cc) that predicts future length-96 windows in well-acknowledged datasets. Totally, more than 20 basclines are included for complete comparison.
B.3 IMPLEMENTATION DETAILS
All the experiments are implemented by PyTorch (Paszke et al. 2019) on NVIDIA A100 Tensor Core GPUs We employ the Adam optimizer (Kingma & Ba 2014) and MSE loss for model optimization.
17

---

Published as a conference paper at ICLR 2025
Table 9: Dataset descriptions. Dim. denotes the number of variables (For univariate forecasting, we adopt channel independence (Nie et al. 2022) train separate models each variable). Dataset Length denotes the number of time points in the (train, validation, test) splits.
Tasks
Dataset
Dim.
Training Setting
Dataset Length
Information (Frequency)
ETThi
|
(24, 96, 168, 672, 2880) -96
(8545, 2881, 2881)
Flectricity (Hourly)
Univariate
321
{24, 672, 2880, 8832)-496
(18317, 2633,
Electricity (Hourly)
Forecasting
Traffic
862
{24, 96, 168, 672, 2880, 8832)
(12185, 1757, 3509)
Transportation (Hourly)
PEMS03
358
(96, 288, 1152, 2016, (064)-996
(15617, 5135, 5135)
Transpartation (5 mins)
ERAS-S
3072-196
(81816, 23376)
Climate Hours)
ETThi, ETT12
(96,672) (96, 192, 336, 720)
(8545, 2881, 288
Fllecricity (Hourly)
ETTml, ETTm2
(92)-62)++1 (96. 192, 336, 720)
(34465, 11521, 11521)
Electricity (15 mins)
ECL
321
(96, 696,672)-+(96 192, 720)
(18317, 2633, 5261)
Electricity (Hourly)
Multivariate
Traffic
862
(96, (16) (966, 192, 336, 720)
(12185, 1757, 3509)
Transportaion (Hourly)
Forecasting
Weather
21
(96, 67.) (96, 192, 336, 720)
(36792, I0540)
(10 mins)
Solar-Einergy
137
{96, 672) [96. 192. 336. 720)
(36601, 5161. 10417)
Energy (10 mins)
ERAS-MS
3072-96
(81816, 11888, 23376)
Climate (3 Hours)
GTWSF
3850
48-+24
(12280, 1755, 3509)
2023
NP
1+2
168-+24
(36500, 5219. 10460)
Electricity (Hourly)
Forecasting
PIM
1+2
168-+24
(36500, 5219, 10460)
Flectricity (Hourly)
with Covariates
BE
142
168-+24
(36500, 10460)
Electricity (Hourly)
FR
1+2
168-+24
(36500, 5219, 10460)
Electricity (Hourly)
DE
1+2
168-+24
(36500, 5219, 10460)
Electricity (Hourly)
ERAS-Large
4920
3072-996
(81816, 23376)
Climate Hours)
Pre-truining
UTSD
2880-196
(868778970, 96530996,
Lio et al 204g
LOTSA
2880-99
(231082956489.
ct 2024
Table 10: Performance robustness of Timer-XL. The prediction settings and results keep the same The standard deviation is obtained from three random seeds.
with Table [2]
Dataset
ECL
ETTh1
Traffic
Horizon
MSE
MAE
MSE
MAE
MSE
MAE
96
0.127:10.001
0.209
0.364:10.002
0.397=1.000
0.340:100002
0.238
192
0.145++0.01
0.236+0.001
0.405=00002
0.424=0.001
0.360=+0001
0.247=0.001
336
0.15940.001
0.252+0.001
0.427+0.003
0.439=0.002
0.377+0.002
0.256-0.002
720
0.187
0.277
0.43910.002
0.459
10.003
0.279_0.002
Dataset
Solar-Energy
Weather
ERAS-MS
Horizon
MSE
MAE
MSE
MAE
MSE
MAE
0.162-10.03
0.22140.002
0.157+00002
0.205=0.001
0.16440.001
0.307=00000
192
0.187-10.03
0.239-0.002
0.206-10.003
0.250-0.002
336
0.205
0.255+
0.259-00000
0.291=0.003
720
0.208
0.209
0.337-10.002
0.344=00002
18

---

Published as a conference paper at ICLR 2025
We adopt channel independence from Nie et al. (0022) in univariate time series forecasting. Based on the prevalence of patch-level tokenization in the time series field, we reproduce typical Transformers: PatchTST (2022), Timer (024c),, and UniTST (2024a) based their official repositories, and keep their model hyperparameters and training configurations the same to evaluate the inherent capability of base models. The results of other baselines based on the benchmark provided by [Lu al. (2023) 2024b); Ding et al. (2024); Wang et al. (2024b). which is fairly built on the configurations provided by their original paper. Detailed experimental configurations are provided in Table [1 We also report the standard deviations under three runs with different random seeds in Table 10 which
exhibits that the performance of Timer-XL, is stable.
For the metrics, we adopt the symmetric mean absolute percentage error (SMAPE), a metric that is independent of the numerical range, to evaluate one-for-all generalization performance on ERA5- Large. For other experiments, we adopt the root mean square error (MSE) and mean absolute error (MAE) that follows previous work. These metrics can be calculated as follows:
200 T
X
X11 X
SMAPE
MSE
Σ
X
MAE
Σ X. X
Here X E I is a univariate time series and X is the corresponding prediction. For multivariate time series, we further calculate the mean metric in the variable dimension.
Table 11: Experimental configurations of XL and other baseline Transformers. All the experi- ments adopt the ADAM (2014) optimizer with the default hyperparameter (81, 32) = (0.9, 0.999).
Configuration
Truining Process
Experiment
Model
Dataset
L
D
de
H
P
LR
Loss
Batch Size
Epochs
ECL.
3
512
64
8
96
0.0005
MSE
2048
10
Timer- XL.
Traffic
3
512
64
8
96
0.001
MSE
2048
10
Univariate Forecasting
PatcTTT
ETTh1
1
512
8
96
0.0005
MSE
256
10
PEMS03
512
0.0005
MSE
2048
10
ERAS-S
512
0.0005
MSE
2048
10
Global Temp.
3
1024
128
8
0.0001
MSE
10
Global Wind
3
1024
128
8
0.0001
MSE
10
Timer-XL
ECL
5
512
8
0.0005
MSE
10
UTiTST
Traffic
4
512
64
8
96
0.0005
MSE
10
Multivariate Forecasting
Timer
ETThl
1024
128
8
96
0.0001
MSE
32
10
PatchTST
Weather
4
512
64
8
96
0.0005
MSE
32
10
Solar.
6
512
64
8
96
0.0001
MSE
16
10
ERAS-MS
3
512
64
8
96
0.0001
MSE
256
10
Timer-XL
NP
3
512
64
8
24
0.0001
MSE
10
TimeXer
PJM
2
512
64
8
24
0.0001
MSE
16
10
Forecasting with Covariates
Timer
BE
2
512
64
8
24
0.0001
MSE
16
10
PatcITT
FR
2
512
8
0.0001
MSE
16
10
DE
2
512
0.0001
MSE
Timer XL.
4
512
0.0001
MSE
40960
10
ERAS-Large
PatcTTT
4
512
0.0001
MSE
40960
10
Timer-XL
UTSD
8
1024
128
8
0.00005
MSE
16384
10
Pre-training
Timer
Liu et 2024c
8
1024
128
8
0.00005
MSE
16384
10
Timer-XL
8
1024
128
8
96
0.001
MSE
32768
Moiraisua
LOTSA
6
384
64
6
Moiruinse
Woo et al. 2024
12
768
64
12
Woo et al. 2024
Moirais
24
1024
64
16
L. is the layer number of Transformers, D the dimension of token embedding (the hidden dimension of FEN set as 4D), de the dimension of query, key, and value, // the multi-head number, P is the patch size, and LR is the initial learning rate.
19

---

Published as a conference paper at ICLR 2025
C HYPERPARAMETER SENSITIVITY
We evaluate the hyperparameter sensitivity of Timer-XL on the ERA5-MS benchmark, as illustrated in Figure(8] concerning the following factors: the number of layers L, the patch size P, and the lookback length during inference. Our findings indicate that performance of Timer-XL, generally improves with increases with L, suggesting that Timer-XL of the influence of P reveals that the optimal patch size is generally close to the predicted length, since it avoid multi-step error accumulations. Toward better long-term forecasting performance, leaves a future improvement to adopt different patch sizes of input and output tokens. Finally, investigate the impact of input length during inference. We discover that the optimal lookback length of during is not necessarily the length during training. Given that decoder-only Transformers accommodate inference inputs shorter than those used during training, this finding is noteworthy and indicates the potential improve the performance.
scalable deep forecaster. Furthermore, our analysis
Layer Number
Patch Size
Lookback Lenglh
0.172
0.171
0.1775
0.371
0.170
0.169
0.169
0.1725
0.163
0.368
0.1700
0.167
0.1675
0.365
0.165
1152 1920 2304 2558 3072
Figure 8: Hyperparameter sensitivity of Timer-XI. (input-3022-rred- on ERA5-MS), including the number Transformer blocks L, the patch size P, and the input lookback length during inference.
D SHOWCASES
To facilitate a clear comparison among various models, we present additional prediction visualization Showcases are randomly selected from Timer-XL XL and the following time-series Transformers: PStchTST (2022), Timer (2024c), and UniTST (2024a). Among them, Timer-XL presents the most accurate predictions.
from diverse datasets in Figure 9 and I0
ECL
ETT1
PEMS
Traffic
PatciST
TimarXL
M
Figure 9: Visualization results on univariate time series dataset. We adopt the forecasting setting of 2880-pred-96 on ECL, ETTh1 and Traffic, and 2016-pred-99 on PEMS.
E SUPPLEMENTARY RESULTS
E. FULL RESULT OF MULTIVARIATE FORECASTING
Table 12 provides the complete results of the one-for-all multivariate forecasting benchmark across well-acknowledged datasets. We evaluate Timer-XI. and baseline models by rolling forecasting: each
20

---

Published as a conference paper at ICLR 2025
ETT
Traffic
Figure 10: Visualization results on multivariate time series dataset. We adopt the forecasting setting of 672-pred-96 on ETTh1 (7 Variables) and Traffic (862 Variables).
model is trained with input length 672 and output length 96, and the predicted values are integrated as part of the input in the next iteration until reaching the desired forecast length in (96, 192,,366 720).
We highlight that this benchmark evaluates the fundamental model versatility deep forecasters, which aims to break the awkward situation of extensive training and model storage in pursuit of better practice for real-world forecasting requirements. On this benchmark, time-series Transformers significantly stand out from other baseline models, and proposed Timer-XI. can achieve state-of- the-art performance, making it nice fundamental backbone of a one-for-all forecaster.
E.2 FULL RESULT OF ZERO-SHOR FORECASTING
Table 13 provides the full results of zero-shot forecasting the benchmark from Wu al. (022)). build Timer- XL based on the configuration Table II] which is pre-trained on the aggregated datasets of UTSD (Liu al. 2024c) and LOTSA (Woo al. 2024). The patch size of Timer-XX XL is set as 96 and we conduct rolling forecast to obtain the desired forecast length in {96, 192, 336, 720)
We evaluate most advanced large models based their official model checkpoints, including Time- MoE (Shi et al., 2024), Moirai (Woo et al. 2024), TimesFM (Das et al. (2023). MOMENT Goswami al. (2024), and Chronos (Ansari al., 2024). We conduct zero-shot evaluations datasets that are not included during the pre-training of corresponding models. For each of the evaluated model, use their maximum input length during inference. The metric (MSE/MAE) is averaged from all predicted windows in the test split.
E.3 ABLATION STUDY TIMEATTENTION
We conduct evaluations TimeAttention to validate the effectiveness of position embeddings. As for variable embedding, the distinction between endogenous and exogenous variables can improve performance. Based more attention to tokens of the endogenous variable. It leaves a prior to mask out minor dependencies that focuses less exogenous variables. For the temporal dimension, other position embeddings inferior to RoPE, since it uses the affine transformation, while others are additive, and thereby less confused with the same additive embedding for variables.
observation of the learned u >
find that the token reasonably pays
E.4 SUPPLEMENTARY RESULTS OF LONG-CONTEXT FORECASTING
Long context a basic indicator of foundation models, which can support emergence capabilities such as prompting, in-context learning, vetrieval-augmente generation, etc. However, the long-context forecasting paradigm receives less attention in the current community, which can be due to the lack of benchmarks. In the meteorological ERA5, it is necessary support the context of more than years to contain a specific cycle (such as El Nino). Table [5] the performance of Timer-Xi XL and DL inear generally improves with the increased context length. By contrast, it reveals the performance
21

---

Published as a conference paper at ICLR 2025
Table 12: Full multivariate forecasting results: we conduct rolling forecast with a single model trained on each dataset (lookback length is 672) and accomplish four forecast lengths in (96, 192, 336, 720}.
Models
Timer-XI.
Timer
UniTST 2024n
iTransformer
DLinear
PatchTST
TimesNet (002
Stationary
Autoformer
(Ours)
2024c
2023
2023
(0022
2022h
2021
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.364
0.397
0.371
0.404 0.379
0.415
0.387
0.418
0.369
0.400
0.373
0.403
0.452
0.463
0.452 0.478
00466 0.499
ETThl
192
0.405
0.424
0.407
0.429
0.415 0.438
0.416
0.437
0.405
0.422
0.405
0.425
0.474
0.477
0.484
0.510
0.492 0.523
336
0.427
0.439
0.434
0.445
0.440
0.454
0.434
0.450
0.435
0.445
0.423
0.440
0.493
0.489
0.511
0.522
0.519 0,531
720
0.439
0.459
(1:461
0.466
00.482
0,482
01.447
0.473
01.493
0.508
0.471 0.445
005560
0.534
0.571
0.543
005589 0.560
Avg
[0.409
0.430
0.448
0,436
(0:429
0.447
[0.421
0.445
[0.426
0.444
0.412
0.435
[0.495
0.491
00.505
0.513
(0.517 0.528
96
[0.277
0.343
0.285
0.344 00.343 0.398
00.304
0.362
10.305
0.371 0.289
0.347
0.3340
0.374
0.348
0.403
[0.358 0.397
BTTh2
192
0.348
0.391
0.365
0.400
0.376 0.420
0.372
0.407
0.412
0.439
0.360
0.393
0.402
0.414
0.408
0.448
0.435 0.451
336
0.375
0.418
0.412
0.440
0.399 0.435
0.418
0.440
0.527
0.508
0.389
0.420
0.452
0.452
0.424
0.457
0.454 0.475
720
0.409
0.458
0.468
0.487
0.419 0.457
0.4463
0.476
0.830
0.653 0.398
0.440
0.462
0.468
0.448
0.476
00.479 0.492
Avg
00.352
0.3422
0.402
0.382
0.418 0.384
0.428
[0.389
0.421
|0.518
0.493
0.359
0.400
00.414
0.427
00:407
0.446
0.446|0..431
|0.431 0.454
96
0.2990
0.34|
00.281
0.338
0,289
0.348
0.311
0.365
|0.307
0.350 0.285
0.346
0.338
0.375
0.414
0.414
|0.466 0.466
mT
192
0.337
0.369
0.330
0.368
0.332
0.375
0.353
0.390
0.337
0.368 0.329 0.372
0.371
0.387
0.524
0.482
0.504 0.496
336
0.374
0.392
0.367
0.393
0.365
0.397
0,387
0.411
0,366
0.387 0,363 0,394
0.410
0.411
0.541
0.497
0.574 0.530
720
0.437
0.428
0.432
0.433
0.421
0.431
0.452
0.445
0.419
0.419 0.421 0.426
0.478
0.450
0.578
0.509
0..59 0.558
Avg
|0.359
0.382
00.352
0.383
0.352
0,388
[0.376
0,403
|0.357
0.381 0.349
0,385
|0.399
0.406
00.514
0.475
j0.535 0,512
96
10.175
0.257
0.1175
0.257
0.171
0.260
0,.183
0.272
00.66
0.263
0.172
0.259
0.1187
0.267
0.237
0.306
(0.255 0.339
ml
192
0.242
0.301
0.239
0.301
0.228
0.230
0,250
0.315
0.230
0.311
0.233
0.299
0.249
0.309
0.330
0.387
0.279 0.335
336
0.299
0.337
0.293
0.342
0.282
0.336
0.311 0.356
0.298
0.361
0,280
0.331
0.321
0.351
0.404
0.424
0.331 0.374
720
0.376
0.390
0.392
0.407
0.380
0.398
0.417
0.419
0.432
0.446
0.357
0.382
0.497
0.403
0.525
0.486
00:413 0.450
Avg
[0.271
0.322
01.775
0.327
0.2665
0.306
[0.290
0.340
|0.282
0.345
0.261
0.318
|0.314 0.333
0.374
0.401 |0.320 01.74
96
0.1227
0.219
0.1
0.221
0.130
0.225
0.13
0.229
0,1338
0.238
0,132
0.232
0..184
0,288
0.
0.185
0.287
(0.256 0.357
ECL
192
0.145
0.236
0.148
0.239
0.150
0.244
0,158
0.258
0.152
0.251
0.151
0.250
0.192
0.295
0.282
0.368
0.291 0.376
336
0.159
0.252
0.164
0.256
0.166
0.262
0.168
0.262
0.167
0.268
0.171
0.272
0.200 0.303
0.289
0.377
0.290 0.379
720
0.187
0.277
0.201
0 0.289
0.206
0.297
00.205
0.294
0.200
0.302
0.222
0.318
0.228
0.325
0.305
0.399
0.320 0.403
Avg
|0.155
0.246
0.161
0.251
00.163
0.257
00.164
0.258
00.165
0.265
0.169
0.268
|0.201
0.303
0.265
0.358
|0.289 0.379
96
0.340
0.238
0.348
0.240
0.359
0,250
0.353
0.259
00339
0.285 0.359 0.255
0.593
0.315
0.610
0.322 0.675 0.412
Traffic
192
0.360
0.247
0.369
0,250
0.373
0.257
0.373
0.267
0.409
0.290
0.377 0.265
0.596
0.317
0.626
0.346 0.679 0.423
336
0.377
0.256
0,388
0,260
0.386
0.265
0
0.275
0.422
0.297 0.393 0,276
0.600
0.319
0.633
0,352
0.688 0.440
720
0.418
0.279
0.431
0.285
0.421
0.286
0.425
0.296
0.461
0.319 0.436 0.305
0.619
0.335
0.651 0,366 0.693 0.457
Avg
|0.374
0.255
0.384
0.259
0.385
0.265
00.34
0.274
[0.423
0.298
0.391
0.275
0.602
0.322
0.630
0.347 [0.684 0.433
Weather
96
10.157
0.20510
0.151
0.202
00.152
0.206
17.
0.225
10.169
0.229
0.149
0.202
06
0.228
0. 185 0.241
|0.355 0.409
192
0.2006
0.250
0.66
0.245
0.198
0.249
0..27
0.268
0.211
0.268
0.194 0.245
0.222
0.269
0.286
0.325
00.421 0.450
336
0.259
01.291
0.249
0.288
0.251
0.291
01290
0.309
0.258
0.306
0.244
0.285
0.299 0.310
0.323
0.347
00.452 0.466
720
0.337
0.344
0.330
0.344
0.322
0.340
0.374
0.360
0.320
0.362
0.317
0.338
0.376
0.364
0.436
0.401
00.513 0.496
Avg
[0.240
0
0.273
00.232
0.270
00.231
0.272
[0.266
0.291
|0.239
0.291
0.226
0.268
[0.264 0.293
00.308
0.329
|0.435 0.455
Solar-Energy Energy
96
00162
0.221
00.212
0.230
0.190
0.240
183
0.265
10.193
0.258
0.168
0.237
08080 0.272
0.199
0.290
|0.206 0.296
192
0.187
0.239
0.232
0.246
0.223
0.264
0.205
0.283
0,214
0.274
0.189
0.257
0.199
0,286
0:243
0.307
0.254 0.328
336
0.205
0.255
0.237
0.253
0.250
0.283
0.224
0.299
0,233
0.291
0.212
0.277
0.220
0,301
0.264
0.322
0.272 0.330
Solar-
720
0.238
0.274
01.252
0.266
00.292
0.311
0.239
0.316
0.244
0.307
0.240
0.305
002251
0.321
0.310
0.339
0.326 0.347
Avg
00.198
0.249
0.233
0.249
0.241
0.275
00.213
0.291
0.222
0.283
0.202
0.269
0.213
0.295
0.254
0.315
|0.265 0.325
Count
21
8
0
0
5
14
9
0
0
22

---

Published as a conference paper at ICLR 2025
Table 13: Full results of zero-shot forecasting. lower MSE or MAE indicates a better prediction. 1st Count represents the number of wins achieved by model under all prediction lengths and datasets.
Models
Timer XLa.
Time Moka--
Time- Mine-Mee...r
Time Moleew
Meiraisus
Moiraisae
Moiraicow
TMmesFM
MOMENT
Chrupemuur
Chronok
(Ours)
2024
2024
2024
2022
2024
20244
Metric
MSB
MAR
MSR
MAE
MSB
MAE
MSR
MAF
MSE
MAF
MSF
MAR
MSE MAF
MSE
MAF
MSE
MAR
MSF
MAR
MSE
MAE
10.317
0.356
00.338
0.368
0.357
0.341
16.418
192
0.363 8.356
(.3616
0.370)0.654 527
0.454 0.408
00.457
0,403
192
0.358
0.381
0.353
0.388
0.346
0.381
0.305
0.358
0.431
0.405
0.388
0.375 0.112 0.383 0.114
0.315
0.38
0.405 0.662 0.532
0.567
0:177
0,530 0.150
336
0.386
0:401
[0.381
0.413
0.373
0.408
0.369
0.305
00.433
0:412 0.416
0,392 01436
0.400
0.429/0.672
0.537
0.662
0.525
0.577
0.481
0.430
0:431
(0.504
0.493
0.475
0.477
00.469
0.472
(0:462
0:432
(.455)
0.418
0.418
0.442
0.420 0.471 (0.622 0:551
0.47)
(..000
0.591
(0.660
[0.378
0.392
[0.394
0.415
|0.976
0.405
|0.356
0.391
[0.436
0.410/0.406
0.385 (0442 0. 0.391 0.391|0.433 0.418|j0.77000356......
00.66
0.500
(0.555
0.465
96
10.189
0.277
[0:201
0.2911
10.
0.286
10.198
0.268
10.214
0,788
(0.205 0.273 0.231 0.274 302 0.270 19919 0.274
0.273
10.197 0.271
[1:241
0:315
0.334
00.250
0.322
0.235
0.312
(0.284
0.332 00.275 0.316 (.288) 0.518 0.289 (0.289
0.350
0.261
00.254
336
0.286
0.348
0..324
0.373
0.337
0.375
0.203
0.348
0.331 0,362 0.329 p. 350 0.341 0.355
0.355 0,360
0.334 0.369
0.326
0,366
00,313 0.353
0..775
0.402
[0.488
0.464
00:480
0.428
(0,402
0.408 0.497 0:411
0.42
0.430
0.40)
00.455
(0:416
0.415
[0.273
0.336
[0.317
0.365
|0.316
0.361
10.288
0.344
|0.307
0.347
(0.311 0.337 0 0.329 0.34330 328 0.34600.316
0.365
(0.310 0.350
(0.295 0.338
96
00,399
0.291
[0.357
0
11.350
0.382
10.349
0.379
10.401
0.402
10.336
0.392/0
0.38) 0. 0.388|0.414
0.404
(0.688
0.557
(0.400
0.393
[0.44] 0.390
00.405
0.413
(0.384
0.409
1..0%%
0.399
00.435
0.421
0.412
0.413
0.41300.44
0.434
0.415
0.434
60w
0.560
00.492
0.426
(0.502 0.534
336
0.4188
0.423
(0.411
0.434
0.411
0.430
0.447
0.453
0.438
0.434
0.483
0.428 0.485 0.445 0.503
0.465 0.563
00550
0.462
00.576 0.467
720
[0.423
0.441
00:489
0.477
00.427
0.455
004457
0.462
00.439
0.454 0.447
0.444
0.631
0.510
0.483 68.683 0.585 (0.822
0.591
[0.835 0.583
Avg
[0.40:1
0.417
[0:100
0.424
[0.394
0.119
[0.112
0.126
[0.128
0.427
(o.417 0.119
0.439
1731 0.443 //.6638883 0.566 (0.591 0.168
|0.588 0.466
96
10.283
0.342
(0.305
0.359
[0.302
0.354
[0.292
0.382
10.297
0.336
(0.29c 0.330 0.296
0.330
0.34910.342 0.396 (0.308 0.343
(0.320 0.345
0.340
0.379
00.381
0.386
00.364
0.385
0.347
0.379
0,368
0.381
0.365
0.375 0.361 0.371 0.388 o 0.3950.354 00.354 0.402 0,384
0.392
00,406 0.399
336
0,366
0,400
00.391
0.418
0.417
0.425
0.406
0.419
0.370
0.343 0.3766 0.390 0.390 0.390 o 0.422 0.427 00.366 0.407 0.429
0.430
00,492 0.453
720
[..397
0.431
00:419
0.454
0.537
0.496
0.439
0.447
0.:41
0:426
0.446 0.433 0.423 0.418 0.443 0.454 (0.395 0.434 005011
0.477
[0.603
0.511
Avg
[0.347 0.388
[0.366
0.404
00,405
0.415
10.371
0.399
|0.361
0.384 |0.362
p. 1.382210 0.382 01366 0.377/0
0.406/o 361 0.409 (0.405
0.410
(0.455
0.427
96
00.141
0.237
10.189 0.280
16.
0.250/0.155
0,241
10,745 0.680
0.154
0.231
10.152 0.229
192
0.159
0.254
0.205 0.292
0.175 0.2633 0.169
0.255
0,755 0.683
0.179
0.254
0,.12
0.250
0.117
0.221 0.307
0.273
0,766 0.687
0.211
0.203
0.276
720
(0.219
0.308
0.258 0.335
[122%
0.309
0.237
0.313
0..744 0.696 (0.311 0,346
(0.289
0.337
Avg
[0.174 0.278
[0:218 0.303
187
0,77-/a
(12770
[0.765 0.686 (0.214 0.27%
[0.704
0.273
10.171
0.225
0.214
15.59
0.213
10.198 0.222
00.20
0.217 0.211
10.243 0.255 0.203 0.238
10.199 0.235
Weather
192
0.221
0.271
01210
0.260
0.215
0.266
0.208
0.256
0.247
0.265
0.271
0,259
01.44
0.251
0.2778 0.329 0..55 0.250
00.249
0.285
[1.274
0.311
0.2774
0.309
0.291
0.322
0.255
0.288
0.303
0.286
0.297 (1.274
00.306 0.346
(0:314
0.336
(0.302 0.327
720
0.356
0.370
0.418
0.405
0.415
0
0.400
0.405
0.397
0.373
0.354
0.373 0.35g
00.337 0.340
(0.350 0.374 00.997 0.396
(0,372 0.378
Avg
|0.256
0.294 0..655
0.297
10.270
0.300
25225
10.275
0.296 (0226 0.29 1 0.273
0.326 0.292
0.315
(0.279
Coum|
15
Datasct for pro-training not evaluated on corresponding models, which is denoted by dash (-).
Traflic from (PEMS) is generally used during the pre-truining of large models and thus not evaluated here.
Our model checkpoint is available at athtts://ugginggicc http:/aacccooottm/iieoeeoooeeeee coythuml/timee-busse buse-84mm
Table 14: Embedding ablation in TimeAttention. For the temporal dimension, we compare prevalent relative and absolute position embeddings. As for the variable dimension, we explore the effectiveness of the variable embedding that distinguishes endogenous and exogenous variables.
Traffic
Weather
Solar-Energy
ERAS-MS
Design
Temporal
Variable
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Timer-XL
RoPE
2024
with
0.340
0.238
0.157
0.205
0.162
0.221
0.164
0.307
ALiBi 2021
with 0.351
0.246
0.162
0.212
0.188
0.210
0.167
0.308
Replace
Relative
2002
with
0.361
0.250
0.163
0.214
0.197
0.215
0.168
0.309
Absolute
2017
with 0.381
0.270
0.159
0.207
0.171
0.204
0.165
0.306
RoPE
2024)
w/o 0.361
0.254
0.171
0.217
0.181
0.221
0.235
0.373
w/o
w/o
w/o
0.363
0.253
0.164
0.215
0.194
0.215
0.167
0.309
23

---

Published as a conference paper at ICLR 2025
Encoder-only
Decoder-only
Focused of each token
Encoderonly
Focused port taken
Decoderonly
Encoder-nly
Decodar-only
Focused each token
Encoder-only
Facused part of the last token
Decoder-only
Figure 11: Case studies of learned attention in encoder-/deccoderoonly Transformers.
degradation of PatchTST. Similar to the observations in Figure 3 the encoder-only architecture produces inferior predictions after thousands time points, which can be concealed due the short context adopted in previous benchmarks. Although PatchTST has conducted an initial exploration in the context of hundreds time points, inappropriately works in ever-long contexts. Therefore, we believe that context bottlenecks deserve further exploration in this community.
Table 15: Performance on ERA5 (pred- Iday). Lookback lengths vary from daily to yearly contexts.
Models
Timer-Xi XL
PatchTST MSE
DLinear MSE
Metric
MSE
MAE
MAE
MAE
Lookback-8 (1 Day)
0.0847
0.2100
0.0897
0.2196
0.0970
0.2276
Lookback-33 32 (4 Day)
0.0713
1.19
0.0778
0.2080
0.0841
0.2113
Lookback-56 (1 Week)
0.0688
0.1891
0.0785 0.2082
0.0814
0.2081
Lookback 224 (1 Month)
0.0675
0.1868
0.0745 0.2042
0.0788
0.2048
Lookback 960 (4 Month)
0.0667
0. 0.1863
0.1194 0.2696
0.0773
0.2031
Lookhack-2944 (1 Year)
0.0663
0.1857
0.1 0.1109 0.2638
0.0763
0.2024
Representation Analysis We further delve into long-context modeling from the perspective of learned representations. As shown in Figure [1 the decoder-only model can selectively focus on the previous context while PatchTST wrongly focuses noisy parts. Since causality is the basis of forecasting, using causal masks leads to coherent token embeddings, while the unmasked attention mechanism may break the causality and prevent the model from telling each tokens.
Normalization Section 4. has discussed instance normalization (Kim et al. 2021). It generally improves the performance of the previous encoder-only Transformers but leads special problems in decoder-only Transformers (e.g., unmatched statistics in multi-step autoregression) However, it is indicative that Timer-Xi XL without ReVIN can achieve competitive performance on well-acknowledged benchmarks in Table 16 while the performance of PStchTST may heavily rely on this normalization.
E.5 ILLUSTRATION OF TIMEATTENTION
Although the formulation to generalize from ID sequences to multivariate time series straightfor- ward, Timer-XL. XL is built decoder-only Transformer, underexploited backbone among current time series models. As shown in Figure [2, challenges lie capturing fine- meee-grined dependencies between all variables in the patch level, while maintaining temporal causality multiple sequences.
24

---

Published as a conference paper at ICLR 2025
Table 16: Evaluations (672-pred-96) the effect of ReVIN (Kim al. 2021
Transformers.
Models
Timer-XL with ReVIN
Timer-XL w/o ReVIN
PatchTST with ReVIN
PatchTST w/o ReVIN
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTh1
0.364
0.397
0.370
0.401
0.370 0.399
0.421
0.448
Weather
0.157
0.205
0.151
0.205
0.149 0.198
0.173
0.242
ECL
0.127
0.219
0.130
0.225
0.129 0.222
0.138
0.244
(a) Univariate
(b) Multivariate
Depadencies
Token Depedencies
21
31
5
Dapandencies
BTolers
Attention Map
Unmasked
Depandences
Time Series A
Time Series
Patch Tokens
Masked
Patch Tokens
C
Lookbock
Forecast
Lookback
Forecast
Map
Figure 12: Illustration of TimeAttention for modeling univariate and multivariate time series.
Technically, we introduce the masking formulation, whose key lies in the grouped causality of flattened 2D sequences. We derive it based on the Kronecker Product, which disentangles the large attention map into formalizable temporal and variable dependencies. It can be naturally extended to covariates pre-defined variable dependencies, which may inspire lot of future explorations.
LIMITATIONS
Timer-XL XL unified model for time series forecasting. can be used for task-specific training or scalable pre-training. handling varying-length and multivariate time series. As an autoregressive model, Timer-XI, necessitates iterative generation for long-term forecasting, which may lead to emor accumulation and inflexibility in the output length. In the future, we plan to incorporate multi- resolution patches for input and output series. Furthermore, given that Timer-XI. explicitly captures fine-grained token dependencies, there remains significant potential to reduce the complexity of Attenttnn particularly in high-dimensional and lengthy time series. Finally, we will investigate the factors contributing to the stagnation of Transformer performance in extremely long contexts, and seek insights in the time series modality to improve context efficiency.
25


Reviews:
[{"review_id": "I1nTRDjveL", "reviewer": "Reviewer_r6jB", "summary": "The paper proposes Timer-XL, a transformer decoder model for time series forecasting. Building upon the existing Timer model, Timer-XL extends the model with a longer context and a masking-based approach called TimeAttention to handle multivariate/covariate scenarios. Empirical results have been reported on univariate, multivariate and covariate experiments on some benchmark datasets.", "questions": "See above.\n\n- Which version of Moirai was used for the experiments? 1.0 or 1.1?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "a) The paper studies an interesting problem of long context modeling in the context of time series forecasting. Authors attempt to connect long context to scenarios beyond univariate modeling through next-token style modeling of multivariate and covariate-informed time series. \n\nb) Experiments have been conducted on many diverse settings although the experiments themselves have some limitations.", "weaknesses": "a) While the problem of long context modeling is interesting, the primary weakness of this work is the lack of clarity about the goal and a proper scope. The discussion is confusing and often only loosely relates to the long context setting which appears to be the primary goal. Authors claim that \"existing transformers in the time series field crucially encounter the context bottleneck\" which is not as critical of a problem as being portrayed here. Such claims require serious empirical justification which is missing from the paper (experiment 1 does not go too far, see below). In reality, long context scenarios may be helpful but _mostly_ in specific high-frequency scenarios. Consider a 5min granularity time series with weekly seasonal behavior. One would need a context larger than 2K to understand the seasonal behavior from the time series history. Such cases should have been better highlighted to justify the central claim of the paper. That said, long context univariate modeling may not always yield improvements. It may also worsen the accuracy, e.g., in the case of distribution shifts. \n\nComing to the utility of long context for multivariate/covariate-informed forecasting, this has been studied (although not explicitly) in Moirai. While the explicit perspective in this paper is interesting, the primary problem is not that one can do multivariate/covariate modeling through long context but that one needs to be able to do it in a zero-shot sense, as attempted in Moirai. As per my understanding, the settings being studied here are task-specific and not a single pretrained Timer-XL model being used for all experiments. Please correct me if I am wrong. \n\nb) The technical novelty of this work is limited in light of works such as Moirai. \"TimeAttention\" is a masking scheme that extends causal univariate patch-based modeling to the multivariate setting.\n\nc) The empirical results lack comprehensiveness and are not particularly strong.\n\n**Experiment 4.1**: The benchmark is fairly small to draw conclusions. Even in these scenarios, the benchmark shows that long context yields to diminishing returns (or worse performance) beyond a point, 1 month in most cases. 1 month @ 1h granularity amounts to a context length of 720 which is not far off from the context length of many time series models (Chronos, Moirai, TimesFM, etc.). Furthermore, I don't understand why PatchTST is the only baseline being studied here. I also don't quite understand what's unique about Timer-XL here that yields better performance over PatchTST. Could it be the larger number of parameters? \n\nMinor: The analysis on normalization does not belong to this section and brings limited value to the discussion. Normalization mostly helps when time series in a dataset have drastically different scales. Models working fine without normalization for a single task of land-surface temperature forecasting is not a surprising result.  \n\n**Experiment 4.2**: The results in Table 2 and 4 are not strong when compared to pretrained baselines such as Moirai. Why is Moirai missing from Fig 4? \n\nMinor: DeepAR and N-BEATS are not numerical simulation based methods. \n\n**Experiment 4.3**: More baselines that can incorporate covariates are needed before conclusions can be drawn. For example, you can consider adding N-BEATSx, NHITS, DeepAR, etc. GluonTS also provides an implementation of PatchTST which can incorporate covariates. \n\n**Experiment 4.4**: The benchmark selected in this experiment for out of domain generalization is severely limited. 5/7 datasets belong to the same domain and with 4 being essentially the same dataset (ETTh1, ETTh2, ETTm1, ETTm2). This is not enough to draw conclusions about a \"pretrained model\". Please check the benchmarks in the Chronos or TimesFM papers. \n\nd) The model only enables point predictions. In forecasting, one is often interested in the entire distribution of future possibilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7jSaM7SC1f", "reviewer": "Reviewer_MkP7", "summary": "This paper proposes Timer-XL, a generative Transformer model for uni & mul-variate time series forecasting, which addresses the limitation of short context lengths in existing time series Transformers and allows the model to capture both intra- and inter-series dependencies. Specifically, the authors design TimeAttention, which incorporates relative position embeddings and causal masking to effectively learn temporal dependencies. Evaluations across various benchmarks demonstrate that Timer-XL achieves state-of-the-art performance in univariate, multivariate, and covariate-informed forecasting tasks.", "questions": "Please see the weaknesses. \nAlso, it will be good if the authors can use a figure to illustrate patch token for univariate and multivariate data.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes multivariate next token prediction for time series forecasting. This paradigm unifies univariate, multivariate, and covariate-informed forecasting, by treating them as a long-context generation problem.\n2. This paper introduces TimeAttention, a novel self-attention mechanism for time series data. TimeAttention captures fine-grained intra and inter-series dependencies, preserves causality in forecasting, and incorporates position embeddings.\n3. Experiments show that extending the context length generally improves accuracy, highlighting the context bottleneck issue.", "weaknesses": "1. This paper focuses heavily on comparing Timer-XL with other Transformer models, particularly PatchTST, Timer, and lacks a broader comparison with other non-Transformer time series forecasting models. Also, how is Timer-XL compared with some recent LLM-based models? e.g., [1], [2].\n2. This paper doesn't extensively discuss the computational cost of Timer-XL. Though it provides a theoretical derivation, a more detailed analysis of the computational resources required, especially when handling high-dimensional time series with long contexts. E.g., how about the number of parameters or training/inference time of Timer-XL as compared with existing solutions. \n3. The extension from univriate to multivariate seems to be an over-simplied way, in that using RoPE to as a positional embedding. Take traffic data as an example, how can RoPE reflect the spatial correlations among traffic network?\n4. The authors emphasize long context for time series forecasting, however, for some domains, it may not be necessary for such a long context, e.g., traffic data with periodicities. This is also shown in Figure 3.  Also, one can define a large patch token (more time points). In this way, can the context length also be shortened?\n\n[1] Time-LLM: Time Series Forecasting by Reprogramming Large Language Models, ICLR 2024\n\n[2] Large Language Models Are Zero-Shot Time Series Forecasters, NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SRe2oQHENd", "reviewer": "Reviewer_V4mu", "summary": "This paper introduces Timer-XL, a method for unified time series forecasting, which addresses challenges in hard to uniformly predict 1D and 2D time series. It utilizes generative Transformer with relative position embedding to capture the correlations between the temporal and variable dimensions. Also, it proposes a novel TimeAttention to capture causal patch-wise dependencies within and among all variables. Experiments demonstrate that Timer-XL achieves state-of-the-art performance across challenging forecasting benchmarks through a unified approach,", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript demonstrates a high level of completeness.\n2. It provides an effective large-scale framework for advancing large models in time series analysis.\n3. The empirical evaluation is comprehensive and promising results are shown.", "weaknesses": "1. My biggest question lies with Equations 4, 5, 7, and 8. Since Timer-XL is a unified time series framework, these equations include processing steps that sort each time series. For example, with 𝑁 time series, does the order after flattening the sequences significantly affect the causal relationship in Equations 4 and 5? The same question applies to Equations 7 and 8.\n2. As a unified time series framework, the author needs to compare it with Moirai in a zero-shot scenario. In Figure 5 of the manuscript, a corresponding comparison experiment should be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)