Timer-XL: Long-Context Transformers for Unified Time Series Forecasting

## Abstract

Abstract We present Timer-XL, a generative Transformer for unified time series forecasting. To uniformly predict 1D and 2D time series, we generalize next token prediction, predominantly adopted for causal generation of 1D sequences, to multivariate next token prediction . The proposed paradigm uniformly formulates various forecasting scenarios as a long-context generation problem. We opt for the generative Transformer, which can capture global-range and causal dependencies while providing contextual flexibility, to implement unified forecasting on univariate series characterized by non-stationarity, multivariate time series with complicated dynamics and correlations, and covariate-informed contexts that include both endogenous and exogenous variables. Technically, we propose a universal TimeAttention to facilitate generative Transformers on time series, which can effectively capture fine-grained intra- and inter-series dependencies of flattened time series tokens (patches) and is further strengthened by position embeddings in both temporal and variable dimensions. Timer-XL achieves state-of-the-art performance across challenging forecasting benchmarks through a unified approach. As a large time series model, it demonstrates notable model transferability by large-scale pre-training, as well as contextual flexibility in token lengths, positioning it as a one-for-all forecaster.

## 1 Introduction

Transformers have contributed significantly to the fields of natural language and computer vision (Radford et al., ; Dosovitskiy et al., ), and been extensively applied in time series forecasting, becoming the foundation of specialized forecasters (Zhou et al., ; Wu et al., ) and large models (Das et al., ). As a typical generative task, the quality of predictions heavily relies on the context (Dai et al., ). Reliable predictions are made by thoroughly considering endogenous temporal variations and retrieving relevant exogenous correlations into the context (Box, ). Further, the context length of training, which serves as an indicator of scaling (Kaplan et al., ), determines the maximum input and output of generative Transformers, ultimately enabling long-sequence, high-resolution, and high-frequency generation (Yin et al., ; Wang et al., ).

However, existing Transformers in the time series field crucially encounter the context bottleneck. As shown in Figure 1, unlike Transformers for natural language and vision that learn dependencies among thousands to millions of tokens (Kirillov et al., ; OpenAI, ), time-series Transformers typically work around limited contexts of up to hundreds of time series tokens (patches) (Nie et al., ). For univariate time series, a short context length leads to an insufficient perception of global tendencies, overlooking widespread non-stationarity in real-world time series (Hyndman, ). The excessive reliance on stationarization, such as normalization (Kim et al., ), restricts the model capacity and leads to overfitting of Transformers (Liu et al., ). Moreover, instead of regarding multivariate time series as independent channels (Nie et al., ), increasing Transformers explicitly capture intra- and inter-channel dependencies (Zhang & Yan, ; Liu et al., ; ), leading to an urgency to increase the context length to encompass inter-correlated variables.

Recently, generative Transformers, which present a predominant scalable choice of large language models (Zhao et al., ) characterized by the decoder-only architecture, have gained increasing attention in the development of large time series models (Rasul et al., ; Ansari et al., ) due to their generalization performance and contextual flexibility, that is, one Transformer accommodates all input lengths during inference (Liu et al., ). Therefore, training on longer contexts not only empowers them with the fundamental capability to incorporate more contextual information but also enhances the model versatility toward a one-for-all foundation model, which regards any-variate and any-length time series as one context. Even if previous work has achieved unified modeling on flattened tokens in encoder-only Transformers (Woo et al., ), the full dependencies among tokens are incompatible with the causality of forecasting, supported by our experiments revealing that their supervised and generalization performance is still inferior to those of decoder-only ones.

In pursuit of unified time series forecasting, we propose *multivariate next token prediction*. It unifies forecasting tasks of Figure 1 into the patch-level generation based on long-context sequences. To fully leverage the global-range modeling ability and contextual flexibility of generative Transformers, we develop *TimeAttention* that learns causal temporal dependencies under channel dependence, elegantly implemented as a masking mechanism without modifying self-attention. With incorporated relative position embeddings for multivariate series, TimeAttention is further enhanced with chronological awareness of time points and the distinction of variables. We propose *Timer-XL* as the extra long version of a generative time-series Transformer (Timer) (Liu et al., ). We firstly enlarge the context to thousands of tokens and achieve state-of-the-art in univariate, multivariate, and covariate-informed forecasting. With notable contextual flexibility, Timer-XL is generalizable across different temporal dynamics, variables, and datasets. Our contributions lie in three aspects:

- •
We propose multivariate next token prediction and unified time series forecasting, strengthening Transformers on enriched forecasting contexts to make reliable predictions.
- •
We introduce TimeAttention, a novel causal self-attention tailored for the time series modality on our proposed paradigm, which enables intra- and inter-series modeling with position perception, and maintains the causality and flexibility of generative Transformers.
- •
We present Timer-XL that achieves state-of-the-art in well-acknowledged benchmarks. We establish new benchmarks, conduct large-scale pre-training, and provide extensive analyses, demonstrating that long-context Transformers are one-for-all forecasters.

Figure: Figure 1: We compare the context length (measured in the number of tokens) of Transformers in different modalities and propose Timer-XL that increases the length to thousands of patch tokens. Given the generality across contexts, Timer-XL is a versatile solution for various forecasting tasks.
Refer to caption: x1.png

## 2 Related Work

Transformers (Vaswani et al., ) for time series forecasting have undergone rapid advancements.

Based on the global-range modeling ability of tokens, Transformers have shown great power in time series, rapidly standing out from RNN-, CNN- and MLP-forecasters, especially on long sequences. Initial Transformer-based forecasters primarily focused on *long-term* prediction, aiming to extend the forecasting horizon while mitigating the quadratic computational growth associated with increasing sequence length (Li et al., ; Zhou et al., ; Wu et al., ). However, the context (lookback) length of previous models is not growing in pace, which hinders Transformers from making fully informed predictions, producing oversmooth results (Liu et al., ). Meanwhile, another advancement has occurred from univariate to multivariate forecasting. Unlike natural language, time series are high-dimensional and inherently correlated (Hyndman, ), further requiring large contexts that contain relevant endogenous and exogenous variables. To effectively exploit the intra- and inter-series dependencies, tokenization of Transformers has been extensively developed in temporal-wise (Lim et al., ), patch-wise (Nie et al., ), and variable-wise (Liu et al., ) approaches, with deftly designed Transformers for inter-series modeling (Zhang & Yan, ; Wang et al., ).

Despite the prevalent issue of insufficient context in the current situation, few works highlight that these challenges can be uniformly tackled by *long-context* Transformers. Consequently, we leverage the fundamental sequence modeling capability of the vanilla Transformer, unify various forecasting tasks into multivariate next token prediction, and significantly extend the context length.

Besides, time-series Transformers have experienced an evolution from small scenario-specific models to large foundation models (Das et al., ; Woo et al., ; Ansari et al., ). Among them, decoder-only Transformers, predominantly adopted as the backbone of large language models (Touvron et al., ; OpenAI, ), have positioned as a scalable and generalizable choice for general time series analysis (Liu et al., ). By independently predicting each token based on preceding contexts, decoder-only models are also multi-length forecasters (Liu et al., ), avoiding resource-intensive training and storage. However, existing decoder-only forecasters are typically trained on a single channel (Nie et al., ), making them inaccessible to inter-series dependencies.

Prior work has employed encoder-only Transformers to fully capture dependencies within 2D time series (Woo et al., ). The incompatibility between this architecture and next token prediction constrains the flexibility and performance of Transformers. To date, the implementation of next token prediction and multivariate time series forecasting in a single Transformer remains a fundamental challenge. We cope with this issue by incorporating the variable dependency and temporal causality into a fine-grained attention mask, thereby leveraging contextual flexibility and capturing inter-series dependencies simultaneously without altering the self-attention mechanism.

## 3 Approach

In this section, we first introduce a generative Transformer to illustrate the procedure of next token prediction on 1D time series. As an extension, we design *TimeAttention* and propose *Timer-XL* for unified time series forecasting. It is applicable to univariate, multivariate, and covariate-informed scenarios by generalizing the context from 1D sequences to 2D time series.

### 3.1 Timer

Timer (Liu et al., ) is a generative Transformer trained by next token prediction (Bengio et al., ), which regards single-dimensional time series as non-overlapping patch tokens.

#### Next Token Prediction

Given an univariate time series $\mathbf{X}=\{x_{1},\dots,x_{TP}\}$ of length $TP$, a time series token is defined as $P$ consecutive time points, also termed as the *patch token*:

$$ $\mathbf{x}_{i}=\{x_{(i-1)P+1},\dots,x_{iP}\}\in\mathbb{R}^{P},\ i=1,\dots,T.$ (1) $$

The training objective is to independently predict the next patch token to maximize the likelihood:

$$ $P(\mathcal{\mathbf{X}})=\prod_{i=1}^{T}p(\mathbf{x}_{i+1}|\mathbf{x}_{\leq i}),$ (2) $$

which is realized by a decoder-only architecture with the block number $L$ and model dimension $D$:

$$ $\displaystyle\mathbf{h}_{i}^{0}$ $\displaystyle=\mathbf{W}_{e}\mathbf{x}_{i},\ i=1,\dots,T,$ (3) $\displaystyle\mathbf{H}^{l}$ $\displaystyle=\operatorname{TrmBlock}(\mathbf{H}^{l-1}),\ l=1,\dots,L,$ $\displaystyle\{\hat{\mathbf{x}}_{i+1}\}$ $\displaystyle=\mathbf{H}^{L}\mathbf{W}_{d},\ i=1,\dots,T.$ $$

For simplicity, we omit the block index $l$. Timer adopts $\mathbf{W}_{e},\mathbf{W}_{d}\in\mathbb{R}^{D\times P}$ that independently embed and project the token embeddings as $\mathbf{H}=\{\mathbf{h}_{i}\}\in\mathbb{R}^{T\times D}$. $\operatorname{TrmBlock}$ includes feed-forward network and self-attention with the temporal causal mask $\mathcal{T}\in\mathbb{R}^{T\times T}$. $\mathbf{h}_{i}\in\mathbb{R}^{D}$ is the contextual representation of the previous $i$ tokens. Predicted $\hat{\mathbf{x}}_{i+1}$ are supervised with ground truth by MSE loss.

### 3.2 Generalize 1D Sequences to 2D Time Series

For the enlarged context with the additional dimension, our proposed attention mechanism aims to (1) thoroughly capture intra- and inter-series dependencies and (2) preserve causality within the temporal dimension. Without loss of generality, we illustrate this with the case of multivariate forecasting.

#### Multivariate Next Token Prediction

Given a multivariate time series $\mathbf{X}\in\mathbb{R}^{N\times TP}$ with the number of variables $N$, the time series token $\mathbf{x}_{m,i}$ is defined as the $i$-th patch of the $m$-th variable:

$$ $\mathbf{x}_{m,i}=\{\mathbf{X}_{m,(i-1)P+1},\dots,\mathbf{X}_{m,iP}\}\in\mathbb {R}^{P},\ m=1,\dots,N,\ i=1,\dots,T.$ (4) $$

The training objective is still to independently predict the next token. Unlike before, each prediction is made based on tokens of the previous time ($\leq i$) from all $N$ variables:

$$ $P(\mathcal{\mathbf{X}})=\prod_{m=1}^{N}\prod_{i=1}^{T}p(\mathbf{x}_{m,i+1}| \mathbf{x}_{:,\leq i})=\prod_{m=1}^{N}\prod_{i=1}^{T}p(\mathbf{x}_{m,i+1}| \mathbf{x}_{1,\leq i},\dots,\mathbf{x}_{N,\leq i}).$ (5) $$

Compared with Equation 2, the multivariate context length increases from $T$ to $NT$. By contrast, the benefit is that this paradigm learns causal dependencies within each sequence while incorporating fine-grained variable correlations from other sequences, making it a universal forecasting mechanism and outperform channel independence and coarse-grained variable-wise modeling experimentally.

Technically, we still adopt the token embedding $\mathbf{W}_{e}\in\mathbb{R}^{D\times P}$ to obtain the patch-wise representation $\mathbf{h}_{m,i}\in\mathbb{R}^{D}$, which will encompass contextual information from $Ni$ tokens through Transformer blocks and be eventually projected by $\mathbf{W}_{d}\in\mathbb{R}^{D\times P}$ into the predicted patch token $\hat{\mathbf{x}}_{m,i+1}$.

#### Position Embedding

Position embedding has not been sufficiently explored in time-series Transformers. To address the permutation-invariance inherent in Transformers, the attention mechanism should reflect the relative position (chronological information) of tokens that vary on the temporal dimension. Conversely, shuffling the input order of variables should not affect anything other than the output order of variables, necessitating that variate embedding should be permutation-invariant.

To meet the above requirements, we adopt RoPE (Su et al., ), a widely utilized relative position embedding in LLMs, as the temporal embedding. In terms of the variable dimension, we use two learnable scalars to keep the reflexivity of variables (Woo et al., ). Beyond simply incorporating them together, we provide detailed ablations in Section 4.5 to demonstrate the effectiveness:

$$ $\mathcal{A}_{mn,ij}=\mathbf{h}_{m,i}^{\top}\mathbf{W}_{q}\mathbf{R}_{\theta,i- j}\mathbf{W}_{k}^{\top}\mathbf{h}_{n,j}+u\cdot\mathds{1}{(}{m=n})+v\cdot \mathds{1}{(}{m\neq n}),$ (6) $$

where $\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}\in\mathbb{R}^{D\times d_{k}}$ and $d_{k}$ is the dimension of the query, key, and value. $\mathbf{R}_{\theta,t}\in\mathbb{R}^{d_{k}\times d_{k}}$ is the rotary matrix with rotation degree $t\cdot\theta$, $\mathds{1}{(}\cdot)$ is the indicator function, and $u,v\in\mathbb{R}$ are learnable parameters for the token to distinguish its endogenous and exogenous variables.

Figure: Figure 2: Illustration of TimeAttention. For univariate series, temporal mask $\mathcal{T}$ keeps the causality. Given multivariate time series tokenized in a temporal-first order, we adopt the variable dependency $\mathcal{C}$, an all-one matrix, as the left-operand of Kronecker product, expanding temporal mask to a block matrix, which exactly reflects dependencies of multivariate next token prediction. The formulation is also generalizable to univariate and covariate-informed contexts with pre-defined variable dependency.
Refer to caption: x2.png

#### TimeAttention

In contrast to variable-wise (Liu et al., ) and non-causal patch-wise tokens (Nie et al., ; Woo et al., ), our TimeAttention can capture causal patch-wise dependencies within and among all variables. We mask out non-causal dependencies in the calculated attention map $\mathcal{A}$. Before masking, we reorder the tokens by flattening their 2D indices in the temporal-first manner, which is illustrated in the upper left of Figure 2.

We provide an intuitive example to illustrate the causal dependencies within multivariate time series: considering the 2nd token of time series A. To predict its next token, its representation $\mathbf{h}$ should be exactly dependent on the tokens-$\{1,2,4,5\}$. Similarly, we provide the causal dependencies of each token in the upper middle of Figure 2. Starting from the observation of coarse-grained variable dependency, where all variables are inter-correlated in this multivariate task, we find the fine-grained masking on $\mathcal{A}$ can be formally derived as the Kronecker product of two matrices: the adjacency matrix of the variable dependency graph as $\mathcal{C}\in\mathbb{R}^{N\times N}$, and the causal temporal mask $\mathcal{T}\in\mathbb{R}^{T\times T}$:

$$ $\mathcal{T}_{i,j}=\begin{cases}1&\text{ if }j\leq i,\\ 0&\text{ otherwise},\end{cases}\ \mathcal{C}_{m,n}=\begin{cases}1&\text{ if variable $m$ is dependent on $n$},\\ 0&\text{ otherwise}.\end{cases}$ (7) $$

Let the Kronecker product $\otimes:(\mathbb{R}^{N\times N},\mathbb{R}^{T\times T})\mapsto\mathbb{R}^{NT
\times NT}$ take two matrices and produce a block matrix. Consequently, TimeAttention is formulated as follows:

$$ $\operatorname{TimeAttention}(\mathbf{H})=\operatorname{Softmax}\left(\frac{ \operatorname{Mask}(\mathcal{C}\otimes\mathcal{T})+\mathcal{A}}{\sqrt{d_{k}}} \right)\mathbf{H}\mathbf{W}_{v},\ \operatorname{Mask}(\mathcal{M})=\begin{ cases}0&\text{if }\mathcal{M}_{i,j}=1,\\ -\infty&\text{if }\mathcal{M}_{i,j}=0.\end{cases}$ (8) $$

As before, token representations in $\mathbf{H}=\{\mathbf{h}_{m,i}\}\in\mathbb{R}^{NT\times D}$ will be independently processed by feed-forward network and layer normalization, and fed into the next Transformer block.

#### Unified Time Series Forecasting

In multivariate forecasting, the variable dependency forms the complete graph, presenting an all-one matrix $\mathcal{C}$. By generalizing TimeAttention on multiple sequences, generative Transformers can leverage contextual flexibility to encompass relevant covariates as well. In this case, Timer-XL is adapted in two steps: (1) formulate the customized variable dependency as $\mathcal{C}$ and (2) optimize the model using the supervision of target variables. An example (target-$A$-covariate-$B$) of attention is illustrated on the right of Figure 2. In a nutshell, we introduce position embedding and extend the context length of generative Transformers. Toward unified time series forecasting, we flatten 2D time series into a long context and capture fine-grained causal token dependencies.

## 4 Experiments

We conduct thorough evaluations of the performance and generality of Timer-XL. Given the saturated performance on previous becnchmarks (Wu et al., ; Nie et al., ; Liu et al., ), where Transformers tend to overfit on these tasks due to their data-hungry nature (Touvron et al., ), we establish new long-context benchmarks and will release our datasets for the advancement of this field. Detailed datasets, baseline models, and experimental configurations are provided in Appendix B. We also dive into commonly adopted techniques, such as channel independence (Nie et al., ) and normalization (Kim et al., ). We conclude that long-context Transformers can tackle these challenges without any task-specific designs.

### 4.1 Univariate Time Series Forecasting

#### Setups

Due to the insufficient dataset length when extending contexts in univariate datasets (Makridakis et al., ), we adopt wide-recognized benchmarks from Liu et al. (). Although these datasets are originally multivariate, they will be predicted in a univariate approach with the implementation of channel independence. Different from the previous long-term forecasting setting, we focus on reliable prediction based on a long context. Thus, we fix the prediction horizon and increase the lookback length to monthly and yearly levels. We also establish a long-context univariate benchmark based on the challenging 40-year ECMWF Reanalysis v5 dataset (Hersbach et al., ), where yearly contexts are adopted to predict the land-surface temperature of a single site (ERA5-S).

Figure: Figure 3: Univariate forecasting (pred-$96$) of well-acknowledged benchmarks under channel independence (Nie et al., ). We increase the lookback length to encompass monthly and yearly contexts.
Refer to caption: x3.png

#### Results

As shown in Figure 3, the accuracy of univariate prediction can be generally improved by extending the daily context to monthly, thus emphasizing widespread context bottleneck in this field. We compare Timer-XL with PatchTST, where Timer-XL can surpass the state-of-the-art encoder-only Transformer when the context length persistently increases. It demonstrates that Timer-XL is proficient in handling extremely long contexts, and thus achieves breakthrough performance. It is also notable that the performance of monthly and yearly contexts improves slowly or even deteriorates, termed *context oversatuation*. It may stem from increased noise and training difficulty, which leaves a future direction to improve the context efficiency. Table 1 provide results on ERA5-S. Timer-XL consistently outperforms PatchTST on all sites, which can be credited to the maintenance of causality and token-wise supervision in the decoder-only architecture.

#### Analysis

Furthermore, we analyze the widespread non-stationary challenge in univariate tasks. It is commonly tackled by normalization (Kim et al., ) that greatly improves Transformer performance in previous benchmarks. However, we find it may be caused by the insufficient time span and training samples in these datasets. Thus, normalization enriches training samples by aligning time series with different means and variances to the same distribution. Instead, it makes Transformer constrained on the temporal variation within windows, preventing them from learning variations among windows and resulting in oversmooth predictions and failures in long contexts. In Table 1, we evaluate the performance on ERA5-S that includes sufficient samples. It validate the claim that long-context Transformers can achieve better results even without normalization.

**Table 1: Univariate forecasting (input-$3072$-pred-$96$) of ERA5-S ($40$ years), encompassing $117$k time points in each station. We evaluate PatchTST and Timer-XL with and without normalization (Kim et al., ). *+ Norm.* indicates using the normalization. We train one model for each site separately.**
| Station | Beijing | Hongkong | London | New York | Paris | Seoul | Shanghai | Average |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| PatchTST | 0.0791 | 0.221 | 0.189 | 0.327 | 0.277 | 0.415 | 0.186 | 0.334 | 0.266 | 0.407 | 0.0940 | 0.238 | 0.137 | 0.289 | 0.175 | 0.319 |
| + Norm. | 0.0797 | 0.220 | 0.191 | 0.323 | 0.281 | 0.419 | 0.184 | 0.334 | 0.272 | 0.411 | 0.0914 | 0.233 | 0.136 | 0.287 | 0.176 | 0.319 |
| Timer-XL | 0.0739 | 0.210 | 0.179 | 0.316 | 0.262 | 0.404 | 0.182 | 0.327 | 0.254 | 0.399 | 0.0901 | 0.229 | 0.134 | 0.282 | 0.168 | 0.310 |
| + Norm. | 0.0742 | 0.210 | 0.183 | 0.317 | 0.278 | 0.418 | 0.181 | 0.330 | 0.264 | 0.407 | 0.0896 | 0.227 | 0.133 | 0.281 | 0.172 | 0.313 |

### 4.2 Multivariate Time Series Forecasting

#### Setups

We follow iTransformer (Liu et al., ) to evaluate multivariate forecasting performance. Toward a one-for-all forecaster, we also evaluate rolling forecast performance, that is, we trained one model for all prediction horizons by integrating the previous prediction into the lookback window in the next iteration. We further establish long-context multivariate forecasting benchmarks: ERA5 multi-station land-surface temperature prediction (ERA5-MS), and the global temperature and wind speed forecasting challenge (GTWSF) (Wu et al., ), to learn complex temporal dynamics and variable correlations with sufficient training samples.

#### Results

As shown in Tables 2-4 and Figure 4, Timer-XL achieves the best results on both previous and new benchmarks. Essentially, Transformers that explicitly capture inter-series dependencies, such as Moirai (Woo et al., ) and iTransformer, reasonably achieve decent performance in Table 2. Beyond iTransformer, Timer-XL can model fine-grained patch-wise temporal dependencies. With TimeAttention, Timer-XL outperforms Timer especially on high-dimensional time series ($13.2$% in ECL and $6.3$% in Traffic, with thousands of tokens in the context). Compared with the encoder-only Moirai, decoder-only Transformers excel at generalizing across varying prediction lengths in Table 3.

Figure: Figure 4: Multivariate forecasting of GTWSF ($2$-day-pred-1-day), involving 3850 worldwide stations spanning two years. Results of the baseline models are officially reported by Ding et al. ().
Refer to caption: x4.png

**Table 2: Multivariate forecasting ($96$-pred-$96$) of well-acknowledged benchmarks. Timer and Moirai are reproduced for evaluation. Results of other baselines are officially reported by Liu et al. ().**
| Models | Timer-XL | Timer | Moirai | iTransformer | DLinear | PatchTST | TimesNet | Stationary | Autoformer |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (Ours) | () | () | () | () | () | () | () | () |  |  |  |  |  |  |  |  |  |  |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ECL | 0.138 | 0.233 | 0.159 | 0.244 | 0.139 | 0.235 | 0.148 | 0.240 | 0.197 | 0.282 | 0.181 | 0.270 | 0.168 | 0.272 | 0.169 | 0.273 | 0.201 | 0.317 |
| ETTh1 | 0.381 | 0.399 | 0.386 | 0.401 | 0.385 | 0.402 | 0.386 | 0.405 | 0.386 | 0.400 | 0.414 | 0.419 | 0.384 | 0.402 | 0.513 | 0.491 | 0.449 | 0.459 |
| Traffic | 0.387 | 0.260 | 0.413 | 0.265 | 0.389 | 0.265 | 0.395 | 0.268 | 0.650 | 0.396 | 0.462 | 0.295 | 0.593 | 0.321 | 0.612 | 0.338 | 0.613 | 0.388 |
| Weather | 0.165 | 0.209 | 0.176 | 0.215 | 0.165 | 0.210 | 0.174 | 0.214 | 0.196 | 0.255 | 0.177 | 0.218 | 0.172 | 0.220 | 0.173 | 0.223 | 0.266 | 0.336 |
| Solar-Energy | 0.200 | 0.229 | 0.204 | 0.234 | 0.203 | 0.232 | 0.203 | 0.237 | 0.290 | 0.378 | 0.234 | 0.286 | 0.250 | 0.292 | 0.215 | 0.249 | 0.884 | 0.711 |

**Table 3: Multivariate forecasting ($672$-pred-$\{96,192,336,720\}$) of well-acknowledged benchmarks. We evaluate one-for-all forecasters following Liu et al. (): rolling forecasting for four forecast lengths with one model. Averaged results are reported here and full results are provided in Table 11.**
| Models | Timer-XL | Timer | Moirai | iTransformer | DLinear | PatchTST | TimesNet | Stationary | Autoformer |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (Ours) | () | () | () | () | () | () | () | () |  |  |  |  |  |  |  |  |  |  |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ECL | 0.155 | 0.246 | 0.161 | 0.251 | 0.163 | 0.257 | 0.164 | 0.258 | 0.165 | 0.265 | 0.169 | 0.268 | 0.201 | 0.303 | 0.265 | 0.358 | 0.289 | 0.379 |
| ETTh1 | 0.409 | 0.430 | 0.418 | 0.436 | 0.429 | 0.447 | 0.421 | 0.445 | 0.426 | 0.444 | 0.412 | 0.435 | 0.495 | 0.491 | 0.505 | 0.513 | 0.517 | 0.528 |
| Traffic | 0.374 | 0.255 | 0.384 | 0.259 | 0.385 | 0.265 | 0.384 | 0.274 | 0.423 | 0.298 | 0.391 | 0.275 | 0.602 | 0.322 | 0.630 | 0.347 | 0.684 | 0.433 |
| Weather | 0.240 | 0.273 | 0.232 | 0.270 | 0.231 | 0.272 | 0.266 | 0.291 | 0.239 | 0.291 | 0.226 | 0.268 | 0.264 | 0.293 | 0.308 | 0.329 | 0.435 | 0.455 |
| Solar-Energy | 0.198 | 0.249 | 0.233 | 0.249 | 0.241 | 0.275 | 0.213 | 0.291 | 0.222 | 0.283 | 0.202 | 0.269 | 0.213 | 0.295 | 0.254 | 0.315 | 0.265 | 0.325 |

#### Analysis

Patching (Nie et al., ) has been demonstrated as an effective tokenization approach for the time series modality, leading to the boom of Transformers in supervised deep forecasters and large time series models. To better cope with multivariate time series forecasting, we compared these Transformers on ERA5-MS to answer the following questions: (1) whether to conduct explicit inter-series modeling or not (channel independence) and (2) whether to use decoder-only or encoder-only Transformers. The combination presents four typical Transformers in Table 4, which shows that Timer-XL combines the advantages of explicit inter-series modeling and the decoder-only architecture, which is suitable for multivariate time series forecasting with arbitrary prediction horizons.

### 4.3 Covariate-Informed Time Series Forecasting

**Table 4: Multivariate forecasting (input-$3072$-pred-$96$) of ERA5-MS (40 years and 7 stations). We fairly evaluate Transformers that adopt patched time series. *CI.* indicates whether the Transformer uses channel independence (Nie et al., ). *Arch.* categorizes them into the encoder-only (E) and decoder-only (D) architectures. Different from ERA5-S in Table 1, we train one model for all sites.**
| Station | Beijing | Hongkong | London | New York | Paris | Seoul | Shanghai | Average |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | CI. | Arch. | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| PatchTST | Yes | E | 0.0815 | 0.222 | 0.190 | 0.326 | 0.275 | 0.414 | 0.185 | 0.333 | 0.265 | 0.407 | 0.0977 | 0.240 | 0.139 | 0.290 | 0.176 | 0.319 |
| Moirai | No | E | 0.0753 | 0.213 | 0.179 | 0.318 | 0.269 | 0.410 | 0.185 | 0.330 | 0.256 | 0.401 | 0.0901 | 0.230 | 0.135 | 0.284 | 0.170 | 0.312 |
| Timer | Yes | D | 0.0734 | 0.210 | 0.182 | 0.319 | 0.268 | 0.407 | 0.183 | 0.329 | 0.255 | 0.399 | 0.0877 | 0.226 | 0.132 | 0.281 | 0.169 | 0.310 |
| Timer-XL | No | D | 0.0736 | 0.209 | 0.174 | 0.309 | 0.263 | 0.404 | 0.182 | 0.327 | 0.252 | 0.396 | 0.0872 | 0.225 | 0.130 | 0.278 | 0.166 | 0.307 |

#### Setups

For the covariate-informed forecasting, we adopt the well-acknowledged electricity price forecasting (EPF) task (Lago et al., ). Each subset contains electricity price as the endogenous variable and two exogenous variables. Therefore, the variable dependency for Timer-XL is formulated as $\mathcal{C}=[[1,1,1],[0,1,0],[0,0,1]]$. To investigate whether to learn causal or noncausal patch-wise dependencies in covariates, we implement two versions of Timer-XL: the original one with temporal causal mask $\mathcal{T}$, and the noncausal one with $\mathcal{T}$ replaced by an all-one matrix.

#### Results

As shown in Table 5, Timer-XL outperforms state-of-the-art models in covariate-informed tasks. Compared with TimeXer (Wang et al., ), which treats an entire covariate as a token, Timer-XL learns fine-grained patch-wise dependencies. By the noncausal version of Timer-XL, we surprisingly find consistent conclusions with endogenous variables: results will be better if Timer-XL learns causal dependencies within exogenous variables. It again validates that next token prediction that maintains causality has a higher upper limit of performance.

**Table 5: Covariate-informed forecasting ($168$-pred-$24$) of EPF. We implement two versions of Timer-XL: *Noncausal* indicates that we do not maintain the causality within covariates by replacing temporal causal mask with all-one matrix. Results of baselines are officially reported by Wang et al. ().**
| Models | Timer-XL | Timer-XL | TimeXer | iTransformer | DLinear | PatchTST | Crossformer | TimesNet | Autoformer |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (Ours) | (Noncausal) | () | () | () | () | () | () | () |  |  |  |  |  |  |  |  |  |  |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| NP | 0.234 | 0.262 | 0.237 | 0.265 | 0.238 | 0.268 | 0.265 | 0.300 | 0.309 | 0.321 | 0.267 | 0.284 | 0.245 | 0.289 | 0.250 | 0.289 | 0.402 | 0.398 |
| PJM | 0.089 | 0.187 | 0.092 | 0.188 | 0.088 | 0.188 | 0.097 | 0.197 | 0.108 | 0.215 | 0.106 | 0.209 | 0.149 | 0.198 | 0.097 | 0.195 | 0.168 | 0.267 |
| BE | 0.371 | 0.243 | 0.410 | 0.279 | 0.379 | 0.243 | 0.394 | 0.270 | 0.463 | 0.313 | 0.403 | 0.264 | 0.436 | 0.294 | 0.419 | 0.288 | 0.500 | 0.333 |
| FR | 0.381 | 0.204 | 0.406 | 0.220 | 0.384 | 0.208 | 0.439 | 0.233 | 0.429 | 0.260 | 0.411 | 0.220 | 0.440 | 0.216 | 0.431 | 0.234 | 0.519 | 0.295 |
| DE | 0.434 | 0.415 | 0.435 | 0.415 | 0.440 | 0.418 | 0.479 | 0.443 | 0.520 | 0.463 | 0.461 | 0.432 | 0.540 | 0.423 | 0.502 | 0.446 | 0.674 | 0.544 |
| Average | 0.302 | 0.262 | 0.316 | 0.273 | 0.306 | 0.265 | 0.335 | 0.289 | 0.366 | 0.314 | 0.330 | 0.282 | 0.362 | 0.284 | 0.340 | 0.290 | 0.453 | 0.368 |

### 4.4 Pre-Trained Time Series Transformers

#### Setups

Pre-training aims to teach time-series Transformers transferable temporal dynamics. The outcome large time series model can better cope with widespread challenges of few-shot and zero-shot forecasting. To scale Timer-XL as large models, we conduct pre-training on UTSD (1 billion time points) from Liu et al. () for a domain-universal model, and our curated dataset ERA5-Large (40 years and 4920 stations) for a domain-specific model. On ERA5-Large, we evaluate in-dataset generalization performance of PatchTST and Timer-XL: pre-training on $80$% stations and $80$% time span and directly forecast on the remaining stations (variable generalization), time span (temporal generalization), and the cross-time and cross-station split. To evaluate out-of-dataset generalization performance, we perform zero-shot forecasting following Timer (). Compared with Timer, the context length of Timer-XL is increased from $672$ to $2880$.

#### Results

We provide in-dataset generalization performance of ERA5-Large in the middle of Figure 5. Timer-XL achieves better results than PatchTST in all cases, supporting that decoder-only architecture has stronger generalization performance. Figure 5 compares zero-shot forecasting results, where Timer-XL outperforms Timer on all benchmark datasets, indicating a promising direction to improve zero-shot forecasting by pre-training on a larger context.

Figure: Figure 5: Illustration of one-for-all generalization (left). Based on the contextual flexibility, Timer-XL can predict heterogeneous time series, indicating three directions of generalization shown on the left. We compare performance when generalizing across the time and variables (middle), and zero-shot results across datasets (right), emphasizing the benefit of long-context pre-training.
Refer to caption: x5.png

### 4.5 Model Analysis

#### Ablation Study

As shown in Table 6, we conduct evaluations on TimeAttention to validate the effectiveness of position embeddings. As for variable embedding, the distinction between endogenous and exogenous variables can improve performance. Based on our observation of the learned $u>v$, we find that the token reasonably pays more attention to tokens of the endogenous variable. It leaves a prior to mask out minor dependencies that focuses less on exogenous variables. For the temporal dimension, other position embeddings are inferior to RoPE, since it uses the affine transformation, while others are additive, and thereby less confused with the same additive embedding for variables. As a counterpart Transformer, Moirai adopts the same combination of position embeddings, but its performance is still inferior to Timer-XL in Table 3 and 4, indicating that decoder-only Transformers are more compatible with such position embeddings.

**Table 6: Embedding ablation in TimeAttention. For the temporal dimension, we compare prevalent relative and absolute position embeddings. For the variable dimension, we explore the effectiveness of the variable embedding that distinguishes endogenous and exogenous variables.**
| Design | Temporal | Variable | Traffic | Weather | Solar-Energy | ERA5-MS |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |  |  |
| Ours | RoPE () | with | 0.340 | 0.238 | 0.157 | 0.205 | 0.162 | 0.221 | 0.164 | 0.307 |
| Replace | ALiBi () | with | 0.351 | 0.246 | 0.162 | 0.212 | 0.188 | 0.210 | 0.167 | 0.308 |
| Relative () | with | 0.361 | 0.250 | 0.163 | 0.214 | 0.197 | 0.215 | 0.168 | 0.309 |  |
| Absolute () | with | 0.381 | 0.270 | 0.159 | 0.207 | 0.171 | 0.204 | 0.165 | 0.306 |  |
| w/o | RoPE () | w/o | 0.361 | 0.254 | 0.171 | 0.217 | 0.181 | 0.221 | 0.235 | 0.373 |
| w/o | w/o | 0.363 | 0.253 | 0.164 | 0.215 | 0.194 | 0.215 | 0.167 | 0.309 |  |

#### Model Efficiency

To evaluate the model efficiency of Timer-XL with respect to the context length, it is essential to recognize the distinct characteristics of time series data compared to 1D sequences. Unlike natural language, the time series modality is characterized by the variable number $N$ and the input length. We adopt two representative multivariate datasets with different $N$, and provide the memory footprint and training speed under gradually prolonged input. We evaluate typical approaches to handle multivariate series: (1) Timer-XL and Moiria that adopt channel dependence; (2) Timer that adopts channel independence. Intuitively, the complexity of the first type is $\mathcal{O}(N^{2}T^{2})$ while the complexity of self-attention under channel independence is $\mathcal{O}(NT^{2})$. However, results shown in Figure 6 reveal that the measured cost of Timer-XL is much less than $N$ times of Timer.

Since the previous analysis of model efficiency on time-series Transformer predominantly focuses on the self-attention on 1D time series, we initially present a theoretical derivation of the computational complexity of Transformers on 2D time series, including the parameter counts, memory footprint, and FLOPs in Table 7. We find that other parts of Transformers, such as feed-forward network, have a complexity of $\mathcal{O}(NT)$ no matter which approach is adopted to handle multivariate time series. They also cause the dominant overhead in existing benchmarks, since their context length is not large enough, confirming our empirical results. Besides, we can introduce FlashAttention (Dao et al., ) to reduce the memory footprint and training speed, which is computationally equivalent and reduces the overall memory footprint of Timer-XL to $\mathcal{O}(NT)$ without affecting performance.

Figure: Figure 6: Efficiency analysis. We compare representative time-series Transformers on multivariate datasets with variable numbers ranging from ten to hundred and increase the lookback length.
Refer to caption: x6.png

Figure: Figure 7: Visualization of TimeAttention. It is from the first sample of a length $672$ in the test split of Traffic. We visualize the last $10$ variables and each contains $7$ tokens. We present auto-correlation function plot. Auto-correlation can be reflected by the distribution of the attention (bottom right). We average TimeAttention across sub-blocks, which can reveal Pearson correlations (upper right).
Refer to caption: x7.png

#### Representation Analysis

In addition to the enhanced performance, fine-grained token dependencies offer improved interpretability. We present a showcase visualization from Traffic in Figure 7. It is observed that sub-matrices along the diagonal generally receive greater attention, which reasonably reveals predominant dependencies within the endogenous variable. By zooming in a sub-block that corresponds to Variable-$3$, we observe that the attention distribution of the last row can indicate certain strong dependencies among patch tokens. This observation is also supported by the auto-correlation function plot (ACF), which reveals auto-correlations with certain lags and thus the model pays special attention to these tokens. Furthermore, we average each sub-matrix into one scalar. The outcome matrix can also illustrate Pearson correlations presented in the raw data.

## 5 Conclusion and Future Work

Grounded in the principles of forecasting, we highlight the urgency to extend the context length in the time series field. To facilitate long-context forecasters on diverse tasks, we propose multivariate next token prediction, a novel paradigm to predict 1D and 2D time series with covariates. We present Timer-XL enhanced by TimeAttention as an extra-long version of generative time-series Transformers. It simultaneously captures temporal dynamics and variable correlations by causal self-attention. In addition to achieving state-of-the-art performance on extensive datasets, we establish challenging benchmarks for long-context forecasting. Further, by pre-training on large-scale heterogeneous time series, Timer-XL demonstrates significant generalization capabilities as a one-for-all large model. In the future, we will improve the context utilization and computational efficiency.

## Appendix A Proof of Model Efficiency

### A.1 Setups

Given an input univariate time series divided into $T$ tokens according to the patch size $P$, which is fed into the vanilla Transformer. The training objective is to predict the next token of $P$ time points. We will generalize the derivation from 1D sequences to 2D time series based on different approaches to handle multivariate data with the variable number $N$. We adopt the same denotations as before: Transformer consists of $L$ blocks with model dimension $D$. The multi-head attention mechanism has ${H}$ heads, each with a dimension of $d_{k}$ for query, key, and value, and $d_{k}=\frac{D}{H}$. The intermediate dimension of feed-forward network is set as $D_{\text{ff}}=\alpha D$. The results are summarized in Table 7, we provide the detailed proof in the following sections.

**Table 7: Parameters count and computational complexity of Transformers for multivariate time series.**
| Metric | Type | Count / Complexity |
| --- | --- | --- |
| FLOPs | Channel Independence (Nie et al., ) | $\mathcal{O}(PDNT+L(D+H)NT^{2}+(2+\alpha)LD^{2}NT)$ |
| (Training Speed) | Channel Dependence (Ours) | $\mathcal{O}(PDNT+L(D+H)N^{2}T^{2}+(2+\alpha)LD^{2}NT)$ |
| Parameters | Flatten head (Nie et al., ) | $(4+2\alpha)LD^{2}+4LD+(1+T)PD$ |
| Token-wise projector (Liu et al., ) | $(4+2\alpha)LD^{2}+4LD+2PD$ |  |
| Memory | Self-Attention (Vaswani et al., ) | $4(D+P)NT+(32+8\alpha)LDNT+4LHN^{2}T^{2}$ |
| Footprint | FlashAttention (Dao et al., ) | $4(D+P)NT+(32+8\alpha)LDNT$ |

### A.2 FLOPs

As a preliminary, the multiplication between matrix $\mathbf{A}\in\mathbb{R}^{n\times m}$ and matrix $\mathbf{C}\in\mathbb{R}^{m\times p}$ requires $mnp$ multiplications and $mnp$ additions, resulting in $2mnp$ floating-point operations. Given batched matrices $\mathbf{A}\in\mathbb{R}^{B\times n\times m}$ and $\mathbf{C}\in\mathbb{R}^{B\times m\times p}$ , $B$ times matrix multiplications will be performed. It is evident that the batch size is a linear multiplier. Thus, we first omit $B$ to calculate the operations of dealing with one univariate series, and then we will reintroduce it to analyze channel independence.

The computational cost of Transformers can be primarily categorized into two types: (1) multi-head attention calculation and (2) linear transformations. In contrast, the operations of layer normalization, residual connection, activation functions, and position embedding with the complexity of $\mathcal{O}(TD)$ are less significant. Therefore, we derive the computational complexity mainly with respect to the above two types by delving into the forwarding process of one univariate series.

#### Patch Embedding

The tokenized time series $\{\mathbf{x}_{i}\}\in\mathbb{R}^{T\times P}$ is mapped into the embedding space through the patch-wise embedding $\mathbf{W}_{e}\in\mathbb{R}^{D\times P}$ , resulting in $2PDT$ operations.

#### Self-Attention

The calculation of self-attention begins with the computation of query, key and value by multiplying the patch embeddings with matrices $\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}\in\mathbb{R}^{D\times d_{k}}$ respectively in $H$ heads, which incurs a computational cost of $6HDd_{k}T=6D^{2}T$ and yields $\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{H\times T\times d_{k}}$. Next, the dot product $\mathbf{QK}^{\top}\in\mathbb{R}^{H\times T\times T}$ is conducted in each head, leading to $2Hd_{k}T^{2}=2DT^{2}$ operations. Following this, the Pre-Softmax map is divided by $\sqrt{d_{k}}$ and processed through $\operatorname{Softmax}$, which includes exponentiation, summation, and normalization of each element, resulting in $4HT^{2}$ operations. The subsequent multiplication with $\mathbf{V}$ incurs $2Hd_{k}T^{2}=2DT^{2}$ operations. Finally, multiple heads are concatenated and multiplied by $\mathbf{W}_{o}\in\mathbb{R}^{D\times D}$, contributing $2D^{2}T$ operations.

#### Feed-Forward Network

It first projects the token representations into the dimension of $D_{ff}$ and subsequently projects it back to the dimension $D$, resulting in a total operations of $4\alpha D^{2}T$.

#### Patch Projection

For encoder-only models, all token representations are flattened and mapped directly to $P$ time points by $\mathbf{W}_{d}\in\mathbb{R}^{TD\times P}$. In contrast, token-wise projector $\mathbf{W}_{d}\in\mathbb{R}^{D\times P}$ in decoder-only models independently map each token to the predicted next token. In both cases, the number of operations is $2PDT$, but the token-wise projector will result in a smaller parameter count.

The forwarding operations in $L$-layers Transformer is $4PDT+4L(D+H)T^{2}+(8+4\alpha)LD^{2}T$ in sum. Considering that the majority of operations in Transformers are binary operations (e.g., matrix multiplications), the gradients for both matrices are computed separately. As a result, the number of operations in backpropagation is the twice of forwarding. Therefore, the total operations of training a Transformer on a univariate series consisting of $T$ patches, each of length $P$, is derived as:

$$ $f(T)=12PDT+12L(D+H)T^{2}+(24+12\alpha)LD^{2}T.$ $$

We plug typical hyperparameters in the current time-series Transformers and forecasting benchmarks: $D=512,H=8,L=4,\alpha=4,T=7$, and $P=96$, we obtain that:

$$ $f(T)=24960T^{2}+76087296T\propto 3.28*10^{-4}T^{2}+T.$ $$

Due to the prevalence of short contexts in the time series field, where $T\ll D$ leads to a significant coefficient in $\mathcal{O}(T)$, we find the primary computational burden of time-series Transformer lies in linear transformations with $\mathcal{O}(T)$, rather than in multi-head self-attention with the $\mathcal{O}(T^{2})$ complexity.

For multivariate series with $N$ variables, FLOPs is influenced by the handling of multivariate data. When adopting channel independence (Timer and PatchTST), $N$ can be regarded as the batch size $B$:

$$ $Nf(T)=12PDNT+12L(D+H)NT^{2}+(24+12\alpha)LD^{2}NT.$ (9) $$

For models that capture fine-grained intra- and inter-series dependencies (Timer-XL and Moirai) in multivariate series, $N$ is reflected as the enlarged number of tokens:

$$ $f(NT)=12PDNT+12L(D+H)N^{2}T^{2}+(24+12\alpha)LD^{2}NT.$ (10) $$

Notably, FLOPs is not entirely equivalent to actual runtime. While FlashAttention increases the overall FLOPs due to its recomputation process, it reduces the number of memory reads and writes. Given that on GPUs, computation is significantly faster than memory access, using FlashAttention can actually lead to further improvements in runtime performance.

### A.3 Parameter Count

From the above analysis, we observe that the parameter count of Transformers includes the following:

#### Patch Embedding

$\mathbf{W}_{e}\in\mathbb{R}^{D\times P}$ to obtain patch embeddings.

#### Self-Attention

$\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}\in\mathbb{R}^{D\times d_{k}}$ of $H$ heads and $\mathbf{W}_{o}\in\mathbb{R}^{D\times D}$ for all heads.

#### Feed-Forward Network

$\mathbf{W}_{\text{ffn1}},\mathbf{W}_{\text{ffn2}}\in\mathbb{R}^{D\times D_{
\text{ff}}}$ in feed-forward network.

#### Layer Normalization

It contains the weight $\mathbf{W}\in\mathbb{R}^{D}$ and the bias $\mathbf{b}\in\mathbb{R}^{D}$. Every Transformer block includes two normalizations after multi-head attention and feed-forward network respectively.

#### Patch Projection

$\mathbf{W}_{d}\in\mathbb{R}^{TD\times P}$ in flatten head and $\mathbf{W}_{d}\in\mathbb{R}^{D\times P}$ in token-wise projection.

In sum, the total count of parameters in time-series Transformers can be expressed as:

$$ $\text{Parameter Count}=\begin{cases}(4+2\alpha)LD^{2}+4LD+(1+T)PD,&\text{using flatten head,}\\ (4+2\alpha)LD^{2}+4LD+2PD,&\text{using token-wise projection.}\end{cases}$ (11) $$

### A.4 Memory Footprint

The memory footprint during training can be primarily categorized into three parts: activation values stored for backpropagation, model parameters, and optimizer parameters.

Regardless of other precision types (e.g., FP16), model parameters and gradients are typically stored as 32-bit floating-point numbers, with each parameter occupying 4 bytes of memory. For time-series Transformers, memory footprint of activation values is given as follows:

#### Patch Embedding

Gradient computation for $\mathbf{W}_{e}$ preserves its input $\{\mathbf{x}_{i}\}\in\mathbb{R}^{T\times P}$ of $4PT$ bytes.

#### Self-Attention

Gradient calculation for $\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}\in\mathbb{R}^{D\times d_{k}}$ requires their inputs $\mathbf{H}\in\mathbb{R}^{T\times D}$, amounting to a total of $4DT$ bytes. The dot product for attention map also needs to store $\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{H\times T\times d_{k}}$, which collectively require a total of $12DT$ bytes of memory. Gradient computation of $\mathbf{W}_{o}\in\mathbb{R}^{D\times D}$ necessitates the concatenated multi-head attention representations $\mathbf{H}\in\mathbb{R}^{T\times D}$, which occupies $4DT$ bytes. If memory-efficient attention mechanisms like FlashAttention (Dao et al., ) is not applied, the outcome $\mathbf{QK}^{\top}$ will be stored and occupy $4HT^{2}$ bytes. Instead, if FlashAttention is adopted, the storage overhead can be avoided.

#### Feed-Forward Network

ReLU activation function is typically employed in this module. The input $\mathbf{H}\in\mathbb{R}^{T\times D}$ must be retained, requiring a total of $4DT$ bytes. Additionally, the product $\mathbf{W}_{\text{ffn1}}\mathbf{H}$ also needs to be stored, amounting to $4D_{\text{ff}}T$ bytes. Similarly, the output activations of ReLU, which serve as the input for subsequent linear transformations, necessitate another $4D_{\text{ff}}T$ bytes.

#### Layer Normalization

Each block of Transformer encompasses two layer normalizations, with each normalization retaining its input, resulting in the memory requirement of $8DT$ bytes.

#### Patch Projection

To perform backpropagation for $W_{d}\in\mathbb{R}^{D\times P}$, it is necessary to retain its input $\mathbf{H}\in\mathbb{R}^{T\times D}$, resulting in a total memory requirement of $4DT$ bytes.

The formula for the total activation values of the entire model occupying GPU memory is as follows:

$$ $\text{Memory Footprint}=\begin{cases}4(D+P)T+(32+8\alpha)LDT+4LHT^{2},&\text{w /o FlashAttention,}\\ 4(D+P)T+(32+8\alpha)LDT,&\text{with FlashAttention.}\end{cases}$ (12) $$

The derived occupancy of activation values increases proportionally with the batch size $B$. For multivariate series, $N$ can be used as a multiplier in channel independence. For channel independence models, we can substitute $T$ with $NT$ as before. The total memory footprint is the sum of activation values and parameters of model and optimizer, which are proportional to the parameter count derived in Equation 11. Due to the limited model size in the time series field, the memory consumption of parameters is minimal and can be considered negligible in practice. Therefore, the overall memory footprint can be predominantly determined by the occupied memory of activation values.

## Appendix B Experimental Details

### B.1 Datasets

We conduct experiments on well-acknowledged benchmarks to evaluate performance of the proposed Timer-XL, which includes (1) ETT (Zhou et al., ) contains 7 factors of electricity transformers from July 2016 to July 2018, which is recorded every hour or 15 minutes. (2) Weather (Wu et al., ) includes 21 meteorological factors collected every 10 minutes from the Max Planck Biogeochemistry Institute Weather Station in 2020. (3) ECL (Wu et al., ) records the hourly electricity consumption data of 321 clients. (4) Traffic (Wu et al., ) collects hourly road occupancy rates measured by 862 sensors on the San Francisco Bay area highways from January 2015 to December 2016. (5) Solar-Energy (Lai et al., ) records the solar power production of 137
PV plants in 2006, which are sampled every 10 minutes. (7) PEMS (Liu et al., ) contains records from the public traffic network in California collected in 5-minute time windows. (8) EPF (Lago et al., ) includes five subsets that span six years. Each contains the electricity price as the endogenous variable to be predicted and two exogenous variables of the day-ahead electricity markets. (9) GTWSF (Wu et al., ) is a dataset collected from the National Centers for Environmental Information (NCEI). This large-scale collection contains hourly averaged wind speed and
temperature data from 3850 stations with different geographical scales and densities each, spanning from 2019 to 2021. (10) UTSD (Liu et al., ) is a multi-domain time series dataset, which includes seven domains with a hierarchy of four volumes. We adopt the largest volume that encompasses 1 billion time points for pre-training.

We further establish challenging forecasting benchmarks based on the ECMWF Reanalysis v5 (ERA5) dataset (Hersbach et al., ) to prevent potential overfitting and performance saturation of deep forecasters in existing benchmarks. Concretely, ERA5 is the fifth generation ECMWF atmospheric reanalysis of the global climate covering the period from January 1940 to the present, which provides hourly estimates of a large number of atmospheric, land, and oceanic climate variables, and includes information about uncertainties for all variables at reduced spatial and temporal resolutions. Due to its pattern sufficiency of temporal dynamics and variable correlations, we could establish practical benchmarks to thoroughly evaluate the performance for univariate and multivariate forecasting, as well as adopt it for large-scale pre-training to develop domain-specific large time series models.

Our datasets are constructed as follows:

- •
ERA5-S: To establish a realistic univariate forecasting benchmark, we start from the basic principle of forecastability and make the prediction on sufficient lookback lengths. Instead of the short time span of training in previous benchmarks (generally no more than 2 years), we curated a three-hour frequency dataset spanning 40 years (January 1979 to December 2018) from ERA5, encompassing 116880 time points. In order to prevent overfitting on a single time series, we selected worldwide stations to form seven subsets.
- •
ERA5-MS: Each univariate series of ERA5-S provides partial observations governed by the spatio–temporal global weather system. Since discovering the global spatio-temporal correlations presents a fundamental challenge in meteorology, we convert ERA5-S into ERA5-MS by using seven subsets as a challenging multivariate forecasting benchmark. Based on the average results in Tables 1 and 4, we can validate the existence of multi-station correlations among selected stations, which have enhanced the average prediction accuracy.
- •
ERA5-Large: To explore the pure data-driven approach to build domain-specific large time series models, we further expanded the number of stations as ERA5-Large, a dataset that evenly covers meteorological 4920 worldwide stations and spans 40 years. We establish the dataset for pre-training, which is expected to generalize across the time (train on the past observations and generalize to the future) and across stations (train on partial stations and generalize to other unseen stations). The total number of time points is around half a billion.

We follow the same data processing and train-validation-test split protocol used in TimesNet (Wu et al., ), where the train, validation, and test datasets are divided according to chronological order to prevent data leakage. Detailed dataset descriptions and prediction settings are provided in Table 8.

**Table 8: Dataset descriptions. *Dim.* denotes the number of variables (For univariate forecasting, we adopt channel independence (Nie et al., ) or train separate models on each variable). *Dataset Length* denotes the number of time points in the (train, validation, test) splits.**
| Tasks | Dataset | Dim. | Prediction Setting | Dataset Length | Information (Frequency) |
| --- | --- | --- | --- | --- | --- |
|  | ETTh1 | 7 | {24, 96, 168, 672, 2880}$\to$96 | (8545, 2881, 2881) | Electricity (Hourly) |
| Univariate | ECL | 321 | {24, 96, 168, 672, 2880, 8832}$\to$96 | (18317, 2633, 5261) | Electricity (Hourly) |
| Forecasting | Traffic | 862 | {24, 96, 168, 672, 2880, 8832}$\to$96 | (12185, 1757, 3509) | Transportation (Hourly) |
|  | PEMS03 | 358 | {96, 288, 1152, 2016, 8064}$\to$96 | (15617, 5135, 5135) | Transportation (5 mins) |
|  | ERA5-S | 7 | 3072$\to$96 | (81816, 11688, 23376) | Climate (3 Hours) |
|  | ETTh1, ETTh2 | 7 | {96, 672}$\to${96, 192, 336, 720} | (8545, 2881, 2881) | Electricity (Hourly) |
|  | ETTm1, ETTm2 | 7 | {96, 672}$\to${96, 192, 336, 720} | (34465, 11521, 11521) | Electricity (15 mins) |
|  | ECL | 321 | {96, 672}$\to${96, 192, 336, 720} | (18317, 2633, 5261) | Electricity (Hourly) |
| Multivariate | Traffic | 862 | {96, 672}$\to${96, 192, 336, 720} | (12185, 1757, 3509) | Transportation (Hourly) |
| Forecasting | Weather | 21 | {96, 672}$\to${96, 192, 336, 720} | (36792, 5271, 10540) | Climate (10 mins) |
|  | Solar-Energy | 137 | {96, 672}$\to${96, 192, 336, 720} | (36601, 5161, 10417) | Energy (10 mins) |
|  | ERA5-MS | 7 | 3072$\to$96 | (81816, 11688, 23376) | Climate (3 Hours) |
|  | GTWSF | 3850 | 48$\to$24 | (12280, 1755, 3509) | Wu et al. () |
|  | NP | 1+2 | 168$\to$24 | (36500, 5219, 10460) | Electricity (Hourly) |
| Forecasting | PJM | 1+2 | 168$\to$24 | (36500, 5219, 10460) | Electricity (Hourly) |
| with Covariates | BE | 1+2 | 168$\to$24 | (36500, 5219, 10460) | Electricity (Hourly) |
| (EPF) | FR | 1+2 | 168$\to$24 | (36500, 5219, 10460) | Electricity (Hourly) |
|  | DE | 1+2 | 168$\to$24 | (36500, 5219, 10460) | Electricity (Hourly) |
| Pre-training | ERA5-Large | 4920 | 3072$\to$96 | (81816, 11688, 23376) | Climate (3 Hours) |
| UTSD | - | 2880$\to$96 | (868778970, 96530996, -) | Liu et al. () |  |

**Table 9: Performance robustness of Timer-XL. The prediction settings and results keep the same with Table 11. The standard deviation is obtained from three random seeds.**
| Dataset | ECL | ETTh1 | Traffic |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Horizon | MSE | MAE | MSE | MAE | MSE | MAE |
| $96$ | 0.127<br>$\pm$0.001 | 0.219<br>$\pm$0.001 | 0.364<br>$\pm$0.002 | 0.397<br>$\pm$0.001 | 0.340<br>$\pm$0.002 | 0.238<br>$\pm$0.001 |
| $192$ | 0.145<br>$\pm$0.001 | 0.236<br>$\pm$0.001 | 0.405<br>$\pm$0.002 | 0.424<br>$\pm$0.001 | 0.360<br>$\pm$0.001 | 0.247<br>$\pm$0.001 |
| $336$ | 0.159<br>$\pm$0.001 | 0.252<br>$\pm$0.001 | 0.427<br>$\pm$0.003 | 0.439<br>$\pm$0.002 | 0.377<br>$\pm$0.002 | 0.256<br>$\pm$0.002 |
| $720$ | 0.187<br>$\pm$0.003 | 0.277<br>$\pm$0.003 | 0.439<br>$\pm$0.002 | 0.459<br>$\pm$0.004 | 0.418<br>$\pm$0.003 | 0.279<br>$\pm$0.002 |
| Dataset | Solar-Energy | Weather | ERA5-MS |  |  |  |
| Horizon | MSE | MAE | MSE | MAE | MSE | MAE |
| $96$ | 0.162<br>$\pm$0.003 | 0.221<br>$\pm$0.002 | 0.157<br>$\pm$0.002 | 0.205<br>$\pm$0.001 | 0.164<br>$\pm$0.001 | 0.307<br>$\pm$0.000 |
| $192$ | 0.187<br>$\pm$0.003 | 0.239<br>$\pm$0.002 | 0.206<br>$\pm$0.003 | 0.250<br>$\pm$0.002 |  |  |
| $336$ | 0.205<br>$\pm$0.003 | 0.255<br>$\pm$0.002 | 0.259<br>$\pm$0.003 | 0.291<br>$\pm$0.003 |  |  |
| $720$ | 0.238<br>$\pm$0.003 | 0.279<br>$\pm$0.003 | 0.337<br>$\pm$0.002 | 0.344<br>$\pm$0.002 |  |  |

### B.2 Baseline Models

We aim to present Timer-XL as a foundation model for unified time series forecasting. We thoroughly include well-acknowledged and advanced models in each forecasting task. For univariate time series forecasting, we compare Timer-XL with PatchTST (Nie et al., ) under channel independence. For multivariate time series prediction, we report official results from Liu et al. (; ); Ding et al. (), including UniRepLKNet (), iTransformer (), Corrformer (), DLinear (), TimesNet (), Non-stationary Transformer (), Pyraformer (), Autoformer () , StemGNN (), DeepAR (), and N-BEATS (). We further reproduce the performance of related Transformers: Timer () and Moirai () based on their official repositories. For covariate-informed time series forecasting, we report the official results of TimeXer (). For zero-shot forecasting, we follow Liu et al. () that predicts future length-$96$ windows in well-acknowledged datasets. In general, more than 20 baselines are included for a complete comparison.

### B.3 Implementation Details

All the experiments are implemented by PyTorch Paszke et al. () on NVIDIA A100 Tensor Core GPUs. We employ the Adam optimizer (Kingma & Ba, ) and MSE loss for model optimization. We adopt channel independence from Nie et al. () in univariate time series forecasting. Based on the prevalence of patch-level tokenization in the time series field, we reproduce typical Transformers: PatchTST (), Timer (), and Moirai () based on their official repositories, and keep their model hyperparameters and training configurations the same to evaluate the inherent capability of base models. The results of other baselines are based on the benchmark provided by Liu et al. (; ); Ding et al. (); Wang et al. (), which is fairly built on the configurations provided by their original paper. Detailed experimental configurations are provided in Table 10. We also report the standard deviations under three runs with different random seeds in Table 9, which exhibits that the performance of Timer-XL is stable.

For the metrics, we adopt the symmetric mean absolute percentage error (SMAPE), a metric that is independent of the numerical range, to evaluate one-for-all generalization performance on ERA5-Large. For other experiments, we adopt the root mean square error (MSE) and mean absolute error (MAE) that follows previous work. These metrics can be calculated as follows:

$$ $\text{SMAPE}=\frac{200}{T}\sum_{i=1}^{T}\frac{|\mathbf{X}_{i}-\widehat{\mathbf {X}}_{i}|}{|\mathbf{X}_{i}|+|\widehat{\mathbf{X}}_{i}|},\ \text{MSE}=\sum_{i=1 }^{T}|\mathbf{X}_{i}-\widehat{\mathbf{X}}_{i}|^{2},\ \text{MAE}=\sum_{i=1}^{T} |\mathbf{X}_{i}-\widehat{\mathbf{X}}_{i}|.$ $$

Here $\mathbf{X}\in\mathbb{R}^{T}$ is a univariate time series and $\widehat{\mathbf{X}}$ is the corresponding prediction. For multivariate time series, we further calculate the mean metric in the variable dimension.

**Table 10: Experimental configurations of Timer-XL and other baseline Transformers. All the experiments adopt the ADAM () optimizer with the default hyperparameter $(\beta_{1},\beta_{2})=(0.9,0.999)$.**
| Experiment | Model | Dataset | Configuration | Training Process |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $L$ | $D$ | $d_{k}$ | $H$ | $P$ | LR | Loss | Batch Size | Epochs |  |  |  |
| Univariate Forecasting |  | ECL | 3 | 512 | 64 | 8 | 96 | 0.0005 | MSE | 2048 | 10 |
| Timer-XL | Traffic | 3 | 512 | 64 | 8 | 96 | 0.001 | MSE | 2048 | 10 |  |
| PatchTST | ETTh1 | 1 | 512 | 64 | 8 | 96 | 0.0005 | MSE | 256 | 10 |  |
|  | PEMS03 | 3 | 512 | 64 | 8 | 96 | 0.0005 | MSE | 2048 | 10 |  |
|  | ERA5-S | 1 | 512 | 64 | 8 | 96 | 0.0005 | MSE | 2048 | 10 |  |
| Multivariate Forecasting |  | Global Temp. | 3 | 1024 | 128 | 8 | 24 | 0.0001 | MSE | 8 | 10 |
|  | Global Wind | 3 | 1024 | 128 | 8 | 24 | 0.0001 | MSE | 8 | 10 |  |
| Timer-XL | ECL | 5 | 512 | 64 | 8 | 96 | 0.0005 | MSE | 4 | 10 |  |
| Moirai | Traffic | 4 | 512 | 64 | 8 | 96 | 0.0005 | MSE | 4 | 10 |  |
| Timer | ETTh1 | 1 | 1024 | 128 | 8 | 96 | 0.0001 | MSE | 32 | 10 |  |
| PatchTST | Weather | 4 | 512 | 64 | 8 | 96 | 0.0005 | MSE | 32 | 10 |  |
|  | Solar. | 6 | 512 | 64 | 8 | 96 | 0.0001 | MSE | 16 | 10 |  |
|  | ERA5-MS | 3 | 512 | 64 | 8 | 96 | 0.0001 | MSE | 256 | 10 |  |
| Forecasting with Covariates | Timer-XL | NP | 3 | 512 | 64 | 8 | 24 | 0.0001 | MSE | 4 | 10 |
| Moirai | PJM | 2 | 512 | 64 | 8 | 24 | 0.0001 | MSE | 16 | 10 |  |
| Timer | BE | 2 | 512 | 64 | 8 | 24 | 0.0001 | MSE | 16 | 10 |  |
| PatchTST | FR | 2 | 512 | 64 | 8 | 24 | 0.0001 | MSE | 16 | 10 |  |
|  | DE | 2 | 512 | 64 | 8 | 24 | 0.0001 | MSE | 16 | 10 |  |
| Pre-training | Timer-XL | ERA5-Large | 4 | 512 | 64 | 8 | 96 | 0.0001 | MSE | 40960 | 10 |
| PatchTST | 4 | 512 | 64 | 8 | 96 | 0.0001 | MSE | 40960 | 10 |  |  |
| Timer-XL | UTSD | 8 | 1024 | 128 | 8 | 96 | 0.00005 | MSE | 16384 | 10 |  |
| Timer | 8 | 1024 | 128 | 8 | 96 | 0.00005 | MSE | 16384 | 10 |  |  |

## Appendix C Hyperparameter Sensitivity

We evaluate the hyperparameter sensitivity of Timer-XL on the ERA5-MS benchmark, as illustrated in Figure 8, concerning the following factors: the number of layers
$L$, the patch size $P$, and the lookback length during inference. Our findings indicate that performance of Timer-XL generally improves with increases with $L$, suggesting that Timer-XL is a scalable deep forecaster. Furthermore, our analysis of the influence of $P$ reveals that the optimal patch size is generally close to the predicted length, since it avoid multi-step error accumulations. Toward better long-term forecasting performance, it leaves a future improvement to adopt different patch sizes of input and output tokens. Finally, we investigate the impact of input length during inference. We discover that the optimal lookback length of during is not necessarily the length during training. Given that generative Transformers can accommodate inference inputs shorter than those used during training, this finding is noteworthy and indicates the potential to improve the performance of generative Transformers.

Figure: Figure 8: Hyperparameter sensitivity of Timer-XL (input-$3072$-pred-$96$ on ERA5-MS), including the number of Transformer blocks $L$, the patch size $P$, and the input lookback length during inference.
Refer to caption: x8.png

## Appendix D Supplementary Results

Table 11 provides the complete results of the one-for-all multivariate forecasting benchmark across well-acknowledged datasets. We evaluate Timer-XL and baseline models by rolling forecasting: each model is trained with input length $672$ and output length $96$, and the predicted values are integrated as part of the input in the next iteration until reaching the desired forecast length in $\{96,192,336,720\}$.

We highlight that this benchmark evaluates the fundamental model versatility of deep forecasters, which aims to break the awkward situation of extensive training and model storage in pursuit of better practice for real-world forecasting requirements. On this benchmark, time-series Transformers significantly stand out from other baseline models, and our proposed Timer-XL can achieve state-of-the-art performance, making it a nice fundamental backbone of a one-for-all forecaster.

**Table 11: Full multivariate forecasting results: we conduct rolling forecast with a single model trained on each dataset (lookback length is $672$) and accomplish four forecast lengths in $\{96,192,336,720\}$.**
| Models | Timer-XL | Timer | Moirai | iTransformer | DLinear | PatchTST | TimesNet | Stationary | Autoformer |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (Ours) | () | () | () | () | () | () | () | () |  |  |  |  |  |  |  |  |  |  |  |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| ECL | 96 | 0.127 | 0.219 | 0.129 | 0.221 | 0.130 | 0.225 | 0.133 | 0.229 | 0.138 | 0.238 | 0.132 | 0.232 | 0.184 | 0.288 | 0.185 | 0.287 | 0.256 | 0.357 |
| 192 | 0.145 | 0.236 | 0.148 | 0.239 | 0.150 | 0.244 | 0.158 | 0.258 | 0.152 | 0.251 | 0.151 | 0.250 | 0.192 | 0.295 | 0.282 | 0.368 | 0.291 | 0.376 |  |
| 336 | 0.159 | 0.252 | 0.164 | 0.256 | 0.166 | 0.262 | 0.168 | 0.262 | 0.167 | 0.268 | 0.171 | 0.272 | 0.200 | 0.303 | 0.289 | 0.377 | 0.290 | 0.379 |  |
| 720 | 0.187 | 0.277 | 0.201 | 0.289 | 0.206 | 0.297 | 0.205 | 0.294 | 0.203 | 0.302 | 0.222 | 0.318 | 0.228 | 0.325 | 0.305 | 0.399 | 0.320 | 0.403 |  |
| Avg | 0.155 | 0.246 | 0.161 | 0.251 | 0.163 | 0.257 | 0.164 | 0.258 | 0.165 | 0.265 | 0.169 | 0.268 | 0.201 | 0.303 | 0.265 | 0.358 | 0.289 | 0.379 |  |
| ETTh1 | 96 | 0.364 | 0.397 | 0.371 | 0.404 | 0.379 | 0.415 | 0.387 | 0.418 | 0.369 | 0.400 | 0.373 | 0.403 | 0.452 | 0.463 | 0.452 | 0.478 | 0.467 | 0.499 |
| 192 | 0.405 | 0.424 | 0.407 | 0.429 | 0.415 | 0.438 | 0.416 | 0.437 | 0.405 | 0.422 | 0.405 | 0.425 | 0.474 | 0.477 | 0.484 | 0.510 | 0.492 | 0.523 |  |
| 336 | 0.427 | 0.439 | 0.434 | 0.445 | 0.440 | 0.454 | 0.434 | 0.450 | 0.435 | 0.445 | 0.423 | 0.440 | 0.493 | 0.489 | 0.511 | 0.522 | 0.519 | 0.531 |  |
| 720 | 0.439 | 0.459 | 0.461 | 0.466 | 0.482 | 0.482 | 0.447 | 0.473 | 0.493 | 0.508 | 0.445 | 0.471 | 0.560 | 0.534 | 0.571 | 0.543 | 0.589 | 0.560 |  |
| Avg | 0.409 | 0.430 | 0.418 | 0.436 | 0.429 | 0.447 | 0.421 | 0.445 | 0.426 | 0.444 | 0.412 | 0.435 | 0.495 | 0.491 | 0.505 | 0.513 | 0.517 | 0.528 |  |
| Traffic | 96 | 0.340 | 0.238 | 0.348 | 0.240 | 0.359 | 0.250 | 0.353 | 0.259 | 0.399 | 0.285 | 0.359 | 0.255 | 0.593 | 0.315 | 0.610 | 0.322 | 0.675 | 0.412 |
| 192 | 0.360 | 0.247 | 0.369 | 0.250 | 0.373 | 0.257 | 0.373 | 0.267 | 0.409 | 0.290 | 0.377 | 0.265 | 0.596 | 0.317 | 0.626 | 0.346 | 0.679 | 0.423 |  |
| 336 | 0.377 | 0.256 | 0.388 | 0.260 | 0.386 | 0.265 | 0.386 | 0.275 | 0.422 | 0.297 | 0.393 | 0.276 | 0.600 | 0.319 | 0.633 | 0.352 | 0.688 | 0.440 |  |
| 720 | 0.418 | 0.279 | 0.431 | 0.285 | 0.421 | 0.286 | 0.425 | 0.296 | 0.461 | 0.319 | 0.436 | 0.305 | 0.619 | 0.335 | 0.651 | 0.366 | 0.693 | 0.457 |  |
| Avg | 0.374 | 0.255 | 0.384 | 0.259 | 0.385 | 0.265 | 0.384 | 0.274 | 0.423 | 0.298 | 0.391 | 0.275 | 0.602 | 0.322 | 0.630 | 0.347 | 0.684 | 0.433 |  |
| Weather | 96 | 0.157 | 0.205 | 0.151 | 0.202 | 0.152 | 0.206 | 0.174 | 0.225 | 0.169 | 0.229 | 0.149 | 0.202 | 0.169 | 0.228 | 0.185 | 0.241 | 0.355 | 0.409 |
| 192 | 0.206 | 0.250 | 0.196 | 0.245 | 0.198 | 0.249 | 0.227 | 0.268 | 0.211 | 0.268 | 0.194 | 0.245 | 0.222 | 0.269 | 0.286 | 0.325 | 0.421 | 0.450 |  |
| 336 | 0.259 | 0.291 | 0.249 | 0.288 | 0.251 | 0.291 | 0.290 | 0.309 | 0.258 | 0.306 | 0.244 | 0.285 | 0.290 | 0.310 | 0.323 | 0.347 | 0.452 | 0.465 |  |
| 720 | 0.337 | 0.344 | 0.330 | 0.344 | 0.322 | 0.340 | 0.374 | 0.360 | 0.320 | 0.362 | 0.317 | 0.338 | 0.376 | 0.364 | 0.436 | 0.401 | 0.513 | 0.496 |  |
| Avg | 0.240 | 0.273 | 0.232 | 0.270 | 0.231 | 0.272 | 0.266 | 0.291 | 0.239 | 0.291 | 0.226 | 0.268 | 0.264 | 0.293 | 0.308 | 0.329 | 0.435 | 0.455 |  |
| Solar-Energy | 96 | 0.162 | 0.221 | 0.212 | 0.230 | 0.190 | 0.240 | 0.183 | 0.265 | 0.193 | 0.258 | 0.168 | 0.237 | 0.180 | 0.272 | 0.199 | 0.290 | 0.206 | 0.296 |
| 192 | 0.187 | 0.239 | 0.232 | 0.246 | 0.223 | 0.264 | 0.205 | 0.283 | 0.214 | 0.274 | 0.189 | 0.257 | 0.199 | 0.286 | 0.243 | 0.307 | 0.254 | 0.328 |  |
| 336 | 0.205 | 0.255 | 0.237 | 0.253 | 0.250 | 0.283 | 0.224 | 0.299 | 0.233 | 0.291 | 0.212 | 0.277 | 0.220 | 0.301 | 0.264 | 0.322 | 0.272 | 0.330 |  |
| 720 | 0.238 | 0.279 | 0.252 | 0.266 | 0.292 | 0.311 | 0.239 | 0.316 | 0.246 | 0.307 | 0.240 | 0.305 | 0.251 | 0.321 | 0.310 | 0.339 | 0.326 | 0.347 |  |
| Avg | 0.198 | 0.249 | 0.233 | 0.249 | 0.241 | 0.275 | 0.213 | 0.291 | 0.222 | 0.283 | 0.202 | 0.269 | 0.213 | 0.295 | 0.254 | 0.315 | 0.265 | 0.325 |  |
| $1^{\text{st}}$ Count | 19 | 17 | 0 | 5 | 0 | 0 | 0 | 0 | 1 | 1 | 7 | 5 | 0 | 0 | 0 | 0 | 0 | 0 |  |

## Appendix E Limitations

As a generative Transformer, Timer-XL necessitates iterative generation for long-term forecasting, which may lead to error accumulation and inflexibility in the output length. In the future, we plan to incorporate multi-resolution patches for input and output series. Furthermore, given that Timer-XL explicitly captures fine-grained token dependencies, there remains significant potential to reduce the complexity of TimeAttention, particularly in high-dimensional and lengthy time series. Finally, we will investigate the factors contributing to the stagnation of Transformer performance in extremely long contexts, and seek insights in the time series modality to improve context efficiency.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: See above.

- Which version of Moirai was used for the experiments? 1.0 or 1.1?
soundness: 2
strengths: a) The paper studies an interesting problem of long context modeling in the context of time series forecasting. Authors attempt to connect long context to scenarios beyond univariate modeling through next-token style modeling of multivariate and covariate-informed time series. 

b) Experiments have been conducted on many diverse settings although the experiments themselves have some limitations.
summary: The paper proposes Timer-XL, a transformer decoder model for time series forecasting. Building upon the existing Timer model, Timer-XL extends the model with a longer context and a masking-based approach called TimeAttention to handle multivariate/covariate scenarios. Empirical results have been reported on univariate, multivariate and covariate experiments on some benchmark datasets.
weaknesses: a) While the problem of long context modeling is interesting, the primary weakness of this work is the lack of clarity about the goal and a proper scope. The discussion is confusing and often only loosely relates to the long context setting which appears to be the primary goal. Authors claim that "existing transformers in the time series field crucially encounter the context bottleneck" which is not as critical of a problem as being portrayed here. Such claims require serious empirical justification which is missing from the paper (experiment 1 does not go too far, see below). In reality, long context scenarios may be helpful but _mostly_ in specific high-frequency scenarios. Consider a 5min granularity time series with weekly seasonal behavior. One would need a context larger than 2K to understand the seasonal behavior from the time series history. Such cases should have been better highlighted to justify the central claim of the paper. That said, long context univariate modeling may not always yield improvements. It may also worsen the accuracy, e.g., in the case of distribution shifts. 

Coming to the utility of long context for multivariate/covariate-informed forecasting, this has been studied (although not explicitly) in Moirai. While the explicit perspective in this paper is interesting, the primary problem is not that one can do multivariate/covariate modeling through long context but that one needs to be able to do it in a zero-shot sense, as attempted in Moirai. As per my understanding, the settings being studied here are task-specific and not a single pretrained Timer-XL model being used for all experiments. Please correct me if I am wrong. 

b) The technical novelty of this work is limited in light of works such as Moirai. "TimeAttention" is a masking scheme that extends causal univariate patch-based modeling to the multivariate setting.

c) The empirical results lack comprehensiveness and are not particularly strong.

**Experiment 4.1**: The benchmark is fairly small to draw conclusions. Even in these scenarios, the benchmark shows that long context yields to diminishing returns (or worse performance) beyond a point, 1 month in most cases. 1 month @ 1h granularity amounts to a context length of 720 which is not far off from the context length of many time series models (Chronos, Moirai, TimesFM, etc.). Furthermore, I don't understand why PatchTST is the only baseline being studied here. I also don't quite understand what's unique about Timer-XL here that yields better performance over PatchTST. Could it be the larger number of parameters? 

Minor: The analysis on normalization does not belong to this section and brings limited value to the discussion. Normalization mostly helps when time series in a dataset have drastically different scales. Models working fine without normalization for a single task of land-surface temperature forecasting is not a surprising result.  

**Experiment 4.2**: The results in Table 2 and 4 are not strong when compared to pretrained baselines such as Moirai. Why is Moirai missing from Fig 4? 

Minor: DeepAR and N-BEATS are not numerical simulation based methods. 

**Experiment 4.3**: More baselines that can incorporate covariates are needed before conclusions can be drawn. For example, you can consider adding N-BEATSx, NHITS, DeepAR, etc. GluonTS also provides an implementation of PatchTST which can incorporate covariates. 

**Experiment 4.4**: The benchmark selected in this experiment for out of domain generalization is severely limited. 5/7 datasets belong to the same domain and with 4 being essentially the same dataset (ETTh1, ETTh2, ETTm1, ETTm2). This is not enough to draw conclusions about a "pretrained model". Please check the benchmarks in the Chronos or TimesFM papers. 

d) The model only enables point predictions. In forecasting, one is often interested in the entire distribution of future possibilities.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see the weaknesses. 
Also, it will be good if the authors can use a figure to illustrate patch token for univariate and multivariate data.
soundness: 2
strengths: 1. The paper proposes multivariate next token prediction for time series forecasting. This paradigm unifies univariate, multivariate, and covariate-informed forecasting, by treating them as a long-context generation problem.
2. This paper introduces TimeAttention, a novel self-attention mechanism for time series data. TimeAttention captures fine-grained intra and inter-series dependencies, preserves causality in forecasting, and incorporates position embeddings.
3. Experiments show that extending the context length generally improves accuracy, highlighting the context bottleneck issue.
summary: This paper proposes Timer-XL, a generative Transformer model for uni & mul-variate time series forecasting, which addresses the limitation of short context lengths in existing time series Transformers and allows the model to capture both intra- and inter-series dependencies. Specifically, the authors design TimeAttention, which incorporates relative position embeddings and causal masking to effectively learn temporal dependencies. Evaluations across various benchmarks demonstrate that Timer-XL achieves state-of-the-art performance in univariate, multivariate, and covariate-informed forecasting tasks.
weaknesses: 1. This paper focuses heavily on comparing Timer-XL with other Transformer models, particularly PatchTST, Timer, and lacks a broader comparison with other non-Transformer time series forecasting models. Also, how is Timer-XL compared with some recent LLM-based models? e.g., [1], [2].
2. This paper doesn't extensively discuss the computational cost of Timer-XL. Though it provides a theoretical derivation, a more detailed analysis of the computational resources required, especially when handling high-dimensional time series with long contexts. E.g., how about the number of parameters or training/inference time of Timer-XL as compared with existing solutions. 
3. The extension from univriate to multivariate seems to be an over-simplied way, in that using RoPE to as a positional embedding. Take traffic data as an example, how can RoPE reflect the spatial correlations among traffic network?
4. The authors emphasize long context for time series forecasting, however, for some domains, it may not be necessary for such a long context, e.g., traffic data with periodicities. This is also shown in Figure 3.  Also, one can define a large patch token (more time points). In this way, can the context length also be shortened?

[1] Time-LLM: Time Series Forecasting by Reprogramming Large Language Models, ICLR 2024

[2] Large Language Models Are Zero-Shot Time Series Forecasters, NeurIPS 2023

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to Weaknesses.
soundness: 3
strengths: 1. The manuscript demonstrates a high level of completeness.
2. It provides an effective large-scale framework for advancing large models in time series analysis.
3. The empirical evaluation is comprehensive and promising results are shown.
summary: This paper introduces Timer-XL, a method for unified time series forecasting, which addresses challenges in hard to uniformly predict 1D and 2D time series. It utilizes generative Transformer with relative position embedding to capture the correlations between the temporal and variable dimensions. Also, it proposes a novel TimeAttention to capture causal patch-wise dependencies within and among all variables. Experiments demonstrate that Timer-XL achieves state-of-the-art performance across challenging forecasting benchmarks through a unified approach,
weaknesses: 1. My biggest question lies with Equations 4, 5, 7, and 8. Since Timer-XL is a unified time series framework, these equations include processing steps that sort each time series. For example, with 𝑁 time series, does the order after flattening the sequences significantly affect the causal relationship in Equations 4 and 5? The same question applies to Equations 7 and 8.
2. As a unified time series framework, the author needs to compare it with Moirai in a zero-shot scenario. In Figure 5 of the manuscript, a corresponding comparison experiment should be added.

Meta-review: This paper introduces a generative transformer model for unified timeseries forecasting. They propose a new form of TimeAttention which allows for the application of generative transformers on multivariate data and demonstrate improvements in long-context regimes. They provide results with both supervised training as well as unified pretraining across many datasets. 

The reviewers all agreed that the paper addresses an important and timely topic, and that the work has a number of noteworthy contributions. However, the reviewers also noted a few weaknesses in the initial submission, including limited novelty over approaches like Morai and that the move from univariate to multivariate data is handled in an oversimplistic manner. Reviewers also brought up some concerns around the motivation of long context and whether the experiments adequately demonstrated that this improvement is helpful in practical settings.

During the rebuttal phase, the authors did an excellent job of responding to these concerns, and further clarified the novelty of their work.

Decision: Accept (Poster)