Timer-XL: Long-Context Transformers for Unified Time Series Forecasting

Authors: Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, Mingsheng Long
Primary area: learning on time series and dynamical systems
Keywords: Time Series Forecasting, Transformer, Foundation Model

Abstract:
We present Timer-XL, a causal Transformer for unified time series forecasting. To uniformly predict multidimensional time series, we generalize next token prediction, predominantly adopted for 1D token sequences, to multivariate next token prediction. The paradigm formulates various forecasting tasks as a long-context prediction problem. We opt for decoder-only Transformers that capture causal dependencies from varying-length contexts for unified forecasting, making predictions on non-stationary univariate time series, multivariate series with complicated dynamics and correlations, as well as covariate-informed contexts that include exogenous variables. Technically, we propose a universal TimeAttention to capture fine-grained intra- and inter-series dependencies of flattened time series tokens (patches), which is further enhanced by deft position embedding for temporal causality and variable equivalence. Timer-XL achieves state-of-the-art performance across task-specific forecasting benchmarks through a unified approach. Based on large-scale pre-training, Timer-XL achieves state-of-the-art zero-shot performance, making it a promising architecture for pre-trained time series models. Code is available at this repository: https://github.com/thuml/Timer-XL.

Decision: accept (poster)
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730350042297

thread_type:
official_review

<thread_object>
creator: reviewer_v4mu

<content>
summary:
This paper introduces Timer-XL, a method for unified time series forecasting, which addresses challenges in hard to uniformly predict 1D and 2D time series. It utilizes generative Transformer with relative position embedding to capture the correlations between the temporal and variable dimensions. Also, it proposes a novel TimeAttention to capture causal patch-wise dependencies within and among all variables. Experiments demonstrate that Timer-XL achieves state-of-the-art performance across challenging forecasting benchmarks through a unified approach,

strengths:
1. The manuscript demonstrates a high level of completeness.
2. It provides an effective large-scale framework for advancing large models in time series analysis.
3. The empirical evaluation is comprehensive and promising results are shown.

weaknesses:
1. My biggest question lies with Equations 4, 5, 7, and 8. Since Timer-XL is a unified time series framework, these equations include processing steps that sort each time series. For example, with 𝑁 time series, does the order after flattening the sequences significantly affect the causal relationship in Equations 4 and 5? The same question applies to Equations 7 and 8.
2. As a unified time series framework, the author needs to compare it with Moirai in a zero-shot scenario. In Figure 5 of the manuscript, a corresponding comparison experiment should be added.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730362106755

thread_type:
official_review

<thread_object>
creator: reviewer_mkp7

<content>
summary:
This paper proposes Timer-XL, a generative Transformer model for uni & mul-variate time series forecasting, which addresses the limitation of short context lengths in existing time series Transformers and allows the model to capture both intra- and inter-series dependencies. Specifically, the authors design TimeAttention, which incorporates relative position embeddings and causal masking to effectively learn temporal dependencies. Evaluations across various benchmarks demonstrate that Timer-XL achieves state-of-the-art performance in univariate, multivariate, and covariate-informed forecasting tasks.

strengths:
1. The paper proposes multivariate next token prediction for time series forecasting. This paradigm unifies univariate, multivariate, and covariate-informed forecasting, by treating them as a long-context generation problem.
2. This paper introduces TimeAttention, a novel self-attention mechanism for time series data. TimeAttention captures fine-grained intra and inter-series dependencies, preserves causality in forecasting, and incorporates position embeddings.
3. Experiments show that extending the context length generally improves accuracy, highlighting the context bottleneck issue.

weaknesses:
1. This paper focuses heavily on comparing Timer-XL with other Transformer models, particularly PatchTST, Timer, and lacks a broader comparison with other non-Transformer time series forecasting models. Also, how is Timer-XL compared with some recent LLM-based models? e.g., [1], [2].
2. This paper doesn't extensively discuss the computational cost of Timer-XL. Though it provides a theoretical derivation, a more detailed analysis of the computational resources required, especially when handling high-dimensional time series with long contexts. E.g., how about the number of parameters or training/inference time of Timer-XL as compared with existing solutions. 
3. The extension from univriate to multivariate seems to be an over-simplied way, in that using RoPE to as a positional embedding. Take traffic data as an example, how can RoPE reflect the spatial correlations among traffic network?
4. The authors emphasize long context for time series forecasting, however, for some domains, it may not be necessary for such a long context, e.g., traffic data with periodicities. This is also shown in Figure 3.  Also, one can define a large patch token (more time points). In this way, can the context length also be shortened?

[1] Time-LLM: Time Series Forecasting by Reprogramming Large Language Models, ICLR 2024

[2] Large Language Models Are Zero-Shot Time Series Forecasters, NeurIPS 2023

questions:
Please see the weaknesses. 
Also, it will be good if the authors can use a figure to illustrate patch token for univariate and multivariate data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577935807

thread_type:
official_review

<thread_object>
creator: reviewer_r6jb

<content>
summary:
The paper proposes Timer-XL, a transformer decoder model for time series forecasting. Building upon the existing Timer model, Timer-XL extends the model with a longer context and a masking-based approach called TimeAttention to handle multivariate/covariate scenarios. Empirical results have been reported on univariate, multivariate and covariate experiments on some benchmark datasets.

strengths:
a) The paper studies an interesting problem of long context modeling in the context of time series forecasting. Authors attempt to connect long context to scenarios beyond univariate modeling through next-token style modeling of multivariate and covariate-informed time series. 

b) Experiments have been conducted on many diverse settings although the experiments themselves have some limitations.

weaknesses:
a) While the problem of long context modeling is interesting, the primary weakness of this work is the lack of clarity about the goal and a proper scope. The discussion is confusing and often only loosely relates to the long context setting which appears to be the primary goal. Authors claim that "existing transformers in the time series field crucially encounter the context bottleneck" which is not as critical of a problem as being portrayed here. Such claims require serious empirical justification which is missing from the paper (experiment 1 does not go too far, see below). In reality, long context scenarios may be helpful but _mostly_ in specific high-frequency scenarios. Consider a 5min granularity time series with weekly seasonal behavior. One would need a context larger than 2K to understand the seasonal behavior from the time series history. Such cases should have been better highlighted to justify the central claim of the paper. That said, long context univariate modeling may not always yield improvements. It may also worsen the accuracy, e.g., in the case of distribution shifts. 

Coming to the utility of long context for multivariate/covariate-informed forecasting, this has been studied (although not explicitly) in Moirai. While the explicit perspective in this paper is interesting, the primary problem is not that one can do multivariate/covariate modeling through long context but that one needs to be able to do it in a zero-shot sense, as attempted in Moirai. As per my understanding, the settings being studied here are task-specific and not a single pretrained Timer-XL model being used for all experiments. Please correct me if I am wrong. 

b) The technical novelty of this work is limited in light of works such as Moirai. "TimeAttention" is a masking scheme that extends causal univariate patch-based modeling to the multivariate setting.

c) The empirical results lack comprehensiveness and are not particularly strong.

**Experiment 4.1**: The benchmark is fairly small to draw conclusions. Even in these scenarios, the benchmark shows that long context yields to diminishing returns (or worse performance) beyond a point, 1 month in most cases. 1 month @ 1h granularity amounts to a context length of 720 which is not far off from the context length of many time series models (Chronos, Moirai, TimesFM, etc.). Furthermore, I don't understand why PatchTST is the only baseline being studied here. I also don't quite understand what's unique about Timer-XL here that yields better performance over PatchTST. Could it be the larger number of parameters? 

Minor: The analysis on normalization does not belong to this section and brings limited value to the discussion. Normalization mostly helps when time series in a dataset have drastically different scales. Models working fine without normalization for a single task of land-surface temperature forecasting is not a surprising result.  

**Experiment 4.2**: The results in Table 2 and 4 are not strong when compared to pretrained baselines such as Moirai. Why is Moirai missing from Fig 4? 

Minor: DeepAR and N-BEATS are not numerical simulation based methods. 

**Experiment 4.3**: More baselines that can incorporate covariates are needed before conclusions can be drawn. For example, you can consider adding N-BEATSx, NHITS, DeepAR, etc. GluonTS also provides an implementation of PatchTST which can incorporate covariates. 

**Experiment 4.4**: The benchmark selected in this experiment for out of domain generalization is severely limited. 5/7 datasets belong to the same domain and with 4 being essentially the same dataset (ETTh1, ETTh2, ETTm1, ETTm2). This is not enough to draw conclusions about a "pretrained model". Please check the benchmarks in the Chronos or TimesFM papers. 

d) The model only enables point predictions. In forecasting, one is often interested in the entire distribution of future possibilities.

questions:
See above.

- Which version of Moirai was used for the experiments? 1.0 or 1.1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732196057940

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve our paper.

In this work, we propose **a unified backbone for one-for-all forecasting**, including one model for varying forecasting horizons (based on Generative Transformers) and one architecture for different forecasting tasks (based on causal self-attention for multivariate time series). Insightfully, we find that **Generative Transformers can better tackle long-context time series**, while previous deep models generally fail. Empirical results include **comparison with state-of-the-art task-specific models** and **zero-shot evaluation by pre-training on billion-level time series points**. 

We're pleased that the reviewers agree our paper "**highlights the context bottleneck issue**" (Reviewer r6jB, MkP7), which we further delved into in this revision, "**demonstrates a high level of completeness with promising empirical evaluation**" (Reviewer V4mu), and "**the proposed self-attention is novel**" (Reviewer MkP7, V4mu).

The reviewers raised insightful and constructive concerns. We **spent nine days** addressing all the issues by providing sufficient evidence and requested results. Here is the summary of the major revisions:

* **Motivation and Scope  (Reviewer r6jB)**: We provided empirical evaluations to emphasize the motivation of long-context forecasting, reclarified the scope of our work, and revised the overall writing of the paper (extra **3 pages** compared with the initial submission).
* **Technical Contributions (Reviewer r6jB, MkP7)**: We clarified the challenges of adopting Generative Transformers for unified time series forecasting. Our implementation facilitates channel-dependent modeling in the flexible decoder-only architecture, an unexploited backbone in this community.
* **Clarification of Experiment Details  (Reviewer r6jB)**: To improve the clarity, we provided the concerned experimental configurations and eliminated potential misleading in the establishment of baseline models.
* **Comprehensive Evaluations (Reviewer r6jB, MkP7, V4mu)**: We included all requested evaluations (100+ experiments in total), including new task-specific baselines, zero-shot benchmarks based on large-scale pre-training, computational cost statistics, and long-context representation analysis. 

All updates are highlighted in blue $\underline{\text{in the revised paper}}$. The valuable suggestions from reviewers are very helpful for us to revise the paper in a better shape. We hope our response has fulfilled the reviewer's expectations and would be very happy to answer any further questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a generative transformer model for unified timeseries forecasting. They propose a new form of TimeAttention which allows for the application of generative transformers on multivariate data and demonstrate improvements in long-context regimes. They provide results with both supervised training as well as unified pretraining across many datasets. 

The reviewers all agreed that the paper addresses an important and timely topic, and that the work has a number of noteworthy contributions. However, the reviewers also noted a few weaknesses in the initial submission, including limited novelty over approaches like Morai and that the move from univariate to multivariate data is handled in an oversimplistic manner. Reviewers also brought up some concerns around the motivation of long context and whether the experiments adequately demonstrated that this improvement is helpful in practical settings.

During the rebuttal phase, the authors did an excellent job of responding to these concerns, and further clarified the novelty of their work.