TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables
Authors: Yuxuan Wang
Haixu Wu
Jiaxiang Dong
Guo Qin
Haoran Zhang
Yong Liu
Yunzhong Qiu
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2024 poster (2024)
Keywords: Transformer
Time Series Forecasting
Exogenous Variable
Primary area: deep_learning_architectures

Abstract: Deep models have demonstrated remarkable performance in time series forecasting. However, due to the partially-observed nature of real-world applications, solely focusing on the target of interest, so-called endogenous variables, is usually insufficient to guarantee accurate forecasting. Notably, a system is often recorded into multiple variables, where the exogenous variables can provide valuable external information for endogenous variables. Thus, unlike well-established multivariate or univariate forecasting paradigms that either treat all the variables equally or ignore exogenous information, this paper focuses on a more practical setting: time series forecasting with exogenous variables. We propose a novel approach, TimeXer, to ingest external information to enhance the forecasting of endogenous variables. With deftly designed embedding layers, TimeXer empowers the canonical Transformer with the ability to reconcile endogenous and exogenous information, where patch-wise self-attention and variate-wise cross-attention are used simultaneously. Moreover, global endogenous tokens are learned to effectively bridge the causal information underlying exogenous series into endogenous temporal patches. Experimentally, TimeXer achieves consistent state-of-the-art performance on twelve real-world forecasting benchmarks and exhibits notable generality and scalability. Code is available at this repository: https://github.com/thuml/TimeXer.

TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables
Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, Mingsheng Long" School of Software, BNRist, Tsinghua University, Beijing 100084, China
(wangyuxu22, whx20, djx200, qinguo24, zhanng -h224, liuyong21, pacp:ngg2,quuuy2200 qiuyz24)mmails. twwngnua. edu. (jimwang,
edw.einnguaeeu......mm edu. en
Abstract
Deep models have demonstrated remarkable performance in time series forecasting. However, due to the partially-observel nature of real-world applications, solely focusing the targer of interest, so-called endogenous variables, is usually insuf- ficient to guarantee accurate forecasting. Notably, a system often recorded into multiple variables, where the exogenous variables can provide valuable external information for endogenous variables. Thus, unlike well-cstablished multivariate or univariate forecasting paradigms that either treat all the variables equally ignore exogenous information, this paper focuses on a more practical setting: time series forecasting with exogenous variables. We propose novel approach, TimeXer, to ingest external information to enhance the forecasting of endogenous variables. With deftly designed embedding layers, TimeXer empowers the canonical Trans- former with the ability to reconcile endogenous and exogenous information, where patch-wise self-attention and variate-wise cross-attention are used simultaneously. Moreover, global endogenous tokens are learned to effectively bridge the causal information underlying exogenous series into endogenous temporal patches. Ex- perimentally, TimeXer achieves consistent state-of-the-a performance on twelve real-world forecasting benchmarks and exhibits notable generality and scalability. Code is available at this repository: https::////// http::///gthttt com Schhhl/Timeell
Introduction
Time series forecasting is of pressing demand in real-world scenarios and have been widely used in various application domains, such metcorology 38 42
electricity [34], and transportation [27]].
Thereof, forecasting with exogenous variables is prevalent and indispensable forecasting paradigm since the variations within time series data are often influenced by external factors, such as economic indicators, demographic changes, and societal events. For example, electricity prices are highly dependent on the supply and demand of the market, and it is intrinsically impossible to predict future prices solely based on historical data. Incorporating external factors in terms of exogenous variables, as illustrated in Figure T (Left), allows for more comprehensive understanding of the correlations and causalities among various variables, leading to better performance and interpretability.
From the perspective of time series modeling, exogenous variables are introduced to the forecaster for informative purposes and do not need to be predicted. The distinction between endogenous and exogenous variables poses unique challenges compared to existing multivariate forecasting methods. First, there are always multiple external factors that are illuminating to the prediction of the target series, which requires models reconcile the discrepancy and dependency among endogenous
"Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Problem Formulation
Practical Situations Exogenous Variables
Performance of TimeXer
Missing Values
Mismatch Frequeecyy
PIM
DE
Exagentnss Variables
Exageriso Varialiie
Forecaster
Temporal Misaligned
Mismatch Length
Renkeer (Ours
Chartomm
Dliverr
(1) x 20))
Exnggemous
Exegertius
PatdEST
Figure I: Left: The forecasting with exogenous variables paradigm includes inputs from multiple ex- ternal variables as auxiliary information without the need for forecasting. Right: Model performance comparison on existing electricity price forecasting with exogenous variables benchmarks.
and exogenous variables. Regarding exogenous variables equally with endogenous ones will not only cause sigmificant time and memory complexity but also involve unnecessary interactions from endogenous series to external information. Second, external factors may have causal effect on endogenous series, so models are expected to reason about the systematic time lags among different variables. Moreover, essential for models to tackle irreuular and heterogeneous exogenous series, including value missing, temporal misalignment, frequency mismatch, and length discrepancy as showcased in Figure Left).
practical forecasting paradigm applied extensively in real scenarios, it is
Despite the success of deep learning models in capturing intricate temporal dependencies time series data, the incorporation of exogenous variables remains underexplored. A common practice to import them is adding or concatenating exogenous features to the endogenous ones. However, given the crucial role of exogenous variables forecasting, it is imperative to incorporate them precisely and properly. Recent Transformers [32 have exhibited remarkable performance in time series forecasting due to their capability of capturing both temporal dependencies and multivariate correlations. Based on the working dimensions of the attention mechanism, existing Transformer- based works can be roughly divided into patch-oriented models and variate-oriented models. Patching is a basic module to preserve the semantic information underlying temporal variations. Therefore, the attention mechanism is applied over patches to unearth the intricate temporal patterns. Based on the channel independence assumption, PatchTST and follow-ups (28) are capable of capturing temporal dependencies but weak at capturing multivariate correlations. In contrast, variate-oriented models represented by iTransformer (3 successfully reason about interrelationships between variables by considering each variate of time series as a single token and applying attention over multiple variate tokens. Unfortunately, they lack the ability to capture internal temporal variations since the whole series embedded into a coarse variate token by a temporal linear projection.
To enable accurate forecasting with exogenous variables in real-world scenarios, it is indispensable to capture both the intra-endogenous temporal dependencies and inter-series correlations between endogenous and exogenous variables. Based on the above observations, we believe that modeling the temporal-wise and variate-wise dependencies within time series data requires hierarchical repre- sentations at different levels. In this paper, we unleash the potential of the canonical Transformer without modifying any component, and propose a Time Series Transformer with eXogenous variables (TimeXer). Technologically, we leverage representations and perform attention mechanisms at both patch and variate levels. First, the endogenous patch-level tokens are applied to capture temporal dependencies. Second, to tackle the arbitrarily irregular exogenous variables, TimeXer adopts their variate-level representations seamlessly ingest the impact of external factors on endogenous ones. Third, inspired by Vision Transformers [10), we introduce learnable global tokens for each endoge- nous series to reflect the macroscopic information of the series, which interact simultaneously with patch-level endogenous tokens and variate-level exogenous tokens. Throughout this information pathway, the external information can be propagated effectively and selectively to corresponding endogenous patches. In summary, our contributions can be listed as follows.
Motivated by the universality and importance of exogenous variables in time series fore- casting, we empower the canonical Transformer to simultaneously modeling exogenous and endogenous variables without any architectural modifications.
We propose a simple and general TimeXer model, which employs patch-level and variate- level representations respectively for endogenous and exogenous variables, with an en-

---

dogenous global token as a bridge in-between. With this design, TimeXer can capture eara--endggeous temporal dependencies and Exogemous-to-cndogenoo correlations jointly,
Extensive experiments twelve datasets show that TimeXer better utilize exogenous information to facilitate endogenous forecasting, in both univariate and multivariate settings.
2 Related Work
2.1 Transformer-based Time Series Forecaster
Motivated by the great success in the field of natural language processing 7 and computer vision 26]. Transformers have garnered significant interest in time series data due their ability to capture
long-term temporal dependencies and complex multivariate correlations. Categorized based on the granularity of representation used in the attention mechanism, Transformer-bic divided into point-wise, patch-wise, and variate-wise. Due to the serial nature of time series, most previous works use a point-wise representation time series data and apply attention mechanisms to capture the correlations among different time points. Therefore, many efficient Transformers 45 were proposed to reduce the complexity caused by point-wise modeling. Informer designs ProbSparse self-attention reduce the quadratic complexity in time and memory. Autoformer (37] replaces canonical self-attention with Auto-correlation
models can be
22
discover sub-series similarity within time series data. Pyraformer [22 develops pyramidal attention module to capture both short- and long-temporal dependencies with linear time and space complexity.
Considering point-wise representations fall short in revealing local semantic information lies in the temporal variation, PatchTST (88 split time series data into subseries-level patches and then capture dependencies between patches. Pathformer [4] utilizes multi-scale patch representations and performs dual attention over these patches capture global correlations and local details as temporal dependencies. Recent large-scale time series models 22 6 [5] 46, 8 have widely included
patch-level representation learn the complex temporal patterns. Beyond capturing the patch-level
temporal dependencies within one single series, recent approaches have endeavored to capture interdependencies among patches from different variables over time. Crossformer 43 introduces Two-Stage Attention layer to efficiently capture the cross-time and cross-variate dependencies of each patch. Further expanding the receptive field, iTransformer [23] utilizes the global representation of the whole series and applics attention to these series-wise representations capture multivariate correlations. Yet, as shown Table [ most of the cxisting Transformer-baa based approaches only focus multivariate univariate time series forecasting paradigms and do not conduct special designs for exogenous variables, which is different from the scenario we studied in this paper.
Table I: Comparison of related methods with its forecasting capability. The character "" in the Transformers denotes the name of *former. The character + indicates that the model can be applied multivariate forecasting scenarios but not explicitly model the cross-variae dependency.
Methods
TimeXer iTran. 23 PatchTST
(2
Cross.
43
Auto.
37
TFT [6
NBEATSx
29 TiDE 5
Univariate
/
X
x
X
Multivariate
/
X
Exogenous
X
X
V
/
2.2 Forecasting with Exogenous Variables
Time series forecasting with exogenous variables has been widely discussed in classical statistical methods. vast majority statistical methods have been extended to include exogenous variables as part of imput. Extending the well-acknowledged ARIMA model, ARIMAX [35] and SARIMAX BT incorporate the correlations between exogenous and endogenous variables along with the autoregression on endogenous variables. Although time series modeling methods have evolved from statistical to deep models, most of the existing deep models incorporating covariates, such as Temporal Fusion Transformer (TFT) 20), primarily focus on variable selection. Some approaches, including NBEATSx 29) and TiDE 5 contend that forecasting models are capable of accessing future values of exogenous variables during the prediction of endogenous variables. It is notable that previous models concatenate exogenous features with endogenous features at each time point and then map them to a latent space, necessitating the alignment of the endogenous and exogenous
3

---

series in time. However, time series in real-world applications often suffer from problems such as missing value and uneven sampling, which leads to significant challenges in modeling the effects of exogenous variables on endogenous variables. In contrast, TimeXer introduces external information to Transformer architecture through a deftly designed embedding strategy, which can effectively introduce the external information into patch-wise representations of endogenous variables, thereby being able to adapt to time-lagged or data-missing records.
3 TimeXer
In forecasting with exogenous variables, the endogenous series is the target to be predicted, while the exogenous series are covariates that provide valuable information boost endogenous predictability,
Problem Settings In forecasting with exogenous variables, we are given an endogenous time series
X1:7 (2., E RTTx and multiple exogenous series 21:T, = (21:T, 2..,, 1:T. E
(C)
RTaxC_ Here x; denotes the value at the i-th time point, 21.r. represents the i-th exogenous variable, and C' is the number of exogenous variables. In addition, T and Tex are the look-back window lengths of the endogenous and exogenous variables respectively. Noteworthily, any series that provides useful information for endogenous forecasting can be used as an exogenous variable, regardless of their look-back lengths, so we relax to the most flexible settings with Tex + T. The goal of forecasting model Fo parameterized by 8 is to predict the future S time steps x = {Ir+1, TT-2: Tr+s} based on both historical observations X1:7 and corresponding exogenous series Z1:T..
11+1T+5 Fo (X1:7, Z1:T,.)
(1)
Structure Overview As shown in Figure 2 the proposed TimeXer model repurposes the canonical Transformer without modifying any component, while endogenous and exogenous variables are manipulated by different embedding strategies. TimeXer adopts self-attention and cross-attention to capture temporal-wise and variate-wise dependencies respectively.
Endogenous Embedding Most of the existing Transformer-be bbased forecasting models embed each time point or segment of time series as a temporal token and apply self-attention to learn temporal dependencies. To finely capture temporal variations within the endogenous variable, TimeXer adopts patch-wise representations. Concretely, the endogenous series is split into non-overlapping patches, and each patch is projected temporal token. Given the distinct roles of endogenous and exogenous variables in the prediction, TimeXer embeds them at different granularity. Therefore, directly combining endogenous tokens and exogenous tokens at different granularity will result in information misalignment. To address this, we introduce a learnable global token for each endogenous variable that serves as the macroscopic representation to interact with exogenous variables. This design helps bridge the causal information from the exogenous series to the endogenous temporal patches. The overall endogenous embedding is formally stated as:
{ss S2, ...y SN} } Patchify (x).
P PalchEmbed (81, 82, SN),
Gon Learnable (x).
Denote by P the patch length, by N and by s; the i-th patch. (mbed() maps each length-P patch, added by its position embedding, into a D-dimensional vector via a trainable linear projector. In all, N patch-level temporal token embeddings Pon cn and series-level global token embedding Gen are fed into the Transformer encoder.
N=
==17 [J the number of patches split from the endogenous series,
Exogenous Embedding The primary use of exogenous variables to facilitate accurate forecasting of endogenous variables. We will show in Appendix B.3 that the interactions of different variables can be captured more naturally by variate-level representations, which are adaptive to arbitrary irregularities such missing values, misaligned timestamps, different frequencies. look-back lengths. In contrast, patch-level representations are overly fine-graine one-grained for exogenous only significant computational complexity but also unnecessary noise
discrepant
variables, introducing information. These insights lead to a design that each exogenous series is embedded in a series-wise variate token, which is formalized as:
V ex EabieteEmbed (z (0), E (1,... .C).

---

Oulput
(ii Exogenous Series Embedding
Projection
Wb Exngenous Exogenoax-o--Flloggnn Ennloggoous Cross-Altention
LayerNorm
Mwww MM-Y
Variate-aiin
Altention
Mmm
Ford-Forward
CrossAAtvntion
myy
Multtppe Variates
LayerNorm
Query
Cross-Attention
Forward
Layer
Exogenous Input
Embedding
LayerNorm
Endogenous Self-Altention
on Indagenous Series Embedding
Temporal-wise Altention
Input Series
Random-iitiallied
Global Token
Self-Altention
Key
Seli-Attention
Embeddding
Non Non-overlap Patches
Query
(N-1) (N--)
Endagenous Input
The schematic of TimeXer, which empowers time series forecasting with exogenous variables. (a) The endogenous embedding module yields multiple temporal token embeddings and one global token embedding for the endogenous variable. (b) The exogenous embedding module yields variate token embedding for each exogenous variable. (c) Self-attention applied simultaneously over the endogenous temporal tokens and the global token to capture patch-wise dependencies. (d) Cross-attention applied over endogenous and exogenous variables to integrate external information.
Figure 2:
Here VariateFmbed RT - RDD is trainable linear projector, Tex is the look-back length of exogenous series, and Vex = (Vex. ] is the set of representations for multiple exogenous series.
Endogenous Self-Attention For accurate time series forecasting, it is vital to discover intrinsic tem- poral dependencies within the endogenous variable, as well as the interactions with the variate-level representations from exogenous variables. In addition to self-attention kens (Patch-to-Patch) the learnable global token builds a bridge between endogenous and exogenous variables. Concretely, the global token plays asymmetric role in cross-attention: (1) Patch-to- Global: the global token attends to temporal tokens for aggregating patch-level information across the entire series; (2) Global-to- Patch: each temporal token attends to the global token for receiving the variate-level correlations. This provides a comprehensive view of the temporal dependencies within the endogenous variable, well better interactions with the arbitrarily irregular exogenous variables. The attention mechanism can be formalized as follows:
endogenous temporal to-
Patch-to-Patch: 1 LayerNorm (PL en Sclf-Attention (P'))
Global-to-Patch: pL.2 LayerNorm (PL Cross-Attention (P' G
(4)
Patch-to-Global: G = LayerNorm (G' + Cross-Attention (G P))))
The overall process can be simplified into an endogenous self-attention computation:
pl en G LayerNorm
[P G'n Self-Attention ((P G²ul)).
(5)
where E (O...., L - 1} denotes the 1-th TimeXer block, and P
en = Pen- Go = G Here, I.1 denotes the concatenation of the patch-wise tokens and global token of the endogenous variable along the sequence dimension. By adopting a self-attention layer over the concatenated tokens [PL on G of the endogenous series, TimeXer relationships between each patch to the entire series simultaneously.
en
capture temporal dependencies between patches and the
Exegenous-tov Eon---ooo-ddonouus Cross-Attention Cross-attention has been widely used in multi-modal learning [17] to capture the adaptive token-wiss dependencies between different modalities. In TimeXer, the tross-attention layer takes the endogenous variable as query and the exogenous variable as key and value to build the connections between the two types of variables. Since the exogenous variables embedded into variate-level tokens,
the learned global token of the endogenous variable to aggregate information from exogenous variables. The above process can be formalized as
Variate-to-Glooal: G. LayerNorm (G', Cross- Atross-Attention (G) Vex)).
(6)
5

---

Finally, all temporal tokens and the learnable global token will be transformed by the feedforward layer, which is formally stated as:
p Freed-Forwadd (P'n), G Feed-Forward (.)
en
where E (1,..... L.. We write each Transformer block as P G = TrmBlock en: G.
en
Forecasting Loss In time series forecasting with exogenous variables, the exogenous variables do not need to be predicted. So we generate the forecast x by applying a linear projection on the endogenous output embeddings [P/n G common practice in the encoder-only forecasters. We employ the squared loss (1.2) to measure the discrepancy between the prediction and the ground truth:
Loss Σ
x
1 where X = Projection (PP G
&
(8)
Parallel Multivariate Forecasting Multivariate forecasting can be viewed predicting each variable in the multivariate data, with the other variables treated as exogenous ones. So for each variable, the other variables are leveraged by TimeXer to facilitate more accurate and causal prediction. Our key discovery that forecasting with exogenous variables can unified forecasting paradigm that generalizes straightforwardly to multivariate forecasting. By employing the channel independence mechanism, for each variable of the multivariate, it treated as the endogenous one. Then TimeXer is applied in parallel manner for all variables with shared self-attention and cross-attention layers.
4 Experiments
To verify the effectiveness and generality of TimeXer, we extensively experiment under two different time series paradigms, i.e. short-term forecasting with exogenous variables and long-term multivariate forecasting, on a diverse range of real-world time series datasets from different domains. We also conduct experiments long-term forecasting with exogenous variables on the multivariate benchmark, which are presented in Appendix L.3
Datasets For short-term forecasting tasks, include short-term electricity price forecasting datasets (EPF) [[5], which is a real-world forecasting with exogenous various benchmarks derived from five major power market data spanning six years each. Each dataset contains electricity price as an endogenous variable and two influential exogenous variables in practice. Meanwhile, we adopt seven well-established public long-term multivariate forecasting benchmarks performance of TimeXer in multivariate forecasting.
33
evaluate the
Baselines We include nine state-of-the of-the-artt deep forecasting models, including TYansformer-basee
models: iTransformer [23], PatchTST (28). Crossformer [43] Autoformer (37). CNN-based models:
TimesNet (36), SCINet (21), and linear-based models: RLinear [19], DLinear [41), TiDE (5). Notably,
TiDE is a recently developed advanced forecaster specifically designed for exogenous variables.
Implementation Details
For short-term electricity price prediction, follow the standard protocol of NBEATSx [29]. where the input series length and prediction length are respectively set as 168 and 24. In addition, we set the patch length as 24 without overlapping. For long-term forecasting datasets, we uniformly use the patch length 16 and fix the length of the look-back series at 96, while the prediction length varies across four lengths (96, 192, 336, 720).
4.1 Main Results
Comprehensive forecasting results for short-term and long-term forecasting are listed Table 2 and
Table 3
A lower MSE or MAE indicates better forecasting performance.
The short-term electricity price forecasting task is derived from real-world scenarios, and presents unique challenge for the forecasting model for the endogenous variable has been shown to be highly correlated with two exogenous variables the dataset. Since the interactions between different variables are crucial for this task, linear forecasters, including RLinear [19 and DLinear 41, fail to triumph over Transformer-basee
forecasters. Similar to TimeXer, Crossformer divides all input series into different segments and captures multivariate correlations over all segments; However,

---

it fails to outperform other baselines which indicates that modeling all variables at granular level introduces unnecessary noise into the forecasting. Also designed for capturing cross-variate dependency, i'Transformer neglects the temporal-wise attention module, indicating that there are still limitations in capturing temporal dependencies solely through linear projection. By contrast, our proposed TimeXer effectively integrates information from exogenous variables while capturing TimeXer achieves consistent
temporal dependencies of endogenous series. As shown in Table 2 state-of-the-ar performance on all five datasets, outperforming various baseline models.
Table 2: Full results of the short-term forecasting task on EPF dataset. We follow the standard protocol in short-term electricity price forecasting, where the input length and predict length are set to 168 and 24 respectively for all baselines. Avg means the average results from all five datasets.
Model|
TimeXer iTransfommer RLinear PatchTST
Crossformer
TiDE
TimesNet
DLinear
SCINet
Autoformer
Metric
MSE MAE MSE MAE MSE MAE MSE MAE MAE MSE MSE MAE MSE MAE MAE MSE MAE
NP 0.2336 0. 0.268 0.265 0.300 |o.335 O. 0.27 0.284/0.240 0,285 0.335 O. 0.34000..00000000000 0.99 0.309 0. (0.373 0.368 0.402 0.398
PJM
(0.0930.1920.099 0.197 (o. |o.124 0.429
0.209/0.101 101 0.199 12422422 0.124 0.2.2/0.022092. 0.19 0. 10 0.215 143.0.253099900 4....55500.66.. 0.267
BE |0.379 0.3394 0.270 |0.520 000.:33770.4002222222..
0.290 0.523 0.3366003119000...
0.463 0.0. 0.412
FR 0.20850. 0.439 0.233 01..077029000411122220
0.208
DR
|0.440
0.443 |0.574 0.498 0.5740.44880.461 0.461 0.4444999990441114434330 0.430 0.568
AVG 0.2657..265003335 0.289
0.284 0.412 http::/.................003333.0
We also evaluate XemeXer on well-established public benchmarks for conventional multivariate long- term forecasting. As mentioned above, TimeXer has the ability perform multivariate forecasting by employing the channel independence mechanism. We present the results averaged from all four prediction lengths in Table It can be observed that TimeXer achieves consistent state-of-the-at performance on most of the datasets, highlighting its effectiveness and generality. In addition, since TimeXer is initially designed for exogenous variables, we also conduct vanilla forecasting with exogenous variables on these datasets by taking the last dimension of the multivariate data as endogenous series and others as exogenous variables. Detailed results are listed in Appendix [.3
Table 3: Multivariate forecasting results. We compare extensive competitive models under different prediction lengths following the setting of i'Transformer (3). The look-back lenghh is set to 96 for all baselines. Results are averaged from all prediction lengths $ = (96, 192, 336, 720).
Model
TimeXer iTrunstormer RLineur
PatchTST
Crossformer
TiDE
TimesNet
DLinear
SCINet
Autoformer
Metric MSE MAE|MSE MAF MSE MAE] MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSF MAE
ECL (0.171 0.270Jo.178 0.270 0.2190.29880. 205 0.20/00.244 244 0.334 0.251 0. 0.244/0. 192.2444...2902220 090.220090002222.000000...
Weather 0.271 (0.258 (0.258 0.278 0.2720.291(0.2990288111
0.315 (0.271 222.:/10.....20000......2....331 0.382
ETThl
|0.437 0.437 0.447 0.446 0.44660.4440.444000455500000 0a446..s|e4..000..00.00020000110 011.1//05...................0440
0.541
ETTh2 |0.367 0.396|0.338 0.407 0.374
mm |0.382 0.397/0.407 0.410 wtw.::/www0.............00..0000
ETTm2
0.32| 0.332
Traffic
|0.4660.28770.425 0.282 0.626 0.3.60..7....811..4000..8..3 00t00000000000000000000000000000
4.2 Ablation Study
In TimeXer, three types of tokens are used to capture temporal-wise and variate- wise dependencies, including multiple patch-level temporal tokens, learnable global tokens of the endogenous variables. and multiple variate-level exogenous tokens. Besides, to incorporate the information from exogenous variables, TimeXer adopts cross-attention layer to model the mutual relationship between different variables. To elaborate on the validity of TimeXer, we conducted detailed ablations covering both the embedding module and the inclusion of exogenous factors. Specifically, for the embedding design, we replace or remove existing components of the embedded vector from exogenous and endogenous variables respectively. Moreover, we keep the existing embedding design and replace the cross-attention by adding the variate token of exogenous variables to the variate token of endogenous variables or concatenating all the variate tokens and temporal tokens. As listed in Table TimeXer exhibits superior performance compared to various architectural designs across all datascts.
7

---

Table 4: Ablation Results. Ex. and En. are abbreviations for Exogenous variable and Endogenous variable, P, G and V denote patch token, learnable global token, and variate token respectively.
Design
NP
PJM
BE
FR
DR
AVG
En.
MSE
MAE
MSE
MAE
MSR
MAF
MSE
MAE
MSR
MAE
MSE
MAE
Ours
P+G
0.236
0.268
0.093
0.192
0.379
0.243
0.385
0.208
0.440
0.415
0.307
0.265
Cross
Replace
P+G
0.237
0.269
0.101
0.196
0.376
0.246
0.390 0.206
0.457
0.422
0,312
0.268
Remove
0.239
0.273
0.106
0.200
0.381
0.260
0.393
0.208
0.468
0.425
0.316
0.273
Add
P+G
0.247
0.272
0.125
0.206
0.387
0.247
0.404
0.209
0.483
0.430
0.329 0.273
Concatenate
P+G
V 0.237
0.266
0.098
0.196
0.383
0.255
0.390
0.209
0.450
0.423
0.312 0.270
4.3
TimeXer Generality
4.3.1
Practical Situations
Increasing Look-back Length Theoretically, the forecasting performance of the model could potentially benefit from increasing the look-backe back length of time series, as a longer historical context encompasses more comprehensive information. However, the attention will be distracted when the look-back length becomes excessively long. In TimeXer, we use the variate-leve representation of exogenous variables which allows for the misalignment between endogenous and exogenous variables. This is particularly valuable in real-world scenarios where the time series data may be collected from newly introduced sensor that has limited historical data. Therefore, we conducted three different experimental settings to assess the generality of TimeXer by increasing the length of either the endogenous exogenous series, which include "Fix Endogenous and increase Exogenous", "Increase Endogenous and Fix exogenous", and "Increase Endogenous and Exogenous". Results shown in Figure reveal that XimeXer can be adapted to situations where the look-back of endogenous and exogenous are mismatched. Moreover, extending the look-back length indeed yields improvements in forecasting performance. Compared to enlarging the historical exogenous series, increasing the look-back length of the endogenous series brings greater benefits to the model, and the performance further improved with both increases.
Endogenous and Increase Exogenous
Exogenous and Endogenous
Increase Endogenous and Exogenous
0.15
0.14
0.14
0.13
0.13
0.13
0.12
0.12
0.12
336
770
336
3.16
512
pred_eeme Jen
pred_len- 152
pred_een= 336
pred_ Je0
Figure 3: Performance with the enlarged look-back length varying from {96, 192, 336, 512, 720]. Different styles of lines represent different prediction lengths. most cases, the forecasting perfor- mance benefits from enlarged look-back lengths of both endogenous and exogenous series.
To further verify the generalizability of TimeXer in complex real-world scenarios, we conduct experiments in scenarios where the historical information of time series is missing. Specifically, for both exogenous and endogenous series, we adopt two strategies to evaluate TimeXer's adaptability to series with missing historical information: (1) Zeros: filling the whole series with the scalar value O. (2) Random: substituting the whole series with random values from uniform distribution on the interval [0, 1). As shown in Table 5 the forecasting results deteriorate when exogenous variables are replaced with meaningless noise, indicating that the model's performance benefits from the inclusion of informative exogenous variables. Interestingly, neither using zero-filled exogenous series nor employing exogenous series with random numbers results in significant
Missing Values
8

---

Table 5: Model performance under missing values. Zeros and Random represent the cases that the corresponding series is set as zeros random values respectively.
NP
PJM
DE
AVG
Variate
Strategies
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Zeros
2.954
1,396
0.188
0.288
0.930 0.664
0.781 0.534
0.774 0.559
1.125 0.688
Endogenous
Random
3.140
1,450
0.233
0.325
0.926 0.667
0.761
0.527
0.692 0.533
1.150 0.701
Zeros
0.257
0.278
0.108
0.210
0.400
0.254
0.446
0.214
0.471
0,430
0.330
0.277
Exogenous
Random
0.258
0.280
0.110
0.212
0.399
0.253
0.424
0.221
0.475
0.432
0.333
0.280
TimeXer
0.236
0.268
0.093
0.192 0.379 0.243
0.385
0.208 0.440
0.415
0.307 0.265
decline in model performance. This robustness can be attributed to TimeXer's design, which uses two attention layers to model endogenous temporal dependencies and the multivariate correlations between endogenous and exogenous variables respectively. This architecture allows endogenous temporal representations to dominate the predictions, ensuring consistent performance even in the presence of uninformative exogenous data. Consequently, can be observed that when the endogenous series is replaced with meaningless zeros or random values, rendering the time series unpredictable, there is significant decline model performance. This underscores that TimeXer's performance is closely tied to the quality of endogenous series, deteriorating markedly when the historical information severely limited.
4.3.2 Scalability
Since recent Transformer-basee forecasters have demonstrated promising scalability, leading to the success of Large Time Series Models, we explore the scalability of TimeXer on large-scale time series data. Specifically, we build a large-scale weather dataset for forecasting with exogenous variables. The endogenous series the hourly temperature of 3,850 stations worldwide, spanning from January 1, 2019, to December 31, 2020, which can be downloaded from the National Centers for Environmental Information (NCEI) I and has been well-processed by [8). Further, we utilize meteorological indicators of corresponding adjacent areas from ERA5 [ exogenous variables, which is with a sampling interval of hours. The adjacent area is defined as the 3x3 grid centered on the endogenous weather station, with four meteorological variables per grid cell, totaling 36 exogenous variables. We set the historical horizon endogenous and exogenous to be 7 days to predict the endogenous variable for the next 3 days. Noteworthily, this is complex forecasting scenario as we aforementioned where the frequencies of endogenous and exogenous are different. We choose existing state-of-the- thetthe-rt multivariate forecasters baselines and use identical hidden dimensions and batch sizes for a fair comparison. Since baseline forecasters cannot handle mismatched series, we interpolate the exogenous series into hourly data using the nearest values. Figure 4 demonstrates that TimeXer surpasses other baselines, verifying its capability to handle large-scale forecasting tasks.
Partial Observations From Stations
Problem Formulation
Model Performance
Endogenous Variable
0.216
0.212
MSE
0.207
0.208
Exogenous Variables
0.200
Figure Forecasting performance on large-scale time series datasets. Left: forecasting scenario. The endogenous is the temperature collected from weather stations, and the exogenous variables are meteorological indicators from the surrounding 3x3 grids including the weather station. Each area contains four types of information, namely, temperature, pressure, and V- components of wind. Right: TimeXer outperforms other advanced forecasters.
Illustration of the

---

4.4 Model Analysis
Wiri Correlations TimeXer adopts Pross-attentionn between the global endogenous token and variable-level exogenous tokens to capture the multivariate correlation, enhancing the inter- pretability of the learned attention map. To validate the rationale behind attention on variate tokens, we visualize the learned attention map alongside the time series of the highest and lowest attention scores. As illustrated in Figure (Left), the case study on the Weather dataset reveals a notable distinction in the attention maps of endogenous variables with different exogenous variables. This demonstrates that TimeXer has the ability to distinguish between exogenous variables, allocating greater attention to those that are most informative for prediction, thereby resulting in more focused and interpretable attention map. Additionally, it observed that exogenous series exhibiting similar shapes to the endogenous series tend to receive more attention. This phenomenon may arise because time series with analogous shapes often share temporal features, leading to higher similarity scores. Consequently, the exogenous series most prominently highlighted by the attention mechanism may intuitively resemble the endogenous variable. Furthermore, physical interpretations for the visualized provided. For the endogenous variable CO2-Concentration there is indeed strong correlation between it and Air Density, while the Maximum Wind Velocity has a relatively minor impact, which validates the effectiveness of TimeXer.
Model Efficiency To evaluate the efficiency of TimeXer, we evaluate the training time and memory footprint of TimeXer forecasting with exogenous variables compared with six baseline models with the identical hidden dimension and batch size for a fair comparison. We present the results on the ECL dataset with 320 exogenous variables in Figure 5 (Right). It is notable that when faced with numerous variables TimeXer exhibits its advantage by outperforming iTransformer in terms of memory footprint. Notably, iTransformer embeds each variate series into one token and applies self-attention mechanism among all variate tokens, whether endogenous or exogenous. Although this design can keep refining the learned variate token in multiple layers, it does cause more complexity. As for TimeXer, exogenous variables will be embedded to variate tokens at the beginning, which will be shared in all layers and interact with the endogenous global token by cross-attention Thus, TimeXer omits the interaction among learned exogenous variate tokens, resulting in favorable efficiency. We provide a comprehensive theoretical analysis of the model efficiency in Appendix
Endogenous Variable
Exogenous Variable 10 Air Density
FCL (320 Exogenous Varlables)
CO2-Concentration
Attention
Value
30
40
6o
2o
C.40
Exogenous Variable 12
Maximum Wind Velocily
0.36
Consulorion
0
40
20
20
40
60
40
.0
Time
Time
Training Time (ms/iter)
Figure 5: Model analysis of TimeXer. Left: Visualization of learned attention map and the endogenous time series and exogenous time series with highest and lowest attention scores. Right: Model efficiency comparison under the forecasting with exogenous variables paradigm on the ECL dataset.
5 Conclusion
Considering the prevalence of exogenous variables in real-world forecasting scenarios. we empower the canonical Transformer architecture with the ability to incorporate exogenous information without architectural modificationss. Technologically, TimeXer revisits the attention mechanism in per-patch- per-variate manner to capture both endogenous temporal dependencies and multivariate correlations between endogenous and exogenous variables. With a deftly designed global token, our proposed TimeXer is able to reconcile variables of different purposes. Experimental results demonstrate that our proposed TimeXer effectively ingests exogenous information to facilitate the prediction of endogenous series, in both univariate and multivariate settings. Besides, TimeXer has shown the potential scalability and promising abilities to address complex real-world forecasting scenarios, including challenges such as value missing, temporal misalignment, or series heterogeneity.
10

---

Acknowledgments
This work was supported by the Ministry of Industry and Information Technology of China.
References
[1] Global forecast system. https: /tt::://wwwwwww noaa /ovego
[2] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sehastian Pineda Arango, Shubham Kapoor, al. Chronos: Learning the language of time series. arXiv preprint arXiv2240..08815, 2024.
[3] Wanlin Cai, Yuxuan Liang, Xianggen Liu, Jianshuai Feng, and Yuankai Wu. Msgnet: Learning sulti-scale inter-series correlations for multivariate time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1 11141-11149, 2024.
[4] Peng Chen, Yingying Zhang, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. Multi-scale transformers with adaptive pathways for time series forecasting. In ICLR, 2023.
[5] Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprint (rXii:2300408424,, 2023.
[6] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. ICML, 204.
[7] Jacob Devlin, Ming-Woi Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1.I.:18100008055 2018.
[8] Jiaxiang Dong, Haixu Wu, Yuxuan Wang, Yunzhong Qiu. Li Zhang, Jianmin Wang, and Mingsheng Long. Timesiam: A pre-training framework for siamese time-serics modeling. In ICML, 2024.
[9] Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. In NeurlPS, 2023.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[11] Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz- Sabater, Julien Nicolas. Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis.
Quarterly Journal of the Royal Meteorological Sociery, 2020.
[12] Qihe Huang, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. Crossgnn: Confronting noisy multivariate time series via cross interaction refine- ment. Advances in Neural Information Processing Systems, 36:46885-46699 2023.
[13] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6688. 2014.
[14] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In /CML, 2019.
[15] Jesus Lago. Grzegorz Marejasz, Bart De Schutter, and Rafal Weron. Forecasting day-ahead electricity prices: A review of state-of-the-an algorithms, best practices and an open-access benchmark. Applied Energy, 293:1 16983, 2021.
[16] Colin Lea, Michael Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. In CVPR, 2017.
11

---

[17] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare. Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, 2021.
[18] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, In NeurIPS, 2019.
[19] Zhe Li, Shiyi Qi. Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:230S. 10721, 2023.
[20] Bryan Lim, Sercan Anık, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 2021.
[21] Minhao Liu, Ailing Zeng. Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. In NeurIPS, 2022.
[22] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In ICLR, 2022.
[23] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625. 2023.
[24] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. In NeurlPS, 2022.
[25] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Transformers for time series analysis scale. In 2024.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[27] Yisheng Lv, Yanjie Duan. Wenwen Kang, Zhengxi Li, and Fei-Yue Wang. Traffic flow prediction with big data: A deep learning approach. IEEE Transactions on Intelligent Transportation Systems, 16(2):865-88) 873,, 2014.
[28] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2022.
[29] Kin G Olivares, Cristian Challu, Grzegorz Marcjasz, Rafal Weron, and Artur Dubrawski. Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with nbeatsx. International Journal of Forecasting, 2023.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style. high peeh-rrrffomeeeeee deep learning library. In NeurlPS, 2019.
[31] Stylianos Vagropoulos, GI Chouliaras, Evaggelos
Kardakos, Christos K Simoglou, and Anastasios G Bakirtzis. Comparison of sarimax, sarima, modified sarima and ann-based models for short-term pv generation forecasting. In ENERGYCON, 2016.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[33] Yuxuan Wang. Haixu Wu, Jiaxiang Dong. Yong Liu, Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark. arXiv preprint (xXiv:2407713278 2024.
[34] Rafal Weron. Electricity price forecasting: A review of the state-of-the-art with a look into the future. International
journaa journal of forecasting, 2014.
12

---

[35] Billy M Williams. Multivariate vehicular traffic flow prediction: evaluation of arimax modeling. Transportation Research Record, 2001.
[36] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. In ICLR, 2023.
[37] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurlPS, 2021.
[38] Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with unified deep model. Nature Machine Intelligence, 2023.
[39] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 753-763, 2020.
[40] Kun Yi, Qi Zhang. Wei Fan, Hui He. Liang Hu, Pengyang Wang, Ning An, Longbing Cao, and Zhendong Niu. Fouriergnn: Rethinking multivariate time series forecasting from a pure graph perspective. Advances in Neural Information Processing Systems, 36, 2024.
[41] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, 2023.
[42] Yuchen Zhang, Mingsheng Long, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael Jordan, and Jianmin Wang. Skilful nowcasting of extreme precipitation with nowcastnet. Nature, 2023.
[43] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In ICLR, 2022.
[44] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
[45] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, 2022.
[46] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. In NeurIPS, 2023.
13

---

A Implementation Details
A.1 Dataset Descriptions
We conduct long-term forecasting experiments on 7 real-world datasets to evaluate the performance of our proposed TimeXer, including: (1) ECL [88 includes hourly electricity consumption data from 321 clients. We take the electricity consumption of the last client as endogenous variable and other clients as exogenous variables. (2) Weather 44 records 21 meteorologica factors collected every 10 minutes from the Weather Station of the Max Planck Biogeochemistry Institute in 2020. In experiment, we use the Wet Bulb factor as the endogenous variable to be predicted and the other indicators exogenous variables. (3) ETT 44 contains four subsets where ETThl and ETTh2 hourly recorded, and ETTml and ETTm2 are recorded every 15 minutes. The endogenous variable is the oil temperature and the exogenous variables are 6 power load features. (4) Traffic (36) records hourly road occupancy rates measured by 862 sensors of San Francisco Bay area freeways. We take the measurement of the last sensor as an endogenous variable and others as exogenous variables.
In addition to the public multivariate time series datasets, we perform short-term forecasting on the electricity price forecasting datasets [5]. which contains five datasets representing five different day-ahead electricity markets spanning six years each. Here are the descriptions of the datasets: (1) NP represents The Nord Pool electricity market, recording the hourly electricity price, and corresponding grid load and wind power forecast from 2013-01-01 to 2018-12-24 12-24. (2) PJM represents the Pennsylvania-Nen Jersey-Maryland market, which contains the zonal electricity price in the Commonwealth Edison (COMED), and corresponding System load and COMED load forecast from 2013-01-01 to 2018-12-24. (3) BE represents Belgium's ' electricity market, recording the hourly electricity price, load forecast in Belgium, and generation forecast in France from 2011-011 1-01-09 to 2016-12-31. (4) FR represents the electricity market in France, recording the hourly prices, and corresponding load and generation forecast from 2012-01 --0 to 2017-12-32 12-31 1. (5) DE represents the German electricity market, recording the hourly prices, the zonal load forecast in the TSO Amprion zone, and the wind and solar generation forecasts from 2012-01-09 to 2017-12-31.
Table 6: Dataset descriptions, Ex. and En. are abbreviations for the Exogenous variable and Endogenous variable, respectively. The dataset size organized in (Train, Validation, Test)
Datasct
#Num
Ex. Descriptions
En. Descriptions
Sampling Frequeney
Datasct Size
Electricity
320
Electricity Consumption
Electricity Consumption
Hour
(18317, 2633, 5261) (36792, 52711, 10540)
Weather
302-Concentration
Minutes
ETTh
6
Power Load Feature
Oil Temperature
Hour
(8545, 2881, 2881)
ETTm
Power Load Feature
Oil Temperature
15 Minutes
(34465, 11521, 11521)
Traflie
861
Road Occupancy Rates
Road Occupancy Rates
Hour
(12185, 1757, 3509)
NP
2
Grid Load, Wind Power
Nord Pool Electricity Price
Hour
(36500, 5219, 10460)
System load, SyZonal COMED load
PeansyIvanis- Now Jarsey-Mayyland
Electricity Price
(36500, 5219, 10460)
BB
2
Generation, System Lond
Belgium's 's Electricity Price
Hour
(36500, 5219, 10460)
Generation, System
France's Electricity Price
(36500, 10460)
DE
2
Wind power, Ampirion zonal load
German's 's Electricity Price
Hour
(36500, 5219, 10460)
A.2
Implementation Details
All the experiments are implemented in PyTorch [30 and conducted on single NVIDIA 4090 24GB GPU. We utilize ADAM 13] with an initial learning rate 10-4 and L.2 loss for the model optimization. The training process is fixed to 10 epochs with early stopping. We set the number of TimeXer blocks in our proposed model I. E (1, 2.3 The dimension of series representations dmode searched from { {128, 256, 512). The patch length uniformly set to 16 for long-term forecasts and 24 for short-term forecasts. We reproduced the compared baseline models based on the benchmark of TimesNet (36 Repository.
B Ablation Study
B.1 Using Overlapped Patch
In this paper, the proposed TimeXer adopts patch-wise representations of endogenous series via splitting the series into non-overlapping patches. Here conduct ablation study on the patching
14

---

method. Following PatchTST (28), we set the patch length to 24, consistent with TimeXer, and the stride set to 12, to generate a sequence of overlapped patches. Compared to the overlapping method, TimeXer has the lowest complexity while having the optimal performance. It is also notable that not only NL.P and CV, contemporary time series approaches also use non-overlapping patches. This preference might stem from the limited redundancy present in time series data, as excessive overlap can result in a smoothed representation for each patch, consequently failing to capture correct temporal dependencies.
Table 7: Model performance with overlapped patches.
NP
PJM
BE
FR
DE
AVG
Design
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
TimeXer
0.236 0.268
0.093 0.192
0.379 0.243
0.385 0.208
0.440 0.415
0.307 0.265
TimeXer-overiap 0.240 0.267
0.095 0.194 0.383 0.248
0.409 0.214
0.453
0.419
0.316 0.269
B.2 Varying Patch Length
In this section, we experiment with the effect of patch lengths on the forecasting performance. We fix the look-back window to 96 and vary the patch lengths P E {2, 4, 6, 8, 12. 24]. We conduct experiments on five short-term electricity price forecasting datasets with consistent parameters except patch-length, and the results are shown in Figure [6] can be seen that the average prediction performance does not vary dramatically with different patch lengths, which indicates that our model robust to the patch length hyperparameter. Notably, the model generally performs lower when the patch length is small, which is probably because small patches are not enough to represent the semantic information in time series data.
NP
PJM
0.223
0.32
0.048
0.30
1.5%
0.2
0054
0.34
0.033
1
12 Patch Length
Patch Length
BE
DE
0.400
0.41
MSE
0.40
MSE
0.450
0.475
0.400
0.35
0.425
0.445
0.480
0.38
0.400
3
5
2
6
8
6
Lengis
Langgh
Patch Lengh
Figure 6: Hlper-parameeer sensitivity analysis of TimeXer on short-term forecasting benchmarks.
B.3 Alternative Embedding Approach
In our above-mentioned analysis, we apply representation from different level capture the temporal dependencies and multivariate correlations. For the endogenous variable, we apply petch-level temporal token and a learnable global token to capture both temporal dependencies and cross-attention between different variables. And for exogenous variables, we utilize variate-level representation directly embedded from the whole series. To verify the rationality of our proposed architecture, we modify the embedding design and inclusion of exogenous variables. In this section, provide
15

---

Table 8: Ablation study on long-term forecast.
Design
Endogenous
Exogenous
Horizon
ETTh1
ETTml
Traffic
MSE
MAE
MSE
MAE
MSE
MAE
96
0.056
0.179
0.028
0.125
0.150
0.225
192
0.071
0.205
0.043
0.158
0.152
0.228
Ours
Putch+Globul
Variate
336
0.080
0.222
0.057
0.185
0.150
0.231
720
0.084
0.229
0.079
0.217
0.172
0.253
Avg
0.073
0.209
0.052
0.171
0.156
0.234
96
0.057
0.182
0.028
0.125
0.156
0.232
192
0.072
0.154
0.232
Replace
Paoch+
Patch
336
0.083
0.226
0.058
0.186
0.134
0.238
720
0.088
0,234
0.079
0.216
0.175
0.258
Avg
0.075
0.212
0.052
0.171
0.160
0.240
96
0.058
0.183
0.028
0.126
0.153
0.229
192
0.072
0.207
0.044
0.159
0.152
0.231
Remove
Patch
Variate
336
0.081
0,222
0.058
0.187
0.152
0.235
720
0.092
0,239
0,080
0.217
0.175
0.258
Avg
0.076
0.213
0.052
0.172
0.160
0.240
96
0.058
0.183
0.029
0.128
0.168
0.240
192
0.074
0,208
0.044
0.161
0.169
0.244
Add
Patch+Globull
Variate
336
0.085
0.227
0.060
0.189
0.165
0.246
720
0.093
0.240
0.083
0.221
0.184
0.266
Avg
0.078
0.214
0.054
0.175
0.171
0.249
96
0.063
0.196
0.028
0.125
0.146
0.222
192
0.074
0,210
0.045
0.162
0.148
0.225
Concatenate
Palch+Gilobal
Variate
336
0.094
0.242
0.061
0.190
0.147
0.227
720
0.088
0.233
0.080
0.219
0.156
0.239
Avg
0.080
0.220
0.053
0.174
0.149
0.228
ablation results on long- term forecasting datasets. As shown in Table 8 among various architectural designs, TimeXer generally exhibits optimal performance. Notably, when replacing variate embedding of exogenous variables with patch embedding. prediction accuracy declines. This observation suggests that a series-wise representation of the exogenous variables is more advantageous for predicting the endogenous variable. However, it is important note that applying patch- wise representations for all exogenous variables would significantly increase computational complexity. In contrast, TimeXer offers a more efficient and effective design. Nevertheless, we notice that on the Traffic dataset when concatenating the variate-level exogenous tokens with those endogenous tokens and performing a Self-Attention over them outperforms our proposed design. Since TimeXer only adopts Pross-Attention between endogenous and exogenous variables, the main distinction between these two designs is whether there attention within multiple exogenous tokens. These ablation results indicate that the interaction within exogenous variables also be viewed external factors that may facilitate the prediction. However, this kind correlation not valid in all cases. As listed in Table & there is no improvement in ETTh] and ETTml datasets but decreased.
C TimeXer Generality under Missing Exogenous Values
Real-worke time series encounter problems such as missing data that result in low-quality data. In this section, we use random masking to replicate these situations and further explore the forecasting
NP
BE
DE
PM0
0.20
0.395
0.400
MSE
0.390
MSF
0.700
0.53-
0.380
0.90
09. 5%
25%
50%
75%
50%
0%
3%
23%
50%
75%
50%
0%.
3%
25%
50%
49%
Mask Ratio
Ratio
TimeXer
iTansformer
PatchTST
Figure 7: Forecasting performance with the masked exogenous series on three EPF datasets, simulat- ing the missing values scenario.
16

---

performance when fed low-quality data. Previous works [9] have demonstrated that the semantic information of time series lies in temporal variation, We use complete, high-quality historical data for the endogenous variables and progressively reduce the quality of the data for the exogenous variables by increasing the masking ratio from 0% (i.e., using complete historical data for the exogenous variables) to 99% as shown in Figure 7 It can be observed that with the decrease in the quality of the exogenous series, the forecasting performance of the model also decreases. Notably, our model maintains a competitive performance when the exogenous series are masked by small amount, which indicates that our proposed TimeXer is capable of supporting low-quality data scenarios.
D Increasing the Look-back Length
In the main text, we have already evaluated the forecasting performance with the increase of the look-back length of endogenous or exogenous series under forecasting with the exogenous variables paradigm. Here, we present results on multivariate forecasting with increased historical length. Results shown in Figure 8 (Right) indicate that TimeXer can benefit from the extended look-back length for a better performance.
Model Efficiency Weather
Multivariate Forecasting on Weather
0.35
pest keg
0.28
SCINel
0.sca
1AGB
2
0.30
0.24
MSE
Dinnear BANGK 3m
PatchTST 1.35K,, 15mm
Memory Factprint
TIDE 1,4GG 125ra
MSE
0.25
0.20
Trarsformer ASCK 1em
0.20
0.16
Crossformer
2.70CK, 9hre
TimeXer 113GR, zire
0.15
0.12
512
720
Training Time (ms/iter)
Look-back Length
Figure 8: Model analysis on Weather dataset under multivariate forecasting paradigm. Left: Model efficiency comparison under input-96-peedict-96 Right: Forecasting performance with an enlarged look-back length 7' E {96, 192, 336, 512, 720].
E
Model Efficiency
In this section, we first provide theoretical analysis of the computational complexity of TimeXer with other advanced Transformer-based forecasters. Suppose the look-back length and prediction length are T and S respectively, C' is the number of exogenous variables, and /' is the length of the patch. As we presented Figure in the main text, our proposed TimeXer has a clear advantage when the number of exogenous variables C' is large and achieves a favorable balance between modeling fineness and efficiency. This mainly benefits from the Cross-Attention to obtain a OCC)) complexity in variate dimensions, whereas iTransformer will increase to OCC2)
Beyond vanilla forecasting with exogenous variables, we also compare the efficiency under the multivariate forecasting paradigm on the Weather dataset when using historical 96-time steps to predict future 96-time steps. The result in (igure&Jclaati demonstrates that TimeXer shares similar performance with Crossformer, but significantly outperforms in terms of training time and memory usage. Moreover, the efficiency of TimeXer close to PST which is because the parallel multivariate forecasting is achieved by the channel independence mechanism on the self-attention over endogenous variables, which introduces a quadratic complexity O(L + +1))). Notably, this quadratic complexity can be reduced through (1) Adapt the patch length: The computational complexity is highly related to the patch length. (2) Incorporate advanced altention module: Since TimeXer does not modify any component in the Transformer, which results in a quadratic complexity. By replacing the full attention with the advanced attention module, such as linear attention, the complexity can decrease to O((+) ) C).
17

---

F Representation Analysis
To evaluate the performance of TimeXer from the perspective of representation, we adopt centered kernel alignment (CKA) similarity [14] in this section. Previous works [36 9 reveal that for low-level time series tasks including forecasting, there is a great similarity among representations from the different layers and higher similarity corresponds to a better performance. Technologically. we calculate the CKA between the output features of the first and the last block obtained from TimeXer as well as other baselines. In particular, since ITrranffrmmr [23] is multivariate forecaster that treats all variables equally, we provide the similarity corresponding to the endogenous variate in addition to the global-view CKA. As shown in Figure the representations from the first and the last layers of TimeXer enjoy great similarities, verifying that TimeXer can learn the appropriate representations for the prediction. It is notable that iTransformer does not distinguish between endogenous and exogenous variables and the output of the model contains representations of all variables. However, the result of the CKA analysis shows that despite the high similarity of the series representations of all variables, the representation the endogenous variables was not well learned. This result also suggests that directly applying multivariate model to perform forecasting with exogenous variables introduces unnecessary noise into the model thus interfering with its forecasting performance.
DE
PIM
0.13
0.60
0.12
0.55
MSE
MSE
0.11
0.50
0.10
0.45
0.09
0.40
0.7
0.6
0.8
CKA Similarity
CKA Similarity
BE
FR
NP
0.55
0.32
0.50
0.30
MSE
MSE
0.45
0.45
0.28
0.40
0.40
0.26
0.24
0.35
0.35
0.7
0.9
0.7
0.8 CKA Similarity
0.70
0.75
0.80
0.85
0.90
0.95
CKA Similarity
CKA Similarity
iTm-AAl
iTm-Ea
PatckTST
TIDE
TimeXer
Figure 9: Series Representation Analysis three EPF datasets. iTrm-All denotes the series represen- tation of all variables learned by iTransformer, iTrm-En is the learned series representation of the endogenous variable.
G Discussion
We find that the multivariate forecasting results on Traffic datasets in Table are different from the other datasets. Noteworthily, the mean absolute error (MAE) result of TimeXer is close to iTransformer, which is the statc-of-the-art model on the Traffic dataset, but there is still a large margin in mean squared error (MSE). This unexpected result prompted us further investigate. As result, we provide visualization of the forecasting results from TimeXer and iTransformer in Figure TT] Upon analysis, we observed that while TimeXer effectively predicted the overall trends of the future horizon, it struggled to accurately forecast the numerical value of future spikes. Subsequently. the squared calculation in MSE will amplify this error, resulting in a drastic difference compared to MAE. To further explore this performance, we also visualize the forecasting results given by PatchTST. As illustrated in Figure TO, we discover that PStchTST performs quite similarly to TimeXer. The commonality of these two models is that they both utilize patch-level representation to model the
18

---

TimeXer
PatchTST
iTransformer
125
100
25
220
Figure 10: Multivariate forecasting showcase on Traffic dataset.
temporal dependencies, which we think might be the answer to the performance decrease. The utilization of patch-level representation in TimeXer and PatchTST contributes to their ability to capture temporal dependencies and contributes to the accurate prediction of the trends. Conversely, iTransformer focuses on the variate-wise correlation while the temporal correlation is only obtained through a linear projection. In our proposed TimeXer, we employ a linear projection over all the endogenous tokens, including multiple patch-level temporal and only global variate-level token. Therefore it can be inferred that the excessive number of patches may make the prediction pay more attention to the overall trend change and fail to predict the precise value of changing points. Based on the above analysis, we believe that to alleviate this problem, we need to address the imbalance problem of multiple temporal tokens and only one global token, which can solved by increasing the patch length or learnable tokens.
H Showcase
H.1 Intuitive Showcases for Exogenous Variable Utilization
To enhance the understanding the role of exogenous variables in prediction, we visually present the prediction results in two distinct scenarios: with and without exogenous variables. In the case where exogenous variables are incorporated, we introduce a special scenario in which the model has access to predictions of these exogenous variables. This is practical scenario for the EPF datasets where the exogenous variables are the day-ahead predictions the source generation. Additionally, add extreme case where there is no historical information endogenous series explore whether TimeXer can learn from exogenous variables in scenarios where only external information is available. By exploring these varied scenarios, aim to provide comprehensive analysis of how exogenous variables influence the model's 's forecasting performance. As illustrated Figure [T] observe that removing either endogenous from the model. This indicates that both types of variables play a crucial role in enhancing the forecaster's ' performance. Notably, when the model has access future predicted values of the exogenous variables, the performance achieves the best. This finding underscores the importance of incorporating external information, particularly the predictive insights from exogenous variables. which is vital to guarantee an accurate prediction.
exogenous variables leads to poorer predictions
H.2 Visualization of Prediction Results
To have an intuitive concept of the forecasting process, we visually present endogenous and ex- ogenous variables from selected datasets BE, DE, and PJM in Figure 12 Figure 13 and Figure
19

---

(a) Wind power (Exogenous)
(b) Ampirion zonal load (Exogenous)
(c) TimeXer (PastEx, PastEn)
With
M.
(d) TimeXer (PastEx, None)
(c) TimeXer (None, PastEn)
(i TimeXer (Past=FutureEx, PastEN)
With mamme
Wnth
W
100
106
10
100
100
(a) Wind power (Exogenous)
(b) Ampirion zonal load (Exogenous)
(c) TimeXer (PastEx, PastEn)
1n
(d) TimeXer (PastEx, None)
(e) TimeXer (None, PastEn)
( TimeXer FPast*FutureEx, PastEN)
My Mym
Mm 180
My M
15
Two showcases (placed at the top and bottom respectively) of TimeXer in forecasting with exogenous variables from DE datasets. (a) and (b): The showcases of two exogenous variables Wind power and Ampirion zonal load. (c) The prediction results of TimeXer using the historical information of endogenous and exogenous variables. (d) and (e): The prediction results of TimeXer only use the historical information of exogenous endogenous variables. (f) The prediction results of TimeXer using the historical information of endogenous and exogenous variables, and the future prediction values of exogenous variables.
Figure 11:
14 respectively. For each case, we display the ground truth values for both the endogenous and exogenous variables, alongside the forecasting results for the endogenous variable. We compare the forecasting performance of the proposed TimeXer with five comparable baseline models, including Crossformer, iTransformer, PatchTST, TiDE, and DLinear. Each model takes the input time series data with a length of 168 and performs forecasting tasks with a prediction horizon of 24. evaluate utilize points of inflection on the curves. If the predicted value falls
the quality of the forecasts, within a range of 0.05 from the ground truth, we consider this prediction successful and highlight with green circle of 0.05 radius. Conversely, if the predicted value exceeds this range, classify it as an oft-of-range forecast and mark with a red circle the same radius to indicate its failure. This visual representation facilitates clear comparison of the performance across different models.
By counting the green and red circles on all injection points in Figure 12 Figure [3 and Figure [4
is clear that the TemeXer can forecast target endogenous more precisely, especially at inflection points
20

---

Geaeattin
TimaXer
CroaSonter
Toansourerr
"M
PocchTST
Figure 12: Showcases TimeXer in forecasting with exogenous variables from BE datasets. The two leftmost plots of the title "Generation" and "System Load" the exogenous variables in the BE dataset. TimeXer outperforms all of challengers by predicting all 4 injections in 24 prediction time points.
Wind Poweer
TimeNer
Crossomme
iTrasSoommr
10
Amppeion Zoral Load
PatckTST
DXLinerr
Figure 13: Showcases TimeXer in forecasting with exogenous variables from DE datasets. The two leftmost plots of the title "System Load" and "Zonal COMED Load" are the exogenous variables in the DE dataset. TimeXer outperforms all of challengers by predicting all injections in 24 prediction time points.
Syeem Load
TimeXer
Chnsforter
Zoeol COMED Lecal
PREETST
TIDE
DLince
M
Figure 14: Showcases of TimeXer in forecasting with exogenous variables from PJM datasets. The two leftmost plots of the title "System Load" and "Zonal COMED Load" are the exogenous variables in the PJM dataset. TimeXer outperforms all of its challengers by predicting all injections in 24 prediction time points.
21

---

where we marked with green circles. In contrast, other models tend to oscillate around or exceed the ground truth, suggesting that TimeXer exhibits superior robustness compared to existing alternatives. By learning from the endogenous variable's s temporal dependencies and relations between endogenous and exogenous variables, TimeXer not only acquires abundant contextual information about its own history but also obtains nutritive relation information about correlated variables. Such architectural design makes TimeXer more aware of the potential pattern of the target dataset, leading to enhanced forecasting performance compared to known Transformer-basee models.
I
Full Results
I.1 More Baselines in Short-term Forecasting
To better evaluate the performance of our proposed TimeXer, we take previous works designed for the inclusion of exogenous variables as our baselines. We also report the standard deviation of TimeXer performance on EPF dataset under five runs with different random seeds. The result in Table indicates that the performance of TimeXer is stable.
Table 9: More baselines in the short-term forecasting with exogenous variables task.
Model
AVG
Metric
MSE
MAE
MSE
MAE
MSF
MAB
MSE
MAE
MSE
MAR
MSF
MAF
TimeXer
0.885:00000000000000000
Stationary
X
0.291
0,308
0.122
0.228
0.133
0.289
0.166
0.242
0.183
0.117
0.360
0,303
NBEATSx (9
0,272
0,301
0.189
0.265
0.393
0.330
TFT
20
0,369
0,391
0.141
0.241
0,479
0.305
0.461
0.219
0.559
0
0.102
0.335
1.2
More Baselines in Long-term Forecasting
Beyond Transformer based architecture, GNN-based models have emerged as potential choice for modeling the underlying dynamic spatial correlations between time series. To fully evaluate the performance of Xere we include classic GNN-based models, including MTGNN [39]. CrossGiNN 12.. MSGNet B, and FourierGNN [40] as our baselines for multivariate forecasting. The result in
Table 10 shows that TimeXer consistently achieves the best.
Table 10: More baselines in the multivariate long-term forecasting. denotes there is no officially reported results.
Model
BCL
Weather
ETTh1
ETTh2
ETTml
ETTm2
Traflic
Metric
MSE
MAF
MSR
MAE
MSE
MAE
MSF
MAR
MSF
MAE
MSE
MAE
MSF
MAR
TameXer
0.171
0.270
0.241
0.271
0.437
0.437
0.366
0.395
0.382
0.397
0.274
0.322
0.467
0.288
MIGNN
39
0.251
0.347
0.314
0.355
0.572
0.553
0.465
0.509
0.468
0.446
0.324
0.365
0.650
0.446
CrossGNN P2
0.201
0.271
0.247
0.289
0.437
0.434
0.363
0,418
0.393
0.404
0.282
0.330
0.583
0.323
MSCNet
0.194
0.278
0.0.452
0.452
0.396
0.398
FourierGNN
40
0.228
0.324
0,249
0.302
0.557
0,342
I,3 Full Results of Long-term Forecasting with exogenous variables
To evaluate the performance of our proposed TimeXer, we conduct long-term forecasting with exogenous variables on acknowledged real-world multivariate datasets. The look-back length is set 96, and the prediction length varies from (96, 192, 336, 720]. The results are listed in Table 11
L.4 Full Results of Long-term Multivariate Forecasting
To evaluate the generality of our proposed TimeXer, we conduct long-term multivariate forecasting on existing real-world multivariate benchmarks. The look-back length is set to 96, and the prediction length varies from {96, 192, 336, 720}. The results are listed in Table 12
22

---

Table 11: Full results of the long-term forecasting with exogenous variables task. memory (OOM) problem.
denotes out of
TimeXer
iTrans.
RLinear
PatchTST
Cross,
TIBE
TimesNat
DLincar
SCINet
Stationary
Models
Metric
MSE MSE MAE MSE MAE MSE MAE MSE MAE
MSE MSE MAE MAE
MSE MAE
MSE MAE
MSE MAE
96
0.261
0.554 480
46
06226 0.408
0.432 0,502
192
0.3160.397..388 0,321 0.473 0,36
0.492 0.492
336
http:///...............00044000. 0.508 0,548
720
0.547 0.569
0.327 A00.0000000000000333344040000303
AVG
0.495 0.528
96
0.001
0.007 0.066
Weather
192
0.007 0.061
0.007 0.062
00000000000000000000000000000000
720
0.053
AVG (0t60000000000000000000000000000
0.006 0.060
96
0.119 0,263
0.130 0,282
0,335
0.2230.377
0.100 0,245
192
0.101 0.236
otg.:/000.......0...0.0000000000 0.300 0.143 0.294
336
0.1300.275
0.156 0.308
720
0.184 0.333
AVG 0.1200.258
0.154 0.305
96
0..510.2241.15602226....2
0.299 0.290
192
11.220.2291115002221 015.0000000101010001000000000110
0.291 0.291
Tralfs
336
11.500.332..1540.200 0...80..22...5002328880800000..0
0.322 0.416
720
0.1720.253 0.277 0...7......00.88822..... 0...
0.2433.329
310. 0.34
0.507
0.307 0.414
AVG
0.1560.234/0.100
p.2400.320
0.266 000
0.361 0.361 0.302 0.353
Count 23
23
2
8
2
23

---

Table 12: Full results of the long-term multivariate forecasting task.
TimeXer
TTrans
RLinear
PatchTST
Cross
TIDE
TimesNet
DLineur
SCINet
Stationury
Auto.
Models Metric
MSE MAE MSE MAE
MSE MAE
MSE MAD
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE MSE MAB
96
0,1570,25660 0.2253300300 0.283 0.299 0.209999..0000002222020000000000 0.209 0.240400
0.1400.242 0.240.00..00 0.281
0.169 0.273 0.201 0.317
192
0.355
0,182 0...8.2.66111 0.2320.334
00T00110000001000000000000001010 010.:/299900004400009.0000000000 0.00000000000 0.369 0.200
0.23 0.338
720
0.211
0.2540.361
0.171
0.2270,.38
96
0.266 0.336
Weather
0.30885500770000 0.367
0.261
0.338 0.359
0.4190..44
0.241
0.3820.40300:38600400030
0.449..459
0.2290.4350.44100.700....
720
0.469
Avg
336
0.482 0.486
720
0.408
0.367
0.362
0.4440.5330.69
336
720
0.4620.441 01220.4410..411..55999999
Avg
0.382
96
0.171
0.25
192
0.237
0.339
0.339 0.281 0,340
336
0.296
0.334 0.361 0,372
720
03.417 0.413 0.433
0.3470.327..377
0.612 0.338 0.313
0.611 0.636
Tnffe
0.4730.28%
0.6188 0.328 0.622
0.5160.307 0.467 0tw.:6666..00..6........ 01w.:002...670..777.55444.4
0.523
0.355 0.66000408
1.2660.287004233 0.282 0.6266..7773
0.304
036.0.333...33 0.383
0.6 0.624 0.340
0.6280..68....
Count
22
5
5
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and Section | have claimed the contributions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the discussion section in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The architecture design have been fully described in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: Please refer to the implementation details listed in Appendix A.2 2 The code can be found in supplementary materials, also available at this repository: com/thuml /Timeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides wwwe://oooooeioooioooooooooooooa
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//ipss.ccppul wecc/pbllicgguuiiice wwwiii/ggge///eeeeeeeeeeieeeeeee /utpi////gideee/eeuuuubbiiiii ssi osiooiicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Appendix
I.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
27

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http::////..aaaasscccccc/l///aaa
Answer: [Yes]
Justification: Ethics.
We have reviewed and the reasearch conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the case study in the Section 4.4 the main text.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators of datasets are properly credited by citations.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets. The code and training details provided in supplementary materials.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "jNF7arSFv5", "reviewer": "Reviewer_r8UR", "summary": "The paper presents TimeXer, a Transformer-based model for time series forecasting that integrates exogenous variables using innovative attention mechanisms. Unlike traditional models that either focus solely on endogenous variables or treat all variables equally, TimeXer integrates exogenous information using patch-wise self-attention and variate-wise cross-attention mechanisms.", "questions": "1. The paper would benefit from more intuitive examples demonstrating how TimeXer effectively learns and utilizes exogenous variables. Can you provide practical examples or case studies where TimeXer successfully learns from exogenous variables in scenarios where only external information is available?\n\n2. In Figure 5 (left), the paper shows score differences when two series look similar versus when they do not, but it does not convincingly establish that the presence of similarly patterned time series aids in prediction accuracy. Can you provide more evidence or analysis to support the claim that similarity in variable patterns enhances prediction accuracy?\n\n3. In the experiments, univariate forecasting with exogenous variables and multivariate forecasting were not distinctly separated. Why did you choose not to clearly differentiate between these two types of forecasting scenarios in your descriptions?", "limitations": "Limitations are discussed briefly.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Given the growing emphasis on multivariate time series forecasting in recent research, this paper distinguishes itself by focusing on exogenous variables. By integrating the strengths of recent studies and introducing novel ideas, it advances the field in a meaningful way.\n\n2. Unlike other models that specialize in specific types of forecasting, TimeXer presents a versatile framework applicable to univariate forecasting, multivariate forecasting, and forecasting with exogenous variables. This adaptability makes it a comprehensive tool for a wide range of forecasting scenarios.", "weaknesses": "1. Methodological Gaps in Handling Practical Situations\n\n   While the paper presents various practical situations involving exogenous variables, the methodology, specifically the variate embedding section, only proposes embedding through learning parameters. It lacks a detailed methodological approach to address these practical issues effectively.\n\n2. Lack of Intuitive Examples for Exogenous Variable Utilization\n   The paper does not provide intuitive examples of how well the model learns and uses exogenous variables. Including examples where TimeXer can effectively learn from other variables in scenarios where information is only available from external variables would significantly emphasize the model's necessity and practical relevance.\n\n3. Lack of Causality Between Variable Similarity and Prediction Accuracy\n   There is an inherent lack of causality between the similarity of shapes among variables and the accuracy of predictions. In Figure 5 (left), the paper demonstrates score differences when two series look similar versus when they do not. However, this does not convincingly show that the presence of similarly patterned time series aids in prediction accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer r8UR\nWe would like to sincerely thank Reviewer r8UR for providing a detailed review and insightful suggestions.\n\n> **W1**: Methodological Gaps in Handling Practical Situations\n\nAs the reviewer mentioned, we focus on forecasting with exogenous variables and have provided a comprehensive analysis for practical forecasting scenarios in $\\underline{\\text{Lines 31-39 of main text}}$. Although the real-world scenarios are multifarious, our proposed TimeXer provides a **simple, flexible, and effective** method to tackle practical challenges, such as missing value or mismatch information. \n\nWith our specially designed variate embedding layer (learned embedding and cross-attention module), TimeXer can naturally handle the mismatch series length and diverse information, which has been comprehensively verified in our experiments:\n\n(1) **Mismatch Length**: In $\\underline{\\text{Lines 255-256 and Figure 3 of main text}}$, we enlarge the length of exogenous and endogenous series respectively, resulting in unequal input lengths.\n\n(2) **Mismatch Frequency and Temporal Misaligned**: We conduct experiments on large-scale time series data ($\\underline{\\text{Figure 4 of main text}}$). As we stated in $\\underline{\\text{Lines 275-276 of main text}}$, the sampling frequencies of the endogenous and exogenous variables are 1h and 3h respectively. Also, the temperature, pressure, and wind meteorological indicators exist inherent temporally misaligned relation (temperature will change earlier, then pressure and finally wind).\n\n(3) **Missing Values**: In $\\underline{\\text{Section C of the Appendix}}$, we randomly mask the exogenous series to obtain cases with missing values.\n\nThe above results verify that **our variate embedding design provides a simple and neat way to bridge Transformer to practical situations, whose flexibility and effectiveness should not be overlooked.**\n\n> **W2&Q1**: Lack of Intuitive Examples for Exogenous Variable Utilization\n\nFollowing your suggestion, we have included more showcases in $\\underline{\\text{Figure 1 of the Author Rebuttal PDF}}$. \n\n- We visually present the prediction results in two cases, with and without exogenous variables, to validate the role of exogenous variables. \n- As per your request in **Q1**, we add an extreme case where there is no historical information on endogenous series to explore whether TimeXer can learn from exogenous variables in scenarios where only external information is available. \n- In addition, we also add a special case where the prediction of exogenous variables is available to the model. This is a practical scenario for the $\\underline{\\text{EPF datasets}}$ where the exogenous variables are the day-ahead predictions of the source generation. \n\nWe can observe that the inclusion of exogenous variables successfully enhances model performance, and when further exogenous variable predictions are utilized, the performance achieves the best.\n\n\n> **W3&Q2**: Lack of Causality Between Variable Similarity and Prediction Accuracy\n\nIn our paper, we adopt $\\underline{\\text{Figure 5 of main text}}$ to demonstrate that the cross-attention module empowers TimeXer with special interpretability. We would like to highlight that the similarity we mentioned is not directly equivalent to \"shape similarity\"; rather, it is derived from the attention mechanism, which captures the intrinsic temporal patterns present within time series data. Intuitively, time series with similar shapes may exhibit shared temporal features, resulting in a higher similarity score. Thus, the most similar exogenous series learned by the attention mechanism may intuitively resemble the endogenous variable, but it cannot be said that all similar series will definitely contribute to the prediction.\n\nAs per the reviewer's request, we conduct analysis studies on the causality between variable similarity and model performance. Technologically, we select the most similar and least similar variable to the endogenous variables through the learned attention map and further conduct experiments by removing this exogenous variable. The results are listed as follows, which can observe that series with larger similarity (learned by TimeXer) is more beneficial to the prediction.\n\n|Short-term(MSE\\|MAE)|NP|PJM|BE|FR|DE|AVG|\t\n-|-|-|-|-|-|-\n| TimeXer    | **0.236** \\| **0.268** | **0.093** \\| **0.192** | **0.379** \\| **0.243** | **0.385** \\| **0.208** | **0.440** \\| **0.415** | **0.307** \\| **0.265** |\n|TimeXer-w/o Most Similar| 0.295 \\| 0.299 | 0.096 \\|0.194 | 0.385 \\| 0.246 | 0.393 \\| 0.210 | 0.483 \\| 0.439 | 0.331 \\|0.278 |\n|TimeXer-w/o Least Similar| 0.244 \\| 0.273 | 0.093 \\|0.192 | 0.382\\| 0.242 | 0.387 \\| 0.211 | 0.462 \\| 0.422 | 0.313 \\|0.268 |\n\n\n> **Q3**: In the experiments, univariate forecasting with exogenous variables and multivariate forecasting were not distinctly separated. Why did you choose not to clearly differentiate between these two types of forecasting scenarios in your descriptions?\n\nAs stated in $\\underline{\\text{Line 193 of main text}}$, we find that **forecasting with exogenous variables can be a unified forecasting paradigm** that generalizes straightforwardly to multivariate forecasting by taking each variable as endogenous, the other variables are exogenous. \n\nTo verify the effectiveness and generality of TimeXer, we conduct experiments under two different forecasting paradigms, namely short-term forecasting with exogenous variables on EPF benchmarks and long-term multivariate forecasting on well-established benchmarks. The experimental results for these two distinct forecasting tasks are given in $\\underline{\\text{Tables 2, 3 of main text}}$, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jh7GlXt4C3", "reviewer": "Reviewer_Mh7v", "summary": "This article proposes a method based on Transformer modeling to enhance the prediction accuracy of endogenous variables by incorporating exogenous variables.", "questions": "The influence of exogenous variables on endogenous variables is multi-grained, including aspects like periodicity, trend, noise, and stability. Does the Transformer have the capability to capture such multi-grained influences, or does the author think that more promising methods (such as LLMs) could be exploited to capture these associations?", "limitations": "The limitation is discussed sufficiently in appendix.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research topic is interesting and has strong practical value.\n2. The paper is well-structured, making it easy for readers to understand.\n3. The experiments are extensive, demonstrating the effectiveness of the method across various datasets and experimental settings.", "weaknesses": "1. The implementation details on how to extend TimeXer to multivariate time series prediction are not sufficiently detailed (lines 190 - 196).\n\n2. The article does not provide a specific explanation for why the non-overlapping patch method is used instead of the overlapping method.\n\n3. Figure 3 shows the prediction performance corresponding to different step lengths of exogenous/endogenous variables, but it lacks experimental results for some extreme cases, such as reducing the step length of exogenous/endogenous variables to 1.\n\n4. It is recommended to explain more clearly in the methods and introduction sections why the global token is applied to the time series prediction task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Mh7v\nMany thanks to Reviewer Mh7v for providing a detailed review and insightful questions.\n> **W1**: The implementation details on how to extend TimeXer to multivariate time series prediction are not sufficiently detailed.\n\nSorry for the missing description. The success of the channel independence mechanism has demonstrated that multivariate forecasting can be viewed as multiple independent univariate forecasting with a shared backbone. Inspired by this, we find that TimeXer can be generalized to multivariate forecasting by employing the channel independence mechanism. \n\nSpecifically, For multivariate time series [B, T, C], we obtain endogenous embedded vector [B\\*C, N, D] and exogenous embedded vector [B, C, D], where N is the number of patches and D is the hidden dimension. Note that the endogenous embedded vector is already channel-independent and further fed into the shared Endogenous Self-Attention Module to capture temporal dependencies within each variable. Further, we select the global token of each variable (which is then reshaped from [B\\*C, 1, D] to [B, C, D]) to perform cross-attention with the exogenous embedded vector to capture cross-variate dependencies. We have provided the relevant code in $\\underline{\\text{supplementary materials}}$.\n> **W2**: The article does not provide a specific explanation for why the non-overlapping patch method is used instead of the overlapping method.\n\nFollowing your suggestion, we have completed the ablation study on the effects of patching in TimeXer. We adopt the patching method used in PatchTST (ICLR 2023), setting the patch length to 24, consistent with TimeXer, and the stride to 12, to generate a sequence of overlapped patches. Compared to the overlapping method, TimeXer has the lowest complexity while having the optimal performance. \n|Short-term(MSE\\|MAE)|NP|PJM|BE|FR|DE|AVG|\t\n-|-|-|-|-|-|-\n|TimeXer|**0.236**\\|**0.268**|**0.093**\\|**0.192**|**0.379**\\|**0.243**|**0.385**\\|**0.208**|**0.440**\\|**0.415**|**0.307**\\|**0.265**\n|TimeXer-overlap|0.240\\|0.267|0.095\\|0.194|0.383\\|0.248|0.409\\|0.214|0.453\\|0.419|0.316\\|0.269|\n\nIt is also notable that not only in NLP and CV, contemporary time series approaches ([41 of our paper] and [1]) also use non-overlapping patches. This preference might stem from the limited redundancy present in time series data, as excessive overlap can result in a smoothed representation for each patch, consequently failing to capture correct temporal dependencies.\n\n[1] \"Timer: Generative Pre-trained Transformers Are Large Time Series Models.\" ICML 2024.\n> **W3**: Figure 3 shows the prediction performance corresponding to different step lengths, but it lacks experimental results for some extreme cases, such as reducing the step length of exogenous/endogenous variables to 1.\n\nAs per your request, we conduct experiments in some extreme cases. Concretely, we set the look-back length of exogenous/endogenous variables to 1 respectively and list the results as follows, where EN and EX denote the endogenous and exogenous series respectively. \n\nShort-term (MSE\\|MAE)|NP|PJM|BE|FR|DE|AVG\n-|-|-|-|-|-|-\n|TimeXer|**0.236**\\|**0.268**|**0.093**\\|**0.192**|**0.379**\\|**0.243**|**0.385**\\|**0.208**|**0.440**\\|**0.415**|**0.307**\\|**0.265**\n|TimeXer-EN1|0.269\\|0.296|0.110\\|0.209|0.482\\|0.324|0.442\\|0.245|0.500\\|0.453|0.361\\|0.305 \n|TimeXer-EX1|0.252\\|0.277|0.106\\|0.207|0.398\\|0.255|0.415\\|0.215|0.467\\|0.429|0.328\\|0.277 \n\nWe can find that TimeXer’s performance is relevant to the look-back length of the endogenous variable, deteriorating when historical information is severely limited. Additionally, the length of the exogenous variable also influences forecasting performance, with better results observed when more external information is available. \n> **W4**: It is recommended to explain more clearly in the methods and introduction sections why the global token is applied to the time series prediction task.\n\nThanks for your suggestion, we will provide a more detailed description in $\\underline{\\text{Line 146-148 of main text}}$ to better explain the role of the global token in the revision: \"Given the distinct roles of endogenous and exogenous variables in the prediction, TimeXer embeds them at different granularity. Directly combining patch-level endogenous tokens and variate-level exogenous tokens results in information misalignment. To address this, we introduce a learnable global token for each endogenous variable that serves as the macroscopic representation to interact with exogenous variables. This design helps bridge the causal information from the exogenous series to the endogenous temporal patches.\"\n> **Q1**: The influence of exogenous variables on endogenous variables is multi-grained, including aspects like periodicity, trend, noise, and stability. Does the Transformer have the capability to capture such multi-grained influences, or does the author think that more promising methods (such as LLMs) could be exploited to capture these associations?\n\nThank you for your insightful question. Previous works, e.g. Autoformer, have attempted to use the seasonal-trend decomposition method to reveal the entangled temporal patterns in time series data and better capture underlying temporal dependencies. We think it will be reasonable to apply more elaborative designs (e.g. decomposition or LLMs) to the endogenous and exogenous series and leverage our proposed architecture to capture these multi-grained influences.\n\nSince we attempt to \"unleash the potential of the canonical Transformer without modifying any component\" ($\\underline{\\text{Lines 60-61 of Introduction}}$), we incorporate external information into TimeXer predictions through series-level dependencies, which works well. We would like to leave the exploration of multi-granularity effects in future work. As for the exploitation of LLMs, Reviewer 1zKu has a similar question, you can refer to our rebuttal to 1zKu Q1 for experiments on leveraging LLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OjsxsuBSDW", "reviewer": "Reviewer_1zKu", "summary": "The article designs TimeXer, Empowering Transformers for Time Series Forecasting with Exogenous Variables. TimeXer effectively utilizes exogenous information to enhance the accuracy of time series forecasting. Extensive experimental results validate the effectiveness of the proposed method.", "questions": "Q1. Have you considered exploiting large language models (LLM) and leveraging rich textual knowledge, to improve the model's ability to understand exogenous variables at multiple granularities?\n\nQ2. In most forecasting scenarios, do you think that the historical information of the target sequence itself is more useful, or is the information from exogenous variables more valuable? You can discuss on it freely.", "limitations": "The authors have thoroughly discussed the limitations of this work in the appendix.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1. Unlike general multivariate time series forecasting, this paper introduces a new task for time series forecasting by integrating exogenous variables to predict endogenous sequences, which is a meaningful research direction.\n\nS2. The experiments are comprehensive, with detailed validations on datasets from different domains and under various prediction horizons.\n\nS3. The method is simple and intuitively effective.\n\nS4. The scalability validation in Figure 4 is interesting and convincing.", "weaknesses": "W1. There is a potentially slight inconsistency between the description of the method and the scale of the data used in the experiments. The method section suggests that the length of the exogenous variables differs from that of the endogenous variables. However, in the datasets used for both long-term and short-term forecasting, the lengths of the exogenous and endogenous variables are the same.\n\nW2. Although the authors extensively discuss related work on considering relationships between variables in time series forecasting, some highly relevant literature is missing [1,2]. It is recommended to include these in the camera-ready version.\n  \n[1] CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement. NeurIPS 2023.\n  \n[2] FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective. NeurIPS 2023.\n\nW3. Figure 5 presents the memory usage of TimeXer and some baselines. Essentially, TimeXer and iTransformer share a similar model structure, yet TimeXer consumes less memory. What do you believe accounts for TimeXer being more memory-efficient than iTransformer?\n\nW4. Table 4 presents a comprehensive ablation study. However, I would like to see the complete removal of the exogenous variable enhancement module, for instance, by setting the exogenous variables to zero or random numbers, to explore TimeXer's adaptability to exogenous variables. If the performance deteriorates, it may indicate that TimeXer's results depend on the quality of the exogenous variables. If the performance remains stable, it suggests that TimeXer has good adaptability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 1zKu\n\nMany thanks to Reviewer 1zKu for providing a detailed review and questions.\n> **W1**: There is a potentially slight inconsistency between the description of the method and the scale of the data used in the experiments. \n\nThank you for your careful reading. Since most of the baseline forecasters can only support cases where the lengths of the exogenous and endogenous variables are the same, we set the lengths of the exogenous and endogenous variables to be the same to align with previous benchmarks for a fair comparison.\n\nWe have **experimented with the case where the exogenous and endogenous variables are of unequal length** in $\\underline{\\text{Section 4.3 of main text}}$, encompassing two common real-world scenarios: (1) unequal look-back length (2) different sampling frequencies. As presented in $\\underline{\\text{Figures 3,4 of main text}}$, TimeXer exhibits superior generality over various practical cases. \n> **W2**: Some highly relevant literature is missing.\n\nThank you for the recommendation. We will include them as baselines in the camera-ready version. Considering these two models are proposed for multivariate data, we compare the TimeXer performance in multivariate forecasting with those given in their original papers. Note that the experimental setup in FourierGNN is not the same as TimeXer, we use the results reported by ForecastGrapher[1].\n\n|MSE\\|MAE|Electricity|Traffic|Weather|\n|-|-|-|-|\n|TimeXer|**0.172**\\|**0.272**|**0.472**\\|**0.283**|**0.242**\\|**0.283**|\n|CrossGNN|0.201\\|0.297|0.583\\|0.323|0.247\\|0.289|\n|FourierGNN|0.228\\|0.324|0.557\\|0.342|0.249\\|0.302|\n\n[1] \"ForecastGrapher: Redefining Multivariate Time Series Forecasting with Graph Neural Networks.\" arXiv preprint arXiv:2405.18036 (2024).\n> **W3**: Essentially, TimeXer and iTransformer share a similar model structure, yet TimeXer consumes less memory. What do you believe accounts for TimeXer being more memory-efficient than iTransformer?\n\nMany thanks for your insightful question. We have included a theoretical efficiency analysis w.r.t. iTransformer in $\\underline{\\text{Appendix E}}$. Here is an intuitive understanding.\n\nThe key design of iTransformer is embedding each variate series into one token. Afterwards, the embedding variate tokens will be fed to Transformer, which will apply a self-attention mechanism among variate tokens. Although this design can keep refining the learned variate token in multiple layers, it does cause more complexity.\n\nAs for TimeXer, as presented in $\\underline{\\text{Figure 2(b) of main text}}$, exogenous series will be embedded to variate tokens at the beginning, which will be shared in all layers and interact with the endogenous global token by cross attention. Thus, **TimeXer omits the interaction among learned exogenous variate tokens**, which makes TimeXer more efficient than iTransformer.\n> **W4**: I would like to see the complete removal of the exogenous variable enhancement module, for instance, by setting the exogenous variables to zero or random numbers, to explore TimeXer's adaptability to exogenous variables. \n\nWe have provided the experimental results on TimeXer's generality under missing exogenous values in $\\underline{\\text{Appendix C}}$. As per your request, we have conducted more ablation studies on the quality of the exogenous variables. \n\nTechnologically, we use two kinds of exogenous variables: (1) set the series into zeros (TimeXer-zerosEX) and (2) set the series to random numbers with mean 0 and variance 1 (TimeXer-randomEX). As demonstrated in the $\\underline{\\text{Table 1 in the Author Rebuttal PDF}}$, the inclusion of exogenous variables leads to an improvement in model performance, while the performance decreases when the exogenous variables are meaningless noise.\n> **Q1**: Have you considered exploiting large language models (LLM) and leveraging rich textual knowledge, to improve the model's ability to understand exogenous variables at multiple granularities?\n\nFollowing the reviewer's suggestion, we have endeavored to leverage large language models for forecasting with exogenous. Technologically, we design a multi-grained prompt and use a T5 tokenizer to extract textual information, which is then integrated into TimeXer by adding it to the corresponding variate tokens. The prompt consists of the name and sampling frequency of each variable, for example, \"This exogenous variable is load generation, with a sampling frequency of 1h. We use it to enhance the prediction of electricity price in the Nord Pool market\".\n\n|Short-term(MSE\\|MAE)|NP|PJM|BE|FR|DE|AVG|\t\n-|-|-|-|-|-|-\n|TimeXer-LLM|0.236\\|0.266|0.106\\|0.198|0.382\\|0.245|0.401\\|0.218|0.450\\|0.422|0.315\\|0.270|\n\nOur experiments indicate that there was no improvement in model performance, likely due to insufficient textual information. We believe that a more sophisticated design for prompts is necessary to capture exogenous variables at multiple levels of granularity, which we will address in future work.\n> **Q2**: In most forecasting scenarios, do you think that the historical information of the target sequence itself is more useful, or is the information from exogenous variables more valuable?\n\nThanks for the reviewer's valuable suggestion. As per your request, we set the exogenous and endogenous series to zeros respectively (TimeXer-zerosEX and TimeXer-zerosEN), and the results are listed in $\\underline{\\text{Table 1 in the Author Rebuttal PDF}}$. We find that TimeXer benefits more from the historical information of endogenous series.\n\nTherefore, in most forecasting scenarios, the historical information of the target sequence itself is more useful. However, there are specific cases where the exogenous variable has a time-lagged influence on the non-stationary endogenous variable, in which case the exogenous series may be more useful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pg1gFAN6pJ", "reviewer": "Reviewer_N2A5", "summary": "The paper introduces `TimeXer`, a novel approach for time series forecasting with exogenous variables. Its main contributions include a new Transformer-based model architecture designed to effectively integrate and utilize exogenous information alongside endogenous time series data. The paper demonstrates the effectiveness of `TimeXer` through extensive experiments on various real-world datasets, showcasing its superior performance compared to **state-of-the-art** methods.", "questions": "- How is the memory consumption of the model measured? It seems that there is no implementation in the provided code.\n- The prediction task of TimeXer and previous prediction task such as `iTransformer` and `DLinear` are essentially different types of prediction targets. More details need to be disclosed on how to transfer previous prediction tasks in the text.", "limitations": "- The paper adequately discusses the limitations, such as the need for more extensive experimentation and the potential challenges with a large number of exogenous variables.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper introduces a novel approach to **integrating exogenous variables** into **Transformer-based** models for time series forecasting.\n- The model design is well-thought-out, and the experiments are carefully executed.\n- The paper is well-written and clearly articulates the problem, solution, and experimental results.\n- The work has potential implications for improving forecasting accuracy in various real-world applications.", "weaknesses": "- There is no comparison between native models that can be used for forecasting with Exogenous Variables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer N2A5\n\nWe would like to sincerely thank Reviewer N2A5 for providing a detailed review and insightful suggestions.\n\n> **W1**: There is no comparison between native models that can be used for forecasting with Exogenous Variables.\n\nIn $\\underline{\\text{Table 7 of the Appendix}}$, we have compared the performance of TimeXer with previous approaches, NBEATSx and TFT, both of which are natively designed for forecasting with exogenous variables. To make this clearer, we present the results here and TimeXer surpasses these two native models.\n\n|Short-term(MSE\\|MAE)| NP           | PJM          | BE           | FR           | DE           | AVG          |\n|-|-|-|-|-|-|-|\n| TimeXer    | **0.236** \\| **0.268** | **0.093** \\| **0.192** | **0.379** \\| **0.243** | **0.385** \\| **0.208** | **0.440** \\| **0.415** | **0.307** \\| **0.265** |\n| NBeatsX     | 0.272 \\| 0.301 | 0.097 \\| 0.189 | 0.389 \\| 0.265 | 0.393 \\| 0.211 | 0.499 \\| 0.447 | 0.330 \\| 0.283 |\n| TFT     | 0.369 \\| 0.391 | 0.141 \\| 0.241 | 0.479 \\| 0.305 | 0.461 \\| 0.249 | 0.559 \\| 0.490 | 0.402 \\| 0.335 |\n\n\n> **Q1**: How is the memory consumption of the model measured? It seems that there is no implementation in the provided code.\n\nWe assess memory consumption by monitoring GPU memory usage during the model training phase.\n\n> **Q2**: The prediction task of TimeXer and previous prediction task such as iTransformer and DLinear are essentially different types of prediction targets. More details need to be disclosed on how to transfer previous prediction tasks in the text.\n\nSorry for the missing details. As stated in $\\underline{\\text{Line 193 of main text}}$, we discover that the forecasting with exogenous variables task can be generalized into multivariate forecasting tasks through the channel independence mechanism. Technologically, we consider the variables in the dataset as mutually independent endogenous variables, with each variable considering all other variables except itself as exogenous. We also have provided the code for multivariate forecasting in the $\\underline{\\text{supplementary materials}}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Summary of Revisions and Global Response

We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve our paper further.

In this paper, we dive into a practical forecasting setting in real-world applications, i.e. time series forecasting with exogenous variables. We propose TimeXer as a simple and general model that explicitly distinguishes between endogenous and exogenous variables through two different embedding modules, with an endogenous global token as a bridge in-between. With this design, we empower Transformer architecture to the inclusion of exogenous variables without architectural modifications. **Experimentally, TimeXer surpasses 9 advanced baselines in 12 well-established datasets in both short-term forecasting with exogenous variables and long-term multivariate forecasting with favorable efficiency and interpretability.**

The reviewers generally held positive opinions of our paper, in that the proposed method is "**novel**", "**well-thought-out**", **a versatile framework**", "**intuitively effective**", and "**advances the field in a meaningful way**", "**has a strong practical value**", "**has potential implications**"; the experiments are "**extensive**", "**comprehensive**", "**interesting and convincing**", "**carefully executed**" and demonstrate "**adaptability**", "**the effectiveness of the method across various datasets and experimental settings**".

The reviewers also raised insightful and constructive concerns. We made every effort to address all the concerns by providing detailed descriptions and requested results. Here is the summary of the major revisions:

- **Provide more intuitive examples for exogenous variable utilization (Review r8UR):** We visually present the prediction showcases in four different cases in the following PDF, including only using endogenous or exogenous series, using both of these two kinds of series and using the prediction values of exogenous series, to validate the role of exogenous variables. 

- **Explain why our method uses the non-overlapping patch method (Reviewer Mh7v):** Following the reviewer's question, we conduct an ablation study on the effects of patching, we find that the non-overlapping has the lowest complexity with the optimal performance.

- **Explain how to extend TimeXer to multivariate forecasting tasks (Reviewer Mh7v, N2A5):** Following the reviewer's suggestion, we elucidate the rationale behind TimeXer's adaptability to multivariate prediction tasks.

- **Experiments under extreme cases (Reviewer Mh7v, 1zKu):** As per the reviewers' request, we have completed comprehensive ablation studies in five distinct scenarios, including reducing the step length of exogenous/endogenous variables to 1, setting the exogenous/endogenous variables to zeros, and setting the exogenous variables to random numbers. TimeXer exhibits adaptability across these diverse scenarios.

- **Add comparison with CrossGNN and FourierGNN (Reviewer 1zKu):** Following the reviewer's suggestion, we have compared the above two baselines in short-term forecasting benchmarks. Compared to these new baselines, TimeXer still performs best.

- **Explore the usage of LLMs in forecasting with exogenous variables (Reviewer 1zKu):** Following the reviewer's suggestion, we have designed a textual prompt to describe the exogenous variable information and tested whether the exploitation of large language models can improve the model's ability to understand exogenous variables at multiple granularities.

The valuable suggestions from reviewers are very helpful for us to revise the paper to a better shape. We'd be very happy to answer any further questions.

Looking forward to the reviewer's feedback.

#### **The mentioned materials are included in the following PDF file.**

- **Figure 1 (Reviewer r8UR)**: Intuitive showcase of the prediction results under different scenarios.
- **Table 1 (Reviewer 1zKu)**: Experimental results for W4 and Q2.

Meta-review: This paper proposes a new time series forecasting model that uses exogenous variables in the prediction. The exogenous variables are valuable for forecasting but they are often ignored in the existing models. The proposed method unifies exogenous and endogenous variables using patch-wise self-attention and variate-wise cross-attention. The proposed method shows state-of-the-art performance on real-world forecasting benchmarks. The reviewers consider this work as novel and of strong practical value. The experimental studies in this paper are extensive, which makes the results trustable. I would like to recommend it for acceptance.
Decision: Accept (poster)