ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer
Authors: Jiawen Zhang
Shun Zheng
Xumeng Wen
Xiaofang Zhou
Jiang Bian
Jia Li
Venue: NeurIPS 2024 poster (2024)
Keywords: time-series forecasting
arbitrary-horizon forecasting
transformer
Primary area: deep_learning_architectures

Abstract: Numerous industrial sectors necessitate models capable of providing robust forecasts across various horizons. Despite the recent strides in crafting specific architectures for time-series forecasting and developing pre-trained universal models, a comprehensive examination of their capability in accommodating varied-horizon forecasting during inference is still lacking. This paper bridges this gap through the design and evaluation of the Elastic Time-Series Transformer (ElasTST). The ElasTST model incorporates a non-autoregressive design with placeholders and structured self-attention masks, warranting future outputs that are invariant to adjustments in inference horizons. A tunable version of rotary position embedding is also integrated into ElasTST to capture time-series-specific periods and enhance adaptability to different horizons. Additionally, ElasTST employs a multi-scale patch design, effectively integrating both fine-grained and coarse-grained information.  During the training phase, ElasTST uses a horizon reweighting strategy that approximates the effect of random sampling across multiple horizons with a single fixed horizon setting. Through comprehensive experiments and comparisons with state-of-the-art time-series architectures and contemporary foundation models, we demonstrate the efficacy of ElasTST's unique design elements. Our findings position ElasTST as a robust solution for the practical necessity of varied-horizon forecasting.

ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer
Jiawen Zhang' DSA, HKUST(GZZ) Guangzhou, China j jiawe. zh@gmail.com
Shun Zheng Microsoft Research Asia Beijing, China shun. www.gggnccooooot.toommm com
Xumeng Wen Microsoft Research Asia Beijing. China xumengeenndnicrooofff. com
Xiaofang Zhou CSE, HKUST Hong Kong SAR, China exffust.. hk
Jiang Bian
Jia Li DSA, HKUST(GZZ) Guangzhou, China j jialeeQust. hk
Microsoft Research Asia Beijing, China
j jiang. bi www.mmicooooottttoooo.oooooooo.. com
Abstract
Numerous industrial sectors necessitate models capable of providing robust fore- casts across various horizons. Despite the recent strides in crafting specific ar- chitectures for time-series forecasting and developing pre-trained universal mod- els, comprehensive examination of their capability in accommodating varied- horizon forecasting during inference still lacking. This paper bridges this gap through the design and evaluation of the Elastic Time-Series Transformer (ElasTST). The ElasTST model incorporates non-autoregressive design with placeholders and structured self-attention masks, warranting future outputs that are invariant to adjustments in inference horizons. tunable version of rotary position embedding is also integrated into ElaSTST to capture time- seme--srres-speciiie periods and enhance adaptability employs multi-scale patch design, effectively integrating both fine-grained and coarse-grained information. During the training phase, ElasTST uses a horizon reweighting strategy that approximates the effect of random sampling across mul- tiple horizons with a single fixed horizon setting. Through comprehensive ex- periments and comparisons with state-of-the-e time-series architectures and temporary foundation models, we demonstrate the efficacy of ElasTST's unique design elements. Our findings position ElasTST robust solution for the practical necessity of varied-horizon forecasting. ElasTST open-sourced at
different horizons. Additionally, ElasTST
Introduction
Time-series forecasting plays a crucial role in diverse industries, where is essential to provide forecasts over various time horizons, accommodating both short-term and long-term planning re- quirements. This includes predicting COVID-19 cases and fatalities one and four weeks ahead to hourly, weekly,
allocate public health resources [[Z], estimating future electricity demand monthly basis to optimize power management [16], and projecting both immediate and long-term traffic conditions for efficient road management [Z] 27], among others.
"This work was done during the internship at Microsoft Research Asia. Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite this, a majority of advanced time-series Transformer [+| variants developed in recent years still necessitate per-horizon training and deployment [17 90. 33, 32,
20. [9]. These models struggle to handle longer inference horizons once trained for specific horizon, and may yield sub- optimal performance when assessed for shorter horizons. These constraints lead to the practical inconvenience of maintaining distinct model checkpoints for different forecasting horizons required by real-world applications.
22
Even though recent studies on pre-training universal time-series foundation models have made some progress in facilitating muriedhorizon forecasting [26. 9. B. ETT] they primarily concentrate on as- sessing the overall transfer performance from pre-training datasets to zero-shot scenarios. However, they lack an in-depth investigation into the challenges of generating robust forecasts for different horizons. To be specific, TimesFM [+], decoder-only Transformer, is capable of arbitrary- forecasting, but this approach could potentially lead to substantial error propagation forecasting scenarios due to autoregressive decoding. DAM [8], though free from this issue thanks to novel output design composing sinusoidal functions, cannot effectively capture abrupt changes in time-series data, thereby limiting its utility in critical domains such energy and traffic. More- over, while MOIRAI [[T]] employs a full-attention encoder-only Transformer architecture and sup- ports Aibitrary-horizon forecasting via non-autorrgresssvv manner by introducing mask tokens into forecasting horizons, it remains uncertain how well MOIRAI adapts to different horizons. For ex- ample, its architecture design does not ensure the
long-term
property: the model output for a specific future position should be invariant to arbitrary extensions in forecasting horizons beyond that. Besides, its performance could drop significantly for moderate context lengths.
To address this research gap, we introduce a comprehensive study to explore how to construct a time-series Transformer variant that can yield robust forecasts for varied inference horizons once trained. We name the developed model as Elastic Time-Series Transformer (ElasTST), ElasTST adopts non-autoregressive design by incorporating placeholders into forecasting horizons, which is inspired by diffusion Transformers [[a]] and the success of SORA [G] in video generation. Here we impose structured self-attention masks, only allowing placeholders to attend to observed time- series patches. This design ensures the aforementioned horizon-invariant property by blocking the information exchange across placeholders. Additionally, we devise tunable version of rotary posi- tion embedding (RoPE) [28] to capture customized period coefficients for time series and to learn the adaptation to varied forecasting horizons. Furthermore, we introduce multi-patch design to balance fine-grained patches beneficial to short-term forecasting with coarse- -rrniied patches pre- ferred by long-term forecasting, and use shared Transformer backbone handle these multi-scale patches. Alongside core model designs, during the training phase, we deploy a horizon reweight- ing approach that approximates the effects of random sampling across multiple training horizons using just one fixed horizon, eliminating the need for additional sampling efforts. Collectively, these key customizations facilitate ElasTST to produce consistent and accurate forecasts across various horizons.
Our extensive experiments affirm the effectiveness of ElasTST in varied-horizen forecasting. First, we evaluated FlasTST, trained with a fixed horizon and employing a reweighting scheme, against state-of-the-arr models trained for specific inference horizons. The results demonstrate that ElasTST delivers competitive performance without requiring per-horizon tuning. Then, we examined varied- horizon forecasting for these models, and the advantages of ElasTST are much more outstand- ing, demonstrating remarkable extrapolations to longer horizons while preserving robust results for shorter ones. Moreover, we also compared ElasTST with some pre-trained time-series models, such as TimesFM and MOIRAL, and found that dataset-specific tuning still offers prominent advantages over sero-shot inference in challenging datasets, such as Weather and Electricity, and that ElasTST can provide more robust performance across different forecasting horizons. At last, we conducted comprehensive ablation tests to highlight the significance of each unique design element of EST
In summary, our contributions comprise:
Conducting a systematic study on saried-horizon forecasting, a critical requirement across various domains, yet an underexplored area in time-series research.
Developing a novel Transformer variant, ElasTST, which incorporates structured atten- tion masks for horizon-invariance, tunable RoPE for time-series-specife periods, multi- patch representations to balance fine-grained and moarse-graines information, and a horizon reweighting scheme to effectively simulate varied-horizes training.

---

Demonstrating the effectiveness of ElasTST through experiments comparing it with state- of af-the-art time-series architectures and some up-to-date foundation models. Our ablation tests further reveal the importance of its key design elements.
2 Related Work
Traditional Neural Architecture Designs for Time-Series Forecasting The field of time-series forecasting has witnessed a significant evolution of neural architectures, transitioning from early multi-layer perceptrons [RK]] convolutional [S], and recurrent networks [[]], to more recent focus on various Transformer variants [37, BIO 63. G2, 22, 20, [9]. However, the challenge of varied- horizon forecasting remains underexplored in these studies, these models often require specific tuning to optimize performance for each inference horizon. Additionally, many models, including PatchTST [22]. iTransformer [20]], and MTST [39], utilize horizon-specitfe projection heads, which inherently complicates the extension of their forecasting horizons.
Developing Foundation Models for Time-Series Forecasting Inspired by the remarkable suc- cesses in the creation of foundational models in the language and vision domains [. 25, 3], the trend of pre-training universal foundation models has emerged in time-series forecasting research. Notable works in this area include Lag-Llama [26]. DAM [8], TimesFM [9], and MOIRAI [3I]. These studies employ unique designs to address the challenges posed by varied variate numbers and forecasting horizons when adapting new scenarios. Lag-IJama, DAM, and TimesFM adopted the univariate paradigm to circumvent the difficulties associated with handling different variates. In contrast, MOIRAL AI has taken a different approach by flattening multi-variate time series into single sequence to facilitate variatv learning. While this method has its merits, it is worth noting that may introduce efficiency issues when handling a substantial number of variates and long forecast- ing horizons. As result, this paper also adopts the univariate setup to maintain efficiency. When it comes varied forecasting horizons, Lag- Llamm and TimesFM both utilized the decoder-only Trans- former and relied on autoregressive decoding to manage arbitrarily long horizons. DAM introduced a novel output scheme that comprises numerous sinusoidal basis functions, enabling it to project into arbitrary future time points. MOIRAI, on the other hand, used composite input scheme, combin- ing observed time-series patches with variable placeholders that indicate forecasting horizons, and built full-attention encoder-only Transformer on top of this. Interestingly, this non-autoregressive generation paradigm originates from diffusion transformers used video generation [24,, EJ.. In this paper, we also embrace this paradigm for generating variable-length time-series. Unlike MOIRAI, which has made considerable strides in time-series pre-training using a moderately designed Trans- former variant, improve robustness in time-series forecasting across various horizons. We believe that constructing more robust, resilient, and universal architecture will pave the way for more powerful foundational time-series models to be pre-trained in the future.
focus lies in systematically examining critical architectural enhancements to
Position Encoding in Time-Series Transformers Position encoding plays a pivotal role in Trans- formers as both self-attention and feed-forward modules lack inherent position awareness. The majority of existing time-series Transformer variants have roughly adopted absolute position en- coding [B0] with minor modifications across different studies. For instance, Informer [6[]] and Pyraformer [19] have combined fixed absolute position embeddings with timestamp embeddings such day, week, hour, minute, etc. Meanwhile, Autoformer [[3] and Fedformer [41] have omitted absolute position embeddings and relied solely on timestamp embeddings. Other models like Log-
Trans [ and PatchTST [22]] have explored learnable position embeddings. However, the challenge with absolute position embedding is its inability to extrapolate into unseen horizons, posing a signif- icant challenge for varied-horizon forecasting. To address this issue, MOIRAI has utilized relative position embedding technique, RoPE [28]] which has been broadly adopted in the language domain to handle variable-length sequences | In our work, we also adopt RoPE to introduce relative position information into self-attention operations. What we uniquely reveal is that the direct appli- cation of the RoPE configuration from the language domain to time-series forecasting is not ideal. The reason being that the predefined coefficients do not align well with the typical periodic patterns observed in time-series data. As solution, we suggest redefining the period range encompassed by the initial RoPE coefficients and making data-driven adjustments to these coefficients.
3

---

Multi-Patch Assembly
(d)
Reweighting Function (T)
Structured
Attention Mask M E B-
Loss Func.
Dec.
Dec.
Dec.
Transformer Block
Feed Forward
Model Outputs & ut 12
(b)
Tuned Period
720
Layer Norm
Mantetlen /vctt /wd/// W
Coefficients P)
Masked Self-Att.
TluPe (.))
Norm
ElasTST
Tuning
E (.,)
Enc.
Enc.
Enc.
Model Inputs X E RL+T
1000
Initialized Period
100
Coefficients Py
Patching
Placeholder
Figure 1: Overview of the ElasTST Architecture. ElasTST employs (a) structured attention masks for placeholders ensure consistent outputs across varied forecasting horizons. It incorporates (b) tunable RoPE customized to time series periodicities, enhancing its robustness. The architecture also integrates (c) multi-scale patch assembly that merges fine-grained and coarse-grained details for improved forecasting accuracy. Furthermore, we implement (d) training horizon reweighting scheme during the training phase, which effectively simulates random sampling forecasting hori- zons, reducing the need for additional sampling efforts.
Input Patches in Time-Series Transformers PatchTST [2 spearheaded the concept of segment- ing time-series data into patches instead of feeding raw time-series values directly into Transformer models. This straightforward yet effective approach has been widely adopted in subsequent studies, including MTST [39]. TSMixer [IO]) HDMixer [17]], and MOIRAI. It noteworthy that MOIRAI has been trained with a diverse range of time-series patches with varying patch sizes. When adapting it to new dataset, practitioners need to search through range of patch sizes and rely valida- tion performance to select single patch size. In our work, however, we have demonstrated that segmenting time series into multiple patch sizes to create multi-scale patch representations is more advantageous. This approach further aids stabilizing accurate forecasting across various horizons.
Elastic Time-Series Transformers
In Figure II, we present an overview of ElaSTST. Different from other encoder-only Transformer ar- chitectures, ElasTST equipped three designs facilitate varied-horizon forecasting: structured self-attention masks for placeholders, tunable rotary position embedding (TRoPE) with custimized period coefficients, and multi-scale patch representation learning. Additionally, utilize hori- zon reweighting scheme to achieve the effects of varied-horizon training.
Notations We define univariate time series æ1.7 {are)== with v, R indicating the value at time index f. The learning objective of a varied-horizon forecasting can be formulated as: max, Ee-p(D))(e (t. (.TT)())( log Ps (#t441444| -1,- where p(D) is the data distribution from which time series samples are drawn, and p(T) is the task distribution, from which the timestamp t, look-back window L, and the prediction horizon T are sampled.
Model Inputs To accommodate varied forecast horizons, our model combines the historical context series Fi-L+1: with placeholders E RT' through concatenation, forming the input X Concat(z 0). This approach allows for flexible adjustment of the input and output di- mensions to suit different forecasting scenarios. We further segment X into non-overlapping patches X" E NNP,, where P is the patch length and N (L+T) represents the number of patches. Each input patch is then transformed into latent space by the encoder H Enc(X²), H E NDD.

---

Masked Self-Attention robust -hrizon forecasting method should deliver consistent out- puts across different forecasting horizons while maintaining high accuracy on unseen horizons. Ex- isting time series Transformers, however, typically directly adapt techniques from video genera- tion and natural language processing without considering the unique characteristics of time series. To address this deficiency, ElasTST modifies standard Transformer Encoder with two crucial en- hancements: structured attention masks and a tunable RoPE to encode relative position information effectively. We formulate the attention scores within masked self-attention
am,. (hi W9.m),
m), ff FTRPP (h, W .n)) Mm.n.
where W W E Dxd denote the linear mappings for the query and key, respectively. A tunable RoPE f fIRoPE dynamically adjusts the relative position encoding manner to best suit each dataset, with further details provided in the following subsection. The structured attention mask M. is set to 0 for patches X consisting solely of placeholders and otherwise, ensuring that tokens attend only to context-carryng patches. This structured masking, in conjunction with the relative position encoding, prevents the influence of placeholders on prediction outcomes, thus ensuring consistent outputs across varied forecasting horizons.
Tunable Rotary Position Embedding Position embedding is crucial for the attention mecha- nism to maintain accuracy over unseen horizons. To overcome the limitations of absolute posi- tion embedding in extrapolation scenarios, RoPE has been widely adopted in the NLP domain for handling variable-length sequences. It rotates vector x E I² onto an embedding curve on a sphere in Cd/2, with the rotation parameterized by base frequency b. The function is defined as (RoPP tzx,t); (#2j-1 izzy)e -b---//)// where j E [1, ..., d/2] [[a]]. Typically in NLP, the base frequency is set to a constant, such as 10,000. However, due to the unique characteristics of time series data, specific adaptations of RoPE necessary. In this paper, propose to use the for parameterization:
period coefficients Py
b 20 17a
STROPE
(x, t)j = (22j-1 + ix2;)e )
(2)
P Pmax
P3 23=0mias Pmin 20(i-1), (a(i)2
a d In
Pmin
where Pmin and Pmax represent the predefined minimum and maximum period coefficients, respec- tively. This formula maintains an exponential distribution but adjusts the range to better align with the periodic characteristics of time series data. By setting Pinin approach mirrors the original RoPE setup.
2Tr and Piaax = 2mbl- á this
In addition to adjusting the period range, the distinct and varying periodicities inherent in time series data necessitate more flexible period coefficients. Therefore, in ElasTST, we consider period coef- ficients SP as tunable parameters, optimizing it along with varied datasets and forecasting horizons. This adaptive approach allows for more precise and effective forecasting across diverse conditions. We provide a detailed exploration of this design in Section D.4, and illustrate the optimized period coefficients for each dataset in Appendix B
Multi-Scale Patch Assembly To ensure robust performance across various forecasting horizons, integrating both fine-grained and coarse-grained features from time series data is essential. Different from earlier multi-patch models that utilize separate processing branches for each patch size [[]] ElasTST features a multi-scale patch design within shared Transformer backbone, capable of both parallel and sequential processing. We chose sequential processing for our implementation, keeping the memory consumption comparable to baselines such as PatchTST. The implications of this design on memory usage are further discussed in Appendix E. Specifically, we define each patch size as p (Pı,..., ps}, with each size corresponding to a dedicated MI.P encoder fEne TRP + RDD
and decoder fD. IR. + I... The outputs from each size are flattened and then averaged to produce the final forecast X. During training, losses under individual patch size are calculated and averaged with the assembled forecast losses, to enhance accuracy and consistency across different scales. Further details on the effectiveness of this design is provided in Section
Training Horizon Reweighting To effectively manage varied forecasting horizons, training mod- els across multiple horizon lengths, rather than using fixed ones, is a practical approach [EI]. In
5

---

this study, we propose to use reweighting scheme for loss computation that simulates this process, without the need for additional sampling efforts. Formally, the conventional implementation, at each training step forecasting horizon T, is randomly selected from the range [1, T [ Then the loss L, at step is computed as:
T. L, = Σ (())( i4t)), w(T)= 1
T.
Theoretically, the expectation of this random sampling process can be represented as weighted loss over a fixed horizon T Tmax. To be specific, the expected value ((T) is calculated as: [E[[[T]] = We further approximate the reweighting function by harmonic series as:
Tun
T= T.
1 (In(T - In(T)).
w(T)
(5)
Tmax
By employing this weighted loss w(T during training, we replicate the effect achieved by randomly sampling horizons at infinite number of training steps. In addition, the function w(T) can be adapted follow any desired distribution family and can be made differentiable.
Experiments
To validate the effectiivnness of ElasTST, we systematically assess its performance across various forecasting scenarios, benchmarking it against established models. The results, detailed in Sec- tion I.l showcase ElasTSTss adaptability to diverse forecasting horizons. Subsequently. we perform an extensive ablation study in Section 2 to examine the impact of its key designs.
Datasets Our experiments leverage well-recognized datasets, including 4 from the ETT series (ETTh1, ETTh2, ETTm1, ETTm2), and others include Electricity, Exchange, Traffic, and Weather. These datasets cover a wide array of real-world scenarios and are commonly used as benchmarks in the field. Detailed descriptions of each dataset provided in Appendix CTl Following the setup described in [[S], all models use standard lookback window of 96, except TimesFM [5] and MOIRAI [3T], which utilize extended lookback windows of 512 and 5000, respectively.
Baselines For our comparative analysis, we select 6 representative forecasting models as baselines: (1) Advanced but non-elastic forecasting models, such as iTransformer [[0]]. PatchTST [20]. and DLinear [36]] (2) Autoformer [3%], which supports varied-horizon forecasting but requires horizon- specific tuming: (3) the cutting-edge time series foundation model like TimesFM [M] and MOIRAI
[ВI], which are pre-trained for general-purpose forecasting across varied horizons. Our analysis pri- marily assesses the saried-horizon forecasting capabilities, considering their pre-training subsets of the datasets used.
Implementation ElasTST is implemented using PyTorch Lightning [12], with training regimen of 100 batches per epoch, batch size of 32, and a total duration of epochs. We use the Adam optimizer with learning rate of 0.001, and experiments are conducted on NVIDIA Tesla V100 GPUs with CUDA 12.1. To ensure fairness, we conducted an extensive grid search for critical hyperparameters across all models in this study. The range and specifics of these hyperparameters are documented in Appendix DA For parameters not mentioned in the table, we adhered to the best practice settings proposed in their respective original papers. For evaluation, we use Normalized Mean Absolute Error (NMAE) and Normalized Root Mean Squared Error (NRMSE) as they are inlee-insensitive and widely accepted in recent studies
More details are in Appendix
4.1 Main Results
Comparing ElaSTST with Horizon Reweighting to Neural Architectures Tuned for Specific Inference Horizons Experimental results demonstrate that ElasTST consistently delivers excep- tional performance across all horizons without the need for per-horizon tuning. As evidenced in Ta- ble M1, ESaSTST outperformed SOTA models on diverse datasets including ETTm1, ETTh1, ETTh2,
'The look-back window is fixed.
2 Unless stated otherwise, horizon reweighting scheme is deactivated ablation study.

---

Traffic, Weather, and Exchange, despite these models undergoing specific horizon barizon-based training and tuning. This clearly demonstrates ElasTST's inherent robustness and its remarkable capacity to generalize effectively across varied forecasting scenarios.
Table 1: Results (mean on long-term forecasting scenarios with the best in bold and the second underlined. Each result contains three independent runs with different seeds. During the training phase, ElasTST utilizes a loss reweighting strategy where single trained model is applied across all inference horizons, where the max is set to 720. Other baseline models undergo -prizin-specific training and tuning. Additional baseline results are detailed in Appendix D.II
pred len
EST
iTranformer
PaccTTTT
DLineur
Auteformer
NMAE
NRMSE
NMAE
NRMSE
NMAE
NRMSE
NMAE
NRMSE
NMAE
NRMSE
96 0.273, 80o
0.488 000
00.27..co .000
0.508, 00o
0.272 001
0.565.om 0011
0.2882 002
0.573.o01
0..88, .011
0.711 Loo
192
0.289..om
0.520.oo0 .000
0.301
0.614 800
0.295
0.602.cm
0.309
0.617
0.442.cm
0.820 pos
336
0.314.000 000
0.575 000
0.333 o00
0.668 000
0.323.oo 001
0.645.008
0.338 008
0.654,007
0.429.oom
0.774, 001
720
0.346, uoa
0.645.ou0
0.376. .00
0.741. m00
0.353.om
0.700.oms
0.387 uos
0.737 oon
0.440.om
0.793. poo
96
0.150 ooo
0.227 ooo
0.137 .00
0.227 000
0.132. _011
0.220.om oon2
0,138 po0
0.226 mne
0.1158. ooo0
0.254 poo
E1
192
0.174 000
0.264 000
0.161.000 0000
0.266 000
0.157 .001
0.259. .022
0.163 oc3
0,264
0.175.00
0.283 000
336
0.191
0.289 poo
0.180
0.293
0.176.om ,oo0
0.286 _000
0.188 noa
0.291
0.191 100
0.307
720
0.211
oco
0.318.000
000
0.330 p00
0.205.o01
0.324.002
0.219 oc3
0.327 ooo
0.217 oo00
0.338 po0
96
0.342 000
0.619.oo0
0.321 .000
0.626 m00
0.328 ond
0.640.o0z
0.352 011
0.668.o12
00.667 .ovo
0.656 0oo
ETh1
192
0.304
800
0.661 .000
0.359.000
0.690 800
0.359.oo ,002
0.705.001
0.393.cm 001
0.745, o0o3
0.392.cmm
0.706 000
336
0.371 oo0
0.666.o08
0.388
0.723
800
0.384 oorz
0.740.0m4
0.419.
0.778 .o0
0.398 0.0
0.71
pou
720
0.376 000
0.679.000
0.408 .00
0.735 000
0.397 0022
0.738,001
0.502 029
0.860 oo4
0.433,onc
0.739.
.000
ET
192
96
0.170.00 000
0.158,
000
0.239.cm 000
0.259.00 000
0.177 .00
0.203 500
0.279 000
0.314 000
0.201 001
0.177 000
0.314.001
0.281 0011
0.211 027
.28
0.353 ooso
0.320 osa
00.203,oom o000
0.226.000
0.346 o00
0.317 0.00
3.6
0.188,00
0.282.co
0.243,
0.372
000
0.240 001
0.366
0.284,
0.407
0.264 ooc
0.398 oo0
720
0.215,000 000
0.319.00
0000
0,2611,00
0.386 000
0.252 000
0.371 .00
0.307 0c0
0.426.o07
0.2887.onc
0.416. ooo
Eleetrictt
96
0.085. o00
0.777 oo0
0.098 000
0.772.000 000
0.086.cm 001
0.816 oo5
0.090 001
0.863 oog
0.1140, .000
0.977 o16
192
0.0933 non
0.933 po0
0.106.cm ..00
0.916.com .00
0.092.0ox
0.942.
0.095 no1
0.97/1.om
0.136.cm 0o0
1.017 noo
336
0.101
1.063
0.115
0.985.00
0.100.000
1.035.om
0.104.
1.066
0.147 o000
1.080
720
0.117 no0
1.289 pou
0.133 .00
1.110.00x
000
0.116.000
1.213.om
0. 0.122,u03 n03
1.259 .00
0.159.cuv
1.283, pos
96
10.195.0mm
000
0.461 .000
0.246 .000
0.511
000
0.248.001
0.527
0.356 0c9 0.645.
0.299,.oo
0.560 oo0
Traffi
192
0.393, 000
0.459. 000
0.259 000
0.543
000
0.245 0011
0.528 0011
0.346 0c9
0.628 .00e
0.318, onc
0.594 6o0
336
0.199.00
.000
0.468
0.283,
0.571
.000
0.257 .002
0.550.00
0.350
0.631
0.332
0.630 .000
720
0.218.000 000
0.497 _000
0.275 000
0.563 000
0.266.001
0.559, p01
0.365 oc9
0.659 .00
0.341. ,003
0.611 o00
96
0.086.
0.287_000
0.089
0,295
0.087
0.294 ouz
0,112
0.316
0..399.cmm
0.614 o22
Waather
192
0.092 oco
0.312 oon
0.093. 000
0.299.com 000
0.090.oo1
0.299.co .o01
0,122 001
0.331 0000
0.213.cmm 0o133.occ
0.533 oo3
336
0.091
0.307 pou
0.096 .00
0,297 .000
0.092.co oz2
0.297.am
0.130
0.3-10
0.176.com
0.413 po1
720
0.093.o00 000
0.308 po0
0.099.0m o00
0.298.con
0.094.001
0.298.cm ooo
0.144. 001
0.358 .003
0.170.o01
0.434 010
Exchange
96
0.026
0.039 poo
0.025
0.039
0.023.0o0
0.036.com
0.024
0.037 .000
00.32.com
0.049
192
0.033, 000
0.050.000 000
0.036.a00 .00
0.056 000
0.034.oa0
0.05-4.000
0.036 aco
0.055.omm .000
0.041 Lonc
0.005 ooo
336
0.041.
0.062
0.048. .00
0.072
0.045
0.076.0m
0.048
0.072
0.056
0.091 poo
720
0.059.000 000
0.089.000 000
0.076.
0.114. 000
0.072.000
0.106,001
0.075.002
0002
0.118 .004
0.112,o02
0.164, 004
Comparing the Robustness of Different Models for Varied Inference Horizons The results clearly position ElasTST as the most robust option for deploying single, well-trained model across various inference horizons and application scenarios. As demonstrated in Figure ElasTST consis- tently maintains strong performance across both seen and unseen horizons, underscoring its ability to navigate beyond trained scopes with consistent accuracy across a wide range of forecasts.
In contrast, other models face significant challenges in paried-horizon forecasting. State-of-the-art models like iTransformer and PatchTST excel within their trained horizons but struggle when ex- tended beyond these limits. Models that require horizon-specific tuning, such as Autoformer, often experience abrupt declines in performance, illustrating that scalability alone is insufficient without tailored optimization. TimesFM, with its autoregressive nature, shows substantial error propaga- tion in unseen datasets like ETT and Exchange, and increased errors in pre-trained datasets such as Weather as the horizon extends. While MOIRAI demonstrates strong zero-shot performance on datasets like ETT and Exchange, we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages. Furthermore, MOIRAls performance sig- nificantly diminishes with shorter context lengths, as discussed in their paper [А]]] In comparison, EST operate effectively with much shorter context lengths.
7

---

Model
ElasTST
mannsfermr
DLinear
PatchTST
Autoformer Electricky
Times FM
MOIRA
ETT
Exchange
Weather
Traffic
0:38
030
0.58
0:35
0.00
0.20
9.00
0.55
0.39
0.30
0.38
0:1
0.15
0:04
0:30
623
0.02
192
336
720
152
226
720
152
235
720
352
2.36
724
342
2.20
728
Forecasting Horizon
Forecasting Horizon
Forocasting Horizon
Forocasting Horizon
Forecasting Honzzn
Figure 2: Performance of trained once and inference over varying forecasting horizons. Models ex- cept TimesFM and MOIRAI are trained with a forecasting horizon of 720 and tasked with predicting across multiple horizons. A vertical red dashed line distinguishes between their seen horizons (96, 192, 336, 720) and unseen horizon (1024). We use dashed line to denote the datasets on which the model was pre-trained, e.g., both TimesFM and MOIRAI have leveraged Traffic datasets for their pre- tre-training. The ETT encompasses averaged results from datasets ETTh1, ETTh2, ETTml, and ETTm2. Models lack inherent elasticity use truncation strategy for shorter forecasts, and the foundation models use their pre-trained checkpoints and recommended configurations for inference.
w/o Mask
w/o Tunable RoPE
--- w/o Multi-patch
ElasTST
ETT
Weather
Electricity
Traffic
0.42
0.12
NMAE
0.35
0.11
0.23
0.33
0.10
0.25
0.15
0.33
0.09
0.2
Inter hor.
Infer hor.
hor.
Infer hor.
Figure 3: Ablation study for the structured attention masks, tunable RoPE, and multi-patch assem- bly. vertical red dashed line indicates the training horizon.
4.2 Ablation Study
Structured Attention Masks The ablation study confirms that structured attention masks are es- sential for robust inference across horizons that differ from the training phase. As illustrated in Figure EL removing structured masks from TST results in significant performance declines, par- ticularly in the Weather dataset. Furthermore, as demonstrated Figure 0 (see Appendix (D.2), the benefits of structured masks consistent across all forecasting horizons. This underscores the im- portance of the invirriann property for enhancing the stability of time series forecasting, an aspect often overlooked in current research.
Tunable Rotary Position Embedding Experimental results indicate that tunable RoPE signif- icantly improves the models ability to extrapolate. Figure Aal shows that while other positional embedding methods are effective on seen horizons, they falter when applied to horizons extending beyond the training range. Although the original RoPE excels in NLP tasks, it underperforms in time series forecasting. Besides, data-driven adjustments of these coefficients enable far more ro- bust extrapolation. Dynamically tuning RoPE parameters according to the periodic patterns of the dataset proves highly beneficial, especially when inferring over unseen horizons.
Furthermore, a range from 1 to 1000 for the period coefficients SP is more suitable for time series forecasting. As demonstrated in Figure HiR using the commonly-used NLP settings with Amin = 1 and Pm 10000 does not fully exploit the potential of RoPE time series forecasting. Setting Punx to 1000 results in better performance. We hypothesize that this is because, unlike textual data which benefits from attention over longer contexts, the time series data, especially when segmented into patches, benefits from focusing on shorter, more recent intervals. By adjusting the maximum period coefficient to a lower value, the model captures a richer spectrum of mid-to-high frequency patterns, thereby enhancing its effectiveness. Detailed analyses of these findings are available in Appendix D..1 Appendix E includes visualizations demonstrating how different initial ranges impact frequeney components, along with illustrations of the tuned period coefficients for each dataset.

---

w/o PE
Abs PE
Learn PE
RoPE
Tunable RoPE
ETT
Weather
Electricity
Traffic
0.03
0.12
0.35
NMAE
0.35
0.13
0.30
0.30
0.43
0.25
336 Infer hor.
Infer hor.
Infer hor.
Infer hor.
(a) Comparison of various positional embeddings with the proposed Tunable RoPE.
10
100
1000
10000
Training Horizon: 96
Training Horizon: 192
Training Horizon: 336
Training Horizon: 720
0.28
0,28
0.26
0.26
NMAE
0.24
0.25
0.24
0.23
0.22
0:22
0.20
0.20
0.20
0.18
0.18
96
192
336
720
96
720 Forecasting Horizon
192
336
1024
96
192
336
720
1024
192
720
1024
Forecasting Horizon
Forecasting Harizon
Forecasting Horizon
(b) The effect of selecting Pmxx with Pain fixed at and @ is untunablc during training.
Figure 4: Ablation study for designs in position embedding. vertical red dashed line distinguishes between seen horizons and unseen horizons.
0.30
Patch
MMAE
16
32
0.20
64
8.16
8.16
1024
Forecasting Horizons
8.16 32,64 64
Figure 5: Performance of patch size selections. Results are averaged across all datasets and
training horizons of {96, 192, 336, 720). '8 $8_16_32 represents multi-patch configuration of {8, 32).
Multi-Patch Design These experiments demonstrate that multi-patch configurations generally out- perform single patch sizes across various forecasting horizons. Figure E shows that the configu- ration p = {8, 16, 32} consistently achieves the lowest NMAE values, effectively balancing the capture of short-term dynamics and long-term trends. However, adding larger patches, such as = {8, 16, 32, 64), does not consistently improve performance and can sometimes increase the NMAE. This suggests that more complex configurations may not always provide additional benefits and could even be counterproductive.
Moreover, the patch size selection is particularly critical in the varied-horizon forecasting scenarios. As demonstrated in the Figure I| (see Appendix D.S), various combinations training and forecast- ing horizons exhibit distinct preferences for patch sizes. For instance, when the training forecasting horizon 720, during the inference stage, longer forecasting horizons prefer larger patch sizes. Conversely, shorter training horizons, such 96 and 192, choosing large patch sizes for longer horizons can lead to performance collapse. This difference underscores the complexity and neces- sity of optimal patch size selection in achieving effective elastic forecasting. Detailed results for four training horizons and further analysis are provided in Appendix D.Y
The Impact of Training Horizons Further experiments validate the effectiveness of our proposed training horizon reweighting scheme enhancing varied-horizon inference. As illustrated in Fig- ure E reweighting longer horizons simplifies the training process, yielding better outcomes than

---

Train Hor.
96
192
336
720
Random Sample
Reweighting
ETTh1
Weather
Exchange
Electricity
Traffic
0.50
0.50
0.45
0.11
0.
0.45
0.06
0.40
0.43
0.30
0.39
91
0.25
0.30
0.02
0.30
7201824 infer hor.
1201024
95 7203024
36 192 7201024
122 336 1201024
Infer hor.
Infer hor.
infer hor.
Infer hor.
Figure 6: Impact of forecasting horizon selection during the training phase.
selecting a fixed horizon and mitigating the uncertainties associated with random sampling. Cru- cially, this training approach is model-agnostic and can be applied to different forecasting training scenarios. These results also highlight the advantages of a flexible forecasting architecture, which allows training horizons be customized to the unique characteristics of each dataset.
We also observe that different datasets have distinct preferences for training horizons. For example, in the Exchange dataset, the longest training horizon led to worse results compared to shorter hori- zon of 96, suggesting risks of overfitting or forecast instability with prolonged horizons. Besides, in the ETThl, employing random sampling for training horizons proved suboptimal. These insights show that tailoring the training horizon selection strategy to the specific dataset can yield improve- ments. One potential enhancement could involve dynamically optimizing the horizon reweighting scheme alongside model training.
5 Conclusion
This study introduces the Elastic Time-Series Transformer (ElasTST)), a pioneering model designed to tackle the significant and insufficiently explored challenge of varied-horizon forecasting. ElasTST integrates non-autoregressive framework with innovative elements such as structured self-attention masks, tunable Rotary Position Embedding (RoPE), and a versatile multi-scale patch system. Ad- ditionally, we implement& a training horizon reweighting scheme that simulates random sampling of forecasting horizons, thus eliminating the need for extra sampling efforts. Together, these elements enable TST to adapt to wide range of forecasting horizons, delivering reliable and competitive outcomes even when facing horizons that were not encountered during the training phase.
Limitations While ElasTST demonstrates robust performance across various forecasting tasks, several limitations have been identified that highlight opportunities for future enhancements. First, the current version of ElasTST does not incorporate a pre-training phase, which could significantly improve the models initial grasp of time-series dynamics and boost its efficiency during task- specific fine-tuning. Further exploration is needed to ascertain optimal training methodologies that maximize the architectural benefits of EST Additionally, while the training horizon reweighting scheme straightforward and effective in enhancing performance across different inference horizons, is not the optimal solution for all datasets. Moreover, the evaluation of ESa is limited to select number of datasets, which may not fully represent the broader challenges encountered in more complex or diverse real-world scenarios.
Future Work In response to these limitations, our forthcoming research efforts will concentrate on developing and validating pre-training protocols for ESa to elevate its foundational performance and extend its applicability across universal forecasting tasks. We aim to incorporate a reasonable training approach that will fine-tune the models ability to seamlessly manage forecasts of varying lengths, thus bolstering its utility in dynamic real-world environments. Furthermore, by broadening the range of datasets used for model evaluations, we intend to rigorously test ESasTSTss effectiveness across an expanded spectrum of andustry-specifie challenges. This comprehensive approach will not only solidify FlasTSTss standing as a cutting-edge solution for time-series forecasting but also en- hance our understanding of its practical implications and potential in diverse industrial applications.
10

---

References
[1] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango. Shubham Kapoor, et al. 2024. Chronos: Learning the language of time series. arXiv preprint
20X3::2403.07815 (2024).
[2] Joaquim Barros, Miguel Araujo, and Rosaldo JF Rossetti. 2015. Short-term real-time traffic prediction methods: A survey. In 2015 International Conference on Models and Technologies for Intelligent Transportation Systems.
[3] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. 2024. Video generation models world simulators. (2024). https://openai. com/research/
Ladeo-generation woweeioooooooessssee
models-aa wor -dd-ddd-- simulators
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In NeurIPS.
[5] Yitian Chen, Yanfei Kang, Yixiong Chen, and Zizhuo Wang. 2020. Probabilistic forecasting with temporal convolutional neural network. Neurocomputing 399 (2020), 491-501.
[6] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empir- ical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv:/412.3335 1412.3555 (2014).
[7] Estee Cramer, Evan Ray, Velma Lopez, Johannes Bracher, Andrea Brennen, Alvaro Castro Rivadeneira, Aaron Gerding, Tilmann Gneiting, Katie H House, Yuxin Huang, et al. 2022. Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States. Proceedings of the National Academy of Sciences (2022).
[8] Luke Nicholas Darlow, Qiwen Deng, Ahmed Hassan, Martin Asenov, Rajkam Singh, Artjom Joosen, Adam Barker, and Amos Storkey. 2024. Dam: foundation model for forecasting. In ICLR.
[9] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. 2023. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310. 10688 (2023).
[10] Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. Tsmixer: Lightweight mip-mixer model for multivariate time series forecasting. In SIGKDD, 459-469.
[11] Vijay Ekambaram, Arindam Jati, Nam H Nguyen, Pankaj Dayama, Chandra Reddy, Wes- ley M Gifford, and Jayant Kalagnanam. 2024. Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Sher Forecasting of Multivariate Time Series. arXiv preprint arXiv:2401.0395 (2024).
[12] William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning. https://doi. org/10. 5281/zenode 3828935
[13] Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis. and Marinka Zitnik. 2024. UniTS: Building Unified Time Series Model. arXiv preprint arXiv:2403.0013 (2024).
[14] Azul Garza, Cristian Challu, and Max Mergenthaler-Canseeo. 2023. TimeGPT-1. arXiv preprint ar..092310.03588 (2023).
[15] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. 2024. MOMENT: A Family of Open Time-series Foundation Models. In ICML.
11

---

[16] Luis Hernandez, Carlos Baladron, Javier M Aguiar, Belén Carro, Antonio J Sanchez- Esguevillas, Jaime Lloret, and Joaquim Massana. 2014. A survey on electric power demand forecasting: Future trends in smart grids, microgrids and smart buildings. IEEE Communica- sions Surveys Tutorials (2014).
[17] Qihe Huang, Lei Shen, Ruixin Zhang, Jiahuan Cheng, Shouhong Ding, Zhengyang Zhou, and Yang Wang. 2024. HDMixer: Hierarchical Dependency with Extendable Patch for Multivariate Time Series Forecasting. In AAAI, Vol. 38.
[18] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. In NeurIPS. 5244-5254.
[19] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. 2021. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, In ICLR.
[20] Yong Liu, Tengge Hu, Haoran Zhang. Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2023. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.0662. (2023).
[21] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. 2024. Timer: Transformers for Time Series Analysis at Scale. In ICML.
[22] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In ICLR.
[23] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2020. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. In ICLR.
[24] William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In ICCV. 4195 4205.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning trans- ferable visual models from natural language supervision. In ICML.
[26] Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, et al. 2023. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08276 (2023).
[27] Fei Su, Honghui Dong, Limin Jia, Yong Qin, and Zhao Tian. 2016. Long-tern forecasting oriented to urban expressway traffic situation. Advances in mechanical engineering (2016).
[28] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Ro- former: Enhanced transformer with rotary position embedding. Neurocomputiing 568 (2024), 127063.
[29] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. / 1397/ (2023).
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurlPS. 1998-6008..
[31] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sa- hoo. 2024. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592 (2024).
[32] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2023. TemesNet: Temporal 2D- Variatitn Modeling for General Time Series Analysis. In ICLR.
12

---

[33] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021.
Autoformer: Decom- position Transformers with Auto-Correlation for Long-Term Series Forecasting. NeurIPS. 22419-22430..
[34] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023. Effective long-context scaling of foundation models. arXiv preprint arXhv:239..10333 (2023).
[35] Jiexia Ye, Weiqi Zhang, Ke Yi, Yongzi Yu, Ziyue Li, Jia Li, and Fugee Tsung. 2024. A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Mode. arXiv preprini wrXiv22455.02355 (2024).
[36] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are Transformers Effective for Time Series Forecasting?. In AAAI. 11121-11128.
[37] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Fickhoff. 2021. transformer-basee framework for multivariate time series representation learning. In SIGKDD. 2114-2124.
[38] Jiawen Zhang, Shun Zheng, Wei Cao, Jiang Bian, and Jia Li. 2023. Warpformer: A multi-scale modeling approach for irregular clinical time series. SIGKDD. 3273-3285.
[39] Yitian Zhang, Liheng Ma, Soumyasundar Pal, Yingxue Zhang, and Mark Coates, 2024. Multi- resolution time-series transformer for long-term forecasting. In International Conference Artificial Intelligence and Statistics. 4222-4230.
[40] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan- cai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-series Forecasting. In AAAI. 11106-11115.
[41] Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. Fedformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. In ICML. 77268-27286.
13

---

A Details of Methods
A.1 Details of Rotary Position Embedding
Rotary position embedding (RoPE) [28]] is a method used to encode the position of tokens the in- put sequence for transformer-bused models, enhancing the capability utilize the positional context of tokens. RoPE uniquely incorporates the geometric property of vectors, transforming them into a rotary matrix that interacts with the vector embeddings.
Here, we have adopted the formal definition from the original RoPE paper. the simplest two- dimensional (2D) case, RoPE considers a dimension d = 2, where each position vector is treated in its complex form. The formulation is given by:
(Z m) (Wymmme
Sa (N n. (WLI,)e"
(Imm Tn, m n) Re( (W[,,,))WW,,app (W c(m=-)))
This expression shows that the embedding rotates the seiine-trnsformed word embedding vectors by angle multiples relative to their position indices. This rotation is mathematically represented through a multiplication matrix:
Coos m0
- sin m0 cos ml
( q.k w q.k
q.k w((2))
(1)
fq.k (Tm: m) =
sin ml
w2 q.k
(2)
For generalized form in any dimension d, RoPE divides the space into d/2 sub-spaces and com- bines them to utilize the linearity of the inner product. The generalized rotary matrix Iy.... is defined as:
cos md1
sin m8
0
sin m01
cos m01
0
0
0
0
0
cos m02
sin m0,
0
0
0
sin m02
cos m00
0
R
0
0
COS m/a/2 sin m02a22
sin mdd/2 cos m0
0
0
0
where θ (0; 10000 1(1-1)/4 10000-2((-11/4
E [1. d/2]}. This formulation ensures that RoPE is computationally efficient and stable due to the orthogonality of R The use of sparse matrices further improves computational efficiency, making RoPE practical approach to incorporate large- scale transformer models.
B Additional Related Work on Foundation Models
We summarize existing time series foundational models in Table A excluding the LLM--oieetted ones [G5]. 'Teee models typically use standard architecture designs, position encodings, and patch- ing approaches. primarily aiming to enhance transferability in zero-shot scenarios. However, they generally do not deeply explore the challenges of producing robust forecasts across varied hori- zons. Our work specifically addresses this gap by improving model design to enhance robustness for varied-horizon forecasting.
C Additional Details of Experiment Setting
C.1 Dataset Details
Our experiments utilize widely recognized datasets, including from the ETT series (ETTh), ETTh2, ETTml, ETTm2), well as the Electricity, Exchange, Traffic, and Weather datasets. These
14

---

extensive corpus of real-world and Table 2:
Summary of Time Series Foundation Models.
Model
Backbone
Dec. Scheme.
Pos. Emb.
Token.
TimeGPT-I
Enc-Dec Transformer
AR
Abs PE
Lag-Llama
Decuder-only Transformer
AR
RoPE
Chronos
Enc-Dec Transformer
AR
Simplified relative PE
Quantization
Timer [2]]]
Decoder-only Transformer
AR
Abs PE
Patching
TimesFM [8]]
Decoder-only Transformer
AR
Abs PE
Patching
UniTS [[]]]]
Transformer Encoder
NAR
Learnable PE
Patching
DAM [
Transformer Encoder
NAR
Abs PE
ToME
Tiny Time Mixers
||]
TSMixer
NAR
Patching
MOIRAI [51
Transformer Encoder
NAR
RoPE Learnable relative PH:
Patching
MOMENT 1
Transformer Encoder
NAR
Patching
datasets encompass a broad range of real-world applications and are frequently used as benchmarks in the fieldS. Consistent with common practices in long-term forecasting [G3, 36, 122 RQJ, all models are tested under the forecasting horizons 7 E {96, 192, 336, 720). Except for TimeFM [9], which uses lookback window of 512, standard lookback window of 96 is employed across all other
models as [66]]
Table 3: Dataset Summary.
Dataset
#var.
range
freq.
timesteps
Description
ETTh1/h2
7
R+
H
17,420
Electricity transformer temperature per hour
ETTm1/m2
7
R+
15min
69,680
Electricity transformer temperature every 15 min
Electricity
321
R+
H
26,304
Electricity consumption (Kwh)
Traffic
862
(0.1)
17,544
Road occupancy rates
Exchange
R+
Busi. Day
7,588
Daily exchange rates countries
Weather
21
R+
10min
52,696
Local climatological data
C.2 Implementation Details
ElasTST implemented using PyTorch Lightning [022]. Training consists of 100 batches per epoch, capped at 20 epochs, with the NMAE metric used for model checkpointing. We use the Adam optimizer with learning of 0.001, and experiments conducted on NVIDIA Tesla V100 GPUs with CUDA 12.1. The code for Transformer Block is adapted from [[8].
Baselines
We select five representative forecasting models baselines:
iTransformer
A transformer-based model that inverts the dimensions of time and variates to effectively capture multivariate correlations, enhancing generalization across different variates.
|00]:
PatchTST |
A transformer-based model segmenting time series into subseries-level patches and employing channel-independent processing to improve forecasting accuracy.
An MLP-based model that has demonstrated superior performance over more complex transformer-based models in multiple real-life datasets.
DLinear [6
Integrates a novel Auto-Correlation mechanism that exploits series periodicity for enhanced dependency discovery and representation aggregation.
Autoformer 0 |
TimeFM M A foundation model for time series forecasting, pretrained using a decoder- style attention mechanism and input patching synthetic data.
Datasets available
www.:/hww...oot.u.u..a..a. Autoformer hotpa//bocrccommmmmm/ttttotorrrr under MIT License.
htttge///gaaal..ooo////////aaaaa MIT License.
http:///iiibb.cc/////aaa////tttt Apache-2.1 license.
httpg///gghaaooooaa/eeaa////1111 2pa license.
hatpaa///gthhl...cc////aaaallaat License.
http:://ggghhh.oooooooo/o/eeerer Apache- 2.0 license.
15

---

MOIRAI (UNI2TS) 8 : A foundation model for time series forecasting, pretrained using masked encoder- based Transformer on the extensive Large-scale Open Time Series Archive (LOTSA) with over 27 billion observations.
Hyper-parameter Tuning To ensure fairness, we conducted an extensive grid search for critical hyperparameters across all models in this study. The range and specifics of these hyperparameters are documented in Table For parameters not mentioned in the table, we adhered to the best practice settings proposed in their respective original papers.
Table Hyper-parameters values fixed or range searched in hyper-parameter tuning. Models
Hyper-parameter
Value or Range Searched
learning rate multi multi_patch__e patch_sizee min fain__eriod_ceoe ccoeff
0.001
8-16-32
1
max Boox__eriod__o___e coeff
1000
ElasTST
hehadds n n_layers hidden_size
[2, 4, 8, 16]
[1,2,4]
[128,256,512
d_v
[64,128]
learning_ran
i'Transfommer
hidden_size
[128,256,512,100 1024]
e e_layers
[1,2,3,4]
learning rate
[0.0001, 0.001]
patch_lex
16
stride
8
n layers
3
PatchTST
d model
[16, 128,256,512]
d_ff
[128,256,512
n_heads
[4,8,66]
dropout
[0.2, 0.3]
learning_rate kemel_siize
10.0001,000005,000001, 0.000, 0.05]
DLinear
25
learning_ran rate
[0.0001, 0.001]
e_ layers
[1,2,3]
Autoformer
d layers
[1,2,3]
factor
[1,3]
C.3 Evaluation Metrics
We employ the Normalized Mean Absolute Error (NMAE) and Normalized Root Mean Squared Error (NRMSE) as our evaluation metrics because they provide relative measure of error that is independent of the data scale. It's important to note that some original papers reported metrics prior to re-scaling forecasts to their original magnitude, which can affect metric calculations. In this study, we have carefully ensured that our reproduced results are consistent with those reported in the original studies and have applied these unified metrics enable a comprehensive and fair comparison.
Normalized Mean Absolute Error (NMAE) The Normalized Mean Absolute Error (NMAE) is normalized version of the MAE, which is dimensionless and facilitates the comparability of the error magnitude across different datasets or scales. The mathematical representation NMAE given by:
ΣK E
NMAE
Σ DE=
Apache-2.0 2.0 license.
16

---

Normalized Root Mean Squared Error (NRMSE) The Normalized Root Mean Squared Error (NRMSE) normalized version of the Root Mean Squared Error (RMSE), which quantifies the average squared magnitude of the error between forecasts and observations, normalized by the ex- pectation of the observed values. can be formally written as:
K KXT -= Ei= (Fi 22,e2
NRMSE
KxT 1= 21=1 [0ia]
D Additional Experimental Results
D.1 Comparing ElaSTST with More Neural Architectures
Table 5: Results (meanga) on long-term forecasting scenarios with the best in bold and the second underlined. Each result containing three independent runs with different seeds. During the training phase, ElasTST utilizes an loss reweighting strategy where a single trained model is applied across all inference horizons, the Hmax is set to 720. Other baseline models undergo horizon-spechio training and tuning.
pred Ion
FlasTST NMAE
iTranformmr NMAE NRMSE|
PatchTST NMAB NRMSE
DI DLinear NMAB
MSMixer NMAE
Autoformer NMME
Trunsformer Enc
NRMSB|
NRMSE
NRMSE
NRMSE
NMAE
NRMSE
96
0.273
0.488
0.271
0.568
0.272
0.565
0.282
0.573
0.369
0.620
0.388
0.711
0.320
0.561
192
0.289
0.520
0.301
0.614
0.295
0.602
0.309
0.617
0.393
0.673
0.442
0.820
0.348
0.632
336
0.314
0.575
0.333
0.668
0.323
0.645
0.338
0.654
0.426
0.727
0.429
0.771
0.373
0.679
720
0.346
0.645
0.376
0.741
0.353
0.700
0.387
0.737
0.464
0.788
0.440
0.793
0.397
0.712
96
0.150
0.227
0.137
0.227
0.132
0.220
0.138
0.226
0.560
0.668
0.158
0.254
0.156
0.233
ETTm2
192
0.174
0.264
0.161
0.266
0.157
0.259
0.163
0.264
0.556
0.605
0.175
0.283
0.183
0.275
336
0.191
0.289
0.180
0.293
0.176
0.286
0.188
0 0.291
0.553
0.661
0.191
0.307
0,201
0.303
720
0.211
0.318
0.330
0.205
0.324
0.219
0.327
0.548
0.661
0.338
0.336
96
0.342
0.619
0.321
0.626
0.328
0.6 0.640
0.352
0. 0.668
0.3/13
0. 0.628
0.367
0.656
0.389
0.693
192
0.564
0,681
0.359
0.690
0.359
0.705
0.393
0.745
0.399
0.706
0.392
0.706
0.414
0.722
336
0.371
0.666
0.388
0.723
0.384
0.740
0.419
0.778
0.449
0.749
0.398
0.711
0.459
0.799
720
0.376
0.679
0.408
0.735
0.397
0.738
0.502
0.860
0.499
0.779
0.433
0.739
0.473
0.806
96
0.158
0.239
0.177
0.279
0.177
0.281
0.211
0.320
0.199
0.283
0.203
0.317
0.210
0.312
Em7h2
192
0.170
0.259
0.203
0.314
0.201
0.314
0.238
0.353
0.228
0.322
0.226
0.346
0.228
0.339
136
0.188
0.282
0.243
0.372
0.240
0.366
0.284
0.407
0.253
0.354
0.264
0.398
0.244
0.361
720
0.215
0.319
0.264
0.386
0.252
0.371
0.307
0.426
0.288
0.390
0.287
0.416
0.262
0.391
Electricity
96
0.085
0.777
0.098
0.772
0,086
0.816
0.090
0.863
0.101
0.862
0.140
0.977
0.107
0.924
192
0.093
0.933
0.106
0.916
0.092
0.942
0.095
0.974
0.5
0.95
0.6
1.017
0. 109
0.957
336
0.101
1.063
0.115
0.985
0.100
1.035
0.104
1.066
0.109
1.022
0.147
1.080
1.057
720
0.117
1.289
0,133
1.110
0.116
1.213
0,122
1.259
0.120
1.194
0.159
1.283
0.123
1.118
96
0.195
0.461
0,246
0.511
0.248
0.527
0.356
0.645
0.313
0.590
0.293
0.560
0.319
0.576
Thaffic
192
0.193
0.459
0.259
0.343
0.245
0.328
0.346
0.628
0.295
0.555
0.318
0.591
0.315
0.569
336
0.199
0.488
0,283
0.571
0.257
0.550
0.350
0.631
0.316
0.575
0.332
0.630
0.314
0.567
720
0.218
0.497
0.275
0.563
0.266
0.559
0.365
0.659
0.332
0.598
0.341
0.611
0.334
0.587
Weaterr
96
0.086
0.287
0.089
0.295
0.087
0.294
0.112
0.316
0.118
0.309
0.239
0.614
0. 0.106
0.309
192
0,092
0.312
0.093
0.299
0.090
0.299
0.122
0.331
0.123
0.325
0.213
0.533
0.327
336
0.091
0.307
0.096
0.297
0.092
0.297
0,130
0.340
0.126
0.328
0.176
0.413
0.114
0.330
720
0.093
0.308
0.099
0.298
0.094
0.298
0.144
0.358
0.129
0.330
0.170
0.434
0.113
0.324
Exchange
96
0.026
0.039
0.025
0.039
0.023
0.036
0.024
0.037
0.040
0.055
0.032
0.049
D.030
0.045
192
0.033
0.050
0.036
0.058
0.034
0.054
0.035
0.055
0.052
0.072
0.041
0.065
0.037
0.055
336
0.041
0.062
0.048
0.072
0.048
0.076
0.048
0.072
0.067
0.092
0.056
0.091
0.049
0.074
720
0.059
0.089
0.076
0.111
0.072
0.106
0.075
0.118
0.091
0.128
0.112
0.161
0.120
D.2 Performance Gains Across the Forecasting Horizon
In Figure M we compare the performance gains of each model design at different points within the forecasting window. The benefits of structured masks are consistent across the entire horizon, while the advantages of tunable RoPE and multi-patch designs become more prominent when handling un- seen horizons. Notably, the tunable RoPE plays a critical role in enhancing the models extrapolation capability.
17

---

Model BBSTST.NG
Modill Maui Rl
2.5
Tunul RoPE. BlsiTT
2.0
120
200
520
302
500
750
HDc.
Figure 7: Performance gain of each model design across the forecasting horizon. A relative perfor- mance greater than indicates a gain, while values less than indicate drop. Results are averaged across all datasets, with the vertical red dashed line marking the training horizon.
Tra ning Horizon
96
192
336
720
ETT
Exchange
Weather
Electricity
Traffic
0.34
3:30
0.00
0.2
0.30
0.20
0.26
0.64
0.25
8.9
0.00
0.20
192
1024 Forecasting Horizon
720 Forecasting Horizon
192
1024
Forecasting Horizon
Forecasting Horizon
Forecasting Horizon
Figure 8: Impact of forecasting horizon selection during the training phase.
D.3 More Analysis of the Impact of Training Horizon
The scalable architecture of ElasTST allows to treat the training horizon as hyperparameter. This adaptability prompts us to evaluate performance across various training and inference horizons (see Figure H), yielding several interesting insights.
Model performance deteriorates as the forecasting horizon increases, particularly in models trained on shorter horizons, as seen in the ETT and Electricity datasets. This pattern suggests the importance of training models extended horizons capture adequate contextual informa- tion.
Tuning models specifically for a given horizon does not guarantee improved performance on that horizon, noted in the Weather dataset. This indicates that optimal model settings depend sig- nificantly on dataset-specific characteristics, and horizon-specific tuning may not be a reliable strategy.
The longest training horizons do not always produce the best forecasting results. In the Exchange dataset, for example, the longest horizon yielded poorer results compared to a shorter training horizon of 96. This points to the potential risks of overfitting or forecast instability when training with long-term series only.
These observations underscore the importance of tailoring training horizons to the unique character- istics of each dataset and underscore the benefits of an architecture designed for elastic forecasting. Furthermore, they suggest the potential advantages implementing a mixed-horizon training strat- egy, which leverages multiple horizons to produce more resilient forecasts.
D.4 More Analysis of the Impact of Tunable Rotary Position Embedding
Beyond its scalable architecture, the position embedding plays a crucial role in enhancing the elas- ticity of EI.sTST. We analyze the Tunable RoPE in ElasTST by examining the effects of the initial- ization of period coefficients, specifically Pmin and Pmxx, and the benefits parameter optimization during the training process.
Experimental results, presented in Figure E. indicate that using settings similar to the commonly- used one in NLP, with Pmin and Pmax 10000, does not fully exploit its potential in time series
18

---

Training Horizon:
Training Horizon: 192
Training Horizon: 336
Training Horizon: 720
0:28
period
0
period
26
min period
NMAE
0.25
0.25
0.1
0.24
0.1
0.24
0.1
0.23
min period
0.23
0.22
0.01
0.20
0.1
0.20
0.20
0.20
0.18
0.01
0.18
0.18
96
192
336
720
96
192
336
720 1024
96
192
336
720 1024
96
192
336
720 1024
Forecasting Horizon
Forecasting Horizon
Forecasting Horizon
Forecasting Horizon
(a)
The effect of selecting Pmin. with Pmx fixed at 10,000 and @ is untunable during training.
Training Horizon: 96
Training Horizon: 192
Training Horizon: 336
Training Horizon: 720
0.28
0.26
max period
max period
0.26
max period
NMAE
period
100
0.24
100
0.24
100
0.22
100
10000
1000
0.22
1000
10000
0.22
1000
10000
1000
0.20
10
10000
0.18
0:10
0.18
96
192
336
720
96
192
336
720 1024
192 336 720 1024
192
336
720
1024
Forecasting Horizon
Forecasting Horizon
Forecasting Horizon
Forecasting Horizon
(b) The effect of selecting Pmax. with Pmin fixed at 1 and 0 is untunable during training.
Training Horizon: 96
Training Horizon: 192
Training Horizon: 336
Training Horizon: 720
0.28
0.28
tunab
0.26
tunable
0.26
tunable
NMAE
0.24
0.25
talse
0.24
faise
faise
0.22
tunable
0.23
0.22
0.20
0.20
0.20
0.18
false
0.18
0.18
336
336 720 Forecasting Horizon
192
1024
720 1024
Forecasting Horizon
Forecasting Horizon
Forecasting Horizon
(c) The effect tuning with Pain set at and Pmux set at 10,000.
Figure 9: Ablation study for designs in position embedding. Here we analyze the tumable RoPE in ElasTST by examining the effects of the initialization of period coefficients, specifically Pmin and Pmax. and the benefits of parameter optimization during the training process. Results are averaged across all datasets. A vertical red dashed line distinguishes between seen horizons and unseen hori- zons.
forecasting. This discrepancy stems from fundamental differences between the data types: in text, discrete tokens are the smallest units, requiring attention over longer contexts, while time series data, particularly when patched, may benefit from focusing on shorter, more recent tokens.
Furthermore, our findings reveal that tuning parameters in RoPE during training significantly im- proves forecasting accuracy, particularly over varying and extended horizons. When the training horizon is set to 96, a tunable feature shows minimal impact, suggesting that a short-seen horizon does facilitate learning effective period coefficients. However, as the training horizon extends, the advantages of tunable theta become more pronounced, especially for unseen horizons.
These results emphasize the importance of customizing RoPE's period coefficients settings and uti- lizing tunable RoPE to enable flexible and accurate forecasting in time series analysis. Appendix E offers visualizations demonstrating how different initial ranges impact the frequency components, along with a detailed analysis of the distribution of RoPE periods optimized for each dataset.
D.5
The Impact of Patch Size Selection
These experiments highlight the critical impact of patch size selection, particularly in varied-horizon forecasting scenarios. As demonstrated in Figure IIA various combinations of training and forecast- ing horizons exhibit distinct preferences for patch sizes. For instance, when the training forecasting horizon 720, during the inference stage, longer forecasting horizons prefer larger patch sizes. Conversely, on shorter training horizons, such as 96 and 192, choosing large patch sizes for longer horizons can lead to performance collapse. This difference underscores the complexity and necessity of optimal patch size selection in achieving effective elastic forecasting.
19

---

16
96
192
336 Forecasting Hortzons
720
1024
32 64
(a) Training Horizon: 96
Patch
0.25
1.6
32
0.20
8.16
96
192
336 Forecasting Horizons
720
1024
8,16_32
(b) Training Horizon: 192
0..00
Patch
0.25
16
0.15
8,16.32
8.16, 32.52...4
(c) Training Horizon: 336
0.30
Patch
32
0.20
64
8,16
96
192
336
720
1024
16
Forerasting Hortzons
(d) Training Horizon: 720
Figure 10: The performance difference patch size selection.
Moreover, Figure clearly show that multi- salli--pach configurations typically surpass single patch sizes across most forecasting horizons. The configuration lowest NMAE values, striking an optimal balance between capturing short-term dynamies and long- term trends. However, introducing larger patches (p = {8, 16, 32, 64)) does not always enhance performance and can sometimes increase the NMAE, indicating that overly complex configurations may not yield additional benefits and could be counterproductive.
{8, 16, 32} consistently offers the
These findings emphasize the advantages of employing multi-patch configurations improve the accuracy of varied- horizon forecasting. They also highlights the importance of carefully selecting patch size combinations optimize performance and minimize computational expenses.
E
Visualization of Periods in RoPE
E.1
Initialized Periods
To provide intuitive visualization of the initial distribution of periodicity coefficients, present this in Figure IT].
E.2 Tuned periodicity coefficients
In Figure M
present the distribution of tuned periodicity cocfficients for each dataset.
20

---

1000
10000
750
7500
500
5000
250
2500
0
5
10
10
(a)
[1. 1000]
(b) P E [1, 10000] (Same as RoPE paper).
sini2 00.00
sint2m2.774
4ini2 2.7
5in12m113
9in.2/y19.300
sin/2/551.8 8x0
sin(2m5
sin(2m1193.1x
sini2m/138.0
sin/2m//9.7xx
sint2 3/72.8
sin/2m22682.24
0.0
0.5
sin/2m/10000.on
0.0
0.00
(c) Frequency distribution when s E [1, 1000). (d) Frequency distribution when P [1, 10000] (Same as RoPE paper).
Figure 11: The initialized periodicity coefficients RoPE. Here we set the dimension 16.
Model Efficiency
Overall Computational Efficiency Comparison
Table 6: Computation memory.
The batch size 1 and the prediction horizon set to 96.
Metric
Dataset
DLinear
PatchTST
Autoformer
i'Transorrmer
ElasTST
ETTml
0.075
2. 2.145
23.273
3,366
2.240
Electricity
0.076
2.146
29.701
3,366
2.240
NPARAMS (MB)
Traffic
0.078
2.149
40.783
3.366
2.240
Weather
0.075
2.145
23.560
3.366
2.240
Exchange
0.075
0.135
23.286
3.366
2.240
ETTml
0.002
0.009
0.227
0,026
0,024
Electricity
0,060
0.068
0.246
0.037
0.112
Max GPU Mem. (GB)
Traffic
0.161
0.168
0.279
0.157
0.263
Weather
0.004
0.012
0.228
0.027
0.028
Exchange
0.002
0.002
0.227
0.026
0.024
In Table E, compare the memory usage of ElasTST with other baseline models. The comparison reveals that ElasTST does not require more computational resources than other Transformer-bar models.
F.2 Memory Usage Introduced by TRoPE
In Table Z. we compare the memory usage of ElasTST with different position encodings. The results show that RoPE adds almost negligible number of parameters compared to vanilla absolute position encoding. While applying rotation matrix does introduce slightly higher memory usage, remains within acceptable limits.
21

---

Intialiiation
ETTh1
ETT11
1000
1000
40000
750
750
30000
500
Q
20000
250
250
10000
10
ETTml
ETTm2
Electricity
3000
400
300
2000
200
100
20
30
10
20
30
20
30
Weather
Exchange
Traffic
2000
2000
1500
1500
400
1000
500
200
500
10
20
30
10
20
30
10
20
30
The batch size and the Figure 12: Tuned periodicity coefficients over different datasets.
Table 7: Memory consumption of EST using different position encoding approaches. The batch size is 1 and the forecasting horizon 1024.
Metric
Abs PE
RoPE w/o Tunable
Tunable RoPE
Max GPU Mem. (GB)
0.0480
0.0539 5.1225
0.0539 5.1228
NPARAMS (MB)
5.1225
F.3 Memory Usage Introduced by Multi-scale Patch Assembly
Table Memory consumption under different patch size settings. forecasting horizon is 1024.
Metric
p=l
p-8
p=16
p=32
p=(1,8,16,322)
(=,6,16,32)
Max GPU Mem. (GB)
0.6747
0.0536
0.0415
0.0360 5.0738
0.6751
0.0539
NPARAMS (MB)
5,0130
5.0267
5.0424
5.1257
5.1228
While model uses shared Transformer backbone process all patch sizes, this be done either in parallel or sequentially, depending on whether shorter computation times or lower memory practice, we chose the sequential approach, where each patch size pro-
usage is prioritized. cessed individually, and the total forecast is assembled afterward. This approach ensures that the memory bottleneck depends on the smallest patch size used.
As indicated Table & memory usage is primarily influenced by the smallest patch size, not by the number of patch sizes employed. The additional parameters introduced by using multiple patch sizes are almost negligible. For example, the multi-patch setting 8, 1,322 used in the paper requires the same maximum memory using single patch size of 8. Under resource constraints, the minimum patch size can be adjusted to balance model performance and memory usage.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction are well-aligned with the paper's contributions and scope as detailed in Sections B and EL where we elaborate on the theoret- ical advancements and experimental validations.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of our work are discussed in Section S
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
answer: [NA]
Justification: There is no theoretical result in this paper.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All necessary details required for reproducing the main experimental results are comprehensively disclosed in Section M2 and Appendix M
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code available at hitttper//ggcaah.ccooccccrsessts
elastst.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All training and testing details, including data splits, hyperparameters, their selection rationale, and the type of optimizer used, fully specified in Section en> and Appendix
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the experiments presented the main result tables, we conducted each experimental setup three times using random seeds set 0, 1, and The mean and variance these three trials are reported in the tables, providing a measure the experiments' statistical robustness.
8. Experiments Compute Resources
23

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory. time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed specifications of the compute resources provided in Ap- pendix C and Appendix
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research presented this paper adheres fully to the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No negative societal impact of the work performed.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The methodologies developed in this study do not involve such risk.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
All relevant assets used in our research are duly credited. We have explicitly listed the licenses and terms of use for each asset, including URLs the original sources, in Appendix C. and Appendix ee
Justification:
13. New Assets
Question: Are assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: No new asset have been released so far.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human
Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.


Reviews:
[{"review_id": "Tv3NsKFno7", "reviewer": "Reviewer_qEB8", "summary": "The paper introduces ElasTST, a novel approach designed to enhance robustness in forecasting across various horizons. The proposed architecture dynamically adapts to different forecasting horizons, addressing a significant challenge in the literature. Previous methods typically relied on recursive approaches or utilized masks that failed to maintain horizon invariability. ElasTST aims to achieve high forecasting accuracy while ensuring adaptability across different horizons by employing a transformer encoder as its backbone. The approach incorporates structured attention masks through patching and rotary position embeddings (RoPE) to encode relative positional information for improved forecasting.", "questions": "1. What is the computational complexity added by the Rotary Position Embeddings (RoPE)?\n2. In some cases, the results are very similar to PatchTST, such as with the ETTm2 dataset. Is there an intuition or explanation regarding the types of data that explain these similarities and differences in performance?\n3. It would be interesting to compare the performance gains for each point in the forecasting window. Are the gains through the whole window or in the earliest/latest points?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper effectively tackles a crucial issue in the forecasting literature—enhancing deep learning architectures to produce forecasts for any horizon while maintaining horizon invariability and reducing the concatenation of errors.\n2. The approach leverages well-established research, such as patching from the PatchTST paper, and also rotary position embeddings, to handle long-horizon forecasting tasks.\n3. The paper considers recent advancements in foundation models for time series forecasting.", "weaknesses": "1. The literature review on foundational models is lacking. Notable models such as TimeGPT-1 [1], Chronos [2], TinyTimeMixers [3], and Moment [4] are not discussed.\n2. A traditional encoder transformer is not included as a baseline, which would help demonstrate the improvements offered by the proposed method. Additionally, NLP-based architectures such as NHITS [5] and TsMixer [6] are absent from the comparisons.\n3. The paper does not adequately address the computational scalability of ElasTST when applied to very large datasets or extremely long time series.\n4. The authors modified the traditional metrics used in long-horizon literature (MSE and MAE) to scaled metrics. They argue this adjustment reports fairer results, but it overlooks the fact that these datasets are typically already scaled, making additional scaling in the metric unnecessary.\n5. The paper lacks a detailed discussion on potential failure cases or scenarios where ElasTST might underperform, which would provide a more balanced view of its applicability.\n\n[1] https://arxiv.org/abs/2310.03589\n[2] https://arxiv.org/abs/2403.07815v1\n[3] https://arxiv.org/abs/2401.03955\n[4] https://arxiv.org/abs/2402.03885\n[5] https://arxiv.org/abs/2201.12886\n[6] https://arxiv.org/abs/2303.06053", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your in-depth reviews, insightful questions, and constructive suggestions. We hope the following responses can help to address all your questions and concerns.\n\n## Response to Weakness 1\n\nWe will expand our literature review to provide a more comprehensive discussion of recent time series foundational models. We briefly summarize these models in the table below. Most of these efforts employ standard architecture designs, position encodings, and patching approaches. However, they often lack in-depth investigation into the challenges of generating robust forecasts across varied horizons. Our work directly addresses this gap by enhancing model design for varied-horizon robustness. Detailed discussions will be incorporated into the revision.\n\n| Model | Backbone | Dec. Scheme. | Pos. Emb. | Token. |\n| --- | --- | --- | --- | --- |\n| TimeGPT-1 | Enc-Dec Transformer | AR | Abs PE | - |\n| Chronos | Enc-Dec Transformer | AR | Simplified relative PE | Quantization |\n| TimesFM | Decoder-only Transformer | AR | Abs PE | Patching |\n| DAM | Transformer Encoder | NAR | Abs PE | ToME |\n| Tiny Time Mixers | TSMixer | NAR | - | Patching |\n| MOIRAI | Transformer Encoder | NAR | RoPE | Patching |\n| MOMENT | Transformer Encoder | NAR | Learnable relative PE | Patching |\n\n## Response to Weakness 2\n\nWe acknowledge the importance of including a more comprehensive baseline. Due to the word limitation, we have provided the averaged performance of these baselines across all datasets in the table below for reference. The revised manuscript will include detailed results and analysis to provide a more thorough evaluation.\n\n| Infer Hor. | ElasTST (NMAE) | NHiTS (NMAE) | TSMixer (NMAE) | TransformerEnc (NMAE) |\n| --- | --- | --- | --- | --- |\n| 96 | 0.16775 | 0.205125 | 0.25375 | 0.2055 |\n| 192 | 0.177875 | 0.220375 | 0.2645 | 0.219375 |\n| 336 | 0.188125 | 0.247875 | 0.283 | 0.23375 |\n| 720 | 0.21375 | 0.29075 | 0.30675 | 0.2505 |\n\n## Response to Weakness 3\n\nThank you for raising this important point. ElasTST introduces minimal additional computation compared to other non-autoregressive Transformer-based models, as shown in Table 1 in the global response. However, we acknowledge the need for further scalability improvements. To address this, we are exploring solutions such as dimensionality reduction, quantization, and memory-efficient attention mechanisms to reduce computational overhead for large datasets.\n\n## Response to Weakness 4\n\nThank you for your observation regarding the evaluation metrics. As noted in Appendix B.3, some existing models apply z-score standardization during data preprocessing and compute metrics before reversing the standardization, which can complicate direct comparisons when different scaler are used. We chose NMAE (Normalized Deviation in the M4 and M5 literature) as our primary evaluation metric because it is scale-insensitive and widely accepted in recent studies [1].\n\nAdditionally, we have also calculated MAE and MSE both before and after de-standardization. These results will be included in the Appendix in future revisions.\n\n[1] Oreshkin, B. N., Carpov, D., Chapados, N., & Bengio, Y. (2020). N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. *ICLR*.\n\n## Response to Weakness 5\n\nWhile our method improves varied-horizon forecasting, as shown in Figure 3, the training horizon is a tunable hyper-parameter, and the optimal value varies across datasets. An inappropriate choice, like 96, can significantly degrade performance. Besides, the selected horizon of 336 used in the main results is not always optimal, suggesting that tailoring the training horizon for each dataset can further enhance the model’s effectiveness.\n\n## Response to Question 1\n\nIn Table 2 of the global response, we compare the memory usage of ElasTST with different positional encodings. The results show that RoPE adds a negligible number of parameters compared to vanilla absolute position encoding. Although applying the rotation matrix introduces slightly higher memory usage, it remains acceptable.\n\n## Response to Question 2\n\nThank you for your insightful observation. To understand the performance similarities, we analyzed the trend and seasonality of these datasets [1]. We find that the similarities between ElasTST and PatchTST could be attributed to varying levels of seasonality. Specifically, ETTm1 and ETTm2 have minor seasonality, accompanying similar performance, while ETTh1 and ETTh2 exhibit stronger seasonal patterns, where ElasTST shows clear advantages.\n\nWithout the additional designs for robust varied-horizon forecasting, ElasTST essentially functions as a variable-length version of PatchTST. Therefore, the differences in performance could reflect the datasets’ sensitivity to our specific architecture designs.\n\n|  | ETTh1 | ETTh2 | ETTm1 | ETTm2 |\n| --- | --- | --- | --- | --- |\n| Seasonality | 0.4772 | 0.3608 | 0.0105 | 0.0612 |\n\n[1] Wang, X., Smith, K. A., & Hyndman, R. J. (2006). Characteristic-based clustering for time series data. *Data Mining and Knowledge Discovery*, *13*(3), 335–364.\n\n## Response to Question 3\n\nIn Figure 2 of the global response, we compare the performance gains of each model design across different points within the forecasting window. The benefits of structured attention masks are consistent throughout the entire horizon, while the advantages of tunable RoPE and multi-scale patch assembly become more pronounced when dealing with unseen horizons. Notably, tunable RoPE significantly enhances the model’s extrapolation capability. Due to the word limitation, we will include a more detailed analysis in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81JR6EgJhQ", "reviewer": "Reviewer_eA7o", "summary": "This paper introduces ElasTST, which aimed at addressing the challenge of varied-horizon time-series forecasting. ElasTST integrates a non-autoregressive architecture with placeholders for forecasting horizons, structured self-attention masks, and a tunable rotary position embedding. Furthermore, it employs a multi-scale patch design to incorporate both fine-grained and coarse-grained temporal information, enhancing the model’s adaptability across different forecasting horizons. Experiments in the paper demonstrate the model's performance in providing robust predictions when extrapolating beyond the trained horizons.", "questions": "- Given the significant increase in computational cost from longer token sequences, would independently inputting each patch size into the transformer layers, instead of all at once, reduce memory usage while maintaining model performance?\n\n- With multiple patch sizes used in the study, how are the RoPE applied? Are they applied independently to tokens from each patch size, or collectively to all tokens from different patch sizes?\n\n- Ablation studies indicate that incorporating smaller patch sizes seems to benefit model performance substantially. If my understanding is correct, when the patch size is 1 (excluding tunable RoPE), the structure is similar to the decoder part of models like Informer and Autoformer. Would using a patch size of 1 worsen the model performance? Would adding \"1_\" to the existing best patch size setting \"8_16_32\" improve performance? Using patch size 1 turns the model into a channel-independent Informer-decoder-like model, which significantly increases computational costs, hence it you don't have to consider this on large datasets.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper attempts to address an important issue in time series applications by enabling the model to adapt to varied forecasting horizons during the inference stage, which can make the trained forecasting model more flexible in usage.\n\n- Introducing multi-scale patching operations to construct input tokens for the Transformer can effectively handle local patterns of different granularities.\n\n- The paper introduces a tunable Rotary Position Embedding (RoPE) module tailored for time series tasks, showing potential for broader application across various forecasting methods.", "weaknesses": "- The newly proposed tunable RoPE module lacks comparative analysis with previous methods. A comparison with methods such as no positional embedding, vanilla positional encoding, trainable positional embedding matrices, and original rotary positional embeddings, would clarify the benefits of different approaches. Since tunable RoPE appears easily transferable, demonstrating its performance enhancement by applying it to previous models like PatchTST and Autoformer would provide more intuition.\n\n- Using longer sequence length significantly increases the computational cost in Transformers. Comparing with baselines like PatchTST, the paper uses multiple patch sizes as input processing method, which significantly increases computational costs. Further analysis like how much adding a patch size can increase computational costs or what patch size combination are more reasonable to choose under resource constraints would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your in-depth reviews, insightful questions, and constructive suggestions. We hope the following responses can help to address all your questions and concerns.\n\n## Response to Weakness 1\n\n> A comparison with methods such as no positional embedding, vanilla positional encoding, trainable positional embedding matrices, and original rotary positional embeddings, would clarify the benefits of different approaches. Since tunable RoPE appears easily transferable, demonstrating its performance enhancement by applying it to previous models like PatchTST and Autoformer would provide more intuition.\n> \n\nThank you for highlighting this gap in our experiments. We have updated Figure 3 in the global response to include the performance of ElasTST with various positional embeddings. The results demonstrate that Tunable RoPE significantly enhances the model’s extrapolation capabilities. While other positional embeddings perform well on seen horizons, they struggle when the forecasting horizon extends beyond the training range. Although the original RoPE performs well in NLP tasks, it falls short in time series forecasting. Dynamically tuning RoPE parameters based on the dataset’s periodic patterns proves highly beneficial, especially for unseen horizons, with the benefits becoming more pronounced as the inference horizon lengthens.\n\nWe appreciate your suggestion to apply Tunable RoPE to previous models like PatchTST and Autoformer. However, PatchTST requires per-horizon training and deployment, making it difficult to showcase the benefits of Tunable RoPE since it cannot handle longer inference horizons once trained for a specific horizon. Essentially, our architecture—after removing the tunable RoPE and multi-patch designs—can be viewed as a version of PatchTST that adapts to varied horizons. As for Autoformer, it replaces the self-attention mechanism with an auto-correlation mechanism that inherently captures sequential information, making positional embeddings unnecessary.\n\nYour feedback has been invaluable in refining our work, and we will update the manuscript to include these discussions.\n\n## Response to Weakness 2\n\n> Further analysis like how much adding a patch size can increase computational costs or what patch size combination are more reasonable to choose under resource constraints would be beneficial.\n> \n\nElasTST uses a shared Transformer backbone to process all patch sizes, which can be done either in parallel or sequentially, depending on whether shorter computation times or lower memory usage are prioritized. We chose sequential processing, where each patch size is handled individually, keeping the memory consumption comparable to baselines like PatchTST (see Table 1 in global response).\n\nIn this approach, memory usage is primarily influenced by the smallest patch size, not the number of patch sizes used (see Table 3 in global response). For example, the multi-patch setting in the paper (sizes {8,16,32}) requires the same maximum memory as using a single patch size of 8. Under resource constraints, adjusting the minimum patch size helps balance model performance and memory usage. We provide further analysis on the performance of different patch size combinations in the response to Question 3.\n\n## Response to Question 1\n\n> Given the significant increase in computational cost from longer token sequences, would independently inputting each patch size into the transformer layers, instead of all at once, reduce memory usage while maintaining model performance?\n> \n\nYes, as mentioned in our response to Weakness 2, independently inputting each patch size into the transformer layers can indeed reduce memory usage without degrading model performance.\n\n## Response to Question 2\n\n> With multiple patch sizes used in the study, how are the RoPE applied? Are they applied independently to tokens from each patch size, or collectively to all tokens from different patch sizes?\n> \n\nRoPE is applied independently to each patch size, and tokens from different patch sizes do not interact within the Transformer layer.\n\n## Response to Question 3\n\n> Ablation studies indicate that incorporating smaller patch sizes seems to benefit model performance substantially. Would using a patch size of 1 worsen the model performance? Would adding \"1_\" to the existing best patch size setting \"8_16_32\" improve performance? \n> \n\nThank you for your questions. We would like to clarify that smaller patch sizes do not always improve model performance. In Appendix C.3 of the submitted manuscript, we discuss how different patch size combinations impact performance across various training horizons. For instance, with a training horizon of 720, a smaller patch size can sometimes result in poorer performance.\n\nIn Figure 4 of the global response, we included the case with a patch size of 1 for a more comprehensive comparison. The results indicate that using a patch size of 1, or adding it to a multi-patch combination, does not consistently lead to better or worse performance. The optimal patch size varies by dataset and forecasting horizon. Overall, the 8_16_32 combination provides a good balance between performance and memory consumption for varied-horizon forecasting.\n\nWe sincerely appreciate your suggestions and feedback. We will revise the paper to clarify the workings of Tunable RoPE and multi-patch assembly, and we will include a more comprehensive comparative and computational analysis to provide a holistic view of the proposed model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FJ5nJPGl6I", "reviewer": "Reviewer_i9JA", "summary": "The paper introduces a masked encoder based transformer model for time series forecasting. It performs masking in the observation space, replacing the prediction horizon with 0s. It further applies an attention mask such that tokens only attend to the context, and do not attend to the masked tokens. The paper also uses a tunable version of rotary positional embeddings, and ensembles predictions over various patch sizes.", "questions": "None", "limitations": "Authors have provided limitations of their work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a method which can adapt to multiple forecast horizons, which is a weakness of many recent deep forecasting models. The paper presents several empirical analyses to verify the claims made, and also presents a detailed look at related work.", "weaknesses": "Overall, the proposed approach is very similar to Moirai, with several minor changes:\n\ni) Perform the masking operation in observation space\n\nii) Add an attention mask\n \niii) Tunable rotary embeddings\n \niv) Ensembling results over multiple patch sizes\n\nHowever, it seems that even more features which Moirai had were removed, making it a weaker proposition. Such features include probabilistic predictions, multivariate capabilities, and any context length. Furthermore, few experiments are done to show the benefits of the changes, such as (i), (ii), and (iv). \n\nOthers:\n1. The statement, \"we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages\" (L238-L239) is a little eyebrow raising, in the sense that it is stating the obvious. I believe most researchers expect in-distribution predictions to outperform zero-shot predictions. Such statements should be rephrased.\n2. Effects of tuning rotary embeddings in figure 4 seem very marginal, error margins should be provided for such cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your in-depth reviews and constructive suggestions. We appreciate the opportunity to clarify the distinctions between our approach and MOIRAI, as well as highlight the key contributions of our paper. \n\nWhile there are similarities between our model and MOIRAI due to shared influences from image/video generation Transformer frameworks such as DiT [1] and SORA [2], there are significant differences in our objectives and approaches.\n\nMOIRAI focuses on **universal forecasting, emphasizing pre-training and transferability across datasets**. It introduces any-variate attention and probabilistic predictions to enhance these capabilities. However, these features do not directly address the challenge of robust forecasting across arbitrary horizons. Moreover, while MOIRAI incorporates techniques from recent LLM architectures, such as vanilla RoPE, RMSNorm, and SwiGLU, these were applied without specific adaptations for time series. As shown in the global response (Figure 3), using vanilla RoPE directly in varied-horizon forecasting still presents significant challenges.\n\nOur work **specifically targets robust varied-horizon forecasting** with three key designs: \n\n(i) structured self-attention masks for consistent outputs across different forecasting horizons, \n\n(ii) tunable rotary position embeddings for adapting to varying periodic patterns, and \n\n(iii) multi-scale patch assembly to balance patch settings preferred by different horizons.\n\nThe full ablation study in the global response clearly demonstrates that **each of these designs plays a crucial role in enabling robust varied-horizon forecasting**:\n\n(i) Structured attention masks ensure reliable inference on horizons that differ from the training horizon (Figures 1 and 2).\n\n(ii) Tunable RoPE is crucial for enhancing the model’s extrapolation capability (Figure 3).\n\n(iii) Multi-scale patch assembly reduces sensitivity to patch size hyper-parameters across different horizons (Figure 4).\n\nWhile our model also supports any context length, this feature was not highlighted in the manuscript as it is less central to our key technical contributions.\n\nIn summary, ElasTST and MOIRAI address different challenges in time series forecasting, each offering solutions to distinct problems. In the future, our methods could potentially be integrated within a broader framework to enhance both generality and robustness in time series forecasting.\n\n----------\n**Improve the presentation of this paper**\n\n> 1. The statement, \"we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages\" (L238-L239) is a little eyebrow raising, in the sense that it is stating the obvious. I believe most researchers expect in-distribution predictions to outperform zero-shot predictions. Such statements should be rephrased.\n> 2. Effects of tuning rotary embeddings in figure 4 seem very marginal, error margins should be provided for such cases.\n\nThank you for your valuable suggestions.\n\nWe agree that some statements could be better phrased. Specifically, regarding the statement in L238-L239, our intention was to emphasize that while the zero-shot capability of foundation models is impressive, additional dataset-specific tuning may still be necessary for challenging datasets to achieve optimal forecasting performance. We will revise these statements in the manuscript to convey our findings more clearly and accurately.\n\nIn the experimental section, we will incorporate key results from the global response into the main text and add error margins to Figure 4 to improve result representation.\n\nWe appreciate your feedback and will refine the paper accordingly to ensure a clearer and more thorough presentation in the revised manuscript.\n\n[1] Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. *ICCV.*\n\n[2] Tim Brooks et. al., (2024). Video generation models as world simulators.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their insightful comments.

In response to your suggestions, we have conducted additional experiments, which are detailed in the attached PDF. The ablation study further confirms that each proposed design element is crucial for achieving robust varied-horizon forecasting.

Specifically, the supplementary experiments include:

1. The benefits of each proposed design (Figure 1).
2. Performance gains across the forecasting horizon for each model design (Figure 2).
3. A detailed comparison of tunable RoPE with other positional embedding methods (Figure 3).
4. The performance of different multi-patch size combinations, including the case with a patch size of 1 (Figure 4).

To address concerns about computational efficiency, we have provided analysis in Tables 1, 2, and 3, where we compare our model to other baselines and calculate the additional memory consumption introduced by tunable RoPE and multi-scale patch assembly. Our analysis shows that ElasTST adds minimal computational overhead compared to other non-autoregressive Transformer models, and the inclusion of tunable RoPE and multi-patch design has a negligible impact on overall efficiency.

We will include these experiments and analyses in the revised paper, and we welcome any further questions or suggestions.

Table 1. Memory consumption of ElasTST and baselines. The batch size is 1 and the horizon is set to 1024.

|  | TransformerEnc | Autoformer | PatchTST | ElasTST |
| --- | --- | --- | --- | --- |
| Max GPU Mem. (GB) | 0.1516 | 0.1678 | 0.0389 | 0.0539 |
| NPARAMS (MB) | 3.4151 | 5.5977 | 7.0337 | 5.1228 |

Table 2. Memory consumption of ElasTST that using different position encoding approaches. The batch size is 1 and the forecasting horizon is 1024.

|  | Abs PE | RoPE w/o Tunable | Tunable RoPE  |
| --- | --- | --- | --- |
| Max GPU Mem. (GB) | 0.048 | 0.0539 | 0.0539 |
| NPARAMS (MB) | 5.1225 | 5.1225 | 5.1228 |

Table 3. Memory consumption under different patch size settings. The batch size is 1 and the forecasting horizon is 1024. 

|  | p=1 | p=8 | p=16 | p=32 | p={1,8,16,32} | p={8,16,32} |
| --- | --- | --- | --- | --- | --- | --- |
| Max GPU Mem. (GB) | 0.6747 | 0.0536 | 0.0415 | 0.036 | 0.6751 | 0.0539 |
| NPARAMS (MB) | 5.013 | 5.0267 | 5.0424 | 5.0738 | 5.1257 | 5.1228 |

Meta-review: This paper proposes a non-autoregressive architecture for varied-horizon forecasting.  The approach incorporates structured attention masks through patching and rotary position embeddings (RoPE) to encode relative positional information for improved forecasting. This paper addresses a critical issue in time series forecasting and the proposed method is novel and effective. There are concerns about the scalability of the proposed method, but the authors have provided reasonable responses to address these concerns. Overall, this is a solid paper and I recommend it for acceptance.
Decision: Accept (poster)