Title: ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer
Venue: NeurIPS 2024
Accepted: yes
Keywords: time-series forecasting

arbitrary-horizon forecasting

transformer
Abstract: Numerous industrial sectors necessitate models capable of providing robust forecasts across various horizons. Despite the recent strides in crafting specific architectures for time-series forecasting and developing pre-trained universal models, a comprehensive examination of their capability in accommodating varied-horizon forecasting during inference is still lacking. This paper bridges this gap through the design and evaluation of the Elastic Time-Series Transformer (ElasTST). The ElasTST model incorporates a non-autoregressive design with placeholders and structured self-attention masks, warranting future outputs that are invariant to adjustments in inference horizons. A tunable version of rotary position embedding is also integrated into ElasTST to capture time-series-specific periods and enhance adaptability to different horizons. Additionally, ElasTST employs a multi-scale patch design, effectively integrating both fine-grained and coarse-grained information.  During the training phase, ElasTST uses a horizon reweighting strategy that approximates the effect of random sampling across multiple horizons with a single fixed horizon setting. Through comprehensive experiments and comparisons with state-of-the-art time-series architectures and contemporary foundation models, we demonstrate the efficacy of ElasTST's unique design elements. Our findings position ElasTST as a robust solution for the practical necessity of varied-horizon forecasting.

ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer

Abstract

Numerous industrial sectors necessitate models capable of providing robust forecasts across various horizons. Despite the recent strides in crafting specific architectures for time-series forecasting and developing pre-trained universal models, a comprehensive examination of their capability in accommodating variedhorizon forecasting during inference is still lacking. This paper bridges this gap through the design and evaluation of the Elastic Time-Series Transformer (ElasTST). The ElasTST model incorporates a non-autoregressive design with placeholders and structured self-attention masks, warranting future outputs that are invariant to adjustments in inference horizons. A tunable version of rotary position embedding is also integrated into ElasTST to capture time-series-specific periods and enhance adaptability to different horizons. Additionally, ElasTST employs a multi-scale patch design, effectively integrating both fine-grained and coarse-grained information. During the training phase, ElasTST uses a horizon reweighting strategy that approximates the effect of random sampling across multiple horizons with a single fixed horizon setting. Through comprehensive experiments and comparisons with state-of-the-art time-series architectures and contemporary foundation models, we demonstrate the efficacy of ElasTST's unique design elements. Our findings position ElasTST as a robust solution for the practical necessity of varied-horizon forecasting. ElasTST is open-sourced at https://github.com/microsoft/ProbTS/tree/elastst .

1 Introduction

Time-series forecasting plays a crucial role in diverse industries, where it is essential to provide forecasts over various time horizons, accommodating both short-term and long-term planning requirements. This includes predicting COVID-19 cases and fatalities one and four weeks ahead to allocate public health resources [7], estimating future electricity demand on an hourly, weekly, or monthly basis to optimize power management [16], and projecting both immediate and long-term traffic conditions for efficient road management [2, 27], among others.

Despite this, a majority of advanced time-series Transformer [30] variants developed in recent years still necessitate per-horizon training and deployment [37, 40, 33, 32, 22, 20, 39]. These models struggle to handle longer inference horizons once trained for a specific horizon, and may yield suboptimal performance when assessed for shorter horizons. These constraints lead to the practical inconvenience of maintaining distinct model checkpoints for different forecasting horizons required by real-world applications.

Even though recent studies on pre-training universal time-series foundation models have made some progress in facilitating varied-horizon forecasting [26, 9, 8, 31], they primarily concentrate on assessing the overall transfer performance from pre-training datasets to zero-shot scenarios. However, they lack an in-depth investigation into the challenges of generating robust forecasts for different horizons. To be specific, TimesFM [9], a decoder-only Transformer, is capable of arbitrary-horizon forecasting, but this approach could potentially lead to substantial error propagation in long-term forecasting scenarios due to autoregressive decoding. DAM [8], though free from this issue thanks to a novel output design composing sinusoidal functions, cannot effectively capture abrupt changes in time-series data, thereby limiting its utility in critical domains such as energy and traffic. Moreover, while MOIRAI [31] employs a full-attention encoder-only Transformer architecture and supports arbitrary-horizon forecasting via a non-autoregressive manner by introducing mask tokens into forecasting horizons, it remains uncertain how well MOIRAI adapts to different horizons. For example, its architecture design does not ensure the horizon-invariant property: the model output for a specific future position should be invariant to arbitrary extensions in forecasting horizons beyond that. Besides, its performance could drop significantly for moderate context lengths.

To address this research gap, we introduce a comprehensive study to explore how to construct a time-series Transformer variant that can yield robust forecasts for varied inference horizons once trained. We name the developed model as Elastic Time-Series Transformer (ElasTST). ElasTST adopts a non-autoregressive design by incorporating placeholders into forecasting horizons, which is inspired by diffusion Transformers [24] and the success of SORA [3] in video generation. Here we impose structured self-attention masks, only allowing placeholders to attend to observed timeseries patches. This design ensures the aforementioned horizon-invariant property by blocking the information exchange across placeholders. Additionally, we devise a tunable version of rotary position embedding (RoPE) [28] to capture customized period coefficients for time series and to learn the adaptation to varied forecasting horizons. Furthermore, we introduce a multi-patch design to balance fine-grained patches beneficial to short-term forecasting with coarse-grained patches preferred by long-term forecasting, and use a shared Transformer backbone to handle these multi-scale patches. Alongside core model designs, during the training phase, we deploy a horizon reweighting approach that approximates the effects of random sampling across multiple training horizons using just one fixed horizon, eliminating the need for additional sampling efforts. Collectively, these key customizations facilitate ElasTST to produce consistent and accurate forecasts across various horizons.

Our extensive experiments affirm the effectiveness of ElasTST in varied-horizon forecasting. First, we evaluated ElasTST, trained with a fixed horizon and employing a reweighting scheme, against state-of-the-art models trained for specific inference horizons. The results demonstrate that ElasTST delivers competitive performance without requiring per-horizon tuning. Then, we examined variedhorizon forecasting for these models, and the advantages of ElasTST are much more outstanding, demonstrating remarkable extrapolations to longer horizons while preserving robust results for shorter ones. Moreover, we also compared ElasTST with some pre-trained time-series models, such as TimesFM and MOIRAI, and found that dataset-specific tuning still offers prominent advantages over zero-shot inference in challenging datasets, such as Weather and Electricity, and that ElasTST can provide more robust performance across different forecasting horizons. At last, we conducted comprehensive ablation tests to highlight the significance of each unique design element of ElasTST.

In summary, our contributions comprise:

Conducting a systematic study on varied-horizon forecasting, a critical requirement across various domains, yet an underexplored area in time-series research.

Developing a novel Transformer variant, ElasTST, which incorporates structured attention masks for horizon-invariance, tunable RoPE for time-series-specific periods, multipatch representations to balance fine-grained and coarse-grained information, and a horizon reweighting scheme to effectively simulate varied-horizon training.

Demonstrating the effectiveness of ElasTST through experiments comparing it with stateof-the-art time-series architectures and some up-to-date foundation models. Our ablation tests further reveal the importance of its key design elements.

2 Related Work

Traditional Neural Architecture Designs for Time-Series Forecasting The field of time-series forecasting has witnessed a significant evolution of neural architectures, transitioning from early multi-layer perceptrons [23], convolutional [5], and recurrent networks [6], to a more recent focus on various Transformer variants [37, 40, 33, 32, 22, 20, 39]. However, the challenge of variedhorizon forecasting remains underexplored in these studies, as these models often require specific tuning to optimize performance for each inference horizon. Additionally, many models, including PatchTST [22], iTransformer [20], and MTST [39], utilize horizon-specific projection heads, which inherently complicates the extension of their forecasting horizons.

Developing Foundation Models for Time-Series Forecasting Inspired by the remarkable successes in the creation of foundational models in the language and vision domains [4, 25, 3], the trend of pre-training universal foundation models has emerged in time-series forecasting research. Notable works in this area include Lag-Llama [26], DAM [8], TimesFM [9], and MOIRAI [31]. These studies employ unique designs to address the challenges posed by varied variate numbers and forecasting horizons when adapting to new scenarios. Lag-Llama, DAM, and TimesFM adopted the univariate paradigm to circumvent the difficulties associated with handling different variates. In contrast, MOIRAI has taken a different approach by flattening multi-variate time series into a single sequence to facilitate cross-variate learning. While this method has its merits, it is worth noting that it may introduce efficiency issues when handling a substantial number of variates and long forecasting horizons. As a result, this paper also adopts the univariate setup to maintain efficiency. When it comes to varied forecasting horizons, Lag-Llama and TimesFM both utilized the decoder-only Transformer and relied on autoregressive decoding to manage arbitrarily long horizons. DAM introduced a novel output scheme that comprises numerous sinusoidal basis functions, enabling it to project into arbitrary future time points. MOIRAI, on the other hand, used a composite input scheme, combining observed time-series patches with variable placeholders that indicate forecasting horizons, and built a full-attention encoder-only Transformer on top of this. Interestingly, this non-autoregressive generation paradigm originates from diffusion transformers used in video generation [24, 3]. In this paper, we also embrace this paradigm for generating variable-length time-series. Unlike MOIRAI, which has made considerable strides in time-series pre-training using a moderately designed Transformer variant, our focus lies in systematically examining critical architectural enhancements to improve robustness in time-series forecasting across various horizons. We believe that constructing a more robust, resilient, and universal architecture will pave the way for more powerful foundational time-series models to be pre-trained in the future.

Position Encoding in Time-Series Transformers Position encoding plays a pivotal role in Transformers as both self-attention and feed-forward modules lack inherent position awareness. The majority of existing time-series Transformer variants have roughly adopted absolute position encoding [30] with minor modifications across different studies. For instance, Informer [40] and Pyraformer [19] have combined fixed absolute position embeddings with timestamp embeddings such as day, week, hour, minute, etc. Meanwhile, Autoformer [33] and Fedformer [41] have omitted absolute position embeddings and relied solely on timestamp embeddings. Other models like LogTrans [18] and PatchTST [22] have explored learnable position embeddings. However, the challenge with absolute position embedding is its inability to extrapolate into unseen horizons, posing a significant challenge for varied-horizon forecasting. To address this issue, MOIRAI has utilized a relative position embedding technique, RoPE [28], which has been broadly adopted in the language domain to handle variable-length sequences [29]. In our work, we also adopt RoPE to introduce relative position information into self-attention operations. What we uniquely reveal is that the direct application of the RoPE configuration from the language domain to time-series forecasting is not ideal. The reason being that the predefined coefficients do not align well with the typical periodic patterns observed in time-series data. As a solution, we suggest redefining the period range encompassed by the initial RoPE coefficients and making data-driven adjustments to these coefficients.

Multi-Patch Assembly

Enc.

Patching

Enc.

Enc.

Dec.

Dec.

Dec.

𝑴∈ℝ 𝑁×𝑁

Transformer Block

𝐾×

Feed Forward

Layer Norm

Masked Self-Att.

Layer Norm

Structured

Attention Mask

(a)

(b)

𝑓 𝒫

TRoPE (⋅,⋅)

𝑓 𝒫 ∗

TRoPE

Tuning

Initialized Period

Coefficients

𝒫 𝑗

Tuned Period

Coefficients

𝒫 𝑗

∗

(c)

ElasTST

Reweighting

Function

(d)

ℒ

𝑿 ∈ ℝ 𝐿+𝑇

Placeholder

෡

𝑿 ∈ ℝ 𝐿+𝑇

Input Patches in Time-Series Transformers PatchTST [22] spearheaded the concept of segmenting time-series data into patches instead of feeding raw time-series values directly into Transformer models. This straightforward yet effective approach has been widely adopted in subsequent studies, including MTST [39], TSMixer [10], HDMixer [17], and MOIRAI. It noteworthy that MOIRAI has been trained with a diverse range of time-series patches with varying patch sizes. When adapting it to a new dataset, practitioners need to search through a range of patch sizes and rely on validation performance to select a single patch size. In our work, however, we have demonstrated that segmenting time series into multiple patch sizes to create multi-scale patch representations is more advantageous. This approach further aids in stabilizing accurate forecasting across various horizons.

3 Elastic Time-Series Transformers

In Figure 1, we present an overview of ElasTST. Different from other encoder-only Transformer architectures, ElasTST equipped three core designs to facilitate varied-horizon forecasting: structured self-attention masks for placeholders, tunable rotary position embedding (TRoPE) with custimized period coefficients, and a multi-scale patch representation learning. Additionally, we utilize a horizon reweighting scheme to achieve the effects of varied-horizon training.

Notations We define a univariate time series as x 1: T = { x t } T t =1 , with x t ∈ R indicating the value at time index t . The learning objective of a varied-horizon forecasting can be formulated as: max ϕ E x ∼ p ( D ) , ( t,L,T ) ∼ p ( T ) log p ϕ ( x t +1: t + T | x t -L +1: t ) , where p ( D ) is the data distribution from which time series samples are drawn, and p ( T ) is the task distribution, from which the timestamp t , look-back window L , and the prediction horizon T are sampled.

Model Inputs To accommodate varied forecast horizons, our model combines the historical context series x t -L +1: t with placeholders 0 ∈ R T through concatenation, forming the input X = Concat ( x t -L +1: t , 0 ) . This approach allows for flexible adjustment of the input and output dimensions to suit different forecasting scenarios. We further segment X into non-overlapping patches X p ∈ R N × P , where P is the patch length and N = ( L + T ) P represents the number of patches. Each input patch is then transformed into latent space by the encoder H = Enc ( X p ) , H ∈ R N × D .

Masked Self-Attention Arobust varied-horizon forecasting method should deliver consistent outputs across different forecasting horizons while maintaining high accuracy on unseen horizons. Existing time series Transformers, however, typically directly adapt techniques from video generation and natural language processing without considering the unique characteristics of time series. To address this deficiency, ElasTST modifies a standard Transformer Encoder with two crucial enhancements: structured attention masks and a tunable RoPE to encode relative position information effectively. We formulate the attention scores within a masked self-attention as

where W q , W k ∈ R D × d denote the linear mappings for the query and key, respectively. A tunable RoPE f TRoPE dynamically adjusts the relative position encoding manner to best suit each dataset, with further details provided in the following subsection. The structured attention mask M · ,n is set to 0 for patches X p n consisting solely of placeholders and 1 otherwise, ensuring that tokens attend only to context-carrying patches. This structured masking, in conjunction with the relative position encoding, prevents the influence of placeholders on prediction outcomes, thus ensuring consistent outputs across varied forecasting horizons.

Tunable Rotary Position Embedding Position embedding is crucial for the attention mechanism to maintain accuracy over unseen horizons. To overcome the limitations of absolute position embedding in extrapolation scenarios, RoPE has been widely adopted in the NLP domain for handling variable-length sequences. It rotates a vector x ∈ R d onto an embedding curve on a sphere in C d/ 2 , with the rotation parameterized by a base frequency b . The function is defined as f RoPE ( x , t ) j = ( x 2 j -1 + ix 2 j ) e ib -2( j -1) /d t , where j ∈ [1 , 2 , ..., d/ 2] [34]. Typically in NLP, the base frequency b is set to a constant, such as 10,000. However, due to the unique characteristics of time series data, specific adaptations of RoPE are necessary. In this paper, we propose to use the period coefficients P j = 2 π b -2( j -1) /d for parameterization:

where P min and P max represent the predefined minimum and maximum period coefficients, respectively. This formula maintains an exponential distribution but adjusts the range to better align with the periodic characteristics of time series data. By setting P min = 2 π and P max = 2 πb 1 -2 d , this approach mirrors the original RoPE setup.

In addition to adjusting the period range, the distinct and varying periodicities inherent in time series data necessitate more flexible period coefficients. Therefore, in ElasTST, we consider period coefficients P as tunable parameters, optimizing it along with varied datasets and forecasting horizons. This adaptive approach allows for more precise and effective forecasting across diverse conditions. We provide a detailed exploration of this design in Section D.4, and illustrate the optimized period coefficients for each dataset in Appendix E.2.

Multi-Scale Patch Assembly To ensure robust performance across various forecasting horizons, integrating both fine-grained and coarse-grained features from time series data is essential. Different from earlier multi-patch models that utilize separate processing branches for each patch size [39], ElasTST features a multi-scale patch design within a shared Transformer backbone, capable of both parallel and sequential processing. We chose sequential processing for our implementation, keeping the memory consumption comparable to baselines such as PatchTST. The implications of this design on memory usage are further discussed in Appendix F. Specifically, we define each patch size as p = { p 1 , . . . , p S } , with each size corresponding to a dedicated MLP encoder f Enc p i : R p i ↦→ R D and decoder f Dec p i : R D ↦→ R p i . The outputs from each size are flattened and then averaged to produce the final forecast ˆ X . During training, losses under individual patch size are calculated and averaged with the assembled forecast losses, to enhance accuracy and consistency across different scales. Further details on the effectiveness of this design is provided in Section 4.2.

Training Horizon Reweighting To effectively manage varied forecasting horizons, training models across multiple horizon lengths, rather than using fixed ones, is a practical approach [31]. In

this study, we propose to use reweighting scheme for loss computation that simulates this process, without the need for additional sampling efforts. Formally, in the conventional implementation, at each training step s , a forecasting horizon T s is randomly selected from the range [1 , T max ] . 1 Then the loss L s at step s is computed as:

Theoretically, the expectation of this random sampling process can be represented as a weighted loss over a fixed horizon T max. To be specific, the expected value of ω ( τ ) is calculated as: E [ ω ( τ )] = 1 T max ∑ T max T =1 1 T . We further approximate the reweighting function by harmonic series as:

By employing this weighted loss ω ( τ ) during training, we replicate the effect achieved by randomly sampling horizons at an infinite number of training steps. In addition, the function ω ( τ ) can be adapted to follow any desired distribution family and can be made differentiable.

4 Experiments

To validate the effectiveness of ElasTST, we systematically assess its performance across various forecasting scenarios, benchmarking it against established models. The results, detailed in Section 4.1, showcase ElasTSTs adaptability to diverse forecasting horizons. Subsequently, we perform an extensive ablation study in Section 4.2 to examine the impact of its key designs. 2

Datasets Our experiments leverage 8 well-recognized datasets, including 4 from the ETT series (ETTh1, ETTh2, ETTm1, ETTm2), and others include Electricity, Exchange, Traffic, and Weather. These datasets cover a wide array of real-world scenarios and are commonly used as benchmarks in the field. Detailed descriptions of each dataset are provided in Appendix C.1. Following the setup described in [33], all models use a standard lookback window of 96, except TimesFM [9] and MOIRAI [31], which utilize extended lookback windows of 512 and 5000, respectively.

Baselines For our comparative analysis, we select 6 representative forecasting models as baselines: (1) Advanced but non-elastic forecasting models, such as iTransformer [20], PatchTST [22], and DLinear [36]; (2) Autoformer [33], which supports varied-horizon forecasting but requires horizonspecific tuning; (3) the cutting-edge time series foundation model like TimesFM [9] and MOIRAI [31], which are pre-trained for general-purpose forecasting across varied horizons. Our analysis primarily assesses the varied-horizon forecasting capabilities, considering their pre-training on subsets of the datasets used.

Implementation ElasTST is implemented using PyTorch Lightning [12], with a training regimen of 100 batches per epoch, a batch size of 32, and a total duration of 50 epochs. We use the Adam optimizer with a learning rate of 0.001, and experiments are conducted on NVIDIA Tesla V100 GPUs with CUDA 12.1. To ensure fairness, we conducted an extensive grid search for critical hyperparameters across all models in this study. The range and specifics of these hyperparameters are documented in Appendix C.2. For parameters not mentioned in the table, we adhered to the best practice settings proposed in their respective original papers. For evaluation, we use Normalized Mean Absolute Error (NMAE) and Normalized Root Mean Squared Error (NRMSE) as they are scale-insensitive and widely accepted in recent studies [23]. More details are in Appendix C.3.

4.1 Main Results

Comparing ElasTST with Horizon Reweighting to Neural Architectures Tuned for Specific Inference Horizons Experimental results demonstrate that ElasTST consistently delivers exceptional performance across all horizons without the need for per-horizon tuning. As evidenced in Table 1, ElasTST outperformed SOTA models on diverse datasets including ETTm1, ETTh1, ETTh2,

Traffic, Weather, and Exchange, despite these models undergoing specific horizon-based training and tuning. This clearly demonstrates ElasTST's inherent robustness and its remarkable capacity to generalize effectively across varied forecasting scenarios.

Comparing the Robustness of Different Models for Varied Inference Horizons The results clearly position ElasTST as the most robust option for deploying a single, well-trained model across various inference horizons and application scenarios. As demonstrated in Figure 2, ElasTST consistently maintains strong performance across both seen and unseen horizons, underscoring its ability to navigate beyond trained scopes with consistent accuracy across a wide range of forecasts.

In contrast, other models face significant challenges in varied-horizon forecasting. State-of-the-art models like iTransformer and PatchTST excel within their trained horizons but struggle when extended beyond these limits. Models that require horizon-specific tuning, such as Autoformer, often experience abrupt declines in performance, illustrating that scalability alone is insufficient without tailored optimization. TimesFM, with its autoregressive nature, shows substantial error propagation in unseen datasets like ETT and Exchange, and increased errors in pre-trained datasets such as Weather as the horizon extends. While MOIRAI demonstrates strong zero-shot performance on datasets like ETT and Exchange, we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages. Furthermore, MOIRAIs performance significantly diminishes with shorter context lengths, as discussed in their paper [31]. In comparison, ElasTST operate effectively with much shorter context lengths.

24

48

96

192

336

720 1024

Infer hor.

0.20

0.25

0.30

0.35

0.40

NMAE

ETT

24

48

96

192

336

720 1024

Infer hor.

0.09

0.10

0.11

0.12

Weather

24

48

96

192

336

720 1024

Infer hor.

0.10

0.15

0.20

0.25

0.30

0.35

Electricity

24

48

96

192

336

720 1024

Infer hor.

0.20

0.30

0.40

0.50

Traffic

w/o Mask

w/o Tunable RoPE

w/o Multi-patch

ElasTST

4.2 Ablation Study

Structured Attention Masks The ablation study confirms that structured attention masks are essential for robust inference across horizons that differ from the training phase. As illustrated in Figure 3, removing structured masks from ElasTST results in significant performance declines, particularly in the Weather dataset. Furthermore, as demonstrated in Figure 7 (see Appendix D.2), the benefits of structured masks are consistent across all forecasting horizons. This underscores the importance of the horizon-invariant property for enhancing the stability of time series forecasting, an aspect often overlooked in current research.

Tunable Rotary Position Embedding Experimental results indicate that tunable RoPE significantly improves the models ability to extrapolate. Figure 4a shows that while other positional embedding methods are effective on seen horizons, they falter when applied to horizons extending beyond the training range. Although the original RoPE excels in NLP tasks, it underperforms in time series forecasting. Besides, data-driven adjustments of these coefficients enable far more robust extrapolation. Dynamically tuning RoPE parameters according to the periodic patterns of the dataset proves highly beneficial, especially when inferring over unseen horizons.

Furthermore, a range from 1 to 1000 for the period coefficients P is more suitable for time series forecasting. As demonstrated in Figure 4b, using the commonly-used NLP settings with P min = 1 and P max = 10000 does not fully exploit the potential of RoPE in time series forecasting. Setting P max to 1000 results in better performance. We hypothesize that this is because, unlike textual data which benefits from attention over longer contexts, the time series data, especially when segmented into patches, benefits from focusing on shorter, more recent intervals. By adjusting the maximum period coefficient to a lower value, the model captures a richer spectrum of mid-to-high frequency patterns, thereby enhancing its effectiveness. Detailed analyses of these findings are available in Appendix D.4. Appendix E includes visualizations demonstrating how different initial ranges impact frequency components, along with illustrations of the tuned period coefficients for each dataset.

24

48

96

192

336

720 1024

Infer hor.

0.20

0.25

0.30

0.35

0.40

NMAE

ETT

24

48

96

192

336

720 1024

Infer hor.

0.08

0.09

0.10

0.11

0.12

Weather

24

48

96

192

336

720 1024

Infer hor.

0.10

0.15

0.20

0.25

0.30

0.35

Electricity

24

48

96

192

336

720 1024

Infer hor.

0.20

0.30

0.40

0.50

0.60

Traffic

w/o PE

Abs PE

Learn PE

RoPE

Tunable RoPE

96

192

336

720

Forecasting Horizon

0.18

0.20

0.22

0.24

0.26

NMAE

96

192

336

720

1024

Forecasting Horizon

0.18

0.20

0.23

0.25

0.28

96

192

336

720

1024

Forecasting Horizon

0.18

0.20

0.22

0.24

0.26

0.28

96

192

336

720

1024

Forecasting Horizon

0.18

0.20

0.22

0.24

0.26

10

100

1000

10000

(b) The effect of selecting P max, with P min fixed at 1 and P is untunable during training.

96

192

336

720

1024

Forecasting Horizons

0.15

0.20

0.25

0.30

NMAE

Patch

8

16

32

64

8_16

8_16_32

8_16_32_64

Multi-Patch Design These experiments demonstrate that multi-patch configurations generally outperform single patch sizes across various forecasting horizons. Figure 5 shows that the configuration p = { 8 , 16 , 32 } consistently achieves the lowest NMAE values, effectively balancing the capture of short-term dynamics and long-term trends. However, adding larger patches, such as p = { 8 , 16 , 32 , 64 } , does not consistently improve performance and can sometimes increase the NMAE. This suggests that more complex configurations may not always provide additional benefits and could even be counterproductive.

Moreover, the patch size selection is particularly critical in the varied-horizon forecasting scenarios. As demonstrated in the Figure 10 (see Appendix D.5), various combinations of training and forecasting horizons exhibit distinct preferences for patch sizes. For instance, when the training forecasting horizon is 720, during the inference stage, longer forecasting horizons prefer larger patch sizes. Conversely, on shorter training horizons, such as 96 and 192, choosing large patch sizes for longer horizons can lead to performance collapse. This difference underscores the complexity and necessity of optimal patch size selection in achieving effective elastic forecasting. Detailed results for four training horizons and further analysis are provided in Appendix D.5.

The Impact of Training Horizons Further experiments validate the effectiveness of our proposed training horizon reweighting scheme in enhancing varied-horizon inference. As illustrated in Figure 6, reweighting longer horizons simplifies the training process, yielding better outcomes than

selecting a fixed horizon and mitigating the uncertainties associated with random sampling. Crucially, this training approach is model-agnostic and can be applied to different forecasting training scenarios. These results also highlight the advantages of a flexible forecasting architecture, which allows training horizons to be customized to the unique characteristics of each dataset.

We also observe that different datasets have distinct preferences for training horizons. For example, in the Exchange dataset, the longest training horizon led to worse results compared to a shorter horizon of 96, suggesting risks of overfitting or forecast instability with prolonged horizons. Besides, in the ETTh1, employing random sampling for training horizons proved suboptimal. These insights show that tailoring the training horizon selection strategy to the specific dataset can yield improvements. One potential enhancement could involve dynamically optimizing the horizon reweighting scheme alongside model training.

5 Conclusion

This study introduces the Elastic Time-Series Transformer (ElasTST), a pioneering model designed to tackle the significant and insufficiently explored challenge of varied-horizon forecasting. ElasTST integrates a non-autoregressive framework with innovative elements such as structured self-attention masks, tunable Rotary Position Embedding (RoPE), and a versatile multi-scale patch system. Additionally, we implement a training horizon reweighting scheme that simulates random sampling of forecasting horizons, thus eliminating the need for extra sampling efforts. Together, these elements enable ElasTST to adapt to a wide range of forecasting horizons, delivering reliable and competitive outcomes even when facing horizons that were not encountered during the training phase.

Limitations While ElasTST demonstrates robust performance across various forecasting tasks, several limitations have been identified that highlight opportunities for future enhancements. First, the current version of ElasTST does not incorporate a pre-training phase, which could significantly improve the models initial grasp of time-series dynamics and boost its efficiency during task-specific fine-tuning. Further exploration is needed to ascertain optimal training methodologies that maximize the architectural benefits of ElasTST. Additionally, while the training horizon reweighting scheme is straightforward and effective in enhancing performance across different inference horizons, it is not the optimal solution for all datasets. Moreover, the evaluation of ElasTST is limited to a select number of datasets, which may not fully represent the broader challenges encountered in more complex or diverse real-world scenarios.

Future Work In response to these limitations, our forthcoming research efforts will concentrate on developing and validating pre-training protocols for ElasTST to elevate its foundational performance and extend its applicability across universal forecasting tasks. We aim to incorporate a reasonable training approach that will fine-tune the models ability to seamlessly manage forecasts of varying lengths, thus bolstering its utility in dynamic real-world environments. Furthermore, by broadening the range of datasets used for model evaluations, we intend to rigorously test ElasTSTs effectiveness across an expanded spectrum of industry-specific challenges. This comprehensive approach will not only solidify ElasTSTs standing as a cutting-edge solution for time-series forecasting but also enhance our understanding of its practical implications and potential in diverse industrial applications.

References

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. 2024. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815 (2024).

Joaquim Barros, Miguel Araujo, and Rosaldo JF Rossetti. 2015. Short-term real-time traffic prediction methods: A survey. In 2015 International Conference on Models and Technologies for Intelligent Transportation Systems .

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. 2024. Video generation models as world simulators. (2024). https://openai.com/research/ video-generation-models-as-world-simulators

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In NeurIPS .

Yitian Chen, Yanfei Kang, Yixiong Chen, and Zizhuo Wang. 2020. Probabilistic forecasting with temporal convolutional neural network. Neurocomputing 399 (2020), 491-501.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014).

Estee Y Cramer, Evan L Ray, Velma K Lopez, Johannes Bracher, Andrea Brennen, Alvaro J Castro Rivadeneira, Aaron Gerding, Tilmann Gneiting, Katie H House, Yuxin Huang, et al. 2022. Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States. Proceedings of the National Academy of Sciences (2022).

Luke Nicholas Darlow, Qiwen Deng, Ahmed Hassan, Martin Asenov, Rajkarn Singh, Artjom Joosen, Adam Barker, and Amos Storkey. 2024. Dam: A foundation model for forecasting. In ICLR .

Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. 2023. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688 (2023).

Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. In SIGKDD . 459-469.

Vijay Ekambaram, Arindam Jati, Nam H Nguyen, Pankaj Dayama, Chandra Reddy, Wesley M Gifford, and Jayant Kalagnanam. 2024. Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series. arXiv preprint arXiv:2401.03955 (2024).

William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning. https://doi. org/10.5281/zenodo.3828935

Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. 2024. UniTS: Building a Unified Time Series Model. arXiv preprint arXiv:2403.00131 (2024).

Azul Garza, Cristian Challu, and Max Mergenthaler-Canseco. 2023. TimeGPT-1. arXiv preprint arXiv:2310.03589 (2023).

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. 2024. MOMENT: A Family of Open Time-series Foundation Models. In ICML .

Luis Hernandez, Carlos Baladron, Javier M Aguiar, Belén Carro, Antonio J SanchezEsguevillas, Jaime Lloret, and Joaquim Massana. 2014. A survey on electric power demand forecasting: Future trends in smart grids, microgrids and smart buildings. IEEE Communications Surveys & Tutorials (2014).

Qihe Huang, Lei Shen, Ruixin Zhang, Jiahuan Cheng, Shouhong Ding, Zhengyang Zhou, and Yang Wang. 2024. HDMixer: Hierarchical Dependency with Extendable Patch for Multivariate Time Series Forecasting. In AAAI , Vol. 38. 12608-12616.

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. In NeurIPS . 5244-5254.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. 2021. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In ICLR .

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2023. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 (2023).

Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. 2024. Timer: Transformers for Time Series Analysis at Scale. In ICML .

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In ICLR .

Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2020. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. In ICLR .

William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In ICCV . 4195-4205.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML .

Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, et al. 2023. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08278 (2023).

Fei Su, Honghui Dong, Limin Jia, Yong Qin, and Zhao Tian. 2016. Long-term forecasting oriented to urban expressway traffic situation. Advances in mechanical engineering (2016).

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS . 5998-6008.

Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592 (2024).

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2023. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In ICLR .

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. In NeurIPS . 22419-22430.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039 (2023).

Jiexia Ye, Weiqi Zhang, Ke Yi, Yongzi Yu, Ziyue Li, Jia Li, and Fugee Tsung. 2024. A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Mode. arXiv preprint arXiv:2405.02358 (2024).

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are Transformers Effective for Time Series Forecasting?. In AAAI . 11121-11128.

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A transformer-based framework for multivariate time series representation learning. In SIGKDD . 2114-2124.

Jiawen Zhang, Shun Zheng, Wei Cao, Jiang Bian, and Jia Li. 2023. Warpformer: A multi-scale modeling approach for irregular clinical time series. In SIGKDD . 3273-3285.

Yitian Zhang, Liheng Ma, Soumyasundar Pal, Yingxue Zhang, and Mark Coates. 2024. Multiresolution time-series transformer for long-term forecasting. In International Conference on Artificial Intelligence and Statistics . 4222-4230.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-series Forecasting. In AAAI . 11106-11115.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. Fedformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. In ICML . 27268-27286.

A Details of Methods

A.1 Details of Rotary Position Embedding

Rotary position embedding (RoPE) [28] is a method used to encode the position of tokens in the input sequence for transformer-based models, enhancing the capability to utilize the positional context of tokens. RoPE uniquely incorporates the geometric property of vectors, transforming them into a rotary matrix that interacts with the vector embeddings.

Here, we have adopted the formal definition from the original RoPE paper. In the simplest twodimensional (2D) case, RoPE considers a dimension d = 2 , where each position vector is treated in its complex form. The formulation is given by:

This expression shows that the embedding rotates the affine-transformed word embedding vectors by angle multiples relative to their position indices. This rotation is mathematically represented through a multiplication matrix:

For a generalized form in any dimension d , RoPE divides the space into d/ 2 sub-spaces and combines them to utilize the linearity of the inner product. The generalized rotary matrix R d Θ ,m is defined as:

where Θ = { θ i = 10000 -2( i -1) /d , i ∈ [1 , 2 , ..., d/ 2] } . This formulation ensures that RoPE is computationally efficient and stable due to the orthogonality of R d Θ ,m . The use of sparse matrices further improves computational efficiency, making RoPE a practical approach to incorporate in largescale transformer models.

B Additional Related Work on Foundation Models

We summarize existing time series foundational models in Table 2, excluding the LLM-oriented ones [35]. These models typically use standard architecture designs, position encodings, and patching approaches, primarily aiming to enhance transferability in zero-shot scenarios. However, they generally do not deeply explore the challenges of producing robust forecasts across varied horizons. Our work specifically addresses this gap by improving model design to enhance robustness for varied-horizon forecasting.

C Additional Details of Experiment Setting

C.1 Dataset Details

Our experiments utilize 8 widely recognized datasets, including 4 from the ETT series (ETTh1, ETTh2, ETTm1, ETTm2), as well as the Electricity, Exchange, Traffic, and Weather datasets. These

datasets encompass a broad range of real-world applications and are frequently used as benchmarks in the field 3 . Consistent with common practices in long-term forecasting [33, 36, 22, 20], all models are tested under the forecasting horizons T ∈ { 96 , 192 , 336 , 720 } . Except for TimeFM [9], which uses a lookback window of 512, a standard lookback window of 96 is employed across all other models as [33].

C.2 Implementation Details

ElasTST is implemented using PyTorch Lightning [12]. Training consists of 100 batches per epoch, capped at 20 epochs, with the NMAE metric used for model checkpointing. We use the Adam optimizer with a learning rate of 0.001, and experiments are conducted on NVIDIA Tesla V100 GPUs with CUDA 12.1. The code for Transformer Block is adapted from [38].

Baselines We select five representative forecasting models as our baselines:

iTransformer 4 [20]: A transformer-based model that inverts the dimensions of time and variates to effectively capture multivariate correlations, enhancing generalization across different variates.

PatchTST 5 [22]: A transformer-based model segmenting time series into subseries-level patches and employing channel-independent processing to improve forecasting accuracy.

DLinear 6 [36]: An MLP-based model that has demonstrated superior performance over more complex transformer-based models in multiple real-life datasets.

Autoformer 7 [33]: Integrates a novel Auto-Correlation mechanism that exploits series periodicity for enhanced dependency discovery and representation aggregation.

TimeFM 8 [9]: A foundation model for time series forecasting, pretrained using a decoderstyle attention mechanism and input patching on an extensive corpus of real-world and synthetic data.

MOIRAI (UNI2TS) 9 [31]: A foundation model for time series forecasting, pretrained using a masked encoder-based Transformer on the extensive Large-scale Open Time Series Archive (LOTSA) with over 27 billion observations.

Hyper-parameter Tuning To ensure fairness, we conducted an extensive grid search for critical hyperparameters across all models in this study. The range and specifics of these hyperparameters are documented in Table 4. For parameters not mentioned in the table, we adhered to the best practice settings proposed in their respective original papers.

C.3 Evaluation Metrics

We employ the Normalized Mean Absolute Error (NMAE) and Normalized Root Mean Squared Error (NRMSE) as our evaluation metrics because they provide a relative measure of error that is independent of the data scale. It's important to note that some original papers reported metrics prior to re-scaling forecasts to their original magnitude, which can affect metric calculations. In this study, we have carefully ensured that our reproduced results are consistent with those reported in the original studies and have applied these unified metrics to enable a comprehensive and fair comparison.

Normalized Mean Absolute Error (NMAE) The Normalized Mean Absolute Error (NMAE) is a normalized version of the MAE, which is dimensionless and facilitates the comparability of the error magnitude across different datasets or scales. The mathematical representation of NMAE is given by:

Normalized Root Mean Squared Error (NRMSE) The Normalized Root Mean Squared Error (NRMSE) is a normalized version of the Root Mean Squared Error (RMSE), which quantifies the average squared magnitude of the error between forecasts and observations, normalized by the expectation of the observed values. It can be formally written as:

D Additional Experimental Results

D.1 Comparing ElasTST with More Neural Architectures

D.2 Performance Gains Across the Forecasting Horizon

In Figure 7, we compare the performance gains of each model design at different points within the forecasting window. The benefits of structured masks are consistent across the entire horizon, while the advantages of tunable RoPE and multi-patch designs become more prominent when handling unseen horizons. Notably, the tunable RoPE plays a critical role in enhancing the models extrapolation capability.

0

100

200

300

400

500

600

700

1.0

1.5

2.0

2.5

3.0

3.5

Model.MAE / ElasTST.MAE

Model

w/o Mask

w/o Multi-patch

w/o Tunable RoPE

ElasTST

Hor.

D.3 More Analysis of the Impact of Training Horizon

The scalable architecture of ElasTST allows it to treat the training horizon as a hyperparameter. This adaptability prompts us to evaluate performance across various training and inference horizons (see Figure 8), yielding several interesting insights.

Model performance deteriorates as the forecasting horizon increases, particularly in models trained on shorter horizons, as seen in the ETT and Electricity datasets. This pattern suggests the importance of training models on extended horizons to capture adequate contextual information.

Tuning models specifically for a given horizon does not guarantee improved performance on that horizon, as noted in the Weather dataset. This indicates that optimal model settings depend significantly on dataset-specific characteristics, and horizon-specific tuning may not be a reliable strategy.

The longest training horizons do not always produce the best forecasting results. In the Exchange dataset, for example, the longest horizon yielded poorer results compared to a shorter training horizon of 96. This points to the potential risks of overfitting or forecast instability when training with long-term series only.

These observations underscore the importance of tailoring training horizons to the unique characteristics of each dataset and underscore the benefits of an architecture designed for elastic forecasting. Furthermore, they suggest the potential advantages of implementing a mixed-horizon training strategy, which leverages multiple horizons to produce more resilient forecasts.

D.4 More Analysis of the Impact of Tunable Rotary Position Embedding

Beyond its scalable architecture, the position embedding plays a crucial role in enhancing the elasticity of ElasTST. We analyze the Tunable RoPE in ElasTST by examining the effects of the initialization of period coefficients, specifically P min and P max, and the benefits of parameter optimization during the training process.

Experimental results, presented in Figure 9, indicate that using settings similar to the commonlyused one in NLP, with P min = 1 and P max = 10000 , does not fully exploit its potential in time series

(a)

The effect of selecting

P

min, with

P

fixed at 10,000 and

θ

is untunable during training.

(b)

The effect of selecting

P

P

min

fixed at 1 and

θ

is untunable during training.

forecasting. This discrepancy stems from fundamental differences between the data types: in text, discrete tokens are the smallest units, requiring attention over longer contexts, while time series data, particularly when patched, may benefit from focusing on shorter, more recent tokens.

Furthermore, our findings reveal that tuning parameters in RoPE during training significantly improves forecasting accuracy, particularly over varying and extended horizons. When the training horizon is set to 96, a tunable feature shows minimal impact, suggesting that a short-seen horizon does not facilitate learning effective period coefficients. However, as the training horizon extends, the advantages of a tunable theta become more pronounced, especially for unseen horizons.

These results emphasize the importance of customizing RoPE's period coefficients settings and utilizing tunable RoPE to enable flexible and accurate forecasting in time series analysis. Appendix E offers visualizations demonstrating how different initial ranges impact the frequency components, along with a detailed analysis of the distribution of RoPE periods optimized for each dataset.

D.5 The Impact of Patch Size Selection

These experiments highlight the critical impact of patch size selection, particularly in varied-horizon forecasting scenarios. As demonstrated in Figure 10, various combinations of training and forecasting horizons exhibit distinct preferences for patch sizes. For instance, when the training forecasting horizon is 720, during the inference stage, longer forecasting horizons prefer larger patch sizes. Conversely, on shorter training horizons, such as 96 and 192, choosing large patch sizes for longer horizons can lead to performance collapse. This difference underscores the complexity and necessity of optimal patch size selection in achieving effective elastic forecasting.

96

192

336

720

1024

Forecasting Horizons

0.2

0.3

0.4

NMAE

Patch

8

16

32

64

8_16

8_16_32

8_16_32_64

96

192

336

720

1024

Forecasting Horizons

0.15

0.20

0.25

0.30

NMAE

Patch

8

16

32

64

8_16

8_16_32

8_16_32_64

96

192

336

720

1024

Forecasting Horizons

0.15

0.20

0.25

0.30

NMAE

Patch

8

16

32

64

8_16

8_16_32

8_16_32_64

96

192

336

720

1024

Forecasting Horizons

0.15

0.20

0.25

0.30

NMAE

Patch

8

16

32

64

8_16

8_16_32

8_16_32_64

Moreover, Figure 10 clearly show that multi-patch configurations typically surpass single patch sizes across most forecasting horizons. The configuration p = { 8 , 16 , 32 } consistently offers the lowest NMAE values, striking an optimal balance between capturing short-term dynamics and longterm trends. However, introducing larger patches ( p = { 8 , 16 , 32 , 64 } ) does not always enhance performance and can sometimes increase the NMAE, indicating that overly complex configurations may not yield additional benefits and could be counterproductive.

These findings emphasize the advantages of employing multi-patch configurations to improve the accuracy of varied-horizon forecasting. They also highlights the importance of carefully selecting patch size combinations to optimize performance and minimize computational expenses.

E Visualization of Periods in RoPE

E.1 Initialized Periods

To provide a more intuitive visualization of the initial distribution of periodicity coefficients, we present this in Figure 11.

E.2 Tuned periodicity coefficients

In Figure 12, we present the distribution of tuned periodicity coefficients for each dataset.

0

5

10

j

0

250

500

750

1000

Period P

(a)

P

∈

[1

,

1000]

.

0

5

10

j

0

2500

5000

7500

10000

Period P

(b)

P

∈

[1

,

10000]

(Same as RoPE paper).

0

20

40

60

80

100

1

0

1

sin(2

/1.0

x

)

0

20

40

60

80

100

1

0

1

sin(2

/2.7

x

)

0

20

40

60

80

100

1

0

1

sin(2

/ 7.2

x

)

0

20

40

60

80

100

1

0

1

sin(2

/19.3

x

)

0

20

40

60

80

100

1

0

1

sin(2

/51.8

x

)

0

20

40

60

80

100

1

0

1

sin(2

/138.9

x

)

0

20

40

60

80

100

0

1

sin(2

/372.8

x

)

0

20

40

60

80

100

0.0

0.5

sin(2

/1000.0

x

)

0

20

40

60

80

100

1

0

1

sin(2

/1.0

x

)

0

20

40

60

80

100

1

0

1

sin(2

/3.7

x

)

0

20

40

60

80

100

1

0

1

sin(2

/13.9

x

)

0

20

40

60

80

100

1

0

1

sin(2

/51.8

x

)

0

20

40

60

80

100

0

1

sin(2

/193.1

x

)

0

20

40

60

80

100

0.0

0.5

sin(2

/ 719.7

x

)

0

20

40

60

80

100

0.0

0.2

sin(2

/2682.7

x

)

0

20

40

60

80

100

0.00

0.05

sin(2

/10000.0

x

)

(c) Frequency distribution when P ∈ [1 , 1000] . (d) Frequency distribution when P ∈ [1 , 10000] (Same as RoPE paper).

F Model Efficiency

F.1 Overall Computational Efficiency Comparison

In Table 6, we compare the memory usage of ElasTST with other baseline models. The comparison reveals that ElasTST does not require more computational resources than other Transformer-based models.

F.2 Memory Usage Introduced by TRoPE

In Table 7, we compare the memory usage of ElasTST with different position encodings. The results show that RoPE adds an almost negligible number of parameters compared to vanilla absolute position encoding. While applying rotation matrix does introduce slightly higher memory usage, it remains within acceptable limits.

F.3 Memory Usage Introduced by Multi-scale Patch Assembly

While our model uses a shared Transformer backbone to process all patch sizes, this can be done either in parallel or sequentially, depending on whether shorter computation times or lower memory usage is prioritized. In practice, we chose the sequential approach, where each patch size is processed individually, and the total forecast is assembled afterward. This approach ensures that the memory bottleneck depends on the smallest patch size used.

As indicated in Table 8, memory usage is primarily influenced by the smallest patch size, not by the number of patch sizes employed. The additional parameters introduced by using multiple patch sizes are almost negligible. For example, the multi-patch setting 8,16,32 used in the paper requires the same maximum memory as using a single patch size of 8. Under resource constraints, the minimum patch size can be adjusted to balance model performance and memory usage.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract and introduction are well-aligned with the paper's contributions and scope as detailed in Sections 3 and 4, where we elaborate on the theoretical advancements and experimental validations.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work are discussed in Section 5.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

answer: [NA]

Justification: There is no theoretical result in this paper.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All necessary details required for reproducing the main experimental results are comprehensively disclosed in Section C.2 and Appendix C.2.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available at https://github.com/microsoft/ProbTS/tree/ elastst .

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All training and testing details, including data splits, hyperparameters, their selection rationale, and the type of optimizer used, are fully specified in Section C.2 and Appendix C.2.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For the experiments presented in the main result tables, we conducted each experimental setup three times using random seeds set to 0, 1, and 2. The mean and variance of these three trials are reported in the tables, providing a measure of the experiments' statistical robustness.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Detailed specifications of the compute resources are provided in Appendix C.2 and Appendix F.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research presented in this paper adheres fully to the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: No negative societal impact of the work performed.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The methodologies developed in this study do not involve such risk.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All relevant assets used in our research are duly credited. We have explicitly listed the licenses and terms of use for each asset, including URLs to the original sources, in Appendix C.1 and Appendix C.2.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new asset have been released so far.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a masked encoder based transformer model for time series forecasting. It performs masking in the observation space, replacing the prediction horizon with 0s. It further applies an attention mask such that tokens only attend to the context, and do not attend to the masked tokens. The paper also uses a tunable version of rotary positional embeddings, and ensembles predictions over various patch sizes.\n\nSTRENGTHS:\nThe paper proposes a method which can adapt to multiple forecast horizons, which is a weakness of many recent deep forecasting models. The paper presents several empirical analyses to verify the claims made, and also presents a detailed look at related work.\n\nWEAKNESSES:\nOverall, the proposed approach is very similar to Moirai, with several minor changes:\n\ni) Perform the masking operation in observation space\n\nii) Add an attention mask\n \niii) Tunable rotary embeddings\n \niv) Ensembling results over multiple patch sizes\n\nHowever, it seems that even more features which Moirai had were removed, making it a weaker proposition. Such features include probabilistic predictions, multivariate capabilities, and any context length. Furthermore, few experiments are done to show the benefits of the changes, such as (i), (ii), and (iv). \n\nOthers:\n1. The statement, \"we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages\" (L238-L239) is a little eyebrow raising, in the sense that it is stating the obvious. I believe most researchers expect in-distribution predictions to outperform zero-shot predictions. Such statements should be rephrased.\n2. Effects of tuning rotary embeddings in figure 4 seem very marginal, error margins should be provided for such cases.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nAuthors have provided limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ElasTST, which aimed at addressing the challenge of varied-horizon time-series forecasting. ElasTST integrates a non-autoregressive architecture with placeholders for forecasting horizons, structured self-attention masks, and a tunable rotary position embedding. Furthermore, it employs a multi-scale patch design to incorporate both fine-grained and coarse-grained temporal information, enhancing the model’s adaptability across different forecasting horizons. Experiments in the paper demonstrate the model's performance in providing robust predictions when extrapolating beyond the trained horizons.\n\nSTRENGTHS:\n- The paper attempts to address an important issue in time series applications by enabling the model to adapt to varied forecasting horizons during the inference stage, which can make the trained forecasting model more flexible in usage.\n\n- Introducing multi-scale patching operations to construct input tokens for the Transformer can effectively handle local patterns of different granularities.\n\n- The paper introduces a tunable Rotary Position Embedding (RoPE) module tailored for time series tasks, showing potential for broader application across various forecasting methods.\n\nWEAKNESSES:\n- The newly proposed tunable RoPE module lacks comparative analysis with previous methods. A comparison with methods such as no positional embedding, vanilla positional encoding, trainable positional embedding matrices, and original rotary positional embeddings, would clarify the benefits of different approaches. Since tunable RoPE appears easily transferable, demonstrating its performance enhancement by applying it to previous models like PatchTST and Autoformer would provide more intuition.\n\n- Using longer sequence length significantly increases the computational cost in Transformers. Comparing with baselines like PatchTST, the paper uses multiple patch sizes as input processing method, which significantly increases computational costs. Further analysis like how much adding a patch size can increase computational costs or what patch size combination are more reasonable to choose under resource constraints would be beneficial.\n\nQUESTIONS:\n- Given the significant increase in computational cost from longer token sequences, would independently inputting each patch size into the transformer layers, instead of all at once, reduce memory usage while maintaining model performance?\n\n- With multiple patch sizes used in the study, how are the RoPE applied? Are they applied independently to tokens from each patch size, or collectively to all tokens from different patch sizes?\n\n- Ablation studies indicate that incorporating smaller patch sizes seems to benefit model performance substantially. If my understanding is correct, when the patch size is 1 (excluding tunable RoPE), the structure is similar to the decoder part of models like Informer and Autoformer. Would using a patch size of 1 worsen the model performance? Would adding \"1_\" to the existing best patch size setting \"8_16_32\" improve performance? Using patch size 1 turns the model into a channel-independent Informer-decoder-like model, which significantly increases computational costs, hence it you don't have to consider this on large datasets.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ElasTST, a novel approach designed to enhance robustness in forecasting across various horizons. The proposed architecture dynamically adapts to different forecasting horizons, addressing a significant challenge in the literature. Previous methods typically relied on recursive approaches or utilized masks that failed to maintain horizon invariability. ElasTST aims to achieve high forecasting accuracy while ensuring adaptability across different horizons by employing a transformer encoder as its backbone. The approach incorporates structured attention masks through patching and rotary position embeddings (RoPE) to encode relative positional information for improved forecasting.\n\nSTRENGTHS:\n1. The paper effectively tackles a crucial issue in the forecasting literature—enhancing deep learning architectures to produce forecasts for any horizon while maintaining horizon invariability and reducing the concatenation of errors.\n2. The approach leverages well-established research, such as patching from the PatchTST paper, and also rotary position embeddings, to handle long-horizon forecasting tasks.\n3. The paper considers recent advancements in foundation models for time series forecasting.\n\nWEAKNESSES:\n1. The literature review on foundational models is lacking. Notable models such as TimeGPT-1 [1], Chronos [2], TinyTimeMixers [3], and Moment [4] are not discussed.\n2. A traditional encoder transformer is not included as a baseline, which would help demonstrate the improvements offered by the proposed method. Additionally, NLP-based architectures such as NHITS [5] and TsMixer [6] are absent from the comparisons.\n3. The paper does not adequately address the computational scalability of ElasTST when applied to very large datasets or extremely long time series.\n4. The authors modified the traditional metrics used in long-horizon literature (MSE and MAE) to scaled metrics. They argue this adjustment reports fairer results, but it overlooks the fact that these datasets are typically already scaled, making additional scaling in the metric unnecessary.\n5. The paper lacks a detailed discussion on potential failure cases or scenarios where ElasTST might underperform, which would provide a more balanced view of its applicability.\n\n[1] https://arxiv.org/abs/2310.03589\n[2] https://arxiv.org/abs/2403.07815v1\n[3] https://arxiv.org/abs/2401.03955\n[4] https://arxiv.org/abs/2402.03885\n[5] https://arxiv.org/abs/2201.12886\n[6] https://arxiv.org/abs/2303.06053\n\nQUESTIONS:\n1. What is the computational complexity added by the Rotary Position Embeddings (RoPE)?\n2. In some cases, the results are very similar to PatchTST, such as with the ETTm2 dataset. Is there an intuition or explanation regarding the types of data that explain these similarities and differences in performance?\n3. It would be interesting to compare the performance gains for each point in the forecasting window. Are the gains through the whole window or in the earliest/latest points?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}