ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer

Authors: jiawen  zhang
shun  zheng
xumeng  wen
xiaofang  zhou
jiang  bian
jia  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:34:59

Abstract: Numerous industrial sectors necessitate models capable of providing robust forecasts across various horizons. Despite the recent strides in crafting specific architectures for time-series forecasting and developing pre-trained universal models, a comprehensive examination of their capability in accommodating varied-horizon forecasting during inference is still lacking. This paper bridges this gap through the design and evaluation of the Elastic Time-Series Transformer (ElasTST). The ElasTST model incorporates a non-autoregressive design with placeholders and structured self-attention masks, warranting future outputs that are invariant to adjustments in inference horizons. A tunable version of rotary position embedding is also integrated into ElasTST to capture time-series-specific periods and enhance adaptability to different horizons. Additionally, ElasTST employs a multi-scale patch design, effectively integrating both fine-grained and coarse-grained information.  During the training phase, ElasTST uses a horizon reweighting strategy that approximates the effect of random sampling across multiple horizons with a single fixed horizon setting. Through comprehensive experiments and comparisons with state-of-the-art time-series architectures and contemporary foundation models, we demonstrate the efficacy of ElasTST's unique design elements. Our findings position ElasTST as a robust solution for the practical necessity of varied-horizon forecasting.

Main paper: 
 introduction: 
 Time-series forecasting plays a crucial role in diverse industries, where it is essential to provide forecasts over various time horizons, accommodating both short-term and long-term planning requirements. This includes predicting COVID-19 cases and fatalities one and four weeks ahead to allocate public health resources [7] , estimating future electricity demand on an hourly, weekly, or monthly basis to optimize power management [16] , and projecting both immediate and long-term traffic conditions for efficient road management [2, 27] , among others. Despite this, a majority of advanced time-series Transformer [30] variants developed in recent years still necessitate per-horizon training and deployment [37, 40, 33, 32, 22, 20, 39] . These models struggle to handle longer inference horizons once trained for a specific horizon, and may yield suboptimal performance when assessed for shorter horizons. These constraints lead to the practical inconvenience of maintaining distinct model checkpoints for different forecasting horizons required by real-world applications.Even though recent studies on pre-training universal time-series foundation models have made some progress in facilitating varied-horizon forecasting [26, 9, 8, 31] , they primarily concentrate on assessing the overall transfer performance from pre-training datasets to zero-shot scenarios. However, they lack an in-depth investigation into the challenges of generating robust forecasts for different horizons. To be specific, TimesFM [9] , a decoder-only Transformer, is capable of arbitrary-horizon forecasting, but this approach could potentially lead to substantial error propagation in long-term forecasting scenarios due to autoregressive decoding. DAM [8] , though free from this issue thanks to a novel output design composing sinusoidal functions, cannot effectively capture abrupt changes in time-series data, thereby limiting its utility in critical domains such as energy and traffic. Moreover, while MOIRAI [31] employs a full-attention encoder-only Transformer architecture and supports arbitrary-horizon forecasting via a non-autoregressive manner by introducing mask tokens into forecasting horizons, it remains uncertain how well MOIRAI adapts to different horizons. For example, its architecture design does not ensure the horizon-invariant property: the model output for a specific future position should be invariant to arbitrary extensions in forecasting horizons beyond that. Besides, its performance could drop significantly for moderate context lengths.To address this research gap, we introduce a comprehensive study to explore how to construct a time-series Transformer variant that can yield robust forecasts for varied inference horizons once trained. We name the developed model as Elastic Time-Series Transformer (ElasTST). ElasTST adopts a non-autoregressive design by incorporating placeholders into forecasting horizons, which is inspired by diffusion Transformers [24] and the success of SORA [3] in video generation. Here we impose structured self-attention masks, only allowing placeholders to attend to observed timeseries patches. This design ensures the aforementioned horizon-invariant property by blocking the information exchange across placeholders. Additionally, we devise a tunable version of rotary position embedding (RoPE) [28] to capture customized period coefficients for time series and to learn the adaptation to varied forecasting horizons. Furthermore, we introduce a multi-patch design to balance fine-grained patches beneficial to short-term forecasting with coarse-grained patches preferred by long-term forecasting, and use a shared Transformer backbone to handle these multi-scale patches. Alongside core model designs, during the training phase, we deploy a horizon reweighting approach that approximates the effects of random sampling across multiple training horizons using just one fixed horizon, eliminating the need for additional sampling efforts. Collectively, these key customizations facilitate ElasTST to produce consistent and accurate forecasts across various horizons.Our extensive experiments affirm the effectiveness of ElasTST in varied-horizon forecasting. First, we evaluated ElasTST, trained with a fixed horizon and employing a reweighting scheme, against state-of-the-art models trained for specific inference horizons. The results demonstrate that ElasTST delivers competitive performance without requiring per-horizon tuning. Then, we examined variedhorizon forecasting for these models, and the advantages of ElasTST are much more outstanding, demonstrating remarkable extrapolations to longer horizons while preserving robust results for shorter ones. Moreover, we also compared ElasTST with some pre-trained time-series models, such as TimesFM and MOIRAI, and found that dataset-specific tuning still offers prominent advantages over zero-shot inference in challenging datasets, such as Weather and Electricity, and that ElasTST can provide more robust performance across different forecasting horizons. At last, we conducted comprehensive ablation tests to highlight the significance of each unique design element of ElasTST.In summary, our contributions comprise:• Conducting a systematic study on varied-horizon forecasting, a critical requirement across various domains, yet an underexplored area in time-series research.• Developing a novel Transformer variant, ElasTST, which incorporates structured attention masks for horizon-invariance, tunable RoPE for time-series-specific periods, multipatch representations to balance fine-grained and coarse-grained information, and a horizon reweighting scheme to effectively simulate varied-horizon training.• Demonstrating the effectiveness of ElasTST through experiments comparing it with stateof-the-art time-series architectures and some up-to-date foundation models. Our ablation tests further reveal the importance of its key design elements. 
related work: 
 Traditional Neural Architecture Designs for Time-Series Forecasting The field of time-series forecasting has witnessed a significant evolution of neural architectures, transitioning from early multi-layer perceptrons [23] , convolutional [5] , and recurrent networks [6] , to a more recent focus on various Transformer variants [37, 40, 33, 32, 22, 20, 39] . However, the challenge of variedhorizon forecasting remains underexplored in these studies, as these models often require specific tuning to optimize performance for each inference horizon. Additionally, many models, including PatchTST [22] , iTransformer [20] , and MTST [39] , utilize horizon-specific projection heads, which inherently complicates the extension of their forecasting horizons.Developing Foundation Models for Time-Series Forecasting Inspired by the remarkable successes in the creation of foundational models in the language and vision domains [4, 25, 3] , the trend of pre-training universal foundation models has emerged in time-series forecasting research. Notable works in this area include Lag-Llama [26] , DAM [8] , TimesFM [9] , and MOIRAI [31] . These studies employ unique designs to address the challenges posed by varied variate numbers and forecasting horizons when adapting to new scenarios. Lag-Llama, DAM, and TimesFM adopted the univariate paradigm to circumvent the difficulties associated with handling different variates. In contrast, MOIRAI has taken a different approach by flattening multi-variate time series into a single sequence to facilitate cross-variate learning. While this method has its merits, it is worth noting that it may introduce efficiency issues when handling a substantial number of variates and long forecasting horizons. As a result, this paper also adopts the univariate setup to maintain efficiency. When it comes to varied forecasting horizons, Lag-Llama and TimesFM both utilized the decoder-only Transformer and relied on autoregressive decoding to manage arbitrarily long horizons. DAM introduced a novel output scheme that comprises numerous sinusoidal basis functions, enabling it to project into arbitrary future time points. MOIRAI, on the other hand, used a composite input scheme, combining observed time-series patches with variable placeholders that indicate forecasting horizons, and built a full-attention encoder-only Transformer on top of this. Interestingly, this non-autoregressive generation paradigm originates from diffusion transformers used in video generation [24, 3] . In this paper, we also embrace this paradigm for generating variable-length time-series. Unlike MOIRAI, which has made considerable strides in time-series pre-training using a moderately designed Transformer variant, our focus lies in systematically examining critical architectural enhancements to improve robustness in time-series forecasting across various horizons. We believe that constructing a more robust, resilient, and universal architecture will pave the way for more powerful foundational time-series models to be pre-trained in the future.Position Encoding in Time-Series Transformers Position encoding plays a pivotal role in Transformers as both self-attention and feed-forward modules lack inherent position awareness. The majority of existing time-series Transformer variants have roughly adopted absolute position encoding [30] with minor modifications across different studies. For instance, Informer [40] and Pyraformer [19] have combined fixed absolute position embeddings with timestamp embeddings such as day, week, hour, minute, etc. Meanwhile, Autoformer [33] and Fedformer [41] have omitted absolute position embeddings and relied solely on timestamp embeddings. Other models like Log-Trans [18] and PatchTST [22] have explored learnable position embeddings. However, the challenge with absolute position embedding is its inability to extrapolate into unseen horizons, posing a significant challenge for varied-horizon forecasting. To address this issue, MOIRAI has utilized a relative position embedding technique, RoPE [28] , which has been broadly adopted in the language domain to handle variable-length sequences [29] . In our work, we also adopt RoPE to introduce relative position information into self-attention operations. What we uniquely reveal is that the direct application of the RoPE configuration from the language domain to time-series forecasting is not ideal. The reason being that the predefined coefficients do not align well with the typical periodic patterns observed in time-series data. As a solution, we suggest redefining the period range encompassed by the initial RoPE coefficients and making data-driven adjustments to these coefficients. tunable RoPE customized to time series periodicities, enhancing its robustness. The architecture also integrates a (c) multi-scale patch assembly that merges fine-grained and coarse-grained details for improved forecasting accuracy. Furthermore, we implement (d) training horizon reweighting scheme during the training phase, which effectively simulates random sampling of forecasting horizons, reducing the need for additional sampling efforts.Input Patches in Time-Series Transformers PatchTST [22] spearheaded the concept of segmenting time-series data into patches instead of feeding raw time-series values directly into Transformer models. This straightforward yet effective approach has been widely adopted in subsequent studies, including MTST [39] , TSMixer [10] , HDMixer [17] , and MOIRAI. It noteworthy that MOIRAI has been trained with a diverse range of time-series patches with varying patch sizes. When adapting it to a new dataset, practitioners need to search through a range of patch sizes and rely on validation performance to select a single patch size. In our work, however, we have demonstrated that segmenting time series into multiple patch sizes to create multi-scale patch representations is more advantageous. This approach further aids in stabilizing accurate forecasting across various horizons. 
elastic time-series transformers: 
 In Figure 1 , we present an overview of ElasTST. Different from other encoder-only Transformer architectures, ElasTST equipped three core designs to facilitate varied-horizon forecasting: structured self-attention masks for placeholders, tunable rotary position embedding (TRoPE) with custimized period coefficients, and a multi-scale patch representation learning. Additionally, we utilize a horizon reweighting scheme to achieve the effects of varied-horizon training.Notations We define a univariate time series as x 1:T = {x t } T t=1 , with x t ∈ R indicating the value at time index t. The learning objective of a varied-horizon forecasting can be formulated as: , (t, L, T )∼p(T) log p ϕ (x t+1:t+T |x t-L+1:t ), where p(D) is the data distribution from which time series samples are drawn, and p(T) is the task distribution, from which the timestamp t, look-back window L, and the prediction horizon T are sampled.max ϕ E x∼p(D)Model Inputs To accommodate varied forecast horizons, our model combines the historical context series x t-L+1:t with placeholders 0 ∈ R T through concatenation, forming the input X = Concat(x t-L+1:t , 0). This approach allows for flexible adjustment of the input and output dimensions to suit different forecasting scenarios. We further segment X into non-overlapping patches X p ∈ R N ×P , where P is the patch length and N = (L+T ) P represents the number of patches. Each input patch is then transformed into latent space by the encoder H = Enc(X p ), H ∈ R N ×D . Masked Self-Attention A robust varied-horizon forecasting method should deliver consistent outputs across different forecasting horizons while maintaining high accuracy on unseen horizons. Existing time series Transformers, however, typically directly adapt techniques from video generation and natural language processing without considering the unique characteristics of time series. To address this deficiency, ElasTST modifies a standard Transformer Encoder with two crucial enhancements: structured attention masks and a tunable RoPE to encode relative position information effectively. We formulate the attention scores within a masked self-attention asEQUATIONwhere W q , W k ∈ R D×d denote the linear mappings for the query and key, respectively. A tunable RoPE f TRoPE dynamically adjusts the relative position encoding manner to best suit each dataset, with further details provided in the following subsection. The structured attention mask M •,n is set to 0 for patches X p n consisting solely of placeholders and 1 otherwise, ensuring that tokens attend only to context-carrying patches. This structured masking, in conjunction with the relative position encoding, prevents the influence of placeholders on prediction outcomes, thus ensuring consistent outputs across varied forecasting horizons.Tunable Rotary Position Embedding Position embedding is crucial for the attention mechanism to maintain accuracy over unseen horizons. To overcome the limitations of absolute position embedding in extrapolation scenarios, RoPE has been widely adopted in the NLP domain for handling variable-length sequences. It rotates a vector x ∈ R d onto an embedding curve on a sphere in C d/2 , with the rotation parameterized by a base frequency b. The function is defined as [34] . Typically in NLP, the base frequency b is set to a constant, such as 10,000. However, due to the unique characteristics of time series data, specific adaptations of RoPE are necessary. In this paper, we propose to use the period coefficients P j = 2π b -2(j-1)/d for parameterization:f RoPE (x, t) j = (x 2j-1 + ix 2j )e ib -2(j-1)/d t , where j ∈ [1, 2, ..., d/2]EQUATIONwhere P min and P max represent the predefined minimum and maximum period coefficients, respectively. This formula maintains an exponential distribution but adjusts the range to better align with the periodic characteristics of time series data. By setting P min = 2π and P max = 2πb 1-2 d , this approach mirrors the original RoPE setup.In addition to adjusting the period range, the distinct and varying periodicities inherent in time series data necessitate more flexible period coefficients. Therefore, in ElasTST, we consider period coefficients P as tunable parameters, optimizing it along with varied datasets and forecasting horizons. This adaptive approach allows for more precise and effective forecasting across diverse conditions. We provide a detailed exploration of this design in Section D.4, and illustrate the optimized period coefficients for each dataset in Appendix E.2.To ensure robust performance across various forecasting horizons, integrating both fine-grained and coarse-grained features from time series data is essential. Different from earlier multi-patch models that utilize separate processing branches for each patch size [39] , ElasTST features a multi-scale patch design within a shared Transformer backbone, capable of both parallel and sequential processing. We chose sequential processing for our implementation, keeping the memory consumption comparable to baselines such as PatchTST. The implications of this design on memory usage are further discussed in Appendix F. Specifically, we define each patch size as p = {p 1 , . . . , p S }, with each size corresponding to a dedicated MLP encoder f Enc pi : R pi → R D and decoder f Dec Training Horizon Reweighting To effectively manage varied forecasting horizons, training models across multiple horizon lengths, rather than using fixed ones, is a practical approach [31] . In this study, we propose to use reweighting scheme for loss computation that simulates this process, without the need for additional sampling efforts. Formally, in the conventional implementation, at each training step s, a forecasting horizon T s is randomly selected from the range [1, T max ]. 1 Then the loss L s at step s is computed as:EQUATION)Theoretically, the expectation of this random sampling process can be represented as a weighted loss over a fixed horizon T max . To be specific, the expected value of ω(τ ) is calculated as:E[ω(τ )] = 1 Tmax ∑ Tmax T =1 1T . We further approximate the reweighting function by harmonic series as:EQUATIONBy employing this weighted loss ω(τ ) during training, we replicate the effect achieved by randomly sampling horizons at an infinite number of training steps. In addition, the function ω(τ ) can be adapted to follow any desired distribution family and can be made differentiable. 
experiments: 
 To validate the effectiveness of ElasTST, we systematically assess its performance across various forecasting scenarios, benchmarking it against established models. The results, detailed in Section 4.1, showcase ElasTSTs adaptability to diverse forecasting horizons. Subsequently, we perform an extensive ablation study in Section 4.2 to examine the impact of its key designs. 2Datasets Our experiments leverage 8 well-recognized datasets, including 4 from the ETT series (ETTh1, ETTh2, ETTm1, ETTm2), and others include Electricity, Exchange, Traffic, and Weather. These datasets cover a wide array of real-world scenarios and are commonly used as benchmarks in the field. Detailed descriptions of each dataset are provided in Appendix C.1. Following the setup described in [33] , all models use a standard lookback window of 96, except TimesFM [9] and MOIRAI [31] , which utilize extended lookback windows of 512 and 5000, respectively.Baselines For our comparative analysis, we select 6 representative forecasting models as baselines:(1) Advanced but non-elastic forecasting models, such as iTransformer [20] , PatchTST [22] , and DLinear [36] ; (2) Autoformer [33] , which supports varied-horizon forecasting but requires horizonspecific tuning; (3) the cutting-edge time series foundation model like TimesFM [9] and MOIRAI [31] , which are pre-trained for general-purpose forecasting across varied horizons. Our analysis primarily assesses the varied-horizon forecasting capabilities, considering their pre-training on subsets of the datasets used.Implementation ElasTST is implemented using PyTorch Lightning [12] , with a training regimen of 100 batches per epoch, a batch size of 32, and a total duration of 50 epochs. We use the Adam optimizer with a learning rate of 0.001, and experiments are conducted on NVIDIA Tesla V100 GPUs with CUDA 12.1. To ensure fairness, we conducted an extensive grid search for critical hyperparameters across all models in this study. The range and specifics of these hyperparameters are documented in Appendix C.2. For parameters not mentioned in the table, we adhered to the best practice settings proposed in their respective original papers. For evaluation, we use Normalized Mean Absolute Error (NMAE) and Normalized Root Mean Squared Error (NRMSE) as they are scale-insensitive and widely accepted in recent studies [23] . More details are in Appendix C.3.Comparing ElasTST with Horizon Reweighting to Neural Architectures Tuned for Specific Inference Horizons Experimental results demonstrate that ElasTST consistently delivers exceptional performance across all horizons without the need for per-horizon tuning. As evidenced in Table 1, ElasTST outperformed SOTA models on diverse datasets including ETTm1, ETTh1, ETTh2, Traffic, Weather, and Exchange, despite these models undergoing specific horizon-based training and tuning. This clearly demonstrates ElasTST's inherent robustness and its remarkable capacity to generalize effectively across varied forecasting scenarios. Comparing the Robustness of Different Models for Varied Inference Horizons The results clearly position ElasTST as the most robust option for deploying a single, well-trained model across various inference horizons and application scenarios. As demonstrated in Figure 2 , ElasTST consistently maintains strong performance across both seen and unseen horizons, underscoring its ability to navigate beyond trained scopes with consistent accuracy across a wide range of forecasts.In contrast, other models face significant challenges in varied-horizon forecasting. State-of-the-art models like iTransformer and PatchTST excel within their trained horizons but struggle when extended beyond these limits. Models that require horizon-specific tuning, such as Autoformer, often experience abrupt declines in performance, illustrating that scalability alone is insufficient without tailored optimization. TimesFM, with its autoregressive nature, shows substantial error propagation in unseen datasets like ETT and Exchange, and increased errors in pre-trained datasets such as Weather as the horizon extends. While MOIRAI demonstrates strong zero-shot performance on datasets like ETT and Exchange, we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages. Furthermore, MOIRAIs performance significantly diminishes with shorter context lengths, as discussed in their paper [31] . In comparison, ElasTST operate effectively with much shorter context lengths. Structured Attention Masks The ablation study confirms that structured attention masks are essential for robust inference across horizons that differ from the training phase. As illustrated in Figure 3 , removing structured masks from ElasTST results in significant performance declines, particularly in the Weather dataset. Furthermore, as demonstrated in Figure 7 (see Appendix D.2), the benefits of structured masks are consistent across all forecasting horizons. This underscores the importance of the horizon-invariant property for enhancing the stability of time series forecasting, an aspect often overlooked in current research.Tunable Rotary Position Embedding Experimental results indicate that tunable RoPE significantly improves the models ability to extrapolate. Figure 4a shows that while other positional embedding methods are effective on seen horizons, they falter when applied to horizons extending beyond the training range. Although the original RoPE excels in NLP tasks, it underperforms in time series forecasting. Besides, data-driven adjustments of these coefficients enable far more robust extrapolation. Dynamically tuning RoPE parameters according to the periodic patterns of the dataset proves highly beneficial, especially when inferring over unseen horizons.Furthermore, a range from 1 to 1000 for the period coefficients P is more suitable for time series forecasting. As demonstrated in Figure 4b , using the commonly-used NLP settings with P min = 1 and P max = 10000 does not fully exploit the potential of RoPE in time series forecasting. Setting P max to 1000 results in better performance. We hypothesize that this is because, unlike textual data which benefits from attention over longer contexts, the time series data, especially when segmented into patches, benefits from focusing on shorter, more recent intervals. By adjusting the maximum period coefficient to a lower value, the model captures a richer spectrum of mid-to-high frequency patterns, thereby enhancing its effectiveness. Detailed analyses of these findings are available in Appendix D. 4 . Appendix E includes visualizations demonstrating how different initial ranges impact frequency components, along with illustrations of the tuned period coefficients for each dataset. Multi-Patch Design These experiments demonstrate that multi-patch configurations generally outperform single patch sizes across various forecasting horizons. Figure 5 shows that the configuration p = {8, 16, 32} consistently achieves the lowest NMAE values, effectively balancing the capture of short-term dynamics and long-term trends. However, adding larger patches, such as p = {8, 16, 32, 64}, does not consistently improve performance and can sometimes increase the NMAE. This suggests that more complex configurations may not always provide additional benefits and could even be counterproductive.Moreover, the patch size selection is particularly critical in the varied-horizon forecasting scenarios.As demonstrated in the Figure 10 selecting a fixed horizon and mitigating the uncertainties associated with random sampling. Crucially, this training approach is model-agnostic and can be applied to different forecasting training scenarios. These results also highlight the advantages of a flexible forecasting architecture, which allows training horizons to be customized to the unique characteristics of each dataset.We also observe that different datasets have distinct preferences for training horizons. For example, in the Exchange dataset, the longest training horizon led to worse results compared to a shorter horizon of 96, suggesting risks of overfitting or forecast instability with prolonged horizons. Besides, in the ETTh1, employing random sampling for training horizons proved suboptimal. These insights show that tailoring the training horizon selection strategy to the specific dataset can yield improvements. One potential enhancement could involve dynamically optimizing the horizon reweighting scheme alongside model training. 
conclusion: 
 This study introduces the Elastic Time-Series Transformer (ElasTST), a pioneering model designed to tackle the significant and insufficiently explored challenge of varied-horizon forecasting. ElasTST integrates a non-autoregressive framework with innovative elements such as structured self-attention masks, tunable Rotary Position Embedding (RoPE), and a versatile multi-scale patch system. Additionally, we implement a training horizon reweighting scheme that simulates random sampling of forecasting horizons, thus eliminating the need for extra sampling efforts. Together, these elements enable ElasTST to adapt to a wide range of forecasting horizons, delivering reliable and competitive outcomes even when facing horizons that were not encountered during the training phase.Limitations While ElasTST demonstrates robust performance across various forecasting tasks, several limitations have been identified that highlight opportunities for future enhancements. First, the current version of ElasTST does not incorporate a pre-training phase, which could significantly improve the models initial grasp of time-series dynamics and boost its efficiency during task-specific fine-tuning. Further exploration is needed to ascertain optimal training methodologies that maximize the architectural benefits of ElasTST. Additionally, while the training horizon reweighting scheme is straightforward and effective in enhancing performance across different inference horizons, it is not the optimal solution for all datasets. Moreover, the evaluation of ElasTST is limited to a select number of datasets, which may not fully represent the broader challenges encountered in more complex or diverse real-world scenarios. 
 
 Appendix: In response to these limitations, our forthcoming research efforts will concentrate on developing and validating pre-training protocols for ElasTST to elevate its foundational performance and extend its applicability across universal forecasting tasks. We aim to incorporate a reasonable training approach that will fine-tune the models ability to seamlessly manage forecasts of varying lengths, thus bolstering its utility in dynamic real-world environments. Furthermore, by broadening the range of datasets used for model evaluations, we intend to rigorously test ElasTSTs effectiveness across an expanded spectrum of industry-specific challenges. This comprehensive approach will not only solidify ElasTSTs standing as a cutting-edge solution for time-series forecasting but also enhance our understanding of its practical implications and potential in diverse industrial applications.Rotary position embedding (RoPE) [28] is a method used to encode the position of tokens in the input sequence for transformer-based models, enhancing the capability to utilize the positional context of tokens. RoPE uniquely incorporates the geometric property of vectors, transforming them into a rotary matrix that interacts with the vector embeddings.Here, we have adopted the formal definition from the original RoPE paper. In the simplest twodimensional (2D) case, RoPE considers a dimension d = 2, where each position vector is treated in its complex form. The formulation is given by:f q (x m , m) = (W q x m )e imθ , f k (x n , n) = (W k x n )e inθ , g(x m , x n , m -n) = Re[(W q x m )(W k x n ) * e i(m-n)θ ].This expression shows that the embedding rotates the affine-transformed word embedding vectors by angle multiples relative to their position indices. This rotation is mathematically represented through a multiplication matrix:f q,k (x m , m) = [ cos mθ -sin mθ sin mθ cos mθ ] [ W (11) q,k W (12) q,k W (21) q,k W (22) q,k ] [ x (1) m x (2) m ] .For a generalized form in any dimension d, RoPE divides the space into d/2 sub-spaces and combines them to utilize the linearity of the inner product. The generalized rotary matrix R d Θ,m is defined as: R d Θ,m =           cos mθ 1 -          , where Θ = {θ i = 10000 -2(i-1)/d , i ∈ [1, 2, ..., d/2]}.This formulation ensures that RoPE is computationally efficient and stable due to the orthogonality of R d Θ,m . The use of sparse matrices further improves computational efficiency, making RoPE a practical approach to incorporate in largescale transformer models.We summarize existing time series foundational models in Table 2 , excluding the LLM-oriented ones [35] . These models typically use standard architecture designs, position encodings, and patching approaches, primarily aiming to enhance transferability in zero-shot scenarios. However, they generally do not deeply explore the challenges of producing robust forecasts across varied horizons. Our work specifically addresses this gap by improving model design to enhance robustness for varied-horizon forecasting.Our experiments utilize 8 widely recognized datasets, including 4 from the ETT series (ETTh1, ETTh2, ETTm1, ETTm2), as well as the Electricity, Exchange, Traffic, and Weather datasets. These TimeGPT-1 [14] Enc-Dec Transformer AR Abs PE -Lag-Llama [26] Decoder-only Transformer AR RoPE -Chronos [1] Enc-Dec Transformer AR Simplified relative PE Quantization Timer [21] Decoder-only Transformer AR Abs PE Patching TimesFM [9] Decoder-only Transformer AR Abs PE Patching UniTS [13] Transformer Encoder NAR Learnable PE Patching DAM [8] Transformer Encoder NAR Abs PE ToME Tiny Time Mixers [11] TSMixer NAR -Patching MOIRAI [31] Transformer Encoder NAR RoPE Patching MOMENT [15] Transformer Encoder NAR Learnable relative PE Patching datasets encompass a broad range of real-world applications and are frequently used as benchmarks in the field 3 . Consistent with common practices in long-term forecasting [33, 36, 22, 20] , all models are tested under the forecasting horizons T ∈ {96, 192, 336, 720}. Except for TimeFM [9] , which uses a lookback window of 512, a standard lookback window of 96 is employed across all other models as [33] . ElasTST is implemented using PyTorch Lightning [12] . Training consists of 100 batches per epoch, capped at 20 epochs, with the NMAE metric used for model checkpointing. We use the Adam optimizer with a learning rate of 0.001, and experiments are conducted on NVIDIA Tesla V100 GPUs with CUDA 12.1. The code for Transformer Block is adapted from [38] .Baselines We select five representative forecasting models as our baselines:• iTransformer4 [20] : A transformer-based model that inverts the dimensions of time and variates to effectively capture multivariate correlations, enhancing generalization across different variates. • PatchTST5 [22] : A transformer-based model segmenting time series into subseries-level patches and employing channel-independent processing to improve forecasting accuracy. • DLinear6 [36] : An MLP-based model that has demonstrated superior performance over more complex transformer-based models in multiple real-life datasets. • Autoformer7 [33] : Integrates a novel Auto-Correlation mechanism that exploits series periodicity for enhanced dependency discovery and representation aggregation. • TimeFM8 [9] : A foundation model for time series forecasting, pretrained using a decoderstyle attention mechanism and input patching on an extensive corpus of real-world and synthetic data.• MOIRAI (UNI2TS) 9 [31]: A foundation model for time series forecasting, pretrained using a masked encoder-based Transformer on the extensive Large-scale Open Time Series Archive (LOTSA) with over 27 billion observations.Hyper-parameter Tuning To ensure fairness, we conducted an extensive grid search for critical hyperparameters across all models in this study. The range and specifics of these hyperparameters are documented in Table 4 . For parameters not mentioned in the table, we adhered to the best practice settings proposed in their respective original papers. The scalable architecture of ElasTST allows it to treat the training horizon as a hyperparameter. This adaptability prompts us to evaluate performance across various training and inference horizons (see Figure 8 ), yielding several interesting insights.• Model performance deteriorates as the forecasting horizon increases, particularly in models trained on shorter horizons, as seen in the ETT and Electricity datasets. This pattern suggests the importance of training models on extended horizons to capture adequate contextual information.• Tuning models specifically for a given horizon does not guarantee improved performance on that horizon, as noted in the Weather dataset. This indicates that optimal model settings depend significantly on dataset-specific characteristics, and horizon-specific tuning may not be a reliable strategy.• The longest training horizons do not always produce the best forecasting results. In the Exchange dataset, for example, the longest horizon yielded poorer results compared to a shorter training horizon of 96. This points to the potential risks of overfitting or forecast instability when training with long-term series only.These observations underscore the importance of tailoring training horizons to the unique characteristics of each dataset and underscore the benefits of an architecture designed for elastic forecasting. Furthermore, they suggest the potential advantages of implementing a mixed-horizon training strategy, which leverages multiple horizons to produce more resilient forecasts.Beyond its scalable architecture, the position embedding plays a crucial role in enhancing the elasticity of ElasTST. We analyze the Tunable RoPE in ElasTST by examining the effects of the initialization of period coefficients, specifically P min and P max , and the benefits of parameter optimization during the training process.Experimental results, presented in Figure 9 , indicate that using settings similar to the commonlyused one in NLP, with P min = 1 and P max = 10000, does not fully exploit its potential in time series forecasting. This discrepancy stems from fundamental differences between the data types: in text, discrete tokens are the smallest units, requiring attention over longer contexts, while time series data, particularly when patched, may benefit from focusing on shorter, more recent tokens.Furthermore, our findings reveal that tuning parameters in RoPE during training significantly improves forecasting accuracy, particularly over varying and extended horizons. When the training horizon is set to 96, a tunable feature shows minimal impact, suggesting that a short-seen horizon does not facilitate learning effective period coefficients. However, as the training horizon extends, the advantages of a tunable theta become more pronounced, especially for unseen horizons.These results emphasize the importance of customizing RoPE's period coefficients settings and utilizing tunable RoPE to enable flexible and accurate forecasting in time series analysis. Appendix E offers visualizations demonstrating how different initial ranges impact the frequency components, along with a detailed analysis of the distribution of RoPE periods optimized for each dataset.These experiments highlight the critical impact of patch size selection, particularly in varied-horizon forecasting scenarios. As demonstrated in Figure 10 , various combinations of training and forecasting horizons exhibit distinct preferences for patch sizes. For instance, when the training forecasting horizon is 720, during the inference stage, longer forecasting horizons prefer larger patch sizes. Conversely, on shorter training horizons, such as 96 and 192, choosing large patch sizes for longer horizons can lead to performance collapse. This difference underscores the complexity and necessity of optimal patch size selection in achieving effective elastic forecasting. Moreover, Figure 10 clearly show that multi-patch configurations typically surpass single patch sizes across most forecasting horizons. The configuration p = {8, 16, 32} consistently offers the lowest NMAE values, striking an optimal balance between capturing short-term dynamics and longterm trends. However, introducing larger patches (p = {8, 16, 32, 64}) does not always enhance performance and can sometimes increase the NMAE, indicating that overly complex configurations may not yield additional benefits and could be counterproductive.These findings emphasize the advantages of employing multi-patch configurations to improve the accuracy of varied-horizon forecasting. They also highlights the importance of carefully selecting patch size combinations to optimize performance and minimize computational expenses.To provide a more intuitive visualization of the initial distribution of periodicity coefficients, we present this in Figure 11 .In Figure 12 , we present the distribution of tuned periodicity coefficients for each dataset. While our model uses a shared Transformer backbone to process all patch sizes, this can be done either in parallel or sequentially, depending on whether shorter computation times or lower memory usage is prioritized. In practice, we chose the sequential approach, where each patch size is processed individually, and the total forecast is assembled afterward. This approach ensures that the memory bottleneck depends on the smallest patch size used.As indicated in Table 8 , memory usage is primarily influenced by the smallest patch size, not by the number of patch sizes employed. The additional parameters introduced by using multiple patch sizes are almost negligible. For example, the multi-patch setting 8,16,32 used in the paper requires the same maximum memory as using a single patch size of 8. Under resource constraints, the minimum patch size can be adjusted to balance model performance and memory usage.The look-back window L is fixed.Unless stated otherwise, horizon reweighting scheme is deactivated in ablation study.Datasets are available at https://github.com/thuml/Autoformer under MIT License.https://github.com/thuml/iTransformer, MIT License.https://github.com/yuqinie98/PatchTST, Apache-2.0 license.https://github.com/cure-lab/LTSF-Linear, Apache-2.0 license.https://github.com/thuml/Autoformer, MIT License.https://github.com/google-research/timesfm, Apache-2.0 license.https://github.com/SalesforceAIResearch/uni2ts, Apache-2.0 license. 


Decision: This paper proposes a non-autoregressive architecture for varied-horizon forecasting.  The approach incorporates structured attention masks through patching and rotary position embeddings (RoPE) to encode relative positional information for improved forecasting. This paper addresses a critical issue in time series forecasting and the proposed method is novel and effective. There are concerns about the scalability of the proposed method, but the authors have provided reasonable responses to address these concerns. Overall, this is a solid paper and I recommend it for acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces ElasTST, a novel approach designed to enhance robustness in forecasting across various horizons. The proposed architecture dynamically adapts to different forecasting horizons, addressing a significant challenge in the literature. Previous methods typically relied on recursive approaches or utilized masks that failed to maintain horizon invariability. ElasTST aims to achieve high forecasting accuracy while ensuring adaptability across different horizons by employing a transformer encoder as its backbone. The approach incorporates structured attention masks through patching and rotary position embeddings (RoPE) to encode relative positional information for improved forecasting.
strength_weakness: strength_weakness: 1. The literature review on foundational models is lacking. Notable models such as TimeGPT-1 [1], Chronos [2], TinyTimeMixers [3], and Moment [4] are not discussed.
2. A traditional encoder transformer is not included as a baseline, which would help demonstrate the improvements offered by the proposed method. Additionally, NLP-based architectures such as NHITS [5] and TsMixer [6] are absent from the comparisons.
3. The paper does not adequately address the computational scalability of ElasTST when applied to very large datasets or extremely long time series.
4. The authors modified the traditional metrics used in long-horizon literature (MSE and MAE) to scaled metrics. They argue this adjustment reports fairer results, but it overlooks the fact that these datasets are typically already scaled, making additional scaling in the metric unnecessary.
5. The paper lacks a detailed discussion on potential failure cases or scenarios where ElasTST might underperform, which would provide a more balanced view of its applicability.

[1] https://arxiv.org/abs/2310.03589
[2] https://arxiv.org/abs/2403.07815v1
[3] https://arxiv.org/abs/2401.03955
[4] https://arxiv.org/abs/2402.03885
[5] https://arxiv.org/abs/2201.12886
[6] https://arxiv.org/abs/2303.06053
questions: questions: 1. What is the computational complexity added by the Rotary Position Embeddings (RoPE)?
2. In some cases, the results are very similar to PatchTST, such as with the ETTm2 dataset. Is there an intuition or explanation regarding the types of data that explain these similarities and differences in performance?
3. It would be interesting to compare the performance gains for each point in the forecasting window. Are the gains through the whole window or in the earliest/latest points?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces ElasTST, which aimed at addressing the challenge of varied-horizon time-series forecasting. ElasTST integrates a non-autoregressive architecture with placeholders for forecasting horizons, structured self-attention masks, and a tunable rotary position embedding. Furthermore, it employs a multi-scale patch design to incorporate both fine-grained and coarse-grained temporal information, enhancing the model’s adaptability across different forecasting horizons. Experiments in the paper demonstrate the model's performance in providing robust predictions when extrapolating beyond the trained horizons.
strength_weakness: strength_weakness: - The newly proposed tunable RoPE module lacks comparative analysis with previous methods. A comparison with methods such as no positional embedding, vanilla positional encoding, trainable positional embedding matrices, and original rotary positional embeddings, would clarify the benefits of different approaches. Since tunable RoPE appears easily transferable, demonstrating its performance enhancement by applying it to previous models like PatchTST and Autoformer would provide more intuition.

- Using longer sequence length significantly increases the computational cost in Transformers. Comparing with baselines like PatchTST, the paper uses multiple patch sizes as input processing method, which significantly increases computational costs. Further analysis like how much adding a patch size can increase computational costs or what patch size combination are more reasonable to choose under resource constraints would be beneficial.
questions: questions: - Given the significant increase in computational cost from longer token sequences, would independently inputting each patch size into the transformer layers, instead of all at once, reduce memory usage while maintaining model performance?

- With multiple patch sizes used in the study, how are the RoPE applied? Are they applied independently to tokens from each patch size, or collectively to all tokens from different patch sizes?

- Ablation studies indicate that incorporating smaller patch sizes seems to benefit model performance substantially. If my understanding is correct, when the patch size is 1 (excluding tunable RoPE), the structure is similar to the decoder part of models like Informer and Autoformer. Would using a patch size of 1 worsen the model performance? Would adding "1_" to the existing best patch size setting "8_16_32" improve performance? Using patch size 1 turns the model into a channel-independent Informer-decoder-like model, which significantly increases computational costs, hence it you don't have to consider this on large datasets.
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces a masked encoder based transformer model for time series forecasting. It performs masking in the observation space, replacing the prediction horizon with 0s. It further applies an attention mask such that tokens only attend to the context, and do not attend to the masked tokens. The paper also uses a tunable version of rotary positional embeddings, and ensembles predictions over various patch sizes.
strength_weakness: strength_weakness: Overall, the proposed approach is very similar to Moirai, with several minor changes:

i) Perform the masking operation in observation space

ii) Add an attention mask
 
iii) Tunable rotary embeddings
 
iv) Ensembling results over multiple patch sizes

However, it seems that even more features which Moirai had were removed, making it a weaker proposition. Such features include probabilistic predictions, multivariate capabilities, and any context length. Furthermore, few experiments are done to show the benefits of the changes, such as (i), (ii), and (iv). 

Others:
1. The statement, "we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages" (L238-L239) is a little eyebrow raising, in the sense that it is stating the obvious. I believe most researchers expect in-distribution predictions to outperform zero-shot predictions. Such statements should be rephrased.
2. Effects of tuning rotary embeddings in figure 4 seem very marginal, error margins should be provided for such cases.
questions: questions: None
limitations: limitations: Authors have provided limitations of their work.

Comments: title: Response to Official Comment by Reviewer i9JA
author reviewer discussion
comment: Thanks to the authors for the rebuttal. I will keep my positive rating.
Dear Reviewer i9JA,

Thank you for your response, and we appreciate the opportunity to address your concerns. We believe that our previous responses have covered these issues.

> Moirai has addressed this with a simple approach - during training, train with multiple horizon lengths. This seems to be a simple method which should be compared to.
> 

MOIRAI’s training approach (with maximum forecasting horizon of 256) does not guarantee robust extrapolation capabilities, while the design in ElasTST is specifically aimed at improving performance on unseen horizons.

As shown in Figures 1 and 3 in the global response PDF, stable performance may be achieved within the training range (1-336) even without certain design elements, but these elements become crucial when dealing with longer, unseen horizons. Simply training with multiple horizon lengths alone does not ensure robustness over extended inference horizons.

> Also, I do not see evidence in the paper indicating that the structured mask is critical for varied horizon prediction.
> 

The structured self-attention masks ensure consistent outputs when inference horizons differ from the training horizon. This is evidenced in the global response PDF.

Specifically, Figure 1 shows that removing the structured attention mask (w/o Mask) can lead to issues when forecasting horizons outside the training range, particularly noticeable in the weather dataset. Figure 2 further illustrates that the absence of the structured mask leads to a performance drop for both shorter and longer inference horizons compared to the full model (ElasTST).

We will incorporate these experimental results and analyses into future revisions to clarify our contributions. We hope our response has addressed your concerns, and we welcome any further questions or suggestion you may have.

Best Regards,

All Authors
Thank you authors for the rebuttal. I understand that this paper focuses on the varied horizon setting. Moirai has addressed this with a simple approach - during training, train with multiple horizon lengths. This seems to be a simple method which should be compared to. Also, I do not see evidence in the paper indicating that the structured mask is critical for varied horizon prediction. As such, my rating remains.
Dear reviewer,

The author-reviewer discussion ends soon. If you need additional clarifications from the authors, please respond to the authors asap. Thank you very much.

Best,

AC