WAVE: Weighted Autoregressive Varying Gate for Time Series Forecasting
Authors: Jiecheng Lu
Xu Han
Yan Sun
Shihao Yang
Venue: ICML 2025 poster (2025)
Keywords: Attention
Transformer
Autoregressive Moving-average
Time Series Forecasting
Primary area: deep_learning->sequential_models_time_series

Abstract: We propose a Weighted Autoregressive Varying gatE (WAVE) attention mechanism equipped with both Autoregressive (AR) and Moving-average (MA) components. It can adapt to various attention mechanisms, enhancing and decoupling their ability to capture long-range and local temporal patterns in time series data. In this paper, we first demonstrate that, for the time series forecasting (TSF) task, the previously overlooked decoder-only autoregressive Transformer model can achieve results comparable to the best baselines when appropriate tokenization and training methods are applied. Moreover, inspired by the ARMA model from statistics and recent advances in linear attention, we introduce the full ARMA structure into existing autoregressive attention mechanisms. By using an indirect MA weight generation method, we incorporate the MA term while maintaining the time complexity and parameter size of the underlying efficient attention models. We further explore how indirect parameter generation can produce implicit MA weights that align with the modeling requirements for local temporal impacts. Experimental results show that WAVE attention that incorporates the ARMA structure consistently improves the performance of various AR attentions on TSF tasks, achieving state-of-the-art results.

WAVE: Weighted Autoregressive Varying Gate for Time Series Forecasting
Jiecheng Lu"
Xu Han
Yan Sun
Shihao Yang
Abstract
Liu et al., 2024a), and audio (Borsos et al., 2023). This structure is well-suited for various sequential generation and prediction tasks. However, in typical sequence model- ing tasks like time series forecasting (TSF), there has been less exploration of this architecture compared to other struc- tures. Most of the best-performing recent TSF models are encoder-only Transformers (Liu et al., 2024b; Nie al., 2022), MLPs (Das et al.. 2023: Lu et al., 2024), or even linear models (Zeng et al., 2023; Xu et al., 2024). The few relevant discussions mainly focus on using pretrained au- toregressive LLMs or similar structures for few-shot and zero-shot prediction (Gruver et al., 2023; Jin et al., 2024; Das et al., 2024; Liu et al., 2024c), with little research di- rectly evaluating their TSF performance end-to-end train- ing. Therefore, this paper will first briefly demonstrate that with appropriate tokenization and training methods, basic AR Transformer is enough to achieve results comparable to the state-of-the-art (SOTA) baselines, as shown in Fig. 1.
We propose a Weighted Autoregressive Varying gatE (WAVE) attention mechanism equipped with both Autoregressive (AR) and Moving-average (MA) components. It can adapt to various atten- tion mechanisms, enhancing and decoupling their ability to capture long-range and local temporal patterns in time series data. In this paper, we first demonstrate that, for the time series fore- casting (TSF) task, the previously overlooked decoder-only autoregressive Transformer model can achieve results comparable to the best base- lines when appropriate tokenization and training methods are applied. Moreover, inspired by the ARMA model from statistics and recent advances in linear attention, introduce the full ARMA structure into existing autoregressive attention mechanisms. By using an indirect MA weight generation method, we incorporate the MA term while maintaining the time complexity and pa- rameter size of the underlying efficient attention models. We further explore how indirect parame- generation can produce implicit MA weights that align with the modeling requirements for lo- cal temporal impacts. Experimental results show that WAVE attention that incorporates the ARMA structure consistently improves the performance of various AR attentions on TSF tasks, achieving state-of-the- art results. The code implementation is available at the following link.
Recently, efficient linear autoregressive attention variants have been explored and developed (Katharopoulos et al., 2020; Hua et al., 2022), reducing the time complexity of standard softmax attention from O(N2) to O(N). Re- searchers have found that adding gating decay factor similar exponential moving average (EMA) structure to AR structure, as in gated linear attention (Ma et al., 2022; Yang et al., 2024), enhances linear attention's 's ability to model lo- cal patterns and improves performance. The success of these approaches inspired us to introduce a more comprehensive full autoregressive moving- (ARMA) structure into existing AR attention mechanisms and explore the perfor- mance of these Transformers in TSF.
1. Introduction
In TSF models, EMA, connecting back to the historic work of Winters (Winters, 1960; Holt, 2004), focuses on smoothed local data, which improves the modeling of short- term fluctuations but reduces the ability to capture long- term information. In contrast, ARMA, connecting back to the historic work of Box-Jenkins (Box et al., 1974), classic structure in TSF, considers both historical data and the cumulative impact of prediction errors. This allows it to handle and decouple long-term and short-term effects, significantly improving forecasting performance on data with complicated temporal patterns.
In recent years, autoregressive (AR) decoder-only Transformer-based models (Vaswani, 2017; Radford, 2018)
have been widely used in sequence modeling tasks across fields such NLP (Brown al., 2020; Touvron et al., 2023), CV (Chen et al., 2020; Esser et al., 2021; Chang et al., 2022;
'Georgia Institute of Technology AAWS. Correspondence to: Jiecheng Lu <jlu414@ @a44@a@ggacduu.... Shihao Yang <shi- hao.yang hao.yang@isse..atteea. wwt.syy..tcch..........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We propose the Weighted Autoregressive Varying gatE

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
1p
6p
hp
6p
Nean Ranking
Fixed Altn-ARMA
AMUSA Alteniion
EUn ARMA
AR Attention
Glin AEth-AAiMM
Baseline
AtEn-ARMA
ReiN
Fixed Attn
Elin Attn
Transtormer
Layes
Altn
Enchormer
Postion lirteding
Diinear PatchTSI
1p
ITransformer
FITS
16
10
Rankings
Figure 1. (Left: Overall architecture of decoder Transformer for TSE. (Right: b) Box plots of performance rankings from 48 sub-experiments across 12 datasets. Green represents WAVE Transformers, yellow AR Transformers, and red the baselines, with triangles indicating mean rankings, AR Transformers perform comparably baselines, while WAVE Transformers significantly outperform their AR counterparts. Sce Table and for more details.
(WAVE) attention mechanism equipped with ARMA struc- ture, which integrates moving-average (MA) term into various existing AR attention mechanisms. Our method improves the TSF performance of AR Transformers without significantly increasing computational costs, maintaining O(N) time complexity and original parameter size. We design an indirect MA weight generation obtain the MA output without explicitly computing the MA attention ma- trix, preserving efficiency of linear attentions. We explore specific techniques for generating implicit MA weights to ensure proper decoupling and handling of short-term ef- fects. Extensive experiments and visualization analyses demonstrate that ARMA balances long- and short-term de- pendencies, significantly improving AR Transformers and achieving tfate-of-the-art TSF results.
2. Method
2.1. Time series forecasting
In Time Series Forecasting (TSF), the goal is to predict the future part in multivariate time series S E R R where is the length of the series, and C' is the number of channels or input series. The time series is divided into historical input S, E RR, xC', and future data Sp E RLoxC, where L = L1 + Lp, and Lı and Lp represent the lengths of the input and forecasting periods, respectively. The objective is to learn mapping function f R - R Leex that predicts the future values Sp [Sr) given the historical input S₁.
2.2. Appropriate tokenization for autoregressive forecasting
The main contributions of this paper can be summarized as follows:
Recently, most time series forecasting research utilizes encoder-decoder or encoder-only Transformers for TSF (Li et al., 2019b; Zhou et al., 2021; Wu et al., 2021; Nie et al., 2022; Liu et al., 2024b), with limited focus on end-to-end decoder-only autoregressive Transformer because of error accumulation issue. For long-term forecasts, The autore- gressive Transformers requires iteratively doing one-step prediction, leading error accumulation and higher MSE compared to non-autoreeresssve models that generate the entire forecast at once.
a) We demonstrate that, with appropriate tokenization and preprocessing methods, an AR Transformer is enough to achieve the level of existing SOTA baselines. Furthermore, the introduction of WAVE attention enables the decoder- only Transformer to outperform SOTA baselines.
b) We propose the WAVE attention mechanism, which in- troduces an MA term into existing AR attention without increasing time complexity or parameter size. By adding the MA term to various AR attention mechanisms, the result- ing WAVE Transformers significantly improve forecasting performance compared to their AR counterparts.
To prevent error accumulation, we use an autoregressive Transformer (Fig. 1) that treats one-step prediction as the complete forecast. Inspired by PatchTST (Nie et al., 2022). adopt a channel-independemt approach, predicting each series separately and applying RevIN (Kim al., 2022) to each. For an input series of length L, in S,. we apply non- aanovverlapping patches with patch size Lp, dividing the input into N = Lp for divisibility. This ensures that each out-of-sample predic- tion token covers the entire forecasting length Ipp, thereby
c) We design an indirect MA weight generation method that is computationally efficient while ensuring that the implicit MA weights effectively capture the important short-term effects in TSF, allowing the AR term to focus more on long-term and eyclic patterns.
L,= patches, where P is zero- pardopgdding
2

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
avoiding error accumulation'.
term in standard attention with kernel function k(ac, y) (o(1), resulting in ((z)))((k) (Katharopoulos et al.,
Fig. 1 shows that autoregressive Transformers using this method can achieve performance comparable to existing SOTA models. Additionally, decoder-bases architectures may have significant advantages in extended lookback length and varying output horizon, highlighting their po- tential.
2020). This change reduces the time complexity from O(N²) to O(N) by eliminating the need to compute the full N x N attention matrix. Instead, it computes ((k:) T vs for each i and aggregates over N. Various kernel functions have been explored, with identity kernels without denominators performing well enough (Mao, 2022; Qin et al., 2022; Sun et al., 2023; Yang et al., 2024). In this setup, Linear atten- tion can be viewed as an RNN with hidden state matrix k. E Raxd that updates using the identity function. The output at each step is: O = q: Ei= K; v/-
2.3. Preliminaries: decoder-only Transformer
We use a GPT-2 decoder-only Transformer (Radford et al., 2019) for autoregressive TSF. Token patches of length Lp linearly projected to d-dimensional vectors and com- bined with learnable positional embeddings to form the in- put sequence X E Rg where each token is a E R Each of the m Transformer layers applies layer normal- ization L.).), attention At,( and a channel-wise MIP MLP(-). With single-head softmax attention, a Transformer layer is defined as:
Element-wise linear attention In multi-head linear atten- tion with à heads, we handle h hidden state matrices of size A x d. When h = d, this simplifies h scalar hidden states, effectively transforming linear attention into linear RNN with a d-dimensional hidden state vector (kk,) O o, and enabling element-wise computations of q, k v. This approach, also known as the Attention Free Transformer (AFT) (Zhai et al., 2021), is favored for its simplicity and efficiency in recent works (Peng et al., 2023). We adopt the structure in AFT, where (-) is the sigmoid function, and the output at each step is: o (qq)
Attn(X) = softmax (M O (QK VW.. with Q. V = XW 4. XW, X :- X Aun(LN(X)), then X x X MLP(LN(X))
(L)
Σi= rxp( Els exp(ik;)
where W., WR, W., W. E Rddd are the projection ma-
trices for the query, key, value, and output, respectively, and M E RNNN is the causal mask, defined as M,,- = 1 2
Gated linear attention Recent studies have explored adding a forget gate, commonly used in traditional RNNs, to linear attention, allowing autoregressive models to forget past information and focus on local patterns (Mao, 2022; Sun et al., 2023; Qin et al., 2024; Yang et al., 2024). We implement a simple gating mechanism where each imput D, is converted into a scalar between [0. 1] and expanded into a forget matrix G, matching the shape of k,U, With gating parameters W, E Rdx1, , the output at each step is:
j) 50. 1fi {)
2.4. Preliminaries: efficient linear attention mechanisms
Recent autoregressive efficient attention mechanisms re- duce computational complexity from O(N2) to O(N) by avoiding the explicit calculation of the N N attention matrix (Katharopoulos et al., 2020; Choromanski et al., 2021; Hua et al., 2022; Sun et al., 2023). Most of them can be reformulated as parallel linear RNNs with identity or diagonal state updates. Although these efficient atten- tions do not outperform standard softmax attention for large models, they achieve comparable results on smaller tasks (Katharopoulos et al., 2020; Choromanski et al., 2021). This paper investigates integrating these mechanisms into TSF and shows that adding moving-average term significantly improves their performance. We begin by expressing the recurrent form of standard softmax attention. For a single head without output projection, let qi, kt; Us be the vec- tors at step t from K, V. The output or is given by: Eig explq.k.) je. Eim1 )xplq=k))
O = q: E G2 O. v; =;= = (xx W ,,,TT
Fixed Attention We additionally explore an autoregressive structure with fixed, data- iata-indeeendent weights Wr,s, replac- ing the dynamically generated attention weights d(q.) (k;)) Without dynamic parameter generation, this becomes a lin- car layer with a causal mask M rather than true attention mechanism. We use this structure to examine the effect of adding moving- moving-average term. This autoregressive causal linear layer is expressed as: O = E 1%
2.5. Inspiration: decoupling the short-term impact
In sequence modeling tasks like NLP, context tokens closer to the output token typically carry higher importance. As a result, a gating mechanism with exponential decay in gated linear attention can significantly enhance the performance by assigning greater weights to nearby tokens. Additionally, NLP tasks require retrieval of long-tem information. Even though the decay factor reduces the weights for long-term tokens, the AR weights' ability to capture long-term de-
O,-
Linear attention Linear Attention replaces the exp(q:k:)
Plecase note that, consistent with previous studies on long- term time series forecasting, this paper focuses on one-step pre- diction for the next long time period. Specifically, we predict the next token of length Lr. Readers can view AR tokenization as SStsttets tokenization with an added autoregressive loss.
3

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
pendencies allows for effective retrieval within moderately sized lookback windows.
mechanisms in an AR model structure:
1t+1 of AR +
We 1 v; +
where ry is the AR error. In an ARMA model, the MA term captures short-term fluctuations, allowing the AR compo- nent to focus long-term dependencies. Let e, be the error after introducing the MA term and 0.-1, the MA weights generated by some attention mechanism. We expand the AR error 7x into an MA form and extend the model to an ARMA structure as:
Figure 2. Visualization of different effects with exponential decay strategies and their challenges in gated linear attention. (Left: a) Pure exponential decay strategy in gated linear attention; (Mid: b) Exponential decay facing challenges in capturing long-term dependencies; (Right: c) Exponential decay facing challenges in capturing periodic dependencies
V1+1 O, AR MA
+
+
[-1
Wtd1 1 +
Σ
0 O
6/
6
(2)
However, applying exponential decay to the gated linear attention AR weights may not align with the needs of TSF, which often involves stable periodic patterns alongside short- term impacts. TSF data frequently exhibit seasonal effects that differ from the transient long-term effects NL.. These seasonal effects are stable and persist across the temporal dimension without decaying. As shown in Figure 2, in sce- narios involving seasonal effects, sudden changes, and local effects, relying solely on exponential decay in gated linear attention is not the most suitable approach for modeling TSF. Decoupling local effects into short-term MA weights, allowing the AR term to focus on modeling the seasonal and long-term effects it handles best, would be a better solution for TSF.
= Σ 11
The structure of the MA output o - - O resembles the AR term and could potentially be computed using an attention mechanism. For simplicity, we consider a single channel of the d-dimensional space, with other channels an be handled in parallel. We express the matrix form of the P2 in Eq. (2) for one channel as:
r1
01,1
02.1
02
(3)
11-11
01-11-1
r= (I+ O)e, c= (I+
2.6. WAVE attention mechanism
where θ is a strictly lower triangular matrix of MA weights for this channel. Once the attention mechanism determines AR and 0,-1,j. we can calculate ry = UJ+1 of (token shifting) for all j N 1 and determine Cj via matrix inversion. Substituting these back into Eq. (2) yields the final WAVE attention output of oM for step t.
In the attention mechanisms above, the next-step prediction at time is weighted sum of all previous values v; E R with weights Wa,s E Rixd derived from interactions between q: and kg Naturally, we can write these attention
However, computing o. requires inverting + O. which involves calculating all 0;-1, in the N x N matrix. This increases the complexity of linear attentions back O(N2) and may also cause training instability. To maintain linear time complexity, need method to compute MMA without explicitly calculating all 0--1, values.
Kuyi
Solrnas
2.7. Indirect MA weight generation
W.
W
We need an approach that can leverage linear attention's 's efficiency to compute oMA without the costly N X ma- trix operations. Instead of separately calculating attention weights to determine Cj as value input and recomputing the whole MA output, we aim to use a linear RNN to collect all keys and values at once. We observe from Eq. (3) that
Allesice de Atcettin
WAVE
Figure 3. WAVE attention structure with the indirect MA weight generation method applied to softmax and linear attention. See Table for more calculation details.
4

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table 1. Summary of WAVE attention for various attention mcchanisms, detailing the calculation methods for AR output and MA output,
AR where ry U5+1 ,
Model
AR AR term output Or
Indirect MA term output or
Standard Softmax Attention (Std Attn)
Eis repig: T D maple (4(")
2 (q- (A3
Lincar Attention
q: Σim (R:")) Te:
24 Σ ] 0E (())
(Lin Attn)
Element- wise Linear Attention (ELin Attu)
(gg)) 2iln 1opl
o MM (q: 1) の اj= 6M O.:
El_ (xp(k?")
Gated Linear Attention
(GLin Attn)
q: Eim G, 3 (A:")) Ve
44 ((-1)
(Fixed Atmm)
Fixed Attention
Σim wi.e 2/
d (w
Σj 02 (20 Mhiyk
"ry
there is already a sequential relationship between r and Ej, and rj can be computed directly once oA is determined. Therefore, we implicitly compute the MA weights of ej by using ry as value input for the MA component instead of Ej. Let B:-1.j denote the generated attention weights corre- sponding to rj at step f. and let 0.-1.j here be the implicit MA weights hiddenly linked to the generated B,-1.j- Based
gular, the elements of the subdiagonal in O remain constant, while the elements further down progressively accumulate additional terms formed by the product of different 3 el- ements above. Assuming 3y follows distribution and simplifying by setting each B|]] to the distribution mean b, the elements of O can be expressed as:
on Eq. (3), for each channel, we establish:
B.j = b(1 + b)( where i >>>
(5)
t-1
This simplification offers valuable insights. To prevent longer-term errors from having larger impact, we aim to avoid large absolute values accumulating in O far from the diagonal. We also want A to decay steadily as it moves away from the diagonal. Therefore, constraining B. between -1 and O, with a preference of smaller absolute values, is a practical approach.
E ,r,
Σ
0 1. O6y +
Br = Oe
/=1
1=1
B θ (I+ 1 θ= B B.B.I-B I-
With θ = B. (I-B)-L as long as the indirectly generated @ accurately reflects the characteristics of the MA weights we want, we can use Σ B.-1.5 @ r as OMM.. Since ry is known after computing OR linear attention can be used to compute OMA without increasing the time complexity. To ensure the implicitly generated θ from B captures the desired MA properties, we must carefully design how B is generated. The invertibility of (I (--) is guaranteed since B is strictly lower triangular. To efficiently compute the gen- erated weights, we use the B,- = M (qMA (*MA(M))) (A)MA) to generate B, similar to linear attention. Previous dynamic ARMA models in statistics often update MA weights based on observations (Grenier, 1983; Azrak & Mélard, 2006), so we derive g,ah, and MMA by multiplying the attention input HH with WMA and WMA. Now, the effectiveness of MA weights lies in selecting the most suitable functions dM (-) and MA
We tested various activation function combina- tions for eM^()) (-) and eM^()) (-) to generate BL-1.J df (q )) (RMAA) values, as shown in Fig. 4. We used the sigmoid function @ (AMMA) = a(ak) / /Vi) to obtain values between O and 1, where a = 0.05 and vd are scaling factors to maintain small absolute values. Then, we selected function & (.) to make the product negative. We ultimately chose @MA (g LeakyReLUU-q MA /VV) with a negative slope of 0.02. The inner negative sign maintains directional consistency (for later parameter sharing), and the outer negative sign encourages a negative output.
Fig. 4 shows that LeakyReLU provides balanced lag weight pattern. Unlike ReLU and Sigmoid, which only output values of the same sign, LeakyReLU offers some relaxation while keeping most values negative. This adds flexibility by enabling the desired negative smoothing effect
2.8. Selection of (b)) and characteristics of implicit MA weights
The MA term models short-term effects and local tempo- ral relationships, so we want the implicit O follow a pattern where elements near the diagonal have larger abso- lute values, and those farther away gradually decrease. The expanded form of O is given by θ- = B (I- ()-B)-1- = B + 2 + 3 .... The elements along the diagonal di- rection in B continually accumulate as products into the elements below them in O Since B strictly lower trian-
'In the key activation, a controls the variance of each row in the B matrix, indirectly influencing the amount of long-term information (lower left) in the MA weights O. Increasing a would make the MA weights focus more on modeling long-term infor- mation. However, since we want the AR weights to handle the long-temn term component, we set a to relatively small value. This explains why the rows of the B matrix appear smooth in the visu- alization. Refer to Fig. 7 for more details a, and see Fig. 8 for the effects of reversed positive Qq.
5

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table Summary of main TSF results with forecasting horizons I.p E {12, 24, 48, 96} and 1, = 512. See Table 8 for the original results. Averages of test set MSE for each model on each dataset are presented. Average rankings (AvgRank) of each model, along with the count of first- place rankings (#Topl), are also included.
Pure AR / WAVE Transommer
Baseline
Model
Sid
Std. -ARMA
Lin
Lin +ARMA
Gl Alt
GLin +ARMA
Elin Alln
+ARMA
Fixed Altn
Fixed +ARMA
FITS
iTrans- Tormer
CATS
PatchTST
DI iniie
Former
Enc-
Weather
0.104
0.101
0.104
0.100
0.119
0.105
0.104
0.103
0.105
0.104
0.114
0.117
0.105
0.107
0.124
0.135
Solar
0.134
0.124
0.122
0.119
0.148
0.124
0.136
0.133
0.142
0.135
0.152
0.345
0.122
0.150
0.149
0.125
ECL
0.110
0.106
0.104
0.110
0.108
0.115
0.118
0.124
0.106
0.110
0:111
0.114
ETTh
0.323
3.315
0.318
0.116
0.408
0.321
0.323
0.371
0.330
0.228
0.333
0.351
0.327
0.335
0.329
0.817
ETTh2
0,192
0.192
0,193
0.195
0.217
0.198
0.193
0.190
0.200
0,144
0.197
0.229
0.194
0.201
0,198
0.597
ETTml
0.264
0.239
0.238
0.22
0:407
0.251
0.237
0.259
0.264
ETTm2
0.131
0.12%
0.126
0.121
0.142
0.12
0.134
0.128
0.129
0.122
0.115
0.135
0.116
0.119
0.120
0.311
Traffic
0.341
0.335
0.337
0.330
0.429
0.350
0.352
0.348
0.375
0.365
0,385
0.330
0.372
0.358
0.375
0.547
PEMS03
0.112
0.100
0.100
0.966
0.209
0.066
0.140
PEMS04
0.118
0.106
0.103
0.98
0.167
0.105
0.122
0.119
0.128
0.124
0.151
0,088
0.108
0.164
0.148
0.099
PEMS07
0.092
0.083
0.087
0.027
0.093
0.087
0.101
0.097
0,106
0.100
0.132
0.079
0.094
0,093
0.129
0.102
PEMSOS
0.148
0.132
0.116
0.359
0.125
0.201
0.135
0.121
0.183
AvgRank
7.958
4.252
4.271
23333
12.375
6.229
8,928
7.125
11.145
8.688
12,042
7.271
5,938
9.792
11,250
12:813
#TTpp
0
25
5
Wg. WAR WMA, W as shown in Fig. 3. While WAVE AA' attention has the same time complexity in order of magni- tude as efficient AR attention, its two-stage structure may increase computational costs on constant level. We compare models with different number of layer in the experiements section to show that ARMA's improved performance is due to structural enhancements, but not increased complexity.
MA, (a) 0g MA, LLaakRRLLU, &% 0
(b) 44
MA,
MA. ReLU,, 9k 0
3. Experiments
We conducted comprehensive experiments on 12 widely- used TSF datasets, including Weather, Solar, Electricity (ECL), ETTs, Traffic, and PEMS. See $A.2 for detailed description of datasets.
(c) Pg -OT, 0%
(d) 02 -Swish, 0%
Figure 4. Visualization of the (left) - (right) relationship with different (()) We construct the simulated B matrices using ran- domly sampled q and & (N- 64, 32) from the normal distribution, and display the corresponding implicit O matrices,
Baselines We built AR Transformers using the five attention mechanisms from Table and added MA terms to create WAVE attention for comparison in TSF tasks. Additionally, we included five recent SOTA baselines: FITS (Xu et al., 2024), iTransformer (Liu et al., 2024b), CATS (Lu et al., 2024), PatchTST (Nie et al., 2022), and DLinear (Zeng et al., 2023). We also used a simple channel-dependent encoder-only Transformer, modified by repeating the last input value (like NLinear) to address distribution shift. This model already surpasses older architectures like Autoformer (Wu et al., 2021) and Informer (Zhou et al., 2021), so we excluded these from our comparison.
of the MA term, with occasional positive values to enhance modeling flexibility.
To summarize the WAVE attention process with indirect MA weight generation: First, we compute all oR using the selected attention mechanism. Then, we apply token shifting and compute all ry for j < N 1. Next, using dMA (gMM)) (MM),, dMA (RMMA), and TJ, we calculate MM with the efficient method matching AR attention, as illustrated in Fig. 3. Finally. the ARMA output is o (o + JW.. A summary of MA computation methods for each attention mechanism is in Table 1.
In the main experiments, both pure AR and WAVE Trans- formers use a consistent setup: m = 3 Transformer layers, 8 heads, and model dimension determined by empirical method = 16VC, where C is the number of series. We evaluate their performance using one-step prediction for each test datapoint, aligned with the baselines. Baseline hyperparameters are set to the reported values from their original papers. For more details hyperparameters and implementation, see $A.3.
Computational cost and model performance The intro- duction of MA term adds three weight matrices WMA increasing parameter size. To ensure fair comparison, we use weight-sharing to match the parameter sizes of ARMA and AR models. Specifically, we share W, between the AR and MA terms and set W., to an identity matrix, with mini- mal impact due to the existance of W. and the MLP layer (see Eq. (1)). This reduces ARMA's trainable weights to
We ran all models on all datasets for the four different Lp. In the main text, we report the average test set MSE for each
6

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table Summary showing that pure AR/WAVE Transformers effectively utilize extended lookhack T,, while haselines experience performance degradation. LI E f512, 1024, 2048, 4096) with Lp E {12, 24, 48, 96} are evaluated and averaged. Original results can be found Table 10.
AR/WAVE Trnsformer
Baseline
Model
Sid Altn
Altn +ARMA
Lin Alm
Lin +ARMA
GLin Altn
+ARMA
DLin Altn
ELin Attn +ARMA
Fixed Altn
+ARMA
Atta
ITTS
ITraus- forme:
CATS
PachTST
DLinner
Former
Wealher
Li - 512
0.104
0,101
0.104
11300
0.119
0,105
0.304
0
0,103
0.104
0.114
0.117
0
0.108
0.124
0.135
1024
0.107
0.102
0.102
0.101
0.116
0.104
0.106
0.106
0.108
0.105
0.120
0.117
0.108
0.120
0.118
0.124
20:8
0.102
0.101
0.100
0.108
0.123
0.121
0.119
0.113
0.128
J. 4096
0.108
0.102
0.100
0,300
0.115
0.105
0.109
0.107
0,130
0.108
0.124
0.132
0.123
0.125
0.121
0.136
ET1ml
To = 512
0.266
0.239
0.238
0.222
0.407
0.260
0.346
0.244
0.267
0.251
0.237
0.259
0.222
0.244
0.235
0.429
1024
0.280
0.241
0.239
0.227
0.423
0.236
0.265
0.253
0.261
0.263
0.240
0.258
0.238
0.245
0.239
0.364
2018
0.239
0.233
0.223
0.281
0.252
0.258
0.246
0.261
1006
0.275
0.234
0.237
0.226
0.304
0.229
0.782
0.265
0.287
0.266
0.252
0.274
0.340
0.260
0.428
Table 4. Summary showing that WAVE Transformers with m 3 layers consistently outperform their AR counterparts across wide
range of m. The same experimental settings and data presentation method as in Table 2 are used. See Table 9 for the original results.
Model
WAVE
3
Pure AR
Pure AR
2
21 Pure ARR
Parc AR
4
Pure AR
5
Fure AR
B
Pure AR
Pare AR
8
ET71l Weather
Ams
Astn
0.101
0.104
0.108
0.113
0:111
0.112
0.105
0.122
0.122
0.119
0:121
0.121
0.122
0.121
0:120
ELin
0.103
0.110
0.104
0.108
0.109)
0.311
0:111
Fixed Alin
0,104
0.113
0,109
0,105
0,110
0.112
0.110
0,110
0,110
Std Auta
0.239
0.265
0.270
0,264
0.206
0.269
0.270
0.270
0.272
Lin Artn
0.222
0.341
0.233
0,238
0,232
0.230
0.230
0.231
0.231
GLin Altn
0.260
0.411
0.113
0:407
0:409
0.110
0,110
0. 1.109
0:401
Eliin
0.346
Fixed Alin
0.251
0.069
0,264
0.267
0,260
0.258
0.259
0,258
0,257
model across different Lp on each dataset and provide the full results in $A.5.
term, local effects are absorbed, allowing the decay factor to function properly in the AR forgetting mechanism, leading to significant performance gains.
Short-term TSF results Table highlights the significant performance gains from introducing MA terms the AR Transformers. All WAVE attention mechanisms outperform their AR counterparts in both average test MSE and ranking, with linear and standard attention showing the best results.
Performance of fixed attention Fixed attention, which lacks dynamic parameter generation, performs worse than other attention. However, its significant improvement with MA terms shows that WAVE enhances the model's ability structurally to capture comprehensive sequence patterns.
Long-term TSF results We evaluated pure AR/WAVE Transformers with varying input lengths (Lf) for different prediction horizons (Lp): Lr) (1024, 96). (2048, 192), (2048, 336). (4096, 720). For the baseline models, we selected the best- performing results across multiple input lengths L, E {512, 1024, 2048, 4096} for each prediction horizon. As shown in Table 5, AR models demonstrated comparable performance to baselines, and the incorporation of the ARMA structure consistently yielded improved results over the AR models across all prediction horizons.
Performance and complexity The improvement of adding the MA term comes from its ability to model short-term impacts, allowing the AR term focus on long-term and cyclic effects, not from increased computational costs. Table 4 shows that, regardless of the number of layers (I to 8), pure AR Transformers consistently underperform compared to WAVE Transformers with a fixed m = 3.
ddataabiityy to Longer L, Previous baseline models typi- cally use L, between 96 and 720, as longer L, often leads to overfitting to long-term patterns, ignoring more important local effects (Zeng al., 2023; Nie et al., 2022; Liu et al., 2024b). However, the next-step prediction and varying look- back inputs in pure AR/WAVE Transformers help the model focus on tokens closer to the next step. improving general- ization. As shown in Table 3, increasing L, from 512 to 4096 improves pure AR/WAVE performance, demonstrat- ing scalability and the ability to properly leverage long-term effects. Also, the WAVE structure consistently boosts AR model performance across different L1-
Performance of linear attention Linear attention outper- forms softmax attention in TSF, suggesting that simpler at- tention patterns and normnozmll input shortcuts (without denominator) can improve generalization on time-varying distributions. This aligns with earlier findings where linear models can outperform more complex Transformers in TSF (Zeng et al., 2023; Xu et al., 2024).
Performance of gated linear attention WAVE brought the greatest improvement to gated linear attention. In gated AR models, the decay factor helps the AR term focus on important local patterns, but it weakens the ability to capture long-term or stable cyclic patterns. By introducing the MA
Comparison to MEGA The MEGA structure (Ma et al., 2022) uses an exponential moving average (EMA) in gated attention to model local patterns. However, applying EMA
7

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table Summary of long-term time series forecasting for Lp E {96, 192, 336, 720). Pure AR/WAVE: Transformers uses (7.r, I.p)
(1024, 96), (2048, 192), (2048, 336), (4096, 720) and we choose the best results for the baselines from Li (512, 1024, 2048, 4096) for the 4 Lp settings. See Fig. 11 for the original results.
Pure AREVAVE Transfommr
Baseline
Model
Std Alta
Attn +ARMA
Lin Alta
Altn +ARMA
GLin Altn
GLin Atin
+ARMA
ELin
+ARMA
Fixed. Attn
Fixed Atin +ARMA
FIIS
iTrans- former
CATS
PuichTST
DLinear
Furmer
Ene-
Weather
0:221
0.218
0.218
0.215
0.223
0.216
0.220
0.219
0.220
0.218
0.222
0.232
0.216
0.221
0.233
0.251
0.198
0.195
0.193
0.198
0.195
0.199
0.209
0.206
0216
ETThl
0.414
0.411
0.415
0.411
0.417
0.408
0.409
0.405
0.414
0.410
0.440
0.454
0.408
0.413
01422
0.906
ET
0.340
0.339
0.345
0.339
0.342
0.340
0.357
0.352
0.348
0.344
0.354
0.374
0.320
0.330
0.466
0.77
ETTml
0.347
0.316
0.347
0.345
0.34
ET1m2
0.249
0.246
0.247
0.243
0.250
0.245
0.246
0.244
0.245
0.240
0.247
0.265
0.248
0.247
0.252
0.576
Model: Std Attn
Model: Lin Attn
Model: GLin Attn
Model: ELin Attn
Model: Fixed Attn
Test Laos (Weather)
Std Attn
Lnn Attn
ELin Attn
Fixed Atth
0.14
Std Atin ARMA
Alin ARMA
Altn ARMA
Fixed Alin ARMA
GLin Alln
GLin Alln ARMA
0.12
10
20
30
0
10
15
10
20
30
10
20
30
0
10
20
30
Training Step
Training Step
Training Step
Training Step
Training Step
Model: Std Attn
Model: Lin Attn
Model: GLin Attn
Model: ELin Attn
Model: Fixed Attn
Test Loss (ETTmL)
0.5
Std Attn
Lin Atin
ELin Attn
Fixed Altn
Std Attn ARMA
Lin Attn ARMA
ELin Attn ARMA
Fixed Attn ARMA
0.4
GLin
GLin Attn ARMA
10
20
30
0
10
20
20 Training Step
10
20
30
10
20
30
Training Step
Training Step
Training Step
Training Stop
Figure 5. Visualization test loss curves. We show the testing performance of five attention mechanisms using pure AR/WAVE structures on the Weather and ETTml datasets (L) 512, Lp
18).
alizations in Figs. 9-12 reinforce that there are important long-term stable seasonal patterns for AR weights to cap- ture that should not be disrupted by applying forget gates or EMA. This explains why gated linear attention underper- forms linear attention in our experiments.
Table Summary of the performance comparison with MRGA. See Table 12 for the original result.
Model
Sid Amm
Sid Attn +ARMA
Lin Altn
Lin Altn +ARMA
Gl.in Altn
GLin Attn -ARMA
MRGA
Weather
0.104
0.101
0.104
0.100
0.319
0.105
0.121
Solar
0.134
0.124
0.122
0.119
0.148
0.124
0.236
ETTbi
0.323
0.318
0
0.316
0.408
0.321
0.101
Computational cost Table 7 compares the computational cost of pure AR/WAVE Transformers with baselines on the ETTml dataset. Our tokenization method reduces the token size N. keeping pure AR/WAVE models' computational cost comparable to the baselines. Additionally, parameter sharing ensures the MA term doesn't 't increase the number of parameters, and the extra FLOPs from using WAVE are not significant.
ETTh2
0.192
0.192
0.193
0.217
0.198
0.214
ETTml
0.264
0.239
0.238
0.222
0.407
0,260
0.412
ETTm2
0.131
0.128
0.126
0.121
0.142
0.128
0.137
PRMS01
0.112
0.100
0.100
0.319
0.161
directly to AR weights weakens the model's ability to cap- ture long-term and stable seasonal patterns, making it less effective than ARMA decoupling long-term and short- term effects. Table 6 shows that the performance of using MEGA as the attention mechanism is similar to using gated linear attention without the MA term. It provides less im- provement compared to gated linear attention with ARMA.
4. Conclusion, limitation, and future works
We propose the WAVE attention mechanism, which inte- grates an MA term into existing AR attention using a novel indirect MA weight generation method. This approach main- tains the same time complexity and parameter size while en- suring the validity of the implicit MA weights. Experiments demonstrate that WAVE attention successfully decouples and handles long-term and short-term effects. The WAVE Transformer, enhanced with the MA term, outperforms their
Visualization analysis Fig. 5 shows test loss curves for dif- ferent Pure AR/WAVE attention mechanisms on the Weather and ETTml datasets, with WAVE consistently outperform- ing AR in both convergence speed and final loss. Fig. 6 visualizes attention input sequence, AR weights, B, and θ matrices of a test datapoint on Weather, showing how MA weights decouple local patterns, allowing AR weights to focus on cyclic and long-term patterns. Additional visu-
8

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
AR counterparts and achieves state-of-the-art results, offer- ing consistent improvements in training with minimal added computational cost.
models are few-shot learners. In Larochelle, H.,
Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.),
Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL htttp:::/prooeedddnss.nrrrrr
One limitation is that we have not explored combining the channee-independent WAVE Transformer with multivariate forecasting models to improve its handling of inter-series relationships. For future work. WAVE attention could be applied general sequence modeling tasks beyond TSF. Testing on larger-scale datasets, such using WAVE Trans- formers for large-scale NLP pretraining, is another promis- ing direction.
cc/ cc//spper_fileeppperr//2//fff 1 4557codbbbcc449661111bbbebae pdf.
Paper.
Chandola, V., Banerjee, A., and Kumar, V. Anomaly detec- tion: A survey. ACM computing surveys (CSUR), 41(3): 1-58, 2009.
Impact Statement
Chang, H. Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11315-11325, 2022.
This paper contributes to the field of Machine Learning by presenting model that enhances the accuracy and effi ciency of time series forecasting. The proposed WAVE approach has valuable applications, including improved decision-making in critical domains like transportation and healthcare. Although the societal impacts of this research are largely positive, it is important to ensure responsible implementation and careful oversight, particularly in sensi- tive applications, mitigate any potential risks negative outcomes.
Che, Z.., Purushotham, S., Cho, K., Sontag, D., and Liu, Y. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In International conference on machine learning, pp. 1691- 1703. PMLR, 2020.
References
Choromanski, K. M.,, Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mo- hiuddin, A., Kaiser, I.., Belanger, D. B., Colwell, L. J., and Weller, A. Rethinking attention with performers. In International Conference on Learning Representations, 2021. URL httpht::///ppeeee///// /ppereevee net/ forum? id=Ua 6ZUkORH..
Aghabozorgi, S., Shirkhorshidi, A. S., and Wah, T. Y. Time- series clustering-a decade review. Information systems, 13:16-3,,20 2015.
Azrak, R. and Mélard, G. Asymptotic properties quasi- maximum likelihood estimators for arma models with time-dependent coefficients. Statistical Inference for Stochastic Processes, 9:279-330, 2006.
Das, A., Kong, W., Leach, A., Mathur, S. K., Sen, R., and Yu, R. Long-term forecasting with tiDE: Time- series dense encoder. Transactions on Machine Learn- ing Research, 2023. ISSN 3835-8856. URL https: / //openreview.ce net http///ooummmmmmeeeee wttp:r/wwww.pcccccc..055505
Borsos, Z. Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grang- ier, D., Tagliasacchi, M., et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM trans- actions on audio, speech, and language processing, 31: 2523-2533 2023.
Das, A., Kong, W., Sen, R., and Zhou, Y. A decoder-only foundation model for time-series forecasting. In Forty- first International Conference on Machine Learning, 2024. URL https: wpwp//weeeeeeeeeeeeeeeeeeeeeeewa wwt.:wiwwwwweeeeeeeeee.mee.eeeee forum? id=jn2iTJas6t.
Box, G. E., Jenkins, G. M., and MacGregor, J. F. Some recent advances in forecasting and control. Journal of the Royal Statistical Society: Series C (Applied Statistics), 23(2):158-179, 1974.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11873-12883. 2021.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss. A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J.,
Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language
Grenier, Y. Time-dependent arma modeling of nonstationary signals. IEEE Transactions on Acoustics, Speech, and Signal Processing, 31(4):899-91 1983.

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Gruver, N., Finzi, M. A., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023, URL https:://///...... http///pppeeeeeeeeeeeeeeeeeee/ee met//forue d=md 6 8 7
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.- X., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer time series forecast- ing. Advances in neural information processing systems, 32, 2019a.
Gu, A., Goel, K., Gupta, A., and Ré, C. On the parame- terization and initialization of diagonal state space mod- els. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: http:///ppereeeeei.emmem n forum? d=yJE7iQSAe..
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.- X., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer time series forecast- ing. Advances in neural information processing systems, 32, 2019b.
Li, Y. Yu, R., Shahabi, C., and Liu, Y. Diffusion con- volutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01900 2017.
Hochreiter, S. and Schmidhuber. J. Long short-term memory. Neural computation, 98):1133--7880, 1997.
Holt, C. C. Forecasting seasonals and trends by exponen- tially weighted moving averages. International journal of forecasting, 20(1):5-10, 2004.
Liao, T. W. Clustering of time series data a survey. Pattern recognition, 38(1 1):1877-1874, 2005.
Liu, D., Yu, Y., Tan, L., Wu, W. Zhao, B., Li, Z., Lu, B., and Wen, Y. Seamcarver: Lim-enhanced content-aware image resizing. Preprints, December 2024a. doi: 10. 2044/peppin32004111189....1 URL httsst /doi. org/10. http:///99////ppprrttt2022222110
Hua, W., Dai, Z., Liu, H., and Le, Q. Transformer quality in linear time. In International conference machine learning, pp. 9099-9117, 17. PMLR, 2022.
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., and Muleer, P.-A. Deep learning for time series classifica- tion: a review. Data mining and knowledge discovery, 33 (4):917-963, 2019.
vl.
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024b. URL httphhtsss///peeeiiiiiiiee /ttp/////ppp/eeeeeeoe/ppe. / forum? id-JeP fAI8fah.
Jin, M., Wang, S., Ma, L.., Chu, Z., Zhang. J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y-FF, Pan, S., and Wen, Q TiMe-LLM: Time series forecasting by re- programming large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: http://peeeeeeeeeeeeeeooooee htt/://eee//ruoou id-Unb5CVPtae.
Liu, Y. Qin, G., Huang, X., Wang, J., and Long, M. Auto- times: Autoregressive time series forecasters via large lan- guage models. arXiv preprint urXiv:2402.02370 2024c.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165. PMLR, 2020.
Lu, J., Han, X., Sun, Y., and Yang. S. Cats: Enhancing multivariate time series forecasting by constructing auxil- iary time series as exogenous variables. arXiv preprint arXiv: 2403:2433000 006601673.... 2024.
Kim, T., Kim, J., Tac, Y., Park, C., Choi, J.-H. and Choo, J. Reversible instance normalization for accu- rate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2022. URL https: / wpw..eeeeeeeeeeeeeee/ee/ee wttprrewwwwwwwweeeeeeeeeeeeeeeee n forum? id=cGDAk@olCOp.
Ma, X., Zhou, C., Kong, X., He, J., Gui, L., Neubig, G., May, J., and Zettlemoyer, L. Mega: moving average equipped gated attention. arXiv preprint wwXiv:2209.0065: 2022.
Malhotra, P., Vig, L., Shroff, G., Agarwal, P., et al. Long short term memory networks for anomaly detection in time series. In Proceedings, volume 89, pp. 94, 2015.
Lai, G., Chang, W., Yang, Y., and Liu, H. Modeling long- and short-term temporal patterns with deep neural net- works. In Dolliss-hoomponn K., Mei, Q. Davison, B. D., Liu, Y., and Yilmaz, E. (eds.), The 41st Inter- Research & De-
Mao, H. H. Fine-tuning pre-trained transformers into de- caying fast weights. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10236-10242, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. doi: 10.18653/v12002..2 www.o..eeemmlmmmmmmmmmmmmm main.697. URL https:// aclanthol ogy .rrg emn emnlp-main ---mmii.... 697.
national ACM SIGIR Conference velopment in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 02- 2018, pp. 95-104. ACM, URL https:
2018. doi: 10. 10.1145/3209978.2210000 //doi. org. 10. http///....1////999...
10

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2022.
Schiele, P., Berninger, C., and Rügamer, D. Arma cell: A modular and effective approach for neural autoregressive modeling, arXiv preprint arXiv:2208. 14919, 2022.
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
Orvieto, A., Smith, S. L.., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pp. 16670-26698 PMLR, 2023.
Touvron, H.,, Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023, URL abs 00ts/2302.1999
Peng, B., Alcaide, E.,, Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung. M., Grella, M., et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305. 13048, 2023.
Qin, Z., Han, X., Sun, W., Li, D., Kong, L.,, Barnes, N., and Zhong, Y. The devil in linear transformer. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 7025-7041, Abu Dhabi, United Arab Emirates, December 2022. Association for Compu- tational Linguistics. doi: 10.18653/1//2222.mmmll) main. 473. URL https: htt////laatttttt http///aaaaanoooo.oooooo htw.aaatthhooo.oor///0000 org/2022. emnlp wwallp-main.. main..1. 473.
Truong, C., Oudre, L., and Vayatis, N. Selective review of offline change point detection methods. Signal Process- ing, 167: 2020.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Winters, P. R. Forecasting sales by exponentially weighted moving averages. Management science, 6(3):24433) 342, 1960.
Qin, Z., Yang, S., Sun, W., Shen, X., Li, D., Sun, W., and Zhong. Y. Hgm2: Gated linear mnns with state expansion. arXiv preprint arXiv:2404.0790 2024.
Wu, H., Xu, J., Wang. J., and Long. M. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Informa- tion Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021. virtual, pp. 22419- 22430, 2021. URL https:/ attps://proceeding:
Radford, A. standing by generative pre-training.
Improving language under-
Ope-
nAI technical report, 2018. URL https:
laaguugg-unsssprrrrise///aagggg understandiing_aperr.o
Radford, A., WU, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
neurips httpe/cceeaeeeee//aaaaaa/ 0ccoad-0022819339eccddddda...3. 9aacc0002000 Abstract html.
Rangapuram, S. S., Seeger, M. W., Gasthaus, J., Stella, L., Wang, Y., and Januschowski, T. Deep state space models for time series forecasting. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., BesahBianchi, N., and Garnett, R. (eds.). Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018. NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7796- 7805, 2018. URL https:/ http::///ccccceee.....mm
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186. 2022.
Xu, Z., Zeng. A., and Xu, Q. FITS: Modeling time series with S10k$ parameters. In The Twelfth In- ternational Conference Learning Representations, 2024. URL httpt::t////pptet http:://///eeeee.../m net f 5cFf8669F667 Abattrraat id=bWcnnz2ggMs
neurips. dc wttpp//pppp.p///eeeeeeeeepp hatshhaah/
html.
Yang. S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efffciien train- ing. In Forty-firs piry-fiist International Conference on Machine Learning, 2024. URL ht https: openreview .nt// forum? a 5XVXFUJT.
Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1884 181-1991 2020.

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Yang, W., Fregosi, D., Bolen, M., and and, K. P. Anomaly detection in pv systems using constrained low-rank and sparse decomposition. IISE Transactions, 6776);607-620, 2025. doi: 10. 10.00002442255442444339494
Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11111 1228, 2023.
Zhai, S., Talbott, W., Srivastava, N., Huang, C., Goh, H., Zhang, R., and Susskind, J. An attention free transformer. arXiv preprint arX10:2005.44133, 2021.
Zhang. Y. and Yan, Crossformer:
Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, 11106- 11115. AAAI Press, 2021. URL hUtD.UUUR.ht.ts./ojs..ooo org/index: httpa///////aaaaarrrrr//////////
12

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
A. Appendix
A.I. Related works
Linear Attention Mechanisms The quadratic complexity of traditional attention has motivated extensive research into efficient alternatives. Katharopoulos et al. (2020) pioneered linear attention by replacing the exponential similarity function with kernel functions, achieving linear complexity through reordered matrix operations. While this enabled Transformers be reformulated RNNs during inference, early implementations suffered performance degradation. Various improvements followed: Choromanski et al. (2021) introduced Performers with FAVOR+ for unbiased softmax approximation; Qin et al. (2022) addressed unbounded gradients and attention dilution in TransNormer, and Sun et al. (2023) combined linear attention with retention mechanisms in RetNet. The connection to classical concepts was explored by Mao (2022), who linked linear transformers to Fast Weight Programmers from the 1990s. Recent advances focus on practical large-scale applications. Yang et al. (2024) proposed Gated Linear Attention with hardware-efficient training, while Lighnnning Attention-2 achieved constant training speed regardless of sequence length through innovative tiling strategies.
Linear attention with exponential moving-average Beyond using gating decay factor the hidden state matrix of linear attention (Mao, 2022; Sun et al., 2023; Yang et al., 2024), recent studies have explored incorporating EMA mechanisms into gated linear attention by applying a smoothing factor (summing to 1) to the two terms in the state update (Ma et al., 2022). Similar EMA mechanisms have also been used in many modern RNN structures (Gu et al., 2022; Peng et al., 2023; Orvieto et al., 2023; Qin et al., 2024). Additionally, Schiele et al. (2022) attempted to introduce the ARMA structure into traditional RNNs, but their method could not ensure that the generated MA weights can properly model short-term patterns, and the final results did not significantly surpass traditional RNNs nor compare with recent attention models.
Time Series Analysis Time series analysis encompasses several key tasks. Beyond forecasting, tasks include anomaly detection, which involves identifying abnormal points or patterns in data sequences (Chandola et al., 2009; Malhotra et al., 2015; Yang et al., 2025); classification, which assigns time series data to predefined categories or labels (Ismail Fawaz et al., 2019); clustering, grouping similar time series without predefined labels (Liao, 2005; Aghabozorgi et al., 2015); imputation, addressing missing data points to ensure continuity (Che et al., 2018); and change-point detection, pinpointing moments of significant shifts in statistical properties (Truong et al., 2020). Each of these tasks poses distinct challenges and requires tailored methodological approaches.
Time Series Forecasting Time series forecasting has evolved from classical methods like ARIMA (Box et al., 1974) and exponential smoothing (Holt, 2004) to deep learning approaches. RNN- methods (Hochreiter & Schmidhuber, 1997; Rangapuram et al., 2018; Salinas et al., 2020) captured sequential dependencies but struggled with long-range patterns.
TSF Structures The use of neural network structures for TSF has been widely explored (Hochreiter & Schmidhuber, 1997; Rangapuram et al., 2018; Salinas et al., 2020; Wu et al., 2022). Recently, many Transformer- based TSF models with encoder-only and encoder-decoder structures have emerged (Li et al., 2019a; Zhou et al., 2021: Wu et al., 2021: Zhang Yan, 2023; Nie al., 2022; Liu al., 2024b). Transformers revolutionized the field through various adaptations: LogTrans (Li et al., 2019b) with local convolutions, Informer (Zhou et al., 2021) with ProbSparse attention, and Autoformer (Wu et al., 2021) with auto-correlation mechanisms. However, these complex Transformer architectures have not significantly outperformed simpler MI.P or linear models (Zeng et al., 2023; Das et al., 2023; Xu et al., 2024; Lu et al., 2024). Additionally, these models struggle to handle short-term effects properly with longer lookback windows, where, paradoxically, longer inputs often lead to worse performance, Recent work explores novel perspectives. Nie et al. (2022) proposed PatchTST treating time series as patches, while Liu al. (2024b) applied attention across variates rather than time. The emergence of L.LMs has opened new directions, with Gruver et al. (2023) demonstrating zero-shot forecasting capabilities and Jin et al. (2024) adapting pre-trained models to temporal tasks. Key challenges remain in balancing model complexity with performance and effectively modeling both temporal and cross-variate dependencies (Zhang & Yan, 2023; Lu et al., 2024), while integrating domain-specific biases without sacrificing generality.
A.2. Datasets
Our main MTSF experiments are conducted on 12 widely-used real-world time series datasets. These datasets are summarized
as follows:
13

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Weather Dataset' (Wu et al., 2021) comprises 21 meteorological variables, including air temperature and humidity, recorded at 10-minute intervals throughout 2020 from the Weather Station of the Max Planck Biogeochemistry Institute in Germany.
Solar Dataset al., 2018) consists of high-frequency solar power production data from 137 photovoltaic plants recorded throughout 2006. Samples were collected at 10-minute intervals.
Electricity Dataset' (Wu et al., 2021) contains hourly electricity consumption records for 321 consumers over three-year period from 2012 to 2014.
ETT Dataset"(Zhou et al., 2021) The ETT (Electricity Transformer Temperature) Dataset comprises load and oil temperature data from two electricity transformers, recorded 15-minute and hourly intervals from July 2016 to July 2018. It is divided into four subsets (ETTml, ETTm2, ETThl, and ETTh2), each containing seven features related to oil and load characteristics.
Traffic Dataset (Wu et al., 2021) Sourced from 862 freeway sensors in the San Francisco Bay area, the Traffic dataset provides hourly road occupancy rates from January 2015 to December 2016. This comprehensive dataset offers consistent measurements across two-year period.
PEMS Dataset (Li et al., 2017) The PEMS dataset consists of public traffic network data collected in California at 5-minute intervals. Our study utilizes four widely-adoptee subsets (PEMS03, PEMS04, PEMS07, and PEMS08), which have been extensively studied in the field of spatial-temporal time series analysis for traffic prediction tasks.
A.3.
settings and implementation details
For the hyper-parament parameter settings of the pure AR/WAVE Transformer, we use m = 3 Transformer layers, 8 heads, and set the hidden dimension d based on the number of series C, using the empirical formula d = 16| [C]. We use 4d as the hidden dimension for the feedforward MLP in the Transformer layer. A dropout rate of 0.1 applied to both the AR term and MA term. We initialize the weights of all linear layers and embedding layers using the GPT-2 weight initialization method, with normal distribution and a standard deviation of 0.02. For the output projection layers in the attention and MLP, we additionally scale the standard deviation by a factor of 1//m, aligned with the GPT-2 setting. Normalization layer is applied both before the input to the Transformer and after the Transformer output. We experimented with both standard LayerNorm and R.SNNrm as the normalization layer, finding no significant performance differences, so we opted for RMSNorm for lower computational cost. For token input projection, we use a linear layer to project the Lp-dimensional token to a d-dimensional input vector. In the output projection, we do not tie the weights between the input and output linear layers. learnable position embedding that maps the integer labels from to N (the input sequence length) the corresponding d-dimensional position vectors is used. At the beginning of the model, apply RevIN to input series S,, subtracting the mean and dividing by the standard deviation for each series. Before outputting the final result, we multiply by the standard deviation and add the mean back. All input series are processed independently and in parallel, merging different series dimensions into the batch size for parallel computation. The random seed used in all the experiments is 2024.
All training tasks in this paper can be conducted using a single Nvidia RTX 4090 GPU. The batch size is set to 32. For larger datasets, such as Traffic and PEMS07, we use a batch size of 16 or 8, with 2-step or 4tep gradient accumulation to ensure the effective batch size for parameter updates remains 32. During training, pure AR/WAVE Transformers are trained using the next-step prediction objective with MSE loss. We use the AdamW optimizer with betas=(0.9, 0.95) and weight 1ecay=0.1, following the GPT-2 settings. For a fair comparison, the same optimizer is used for training baseline models. It is important to note that the baseline models trained with this AdamW setup show significantly better TSF performance compared to those trained with the default Adam optimizer settings. As result, the baseline performance presented in this
hhrttp:/ppes.dot.... "http: httttp:/ppes.ddot
14

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
paper may exceed the results reported in their original papers. Since this study focuses on long-term last token prediction results, apply additional weight factor to the training loss for the last token, multiplying it by N. However, this weighting only slightly affects performance smaller datasets with fewer data points, such ETTs, and has little to no effect on larger datasets. Given the minimal impact of this method, the original next-token MSE loss is sufficient for most datasets, without requiring further modifications.
use the same seii--valiaation-eest set splitting ratio as in previous studies by Zeng et al. (2023); Nie et al. (2022); Liu al. (2024b). We also follow the same dataset standardization methods used in these studies. During training, we evaluate the validation and test losses at the end of each epoch, with an carly-stopping patience set 12 epochs. The maximum number of training epochs is 100. We apply linear warm-up for the learning rate, increasing it from 0.00006 0.0006 over the first 5 epochs, and gradually decreasing it in the subsequent epochs.
A.4. Time Complexity of WAVE Attention
Proposition A.1. Let N be the sequence length and d the embedding dimension. Using efficient linear-attentien implementation, WAVE attention has time complexity
O(N df),
which is linear in N.
Proof. We split WAVE attention into its AR (autoregressive) and MA (moving-averaggl parts.
AR Component.
Query, Key, Value projections: Computing Q,K, V. E Nxd via d x d projections costs O(N d²).
Key-Value summary: At each f, update
S. = S + kt
costing (dd) per step, for a total O(N
AR output: Each output
AR = q/ S
costs (dd), summing to O(N df).
Thus, AR costs O(N d=).
MA Component.
Residuals: For j = 1, I,...,N
N - 1, compute
Ty = Uj+1 0f+1:
costing O(N d) overall.
MA projections: Form MA and QMA I)x- via two d x d projections, costing O(N d²).
Running MA summary: At each f, update
T. = T.- + oh (KMA)) (ra)
costing (dd) per step. for O(N d2) total.
MA output: Each
MA (a T T
costs Odd)), summing to O(N d)).
Ignoring the lower-order O(N d) term, MA also costs O(N d2).
15

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Sequence Value
Value
0.1
0.0
40
Timestep
AR Weight Matrix
Beta Matrix
MA Weight Matrix o
1.0
0.00
0.00
0.8
0.6
-0.05
10
-0.05
0.4
1lmestep
0.2
-0.10
-0.10
20
20
20
00.15
-0.15
30
30
30
0.20
-0.20
40
0.0
0
20
40
20
40
20
40
Timestep
Timestep
Timestep
Figure 6. Visualization of the WAVE attention weights (first attention layer, averaged across the multiple heads or d-dimensional channels) for the first test data point in the Weather dataset (T, 4096, T.p 96). More weight visualization be found Fig. 9, 10, 11, and 12.
Conclusion. Combining AR and MA yields
-))
ie.. linear N for fixed d.
A.5. Supplementary experiment results
In the following section, we provide the complete experimental data corresponding to the tables in the main text. Additionally, we include extra visualizations to help illustrate the actual behavior of the MA weights.
Table Comparison of computational costs utilizing the data format of ETTml to build model inputs (L, = 512). The hyper-parameters
for models according their default configurations.
Models
EncFormer
CATS
PatchTST
iTransformer
DLinear
FITS
Metric
FLOPs
Params
FL.OPs
Params
FLOPs
Params
FLOPs
Params
FLOPPs
Params
FLOPs
Params
Lp 96
1.442G
1.646M
262.9M
1.326M
180.9M
1.046M
81.96M
1857M
4.337M
98.50K
334.0K
24.02K
Lp
1.328G
1.646M
243.5M
1.227M
163.6M
652.9K
81.69M
1.51M
2.174M
49.25K
308.1K
22.16K
Lp 24
1.271G
1.645M
233.9M
1.178M
155.0M
456.3K
81.56M
1848M
1.093M
24.62K
294.2K
21.16K
Lp 12
1.242G
1.645M
229.0M
1.154M
150.7M
358.0K
81.49M
1.877M
552.5K
12.31K
288.3K
20.74K
GL Attn
Lin Attn
il Attn
Model
GLin Attn
Lin Attn
ELin Attn
+ARMA
+ARMA
+ARMA
Metric
FLOPs
Params
FLOPs
Params
FLOPs
Params
FLOPs
Params
Params
FLOPS
Params
Lp 96
7.403M
45.81K
7.431M
45.81K
7.387M
45.79K
7.415M
45.79K
7.258M
45.79K
7.266M
45.79K
Lp 48
12.63M
43.97K
12.77M
43.97K
12.60M
43,95K
12.74M
43.95K
12.36M
43.95K
12.37M
43.95K
Lp 24
24,30M
45.22K
24.70M
45.22K
24.25M
45.21K
24.64M
45.21K
23.77M
45.21K
23.80M
45.21K
Lp 12
46.58M
49.82K
47.45M
49.82K
46.46M
49,80K
47.34M
49.80K
45.54M
49.80K
45.60M
49.80K
16

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
33
43
-1.
=15
MA. Leaky ReLU, : (ax 0.5)
(b) 0
MA. ReLU, 0%
MA (a 0.5)
Marric
Siarica
0.00
30
237
(c) @4 -a, 0E
MA
(ax 0.5)
(d) MA. Swish, d (a 0.5)
Mastic
Marrica
Natra
53
(e) 0
MA LeakyReLLL ReLU, MM. (a 2)
MA. (f) Qq
ReLU, MA (a 2)
Xaans
Marrik
Wani
Nasra
-.02
E3c
MA MA (g) 0g -0, 0
(a 2)
(h) 0
MA, -wwish MA, (a 2)
Maunss
Morra
Morri
Marric e
ii 0q -Laky ReUU, MMM (a 10)
MA RReLU, MA (ar = 10)
Marrs
Mardc
Mivricc
Natra
50
-33
(k) @
MA -0,
MA
(ax 10)
(1) MMA. Swish, MMA (a = 10)
Figure 7. Additional visualization of B θ relationship with different d(-) and different a. We construct the simulated B matrices using randomly sampled and & (N- 64, 32) from the normal distribution, and display the corresponding O matrices.
17

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Metri
Matrix
Matra
Matrix
1.0
3.90
10
10
1.0
10
150
06
0.
125
1.00
0.0
0.0
60
50
-0.2
50
50-
50
025
-044
-02
0.0
0.00
(a) Φ4
MA, +LeakyReLU, Re 0% (a = 0.05)
(b) 04
MA.
+ReLU, 0%
MMA
(a 0.05)
Matrix
Matrix
Matrix
Matrix
0-
0
3.06
02
3.05
50
30
30
40
3.02
00
60
80
60
-50
20
40
60
20
40
20
40
60
60
(c) Q4
+a, MAA
0.05)
(d)
04
MA
Swish, d (a 0.05)
Figure 8. Visualization of relationship with positive query activation functions GM (.).
18

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table 8. Detailed results of main TSF experiments with forecasting horizons Lp E (12,24, 48, 96) and L,- 512. Test set MSE and
MAE for each model on each experiment setup are presented.
Eichomer
MAE
0248
1:199
0,200
0:25
1::40
0:301
0:31
0.50
1.50
0
(1750
047
0:39
11:00
0.5
1:300
0.400
B/51
8417
0440
0272
0330
u:99
nrsg
1::0
1220
02288
1:30
XSE
11:12
0.90
IL92
1127
(USS
IU500
1.B0
0.50
(1:2
1L:00
0475
1L39
0.85
0.00
0.00
A17
(UR)
0116
ao2
are
ass
0.00
BLSS
a154
1125
MAE
0:309
0177
0.11
0.30
0161
0.12
8219
0256
0.566
0.310
0.32
0.76
0266
6:37
(LH
0.229
0.288
0220
0223
0226
0:94
0.96
0.26
0.26
0:21
0270
0,29
6360
6218
0.144
0.00
0306
0229
0157
0318
6:57
0179
0.316
0233
0232
0195
DU
0.50
01:22
01:47
00788
11:25
0005
1135
11:30
01.00
1162
0312
0391
0217
0.06
8:31
ILMS
0278
0316
11:40
0.36
0.25
0,500
035S
0.90
0:35
11:00
0655
0300
0125
0:78
RLSS
R228
BEST
0.224
0109
0122
C0P7
0240
6.30
GLES
0.24
6206
0196
01599
0407
0300
0361
6345
034
0.04
0389
034
6002
0341
0.34
0224
0266
0221
6181
025
0214
6361
0253
0335
0386
6.59
0.70
0340
0211
0.96
6183
0163
021
0225
0159
MSE
0.049
0.000
0.29
0.05
O122
0.09
0.08
0333
0.500
024
0222
0174
0175
0.03
0259
0224
0.23
0.57
0.21
0087
0002
0:311
035
0.445
0.311
0158
B1AT
0106
00RR
02311
0214
0.21
0009
0172
0081
018 0083
0086
0
0.26
1001
HOS
0.89
19
0.
0.022
02N
O313
O2N
0214
0E9
0
0.59
0303
920
0NTH
0.29
0260
021
0384
0324
0279
0221
0255
022
910
016
027
62N
038
9,07
029
OIK
0
OLH 0251
630
NOW 0.00
80% 01RE
REA 0257
0212
020 0228
014 028
GID 0302
O
CATS
0
MSE
0.
0:00P
002
GEN
HIS
GIN
DOM
033
0251
122
022
AND
0138
10500
0.50
010
012
0100
012
6106
000
030
0.50
0.00
01S
811
R8S
000
B1S
003
USTI DON
MAE
11 77
11
1.5
0.03
0J04
0226
U22
U.EE
0.00
03395
0381
0300
1130
0310
0225
022
0275
0366
0203
0366
0229
030
021
010
0250
0300
0250
0258
0229
0211
0364
0166
021
0.255
0.065
0.74
0210
1150
0215
0.07
0.
MSE
RJII
GISS
0,50
0350
622
033
0.20
616
R12
0.50
0330
0.50
0.30
AUDN
ON
OUTS
0102
0068
0.50
RUM
MAE
0.200
17777
1U:47
0.12
0330
0330
0240
IESS
0221
0216
0360
035
1150
0298
026
BEW
033
0320
11S
0259
0325
10:00
0200
0256
11200
033
022
0367
021
11200
0250
1199
0230
035
0313
107
1222
0302
UES
1196
F1SS
ASE
0.35
6.755
6.305
0.00
6
GJIS
6336
0,00
620
627
6175
0.35
0.2
0.44
6.44
6:26
0:00
0.35
0.304
0,300
0.55
0255
0.244
0.002
ONN
G
OUPS
GND
6.22
OND
MAE
0.58
0.59
21
0.50
0239
0.41
OIN 0.76
ONG
0215
0300
0.66
0228
0251
01355
0.34
0330
032
035
0224
0228
1022
0.77
027
0365
0251
024
0240
031
0.7H
0.30
0130
0218
0:95
ON1
022
0369
0248
0214
022
-ARNA
0.44
0315
0.9S
0122
007O
O12
0104
0ND
03H
030
9316
022%
0.38
012
0177
628
0360
036
0114
0108
01.2
211
035
0.40
0304
GIN
ONR
0000
910
OIN
010
ONS
OIR
01N
0007
0226
GINI
OIL
OND
Eoodd ATO
0.594
0155
0122
COPP
0215
621
0161
03R
0225
6255
O2
633
639
6TH
626
6200
636
634
635
035
055
0244
0245
020
6177
025
035
0266
6255
6255
6NR
G1FT
02H1
6255
6227
0158
ONE 0266
6227
0IN
O25
635
O22
0255
0 M4T
112
0
OND
OI2
0100
OT1
0.50
0.06
0.50
02M
0215
0158
019
020
034
0.3
0:30
9100
0198
0.00
035
0351
035
0174
0100
0007
01T
0113
RIEE 0109
014
0PN
OUF3
02H
OIN
OTLS
OND
364E
115
0U0S
0.25
BLUSS
GIES
020
1USS
0.60
0559
026
038
11200
032
050
0.69
0.28
826
11
022
U50
0252
0.29
01.
8215
028
R2N
0211
BISS
+41100
0.13
12
00000
OLIES
0.00
1921
0.00
0100
113H
0305
1226
0210
0158
11
11
028
021
015
0324
0128
000
0358
03500
0336
0.00
012
0008
055
0.00
1000
1221
0300
MAR
0.09
0155
0116
0.07
GL92
020
0.90
OND
0,50
0331
055
0.20
0.0
6.22
0,585
0,56
0.00
0.80
0.50
0.55
6217
6172
0,50
036
0.255
0.28
025
0.99
6.73
0,25
0217
QUE
026
0.22
6
6NI
62H
024
0213
0.44
0050
0002
12
0099
036
0250
0212
0.66
1LNS
026
025
0:57
0.228
0:55
0:117
00006
0.38
0565
0.7
0.330
0.75
0:51
6071
0.56
0:38
0.00
0055
8:78
1.10
1071
0229
0L72
0.12
1606
0.
-AINA
MANE
6.00
0155
6/222
0.00
0.A7
0.21
022
6310
0.00
0.08
0,50
0300
6365
6:39
0.551
6:50
6.58
0334
0.339
6:356
0:200
6.722
6:36
6:55
011
6.00
6.77
0.5
6.28
6.00
0.88
0.53
6.00
6.08
0.00
6.58
622
0190
0170
MSD
0.44
0116
0000
0070
0009
0029
0173
0006
0OR8
0,80
0.77
631
0220
0104
6127
6299
0372
022
0744
011
01149
010
6002
0381
0944
0540
0313
0144
0100
6067
0155
0113
0004
0.73
611
6077
0154
0128
0998
OUT1
MEAE
0.310
0.22
0.000
0,21
0.39
0.50
0.50
0.97
0.39
0.34
900
0.59
022
0.00
0:00
0.80
0,02
0.08
0:35
0:09
0.24
22
0.75
0.54
090
0:53
0.530
946
000
8200
0.35
0:50
0.80
0.20
0.55
0.31
63H
0.85
0334
0.5
0.50
0N12
MSS
0
0.00
0000
0141
0003
0125
0991
0340
0340
9351
0255
0231
0163
0154
020
0300
0:71
0356
0474
010T
000
0654
0546
0305
0360
097
010
020
611
ON6
O1H
0000
0225
0101
012
01PS
MAG
0.602
O.II
USB
0.50
1399
0.257
307
0:58
220
0.2
0138
0.28
0.2
022
020
0.00
000
0.30
0.29
0.58
02227
25
200
0.00
0176
0179
0.0
0100
025
038
0.8
6IM
+ARKA
MINE
0.59
0.10
DJRS
0
0IDS
0152
0124
OSTO
OIEE
0000
0330
0.81
0.99
0272
0.23
0213
0.00
0.22
0.25
0.66
0196
0.24
0.02
0107
On7
0.02
0358
0315
03302
0147
ORTS
0302
01F
0.05
OORS
0109
0317
ORL
0
0.25
0105
OIPS
11.56
BLIM
OLIS
GUIS
0250
071C
6220
0350
BUSS
0236
0205
8244
0255
0208
92H
0262
0251
BEW
0250
0258
031
0:00
0200
033
1130
BESS
RJIS
1191
GUTI
XSI
0
1
IURS
GLOSS
ILIO
DOMI
0209
0335
0215
1N
0.75
IUSII
01S
1000
0305
05
11522
031
013
(U66)
0135
1096
(USD)
0US
MDOS
1004
MAE
0L953
0151
CLESS
0086
OZW
GUSO
OIR
G3ST
0200
GOT
025%
022
0335
0265
6241
624
62%
6217
0174
0260
0255
6211
0226
026
0317
GUST
0U6T
0250
0U550
0179
0L99
GL71
6OW
6022
0192
-4HMA
0.42
0.08
0.291
0,255
0.5%
0:05
0,200
0,30
0.16
0.10
0.28
6,301
02%
0225
6:58
0.74
0.3
2012
0,63
0,55
0522
0115
030
0.59
9:00
0579
0.63
0. 1M
0.19
0.02
0,055
0:
0.62
0:0
0.0
0.41
0.10
0.00
MAA
0.555
0157
0091
0150
0211
024
6193
0158
0395
0365
031
6330
6250
632
0.22
6209
631
035
0251
622
0:28
0528
6175
221
0366
8230
0200
62H
G17S
0.65
6346
02.5
0150
0211
0159
0140
6243
62N
639
3H1 ATT
MSE
0.44
0
0.071
0,00
0.776
0000
0.000
0.99
0.34
0.32
0.500
026
0213
01000
0.229
1.36
0227
622
0110
0.51
050
0224
0:35
0363
0.92
0.:76
0.00
0,500
0.772
02222
0
0.14
DOWN
Nosel
9
48
24
12
W
49
24
48
48
24
12
96
48
96
34
12
96
96
24
12
96
46
24
19

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table 9. Results showing that WAVE Transformers with r. = 3 layers consistently outperform their AR counterparts across a wide range of m.. Forecasting horizons Lp {12, 24, 48, 96) and L, 512 are used. Test set MSE and MAE for each model each experiment
setup are presented.
Model
WAVP(m=3)
AR(m-1)
AR(m=2)
Pure AR(m-33
Pure AR(m=4)
Pure AR(m-5)
Pure AR(m-6)
AR(m-7)
A AR(m-8)
Merrics
MSE
MAF
MSF
MAF
MSF
MAF
MSF
MAR
MSE
MAE
MSE
MAE
MSF
MAE
MSE
MAF
MSE
MAE
Sid Alta
0.301
0.354
0.305
0.360
0,308
0.360
0.305
0.359
0.308
0.360
0.304
0.358
0.309
0.36]
0.306
0.359
0.307
0.359
Lin Attn
0.296
0.351
0,310
0.361
0.301
0.358
0.303
0.355
0.303
0.356
0.299
0.352
0.301
0.355
0.299
0.353
0.300
0.354
96
GLia Alla
0.299
0.355
0.337
0.381
0.337
0.387
0.336
0.382
0.334
0.380
0.337
0.382
0.337
0.382
0.335
0.381
0.333
0.379
ELin Attn
0.301
0.354
0.307
0.363
0.309
0.361
0.305
0,356
0.307
0.360
0,306
0.359
0.309
0,362
0.305
0.359
0.308
0.361
Fixed Artn
0.296
0.344
0.299
0.346
0.298
0.349
0.298
0.347
0.299
0.347
0.300
0.348
0.298
0.347
0.302
0.348
0.305
0.351
Sid
0.276
0.333
0.347
0.293
0,347
0.287
0,290
0.345
0.290
0.345
0,344
0.286
0.342
0.291
0.345
Lin Aim
0.266
0.328
0,280
0.336
0.278
0.337
0.278
0.336
0.278
0.336
0.271
0.331
0.272
0.333
0.274
0.332
0,276
0.334
GLin Alta
0.372
0.347
0.496
0.500
0.505
0.467
0.468
0.459
0.469
0.499
0.469
E71ml
ELin Attn
0.282
0.342
0.293
0.350
0,289
0,350
0.284
0,346
0.292
0.3449
0.294
0.352
0.295
0.352
0.292
0.350
0.299
0.355
Fixed Atnn
0,280
0.340
0.345
0.283
0.340
0.284
0.283
0.340
0.284
0.338
0.335
0.282
0.335
0.279
Std Atms
0.223
0.293
0.234
0,300
0.258
0.323
0.246
0,312
0.244
0.311
0,262
0.325
0.260
0.324
0.259
0.323
0,263
0.327
Altn
0.196
0.279
0.297
0,210
0,288
0:218
0.293
0.211
0.288
0,210
0.286
0.285
0.212
0.287
0,209
0.287
24 GLin Attn
0.225
0.305
0,487
0,4.0
0,499
0,440
0.473
0,429
0.476
0,436
0,482
0.436
0.466
0,,43
0,486
0,440
0,463
0.430
ELin Alm
0,241
0.307
0.253
0.328
0,246
0.317
0.239
0.305
0.253
0.318
0.268
0.327
0.263
0.332
0,264
0.325
0,264
0.326
Fixed Attn
0.255
0.321
0,274
0.317
0.272
0.334
0.284
0,332
0,260
0.326
0,254
0.320
0.266
0.329
0,257
0.322
0.255
0.322
Sid Almn
0.156
0.241
0.229
0.293
0,222
0.288
0.218
0.287
0.221
0.291
0.221
0.289
0.223
0.291
0.228
0.288
0.227
0.287
Lin Aim
0.128
0.222
0.148
0.239
0.141
0,232
0.151
0,240
0.137
0.228
0.138
0.234
0.139
0.231
0.137
0.228
0.138
0.229
12
GLin Alnn
0.144
0.237
0.325
0.325
0.321
0.324
0.320
0.335
0.319
0.326
0.322
0.326
0.322
0.325
0.320
0.323
0.320
0.323
ELin Aln
0.153
0.246
0,251
0.160
0.157
0.159
0.250
0.252
0,260
0,163
0.160
Fixed Arnn
0.174
0.261
0.215
0.289
0.204
0.288
0.203
0.282
0.199
0.281
0.194
0.279
0.195
0.278
0.192
0.275
0.189
0.278
Alts
0.142
0.193
0.207
0.153
0,210
0.144
0.152
0.206
0.201
0.209
0.156
0.207
0.156
Lin Attn
0.139
0.191
0,144
0.197
0.143
0.195
0.142
0.194
0.143
0.196
0.143
0.194
0.143
0.194
0,142
0,193
0.144
0.196
Anm
0.142
0.194
0.163
0.213
0.163
0,212
0.161
0.210
0.165
0.213
0.163
0.212
0.164
0.213
0.164
0.216
0.164
0.213
ELin Amm
0.143
0.195
0.157
0.211
0,148
0,207
0.146
0,197
0.151
0:211
0,156
0,208
0.157
0,211
0,156
0.207
0.157
0.208
Atn
0.142
0.198
0.158
0.210
0.151
0,209
0.147
0.194
0.152
0.206
0.154
0.206
0.204
0.153
0.207
0.153
0:205
Std Alta
0.109
0.151
0,116
0.161
0.115
0,159
0.113
0.157
0.116
0.160
0,127
0.177
0.120
0,168
0,128
0,181
0.127
0.177
Lin Aitn
0.110
0.153
0.114
0.156
0.113
0.156
0.115
0.158
0.113
0.153
0.113
0.155
0.112
0.153
0.112
0.155
0.112
0.155
48 GLin Anm
0.116
0.159
0.144
0.190
0,144
0,189
0.144
0.191
0.145
0.193
0,145
0.191
0.144
0.192
0,143
0,189
0.143
0.189
Weather
ELin Attn
0.112
0.156
0.119
0.167
0.116
0.166
0.114
0.158
0.117
0.166
0.118
0.165
0.122
0.172
0.121
0.167
0,123
0.171
Fixed Artn
0.115
0.159
0.124
0.171
0.118
0,170
0.112
0.155
0.122
0.170
0.126
0.173
0.121
0.171
0.122
0.168
0.121
0.167
Sid Atta
0.085
0.115
0.091
0.124
0.091
0.124
0.089
0.118
0.093
0.124
0.096
0.132
0.095
0.130
0.095
0.129
0.094
0.126
0.083
0.117
0.087
0,118
0.088
0.117
0.067
0.115
0.086
0.116
0.087
0.118
0.087
0.117
0.087
0.116
24 GLin Atta
0.090
0.122
0,103
0.152
0.103
0.132
0.101
0.129
0.105
0.132
0.103
0.129
0.103
0.134
0.104
0.132
0.103
0.152
Alnn
0.089
0.119
0.092
0.123
0.092
0,123
0.087
0.116
0.090
0.122
0.092
0.128
0.092
0.128
0.092
0.127
0.092
0.124
Fixed Atmn
0.088
0.120
0.095
0.127
0.093
0.126
0.091
0.122
0.094
0.128
0.094
0.125
0.093
0.125
0.092
0.124
0.094
0.127
Sid Alte
9:067
0.086
0.073
0.001
0.072
0.091
0.071
0.091
0.072
0.093
0.972
0.091
0.072
0.091
0.072
0.094
0.072
0.091
Lin Attn
0.067
0.086
0.069
0.088
0.069
0.089
0.069
0.088
0.069
0.089
0.968
0.088
0.068
0.087
0.068
0.088
0.009
0.090
GLin Atta
9.070
0.091
0.078
0.002
0.077
0.095
0.070
0.090
0.072
0.093
0.971
0.088
0.077
0.093
0.071
0.088
0.071
0.090
ELin Altn
0.069
0.087
0.071
0.090
0.071
0.091
0.069
0.087
0.072
0.091
0.071
0.091
0.071
0.090
0,072
0.091
0.071
0.090
Fixed Atin
0.069
0.090
0.073
0.093
0,072
0.094
0.071
0.089
0.072
0.093
0.072
0.091
0.072
0.090
0.072
0.089
0.072
0.091
20

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table 10. Results showing that pure AR/WAVE Transformers effectively utilize extended lookback Lt, while baselines experience
performance degradation. L, E (512, 1024, 2048, 4096) with Lp E {12, 24, 48, 96) are evaluated. Test set MSE and MAE for each model on each setup are presented.
4096)
De
MAE
0,316
0.99
01.5
0,377
0,312
n
0
0,250
0,10
0157
0.11
0.36
0.14
0100
0110
0000)
0.25
0,30
1.244
0,:87
0.:23
11200
01:01
(1:35
0196
1155
1190
1::05
1::00
XSN
03.58
0.543
11775
II
02213
0139
0000
0.00
0.05
1:303
11.94
10:3
SOAE
0.309
0.57
0.000
0.35
0.22
0:81
0.14
0
0.23
0:7
0.35
0.91
0.52
0,12
0.50
0.229
0,208
0,200
0.25
0.34
0.89
0.312
0.21
0:51
MSS
0,150
0 0122
01
0177
0116
0121
0100
0077
016
0077
0171
013
QIE
0079
0315
022
0.16
0140
0208
0267
0310
01
031
0260
0000
0188
MAE
0.224
0151
0122
0097
0221
0167
0144
0110
0233
0.59
0102
0/0
0301
0311
0.30
0224
0300
0300
0282
0234
6.54
030
Qm
030
023
MANE
0U55
0009
O0F
036
0133
0100
0007
0000
OUTS
0136
0207
0003
0223
0269
0235
1126
0.23
0280
0151
0100
035
B776
MAN
0.208
0.50
0125
0192
0.30
0125
ON0
025
035
0IN
0258
0338
0277
022
0367
035
1220
15
035
RAS
830
C/OTS
MNS
0.14
0.15
0.9
0.09
0.25
0315
0.00
0.03
OUTS
0
0155
0105
01S
0.300
0.305
0.97
0125
0314
0201
0210
0.33
0.23
0:54
030
0.5
020
[Tansforee
01.40
176
13
105
02
0251
0.00
DISE
0.57
0556
1228
1136
19300
1230
12300
1230
11
1M
21
182544
MSE
0.55%
0.32
0.501
0.078
0.644
0127
0.888
000
0.01
0002
0.00
0.73
0300
0085
0332
030
0.25
0.55
0300
030
0228
0155
GTM
0.34
0.56
MANE
0.204
0.77
0.347
0.12
0.22
0.385
0:14
0.11
0.20
0,50
0.00
0.20
0,35
0257
0231
0.20
0
0.55
033
0266
0:377
0256
0:30
BITS
0.55
0.255
000S
0132
0.00
0.009
016
0.35
0
0009
0,35
0:30
0211
0154
0,10
0.38
0222
014
030
0.20
0,55
0.000
0.
MAL
0LU9S
0L9SS
0126
0093
0213
0199
01N
0096
033
0330
0.22
0260
025
828
0:000
0.500
0350
0.260
MINE
0142
0.55
0JSS
0100
01444
014
0196
007)
0072
015
01D
MON
NO1
0.56
0260
0255
017
022
0220
125
81M
3
0100
0166
MAAE
0.594
0.555
0.22
0.009
0.30
0.561
0.20
0.992
035
0.00
022
0007
0337
0.288
0.312
0.29
0.206
0.366
0.92
0.32
0.50
0.3
0 9m
0.38
MSB
0144
0112
OUTI
0140
0116
0195
0072
O15
OU1
9196
00T1
0280
0281
0285
0210
0206
0205
0292
0221
82N
0365
050
0345
01
18.i AMO
MAN
0.05
0125
01
0X97
0.00
01.5
0123
0090
0ONS
0312
0106
012
0097
0551
051
0207
029
0059
0016
02M
6040
0359
0357
61H
0292
MUSE
0.44
0.12
0369
0.44
0.17
0.00
0093
0190
HO72
0,50
0316
0089
0200
026
0261
1151
0200
0226
1251
1155
0220
0101
U151
021
Am
0.90
01558
11116
0159
02
0100
0313
036
0130
1006
0350
0330
0305
1227
0307
036
03M
1150
0.5
055
10300
MIC
0.445
0114
0197
FI1
OU73
OUT2
ONIT
030
0284
022
0157
0517
0315
0206
0100
02N
0.99
0.1
0.80
0251
Glin AME
MOGE
0.094
01.59
0122
DAPS
BAST
OIEE
OOST
0.00
01RS
6100
1089
035
0SH
U32
022
0.22
02500
0228
022
02H
1.36
MSNE
0.44
0.336
0,500
0.350
0.45
0115
0900
0.008
OOST
0.08
ON1
0.55
0086
0988
0.29
0.32
0.35
0:44
0.30
0277
0540
0.20
200
0.72
0.211
0.57
Jom
MOXE
0,200
0.591
0.32
0.590
031
0.99
0.23
0.000
0,20
0.85
0.21
9002
030
0472
0.20
0,35
0.3
0
0.22
0.84
9395
0.00
0.109
0.366
0.16
01
0.70
0114
0/09
0.09
0.66
0.37
0000
000
0.58
650
0475
035)
0359
0557
65H
GMP
1.0
0,58
654
055
0116
MAB
0.993
0U:53
010S
0.057
O117
0096
0.50
0135
OIIS
008
035
0.22
0279
0221
030T
0331
0222
0,300
022
0350
USB
022
2m0
XINE
0.339
500
1OST
114
109
0096
005T
013
010
DOS
100
0.26
0220
1196
1128
0272
332
0110
1
MMM
125
0.5%4
0.228
0:37
0.088
0.998
0.54
0112
0988
0155
0088
0.58
0.5
OUT
0088
0355
0356
0.85
0.40
0.56
0388
0.8
20
0212
550
Km
0231
MSN
0.14
0.35
0008
0.00
0.44
0.300
0.86
91008
0:08
0.38
0.00
0085
0.08
0.50
032
0
0.21
0.30
028
0221
9.788
6.7
0.20
0,50
0.200
0:38
+ARMA
0.003
0151
0.15
02966
0114
0122
011
0996
020
0159
0122
0009
0351
0333
0281
0241
0357
0353
0.00
0299
0.558
013
030
0251
MONE
02
0
0067
01-42
0310
0085
0006
0005
034
0313
0005
0007
020
02266
0221
00550
0205
028
0233
01:49
B2W
0176
MEAS
0.005
01:57
0116
0000
D3NG
01556
01300
0194
0094
0207
0362
0300
B0MS
0259
USH
0312
0366
0388
11356
035
US54
1296
J010
MSTE
0.144
0.113
0109
0MP1
01445
0.12
0.PS5
0.055
0.59
0.14
0172
0.303
0351
020
0215
0316
0.308
0.207
02224
0335
0.24
0.36
0209
24
96
45
24
12
96
90
24
12
24
Moddl
512
1024
2009
4056
512
102-1
2015
4050
21

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table 11. Results of long-term time series forecasting, Baselines are reported with their best-performing results. Test set MSE and MAB for each model on each setup are reported.
Model
Ant +ARMA
+ARMA
Alti
GLin Alla
+ARMA
ELin Alnn -ARMA
Fined Alte
Fined Alta
-ARMA
iTrandoommer
PaisITST
Diiiner
EncFeemar
Metiie
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
MSE
96
0.144
0.142
0.142
0.139
0.161
0.142
0.146
0.143
0.147
0.142
0.149
0.158
0.143
0.149
0.150
0.188
Weather
$36
192
0.245
0.193
0.190
0:342
0.230
0.190
0.158
0.237
03238
0.191
0.191
0.236
0.238
0.192
0.235
0.194
0.236
0.158
0.236
0.185
0.237
0.189
0.201
0.250
0.186
0:335
0.240
0.190
0.255
0.211
0.215
0.270
0.301
0.299
0.300
0.255
0.300
0.256
0.305
0.302
0.309
0.307
0.316
0.297
0.306
0.316
0.332
96
0.196
0.183
0.130
0.209
0.182
0.191
0.182
0.209
0.208
0.201
Solar
0.192
0.193
0.190
0.198
0.197
0.189
0.196
0.194
0.191
0.201
0.196
0.20h
0.216
0.200
0.208
11.221
0.221
720
0.210
0.205
0.208
0.206
0.206
0,204
0.208
0.205
0.204
0.205
0.221
0.218
0.213
0.205
0.227
0.218
96
0.357
0.360
0.358
0.361
0.378
0.368
0,560
0.356
0,362
0.359
0.369
0.396
0.365
0.370
11370
0.956
ETT11
336
192
0.418
0.393
0.415
0.391
0.304
0.426
0.424
0.398
0.419
0.401
0,416
0.396
0.391
0,42
0.415
0.389
0,800
0.423
0.419
0.395
0.468
0.435
0.459
0.451
0.425
0.404
0.412
0.422
1,439
1.403
0.814
0.853
0.487
0.453
0.463
0.458
0.472
0.941
96
0,286
0.273
0.275
0.281
0.263
0.277
0.347
0.313
0:329
0.365
0.375
0.371
0.366
0.375
0,373
0.363
0.366
0.565
0.357
0,361
0.363
0.376
0.329
0.448
0.551
720
0,585
0.382
0.377
0.371
0.385
0
0.579
0.379
0.601
0.398
0.421
0.423
0.265
0.379
0.605
0.714
96
0.305
0
0.303
0.296
0.336
0.259
0.305
0.301
0.298
0.295
0.305
0.325
0.282
0.290
(1299
0.636
ET7ml
536
192
0.355
0.332
0.329
0:354
0.363
0.329
0. 0.362
0.327
0.360
0.332
0.332
0.359
0.332
0.358
0.356
0.329
0.355
0.334
0.326
0.354
0.365
0.334
0.352
0.362
0.326
0:358
0.390
0.328
0.390
0.335
0.791
0.636
0,395
0.409
0.906
0,393
0.393
0.346
96
0.177
0.167
0.172
0.174
0.184
ETTm2
0.220
0.256
0:218
0.256
0218
0.633
0.216
0.258
0.260
0.217
0.217
0.254
0.213
0.211
0.266
0.218
0.263
0.461
720
0.341
0.337
0.339
0.337
0.340
0.334
0,330
0.329
0,332
0.325
0.352
0.358
0.340
0.344
0.341
0.928
Table 12. Experiment results of the performance comparison with MEGA with Lp E [12,224 48,90 48,96).
Std Attn
Lin Attn
GLin Attn
Model
Std Attn
Lin Attn
GLin Attn
MEGA
+ARMA
+ARMA
+ARMA
Metries
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Weather
96
0.144
0.195
0.142
0.193
0.142
0.194
0.139
0.191
0.161
0.210
0.142
0.194
0.164
0.212
48
0.113
0.157
0.109
0.151
0.115
0.158
0.110
0.153
0.144
0.191
0.116
0.159
0.141
0.187
24
0.089
0.118
0.085
0.115
0.088
0.117
0.083
0.114
0.101
0.129
0.090
0.122
0.102
0.128
12
0.071
0.091
0.067
0.086
0.069
0.088
0.067
0.086
0.070
0.090
0.070
0.091
0.077
0.090
96
0.196
0.263
0.192
0.257
0.183
0.247
0.180
0.244
0.209
0.283
0.182
0.243
0.235
0.302
Solar
48
0.160
0.230
0.151
0.223
0.152
0.219
0.149
0.217
0.177
0.258
0.154
0.222
0.342
0.406
24
0.112
0.180
0.098
0.168
0.098
0.166
0.095
0.162
0.143
0.232
0,099
0.167
0,231
0.284
12
0.069
0.137
0.055
0.118
0.056
0.113
0.052
0.111
0.063
0.139
0.059
0.121
0.097
0.170
96
0.357
0.393
0.360
0.395
0.358
0.396
0.361
0.399
0.378
0.406
0.368
0.404
0.373
0.468
PEMS03 ETTm2 ETTml ETTh2 E
48
0.334
0.375
0.331
0.374
0.331
0.375
0.331
0.376
0.349
0.386
0.337
0.381
0.348
0.386
24
0.312
0.365
0.299
0.357
0.299
0.356
0.299
0.357
0.349
0.394
0.303
0.360
0.345
0.387
12
0.290
0.345
0.280
0.340
0.285
0.342
0.272
0.337
0.554
0.503
0.277
0.340
0.551
0.499
96
0.266
0.330
0.268
0.331
0.273
0.336
0.275
0.338
0.285
0.338
0.281
0.345
0.278
0.330
48
0.213
0.290
0.216
0.294
0.215
0.290
0.217
0.293
0.233
0.294
0.220
0.295
0.230
0.295
24
0.0
0.252
0.160
0.252
0.159
0.250
0.162
0.252
0.182
0.263
0.1
0.254
0.180
0.261
12
0.129
0.229
0.125
0.224
0.124
0.224
0.125
0.224
0.168
0.263
0.127
0.224
0.167
0.263
96
0.305
0.359
0.301
0.354
0.303
0.355
0.296
0.351
0.336
0.382
0.299
0.355
0.335
0.378
48
0.287
0.344
0.276
0.333
0.278
0.336
0.266
0.328
0.500
0.472
0.372
0.334
0.507
0.469
24
0.246
0.312
0.223
0.293
0.218
0.293
0.196
0.279
0.473
0.429
0.225
0.305
0.487
0.434
0.218 12
0.287
0.156
0.241
0.151
0.240
0.128
0.222
0.320
0.335
0.144
0.237
0.318
0.322
96
0.177
0.262
0.174
0.261
0.167
0.255
0.162
0.250
0.178
0.260
0.172
0.258
0.176
0.258
48
0.139
0.238
0.137
0.236
0.145
0.248
0.143
0.250
0.167
0.264
0.148
0.249
0.157
0.253
24
0.121
0,219
0.117
0.217
0.110
0,211
0.101
0.199
0.132
0.227
0.110
0.209
0.126
0.223
12
0.086
0.175
0.083
0.1
0.083
0.174
0.078
0.1
0.089
0.178
0.082
0.172
0.088
0.178
96
0.171
0.280
0.153
0.263
0.149
0.258
0.143
0.252
0.360
0.416
0.147
0.257
0.223
0.329
48
0.122
0.234
0.106
0.217
0.105
0.215
0.102
0.210
0.299
0.380
0.109
0.216
0.202
0.317
24
0.089
0.201
0.079
0.187
0.081
0.188
0.075
0.181
0.097
0.209
0.082
0.189
0.129
0.244
12
0.067
0.173
0.063
0.167
0.065
0.169
0.062
0.165
0.078
0.185
0.067
0.177
0.089
0.199
22

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Table 13. Additional ablation studies: without AR loss.
Model
Loss
Std
Original
Sid Amm
YARMA
Otigina
AR
Lin Arta
Onginal
weo
AR
Lit
-ARMA
Orginal
Loss
GLin
Original
GLis Aum +ARMA
Loss
Origitall
Metrics
MSE
MAB
MSB
MAB
MSB
MAB
MSB
MAB
MSB
MAE
MSE
MAE
MAE
MSB
MAE
MSE
ETTml
0.312
0.363
0.305
0.359
01355
0.301
0.354
0.313
0.303
0.355
0.303
0.361
0.351
0.314
0.374
0.336
0.312
0.363
0.299
0.355
0.239
0.346
0.287
0.344
0.387
0.341
0.286
0.345
0.201
0.283
0.343
0.286
0.345
0.266
0.328
0.283
0.343
0.72
0.227
0:336
0.373
0.334
24
0.224
0.294
0.216
0.293
0.217
0.203
0.211
0.259
0.217
0.293
0.212
0.290
0.258
0.255
0.196
00.279
0.212
0.290
0.473
0.429
0.208
0.285
1.225
0.325
12
0.142
0.235
0.136
0.232
0:345
11.237
0.157
0.230
0,142
0:235
0.340
0,230
0.218
0.257
0.156
0.241
0.151
0.240
0.128
0.222
0
0.335
0.144
0.257
96
0.176
0.263
0.172
0.261
0.178
112667
0.172
0.263
0.180
0.270
0.375
0,269
0.177
0.262
0.174
0.261
0.167
0.255
0.162
0.230
0.376
0.260
1.172
0.251
48
0.1.9
0.253
0.137
0.216
0.138
0.340
0.134
0.235
0. 140
0.239
0.138
0.235
0.138
0.258
0.137
0:336
0.145
0.248
0.141
0.250
0.167
0.364
0.148
0.249
0.198
0.108
0.104
0.199
I.195
0.121
0.217
0.110
0.211
0:227
0.110
0.167
0.079
0.170
0.029
0.167
0.106
0.083
0.178
Table 14. Additional ablation studies: multivariate tokenization.
Modd
Ve 1449 Trkenborrios
Sal Are
Mals Tileetrorce
ArE +ARMA
Origrel
94. 30 Telonicatinn
1.6 Art
Origisa
Arm +ARMA
127
02.M333 Trlrsbodis
GLi
Origeal
Vo Mets Titretotee
GLi Ane AARMA
Origiel
01266
0.545
8.285
0.361
01:39
14:307
0.205
01.15
0.238
03:30
01:355
0.318
0.315
0.250
0.300
0.355
11:313
0:355
0.366
0.351
0.338
11382
0.299
01355
012557
0.726
0.258
11:21
0.254
0.328
0.351
0.320
01.54
0.330
0.267
0.344
01.31
0335
11.75
0.390
0316
0.125
11.00
14:73
0.772
01:31
01113
0.36
8.112
0.385
0.:21
01215
0.131
08234
0.116
03235
01114
0.310
0.216
0.267
9.41
01151
0.241
0.125
0.22
01320
11325
018
00237
01100
0.241
0.155
0.35
0.158
0.345
0354
0.344
1160
0.388
01:3
0.16
0.177
0.903
0.131
0.361
0.007
0.995
3.66
0.90
0.33
0.300
0.12
0.55
0.381
01:01
01881
0.60
03883
0.175
0.11
0.319
0217
0.10
0,111
3.001
0199
0.333
8110
0.384
0/004
0.195
0167
6152
0.66
03533
0996
01553
0167
03511
0.966
0.553
0486
0.175
0.811
0131
0081
330%
0.000
01284
01576
0/0
01:33
Table 15. Additional ablation studies: Comparison with NetesNet (Wu et al., 2022).
Mathods
MSE
TemesNet
MAB
MSB
Sil
MAE
MSE
+ARMA MAB
MSL
MAE
Lim
MSE
+ARMA MAE
MSE
MAB
+ARMA GLin
MSE
MAE
BTTbi (96)
0.384
0.402
0.357
0.393
0,350
0.395
0.358
0.395
0.361
0.399
0.578
0.406
0.368
0.504
ETTh1
0.4%6
0.429
0.393
0.391
0.417
0.404
0.429
0.398
0.425
0.401
0.422
0.420
B (336)
0.49]
0.469
0.488
0:434
0.415
0.432
0.428
0.447
0.424
0.444
0.19
0.439
0.416
0,434
BTTbi (720)
0.521
0.500
0.487
0.491
0.478
0.483
0.471
0.458
0.462
0.479
0.469
0.489
0.453
0.473
ETTh2
0,340
0.374
0.266
0.330
0.268
0.331
0.336
0.338
0.285
0.338
0.345
BTTb2 (192)
0,402
0.414
0.336
0.384
0.330
0.382
0.347
0.385
0.336
0.382
0.335
0.381
0.333
0,379
BTTh2 (336)
0.452
0.452
0.371
0.412
0.366
0.408
0,375
0.414
0.373
0.413
0,363
0.411
0.366
0.415
BTTh2
0.463
0.382
0.424
0.426
0.371
0.425
0.585
0.429
0.381
0.425
ETTml (96)
0.338
0.375
0.305
0.359
0.301
0.354
0.303
0.355
0.296
0.351
0.336
0.382
0.355
BTTml (192)
0,374
0.387
0.332
0,374
0,329
0.370
0.329
0.375
0.327
0.373
0.332
0.374
0.332
0.373
bTtml
0.411
0,390
0.354
0.392
0.363
0.397
0.362
0.396
0.500
0.395
0.345
ETTml (720)
0.478
0.400
0.395
0.222
0.395
0.424
0.409
0,322
0.406
0.429
0.398
0.426
0,421
BTTm2 (96)
0.187
0.267
0.177
0,262
0,174
0.261
0.167
0.255
0.162
0,250
0.178
0.260
0,172
0.258
ETTm2 (192)
0.249
0.309
0:220
0.290
0,218
0.292
0.293
0.215
0.289
0.216
0.290
0.289
ETTm2 (336)
0.321
0.351
0.256
0.321
0.256
0.320
0.263
0.326
0.259
0.323
0.264
0.329
0.323
ETTm² (720)
0,408
0.401
0.341
0.378
0.337
0.374
0.359
0.378
0.337
0.360
0.340
0.560
0.334
0.378
Weather (96)
0,172
0.220
0.144
0,195
0,142
0.193
0,142
0.191
0.139
0.191
0.161
0.210
0,142
0,191
Weather (192)
0.219
0.361
0.193
0.190
0.243
0.243
0.188
0.244
0.191
0.344
0.191
Weather (336)
0.280
0.306
0.245
0.289
0.242
0.287
0.239
0.283
0.237
0.283
0.238
0.784
0.236
0.281
Weather (720)
0,365
0.359
0.301
0.331
0.299
0.333
0,300
0.330
0.245
0.325
0.300
0.331
0.296
0.327
Table 16. Additional ablation studies: stability under different random seeds.
Std Attn
Sud Attn +ARMA
Lin Attn
Lin Attn +ARMA
Seed / Method
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
ETTh1 (96)
0.357
0.393
0.360
0.395
0.358
0.396
0.361
0.399
ETTh1 (192)
0.393
0.418
0.391
0.417
0.404
0.429
0.398
0.425
Seed=2024
ETThl (336)
0.418
0.434
0.415
0.432
0.428
0.447
0.424
0.444
ETTh1 (720)
0.487
0.491
0.478
0.483
0.471
0.488
0.462
0.479
ETTh1 (96)
0.358
0.394
0.361
0.396
0.358
0.398
0.361
0.399
ETThl (192)
0.395
0.419
0.393
0.418
0.403
0.430
0.398
0.427
Seed=2025
ETThl (336)
0.417
0.432
0,416
0.431
0.427
0.449
0.425
0.446
ETTh1 (720)
0.486
0.491
0.477
0.484
0.468
0.487
0.462
0.480
ETThl (96)
0.356
0.394
0.361
0.398
0.358
0.396
0.360
0.399
ETThl (192)
0.393
0.419
0.391
0.418
0.404
0.429
0.396
0.426
Secd=2026
ETThl (336)
0.420
0.436
0.415
0.430
0.427
0.444
0.425
0.446
ETThl (720)
0.489
0.489
0.477
0.482
0.472
0.487
0.460
0.480
ETTh1 (96)
0.355
0.389
0.360
0.395
0.357
0.396
0.361
0.401
ETTh1 (192)
0.393
0.416
0.391
0.416
0.405
0.429
0.398
0.425
Seed=2027
ETThl (336)
0.419
0.431
0.415
0.431
0.426
0.446
0.424
0.445
ETTh1 (720)
0.485
0.492
0.478
0.483
0.469
0.490
0.462
0.479
ETThl (96)
0.358
0.392
0.357
0.395
0.358
0.398
0.362
0.400
ETThl (192)
0.393
0.418
0.388
0.415
0.403
0.430
0.397
0.424
Seed=2028
ETTh1 (336)
0.417
0.433
0.417
0.432
0.428
0.447
0.425
0.445
ETThl (720)
0.487
0.491
0.478
0.482
0.469
0.486
0.464
0.478
23

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Sequence Value
Value
20 Timestep
40
AR Weight Matrix
Beta Matrix B
MA Weight Matrix e
0.4
lisestep
0.2
0.0
0.0
-0.1
30
30
30
-03
-044
-04
-0.5
-0.5
20
40
20
40
limestop
Timestep
Timestep
(a) Dataset: Weather, Channel: 1st, Layer: 1st
Sequence Value
Vilue
10
20
30
40
Timestep
Weight
Beta Matrix
MA Weight Matrix
1.0
0.0
0.8
10
timestep
002
20
20
20
0.3
-0.1
-0.2
-0.4
005
0.0
40
40
-0.6
20
40
Timestop
Timestop
Timestop
(b) Dataset: Weather, Channel: 1st, Layer: 2nd
Sequence Value
10
2.0 'Timestep
AR Weight Matrix
Beta Matrix B
MA Weight Matrix e
1.0
0.8
10
0.6
10
0.2
10
0.0
Tmestep
0.4
-0.4
0.2
0044
0.6
0.6
30
0.8
30
-1.0
40
40
Timestep
Timestep
Timestep
(c) Dataset: Weather, Channel: 1st, Layor: 3rd
Figure Visualization of the WAVE attention weights of the first input channel for the first test set data point the Weather dataset (Lz = 4096, Le = 96).
24

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Sequence Value
50
100
150
200
250
300
350
Timestep
AR Weight Matrix
Beta Matrix B
MA Weight Matrix e
0.000
50
50
50
-0.025
100
100
100
00.050
Tisestep
150
150
-0.075
0.000
-0.025
200
00 100
-0.050
250
0.075
250
-0.125
0. 10000
300
300
0.1125 300
0. 1.50
-0.15
0.175
-0.175
100
200
300
200
300
100
200
300
limestop
Limestep
Timestep
(a) Dataset: ETTml, Channel: 1st, Layer: 1st
Sequence Value
50
100
150
200
250
300
350
Timestep
Weight Matrix
Beta Matrix B
MA Weight Matrix
1,0
0.00
0.8
0.6
50
0000
]00
0.4
100
-0.04
Timestep
150
150
150
-0.06
200
200
200
-0.00 0.
250
250
250
300
300
-0.12
0.0
-0.14
0014
100
200
300
100
200
300
100
200
300
Timestop
Timestop
Timestop
(b) Dataset: ETTml, Channel: 1st, Layer: 2nd
Sequence Value
150
200
250
300
Timestep
AR Weight Matrix
Beta Matrix
MA Weight Matrix e
0
0.000
50
50
50
-0 00.025
100
100
100
050
Timestop
150
150
0.000
-0.025
0.100
250
300
250
300
-0.050
-0.1
0.100
0.075
1..125
300
250
00.125
0.150
100
300
100
200
200
100
300
Timestep
Timestep
Timestep
(c) Dataset: ETTml, Channel: 1st, Layer: 3rd
Figure Visualization the WAVE attention weights of the first input channel for the first set data point in the ETTml dataset (Lz = 4096, Lp = 12).
25

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Sequence Value
0.3
10
20
30
40
Timestep
Weight 1.0
Weight Matrix
0.8
0.6
-0.05
10
0005
Timestep
-0.10
0.10
20
20
20
00.5
-0.1
30
30
-0.20
-0.20
40
0.0
20
40
20
40
20 Timestop
40
Timestop
Timestep
(a) Dataset: Weather, Channel: All (Averaged), Layer: 1st
Sequence Value
0.1
Timestep
AR Weight Matrix
Beta Matrix
MA Weight Matrix O
1.0
0.00
0.00
0.8
10
0.6
10
-0.05
10
0.4
Timestep
0.10
0.2
20
-0.15
-0.15
0020
30
40
40
-025
40
0.25
20 Timestop
Timestop
Timestop
(b) Dataset: Weather, Channel: All (Averaged), Layer: 2nd
Sequence Value
0.2
0.0
20
30
40
Timestep
AR Weight Matrix
Beta Matrix
Weight Matrix
0.000
0.025
10
-0.500
Timestep
-0.075
0.075
20
20
20
-0.100
0 -0.100
30
30
-0.125
30
0 -2.
-0. -0.150
00.150
0.175
0.175
20
40
20
40
20
40
Timestap
Timestop
Timestap
Dataset: Weather, Channel: All (Averaged), Layer: 3rd
Figure 11. Visualization of the WAVE attention weights for the first test set data point in the Weather dataset (L, 4096, Lp 96).
26

---

Weighted Autoregressive Varying Gate for Time Series Forecasting
Sequence Value
Vglue
0.2
) A
0.0
50
100
150
200
250
300
350
Timestup
AR Weight Matrix
Matrix
MA Weight Matrix
00.01
0.6
100
0.4
100
100
Timesteg
150
0.2
150
0.03
150
200
200
04
200
250
-0.05
250
-0.05
300
0006
300
00.06
0.0
0.07
0.07
0
100
200
300
100
200
300
0
100
200
300
limestop
Timestep
Timestop
(a) Dataset: ETTml, Channel: All (Averaged), Layer: 1st
Sequence Value
Value
0.2
100
250
200
Timestep
AR Weight Matrix
Beta Matrix B
MA Weight Matrix e
1.0
0.00
0
0.00
0.8
50
0.6
50
-0.01
50
-0.01
100
100
Timestep
-0.02
-0.02
150
150
0.03
200
200
200
250
250
-0.04
250
300
300
360
100
100
200
300
Timestop
Timestop
Timestop
(b) Dataset: ETTml, Channel: All (Averaged), Layer: 2nd
Sequence Value
0.2
0.0
50
100
150
200
250
300
350
Timestep
AR Weight Matrix
Beta Matrix B
Weight Matrix e
0.00
50
50
-0.01
50
100
100
-0.02
100
-0.02
Timette
150
0.2
150
150
0.04
-0.04
200
200
0.05
300
300
-0.07
300
0,0
0
100
200
300
100
200
300
0
100
200
300
Timestop
Timestop
Timestop
(c) Dataset: ETTm), Channel: All (Averaged), Layer: 3rd
Figure 12. Visualization of the WAVE attention weights for the first test set data point in the ETTml datasct (Lr 4096, Lp 12).
27


Reviews:
[{"review_id": "2xVllD1DHl", "reviewer": "Reviewer_nwFT", "summary": "This work introduces a decoder-only Transformer based model for time-series forecasting and introduces the WAVE attention mechanism. The WAVE attention mechanism leverages autoregressive and weighted moving averaging techniques. The authors show that coupling WAVE-based attention and a decoder-only structure outperforms other state-of-the-art models for short-term forecasting and produces comparable results for long-term forecasting. The authors also provide extensive ablation studies into WAVE-based attention mechanisms and their hyperparameters.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Describe how you avoid error accumulation.\n\nThank you for this question. To clarify, this approach is not a contribution but a **prerequisite setting** demonstrating that **pure AR attention** can match previous models. Traditional one-step AR models accumulate significant errors during iterative prediction when $L_P>1$. By making each token contain $L_P$ steps, we **eliminate iterative prediction**, naturally avoiding these errors. This allows AR attention to integrate with existing baseline environments that also directly predict all $L_P$ steps. Essentially, this is similar to PatchTST tokenization with AR loss added between non-overlapping tokens, as illustrated in Figure 1(a).\n\n> Please report long-term experimental results for the same datasets\n\nThank you for this valuable question. We prioritized short-term forecasting as our main experiment, with long-term as supplementary. This choice stems from **baseline limitations** rather than our model structure - previous baselines struggle to efficiently use longer lookback information needed to showcase ARMA attention's effectiveness in long-term forecasting.\n\nOur AR tokenization requires $L_P$ as patch length. For $L_P=720$, we need $L_I=4096$ to maintain sufficient tokens to demonstrate ARMA's benefits (Table 8). However, previous research typically used shorter input lengths.\n\nFor fair comparison, we reran all baselines with $L_I \\in \\{512, 1024, 2048, 4096\\}$ and selected the best results. Some baselines (Encformer, PatchTST) become **computationally infeasible** with high $L_I$, $L_P$, and input series count $C$. For large datasets like electricity ($C=321$) and traffic ($C=862$), using $L_I=4096$ makes training impossible even with batch size = 1. Thus, our long-term setting only included datasets up to Solar ($C=137$). Our code repository's baseline.sh verifies this limitation. We'll add a footnote to explain these constraints.\n\n> The authors do not show any time-series forecasts\n\nWe'll add figures [link (PDFs)](https://anonymous.4open.science/r/WAVE-Rebuttal-B8D2) in the appendix. Time series visualizations typically show just one test datapoint from one series, providing limited information compared to comprehensive evaluation metrics. Papers in this field rarely include such figures in the main text, occasionally placing them in the appendix without using them to support claims.\n\n> Experimental details, train/val/test split, hyperparameters\n\nThis information is detailed in Appendix B.2. Our setup matches DLinear, PatchTST, and iTransformer, using **identical train/val/test splits**. For short-term experiments, we fixed $L_I=1024$; for long-term, we used fixed combinations: $(1024,96)$, $(2048,192)$, $(2048,336)$, $(4096,720)$ for $(L_I,L_P)$. For baselines, we ran all combinations of $L_I \\in \\{512, 1024, 2048, 4096\\}$ for each $L_P$ and selected the best results, ensuring **fair comparison** against optimally-performing baselines, similar to Table 3. Our complete code implementation supports these details.\n\n> Lacks a sufficiently overview of previous works\n\nThank you for this suggestion. Our overview currently focuses on autoregressive attention mechanisms and attention for TSF. We'll expand the background content in the first section and add a related works section in the appendix discussing WMA and EMA applications in attention-based TSF and their relationship with ARMA.\n\n> 1. I cannot read the text on most figures\n> 2. Figure 3 never mentioned in the text\n> 3. Table 1 needs to take up more space\n> 4. bolded and underlined results\n> 5. Tables are in reverse order\n\nThank you for these suggestions. We'll make these corrections:\n\n1. We'll move small text from figures to captions.\n2. We referenced Figure 3 in section 2.8 (line 262), though admittedly late. We'll reposition Figures 3 and 4 to address this ordering issue.\n3. We'll add line breaks for Std and ELin names to provide more space for formulas.\n4. We'll add light color formatting to all tables to highlight performance differences.\n5. We'll reorder tables in ascending order.\n\n> Random periods after some of the symbols ... manuscript was not thoroughly read over by other authors\n\nWe apologize for this confusion. What appeared as periods were actually `\\cdot` **placeholders** in subscripts, particularly in sections 2.7 and 2.8. While normally displayed as $\\phi(\\cdot)$, in subscripts it appears as $\\beta_{\\cdot}$, resembling a period. In our revision, we'll use $\\beta_{(\\cdot)}$ in subscripts to clearly distinguish from periods.\n\n> Plots of oAR and MA.\n\nWhile decomposing AR/MA contributions to forecasting would be informative, our **stacked ARMA attention layers** make this challenging, as components mix with counterparts from previous layers. For understanding AR/MA behavior in each layer, our **visualizations of weights** are more appropriate. Please refer to Figure 9-12, which display AR and MA weights for each layer, providing detailed insights into how ARMA functions within layers.", "claims_and_evidence": "The claims made are supported by empirical evidence of the model's performance as well as plots showing the attention matrices. The plots of the attention matrices validate the authors' claims that the AR and MA attend to long-term and short-term phenomena, respectively. However, the authors do not provide evidence of how this affects the AR and MA predictions. I would like to see plots of oAR and oMA as well to validate the claims that oAR and oMA do in fact contain long-term cyclical and short-term dynamics, respectively.", "methods_and_evaluation_criteria": "This work evaluates multiple attention methods on standard benchmarks. However, it is unclear to me why the authors did not provide long-term prediction results for all the datasets used in the short-term experiments. \n- Please report long-term experimental results for the same datasets as in the short-term experiments or provide sufficient reasons for not doing so.\n- The authors do not show any time-series forecasts. It is very odd that there are no visualizations of the forecast results.", "theoretical_claims": "- The authors attempt to avoid error accumulation by employing non-overlapping patching. However, when linking this patching method to their goal they briefly state \"This ensures that each out-of-sample prediction token covers the entire forecasting length LP, thereby avoiding error accumulation.\" Basically the authors simply state that this works without providing insight or arguments that it should. Specifically, it's not clear how non-overlapping patches in the input mitigate error accumulation in the output, which I interpret as the point of this statement at this stage of the paper.", "experimental_designs_or_analyses": "- This work omits important experimental details making it unclear how exactly the experiments were performed. Please provide a section in the supplementary material providing such details. Below I list a few.\n- How many experiments were run?\n- What is the train/val/test split?\n- Which hyperparameters (e.g. Li and Lp) are dataset dependent or dataset and prediction length dependent?", "relation_to_broader_scientific_literature": "Time-series forecasting is a ubiquitous problem in scientific and industrial tasks. The authors introduce attention mechanisms based on well-established theory within these fields and expand upon them. Their work is broadly applicable to both science and industry.", "essential_references_not_discussed": "The authors interweave discussion of previous works with their own methodological discussions. I find this approach helpful. However, I do think this paper lacks a sufficiently broad overview of previous works that touch on all aspects of this work.\n- Currently, there appears to be one paragraph at the beginning that addresses previous works in general. This needs to be expanded to multiple paragraphs and include more topics discussed in this work. Below are two examples.\n- Some mention of linear attention mechanism and the important works as well as their impact.\n- Some mentions of other works that include weighted moving averages within the attention process. These references are severely lacking in these references, such as ETSformer. The authors can find a good overview of such works in the introduction and related works of \"Powerformer: A Transformer with Weighted Causal Attention for Time-series Forecasting\" and \"TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis\"", "other_strengths_and_weaknesses": "**Strengths**\n- The introduction of well estabilished theory into the attention mechanism through the AR and MA methods.\n- Extensive experiments showing the benefits of the MA methods.\n- Extensive experiments showing how ARMA improves upon multiple types of attention\n\n**Weaknesses**\n- Missing experiments and experimental details (described above).\n- Poor use of figures and tables\n- The claim of how the authors avoid error accumulation is very unclear even thought this seems important\n- In general it seems like this paper needs to be read over by a more experienced writer, there are also random periods after some of the symbols which to me indicates that this manuscript was not thoroughly read over by other authors.", "other_comments_or_suggestions": "I consider each bullet point (outside of the listed strengths) as a concern that needs addressing. Below are some further writing and organization concerns. In general I think this is a compelling work ultimately worthy of publication after addressing the concerns and clarifications throughout this review.\n\n- I cannot read the text on most figures\n- Figure 3 seems very important but is never mentioned in the text. If it's not important enough to mention in the text it should not exist in the main body. The description of the attention mechanisms in the methods section would greatly benefit from pointing to the corresponding panel in Fig. 3. This figure should also be made much larger, it is too busy for how small it is.\n- Table 1 needs to take up more space or be reconfigured, I can't separate the equations in Elin-attn\n- Not all tables have bolded and underlined results, why? This would be very helpful\n- Tables 4, 5, and 6 are mentioned in the paper in reverse order, their order needs to be reversed.", "questions_for_authors": "1) Can you please describe how you avoid error accumulation?\n2) Does this method predict the entire prediction sequence all at once, like PatchTST, or does it predict one time-step and then the sequence in an autoregressive fashion?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ifyNPctRck", "reviewer": "Reviewer_rjH5", "summary": "The Weighted Autoregressive Varying Gate (WAVE) attention is a new mechanism that augments Transformer attention with both an autoregressive component and a moving-average component. By combining ideas from statistical models with efficient Transformer architectures, WAVE expands the modeling capacity for time series forecasting while keeping the model lightweight and fast.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper adequately cites recent Transformer-based TSF models, but lacks statistical forecasting approaches (e.g., deep state-space models and mamba, etc).\n\nThank you for your suggestion. We will add a related works section in our revision to discuss statistical forecasting and recent SSM-based TSF models.\n\n> Performance on multivariate time series remains unexplored.\n> How does WAVE perform on multivariate time series forecasting.\n\nThank you for your suggestion. Following your advice, we added multivariate channel-mixing to our tokenization: after obtaining a tensor of shape $(C, N, L_P)$ using AR tokenization (where $C$ is the number of series, $N$ is the patch token count, and $L_P$ is patch size = forecasting horizon), we applied a **linear projection to the $C$ dimension** to mix channel information at the same position. We report comparison results of pure AR vs. WAVE attention with multivariate AR tokenization in [link (Table 2)](https://anonymous.4open.science/r/WAVE-Rebuttal-B8D2). Results show that introducing multivariate effects enhances overall performance, and the improvements from ARMA structure remain effective. We will include these results in our revision.\n\n> Its potential application to other sequential modeling tasks is not discussed.\n> Could WAVE be generalized to non-time-series applications?\n\nThank you for your suggestion. We've addressed this in the limitations section. Transformers for NLP are typically designed to learn complex, unstructured temporal relationships, unlike time series' common patterns (lag autocorrelation, cyclic patterns, seasonal effects, trends, local decay patterns). Introducing ARMA structure into attention provides **stable, interpretable inductive bias** for handling these effects, better modeling the time series generation process. We briefly explained TSF and NLP differences in section 2.5, which inspired our ARMA integration into AR attention. Further research is needed to verify if this TSF-specific inductive bias is effective for sequence tasks without fixed positional autocorrelation effects. We appreciate your understanding.\n\n> Please elaborate more on the relation between mamba and WAVE.\n\nThank you for your suggestion. Mamba uses the following parameterization method:\n\n$$\n\\mathbf{o}\\_t = C \\overline{B} \\mathbf{x}\\_t + C \\overline{A}\\_{t} \\overline{B} \\mathbf{x}\\_{t-1} + C \\overline{A}\\_{t-2} \\overline{A}\\_{t-1} \\overline{B} \\mathbf{x}\\_{t-2} + \\cdots\n$$  \n\nIts step-level formulation is:\n\n$$\n\\mathbf{o}\\_t = \\sum\\_{i=1}^t \\prod\\_{j=i+1}^{t} C \\overline{A}\\_{j}  \\overline{B} \\mathbf{x}\\_{i}\n$$  \n\nwhere $\\mathbf{x}\\_i$ is a $1 \\times d$ input column vector. The term $\\prod\\_{j=i+1}^{t} C \\overline{A}\\_{j} \\overline{B}$ can be viewed as a vector autoregressive weight matrix for step $i$. If we use a diagonalizable parameterization for $\\overline{A}\\_{j}$, similar to the method in linear recurrent unit (https://arxiv.org/abs/2303.06349), we get the parameterization $ C P (\\prod_{j=i+1}^{t}\\Lambda_j) P^{-1} \\overline{B}$, where $P$ is an invertible matrix. Here, $CP$ can be seen as $W_o$ in attention, $P^{-1} \\overline{B}$ as $W_v$, and each diagonal element in $\\prod_{j=i+1}^{t}\\Lambda_j$ as an attention score $w_{t,i}$ for each channel. The difference is that here $w_{t,i} = \\prod_{j=i+1}^{t}\\lambda_j$, rather than $w_{t,i} = f(x_t W_q W_k^\\top x_i^\\top)$ in attention. In the paper's channel-wise AR format: $o_t = w_{t,i} v_i, \\ v_i = P^{-1} \\overline{B} x_i$\n\nIn summary: Diagonalizable Mamba or S4 (linear recurrent unit) can be viewed as **autoregression on each diagonal channel**, allowing direct incorporation of ARMA structure. However, Mamba or S4 (vanilla) can only be viewed as vector autoregression (SSM system), making it difficult to directly apply our method, which would require exploration in the VARMA domain.\n\n> Further interpretability analysis & notations are not clear.\n\nThank you for your suggestion. We illustrate using the hierarchical AR/MA weights visualization in Figure 9. The AR/MA weights across the 3 layers exhibit different patterns. The first layer's AR weights capture detailed long-term and cyclic relationships corresponding to different input lengths, while the second and third layers' AR weights capture common stable cyclic patterns. The first layer's MA weights focus on **distinctive short-term dependencies** across various input lengths, the second layer focuses on **shared decreasing short-term effects** with fixed block lengths, and the third layer emphasizes **shorter-term correction effects**. These AR/MA weights help make the model's operating mechanism more understandable compared to pure AR models.\n\nWe will add more visualization analyses in the revision and move smaller, less clear text from figures to captions for better readability.", "claims_and_evidence": "The paper claims that WAVE-attention-equipped Transformers outperform state-of-the-art TSF models while maintaining O(N) time complexity, and that its indirect MA weight generation allows efficient modeling of short-term effects without significantly increasing model size.", "methods_and_evaluation_criteria": "Standard TSF benchmarks and baseline comparisons with recent Transformer architectures such as PatchTST, iTransformer, and DLinear.", "theoretical_claims": "The paper builds on ARMA modeling principles and argues that WAVE correctly extends attention mechanisms with an implicit MA term. The mathematical formulation for the indirect MA weight generation is clearly derived, showing that the method preserves the computational efficiency of linear attention while approximating a full ARMA structure.", "experimental_designs_or_analyses": "The one-step forecasting setup ensures fair comparisons between models. The long-term TSF experiments are insightful, demonstrating that WAVE Transformers scale effectively with increased lookback lengths, while other models suffer from performance degradation.", "relation_to_broader_scientific_literature": "The study connects to statistical ARMA modeling, demonstrating how principles from classical time series analysis can enhance modern neural architectures.", "essential_references_not_discussed": "The paper adequately cites recent Transformer-based TSF models, but lacks statistical forecasting approaches (e.g., deep state-space models and mamba, etc).", "other_strengths_and_weaknesses": "Pros\n\n1. Good results. The proposed model consistently outperforms state-of-the-art TSF baselines across 12 datasets\n2. The idea looks interesting and works well. WAVE attention successfully incorporates ARMA modeling principles into Transformer attention, improving both long-term dependency handling (AR) and short-term fluctuation modeling (MA) without increasing computational complexity.\n3. The paper is well-written and well-organized.\n\nCons\n\n1. The model is primarily tested on single-channel TSF tasks, and its performance on multivariate time series remains unexplored.\n2. Its potential application to other sequential modeling tasks (e.g., NLP or reinforcement learning) is not discussed\n3. Please elaborate more on the relation between mamba and WAVE.", "other_comments_or_suggestions": "Further interpretability analysis (e.g., visualization of learned MA weights over time) would strengthen the paper’s conclusions.\nThe notations are not clear.", "questions_for_authors": "+ How does WAVE perform on multivariate time series forecasting, particularly when dealing with highly correlated input series?\n+ Could WAVE be generalized to non-time-series applications, such as language modeling or reinforcement learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UYrEUgKi4m", "reviewer": "Reviewer_ycDF", "summary": "The paper proposes WAVE, a novel attention mechanism integrating autoregressive (AR) and moving average (MA) components for time series forecasting (TSF). The key contributions include:\n\nDemonstrating that a decoder-only autoregressive Transformer, with proper tokenization and preprocessing, achieves performance comparable to state-of-the-art (SOTA) baselines.\n\nIntroducing an ARMA structure into autoregressive attention via an indirect MA weight generation method, which maintains linear time complexity and parameter efficiency.\n\nValidating WAVE's effectiveness across 12 TSF benchmarks, showing that it improves AR-based Transformers and achieves SOTA results. Experiments demonstrate WAVE's superiority over existing methods in both short- and long-term forecasting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> It lacks rigorous theoretical proofs (e.g., asymptotic complexity analysis to substantiate WAVE’s linear time complexity claims)\n\nThank you for your suggestion. We provide the time complexity analysis below:\n\n**Proposition** For a sequence of length $N$ and embedding dimension $d$, WAVE attention based on **efficient linear attention** maintains a time complexity of $O(Nd^2)$, which is linear in the sequence length.\n\n**Proof** We analyze the complexity of each component of WAVE attention separately:\n\n**AR Component Complexity** For linear attention variants, computing $\\mathbf{o}^{\\text{AR}}_t$ for all positions $t$ requires:\n- Computing query, key, value matrices: $O(Nd^2)$\n- For each position $t$, computing a running sum $\\mathbf{S}\\_t = \\mathbf{S}\\_{t-1} + (\\mathbf{k}^{\\text{AR}}\\_t)^\\top \\mathbf{v}\\_t$: $O(Nd^2)$\n- Computing $\\mathbf{o}^{\\text{AR}}\\_t = \\mathbf{q}\\_t \\mathbf{S}\\_t$ for all $t$: $O(Nd^2)$\n\nTotal AR component complexity: $O(Nd^2)$\n\n**MA Component Complexity** The indirect MA weight generation and computation involves:\n- Computing residuals $\\mathbf{r}\\_j = \\mathbf{v}\\_{j+1} - \\mathbf{o}^{\\text{AR}}\\_j$: $O(Nd)$\n- Computing $\\mathbf{q}^{\\text{MA}}$ and $\\mathbf{k}^{\\text{MA}}$ matrices: $O(Nd^2)$\n- Applying activation functions $\\phi^{\\text{MA}}\\_q$ and $\\phi^{\\text{MA}}\\_k$: $O(Nd)$\n- For each position $t$, computing a running sum $\\mathbf{T}\\_t = \\mathbf{T}\\_{t-1} + \\phi^{\\text{MA}}\\_k(\\mathbf{k}^{\\text{MA}}\\_t)^\\top \\mathbf{r}\\_t$: $O(Nd^2)$\n- Computing $\\mathbf{o}^{\\text{MA}}\\_t = \\phi^{\\text{MA}}\\_q(\\mathbf{q}^{\\text{MA}}\\_{t-1}) \\mathbf{T}\\_{t-1}$ for all $t$: $O(Nd^2)$\n\nTotal MA component complexity: $O(Nd^2)$\n\n**Final Output Complexity** Computing $\\mathbf{o}_t = (\\mathbf{o}^{\\text{AR}}_t + \\mathbf{o}^{\\text{MA}}_t)\\mathbf{W}_o$ for all $t$: $O(Nd^2)$\n\nTherefore, the total time complexity of WAVE attention based on linear attention is:\n$O(Nd^2 + Nd^2 + Nd^2) = O(Nd^2)$\n\nThis confirms that WAVE attention maintains linear time complexity with respect to the sequence length $N$ when applied to **efficient linear attention** mechanisms.\n\n**Corollary** For WAVE attention based on **standard softmax attention**, the time complexity remains $O(N^2d + Nd^2)$, matching the underlying softmax attention mechanism.\n\n\n> 1. The ablation studies could be expanded to validate the impact of AR Transformer components (e.g., tokenization strategies, weight-sharing mechanisms) \n> 2. Limited ablation studies to isolate the contributions of key components (e.g., tokenization, AR/MA term interactions)\n> 3. Explore the generalizability of WAVE to multivariate time series\n\nThank you for your suggestions. We provide experimental results in [link (Table 2)](https://anonymous.4open.science/r/WAVE-Rebuttal-B8D2) comparing channel-independent tokenization with channel-mixing tokenization, as well as results without autoregressive training loss. These findings support our claims. Additionally, without weight-sharing, parameter size would exceed pure AR attention, creating an unfair comparison with the same number of layers. We'll discuss this further in the revision appendix. Regarding AR/MA terms, every experiment includes comparisons between pure AR and ARMA, effectively serving as an ablation study. For interpretability, please refer to the following discussion:\n\nWe illustrate using the hierarchical AR/MA weights visualization in Figure 9. The three layers exhibit different patterns. First layer AR weights capture detailed **long-term and cyclic relationships** at different input lengths, while second and third layers capture **common stable cyclic patterns**. First layer MA weights focus on distinct short-term dependencies across input lengths, second layer on shared decreasing short-term effects of fixed block length, and third layer on shorter-term correction effects. These AR/MA weights make the model's operation more interpretable compared to pure AR models.\n\n> 1. Does not cite TimesNet \n> 2. Discussion of TimesNet's 2D temporal modeling approach \n> 3. The absence of a dedicated related work section in the main text\n> 4. Potential inconsistencies in Figure 1\n\nThank you for your suggestion. TimesNet uses 2D convolution on adjacent time series to simultaneously aggregate channel-wise and temporal-wise information, which differs conceptually from AR attention-based methods. We have provided performance comparisons with TimesNet in [link (Table 3)](https://anonymous.4open.science/r/WAVE-Rebuttal-B8D2).\n\nWe will add a related work section in our revision to discuss recent TSF models beyond attention-based methods.\n\nWe have verified that Figure 1 accurately reflects the ARMA structure calculation method. In our revision, we will move smaller fonts from Figure 1 to the caption for improved clarity.", "claims_and_evidence": "The paper's primary claims are largely supported by experiments. For instance, Figure 1 and Table 2 illustrate that AR Transformers perform competitively against baselines, while all WAVE variants outperform their AR counterparts. Table 7 further confirms that WAVE maintains computational efficiency in terms of FLOPs and parameter counts.", "methods_and_evaluation_criteria": "The ARMA structure in WAVE is well-designed: the MA term models short-term impacts via error accumulation, and the indirect MA weight generation avoids explicit matrix inversion through linear attention mechanisms. These design choices are methodologically sound and align with the goal of balancing efficiency and performance.", "theoretical_claims": "While the paper formalizes the ARMA structure and WAVE attention, it lacks rigorous theoretical proofs (e.g., asymptotic complexity analysis to substantiate WAVE’s linear time complexity claims). The validity of the method is primarily empirically justified.", "experimental_designs_or_analyses": "The experimental section supports the main claims but has limitations. For example, the ablation studies could be expanded to validate the impact of AR Transformer components (e.g., tokenization strategies, weight-sharing mechanisms) on performance.", "relation_to_broader_scientific_literature": "This work situates itself within the broader scientific literature through three principal connections. First, it extends the foundational architecture of decoder-only Transformers (Vaswani et al., 2017) and their computationally efficient variants, establishing their applicability to time series forecasting via deliberate tokenization strategies.  Second, the study introduces a novel integration of classical autoregressive moving average (ARMA) principles (Box et al., 1974) into modern attention mechanisms, effectively bridging decades-old statistical forecasting techniques with contemporary deep learning paradigms. Third, the study resolves limitations of exponential decay in gated attention (e.g., MEGA) by explicitly separating MA terms, preserving long-range capabilities while capturing local patterns.", "essential_references_not_discussed": "The paper proposes a decoder-only Transformer with ARMA-enhanced attention (WAVE) for time series forecasting, claiming state-of-the-art performance. However, it does not cite TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (Wu et al., ICLR 2023), which also addresses multi-periodicity modeling and achieved SOTA results in forecasting tasks. A discussion of TimesNet's 2D temporal modeling approach and its differences from WAVE's ARMA mechanism would strengthen the technical context, and inclusion in experiments is recommended for rigorous benchmarking.", "other_strengths_and_weaknesses": "Strengths:\n\nThe novel integration of the ARMA structure with efficient attention mechanisms provides a fresh perspective for time series forecasting (TSF), demonstrating the potential of AR Transformers in this domain.\n\nThe extensive experimental validation across multiple benchmarks strengthens the credibility of the results.\n\nWeaknesses:\n\nThe absence of a dedicated related work section in the main text, which may hinder contextualizing the method within existing literature.\n\nLimited ablation studies to isolate the contributions of key components (e.g., tokenization, AR/MA term interactions).\n    The core novelty of introducing ARMA into TSF could benefit from further justification.\n    Potential inconsistencies in Figure 1 (e.g., arrow directions in the schematic diagram).", "other_comments_or_suggestions": "Future work could explore the generalizability of WAVE to multivariate time series, as the current experiments focus primarily on univariate scenarios.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uHxWoX7An0", "reviewer": "Reviewer_kz7F", "summary": "The author incorporates a moving average term into the autoregressive attention model for linear attention mechanisms, achieving state-of-the-art performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The method in this paper requires increasing the input length when the prediction length is extended. Even though, as the authors suggest, this method can be viewed as a patch for patchtst with added AR loss, patchtst itself does not require a longer input length when increasing the prediction length. Overall, while this design seems to alleviate the issue of error accumulation, it is not entirely natural. \n\nThank you for your suggestion. Our paper's main contribution is designing the ARMA attention mechanism and demonstrating that attention with ARMA structure significantly outperforms pure AR attention. Our tokenization method is merely a low-cost prerequisite problem introduction, aimed at incorporating AR training into the existing baseline experimental environment with minimal modifications. This allows us to isolate the improvement of adding the MA term to AR attention without introducing additional complex designs. We did this **only to control the introduction of irrelevant factors**.\n\nWe use non-overlapping PatchTST-style tokenization + AR loss to maintain a **pure autoregressive relationship** between tokens. This ensures the MA term calculation better aligns with its original design. Otherwise, the MA term calculation would be affected by overlapping portions between the current and previous tokens. While this wouldn't significantly impact performance, using such results for comparison wouldn't sufficiently demonstrate that ARMA structure's performance gain comes from properly introducing the MA term. Similarly, if we mixed channel information in tokenization to model multivariate relationships, it would improve model performance to some extent (see [link (Table 2)](https://anonymous.4open.science/r/WAVE-Rebuttal-B8D2)). However, mixing channel information would place output tokens containing single series information and input tokens containing channel relationships in different spaces, breaking the AR training objective. These additional factors would weaken our experimental results showing ARMA attention's improvement over AR forecasting, requiring additional ablation studies. Therefore, to straightforwardly demonstrate our results, we used this constrained pure AR setting to prove our conclusions while controlling these extra structures.\n\nAs shown in Table 3, this constrained setting provides additional insights: compared to current baselines with L_I≈512, it adapts to longer context lengths. This suggests some advantages for this **low-cost tokenization approach** when transferring AR attention to current experimental settings. However, these experiments were still designed to highlight ARMA attention's improvements over pure AR attention, which remains our primary objective.\n\n> Sensitivity of parameters not analyzed.\n\nThank you for your suggestion. We modified the seed and ran our model five times, with results shown in [link (Table 4)](https://anonymous.4open.science/r/WAVE-Rebuttal-B8D2). We will add these additional experiments to the revision.\n\n> Figure 3 is too small to understand.\n\nThank you for your suggestion. We will move the smaller font in Figure 3 to the caption to help readers understand it better.\n\n> Why is each row of matrix B in the visualization basically uniform?\n\nThis is an intentional design choice, as we explained in detail in the footnote at the bottom right of page 5 and demonstrated in Figure 7. When matrix B lacks smoothness, the long-term components in the lower triangle dominate, preventing effective short-term modeling of MA terms.\n\nIncreasing element variance in matrix $\\mathbf{B}$ causes greater fluctuations in longer-term elements of $\\mathbf{\\Theta}$. This can be understood through the simplified mean form in Eq. 5: $\\theta_{ij} = b(1 + b)^{i-j-1}$. The lower-left portion of $\\mathbf{\\Theta}$ with larger values of $i-j-1$ corresponds to longer-term components. Due to the larger exponents in this area, variance is more **significantly affected** by variance in $b$. This is described in the paper as:\n\n---\nIn the key activation, $ \\alpha $ controls the variance of each row in the $ \\mathbf{B} $ matrix, indirectly influencing the amount of long-term information (lower left) in the MA weights $ \\mathbf{\\Theta} $. Increasing $ \\alpha $ would make the MA weights focus more on modeling long-term information. However, since we want the AR weights to handle the long-term component, we set $ \\alpha $ to a relatively small value. This explains why the rows of the $ \\mathbf{B} $ matrix **appear smooth** in the visualization. Refer to Fig. 7 for more details on $\\alpha$, and see Fig. 8 for the effects of reversed positive $\\phi_q$.", "claims_and_evidence": "1. Effectiveness of the decoder-only autoregressive Transformer  \n   - In time series forecasting (TSF), the previously overlooked decoder-only autoregressive Transformer can achieve results comparable to top baseline methods with appropriate tokenization and training strategies.  \n\n2. Incorporating the full ARMA structure into autoregressive attention  \n   - Inspired by the ARMA model and recent advances in linear attention, this paper integrates the full ARMA structure into autoregressive attention mechanisms, improving long-range and local temporal modeling.  \n\n3. Proposing WAVE attention\n   - WAVE attention combines autoregressive (AR) and moving-average (MA) components, enhancing adaptability to different attention mechanisms while improving their ability to model long-range dependencies and local patterns.  \n   - An indirect MA weight generation method introduces the MA component without increasing time complexity or parameter size in efficient attention models.  \n\n4. Effectiveness of indirect parameter generation for MA weights  \n   - The study shows that indirect parameter generation can implicitly produce MA weights suited for capturing local temporal effects.  \n\n5. Experimental validation of WAVE attention  \n   - WAVE attention consistently enhances autoregressive attention mechanisms and achieves state-of-the-art (SOTA) performance in time series forecasting tasks.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "contains no proofs.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "s1. The WAVE attention mechanism captures short-term effects through the MA term, allowing the AR term to focus more on long-term and periodic patterns, thus balancing long-term and short-term dependencies.\n\ns2. The WAVE attention mechanism proposed in the paper introduces the MA term while maintaining the same time complexity (O(N)) and parameter scale as the underlying efficient attention model.\n\ns3. The experiments in the paper are comprehensive, covering both long-term and short-term time series forecasting, as well as ablation experiments. \n\nw1. The method in this paper requires increasing the input length when the prediction length is extended. Even though, as the authors suggest, this method can be viewed as a patch for patchtst with added AR loss, patchtst itself does not require a longer input length when increasing the prediction length. Overall, while this design seems to alleviate the issue of error accumulation, it is not entirely natural. Existing decoder-only models like Autotimes do not need this design yet still remain state-of-the-art, demonstrating the true capability of decoder-only models: the ability to predict future lengths of any size.\n\nw2. Sensitivity of parameters not analyzed.", "other_comments_or_suggestions": "1. figure 3 is too small to understand", "questions_for_authors": "q1. Why is each row of matrix B in the visualization basically uniform?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qznxq1mplU", "reviewer": "Reviewer_Y1Dg", "summary": "In this paper the authors integrate the attention mechanism used for time series forecasting with the concepts of moving average in used in classic statistical ARMA models. In particular they device the indirect MA weights on top of patched tokenized time series with the emphasis on linear attention level complexities. They show via empirical study that the resulting WAVE mechanism helps improve the forecast accuracy significantly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Pretrained decoder-only models & end to end evaluation\n\nWe'll clarify our introduction regarding design purpose and use this as evidence in Section 2.2 showing AR attention-based TSF models perform comparably to other structures.\n\n> Intuition behind WAVE on a patched level\n> Context of ARMA\n> Inductive biases issue\n\nThanks for this insightful question. Due to MLPs and non-linearities, patch-level AR weights resist direct interpretation on original observations. Our parameterization provides **separate AR/MA attention weights** whose visualizations (Figures 6, 9-13) reveal **token-level cyclic patterns** captured by AR weights and **short-term effects** by MA weights.\n\nEach attention layer's value part can be viewed as a **reconstructed aggregation** of observations through a factor model. The Sequence Value section above each figure shows these layer-internal observations, essentially optimal input observations for ARMA structure learned at each layer.\n\nThis optimization occurs in all multi-layer AR attention models. For non-first layers, each input sequence step differs from the original observation as each layer aggregates information from other steps. While this makes interpreting attention weights on original inputs difficult, interpreting them on the model's constructed optimal input sequence remains meaningful.\n\nIn Figure 9, the three layers' AR/MA weights show distinct patterns. First-layer capture detailed **long-term/cyclic relationships** for different input lengths, while second/third layers capture stable cyclic patterns. MA weights show varying focus: first layer on distinctive **short-term dependencies**, second on shared decreasing short-term effects, and third on shorter corrective effects. This separation helps understand model mechanics better than pure AR models.\n\nRegarding inductive bias, we believe all recent networks works on inductive bias while we need to consider its generalization ability. Our ARMA attention has demonstrated good generalization across 12 widely-used TSF datasets at different scales, proving its effectiveness as a structural prior.\n\n> Other tokenization strategies\n\nWe use this tokenization to maintain next-step autoregressive relationships while covering the entire forecasting horizon. This represents PatchTST-style tokenization with autoregressive loss, enabling direct benchmark comparison. Our focus is efficiently introducing **MA terms to AR attention**, and we experimented with inter-channel mixing (results at [link (Table 2)](https://anonymous.4open.science/r/WAVE-Rebuttal-B8D2)), showing multivariate relationships do improve performance.\n\n> Encoder-only vs Decoder-only\n\nOur decoder-only pure AR model extends PatchTST with **autoregressive loss**. We omitted multi-step autoregressive prediction comparisons since baselines used different training objectives. Experiments confirm AR/ARMA models outperform prior approaches in one-step forecasting. Though our model shows advantages with shorter contexts due to autoregressive training, this is a secondary benefit as longer contexts are preferred when available. Since decoder-only models operate within training windows, our evaluation uses **full contexts**, maintaining consistency with previous benchmarks.\n\nAdditionally, decoder-only attention naturally performs multi-task training across varying context lengths, learning more adaptable representations that enhance generalization for TSF.\n\n> Quadratic attention works worse / are way from optimally trained? Why linear better?\n\nWe agree. Research like DLinear, PatchTST, and iTransformer highlights serious overfitting in TSF Transformers. Softmax attention's non-linear capacity easily fits noise.\n\nLinear attention has a special **dual structure**:\n\nAutoregressive form of attention:\n$$\n\\mathbf{o}\\_t = \\sum\\_{i=1}^t \\mathbf{w}\\_{t,i} \\mathbf{v}\\_i \\ , \\mathbf{w}\\_{t,i} = \\mathbf{q}\\_t \\mathbf{k}\\_i^\\top \\in \\mathbb{R}^d\n$$\n\nVector Autoregressive form of linear attention ($\\mathbf{q}\\_t \\mathbf{k}\\_i^\\top = \\mathbf{k}\\_i \\mathbf{q}\\_t^\\top$):\n$$\n\\mathbf{o}\\_t = \\sum\\_{i=1}^t \\mathbf{k}\\_i \\mathbf{A}\\_{t,i} \\ , \\mathbf{A}\\_{t,i} = \\mathbf{q}\\_t^\\top \\mathbf{v}\\_i \\in \\mathbb{R}^{d \\times d}\n$$\n\nThis dual linearity provides both **AR form and dynamic vector autoregression form**, providing **regularization**, and improving **linear relationship learning** in both channel- and token-wise dimensions.\n\n> Commonly used lookback / forecast horizon pairs\n\nFor long-term forecasting, we used $L_I \\in \\{512, 1024, 2048, 4096\\}$ with all $L_P$ values. We tested baseline setting $L_I=512$ and others. For stable token counts, we used fixed $L_I=1024$ for short-term prediction and specific combinations for long-term prediction. Table 3 demonstrates model's stability across different $L_I$ lengths. Our model works effectively with **very long lookbacks** ($L_I=4096$), showing excellent adaptability to extended contexts, which is a capability rare among existing models.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "N/A.", "essential_references_not_discussed": "Literature on pretrained decoder-only time series specific models have discussions on end to end evaluation usually as part of the empirical study, see e.g., \n\n[1] Liu, Yong, et al. \"Timer: generative pre-trained transformers are large time series models.\" Proceedings of the 41st International Conference on Machine Learning. 2024.\n[2] Das, Abhimanyu, et al. \"A decoder-only foundation model for time-series forecasting.\" Forty-first International Conference on Machine Learning. 2024.", "other_strengths_and_weaknesses": "Strength:\n- The WAVE mechanism, especially the patch level MA formulation, is novel among TSF methods.\n- The design and the empirical study included in the paper are following common rigorous practices thus are relatively convincing.\n\nWeakness:\n- The intuition and the theory behind the proposal are relatively weak, especially compared to the original context when ARMA applies.\n- The empirical study does not fully show the ARMA structure is working as intended.", "other_comments_or_suggestions": "Math in the paper could be better organized. Consider provide more intuition, a central place for notation definitions, and more concise math writings.", "questions_for_authors": "I am overall positive towards this submission, and plan to increase my recommendation once the following concerns / questions are addressed. \n\n1. While ARMA has strong statistical interpretation, ARMA on the patch level in a stacked transformer becomes theoretical hard to understand. While there are examples (e.g., Fig 6) attempting to visualize the ARMA effect, what's the intuition behind WAVE when on a patched level, and how to claim it's working as intended instead of, e.g., merely introducing inductive biases in the model that work for the selected benchmark datasets?\n\n2. Tokenization: while the authors claim an appropriate tokenization is required, the chosen tokenizer in the paper is a MLP(?) structure on top of patched time series which are common in PatchTST line of work. Any consideration of other tokenization strategies?\n \n3. encoder-only vs decoder-only: the benefit of decoder-only mainly comes from its training efficiency and autoregressive decoding. The former is more crucial for pretrained models, so I wonder if we can see see any empirical study results on autoregressive decoding beyond a single step of the proposed method. Otherwise what's the justification of choosing decoder-only?\n\n4. It's counterintuitive to see quadratic attention works worse than the linear attention most of the time, given the effective context length after patching is not long in the empirical study. What's the explanation, or if the quadratic models are way from optimally trained? Can we include more commonly used lookback / forecast horizon pairs in, e.g., the PatchTST paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new weighted attention mechanism specifically designed for time series forecasting, motivated by some classic ideas. The approach is novel and aims to better capture both long-term and short-term dependencies, supported by promising experimental results. Overall, the reviews are positive and detailed.

That said, the reviewers raised several valid concerns, including: (i) limited intuition and theoretical grounding behind the proposed approach; (ii) missing experimental details; and (iii) missing experiments on multivariate time series. The authors made good use of the rebuttal phase to address these and other issues, and one reviewer even increased their score in response.

Overall, I find this to be an interesting and promising paper, with sound methodology and results. My main concern is that the paper did not feel fully ready for publication, as evidenced by the number of concerns raised during the review process. Integrating all this feedback into the manuscript will require substantial revision. I strongly encourage the authors to incorporate all the discussed clarifications and improvements if given the opportunity to submit a camera-ready version. In particular, the presentation and discussion would benefit from further refinement. Moreover, strengthening the theoretical connection to classical time series models, such as ARIMA, would also add valuable context and depth to the work, and possibly help to motivate future work.

I recommend accepting the paper.
Decision: Accept (poster)