Linear Transformers as VAR Models:  Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting
Authors: Jiecheng Lu
Shihao Yang
Venue: ICML 2025 poster (2025)
Keywords: Linear Attention
Transformer
Time Series Forecasting
Vector Autoregression
Primary area: deep_learning->sequential_models_time_series

Abstract: Autoregressive attention-based time series forecasting (TSF) has drawn increasing interest, with mechanisms like linear attention often outperforming vanilla attention. However, deeper Transformer architectures frequently misalign with autoregressive objectives, obscuring the underlying VAR structure embedded within linear attention and hindering their ability to capture the data generative processes in TSF. In this work, we first show that a single linear attention layer can be interpreted as a dynamic vector autoregressive (VAR) structure. We then explain that existing multi-layer Transformers have structural mismatches with the autoregressive forecasting objective, which impair interpretability and generalization ability. To address this, we show that by rearranging the MLP, attention, and input-output flow, multi-layer linear attention can also be aligned as a VAR model. Then, we propose Structural Aligned Mixture of VAR (SAMoVAR), a linear Transformer variant that integrates interpretable dynamic VAR weights for multivariate TSF. By aligning the Transformer architecture with autoregressive objectives, SAMoVAR delivers improved performance, interpretability, and computational efficiency, comparing to SOTA TSF models.

Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecasting
Jiecheng Lu Shihao Yang
Abstract
self-attention, computes attention weights between the cur- rent token and all preceding tokens during prediction. How- ever, using softmax to compute the N x N attention map re- sults in large O(N2) time complexity as the sequence length grows. To address the efficiency bottleneck, researchers have developed efficient variants like Linear Transformers (Katharopoulos al., 2020; Hua al., 2022). By replac- ing softmax with a linearizable kernel, linear attention re- duces complexity from O(N²) to O(N) by maintaining 2d-dimensional hidden state instead of forming the full x N attention map (Choromanski et al., 2021; Hua et al., 2022; Sun et al., 2023). Although often underperforms vanilla attention in complex tasks like NLP, studies show that in simpler tasks, such as time series forecasting (TSF), linear attention can outperform vanilla attention (Patro & Agneeswaran, 2024; Lu et al., 2024a; Behrouz et al., 2024).
Autoregressive attention-based time series fore- casting (TSF) has drawn increasing interest, with mechanisms like linear attention sometimes out- performing vanilla attention. However, deeper Transformer architectures frequently misalign with autoregressive objectives, obscuring the un- derlying vector autoregressive (VAR) structure embedded within linear attention and hindering their ability to capture the data generative pro- cesses TSF. In this work, we first show that single linear attention layer can be interpreted as dynamic VAR structure. We then explain that existing multi-layer Transformers have struc- tural mismatches with the autoregressive forecast- ing objective. which impair interpretability and generalization ability. To address this, we show that by rearranging the MLP, attention, and input- output flow, multi-layer linear attention also be aligned as a VAR model. Then, propose Structural Aligned Mixture of VAR (SAMoVAR), a linear Transformer variant that integrates in- terpretable dynamic VAR weights for multivari- ate TSF. By aligning the Transformer architec- ture with autoregressive objectives, SAMo VAR delivers improved performance, interpretability, and computational efficiency, comparing to SOTA TSF models. The code implementation is avail- able at this link.
Autoregressive modeling has long history in time series forecasting. Traditional methods like ARIMA handle uni- variate series through autoregression, differencing, and mov- ing averages (Winters, 1960; Holt, 2004), while Vector Autoregression (VAR) (Stock & Watson, 2001; Zivot & Wang, 2006) extends this to multivariate settings by captur- ing cross-variable lag dependencies. Although widely used in fields like economics and climate due to their interpretabil- ity and theoretical guarantees (Burbidge & Harrison, 1984; Pretis, 2020), these models' linear assumptions and fixed lag orders limit their ability to capture complex patterns. With growing data seale and complexity, deep learning-based TSF models, especially iitention-based approaches, have outperformed traditional AR/VAR methods (Li et al., 2019; Zhou et al., 2021; Nie et al., 2022; Liu et al., 2024).
1. Introduction
Previous research offers various perspectives on linear at- tention, viewing it as an RNN with linear state updates, a dynamic temporal projection, or fast weight programming (Katharopoulos al., 2020; Schlag al., 2021). In this paper, we show that linear attention naturally contains VAR structure. While single-layer linear attention mod- ule can be directly interpreted as a VAR model, stacking multiple layers introduces structural mismatches with the time series generative process, reducing its effectiveness for TSF. We demonstrate that by reorganizing the input-output flow, multi-layer linear attention can fully align with a VAR
In recent years, autoregressive decoder-only Transform- ers have made significant strides (Vaswani, 2017; Radford, 2018), powering Large Language Models (LL.Ms) (Brown al., 2020; Touvron et al., 2023) capable of handling com- plex sequential data. Their core mechanism, autoregressive
'Georgia Institute of Technology. Correspondence Shihao Yang <shihao.yann yang lann@isyc.gee wisy..gatcch....m www.ch..ccc.........
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Linear Transformers as VAR Models
model. Further, we propose Structural Aligned Mixture of VAR (SAMoVAR), which enhances linear Transformers for TSF, improving both accuracy and interpretability.
1/6 /dd scaling), the mechanism becomes vanilla attention. Re- placing a with an identity mapping simplifies it to linear attention with an identity kernel. Adding an MLP layer after the attention layer, as X := X - MLP(LN(X)), forms a standard autoregressive Transformer block. We will explore the attention from multiple perspectives (P-). An attention layer dynamically computes a temporal mapping weight ma- trix QK E RNN for sequence of length N. For each imput step t, it generates attention map over all N tokens as dynamic weights. In vanilla attention, these weights are to sum to 1. In autoregressive attention, lower-triangular mask M ensures each step only attends to positions up to t. Thus, the attention layer functions as a variable-length dynamic linear layer on the input value se- quence V V(Vaswani, 2017; Katharopoulos al., 2020; Yang et al., 2024).
The main contributions are summarized as follows:
1) We provide a new perspective by interpreting a single- layer linear attention module as a VAR structure, where the key represents the observation and the outer product of value and query forms dynamic VAR weights.
2) We analyze how the designs of existing Transformers lead to misalignments with a VAR model's time series gen- erative objective, including mismatched losses, inconsistent residual streams, and unbalanced observation weighting.
3) We show that properly arranging the input-output flow in a linear Transformer allows multi-layer linear attention to act as a expressive dynamic VAR model. With / layers, each past step's 's influence on future steps is captured through "temporal influence path" involving up to / - 1 intermediate nodes, enhancing interpretability.
P2: Recurrent Form and Autoregression
Autoregressive attention can be viewed as a step-by-step generative process with a recurrent formulation. For the input H, at step f, the output og is: Or Ei- k.). Ei (q.),.)
4) Based on this aligned structure, we propose SAMo VAR for TSF. Experiments demonstrate that it surpasses previous TSF models in accuracy, interpretability, and efficiency.
where q:, ks, 2y R Ixd are query, key, and value vectors at step 1. When (g:: kt) = exp(q.k..) ), this represents vanilla attention, relying on all previous keys ((......)) and values VII....!!. If og k;) is derived from a kernel fea- ture map k(g,, k;) = =>()))((k))) (kk:)) T, the computation is lin- carized as: O, d(q.) Ei o(k))
2. Background
2. Time Series Forecasting
o(q.) Σi= (ok,) This avoids the full N x N attention map by aggregating past information into a hidden state. Ignoring the denominator, the simplified form is: O = q: Σ Σ k. u; Here, attention acts like an RNN with 2D hidden state k2 0; E R and identity state updates. Studies have shown comparable performance without normalization, so we use this simplified form (Zhai et al., 2021; Mao, 2022; Qin et al., 2022; Sun et al., 2023; Yang et al., 2024). In this view, attention is a dynamic au- toregressive model with shared weights Wt, = o k;) across all d channels: o, = -E W,,
Time Series Forecasting (TSF) aims to predict future values in a multivariate sequence S e RxCC' split into a historical part S, E LxC and a future part Sp E RLLxxC where L- Li + Lp are the series lengths, and C is the number of channels. The task is to learn a function f LCC R LC that generates Sp f(S)), given the input Sy.
2.2. Preliminaries: Attention Mechanisms
Previous studies have examined autoregressive attention mechanisms from various angles, emphasizing their com- mon feature of dynamic weights (Katharopoulos et al., 2020; Hua et al., 2022; Sun et al., 2023). For an input sequence of length N and dimension d. represented as X e IR Nxd with each token denoted by æ, E Rixdd a single-head au- toregressive attention layer is formulated as:
P3: Fast Weight Programming Fast weight programming (FWP) refers to the process of dynamically determining a set of linear predictor weights for each step in the input sequence, i.e., Wrwp, g 2x,) E Rdxd In linear attention, this is achieved using a summation aggregator to combine past weight information (Schlag et al., 2021), WFWP, = - (kk)) T Us, which serves as a dynamic linear predictor for qt- The final output at step t is then
(X) (M @ (QKT) VW..
with K, V XW 9: XW XW., ... (1)
X x X + Att)(LN(X))
o = q: W FFW...
Here, Wg. WR. W.. o. W. E Rd are the projection ma- trices for query, key, value, and output. The causal mask M E R. XNN ensures autoregressive behavior, with Myj = [i - 00 lfi < j}. allowing only current and past positions. Attn--) and LN(-) denote the attention and layer normalization functions. When a is softmax (ignoring the
2.3. Linear Attention as VAR
Recent TSF research shows that linear attention- softmax- outperforms vanilla attention (Patro & Agneeswaran, 2024; Lu et al., 2024a; Behrouz et al., 2024). While it can be viewed an RNN form of fast weight
2

---

Linear Transformers as VAR Models
programming (FWP), these perspectives do not directly link to the autocorrelation or generative nature of TSF data. In this section, we demonstrate that linear attention naturally forms a dynamic VAR structure, making it well-suited for modeling TSF data generation.
strict VAR model form:
ki41 01 + u
T qe ki +
+ue
(3)
P4: Vector Autoregression classic Vector Autoregressive model (p) with lag p uses parameter matrices A, E pdxd to model dependencies on p previous time steps:
Ar
y: -A1 + A2y1- + ApDi-p u4
where u, E R is the residual not explained by the dy- namic VAR system. To align linear attention with VAR model, minimizing u, should be part of the training objec- tive. Adding loss term for each layer's 's residual k; o might partially achieve this but would conflict with the over- all Transformer objective.
where y, E lxd represents the observations and u, E Rixd the residual. In the RNN and FWP views, the pa- rameter matrices depend directly on k; and v; and act on the query q, rather than sequentially applying to previous steps as in VAR. However, we can reformulate linear attention to reveal its VAR structure. Since q is a scalar, rearrang- ing terms gives:O: = Σ k. Vt. By transposing this expression, we obtain the VAR form of a single-layer linear attention mechanism:
In a VAR model, the weights are learned to perform for- ward shift for each input position. With attention layers, enforcing this VAR loss would result in / shifts, whereas the Transformer's 's autoregressive objective only requires single-step shift at the output. As result, each layer would need to perform only fractional shift to stay aligned.
0 =
K. A1,; = waa q:
(2)
Residual Stream Next, we focus on how each attention layer operates. In decoder-only Transformers, a common pre-normalization design (Radford, 2018) includes a resid- ual shortcut from input to output, with each block learning the difference between the current and the next step. The at- tention layers gather information from previous steps, grad- ually refining this difference and adding it to the shortcut.
This defines a VAR(t) structure with dynamic weights, where the observations are k. and the rank-1 weight matri- ces Ar.; V. qe are dynamically generated for each step 1. Unlike RNNs or FWP, where weights propagate across steps, these matrices are independently generated at each time step. Thus, autoregressive linear attention forms its own V-R(.) structure at each step, as shown in Figure ((a).
If all attention layers are disabled, the model reduces to a local MLP predictor that maps the current input to the next-step output. With patch tokenization, where each token covers Lp time steps, the model effectively becomes an MLP-based TSF predictor mapping Lp input steps to Lp outputs. Adding an attention layer adjusts the input pattern using past tokens to better match the sext-step difference. Recent studies of in-context learning (Zhang et al., 2023; Akyürek et al., 2022) suggest that with more layers, the model can dynamically approximate predictors like gradient descent, ridge regression, or shallow MLPs. Thus, attention layers prioritize refining local predictors through context, rather than explicitly modeling step-by-step generation con- sidering raw observations.
3. Aligning the Objective of Autoregressive Attention with Autoregressive Forecasting
In this section, we show that while single linear attention layer naturally exhibits a dynamic VAR structure, the de- sign of current multi-layer Transformers diverges from the VAR training objective. As a result, these models lose the beneficial VAR properties for time series forecasting (TSF).
Time series data naturally show temporal dependence. A classic VAR model captures both autocorrelation and cross- correlation in the data generation process, with weight ma- trices Aj decoupling how past values influence future out- comes. Although standard attention and Transformers are effective for modeling complex sequence relationships (c.g., in NLP), their architecture conflicts with VAR's goal of ex- plicitly representing lag-based dependencies. We outline the key sources of this misalignment below.
Input and Output Comparing the attention output in Eq. (I) with the VAR output in Eq. (3), we see that attention out- puts must align with the residual shortcut space, while VAR outputs correspond to the key observations k. VAR repre- sent the data generation process, so their outputs should not be treated as residuals. However, by adjusting the weight matrix at each step, we can align the dynamic VAR struc- ture of linear attention with the residual- based objective for transitioning observations to the next step. We call this a "key shortcut," which adds an identity matrix to the dynamic
VAR Loss and Position Shifting A VAR model is designed to directly capture the relationship between past observa- tions and future values. Suppose we rewrite Eq. (2) into a
3

---

Linear Transformers as VAR Models
2, (1+ paths
222
x
4
X
Lt
X
Zc
J+1 t-1
(b) Travensal and Combination Possible Temporal Influence Paths as the Final
A
k2-2
kg
o2
Illustration of
Step Single Lineur Layer
Mixture Weights
Lin
Of
Output
Output
We
Attn
Atin
Atin
LN
MLP
LN
Ix
Maturn VAR
Nor
Norm
W
W
LN
Attn
bx
LN
MLP
Inpue
"0,
(d) Structural Aligned Mixture of VAR (SAMoVAR) Transformer for Autoregressive Forecasting
(c) Conceptual Illustration of VAR Lincar Alteation Block on Step
(6) Vanilla Transformer Blocks
Figure 1. Visualization of Key Concepts in SAMoVAR. The subfigures highlight different structural and conceptual elements of the model.
VAR weights, guided by an indicator for the output step t.
its ability to model the generative processes of time series data. For a single linear attention layer in Eq. (2), the Trans- former naturally aligns with the VAR objective for one-step shifting, as position -iit operations are confined within the layer. The VAR weights remain balanced across past lags when viewed in the original key observation space. To main- tain this structure, the output and input signals must follow the same recursive equation, and the residual shortcut should share the same normalization as the attention layer's key imputs, avoiding typical pre- pre-normalization shortcuts seen in standard Transformers.
k2t1 k + 0 U
A k +
u
(4)
where A., = o qi
I. I. (i==))-
However, in a pre-normalization setup, the attention layer only processes the transformed input LN(x;) and lacks di- rect access to the original signal x; needed to model the residual. As a result, not possible to establish a strict VAR recurrence involving o and the original signal, re- gardless of adjustments to k or the weight matrices Ar,
4.1. Multi-layer Linear Attention as VAR
Balanced Weights of Observations In a VAR model, all lag positions are initially treated equally, with their influ- ence on future steps determined by learned weights, free from positional bias. In contrast, a multi-layer Transformer requires each token to perform two roles: (1) gather infor- mation via attention to predict its next step and (2) serve as context for future tokens. As layers deepen, accumulated ... x.)
Using a single linear attention layer preserves clear VAR structure but limits the Transformer's expressive power. Each outer product weight v. q: forms a rank-1 matrix, and unlike RNNs or fast weight programming, this VAR for- mulation cannot increase rank through timestep summation. Below, show that when multiple linear attention layers are stacked without MLP layers in between, and o, is di- rectly fed into the next layer, the attention layers can still function as a dynamic VAR model. This model uses the first layer's s key input A the observation, with explicit weight matrices that remain aligned with the autoregressive forecasting objective.
residual updates cause the representation to drift away from the original observation semantically. This drift complicates the VAR-like stepwise shift and leads to uneven weighting of original observations a linear attention-based VAR framework.
4. Structural Aligned Mixture of VAR
Let us denote the output of the first linear attention layer at step by (1) (omitting residuals, normalization, and setting W
We show that the misalignments between linear attention and VAR-based forecasting can be resolved by reorganizing the MLP and attention layers in a linear Transformer. By re- designing the input-output flow, we can enable multi-layer linear attention maintain VAR structure, improving
I). In the single-head case: o, o
Σ(_ Gi
(1)T (1) k where (1) x, w k(")
q.
æ; (!)) (1) x. (!)) Denote the first layer's key input
b V.

---

Linear Transformers as VAR Models
by k!"' = k, and let B be the weight matrix directly
of B "igg." Each path's 's scale controlled by a se- ries of dot-product scalars, e.g., (q) (1 Notably, positions farther from the output step t have more influ- ence paths summed, reflecting the dynamic VAR structure's capacity to capture complex long-term dependencies.
acting on ky. Then:
T A.J B = - =0
vg (T) 1
Robust Path Pruning To maintain numerical stability, we must prevent exploding values in each path's multiplicative chain. Additionally, with distant timesteps, the large number of possible paths can increase weight variance, making it essential for the model to prune unimportant paths. In this setup, any path with query-key dot product of zero is efffectively pruned.
Now take of as input to the second layer. The second-layer
output becomes: of T2) =i=
(v. (2)
q
(2)
k
(2)T where
Li 2) of B = of of = of w!?
After expansions and rearrangements, this can be rewritten in VAR-like form in terms of the original observation ky:
a) Controlling Exploding Values. We use root mean square layer normalization (RMSNorm) on q and v vectors to pre- vent their norms from growing excessively. The weight matrices W, q and W,, are initialized with low variance, en- suring paths start at small scales, especially those with many intermediate points. b) Passive Pruning of Distant Paths. Multi-heads with sufficiently large dimension d increases the chance that qe and v; become orthogonal, resulting in zero dot products that naturally prune unnecessary paths.
0 ())
Σ
B
2 KJ
B
Σ
(2)T q:
(
W
(2)
B
(1)
%
Thus, for / stacked linear attention layers, the final weight [ for the original key observation kj at step t is:
TT P
(1)
WTT
B (1-1)
(5)
Since each layer's output o aggregates multiple influence paths, reusing o directly to form the next layer's q and v can further complicate control over numerical stability. In the structure above, all the q and generate each layer's dynamic VAR weights. Hence, to maintain stability, we choose to compute q!" and o! di- rectly from the original first-layer input signals. Specif- ically, let k = xx(1) For the 1-th attention layer, we
used only to
This shows how stacking multiple linear attention lay- expres-
ers-without MLP layers in between-produse sive dynamic VAR structure for autoregressive forecasting, using the first layer's key input as the observation.
Temporal Influence Path The dynamic weight matrix at step j for step I, Bug is derived by applying the layer- I weights A (1) and W the previous layer's B B((-1) across all intermediate steps i. Repeated multiplication of different WT matrices can lead to numerical instability. To address this, we replace the key projection with the identity matrix I. Under this setup, each term in the summation can be interpreted as a modification (amplification or attenu- ation) of the previous layer's dynamic weights across the intermediate points after step j t. The iterative factor within each path can be expressed as:
parametrize: qi (I) RMSNorm (z) (1)
w(() ), o!!"))
((-1) I. In this setup, the total weight at each position is a combination of multi- rank path matrices. Gradient updates amplify the ranks of important paths while suppressing less significant ones, implicitly applying a low-rank regularization.
RMSNorm' (4) (z, (2) )."), = o
Key Shortcut and Mixture of VAR TTo address instability from initializing all paths with small weights, we introduce a key shortcut by adding I B when t = j. as described in Eq. (4). This effectively incorporates temporal influence path for ( = 0 at each step. Additionally, instead of relying solely on the final layer's output ( we aggregate outputs from all attention layers. This creates mixture of VAR parameters, with the final output expressed as:
p!.) L.i.bi
A A
(
11-1) (1)
11.22
(I)T
(1)
I)T (1-1 (1) ()) Ti
v d 1/ 9.
where t 2 i1 2 i2
1-1 2 j. This describes a temporal influence path from step j to t involving I intermediate timesteps. All possible combinations of in- termediate steps form the complete set of influence paths contributing to B the binomial coefficient path = ((-))+((>>) Because each path representa a rank-1 matrix, the maximum rank
(final) Ec.af (1) Cr =
Σ m=1
B
(m) L.j
+
1 (=
t.jj The number of such paths is given by
Pleasee note that even though we use the name "Mixture of VAR." its final form remains an integrated and complete dynamic VAR model. The "mixture" aspect is reflected in the traversal and combination all possible temporal influence paths.
5

---

Linear Transformers as VAR Models
Here, each y represents the combination of all temporal influence paths from using up to m intermedi- ate points. Consequently, Ct. contains all paths from j to / with up to 1 intermediate points, with total of Em n + paths (as illustrated in Figure I(b)).
8
&
1 &
8541
Transformer
Structural VAR A classic VAR model can be generalized into a Structural VAR (SVAR) Rubbi--Ramieez et al., 2010: Primiceri, 2005) by introducing an invertible matrix D E Rdxd: Dy, instantaneous relationships among variables but requires extra constraints for proper identification. The structural shocks e, are derived by decomposing residuals u, using D. Since interpreting individual channel weights in the learned VAR observations k; can be complex, we use this SVAR form mainly to enhance representational capacity and better interpret residuals. The structural form of our multi-layer linear attention VAR is given as:
Transformer
+ e,", where D captures
4/-2
81
(o) AR Tokenization Univariate Forecasting
(b) ARX Tokenization for Multivariate Forecasting
Figure 2. Illustration of the ARX tokenization, where we use sf to represent the (-th patch token of series j. S settbp.))
tion strategy, which captures univariate relationships while treating other series exogenous inputs model multi- variate dependencies. We partition a time series of length L1 into non-overlapping patches of size Lp. If needed, zero pero-padding P is added so that T. + P is divisible by Lp, resulting in N LI+P
(I)T_ F+1 Og
(fina))T
+e7 Σ D CC
(1)
+ e
patches S L(( E R.xxC For each series j. a linear projection Win E RaxLe trans- forms its patch sfittLLo into an autoregressive token (W wokk S Sp E Ixxdd This token is used to pre- dict the next I. steps for series j. which can be viewed as a PatchTST-stylle tokenization (Nie et al., 2022) with an autoregressive loss.
Lp)
where D can be viewed shared output projection WT across / attention layers. We parameterize W via
LL
a learnable LU factorization: a lower triangular matrix L (with diagonal fixed to 1) and an upper triangular matrix U (diagonal activated by softplus) are multiplied to form D, from which we derive its inverse.
4.2. SAMo Transformer
Inspired by vector autoregressive with exogenous variables (VARX), yt = Em=1 Am + Ex= B, + u, where ét-a represents exogenous factors, we model all other series as exogenous when forecasting series j. Specif- ically. linear projection Wex E RCxC mixes each channel independently generate the exogenous token
The aligned multi-layer linear attention module described above preserves a valid VAR structure, resolving misalign- ments in standard linear attention related training objec- tives, data generation, input/output spaces, and autoregres- sive forecasting. Building this, we introduce SAMoVAR (Structural Aligned Mixture of VAR), which reorganizes MLP and linear attention layers, serving as a drop-in re- placement for standard linear Transformers in time series forecasting (TSF), as shown in Figures I (c) and (d).
E 1l As shown in Figure 2, we combine autoregressive and exogenous tokens along the sequence dimension form the input tokens Kipuu RC2NXxd The channel dimension is treated as part of the batch for independent computation, with each exogenous to- ken placed before its corresponding target token. Trainable position embeddings based on token positions and channel indices are added. The SAMo VAR Transformer processes the sequence, and the outputs corresponding to target to- kens are projected using W Wout E p to generate the next-step ARX predictions.
Wiok | ffs+++ w
A 1-layer SAMoVAR Transformer includes: a) MIP Com- ponent: Consists of MLP layers that learn optimal repre- sentations for VAR observations. After MLP processing, layer normalization ensures VAR outputs align with the transformed input signals. b) SAMoVAR Attention: Com- prises/ / linear attention layers, parameterized with the robust path pruning methods discussed earlier. A unified residual shortcut across all layers stabilizes training. The overall architecture is shown in Figures 1 (d) and (e).
5. Experimental Results
Patch-based ARX Tokenization Prior research highlights the importance of preserving univariate dependencies for effective multivariate time series forecasting (TSF) (Zeng et al., 2023; Nie et al., 2022; Lu et al., 2024b). For multivari- ate inputs S,, we adopt an autoregressive (ARX) tokeniza-
5.1. Synthetic Tasks
Model Setup We use the SAMoVAR architecture described in Fig. ((d) along with the ARX tokenization from Fig. to train our TSF models. Additionally, we construct baseline
6

---

Linear Transformers as VAR Models
Wiight Lap
SANGOOA
AC Himitras
LITTea fempora Weight
Figure 3. Visualization of the validation datapoint and model weights for the synthetic VAR task. See Section 5,1 for more details.
0.10
Traiining Loss
Validation Loss
with coefficients between -0.25 and 0.25 to evaluate extrap- olation. The input length is L, 64, output length Lp 1, and C = 2 series, with ARX tokenization. We use 3 lay- ers and 2 attention heads. As shown in Fig. 4, during training, FixedVAR effectively memorizes the input struc- ture but struggles in validation. contrast, SAMo VAR's dynamic VAR inference generalizes well and significantly outperforms FixedVAR. LinTrans, which does not explicitly model the TSF process, performs worse in both phases.
Lintrans SAMOVAR WAC
0.008
Unniaans SAMcVAR
ReedVAR
0,006
0.005
11
0.02
0.00
0.003
Explainability The left side of Fig. 3 shows the original VAR weights, cumulative temporal contributions, and raw observations for a validation datapoint. The middle part compares the averaged attention maps of SAMOVAR and LinTrans. The upper right visualizes SAMo final out- put contributions using Wont D WT across channels. The lower right shows Trans'sss final-row attention map after reordering. SAMo VAR's VAR-based attention better aligns well with the true contribution heatmap, providing more interpretable results than LinTrans.
Epoch
Epoch
Figure 4. Visualization of the loss curves for synthetic VAR tasks.
1A7
Temporal Influence Path Figure 5 highlights how visual- ized paths reveal intermediary effects between time steps. The displayed values on the edges represent the averaged weights from the path matrix P. The top section shows Series (SI) and the bottom, Series 2 (S2). From the orig- inal VAR weights, we know S2 has a stronger influence on S1 than vice versa. Correspondingly, tion, paths passing through exogenous points (marked "Ex") show higher weights, while in the bottom section, these paths are weaker. This visualization effectively uncovers the transmission dynamics within the VAR structure, enhancing interpretability of TSF results.
0.594
Figure 5. Visualization of the 2 temporal influenee paths from step 124 to step 128 for the two series in the datapoint shown in Fig. where even-numbered steps represent endogenous tokens and odd-numbered steps represent exogenous tokens.
the top sec-
model (Lin'Trans) based on the classic linear Transformer structure shown in Figure 1(c) to highlight the improvements of SAMoVAR. To demonstrate the impact of dynamic VAR weights, we replace SAMoVAR's mixture of VAR modules with fixed-weight VAR layer (FixedVAR).
5.2. Multivariate TSF
VAR Generalization To test SAMo ability to learn and generalize to the underlying data generation process, we generate training and test data using random VAR(p) models with varying lag orders. For training, E {1, 2,33 with co- efficients between -0.5 and 0.5. For validation, p E (3, 4,5)
We conducted comprehensive experiments on 12 widely- used TSF datasets, including Weather, Solar, Electricity
7

---

Linear Transformers as VAR Models
Table I. Summary of Multivariate TSF Results. Averaged test set
datasets like Solar and PEMS, which contain many se- ries with stable long-term patterns, SAMoVAR achieved over 30% improvement compared to previous models. This shows the significant benefit of incorporating dynamic VAR structures and alignment when modeling complex TSF data with stable generative processes. Furthermore, although lin- ear Transformers did not fully align with the autoregressive forecasting targets, they still outperformed most baseline models, indicating their potential in TSF.
MSE are reported. See Table for the original results.
Model
ISAMoVAR
LintTras
V'neVARRCCATS
(Transfomer LVTS PachTST
Dineer Uncevormer
Weather
0.314
0.217
0.257
B1216
0.212
0.222
0.221
0.253
0.251
Solar
0.184
0.189
0.430
0.205
0.219
0.209
0.202
0.216
0212
0.401
0.319
0.364
0.408
0.454
0.640
0.413
0.422
0.905
RTT
0.324
0.346
0.391
0.320
0.314
0.354
0.390
0.466
0877
RTTml
0.339
0.346
0.519
0.365
0.373
0.354
0.346
0.347
0.735
ETTm2
0.340
0.243
0.276
0.243
0.265
0.347
0.267
0.252
0.576
ECL
1.151
0.166
0.345
11.51
0.170
0.167
0.192
0.163
B664
Trallc
0.39
0.438
0.717
0.385
0.414
0.418
1.391
0.411
0824
PEMS03
0.150
1.18
0.375
0.225
0.212
0.234
0.230
0.254
0.443
PEM504
0.102
0.404
0.184
0.171
0.256
0.221
0.246
0.377
PEMS08
0.234
1.261
0.674
0.359
0.271
0.296
0.240
0.357
0.681
AvgRank
1.41
3
8.16
2.56
5.43
3.20
4.00
5.82
8,30
29
3
0
0
2
Ablation Studies In Table 2, we present ablation studies to validate the effectiveness of components within SAMo
Table 2. Summary of Ablation Study Results. Averaged test set
MSF are reported. See Table 4, 5, 6, 7 for the original results.
1) Reintroducing key projection weights: Based on our anal- ysis in $4.1, introducing Wx negatively impacts the scaling control of temporal influence paths and increases numeri- cal instability due to the additional weight matrices. The results show significant performance drop and training instability when We & is added. 2) Removing the inverse matrix D-1: This matrix, corresponding to W. or plays a critical role in controlling the output space. Without it, per- formance degrades, as expected. 3) Removing RMSNorm from queries and values: As discussed in $4.1, norm control over queries and values is essential for learning effective VAR path weights. Removing R.MNNrmm significantly de- grades performance and causes training issues. 4) Increasing the number of attention heads: When the hidden dimension is fixed, using more attention heads reduces the dimension per head and passively disables more influence paths during initialization, leading to performance degradation. This also reduces the parameter capacity of dynamic VAR weights, further hurting performance. 5) Varying the number of Transformer layers I: The layer depth determines the num- ber of intermediate points in the temporal influence paths included in the final VAR weights. The results show that = yields the worst performance due to the lack of in- termediate points. For real-world TSF datasets, = 3 or 4 (2 or intermediate points) sufficient for good performance, while larger values lead to overfitting risks.
Exp
SAMoVAR
We
w/
D
wif
QV
wlo
Norm
Headi-4
dim-16
Headenk
dim-8
Heak-166
dim-4
Headem32
dim-2
ErThl
0.401
04433
0.409
0421
0.401
0.406
0.412
0.413
FTTml
0,339
0,346
0,344
0.300
0.339
0,342
0,343
0,344
Exp
[=1
[=2 1=3
1=5
1=
(=7
(=8
ETThl
0.420
0411
0.401
0.404
0413
0.414
0.408
0.410
ETTml
0,346
0.34
0.339
0.345
0.346
0.318
0,349
0.351
(ECL), ETTs, Traffic, and PEMS 2, See $A.2 for detailed descriptions of the datasets. Detailed hyperparameter set- tings and implementation details can be found in $A.44
For SAMo VAR, LinTrans, and FiX,dVAR, we used / = 3 Transformer layers, with the hidden dimension determined empirically as d = 32 [VC]. The number of attention heads was set to d/16, ensuring that the dimension per head was 16. During testing, we predicted the next L,p time steps corresponding to the last input token, following the same approach as the baselines for consistency.
Baselines In addition to SAMoVAR, Lin LinTraass and Fixed- VAR, we introduced five recent state-of-the-art baselines: CATS (Lu et al., 2024b), iTransformer (Liu et al., 2024), FITS (Xu al.. 2024). PatchTST (Nie al., 2022), and DLinear (Zeng et al., 2023). We also included an encoder- only vanilla Transformer, named Encformer, as a compar- ison to autoregressive Transformers. Encformer uses the same tokenization method as Autoformer (Wu et al., 2021) and Informer (Zhou et al., 2021).
Computational Costs SAMOVAR introduces additional computational overhead compared to the vanilla linear Transformer and even reduces the key projection step. This gives it efficiency advantages over other TSF models. De- tailed comparisons are shown in Table 8.
Fair Comparison All baseline models were trained under the same conditions with input lengths L, E (512, 1024, 2048, 4096), and the best performance was reported. This approach may yield stronger results than the original papers, ensuring a rigorous and fair comparison. els, we used consistent settings (L₁ : Lp) = (1024, 96), (2048, 192), (2048, 336), (4096, 720) to ensure appropriate ARX tokenization.
For the three VAR-based mod-
6. Conclusion and Limitation
This work bridges the gap between linear attention Trans- formers and VAR models for time series forecasting. We demonstrate that single-layer linear attention inherently cap- tures dynamic VAR structures, while standard multi-layer architectures misalign with autoregressive objectives. By structurally aligning the input-output flow and MLP layers. we propose SAMO multi-layer linear Transformer that integrates dynamic VAR weights through temporal in-
Main Results As shown in Table 1, SAMoVAR consistently outperformed other models across all datasets, with a sig- nificantly higher average ranking and t count. Notably,
Due to baseline models failing to train on PEMS07 when using batch size= 1 and L, 1096, this dataset is excluded from the main text, as described in the fair comparison paragraph.
Note:
8

---

Linear Transformers as VAR Models
fluence paths. SAMoVAR achieves superior accuracy, in- terpretability, and efficiency compared to state-of-the-art models across synthetic and real-world benchmarks. As for limitation, we have not yet tested larger SAMoVAR models on large-scale general TSF tasks to evaluate their potential as foundation models. Additionally, we have not explored applying SAMoVAR to general sequence modeling tasks to assess whether the learned dynamic VAR weights are effective beyond TSF tasks.
oil-price rises using vector autoregressions. International economic review, pp. 459-484, 1984.
Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mo- hiuddin, A., Kaiser, I.., Belanger, D. B., Colwell, L.. J., and Weller, A. Rethinking attention with performers. In International Conference on Learning Representations, /tt.tts://ppeeeeiiieee http////ppeeeeeeeeeeeeeeeeeeeeee net forum?
2021. URL https:://///. i id=Ua 6WRH...H
Impact Statement
Das, A., Kong, W., Sen, R., and Zhou, Y. A decoder- only foundation model for time-series forecasting. In Forty-first International Conference Machine Learn- ing. 2024.
This research introduces a novel approach to time series forecasting, advancing the aceuracy and interpretability of attention-based time series models. By enabling better- informed decisions in fields such as economics and health- care, the method demonstrates broad practical value. While its overall societal impact leans toward positíve outcomes, the use of this technology in sensitive areas requires thought- ful management and ethical oversight to minimize potential risks and unintended consequences.
Gruver, N., Finzi, M. A., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URL net/forunt (ttoormmmdmmmmmemi111100
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 98)::173--188) 1997.
References
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm in-context learn- ing? investigations with linear models. arXiv preprint arXiv:2211. 15661, 2022.
Holt, C. C. Forecasting seasonals and trends by exponen- tially weighted moving averages. International joternationaaouunalll of forecasting, 20(1):5-10, 2004.
Hua, W., Dai, Z., Liu, H., and Le, Q. Transformer quality in linear time. In International conference on machine learning, pp. 9099-9117, PMLR, 2022.
Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize test time. arXiv preprint arXiv:2501.0066 006663, 2024.
Jin, M., Wang, S., Ma. L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y-F., Pan, S., and Wen, Q. Time-LLM: Time series forecasting by re- programming large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /pwnreviewweeeeeeeeeeee net forum?
Box, G. E., Jenkins, G. M., and MacGregor, J. Some recent advances in forecasting and control. Journal of the Royal Staristical Society: Series C (Applied Statistics), (3(2):158-179) 1974.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P. Neelakantan, A., Shyam, P., Sastry, G.,
id-UnbSCVPtae.
Askell, A., Agarwal, S., Herbert- A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL cc cc/ 4477C4801e0399741115000011131314 pdf.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rans: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. $156-5163 PMLR, 2020.
Kim, T. Kim, J.. Tae, Y., Park, C., Choi, J.-H., and Choo, Reversible instance normalization for accu- rate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2022. URL htttps://openeeeeeee wwwp///pwweew...e//////. net fortm/frrm? ld=cGDAkoolCCp.
neurips.
Lai, G., Chang, W., Yang, Y., and Liu, H. Modeling long- and short-term temporal patterns with deep neural net- works. In Collins- Tallinsn, K., Mei, Q., Davison,
Burbidge, and Harrison, A. Testing for the effects of

---

Linear Transformers as VAR Models
B. D., Liu, Y., and Yilmaz, E. (eds.), The 41st Inter- national ACM SIGIR Conference Research & De- velopment in Information Retrieval, SIGIR 2018, Ann Arbor. MI, USA, July 12. 2018, pp. 95-104. ACM, URI. https:
Pretis, F. Econometric modelling of climate systems: The equivalence of energy balance models and cointegrated vector autoregressions. Journal of Econometrics, 214(1): 256-273 2020.
2018. doi: 10. 1 10.11445/22099988322100000 //doi. ///.i.oog//// 10. 111111111/1111 3210006.
Primiceri, G. E. Time varying structural vector autoregres- sions and monetary policy. The Review of Economic Studies, 72(3):821-855 2005.
Li, S., Jin, X., Xuan, Y. Zhou, X., Chen, W., Wang, Y.- X., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecast- ing. Advances in neural information processing systems, 32, 2019.
Qin, Z., Han, X., Sun, W., Li, D., Kong, I.., Bames, N., and Zhong, Y. The devil in linear transformer. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 7025-7041, Abu Dhabi, United Arab Emirates, December 2022. Association for Compu- tational Linguisties. doi: 10. 10.18653/v1/0022.emmlg main. 473. URL https: http://aaaaatthoogooooo org/ 2022. emnlp-main.47 473.
Li, Y. Yu, R., Shahabi, C., and Liu, Y. Diffusion con- volutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint 17XXvv/707.0111 2017.
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. itransformer: Inverted transformers are effective for time series forecasting, In The Twelfth International Conference on Learning Representations, 2024. URL https: http://pppereeeeeeeeeeeeeee wttp:/iiewnnne/tttteeeee forum? id=JeP fAI8fah.
Radford, A. standing by generative pre-training. nAI technical report, 2018. URL https: // bangaagg-unsuuerrvvsee///gggggg understanddig__aper.
Improving language
under-
Ope-
Lu, J., Han, X., Sun, Y., and Yang, S. Autoregressive moving-average attention mechanism for time series fore- casting, 2024a. URL pttttps://opeeeeee /ppprevveeeev...///// net./ diiimmii==-999 29N377
Mubio-Ramires J. F., Waggoner, D. F., and Zha, T. Struc- tural vector autoregressions: Theory of identification and algorithms for inference. The Review of Economic Stud- ies, 77(2):665-690) 2010.
Lu, J., Han, X., Sun, Y., and Yang, S. Cats: Enhancing multivariate time series forecasting by constructing auxil- iary time series as exogenous variables. arXiv preprint arXiv:2403.0167. 2024b.
Salinas, D., Flunkert, V. Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3) 2020.
Ma, X., Zhou, C., Kong, X., He, J., Gui, L., Neubig, G., May, J., and Zettlemoyer, L. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209. 10655, 2022.
Schlag, I., Irie, K., and Schmidhuber, J. Linear transform- ers are secretly fast weight programmers. In Interna- tional Conference on Machine Learning, pp. PMLR, 2021.
Mao, H. H. Fine-tuning pre-traine pre-trained transformers into de- caying fast weights. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, 10236, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguisties. doi: 10. 10.18653/v1/002..co www.eemwwwmmmmmmmemmmmmmmmmmmeme 697. URL https:// acl aclanthology www.oorg//22 22222222222222 emm1pmmmmllppmmmmm matpppmmain..m.mmmmmmmm 697.
Stock, J. H. and Watson, M. W. Vector autoregressions. Journal of Economic perspectives, 101):100 2001.
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.0862. 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URL https: pettps://axxiv.or. (bs/2302.13977
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference Learning Representations, 2022.
Patro, B. N. and Agneeswaran, S. Simba: Simplified mamba- based architecture for vision and multivariate time series. arXiv preprint arXiv:2403. 15360, 2024.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
10

---

Linear Transformers as VAR Models
Winters, P. R. Forecasting sales by exponentially weighted moving averages. Management science, 6(3):324-342, 1960.
Zivot, E. and Wang, J. Vector autoregressive models for multivariate time series, Modeling financial time series with S-PLUS@, pp. 385-429, 2006.
Wu, H., Xu, J., Wang. J., and Long, M. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Informa- tion Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 22419- 22430, 2021. URL
Beo0d002879seaaccccaaaaaaaao Abstract. html.
Xu, Z., Zeng, A., and Xu, Q. FITS: Modeling time series with $10k$ parameters. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL
id-bWenv23gMb.
Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient train- ing. In Forty-first International Conference Machine Learning, 2024. URL httph:tt///cc.. http://pppeeeeeeeeeeeeeee n forum? idaaaa
Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11122 128, 2023.
Zhai, S., Talbott, W., Srivastava, N., Huang, C., Goh, H., Zhang, R., and Susskind, J. An attention free transformer. arXiv preprint 105. 2021.
Zhang, R., Frei, S., and Bartlett, P. L. Trained trans- formers learn linear models in-context. arXiv preprint arXiv:2306.099277 2023.
Zhang, Y. and Yan, J. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting, In The Eleventh International Conference on Learning Representations, 2023.
Zhou, H., Zhang, S., Peng, J., Zhang, S., J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Thirty-Fith AAAI Conference on Artificial Intelligncee AAAI 2021, Thirty-Thing Conference Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 11106- 11115. AAAI Press, 2021. htt.tttss://oosssseett org/ i nide. php/ httpp///////tttlllle//e/////33//

---

Linear Transformers as VAR Models
A. Appendix
A.I. Workflow the SAMoVAR Attention Module
We present the details of the attention module in the SAMoVAR Transformer, as shown in Algorithm 1.
Algorithm 1 SAMoVAR Attention Module
Input: Sequence E BxLxD layers Laun: head count head dimension D/H, query projection W DxD
value projection w! E DxD invertible matrix D "xxdxdd dropout rate
Output: Processed sequence BBLLDD
B, L, + (X))
Xong f clone(X) {Store original input for separate Q V computation]
Initialize + X
Generate invertible matrices D using LU factorization
for / = Luin do
Compute Query and Value Projections:
Q + reshape w
g Xocig B, L, H, d) {Q E RBXLLHxx
7( <- reshape( w!!) Xong: B, I., H, d) (V() E [BXLLxXx]
Q(1) f- RMSNorm( Q()), V(O RMSNorm(V())
Compute Keys from Input:
K + reshape B, L, H, d) {K E [
Compute Cumulative Fast- West-Weight Updates:
W + cumsum K @ ((), dim = 1) (W E RxxLxxxdxdy Compute Output using Q();
Y <- -QQ & W {Y BBxxxxxd]
Apply Dropout and Mix Output via Structural Matrix D:
Y drooout(YYy)
Y mransformed einsum (ansumm"bblee ('blhd, hde- blhe', Y,D {Apply per-head invertible transformation)
Y Firansormmdd f- reshape iransfoommdd
B, L, D) (Reshape back to Bx
X
X Yrranffommed
Update Input for Next Layer: X+Y
end for Return: E BBXLLD
A.2. Experimental Datasets
Our multivariate time series forecasting experiments employ twelve real-world benchmark datasets. The original dataset names and their key details are summarized as follows:
Weather Dataset (Wu et al., 2021) This dataset records 21 meteorological indicators (e.g., temperature, humidity) at 10-minute intervals throughout 2020, collected from the weather station of the Max Planck Institute for Biogeochemistry in Germany.
Solar Dataset al., 2018) Comprising solar power generation data from 137 photovoltaic plants, this dataset captures energy production values sampled every 10 minutes during 2006.
Electricity Dataset (Wu et al., 2021) Containing hourly electricity consumption records of 321 clients, this dataset spans three-year period from 2012 to 2014.
ETT (Zhou et al., 2021) The Electricity Transformer Temperature (ETT) dataset monitors operational parame-
we/dd///wweee dee/eeter/////
http:i/d//ssla--ppoeer-taaaaaaae html
ihttp:://archive.iccs.eeu edd//eedmmmmm
2 12014 12014
hattps://githubb..oo//ooo//
12

---

Linear Transformers as VAR Models
ters (including load and oil temperature) from power transformers, recorded 15-minute (TTTml/ETTm2) and hourly (ETTh1/ETTh2) resolutions between July 2016 and July 2018. Each subset contains seven critical operational features.
Traffic Dataset" (Wu et al., 2021) This dataset provides hourly road occupancy rates from 862 highway sensors in the San Francisco Bay Area, collected continuously between January 2015 and December 2016.
PEMS Dataset (Li et al., 2017) A standard benchmark for traffic prediction, the PEMS dataset includes California freeway network statistics recorded at 5-minute intervals. Our experiments utilize four widely adopted subsets: PEMS03, PEMS04, PEMS07, and PEMS08.
A.3. Related Works
Time Series Forecasting Models. Time Series Forecasting (TSF) has seen extensive research spanning traditional statistical models to deep learning-based approaches. Classical methods, including ARIMA (Box et al., 1974) and exponential smoothing (Holt, 2004), effectively model univariate series by capturing trends and seasonality but struggle with complexity in multivariate and nonlinear scenarios. Vector Autoregression (VAR) (Stock & Watson, 2001; Zivot & Wang, 2006) extends autoregressive methods to multivariate series, accounting for cross-variable interactions, and remains prevalent in economics due to interpretability and theoretical robustness (Pretis, 2020).
The emergence of deep learning, particularly RNN-based approaches like LSTM (Hochreiter & Schmidhuber, 1997) and DeepAR (Salinas et al., 2020), significantly improved capturing temporal dependencies. However, RNNs often suffer from challenges in modeling long-range dependencies. Recent advancements leveraging Transformer architectures have transformed TSF through models such as Log LogTrrans (Li et al., 2019), Informer (Zhou et al., 2021), and Autoformer (Wu et 2021), each introducing innovative mechanisms like local convolutions, ProbSparse attention, and auto-correlation respectively. Despite their sophistication, these methods have not consistently outperformed simpler approaches like MLPs or linear models in various contexts (Zeng et al., 2023; Xu et al., 2024; Lu et al., 2024b). Recent innovations explore novel directions, such as treating series as patches (Nie et al., 2022), focusing attention across variates (Liu et al., 2024). and utilizing Large Language Models (L.LMs) for zero-shot forecasting or foundation model training (Gruver et al., 2023; Jin al., 2024; Das et al., 2024). Balancing model complexity, interpretability, and effectively modeling both temporal and cross-variate dependencies remains central challenge in TSF (Zhang& & Yan, 2023; Lu et al., 2024b).
Linear Attention Mechanisms. The quadratic complexity of traditional Transformer attention (Vaswani, 2017) has spurred extensive research into efficient alternatives. Linear attention mechanisms emerged as a promising solution, beginning with kernel-based linear Transformers (Katharopoulos et al., 2020), which approximate attention operations linearly by employing kernel feature mappings. Despite achieving substantial computational efficiency, initial implementations experienced performance drops compared vanilla attention.
Subsequent efforts have focused addressing these shortcomings and enhancing practical usability. Notably, Performers introduced FAVOR+ for unbiased softmax approximation (Choromanski al., 2021), while TransNormer (Qin et al., 2022) tackled issues of gradient instability and attention dilution. RetNet (Sun et al., 2023) integrated retention mechanisms with linear attention to better capture sequential patterns. Further theoretical insights emerged connecting linear attention to classical concepts like Fast Weight Programming (Mao, 2022; Schlag et al., 2021), providing new interpretability perspectives. Recent advancements have emphasized practical large-scale deployment and efficiency. Gated Linear Attention (Yang et al., 2024) improved hardware efficiency during training, and innovative tiling strategies in Lightning Attention-2 ensured constant training speeds irrespective of sequence length. The introduction of exponential moving-average (EMA) into gated linear attention models further stabilized training dynamics and performance (Ma al., 2022). Despite these advancements, effectively aligning linear attention structures with generative processes of time series forecasting data remains an active research area.
A.4. Hyper-pane parameter Settings and Implementation Details
This section explains the hyper-parameter settings for SAMoVAR attention, linear (AR) attention, and fixed VAR models used in the experiments.
Attention Module: We use 3 layers, consisting of 3 ML. layers and 3 attention layers for SAMoVAR attention. For linear
hhttp://pems. dot, ca..aa.... gov/ hhttp://pmms. dot ca wa..ca...oo... /ov/
13

---

Linear Transformers as VAR Models
attention, we also use Transformer blocks. For fixed VAR, we first use 3 MLP layers and replace the final mixture of VAR modules in SAMoVAR with single-layer VAR structure with fixed weights. This VAR structure shares the same architecture as linear attention, but the query and key vectors are replaced with set of fixed position vectors, similar to trainable positional embeddings.
For all Transformers, we set the hidden dimension as d = 32| v VCJ, where C' is the number of multivariate time series. For linear attention and fixed VAR, we use attention heads. For SAMoVAR, the number of heads determined to ensure each head dimension is 16. For example, for the ETT datasets (C = 7), d 64, and the number of heads is 16. For the Weather dataset (C- 21), 128, and the number of heads 8.
Hidden Dimension:
Initialization: All linear layers are initialized using normal distribution with mean of 0 and standard deviation of 0.02. Embedding layers are zero-initialized. For projection layers in MIPs, we use GPTT-2-style initialization with scale factor of 1. where is the number of MLP/attention layers. The MLP structure follows the standard 2-layer Transformer design with an expansion ratio of 4. The dropout rate is set to 0.1 uniformly.
V
Layer Normalization: We add RMSNorM after query and key projections for SAMoVAR. For the other layer normalization modules, experiments showed no significant difference between traditional layer normalization and RMSNorm, so we opted for RMSNorm due to its lower computational cost.
For input time series of shape Lp), where Lp is the input length, we concatenate timestep embeddings along the channel dimension if available (as described in works like Autoformer (Wu et al., 2021) and DLinear (Zeng et al., 2023)). The time series is divided into N non-overlapping patches of size Lp. Zero padding is applied if Lp does not divide L, evenly. The resulting input tokens have shape (C, N. Lp). We apply RevIN (Kim et al., 2022) to normalize each token by subtracting its mean and dividing by the standard deviation of the entire input.
Input Preprocessing:
We then create an additional set of tokens by projecting the channel dimension using C x C/ linear weights. These exogenous tokens are interleaved with the univariate tokens, resulting an ARX input of shape (C, 2N, Lp). The patch size dimension Lp is projected to the hidden dimension d, resulting in input tokens of shape (C, 2N, d). We add d-dimensional channel and position embeddings to the input tokens.
Transformer Module: The input tokens are layer-normalized and passed through the Transformer module. The output has shape (C, 2N, d). We select the outputs corresponding to the original univariate tokens, resulting in (C, NN d). This is followed by layer normalization and projection back to dimension L,,, producing output tokens of shape (C, N, L.)). The RevIN reverse process restores the outputs by multiplying with the previously calculated standard deviation and adding the mean. We compute the MSE loss between the output and the next timestep's 's univariate token values.
Training Setup:
single Nvidia RTX 4090 GPU with batch size of 32. For datasets that cause memory overflow, the batch size is reduced to 16 or 8, with 2-step or 4-step gradient accumulation to maintain an effective batch size of 32. The optimizer is AdamW with a weight decay of 0.1 and 3 values of (0.9, 0.95). All baseline models are retrained under the same settings.
All experiments
conducted
Dataset Splits and Preprocessing: Following Nie et al. (2022); Liu et al. (2024), we use a train-validation split ratio of 0.7, 0.1, and 0.2. Input data is standardized using the mean and standard deviation calculated from the training set.
Training Procedure: We apply early stopping with patience of 12 epochs and a maximum of 100 epochs. For the first 5 epochs, we use warm-up learning rate, gradually increasing it from 0.00006 to 0.0006, followed by linear decay until the maximum epoch is reached.
A.5. Detailed Experimental Results
A.6. Additional Visualization of the Synthetic VAR Task
A.7. Additional Proofs and Clarifications
A.7.1. COMPLEXITY ANALYSIS OF SAMOVAR ATTENTION
Proposition. The computational complexity of SAMoVAR Attention with respect to the sequence length L. O(L).
Proof. We provide detailed complexity analysis for Algorithm (SAMoVAR) here. Let the batch size be B, the model dimension D. the number of heads H. the her-head dimension d = D/H, and the number of attention layers Latu-
14

---

Linear Transformers as VAR Models
Table Full results of Multivariate TSF task. The test set MSE and MAE are reported. The hest results are bolded and the second hest
are underlined.
Models
SAMoVAR
LinTras
FisodVAR
CATS
iTrnsformer
FITS
PaichTST
DLincur
Encformer
Metric
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAB
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
Weather (96)
0,141
0.193
0.145
0.19
0.230
0.143
9.19%
0.158
0.219
0.149
0.204
0.149
0.198
0.150
0.209
0.188
0.245
Weather (192)
0.186
0.240
0.187
0.277
0.188
0.242
0.203
0.254
0.189
0.241
0.241
0211
0.265
0.215
Waather (336)
0.232
0.279
0.237
0.281
0.311
0.235
0.278
0.250
0.291
0.257
0.340
0.279
0.355
0.505
0
0.540
Weather (720)
0.295
0.354
0.299
0.330
0.331
0.355
0.297
0.331
0.316
0.341
0.511
0.332
0.306
0.327
0.316
0,550
0.332
0.382
Solar (96)
0,165
0.230
0.174
0.246
0.440
0.495
0.182
0.239
0.250
0.257
0.189
0.240
0.209
0.251
0.208
0.274
0.201
0.225
Sodar (192)
0.177
0.253
0.17
0.251
0.405
0.496
0.214
0.283
0.204
0.383
0.206
0.249
0.192
0.255
0.308
0.265
0.289
0.289
Solar (336)
0.19
0.262
0.266
0.272
0.222
0.219
0.253
0.221
0.279
Solar (720)
0.199
0.261
0.207
0.467
0.533
0.213
0.267
0218
0.299
0.221
0.205
0.258
0.227
0.291
0.218
(96)
0.357
0.394
0.362
0.396
0.493
0.484
0.365
0.306
0.396
0.42
0.369
0.398
0,370
0.399
0.370
0.394
0.986
0.730
ETTal (192)
0.398
0.419
0.416
0.437
0.548
0.531
0.404
0.420
D431
0.451
0,435
0.444
01422
0.421
0.405
0.416
0814
0.991
ETTal (336)
0.422
0.44
0.427
0.646
0.560
0.538
0.425
0.477
0.459
0.470
0.468
0.467
0.422
0.436
0.439
0.645
0.883
0.680
ETTsl (720)
0.427
0.451
0.471
0.485
0.63
0.607
0.441
0.465
0.528
0.523
0.488
0.497
0.447
0.466
0.422
0,490
0.941
0.739
ETT (192)
0.266
0.329
0.276
0.259
0.337
0.299
0.277
0.546
1.303
0.323
0.386
0.342
0.362
0.315
0.368
0.365
0.348
0,400
0,339
0,379
0.375
0.412
2 (336)
0.34]
0.394
0.375
0,414
0.396
0,440
0.339
0.392
0-407
0.429
0.376
0,426
0,229
0,380
0.448
0.465
0.551
0.544
ETT52 (720)
B.366
0.421
0.389
0.451
0.493
0.513
0.365
0.419
D.25
0.454
0.421
09433
0.379
0.422
0.605
0.551
0714
0.631
ETTm] (96)
0.278
0.33V
0.296
0.344
0.193
0.462
0.282
0.339
0.325
0.376
0.305
0.347
0.290
0.342
0.299
0.345
0.686
0.60
ETTml (192)
0
0.367
0.524
0.371
0.502
0.471
0.326
0.363
0.352
0.388
0.354
0.371
0.328
0.369
0.335
0.365
0.636
0.580
ETTml
0.359
0.396
0.363
0.397
0.358
0.382
0.382
0.405
0.363
0.392
0.359
0.386
0.602
(720)
0.201
0.113
0.409
0.432
0.527
0.414
0.116
0.432
0.434
0,412
0,409
0,405
0.415
0.396
0.409
0.825
ETTm2 (96)
0.154
0.242
0.158
0.246
0.188
0.262
0,158
0.248
0.187
0.281
0.164
0253
0,165
0.255
01184
0.283
0.481
0.525
ETTm2 (192)
0.208
0.282
0.213
0.287
0.244
0.318
0211
0.285
0.252
0.311
0.211
0292
0.214
0.289
0211
0.501
0.434
0.517
ET 122 (336)
0.257
0.333
0,263
0.326
0.302
0.361
0266
0.322
0.281
0.342
0.259
0.134
0.266
0.328
0.263
0.333
0.461
0.531
ETTm2 (720)
0.300
0.272
0,339
0.376
0.377
0.421
0.340
0.371
0.358
0.392
0.352
0.377
0,344
0.376
0.3L
0.388
0.28
0.739
0.129
0.226
0.149
0.265
0,127
0.182
0.226
0.466
0.222
0.125
0.252
0.342
(19)
0,141
0.243
0.160
0.268
0.457
0:24
0.166
02269
0.149
0249
0,247
0.240
0.151
0.289
0.658
0.611
(336)
0.15
0.255
0.171
0.299
0.307
0,398
0.15
0:253
0.176
0.270
0.168
0262
0163
0:259
0.109
0.267
0.888
ECL. (720)
0.176
0.281
0.183
0.279
0.375
0.446
0.12
0.273
0.206
0,283
0.206
02033
0.197
0.290
0.203
0.501
0.781
0.739
Trafic (96)
0.571
0.261
0.423
0.294
0.764
0.487
D355
0.255
0,356
02299
0,404
0.286
0.300
0.249
0.399
0.286
0.915
0.400
Traffic (192)
0.375
0.268
0.434
0.295
0.769
0.460
0.373
0.258
0410
0.278
0.406
0.274
0.379
0.256
0.23
0.287
0.723
0.455
Traffic
0.590
0.279
0.439
0.299
0.384
0.481
0.264
0.4%6
0.29
0.764
0.51
Trallie
0.298
0.425
0.296
0.45k
0.291
0.286
0.466
0.315
0.892
PFMS01 (96)
0.119
0.232
0.144
0.258
0.799
0.437
0.157
0.267
0.135
0.229
0.193
0.274
0.198
0.285
0.198
0.299
066
0.272
PEMS08 (192)
0.148
0.248
0.146
0.264
0.418
0.523
0.197
0.300
0.19%
0.259
0.221
0.299
0.201
0.288
0.231
0.528
0.471
0.513
PEMS01 (336)
0.191
0.27y
0.217
0.315
0.344
0.431
0.222
0.318
0.234
0.330
0.238
0.313
0.218
0.304
0.254
0.351
0.542
0.548
PEM501 (720)
0.142
0.248
0.345
0 0.306
0.440
0,500
0.325
0,406
0.279
0.344
0.285
0.353
0.304
0.392
0.331
0.413
0.597
0.610
PEMS04 (96)
0.092
0.193
0.253
0.215
0.320
0.309
0.232
PEMSO1 (192)
0,100
0.19K
0.132
0.189
0.310
0.165
0.279
0.238
0.294
0.228
0.323
PEMS04 (336)
0.107
0.207
0.139
0.256
0.341
0.434
0.196
0.309
0:175
0.290
0.265
0.355
0.226
0.318
0.252
0.343
0.422
0
PEMSON (720)
0,109
0.209
0.157
0.271
0.422
0.468
0.209
0.327
0.221
0.331
0.306
0.391
0.246
0.346
0.294
0.377
0.523
0.538
PEMSOS (96)
0,135
0.213
0,168
0,249
0.977
0.774
0.212
0.276
0.170
0.225
0.357
0.322
0.181
0.236
0.118
0.326
0.253
0.302
PEMS08 (192)
0,197
0.224
0.347
0,784
0.569
0.503
0.289
0.313
0.269
0 395
0.343
0.361
0.273
0.305
0.325
0.344
0 070
0.619
PEMSO8 (336)
0.295
0.313
0.306
0.328
0.565
0.324
0.338
0.303
0.324
0.382
0,349
0.335
0.332
0.374
0.302
0.831
0.685
PEMSOS (720)
0.304
0.250
0.321
0.340
0.515
0.359
0.369
0.342
0.403
0.369
0.376
0.411
0.415
0.936
312
<X0
by
112
20
Figure 6. Additional Visualization of the VAR Synthetic Task with a Random Datapoint in the Validation Set.
Figure 7. Visualization the 2 temporal influence paths from step 124 to step 128 for the two input time series variable for the datapoint shown above, where even-numbered steps represent endogenous tokens and and-nnumeered steps represent exogenous tokens.
Preprocessing:
15

---

Linear Transformers as VAR Models
Table Full results of the ablation studies of the SAMoVAR module structure. The MSE and MAE of the are reported. The best
results are bolded and the second best are underlined.
Models Metric
SAMoVAR
w/ W
w/o
w/o QV Norm
MSE
MAE
MAE
MSE
MAE
MSE
MAE
ETThi (96)
0.357
0.394
0.362
0.396
0.360
0.395
0.370
0.403
ETThi (192)
0.398
0.419
0.419
0.434
0.404
0.430
0.423
0.441
ETTh1 (336)
0.422
0.442
0.431
0.447
0.436
0.455
0.442
0.461
ETThl (720)
0.427
0.451
0.467
0.435
0.450
0.478
ETTml (96)
0.278
0.339
0.284
0.341
0.282
0,343
0.285
0.346
ETTml (192)
0.318
0.367
0.327
0.371
0.322
0.373
0.334
0.379
ETTml (336)
0.359
0.396
0.368
0,400
0.363
0.397
0.373
0,403
ETTml (720)
0.401
0.413
0.405
0.421
0.408
0.426
0,409
0.427
Table Full results of the ablation studies of the number of heads and dimension. The MSE and MAE of the test set reported. The hest results are bolded and the second hest are underlined.
Models
Heads=4, dim=16
Heads=4.dim- 16
Headss4,ddim=1
Heads=4.dimm- 16
Metric
MSH
MAE
MSH
MAE
MSE
MAH
MSE
MAE
E (96)
0.357
0.394
0.357
0.395
0.363
0.398
0.365
0,401
ETThl (192)
0.398
0.419
0.403
0.424
0,415
0.426
0.418
0,431
ETThl (336)
0.422
0.442
0.436
0.449
0.435
0.449
0,439
0.452
ETThl (720)
0.427
0.451
0.429
0.452
0.433
0.454
0,431
0.453
ETTml (96)
0.278
0.339
0.281
0.343
0.280
0.341
0.283
0.346
ETTml (192)
0.318
0.367
0.322
0.369
0.322
0.371
0.323
0.373
ETTml (336)
0.359
0.396
0.399
0,364
0,401
0.365
0.401
ETTml (720)
0.401
0.413
0.404
0.413
0.406
0.414
0.406
0,414
Weigh Lay
Figure 8. Additional Visualization of the VAR Synthetic Task with a Random Datapoint in the Validation Set.
2014
Figure 9, Visualization of the 2 temporal influence paths from step 124 to step 128 for the two input time series variable for the datapoint shown above, where even-numbered steps represent endogenous tokens and odd nulmbeeeeeed steps represent exogenous tokens.
LU Matrix Generation: Creating the invertible matrix D via LU decomposition is independent of sequence length L, with complexity O(Hd) per layer.
Per-Layer Operations (for each attention layer 1, 1,.... Latt):
1. Linear Projections: Computing query Q()) and value ( projections costs (LLD2).
16

---

Linear Transformers as VAR Models
Table Full results of the ablation studies of different number of layers/ intermediate points in temporal influence paths. The MSE and MAE of the test set reported. The best results are bolded and the second best are underlined.
Models
1
2
3
5
6
I =
7
8
Metrie
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETThl (96)
0.368
0.400
0.356
0.393
0.357
0.394
0.360
0.397
0.362
0.397
0.360
0.395
0.357
0.394
0.358
0.394
ETThl (192)
0.409
0.431
0.402
0.424
0.398
0.419
0.400
0.421
0.400
0,422
0.400
0.424
0.406
0.426
0,400
0.423
ETThl (336)
0,464
0.475
0.439
0.453
0.422
0.442
0.428
0.449
0.439
0,455
0,440
0.450
0.429
0.444
0.426
0.445
ETThi (720)
0.440
0.464
0.445
0.462
0.427
0.451
0.428
0.452
0.451
0.465
0.454
0.466
0.440
0.454
0.456
0.476
ETTml (96)
0.287
0.344
0.280
0.341
0.278
0.339
0.286
0.344
0.288
0.347
0.285
0.343
0.287
0.342
0,290
0.349
ETTml (192)
0.325
0.369
0.322
0,368
0.318
0.367
0.327
0.371
0.329
0,372
0.331
0.373
0.331
0.375
0.332
0.374
ETTml (336)
0.365
0.398
0.364
0.397
0.359
0.396
0.366
0.401
0.366
0,400
0.368
0.402
0.371
0.405
0.372
0.403
ETTml (720)
0.407
0.419
0.403
0.416
0.401
0.413
0.400
0.417
0.401
0.415
0,408
0.424
0.407
0.426
0.409
0.430
Table Effect of Different Random Seeds on the Results of VAR. The results show that there is almost no difference across five runs with different seeds, demonstrating the stability of SAMo with respect random initialization.
Models
Secd=2023
Sccd=2024
Sced=2025
Sced=2026
Sccd=2027
Metric
MSF
MAF
MSF
MAR
MSR
MAF
MSF
MAR
MSR
MAF
ETThl (96)
0.356
0.393
0.357
0.394
0.357
0.394
0.358
0.395
0.357
0.395
ETThl (192)
0.401
0.421
0.397
0.42
0.398
0.419
0.404
0.422
0.401
0.419
ETThl (336)
0.424
0.446
0.421
0.441
0.422
0.442
0.423
0.449
0.425
0.452
ETThl (720)
0.429
0.454
0.431
0.449
0.427
0.451
0.425
0.452
0.428
0.453
ETTml (96)
0.279
0.34
0.279
0.339
0.278
0.339
0,278
0.34
0.277
0.338
ETTml (192)
0.317
0.367
0.318
0.367
0.318
0.367
0.317
0.367
0,319
0.368
ETTml (336)
0.359
0.397
0.358
0.396
0.359
0.396
0.359
0.398
0.361
0.397
ETTml (720)
0.401
0.413
0.402
0.413
0.401
0.413
0,403
0.414
0.4
0,412
NechtMerx L
SANACoA
LiTTer Bempore Waight
Figure 10. Additional Visualization of the VAR Synthetic Task with Random Datapoint in the Validation Set.
124
422
430
Figure 11. Visualization of temporal influence paths from step 124 to step for the two input time series variable datapoint shown above, where even-numbered steps represent endogenous tokens and odd ood-mmbeered steps represent exogenous tokens.
2. Cumulative State Update: The recursive computation
W. = W V!()
incurs complexity of O(Hd²) per timestep, totaling O O /H) across all timesteps.
3. Output Computation: Calculating
Y, = Q @ W.
also results in (LLD2/H)
17

---

Linear Transformers as VAR Models
Table Comparison of computational costs. This comparison utilizes the data format of ETTh1 construct model inputs. T, set 512 for the baselines, 1024 for VAR-based models, and the other hyper parameters for every model are set according to their default configurations.
Models
SAMoVAR
Tians
FiXedVAR
Metric
FL.OPs
Params
FLOPs
Params
FLOPss
Params
Lp 96
43.31M
157.3K
50.37M
181.9K
35.74M
196.5K
Lp 192
25. 25.24M
175.9K
29.08M
200.4K
21.11M
215K
LP 336
18.44M
199.6K
20.99M
224.1K
15.69M
238.7K
Lp 720
11.38M
272.6K
12.63M
297.2K
10M
311.8K
Models
Encformer
PatchTST
iTransformer
Metric
FLOPs
Params
FLOPs
Params
FLOPs
Params
Lp 96
1.328G
1.646M
180.9M
1.841M
42.39M
1.923M
LP 96
1.442G
1.647M
215.5M
3.414M
42.66M
1.935M
LP 96
1.613G
1.648M
267.5M
5.773M
43.07M
1.954M
LP 96
2.068G
1.65M
405.9M
12.07M
44.
2.003M
Figure 12. Additional Visualization of the VAR Synthetic Task with Random Datapoint in the Training Set.
124
AUME
Figure Visualization of temporal influence paths from step to step for the two input series variable datapoint shown above, where wen-nuubereed steps represent endogenous tokens and odd numbeeedd steps represent exogenous tokens.
4. Structural Transformation:
Y trransfoomedd = cinsumm'blus lide + bhe', Yr, D-1)
similarly requires O(LD//H)
Overall Complexity: Summing up each attention layer's operations, each layer requires OLDD²). Given that the number of attention layers Lum is constant with respect to sequence length, the overall computational complexity thus OLLD²), clearly linear with respect to the sequence length L.
Therefore, SAMoVAR Attention achieves linear complexity in sequence length.
A.7.2. CLARIFICATION ON COMPUTATIONAL EFFICIENCY
We further clarify the computational efficiency results presented in Table 8. The values for FLOPs (floating-point operations) and parameter counts were calculated consistently across all compared models using the ETThl dataset. Specifically, all baseline models were evaluated using an input length of I, 512, while linear-attention-basee models (including
18

---

Linear Transformers as VAR Models
SAMoVAR) were evaluated using L, 1024. All other hyperparameters strictly followed the original ETTh1 dataset configurations. For linear attention variants (including SAMoVAR), we the hidden dimension in d = 64 for the ETThl dataset with C' = 7. This setting naturally reduces the number of parameters compared to baseline models (PatchTST/iTranse Traasfss)//rrnsffmmmrr)) which typically use higher dimensions (128/256). SAMoVAR further optimizes computational costs by eliminating the key projection matrices and sharing the output matrices W.. leading to fewer FL.OPs compared to standard weight per lag, avoids the computational overhead associated with dynamically generated weights, resulting lower FLOPs than standard linear-attention Transformers.
332/C] CJ. resulting
secer--ttention-baae Transformers, FixedVAR, despite containing more parameters due to a fixed
A.8. Theoretical Analysis of Robust Path Pruning: Bounded Dot-Product via RMSNorM
Proposition. Applying RMSNorm to query and value vectors bounds the magnitudes of the dot-products temporal influence paths, thus preventing numerical instability in SAMoVAR.
Proof. We first recall that the query q: and value vectors v!" are normalized by R.MSNorm as:
RMSNomm X (2) w(").,
RMSNorm(x: (1) W ).
where RMSNorm is defined as:
y
RMSNorm(y)
g.
Σ ト
j=1 Vj
Then, the absolute value of their dot-product can be bounded as follows:
d
IV (T
Te
S <IV.
12lla 2 112 ~ "ll!!" 12]ls,)))
122
Given that the gain parameters g are typically initialized with small values, e.g. N 1/d), the resulting upper bound is approximately 1. Thus, for given temporal influence path spanning / layers:
P E
(1) Iv L -- (0) qr
(1-IT
(
<1
In practice, these tro-prrd are smaller than one, ensuring numerical stability and preventing gradient explosion during training.
A.9. Theoretical Analysis of Robust Path Pruning: Orthogonality Probability in High Dimensions
Proposition. As the dimension increases, randomly initialized vectors become increasingly orthogonal with high probability, naturally pruning irrelevant temporal influence paths in SAMoVAR.
Proof. Consider two normalized random vectors q, v E Rd, each component initialized independently from a symmetric distribution with mean zero and finite variance. The dot-product of these vectors is given by:
T V
4.
By the central limit theorem, as d + oc, the distribution of the dot-product converges to a Gaussian distribution:
q T d= N 00. 2)
Given small threshold 0, compute the probability that the absolute value of their dot-product is below e:
P(IâT 0 <) R 2 (vV)) 1,
19

---

Linear Transformers as VAR Models
where & denotes the cumulative distribution function of the standard Gaussian distribution.
As dimension d increases, this probability approaches 1, implying that with high probability, random vectors become nearly orthogonal. For temporal influence paths defined as:
P
P (v ( 1 ( ap (v 9 (1-1)). ... (v) ( (2)
the probability of any path having small or near-zero magnitude grows with dimension, thus naturally "pruning" irrelevant paths. During training, important paths are enhanced by gradient-based updates, selectively preserving informative temporal influence patterns.
20


Reviews:
[{"review_id": "g4EUVCK8I2", "reviewer": "Reviewer_ZzwP", "summary": "This paper proposes structural modifications to linear Transformer architectures to better align them with the Vector Autoregressive (VAR) framework, which is widely used in time series forecasting. The authors show that while a single-layer linear attention module can naturally express a dynamic VAR structure, standard multi-layer Transformer designs introduce misalignments that hinder interpretability and forecasting performance. To address this, they introduce SAMoVAR (Structural Aligned Mixture of VAR), a novel Transformer variant that reorganizes the attention and MLP layers to preserve a coherent dynamic VAR interpretation across multiple layers. The paper presents both theoretical formulations and empirical evaluations, demonstrating that SAMoVAR achieves improved accuracy, interpretability, and computational efficiency on a range of standard multivariate time series forecasting benchmarks, outperforming existing state-of-the-art models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Performance differ from those presented in original papers ...\n\nThank you for your valuable question. For each baseline and output length $ L_P $, we run experiments with input lengths $ L_I \\in \\{512, 1024, 2048, 4096\\} $ and report the best result. This avoids biases from models' sensitivity to input length, as noted in prior work. Our reported results may outperform those in the original papers, but never underperform. This evaluation strategy, also used in TiDE (https://arxiv.org/abs/2304.08424), ensures fairer and more strict comparison. Our problem setup follows DLinear, PatchTST, and iTransformer, and can be verified in our code repository (see `baseline.sh`).\n\n```bash\nfor pred_len in 96 192 336 720\ndo\n\nfor seq_len in 4096 2048 1024 512\ndo\n...\n```\n\n>Computational efficiency is less clearly supported\n>Including formal analysis or clearer theoretical justification would strengthen the paper’s contribution.\n\nThank you for the suggestion. Our writing style follows prior work of LinTrans (e.g., Katharopoulos et al., 2020; Yang et al., 2024), focusing more on structure derivations and insights. We’re happy to add rigorous proofs of computational efficiency in the revision. Please see below:\n\n**Proposition** SAMoVAR Attention has $ O(L) $ complexity with respect to sequence length $ L $.\n\n**Proof** We analyze Algorithm 1 (SAMoVAR) in the appendix. Let batch size be $ B $, model dimension $ D $, number of heads $ H $, per-head dimension $ d = D/H $, and number of attention layers $ L_{\\text{attn}} $.\n\n**Preprocessing**:\n- **LU Matrix Generation**: Creating the invertible matrix $ \\mathbf{D} $ via LU decomposition is independent of $ L $, with cost $ O(Hd^2) $ per layer.\n\n**Per-Layer Operations** (for each $ l = 1, ..., L_{\\text{attn}} $):\n\n1. **Linear Projections**: Computing $ Q^{(l)} $ and $ V^{(l)} $ costs $ O(LD^2) $.\n\n2. **Cumulative State Update**: Recursively updating \n   $$\n   W_t = W_{t-1} + K_t \\otimes V_t^{(l)}\n   $$\n   costs $ O(Hd^2) $ per step, totaling $ O(LHd^2) = O(LD^2/H) $.\n\n3. **Output Computation**:\n   $$\n   Y_t = Q_t^{(l)} \\otimes W_t\n   $$\n   also costs $ O(LD^2/H) $.\n\n**Structural Transformation**:\n   $$\n   Y_{t,\\text{transformed}} = \\text{einsum}('bhd,hde \\rightarrow bhe', Y_t, \\mathbf{D}^{-1})\n   $$\n\nhas the same cost: $ O(LD^2/H) $.\n\n**Conclusion**: Each layer has total cost $ O(LD^2) $. Since the number of layers is constant, overall complexity is $ O(LD^2) $, linear in sequence length $ L $.\n\n>Comparison with TimeMixer++\n\nThank you for the suggestion. We added a comparison with TimeMixer++ at [Link (Table 2)](https://anonymous.4open.science/r/SAMoVAR-Rebuttal-F83C), using its official implementation and hyperparameters from `TimeMixer_ETTh1_unify.sh`. We evaluated all $ L_I \\in \\{512, 1024, 2048, 4096\\} $. Note that the $ L_I = 512 $ result here outperforms the original $ L_I = 96 $ result. Our findings show that TimeMixer++ have no robust advantage over prior baselines and performs worse at longer $L_I$, similar to patterns seen in the DLinear and PatchTST papers. Interestingly, while recent NLP models trend toward longer contexts, recent TSF models (e.g., iTransformer, TimeMixer++) keep $ L_I $ as 96, compared to 512 in PatchTST, which is counterintuitive given TSF data (on test set) is typically continuous and not limited the usage of long context. We hope our varying $ L_I $ setup encourages a shift for TSF community toward the standards in line with broader sequence modeling area.\n\n>Why SAMoVAR Better Than CATS\n\nThank you for the insightful question. Our results show that LinTrans with ARX tokenization performs closely to CATS, suggesting that autoregressive loss (as short-context data augmentation) and varying-context multitask training allow linear attention to match MLP-based models.\n\nSAMoVAR goes further by explicitly modeling the VAR data generation process of MLP-constructed input sequences. You can view its MLP module as linearizing ARX-tokenized inputs, enabling dynamic VAR modeling via aligned linear attention. This offers a stronger inductive bias for TSF by better capturing lag, cycles in time series patterns compared to LinTrans.\n\n>Clarification of Table 8 (Efficiency)\n\nThe computational costs in Table 8 align with our analysis. FLOPs and parameter counts are measured on ETTh1 with $ L_I = 512 $ for baselines and $ L_I = 1024 $ for linear attention models. All other hyperparameters follow the original ETTh1 settings.\n\nFor linear attention variants, we set hidden dimensions as $ d = \\text{int}(32 \\sqrt{C}) $; with $ C=7, d = 64 $, this results in fewer parameters than PatchTST/iTransformer (128/256).\n\nSAMoVAR further reduces cost by removing all key projection matrices and sharing all output matrices $ \\mathbf{W}_o $, leading to lower computational cost than LinTrans. FixedVAR, while having more parameters due to fixed weights per lag, avoids dynamic weight generation and thus has lower FLOPs than LinTrans.\n\nIn summary, Table 8 fully aligns with our analysis.", "claims_and_evidence": "The primary claim of the paper is that the proposed SAMoVAR architecture improves over state-of-the-art Transformer-based models for time series forecasting in terms of accuracy, interpretability, and computational efficiency. This claim is generally supported by empirical evidence: the authors report strong performance across a wide range of standard benchmarks, and the improvements in accuracy are consistent across datasets.\n\nHowever, some concerns remain. First, while the reported results show SAMoVAR outperforming prior methods, the performance margins differ from those presented in original papers of some recent baselines, suggesting that the experimental setup may not fully match those prior works. Clarification on baseline implementations and training details would help validate the comparison.\n\nSecond, the claim of superior computational efficiency is less clearly supported. Table 8, which is meant to demonstrate SAMoVAR's efficiency, includes results that appear counterintuitive or internally inconsistent. These discrepancies warrant further clarification from the authors to substantiate the efficiency claim.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are appropriate for the time series forecasting task. The paper follows standard practices in the field, employing widely-used multivariate TSF benchmark datasets and comparing against strong Transformer-based baselines. The experimental setup, including input/output lengths and tokenization strategies, is consistent with prior work, ensuring a fair and relevant evaluation of the proposed model.", "theoretical_claims": "The paper does not include formal theoretical claims or proofs. While it provides intuitive interpretations and derivations connecting linear attention mechanisms to dynamic VAR structures, these are presented in an informal manner without rigorous theoretical guarantees. This absence raises some concern regarding whether the observed performance gains stem from fundamental modeling improvements or from design choices that may implicitly favor the proposed method. Including formal analysis or clearer theoretical justification would strengthen the paper’s contribution.", "experimental_designs_or_analyses": "The experimental design and analysis in the paper are generally sound. The authors evaluate the proposed method across a diverse set of standard time series forecasting benchmarks and compare against strong, commonly used baselines. However, there are two notable concerns. First, the reported results for some prior state-of-the-art methods differ from those presented in their original papers, raising questions about reproducibility and consistency in implementation. Second, the compute efficiency claims—particularly those in Table 8—are not fully convincing and appear inconsistent or counterintuitive. These aspects would benefit from further clarification and verification.", "relation_to_broader_scientific_literature": "This paper contributes to the growing body of work at the intersection of classical statistical models and deep learning architectures. By drawing a connection between Vector Autoregressive (VAR) structures and linear Transformer models, the authors offer a novel perspective on how autoregressive attention mechanisms can be restructured to align with traditional time series modeling principles. If the empirical results are verified, the proposed SAMoVAR model could represent a meaningful step forward in the design of interpretable and effective models for time series forecasting (TSF), potentially influencing future research in both the deep learning and econometrics communities.", "essential_references_not_discussed": "While not strictly required, given the popularity and rapid development in time series forecasting (TSF), it would be helpful for the authors to discuss very recent advances in the field. One such example is TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis by Wang et al., recently accepted at ICLR 2025. This work proposes a highly general and competitive TSF model, and a comparison—either empirical or conceptual—would help situate SAMoVAR more clearly within the current landscape of TSF methods.", "other_strengths_and_weaknesses": "No additional strengths or weaknesses beyond those already discussed.", "other_comments_or_suggestions": "No additional comments or suggestions.", "questions_for_authors": "1. Comparison to Recent SOTA Methods: Given the likely overlap between the authors of this paper and the recent CATs model, could the authors provide more discussion or empirical comparison to other recent strong TSF methods such as TimeMixer++ (ICLR 2025)? A clearer comparison would help better position SAMoVAR in the current TSF landscape.\n\n2. Why SAMoVAR Works Better Than CATs: While the paper clearly explains how SAMoVAR is constructed, it is less clear why it performs better than prior models like CATs. Could the authors elaborate on the intuition or mechanisms that drive the performance gains?\n\n3. Clarification of Table 8 (Compute Efficiency): The reported FLOPs in Table 8 appear to be inconsistent or counterintuitive for several models. Could the authors clarify how these values were computed and confirm their correctness? A more transparent explanation would improve the credibility of the compute efficiency claims.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oscWaHCHWd", "reviewer": "Reviewer_RLuA", "summary": "This paper demonstrates that autoregressive linear attention can be interpreted as a rank-1 vector autoregressive (VAR) model. Building upon this perspective, the authors introduce SAMoVAR, a novel model achieved by stacking multiple linear attention layers. SAMoVAR overcomes the inherent rank-1 limitation of linear attention, thus enhancing the model's expressiveness and forecasting performance on time series tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">However, integrating more explicit discussions about linear attention...\n\nThank you for the valuable suggestion. Currently, related literature is only discussed in the Introduction and Background. We will add a Related Work section in the revision for a more thorough and structured discussion.\n\n>The page header is still shown as \"Submission and Formatting Instructions for ICML 2025\".\n\nThank you for pointing this out. We will ensure the correct header format is used in the revision to comply with the guidelines.\n\n>Could you provide insight into how your VAR framework addresses or mitigates common issues with linear attention and its variants\n\nThank you for the insightful question. Prior work addresses the unbounded gradient issue by normalizing $ \\mathbf{o}_t $. Similarly, in SAMoVAR, we apply RMSNorm to stabilize computations:  \n$$\n\\phi(\\mathbf{q}_t)^{(l)} = \\text{RMSNorm}(\\mathbf{x}_t^{(1)} \\mathbf{W}_q^{(l)}), \n\\quad \\phi(\\mathbf{k}_i)^{(1)} = \\mathbf{x}_i^{(1)}, \n\\quad \\phi(\\mathbf{k}_i)^{(l)} = \\mathbf{o}_i^{(l-1)}, \n\\quad \\mathbf{v}_i^{(l)} = \\text{RMSNorm}(\\mathbf{x}_i^{(1)} \\mathbf{W}_v^{(l)})\n$$\nNote that $ \\mathbf{x}_i^{(1)} $ is also normalized beforehand. This aligns well with the approach used in the referenced work.\n\nThe notion of *attention dilution* suggests that softmax attention tends to focus on local token relationships. However, we caution against a potential causality issue here: in NLP, softmax attention appears to model local relationships not because of an inherent bias, but because NLP data often demands such modeling. Thanks to its strong selectivity, softmax attention can naturally focus on local context even without explicit inductive bias: unlike linear attention, which tends to produce more diffuse attention patterns due to weaker selectivity. In contrast, for time series data, softmax attention, being a smooth approximation of the max operation, can overly concentrate on specific intervals. Yet time series typically exhibit stable temporal patterns such as lagged autocorrelations, cycles, seasonality, and trends. These effects are less variable than token-wise dependencies in NLP. Therefore, a VAR-based inductive bias is better suited to capture these temporal consistent structures, avoiding the overly sharp, localized focus induced by softmax attention, which may result in overfitting as stated in previous Transformer-based TSF works.\n\n>The current explainability analyses primarily highlight smoother temporal weights and locality. Could you further clarify how SAMoVAR's VAR structure explicitly enhances interpretability regarding task-specific insights?\n\nThank you for the question. As noted earlier, attention in NLP rarely models effects tied to absolute lag positions, since the context relationships vary greatly across inputs. This differs significantly from the stable, linearizable lag patterns seen in autocorrelated VAR processes.\n\nIn contrast, tasks involving time series, images, speech, and those in Long Range Arena (https://arxiv.org/abs/2011.04006) are more position-sensitive. Recent studies on efficient linear attention often evaluate on these domains. While many linear attention variants struggle to outperform vanilla Transformers on NLP tasks, they often surpass them on position-related tasks, especially LRA benchmarks.\n\nOur interpretation of linear attention as a form of VAR modeling helps explain this: VAR weights can explicitly capture stable positional patterns, such as autocorrelations in token space, offering a better inductive bias in these settings with consistent positional patterns.\n\n>Have you evaluated SAMoVAR in language modeling tasks? If yes, how does its performance compare with traditional linear attention methods?\n\nThank you for the valuable question. Building on the previous responses, you may understand that our positioning of SAMoVAR is as follows: it is particularly suited for data with clear autoregressive structure, strong autocorrelations, and stable positional effects, as seen in TSF data.\n\nFor NLP tasks, SAMoVAR’s dynamic weight matrix would need to vary more significantly, similar to softmax, to adapt to diverse input patterns. This remains an interesting direction for future exploration.", "claims_and_evidence": "The claims presented in the paper are supported by both theoretical analyses and empirical evidence.", "methods_and_evaluation_criteria": "The authors evaluate the proposed method across multiple multivariate time series forecasting datasets, demonstrating consistent improvements over state-of-the-art methods on most datasets.", "theoretical_claims": "The theoretical derivations in the paper are clearly presented. I did not identify any major issues in the theoretical analysis.", "experimental_designs_or_analyses": "The main comparative experiments are well-designed. However, the explainability section lacks clarity. This aspect needs further clarification (see Questions for Authors).", "relation_to_broader_scientific_literature": "The paper provides a novel perspective by framing linear attention in the VAR context, potentially bridging VAR models and linear attention architectures. However, connections to the linear attention literature could be strengthened further, particularly by directly comparing SAMoVAR with Linear Attention and its variants. Such comparisons could enhance accessibility for audiences from the Linear Transformer and LLM communities.", "essential_references_not_discussed": "The references are sufficiently comprehensive. However, integrating more explicit discussions about linear attention approaches prevalent in the broader literature (e.g., https://arxiv.org/pdf/2210.10340; https://arxiv.org/pdf/2006.16236; https://arxiv.org/pdf/2405.13956) would strengthen connections to related work.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Formatting / Typos:\n\n1. The page header is still shown as \"Submission and Formatting Instructions for ICML 2025\".", "questions_for_authors": "[Q1] Could you provide insight into how your VAR framework addresses or mitigates common issues with linear attention and its variants, such as attention dilution and unbounded gradients described by Qin et al. (https://arxiv.org/pdf/2210.10340)?\n\nUnder the VAR interpretation, what specifically causes these issues in linear attention and its variants?\n\nWhich of these known problems does SAMoVAR solve or alleviate, and which problems persist? Please clarify the underlying reasons.\n\n[Q2] The current explainability analyses primarily highlight smoother temporal weights and locality. Could you further clarify how SAMoVAR's VAR structure explicitly enhances interpretability regarding task-specific insights?\n\n[Q3] Have you evaluated SAMoVAR in language modeling tasks? If yes, how does its performance compare with traditional linear attention methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uUD73P6bED", "reviewer": "Reviewer_aK3c", "summary": "This paper demonstrates that a single linear attention layer behaves like a dynamic VAR model and that deeper Transformers can be restructured to align with autoregressive objectives. Based on these insights, the authors introduce SAMoVAR, a Transformer variant that leverages dynamic VAR weights to enhance forecasting performance, interpretability, and efficiency.\n\nUpdate after rebuttal: Thanks for the detailed rebuttal. I appreciate the authors' efforts in addressing my concerns and helping me better understand the work. Therefore, I will maintain my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Why does the validation loss in Figure 4 exhibit significant oscillations during the training process?\n\nThank you very much for taking the time to read our paper and for acknowledging its contributions.\n\nWe apologize for the misunderstanding. In Figure 4, we cropped the Y-axis of the validation loss curve. Like the training loss, it starts around 0.1 and decreases over epochs. However, using 0.1 as the upper bound would make the convergence of all three curves indistinguishable in this compact figure. As shown, the validation loss fluctuates within ±0.002 in later epochs, consistent with the training loss scale.\n\nNotably, SAMoVAR surpasses the final performance of the other two baselines after just 30 epochs. The Y-axis scaling was chosen purely to highlight this advantage for clarity.\n\nYou can verify more of this in our code repository: see `visualization/model_visualize.ipynb`, specifically `In [5]` and `Out [5]`, where we use `plt.ylim(-0.01, 0.1)` for the training loss and `plt.ylim(0.0025, 0.0085)` for the validation loss.\n\n>Line 18: The paper refers to VAR before defining it.\n\nThank you for pointing this out. We will ensure that VAR is properly defined upon its first appearance in the revision. \n\n>Theoretical Claims:\n>N/A.\n\nThank you for your positive feedback. Even though you didn’t explicitly request it, we’re happy to provide a more detailed derivation of how SAMoVAR’s multi-layer linear attention can be expressed as a VAR structure, to further support your understanding. If you're already familiar with this, feel free to skip the following explanation. Thank you again!\n\n\n**Proposition** For any $l \\geq 1$, the output $\\mathbf{o}_t^{(l)}$ can be expressed as VAR structure:\n$$\\mathbf{o}\\_t^{(l)\\top} = \\sum\\_{j=1}^t \\mathbf{B}\\_{t,j}^{(l)} \\mathbf{x}\\_j^{(1)\\top}$$\n\n**Proof by Induction:**\n\n**Base Case (l=1)** For a single linear attention layer, we have:\n$$\\mathbf{o}\\_t^{(1)\\top} = \\sum_{i=1}^t \\mathbf{A}\\_{t,i}^{(1)} \\mathbf{k}\\_i^{(1)\\top} = \\sum\\_{i=1}^t \\mathbf{A}\\_{t,i}^{(1)} \\mathbf{x}\\_i^{(1)\\top}$$\n\nSince $\\mathbf{k}\\_i^{(1)} = \\mathbf{x}\\_i^{(1)}$, we define $\\mathbf{B}\\_{t,i}^{(1)} = \\mathbf{A}\\_{t,i}^{(1)}$, which gives us the VAR form for the first layer.\n\n**Inductive Step** The key insight is the relationship between outputs and inputs across layers. When we use $\\mathbf{k}\\_i^{(l+1)} = \\mathbf{o}\\_i^{(l)}$, we get:\n$$\\mathbf{o}\\_t^{(l+1)\\top} = \\sum\\_{i=1}^t \\mathbf{A}\\_{t,i}^{(l+1)} \\mathbf{o}\\_i^{(l)\\top}$$\n\nSubstituting the inductive hypothesis and rearranging the sums:\n$$\\begin{aligned}\n\\mathbf{o}\\_t^{(l+1)\\top} &= \\sum\\_{i=1}^t \\mathbf{A}\\_{t,i}^{(l+1)} \\left(\\sum\\_{j=1}^i \\mathbf{B}\\_{i,j}^{(l)} \\mathbf{x}\\_j^{(1)\\top}\\right) \\\\\n&= \\sum\\_{j=1}^t \\sum\\_{i=j}^t \\mathbf{A}\\_{t,i}^{(l+1)} \\mathbf{B}\\_{i,j}^{(l)} \\mathbf{x}\\_j^{(1)\\top} \\\\\n&= \\sum\\_{j=1}^t \\left(\\sum_{i=j}^t \\mathbf{A}\\_{t,i}^{(l+1)} \\mathbf{B}\\_{i,j}^{(l)}\\right) \\mathbf{x}\\_j^{(1)\\top}\n\\end{aligned}$$\n\nTherefore:\n$$\\mathbf{o}\\_t^{(l+1)\\top} = \\sum\\_{j=1}^t \\mathbf{B}\\_{t,j}^{(l+1)} \\mathbf{x}\\_j^{(1)\\top}$$\n\nWhere:\n$$\\mathbf{B}\\_{t,j}^{(l+1)} = \\sum\\_{i=j}^t \\mathbf{A}\\_{t,i}^{(l+1)} \\mathbf{B}\\_{i,j}^{(l)}$$\n\nThis recursive definition of $\\mathbf{B}_{t,j}^{(l)}$ captures all possible influence paths from time $j$ to time $t$ through $l-1$ intermediate points, confirming that multi-layer linear attention maintains an VAR structure throughout the network.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The authors introduce SAMoVAR, a Transformer variant that leverages dynamic VAR weights to enhance forecasting performance, interpretability, and efficiency.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "I am not very familiar with the time series forecasting task, but the paper reads logically and intuitively.\n\nStrengths:\n1. The motivation behind the proposal of SAMoVAR is well-founded.\n2. The theoretical explanations are comprehensive and clear.\n3. The experiments and the analysis of experimental results are thorough and convincing.\n\nQuestion:\n\nWhy does the validation loss in Figure 4 exhibit significant oscillations during the training process?\n\nMinor Issue:\n\nLine 18: The paper refers to VAR before defining it.", "other_comments_or_suggestions": "See strengths and weaknesses.", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SkmEuWBjp", "reviewer": "Reviewer_pxAU", "summary": "This paper addresses the misalignment between deep Transformer architectures and autoregressive objectives in time series forecasting (TSF), proposing Structural Aligned Mixture of VAR (SAMoVAR) to integrate interpretable dynamic VAR weights into multi-layer linear Transformers. By reorganizing the input-output flow and aligning the structure with VAR models, SAMoVAR enhances the ability to capture data generative processes, achieving improved performance, interpretability, and efficiency compared to state-of-the-art TSF models through experiments on synthetic and real-world datasets.\n\n## update after rebuttal\nI have reviewed your responses and those to other reviewers. While some concerns have been addressed, I remain unconvinced regarding the theoretical insights or experimental results on capturing periodicity and non-linear patterns. Therefore, I will maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">No theoretical analysis for robust path pruning\n\nThank you for the insightful question. We provide a proof sketch below.\n\n**Theorem 1**: RMSNorm applied to query and value vectors bounds the magnitudes of dot products in temporal influence paths, preventing numerical instability.\n\n**Proof Sketch**:\nQuery and value vectors are normalized using RMSNorm:\n\n$$\\mathbf{q}_t^{(l)} = \\text{RMSNorm}(\\mathbf{x}_t^{(1)}\\mathbf{W}_q^{(l)}), \\quad \\mathbf{v}_i^{(l)} = \\text{RMSNorm}(\\mathbf{x}_i^{(1)}\\mathbf{W}_v^{(l)})$$\n\nwhere:\n\n$$\\text{RMSNorm}(\\mathbf{y}) = \\frac{\\mathbf{y}}{\\sqrt{\\frac{1}{d}\\sum_{j=1}^{d} y_j^2}} \\odot \\mathbf{g}$$\n\nThis bounds their dot product:\n\n$$|\\mathbf{v}\\_{i}^{(l)\\top} \\mathbf{q}\\_{t}^{(l)}| \\leq \\|\\mathbf{g}\\_v^{(l)}\\|_2 \\cdot \\|\\mathbf{g}\\_q^{(l)}\\|_2 \\approx 1$$\n\nwith proper initialization of small gain parameters $\\mathbf{g}$ (e.g., $\\sim \\mathcal{N}(0,\\frac{1}{d})$).\n\nFor a temporal influence path with $l$ layers:\n\n$$|\\mathbf{P}\\_{t,j,\\{i_1,...,i\\_{l-1}\\}}^{(l)}| = |\\mathbf{v}\\_{i_1}^{(l)\\top} \\mathbf{q}\\_t^{(l)}| \\cdot ... \\cdot |\\mathbf{v}\\_{j}^{(1)\\top} \\mathbf{q}\\_{i\\_{l-1}}^{(1)}| \\leq 1$$\n\nIn practice, path magnitudes decrease with depth as dot products are typically smaller than 1.\n\n**Theorem 2**: As dimension $d$ increases, random vectors become increasingly orthogonal, naturally pruning unnecessary paths.\n\n**Proof Sketch**:\nFor normalized vectors $\\hat{\\mathbf{q}}$ and $\\hat{\\mathbf{v}}$ with random initialization, their dot product:\n\n$$\\hat{\\mathbf{q}}^T \\hat{\\mathbf{v}} = \\sum_{i=1}^d \\hat{q}\\_i \\hat{v}\\_i$$\n\nhas mean 0 and variance $\\frac{1}{d}$. By the Central Limit Theorem, for large $d$, this approaches $\\mathcal{N}(0,\\frac{1}{d})$.\n\nFor threshold $\\epsilon > 0$, the probability of near-orthogonality is:\n\n$$P(|\\hat{\\mathbf{q}}^T \\hat{\\mathbf{v}}| < \\epsilon) \\approx 2\\Phi\\left(\\epsilon\\sqrt{d}\\right) - 1$$\n\nwhich approaches 1 as $d$ increases.\n\nFor a path:\n\n$$\\mathbf{P}\\_{t,j,\\{i_1,...,i\\_{l-1}\\}}^{(l)} = \\mathbf{v}\\_{i\\_1}^{(l)\\top} \\mathbf{q}\\_t^{(l)} \\cdot ... \\cdot \\mathbf{v}\\_{j}^{(1)\\top} \\mathbf{q}\\_{i_{l-1}}^{(1)}$$\n\nWith larger $d$, the likelihood of a near-zero dot product increases, pruning irrelevant paths while training strengthens important ones.\n\n>The claim that SAMoVAR provides interpretable ... could be further strengthened. \n\nOur current temporal influence path visualization offers interpretability similar to attention maps but does not yet match that of traditional MLE-based VAR models. This paper focuses on SAMoVAR’s predictive performance, though full statistical interpretability is possible through:\n\n1. Initializing query/value weights via SVD of training set covariance.  \n2. Breaking symmetry by fixing an element in the q/v weight matrices.  \n3. Removing MLPs, pre-normalization, patching, and projections for full linearity.  \n4. Training with MLE loss.  \n5. Estimating parameter uncertainty using asymptotic covariance for standard errors.  \n6. Applying the Delta method to propagate uncertainty in dynamic VAR weights.\n\nThese changes would enable full interpretability but reduce model flexibility and accuracy. We’re happy to share our ongoing works in this direction, though it is beyond this paper’s current scope.\n\n>Whether the cumulative paths preserve VAR properties (e.g., stationarity)\n>Proof of VAR structure\n\nSAMoVAR is not constrained to classic stationarity. Enforcing eigenvalues within the unit circle, especially with long lags, would limit expressiveness and harm predictive power. As for the structural derivation of the model, Please see the proof in our response to Reviewer aK3c. Thank you!\n\n>Batch size adjustments may introduce bias.\n\nAs noted in Appendix (line 707), we use gradient accumulation to maintain an effective batch size of 32 when reducing batch size. A reference will be added in the revision.\n\n>Ground-truth VAR parameters (Sec. 5.1) are not described, limiting reproducibility.\n\nThe VAR parameters are shown in the top-left heatmap of Figure 3. All experiments and code from Section 5.1 are available in `visualization/model_visualize.ipynb` to ensure maximized reproducibility. \n\n>non-linear patterns ... ability to handle periodicity and non-linear interactions...\n\nDue to space limits, please refer to our response to **Reviewer RLuA** for discussion on NLP and complex data.\n\n>SAMoVAR... in medical forecasting tasks\n\nWe use standard TSF benchmarks, which contain irregular patterns in some data (like Weather, ETTh1, ETTh2). SAMoVAR performs well, suggesting potential for irregular medical data.\n\n>Proof of SAMoVAR maintains O(N) complexity\n\nDue to space limits, please see the **proof** in our response to **Reviewer ZzwP**.\n\n>guidelines for selecting optimal configurations\n\nWe use fixed hyperparameters across all datasets. The fixed $L_I$ settings highlight robustness compared to baselines needing $L_I$ tuning. See our response to Reviewer ZzwP, and Appendix A.3, where full configuration are ready to use without any modification.", "claims_and_evidence": "The assertion that single-layer linear attention aligns with a dynamic VAR structure is backed by rigorous theoretical derivations (Equations 2–3) and validated through synthetic experiments (Figures 3–4). Additionally, the improved performance of SAMoVAR over baseline models is demonstrated across multiple datasets, with significant reductions in MSE (e.g., 30% on the Solar dataset compared to previous models), providing strong empirical support for the effectiveness of the proposed model.\n\nHowever, the claim that SAMoVAR provides interpretable dynamic VAR weights could be further strengthened. While visualizations (e.g., Figure 5) illustrate the temporal influence paths, they lack quantitative analysis.", "methods_and_evaluation_criteria": "The paper introduces \"Robust Path Pruning\" to ensure numerical stability and control weight variance in SAMoVAR. While the ablation studies show that removing RMSNorm degrades performance, further validation is needed to fully support the claims around path pruning. Specifically, the paper lacks detailed metrics on the percentage of paths pruned and the correlation between pruned paths and model performance. Additionally, theoretical analysis on the probability of query-value orthogonality and comparison to active pruning methods would strengthen the claims. Assessing the impact of pruning on the model’s ability to capture long-term dependencies and measuring the rank of the resulting weight matrices would also provide deeper insights into the effectiveness of the pruning mechanism.", "theoretical_claims": "Multi-Layer Alignment: The claim that \"multi-layer linear attention aligns with VAR\" (Sec. 4.1) relies on recursive expansions (Eq. 5) but lacks formal proof. For instance, it is unclear whether the cumulative paths preserve VAR properties (e.g., stationarity).\n\nStability Guarantees: No theoretical analysis ensures that path pruning or RMSNorm prevents gradient explosion in deep layers.", "experimental_designs_or_analyses": "Batch size adjustments for large datasets (e.g., PEMS07 excluded due to batch size=1) may introduce bias. No sensitivity analysis is provided.", "relation_to_broader_scientific_literature": "The work bridges classical VAR models and modern linear Transformers, extending prior efforts like [Katharopoulos et al. 2020] (linear attention as RNNs) and [Zeng et al. 2023] (Transformer-VAR comparisons).", "essential_references_not_discussed": "Limited discussion of concurrent work on efficient attention mechanisms (e.g., Retentive Networks [Sun et al. 2023], H3 [Dao et al. 2023]), which also achieve linear complexity.", "other_strengths_and_weaknesses": "Strengths:\n\n- Originality: The integration of VAR theory into linear attention is novel, offering a fresh perspective on Transformer interpretability.\n\n- Significance: The method’s efficiency and performance improvements are practically valuable for real-world TSF applications.\n\nWeaknesses:\n\n- Clarity: The temporal influence path concept (Sec. 4.1) is overly abstract. A step-by-step example (e.g., for 2-layer SAMoVAR) would improve readability.", "other_comments_or_suggestions": "The current model framework diagram is insufficient for fully understanding the proposed mechanisms, especially the Temporal Influence Path and Robust Path Pruning. Enhancing the visual representation with more detailed diagrams or flowcharts would significantly aid in clarifying these complex components and their integration into the overall architecture.", "questions_for_authors": "1.Time series often exhibit periodic and non-linear patterns, especially in long-term forecasting. How does SAMoVAR effectively capture these characteristics, and are there theoretical insights or experiments demonstrating its ability to handle periodicity and non-linear interactions?\n\n2.Irregular time series data are common in healthcare applications. Can the authors discuss SAMoVAR’s potential advantages in handling such data, particularly in medical forecasting tasks?\n\n3.The paper claims that SAMoVAR maintains O(N) complexity, but the new architecture might introduce additional overhead. Can the authors provide a detailed complexity analysis, especially for the Temporal Influence Path mechanism?\n\n4.The experiments show SAMoVAR’s effectiveness with specific hyperparameters. How robust are these settings across different datasets, and can the authors provide guidelines for selecting optimal configurations in various scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bu6JTmGLQm", "reviewer": "Reviewer_NbcJ", "summary": "The paper addresses time series forecasting by aligning linear-attention Transformer with vector autoregressive (VAR). The authors reveal that while single-layer linear attention mechanisms naturally exhibit a dynamic VAR structure, but multi-layer Transformers can misalign with the autoregressive forecasting objective. To address this, they propose a linear-attention Transformer variant that incorporates dynamic VAR weights by reorganizing input-output flows. Empirical results show the proposed models improve the performance and interpretability compared to several baseline models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Linear attention outperform ... may not be the consensus\n>Linear Transformers are not the dominant ...\n\nThank you for the question. Our claim is based on [1] (https://arxiv.org/abs/2410.03159, cited at line 479), which suggests AR linear attention may outperform vanilla attention in TSF. We also support this by referencing linear attention's performance in other position-sensitive domains.\n\nWe will revise the text to include qualifiers (e.g., sometimes) for clarity. Additionally, we provide more results at [Link (Table 1)](https://anonymous.4open.science/r/SAMoVAR-Rebuttal-F83C) comparing linear and softmax attention, showing advantage.\n\nPrior works (DLinear, PatchTST ...) have noted overfitting in TSF Transformers. Softmax attention’s strong expressiveness can overfit noise. In contrast, linear attention, due to the linearity of query, key, and value, retains an AR form [1] and also introduces a dynamic VAR structure:\n\nAR form of attention [1]:\n\n$$\n\\mathbf{o}\\_t = \\sum\\_{i=1}^t \\mathbf{w}\\_{t,i} \\mathbf{v}\\_i \\ , \\mathbf{w}\\_{t,i} = \\mathbf{q}\\_t \\mathbf{k}\\_i^\\top \\in \\mathbb{R}^d\n$$\n\nVAR form of linear attention [This paper]:\n\n$$\n\\mathbf{o}\\_t = \\sum_{i=1}^t \\mathbf{k}\\_i \\mathbf{A}\\_{t,i} \\ , \\mathbf{A}\\_{t,i} = \\mathbf{q}\\_t^\\top \\mathbf{v}\\_i  \\in \\mathbb{R}^{d \\times d}\n$$\n\nThis allows for dual linear interpretability, with stronger regularization, better suited for channel- and token-wise linear mixing at each layer, suited for TSF data commonly modeled with AR/VAR structures.\n\n>Inconsistent with (Table 1) ... \n\nThis aligns with Table 1: LinTrans outperforms softmax-based models (iTransformer, PatchTST, EncFormer). The only exception is CATS, which is MLP-based and does not use softmax attention.\n\n>Some claims may be oversimplified: Replacing $\\sigma$ ...\n\nApologies—this should read “simplified to linear attention with a linear kernel.” This is a standard baseline in recent works like Gated Linear Attention and RetNet. We’ll clarify this in the revision.\n\n>Evidence that explicit VAR ... \"capability of modeling complex ...\"\n\nDue to space limits, please refer to our response to Reviewer RLuA for discussion on NLP data.\n\n> Incorrect assumptions: (1) ..., (2) ...\n\n(1) Apologies—we’ll clarify this. The scaling factor acts like a softmax temperature and doesn’t affect analysis.\n\n(2) Yes, all attention structures in our methodology use decoder-only attention with a causal mask. A comparison is provided below:\n\nEncoder-only (without causal mask):\n\n$$\n\\mathbf{O} = \\mathbf{Q} \\mathbf{K}^\\top \\mathbf{V} \\\\\n\\mathbf{o}\\_t = \\mathbf{q}\\_t \\sum_{i=1}^N \\mathbf{k}\\_i^\\top \\mathbf{v}\\_i\n$$\n\nDecoder-only (with causal mask, used in this paper):\n\n$$\n\\mathbf{O} = \\mathbf{M} \\odot (\\mathbf{Q} \\mathbf{K}^\\top) \\mathbf{V} \\\\\n\\mathbf{o}\\_t = \\mathbf{q}\\_t \\sum\\_{i=1}^t \\mathbf{k}\\_i^\\top \\mathbf{v}\\_i\n$$\n\nTimesteps are $\\{1, ..., t, ..., N\\}$. For encoder-only, $\\mathbf{q}_t$ sees all tokens; for decoder-only, it sees only up to step $t$. This notation is standard in linear attention literature.\n\n>Evaluate more baseline\n\nNote that LinTrans uses decoder-only attention, while iTransformer, PatchTST, and EncFormer use encoder-only softmax attention. Additional results at [Link (Table 1)](https://anonymous.4open.science/r/SAMoVAR-Rebuttal-F83C) compare with AR softmax and gated linear attention.\n\n>Performance ... more layers\n\nTable 2 shows that ~3 layers of SAMoVAR work well across most datasets, with other choices showing no major drop in performance. The $l$-th layer output aggregates results from $n^{\\text{path}}_{t,j,l}$ paths per key observation:\n\n$$\n\\mathbf{o}\\_t^{(l)} = \\sum\\_{\\text{All Paths}} \\mathbf{P}\\_{t,j,\\{i\\_1, \\cdots, i\\_{l-1}\\}}^{(l)} \\mathbf{k}\\_i^{(1)\\top} = \\sum\\_{\\text{All Paths}} \\mathbf{v}\\_{i\\_1}^{(l)\\top} [\\mathbf{q}\\_t^{(l)} \\,\\mathbf{v}\\_{i\\_2}^{(l-1)\\top} \\mathbf{q}\\_{i\\_1}^{(l-1)} \\;\\cdots\\; \\mathbf{v}\\_{j}^{(1)\\top}] \\mathbf{q}\\_{i_{l-1}}^{(1)} \\mathbf{k}\\_i^{(1)\\top}\n$$\n\nEach path is scaled by the bracketed scalar. Normalizing q and k keeps inner products close to <1, avoiding instability. The only risk of deeper models comes from the increase in path count ($n^{\\text{path}}_{t,j,l}$), which may cause overfitting of too many dynamics for longer lags.\n\n>More recent advancements ... TimesFM ...\n\nWe will add a Related Works section for recent TSF advances to support the use of AR Lin Attn in TSF.\n\n>Is there information leakage... \n\nOur AR/ARX tokenization adds an AR loss between non-overlapping PatchTST tokens. Each token covers the full horizon $L_P$, allowing us to follow the same training/testing setup as prior works, with only one added loss term. The decoder-only enables effective training, functionally similar to shorter-context data augmentation or varying-context multitask training.\n\nThis allows us to integrate AR attention without changing the experimental setup. Like previous works, we use one-step prediction covering $L_P$, avoiding iterative prediction and any risk of data leakage.", "claims_and_evidence": "The claims made in the paper are generally supported, for example, the authors provide a theoretical analysis of how linear attention can be interpreted as a VAR model. However, some claims are made based on empirical results from previous works. They claim that \"linear attention can outperform vanilla attention in time series forecasting tasks\". This may not be the consensus in the field. It is also inconsistent with the evaluations (Table 1), where linear-attention Transformer can not generally outperform the vanilla ones. Some claims may be oversimplified: \"Replacing $\\sigma$ with an identity simplifies it to linear attention,\" Which linear attention is referred to? This assumption makes the Attn() a linear transformation of X.  Some claims can be biased: \"Transformers are effective for modeling complex sequence relationships (e.g., in NLP), their architecture conflicts with VAR’s goal of explicitly representing lag-based dependencies.\" The author does not provide evidence that explicit VAR can help in terms of training models for time series rather than natural language. Therefore, it is better to explain the \"capability of modeling complex sequence relationships\" and why the alignment with explicit VAR is important.", "methods_and_evaluation_criteria": "The authors use standard benchmark datasets for time series forecasting, such as Weather, Solar, and ETT, which are widely adopted in the previous works.", "theoretical_claims": "The proof in Section 3 appears to be correct, but there are some incorrect assumptions: (1) Section 2.2: Equation (1) omits the scaling factor, (2) The recurrent form of Attention relies on the causal mask. However, I am not certain about the proposed architecture adopting the causal mask. It seems that Transformers in Table 1 are not decoder-only ones? Please correct me if I'm mistaken..", "experimental_designs_or_analyses": "* I recommend the author to evaluate more baseline models, including more counterpart linear-attention Transformers (currently 2), and decoder-only Transformers (currently 0).\n* How about the performance when using more Transformer layers? Is the training process of SAMoVAR stable?", "relation_to_broader_scientific_literature": "This paper focuses on refining Transformers for time series forecasting.", "essential_references_not_discussed": "The paper can benefit from discussing more recent advancements in autoregressive Transformers for time series forecasting, such as TimesFM [ICML 2024]. Whether the method proposed in this paper can improve the performance on similar models?", "other_strengths_and_weaknesses": "* Strength: The paper presents a theoretically proof to reveal the similarity between simplified linear Transformers with VAR models.\n* Weakness: The motivations of aligning linear Transformer with VAR are not well supported. To my best knowledge, linear Transformers are not the dominant choice in time series forecasting.", "other_comments_or_suggestions": "See Above.", "questions_for_authors": "Is there information leakage when the above method is used for training encoder-only models? Although the model is autoregressive supervised by the groundtruth in multi-step prediction, there is no groundtruth at each step during inference.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper first shows that single-layer linear attention module can naturally express a dynamic vector autoregressive model structure but standard multi-layer Transformer designs do not. They hypothesize that this mismatch, if resolved, could lead to a more performant class of neural networks for time series forecasting. To this end, they develop SAMoVAR (Structural Aligned Mixture of VAR), a novel Transformer variant that reorganizes the attention and MLP layers so that the computation across layers can be interpreted as a dynamic VAR model. The paper empirically tests the model on a standard suite of benchmarks in time series forecasting and shows that it achieves improved accuracy, interpretability, and computational efficiency. The discussions with the reviewers a] a comparison with TimeMixer (which was provided) b] an expansion on why SAMOVAR performs well c] a proof of how the multi-layer case of SAMOVAR can be seen as a VAR process and an analysis of robust path pruning. Overall, the above questions were responded to in the rebuttal and I encourage the authors to incorporate them into the revised manuscript. Given the novelty of the idea and the work's dive into understanding how to align time series based transformers with VAR models (as well as the latters in classical time-series forecasting problems) I lean towards acceptance.
Decision: Accept (poster)