ICML 2025

Linear Transformers as VAR Models: Aligning Autoregressive Attention
Mechanisms with Autoregressive Forecasting
Jiecheng Lu 1 Shihao Yang 1
Abstract
Autoregressive attention-based time series fore-
casting (TSF) has drawn increasing interest, with
mechanisms like linear attention sometimes out-
performing vanilla attention. However, deeper
Transformer architectures frequently misalign
with autoregressive objectives, obscuring the un-
derlying vector autoregressive (VAR) structure
embedded within linear attention and hindering
their ability to capture the data generative pro-
cesses in TSF. In this work, we first show that
a single linear attention layer can be interpreted
as a dynamic VAR structure. We then explain
that existing multi-layer Transformers have struc-
tural mismatches with the autoregressive forecast-
ing objective, which impair interpretability and
generalization ability. To address this, we show
that by rearranging the MLP, attention, and input-
output flow, multi-layer linear attention can also
be aligned as a VAR model. Then, we propose
Structural Aligned Mixture of VAR (SAMoVAR),
a linear Transformer variant that integrates in-
terpretable dynamic VAR weights for multivari-
ate TSF. By aligning the Transformer architec-
ture with autoregressive objectives, SAMoVAR
delivers improved performance, interpretability,
and computational efficiency, comparing to SOTA
TSF models. The code implementation is avail-
able at this link.
1. Introduction
In recent years, autoregressive decoder-only Transform-
ers have made significant strides (Vaswani, 2017; Radford,
2018), powering Large Language Models (LLMs) (Brown
et al., 2020; Touvron et al., 2023) capable of handling com-
plex sequential data. Their core mechanism, autoregressive
1Georgia Institute of Technology. Correspondence to: Shihao
Yang <shihao.yang@isye.gatech.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(...