MVAR: Visual Autoregressive Modeling with Scale and Spatial Markovian Conditioning
Authors: Jinhua Zhang
Wei Long
Minghao Han
Weiyi You
Shuhang Gu
Venue: ICLR 2026 Poster (2026)
Keywords: Image Generation;Next-Scale Prediction;Markovian Conditioning
Primary area: generative models
TL;DR: MVAR is a Markovian autoregressive model for image generation using scale and spatial Markovian conditioning.

Abstract: Essential to visual generation is efficient modeling of visual data priors. Conventional next-token prediction methods define the process as learning the conditional probability distribution of successive tokens. Recently, next-scale prediction methods redefine the process to learn the distribution over multi-scale representations, significantly reducing generation latency. However, these methods condition each scale on all previous scales and require each token to consider all preceding tokens, exhibiting scale and spatial redundancy. To better model the distribution by mitigating redundancy, we propose Markovian Visual AutoRegressive modeling (MVAR), a novel autoregressive framework that introduces scale and spatial Markov assumptions to reduce the complexity of conditional probability modeling. Specifically, we introduce a scale-Markov trajectory that only takes as input the features of adjacent preceding scale for next-scale prediction, enabling the adoption of a parallel training strategy that significantly reduces GPU memory consumption. Furthermore, we propose spatial-Markov attention, which restricts the attention of each token to a localized neighborhood of size (k) at corresponding positions on adjacent scales, rather than attending to every token across these scales, for the pursuit of reduced modeling complexity. Building on these improvements, we reduce the computational complexity of attention calculation from (\mathcal{O}(N^{2})) to (\mathcal{O}(N k)), enabling training with just eight NVIDIA RTX 4090 GPUs and eliminating the need for KV cache during inference. Extensive experiments on ImageNet demonstrate that MVAR achieves comparable or superior performance with both small model trained from scratch and large fine-tuned models, while reducing the average GPU memory footprint by  3.0x.

Published as a conference paper at ICLR 2026
MVAR: VISUAL AUTOREGRESSIVE MODELING WITH SCALE AND SPATIAL MARKOVIAN CONDITIONING
Jinhua Zhang Wei Long Minghao Han
Weiyi You
Shuhang Gu"
University of Electronic Science and Technology of China jinhua z-jh@gmai com - shuhangguu gma com
ABSTRACT
Essential to visual generation is efficient modeling of visual data priors. Conven- tional next-token prediction methods define the process learning the conditional probability distribution successive tokens. Recently, next-scale prediction meth- ods redefine the process to learn the distribution over multi-scale representations, significantly reducing generation latency. However, these methods condition each scale all previous scales and require each token consider all preceding kens, exhibiting scale and spatial redundancy. To better model the distribution by mitigating redundancy, we propose Markovian Visual RugoRegeesiive model- ing (MVAR), novel autoregressive framework that introduces scale and spatial Markov assumptions reduce the complexity of conditional probability modeling- Specifically, we introduce scale-Markov trajectory that only takes as input the features of adjacent preceding scale for next-scale prediction, enabling the adoption of parallel training strategy that significantly reduces GPU memory consumption. Furthermore, we propose spatial-Marke -Markov attention, which restricts the attention of each token to localized neighborhood of size k at corresponding positions on adjacent scales, rather than attending to every token across these scales, for the pursuit of reduced modeling complexity. Building we reduce the computational complexity of attention calculation from ((N2) to (N))), enabling training with just eight NVIDIA RTX 4090 GPUs and eliminat- ing the need for KV cache during inference. Extensive experiments on ImageNet demonstrate that MVAR achieves comparable superior performance with both small model trained from scratch and large fine-tuned models, while reducing the average GPU memory footprint by 3.0x.
these improvements,
[[ na. p(Pu:tt.... p(r)) 1[ p(r (ma)) plrutaa..it)) pfri) I[ pinl mltaai
603
in3
K
(a) Vanilla next-scale prediction
(b) Scale Markovian conditioning
(c) Scale&spatial Markovian con. (MVAR)
Figure I: Vanilla next-scale prediction vs. MVAR. Left: The vanilla next-scale method predicts each scale based on all previous scales and requires each token consider all preceding tokens. Middle: A next-scale variant predicts the next scale using only the adjacent scale, leveraging scale Markovian conditioning across scales. Right: MVAR further disentangles the spatial constraint and predicts each token using the neighboring positions of adjacent scales, based on spatial Markovian conditioning.
INTRODUCTION
Autoregressive (AR) models (Van Den Oord et al., 2016; Esser et al., 2021; Ramesh et al., 2021; Sun et al., 2024; Li et al., 2024b), inspired by the next-token prediction mechanisms of language models,
"Corresponding author
Project page: ttttps://nuanbabaa. thub. 10/MAAR

---

Published as a conference paper at ICLR 2026
(a) Attention Weights Acrces Scale
(b) Attention Weights across Neighborhood Size
Inforance
160
WAR-316
VGB 100
29023
M6AN-d15
MARR-22 MAR--22
130.00
MAATHE
241 257
19413
1852
15:33
SCAdo 0004
Heigroofhodd S2a
Figure 2:
Left: Accumulated attention weights across scales in VAR. At each scale, the highest attention weight occurs at the adjacent scale, indicating a predominant focus on neighboring scales rather than all preceding ones. Middle: Accumulated attention weights across varying neighborhood sizes between adjacent scales in VAR. As the neighborhood expands, attention weight growth becomes more gradual, indicating a spatially localized focus rather than engagement with all preceding tokens. Right: Memory usage during inference. MVAR reduces memory consumption by 4.2 x compared to VAR-d24 (9, 960MB 40,971MB at batch size 64) through scale and spatial Markovian conditioning, demonstrating significant improvement in memory consumption.
have emerged as powerful paradigm for modeling visual data priors in image generation tasks. Conventional AR models (Esser et al., 2021; Wang et al., 2024) typically adopt a raster-scan order for conditional probability modeling, which suffers from excessive decoding steps and results in substantial generation latency. To mitigate this latency, pioneering studies have explored modifying token representation patterns (Lee et al., 2022; Huang et al., 2025; Pang et al., 2024; Liu al., 2024) or redefining the decoding order (Chang et al., 2022; Yu et al., 2023; Li et al., 2024b). Recently, visual autoregressive modeling (VAR) (Tian et al., 2024) has shifted visual data priors modeling from next-token prediction to a next-scale prediction paradigm, which sequentially generates multi-scale manner. As the next-scale prediction paradigm better preserves the
image tokens in a coarse-to- intrinsic two-dimensions structure of images at each scale, VAR improves both the efficiency of the modeling process and the generation quality compared to conventional AR baselines.
Although vanilla next-scale prediction methods (Tian et al., 2024; Tang et al., 2024; Han al., 2024; Chen et al., 2024) dramatically accelerate the generation process, our key observations, as shown in Fig. 2 and Fig. 2 (b), reveal that approximate conditional independence, both across scales and between adjacent spatial locations, can be further leveraged to mitigate redundancy and improve the generation process. More specifically, vanilla next-scale prediction methods utilize tokens from all preceding scales to predict tokens of next-scale. However, as illustrated in Fig. 2 (a), the analysis of attention weights across scales reveals that each scale predominantly relies on its immediately preceding neighbor and is approximately independent of non-adjacent scales. Such cross-scale sesse conditional independence allows us to establish MarkoM next-scale prediction model for reducing GPU memory consumption and eliminating redundant computation. On the other hand, conventional next-scale prediction methods adopt the features of all preceding tokens as keys and values in attention operations (Vaswani, 2017). Nevertheless, represented in Fig. 2 (b), the analysis of accumulated attention weights across adjacent scales with different neighborhood sizes reveals that each token primarily relies on its immediate neighbors rather than all preceding tokens. Such approximate spatial conditional independence makes it possible to further simplify our scale-Markov model with spatial-Markov constraint.
Building upon the above observations, we introduce scale and spatial Markovian assumptions into the vanilla next-scale prediction model and propose Markovian Visual AutoRegressive (MVAR) model for effective modeling of visual data priors. Instead of relying on all preceding scales to predict the current scale, as shown in Fig. 1 (a) for conventional next-scale prediction method, we introduce Markov assumption which models the transition probabilities between adjacent scales, as illustrated in Fig. 1 (b). This design enables the learning of scale-Markov trajectory with parallel training strategy, significantly reducing GPU memory consumption and computational redundancy. In addition to the scale-Markon assumption, we further introduce a spatial-Markon constraint which confines each token's is receptive field to a neighborhood of size & at its corresponding position. As constraint further reduces the computational
depicted in Fig. (c), the proposed footprint of our model. Based on the above improvements, MVAR reduces the computational complexity of attention calculation from ( to ONK)), enables training on eight NVIDIA RTX 4090 GPUs, and eliminates the need for KV cache during inference. Extensive experiments on
2

---

Published as a conference paper at ICLR 2026
ImageNet x56x256 demonstrate the advantages of model. Compared to the vanilla VAR model, our proposed Markovian constraints not only reduce memory and computational footprints for visual modeling, but also improve modeling accuracy by focusing more important information.
Our main contributions are summarized as follows:
We propose a novel next-scale autoregressive framework with scale Markovian assumption to model conditional likelihood. It substantially reduces memory consumption during both training and inference, enabling MVAR training eight NVIDIA RTX 4090 GPUs.
Based on the learned scale-Markov transition probabilities, we introduce spatial-Marker attention mitigate spatial redundancy across adjacent scales, reducing the computational complexity of attention calculation from ((N2) to (NN).
We conduct comprehensive experiments on both small model trained from scratch and large fine-tuned models. Our method achieves comparable or superior performance to the vanilla next-scale prediction model while reducing memory consumption by 4.2x, as shown in Fig. (c).
RELATED WORK
2.1 GANS AND DIFFUSION IMAGE GENERATION
GANs and diffiusion image generation methods have been extensively investigated for image synthesis. In contrast to AR models, GANs (Goodfellow et al., 2020) learn mapping from noise distribution to the high-dimensional image space in a single pass via adversarial training, enabling efficient sampling. Pioneering efforts include the LAPGAN (Denton al., 2015), which progressively generates images from coarse to fine, and StyleGAN (Sauer et al., 2022), which enables controllable, high-fidelity facial synthesis via unsupervised learning. Further progress has been made by improving architectures (Xu et al., 2022). training strategies (Chen et al.. 2020), conditioning mechanisms (Peng Qi, 2019), and evaluation metries (Gu et al., 2020), advancing both theoretical understanding and practical applications. More recently, unlike the single-step mapping used in GANs, diffusion models decompose image synthesis into sequence of gradual denoising steps, thereby enabling stable training and improved sample quality. Early methods relied on pixel-space Markovian sampling (Ho et al., 2020), which incurred high computational cost. Subsequent works improved efficiency through non-Markovian samplers (e.g.. DDIM) (Song et al., 2020), ODE-based solvers (e.g. DPM-Solver) (Lu et al., 2022). distillation (Salimans Ho. 2022), latent-space sampling (e.g.. LDM) (Rombach et al., 2022b), and modified architectures (Peebles & Xie, 2023b). While GANs and diffusion models both yield high-quality images, GANs suffer from training instability, diffusion models offer stability but suffer from slow sampling, and autoregressive models provide strong causal interpretability and support more efficient sampling.
2.2 AUTOREGRESSIVE IMAGE GENERATION
Autoregressive image generation methods originate from sequence prediction tasks in natural lan- guage processing. By recursively modeling the conditional probability of each element given all previous ones, these models enable stepwise image synthesis. Early work such as PixeICNN (Van den Oord et al., 2016) flattened images into one-dimensional pixel sequences and employed gated con- volutional layers to capture causal conditional distributions for image synthesis. To overcome the computational bottleneck of next-pixel prediction. VQ-VAE (Van Den Oord et al., 2017) introduced a discrete representation space via a two-stage training process. Subsequently, modeling the next- token distribution in this discrete latent space became the mainstream approach. VQ-GAN (Esser et al., 2021) further enhanced the discrete representations for next-token prediction by integrating adversarial training and a perceptual loss. MaskGit (Chang et al., 2022) introduced a bidirectional transformer with random masking that generates multiple discrete tokens in parallel, significantly reducing the number of autoregressive inference steps. More recently, VAR (Tian et al., 2024) redefined the autoregressive modeling next-scale prediction process, mitigating the difficulty of causal modeling. Nonetheless, despite their promising performance, next-scale methods still suffer from excessive memory consumption and high computational costs during training and inference.
sequence-prediction
3

---

Published as a conference paper at ICLR 2026
vix vixd
K
by (c) Attention mechanism
(a) Cross-scale attention map at 110
(b) Sub adjacent-scale attention map
Attention patterns in the vanilla next-scale prediction method (e.g. VAR). Left: The vanilla next-scale prediction exhibits redundant inter-scale dependencies, each scale primarily conditions on its adjacent scale rather than all preceding ones. Middle: The attention operation between adjacent scales shows spatial redundancy, as each token mostly attends its immediate spatial neighbors. Right: An illustration of the receptive field in the attention mechanism observed in VAR, showing that attention restricted to local neighborhoods. Additional visual examples are provided in Appendix
Figure 3:
3 METHODOLOGY
3.1 PRELIMINARY: AUTOREGRESSIVE MODELING VIA NEXT-SCALE PREDICTION
The conventional next-scale (Tian et al., 2024; Han et al., 2024) autoregressive modeling introduced by VAR shifts image representation from the raster-scan order token map to multi-scale residual token map. enabling parallel prediction of all token maps at varying scales instead of individual tokens. Following predefined multi-scale token size sequence {(h,, w)) wr)] it first quantizes the feature map f E RxxwxC into I. multi-scale residual token maps R (r1, -2..... rL), each increasingly higher resolution h x wt, with rı matching the resolution of the original feature map w. Specifically, the residual token map f is formulated as:
F Q(Down (S Up(lookup(44 (, rı-1), ht, w), h1, wi)),
where (--) a quantizer, Down(-; (fy, wi)) and Up(-, (h, w)) denote down-sampling or up-sampling token map to the size (fu, wx) (h, w) respectively, and lookup( Z, r1) refers to taking the 1-th token map from the codebook Z. Then, a standard decoder-only Transformer uses the residual token maps R to model the autoregressive likelihood:
p2r,, T2,
II
I p(ra r1. [=1
where each autoregressive unit ry is the token map at scale I, containing by x wy tokens. All previous token maps from r1 rı-1, together with the corresponding positional embedding map scale I, serve as the prefill condition for predicting connections, where r is conditioned on previous token maps, the KV cache retain the features of all preceding scales to avoid redundant computation.
During inference, due to the dense inter-scale
3.2 KEY OBSERVATIONS
In this subsection, we present the motivation behind MVAR based on two key observations from the vanilla next-scale prediction paradigm (Tian et al., 2024), as revealed by the attention weight analysis in Fig. and the attention map patterns in Fig. 3. First, the attention weights of vanilla next-scale method reveal redundant inter-scale dependencies, leading to excessive GPU memory consumption and computational inefficiency. Second, dense attention operations between adjacent scales display spatial redundancy, where each token primarily attends its immediate spatial neighbors rather than all preceding tokens, presenting an opportunity to further reduce computational complexity.
Observation 1: Attention weights in vanilla next-scale prediction model exhibit scale redundancy. While vanilla next-scale method utilizes the entire sequence of token maps (ri, T2...., rt-1) as the conditioning prefix predict rı, the information encapsulated in the preceding scale token maps is
4

---

Published as a conference paper at ICLR 2026
Markovian Visual Autoregressive Modeling (MVAR) Layer
+
Decoeaaation
procsit
Mir airsall I (slraall 37% IaPrall
wiihh of soc
Ovenall fmmewwrrk proposed MVAR
Implanenittiion of Marrk-Markk Injeettry and spulial-Msskor
Figure 4: Overall Framework of MVAR. First, scale-Markov trajectory predicts using only its adjacent scale Tt- is discarding all earlier scales. This allows for parallel training across scales using a standard cross-entropy loss lossy. Second, a spatial- attention mechanism restricts attention local neighborhood of size k, reducing computational complexity from ((N2) ((Nk).
often redundant for consecutive scale prediction. As demonstrated in Fig. 2 (a) and Fig. 3 (a), the attention weights across scales reveal that queries at scale / pay negligible attention all preceding scales, but exhibit significant concentration on the immediately adjacent scale. This behavior aligns with the hierarchical design, where higher-resolution scales prioritize local refinements rather than reusing features from coarser levels. Consequently, by effectively alleviating such scale redundancy. we can reduce computational complexity and eliminate unnecessary conditioning on earlier scales when predicting next-scale tokens.
Observation 2: Attention weights in vanilla next-scale prediction model exhibit spatial redundancy. Conventional next-scale autoregressive approaches typically utilize Transformer (Radford et al., 2019) blocks to model transition probabilities. At the core of Transformer is the attention operation, which enhances features by computing weighted sums of token similarities. This operation is formally defined as:
Attention( Q', K. V SSoftMax (Q'(K)//V)) v
(3)
where Q' Nex. K' Nixd and V E Ne are linearly projected from the features of token map N₁ = hy x ii is the token number, and d is feature dimension. While attention operations theoretically enable global context modeling, finer scales which primarily capture textural details exhibit spatial redundancy, such as strong locality and 2D spatial organization, akin to convolutional operations (He et al., 2016). As depicted in Fig. 3 (b), the attention map between adjacent scales exhibits a diagonally dominant pattern, indicating that adjacent scales interact predominantly through spatially correlated regions rather than global dispersion. Such geometrically constrained weight distribution mirrors the local connectivity prior inherent in convolutional operations, illustrated in Fig. (b) and Fig. 3 (c). As neighborhood size increases, attention distributions become smoother, implying that attention operations across adjacent scales emphasize local textural details over global structural information (Chen et al., 2024; Guo et al., 2025; Kumbong et al., 2025). Therefore, such spatial redundancy be leveraged to significantly reduce the O(N?) time complexity of attention computation, particularly pronounced at finer scales (e.g., and r10), where attention operations account for up to 60% of the total computational cost.
In summary, the key insights from our observations are as follows: dencies in hierarchical architectures can be mitigated by introducing the scale-Markoviar assumption, which reduces memory usage and enhances computational efficiency. 2) Spatial redundancy in adjacent-scale interactions enables efficient spatial-Marker attention, which approximates localized 2D operations rather than global attention patterns.
1) Redundant inter-scale depen-
3.3 AUTOREGRESSIVE MODELING VIA SCALE AND SPATIAL MARKOVIAN CONDITIONING
To improve the modeling of visual data priors, we introduce MVAR, a novel next-scale autoregressive framework that applies the scale Markovian assumption to inter-scale dependencies and the spatial
5

---

Published as a conference paper at ICLR 2026
(a) VAR
(b)1 scale-Markov trajectory
MVAR
Figure 5: Schematic of causal masks for 256 256 image gemeration. VAR employs full causal mask to model p(r: r=))- In MVAR, scales 71 to rs use a diagonal-patten mask to model p(r: r7-1)- since the receptive field is smaller than the neighborhood size k. Scales rg and Tio are generated using custom CUDA kernels to model p(r (n(r)))).
Markovian assumption to attention across adjacent scales, as illustrated in Fig. 4. this framework, we restrict inter-scale dependencies to adjacent scales and incorporate localized attention operations into the dense computations between them, reducing memory consumption and computational cost.
Next-scale prediction with scale Markovian conditioning. To better mitigate the observed redun- dant dependencies across scales, we reformulate autoregressive modeling for next-scale prediction by shifting from the "previous scales-based" paradigm to an "adjacent scale-based" paradigm. Specifi- cally, the conditioning prefix for scale ry is restricted to immediate predecessor r/-1, discarding all other preceding scales. Formally, the autoregressive transition probability is defined as:
p(ri, F = p(ri) I me(r-1));
1=2
where ()(+) restricts attention to preceding tokens within a neighborhood of size k, as discussed below. The term p(r1) represents the probability distribution of the first scale token, generated from a class embedding referred to as the start token [s]. This pruning strategy aligns with the scale redundancy observed in Fig. 2 (a) and Fig. (a), allowing each scale to focus local refinements rather than redundantly reusing information from preceding scales.
Attention operation with spatial Markovian conditioning. As illustrated in Fig. (b), restricting inter-scale dependencies to adjacent scales reduces redundant interactions; however, dense compu- tations within these adjacent scales remain excessive (see analysis Fig. 2 (b) and attention map patterns in Fig. (b)). As shown in Eq. 3, conventional attention operates densely over all tokens in adjacent scales to produce outputs, resulting in quadratic time complexity O(N2). To mitigate the observed spatial redundancy, we introduce spatial-Markon attention (Hassani & Shi, 2022; Hassani al., 2023), where each query token attends exclusively to & nearest neighbors, as illustrated in Fig. 4. Specifically, given the query, key, and value matrices Q' RR Nixd K e IR Nxxd and v' R xd at scale Tı, we compute the local attention score S glxk for the i-th token within a neighborhood of size l as:
S = [K OT)
(K))
QK (7.
(5)
where (i) denotes the index of the j-th neighboring token. Similarly, the local value matrix V R defined as:
V (v (T))
(Vl V
(v' (L)( (T)
(6)
Since each token attends only to its neighboring tokens rather than all tokens as conventional next-scale paradigms, the computational complexity is reduced from ((N)) to O(Nk).. Finally, the spatial-Markov attention output SA E IR1xd for the i-th token with neighborhood size k is defined
as:
SA SoftMax ///Va)/

---

Published as a conference paper at ICLR 2026
Table Quantitative results on class-conditional ImageNet at resolution 256x256. + //1 indicate that lower higher values are better. We report results for representative generative models including generative adversarial networks (GANs), diffusion models (DMs), token-wise autoregressive models, and scale-wise autoregressive models. Metrics marked VAR evaluated using the official pretrained weights from its GitHub repository. "d16" denotes the depth of the Transformer in the autoregressive network.
are not reported in the original papers.
Type
Models
FID
ISt
Precision
Recallf
#Params
#Stops
BigGAN (Brock et al., 1809)
6,95
224.5
0.89
0.38
112M
GANS
GigaCiAN (Kang et al., 2023)
3.45
225.5
0.84
0.61
569M
StyleGAN-XI (Sauer et al., 2022)
2.30
0.78
0.53
ADM (Dhariwal Nichol, 2021)
10.94
CDM (Ho 2022)
4.88
158.7
8100
LDM-4-G (Rombach et al., 2022d)
3,60
247.7
400M
250
DMs
Simple-Diffusion (Hoogeboom et 2023)
2.44
256.3
2B
DiT-L/2 (Peebles Xie, 2023a)
5.02
167.2
0.75
0.57
458M
250
DIT-XL/2 (Poebles & Xie, 2023a)
2.27
278.2
0.83
0.57
675M
250
MaskGIT (Chang
2022)
6.18
182.1
0.80
0.51
227M
RCG (cond.) (Li et 2024a)
3.49
215.5
502M
20
VQVAE-21 (Razavi et al., 2019)
31.11
4.
0.36
0.57
13.5B
5120
VQGANt (Esser et al., 2021)
18.65
80.4
0.78
0.26
227M
256
VQGAN (Esser et al., 2021)
15.78
74.3
1.4B
256
Token-wise
VQGAN-re (Esser et al., 2021)
5,20
280.3
256
VITQQ (Yu al., 2021)
4.17
175.1
1.7B
1024
VITVQ-re (Yu et al., 2021)
3.04
227.4
1.7B
1024
DART-AR (Gu et al., 2025)
3.98
256.8
812M
RQTraan (leze et al,, 2022)
7.55
134.0)
3.88
68
RQTran, (Lee et al.,, 2022)
3.80
323.7
Scale-wise
VAR-116 (Tian al., 2024)
280.4
0.51
MVAR-d16
3.09
285.5
0.85
0.51
310M
Note that during training, by disentangling dependencies between non-adjacent scales such that each only attends to its immediate predecessor 1z- parallelized training can be adopted per-scale basis. Specifically, for 256 image generation, we maximize GPU utilization by training scales Ti through rы parallel. This is achieved through diagonal-patie -ptteer causal mask (see Fig. 5), which constrains each Ti only attends to its prefix r:-1- Subsequently, scales T9. and T10 are separately generated using custom CUDA kernels (Hassani & Shi, 2022) model p(ri ()(-)))) During inference, our method eliminates the need for KV cache, thereby further streamlining computational efficiency. More details about the mixed training strategy are provided in Appendix D.5.
3.4 ANALYSIS OF COMPUTATIONAL COMPLEXITY
Conventional next-scale paradigms incur high computational cost because the number of tokens at each scale equals the sum of tokens from all preceding scales, and dense computation is performed between adjacent scales. In the token generation process of conventional next-scale autoregressive models, the time complexity of producing N tokens is O(N2) (see Appendix A for more details), whereas our method reduces this to ((N)). Specifically, consider predefined sequence of multi- scale tokens {(4, Wi) For simplicity, let h = wt = /N, for 1 < S L, and denote vN = h = w. We further define VN, d'-1 for some constant a > 1, such that at-1 /N. At the 1-th autoregressive generation stage, the total number of tokens is Ni (t-1), Therefore, the time complexity for this stage is ((Nk).. Summing over all scales yields:
2L
a²N -
2(1-1) k:
k
O(NK).
(8)
a2
2
EXPERIMENTAL RESULTS
In this section, we present the implementation details, report the main results on the ImageNet 256 x 256 generation benchmark, and conduct ablation studies evaluate the contribution of each
7

---

Published as a conference paper at ICLR 2026
Table 2:
Pre-trained rs. Fine-tuning Comparison. Quantitative comparison between pre-trained VAR baselines and our fine-tuned variants (MVAR(). Metrics include inference time (seconds per batch), GFLOPs in attention blocks, KV cache footprint (MB), GPU memory usage (MB), training speed (seconds per iteration), and training memory consumption (MB). All experiments were conducted on NVIDIA RTX 4090 GPU with batch size 32. OOM denotes out of GPU memory.
Methods
Time GFLOPs, KV Cachel
Memory!
Train Speed|
Train Memory.
FID 1St
Precision Recallf
VAR-d16
43.61
5704M
10882M
0.99
34319M
0.84
MVAR-a16
0.27
35.44
3846M (2.8x)
0.61 (1.6x)
20676M
3.40 297.2
0.86
0.48
VAR-d20
0.52
81.52
8500M
16244M
1.35
48173M
2,95
302.6
0.83
0.56
MVAR-d20'
0,45
68.75
5432M (3.0x)
0.79 (1.7x)
2766SM
2.87 295.3
0.86
0.52
VAR-d24
0.81
136.63
12240M
23056M
OOM
2.33 312.9
0.82
0.59
MVAR-d24
0.71
118.25
7216M (3.2%)
0.91
38579M
2.23 300.1
0.85
0.56
Figure 6: Qualitative results of MVAR. Examples Cass-cconitionaa generation on ImageNet 256x256.
component in our proposed methodology. Additional experiments on other resolutions and datasets are provided Appendix D.
4.1 IMPLEMENTATION DETAILS
As shown in Tab. 1, we train MVAR-d model from scratch on eight NVIDIA RTX 4090 GPUs with total batch size of 448 for 300 epochs. For the results reported in Tab. 2, fine-tuning is performed with a different batch size for 80 epochs. Both models are optimized using the AdamW (Kingma & Ba, 2014) optimizer with 3 0.9, B2 0.95, weight decay of 0.05, and base learning rate of I x 10-4, For the predefined multi-scale token length sequence, unless otherwise specified, we adopt the same setting VAR, i.e.. 21, 2, 4, 5, 8, 10, 13, 16). for the 256x256 resolution. Following common practice, we evaluate Fréchet Inception Distance (FID) (Heusel et al., 2017), Inception Score (IS). Precision, and Recall, along with additional reference metries. More implementation details are provided in Appendix B.
4.2
MAIN RESULTS
Overall comparison. We evaluate the performance of MVAR against the original VAR models, conventional next-token prediction models, diffusion models, and GAN-based approaches on the ImageNet- 1K benchmark. As shown in Tab. 1, method consistently outperforms nearly all com- peting models, achieving notable improvements over the VAR baselines. Specifically, compared to VAR, method reduces the FID by 0.46, and increases the IS by 5.1, demonstrating superior overall
8

---

Published as a conference paper at ICLR 2026
71
12
T3
74
T5
Tg
TT
T9
T10
(a) vanilla next-scale prediction
Memory: 10882M. FID: 4.84, IS: 227.1. Pre: 0.85, Roc:0.43, GFLOPs: 43.61
2.8x Reduction
(b) *scale-Markov trajectory
Mernory: 4199M, FID: 4.35, IS: 240.6, Pre: 0.86, Rec:0.45, GFL.OPs: 37.84
Speedup
(c) -esptssl--aarkkeeeee Markov attention
Memory: 3846M,
Pre: 0.85, Rec:0.45, GFLOPss
Figure 7: Ablation study scale and spatial Markovian conditioning, Intermediate outputs at scales T1 to 710 illustrating the progressive emergence of semantic and spatial details, along with memory usage and evaluation metrics including FID, IS, Precision (Pre), Recall (Rec), and attention GFL.Ps. All experiments were conducted on an NVIDIA RTX 4090 with batch size 32.
performance. As illustrated in Fig. 6, MVAR produces high-quality images for class-conditional generation on ImageNet 256x256. Additional qualitative results can be found in Appendix F.
Pre-trained vs. Fine-tuning. To further demonstrate the scale and spatial redundancy observed in conventional next-scale prediction methods, we use pre-trained VAR models as baselines and then fine-tune them, achieving comparable performance with lower GPU memory consumption and computational costs. Specifically, shown in Tab. 2, the fine-tuned VAR model not only achieves significant performance improvements over the original model but also attains an average x.0x reduction in inference memory consumption. These results highlight that approach outperforms the next-scale paradigm, underscoring the benefits of scale and spatial Markovian conditioning.
4.3 ABLATION STUDY
To assess the effectiveness of the core components in our method, we conduct ablation studies focusing on two key aspects: scale Markovian conditioning across scales and spatial Markovian conditioning across adjacent locations. Due to limited computational resources, all results are reported using the 310M model trained under a shorter schedule 50 epochs.
Effects of scale and spatial Markovian conditioning. To assess the impact of key components MVAR, we design three model variants and compare their performance, as illustrated in Fig. Method (a) serves the bascline following the VAR setup. To reveal the redundancy of inter-scale dependencies in this setting, Method (b) uses only the immediate predecessor to model transition probabilities, rather than relying on all preceding non-adjacent scales. Method (c) further introduces - attention, refining the attention mechanism to more effectively capture local textural details. As shown in Fig. 7, even when the strong scale and spatial dependencies are removed, the progressive generation process remains analogous to the conventional next-scale prediction method. Moreover, the combined application of scale-Markov trajectory and spatial-Markoy attention achieves improvements across most metrics, notably reducing FID by 0.68, increasing IS by 13.7, and lowering GFLOPs of attention operations by 8.17. Additional qualitative results can be found in Appendix
Effects of varying the number of scales in the conditioning prefix. We conduct experiments to investigate the impact of different strategies for modeling inter-scale dependencies. Method (a) follows the original VAR setup and serves as the baseline. Methods (b) and (c) explore limiting the conditioning prefix to only the two or three most recent preceding scales. As shown in Tab. reducing redundant dependencies leads lower memory consumption and computational cost. Notably, Method (d). which conditions solely on the immediate predecessor, removes the need for KV cache, improves IS by 13.5, and reduces memory usage by 2.6x. These results suggest that minimizing inter-scale dependencies helps the model concentrate on essential generative patterns rather than redundant historical information, thereby enhancing generation quality.

---

Published as a conference paper at ICLR 2026
Table 3:
Ablation study on the number of preceding scales. Reducing the number of preceding
scales decreases memory usage and improves computational efficiency. "(a)" denotes the baseline
configuration following the VAR setup. All experiments used am RTX 4090 with batch size 32.
Method Number
Memory
KV Cachel Time (s)
FID
1ST
Precisionf
Recallf
10882M
5704M
0.34
43.61
4.84
227.1
0.85
0.43
(b)
3
9518M
3565M
0.32
41.54
4.86
220.3
0.86
0.43
(C)
2
9262M
2147M
0.31
40.15
5.01
208.8
0.84
0.45
(d)
4199M (2.6x)
0
0.29
37.84
4.35
240.6
0.86
0.45
Table Ablation study on neighborhood size in spatial-Markov attention. "_" denotes the baseline configuration following the VAR setup. Additional experiments on other resolutions and on CelebA and FFHQ datasets are provided in Appendix D.4.
k
FID
1S1
Precisiont
Recallf
GFLOPs
4.84
227.1
0.85
0.43
43.61
3x3
4.64
243.9
0.85
0.44
34.89
5x5
4.36
235.8
0.86
0.43
35.11
7x7
4.16
240.8
0.85
0.45
35.44
9x9
4.18
237.4
0.85
0.46
35.89
Effects of neighborhood size k in spatial-Markov attention. The neighborhood size & is crucial in ppatial-Markov attention, as it determines the size of the receptive field. larger neighborhood provides broader context, generally improving performance. We conduct experiments to examine the effect of varying & from x
x 9, as shown Tab. The model improves significantly when increases to 7 x 7, but further increases results in only marginal gains. This is because spatial-Markov attention effectively captures fine mne-grained details within a compact neighborhood, while larger values bring diminishing returns and redundant computation, as illustrated in Fig. 2 (b). To strike balance between performance and computational cost, we choose x 7 for our final models.
CONCLUSION
In this paper, we propose MVAR, novel next-scale prediction autoregressive framework that significantly reduces GPU memory usage during training and inference while maintaining superior performance. We introduce the scale-Markov trajectory, which models the autoregressive likelihood using only adjacent scales and discards non-adjacent ones. This design enables parallel training and substantially lowers GPU memory consumption. Furthermore, to address spatial redundancy in attention across adjacent scales, propose spatial-Markov attention, which restricts attention to a neighborhood of size k; reducing computational complexity of attention calculation from ((N²) ((N)). Extensive experiments demonstrate that our method achieves comparable better performance than existing approaches while significantly reducing average GPU memory usage.
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China (No. 62476051) and Sichuan Natural Science Foundation (No. 2024NSFTD0041),
REPRODUCIBILITY STATEMENT
We provide detailed hyperparameter settings in Section and Appendix B. To further facilitate reproducibility, we will release our implementation and trained model checkpoints, enabling the reported results be reproduced.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arxiv 2018. arXiv preprint
1809. 7
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative
image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11315-11325 2022. 2, 3,7
Runfa Chen, Wenbing Huang, Binghui Huang, Fuchun Sun, and Bin Fang. Reusing discriminators for encoding: Towards unsupervised image-to-image translation. In Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition, pp. 8168-8177, 2020. 3
Zigeng Chen, Xinyin Ma, Gongfan Fang, and Xinchao Wang. Collaborative decoding makes visual auto-regressive modeling efficient. arXiv preprint arXiv:2411.17887, 2024. 2,
2024.
Emily Denton, Soumith Chintala, Rob Fergus, al. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in neural information processing systems, 28, 2015. 3
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 07004,8794400 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883 2021. 2, 3,7
Ian Goodfellow, Jean Pouget-Abadit Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020. 3
Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, and Shuangfei Zhai. Dart: Denoising autoregressive transformer for scalable text-to- image generation. In ICLR, 2025. 7
Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Giqa: Generated image quality assessment. In Computer Pision-ECVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Pari XI 16, pp. 369-385. Springer, 2020.
Hang Guo, Yawei Li, Taolin Zhang. Jiangshan Wang, Tao Dai, Shu-Tao Xia, and Luca Benini. Fastvar: Linear visual autoregressive modeling via cached token pruning. arXiv preprint arXiv:2503.23367, 2025. 5
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint 4rXiv:2412.04431, 2024. 2, 4, 16
Ali Hassani and Humphrey Shi. Dilated neighborhood attention transformer. 2022. URL https:
//arxiv. org/ aaaaaaa22200 220922222222 15001. 6.7
Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6185-6194, June 2023. 6
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 8
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.12598 2022. 16
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1446840-8551 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans.
Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47)11-33, 2022.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In ICMI,, 2023. 7
Zhihao Huang, Xi Qiu, Yukuo Ma, Yifu Zhou, Chi Zhang, and Xuelong Li. Nfi: Autoregressive image generation with next-frequency prediction. arXiv preprint (rXiv:2503.07076, 2025. 2
Siyu Jiao, Gengwei Zhang, Yinlong Qian, Jiancheng Huang, Yao Zhao, Humphrey Shi, Lin Ma, Yunehao Wei, and Zequn Jic. Flexvar: Flexible visual autoregressive modeling without residual prediction. arXiv preprint arXiv:2502.203111 2025. 16
Minguk Kang, Jun-Yan Zhu, Richard Zhang. Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 10124-10134, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019. 18
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 8, 16
Hermann Kumbong, Xian Liu, Tsung- Yi Lin, Ming- Yu Liu, Xihui Liu, Ziwei Liu, Daniel Fu, Christopher Re, and David W Romero. Hmar: Efficient hierarchical masked auto-regressive image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2535-2544, 2025. 5
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, pp. 11523-11532, 2022. 7,7
Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised representation generation method. In The Tiihth- Annual Conference on Neural Information Processing Systems, 2024a. 7
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 18338 2024b. 1,2
Sifei Liu, Shalini De Mello, and Jan Kautz. Cosae: Learnable fourier series for image restoration. Advances in Neural Informaionn Processing Systems, 1022726666077,, 2024. 2
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV) December 2015. 18
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022. 3
Yatian Pang, Peng Jin, Shuo Yang, Bin Lin, Bin Zhu, Zhenyu Tang, Liuhan Chen, Francis EH Tay, Ser-Nam Lim, Harry Yang, et al. Next patch prediction for autoregressive visual generation. arXiv preprint 4rXiv:2412.13322 2024. 2
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 4195 4205,, 2023a. 7
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFE/CVF international conference on computer vision, pp. 4195 4195-4205, 2023b.
12

---

Published as a conference paper at ICLR 2026
Yuxin Peng and Jinwei Qi. Cm-gans: Cross-modal generative adversarial networks for common representation learning. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 15(1):1-24, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 5
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image -moi generation. In International conference on machine learning. pp. 8821-8831. Pmlr, 2021. 1
Ali Razavi, Aaron Van Den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with
vq-vae-2. 2019. 7
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pultern recognition, pp. 10684-10695, 2022b. 3
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 22Xi::2202.00112, 2022. 3
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. volume 0tt/22001000273 2022. URL httpp:://arxiii http://aaww..ooo///////a/// 00273.222. 3,
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020. 3
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.005225 2024. 1
Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. HART: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint 2ix:::4400.10112 2024. 2
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 2024. 2, 3, 4, 7, 15,
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016. 3
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pp. 1747-1756. PMLR, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurlPS, 30. 2017. 3
A Vaswani. Attention is all you need. NeurlPS, 2017.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. EMU3: Next-token prediction is all you need. arXiv preprini arXiv:2409, 200.1889998 2024. 2
Yanbo Xu, Yueqin Yin, Liming Jiang, Qianyi Wu, Chengyao Zheng, Chen Change Loy, Bo Dai, and Wayne Wu. Transeditor: Transformer-basee dual-space gan for highly controllable facial editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7683-7692, 2022.
13

---

Published as a conference paper at ICLR 2026
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. quatti-- image modeling with improved vegan. arXiv preprint arXiv:2110.0667,, 2021. 7
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (0459-10465) 2023. 2
14

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENT FOR APPENDIX
Time Complexity Analysis of Different Autoregressive Models
15
B Additional Implementation Details of MVAR
16
Limitations and Future Work
16
More Experimental Results
17
D.I
Experiments on ImageNet-512x512
17
D.2 Experiments with the 2B Model CelebA
17
D.3 Experiments on CelebA and FFHQ Datasets
18
D.4 Experiments Neighborhood Size CelebA and FFHQ
18
D.5 Experiments on Training Ratio the Mixed Training Strategy
18
The Use of Large Language Models (LLMs)
20
F More Qualitative Results
20
A TIME COMPLEXITY ANALYSIS OF DIFFERENT AUTOREGRESSIVE MODELS
In this section, we demonstrate that the time complexities of conventional next-token prediction methods and vanilla next-scale prediction methods (Tian et al., 2024) are O(N*) and O(N2), respectively. In contrast, MVAR achieves time complexity of O(Nk), established in Section 3.4.
Consider a standard next-token prediction using a self-attention transformer, where the total number of tokens is N h x At generation step i (1 < N), the model computes attention scores between the current token and all previous tokens. This operation necessitates O time due to the quadratic complexity of self-attention each step. Consequently, the overall time complexity given by
N
Σ 2
N(N +1)(2N + +1) O(N%).
On the other hand, consider a predefined multi-scale length {(h, wi)) For simplicity, let h N for all < L L, and denote vN = We further define /Ni a for some constant a > 1 such that d vN. For vanilla next-scale prediction, at the I-th scale generation stage, the total number of tokens can expressed as
21
ΣN, 1=1
a
1(-
(10)
(====
(a"
Thus, the time complexity for the 1-th scale generation process is L scale generation processes, obtain the overall time complexity:
By summing across all
21
a (N22 1) 2a 2a2(Na 1) + loga*) ~(N3). (I))
2
102
1)3
a
a²
15

---

Published as a conference paper at ICLR 2026
Table 5: Detailed hyper-parameters for MVAR models.
config
value
training hyper-parameteess
optimizer
AdamW (Kingma & Ba, 2014)
base learning rate
le-4
weight decay optimizer momentum hatch size
0.05
(0,9, 0.95)
448 (d16) / 192 (d20) 384 (d24) 300 (train) 80 (fine-tune)
total epochs
warmup epochs
60 (train) 2 (fine-tune)
precision
f
grad norm
2.0
dropout rate
0.1
class label dropout rate params
0.1
310M (d16) / 600M (d20) / 1.0B (d24)
embed dim
1024 (d16) 1280 (d20) 1536 (d24)
attention head
16 (d16) / 20 (d20) 24 (d24)
sampling hyper-parameteer
efg guidance scale (Ho & Salimans, 2022) top-k
2.7 (d16) 1.5 (d20) / 1.4 (d24)
1200 (d16) 900 (d20) / 900 (d24) 0.99 (d16) / 0.96 (d20) / 0.96 (d24)
top-p
For our MVAR method, at the I-th autoregressive generation stage, the total number of tokens is Ni 22(1-1), Therefore, the time complexity for this stage is (Nv)). Summing over all scales yields:
2² -
N2N -1 O(NK).
a 21 k k
(12)
a²
a²
B ADDITIONAL IMPLEMENTATION DETAILS OF MVAR
We list the detailed training and sampling hyper-parameters for MVAR models Table 5, For the predefined scale lengths inherited from VAR (Tian et al., 2024) (1, 3, 5, 6, 10, 13, 16), we parallelize training across scales F1-T's maximize GPU utilization, employing diagonal-pattern causal masks. This setup yields a uniform total token length of 255, which closely matches the token lengths of the final two scales (13x 13 and 16x 16). In addition, we describe the procedures used to visualize Fig. and Fig. 3 for completeness. For Fig. 2(a), first use VAR to generate the ImageNet validation set, producing one image for each of the 1,000 classes. We then compute the mean attention weights across all heads and layers at scale with respect to the preceding scales T2,72..... 71-1- For Fig. 2(b), we aggregate the attention weights from p(r r1-1) at each position (i,5) and average them across different neighborhood sizes k. For Fig. 3, we again VAR to generate the ImageNet validation set and visualize the mean attention weights across all layers and heads the final scale 1
C LIMITATIONS AND FUTURE WORK
In this work, we focus on the learning paradigm within the autoregressive transformer framework while retaining the original VQ-VAE architecture from VAR. We anticipate that integrating more advanced image tokenizers (Han et al., 2024; Jiao et al., 2025) could further enhance autoregressive visual generation performance. Future work may explore high-resolution image generation and video synthesis, which remain prohibitively expensive for traditional methods due to KV cache memory demands. Unlike conventional autoregressive models that rely on KV cache, our approach is inherently KV cache-free, reducing both memory consumption and computational complexity.
16

---

Published as a conference paper at ICLR 2026
Table 6: Quantitative results I1ageNet-512x511. All experiments evaluated with batch size of 64 during inference and a batch size of during training, using an RTX 4090 GPU.
Method
Memory4
Cachel
Timel
GFLOPs.
Train speed]
Train Memory, FID4
1St
Procisiont
Recallt
VAR-d16
1.26s
41944M
8.03
MVAR d16 150S0M (2.9k)
1.69%
116.37
0.56x (2.3%)
19510M (2.1x)
7.55
187.9
0.84
0.26
VAR (Memory 43826M, FID 8.03)
2.9x
MVAR (Memory 15090M, HID-7.53)
1500ory-15000M,
Figure 8: Qualitative comparison of VAR-d16 and MVAR-d16 on the ImageNet 512x512.
D MORE EXPERIMENTAL RESULTS
D.1 EXPERIMENTS ON IMAGENETT-1125512
To further evaluate the effectiveness of MVAR. we conducted additional experiments on ImageNet- 512x512. Due to limited computational resources and time constraints (VAR was originally trained using a 2.3B model on 256 100 GPUs with a batch size 768), we were unable to exactly replicate the experimental settings in the original VAR. Nevertheless, ensure fair comparison, the same training configurations for both VAR and MVAR, training each for 100 epochs with batch size of 24. For the predefined scale lengths inherited from VAR (1, 2, 4, 6, 13, 18, 24, 32), we parallelized training across scales ri-Ta to maximize GPU utilization, employing the same diagonal-pattem causal masks as described above. Other settings followed the configuration of MVAR-d20, as listed in Tab. 5. As shown in Tab. 6 and Fig. 8, MVAR consistently outperforms VAR in generation quality. Moreover, MVAR achieves substantial reductions in memory consumption and training/inference time, demonstrating both its effectiveness and efficiency at higher resolutions.
adopted
D.2 EXPERIMENTS WITH THE 2B MODEL ON CELEBA
In Section 4.2, we reported the performance of MVAR on ImageNet-2555 256 using models ranging from 310M to IB parameters. To further evaluate the scalability of MVAR, conducted experiments
17

---

Published as a conference paper at ICLR 2026
with 2B 1B-prammetrr model. Given limited computational resources (the original VAR was trained on 256 A100 GPUs with a batch size of 1024 for approximately four days), we trained both MVAR CelebA-256 256 (Liu et al., 2015), using a batch size of 24
and VAR from scratch for 50 epochs distributed across eight RTX 4090 GPUs. The CelebA dataset contains 30,000 images, from which randomly selected 28,000 for training. As shown in Tab. 7, we report training and inference results on CelebA. MVAR consistently outperforms VAR throughout training. Notably, MVAR achieves 3.1x reduction memory consumption while also enabling faster training and inference.
Table 7: Quantitative results of MVAR and VAR models (2B) on CelebA-256x Experiments performed on an RTX 4090 GPU, using a batch size of for inference and or 12 for training.
Method
Memory4
KV Cachel,
Time.
GFLOPs
Train Speed]
Train Memory.
FID
VAR-d30
46592M
40108M
2.94s
258.60
0.56s
48478M / OOM
2.65
MVAR-d30
14804M (3.1x)
2.35s
229.89
0.36s
47860M 481
1.33
D.3 EXPERIMENTS ON CELEBA AND FFHQ DATASETS
To further evaluate the generalization ability of MVAR on additional datasets, we conducted ex- periments CelebA (Liu et al., 2015) and FFHQ (Karras al., 2019). Specifically, MVAR was trained on CelebA at resolutions of 256 and 512x512, using the same dataset split strategy as in Section D.2. For 2556.256x256, we randomly selected 66,000 images for training and used the remaining images for validation. To ensure fairness and comparability, we adopted identical training settings for VAR and MVAR. The batch sizes were 24 for CelebA-256x 256, 6 for C-lebA-512x55 and 48 for FFHQ-256 256, with all models trained for 50 epochs. As shown in Tab. 8, MVAR consistently outperforms VAR across both datasets, indicating its strong generalization ability across different data distributions.
Table 8: Quantitative results of MVAR and VAR CelebA and FFHQ The notation "(k)" denotes the results obtained the k-th epoch.
Dataset
Method
HD (10),
HI)(22))
FID (30).
HD (40).
HID (50)
VAR-d16
4.71
4.67
3.79
3.31
3.17
CelebA
MVAR-d16
4.32
4.15
3.39
3.05
2.94
VAR-116
3.93
2.92
2.75
2.71
FFHQ
MVAR-d16
3.33
2.92
2.70
2.52
2.42
VAR-d16
5.40
4.88
4.67
4.53
4.45
CelebA-512
MVAR-d16
3.99
3.83
D.4 EXPERIMENTS ON NEIGHBORHOOD SIZE ON CELEBA AND FFHQ
In Section 4.3, we examined the effect of the neighborhood size A: in Spatial-Markov attention on ImageNet- 256 x 256, observing diminishing returns but without conducting a comprehensive robustness analysis (e.g., across datasets image resolutions). Here, extend this study by evaluat- ing the impact neighborhood size on CelebA and FFHQ at different resolutions. Specifically, experiments were performed on CelebA at 256x256 and 512x512, and FFHQ at 256x256 using the same settings as in Tab, 8. As shown in Tab. 9, larger neighborhoods (e.g., 9 x 9) increase computational cost but provide diminishing returns and may even degrade performance. These trends are consistent across datasets and resolutions, aligning with our observations on ImageNet. Overall, the findings suggest that model performance is not highly sensitive to the choice of k. thereby mitigating concerns regarding hyperparameter sensitivity.
D.5 EXPERIMENTS ON TRAINING RATIO IN THE MIXED TRAINING STRATEGY
In Section 3, we introduced the mixed training strategy designed to reduce memory consumption. Due to the strict cross-scale dependency in VAR, where predicting the current scale requires all preceding scales, it is necessary to feed r; through T (a total token length of 680) simultaneously with causal attention mask (see Fig. 5(a)) model p(ri rer). In contrast, our approach conditions
18

---

Published as a conference paper at ICLR 2026
Table 9: Ablation study neighborhood size CelebA 256 and 512x512 resolutions, and
FFHQ at 256x 256 resolution. notation "(k)" denotes the results obtained at the k--hh epoch.
k
Dataset
FID (10)
FID (20)
FID
FID
FID
GFLOPs]
CelehA
4.67
3.79
3.31
3.17
43.61
FFHQ
3.93
3.44
2,92
2.75
2.71
43.61
ColebA-512
5.40
4.88
4.67
4.53
4.45
219.88
CelebA
4.47
4.26
3.42
3.23
35.11
5x5
FFHQ
3.53
3.18
2.86
2.68
2.46
35.11
ColebA-512
5.15
4.76
4.12
4.06
3.94
114.72
CelebA
4.32
35
7x7
FFHQ
3.33
2,92
2.70
35.44
CelehA-511
5.10
4.53
3.99
3.95
3.83
116.37
CelebA
4.24
3.52
3.21
3.04
35.89
9x9
FFHQ
2,99
2,91
2.54
2.44
35.89
CelehA-511
5.13
4.89
4.72
4.35
3.94
118.56
VAR (concate)
WVAR (concate)
+ 4:1.1
+ 6:1
MVAR (8:1:1)
10:111
12:1:1
- 14.1.1
IS910K
FIDG11KK
100
15
40
Procision11K
Recalig10K
03
Pee sam0 D0
2.2
PNcual
0.43
Epoch
Esoch
Figure 9: Ablation study the training ratio for mixed training at 256x256 generation resolution. We compare different ratios for training 71 -ss 79, and 710- Results show that large training ratios (e.g.. 8:1:1) maintain generation quality while substantially reducing GPU memory requirements, demonstrating the robustness and efficiency of the proposed mixed training strategy. The notation "concate" denotes the concatenated training scheme in VAR, "k:1:1" denotes that, within every k++2 training iterations, the first k iterations train rı Ts, followed by iteration training r9 and iteration training F'0.
only on the immediately preceding scale, which enables a parallel training scheme. Specifically, for 256x256 image generation, we train scales i'1 through r' in parallel using liagonal-patteri causal masks (see Fig. 5(c)) that ensure each attends solely to its prefix ri-1- For scales and r10- which together account for 60% all tokens, we train them separately using custom CUDA kernels to model p(r: (T)(r.-)) In fact, MVAR can also be trained using VAR's 's concatenated training scheme by replacing the causal mask Fig. 5(a) with that in Fig. 5(c). Here, we investigate how the training ratio among r1 Ts, T9, and T10 influences the mixed training strategy. We evaluate the MVAR-a R-112 model on ImageNet at a resolution of 256 256. For faireess, we adopt the same training settings VAR-d12 2: all models are trained for 50 epochs with batch size of 64. As shown in Fig.
19

---

Published as a conference paper at ICLR 2026
MVAR consistently outperforms VAR under both the mixed and concatenated training strategies. Moreover, mixed training with larger ratios (c.g., 8:1: 1) achieves generation quality comparable to the concatenated training scheme while requiring significantly less GPU memory than VAR. Overall, these results indicate that model performance is not highly sensitive to the choice of larger training ratios, alleviating concerns regarding performance instability under the mixed training strategy.
E THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (LLMs) aid in polishing the writing. Specifically, LLMs were employed to improve grammar, clarity, and readability of the manuscript. No part of the research ideation, methodological design, experimental analysis relied on LLMs.
F MORE QUALITATIVE RESULTS
In Fig. 10, we present generation results of MVAR model ImageNet at 256x256 resolution. MVAR produces high-quality images with strong diversity and fidelity.
In Fig. 11, we compare the performance of VAR with scale and spatial Markovian constants. Our MVAR demonstrates that, even when strong scale and spatial dependencies removed, it achieves superior results while requiring less GPU memory.
In Fig. 12. Fig. 13, and Fig. 14, we compare the performance of VAR and MVAR across models of 3 310M. 600M, and IB parameters. Our MVAR not only achieves comparable or superior results but also reduces GPU memory consumption by approximately threefold.
In Fig. 15, we show the attention distributions of VAR together with intermediate outputs at different scales. As the scale increases, the model progressively filters out irrelevant information and concentrates on critical regions adjacent scales.
In Fig. 16. Fig. 17, and Fig. 18, we evaluate VAR and MVAR on image in-painting, out- painting, and class-conditional editing tasks. MVAR attains zero-shot performance on par with VAR, while simultaneously reducing GPU memory consumption.
20

---

Published as a conference paper at ICLR 2026
Figure 10: Qualitative results. Examples of class-condittonal generation on ImageNet x56x256.
21

---

Published as a conference paper at ICLR 2026
(u) vanilla next-scale prediction Memory: 10882M, FID: 4.84, IS: 227.1, Pre: 0.85, Rec:0.43, GFLOPs: 4
2.6x Reduction
1.2x Speedup
trujectory
Memory: 4199M, FID: 240..,, Rec:0.45, GFLOOss: 37.84
2.8x Reduction
1.3x
Speedup
(e) +spatialMMarkov attention
Memory: 3846M, FID: 4.16. IS: 240.8, Pre: 0.85, Rec:0.45. GFLOPs: 35.44
Figure 11: More qualitative results scale and spatial Markovian conditioning. MVAR achieves superior visual quality and requires less GPU memory, even when strong dependencies are removed.
22

---

Published as a conference paper at ICLR 2026
VAR-d16 (Memory- 10882M, 10800,,-10882M, FID-3.55)
2.8x
MVAR-dIó (Memory-3846M, FID=3.09)
Figure 12: Qualitative comparison of VAR-d16 and MVAR-d16 the ImageNet 256
23

---

Published as a conference paper at ICLR 2026
VAR-d20 (Memory=16244M, FID=2.95)
3.0x
MVAR-d20 (Memory-5432M, FID=2.87)
Figure 13: Qualitative comparison of VAR-d20 and MVAR-d20 the ImageNet 256x256.
24

---

Published as a conference paper at ICLR 2026
VAR-d24 (Memory=230566, FID=2.33)
3.2x
MVAR-d24 (Memory=7216M,, FID-2.23)
Figure 14: Qualitative comparison of VAR-d24 and MVAR-d24 the ImageNet 256x256.
25

---

Published as a conference paper at ICLR 2026
71
T2
T3
74
75
76
F7
788
f9
710
layer 10
layer 12
layer 14
Cross-scale attention map at P10
T4
78
75
T9
T6
710
VAR-d16 Different adjacent-scale attention map
T2
73
75
T6
T7
79
7'0
layer 10
layer 12
layer 14
Cross-scale attention map at 7'10
T4
7'3
7'
76
T10
VAR-d20 Different adjacent-scale attention map
P1
72
73
74
75
F'6
f7
F's
T9
T10
layer 10
layer 12
layer 14
Cross-scale attention map at Tin
T4
78
7's
T'9
7'6
VAR-d24 Different adjacent-scale attention map
Figure 15: Attention patterns in vanilla next-scale prediction (e.g., VAR). Vanilla next-scale prediction exhibits redundant inter-scale dependencies and spatial redundancy.
26

---

Published as a conference paper at ICLR 2026
VAR-dIó (memory 10882M, FID=3.55)
MVAR-d16 (memory- 3846M, 3.09).00)
VAR-d20 mncmorr-11244M, FID-2.95)
3.0x
MVAR-d20 Hemmory-5432MM FID-2.87)
VAR-d24 (memory=223)) 23056M FID-2.33)
3,2x
MVAR-d24 (memmry-7216M., 7216M, FID-2.23)
original
generated
Figure 16: Zero-shot evaluation in in-painting. The results show that MVAR generalize to novel downstream tasks without special design and finetuning. Zoom in for a better view.
27

---

Published as a conference paper at ICLR 2026
VAR-d16 (memory-10882M, FIID-3.55)
2.8x
MVAR-dI6 (memory 3846M, FID-3.09)
VAR-d20 (memory 16244M, FID-2.95)
3.0x
MVAR-d20 (memory 5432M, FID-2.87)
VAR-d24 (memory-22056M, 23056M, FID-2.33)
3.2x
MVAR-d24 (memory-7216MM 7216M, FID-2.23)
original
generated
Figure 17: Zero-shot evaluation in out-painting. The results show that MVAR can generalize to novel downstream tasks without special design and finetuning. Zoom in for a better view.
28

---

Published as a conference paper at ICLR 2026
VAR-dIó (memory-108822,, FID=3.55)
MVAR-d16 (memory-33846M, FID>3309)
VAR-d20 (m2mory-16244M, FID-22
3.0x
MVAR-d20 4memory-5432M) FID-2.87)
VAR-d24 (memory-2230)) 23056M, FID=2.33)
3,2x
MVAR-d24 (memory- 7216M, FID-2.23)
original
generated
Figure 18: Zero-shot evaluation in cass-conditional editing. The results show that VAR can generalize to novel downstream tasks without special design and finetuning. Zoom in for a better view.
29


Reviews:
[{"review_id": "j0BX93KDmh", "reviewer": "Reviewer_u1aY", "summary": "This paper introduces MVAR, which exploits scale and spatial redundancy based upon Visual Autoregressive Modeling (VAR). Specifically, MVAR only uses the preceding scale for next-scale prediction, and restricts the attention map of each token to a localized neighborhood. These two changes significantly reduces the computational complexity from O(N^2) to O(NK), and eliminate the need for KV cache during inference.", "questions": "1. Based on Table 1, the introduction of markovian scale prediction and localized attention leads to even better performance (FID 3.09 vs 3.55, IS 285.5 vs 280.4)? Would the authors care to explain thsi phenomenon?\n\n2. And how much performance gain does scale markovian conditioning along brings?\n\n3. Why is it called \"spatial Markovian conditioning\"? Localized attention is a common operation, dating back to the days of Swin-Transformer. I do not see any markovian part in localized attention.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is overall well written and rather easy to follow. The figures look nice and captures the core idea in a glimpse.\n\n2. The scale and spatial redundancy in original VAR work makes intuitive sense to me, and the proposed solution is both straightforward and effective as shown in experiments.\n\n3. MVAR shows both performance advantages over vanilla VAR, as well as reduced memory footprint and accelerated inference speed.", "weaknesses": "1. Only model size of 300M is studied in this paper. It is unclear if MVAR shows good scalability or not. The authors are encouraged to show the scaling trend following VAR (up to 2B model).\n\n2. Beyonds ImageNet unconditional generation, The authors are encouraged to try the image in-painting and out-painting task and class-conditional image editing task as in the zero-shot setup in VAR paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7xrdJEh4aX", "reviewer": "Reviewer_3S78", "summary": "This paper proposes Markovian Visual AutoRegressive Modeling (MVAR), an efficient framework for visual autoregressive generation. The method introduces Markovian assumptions in both scale and spatial dimensions: each scale depends only on its immediately preceding scale, and each token attends only to a local neighborhood. This design effectively removes redundant dependencies in next-scale prediction and reduces attention complexity from $O(N^2)$ to $O(Nk)$. Experiments on ImageNet show that MVAR achieves similar or slightly better generation quality compared to vanilla VAR while using significantly less memory and allowing parallel training.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation is clear and well supported by empirical observations of redundancy in next-scale prediction.\n- The Markovian formulation is conceptually simple yet brings strong computational benefits.\n- The method achieves 3× memory reduction without degrading generation quality.\n- Results on ImageNet demonstrate good efficiency–performance trade-offs, making the approach practical for large-scale settings.\n- The paper is overall well written and easy to follow.", "weaknesses": "- The experimental comparison is limited. The paper does not include results against related methods such as Randomized Autoregressive Visual Generation (RAVG, 2024), which also targets efficiency improvement in visual AR models.\n- Table 1 reports results compared with VAR-d16, but it is unclear whether the MVAR model also uses 16 decoder layers. The discrepancy between Table 1 and Table 2 results (different FID/IS scores) suggests inconsistent model settings.\n- Several experiments in the appendix only report FID without IS or precision/recall, making the evaluation incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G33Sk2NSM9", "reviewer": "Reviewer_x3u9", "summary": "This paper proposes a novel visual autoregressive framework, MVAR, which introduces scale-Markov (depending only on adjacent scales) and spatial-Markov (focusing only on a local neighborhood) assumptions. This approach addresses the significant computational and memory redundancy in existing \"next-scale prediction\" models, reducing attention complexity from $O(N^2)$ to $O(Nk)$, cutting down GPU memory usage, and enabling an inference process that is entirely free of a KV cache.", "questions": "1. The ablation study (Table 3) shows that method (d) achieves a better FID than using all prior scales. Does this imply that the original VAR design (relying on all scales) is not just redundant, but potentially a **harmful inductive bias**? Does it force the model to process irrelevant historical information, thereby interfering with the generation?\n2. MVAR uses a fixed $k \\times k$ neighborhood. Have you considered using dynamic or deformable neighborhoods? For example, using a larger $k$ in smooth areas of an image and a smaller $k$ in complex, textured regions?\n3. Appendix B mentions that the causal mask is disabled for the last two scales. Does this mean that within these scales, token generation is parallel (i.e., non-autoregressive)? If so, does this contradict the model's definition as an \"autoregressive\" framework? Please clarify how $p(r_l | \\eta_k(r_{l-1}))$ is modeled for these final two scales.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. The paper accurately identifies and solves a critical bottleneck in existing VAR models related to memory and computation (especially the KV cache). This is a practical problem that has hindered scaling these models to larger sizes and higher resolutions.\n2. MVAR's two core components (scale-Markov trajectory and spatial-Markov attention) are conceptually simple but highly effective. The \"KV-cache-free\" and \"parallel training\" properties drastically reduce the training cost.", "weaknesses": "1. While the scale-Markov assumption proves effective in the current setup, it might be an approximation. In more complex scenarios requiring strong long-range dependencies (e.g., images with complex global structures or multi-object interactions), discarding all information from $r_1$ to $r_{l-2}$ could become a performance bottleneck.\n2. The paper mentions that training for $r_1$ to $r_8$ is parallel, but $r_9$ and $r_{10}$ (which account for 60% of the tokens) seem to be handled separately using single-scale conditional modeling. The complexities of this mixed training strategy and its impact on final performance consistency are not sufficiently discussed in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EXX12sJD5F", "reviewer": "Reviewer_TLfe", "summary": "This paper proposes Markovian Visual AutoRegressive modeling (MVAR). Unlike conventional next-scale prediction methods that condition each scale on all previous scales and require each token to attend to all preceding tokens, MVAR introduces scale and spatial Markovian assumptions to reduce redundancy and computational complexity. Specifically, the model only conditions on adjacent scales (scale-Markov) and restricts attention to local neighborhoods (spatial-Markov), significantly lowering GPU memory usage. Experiments on ImageNet demonstrate that MVAR achieves comparable or superior performance to existing methods while reducing memory consumption by up to 3-4$\\times$.", "questions": "- How was the visualization in Figure 2 obtained? Which dataset and how many samples were used?\n\n- Have the authors designed CUDA kernels for the spatial Markov attention? Does the current implementation affect inference efficiency?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This work introduces Markovian properties into VAR generation process, making the proposed MVAR achieves more efficient training and eliminates the need for a KV cache.\n\n- The paper is well-written and easy to follow.", "weaknesses": "- In the proposed spatial Markov property, it seems that the authors restrict each pixel $i$ to interact only with its neighboring pixels within the same scale. Although the term “Markov” is used, performing attention within local neighborhoods has already been extensively explored in previous works.\n\n- In the visualization of attention maps in Figure 3, the behavior of the attention maps produced by this paper appears quite different from those of prior works on VAR like FastVAR[1]. In previous work, the visualized attention maps exhibit strong cross-scale interactions in their attention visualizations. I would like to know how the authors generated Figure 3.\n\n- In Figure 7(b), the attention mask may be incorrect, the current figure seems to consider only intra-scale token interactions while ignoring tokens from the previous scale.\n\n- The paper lacks comparisons with other recent AR-based approaches, such as MAR[2]. Including these comparisons would better situate the proposed MVAR within the broader context of image generation methods.\n\n> [1]FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning. ICCV25\n\n> [2]Autoregressive Image Generation without Vector Quantization. NIPS24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)