FreqExit: Enabling Early-Exit Inference for Visual Autoregressive Models via Frequency-Aware Guidance
Authors: Ying Li
chengfei lv
Huan Wang
Venue: NeurIPS 2025 poster ()
Keywords: Visual Autoregressive Models
Dynamic Inference
Early Exit
Frequency-Gated Supervision
Primary area: deep_learning
TL;DR: We enable dynamic inference in next-scale autoregressive generation by restructuring intermediate representations with frequency-aware supervision.

Abstract: Visual AutoRegressive (VAR) modeling employs a next-scale decoding paradigm that progresses from coarse structures to fine details. While enhancing fidelity and scalability, this approach challenges two fundamental assumptions of conventional dynamic inference: semantic stability (intermediate outputs approximating final results) and monotonic locality (smooth representation evolution across layers), which renders existing dynamic inference methods ineffective for VAR models. 

To address this challenge, we propose FreqExit, a unified training framework that enables dynamic inference in VAR without altering its architecture or compromising output quality. FreqExit is based on a key insight: high-frequency details are crucial for perceptual quality and tend to emerge only in later decoding stages. Leveraging this insight, we design targeted mechanisms that guide the model to learn more effectively through frequency-aware supervision. The proposed framework consists of three components: (1) a curriculum-based supervision strategy with progressive layer dropout and early exit loss; (2) a wavelet-domain high-frequency consistency loss that aligns spectral content across different generation steps; and (3) a lightweight self-supervised frequency-gated module that guides adaptive learning of both structural and detailed spectral components.
On ImageNet 256×256, FreqExit achieves up to 2× speedup with only minor degradation, and delivers 1.3× acceleration without perceptible quality loss. This enables runtime-adaptive acceleration within a unified model, offering a favorable trade-off between efficiency and fidelity for for practical and flexible deployment.

Exi: Enabling Early-Exit Inference for Visual Autoregressive Models via Frequency-Aware Guidance
Ying Li!
Chengfei Lv²
Huan Wang!*
Westlake University
Alibaba Group
CALM
Original Var
CALM
1.00 SpeeSup
CoDe
LayeSkip
FID
Co
FiogEail
Exit @ Layer 12
Layer
Speedup
Freqliiit
Layer
R-d200
Speedup
1.0
1.6
1.8
2.0
Speedup
Figure 1: We introduce Exiq dynamic inference strategy for next-scale visual autoregressive generation with proposed frequency-avee aware guidance, bridging the gap between heep-wise generation and early-exit-basee acceleration. achieves up to 2x speedup with negligible quality degradation, offering superior trade-off between efficiency and fidelity compared to prior baselines.
Abstract
Visual AutoRegressive (VAR) modeling employs next-scale decoding paradigm that progresses from coarse structures to fine details. While enhancing fidelity and scalability, this approach challenges two fundamental assumptions of conventional dynamic inference: semantic stability (intermediate outputs approximating final results) and monotonic locality (smooth representation evolution across layers), which renders existing dynamic inference methods ineffective for VAR models. To address this challenge, we propose FreqExit, an integrated loss design that enables dynamic inference in VAR without altering its architecture or compromising output quality. FreqExit is based on a key insight: high-frequency details are crucial for perceptual quality and tend to emerge only in later decoding stages. Leveraging this insight, we design targeted mechanisms that guide the model learn more effectively through frequency-
supervision. The proposed framework consists three components: (1) a curriculum-basee supervision strategy with progressive layer dropout and early exit loss; (2) wavelet-domain high-frequency consis- tency loss that aligns spectral content across different generation steps; and (3) lightweight self supervised frequency gated module that guides adaptive learn- ing of both structural and detailed spectral components. On ImageNet 256 x56, FreqExit achieves up to 2 speedup with only minor degradation, and delivers 1.3x acceleration without perceptible quality loss. This enables runtime-adas addapivee acceleration within consistent design tailored for next-scale VAR, offering favor- able trade-off between efficiency and fidelity for practical and flexible deployment. Code is available at https: //ggt///ttttttttt com www./eeeaalliinneeeeee. FreqExit.
"Corresponding author. Email: wanghuan@westlakke.oo edu. cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Autoregressive models [1, 2, 4, 6, 7] have achieved widespread success, but their substantial computational and memory demands during inference [8, 9, 10, 11, 12] pose major obstacles to deployment in resource-limited environments [13, 14]. To address this challenge, dynamic inference has emerged promising strategy for accelerating large models during deployment, enabling the model to adjust its computational path based on the input [15, 16, 17, 18). This flexibility allows the model retain full capacity for complex inputs while reducing computation for simpler ones, supporting a flexible trade-off between performance and efficiency. With the development of visual autoregressive models such as LLama-Gen [19] and VAR [20]. their capacity handle complex vision tasks has attracted increasing attention, motivating efforts to accelerate inference in this domain [21, 22, 23, 24]. VAR has gained popularity for its next-scale prediction paradigm. which replaces token-by-toker generation with hierarchical token map decoding. This generation paradigm has gained increasing popularity for its ability to improve inference efficiency and scalability, while maintaining high-quality image synthesis. However, the application of dynamic inference to VAR presents unique challenges. The decoding paradigm based on token maps in VAR breaks the granularity assumptions behind token-level control methods such as speculative decoding [17, 25]. In addition, our analysis (Sec. 3.1) shows that transformer layer representations remain highly dynamic and non-redundant, which invalidates the assumptions of layer stability required by early-exit [26, 27] and layer compression [28, 29] approaches designed for ooken-wiss models.
To address this challenge, we perform an in-depth analysis of the VAR generation dynamics and identify distinet frequencies domain variations throughout the decoding steps. This progressive shift from low- www-freuency structures in the early steps to high-frequency details in later steps leads to step-dependent feature semanties. These step-dependent frequency shifts and coarse-to-fine token map generation patterns lead to unstable intermediate representations and limited predictability, which undermine the effectiveness of conventional dynamic inference strategies. Based on these insights, propose FreqExit, cohesive training framework that integrates three synergistic mechanisms to support efficient and adaptive inference in VAR: (1) a curriculum- based supervision strategy that progressively activates intermediate layer training, (2) a frequency-awar consistency loss that guides reconstruction at later steps, and (3) a frequency-gated self-reconstrruttinn loss that adaptively regularizes spectral learning during training. Together, these components robustly enhance intermediate representations and enable efficient early exits without compromising generation quality, Experimental results demonstrate the efficiency of our method. As shown in Figure our method achieves up to 2x acceleration with minor quality degradation, and delivers 1.3x speedup with no perceptible loss-offering flexible trade-off's between efficiency and overall generation fidelity. Our main contributions are summarized as follows:
We propose an integrated training framework for accelcrating dynamic inference in visual autoregressive models, specifically tailored to the next-scale generation paradigm of VAR.
We introduce two training- time components: a step-wise high- frequenyy consistency loss and lightweight frequency -gttdd reconstinnttt (FGSR) module, which improve representation quality and accelerate training without incurring any inference-time overhead.
Extensive experiments demonstrate that our method achieves up to 2x inference speedup with negligible quality degradation. To the best of our knowledge, this is the first work to enable early-exit-basee dynamic inference in next-scale generation models like VAR.
2 Related Work
2.1 Dynamic Inference Methods
Dynamic inference refers to the ability of a model to adjust its computational pathway based on input, allowing selective execution of layers, channels, or tokens to improve inference efficiency [30, 31, 32]. Compared to static compression methods, dynamic approaches offer better adaptability across inputs of varying complexity. Early works have explored the depth of dynamic layers, channel width, and routing mechanisms [16, 33, 34, 35]. More recent research extends this idea to transformer-based architectures in both language and vision domains, where dynamic inference is used to adaptively control the depth of decoding the granularity of token-level computation [27, 28, 36, 37, 38).

---

Early Exiting As representative form of dynamic inference, early exiting enables a model terminate inference intermediate layers based on signals such as confidence or consis- tency [26, 36, 39, 40]. Sample-level approaches determine the required depth based on overall input complexity, typically by attaching lightweight classifiers [41, 42] or monitoring output consis- tency across layers [39, 43J. At the token level, early exiting has been applied to both encoder and decoder architectures. CALM [26] uses confidence-based classifiers and handles KV cache incon- sistencies via hidden state reuse. BERxiT [44], BE3R [45], and EE-LLM [46] extend token-level exits using learned classifiers. SkipDecode [36] improves batch-level efficiency by enforcing unified and monotonic exits. Raposo al. [28] introduce trainable routers for dynamic depth control. Layer- Skip [27] eliminates explicit confidence estimation by unifying early-exit training with speculative decoding. Recent efforts also extend early exit to token-wise visual generation, including MuE [40], AdaNAT [47], and DeeR-VLA [38].
2.2 Inference Acceleration for Visual Generation
Autoregressive image generation has evolved from early raster-scan models [48, 49] to transformer- based frameworks that sequentially generate discrete image tokens via VQVAE [50] or VQGAN [51]. Recent models such LlamaGen [19] and MAR [52] achieve strong results in high-resolution and multimodal generation [53, 54]. VAR [20] introduces a next-scale paradigm that outputs entire token maps per step. This hierarchical design enhances scalability and has since inspired a wide range of follow-up works spanning different modalities and tasks [55, 56, 57, 58, 59, 60].
Despite the rapid progress in visual autoregressive modeling, research inference acceleration remains at an early stage. Several approaches have been proposed to reduce decoding latency and computational cost. Token-parallel acceleration methods such speculative decoding [23, 25] aim to predict multiple tokens simultaneously or restructure the inference schedule for faster generation. However, these methods are fundamentally incompatible with next-scale generation models like VAR, as the latter produce entire token maps per step instead of single-token outputs, undermining the assumptions of token-level parallelism. CoDe [61] proposes collaborative decoding strategy to accelerate VAR by assigning large and small models to different scales. Nonetheless, it relies on separately fine-tuned models with fixed inference paths, and its dependence limits applicability in resource-constraines settings where a single lightweight model is preferred. To address these limitations, we introduce a dynamic inference framework tailored for the hierarchical next-scale generation paradigm, enabling efficient acceleration through early exits.
large backbone
3 Method
3.1
Comprehensive Analysis of VAR Generation Dynamics
1.0
80
1.0
Typical eally-oua dicehald (0.3)
Low Frequency Enengy
FID
Casine simiarit witts presoous
2500
High Frequency Fnerg
0.
High Brquencyy Dustoctinn Reduction (AFID)
Energy
2000
1500
HighLow Rani
0.66
Fiihht Low Presqiencyy
I1D Score
6th
33.5
35.1
38.1
0.6
0.4
0.4
500
0.2
Generation Steps
6
0.0
VAR-316
3.55
VAR-020
2.95
VAR-I24
2.33
VAR-030
1.97
0.0
10
Leyer
12
16
20
(a) Frequency Growth
(b) AFID Breakdown
(c) Layer Similarity
Figure 2: Comprehensive visualization VAR generation dynamics. (a) Frequency component evolution across
steps in VAR-d20. (b) FID degradation caused by high-frequency removal, (c) Cosine similarity between token
maps from consecutive layers.
To enable dynamic acceleration tailored for next-scale generation, we analyze the generative behavior of VAR [20] through discrete wavelet transform -based frequency decomposition, technique shown effective in prior generative model studies [62, 63), to reveal its step-wise spectral characteris- tics. Specifically, we decompose token maps at each generation step into low- and high-frequency components, and study their evolution across steps and layers. We further compare model variants (d16, d20, d24, d30) to assess how high-frequency removal affects generation quality. In parallel, we
3

---

conduct layer analysis by recording the output token map at each transformer block and computing its cosine similarity with the previous layer, quantifying how representations shift throughout the network. Fig. 2 illustrates three key findings that characterize the generative behavior of VAR.
Observation 1. Frequency progression across steps As shown in Fig. 2a, the first six steps predominantly reconstruct low-frequency structures, while steps 7-10 introduce a sharp increase in gegh-rreueeccy content. This step-wise divergence reveals a two-stage generation process, where the first steps establish the global structure and later steps refine fine-grained details, making it difficult for the model to maintain consistent representations across stages.
Observation 2. Hrig-frr accuracy drives quality All models experience substantial performance degradation when high-frequency details removed, with the impact increasing with model sigg-biig that deeper models primarily improve visual quality by modeling fine details more precisely. This highlights the importance of high-frequency modeling for improving early-layer exit capability, as generative quality relies heavily on accurate reconstruction of fine textures rather than fow-freueeccy structure alone.
Observation No redundant layers We assess layer-wise changes using truncated inference and cosine similarity between successive token maps (Fig. 2c). Similarity values remain consistently low (ranging between 0.2-0.6), even the final layers (e.g.. between layer 19 and 20), suggesting that each transformer block continues to modify the representation meaningfully. This contradicts the layer stabilization assumption commonly adopted in early-exit strategies such as CALM [26), leaving no obvious "safe" point for early termination.
3.2 Our Method
Motivation. Our analysis (Sec. 3.1) reveals that VAR follows a hierarchical next-scale decoding paradigm, where generation gradually shifts from low-frequency structures to high-frequency details. This results in step-wise frequency transitions and unstable inter-layer representations that hinder reliable early-exit decisions. In contrast to conventional autoregressive models, intermediate outputs in VAR are highly dynamic and lack semantic stability, as tokens are repeatedly refined across steps rather than converging to fixed representations. Accurately modeling sigh-frrquency components therefore becomes essential for maintaining output quality during early termination.
To address these challenges, we propose an integrated training framework that restructures in- termediate representations to support early exit without degrading output quality. It integrates: (1) curriculum- supervision to enhance shallow layers, (2) frequency- consistency loss to align multi-step predictions, and (3)a frequency-gated self-reconstruuctinn (FGSR) module for modeling cross-band dependencies. An overview of the proposed framework is illustrated in Fig. 3.
3.2.1 Curriculum-Base Early-Exit Supervision
We adopt a curriculum learning-basee early-exit training framework that jointly supervises inter- mediate layers support efficient dynamic inference, shown in Fig. 3(a). It combines depth-aware layer dropout with cross-entropy loss at each supervised layer, and further introduces a lightweight, layer-adaptive distillation mechanism that adjusts supervision strength by depth.
Layer Dropout. To encourage shallow layers learn stronger representations. we apply layer dropout during training by randomly skipping layers with depth-dependent probabilities. Specifically, deeper layers are dropped more frequently to promote early-layer expressiveness. The hidden state update for layer at iteration / is:
Fe = Fty + M(Pr.) fe(
where (p) is Bernouli mask, and the dropout probability Pe, increases with depth.
Early-Exit Loss. We apply an early-exit loss to intermediate layers, enabling dynamic inference without modifying the architecture. Specifically, each transformer block is connected to shared LM

---

Layer Dropout
Ground Truth
Teacher Logits
(Prr) Bermouli() Pos
LM Head
CE Loss
Pu.t
VAR Layer
Pi.c
Loss
VAR Layer
Tk
Pi.e
Early-Exit Loss
VAR Layer
Intermediate Layer Outputs
T
Early-Exit Supervision
Reconstruction
iDWT
LL
Frequency Gate
www-Freqeency stage
Mixed generation stage
LH
Tk
DWT
Diagonal Projection
Pin
DWT
Origin
Reshupe
Conv
H
VAR Model
1x1 Conv
HF Loss
H,
Th-1
Wavelet Sub-Bands
(b) Progressive High-Frequency Consistency Loss
(c) Auxiliary Frequency-Gated self-Reconstruction Loss
Figure 3: Overview of our proposed ExiqExit framework. (a) Curriculum-Basse Early-Exit Supervision: integrates depth-aware layer dropout and adaptive early-exit loss with layer-specific knowledge distillation, encouraging shallow layers to learn expressive features under dynamic supervision curriculum. (b) High-Frequency Consistency Loss: enforces step-wise spectral align- ment between student and teacher predictions in the wavelet domain, stabilizing high-frreqency learning across generation steps without disrupting early training behavior. (c) Frequency- Saff-Reconstructinn Loss: provides auxiliary loss branch with learnable sub-band gates that modulate wavelet components and guide the model toward frequency-awan spectral reconstruction, improving convergence and generation quality without affecting inference runtime.
head y and supervised by cross-entropy loss:
L-1 ( (oddl)eeoll((e a(t, C(I,E)((() LEE Σ
(2)
Σ.J C((,i)e(1)
Here C't, E)) is a binary curriculum gate that activates exits gradually, while e(() increases quadrati- cally with depth to place more weight later layers.
To improve learning at shallow exits, we introduce a adaprivd knowledge distillation loss that provides softened teacher guidance with temperature annealing:
E I
LKD
Σ we T KL (g:/T /T), T Tmax (Tmax T Trinn EES
Etes WE
where qe and 9, denote the teacher (full (20) and layer- wise student logits, respectively, and the temperature T₁ decreases from 4.0 (shallow layers) to 1.0 (deep layers), providing softer targets to stabilize shallow-layer training, where semantic information is limited. The early-exit loss combines LEe and LKD. guiding intermediate exits under the curriculum schedule.
Curriculum Scheduling. To ensure stable training and effective supervision, we adopt a rotational curriculum C(I, f) that activates early-exit loss on a subset of layers at each iteration. Specifically, fixed number of layers are selected in a round-robin manner, allowing layers to periodically updated while avoiding conflicting gradient signals between shallow and deep layers. This strategy

---

mitigates the dominance of gradients in deeper layers and promotes more balanced learning across depths. Details of the curriculum schedule for C(t, E) and S(t) are provided in Appendix A in the supplementary material.
3.2.2 High-Frequency Consistency Loss
As shown Fig. 3(b), we introduce a tigh-frequeney consistency loss to guide intermediate rep- resentations during training. While high-frequency components are essential in the later stages of next-scale decoding (Sec.3.1). they are often underrepresented earlier layers due to resolution mismatch and immature semantics. Direct supervision at these stages may destabilize training. To mitigate this, we propose progressive loss that softly aligns student and teacher predictions in the wavelet domain through step-wise and epoch-wise scheduling.
For each layer é and generation step n with Pv Pn. patches, reshape the token map into spatial grid and apply 2D Haar DWT:
(e), [.((0)) (re(respppoxxe..))
and extract the high-frequency bands Hg(()) and H, from student and teacher. To balance gradient contributions across resolutions, we apply step-aware weights wn:
Pn
Wn
=
(5)
Pmax
where y is tunable scaling factor. The total high-frequency consistency loss is defined as:
CHF ΣnEP Wn REP
ΣΣ Wn | [|H000 H7ll2
(6)
To prevent premature constraints insufficiently trained layers, Lue is initially disabled and gradually activated as training progresses. A dynamic scaling mechanism adjusts its contribution based on the moving average ratio between ChP and other losses, ensuring stability throughout.
3.2.3 Frequency- Gateed RelffReconstructinn Loss
Reonnstconttinnonnn (FGSR) module is proposed to introduce a residual training path with learnable gates over wavelet sub-bands, as shown in Fig. 3(c). Unlike traditional auxiliary branches, FGSR provides interpretable frequency-aware supervision while leaving the main architecture unchanged. Frequency-aware optimization during next-scale generation achieved by incorporating a learnable gating mechanism over wavelet sub- bands. Let ra denote the token map generation step t, and Faux,t the auxiliary output reconstructed from gated frequency components. The training objective is:
lightweight Frequency-Gater
Lisal LCE(r.) X |/Faux, rx113
where LCE is the main task loss and À controls the strength of reconstruction guidance. Each sub-band b E {LL, LH, H L, HH} modulated by a learnable gate 76 b' b. ? The gradient of Lioual w.r.t. comprises components from both the main loss and reconstruction loss. Under the assumption r: :DWT(.... b- e ...,,,,, ), the regularizing term becomes:
i r Tuux,t rel1
2(70 1) |b6
8
0%
which pulls " toward unless overridden by task gradients. At convergence, the closed-form update that balances task alignment and reconstruction stability is:
1 DCe
76
|
Gates 7 are enhanced when their frequency band aligns with task gradients, guiding the model from low-frequency structure to high-frequency detail over time. This behavior realized through the following frequency-gatee reconstruction process.

---

FGSR operates on intermediate feature maps E BBXCXHXW, where C' denotes the number of channels. Each feature map is reshaped into 2D spatial layout and decomposed via discrete wavelet transform (DWT) into four frequency sub-hands. A learnable exponential gate "6 is applied to each
sub-band, yielding the modulated band V/ b e"e The gated sub-bands are concatenated and projected through shared linear transformation Wgp E ERCxIC then reconstructed via its transpose
and inverse DWT:
f = iDWT (Wigg WEo LII', ,HL',HHH"
(10)
The frequency- gated reconstruction loss is defined over generation steps A and supervised layers S:
FEGSR
1 [|]
1 A
Σ [(oOS)(())) AEA
A Latign
(11)
EES
where fEOSK denotes the reconstruction function, and À balances the auxiliary loss. A projection regnmanntegglaariiin is introduced promote numerical stability and reversibility, where Sym(A) (A + J is the symmetrization operator, and Win approximates WT.
Calign = ||Sym (W.
|aatenI|SSyyagl fop Wip - DII + || l
(12)
FGSR serves lightweight, inference-free auxiliary branch designed to address the mismatch in generation patterns across different stages. Early steps tend to produce low-frequency structures, while later steps emphasize both low- and high-frequency details. This shift poses challenges for joint training across different stages. FGSR resolves this by introducing frequency-gated supervision that softly guides optimization without interfering with the main gradient flow. Specifically, when the reconstruction loss remains small, the model maintains emphasis on low-frequency components. As high-frequency details become important, the loss increases, prompting the gates to adaptively adjust their influence- -thereyy signaling the model to refine its spectral representation. Through this loss-driven feedback loop, FGSR enables stage-aware. spectrum spectrun-adaptive training that improves convergence without modifying architecture. The specific implementation details and pseudocode of FGSR can be found in Appendix A of the supplementary material.
Overall Training Objective. Following the integration of all supervision components, the final training objective of FreqExit is formulated as:
Lioall CEE + AdCKD + ALCHF + C&
(13)
where LI and LKD jointly constitute the early-exit supervision loss, providing primary learning signals for intermediate layers, while Lui and LEGSs form the frequency adaptation loss that enhances spectral consistency and improves high-frequency reconstruction quality. The weighting coefficients X control the relative contribution of each component and are provided in Appendix C.
4
Experimental Results
4.1 Experimental Setup
Baseline Methods. We evaluate our method in comparison with several representative baselines. CALM [26] a training-free method that exits based on the softmax gap between top-1 and top-2 logits. As the default threshold of 0.9 rarely triggers exits under layer-wise dynamics of VAR, it lowered to 0.6 to enable meaningful comparison. LayerSkip [27] enables flexible truncation via shared early-exit loss and layer dropout. Its layer-wise mechanism is adapted to VAR, without speculative decoding, which is incompatible with scale-wise generation. CoDe [61] adopts a two- model collaboration between drafter and a refiner. In setup, VAR-d20 and VAR-d16 are used as the drafter and refiner, enabling step-wise refinement during inference. However, CoDe relies on two independently trained models with fixed inference paths, whereas FreqExit enables adaptive acceleration within a single model without architecture modification.
Implementation Details. All models are trained on ImageNet- IK [64]. CALM requires no training. LayerSkip and CoDe are trained for 80 epochs using Adam W with a batch size of 1024. Our method uses the same training setup as LayerSkip, with a learning rate of le-5 and no weight decay for 80 epochs. For CoDe, we evaluate both training-fres and fine-tuned variants. the fine-tuned setting, the drafter (VAR-d20) is trained for 15 epochs (learning rate 1c-6, weight decay 0.08), and the refiner
7

---

Table 1: Comparison of inference efficiency and generation quality across methods. Param: active parameters; Layers: average number of transformer layers used; Speedup: latency- acceleration relative to VAR-d20, Latency (8): per-image inference time; Thrpt (it/s): throughput, i.e., images
processed per second; GFLOPs: total floating-poin operations; FID, IS: standard metrics for
generation quality: Prec.: Precision; Rec.: Recall.
Method
Param Layers Speedup Latency
Thrpt
GFLOPs
FID 4 IS f Prec.
Rec. f
VAR-d20
600M
1.00x
0.265
3.80
1879
2.78
252
0.84
0.55
CALM
546M
18.2
1.05x
0.252
3.07
1779
8.20
249
0.68
0.60
CoDe (w/o training)
900M
18
1.18x
0.224
4.66
1440
9.35
251
0.79
0.52
CoDe (w/ training)
900M
18
5.86
252
0.80
0.54
l.aeerkki(a)
480M
16
1.27x
0.209
4,36
1664
3.25
253
0.80
0.58
LayerSkip (8)
420M
14
1.30x
0.204
4.92
1466
3.99
251
0.78
0.57
LayerSkip (+)
360M
1.63x
0.162
6.21
1448
4.78
250
0.76
0.60
FreqExit(")) (Ours)
480M
1.27x
0.209
4.36
1664
2.72
252
0.81
0.55
FreqExit (0) (Ours)
420M
1.30x
0,204
4.92
1466
2.90
250
0.78
0.57
(re (Ours)
360M
12
1.63x
0.162
6.21
1448
3.16
251
0.71
0.57
Exeq (4) (Ours)
300M
2.00x
0.130
7.72
1341
3.58
251
0.73
0.56
Note. VAR generates images in 10 autoregressive steps. evaluate four early exit configurations reflecting
different cost-accuracy trade-offs, where each vector lists transformer layers used at each generation step.
(VAR-d16) is distilled for 65 epochs (learning rate le-5, no weight decay), following the original CoDe setup. In practice, FreqExit does not require retraining VAR from scratch. We initialize from the official VAR-d20 checkpoint (trained for 250 epochs on ImageNet-IKg and fine-tune for an additional 80 epochs (~300 cost) enable early-exit capability. Detailed training configurations are detailed in Appendix C of the supplementary material.
Evaluation Metrics. For each baseline method, generate 50,000 images (50 per class) across 1,000 ImageNet- IK classes, using the same inference settings on a single RTX 4090 GPU. Efficiency evaluated by latency, FLOPSs (measured using the Torch. profiler) throughput, memory usage, active parameters, and transformer layers (Param and layers). Speedup is computed relative to VAR-d20. Generation quality is assessed using four standard metrics: FID, IS, Precision, and Recall.
4.2
Main Results
On the ImageNet 256 x 256 benchmark, our FreqExit framework achieves up to 2.0x acceleration with only minor quality degradation (FID increases from 2.78 to 3.58), and provides 1.3x speedup with no perceptible loss in generation quality (FID 2.72). These results are achieved under different early-exit configurations, demonstrating the flexibility of the method to support un-timm-addativy inference with a single model, allowing dynamic selection of exit strategies without retraining or architectural changes. Compared to baseline methods including CALM, LayerSkip, and CoDe (see Table 1), FreqExit consistently delivers better sfficiency-quality trade-offs. It achieves lower latency, fewer FL.OPs, and higher throughput under comparable or improved fidelity. For example, at the most aggressive setting, FreqExit reaches throughput of 7.72 it/s, outperforming all baselines. When compared to LayerSkip, our method shows clear advantages, especially at shallower configurations. As the number of active layers decreases, Exiq maintains better generation quality with lower FID scores- highlightng its ability to preserve fidelity even under tight computational budgets.
Overall, these results confirm that FreqExit enables efficient, flexible, and high- autoregressive generation. It effectively bridges the gap between next-scale decoding and practical dynamic inference, offering scalable solution for deployment under varying resource budgets.
4.3 Dynamic Inference Capability
To evaluate the dynamic inference capability of our method, we construct series of early-exit strategies with different layer configurations across the 10 autoregressive generation steps. Each strategy generates 50,000 images on a single NVIDIA RTX 4090 GPU, and the results are summarized in Table 2. Progressively reducing the average number of layers leads to substantial efficiency gains.
8

---

Table 2: Early exit strategies used in dynamic inference. Each vector indicates the number of transformer layers used at each of the 10 generation steps.
Strategy
Exit Layers at Steps
Avg. Params
Avg. Layers
Max Batch Size
FID V
Full
[20, .... 20] 220.0.20.22022111111.6....6
600M
20.0
80
2.71
540M
18.0
96
2.72
[16, .... 16]
480M
16.0
96
2.72
110.16116.66111.2.111.....2.
420M
12.0
120
2.90
[16, 12]
360M
12.0
120
3.08
116.2.12..2212.10000110..0.00100
330M
12.0
128
3.27
[16,12,10, .... 10]
300M
130
3.37
Table Ablation study on training efficiency. FID values intermediate layers (Layer 8 and Layer 12) across training epochs, comparing models with and without HF loss and FGSR. The proposed modules lead faster convergence and significantly lower FID, especially shallower layers.
FID with exit at Layer 8
FID with exit at Layer 12
Epoch
w/o HF+FGSR w/ HF+FGSR
w/o HF+FGSR
w/ HF+FGSR
40
22.73
13.20
6.87
4.59
48
20.53
12.54
5.52
4.86
56
19.40
11.74
5.31
4.36
64
18.52
10.90
5.21
3.92
72
18.78
10.51
5.25
3.48
80
17.38
10.02
4.78
3.26
Compared to the full inference baseline, the most lightweight configuration uses only the first 10 layers per step, reducing active parameters by 50% and increasing the maximum batch size by 1.63x (from 80 to 132). This comes with only a modest drop in quality, as FID increases by just 0.94.
We analyze these results from perspectives: First, dynamic configurations significantly reduce computation while preserving image quality. Strategies averaging 16 or more layers achieve FID scores comparable to the full model, demonstrating that early exits can reduce cost without sacrificing fidelity. The reduced memory usage also allows for larger batch sizes and improved throughput. Second, this dynamic behavior aligns well with the autoregressive nature of VAR, where token counts increase across steps. Early steps, with fewer tokens and lower-resolution representations, primarily capture low-frequency structures and can benefit from deeper layers. In contrast, later steps more expensive due to larger token maps and can use shallower exits to control cost. This per-step flexibility enables fine-grained, step-aware adjustment of inference depth, allowing the model to adaptively balance quality and efficiency throughout the generation process.
4.4 Ablation Study
To evaluate the effectiveness of our proposed training enhancements, we conduct an ablation study comparing models trained with and without the HF loss and FGSR modules. In this study, track the FID scores obtained under early exit at Layer and Layer 12 across training epochs. The results, presented in Table 3, show that models equipped with these modules consistently achieve lower FID and converge more rapidly than their counterparts. The improvement is particularly pronounced when exiting at shallower layers (e.g., Layer 8), where the FID decreases more sharply and stabilizes an earlier stage of training. These findings suggest that our method not only improves final performance, but also accelerates representation learning, especially the early layers. This can be attributed to two key factors: (1) the HF loss provides explicit supervision in the frequency domain, helping the model adapt to different frequency characteristics across generation steps; and (2) the FGSR reconstruction signals that guide optimization toward step-specific generation behaviors. Together, these components mitigate the mismatch between model capacity and evolving generation patterns, enabling more efficient learning in shallow layers.
module delivers frequency-avee

---

Conclusion
We present ExeqExit, an integrated training framework that enables dynamic inference in next-scale visual autoregressive models. By thoroughly analyzing the spectral and representational behavior of VAR, we identified key obstacles efficient early exit, such as frequency progression across steps and unstable intermediate representations. FreqExit addresses these challenges through three targeted modules: a curriculum-base
early-exit supervision strategy, a progressive wavelet-based high- frequency consistency loss, and a lightweight frequency-gater seff-reconstruction module. Extensive experiments on ImageNet x56 demonstrate that Exiq achieves up to 1.3x speedup with no perceptible quality degradation, and up to 2x acceleration with only minor negligible losses. This flexible trade-off highlights the adaptive nature of FreqExit, which can dynamically adjust computational depth to meet diverse efficiency requirements. Overall, Exiq provides practical and generalizable solution for efficient early-exit acceleration next-scale generation models.
Acknowledgements
This paper is supported by Young Scientists Fund of the National Natural Science Foundation of China (No. 62506305), Zhejiang Leading Innovative and Entrepreneur Team Introduction Program (No. 2024R01007), Key Research and Development Program of Zhejiang Province (No. 2025C01026), Scientific Research Project of Westlake University (No. WU2025 Chinese Association for Artificial Intelligence (CAAT)& & Ant Group Research Fund- AGI Track (No. 2025CAA1-ANT-13), and the Special Support Talents Program of "Xi Hu Ming Zhu Program" in Hangzhou.
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint ar arXiv:2303.08774. 2023.
[2] W. X. Grattafiori, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve, "Code Ilama: Open foundation models for code," arXiv preprint arXiv:2308. /2950, 2023.
[3] J. Y. Koh D. Fried, and R. Salakhutdinov, "Generating images with multimodal language models," " in NeurlPS, 2023.
[4] W. Wang, Chen, X. Chen, Wu, X. Zhu, Zeng, Luo, T. Lu, Zhou, Y. Qiao al., "VisionlIm: Large language model is also an open-ended decoder for vision-centric tasks," in NeurlPS, 2023.
[5] D. Guo, D. Yang, H. Zhang, Song, R. Zhang, R. Xu, Q Zhu, Ma, P. Wang, X. Bi et al., "Deepseck-rl: Incentivizing reasoning capability in Iims via reinforcement learning," arXiv preprint arXiv:2501. 12948, 2025.
[6] X. Tu, Z. He, Y. Huang, Z.-H. Zhang, M. Yang, and J. Zhao, "An overview of large ai models and their applications," Visual Intelligence, vol. 2, no. 34, 2024.
[7] Y. Jiang, X. Yan, G.-P. Ji, K. Fu, M. Sun, H. Xiong, D.-P. Fan, and F. S. Khan, "Effectiveness assessment of recent large vision-language models," " Visual Intelligence, vol. 2, no. 17, 2024.
[8] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, "Flexgen: High-throughput generative inference large language models with single gpu," in ICML, 2023.
[9] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett et al., "H2o: Heavy-hitter oracle for efficient generative inference of large language models," in NewrIPS, 2023.
[10] K. Shao, K. Tao, K. Zhang, S. Feng, M. Cai, Y. Shang, H. You, C. Qin, Y. Sui, and H. Wang, "When tokens talk too much: A survey of multimodal long-context token compression across 2025.
images, videos, and audios," arXiv preprint
10

---

[11] K. Shao, K. Tao, C. Qin. H. You, Y. Sui, and H. Wang, "Holitom: Holistic token merging for fast video large language models," " in NeurIPS, 2025.
[12] K. Tao, C. Qin, H. You, Y. Sui, and H. Wang. "Dyeoke: Dynamic compression of tokens for fast video large language models," in CVPR, 2025.
[13] 7. Liu, C. Zhao, F. landola, C. Lai, Y. Tian, I. Fedorov, Y. Xiong, E. Chang, Y. Shi, R. Krish- namoorthi et al., "Mobilellm: Optimizing sub-billion parameter language models for on-device use cases," in ICML, 2024.
[14] Z. Gao, Z. Chen, E. Cui, Y. Ren, W. Wang, J. Zhu, H. Tian, S. Ye, J. He, X. Zhu, L. Lu, T. Lu, pocket multi-modal model
Y. Qiao, J. Dai, and W. Wang. "Mini- internvl: a flcxible-trame with 5% parameters and 90% performance," Visual Intelligence, vol. 2, no. 32, 2025.
[15] Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L. S. Davis, K. Grauman, and R. Feris, "Blockdrop: Dynamic inference paths in residual networks," in CVPR, 2018.
[16] Y. Han, Y. Pu, Z. Lai, C. Wang. S. Song. J. Cao, W. Huang, C. Deng, and G. Huang, "Learning weight samples for dynamic early-exiting networks," in ECCV, 2022.
[17] Y. Leviathan, M. Kalman, and Y. Matias, "Fast inference from transformers via speculative decoding,"" in ICML. PMLR, 2023.
[18] X. Chen, K. Tao, K. Shao, and H. Wang, "Strcamingtom: Streaming token compression for efficient video understanding," arXiv preprint arXiv:2510.18266 2025.
[19] Sun, Y. Jiang. S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan, "Autoregressive model beats diffusion: Llama for scalable image generation," arXiv preprint arXiv:2406.06522, 2024.
[20] V. autoregressive modeling: Scalable image generation via next-scale prediction, "Tian, keyu and jiang, yi and yuan, zehuan and peng. bingyue and wang, liwei," in NeurIPS, 2024.
[21] Y. Li, F. Wei, C. Zhang, and H. Zhang, "EAGLE: Speculative sampling requires rethinking feature uncertainty," in ICML, 2024.
[22] H. Deng, T. Pan, H. Diao, Z. Luo, Y. Cui, H. Lu, S. Shan, Y. Qi, and X. Wang, "Autoregressive video generation without vector quantization," " in ICLR, 2024.
[23] Y. Teng, H. Shi, X. Liu, X. Ning, G. Dai, Y. Wang, Z. Li, and X. Liu, "Accelerating auto- regressive text-to-image generation with training-free speculative jacobi decoding," in ICLR, 2024.
[24] Jian and H. Wang, "Ssd: Spatial-semantic head decoupling for efficient autoregressive image generation," arXiv preprint arXiv:2510./87/6 2025.
[25) D. Jang, S. Park, J. Y. Yang, Y. Jung. J. Yun, S. Kundu, S.-Y. Kim, and E. Yang. "Lantem:: Accelerating visual autoregressive models with relaxed speculative decoding," arXiv preprins arXiv:2410.03355 2024.
[26] T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Q. Tran, Y. Tay, and D. Metzler, "Confident adaptive language modeling," in NewrIPS, 2022.
[27] M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman et al., "Layerskip: Enabling early exit inference and seefuupecul decoding," in ACL, 2024.
[28] D. Raposo, S. Ritter, B. Richards, T. Lillicrap, P. C. Humphreys, and A. Santoro, "Mixture-of- depths: Dynamically allocating compute in transformer- based language models," arXiv preprint arXivv2404.022558 2024.
[29] A. Liu, J. Liu, Z. Pan, Y. He, R. Haffari, and B. Zhuang, "Minicache: Kv cache compression in depth dimension for large language models," in NeurIPS, 2024.
[30] C. Li, G. Wang, B. Wang, X. Liang, 7. Li, and X. Chang, "Dynamic slimmable network," in CVPR, 2021, pp. 8607-8617.
11

---

[31] Z. Zheng, L. Yang, Y. Wang, M. Zhang, L. He, G. Huang, and F. Li, "Dynamic spatial focus for efficient compressed video action recognition." IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 2. pp. 695-708, 2023.
[32] Y. Pu, Y. Wang, Z. Xia, Y. Han, Y. Wang. W. Gan, Z. Wang. S. Song, and G. Huang. "Adaptive rotated convolution for rotated object detection," in CVPR, 2023.
[33] S. Teerapittayanon, B. McDanel, and H.-T. Kung, "Branchynet: Fast inference via early exiting from deep neural networks," in ICLR, 2016.
[34] T. Bolukbasi, J. Wang, O. Dekel, and V. Saligrama, "Adaptive neural networks for efficient inference," in ICML, 2017.
[35] L. Yang, Y. Han, X. Chen, S. Song, J. Dai, and G. Huang, "Resolution adaptive networks for efficient inference," in CVPR, 2020.
[36] L. Del Corro, A. Del Gioro, S. Agarwal, B. Yu, A. Awadallah, and Mukherjee, "Skipdecode: Autoregressive skip decoding with batching and caching for efficient Ilm inference," arXiv preprint arXiv:2307.022 02628,, 2023.
[37] X. Hu, Z. Huang, A. Huang. J. Xu, and S. Zhou, "A dynamic multi-scale voxel flow network for video prediction," in CVPR, 2023.
[38] W. Zhao, J. Tang, Y. Han, Y. Song, K. Wang. G. Huang, F. Wang, and Y. You, "Dynamic tuning towards parameter and inference efficiency for vit adaptation." in NeurlPS, 2024.
[39] W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei, "Bert loses patience: Fast and robust inference with early exit," in NeurlPS, 2020.
[40] S. Tang, Y. Wang, Z. Kong, T. Zhang, Y. Li, C. Ding, Y. Wang, Y. Liang, and D. Xu, "You need multiple exiting: Dynamic early exiting for accelerating unified vision language model," in CVPR, 2023.
[41] Xin, R. Tang, J. Lee, Y. Yu, and J. Lin, "Deebert: Dynamic early exiting for accelerating bert inference," in ACL, 2020.
[42] J. Kong, J. Wang, L.-C. Yu, and X. Zhang, "Accelerating inference for pretrained language models by unified multi-perspective early exiting," in COLING, 2022.
[43] M. Jazbec, J. Allingham, D. Zhang, and E. Nalisnick, "Towards anytime classification in early-exit architectures by enforcing conditional monotonicity," in NeurIPS, 2023.
[44] J. Xin, R. Tang, Y. Yu, and J. Lin, "Berxit: Early exiting for bert with better fine-tuning and extension to regression." in EACL, 2021.
[45] S. Mangrulkar, A. MS, and V. Sembium, "Be3r: Bert based early-exit using expert routing," in KDD, 2022.
[46] Y. Chen, X. Pan, Y. B. Ding, and J. Zhou, "Ee-Ilm: Large-scale training and inference of exrty-cxit large language models with 3d parallelism," in ICML, 2023.
[47] Z. Ni. Y. Wang, R. Zhou, R. Lu, J. Guo, J. Hu, Z. Liu, Y. Yao, and G. Huang, "Adanat: Exploring adaptive policy for token-based image generation," " in ECCV, 2024.
[48] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves et al., "Conditional image generation with pixelenn decoders." in NeurIPS, 2016.
[49] X. Chen, N. Mishra, M. Rohaninejad, and Abbeel, "Pixelsnail: An improved autoregressive generative model," in ICML, 2018
[50] A. Van Den Oord, O. Vinyals al., "Neural discrete representation learning." in NeurlPS, 2017.
[51] Esser, R. Rombach, and B. Ommer, "Taming transformers for high-resolution image synthesis," in CVPR, 2021.
12

---

[52] T. Li, Y. Tian, H. Li, M. Deng, and K. He, "Autoregressive image generation without vector quantization," in NeurIPS, 2024.
[53] Li, Z. Zhang, and Yang. "Tp2o: Creative text pair-to-object generation using balance swap-sampling, in ECCV, 2024.
[54] Z. Xiong, Z.-d. Zhang. Z. Chen, S. Chen, X. Li, G. Sun, J. Yang, and J. Li, "Novel object synthesis via adaptive text-image harmony," in NewrIPS, 2024.
[55] J. Han, J. Liu, Y. Jiang, B. Yan, Y. Zhang,, Z. Yuan, B. Peng, and X. Liu, "Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis," arXiv preprint arXiv:2412.04431, 2024.
[56] X. Li, K. Qiu, H. Chen, J. Kuen, Z. Lin, R. Singh, and B. Raj. "Controlvar: Exploring controllable visual autoregressive modeling," arXiv preprint MKKv:22406009550 2024.
[57] X. Ma, M. Zhou, T. Liang, Y. Bai, T. Zhao, H. Chen, and Y. Jin, "Star: Scale-wise text-to-image generation via rego-segressive representations, arXiv preprint arXhv:2406.10797, 2024.
[58] H. Tang, Y. Wu, S. Yang, E. Xie, J. Chen, J. Chen, Z. Zhang. H. Cai, Y. Lu, and S. Han, "Hart: Efficient visual generation with hybrid autoregressive transformer," in JCLR, 2024.
[59] Q. Zhang, X. Dai, N. Yang, X. An, Z. Feng, and X. Ren, "Var-clip: Text-to-image generator with visual rugo-segressive modeling," " arXiv preprint arXiv:2408. 01181, 2024.
[60] S. Ren, Q. Yu, J. He. X. Shen. A. Yuille, and L.-C. Chen, "Beyond next-token: Next-x predictio for autoregressive visual generation," arXiv preprint arXiv:2.50. 2025.
[61] Z. Chen, X. Ma. G. Fang, and X. Wang, "Collaborative decoding makes visual auto-regressive modeling efficient," in CVPR, 2024.
[62] W. Ren, H. Yang, G. Zhang, C. Wei, X. Du, S. Huang, and W. Chen, "Consisti2v: Enhancing visual consistency for image-to-vider vo-ideo generation." arXiv preprint arXiv:2402.04322, 2024.
[63] H. Li, S. Li, W. Dai, M. Cao, Kan, C. Li, Zou, and H. Xiong, "On disentangled training for nonlinear transform in learned image compression," in ICLR, 2025.
[64] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in CVPR, 2009.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction faithfully reflect the paper's s contributions. The proposed framework (FreqExit) and its components are detailed in Section 3, and the empirical results claimed are validated in Section 4.2 and Table 1. The motivation and observed challenges with dynamic inference VAR are also supported by the analysis in Section 3.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations discussed in Appendix C of the supplementary material, where the authors reflect on the scope and assumptions of the proposed method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include formal theoretical results proofs. Instead, it
focuses on empirical analysis and the design of a practical training framework for dynamic inference in visual autoregressive models.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides sufficient details to reproduce the main experimental results. Section.4 4 describes all baseline methods, implementation settings, and evaluation metrics. Model configurations are listed in Tables and 2, with additional training configurations included in the Appendix B of the supplementary material. These descriptions allow independent reproduction of the main results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
The authors should provide scripts to reproduce all experimental results for the new reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide anonymized access to our code and datasets, along with instructions to reproduce all main results, including training and evaluation procedures.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
awsssnoooiicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: wwwpc//gg///////gg / /niip. htabli//guuddes//deSuuuuiiii /wtde/euuiiseiiiiieeeeeeeeeeaeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specifies all relevant training and evaluation details in the Experi- mental Setup section, including dataset, data split, optimizer (AdamW), learning rate, batch size, training schedule (number of epochs), and model configurations. Additional implemen- tation and hyperparameter details provided in Appendix B of the supplementary material to ensure full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
16

---

Justification: Due to the prohibitive compute cost of training large- scale visual autoregres- sive models, we do not report error bars or repeated runs. That said, have thoroughly evaluated our method across standard datasets (e.g., ImageNet- K) and multiple configura- tions (see Table I, Figure 1) to confirm the robustness and consistency of our performance improvements.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
The paper specifies the compute resources used for both training and evaluation. Specifically, model training was performed on 8 NVIDIA A 100 GPUs with 80GB memory each. Evaluation and inference were conducted noted the Experimental Setup section. While we do not report exact training time, all experiments follow fixed schedule of 80 epochs with batch size 1024.
Justification:
single NVIDIA RTX 4090 GPU, as
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerrrp.preee//ee cc http:pullc/tthisssiiiiieeeeeeeee
Answer: [Yes]
Justification: The research strictly adheres to the NeurlPS Code of Ethics. All experiments are conducted using publicly available datasets (e.g., ImageNet), without involving any personally identifiable information, sensitive content, or human subjects. The work focuses on methodological development for visual generative models and does not raise foreseeable ethical concerns regarding faimess, misuse, societal harm.
17

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [No]
Justification: The paper does not include explicit discussion of broader societal impacts. However, the work focuses on improving inference efficiency for generative models and does not introduce new capabilities that raise direct ethical societal concerns.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
The paper does not involve models datasets with high risk for misuse. The method focuses on improving the efficiency of visual autoregressive generation and does not entail the release of pretrained generative models or scraped datasets.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
18

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All used assets, including pretrained models and datasets, are publicly available under appropriate licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper provides newly developed training code for the proposed method. An anonymized version of the codebase submitted alongside the paper to support repro- ducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methods and contributions of this research do not involve large language models (LLMs) as important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Enabling Early-Exit Inference for Visual Autoregressive Models via Frequency-Aware Guidance
Exit:
Appendix
This appendix provides additional details the proposed methodology, training configurations, and experimental results. Section A elaborates on implementation components such as early-exit super- vision, frequency-gated reconstruction, and gradient-based loss tuning. Section C outlines training setups and hyperparameters for baseline comparisons. Section D presents qualitative visualizations under conditional generation and zero-shot inpainting. Section E concludes with discussion of limitations and future directions, along with information for reproducibility.
A Details of Methodology
A.1 Curriculum-Ban Early-Exit Supervision
Depth-Aware Layer Dropout. To promote the expressiveness of the shallow layer during train- ing, we implement a depth-sensitive layer dropout strategy. Each transformer block is skipped independently with a depth-increasing probability:
Pt = min maxx
(c)
Pmax:
(14)
where L is the total number of layers and Pmax is global dropout scaling factor (default 0.1). This formulation ensures that early layers are less likely to be skipped during training, thereby encouraging them to learn stronger and more generalizable representations.
Layer-Wise Exit Weights. To prioritize deeper exits while still training shallow ones, we assign normalized weight to the loss of each layer based on its relative depth. Let S C O,..., L- be the set of supervised layers at training step t; the unnormalized weight is defined as:
il
Wi E
DL-2,, 1, i-
(15)
These weights are then normalized across S and used in both the cross-entropy loss and knowledge distillation loss to ensure that deeper exits dominate the overall optimization process, thereby effectively preventing performance degradation the final output layers.
Rotational Curriculum Scheduling. We adopt a rotational curriculum mechanism to activate a subset of exit layers per iteration, ensuring all layers are periodically updated while avoiding conflicting gradients. Let S(L) be the selected subset of layers at iteration i. For a fixed number of partitions R and block size = L/R], we define the active block index as:
(()) = (1) E [(1 mod R) B, min(( mod R+ 1) B, L))}
(16)
This schedule ensures that each layer supervised once every R steps, enabling more diverse gradient signals across training steps while also improving overall training stability:
Pe. S(L) ((E) Pinax:
(17)
where D(f) controls the per-layer profile and S(t) adjusts the global schedule over training steps.
A.2 Frequency-Gated Rell-Reeonstruction:: Implementation Details
Given the progressive decoding nature of next-scale generation in VAR, each generation step exhibits distinct spatial resolution and frequency characteristics. To accommodate this, we adopt step- wise Frequency-Gan
Self-Reet Reconstruction (FGSR) strategy: a lightweight FGSR module instantiated for each generation step, allowing frequency-specific supervision that aligns with the evolving spectrum across decoding stages. This setup supports stage-aware, spectrum-adaative learning while preserving architectural simplicity and inference efficiency.
21

---

Algorithm Step-Wise Frequency-Gated Self-Reconstruction Loss
1: Input: Token maps { patch sizes {p:}. FGSR modules (FGSR()), weight À
2: Output: Total loss LEGSR + X Lalign
3: Initialize LEGSR +- 0, Latign t- 0
4: for layer E E do
5: Extract r² <- Zf
6: for step A do
7: Extract token slice of of shape [B, C, P.Ps: Pt]
8:
Apply step-specific module: of + FGSR (r!)
9:
LFGSR += ||A -
Extract W((1) w from FGSR (8)
1sp inv
11: Lalign += [|Sym(W
(W w
1) - ||
||
fop fgp
inv
fgp
12: end for
13:
end for
14:
return CFGSR + A [3- Calign
Initialization Details. Each FGSR module operates the token map r RBBxCXHX step I and applies a wavelet-based frequency decomposition. The learnable exponential gates ? for each sub-band b [LL, LH, HI, HH} are initialized as:
IL log(1) (1).
"H = MHL. = l0.5).
Mu 1o)
Each FGSR module contains a pair of x convolutional layers: one forward projection Wig RCx4C initialized to be near orthogonal, and one inverse projection Wi.. E RACCC initialized as the transpose of WEp These projection layers are shared within the module and are explicitly regularized during training to ensure long-term numerical consistency and stable convergence behavior, We apply FGSR loss to the supervised layers and all decoding steps, using step-specific FGSR modules instantiated for each generation step i, as shown in Algorithm 1.
Adaptive Gating Across Generation Stages. FGSR enables adaptive frequency gating by respond- ing to the evolving spectral structure across decoding steps. Early token maps primarily capture coarse, low-frequency components, while high-frequency details emerge progressively in later stages, To justify the gating design of FGSR, we provide view from the information bottleneck principle.
At generation step t, the objective is to retain the most relevant information under a distortion constraint. This trade-off can be formulated as rate-distortion problem:
RDDD) min I (z; s.t. B[d(z, 2,)] < D,
(18)
p(re(m)
where / ((xrr)) denotes the mutual information between input æ and representation Tr. and d(-) is distortion measure. Let Em ||H|| 2 denote the eigh-frequeccy energy. FGSR indirectly controls Eur via learned sub-band gates 76. From an information-theorte function to frggufreyuency energy is given by:
view, the sensitivity of the rate
0R:
A
HH 22
(19)
θEEF
1
1+X
1rel
where À is regularization strength associated with frequency emphasis. This term increases with the decoding steps, which justifies the design of FGSR to progressively activate the high sighfrreuency sub-band gates. Such adaptive behavior ensures early stages prioritize structure and layout, while later stages refine details, maintaining reconstruction fidelity with minimal redundancy. It supports information-efftcent frequency allocation and aligns with the coarse-to-fine generation paradigm.
A.3 Gradient-Based Loss Weight Tuning
To achieve a proper balance among the multiple auxiliary losses used during training, such as early- exit supervision, knowledge distillation, and frequency- aware objectives, we perform gradient-based
22

---

analysis in the debugging phase. This analysis serves to inform the manual adjustment of the fixed loss weights prior the final training. At selected training intervals, each loss component L3 is isolated, and its relative gradient influence with respect to the total loss Lioal is computed as:
(0,
(V Lroral VLk)
C₁ max
(20)
Coxall
where VL2 and Lioal denote the flattened gradients with respect to all trainable parameters. The resulting contribution scores C₁ are used as a reference to adjust the relative weights of different loss terms improve the stability and effectiveness of training.
B Additional Discussion on High-Frequency Behavior
VAR generates images in a scale-by-seale manner, where coarse-to-fim decoding naturally induces a frequency progression from low-frequency structure to high-frequency details. This behavior follows the Nyquist sampling principle: early steps with low spatial resolution can only represent coarse, low- frequency components, while upsampled later steps allow finer textures to emerge. Moreover, natural images exhibit a I/ʃ power spectrum, implying that most signal energy lies low frequencies; during training, cross-entropy loss first reduces www-freuuency errors, leaving high-frequency residuals to dominate later-stage gradients. This explains the observed "structure first, detail-later" dynamics in next-scale generation and motivates the frequency-awer supervision strategy used in FreqExit.
C Experimental Details
We compare our method FreqExit with two representative early-exit baselines, LayerSkip[27] and CoDe[61]. Since both methods require additional training beyond the original autoregressive model, we detail their training configurations alongside ours below. All methods are trained on the ImageNet- IK dataset using the Adam W optimizer with global batch size of 1024 and progressive patch scheduling (1, 3, 4, 5, 6, 8, 10, 13, 16).
CoDe. Following the original two-stage pipeline of CoDe, the drafter (VAR-d20) is fine-tuned for 15 epochs using cross-entropy loss the first N-6 decoding steps, with learning rate of 1e-6 and weight decay of 0.08. The refiner (VAR-d16) is then distilled for 65 epochs from the drafter using soft-label supervision. The loss over the first N steps down- woin-weeghted progressively using a dynamic coefficient a = 1 - A, where À is the normalized training progress.
LayerSkip. We adapt LayerSkip our autoregressive model and train it for 80 epochs with learning rate of le-5 and weight decay of 0.01. At each step, a subset of transformer layers is supervised in a round-robin fashion with rotation interval of R-4, and layer-wise sample dropout is applied with a maximum dropout probability Praxx = 0.1.
ExiqExit. Our method adopts the optimizer and schedule setup the other basclines. At each training step, a subset transformer layers is supervised in a round-robin fashion with group size R=4 and dropout probability Pmuxa 0.1. Three types of losses used during training:
Early-exit loss: Applied to selected intermediate layers, this loss combines cross-entropy and distillation terms, each contributing fixed ratio of 0.5.
HF loss: This wavelet-domain consistency loss is disabled for the first 25% of training, then linearly increased to full strength. Its weight is dynamically computed as
Letteer
a Lut +6'
with a=0.3,
(21)
where Lother denotes the average of CE, early-exit, and distillation losses.
FGSR loss: This loss promotes frequeney- aware representation learning via self- reconstruction. A fixed weight of 5.0 applied to the FGSR term, along with an or- thogonality regularization term weighted by 0.1.
The total loss computed as the weighted sum of these components. Both HF and FGSR losses are further modulated during training via gradient-based dynamic weight scaling.
23

---

Table Training hyperparameters and loss types for methods requiring fine-tuning. BS batch size, LR learning rate, WD = weight decay.
Method
Model
Epochs
BS
LR
WD
Pmax
Loss Type
CoDe (drafter)
VAR-d20
15
1024
le-6
0.08
CE KD CE
CoDe (refiner)
VAR-d16
65
1024
le-5
LayerSkip
VAR-d20
80
1024
le-5
0.01
0.1
ExiqExit
VAR-d20
80
1024
1e-5
0.1
CE KD, HE, FGSR
Original VAR-d20
Speedup 1.0x
Exit @ Layer 16
Speedup 1.27x
Exit @ Layer 14
Speedup 1.30x
Exit @ Layer 12
Speedup 1.63x
Exit @ Layer 10
Speedup x
Figure 4: Visualization generated images from different exit layers under conditional generation. Despite the reduced number of layers, the image quality remains nearly unchanged across exits, while the inference speed can be improved by up 2x.
Qualitative Generation Results
Condition Generation. We provide extensive qualitative comparison between the original VAR-d20 model and our proposed FreqExit model fine-tuned with early-exit supervision. As shown in Fig. 4, we visualize outputs from different exit layers (Layer 16, 14, 12, and 10), corresponding to speedups of x,27x, 1.30x, 1.63x, x, and 2.00x, respectively. From these results, is evident that even with two-fold reduction in inference cost, the generated images maintain high visual quality, semantic consistency, and fine-grained detail. This confirms the effectiveness of our approach in enabling efficient autoregressive decoding without compromising fidelity.
Zero-Shot Inpainting. We further evaluate the model's capability under zero-shot inpainting settings. As illustrated Fig. 5, we begin with complete original images and apply random binary masks to simulate missing regions. These masked images, along with class labels, are provided as inputs, and the model performs inpainting using different exit layers. The results demonstrate that our method retains strong generative performance even with early exits. The inpainted regions remain semantically coherent and visually indistinguishable from the ground truth, indicating that the intermediate layers are well-equipped to capture both global structure and local detail. This highlights
24

---

Exit @ Layer 16
Exit @ Layer 14
Exit @ Layer 12
Exit @ Layer 10
Figure 5: Visualization of zero-shot inpainting results at different exit layers. The model demonstrates strong capability in reconstructing missing regions, and the inpainted content remains visually consistent across different exit depths without introducing artifacts or semantic mismatches.
the capacity of the model for semantic understanding at multiple depths and validates its robustness and adaptability under varying inference conditions.
Discussion and Future Work
We propose Exit a unified framework for enabling efficient early-exit inference in next-scale visual autoregressive models such as VAR. By analyzing the spectral dynamics of VAR and addressing the instability of intermediate representations, our method introduces a curriculum-base supervision strategy, progressive high-frequency consistency loss, and lightweight frequency-gated self- reconstruction module. These components collectively enhance representation quality and support dynamic exit behavior with minimal impact on generation quality. Experiments ImageNet 256x256 demonstrate that FreqExit achieves up to x speedup with little perceptual degradation, and performs well in both conditional generation and zero-shot inpainting scenarios.
Limitations. While FreqExit demonstrates strong dynamic inference performance, it introduces multiple loss terms to guide intermediate learning. As a result, careful tuning of their weights is needed during training. In particular, we find that the high-frequency consistency loss must be appropriately balanced-if overemphasized, it can hinder the learning of low-frequency structures. Nevertheless, this tuning process is manageable and only needs to be done once per configuration, and gradient-based analysis used to guide weight selection is provided in Appendix A.
Future Work Future work will explore more adaptive training strategies to better coordinate frequency-aware supervision with semantic objectives, aiming to further improve dynamic inference performance while minimizing the need for manual tuning efforts.
25

---

Reproducibility. To support reproducibility and further research, we provide training and evaluation code in the supplementary FreqExit, zip file and at the anonymous repository https: // github. www.euraalyinge.. iiggaii/rFreeriit...
26


Reviews:
[{"review_id": "NDGIPsVJqc", "reviewer": "Reviewer_NKKY", "summary": "The paper introduces FreqExit, a methodology that allows dynamic inference Visual Autoregressive (VAR) models. VAR models are characterized by generating images in a coarse-to-fine, next-scale manner. Therefore, existing early-exit strategies fail in this setting due to unstable intermediate representations and the progressive existense of high-frequency details in later steps. The authors achieve dynamic inference with early exit by combining three key strategies:  1) Curriculum-Based Early-Exit Supervision, 2) Progressive High-Frequency Consistency Loss, and 3) A Lightweight Frequency-Gated Self-Reconstruction Loss (FGSR). The first strategy improves the performance of shallow layers. It ensures that all layers are trained to be predicted under supervision with dropout and KD loss relying on layer-wise supervision, depth-aware layer dropout, a rotational curriculum schedule, and a combination of cross entropy and KD for early exit losses. The second techniques aligns intermediate outputs with ground truth in the frequency domain over time. Early generation steps emphasis low-frequency while later steps add high frequency details. This encourages the model to learn frequency-consistent features across steps without disrupting early training dynamics.The FGSR loss tries to enhance frequency learning  via a lightweight auxiliary path using wavelet gates. It acts as an auxiliary loss to guide spectral learning. Empirical evaluation on ImageNet shows that by combining these techniques, FreqExit is able to stabilizes training, support early exists and maintain high fidelity in visual generation especially for models that have coarse-to-fine autoregressive generation like VAR. Overall, FreqExit is able to save computational cost without any architectural modifications and compromising model quality ( 2x inference speedup with negligible quality degradation).", "questions": "1- What is the inference-time criterion for triggering early exits? Is it confidence-based or entropy-based? The paper proposes early exist during generation but no clear explanation when to stop at an intermediate layer during inference. In existing early-exit methods, common heuristics include: entropy threshold, confidence threshed or margin criterion (e,g., exit when the gap between top-1 and top-2 logits exceeds a threshold). It would strengthen the paper if the authors clearly explain the exist policy at inference and evaluate it sensitivity and reliability. \n2- FGSR module appears to be a key component for stabilizing training. How sensitive is the final performance to the hyperparameter λ, which balances the main task loss with the reconstruction guidance? do you have a sensitivity analysis?\n3- The paper shows very good results on the family of VAR models. How generalizable is the FreqExit approach? Could it be applied to other hierarchical or next-scale generative models, or in other modalities such as audio, video or text,  that exhibit similar coarse-to-fine generation dynamics?\n4- The early-exit strategies outlined in Table 2 are manually predefined schedules. Have you considered learning an adaptive policy that determines the exit layer for each generation step based on properties of the input image or the intermediate representation? This could potentially lead to even better efficiency-fidelity trade-offs.\n5- The paper doesn't provide runtime savings benchmarks. For example FLOPS, latency for the different exit schedules. It would be good to show layer-wise FLOP savings, wall-clock latency improvements, and quality degradation as a function of easy exist. The results only show the speedup. I recommend including a plot showing performance vs. average depth or latency and reporting FLOPS or latency for several early exit schedules. \n6- Could the shared LM head limit the informational/representational diversity? There is no analysis or ablation of shared vs. per-later heads. It would be good to compare shared vs. separate heads for shallow and deep layers. Then analyze if deep leaders still learn richer presentations or get regularized.", "limitations": "Yes, the authors have adequately addressed the limitations in the NeurIPS checklist included with the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- This work is the first to implement the early-exist strategy in dynamic inference for VAR models. \n- The limitations of existing early-exit approaches are well explained and motivated. The empirical analysis of VAR's generation dynamics using DWT (Sec 3.1)  clearly illustrates the problem of frequency progression and layer instability\n- The paper is well written and provides good illustrations in Figure 1 to explain the overall methodology. The 3 proposed techniques are well-designed, with each component directly targeting an identified weakness of the baseline model. \n- The methodology used is novel and provides a principles approach to supervise models in the frequency space where high-frequency details often correlated with semantic correctness. The combination of early-exist loss, curriculum scheduling and temperature-aware distillation is well-motivated and has sound intuition. \n- The experiments are thorough, with comparisons to relevant baselines (CALM, LayerSkip, CoDe)\n- The ablation study in Table 3 validates the contribution of the proposed HF loss and FGSR modules.\n\nWeaknesses:\n- The scope of evaluation is limited to ImageNet 256x256. While the presented results are strong, the benefits of dynamic inference and especially inference acceleration are more profound at high resolution where the the computational needs are significantly higher. \n- The comparison with CALM required modifying the threshold to enable it to work. This kid of indicates the incompatibility of this baselines. It would have been better to frame this as a limitation focusing on why existing methods fail. \n- The mathematical explanation of the FGSR module was hard to follow especially the derivation of the closed form update in Equation 9. It would be good if the authors can simplify the explanation in the main body. \n- Minor detail: HH, LH, etc symbols are not defined. They might be obvious for some readers as high-frequency bands. It would be better to define them for clarity. \n- The inference-time criterion's for triggering early exists is not well explained in the paper \n- The paper doesn't provide runtime benchmarks or latency vs. accuracy tradeoff figures. \n- The shared LM head across layers could introduce informational or representational bottlenecks. This is not explained or discussed in the paper.\n- No deep evaluation of the impact of early exist on the quality of the generation of multi-token contexts. \n- No discussion on how this can be generalized to no visual tasks such as audio, video or NLP.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive and insightful feedback. We greatly appreciate your recognition of the strengths of our work and your valuable suggestions. We address each point in detail below.\n\n---\n\n**W1: Lack of High-Resolution Experiments**\n\n**W1-Ans:** We thank the reviewer for the insightful comment. We regret that we are unable to provide high-resolution results during the rebuttal period. Training at 512×512 resolution is prohibitively expensive. Running 80 epochs with VAR-d36 would take about **24 days on 8×A100 GPUs** (approximately 23% of the original VAR-d36 training time on A100 GPUs). We plan to include high-resolution experiments in the revision.\n\nNotably, **FreqExit is resolution-agnostic**: it adjusts layer depth per generation step without modifying patching or tokenization. Although higher resolutions produce larger token maps, the layer-wise structure within each step remains unchanged, allowing seamless application of our method. Moreover, our baseline **CoDe** is also only evaluated at **256×256** resolution.\n\nFinally, recent work such as **Infinity** [1] adopts the same next-scale generation paradigm and per-step Transformer structure at resolutions beyond 1024, further supporting the generalizability of FreqExit to high-resolution settings.\n\n---\n\n**W2: Limitations of CALM under Next-Scale Generation**\n\n**W2-Ans:**  We evaluated CALM-style exit criteria (*Softmax Max*, *Diff*, and *Entropy*) on VAR-d16 and VAR-d20 (see Table below), and found that all metrics remain far below typical thresholds, indicating the failure of confidence-based signals to determine viable exits.\n\n**Table 1: Layer-wise CALM Confidence Metrics Comparison**\n\n*Softmax Max = probability of the top-1 token; Softmax Diff = difference between top-1 and top-2 probabilities*\n\n| **Layer** | **Softmax Max(d-16)** | **Softmax Diff** | **Entropy** | **Softmax Max(d-20)** | **Diff** | **Entropy** | **Exit Threshold** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| 10 | 0.070 | 0.034 | 6.02 | 0.045 | 0.020 | 6.50 | No |\n| 12 | 0.027 | 0.010 | 6.91 | 0.018 | 0.006 | 6.23 | No |\n| 14 | 0.030 | 0.011 | 6.75 | 0.010 | 0.003 | 6.55 | No |\n| 16 | -- | -- | -- | 0.005 | 0.001 | 6.86 | No |\n| 18 | -- | -- | -- | 0.034 | 0.013 | 5.61 | No |\n\nThis is due to CALM’s core assumption that **representations stabilize across layers**, which does not hold in VAR. As next-scale prediction introduces high-frequency details in later steps, each layer continues to substantially update token representations. This leads to large inter-layer changes, also reflected in the low cosine similarity shown in Fig. 2(c).\n\nThese results will be included in the revision.\n\n---\n\n**W3:  FGSR Closed-Form Derivation**\n\n**W3-Ans:** We will revise the main text to improve the explanation of the FGSR module and simplify the derivation of the closed-form update in Eq. (9).\n\n---\n\n**W4: Lack of Definition**\n\n**W4-Ans:** All frequency band notations (e.g., HH, LH) will be defined upon first use to ensure clarity. Thanks for pointing out the issue!\n\n---\n\n**W5: Early Exit Strategy Clarification**\n\n**W5-Ans:** We report typical exit paths to enable comparison across typical Latency/FLOPs–FID trade-offs, serving as consistent baselines for evaluating inference cost and quality.\n\n---\n\n**W6: Lack of benchmark Figures**\n\n**W6-Ans:** We will include visual plots showing performance vs. average depth and latency in the revised version. These cannot be included in the rebuttal per the NeurIPS rebuttal guidelines.\n\n---\n\n**W7: Discussion of Shared LM Head**\n\n**W7-Ans:** We thank the reviewer for raising this point and are glad to clarify. Sharing the LM head promotes consistent representations and serves as a form of regularization. It also avoids significant parameter overhead. Adding separate heads from layer 10 onward increases parameters by 23.8%–29.9% (see Table 2 below) without improving performance.\n\n**Table 2: Overhead from adding individual LM heads to intermediate layers starting at layer 10 (excluding the final layer).**\n| **Model** | **Added Params** | **Param Overhead (%)** |\n| --- | --- | --- |\n| VAR-d16 | 73.7M | 23.8% |\n| VAR-d20 | 151.3M | 25.2% |\n| VAR-d24 | 287.2M | 27.8% |\n| VAR-d30 | 601.2M | 29.9% |\n\n---\n\n**W8: Impact of Early Exit on Multi-Token Generation**\n\n**W8-Ans:** We thank the reviewer for raising this important concern. FreqExit adjusts inference depth but **does not alter the underlying autoregressive generation scheme**. The reported FID reflects the quality of full images, thereby capturing coherence across token sequences. Moreover, Fig.4 in the appendix (Zero-Shot Inpainting) shows that **local consistency is well preserved**.\n\n---\n\n**W9: Generalization of FreqExit**\n\n**W9-Ans:** FreqExit targets next-scale autoregressive visual generation (e.g., image synthesis, inpainting) and already offers practical, resource-aware inference within vision. Because it operates purely at the architectural level, the same depth-adaptive strategy can transfer to hierarchical coarse-to-fine generators in NLP, audio, and video without changing the core training scheme.\n\n---\n\n**Q1: Inference-Time Exit Criterion**\n\n**Q1-Ans:** In our current paper, **early exits follow fixed depth paths chosen offline**. We benchmark several representative paths and report their **speed (latency / FLOPs) and quality (FID)** so readers can make direct trade-off comparisons. No confidence or entropy threshold is applied at inference time. These fixed paths serve as practical heuristics and reproducible baselines for typical deployment budgets. In future work, we will explore adaptive per-sample exit rules, such as confidence signals or lightweight policy networks, to further improve flexibility.\n\n---\n\n**Q2: Sensitivity Analysis of λ**\n\n**Q2-Ans:** FGSR uses two mechanisms for stable training: (i) a learnable sub-band gate γ_b and (ii) an auxiliary reconstruction loss. The hyper-parameter λ scales only the orthogonal alignment term L_align, regularizing sub-band projections **without perturbing the main gradients**. We have added a sensitivity analysis on VAR-d16 that varies λ and confirms training remains stable across a broad range of values.\n\n**Table 3** reports FID scores at two checkpoints (Epoch 8 and Epoch 16) for three λ values and three early-exit layers.\n| λ | L10 (Epoch 8) | L12 (Epoch 8) | L14 (Epoch 8) | L10 (Epoch 16) | L12 (Epoch 16) | L14 (Epoch 16) |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0.05 | 26.8 | 20.2 | 7.31 | 18.3 | 12.6 | 6.26 |\n| 0.10 | 26.8 | 20.2 | 7.45 | 18.3 | 12.7 | 6.24 |\n| 0.20 | 27.1 | 20.0 | 7.37 | 18.1 | 12.6 | 6.16 |\n\nThe differences across λ in [0.05, 0.20] are marginal, indicating that FGSR is **insensitive** to reasonable choices of λ. We therefore set λ = 0.1.\n\n---\n\n**Q3: Applicability of FreqExit Beyond VAR**\n\n**Q3-Ans:** FreqExit is an architectural mechanism: every layer learns to emit a full sample, and inference depth is selected on demand. This layer-centric design is **token-agnostic and thus transferable**. We have already confirmed its effectiveness on visual generation with VAR models. We believe the same depth-adaptive scheme could be applied to other hierarchical or next-scale generators, including those in NLP, audio, and video. We will explore these modalities in future work.\n\n---\n\n**Q4: Adaptive Early-Exit Strategy**\n\n**Q4-Ans:** We agree this is a valuable suggestion. FreqExit presently uses a few fixed exit paths to cover typical latency versus FID trade offs. Learning a per sample exit rule by using confidence cues is a promising next step. We plan to investigate it in future work.\n\n---\n\n**Q5: Runtime Savings Benchmarks**\n\n**Q5-Ans:** We clarify that Table 1 reports latency, throughput, and GFLOPs together with FID for each exit schedule, providing a comparison of speed-quality trade offs. To make these trends clearer, We will include a performance vs. latency plot in the revision.\n\n---\n\n**Q6: Impact of  Shared and Sperate LM Heads**\n\n**Q6-Ans:**  A shared LM head keeps representations consistent, serves as implicit regularization, and avoids the extra parameters incurred by per-layer heads. As shown in Table 2, adding separate heads from layer 10 onward would raise parameters by roughly 24%–30%. Consistent with prior work [2-3], we therefore keep a single shared head.\n\n---\n\n**Reference**\n\n[1] Han J, Liu J, Jiang Y, et al. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 15733-15744.\n\n[2] Elhoushi M, Shrivastava A, Liskovich D, et al. LayerSkip: Enabling early exit inference and self-speculative decoding[J]. arXiv preprint arXiv:2404.16710, 2024.\n\n[3] Liu F, Tang Y, Liu Z, et al. Kangaroo: Lossless self-speculative decoding for accelerating llms via double early exiting[J]. Advances in Neural Information Processing Systems, 2024, 37: 11946-11965."}, {"review_id": "5ZFZ1YIzZ3", "reviewer": "Reviewer_NPSH", "summary": "FreqExit proposes a unified training framework to accelerate Visual AutoRegressive Modeling through frequency-aware supervision. They observe that high-frequency details, which are important for image perception, appear in the later stages of the generation, and there is an instability in the intermediate representations. To solve these problems and encourage early exit, they use a layer and time-dependent dropout, early exit loss, and a curriculum scheduling. They also introduce a frequency-gated residual connection to learn high-frequency details at earlier stages without affecting the main architecture. Through empirical analysis, they show that they are able to achieve almost 2x speed up in image generation with minimal degradation in quality.", "questions": "1. Other datasets and different resolutions can provide more strength to the generalizability of the methods used in the paper\n2. Though the appearance of high-frequency terms in the later stages of generation is observed empirically, the reasons behind it are not made clear in the paper\n3. The details of the wavelet transform used is not specified in the paper. What about different frequency analysis like the frequency or cosine transforms?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The authors introduce a suite of techniques to accelerate image generation in VAR modeling, tackling multiple challenges\n2. The observation frequency characteristics developing over time are new and helped guide their methodology for FGSR and high-frequency consistency loss\n3. Curriculum-based early exit supervision encourages the earlier layers of the model to learn robust representations in the early layers, thereby enabling speed-up\n4. Post training, their framework allows dynamic early-exit strategies trading off generation quality and speedup.\n\nWeaknesses:\n1. Their hypothesis that high frequency terms improve perception quality runs counter to the idea of natural image compression techniques, such as JPEG, that prune high frequency terms while keeping low frequency terms\n2. Their distilling process needs two models, which increases training costs for large-scale models\n3. All the experiments are shown on ImageNet generation, lacking diversity in the evaluations.", "was_revised": true, "final_justification": "Some of my concerns were addressed in the rebuttal by the authors; I am keeping my favorable score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we provide point-by-point responses to the comments and questions.\n\n---\n\n**W1: Role of High Frequencies**\n\n**W1-Ans:** We thank the reviewer for raising this point. To clarify upfront: image compression (e.g., JPEG) and visual image generation **pursue fundamentally different goals**. Compression removes redundant information from complete images to reduce bitrate. In contrast, generative models synthesize content from scratch and **rely on high-frequency details** to achieve perceptually convincing results. In generative methods such as GANs [1] and diffusion models [2], insufficient high-frequency modeling typically leads to over-smoothing and loss of texture, degrading visual quality.\n\nOur work investigates this issue in autoregressive visual generation, with a focus on **next-scale generation**, where images are produced progressively across resolution scales. We identify a **previously underexplored frequency pattern**: low-frequency components dominate early stages, while high-frequency content becomes dominant in later steps. This frequency inconsistency limits the effectiveness of confidence-based, training-free early-exit strategies, which assume uniform information distribution across layers. To address this, we propose FreqExit, a frequency-adaptive training framework that explicitly aligns learning dynamics with this progression. It integrates frequency-aware supervision and a frequency-gated residual mechanism to promote earlier modeling of high-frequency features, enabling more effective early exits with minimal loss in image quality.\n\n---\n\n**W2: Distillation Overhead**\n\n**W2-Ans:** In our setup, the teacher model is frozen and performs only a single forward pass per batch. Its activations are not stored, and memory is immediately released after inference, resulting in minimal resource usage. To quantify this, we conducted a profiling experiment using PyTorch’s `torch.cuda.max_memory_allocated()` to capture peak memory during the teacher’s forward pass. As shown in Table 4, the teacher model accounts for only **5.3%** of peak memory usage and **8.8%** of total training time. \n\n**Table 4: Resource Usage Breakdown During Distillation Training**\n\n| Component                      | Peak Memory Usage (%) | Time Usage (%) |\n|-------------------------------|------------------------|----------------|\n| Teacher Model (Forward Only)  | 5.3%                   | 8.8%           |\n| Student Model (Training Total)| 94.7%                  | 91.2%          |\n\n\nThis confirms that our distillation strategy imposes **minimal overhead**. The training process is also fully compatible with widely-used optimization tools such as mixed-precision training, ZeRO, and DeepSpeed, which can further reduce the cost if necessary. The modest additional training cost is acceptable, and this frozen-teacher strategy is widely adopted in model compression research [3-4].\n\n---\n\n**W3: Dataset and Resolution Generalization**\n\n**W3-Ans:**  We chose ImageNet-256 as our evaluation benchmark because existing VAR-style models are trained and publicly released on this dataset. This ensures reproducibility and allows for direct, fair comparison with baseline methods. In addition, ImageNet’s 1,000-category diversity provides rich variations in texture, scale, and semantics, making it a strong testbed for evaluating convergence, detail recovery, and long-range dependencies.\n\nFreqExit is **resolution- and dataset-agnostic**, as it operates solely along the depth axis by adjusting the number of active layers at each generation step, without modifying the patching scheme, tokenization process, or data modality. For example, in VAR models, increasing resolution from 256×256 to 512×512 enlarges the token maps at each step (e.g., step-wise `patch_nums` from (1, 2, …, 16) to (1, 2, …, 32)), but the per-step computation graph remains unchanged, allowing FreqExit to scale without modification. Recent work such as Infinity [5] has already extended VAR-style architectures to 1024+ resolutions while preserving their core structure, indirectly supporting the transferability of our method.\n\nWe agree that broader evaluation across datasets and resolutions would further demonstrate generalizability. We plan to extend FreqExit in these directions, but due to time and resource constraints, we are unable to provide new experiments during the rebuttal phase and appreciate the reviewer’s understanding.\n\n---\n\n**Q1: Generalization of FreqExit**\n\n**Q1-Ans:**   All publicly available VAR baselines are trained on ImageNet at 256 × 256, so we used the same setting for reproducible, fair comparison. FreqExit itself is resolution- and dataset-agnostic because it only adjusts the active‐layer depth. Training higher-resolution models (e.g., 512 × 512) would take about 24 days on 8 × A100 GPUs, which is infeasible during the rebuttal period. We therefore defer larger-scale and cross-dataset experiments to future work.\n\n---\n\n**Q2: Discussion on High-Frequency Behavior**\n\n**Q2-Ans:** VAR generates images in a scale-by-scale manner, where each step operates on a low-resolution token map that is upsampled before the next step. In early steps, the coarse spatial resolution limits the range of frequency components that can be represented—only large-scale structures and smooth regions (i.e., low-frequency information) can be reliably synthesized. This limitation aligns with the Nyquist sampling principle, which states that **higher-frequency content requires denser spatial sampling**. As the token map is upsampled, the spatial granularity increases, allowing later layers to generate finer textures and high-frequency details. This leads to a natural “structure-first, detail-later” generation process.\n\nFrom an information-theoretic perspective, natural images typically follow a $1/f$ power spectrum, where most of the signal energy is concentrated in the low-frequency range [6]. During training, cross-entropy loss tends to reduce large-magnitude errors first—primarily from low-frequency components. As structural errors diminish, high-frequency residuals become more prominent in the gradient signal, which explains the gradual emergence of fine details in later stages.\n\n---\n\n**Q3: Wavelet Details**\n\n**Q3-Ans:** We use a single-level 2D Haar wavelet transform (`wave=‘haar’`) in FreqExit. Each $2\\times2$ spatial block in the token map is linearly projected into four subbands: LL (low-frequency structure) and LH/HL/HH (directional high-frequency details). This fixed-stride decomposition preserves a one-to-one spatial correspondence between inputs and outputs, allowing frequency-aware gating and reconstruction in a unified coordinate system without extra alignment.\n\nCompared to global Fourier transforms or block-wise DCT, Haar wavelets are better suited to our setting for three main reasons:\n\n (1) **Spatial-frequency alignment**: FFT and DCT distribute frequency information globally or within local blocks, weakening spatial localization. In contrast, Haar wavelets retain both **spatial and frequency information**, which is better suited for frequency-aware control in spatially structured generation; \n\n(2) **Hierarchical consistency**: The structure-to-detail nature of next-scale generation in VAR naturally aligns with Haar’s hierarchical LL → LH/HL/HH decomposition, in contrast to the less localized zig-zag ordering used in DCT.\n\n(3) **Computational simplicity**: Haar wavelets are computationally lightweight and easy to integrate.\n\nOverall, Haar-DWT offers aligned coordinates, hierarchical structure, and low overhead, making it a practical fit for our frequency-aware training design. We will include this discussion in the paper.\n\n ---\n\n**Paper Formatting Concerns**: We thank the reviewer for pointing out the typos and the formatting issue with Figure 3. We will correct these issues in the paper.\n\n---\n\n**Reference**\n\n[1] Zhang B, Gu S, Zhang B, et al. Styleswin: Transformer-based gan for high-resolution image generation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 11304-11314.\n\n[2] Yang X, Zhou D, Feng J, et al. Diffusion probabilistic model made slim[C]//Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2023: 22552-22562.\n\n[3] Huang T, You S, Wang F, et al. Knowledge distillation from a stronger teacher[J]. Advances in Neural Information Processing Systems, 2022, 35: 33716-33727.\n\n[4] Jin Y, Wang J, Lin D. Multi-level logit distillation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 24276-24285.\n\n[5] Han J, Liu J, Jiang Y, et al. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 15733-15744.\n\n[6] Torralba A, Oliva A. Statistics of natural image categories[J]. Network: computation in neural systems, 2003, 14(3): 391."}, {"review_id": "voOEaENyHz", "reviewer": "Reviewer_sjr1", "summary": "This paper proposes a dynamic inference framework tailored for the next-scale prediction (VAR) paradigm by integrating an early-exit mechanism. The approach achieves up to 2× speedup with minimal performance degradation.", "questions": "See weaknesses, more concise and clear explanations and comprehensive analysis are needed.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is clearly written and well-structured. It provides a thorough analysis of the frequency-domain properties and intermediate feature distributions in VAR, and designs a fine-tuning strategy for early-exit accordingly. Compared to prior acceleration methods for VAR, this approach achieves better performance under higher speed-up ratios.\n\nWeaknesses:\n\n1. In Equation (2), the meaning of $Y$ is not explained. The term $\\tilde{e}$ seems to be a dynamic weight, but a more detailed explanation is needed. Moreover, is the total loss simply a summation of all the losses proposed in the Method section?\n2. Although many losses are introduced in the Method section, the ablation study only investigates the HF and FGSR losses. It would be helpful to see how other components affect performance—for instance, the role of the $\\tilde{e}$ and gate terms in the early-exit loss, and the temperature scheduling in Equation (3).\n3. The overall framework feels a bit complex, making it hard to pinpoint which parts contribute most to the performance. A more concise and clearer explanation would greatly improve readability.\n4. If possible, please provide some visual comparison with the baseline model (e.g., CoDe and LayerSkip).", "was_revised": true, "final_justification": "This work conducts thorough experiments on each component and systematically compares with existing baselines, clearly demonstrating the superiority of the proposed method. I believe it is a valuable contribution to the community.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we provide point-by-point responses to the comments and questions.\n\n---\n**W1: \"In Equation (2), the meaning of $Y$ is not explained. The term $\\tilde{e}$ seems to be a dynamic weight, but a more detailed explanation is needed. Moreover, is the total loss simply a summation of all the losses proposed in the Method section?\"**\n\n**W1-Ans**: We thank the reviewer for pointing this out. We provide the following clarifications and will annotate the definitions in the revised version:\n\n- Definition of $Y$: $Y$ denotes the ground-truth token sequence obtained by encoding the target image with a pre-trained VQ-VAE, which serves as the supervision target for predictions.\n\n- Definition of $\\tilde{e}(t,\\ell)$: $\\tilde{e}(t,\\ell)$ follows the formulation in Eq. (2), and is a normalized dynamic weight that determines the contribution of each intermediate layer $\\ell$ to the early-exit loss at training step $t$:\n\nwhere \n\n- $C(t,\\ell)$ is a curriculum gate that selects a subset of layers to be supervised at each training step. Specifically, all layers are divided into $R$ blocks (with $R=4$ for VAR-d20), and one block is activated per step in a round-robin schedule: block_id = $t$ mod $R$.\n\n- $e(\\ell)$ is a depth-dependent static weight defined as: (1) If $\\ell < L-1$ , $e(\\ell) =     \\sum_{i=0}^\\ell i$, (2) if $\\ell= L-1$ , $e(\\ell) =  (L{-}1) + \\sum_{i=0}^{L{-}2} i $. This formulation ensures that deeper layers receive quadratically increasing weights, with additional emphasis on the final layer to promote more comprehensive supervision. This design corresponds to Eq. (14) in Appendix A.1.\n\n- The normalization ensures that the sum of $\\tilde{e}(t,\\ell)$ over all active layers is 1. This design emphasizes deeper layers during training, ensuring that updates to shallow layers do not compromise the performance of deeper ones.\n    \n**Total loss structure:** The total loss is a weighted combination of the main task loss (CE + KD), the high-frequency (HF) loss, and the FGSR loss. The main loss is applied to both the final and intermediate layers using $\\tilde{e}(t,\\ell)$. The HF loss is activated after 25% of training and dynamically scaled (see Appendix B, Eq. (20)), while the FGSR loss is applied with a fixed weight of 5.0, along with an orthogonality term weighted by 0.1.\n\nDetails on the layer-wise loss weights and training schedule can be found in Appendix B (FreqExit). We hope this clarifies the formulation and thank the reviewer again for the valuable feedback.\n\n---\n\n**W2: \"Although many losses are introduced in the Method section, the ablation study only investigates the HF and FGSR losses. It would be helpful to see how other components affect performance—for instance, the role of the $\\tilde{e}$ and gate terms in the early-exit loss, and the temperature scheduling in Equation (3).\"**\n\n**W2-Ans**: The roles of the dynamic weighting $\\tilde{e}(t,\\ell)$ and the gating mechanism $C(t,\\ell)$ have been detailed in our response to Comment 1. Briefly, $e(\\ell)$ emphasizes deeper layers to protect their optimization from shallow-layer interference, while $C(t,\\ell)$ reduces per-step computational cost and gradient conflict by supervising only a subset of layers in a rotational schedule. Together, they help stabilize training and enhance early-exit performance.\n\nWe also provide further clarification on temperature scheduling in Eq. (3). The temperature $T_\\ell$ decreases linearly from 4.0 at shallow layers to 1.0 at the final layer.  This design is based on the observation that shallow layers have limited semantic capacity during early training. A higher temperature smooths the teacher distribution, providing softer and more tolerant supervision targets. As depth increases, semantic representations become more refined; hence, lower temperatures help retain sharper predictions and improve alignment. This strategy aligns with prior work on progressive distillation and improves intermediate representation quality.\n\n---\n\n**W3: \"The overall framework feels a bit complex, making it hard to pinpoint which parts contribute most to the performance. A more concise and clearer explanation would greatly improve readability.\"**\n\n**W3-Ans**: We thank the reviewer for the helpful suggestion. To clarify the contribution of individual components, we conducted additional ablation studies to further isolate the effects of the FGSR module and temperature scheduling.\n\n\n----\n\n**W4: \"If possible, please provide some visual comparison with the baseline model (e.g., CoDe and LayerSkip).\"**\n\n**W4-Ans**: We thank the reviewer for the helpful suggestion. We agree that visual comparisons with baseline methods would provide valuable qualitative insight. Due to rebuttal constraints, we are unable to include figures at this stage. However, we will incorporate representative visual comparisons in the revised version of the paper to highlight the differences in generation quality."}, {"review_id": "cRBErRtIC4", "reviewer": "Reviewer_paNB", "summary": "This paper proposes a unified training framework to address the challenges of applying dynamic inference techniques to next-scale autoregressive generation without altering the architecture or compromising output quality. It introduces two components in the curriculum-based training strategy: 1) a high-frequency consistency loss and 2) a lightweight frequency-gated self-reconstruction module, achieving 1.3x speedup without decrease in generation quality.", "questions": "1. Can you explain more about what the assumptions of dynamic inference are: semantic stability, monotonic locality, and granularity (line 38); and why these assumptions break in the next-scale generation paradigm? It would strengthen the motivation by explaining the need for a new training strategy for acceleration. \n2. How does removing the low-frequency content affect quality? This comparison should be made to support Observation 2 in Section 3.1. Even if this claim is true, why is it important for improving early-exit rather than output quality (line 126)?\n3. What is the setting of each strategy in Table 2? Which component enables the early exit? Is it mainly because of the layer dropout?", "limitations": "Yes, the authors include limitations in the supplementary materials. I’m currently concerned about the extensibility to other autoregressive modes beyond the next-scale paradigm, and that the high computational cost may outweigh the improvement in efficiency.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The speedup results are decent: around 30% without quality drop.\n2. The observation that the high frequency component matters is quite inspiring.\n\nWeaknesses:\n1. Since the proposed acceleration strategy needs to retrain the VAR model from scratch, I am worried that the high computational demand overshadows its gain in efficiency (~30%), and this specifically tailored training strategy may also limit its broader use to other VAR models.  \n2. The method needs to be evaluated on a broader range of VAR models before it can be claimed as a “unified” training framework.\n3. The motivation is not strong enough. The observations made can hardly inform the proposed method. For instance, it’s unclear to me why the three observations, which are made during inference, motivate the need for a new training strategy.\n4. It is quite hard to identify which loss function among Eq.(2)(3)(6)(7)(11)(12) contributes the most and may need ablation studies on this. More task-specific insights could improve the logic flow a lot.", "was_revised": true, "final_justification": "The authors' justifications make me realize this is a fine-tuning strategy to improve the generation efficiency of VAR model, which is a technique studied before, thus clearing up my initial concern about the need to do so.\n\nThe authors' clarifications on the combination of loss functions are valid.\n\nFinally, the authors acknowledge their initially flawed narrative and make a reasonable refinement without over-claiming or causing too much confusion.  \n\nBased on the above efforts, I would recommend a borderline accept rating.", "rebuttal": "We sincerely thank you for the recognition and detailed comments. Below, we address each point and clarify the motivation, assumptions, and design choices of our method, hoping to better convey its contributions and resolve your concerns.\n\n**W1: \"Since the proposed acceleration strategy needs to retrain the VAR model from scratch ...\"**\n\n**W1-Ans:** In practice, **FreqExit does not require retraining the VAR model from scratch**. For example, we start from the official VAR-d20 checkpoint (trained for 250 epochs on ImageNet-1K) and fine-tune for only **80 additional epochs (~30% extra cost)** to enable early-exit capability. Our rotating supervision further reduces computation by selectively activating losses across layers. Thus, the added training overhead is modest and far from the cost of full model training.\n\nThis small investment yields sustained inference benefits: ~30% latency reduction without FID loss, and up to 2× speedup with mild quality trade-offs. FreqExit is deployment-friendly and allows runtime adaptation to diverse resource budgets.\n\nFurthermore, FreqExit is the first to apply early exit in next-scale generation, revealing a progressive low-to-high frequency pattern and addressing frequency conflicts via wavelet-based supervision. Our training strategy is general to transformer-style architectures with layered outputs and could extend to autoregressive models in other domains where frequency-phase behaviors are common.\n\n---\n\n**W2: \"The method needs to be evaluated on a broader range of VAR models before it can be claimed as a “unified” training framework.\"**\n\n**W2-Ans:**  We supplement our analysis with additional experiments on both VAR-d16 and VAR-d24 models. As shown in the tables, our method significantly reduces the FID scores of intermediate layers, demonstrating its effectiveness in enabling early exits with high-quality outputs.\n\n**Table 5: VAR-d16 experimental results**\n\n| Layer   | Epoch 0 | Epoch 16 | Epoch 32 | Epoch 48 |\n|---------|---------|----------|----------|----------|\n| Layer10 | 120.9   | 26.82    | 18.27    | 10.37    |\n| Layer12 | 90.83   | 20.22    | 12.68    | 8.11     |\n| Layer14 | 12.77   | 7.40     | 6.24     | 5.61     |\n\n\n**Table 6: VAR-d24 experimental results**\n\n| Layer   | Epoch 0 | Epoch 4 | Epoch 8  |\n|---------|---------|---------|----------|\n| Layer10 | 134.0   | 68.95   | 27.56    |\n| Layer12 | 129.9   | 60.80   | 17.84    |\n| Layer16 | 127.6   | 41.19   | 8.69     |\n| Layer20 | 85.43   | 21.72   | 5.53     |\n\n---\n\n**W3: \"The motivation is not strong enough.\"**\n\n**W3-Ans:** \n\n**Table 7: Token Matching Rate with Final Transformer Layer Output (VAR-d20, Last Step)**  \nAt the last generation step, we apply greedy sampling to each layer’s output and compute the per-token match rate with the final layer’s output tokens.\n\n| Layer | Average Match Rate (%) |\n|-------|------------------------|\n| 14    | 0.51                   |\n| 16    | 0.70                   |\n| 17    | 3.63                   |\n| 18    | 25.2                   |\n| 19    | 59.6                   |\n\n**Motivation:** Our motivation stems from a mismatch between dynamic inference assumptions and next-scale generation behavior in VAR models. CALM-style confidence signals (e.g., Softmax Max, Entropy) remain far below usable thresholds, revealing that training-free heuristics are unreliable. This is because assumptions like semantic stability, monotonic locality, and granularity consistency break down. FreqExit addresses this by introducing training-time supervision to enable reliable early exits.\n\n**Semantic stability:**  assumes that token representations converge across deeper layers. However, as shown in our analysis (Table 1 in Rebuttal ), softmax confidence remains low and unstable throughout the layers of VAR models. This is likely due to the increasing resolution and complexity of token maps, which evolve from coarse global structures to fine-grained high-frequency content. This continual refinement prevents logits from stabilizing and invalidates the assumption that later layers produce redundant outputs.\n\n**Monotonic locality:** assumes that tokens generated at each step can serve as a fixed and verifiable prefix. In next-scale generation, however, token maps are not immutable—tokens generated at step $i$ are frequently updated in step $i{+}1$ through upsampling and refinement. To validate this, we measured the token-wise matching rate between intermediate layers and the final output at the last generation step of VAR-d20. As shown in Table 7, most layers between 10 and 17 yield match rates below 5%, and even Layer 19 reaches only ~60%, confirming the instability of intermediate outputs and the failure of prefix-based verification.\n\n**Granularity consistency:** assumes that dynamic inference operates over fixed, bounded computation units (e.g., one token or one layer). This assumption breaks in next-scale generation due to VAR’s hierarchical structure: each step generates a $p_n \\times p_n$ token map, which is upsampled to full resolution and then downsampled for the next step. These tokens are intermediate and overwritten, leading to unstable computation units and growing step-wise cost. As a result, fine-grained control (e.g., per-token or per-layer gating) becomes ineffective.\n\nWe believe this unified analysis and supporting experiments clearly demonstrate that effective dynamic inference in next-scale generation requires rethinking beyond inference-time heuristics. \n\n---\n\n**W4: \"It is quite hard to identify which loss function...\"**\n\n**W4-Ans:**  Our ablation（Table 8-10）shows that removing either HF-loss or FGSR significantly degrades early-exit quality (higher FID), while final-layer outputs are only marginally affected—supporting our claim that high-frequency mismatch is the main constraint on early-exit performance.\n\n**Table 8: VAR-d20 with Early Exit loss**\n| Layer   | Epoch 40 | Epoch 48 | Epoch 56 | Epoch 64 |\n|---------|----------|----------|----------|----------|\n| Layer 8 | 22.73    | 20.53    | 19.40    | 18.52    |\n| Layer 12| 6.87     | 5.52     | 5.31     | 5.21     |\n| Layer 16| 3.64     | 3.55     | 3.45     | 3.42     |\n\n**Table 9: VAR-d20 with Early Exit loss and HF loss**\n| Layer    | Epoch 40 | Epoch 48 | Epoch 56 | Epoch 64 |\n|----------|----------|----------|----------|----------|\n| Layer 8  | 16.22    | 15.43    | 14.67    | 13.92    |\n| Layer 12 | 4.83     | 5.32     | 4.93     | 4.72     |\n| Layer 16 | 3.41     | 3.38     | 3.10     | 3.06     |\n\n**Table 10: VAR-d20 with Early Exit loss, HF loss, and FGSR loss (FreqExit)**\n| Layer    | Epoch 40 | Epoch 48 | Epoch 56 | Epoch 64 |\n|----------|----------|----------|----------|----------|\n| Layer 8  | 13.20    | 12.54    | 11.74    | 10.90    |\n| Layer 12 | 4.59     | 4.46     | 4.37     | 3.92     |\n| Layer 16 | 3.27     | 3.06     | 2.91     | 2.87     |\n\n---\n\n**Q1: \"Can you explain more about what the assumptions of dynamic inference are: semantic stability, monotonic locality, and granularity...\"**\n\n**Q1-Ans:**  VAR models exhibit unstable intermediate tokens, overwritten outputs across steps, and variable computation granularity, undermining conventional early-exit heuristics. These failures motivate our training-time solution. Please refer to W3-Ans for full analysis and evidence.\n\n---\n\n**Q2: \"How does removing the low-frequency content affect quality?\"**\n\n**Q2-Ans:**  We clarify the roles of low- and high-frequency components in image generation: low-frequency (LL) sub-bands capture global semantics, while high-frequency (LH/HL/HH) sub-bands encode fine textures. Evaluating VAR-d16/d20/d24/d30, we found that removing low-frequency components leads to **FID > 200**,  indicating **major semantic distortion**, whereas removing high-frequency components mainly affects perceptual quality (Fig.~2(b)). This confirms that low-frequency governs semantics, while high-frequency governs realism.\n\nAs shown in Fig.~2(a), high-frequency energy rises steadily during generation, reaching ~50% in later stages. This makes high-frequency mismatch a key source of residual error, as shallow layers struggle to capture the needed fine details. In such cases, enhancing low-frequency features helps little, while improving high-frequency modeling yields significant FID gains. FreqExit introduces two high-frequency enhancement strategies:\n\n**(1) HF Consistency Loss:** aligns the wavelet-domain high-frequency outputs of shallow layers with the final output.\n\n**(2) FGSR Loss:** provides stage-aware, spectrum-adaptive supervision, guiding the model to adjust to frequency behavior at each generation step and improve texture reconstruction.\n\nOur ablation（Table 8-10）shows that removing either HF-loss or FGSR significantly degrades early-exit quality (higher FID), while final-layer outputs are only marginally affected. In conclusion, Observation 2 highlights a critical insight: once low-frequency information is learned through early-exit supervision, remaining quality gaps are due to insufficient high-frequency modeling. \n\n\n---\n\n**Q3: \"What is the setting of each strategy in Table 2?\"**\n\n**Q3-Ans:**  The strategies in Table 2 represent different early-exit paths applied to the same trained FreqExit model. Each is a 10-dimensional vector specifying the number of layers used at each generation step. For example,\n“7” = [16, 12, 10, …, 10] uses 16 layers at the first step, 12 at the second, and 10 for the rest.\nThis enables flexible quality–latency trade-offs without retraining or extra modules.\n\nThis early-exit capability is not achieved by Layer Dropout. It is enabled by our training design, which equips each layer with independent generation ability via supervised learning. Layer Dropout is only used during training as a regularizer to improve robustness and has no effect at inference time."}]

Author rebuttal:

Meta-review: This paper introduces FreqExit, a novel and well-motivated framework for enabling efficient early-exit inference in Visual Autoregressive (VAR) models. By identifying that VARs' coarse-to-fine generation process violates the assumptions of standard dynamic inference methods—specifically through the late emergence of high-frequency details—the authors propose a technically sound, frequency-aware supervision strategy. The work received a consensus for acceptance (final scores: 5, 4, 4, 4) following a highly constructive discussion phase.

The recommendation to accept is based on the paper's novelty in addressing an unsolved problem and its strong technical execution. Reviewers unanimously recognized the insightful analysis in the work and found the proposed method to be effective, demonstrating significant inference speedups. Initial concerns regarding motivation, the contribution of individual components, and the paper's positioning were thoroughly resolved through rebuttal and discussion. The authors provided detailed clarifications, new ablation studies during the review period, and committed to refining the paper's claims, which was instrumental in convincing initially borderline reviewers.

While the evaluation is limited to a single dataset and resolution, this is an acceptable limitation given the high computational cost of training these models. The engagement during the review process has substantially improved the paper and solidified its contribution.
Decision: Accept (poster)