SoftCFG: Uncertainty-guided Stable Guidance for Visual Autoregressive Model
Authors: Dongli Xu
Aleksei Tiulpin
Matthew B. Blaschko
Venue: ICLR 2026 Poster (2026)
Keywords: visual autoregressive model
Primary area: generative models
TL;DR: We propose SoftCFG, an uncertainty-guided and context-aware regularizer for AR image generation that stabilizes classifier-free guidance and achieves SOTA FID of ARs on ImageNet $256\times256$ without extra training or changes.

Abstract: Autoregressive (AR) models have emerged as powerful tools for image generation by modeling images as sequences of discrete tokens. While Classifier-Free Guidance (CFG) has been adopted to improve conditional generation, its application in AR models faces two key issues: guidance diminishing, where the conditional–unconditional gap quickly vanishes as decoding progresses, and over-guidance, where strong conditions distort visual coherence. To address these challenges, we propose SoftCFG, an uncertainty-guided inference method that distributes adaptive perturbations across all tokens in the sequence. The key idea behind SoftCFG is to let each generated token contribute certainty-weighted guidance, ensuring that the signal persists across steps while resolving conflicts between text guidance and visual context. To further stabilize long-sequence generation, we introduce Step Normalization, which bounds cumulative perturbations of SoftCFG. Our method is training-free, model-agnostic, and seamlessly integrates with existing AR pipelines. Experiments show that SoftCFG significantly improves image quality over standard CFG and achieves state-of-the-art FID on ImageNet 256 × 256 among autoregressive models.

Published as a conference paper at ICLR 2026
S UNCERTAINTY-GUIDED STABLE GUID- ANCE FOR VISUAL AUTOREGRESSIVE MODEL
Dongli Alekseei Tiulpin 2.3 Matthew B. Blaschko' KK Leuven University of Oulu Weill Cornell Medicine
Aleksei
ABSTRACT
Autoregressive (AR) models have emerged as powerful tools for image genera- tion by modeling images as sequences of discrete tokens. While Classifier-Free Guidance (CFG) has been adopted to improve conditional generation, appli- cation in AR models faces two key issues: guidance diminishing, where the con- ditional-unconndtionallll gap quickly vanishes as decoding progresses, and over- guidance, where strong conditions distort visual coherence. To address these challenges, we propose SoftCFG, an uncertainty-guides inference method that distributes adaptive perturbations across all tokens the sequence. The key idea behind SofICFG is to let each generated token contribute certainty-weighted guid- ance, ensuring that the signal persists across steps while resolving conflicts be- tween text guidance and visual context. To further stabilize long- sequence gener- ation, we introduce Step Normalization, which bounds cumulative perturbations of SoftCFG. Our method is training-free, model-agnostic, and seamlessly inte- grates with existing AR pipelines. Experiments show that SofCCG significantly improves image quality over standard CFG and achieves state-of- the-art FID on ImageNet 256 256 among autoregressive models. Codebasc is available.
Conditional Token
Unconditional Imput
Roseline
+SenCRG
AR Model
Generated Token
2,45
Order)
predict step Perturbation Certainty Degree
Unknown Token
Guidance Flow
16
1.4
(a) Classifier-free Guidance
(b) Soft Classifier-fre- Guidance
(c) FID comparison of CFG and SofCCG
Comparison of standard CFG and SoftCFG. (a) Standard CFG for AR replaces the first class token with an empty token. (b) SoftCIG scales the perturbation by model uncertainty, yielding smoother, more informative guidance. (c) On ImageNet-256 x 256, SoftCFG consistently improves FID across diverse AR models vs. baseline sampling and standard CFG.
Figure 1:
INTRODUCTION
Visual Autoregressive (AR) models (Van den Oord et al. 2016 Chang al. 2022 Sun et al. 2024) formulate image generation as next-token prediction task over sequences of discrete visual to- kens, typically obtained via vector-quantized autoencoders (Van Den Oord et al. 2017). Given sequence of previously generated tokens, a visual AR model predicts the next one with decoder- only transformer, following the same autoregressive formulation that underlies large language mod- els (LLMs) (Touvron et al., 2023). This unified design enables conceptual and architectural align- ment between vision and language, offering simplicity, scalability, and potential for cross-modal
integration (Wu et al. 2025a Xie et al. 2025 Xin et al. 2025).
"Corresponding Author: Longlixx@@gaill.om

---

Published as a conference paper at ICLR 2026
CFG
SofCCG
CFG
SoftCFG
CFG
SoftCFG
Input: photo white banana black elephant"
Input: photo yellow and black motorcycle"
Input: highly detailed, 3D-rendered, stylized representation of travellers, 40 year man a year old woman
Imput: "a photo of pink dining table and black sandwich"
Input: photo orange cow purple sandwich"
Input: "... single, ornate globe, its vibrant shades of green, bluc, and brown continents contrasting beautifully against the deep hlue the oceans.
Figure 2: Comparison images generated by standard Classifier-Free Guidance (CFG) and our proposed SoftCFG on LuminaGPT-8B Xin et al. (2025). Unlike CFG, which applies the same ditional offset regardless of generation history, SoftCFG adaptively incorporates uncertainty from the already generated content. As result, SoftCFG effectively reduces unreasonable artifacts, such as motorcycles collapsing into tangled shapes, extra trunks emerging from nowhere, or redundant hands in humans. This demonstrates that by aligning guidance with generated content, SoftCFG yields more coherent and visually plausible generations.
Despite these advances, the qual- ity of AR-based image generation still heavily influenced by the in- ference process. Recent work (Sun al., 2024) has introduced classifier- free guidance (CFG) (Ho & Sali- mans 2022) into the AR setting, aim- ing to improve conditional genera- tion by amplifying the difference be- tween conditional and unconditional predictions during inference. As il-
Normalized Entropy Over Steps
1.0
0.8
Nomdlied Entey
0.6
lustrated in Fig. I (a), in autoregres- sive models, CFG typically ap- plied by generating two parallel pre- dictions at each generation step: one conditioned on the target class (by in- cluding a class token or embedding at the start of the input sequence), and one unconditional version where the first class token is replaced with an empty token. This technique, orig- inally developed for diffusion mod- els, has shown promising results (Yu et al. 2024a Ren et al., 2025 (W
Benelne
Pemutiation
9.0
Gunlance (Boaling- Pomerhotimn)
50
100
150
200
250
Figure 3: Diminishing effect of classifier-free guidance (CFG) in AR model Alitok-XL/Wa et al.) (2025b). We plot the normalized entropy over generation steps. As gener- ation progresses, the difference (i.e., the guidance signal, green line in the plot) between baseline (blue line) and CFG perturbation (orange line) entropy quickly vanishes. Here, normalized entropy close to indicates that guidance no longer provides informative guidance, please refer to Ap- pendix C for more details of normalized entropy.
al. 2025bb in steering AR genera- tion towards class conditions without requiring auxiliary classifiers.
2

---

Published as a conference paper at ICLR 2026
"a photo of a white banana and black elephant
Large guidance strength
banana
Over-
guidance
Figure 4: Illustration of the werrguidance phenomenon. When applying a large guidance strength, the model over-emphasizes certain words in the prompt (e.g. "banana" "), leading distorted gen- erations. In this example, the model incorrectly maps the word "banana" to the elephant's tusk, highlighting how excessive guidance strength can harm semantic alignment.
However, applying CFG to AR models introduces two fundamental challenges: First, the guidance signal can diminish over time: in contrast to diffusion, where guidance is injected at every denois- ing step, AR models rely heavily on conditioning tokens at the beginning of the sequence. As decod- ing proceeds, these tokens drift away from the local context, causing the conditional-uncondtiinn gap to shrink and eventually vanish, even within short sequences (e.g.. 16x 16 grids), as shown in Fig. 3 Some recent methods (Tian et al., 2024 Yu et al. 2024a Han et al. 2025) mitigate this by injecting conditional embeddings directly into the prediction head (e.g., AdaLN (Prrez et al. 2018
ensuring conditional information remains accessible.
Second, CFG can also suffer from over-guidance. Since guidance depends solely on external con- ditions such class labels text prompts, increasing the guidance scale often enforces semantics too aggressively, conflicting with visual coherence. This leads to artifacts such as duplicated limbs, redundant handlebars, or spurious object parts, as illustrated in Fig. 4 Even methods that inject conditional embeddings at every step remain tied to external signals and thus cannot fully resolve this cemantic-visual conflict.
Together, this duality mirrors gradient vanishing and explosion in neural network training, which are usually mitigated by normalization and regularization methods. Motivated by this analogy, we propose SoftCFG: an uncertainty-guider inference method that distributes guidance more stably across the generation sequence. The key idea behind SofCCFG is to let every generated token con- tribute
guidance, ensuring the signal persists across steps while naturally regularizing conflicts between text guidance and context, as illustrated Fig. (b). this work, instantiate the weighting by prediction confidence which can be aligned well with the semantics of generated content (as illustrated in Fig. 55) but the framework is general and can accommodate learned scorers perceptual alignment measures. Similar how CFG perturbs the class token to encourage alignment with the class condition, SoftCFG perturbs high-confidence tokens to encour- age future tokens to align with the most semantically reliable content generated so far. Therefore, SoftCFG not only alleviates the fading guidance problem but also reconciles the conflict between semantic alignment and visual coherence, leading more stable and plausible generations.
To realize this idea, we need a mechanism that allows generated tokens to directly influence future decoding steps. Since the value cache stores the representations past tokens that are repeatedly attended to by subsequent predictions. it provides a natural handle for injecting token-wise guid- ance. We therefore compute each token's 's maximum predicted probability Pmax measure of its confidence, and scale its value cache across all attention layers by factor of (1 Pmax) during inference, as illustrated in Figure bb. In this way, tokens with higher confidence receive stronger perturbations on their cached representations, which amplifies their downstream impact and dis- tributes guidance signals throughout the sequence. To avoid degenerate behavior when many tokens accumulate large perturbations or when Pmax approaches 1, we further introduce Step Normaliza- tion procedure: at each step, we normalize the set of perturbation weights so that their sum remains constant (e.g., 1), ensuring stable and balanced guidance throughout the generation process.
3

---

Published as a conference paper at ICLR 2026
Figure 5: Heatmaps of token confidence overlaid on generated images by LuminamGPT2 (Xin al. 2023). nigh-confidence regions align well with salient semantic structures (e.g., object parts), while low-confidence regions occur in ambiguous backgrounds, supporting high-confidence tokens can be effective guidance signals.
SoftCFG requires additional training, introduces no architectural change, and adds negligible computational overhead, making fully compatible with existing AR models and CFG setups. By leveraging uncertainty-weighee perturbations and step normalization, it improves both the stability and effectiveness of guidance during inference. Experiments on ImageNet show that SofCCG sig- nificantly improves generation quality, reducing the FID from 1.37 1.27 over a SOTA AR base- line (Wu et al. 2025b), setting new state of the art for autoregressive models on 256 x 256 Deng al. (2099) benchmark.
We conclude our contributions as follows:
We demonstrate the necessity of leveraging generated visual content as guidance for sub- sequent token prediction, extending the notion of guidance beyond class or text tokens.
We propose SoftCFG, general framework that introduces mosen-wise soft weighting to integrate such guidance. In this work, we instantiate the weighting by token confidence, while other scoring functions learned modules can be naturally incorporated.
Through comprehensive experiments on class-conditional benchmarks (e.g.. ImageNet 256 x 256), we show that SoftCFG achieves state-of-the-art FID among autoregressive models, consistently improving quality and alignment with negligible overhead.
2 SOFT CLASSIFIER-FREE GUIDANCE
2.1 PRELIMINARY: CLASSIFIER-FRES GUIDANCE AR MODELS
Classifier-Free Guidance (CFG) is widely used technique to enhance controllability in generative models by interpolating predictions from conditional and unconditional branches. At each decoding step t, an autoregressive model maintains key/value caches (K<t. V<t) summarizing previously generated tokens Σer- The branch logits are
cond = fe (Koond Vroodd 1),
Z
uncond
(K uncond Vuncond -It-1,0).
(1)
and the guided logits are formed as
CFG 2t zi
uncond y (2,00d z
uncond)
Pe
CPG
xZC, c) softmax (z( CFG
where > 0 is the guidance scale. Here, we use A. 2000d guidance offset, which only decided by input text condition.
uncond to denote the step-wise
Standard CFG applies hard offset A, at every step. While this improves conditional alignment, it can also over-amplify the conditional signal, especially when ||A,]| or y is large. This phenomenon, which we term over-guidance as illustrated in Fig. often manifests as structural artifacts (e.g., duplicated parts or implausible object completions). Moreover, AR models, once F; is sampled, this offset does not propagate explicitly to future steps, so the effect of A, may decay too quickly across long sequences, as shown in Fig. 3 These two limitations motivate the development a softer, regularized alternative to CFG.
4

---

P Published as a conference paper at ICLR 2026
Purtcrbation & Certainty Degreo
i
x
AR Model (Raster Order)
Visual AR Model
Visual AR Model
K
K
K.
V
V.
V.
Unconditional Input
X. X. X.
X
x.
Generated Token
Max Probability Prediction of X
Max Probability Prediction of X
To prodict in this
P
P.
P:
P..
K Cache Cache
Perturb token C
ignoring
Perturb V cache with (1-
Certainty of Token
(a) Vamilla Uncondtional Perturbation
(b) Soft Perturbation (Ours)
Figure 6: Two perturbation strategies for Visual AR models. (a) Unconditional Perturbation modifies
first class conditional token regardless the certainty score. (b) Uncertainty-guided Perturbation applies softer, weighted changes all V cache entries, with strength (1 P), offering stronger perturbation to high-confidence tokens.
2.2 SOFTCFG: UNCERTAINTY-GUIDE PERTURBATION
At high level, our goal is to address the over-guidance and guidance diminishing issues identified in Sec. 2.T and make guidance both context-aware and stable over long horizons. Instead of rely- ing solely on external conditions, we incorporate already generated content and regularize guidance through the model's context. Concretely, SoftCFG acts as a context-aware regularizer on the uncon- ditional branch: it softly adjusts internal states so that reliable tokens contribute more and uncertain ones less. This redistribution embeds guidance into the model's memory, enabling it to persist across future decoding steps without retraining or architectural changes.
Specifically, for each previously generated token i t. compute its predictive confidence using the maximum probability from the conditional distribution:
(2i),
Pmna (x; max, P6 pg ().
Tokens with higher confidence (low w:) are considered more reliable and thus receive stronger perturbations. We also test different confidence indicators in Sec. 3.4 We scale their unconditional value vectors as:
uncond, pertoontext
v!
where Vi uncond, pertoontext denotes this value cache has no conditional information and its context infor- mation is also perturbed. Then we can produce context-perturbee unconditional logits as follows:
2,
uncond, pertcontext
fe (K uncond V-nncond, perteontext Tt-1, 9).
<t
<t
(5)
SoftCFG then combines the conditional branch with the perturbed unconditional branch:
Z1
SoftCFG
2z
cond
+ (z) 2,
-uncond, pertcontext,
(6)
In this way, SoftCFG weakens the unconditional context based on token confidence: reliable to- kens are downscaled more, while uncertain ones are preserved. This adaptive adjustment redefines the guidance itself: instead of relying solely on the external condition, SoftCFG incorporates nor- malized context from already generated tokens, allowing the guidance signal be maintained and propagated consistently across future steps.
Equivalently, SoftCFG can be written as regularized variant of CFG in Eq.
2: SonCFG .GG 7 A context
context
2,
uncond
uncond, pertoontext
The correction gontext is induced by selectively weakening the unconditional value cache, and hence bounded by the perturbation weights [w.).
Geometrically, the additional term Aoniext acts as a context-aware regularizer. When A aligned with the original guidance offset A, zonmd 2 unncoodd SoftCFG amplifies the conditional signal; when misaligned, shrinks it. This mirrors the role of classical & regularization, which constrains parameter updates by pulling them back toward the origin. Here, instead of parameters, we regularize the contextual representations, ensuring that the guidance signal remains bounded and smoothly propagated through the sequence.
5

---

Published as a conference paper at ICLR 2026
Algorithm SoftCFG sampling with Step Normalization
1: Initialize zo <- -bos);; empty caches {Krodd Veond K uncond Vuncond
P Pmax }
2: for t 1 T do
3: ztnd - (K
1iKkean cond
roond
I2)
<t
<t
4: for l.....
Wi 1 Pmax (2;)
get the confidence of all generated token
Wi
5: for l...., t- 1: dú f 1
> step normalization
j= - (W +
6: Vuncond Viincond for i 1, ..... 1 v.mcond W; V
uncond
apply soft perturbation
mnsoond.eertoooox f- fo uncond V Ix 0)
7:
2
<t
8:
2
SonCFo + (1 72200o 2 pnnoondpperiooonet
SoftCFG with context-regularriztion
Sampld Sampldsoftmaxx )
9:
10:
Update original caches with obtain
[Koand Veend
K
unccond
St
V
Vuncond
P
max
11:
end for
2.3 STEP NORMALIZATION
However, we observed an explosion of A Agontest in Fig. 7 where the normalized entropy of the guid- ance increases rapidly, indicating that unbounded SoftCFG may cause the guidance to explode. We attribute this to the cumulative growth of perturbations in the unconditional branch.
Formally, the deviation of SofCCG from vanilla CFG, defined by the context-aware regularizer cconext in Eq. 7 is bounded by: S. -(1 - w:: uncond i<E
A context
(8)
where Le is the Lipschitz constant fo with respect to the value cache. Please refer to Appendix D for more details. If E - ws) were bounded, the deviation would be at most O(7) within
controlled trust region.
However, the sequence length / increases, the cumulative deviation >1 wx) can grow estficcaallyyeeecallllllllll significantly, especially when many tokens have high confidence (w, - 0, 1- - 1), potentially causing A comtext become excessively large and leading SoftCFG to deviate far from vanilla CFG. This also explains the degeneration observed in practice: the unconditional branch gradually loses all contextual information, causing instability in long-horizon generation. Formally, if w; for high-confidence tokens, the contribution of Vuncond is suppressed and the unconditional branch degenerates: i
perceeee.. + 0.
To mitigate this, we introduce Step Normalization, which renormalizes the perturbation weights at every step:
t-1
W Σ)=1
such that Σ w.
i
ws)
Proposition 1 (Bounded Deviation of Step-Normalized SoftCFG), Let So be Li-Lipschitz with re- spect to its value-cache input at step Then the deviation of SofCCFG from vanilla CFG bounded as
A
context
perond.eer
uncoond
< L max ict
||v
(10)
pencondeert denotes the unconditional logits under step-normalized perturbation. where Z,
Without normalization, the cumulative perturbation of past tokens grows proportionally with se- quence length. Step normalization rescales the perturbation weights {1 - w/} so that Ei=((1 tb;)) 1 at each step, effectively allocating unit perturbation budget over the context. Therefore the perturbation magnitude at step f is controlled by at most one token's 's contribution, leading to the bound context| < L, maxict [ynoood

---

Published as a conference paper at ICLR 2026
Comparison of Normalized Entropy
Other Operations
KV modificction
Inference
1.000
25
400
0.975
0,950
A100 Inference speed (s/25ling)
20
4.6
A100 Inlerencs g (slmn))
Norniizal Entropy
0.925
18
250
0.900
200
0.875
10
150
0.850
100
StepNorm orm
100
250
CFG
SoRCEG
CFG
SofCFG
Step
AlitokXL-0.6B
LuminaGPT2-7B
Figure 7: Comparison of normalized entropy across different sampling strategies. Step nor- malization stabilizes SofCCFG by controlling cumulative perturbations of the unconditional branch.
Figure 8: Inference speed comparison of CFG and SoftCFG (w Step Norm) on AliTok-0.6B (256 x 256) and LuminaGPP22-7B (756 x 756). SoftCFG adds negligible overhead.
3 EXPERIMENTS
3.1 SETUP
Dataset and Benchmark. For class-conditional generation, evaluate on ImageNet (Deng et al. 2009) with the standard 256x class-conditional generation protocol. For sext-to-image generation, evaluate two widely used benchmarks, i.e., GenEval benchmark (Ghosh et al. 2023 and DPG-Bench (Hu et al. 2024).
Models. We evaluate SofCCFG state-of-the-art (SOTA) autoregressive (AR) generators to en- sure that improvements are not due to weak baselines. For class-conditional generation, we use the strongest published AR model on ImageNet 256x256 (Deng et al. 2009), AliTok (Wu et al., 2025b). For text-to-image generation, we use LuminaGPT2 Xin et al. (2025), a recently released large-scale AR model that achieves the best performance and DPG-Bench (Hu et al. 2024). All models adopt a VQ-style discrete tokenizer and support classifier-free guidance (CFG) inference. Our method, SoftCFG, is applied only at inference, without retraining or modifying the architecture.
both GenEval Ghosh et al. (2023)
Metrics. For class-conditional generation task, we generate 50k samples on the validation label set and report Fréchet Inception Distance (FID; lower is better). For completeness, we also track Inception Score (IS), precision/recall for generative models.
Hyperparameter settings. We follow the baseline sampling setup and keep all decoding hyper- parameters identical to CFG (temperature, guidance scale 7, etc.). SoftCFG uses token-wise uncer- Pmax((x)) measured on the unconditional branch at the time token z; is generated, and perturbs only the unconditional value cache as defined in Sec. 2.2 Guidance scale matches the CPG baseline unless stated otherwise.
tainty weights w;
3.2 IMPLICATIONS AND TIME COMPLEXITY ANALYSIS
We summarize the inference procedure of SoftCFG with step normalization in Alg. At each decoding step, the conditional logits are computed as vanilla CFG, while the unconditional branch is temporarily perturbed by rescaling its value-cache entries according to token-wise confidence. Step normalization ensures that the perturbation weights are re-normalized at every step, allocating fixed perturbation budget across the context. Importantly, the perturbation is applied by in-place scalar rescaling of the stored unconditional V-cache, which is negligible compared to Iitentiin/MLLL compute; As illustrated in Fig. the SoftCFG sampling time matches CFG.

---

Published as a conference paper at ICLR 2026
Table ImageNet- 1K (Deng et al.
2009) 256 x 256 generation results evaluated with ADM (Dhari-
wal & Nichol 2021
indicates results from our own implementation.
Type
Generator
Venue
#Params
FID1
18T
sFID.
Pre.7
Rec.1
LDM-8
Rombach al.
2022
CVPR 22
258M
7.76
209.5
0.34
0.35
LDM-4
Kombach ot 4022
CVPR 22
400M
3.60
247.7
0.87
0.48
UVIT-L
2 Bão ct a. 2025
CVPR 23
287M
3.40
219.9
0.83
0.52
UVIT-H/2 Bao et 2025
CVPR
501M
2.29
263.9
0.82
0.57
DIT-L/2 Peentes
ICCV 23
458M
5.02
167.2
0.75
0.57
Diff.
DiT-XI (Pochles X1e
20723
ICCV 23
675M
2.27
278.2
4.60
0.83
0.57
SIT-XL Ma mi 2024
ECCV 24
675M
2.06
270.3
4,50
0.82
0.59
DiMR-XL/ZK Liu
2024
NeulPS 24
505M
1.70
289.0
0.79
0.63
MDTV2-XLL XL/2 Gao et 2023
676M
1.58
314.7
4.52
0.79
0.65
REPA Yu
202g
ICLR 25
675M
1.42
305.7
4,70
0.80
0.65
REPA - Leng dl. 2025
ICCV 25
675M
1.26
314,9
4.11
0.79
0.66
MaskGl1 Chang
2022
CVPR 22
177M
6.18
182.1
0.80
0.51
TiTok-S 128 Yü al
2024b
NeulPS 24
287M
1.97
281.8
Mask.
MAGVIT-v2 Yu
2023
CVPR 23
307M
1.78
319.4
MaskBit Weber
Arxiv 24
305M
1.52
328.6
VAR-d30
A024
NeulPS 24
1.92
0.82
0.59
VAR
VAR-d30-Te
2024
NeulPS 24
2.0B
1.73
325.0
0.82
0.60
MAR-B
2124
NeulPS 24
208M
2.31
281.7
0.82
0.57
MAR
MAR-1 1 al
20244
NeulPS 24
479M
1.78
296.0
0.81
0.61
MAR-I La cl
2024
NeulPS 24
943M
1.55
303.7
0.81
0.62
FlowAR
FlowAR-3 Ren et al.
2025
ICML
170M
3.61
234.1
0.83
0.50
FlowAR-F
ICML
1.98
1.65
296.5
0.83
0.60
GPT2 Esser
CVPR 21
1.4B
15.78
74.3
GPT2-76 Esser cf al.
2021
CVPR 21
1.48
5.20
280.3
VIM-I Tu al. 2012
ICLR 22
1.7B
4.17
175.
VIM-L Yu all 2022
ICLR 22
1.7B
3.04
227.4
Open-MAGVIIZ-E Luo et
22044
Arxiv 24
343M
3.08
258.3
0.85
0.51
Open-MAGVIT2-X
2024
24
1,5B
2.03
286.0
0.84
0.54
LlamaGen-I Sun
4024
Arxiv 24
343M
3.07
256.
0.83
0.52
AR
LlamaGen-33 30 Sun el
2424
Arxiv 24
3.1B
2.18
263.3
0.81
0.58
RandAR-L Pang ct al 2025
CVPR 25
343M
2.55
288.8
0.81
0.58
LandAR-XX et 2025
CVPR
1.4B
2.15
322.0
0.79
0.62
RAR-B Yu al.
20248
Arxiv 24
261M
1.95
290.5
0.82
0.58
RAR-XI, Yu
20244
Arxiv 24
955M
1.50
306.9
0.80
0.62
Alitok-B Wu al. 20/25
Arxiv25
177M
1.50
305.9
0.78
0.64
Alitok-L
Wi ct al
20256
Arxiv25
318M
1.42
326.6
0.78
0.65
- (i
2025b
Arxiv25
662M
1.35
317.1
6.96
0.79
0.64
Alitok-B*SSniICFU (oirs)
177M
1.40
271.0
5.95
0.78
0.66
Alitok-L*+SoliCPe (ours)
318M
1.39
272.3
6.00
0.78
0.66
Alitok-XL x SofiCCG (ours)
662M
1.27
302.4
6.76
0.78
0.65
3.3 STATE-OF-THE-AIR COMPARISONS.
Conditional Image Generation. Table summarizes results on ImageNet-IK 256x256. Diffu- sion and masking models achieved strong IS and better FID. Recent AR models (e.g., LlamaGen Sun et al. (002)). RandAR Pang et al. (2025) ) reduce this gap. Our AliTok+SofiCFG attains an FID of 1.27, narrowing the gap between AR and diffusion models, while maintaining competitive IS and improved recall. With SoftCFG, AR demonstrates stronger potential to replace diffusion in future architectural designs.
3.4 ABLATIONS
Effect of SoftCFG As shown in Table 2 replacing vanilla CFG with SoftCFG reduces FID (1.32 vs. 1.37), indicating more effective guidance. We also show the visualization comparison on Lu- minamGPT2 (Xin et al.) (2025) in Fig. However, discussed in Sec. 2.3 directly applying Soft- CFG could make the unconditional branch lose too much contextual information, therefore SoftCFG slightly destabilizes IS due to amplified step-wise perturbations.
Effect of Step Normalization Adding step normalization top of SoftCFG mitigates the above issue, yielding both stable IS and improved sFID (7.16). This confirms that step normalization complements SoftCFG by regularizing its perturbation strength over time. Figure 7 illustrates the normalized entropy on AliTok-XX XL with = 13, = 14, where the confidence score is defined as the conditional probability after guidance.
8

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study on ImageNet 256 x 256 class-conditional generation. We use the optimal 7 13 (in Eq. and 1.4 (in Sec. 3.4) for CFG Wu all (2025b). We report FID, IS and sFID. SoftCGG improves FID over vanilla CFG, and step normalization further stabilizes IS and sFID. Opt. indicates that jointly tune (7, k) via small grid. We bold the top-2 values for each metric.
Method
FID L
IS 1
sFID +
Baseline
1.76
221.2
5.55
CFG + Opt.
1.35 (-0.41)
317.1 (495.9)
6.96 (+1.41)
SoftCFG
1.32 (-0.44)
288.1 (+66.9)
7.62 (+2.07)
SoftCFG + StepNorm
1.32 (-0.44)
302.0 (+80.8)
7.16 (+1.61)
SoftCFG StepNorm + Opt.
1.27 (-0.49)
302.4 (+81.2)
6.70 (+1.15)
Table Effect of perturbing different cache components in AliTok XL on ImageNet-256. SoftCFG with perturbations applied only to the value cache achieves the best FID.
SoftCFG
FID
IS f
sFID +
Precision 1
Recall
Perturb K and V
1.31
287.6
6.94
0.79
0.65
Perturb K only
1.39
302.5
7.66
0.80
0.64
Perturb V only
1.27
302.4
6.70
0.79
0.65
Effeet of V perturbation We ablate which part of the attention cache SoftCFG perturbs. Since queries of generated tokens are not needed attention calculation, we only compare perturbing K, or both K and V on AliTok-XL. (ImageNet-25) 50k samples). As shown in Table perturbing V only gives the best FID/sFID and highest recall while keeping IS and precision competitive, whereas perturbing K (with or without V) slightly improves IS/precision but consistently worsens FID, sFID, and recall, suggesting that modifying K disrupts attention routing. This empirically supports our design choice of perturbing V as the most robust and practical option for AR decoding.
1.350
k-1,4
9
1.325
k-1.5
9
y= 10
1.295
1,250
8.0
8.5
9.0
9.5
10.0
10.5
11.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
hids
Pors (Coed) Frok. (laboll
Entragy
(a) Guidance scale y sweep
(b) Cosine expoment swoep
Confidence variants
Figure 9: Adaptive hyperparameter ablations of SoftCGG (a) Guidance scale 7 sweep ent cosine exponents k. (b) Cosine exponent & sweep under fixed guidance scales y, (c) Comparison of different confidence score definitions. Results show that SoftCFG robust to hyperparameter choices and that the max probability of conditional branch yields the best trade-off.
differ-
Hyperparameter Varying We analyze the sensitivity of SoftCFG to (i) the guidance scale y (in Eq. and Eq. and (ii) the power parameter in cosine scheduling introduced by Yu et al. (0024a). Here, cosine schedule for the guidance scale is: = (7 1) 1 cos( (1/7 " where t is the current step and & controls how guidance is distributed: larger shifts to later steps, smaller & to earlier steps. As shown in Fig. 9 (a) and Fig. 9 (b), SoftCFG consistently improves FID across wide range of 7, while vanilla CFG quickly deteriorates for large scales. For cosine scheduling, moderate powers (k 1.5 or 1.6) yield the best trade-off, whereas extreme values hurt performance. These results highlight that tuning is still required, and the effective ranges of x and & are broader and the optimal values are typically smaller than those of vanilla CFG. This is because SoftCFG generally produces stronger guidance magnitudes than vanilla CFG. In addition, Fig. (c) compares different definitions of the confidence score. We find that using the max probability of the conditional branch yields the most stable improvements, further showing that SoftCGG is robust confidence variations.

---

Published as a conference paper at ICLR 2026
Table 4: SoftCFG on text-to-image generation with SimpleAR. We report overall score and fine- grained compositional metries.
Method
Overall
% Correct Images
% Correct Prompts
Single Ohj.
Two Obj-
Position
Colors
Color Attr.
Counting
SimpleAR 0.61744
80.65% 80.29%
100.00% 99 99.69%
30.25% 26.00%
81 82.18%
33.44% 27.81%
SofCFG 0.61266
60.80%
91.41% %
40.50
Artifacts
BUTUAL
CFG
P
Status: Correct
Status: Wrong
Status: Correct
Status: Correct
SofCFG
Status: Wrong
Status: Correct
Status: Wrong
Status: Wrong
Reason:
Reason:
Reason:
found right of target
not above target
found left of target
Qualitative comparison between CFG and SoftCFG text-to-image generation. Soft- CFG yields cleaner, more coherent foregrounds with fewer artifacts, but occasionally misplaces objects w.r.f. spatial keywords. The DPG-Bench Hu al. (2024) evaluator relies strict rule-based text-image alignment and largely ignores image quality.
Figure 10:
Text-to-image Generation Table 4 shows that adding SoftCFG to SimpleAR slightly lowers
overall DPG-Bench Hu et al. (2024) score (0.617->0.113), but improves several compositional
pects: two-object prompts (89.39% 19..1111)) color consistency (81.38%-82.1887) and color- attribute correctness (36.00% +40.50%) As shown in Fig. TO position remain challenging and show small regressions, suggesting that current text-to-image evaluation focuses on strict rule- based text-image alignment instead of image quality and coherence. Overall, SoftCFG enhances image quality in T2I AR models, while leaving room for future work on text-image alignment.
4 CONCLUSION
In this work, we introduced SoftCFG, a lightweight modification to classifier-free guidance for au- toregressive generation. Our key insight is that, similar class tokens, previously generated visual content can also provide guidance to subsequent tokens. To fully exploit this effect, SoftCFG ap- plies soft weighting mechanism that adaptively modulates the influence of past tokens. While in this work we instantiated the weights with token-level confidence, our framework naturally accom- modates more sophisticated scoring functions, including alternative scoring mechanisms, such as leamned discriminators or perceptual evaluators. Extensive experiments on class-conditional gen- cration demonstrate that SoftCFG consistently improves fidelity and alignment without retraining increasing inference cost. We believe this simple yet general principle opens directions for integrating fine- grained guidance signals into autoregressive generation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We acknowledge funding from the Flemish Government (AI Research Program) and the Research Foundation Flanders (FWO) through project number G0G292IN. This project was also supported by Sigrid Juselius Foundation and the Finnish Research Council (Profi6 336449 funding pro- gram), the strategic funding of the University of Oulu. We acknowledge EuroHPC JU for award- access to MareNostrums ACC hosted by BSC at
ing the project ID EHPC-AIF-2025SCOO Barcelona, Spain.
ETHICS & REPRODUCIBILITY STATEMENT
This work relies only on public datasets, and while generative models may be misused, our method does increase such risks. All key settings are reported, and we have released the core code for SoftCFG.
REFERENCES
Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. In Proc. Eur. Conf. Comput. Vis. (ECCV). Springer, 2024.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022.
Huayu Chen, Hang Su, Peize Sun, and Jun Zhu. Toward guidance-free visual generation via condition contrastive alignment. arXiv preprint arXiv:2410.093.4 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEF/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognis. (CVPR), 2021.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2023.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt, Geneval: An oojucc-focuse framework for evaluating text-to-image alignment. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2023.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng. and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In Proc. TEFF/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Susung Hong. Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2024.
Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2023.
11

---

Published as a conference paper at ICLR 2026
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.05133, 2024.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2024.
Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2025.
Pengxiang Li, Shilin Yan, Joey Tsai, Renrui Zhang, Ruichuan An, Ziyu Guo, and Xiaowei Gao. Adaptive classifier-free guidance via dynamic low-confidence masking. arXiv preprint arXiv:2505.20199 2025a.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In Proc. Adv. Neural Inf. Process. Syst. (Neur!PS), 2024.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and 7he Lin. Imagefolder: Autoregressive image generation with folded tokens. In Proc. Int. Conf. Learn. Representations (ICLR), 2025b.
Qihao Liu, Zhanpeng Zeng, Ju He, Qihang Yu, Xiaohui Shen, and Liang- Chinh Chen, Alleviating distortion in image generation via multi-resolution diffusion models. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2024.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge. Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.044/11 04040, 2024.
Nanye Ma, Mark Goldstein, Michael Albergo, Nicholas M Boffi, Eric Vanden- and Sain- ing Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In Proc. Eur. Conf. Comput. Vis. (ECCV), 2024.
Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proc. AAAI Conf. Artif. Intell. (AAAT), 2018.
Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Ze- huan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. In Proc. IEEE/CVE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025.
Javad Rajabi, Soroush Mehraban, Seyedmorteza Sadat, and Babak Taati. Token perturbation guid- ance for diffusion models. arXiv preprint arXiv:2506. 22225066.0006 10036, 2025.
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chiee Chen. Flowar: Scale- wise autoregressive image generation meets flow matching. In Proc. Int. Conf. Mach. Learn. (ICML), 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022.
Dazhong Shen, Guanglu Song, Zeyue Xue, Fu- Yun Wang, and Yu Liu. Rethinking the spatial inconsistency in classifier-free diffusion guidance. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024.
12

---

Published as a conference paper at ICLR 2026
Fengyuan Shi, Zhuoyan Luo, Yixiao Ge, Yujiu Yang, Ying Shan, and Limin Wang. Taming scal- able visual tokenizer for autoregressive image generation. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2025.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 010222508.0004.... 2025.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.000 065525 2024.
Zhicong Tang. Jianmin Bao, Dong Chen, and Baining Guo. Diffusion models without classifier-free guidance. In Proc. Int. Conf. Mach. Learn. (ICML), 2025.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Version foundation LLaMA-1 re-
lease from Meta Al.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2017.
Yuqing Wang, Shuhuai Ren, Zhijie Lin, Yujin Han, Haoyuan Guo. Zhenheng Yang, Difan Zou, Jiashi Feng, and Xihui Liu. Parallelized autoregressive visual generation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025.
Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chieh Chen. Maskbit: Embedding-free image generation via bit tokens. arXiv preprint arXiv:2409. 162/1, 2024.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified mul- timodal understanding and generation. In Proc. IEEF/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025a.
Pingyu Wu, Kai Zhu, Yu Liu, Longxiang Tang, Jian Yang. Yansong Peng. Wei Zhai, Yang Cao, and Zheng-Jun Zha. Alitok: Towards sequence modeling alignment between tokenizer and autore- gressive model. arXiv preprint arXiv:2506.055 0528899960089..... 2025b.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In Proc. Ins. Conf. Learn. Representations (ICLR), 2025.
Yi Xin, Juncheng Yan, Qi Qin, Zhen Li, Dongyang Liu, Shicheng Li, Victor Shea-Jay Huang, Yupeng Zhou, Renrui Zhang, Le Zhuo, al. Lumina-mgpt 2.0: Stand-alone autoregressive image modeling. arXiv preprint 2rXiv:2507.17801, 2025.
Yijia Xu, Jianzhong Ju, Jian Luan, and Jinshi Cui. Direction-awarr aware diagonal autoregressive image generation. arXiv preprint arXiv:2503.1 /29, 2025.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. quattr-qu image modeling with improved vggan. In Proc. Int. Conf. Learn. Representations (ICLR), 2022.
13

---

Published as a conference paper at ICLR 2026
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsune -suan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Randomized autoregres- sive visual generation. arXiv preprini arXiv:2411.00770 2024a.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2024b.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In Proc. Int. Conf. Learn. Representations (ICLR), 2025.
14

---

Published as a conference paper at ICLR 2026
Figure 11: Samples generated by AliTok-XL with SoftCFG and Step Normalization on ImageNet 256 256. The results demonstrate high visual fidelity and semantic consistency across diverse classes.
A APPENDIX: VISUALIZATION
We also provide qualitative samples generated by AliTok-XL with SoftCFG and Step Normalization on ImageNet 256 x 256 (Fig. ITI. These results illustrate the visual diversity and semantic fidelity that can be achieved with our method, complementing the quantitative evaluations in Sec. 3.4
B APPENDIX: RELATED WORK
B. VISUAL AR MODELS
Recent advances have significantly improved AR-based generation by enhancing both the visual tokenizer (Yu et al. 2024b Wu et al.
2025b et al. 2025b Qu et al.
2025 Shi et al. 2025 Pang et al. 2025 Yu et al. 2024a Xu et al. 2025
and the generation paradigm (Tian et al.
2024
Wang et al. 2025). leading to sharper s spatial alignment and more coherent semantics. As a result, AR models Wu et al. 2025b Xu et al.
2025 have recently demonstrated generation quality par
with state-of-the-ane diffusion based and flow-based models (Yu et al. 2025
Leng et al. 2025 on
standard benchmarks such ImageNet (Deng et al. 2009).
15

---

Published as a conference paper at ICLR 2026
B.2 CLASSIFIER- GUIDANCE IN GENERATIVE MODELS
Classifier- Freeff Guidance (CFG) first introduced in diffusion models improve conditional generation performance without relying on external classifiers. It interpolates between conditional and unconditional predictions during sampling, trading off between sample fidelity and diversity (Ho) & Salimans, 2022). Since then, CFG has become a fundamental technique in generative modeling.
A growing body of research explores ways to refine or extend CFG, partieularly by injecting per- turbations or structural modifications to guidance: Self-Attention Guidance (Hong et al. (202)) pro- poses using attention-based modifications guide generation in diffusion models, improving sam- ple quality without needing new objectives. This work highlights the importance of internal structure manipulation as an alternative direct logits-basee guidance. et al. 2024) also introduces perturbed-attenn -atuubbeddattnntoon into diffusion sampling to correct misaligned atten- tion signals. Smoothed Energy Guidance (Hong, 2024) further addresses unstable energy landscapes in CFG by smoothing the attention curvature, resulting in more consistent guidance effects across different prompts and steps. Guiding a Diffusion Model with a Bad Version of Itself (Karras et al.) 2024) shows that strong generation quality can be achieved by replacing the unconditional branch in CFG with a smaller or less- dess-trained model.
Snrturbed-Attettinn Guidance (Ahn
Several recent works (Shen et al. 2024: Rajabi et al.
2025
ct al.
2025a also focus improving
CFG via dynamic and token-level perturbations: Semantic- aware CFG (Shen et al.] 2024), which adjusts guidance based on spatial semantic regions to mitigate inconsistency. Token Perturbation Guidance (Rajabi et al. (025), which locally perturbs input tokens or hidden states to encourage more robust generation paths. Adaptive CFG (A-CFG) (Li et al. (025a), which applies dynamic low-confidence masking to the unconditional branch based on step-wise uncertainty.
While these approaches offer significant improvements for diffusion and masked generative models, they are not directly transferable to AR generation, where conditioning signals degrade rapidly due the sequential nature of decoding. One of the few works examine CFG in AR setting is Con- dition Contrastive Alignment (CCA) (Chen et al.) (202)), which enables guidance-free sampling by fine-tuning the model align conditional and unconditional outputs. However, still underperforms standard CFG in most cases and requires additional training. In contrast, our method introduces Soft- CFG, a lightweight uncertainty-io aware guidance mechanism for AR models, designed to stabilize the guidance signal across the entire generation sequence without modifying training, architecture, or inference efficiency. Beyond improving AR inference quality, SoftCFG provides a more stable and informative form of conditional guidance, which can potentially benefit future alignment-based or distillation- based approaches such as CCA and Model-guidance (Tang et al.) 2025).
C APPENDIX: NORMALIZED ENTROPY COMPUTATION
Given the logits 2, RV at decoding step we first compute the probability distribution via softmax:
exp(2r,)
(i()) = Ey= exp(215)
i= i=
(11)
where V is the vocabulary size. The entropy of this distribution is
((p)) Σ (i) log (i)).
(12)
To make entropy values comparable across vocabularies of different sizes, we normalize by the maximum entropy log V:
H(x)) (()))) [0.
(p:))
(13)
log V
Here, (p:)) ss indicates highly confident predictions, while ((p)) corresponds to high uncertainty.
16

---

Published as a conference paper at ICLR 2026
D APPENDIX: DERIVATION OF THE SOFTCFG DEVIATION BOUND
In this appendix, we derive the bound on the deviation of SoftCFG from vanilla CFG, given by:
A Ammiest
S L
Σ w:) || ! ==1
(14)
suuncond. pertconlext uncond is the context-aware perturbation, and L, is the Lipschitz constant of the model So with respect to the value cache.
where A "
D.1 PERTURBED VALUE CACHE
In SoftCFG, the unconditional value cache Vuncond
[
uncond
vincood
unoondd] is perturbed as:
Vinnond W Vuncond
(15)
where Wet diag w1, W2,.... ..... Wt-1): W; Pmax (xi) [0, 1). and denotes clement-wise
scaling. Thus:
unmond
uncond
=1,2, 1,2,.....
(16)
The unconditional logits are computed as:
2
uncond
fe uncond
Vincond
It 11 0),
(17)
z
uncond. perteontext
fe K uncond Vruncond I1-1: 0).
(18)
The deviation Agontert arises from the perturbation Vruncond Tuncond
D.2 LIPSCHITZ CONTINUITY
We assume that fo is Li-iisschizz continuous with respect to the value cache:
2
uncond, perteontext
Zt
unconnd
L, Trunoodd
Vuncond
(19)
This assumption is justified by the structure of Transformer models, which consist of multi-head attention (MHA), feed-forward networks (FFN), residual connections, and LayerNorm:
Attention Mechanism: The attention output is computed Attention Q K,K))
QK" Vdz
softmax
V. The softmax function is smooth with a Lipschitz constant of 1, as
its Jacobian is bounded (
< 1). The value cache V contributes linearly, and as- suming bounded key and query matrices, attention mechanism is Lipschitz continuous.
Osoftmax, 8e;
The FFN, FFN = (WWix x + W2 + b2, uses activation functions like ReLU or GELU, which are Lipschitz continuous.
Feed-Forward Network:
Residual Connections and LayerNorm: Residual connections (x + Attention(x)) and LayerNorm have bounded gradients.
Composition: The composition of Lipschitz continuous functions (attention, FFN, Layer- Norm, and final linear projection) is itself I. Lipschitz continuous.
Form Eq. 19 have:
context
S L Vuncond
runcond
(20)
D.3
VALUE CACHE DIFFERENCE
The norm is:
f-1
runcund
Vuncond <t.
Σ
ws)( -(-v.mccom)
(21)
<t
17

---

Published as a conference paper at ICLR 2026
Applying the triangle inequality:
ws)( m
w;)(( (-;)(-vnnoon))
(22)
Since 1 - wi 0:
[[1 - w/)( v uncond
= (1 w
|| !
(23)
Thus:
(-1
V unccodd
Vuncond
w/y
mncond
||
(24)
<t
D.4 FINAL BOUND
Combining the Lipschitz condition in Eq. 20 and the value cache difference in Eq. 24
t-1
|اا context
S L.
Σ (1 w:) w uneond
(25)
Thus, the deviation between SoftCFG and vanilla CFG is:
| SonCPO
[H = Y | S 7. L. E (1 w:
(26)
This bound depends on the cumulative perturbation E1 - ws). which grows with sequence length especially when many tokens have high confidence (w, 0). If (1= wx) were bounded, the deviation would be O(y), ensuring SofCFG remains within controlled trust region. However, without Step Normalization, this sum can become large, leading to excessive deviation and potential degeneration of the unconditional branch, as discussed in Sec. 2.2
E APPENDIX: LIMITATIONS AND FUTURE WORK
"A calico cat, sporting a patchwork of orange. black, and white for, is confortably nestled on top of sleek
Mercedes Benz. The luxory
vehicle, with its amblem glinting in the light. is stationary. possibly quiet residential area. The cat, with its eyes gently shut, appears be in state of serene repose.
oblivious to the world ground
Figure 12: Failure case of SoftCFG. The prompt describes a calico cat on a Mercedes, but the "patchwork" attribute is wrongly applied to the car. Confidence focuses on the vehicle rather than the cat, leading to semantic drift.
While SoftCGG improves the stability and effectiveness of classificr-free guidance in visual AR models, several limitations remain.
First, the current instantiation relies on token confidence as the weighting function. In complex cases, the confidence distribution may collapse onto single object, leading to concept drift where generation is overly biased toward that object. Figure illustrates a representative failure mode of SoftCFG. Given the input prompt "A calico cat, sporting patchwork of orange, black, and white
18

---

Published as a conference paper at ICLR 2026
fur, is confortably nestled top of sleek Mercedes Benz...", the model generates image where the "patchwork" attribute is incorrectly transferred to the car rather than the cat. The corresponding confidence map further shows that high-confidencee regions concentrated the cat receives low confidence. As result, SoftCFG amplifies the vehicle features and neglects the cat, leading semantic drift. This highlights that SoftCFG is sensitive to the accuracy of the confidence estimates, which current AR models cannot yet provide reliably.
the vehicle, while
Second, is StepNorm too strict for SoftCFG? While step normalization effectively prevents the deviation of SoftCGG from exploding, its strict renormalization may also be overly restrictive. By enforcing E úi;) 1 at every step, the perturbation budget is capped uniformly across sequence lengths. This implies that no matter how long the context grows, the total perturbation injected into the unconditional branch remains equivalent to at most single token's ' information. From another perspective, step normalization can be seen as discarding the cumulative contribution of multiple tokens and retaining only one token's worth of perturbation each step. As result, while stabilizes long-horizon generation, its advantage in later steps may diminish compared to the unnormalized variant, the orange line in Fig where a larger perturbation budget could potentially provide stronger guidance.
Future work could address these limitations: (1) a promising direction is to incorporate auxiliary perceptual models (such as DINOV3 Siméoni et al. 2025)) to score token-level semantics, yielding more robust confidence signals for guidance. (2) A Although step normalization mitigates degenera- tion in long sequences, extremely long-horizon generation (e.g., high-resolution images or video) may still suffer from diminished guidance, calling for more adaptive normalization schemes. (3) Our study focuses primarily on class-conditions and text-to-image generation; broader applications such as multi-mode AR tasks, video synthesis, and reinforcement learning from human feedback (RLHF) remain unexplored.
F APPENDIX: LLM USAGE
During the preparation of this work, we made limited use of large language models (LLMs), specifi- cally GPT, to assist with non-scientific tasks. These included (i) generating and formatting auxiliary visualization code (e.g., plotting scripts for ablation studies) and (ii) refining the writing style of the manuscript through grammar correction and phrasing suggestions. All scientific ideas, theoretical formulations, experimental design, and analysis were solely developed and validated by the authors.
19


Reviews:
[{"review_id": "h24wdFFSBk", "reviewer": "Reviewer_ny6c", "summary": "The paper proposes **SoftCFG**, a training-free and model-agnostic inference modification for **visual autoregressive (AR)** image generation models.  \nIt addresses two common problems when applying Classifier-Free Guidance (CFG) to AR models:  \n(1) *guidance diminishing* (conditional signal fading as decoding progresses), and  \n(2) *over-guidance* (visual distortions caused by high guidance scales).\n\nSoftCFG introduces **uncertainty-guided token-wise perturbations** to the unconditional branch: each past token contributes to guidance proportionally to its **prediction confidence**, influencing future decoding steps through the cached value vectors.  \nA **Step Normalization** mechanism further ensures stability by normalizing cumulative perturbations at each step.  \nThe method requires no retraining, adds negligible computational cost, and yields improved FID on ImageNet-256 (1.37 → 1.27) compared to vanilla CFG.", "questions": "1. Why only perturb the **value cache (V)** and not keys or queries?  \n2. How sensitive is the performance to **confidence miscalibration**? Would temperature scaling or learned uncertainty improve robustness?  \n3. Could **Step Normalization** be relaxed or made adaptive for longer contexts?  \n4. Are improvements consistent under **different tokenizers** or **generation orders** (e.g., folded or diagonal AR)?  \n5. How does SoftCFG behave on **text-to-image benchmarks** quantitatively (e.g., COCO-FID, GenEval metrics)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Clear motivation and problem framing** – The issues of guidance fading and over-guidance in AR models are well-illustrated with entropy plots and examples.  \n2. **Elegant, simple solution** – The token-wise confidence weighting and step normalization are easy to implement and integrate into existing AR inference pipelines.  \n3. **Training-free and architecture-agnostic** – Works as a plug-in for existing models like AliTok and LuminaGPT without retraining or modifying the transformer architecture.  \n4. **Empirical improvement** – Achieves state-of-the-art FID among AR models on ImageNet-256 with negligible runtime overhead.  \n5. **Good ablations and qualitative examples** – The paper studies the impact of StepNorm, guidance scale γ, and scheduling power k, and presents clear visual comparisons showing fewer artifacts.  \n6. **Transparency and reproducibility** – The paper includes clear algorithms, theoretical bounds, and a stated plan to release code.", "weaknesses": "1. **Limited novelty** – Conceptually extends prior ideas (adaptive guidance, token-level perturbation) from diffusion models to the AR setting.  \n2. **Fragile confidence heuristic** – The reliance on max probability as a proxy for uncertainty can mislead guidance, as shown in the “cat–car” failure case.  \n3. **Partial perturbation design** – Only value caches are scaled; effects on keys or multi-head attention routing are not explored.  \n4. **Step Normalization rigidity** – The fixed-sum normalization may underutilize guidance in long sequences; no adaptive scheduling is tested.  \n5. **Loose theoretical analysis** – The Lipschitz-based bound is general but too weak to predict actual model stability.  \n6. **Limited experimental breadth** – Results are restricted to ImageNet-256 and a few text-to-image examples; no tests on higher-resolution or multi-modal AR tasks.  \n7. **Hyperparameter fairness** – CFG baselines may not have been re-tuned under identical γ/k sweeps, potentially overstating SoftCFG’s advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DPJOPWinar", "reviewer": "Reviewer_HN4L", "summary": "This paper introduces SoftCFG, a new uncertainty-guided inference method for discrete visual autoregressive (AR) models. The method aims to mitigate the problem of the guidance signal diminishing over time by reweighing the confidence of previous tokens within the unconditional KV cache. And it introduces the Step Normalization to avoid the explosion of the guidance caused by this reweighing step. Experimental results demonstrate that SoftCFG can improve the generation quality.", "questions": "see Weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed SoftCFG method is intuitive, easy to understand, and appears simple to implement.\n2. The method shows a notable improvement in generation quality on the ImageNet-256x256 dataset.", "weaknesses": "1. **Limited Generalizability:** The paper presents SoftCFG as a general method, yet its effectiveness is only thoroughly validated on a single model. This narrow experimental scope is insufficient to support the claim of generality. Furthermore, the qualitative results shown for the RAR model in Figure 1 are not convincing and do not demonstrate a clear benefit. (Beside, the new versions (10 October 2025) of the baseline model alitok can achieve CFG results comparable to those reported in your paper with SoftCFG. This raises a critical question: is the reported FID improvement a genuine algorithmic contribution of SoftCFG, or does it merely compensate for a sub-optimally tuned CFG baseline? )\n2. The 'diminishing guidance' problem may be an artifact of the baseline using only a single class token. Would this problem already be alleviated in standard CFG if the class token were repeated 64 times as the condition, similar to MAR?\n3. There appears to be a significant error in Equation 6. The equation as written is inconsistent with the line 8 in Algorithm 1 and does not align with Equation 7. I suspect the correct formulation should be $$z_t^{SoftCFG}=z_t^{cond}+scale*(z_t^{cond}-z_t^{uncond,pertcontext})$$?\n4. **Missing Quantitative Results:** The paper text explicitly states in Section 3.1 (lines 349-350, 355-358) that quantitative results (GenEval benchmark, DPG-Bench) for Text-to-Image (T2I) generation are provided. However, no such quantitative results are present anywhere in the paper. This is a major omission that leaves the T2I claims entirely unsubstantiated.\n5. **Overall Presentation:** The paper is not well-written. The presentation suffers from a lack of clarity, and the significant issues noted above (e.g., the error in Equation 6, the missing T2I results) make the paper difficult to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKl7dKIQFI", "reviewer": "Reviewer_Yrai", "summary": "This paper introduces SoftCFG, a plug-and-play inference-time method that improves the stability and fidelity of visual autoregressive models. The method replaces the fixed classifier-free guidance with a soft, uncertainty-weighted guidance that adaptively adjusts the influence of each generated token. Additionally, a Step Normalization mechanism is proposed to bound accumulated perturbations across steps. Experiments on ImageNet and text-to-image benchmarks show consistent FID improvements over standard CFG without retraining.", "questions": "1. What is the standard deviation of FID across runs? Are the reported improvements statistically significant under multiple random seeds or sampling temperatures?\n2. Since SoftCFG strengthens high-confidence tokens, what happens if the model is confidently wrong? Could this amplify hallucination or bias?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The method is lightweight, easy to integrate into existing AR inference pipelines, and does not require additional training or data.\n2. The motivation is clear and relevant to current challenges in autoregressive generation.\n3. The paper is well-written, logically structured, concise, and clear, making it easy for readers to understand.", "weaknesses": "1. SoftCFG largely reinterprets existing CFG dynamics rather than proposing a fundamentally new principle. The main innovation lies in weighting and normalization heuristics.\n2. The reported FID gains (~0.1–0.2) are within or close to the variance commonly observed across runs and sampling seeds. It is unclear whether these improvements are statistically significant or perceptually meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)