Visual Generation Without Guidance
Authors: Huayu Chen
Kai Jiang
Kaiwen Zheng
Jianfei Chen
Hang Su
Jun Zhu
Venue: ICML 2025 poster (2025)
Keywords: Visual Generative Modeling
Guidance
Guided Sampling
CFG
Efficiency
Distillation
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: GFT proposes an efficient alternative to guided sampling, reducing the computational cost of CFG by half while maintaining comparable performance.

Abstract: Classifier-Free Guidance (CFG) has been a default technique in various visual generative models, yet it requires inference from both conditional and unconditional models during sampling. We propose to build visual models that are free from guided sampling. The resulting algorithm, Guidance-Free Training (GFT), matches the performance of CFG while reducing sampling to a single model, halving the computational cost. Unlike previous distillation-based approaches that rely on pretrained CFG networks, GFT enables training directly from scratch. GFT is simple to implement. It retains the same maximum likelihood objective as CFG and differs mainly in the parameterization of conditional models. Implementing GFT requires only minimal modifications to existing codebases, as most design choices and hyperparameters are directly inherited from CFG. Our extensive experiments across five distinct visual models demonstrate the effectiveness and versatility of GFT. Across domains of diffusion, autoregressive, and masked-prediction modeling, GFT consistently achieves comparable or even lower FID scores, with similar diversity-fidelity trade-offs compared with CFG baselines, all while being guidance-free.

Visual Generation Without Guidance
Huayu Chen*1 Kai Jiang*12 Kaiwen Zheng'
Jianfei Chen
Hang Su
Jun Zhu
Abstract
(a) CFG Training
(b) GFT (ours)
Classifier-Fre: Guidance (CFG) has been a de- fault technique in various visual generative mod- els, yet it requires inference from both condi- tional and unconditional models during sampling. We propose to build visual models that are free from guided sampling. The resulting algorithm, Guidance- Free Training (GFT), matches the per- formance of CFG while reducing sampling to a single model, halving the computational cost. Un- like previous distillation-bar
[Dog]
P
L
[Dugl
La
IMASK
[Mask]
pg
L Uncond
Implicit conditional model
Guided Sampling
G Guidance-Free Generation
PS
[Dog]
approaches that rely on pretrained CFG networks, GFT enables training directly from scratch. GFT is simple implement. It retains the same maximum like- lihood objective as CFG and differs mainly in the parameterization of conditional models. Im- plementing GFT requires only minimal modi- fications existing codebases, as most design choices and hyperparameters ited from CFG. Our extensive experiments across five distinct visual models demonstrate the ef- fectiveness and versatility of GFT. Across do- mains of diffusion, autoregressive, and masked- prediction modeling, GFT consistently achieves comparable even lower FID scores, with similar fiveestty-fidelity trade-offs compared with CFG baselines, all while being guidance-free. Code: https:/ /ttes:///tttaie /tttttbtbb./////////////g com/ thu mou-ml/Gee m1/GFT.
[Dogl
Pb
IMAS
8
Implicit sampling model
Figure 1: Comparison of GFT and CFG method. GFT shares CFG's training objective but has different parameterization technique for the conditional model. This enables direct training of an explicit sampling model.
directly inher-
By altering the guidance scale can flexibly trade off image fidelity and diversity, while significantly improving the sample quality. Due to its effectiveness, CFG has been adopted default technique for wide spectrum of visual generative models, including diffusion (Ho et al., 2020), autoregressive (AR) (Chen et al., 2020; Tian et al., 2024), and masked masked-prediction models (Chang et al., 2022; Li et al., 2023).
However, CFG is not problemless. First, the reliance on an extra unconditional model doubles the sampling cost compared with vanilla conditional generation. Second, this reliance complicates the post-training of visual models when distilling pretrained diffusion models (Meng et al., 2023; Luo et al., 2023; Yin et al., 2024) for fast inference or applying RLHF techniques (Black et al., 2023; Chen et al., 2024b), the extra unconditional model needs to be specially considered in the algorithm design. Third, this also renders a sharp difference from low-temperature sampling in language models (LMs), where single model is suffi- cient to represent the sampling distributions across various temperatures. Similarly following LMs' approach to divide model output by a constant temperature value generally ineffective in visual sampling (Dhariwal & Nichol, 202 1), even for visual AR models with similar architecture LMS
1. Introduction
Low-temperature sampling is critical technique for en- hancing generation quality by focusing only on the model's high-likelihood areas. Visual models mainly achieve this via Classifier-Frss Guidance (CFG) (Ho Salimans, 2022). As illustrated Fig. (left), CFG jointly optimizes the target conditional model and extra unconditional model during training, and combines them to define the sampling process.
Equal contribution 'Deaarment of Computer Science & Tech- nology, Tsinghua University ShunngShu, Beijing, China. Corre- spondence to: Jun Zhu <deszj @ tsinghua.cdu.cn>>
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Visual Generation Without Guidance
B 1.0
== 0.5
8 = 0.25
B == 0.1
Figure 2: Impact of adjusting GFT sampling temperature 3 for guidance-free DiT-XL/2. GFT achieves similar results to CFG without requiring dual model inference at each step. More examples are in Figure 13.
(Sun et al., 2024). All these lead us to ask, can we effectively control the sampling temperature for visual models using one single model?
with most design choices and hyperparameters inherited.
We verify the effectiveness and efficiency of GFT in both class-to-image and text-to-image tasks, spanning distinc- tive types of visual models: DiT (Peebles & Xie, 2023), VAR (Tian et al., 2024), LlamaGen (Sun et al., 2024), MAR (Li et al., 2024) and LDM (Rombach et al., 2022). Across all models, GFT enjoys almost lossless FID in fine-tuning existing CFG models into guidance-free models (Sec. 5.2). For instance, we achieve guidance- FID of 1.99 for the DiT-XL model with only 2% of pretraining epochs, while the CFG performance is 2. This surpasses previous distillation and alignment methods in their respective appli- cation domains. GFT also demonstrates great superiority in building guidance-free models from scratch. With the same amount of training epochs, GFT models generally match or even outperform CFG models, despite being 50% cheaper in sampling (Sec. 5.3). By taking in temperature parameter model input, GFT can achieve a flexible diversity-fidelie trade-off similar to CFG (Sec. 5.4).
Existing attempts like distillation methods for diffusion mod- els (Meng et al., 2023; Luo et al., 2023; Yin et al., 2024) and alignment methods for AR models (Chen et al., 2024b) are not ultimate solutions. They all rely heavily on pretrained CFG networks for loss definition and do not support train- ing guidance-free models from scratch. Their two-stage optimization pipeline may also lead to performance loss compared with CFG, even after extensive tuning. General- izability is also concern. Current methods are typically tailored for either continuous diffusion models or discrete AR models, lacking the versatility to cover all domains.
We propose Guidance-FFree Training (GFT), a foundational algorithm for building visual generative models with no guidance. GFT matches CFG in performance while requir- ing only single model for lemperature-controllee sam- pling, effectively halving sampling costs compared with CFG. GFT offers stable and efficient training with the same convergence rate as CFG, almost no extra memory usage, and only 10-20% additional computation per training up- date. GFT highly versatile, applicable in all visual do- mains within CFG's scope, including diffusion, AR, and masked models.
2. Background
2.1. Visual Generative Modeling
Continuous diffusion models. Diffusion models (Ho et al., 2020) define forward process that gradually injects noises into clean images from data distribution p(æ)::
The core idea behind GFT is to transform the desired sam- pling model into easily learnable forms. GFT optimizes the same conditional objective as CFG. However, instead of aiming to learn an explicit conditional network, GFT detines the conditional model implicitly the linear in- terpolation of sampling network and the unconditional network (Figure 1). By training this implicit model, GFT directly optimizes the underlying sampling network, which is then employed for visual generation without guidance. In essence, one can consider GFT simply as conditional pa- rameterization technique in CFG training. This perspective makes GFT extremely easy implement based on existing codebases, requiring only few lines of modifications and
at = 0x + 026,
where E [0, 1]. and e is standard Gaussian noise, Oz,, oz defines the denoising schedule. We have
p((x.) f N(xxlou (p)x)dd
where Po(x:) p(xx) and pr 100,
Given data following p(æ, c). we can train conditional dif- fusion models by predicting the Gaussian noise added to Ce..
min Ep(m..), [[[c((0/]c) - -c|[]]
(1)
9
2

---

Visual Generation Without Guidance
More formally, Song al. (2021) proved that Eq. (1) is essentially performing maximum likelihood training with evidence lower bound (ELBO), Also, the denoising model eg eventually converges to the data score function:
Though the guidance operation performs the logit space instead of the score field:
Eg (2 <en, C)
(0 Zen c) + (2n c) (z,
(2x = at Vay an pa (c)
(2)
Given ls x log it can be proved the sampling distribu- tion for discrete visual models also satisfies Eq. (5).
Given condition eg can be leveraged to generate images from pa (x)lc) by denoising noises from pi iteratively.
3. Method
Discrete AR& & masked models. AR models (Chen et al., 2020) and masked-prediction models (Chang et al., 2022) function similarly. Both discretize images æ into token sequences 21:N and then perform token prediction. Their maximum likelihood training objective can be unified as
Despite its effécctiveness, CFG requires inferencing an extra unconditional model to guide the sampling process, directly doubling the computation cost. Moreover, CFG complicates the post-training of visual generative models because the unconditional model needs to be additionally considered in algorithm design (Meng et al., 2023; Black et al., 2023).
min E (e()
Σ (e)
We propose Guidance-Free Training (GFT) an alterna- tive method of CFG for improving sample quality in visual generation without guided sampling. GFT matches CFG in performance but only leverages single model to represent CFG's sampling distribution p'(x)c).
For AR models, een represents the first i tokens in pre- determined order, and æ, is the next token to be predicted. For masked models, æ; represents all the unknown tokens that are randomized masked during training, while renn are the unmasked ones. Due to discrete modeling, the data likelihood pe in Eq. (3) can be easily calculated.
We derive GFT's training objective for diffusion models in Sec. 3.1, discuss its practical implementation in Sec, 3.2, and explain how it can be extended to discrete AR and masked models in Sec. 3.3.
2.2. Classifier-Free Guidance
Continuous CFG. In diffusion modeling, vanilla temper- ature sampling (dividing model output by constant value) is generally found ineffective in improving generation qual- ity (Dhariwal & Nichol, 2021). Current methods typically employ CFG (Ho & Salimans, 2022), which redefines the sampling denoising function e (x+lc) using two models:
3.1. Algorithm Derivation
The key challenge directly learn the target sampling model is the absence of dataset that aligns with the distribution ya(llc in Eq. (5). This makes it impractical to objective like
optimize a
(x(())) = (z,(c)) + (
(xx (x,)),
(4)
min (x..e) (a.
<l
where og and o respectively model the conditional data distribution p(r|c) and the unconditional data distribution p(z). In practice, ej can be jointly trained with eg. by randomly masking the conditioning data in Eq. (1) with some fixed probability.
as we cannot draw samples from p'. In contrast, training é (ax (e) and (ar) separately as in CFG is feasible because their corresponding datasets, {(x e) ~ p(x, c)} and {æ p( [<)), can be easily obtained.
To address this, reformulate Eq. (4):
According to Eq. (2), CFG's sampling distribution p² p=(xlc) has shifted from standard conditional distribution p(z|c) to
(z2l())
+ 8
(xx))
(6)
p(a/e) c)) 1 1+1
+
p(x(cc) p(a:)
(m)c)
p(x)
(xlc) p(alee
(5)
Learnable
Target sampling model
Learnable
Although learning c directly is difficult, note it can be combined with unconditional model e to represent the standard conditional Cg which is learnable. Thus, we can leverage the same conditional loss in Eq. (1) to train ej. namely:
CFG offers an effective approach for lowering sampling temperature in visual generation by simply increasing s > 0, thereby substantially improving sample quality.
Discrete CFG. Besides diffusion, CFG also a critical
min Ep || 1 (x + 1+s
8 (x,) - e11
sampling technique in discrete visual modeling (Li et al., 2023; Team, 2024; Tian et al., 2024; Xie et al., 2024).
3

---

(1) Alignment with CFG. The practical GFT algorithm (Eq. Visual Generation Without Guidance
Algorithm 1 Guidance-Free Training (Diffusion)
CFG
1: Initialize 0 from pretrained models or from scratch.
x1.54
x1.50
GFT (wlo sg)
2: for each gradient step do
x1.19
GFT (w/
x1.00
3: // CFG training pseudo-tenperature
x1.00
4: x, c ~ p(a, c)
5:
[0, 1), 8 1))
x0.50x0.55 50
6:
e~ 19)
7:
+
8:
Cg masked by Ø with 10% probability
Training time Updatte
GPU memory usage
# Inference Sample
9:
Calculate eă (z. Caa, (3 in training mode
Figure 3: Comparison of computational efficiency between GFT and CFG. Estimated based the DiT-XL model.
10:
// Additional to CFG
11:
Calculate 6 (mx 25, in evaluation made
12:
= (2x Ca, (3 + (1 B)sg.cā (xx
// Standard Maximun Likelihood Training
14:
0 - 0
XVolcco - <cl2 (Eq. 9)
Stopping the unconditional gradient. The main differ- ence between Eq. (9) and Eq. (8) is that ég computed in evaluation mode, with model gradients stopped by the sg] operation. To train the model unconditionally, mask conditions with 2 when computing eg. We show this design does not affect the training convergence point:
end for
randomly
where 6 is standard Gaussian noise, 0t = 0g + drE are diffused images. O2 and o, define the forward process.
Theorem 1 (GFT Optimal Solution). Given unlimited model capacity and training data, the optimal că- for opti- mizing Eq. (9) and Eq. (8) are the same. Both satisfy
To this end, we have a practical algorithm for directly learn- ing guidance-free models eÿ. However, unlike CFG which allows controlling sampling temperature by adjusting guid- ance scale to trade off fidelity and diversity, our method still lacks similar inference-time flexibility as Eq. (7) is performed for a specific 8.
cg- (xx cc, .3)
1 0z
Va, logp:( - - 1)V logp((at))
To solve this problem, we define a pseudo-temperatture = 1/(1+ s) and further condition our sampling model <x,lc,,, B) on the extra B input. We can randomly sample e [0, 1] during training, corresponding to [0, +0o).
Proof. In Appendix B.
The stopping-gradient technique has the following benefits:
The GFT objective in Eq. (7) now becomes:
9) differs from CFG training by single unconditional in- ference step. This allows us implement GFT with only few lines of code based existing codebases.
min E tf [1.3e](2,1e,,)) 8) + (1- (1=)) <|
(8)
When B = 1, Eq. (8) reduces to conditional diffusion loss || (x, <ll When 8 = 0, Eq. (8) becomes unconditional loss (x,) -- ell2. This allows simultaneous training of both conditional and unconditional models.
(2) Computational efficiency. Since the extra unconditional calculation is gradient-free. GFT requires virtually no extra GPU memory and only 19% additional train time per update vs. CFG (Figure 3). This stands in contrast with the naive implementation without gradient stopping (Eq. 8), which is equivalent to doubling the batch size for CFG training.
As pseudo-temppratuuee 3 decreases 1 + O, the modeling target for eg gradually shifts from conditional data distri- bution p(zr c) to lower-temperature distribution p² (a: defined by Eq. (5) (See Fig. 2).
(3) Training stability. We empirically observe that stopping the gradient for the unconditional model could lead to better training stability and thus improved performance.
3.2. Practical Implementation
A desirable algorithm should not only ensure soundness but also offer computational efficiency, seamless integration, and practical deployability. To achieve this, we further present Eq. (9) as practical loss function of GFT. The implementation in Algorithm
Input of B. GFT requires an extra pseudo-temperature imput comparison with CFG. For this. we first process using the similar Fourier embedding method for diffusion time t (Dhariwal & Nichol, 2021; Meng et al., 2023). This is followed by some MIP layers. Finally, the temperature embedding is added to the model's original time or class embedding. fine-tuning, we apply zero initialization for the final MLP layer so that would not affect model output at the start of training.
cx, Co, 3) (2, 8) + (1 1-))s ()sg(e"(0,12), 1)] -ell (9)

---

Visual Generation Without Guidance
Table 1: Comparison of GFT (ours) and other guidance-free methods. Numbers are reported based on experiments for the DiT-XL model or the VAR-d30 model. We use x 80GB H100 GPU cards.
Method
Guidance Distillation
Condition Contrastive Alignment
Guidance-Free Training
Modeling target
P + p3 - pf(()c))
pd + P& -
p(20/e) p(a)
Pg + P/ - -p(xcc)
Applicable area
Diffusion
AR & Masked
All
Loss form Reference equation
||0 - (1+s)E -es0g-se,) selll2
llogo (r)) (-rā); ro= log PA P
356 + (1- + Diff/R Loss
Eq. (13)
Eq. (14)
Eq. (9) & Eq. (11)
Train from scratch?
Not allowed
Not allowed
Allowed
# Inference / Update
3
4
2 x 1.00
Train time Update
x x1.19
x x1.69
GPU memory usage
x 1.15
x 1.39
x 1.00
Hyperparameters. Due to the high similarity between CFG and GFT training, we inherit most hyperparameter choices used for existing CFG models and mainly adjust parameters like learning rate during finetuning. When train- ing from scratch, we find simply keeping all parameters the same with CFG is enough to yield good performance.
models and aligmment methods for AR models. Alongside GFT, these methods all transform the sampling distribu- tion p into simpler, Icarnable forms, differing mainly in how they decompose the sampling distribution and set up modeling targets (Table 1).
Guidance Distillation (Meng al., 2023) is quite straight- forward, it simply learns single model to match the output of pretrained CFG targets using L2 loss:
Training epochs. When fine-tuning pretrained CFG mod- els, we find 1% 5% of pretraining epochs are sufficient to achieve nearly lossless FID performance. When from scratch, we always use the same training epochs compared with the CFG baseline.
cg = (,, 8)
(xx)) (13)
(2x))
where c and cg. are pretrained models. L breaks down the sampling model into a linear combination of conditional and unconditional models, which can be separately learned.
3.3. GFT for AR and Masked Models
Similar to Sec. 3.1, we can derive the GFT objective for AR and masked models as standard cross-entropy loss:
Despite being effective, Guidance distillation relies on pre- trained CFG models as teacher models, and cannot be lever- aged for ttom-scratch training. This results in an indirect, two-stage pipeline for learning guidance-fres models. In comparison, our method unifies guidance-frr training in one singular loss, allowing learning in an end-to-end style. Besides, GFT no longer requires learning an explicit con- ditional model cg. This saves training computation and VRAM usage. A detailed comparison is in Table 1.
CAR (x, ca, B) = Σ log (x xen, CB, B)
(10)
e (alealmccns..1)
E log Ewey ca
(11)
where w is a token in the vocabulary V. and
Eg (5(w 3)
Co. 3) + (1 -3)gg ()sg(( w)) (12)
Condition Contrastive Alignment (Chen et al., 2024b) constructs a preference pair for each image æ in the dataset and applies similar preference alignment techniques for language models (Rafailov al., 2023; Chen et al., 2024a) to fine-tune visual AR models:
=
In Sec. 5, we apply GFT to wide spectrum of visual generative models, including diffusion, AR, and masked models, demonstrating its versatility.
4. Connection with Other Guidance-Free Methods
CCCA logo [ro(0, c²) logo [-ro e²)], (14)
where cº the preferred positive condition corresponding to the image x, c" a negative condition randomly and in- dependently sampled from the dataset. Given conditional
Previous attempts to remove guided sampling from visual generation mainly include distillation methods for diffusion
5

---

Visual Generation Without Guidance
Table 2: Model comparisons on the class-conditional Ima- geNet 256 x 256 benchmark.
Table 3: Model comparisons for zero-shot text-to-image
generation on the COCO 2014 validation set.
FID !
FID
Model Type
w/o Guidance
w/ Guidance
Text Image Models
w/o Guidance
w/ Guidance
Diffusion Models
GLIDE (Nichol al., 2021)
12.24
ADM (Dhariwal & Nichol, 2021)
7.49
3,94
LDM (Rombach et al., 2022)
12.63
DALL-E (Ramesh
2022)
LDM-4 (Rombach et al., 2022)
10.56
3.60
Stable Diffusion 1.5 (Rombach et al., 2022)
22.55
7.87
U-ViT-H/2 (Bao et al., 2023)
2.29
+ Distillation (Meng et al,, 2023)
8.16
MDTv2-XL/2 (Gao al., 2023)
5,06
1.58
GFT (Ours)
8.10
DiT-XL/2 (Peebles Xic, 2023)
9.34
2.11
-Distillation (Meng al., 2023)
2.11
-GFT (Ours)
3. GFT's capability of controlling diversity-fidelity trade- off through temperature parameter [3. (Sec. 5.4)
Autoregressive Models
VQGAN (Esser al., 2021)
15.78
5.20
VIT-VQGAN (Yu et al. 2021)
4.17
3.04
RQ-Transformer (Lee al., 2022)
7.55
3,80
LlamaGen-3B (Sun
2024)
9.44
2.22
5.1. Experimental Setups
CCA (Chen al., 2024b)
2.69
-FFT (Ours)
2.21
Tasks & Models. We evaluate GFT in both class-to-image (C2I) and text-to-image (T2I) tasks. For C2I, we exper- iment with diverse architectures: DiT (Peebles Xie, 2023) (transformer-basse latent diffusion model), MAR (Li et al., 2024) (masked-token prediction model with diffu- sion heads), and autoregressive models: VAR (Tian et al., 2024) and LlamaGen (Sun et al., 2024). For T2I, we use Stable Diffusion 1.5 (Rombach et al., 2022), a text-to-image model based on the U-Net architecture (Ronneberger et al., 2015), to provide a comprehensive evaluation of GFT's per- formance across various conditioning modalities. All these models rely guided sampling critical component.
VAR-d30 (Tian et al., 2024)
5.26
1.92
CCA (Chen et al., 2024b)
2.54
-FTT (Ours)
1.91
Masked Models
MaskGIT (Chang
2022)
6.18
MAGVIT-v2 (Yu al., 2023b)
3.65
1.78
MAGE (Li et al., 2023)
6.93
MAR-B (I.i et al., 2024)
4.17
2.27
GFT (Ours)
2.39
reference model ps. the implicit reward is defined as
1 P5 ( c) p (xlc)
(2 c) == log
Training Evaluation. We train C2I models on 1mageNet-256x256 (Deng et al., 2009). For T2I models, we subset of the LAION-Aesthetic 5+ (Schuhmann et al., 2022), consisting of 18 million image-text pairs. Our codebases are directly modified from the official CFG im- plementation of each respective baseline, keeping most hy- perparameters consistent with CFG training. We use official OPENAI evaluation scripts to evaluate our C2I models. For T2I models, we evaluate our model on zero-shot COCO 2014 (Lin et al., 2014). The training and evaluation details for each model can be found in Appendix D.
CCA proves the optimal solution for solving Eq. (14) is r log p(ale) thus the convergence point for p (a/cc also p(z) Satisfies Eq. (5).
Both CCA and GFT train sampling model p directly by combining it with another model to represent a Iearnable distribution. GFT leverages 3pg + =11 8)pg to represent standard conditional distribution p(æ) c). while CCA com- bines p and pretrained Po (a)(e) to represent the conditional residual log p(ælc),
p(x) They also differ in applicable areas. CCA is based on language alignment losses, which requires calculating model likelihood log during training. This forbids its direct application to diffusion models, where calculating exact likelihood is infeasible.
5.2. Make CFG Models Guidance-Free
Method Effectiveness. In Table 2 and apply GFT to fine-tune wide spectrum of visual generative models. With less than 5% pretraining computation, the fine-tuned models achieve comparable FID scores to CFG while being 2x faster sampling. Figure 4 visually demonstrates this quality improvement.
5. Experiments
Our experiments aim to investigate:
1. GFT's effectiveness and efficiency in fine-tuning CFG models into guidance-free variants (Sec. 5.2)
Comparison with other guidance-free approaches. GFT achieves comparable performance to guidance distilla- tion (designed for diffusion models) and outperforms AR alignment method (Table and 3). Notably, GFT demon-
2. GFT's ability in training guidance-frr models from scratch, compared with classic CFG training (Sec. 5.3)
6

---

Visual Generation Without Guidance
Vanilla conditional generation (w/o CFG)
w/ CFG
GFT (w/o Guidance)
Figure 4: Qualitative T21 comparison between vanilla conditional generation, GFT, and CFG on Stable Diffusion 1.5 with prompt "Elegant crystal vase holding pink peonies, soft raindrops tracing paths down the window behind it". More examples are in Figure 14.
strates superior efficiency compared to both methods (Ta- ble 1). We attribute this effectiveness to GFT's end-to-end training style, and to the nice convergence property of its maximum likelihood training objective.
0.24
CFG (Base)
7.3
CFG (Base)
Train Lets
GFT
GFT
9.15
Step
750k
Finetuning Efficiency Figure 5 tracks the FID progres- sion of DiT-XL/2 during fine-tuning. The guidance-fiee feeee FID rapidly improves from 9.34 to 2.22 in the first epoch, followed by steady optimization. After three epochs, our model achieves better FID than the CFG baseline (2.05 2.11). This computational is almost negligible compared with pretraining, demonstrating GFT's efficiency.
Figure 6: Comparison of convergence speed between GFT and CFG on diffusion and autoregressive models in from- scratch training.
Performance. Table 4 shows that GFT models trained from scratch outperform CFG baselines across DīT-B/2, MAR-B, and LlamaGen-L models, while reducing evalu- ation costs by 50%. Notably, these from-scratch models outperform their fine-tuned counterparts, demonstrating the advantages of direct guidance-free free training.
9.34
DiT-XL/2
5.72
GFT
FID
CFG
(28, 1.99
2.11
01
3
10
28
GFT training epochs
Training stability. An informative indicator of al- gorithm's stability and scalability is its loss convergence speed. With consistent hyperparameters, we find GFT con- vergences at least as fast as CFG for both diffusion and autoregressive modeling (Figure 6). Direct loss comparison is valid as both methods optimize the same objective: the conditional modeling loss for the dataset distribution. The only difference is that the conditional model for CFG is single end-to-end model, while for GFT it is constructed as linear interpolation of two model outputs.
Figure Efficient convergence of FID scores for GFT using DiT-XL/2 model across training epochs.
5.3. Building Guidance-Free Models from Scratch
Training Guidance-Free Models from scratch is more tempt- ing than the two-stage pipeline adopted by See 5.2. However, this is also more challenging due to higher requirements for the algorithm's stability and convergence speed. We inves- tigate this by comparing from-scratch GFT training with classic supervised training using CFG across various archi- tectures, maintaining consistent training epochs. We mainly focus on smaller models due to computational constraints.
Based on the above observations, we believe that GFT is at least as stable and reliable as CFG algorithms, providing a new training paradigm and viable alternative for visual generative models.

---

Visual Generation Without Guidance
DiT-B/2
MAR-B
LlamaGen-L
VAR-d16
Method
Guidance
(Peebles & Xie, 2023)
(Li al.,
2024)
(Sun al., 2024)
(Tian et al., 2024)
FID
IS
FID
IS
FID
IS
FID
IS f
Base
w/o
44.8
30.7
4.17
175.4
19.0
64.7
11.97
105.5
+CFG
w/
9.72
161.5
2.27
260.7
3,06
257.1
3.36
280.1
+GFT
w/o
10.9
128.5
2.39
264.7
2.88
238.4
3.28
251.3
GFT
w/o
9.04
166.6
2.27
279.5
2.52
270.5
3.42
254.8
Table 4: Performance comparison between GFT from-scratch training, CFG, and GFT fine-tuning variants across different model architectures. GFT and the base model are trained for the same number of epochs.
5.4. Sampling Temperature for Visual Generation
DFT-XL/2
CFG
A key advantage of CFG is its flexible sampling temperature for iiversity-fidellin trade-offs. Our results demonstrate that GFT models share this capability.
10
Distill
GFT
FID
We evaluate diversity-fidelity trade-off's els, with FID-IS trade-off for c2i models and FID-CLIP trade-off for 12i models. Results for DiT-XI/2 (fine-tuning), LlamaGen-L. (from-scratch training) and Stable Diffusion 1.5 (fine-tuning) are shown in Figures and 8, with addi- tional trade-off curves provided in Appendix C. For DiT- XI.2 and Stable Diffusion 1.5, we also compare GFT with Guidance Distillation, showing that GFT achieves results comparable CFG while outperforming Guidance Distilla- tion on CLIP scores.
various mod-
200
300
400
LlamaGen-L
15
CFG
GFT
FID
Figure 2 shows how adjusting temperature 3 produces ef- fects similar to adjusting CFG's scale s. This similarity results from both methods aiming to model the same dis- tribution (Eq. 5). The key difference that GFT directly learns series sampling distributions controlled by through training, while CFG modifies the sampling process to achieve comparable results.
5-
100
200
300
400
Inception Score (IS)
Figure 7: FID-IS trade-off comparisons ImageNet. Up- per: DiT-XL/2 with GFT (fine-tuned), CFG, and Guidance Distillation. Lower: LlamaGen-L with GFT (trained from scratch) and CFG.
6. Conclusion
In this work, we proposed Guidance-Free Training (GFT) as an alternative to guided sampling in visual generative models, achieving comparable performance Classifier- Free Guidance (CFG). GFT reduces sampling computational costs by 50%. The method is simple to implement, requiring minimal modifications to existing codebases. Unlike previ- ous distillation based methods, GFT enables direct training from scratch.
20
EDD
15
SD 1.5
CFG
Distill
10
GFT
Our extensive evaluation across multiple types of visual models demonstrates GFT's effectiveness. The approach maintains high sample quality while offering flexible con- trol over the diversity-fideli trade-off through temperature adjustment. GFT represents an advancement in making high-quality visual generation more efficient and accessible.
0.26
0.28
0.30
0.32
CLIP Score
Figure 8: FID-CLIP trade-off comparison on COCO-2014 validation set. Methods compared using Stable Diffusion 1.5.
8

---

Visual Generation Without Guidance
Acknowledgement
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for con- trastive aunuuge-image learning, In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pp. 2818, 2023.
We thank Huanran Chen, Xiaoshi Wu, Cheng Lu, Fan Bao, Chengdong Xiang, Zhengyi Wang, Chang Li and Peize Sun for the discussion. This work was supported by the NSFC Project (No. 62376131, 92270001, 92370124, 92248303) and the High Performance Computing Center, Tsinghua University, J.Z is also supported by the XPlorer Prize.
Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., and Ye, J. C. Diffusion posterior sampling for general noisy in- verse problems. arXiv preprint arXiv:2209. 0222099.1677,,. 2022.
Impact Statement
Our Guidance-Free Training (GFT) method significantly reduces the computational costs of visual generative models by eliminating the need for dual inference during sampling, contributing to more sustainable AI development and re- duced environmental impact. However, since our method accelerates the sampling process of generative models, it could potentially be misused to create harmful content more efficiently, emphasizing the importance of establishing ap- propriate safety measures and deploying these models sponsibly with proper oversight mechanisms.
Chung, H., Kim, J., Park, G. Y., Nam, H., and Ye, J. C.
Cfg++: Manifod--oonstrineed classifier free guidance for diffusion models. arXiv preprint vrXiv22406.08070 2024.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li. K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and patlern recognition, pp. 248 255.. leee, 2009.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 0794, 2021.
References
Bao, F., Nie, S., Xue, K., Cao, Y., Li, All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14669-22679, 2023.
C.,, Su, H., and Zhu,
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision vision and pattern recognition, pp. 22873-12882 2021.
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint arXiv: 13301, 2023.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F, Podell, D., Dockhorn, T., English, Z., Lacey, K., Good- win, A., Marek, Y., and Rombach, R. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
Chang, H., Zhang, H., Jiang, I.., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 12915-11325, 2022.
Fan, L.,, Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M., Sun, D., He, K., and Tian, Y. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410. 13863, 2024.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image genera- tion via masked generative transformers. arXiv preprint arXiv:2301.000 00704, 2023.
Gao, S., Zhou, P., Cheng, M.-M., and Yan, S. Masked diffusion transformer is a strong image synthesizer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23164-23173, 2023.
Chen, H., He, G., Su, H., and Zhu, J. Noise contrastive align- ment of language models with explicit rewards. Advances in neural information processing systems, 2024a.
Geng, Z., Pokle, A., Luo, W., Lin, J., and Kolter, J. Z. Consistency models made easy, arXiv preprint urXiv:2406.1455 14548, 2024.
Chen, H., Su, H., Sun, P., and Zhu, J. Toward guidance-fres ar visual generation via condition contrastive alignment. arXiv preprint arXiv:2410.0934/ 2024b.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In International conference on machine learning, pp. 1691 1703. PMLR, 2020.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6857 2020.

---

Visual Generation Without Guidance
Hong, S., Lee, G., Jang, W., and Kim, S. Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 7462-7471,202
representation learning and image synthesis. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2142-2152, 2023.
Li, T., Tian, Y., Li, H., Deng, M., and He, Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406. /838, 2024.
Ilharco, G., Wortsman, M., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Ha- jishirzi, H., Farhadi, A., and Schmidt, L. Openclip, July 2021. URL https: /do:i wrw/10.../.... 5281/ zenodo.. 5143773.
Lin, S. and Yang, X. Diffusion model with perceptual loss. arXiv preprint arXiv:2401. 10, 2023.
Lin, S., Wang, A., and Yang. X. Sdxl-lightning Progres- sive adversarial diffusion distillation. arXiv preprint arXiv:2402. 13929, 2024.
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E.,, Paris, S., and Park, T. Scaling up gans for texl-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10124 10134, 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 26565-26577, 2022.
Lu, C. and Song, Y. Simplifying, stabilizing and scal- ing continuous-time consistency models. arXiv preprint arXiv:2410.1108. 2024.
Karras, T., Aittala, M., Kynkäänniemi, T., Lehtinen, J., Aila, T., and Laine. S. Guiding a diffusion model with bad version of itself. arXiv preprint (xXi:22406002507 2024.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models, arXiv preprint arXiv:2211.01095. 2022.
Kim, D., Kim, Y., Kwon, S. J., Kang, W., and Moon, L.- C. Refining generative process with discriminator guid- ance in score-based diffusion models. arXiv preprint (tti::2211.1099) 2022.
Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Con- trastive energy prediction for exact energy gerrgy- diffu- sion sampling in offline reinforcement learning. In Inter- national Conference on Machine Learning, pp. 22825- 22855. PMLR, 2023.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Koulischer, F., Deleu, J., Raya, G., Demeester, T., and Am- brogioni, L. Dynamic negative guidance of diffusion models: Towards immediate content removal. In Neurips Safe Generative Al Workshop 2024.
Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H. La- tent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04376 2023.
Kynkälänniemi. T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. Applying guidance in limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724 2024.
Ma, X., Zhou, M.. Liang, T., Bai, Y., Zhao. T., Chen, H., and Jin, Y. Star: Scale-wise text-to-image genera- tion via auto-regressive representations. arXiv preprint arXiv:2406. 10797, 2024.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, T. On distillation of guided diffu- sion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 14297-14306, June 2023.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S. Autore- gressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 11523-11532, 2022.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12,741 2021.
Li, T. Chang, H., Mishra, S., Zhang, H., Katabi, D., and Krishnan, Mage: Masked generative encoder unify
10

---

Visual Generation Without Guidance
Parmar, G., Zhang, R., and Zhu, J.-Y. On aliased resizing and surprising subtleties in gan evaluation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11410-11420, 2022.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294 2022.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Shenoy, R., Pan, Z.,, Balakrishnan, K., Cheng, Q., Jeon, Y., Yang, H., and Kim, J. Gradient-free classifier guid- ance for diffusion model sampling. arXiv preprint arXi::2411.15393 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamies. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Song. J., Meng, C.,, and Ermon, Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502, 2020a.
Song, J., Zhang, Q., Yin, H., Mardani, M.. Liu, M.-Y., Kautz, J., Chen, Y., and Vahdat, A. Loss-guided diffu- sion models for plug-an and-play controllable generation. In International Conference on Machine Learning, pp- 32483-32498. PMLR, 2023a.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text- to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical cont-conditional image generation with clip latents. arXiv preprint arXiv:2204.00 0615, 1(2):3, 2022.
Song. Y., Sohl- J., Kingma. D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 133456,, 2020b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffuuson models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp- 10684-10695, 2022.
Song, Y., Durkan, C.,, Murray, I., and Ermon, S. Maxi- mum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34: 2021
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and cosputer-assisted intervention-MICCA. 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Song. Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. arXiv preprint arXiv:2303.01460 2023b.
Sun, P., Jiang. Y., Chen, S., Zhang. S., Peng, B.. Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv: 2406.065255 2024.
Saharia, C., Chan, W. Saxena, S., Li, L., Whang. J., Denton, E. L.. Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in Neural Information Processing Systems, 35: 36479-36494, 2022.
Tang, H., Wu, Y., Yang, S., Xie, E., Chen, J., Chen, J., Zhang, Z., Cai, H., Lu, Y., and Han, S. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arXiv:2410.10812 2024.
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Ad- versarial difffusion distillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2025a.
Team, C. Chameleon: Mixed-modal early- foundation models. arXiv preprini arXiv:2405. 09800 2024.
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Ad- versarial diffuusin distillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2025b.
Tian, K., Jiang, Y., Yuan, Z., Peng. B., and Wang, L. Visual autoregressive modeling: Scalable image generation via

---

Visual Generation Without Guidance
next-scale prediction. arXiv preprini arXiv:444..02005 2024.
of pretrained diffusion models for one-step generation. In Forty-first International Conference Machine Learn- ing, 2024c.
Xie, J., Mao, W., Bai, Z., Zhang, J. Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. and Park, T. One-step diffusion with distribution matching distillation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 6613-6623 2024.
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:21 2tff:2210004627. 2021.
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang. H., Chang, H., Hauptmann, G., Yang, M.-H., Hao, Y., Essa, I., et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pultern Recognition, pp. 10469, 2023a.
Yu, L., Lezama, J., Gundavarapu, B., Versari, L., Sohn, K., Minnen, D., Cheng, Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprini arXiv:2310.05733 2023b.
Yu, Q., Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.C. An image is worth 32 tokens for reconstruc- tion and generation. arXiv preprint arXiv:2406.07550 2024.
Zhang, Q. Dai, X., Yang, N., An, X., Feng, Z., and Ren, X. Var-clip: Text-to-image generator with visual auto- regressive modeling. arXiv preprint arXiv:2408.01 181, 2024.
Zhao, M., Bao, F., Li, C., and Zhu, J. Egsde: Unpaired image-to- tongg-et- translation via energy-gun guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623 2022.
Zhou, M., Wang, Z.., Zheng, H., and Huang, H. Long and short guidance in score identity distillation for one-step text-to-image generation. arXiv preprint CrXiv:2406.0156h) 2024a.
Zhou, M., Zheng, H., Gu, Y., Wang, 7., and Huang, H. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. arXiv preprint 41X1v22410.14919 2024b.
Zhou, M., Zheng, H., Wang. Z., Yin, M., and Huang, H. Score identity distillation: Exponentially fast distillation
12

---

Visual Generation Without Guidance
A. Related Work
Visual generation model with guidance. Visual generative modeling has witnessed significant advancements recent years. Recent explicit-likelie likelihod approaches can be broadly categorized into diffusion basodd models SScht-Dickstein et al., 2015; Song & Ermon, 2019; Ho al., 2020; Song et al., 2020b; Dhariwal Nichol, 2021; Kingma al., 2021; Rombach al., 2022; Ramesh al., 2022; Saharia al., 2022; Karras al., 2022; Bao et al., 2023; Peebles & Xie, 2023; Esser et al., 2024; Xie et al., 2024), auto-regressive models (Chen et al., 2020; Esser et al., 2021; Ramesh et al., 2021; Yu et al., 2021;
Tian et al., 2024; Team, 2024; Sun et al., 2024; Ma et al., 2024; Zhang et al., 2024; Tang et al., 2024), and musskedpreeicction models (Chang et al., 2022; Yu et al., 2023a; Chang et al., 2023; Li et al., 2024; Yu et al., 2024; Fan et al., 2024). The introduction of guidance techniques has substantially improved the capabilities of these models. These include classifier guidance (Dhariwal Nichol, 2021), classifier-fre guidance (Ho Salimans, 2022), energy guidance (Chung et al., 2022; Zhao et al., 2022; Lu et al., 2023; Song al., 2023a), and various advanced guidance methods (Kim al., 2022; Hong et al., 2023; Kynkäänniemi al., 2024; Karras et al., 2024; Chung et al., 2024; Koulischer et al.: Shenoy al., 2024).
Guidance distillation. To address the computational overhead introduced by classifier-free guidance (CFG), One widely used approach to remove CFG is guidance distillation (Meng et al., 2023), where student model is trained to directly learn the output of a pre-trained teacher model that incorporates guidance. This idea of guidance distillation has been widely adopted in methods aimed at accelerating diffusion models (Luo et al., 2023; Yin et al., 2024; Lin et al., 2024; Zhou et al., 2024a). By integrating the teacher model's guided outputs into the training process, these approaches achieve efficient few-step generation without guidance.
Alternative Methods for Building Guidance-free Models. Recent studies in diffusion models show that perceptual
losses (Lin & Yang, 2023), score-based distillation (Sauer al., 2025a; Zhou al., 2024c; Sauer et al., 2025b; Zhou et al., 2024b), and consistency models (Song al., 2023b; Geng al., 2024; Lu & Song, 2024) can also achieve comparable FID scores to CFG. For auto-regressive models, Condition Contrastive Alignment (Chen et al., 2024b) could enhance guidance-free performance through alignment (Rafailov et al., 2023; Chen et al., 2024a) in self-contrastive manner.
13

---

Visual Generation Without Guidance
B. Proof of Theorem 1
first copy the training objective Eq. (9) and Eq. (8).
Epiz.e). [1 B) (xc)(()) c|1²]
(15)
Opractical
Eelx.ce 3e 8) + (1 - B)sg(a (z (0, 1)] cll2.
(16)
Theorem (GFT Optimal Solution), Given unlimited model capacity and training data, the optimal egg for optimizing Eq. (15) and Eq. (16) are the same. Both satisfy
1 vx (ogpe(()lc)) - - 1)V log 3
(2
Proof. The proof is quite straightforward.
First consider the unconditional part of the model. Let / = 1 in L we have
CEAW (,,,. (x,, <
which is standard unconditional diffusion loss. According to Eq. (2) we have
Eg. (æt) o Va, æ, log Pc
(17)
Then prove for v E (0, 1], stopping the unconditional gradient does not change this optimal solution. Taking derivatives of epractical
L
we have:
Vo
practiaal
Co
Ø) = Ep(a),t.e. (1-
- =
Tueexxx..22Tooocca1l33311ll11111 (0, c|l2
[[c] (z,) c|
[2 Cra (B
Since [2E,3] is a constant, this does not change the convergence point c The optimal unconditional solution for ppractical remains the same.
Cp
For the conditional part of the model, since both 2 and 2
Epractical are standard conditional diffusion loss, we have
Beg (2 + 9)c (x:) or Va, (x;
Combining Eq. (17), we have
c (x, -ot
va, (ote) -1) log
Let 1, we can see that GFT models the same sampling distribution as CFG (Eq. 4).
14

---

Visual Generation Without Guidance
C. Additional Experiment Results.
LlamaGen-3B
VAR-d30
CFG
6
+CFG
CCA
+CCA
GFT
GFTT
2
2
100
200
300
400
200
250
300
350
400
Inception Score (IS)
Inception Score (IS)
DIT-XL/2
4.0
MAR-B
+ CFG
CFG
10
Distill
3.5
GFT
FID
GFT
3.0
5
200
300 Inception Score (IS) f
400
200
250
300
350
Inception Score (IS) f
Figure 9; FID-IS trade-off comparison in fine-tuning experiments.
LlumaGen-L
VAR-d16
15
CFG
15
GFT
GFT
CFG
FID
FID
10
5
5
100
300
100
300
400
Inception Score (IS)
Inception Score (IS)
DiT-B/2
4.0
MAR-B
40
CFG
CFG
GFT
3.5
GFT
30
FID
3.0
20
2.5
10
50
100
150
200
200
250 Inception Score (IS) t
300
350
Inception Score (IS)
Figure 10: FID-IS trade-off comparison in hoim-scratch-trainnng experiments.
15

---

Visual Generation Without Guidance
0.9
0.9
Presicion
Presicion
0.8
0.8
DIT-XL/2
LlamaGGen3B
CFG
CFG
0.7
GFT
0.7
GFT
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.4
0.5
0.6
Recall 1
Recall
Figure 11: Precision-Recal trade-off comparison for DiT and LlamaGen in fine-tuning experiments.
0.6
0.9
Presicion
0.4
Presieion
0.8
D/T-B/2
LlamaGen-L
0.7
0.2
+ CFG
+ CFG
GFT
GFT
0.6
0.5
0.6
0.7 Recall f
0.8
0.9
0.2
0.3
0.4
0.5
0.6
Recall 1
Figure 12: Precision-Resed Recaal trade-off comparison for DiT and LlamaGen in nonn-sraatc-trrainin experiments.
16

---

Visual Generation Without Guidance
3= 1.0
0.5
B 0.25
B
0.1
Figure 13: Additional results of temperature sampling (3) impact on DiT-XL/2 after applying GFT.
17

---

Visual Generation Without Guidance
w/o Guidance
GFT (w/ Guidance)
w/ CFG Guidance
Figure 14: Additional results of qualitative T2I comparison between vanilla conditional generation, GFT, and CFG on Stable
Diffusion 1.5.
18

---

Visual Generation Without Guidance
EVOIOL
w/o Guidance
GFT (w/o Guidance)
w/ CFG Guidance
Figure 15: Additional results of qualitative C2I comparison between vanilla conditional generation, GFT, and CFG on XL/2
19

---

Visual Generation Without Guidance
D. Implementation Details.
For all models, we keep training hyperparameters and other design choices consistent with their official codebases not otherwise stated. We employ a mix of H100, A100 and A800 GPU cards for experimentation.
DIT. We mainly apply GFT fine-tune DiT-XL/2 (28 epochs. 2% of pretraining epochs) and train DiT-B/2 from scratch (80 epochs, following the original DiT paper's 's settings (Peebles & Xie, 2023)). Since the DiT-B/2 pretraining checkpoint is not publicly available, reproduce its pretraining experiment. For all experiments, we use batch size of 256 and learning rate of le 4. For DÝT-XL/2 fine-tuning experiments, we employ a cosine-decay learning rate scheduler.
For comparison, we also fine-tune D/T-X1/2 using guidance distillation, with a scale range from 1 to 5, while keeping all other hyperparameters aligned with GFT.
The original DiT uses the old- add-aosho DDPM (Ho al., 2020) which learns both the mean and variance, while GFT is only concerned about the mean. We therefore abandon the variance output channels and related losses during training and switch to the Dpm-solver++ (Lu et al., 2022) sampler with 50 steps at inference. For reference, our baseline, DiT-XL/2 with CFG, achieves an FID of 2.1 11 using +. compared with 2.27 reported in the original paper. All results are evaluated with EMA models. The EMA decay rate is set to 0.9999.
VAR. We mainly apply GFT to fine-tune VAR-d30 models (15 epochs) or train VAR-d16 models from scratch (200 epochs). Batch size is 768. The initial learning rate is le- 5 in fine-tuning experiments and 1e pretraining experiments. Following VAR (Tian et al., 2024), we employ learning rate scheduler including a warmup and linear decay process (minimal is 1% of the initial).
VAR by default adopts pyramid CFG technique predicted logits. The guidance scale 0 decreases linearly during the decoding process. Specifically, let be the current decoding step index, and N be the total steps. The n-step guidance scale 8n is
Sp N 80-
We find pyramid CFG is crucial to an ideal performance of VAR, and thus design a similar pyramid B schedule during training:
Bn = [N- )a( 1 - -)+1
1 30
where 3n represents the token-specific B value applied in the GFT AR loss (Eq. 12). 2 0 is a hyperparameter be tuned.
When O, we have Sn, So. standing for standard GFT. When 1.0, we have +. corresponding to the default pyramid CFG technique applied by VAR. In practice, we set a = 1.5 in GFT training and find this slightly outperforms a = 1.0.
(+ 1),
Lea We mainly apply GFT to fine-tune LlamaGen-31 models (15 epochs) or train LlamaGen-L models from scratch (300 epochs). For fine-tuning, the batch size 256, and the learning rate 2e For pretraining, the batch size 768, and the learning rate is le 4. We adopt a cesane-decay learning rate scheduler all experiments.
We apply GFT to MAR-B, including both fine-tuning (10 epochs) and training from scratch (800 epochs). We find the batch size crucial for MAR and use 2048 following the original paper. For fine-tuning, we employ learning rate scheduler including a 5-epoch linear warmup to Se - and cosine decay process to le - 4. For training from scratch, we employ 100-epoch linear Ir warmup to 8e 4, followed by a constant Ir schedule, which is the same configuration as the original MAR pretraining.
MAR.
The original MAR follows the old-fashioned DDPM (Ho et al., 2020) which learns both the mean and variance, while GFT only concerned about the mean. We therefore abandon the variance output channels and related losses during training and switch to the DDIM (Song al., 2020a) sampler with 100 steps at inference. As the & condition may not precisely capture the effects of the guidance scale after training, we tune the inference B schedule to maximize the performance. Specifically, we adopt power-cosine schedule
cos((nn - 1))
1
8n
1) +
2
20

---

Visual Generation Without Guidance
where choose 0.4.
Stable Diffusion 1.5. We apply GFT to fine-tune Stable Diffusion 1.5 (SD1.5) (Rombach al., 2022) for 70,000 gradient updates with batch size of 256 and constant learning rate of 1c 5 with 1,000 warmup steps. We disable conditioning dropout as we find it improves CLIP score. For comparison, we also fine-tune SD1.5 using guidance distillation with scale range from 1 to 14, while keeping other hyperparameters aligned with GFT.
For evaluation, following GigaGAN (Kang et al., 2023) and DMD (Yin et al., 2024), we generate images using 30K prompts from the COC02014 (Lin et al., 2014) validation set, downsample them to 256x256, and compare with 40,504 real images from the same validation set. We use clean-FID (Parmar al., 2022) to calculate FID and OpenCLIP-G (Ilharco et al., 2021; Cherti al., 2023) to calculate CLIP score (Radford et al., 2021). All results evaluated using 50 steps DPM-solver++ (Lu et al., 2022) with EMA models. The EMA decay rate is set 0.9999.
E. Prompts for Figure 14
We use the following prompts for Figure 14.
A vintage camera in park, autumn leaves scattered around it.
Pristine snow globe showing winter village scene, sitting on frost-covered pine windowsill at dawn.
Vibrant yellow rain boots standing by a cottage door, fresh raindrops dripping from blooming hydrangeas.
Rain-soaked Parisian streets at twilight.
21


Reviews:
[{"review_id": "70A9JoGirF", "reviewer": "Reviewer_5kGx", "summary": "In this work the authors have proposed a technical to improve the standard classifier-free guidance. The key idea is to directly optimize the target guided noise regressor (as described in Eq. 6) after converting Eq. 4 into another form amenable for this purpose. The standard CFG demands running the diffusion model twice, with or without prompt separately. The proposed GFT method incapsulates all compute into a single model, thus saving much computations.\n\n## update after rebuttal\n\nMy previous concerns have been mostly addressed in a convincing way. Thus I will raise my score to be weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer 5kGx (Part 1/2)\nWe thank the reviewer for the very detailed comments! We summarize concerns into four categories:\n## Computational/Memory efficiency\n**Q1: The reformulation results in two different models active and up to optimization during training, ... Are they sharing the same parameters as in CFG? If not, that makes the advantages of GFT less obvious.**\n\n**A1:**\n**They do share parameters just like CFG**, Throughout all our experiments, there is ALWAYS only one model kept in memory. This makes GFT extremely memory-efficient.\n\nAs noted in Figure 3, GFT has the **same** GPU memory usage as CFG training. This distinguishes GFT from all previous distillation approaches, which all require more GPU memory than CFG.\n\nLlamaGen exp: 8H100, bz 256, FSDP\n| Model | Memory Per Card | Ratio |\n|:-:|:-:|:-:|\n| CFG | 59.3G | 1.0000|\n| GFT | 59.4G | 1.0016|\n\n**Q2: Did not specify the basis for comparing the training time in a convincing way. ... superficially claimed \"less than 5\\% pretraining computation\" and \"2x faster in sampling\" .... should be more rigorous.**\n\n**A2:**\nWe thank the reviewer for the suggestion. We use VAR and DiT to show how \"less than 5\\% pretraining computation\" is calculated in detail.\n\n(8*H100 GPUs)\n\n|Model|Size|Batch Size|Acc. step|Pretrain Epoch|Time/Epoch (CFG)|Total (CFG)|Finetune Epoch|Time/Epoch (GFT)|Total (GFT)|Ratio GFT/CFG|\n|--|-|-|-|-|-|-|-|--|--|--|\n|VAR-d30|2.0B|768|12|350|2.07h|724.5h|15|2.37h|35.55h|4.90\\%|\n|DiT-XL|675M|256|1|1400|0.29h|406h|28|0.33h|9.24h|2.27\\%|\n\n\"2x faster in sampling\": our method simply halves the model inference, ==> allows doubling the batch size while keeping the same sampling time.\n\n## Method motivation and Evaluation\n\n**Q3: The reported experimental results are not consistent..... Figure 7 : GFT and baselines perform similarly..... But Tables 2/3 report a clear, large margin between GFT and CFG.**\n\n**A3:**\nWe feel there is clearly some misunderstanding here. **Figure 7 and Tables 2/3 show consistent results, they just focus on different evaluation metrics.** Take DiT-XL for instance.\n\nCFG:\n\n| Guidance $s$ | FID | IS | Guidance-free? |\n|:----:|:---:|:----:|:---:|\n| 1.0 | **9.34** |117.1 | **Yes** (Table 2) |\n|1.35 | 2.22 | 230.8 | No |\n|1.4 | **2.11** | 245.7| **No** |\n|1.45 | 2.14 |258.6| No |\n|1.5 | 2.14 | 271.2 | No |\n\nGFT:\n\n| Beta $\\beta$| FID | IS | Guidance-free? |\n|:--:|:-:|:--:|:--:|\n|1.0|6.77|152.8 | Yes |\n|1/1.35|2.29| 203.5 | Yes |\n|1/1.4|2.07| 229.7 | Yes |\n|1/1.45|1.99| 240.0 | Yes |\n| 1/1.5| **1.99** |249.6 |**Yes**|\n\nTwo ways to interpret this data:\n\n1. If restricted to **guidance-free** sampling,  ==> can only use $s=1$ for CFG.   ==>  GFT's FID  1.99 **significantly outperforms** CFG FID 9.34. ====> **Table 2/3**.\n\n2. If only focus on the FID-IS trade-off.  ===> we can tune CFG $s$.  ==>  GFT (FID 1.99) slightly outperforms CFG (FID 2.11) , similar FID-IS trade-off.  ====> **Figure 7**\n\n**Q4: Figure 2 is not very informative since most related works can demonstrate such evolution under a stronger level of guidance.**\n\n**A4:** Following **A3**, Figure 2 does NOT mean to prove GFT \"outperforms\" existing methods like CFG. Instead, it demonstrates GFT can achieve previous work performance **without** guided sampling. Thus, the reviewer feeling \"GFT is similar to related works with a strong level of guidance\" is exactly what we want to see.\n\nWe thank the reviewer's question and have updated Figure 2's caption to avoid possible misleading [1].\n\n[1] https://anonymous.4open.science/r/Additional-Results-4CDD/updated.pdf\n\n**Q5: The claim is somewhat counting intuition. Eq. 6 is essentially the same as Eq. 4... However, the authors claim GFT can bring notable performance gain as shown in Tables 2/3.**\n\n**A5:**\nWe respectfully disagree with the reviewer. We hope **A3** and **A4** address the reviewer's concern. **\n\n1.  Theoretically, GFT and CFG are equivalent.  (Eq. 4 -> Eq. 6)\n2. **guidance-free** performance: GFT is significantly better. (Table 2/3)\n3. FID-IS trade-off: GFT and CFG perform similarly. GFT is more efficient in sampling. (Figure 7)\n\n**Q6: In comparing GFT and other baselines, it is critical to ensure they are on the same level of guidance strength. However, this is missing ...**\n\n**A6:**\nWe respectfully disagree. For GFT and all baselines, we report their **best** performance by tuning their respective guidance $s$ or temperature $\\beta$. This ensures fairness.\n\nIf have to align guidance strength, we can easily select a hyperparameter suitable for GFT but not optimal for CFG. For example, for DiT-XL, GFT achieves optimal FID 1.99 with $\\beta = 1/1.5 = 0.667$. Under the same level of guidance, CFG FID is **2.14** at $s=1.5$. However, we choose to compare with the CFG optimal FID **2.11** at $s=1.4$.\n***\n# Reminder for Part (2/2)\nDue to very detailed questions posted, we borrowed some space from **Reviewer 2PrL** for **Part (2/2)** response to answer **Q7-Q11**.  Thank you for understanding!", "claims_and_evidence": "The major claims include the efficacy and effectiveness of the proposed GFT method, compared with classifier-free guidance and two existing relevant baselines (guidance distillation and condition contrastive alignment). The authors provide empirical evaluations on several widely-used benchmarks. However, the reported experimental results are not consistent in all tables or figures. For example, from Figure 7, it seems by properly tuning some hyper-parameters (in particular, the diffusion temperature related s or beta) the proposed method and baselines perform similarly. But Tables 2 and 3 report a clear, large margin between GFT and others.", "methods_and_evaluation_criteria": "The evaluations are conducted following previous practice (class-conditional image generation on ImageNet and text-to-image generation on LAION). All dataset and metrics are reasonable.", "theoretical_claims": "The authors provided proof for Theorem 1 in appendix B. I checked the proof yet was not fully convinced. According to my understanding, beta is part of the target model and can be adjusted for different strength of guidance. Why is beta set to be 1 in the proof?", "experimental_designs_or_analyses": "1. Some reported results are superficially treated. For example, the authors did not specify the basis for comparing the training time and GPU memory usage in a convincing way. In Section 5.2, it was superficially claimed that \"less than 5% pretraining computation\" and \"being 2x faster in sampling\". The comparison should be in more rigorous setting.\n\n2. The claim is somewhat counting intuition. Eq. 6 is essentially the same to Eq. 4. The major difference is to set epsilon^s directly to be optimized such that much computation can be saved. The authors claim that such a reformulation can bring notable performance gain as shows in Tables 2 and 3, which needs further clarification.", "relation_to_broader_scientific_literature": "Essentially the proposed model is an incremental improvement to the standard classifier-free guidance. It saves much compute in the inference time (since only one model will be executed, rather than two as in CFG), however it comes at a cost. The guidance hyper-parameter becomes part of the model, which complicates the training of the model. \n\nI regard the work to be interesting to a broad spectrum of readers, since CFG is crucial for diffusion based generation. However the proposed method needs further justification to be more convincing.", "essential_references_not_discussed": "The references are sufficient.", "other_strengths_and_weaknesses": "Improving classifier guidance is an important research topic in AIGC. I am surprised that the simple change in the proposed method can bring much improvement in the experiments. However, the experiments in their current form are not fully convincing to me, which makes me hesitate to recommend acceptance. \n\nIn other sections of the reviews, I have discussed about several potential weaknesses or problems in the work, including the proof of theorem 1, the claimed superiority in terms of FID. There are some other issues that I feel to be critical for the final evaluation of this work. First, the hyper-parameter beta now becomes part of the model, such as one can directly adjust beta for obtaining different levels of guidance. However, this makes the optimization hard as one needs to sample different values of beta (if I understand well). Also very importantly, in comparing the proposed method and other baselines, it is critical to ensure that they are on the same level of guidance strength (i.e., with proper s and beta), such that a fair comparison can be guaranteed. However, this is missing in many experiments, such as the one in Figure 5.\n\nThe reformulation makes there are two different models active and up to optimization during training, namely the s-type and u-type models. Are they sharing same parameters as in standard classifier-free guidance? If not, that makes the advantages of GFT less obvious.\n\nFigure 2 is not very informative since most related works can demonstrate such evolution under stronger level of guidance.", "other_comments_or_suggestions": "There are some ad hoc parameters in the proposed model. For example, in Algorithm 1, it was shown that c will be masked with a 10% change. How was this 10% chosen? Any specific reason for such a parameter?\n\nSome notations are used without any proper definitions, such as p^s. \n\nThe loss forms in Table 1 are not reasonable, in particular the loss for guidance-free training. The authors are suggested to revise them and provide more clarification.", "questions_for_authors": "In the training stage, are there two models (one with beta, and the other the unconditional version) optimized in the proposed GFT, or just one? If the former case were true, does it require much more memory space in comparison with CFG, where only one model is kept, with the prompt turned on or off?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "Au7g4eD4WH", "reviewer": "Reviewer_oBo4", "summary": "* “Visual Generation Without Guidance” presents Guidance-Free Training (GFT), a novel approach for visual generative models that aims to eliminate the need for guided sampling and reduce computational costs.\n\n* The core of GFT design is to transform the target sampling model into an easily learnable form. Instead of explicitly learning a conditional network as in CFG, GFT defines the conditional model implicitly as a linear interpolation of a sampling network and an unconditional network. During training, GFT optimizes the same conditional objective as CFG.\n\n* GFT is an alternative to guided sampling in visual generative models. It achieves comparable performance to CFG while reducing sampling computational costs by 50%. The method is simple to implement, requiring minimal modifications to existing codebases, and can be trained directly from scratch. It represents an advancement in making high - quality visual generation more efficient and accessible.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer oBo4 (Part 1/1)\nWe thank the reviewer for the insightful review. We are really glad to see the reviewer shares the same belief as us that Guided sampling should eventually be removed from visual modeling. We are also greatly motivated by the high praise given to our work.\n\n**Q1: Algorithm 1, line 8: We already have the pseudo-temperature $\\beta \\in [0,1]$, why do we still need to perform dropout on the condition?**\n\n**A1:** The most direct reason for dropping out condition is that we have detached the gradient from unconditional model ($\\mathrm{\\mathbf{sg}}(\\cdot)$) during training (Algorithm 1 line 12). If we do not randomly dropout the condition, the unconditional model would not be trained at all.\n\n$L_\\theta\n       =\\|\\beta  \\epsilon\\_\\theta^s(x\\_t|c\\_\\emptyset,\\beta ) + (1-\\beta) \\mathrm{\\mathbf{sg}}\\[ \\epsilon\\_\\theta^u (x\\_t| c=\\emptyset , \\beta=1)\\] - \\epsilon\\|^2.\n$\n\nThe question now becomes why we have to detach the unconditional gradient. \n\nIn short, without detaching the unconditional gradient, loss becomes\n\n$L_\\theta\n       =\\|\\beta  \\epsilon\\_\\theta^s(x\\_t|c,\\beta ) + (1-\\beta) \\[ \\epsilon\\_\\theta^u (x\\_t| c=\\emptyset , \\beta=1)\\] - \\epsilon\\|^2.\n$\n\nSince $\\beta \\sim \\text{U}[0,1]$, the unconditional model can still be trained. However, the loss function now spends 50 \\% of its energy to optimize the unconditional model, which is way too much because the unconditional part is not what we finally want. **This hurts performance in practice.** Also, this causes misalignment with CFG training pipeline. **We believe GFT should not only ensure soundness but also offer seamless integration and extreme compatibility.**\n\nWe refer the reviewer to Section 3.2 (lines 194 to 210) in our paper for detailed response. \n\n\n**Q2: When dropping out the condition, wouldn't the two branches $\\beta$ and$ (1-\\beta)$ be redundant?**\n\n**A2:**\nYes, they will be redundant. However, when dropping out condition, we are training the unconditional model, the loss is \n\n$L_\\theta\n       =\\|\\beta  \\epsilon\\_\\theta^u(x\\_t|\\beta ) + (1-\\beta)  \\mathrm{\\mathbf{sg}} \\[ \\epsilon\\_\\theta^u (x\\_t| \\beta)\\] - \\epsilon\\|^2.\n$\n\nWe have proved in Appendix B (line 737-748) that this objective has exactly the same solution as classic unconditional loss:\n$L_\\theta\n       =\\| \\epsilon\\_\\theta^u(x\\_t|\\beta )  - \\epsilon\\|^2.\n$\n\n\nTherefore, $\\mathbf{sg}$ does not affect model convergence.\n\nWe could have removed the $1-\\beta$ branch when dropping out condition, but again, this causes the unconditional model part to be trained too much and also breaks compatibility with CFG design. We considered and tried many versions of this loss equation, but decided the first loss form is the most **elegant** one.\n\n\n**Q3: I am very curious about what impact removing the stop gradient would have on performance.**\n\n**A3:**\nWe re-ran the DiT-XL/2 experiments following the settings in the paper, with and without stop-gradient. The results show that using stop-gradient slightly improves the performance.\n\n| stop gradient? | FID |\n|:-------:|:-----------------:|\n| Yes | 1.93 |\n| No | 2.04 |\n\n**Q4: I am also curious about the effect that the size of the MLP model following the $\\beta$ parameter has on the results.**\n\n**A4:** We ablated how the $\\beta$ MLP encoder size affects training performance on DiT-XL/2. Overall, we find GFT to be insensitive to the MLP encoder size:\n\n| MLP layers | FID |\n|:-------:|:-----------------:|\n| 1 | 1.93 |\n| 2 | 1.92 |\n| 3 | 1.92 |\n\n**Q5: If it's a T2I task, how should the negative prompt and β be implemented?**\n\n**A5:**\n\n1. Replace the unconditional mask with a negative prompt $c\\_n$ randomly sampled from a pool of negative candidates.\n2. We assume randomly masking out \"condition (prompt)\" might not be necessary anymore. Because these negative prompts should already appear in the dataset several times. We are not sure, this one can be tested out.\n\n$L_\\theta\n       =\\|\\beta  \\epsilon\\_\\theta^s(x\\_t|c,\\beta ) + (1-\\beta) \\mathrm{\\mathbf{sg}}\\[ \\epsilon\\_\\theta^s (x\\_t| c\\_n , \\beta=1)\\] - \\epsilon\\|^2.\n$", "claims_and_evidence": "The claims made in the submission are clear", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense for the problem.", "theoretical_claims": "Yes, I  check the correctness of most proofs for theoretical claims.\n\nIn line 8 of Algorithm 1 in the paper, (c_{\\varnothing}= c) masked by (\\varnothing) with a 10% probability, I am a bit confused. Given that we already have the pseudo-temperature (\\beta) ~ ( (0, 1) ), why do we still need to perform dropout on the condition? When dropping out the condition, wouldn't the two branches ((\\beta) and (1-\\beta)) be redundant?", "experimental_designs_or_analyses": "Yes, I checked the soundness and validity of the experimental designs and analyses. GFT conducted experiments on the baselines of many mainstream methods(DIT, VAR, MAR, LlamaGen) and validated the results across multiple benchmarks. The experimental design and analysis are reasonable.", "relation_to_broader_scientific_literature": "I believe this paper significantly contributes to the acceleration and improvement of generative models. Additionally, I think that the classifier-free guidance (CFG) technique in visual generation will inevitably be replaced in the future.", "essential_references_not_discussed": "Essentially, all related works have been properly cited and discussed in the paper.", "other_strengths_and_weaknesses": "After thoroughly reading the paper and following the derivations of the formulas, I find this paper significantly meaningful and quite enjoyable. I believe that the classifier-free guidance (CFG) technique in visual generation will inevitably be replaced in the future, and this paper theoretically demonstrates the feasibility of this transition.\n\nThe paper is well-written, the experiments are comprehensive(especially the experiment for FID-IS trade off experiments), and the results are demonstrated across a substantial number of related works. However, I have a few minor questions:\n\n1. The paper mentions the stop gradient part, and I am very curious about what impact removing the stop gradient would have on performance.\n2. I am also curious about the effect that the size of the MLP model following the β parameter has on the results.\n3. If it's a T2I task, how should the negative prompt and  β be implemented?", "other_comments_or_suggestions": "refer to \"Other Strengths And Weaknesses\"", "questions_for_authors": "refer to \"Other Strengths And Weaknesses\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mDpCyFLDy4", "reviewer": "Reviewer_DEzb", "summary": "This paper introduces Guidance-Free Training (GFT), a novel method for training visual generative models that eliminates the need for Classifier-Free Guidance (CFG) during inference while maintaining comparable generation quality. The key insight is to do direct optimization of the desired sampling distribution during training by replacing the prediction with the linear interpolation with actual CFG target. GFT achieves comparable FID scores to CFG across multiple visual models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer DEzb\n\n**Q1: GFT introduces an additional hyperparameter $\\beta$ that requires tuning in training and still in inference.**\n\n**A1:**\nWe believe there is some misunderstanding over how we tune and inject $\\beta$. \n\n1. $\\beta$ is not a training hyperparameter and **it does not require tuning during training**. \n\n(Algorithm 1): GFT is essentially learning various sampling models under different $\\beta$ **at the same time**. \n\n$\\beta$ is always randomly sampled from the uniform distribution [0,1] for each data point.\n\n**$\\beta$ is very like the diffusion time-step $t$**, it is only a scalar model input instead of a hyperparameter to be tuned.\n\n2. $\\beta$ indeed needs to be adjusted in **sampling**. However, guidance $s$ also needs to be tuned for CFG. So,  **GFT requires no more tuning than CFG**.\n\n**Q2: Providing more intuition for how to select values for both training and inference.**\n\n**A2:** Following **A1**, we do not \"select\" $\\beta$ in inference. \n\nIn sampling, according to Theorem 1 and Line 192 in paper,  there exists a one-to-one correspondence between CFG $s$ and GFT $\\beta$.\n\n$\\beta = \\frac{1}{1+s}$.\n\nSuppose we already know optimal CFG $s$ is 0.4, then the optimal GFT $\\beta$ should be $\\frac{1}{1.4}$. In practice, this value is usually not accurate due to training bias, but it provides a good starting point.\n\nCFG:\n\n| Guidance Scale $s$ |FID|IS|Guidance-free?|\n|:-:|:-:|:-:|:-:|\n| 0.0|9.34|117.1|Yes|\n|0.35 |2.22 | 230.8| No|\n|**0.4**| **2.11**| 245.7|No|\n|0.45|2.14|258.6| No|\n|0.5|2.14| 271.2 | No|\n\nGFT:\n\n| Beta $\\beta$| FID| IS | Guidance-free? |\n|:-:|:--:|:--:|:--:|\n|1.0|6.77|152.8|Yes|\n|1/1.35 | 2.29 | 203.5 | Yes |\n|**1/1.4** | **2.07** | 229.7 | Yes |\n|1/1.45|1.99|240.0| Yes |\n|1/1.5| 1.99|249.6| Yes |\n\n\n**Q3: $\\beta$ may introduce various issues in how to inject this condition. However there lacks ablation study on this.**\n\n**A3:**\nWe are happy the reviewer is interested in this. \n\nActually, how to inject a scalar condition into a diffusion model is well-explored, because diffusion-time $t$ is similar stuff. We simply borrowed their design and found it works well enough. \n\nWe indeed tried several ablation choices initially. \n\n1. Instead of posing $\\beta$ as model input, we leverage $s = \\frac{1}{\\beta} -1$ in Theorem 1, and model \n\n$\n\\epsilon\\_\\theta(x\\_t|c, \\beta) := \\epsilon\\_\\theta^1(x\\_t|c) + (\\frac{1}{\\beta} - 1) \\epsilon\\_\\theta^2(x\\_t|c),\n$\n\nwhere $\\epsilon\\_\\theta^1$ is the pretrained network, and $\\epsilon\\_\\theta^2$ the pretrained network with a new MLP head. Our hope is to avoid making $\\beta$ as network input. However, this design performs poorly:\n\n| Model | CFG FID| $\\beta$ as input FID|$\\beta$ **as linear coefficient FID** |\n|:--:|:--:|:--:|:--:|\n| LlamaGen-3B | 2.22|**2.21** | 2.44 |\n| VAR-d30 | 1.92|**1.91** | 2.30 |\n\nWe suspect the main reason is that '$\\beta$ as model input' allows better leveraging the full potential of pretrained model parameters.\n\n2. We ablated how the $\\beta$ input MLP encoder size on DiT-XL. (Turns out to be insensitive)\n\n| MLP encoder layers | FID |\n|:-:|:--:|\n| 1 | 1.93 |\n| 2 | 1.92 |\n| 3 | 1.92 |\n\n**Q4: Limited exploration of how the approach extends to text-to-image generative models.**\n\n**A4:** We feel the reviewer might miss Table 3, Figure 4, Figure 8, and Figure 12 in our paper, where we have **already conducted experiments on T2I generative models** using Stable Diffusion as base model, and LAION 5+ as the dataset. \n\nIn short, GFT significantly increases guidance-free FID from 22.55 to 8.10, CLIP score from 0.252 to 0.313, achieving same level of CFG performance.\n\n**Q5: When GFT might not be advantageous compared to CFG?**\n\n**A5:**\n\n1. If training from scratch, GFT still requires  20\\% more computation than CFG, but is 2x more efficient in inference.\n\n2. When it comes to advanced guidance methods, such as dynamically adjusting the guidance scale during decoding [1]. CFG only requires modifying the sampling code. However, GFT requires redesigning the training code (though only 1-3 lines).\n\n[1] Applying guidance in a limited interval improves sample quality in diffusion models. \n\n**Q6: How does CFG/GFT training computation trade-off change with model scale?**\n\n**A6:** **In short, the influence of model size is almost negligible.**  \n\nVAR as an example:\n\n|Model|Size|Batch Size|Acc. step|Time/Epoch (CFG)|Time/Epoch (GFT)| Time (GFT/CFG)|\n|--|--|--|--|---|---|--|\n|VAR-d16|300M|768|1|0.82h|0.93h|1.134|\n|VAR-d20|600M|768|4|1.11h|1.26h|1.135|\n|VAR-d24|1B|768|4|1.33h|1.52h|1.142|\n|VAR-d30|2B|768|12|2.07h|2.37h|1.145|\n\n**Q7: CFG doesn't work well with semantic latent space. Does GFT present any advantage over CFG with such semantic tokenizers?**\n\n**A7:**\nUnfortunately, due to the similar theoretical property between GFT and CFG  as in Theorem 1, we find it difficult to expect GFT to solve some potential issues in which CFG has failed. Likewise, if GFT does not work well on some domains, we believe CFG may also struggle.", "claims_and_evidence": "The paper's claims about computational efficiency and model performance are well-supported by extensive experiments across five different model architectures.", "methods_and_evaluation_criteria": "The evaluation is appropriate, using standard metrics (FID, IS) and datasets (ImageNet, COCO) that are widely accepted in the field of visual generation. The comparison with state-of-the-art techniques like CFG, guidance distillation, and contrastive alignment provides a comprehensive assessment.", "theoretical_claims": "I verified the correctness of Theorem 1, which provides the optimal solution for GFT. The proof in Appendix B logically demonstrates that stopping the unconditional gradient does not change the convergence point of the objective function.", "experimental_designs_or_analyses": "The experimental design is sound, covering both fine-tuning and from-scratch training scenarios across multiple architectures.\nThe analysis of training dynamics (Fig. 6) and temperature control (Fig. 2) effectively supports the method's stability and flexibility.", "relation_to_broader_scientific_literature": "This paper addresses a inefficiency issue in current visual generation pipelines by eliminating CFG linear inference of both conditional models and unconditional models.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths** \n* A method application to different visual generation models.\n* Practical value in reducing inference computation by 50% eliminating the need of a unconditional inference in CFG.\n* Simple implementation requiring minimal code changes to existing models\n\n\n** Weaknesses** \n* Introduces an additional hyperparameter $\\beta$ that requires tuning in training and still in inference.\n* $\\beta$ is an input condition to the model, this may introduce varies issues of on how to inject this condition into the model. However there lacks ablation study on this.\n* Limited exploration of how the approach extends to text-to-image generative models.", "other_comments_or_suggestions": "* Consider providing more intuition for how to select $\\beta$ values for both training and inference.\n* The paper could benefit from a more detailed analysis of when GFT might not be advantageous compared to CFG.", "questions_for_authors": "1. How would GFT perform on more complex generation tasks such as text to image generation?\n2. The paper mentions a 10-20% increase in training computation. How does this trade-off change with model scale? \n3. Recently it is found that CFG doesn't work well with semantic latent space [1], does CFT present any advantage over CFG with such semantic tokenizers?\n\n[1] Masked Autoencoders Are Effective Tokenizers for Diffusion Models.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xeLvZ8b0oL", "reviewer": "Reviewer_2PrL", "summary": "1. They proposed guidance-free training, which reparameterizes the conditional model as a combination of trainable sampling model and frozen unconditional model.\n\n2. They introduced pseudo-temperature input (β) to control the fidelity-diversity trade-off.\n\n3. They reached similar performance compared to CFG across several tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Official Response to Reviewer 2PrL  (Part 1/1)\nWe are glad the reviewer finds our method to be versatile and generally applicable. This is exactly what we are trying to convey in this paper.  \n**Q1: More comprehensive evaluation metrics like sFID, precision, Recall**\n\n**A1:**\nWe have conducted additional evaluation, and now report their full evaluation metrics. These results are consistent with our main findings and further support the effectiveness of GFT.\n\n**For fine-tuning experiments**, GFT consistently achieves better or comparable performance across all metrics, as shown below:\n\nDiT-XL/2\n\n| Method | FID | sFID | IS | Presicion | Recall |\n|:----:|:---:|:--:|:--:|:---:|:--:|\n| CFG | 2.11  | 4.81 | 245.7 | **0.81** | 0.59 |\n| GFT | **1.99** | **4.67** | **266.6** | **0.81** | **0.60** | \n\nLlamaGen-3B\n\n| Method | FID | sFID | IS | Presicion | Recall |\n|:----:|:---:|:---:|:---:|:--:|:---:|\n| CFG | 2.22  | 6.01 | 264.1 | 0.82 | 0.58 |\n| GFT | **2.21** | **5.85** | **279.0** | **0.83** | **0.59** | \n\n**For pretraining experiments**, GFT also shows consistent improvement, especially in FID, sFID, and IS:\n\nDiT-B/2\n\n| Method | FID | sFID | IS | Presicion | Recall |\n|:----:|:---:|:----:|:----:|:----:|:---:|\n| CFG | 9.72  | 8.75 | 161.5 | 0.84 | **0.34** |\n| GFT | **9.04** | **8.29** | **166.6** | **0.86** | **0.34** |\n\nLlamaGen-L\n\n| Method | FID | sFID | IS | Presicion | Recall |\n|:--:|:-:|:-:|:-:|:-:|:-:|\n| CFG | 3.06  | 6.15 | 257.1 | **0.83** | 0.52 |\n| GFT | **2.52** | **5.98** | **269.5** | 0.82 | **0.57** | \n\nMore detailed numbers & visualizations at:\nhttps://anonymous.4open.science/r/Additional-Results-4CDD/README.md\n***\nBelow, we borrow some space to further respond to **Reviewer 5kGx** due to space limit.  Thank you for understanding!\n***\n# Official Response to Reviewer 5kGx  (Part 2/2)\n\n## Additional Input of beta\n\n**Q7: GFT saves much compute ..... however it comes at a cost. The guidance hyper-parameter becomes part of the model, which complicates the training of the model.**\n\n**A7:**\nWe want to offer the reviewer a new perspective for incorporating $\\beta$ as the model's input.\n\n1. $\\beta$ does not complicate **training**, because it is merely a **sampling** hyperparameter. In training, $\\beta$ is always sampled from the uniform distribution [0,1] and does not need to be \"tuned\" (Algorithm 1 in paper). Thus **$\\beta$ is very like diffusion time-step $t$**, which also serves as a scalar input of the model, but no one should think input $t$ is a \"cost\" or \"training complication\".\n\n2. $\\beta$ needs to be adjusted for GFT during **sampling**. However, guidance scale $s$ also needs to be tuned for CFG. So, we can conclude **GFT is no more complicated than CFG** during inference. Plus, GFT is 2x more efficient.\n\n**Q8: The hyper-parameter beta now becomes part of the model, ..., However, this makes the optimization hard as one needs to sample different values of beta.**\n\n**A8:** We are glad the reviewer mentions this concern, because **our experiments exactly prove \"sampling different values of beta\" does NOT complicate optimization**.\n\nThe most clear evidence is Figure 6 (convergence plot) in the paper. In large-scale from-scratch training, the losses for GFT converge as fast as the classic CFG training w/o a $\\beta$ input. \n\nAlso in Table 4, performance numbers across 3 distinctive models clearly show that GFT w/ $\\beta$ as model input slightly outperforms CFG baselines.\n\n**Q9: I checked the proof yet was not fully convinced. ... beta is part of the target model and can be adjusted .... Why is beta set to be 1 in the proof?**\n\n**A9:**\nThe reviewer is referring to the proof for the unconditional model convergence point in Line 730-736 in Appendix B, where we select $\\beta = 1$ to derive the conclusion.\n\nAs a matter of fact, for other $\\beta < 1$, the convergence point is the same. **We have already proved this point** in the following lines 737-748. At line 749, we summarized: \"... this does not change the convergence point of loss. The optimal unconditional solution remains the same\".\n\nWe thank the reviewer for the comment and have updated our proof for better clarity [1]. \n\n## Formatting and clarity\n\n**Q10: Some notations are used without any proper definitions, such as $p^s$.**\n\n**A10:**\n\nWe had explicitly defined $p^s$ in the background section $2.2$, Eq. 5, and referred to this definition in the Method section $3.1$, Line 141.\n\nIn short, $p^s$ corresponds to our target sampling distribution. $p^s_\\theta$ is our sampling model.\n\n$\n    p^\\text{s}(x|c) \\propto p(x|c) \\left[\\frac{p(x|c)}{p(x)}\\right]^s.\n$\n\n**Q11: The loss forms in Table 1 are not reasonable....  Suggest revising them and providing more clarification.**\n\n**A11:**\nWe thank the reviewer for the suggestion. Previously, we abbreviated some terms in Table 1 due to space constraints. We have now updated Table 1 to focus more on equation clarity and linked it to the formal equation in the paper [1].", "claims_and_evidence": "1. The quantitative results demonstrate that their method achieves comparable FID or even better FID than CFG.\n\n2. The qualitative results show similar synthetic images compared with CFG.\n\n3. Figure 3 demonstrates their computational efficiency.", "methods_and_evaluation_criteria": "1. The evaluation metrics are insufficient, as the paper does not report sFID, precision, or recall. Those metrics were used in the original DiT experiments to assess fidelity and diversity.\n\n2. The method is evaluated across various models, demonstrating its versatility and general applicability.", "theoretical_claims": "Theorem 1 is mathematically proven in Appendix B, and shows GFT models the same sampling distribution as CFG.", "experimental_designs_or_analyses": "1. Fine-tuning efficiency experiments shown in Figure 5 track FID over epochs, demonstrating rapid convergence with minimal training.\n\n2. The diversity-fidelity tradeoff experiments shown in Figures 7 and 8 vary the temperature parameter, providing a fair comparison with guided approaches.", "relation_to_broader_scientific_literature": "GFT matches the performance of CFG while using half-time cost during inference.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength: \n1. Compared to distillation methods, they can train from scratch.\n\n2. The performance superior is convincible across many of different tasks.", "other_comments_or_suggestions": "No", "questions_for_authors": "Can you report more comprehensive evaluation metrics to better support your claims?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes Guidance-Free Training (GFT), a novel method that optimizes classifier-free guidance (CFG) more efficiently by reformulating the guidance mechanism to eliminate the need for guided sampling at inference. Reviewers appreciate the theoretical clarity, practical effectiveness, and the computational efficiency improvements (approximately 2x faster inference) demonstrated on standard benchmarks. The authors also effectively addressed the concerns raised by the reviewers during rebuttal. Given the positive consensus among reviewers, I recommend acceptance for this paper.
Decision: Accept (poster)