Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator
Authors: Kaiwen Zheng
Yongxin Chen
Huayu Chen
Guande He
Ming-Yu Liu
Jun Zhu
Qinsheng Zhang
Venue: ICML 2025 spotlightposter (2025)
Keywords: Diffusion Models
Visual Autoregressive Models
GAN
Generation Quality
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: an efficient and effective finetuning method for enhancing diffusion models and visual autoregressive models

Abstract: While likelihood-based generative models, particularly diffusion and autoregressive models, have achieved remarkable fidelity in visual generation, the maximum likelihood estimation (MLE) objective, which minimizes the forward KL divergence, inherently suffers from a mode-covering tendency that limits the generation quality under limited model capacity. In this work, we propose Direct Discriminative Optimization (DDO) as a unified framework that integrates likelihood-based generative training and GAN-type discrimination to bypass this fundamental constraint by exploiting reverse KL and self-generated negative signals. Our key insight is to parameterize a discriminator implicitly using the likelihood ratio between a learnable target model and a fixed reference model, drawing parallels with the philosophy of Direct Preference Optimization (DPO). Unlike GANs, this parameterization eliminates the need for joint training of generator and discriminator networks, allowing for direct, efficient, and effective finetuning of a well-trained model to its full potential beyond the limits of MLE. DDO can be performed iteratively in a self-play manner for progressive model refinement, with each round requiring less than 1\% of pretraining epochs. Our experiments demonstrate the effectiveness of DDO by significantly advancing the previous SOTA diffusion model EDM, reducing FID scores from 1.79/1.58/1.96 to new records of 1.30/0.97/1.26 on CIFAR-10/ImageNet-64/ImageNet 512$\times$512 datasets without any guidance mechanisms, and by consistently improving both guidance-free and CFG-enhanced FIDs of visual autoregressive models on ImageNet 256$\times$256.

Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator
Kaiwen Zheng" 12
Yongxin Chen
Huayu Chen
Guande He
Ming-Yu Yu Liu' /aaab///dr//d/////
Jun Zhu'
Qinsheng Zhang
Abstract
While likelihood-basse generative models, partic- ularly diffusion and autoregressive models, have achieved remarkable fidelity in visual generation. the maximum likelihood estimation (MLE) objec- tive, which minimizes the forward KL divergence, inherently suffers from mode-covering tendency that limits the generation quality under limited model capacity. In this work, we propose Direct Discriminative Optimization (DDO) as a unified framework that integrates likelihood-baad gener- ative training and GAN-type discrimination to by- pass this fundamental constraint by exploiting re- verse KI. and self-generated negative signals. Our key insight is to parameterize discriminator im- plicitly using the likelihood ratio between a learn- able target model and a fixed reference model, drawing parallels with the philosophy of Direct Preference Optimization (DPO). Unlike GANs, this parameterization eliminates the need for joint training of generator and discriminator networks, allowing for direct, efficient, and effective fine- tuning of a well-trained model to its full potential beyond the limits of MLE. DDO can be performed iteratively in a self-play manner for progressive model refinement, with each round requiring less than 1% of pretraining epochs. Our experi- ments demonstrate the effectiveness of DDO by significantly advancing the previous SOTA dif- fusion model EDM, reducing FID scores from 1.79/1.558 05/11.911.9 1.966 to new records of 1.30/0.97/1..0 .26 512x512
(a) EDM2-L (Karras et al., 2024b) (FID 1.96)
on CIFAR- 1aaa--110/mmageeeee-6////eeeeeee datasets without any guidance mechanisms, and by consistently improving both guidance- free and CFG-enhanced FIDs of visual autoregressive models ImageNet 256.
(b) EDM2-L. DDO (Ours, FID 1.26)
Figure I. Samples ImageNet 512x512, without any guidance.
1. Introduction
Modeling the distribution of high-dimensional data is fundamental challenge in machine learning (Bishop & Nasrabadi, 2006; Goodfellow et al., 2016). Recent years have witnessed the domination of diffusion (Ho et al., 2020; Song et al., 2021b) and autoregressive (Van Den Oord et al., 2016) paradigms in generative modeling of continuous data
'NVIDIA "Tsinghua University 'The University of Texas at Austin. Correspondence to: Jun Zhu <deszj @ tsinghua.edu.cn>>
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Direct Discriminative Optimization
Paza
Prets
To bypass MLE's mode-covering nature, we aim lever- age GAN-type loss to discriminate between the model and data distributions and produce contrastive forces that guide the model. However, typical GANs require parameterizing extra discriminator networks and alternating optimization, creating engineering complications. Applying GAN-type training trivially to diffusion or autoregressive models es- pecially inefficient due to their iterative sampling processes.
Ppstrand
Pireteed
Contrastive Force
(a)
(b)
Figure 2. Toy example illustrating DDO. (a) Models pretrained via maximum likelihood estimation (MLE) exhibit dispersed density, while DDO imposes contrastive forces toward the data distribution. (b) The finetuned model concentrates better on the main mode,
In this work, we introduce Direct Discriminative Optimiza- tion (DDO), a framework that bridges fikeiihood-base gen- erative models with GANs to push their performance beyond the limits of MLE. Our key insight is to implicitly parame- terize the discriminator using the likelihood ratio between learnable target model and a fixed reference model, both initialized from the pretrained model. This parameterization, inspired by Direct Preference Optimization (DPO) (Rafailov et al., 2024), offers theoretical guarantees of optimality, di- vergence bounds, and connections to guidance methods. It also enables direct finetuning of the pretrained model with- out altering the network architecture or inference protocol and supports iterative refinement via multi-round self-play.
and discrete data. They have achieved both theoretical and empirical success in visual tasks including image and video synthesis (Dhariwal & Nichol, 2021; Esser et al., 2021:
Ramesh et al., 2021; Karras et al., 2022; Ho et al., 2022:
Rombach al., 2022; Balaji al., 2022; Gupta et al., 2023;
Esser et al., 2024; Brooks et al., 2024; Bao al., 2024;
Tian et al., 2024), forming the cornerstone of large-scale generation systems in the era of Al-generated content.
Diffusion and autoregressive models are representatives of likelihood- based generative models. Compared to Genera- tive Adversarial Networks (GANs) (Goodfellow et al., 2014) which often face unstable training and mode collapse issues, these models are distinguished by their stability, scalability, and generalizability. Besides, their iterative generation pro- cess imposes fewer constraints on network Lipschitzness, potentially facilitating superior generative capability over GAN's single-step generation. Likelihood based generative models aim to learn the underlying data distribution Paata by maximizing the likelihood of the observed data under a parameterized probabilistic model po, which is equivalent to minimizing the forward Kullback-Leibe (KL) divergence:
DDO achieves significant performance gains for both dif- fusion and autoregressive models sufficiently pretrained on standard image benchmarks. By finetuning state-of-the diffusion models EDM (Karras et al., 2022) and EDM2 (Kar- ras et al., 2024b), we achieve unprecedented guidance- free FID scores of 1.30/0.97/1.26 on CIFAR-10/ImaeNNer 64/1mageNet 512x datasets. Finetuning the visual au- toregressive model VAR (Tian et al., 2024) on ImageNet 256 256 reduces the FID from 1.92 1.73 while removing sampling tricks. Notably, even without classifier-free guid- ance (CFG) (Ho & Salimans, 2021), the finetuned model achieves an FID of 1.79, surpassing the CFG-enhanced pre- trained model while cutting the inference cost by half.
max E (x) [lg (a:) > min (Pama I| po)
8
However, this maximum likelihood estimation (MLE) ob- jective entails inherent limitations. Forward KL is known to prioritize "modc-covering" and imposes extreme penalties the model severely underestimates the likelihood of any training sample (Karras al., 2024a). Under limited model capacity, this property forces the learned density spread out excessively (Figure 2(a)), potentially leading to blurry samples-a phenomenon commonly observed in Variational Autoencoders (VAEs) (Kingma & Welling, 2014) and in likelihood training of diffusion models (Song et al., 2021a;
2. Background
2.1. Rkkelhoodd-Bssed Generative Models
Likelihood-based generative models parameterize prob- ability distribution pa to learn the data distribution Paata enabling explicit likelihood evaluation and density estima- tion. Among them, diffusion and autoregressive models are two prominent types that excel in visual generation.
Autoregressive (AR) models (Van Den Oord et al., 2016; Brown et al., 2020) learn discrete data distributions via the next-token prediction mechanism:
Kingma et al., 2021; Lu et al., 2022a; Zheng et al., 2023b). Consequently, these models often rely heavily on guidance methods (Ho& & Salimans, 2021; Kim et al., 2023a; Karras et al., 2024a) steer the samples away from unlikely low- probability regions and toward the core of the data manifold in order to improve overall generation quality. In contrast, GANs, which are theoretically grounded in (IS) divergence or Wasserstein distance (Arjovsky et al., 2017), tend to produce sharper and more realistic samples.
log (x:) =
Σ log po (2( n=1
'AR can also be adapted to model continuous data (Tschannen et al., 2024; et al., 2024).
2

---

Direct Discriminative Optimization
where d denotes the data dimension (sequence length). It factorizes the joint distribution into product of conditional probabilitiis, allowing exact likelihood computation. Each pe(((z((<)) is parameterized via a Softmax operation over the model's output logits and optimized using cross-entropy loss against the ground-truth token. In visual autoregres- sive modeling, images are first quantized to discrete to- kens within compact latent space using autoencoders (Van Den Oord et al., 2017; Esser et al., 2021).
generator network go R + Rd that maps latent variables
= E Rd p(z) (typically Gaussian noise) into synthetic samples, and a discriminator network do R + [0. 1] that classifies samples as real or fake:
min max (1)) [log dz(2)) + Epo(a) [log(1 - (x)))) #
)-
(3)
Here (z) is the generator distribution, whose exact den- sity is intractable but can be easily sampled from via x = go go(2), = ~ p(z). In the inner loop, the discriminator is optimized using binary cross-entropy loss (also known as noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010)), and its optimal solution can be derived as:
Diffusion models (Ho et al., 2020; Song et al., 2021b) learn continuous data distributions by gradually perturbing clean data To ~ Pdsta with Gaussian noise, which generates a trajectory { and then learning to reverse this process. The forward and backward dynamics can be formulated as either stochastic or ordinary differential equations (SDEs or ODEs) (Song et al., 2021b). The forward process follows closed-form transition kernel lt o o mn)) = ((r r) with predefined noise schedule a,, dt, enabling reparame- terization as 22 = 0; + 0r6, ~ (N, I). The model is typically parameterized as a noise prediction network co t) trained to estimate e via mean squared error (MSE) regression, which forms an evidence (or variational) lower bound (ELBO) on the likelihood (Song et al., 2021a):
Pdata (2)
d' (xx) Pdltta (a:) + (a
(4)
under which the minimax game becomes
mjn 2D Das( (Paam || po) 2 log 2
(5)
where Pas(P || 1) = is the Jensen-Shannon (IS) divergence. This theoretically ensures that the optimal generator distribution matches the data distribution. However, in practice, training instability arises due to gradient vanishing and mode collapse, inspiring variants such as Wasserstein GANs (Arjovsky et al., 2017).
2 (KL( || ( + 2 DKL ||
2
logPo (200) 2 C C.-.. [we()]] () e 2
(2)
GANs can be incorporated to enhance other generative models. For example, Discriminator Guidance (Kim et al., 2023a) utilizes the gradient information from the discrimina- tor corrective term to refine the score function in diffu- sion models (discussed in Section 4.2). Additionally, GANs are commonly employed as an auxiliary loss to improve one-step generation such as in diffusion distillation (Kim et al., 2023b; Yin et al., 2024; Zhou et al., 2024b).
where Fz = a+æ + ore, C is a constant irrelevant to 0, and p(t), w(t) are certain time distribution and weight- ing function. The ELBO provides a reasonable likeli- hood approximation compared to the exact but cumber- some instantaneous change-of-variabe formula in neural ODEs (Chen et al., 2018). Moreover, while the likeli- hood bound is tight only for specific p(t), wo(), alternative choices share the same optimum and can serve as surrogate objectives (Kingma& & Gao, 2024).
3. Direct Discriminative Optimization
From the perspective of score matching (Song et al., 2021b), the optimal noise predictor is linked to the score function s" (mx,() ()= == V n, log ge (ar,) by <=x)))
Motivated by the benefits of adversarial training in enhanc- ing generation quality, we aim to bridge likelihood-based generative models with GANs to derive an alternative train- ing paradigm to MLE. Unlike prior works that incorporate GAN as an auxiliary loss and require additional engineer- ing overhead, our approach seeks to (I) directly optimize likelihood-baase generative models without modifying the network architecture, adding extra discriminators, compli- cating the training procedure or increasing inference costs, and (2) eliminate the need for backpropagation through the sampling process, making it applicable to diffusion and autoregressive models that rely on iterative sampling.
ao,s"(a,, l), where denotes the marginal distribution at time t in the forward process. Due to the properties of MSE, the net- work can be parameterized in alternative yet theoretically equivalent forms, such as velocity predictor (Salimans & Ho, 2022: Zheng et al., 2023b) that estimates the tan- gent of the diffusion trajectory, commonly known as flow matching (Lippman et al., 2022). In our experiments, we adopt the more generalized F-parameterrization introduced in EDM (Karras et al., 2022) (detailed in Appendix C).
2.2. GANs
3.1. Your I. Likelihood-Basee Generative Model is Secretly a Discriminator
GANs (Goodfellow et al., 2014) do not explicitly model the likelihood pa but instead directly optimize the data gen- eration process through adversarial training. Specifically, the optimization involves an adversarial interplay between a
Unlike one-step generators that learn a direct mapping from noise to data, likelihood-bane based generative models are
3

---

Direct Discriminative Optimization
Training Dataset
Ertee
trainable
Po (Target)
Pe pa
log
log
init:
log
Po Po || def dg
Fake/
po
Porer (Pretrained)
log
1 - a
log
po..t
Discriminator
Samples
max Pauztals) log de (1) + po 00 log(1
de (x))
GAN discriminator loss
Figure 3. Illustration of DDO. (1) Models. fier is the (pretrained) referenee model frozen during training. 0 is the learnable model initialized as ler: (2) Data. Samples from Piu are drawn from the training dataset. Samples from po are generated by the reference model, either offline online. (3) Objective. The target model 0 is optimized by applying the GAN discriminator loss with the implicitly parameterized discriminator de to distinguish between real samples from Paua and fake samples from pe..
grounded in the probabilistic definition of the likelihood function pe, which enables both the generation samples ~ Pe and the evaluation of the likelihood po(2), either exactly approximately, while retaining the tractability of backpropagation through the likelihood computation. This inspires us to utilize the likelihood information embedded in the optimal discriminator (Eqn. (4)).
In contrast to previous GAN-based methods that introduce separate discriminator network dæ, our approach implic- itly defines the discriminator through target generative model pe- While it is theoretically feasible initialize 6, 6eee arbitrarily and train from scratch, strong initial condi- tions facilitate optimization (Section 3.2). In practice, we initialize 0, 0et from widely available pretrained models, promoting steady improvement. We refer to this approach as Direct Discriminative Optimization (DDO), drawing par- allels with Direct Preference Optimization (DPO) (Rafailov et al., 2024), which aligns language models with human preferences by expressing the reward model in terms of the likelihood ratio between two policies (discussed in Sec- tion 4.1). The DDO pipeline is illustrated in Figure 3.
Specifically, consider a pretrained model pa.. as a reference to generate fake samples. The optimal discriminator de for
min- (x) [log do (a:)] -Fppe (x) [log(1 do(2)))] (6)
can be rewritten as:
Paata
log Pdata (2x) pe (x)
d' (x: Pdata (x) + Pe (x)
(7)
What does the DDO update do? For a mechanistic un- derstanding of DDO, we can analyze the gradient of the loss function with respect to parameters A:
where (x) T4e is the Sigmoid function. The data distribution Pdata is available from d', Therefore, if we parameterize the discriminator de using a likelihood-baai target generative model po as:
(el(O) [(1 (1- (z) (po Peata (x)) log pe (æ)d
E|0.1]
Pio (aejt <0
(10)
Intuitively, gradient descent increases the model likelihood (a:):) for data points that satisfy po(x) Paata (x)), and
pe (x) Pa (x)
de (x(x)) = log
(8)
decreases it otherwise, pushing pa closer Paam- Further- more, the gradient magnitude is weighted by both the dis- tance (P) Puaua (x)) and 1 - do(x), assigning higher weights to samples discriminated as fake.
then the optimal target model that minimizes the GAN dis- criminator loss matches the data distribution. We formalize this induced objective in the following theorem.
Theorem 3.1 (Optimality). With unlimited model capacity, the optimal likelihood- based model po under the objective
3.2. Theoretical Analysis
Apart from the optimality guarantee, we also examine the be- havior of the DDO objective when 0 is not optimal. Specifi- cally, we investigate the following question:
log (x(x)) )
min L(8) Eppcn (z)
logo
pe (x) (x))
log (a(
Eppo,, (z))
log 1 -
Is pa closer to Paata with a lower C(0)?
po (x)
Under certain assumptions, we can establish bounds on the divergence between Po and Paaaa in terms of the difference
satisfies po- = Paatu-

---

Direct Discriminative Optimization
between (() to the optimal loss value L³, as formalized in the following theorem.
performance. We sweep over them for the best results.
Handling Compute-Intensive Likelihood Evaluating the model likelihood for specific data point can be computa- tionally intensive. In particular, unlike autoregressive mod- els, which only require a single forward pass through the network to compute log pa (x) (Eqn. (1)) due to the causal structure imposed by attention masks, diffusion models ne- cessitate multiple forward passes over different timesteps to approximate log (0 through the ELBO (Eqn. (2)). Specifically, the log-likelihood ratio in the DDO loss is:
_u_ Theorem 3.2 (Divergence Bounds). log Pl and log pew are bounded, there exist some constants C1, C2 such that
Paase
(Pdota Il po) S Civ ((0) C³ DKL (pe Paaaaa S C2V C (12)
(11)
The assumption of bounded log Pe- implies that the opti- mized distribution does not deviate significantly from the reference distribution, which is reasonable when finetuning for a short duration. The assumption of bounded log Pead. Pau imposes a constraint to the reference model regarding its mutual density coverage with the data distribution. We Pird to be lower bounded, i.e., Pa suffi- ciently covers Pdata. which aligns with the characteristics of MLE-trained models. Under this condition, the forward KL DKL (Paaa Pe) remains bounded by VC(0)- L. How- ever, bounding the reverse KL requires upper bound plat which imposes a stronger constraint on pr Pea
pdat
log po( Er,e [A...ee
(14)
p0 (
where 2t = 02 + 0,E and
can expect log paau
Amette = w(t)
(1))((2,,)) ( c112 (x,, - <l1)
(15)
We apply Jensen's inequality pointwise derive an up- per bound for the loss using the convexity of the function -a log (xx b blog(1 - az)) for any a, b 2 0:
3.3. Practical Implementation
[(0)
~ (xen(() log (Er,e (A])
Epp (e) log(1 a (Er.c [A]))
We introduce several practical techniques that make DDO applicable to nigh-dimensional real-world data and diffusion models whose likelihood computation is expensive.
s Er, | log (A) + Eyg (w) log(1 o(Ai))
(16)
This treatment, analogous the one used in Diffusion- DPO (Wallace et al., 2024), enables us to approximate the diffusion DDO loss using single forward pass for each æ.
Generalized Objective with Extra Coeffecients The log- likelihood log po of likelihood- generative models often scales with the data dimension and can reach magni- tudes of 1 As the DDO objective in Eqn. (9) involves Sigmoid operation log po (ax), this can lead to gradient vanishing and numerical precision issues. To address this, we add hyperparameters a, 3 to control the relative weights of loss terms and scale the probability ratio:
Multi- Multi-Round Refinement via Self-Play Due to the practi- cal modifications for applicability, the optimization process of DDO provides useful gradient information in the early stage but does not converge the data distribution in the final. To maximize the fine-tuning performance, we adopt a multi-round refinement strategy, where the reference model Pe is iteratively updated by replacing it with an improved version from the previous round:
pa (xr)
(0) E pa)((a)
log a BBlog
Pord (xx)
po -eEEpop (a)) log 1 1 Blog
pe (x)
pa
(13)
Round
pe
log
par
The modified loss retains the same optimization trend, namely, increasing (x) for x ~ Pdata and decreasing po(x) for x ~ Pe... but the optimum may "overshoot" the data distribution for B < 1. Specifically, we have:
Reference
Disenminator
Round n + 1:
per
Theorem 3.3. With unlimited model capacity. the optimal likelihood-baa based generative model that minimizes (()) 1-1/8 1/9 satisfies pa- x P Pduna for certain a.
where 8 represents the berf-performing model across differ- ent hyperparameter configurations and training iterations in round n. In each round, the reference model acts as fixed generator, making the multi-round optimization analogous to the mnmerator-discrmmnaa interplay in GANs. However, unlike GANs, where both networks are explicitly optimized, we never update the reference (generator) model directly. Instead, the generator is obtained from the discriminator in the previous round, leading to a form of self-play. This iter- ative refinement process is conceptually similar to Iterative
This establishes deep connection with guidance methods (discussed Section 4.2). In practice, we observe that a and 8 across a wide range of values3 yield reasonable
A code example is provided in Appendix D. 'Typical choices a E [0.5, 50] and 3 E [0.01, 0.1), while the optimal values depend on the specific model and settings.
5

---

Direct Discriminative Optimization
DPO
Human Preference (paired)
DG
AG
x1.71 DDO
Gap?
x1.00
x1.00 00
base
Alignment
Params
Inference Time
(unpalient
DDO
Figure 5. Comparison of model parameter counts and inference time across different guidance methods and DDO. For DG, measure the statistics on class-conditional CIFAR-10. For AG, we measure the statistics ImageNet-664.
Figure 4. Comparison with DPO.
DPO (Xu al., 2023) and SPIN (Chen al., 2024c), which extend DPO for better language model alignment.
optimization of pretrained language models on preference data without training a separate reward model:
3.4. Discussion
CDPO
Connection to RL At high level, DDO enables visual generative models to utilize negative signals from self- generated samples-a characteristic deeply rooted in rein- forcement learning (RL) that underpins modern language models (Achiam et al., 2023; Guo et al., 2025). Distin- guished from works that employ similar contrastive loss merely off-the-shelf data (Chen al., 2024a), DDO can fundamentally improve the base model's ability.
Blog (V TT (bwl
To( (z))
(s (owan log
log
Tea (wn
(19)
where the reward function (8(y)x)) is implicitly parameter- ized by the log- likelihood ratio log
ma(ulz)
ste, (ulæ))
Despite sharing similar insights in parameterization, DDO is fundamentally different from DPO. As illustrated Fig- ure 4, DPO is designed for preference learning, requiring additional paired human-annotated data and maximizing the likelihood gap between preferred (winning) and non- preferred (losing) responses without considering the whole distribution. In contrast, DDO focuses distribution align- ment, directly aligning the model with the ground ground-truth data distribution. It requires only the original training data that are unpaired with the genel-ggnerated samples.
Extension to f-divergence The GAN discriminator loss can be generalized to f-divergences (Nowozin et al., 2016);
Dy(p ll g) = sup Ep() [x)) Eg(x)) [f (T(x)) (17)
where f is the convex conjugate of f. DDO can be ex- tended this family as the optimal (x) f explicitly involves the density ratio.
p()))
((x)
4.2. Guidance Methods
4. Comparison with Existing Methods
We review several types of guidance methods that enhance diffusion models* at inference time. Let (0 t) denote the score function representation introduced Section 2.1.
4.1. Direct Preference Optimization (DPO)
DPO (Rafailov et al., 2024) is a lightweight surrogate ob- jective designed for reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Achiam et al., 2023) that enhances the instruction- instruction-following following ability of pre- trained language models. Standard RLHF involves two stages: (1) learning a reward model тe and (2) aligning the reference policy Th... to the target policy Tee (y)2)) ox a ( root)))))) using RI., where x the prompt and is the response. The Bradley-Terry preference mode (Bradley & Terry, 1952) links preferences and rewards by
Classifier-Free Guidance (CFG) (Ho Salimans. 2021) combines the unconditional/conditton model to obtain new score predictor 8 (x,,, c) = sp( L, c) + w (z,, t, e) 59 t, 0)), where 0 represents the un- conditional case, and w is the guidance scale. The uncondi- tional model shares parameters with the conditional model and learned by random label dropout during training.
Discriminator Guidance (DG) (Kim et al., 2023a) trains dime-dependent discriminator network dg that distinguishes between perturbed real data and model-generated sam- ples. Its gradient is then used to refine the score function:
p(y > 3ele x)
(r((I, ) - r (x, 31))
(18)
(a Similar $0,0 s0 (xt, ; w wVn m, log 1=d.(
where yo and y denote the winning and losing responses for a given prompt x, annotated by human. DPO enables direct
*Guidance can be easily adapted autoregressive models.
6

---

Direct Discriminative Optimization
DDO, DG leverages the optimal discriminator (Eqn. (4)) for theoretical guarantees. However, it explicitly parameterizes the discriminator separate, time-aware network.
enforce it. For diffusion models, finetune over multiple rounds until further improvement is negligible. For VAR, we observe rapid convergence and only finetune for 2 rounds. The finetuming is highly efficient, with each round requiring less than I %% of pretraining iterations. Further experiment details can be found in Appendix C, and visualizations of generated samples are provided in Appendix E.
Autoguidance (AG) (Karras et al., 2024a) operates similarly to CFG but refines the score function by guiding the base model with an inferior variant: (z), == se 60 t) + (( t) s sp(x;,t)) t)), where & is a degraded version of se obtained via reduced model capacity or under-training,
5.2. Results on Diffusion Models
For unified perspective, all these guidance methods im- prove well-trained distribution po by amplifying its dif-
Table I. Results on unconditional and consiionniiional CIFAR- CIFAR-10. "Inlluding diffusion distillation methods with auxiliary GAN loss. The reported parameter count excludes those of the discriminator (for GANs) and VAE cncoder/decoder (for latent-space models).
ference from a degraded or less informative distribution pei
Pa.e O po Po
Pd This superposition sharpens the MLE-
optimized model distribution and suppresses low-probability outliers (Karras et al., 2024a). According to Theorem 3.3, DDO with 3 < 1 induces a similar overshooting effect, highlighting the theoretical connection.
Type
Model
Paramss
NFE
Unzoed
Cond
FID]
FIDJ
StyleGAN2 (Karas 2020)
20M
StyleGAN-XL XL. (Smeer et al., 2022)
18M
1.85
GAN
REGAN (Hunng zl. 2025)
21M
1.96
CTM al., 2023b)
59M
1.98
Unlike guidance methods, DDO enhances sample quality without increasing inference costs compared to the base model (Figure 5). Moreover, in scenarios where CFG is cru- cial for balaneing image-condition alignment and diversity (e.g.. FID-IS curve), DDO can be seamlessly integrated with CFG to achieve overall improved trade-off (Section 5.3).
SiD A (Zhou al. 2024bb
56M
1.50
1.40
DDPM (Ho et 2020)
36M
1000
3.17
IDPM (Nichol Dhhriwall 2021)
50M
40001
2.90
DDIM (Ilo et all, 2020)
36M
100
4.16
DPM-Soller
2022h)
107M
4%
DPM 9PM--ollvr-y3 (Zheng ct al., 2023a)
56M
12
2:24
Diflusion
NCSN+ (Song al., 2021b)
108M
2000
2.20
LSGM (Vahdat
2021)
376M
136
VDM (Kingma et al., 2021)
31M
1000
7.41
Flow Matching (1 ipmon et al., 2022)
142
6.35
i-ODE (Zaeng al., 20236)
139M
215
EDM (Kums et all, 2022)
56M
35
1.97
1.79
5. Experiments
DG (Kim al., 2023a)
107M
53
1.77
1.64
Our experiments aim to investigate the following aspects:
Ours
+ DDO
EDM (reteszed)
56M
56M
35
1.38
1.97
1.85
1.30
1. The effectiveness and efficiency of DDO in enhanc- ing the visual quality of well-trained diffusion models (Section 5.2) and autoregressive (Section 5.3) models.
Table 2. Results on class-conditioaal ImageNet-644
Type
Model
#Purams
NBE
FID
StyleGAN-XL (Sauer et al.,, 2022)
135M
1
1.51
2. The impact of the hyperparameters a, 3, as well as the benefits of multi-round refinement.
REGAN (Huang et 2025)
104M
2.09
GAN
CTM (Kim et al., 20236)
324M
1
1.92
DMD2 (Yin al., 2024)
296M
1,28
PaGoDA (Kim et al., 2024)
296M
1
1,21
5.1. Experimental Setups
SID'A (7hom al. 20244)
296M
IDDPM (Nichol & Dhariwal, 2021)
270M
250
2.92
Datasets & Models We experiment on standard image benchmarks including CIFAR-10 (Krizhevsky et al., 2009) in 32x32 resolution and ImageNet (Deng et al., 2009) in multiple resolutions 664 256x256, 512x512). For each dataset, we apply DDO to finetune state-of-the-art dif- fusion or autoregressive models, including EDM (Karras et al., 2022), EDM2 (Karras et al., 2024b) and VAR (Tian et al., 2024). We compare with range of advanced gener- ative baselines, including diffusion models, autoregressive (AR) models, masked models, and GAN-based approaches.
(Dhariwal Nichol, 2021)
296M
250
RIN (lahri et al., 2022)
281M
1000
1,23
EDM (Kamas al.. 2022)
296M
511
1,36
VDM++ (Kingma Gao, 2024)
296M
511
1.43
DisCo- (Ku al., 2024)
623
1.22
Diffusion
EDM2-8 (Karras al., 2024h)
280M
63
1.58
CFG (Ho & Salimans, 2021)
560M
126
1.48
AG (Kamms et al., 2024.)
405M
126
1.01
EDM2-M
498M
63
1.43
EDM2-L
777M
63
1.33
EDM2-XL
I.IB
63
1,33
EDM2-S (retested)
280M
63
1.60
Ours
DIXX
280M
63
0.97
The EDM and EDM2 base models on CIFAR- and Ima- geNet are implemented as separate unconditional or class- conditional networks. Since CFG provides limited benefits these datasets, we directly apply the diffusion DDO loss without considering the interaction with CFG.
Training& & Evaluation We evaluate Fréchet inception distance (FID) (Heusel et al., 2017) on 50k images as the primary benchmark metric for all experiments, and addition- ally measure Inception Score (IS) (Barratt& & Sharma, 2018) for ImageNet 256x 255. We report the number of function evaluations (NFE) as a quantification of inference efficiency. We find strict class balance crucial for FIDs on ImageNet and slightly modify the original EDM sampling scripts to
Main Results Table 1, Table 2 and Table 3 present the quantitative results on CIFAR-10 10 and ImageNet. Figure 6(a) illustrate the FID reduction over multiple rounds. We high- light the advantages of DDO as follows:
7

---

Direct Discriminative Optimization
2.0
EDM 1O, Canditional)
EDM (CIFAR-10, Unconditional)
EDW2. (4 Conditional)
1.90
1.85
Continue Trarting MLE
8=0.02
8-0.01
0-0.03
1,90
1.05
Trainin with SUL
2=2.0
e=1.
2-3.0
1.8
EDM24 (1mageNet-522. Conditiona)
1.80
8-0.05
4,4.
6 0.1
1.80
2-5.0
1.6
1.75
6-6.0
1.75
1.4
1.70
1.65
1.70
1.60
1.65
1.0
20
250
500
1000
1250
1500
250
500
750
2000
1250
15:00
Round
Iterations
Iterations
(a) FID-RRund
(b) d = 4.0
(c) 3 = 0.05
Figure 6. Illustrations of DDO on diffusion models, including (a) multi-round refinement and (b)(c) training curves under different a, S.
The ImageNet FIDs in this figure evaluated without class rehalancing.
Table 3. Results on class-conditions ImageNet 512x 512. "G" denotes guidance (CFG by default). The reported NFE values are without guidance: applying guidance doubles them. With classi- fier guidance. With autoguidance (AG) (Karras et al., 2024a).
larger) with only 4 rounds, demonstrating the parameter efficiency unlocked by DDO.
Effects of or, As shown in Figure 6(b)(), we visualize the training curves under different a, 3 for class-conditional CIFAR-10 10 during the first round. We empirically find that a wide range of a, 3 consistently improves the base model, though identifying the optimal hyperparameters requires grid searching. Moreover, tuning a while keeping 3 fixed or adjusting 3 under appropriate a yields similar effects.
Type
Madel
wlo G
w/G
*Params
NFE
HDD
FID
N-dep (Brack, 2018)
112M
8.45
GAN
StyleGAN-XL (Saser 2022)
168M
2.41
SID' A (Zhou et al., 2024b)
1.58
1.37
ADM (Dbariwal Nichol, 2021)
559M
250
23.24
7.72
ADM-U
731M
9.96
3.851
XU/2 (Peebles & Xie, 2023)
675M
250
12.03
3.04
SIT-XL (Ma et al. 2014)
675M
2.62
REPA et 20144)
675M
250
2.08
Comparison to MLE Figure 6(b)(c) also includes the FID curve of extended training using the original diffusion loss (i.e., MLE). Unlike DDO, continued MLE training fails to improve performance and even leads to degradation when we do not retain previous optimizer states. This is partly because the base model is already extremely optimized and more fundamentally due to the forward KL objective's 's in- herent limitations. In contrast, DDO with various , con- figurations consistently demonstrates clear advancements.
RIN Uabti et al., 2022)
410M
1000
3.95
U-VIT, (Hoogghoom
2023)
3.54
3.02
Diffusion
VDM++ (Kingma & Gun, 2024)
2B
512
2.99
2.65
USIT-2B (Chen et al., 2024h)
2B
2,90
1.72
EDM2-XS
2024b)
125M
EDM2-S
250M
2,56
2.23
AG
1.34
777M
2,06
1.88
EDM2-XXL
1.5
63
1,91
1.81
AG
1,25t
MaskGIT (Chung et 2022)
237M
7.32
Masked
MAGVIT-22 (Yu et 2023)
307M
64
3.07
1.91
MAR.L (Li 2024)
481M
1024
2.74
1.73
AR
VAR-136-s (Tinn et 2024)
2.38
10
2.63
Accelerate Sampling By leveraging the advanced sam- pler DPM-Solver-v3 (Zheng al., 2023a), we reduce the in- ference steps of EDM2-L+DDO to 25 while preserving gen- eration quality. Our model achieves FID scores of 1.29/1.21 for 512x512 12 images, with an end-to-end per-image latency of 1.03s/1.966 on a single H100 GPU (batch size = 1).
EDM2 L. (ratested)
777M
63
1.96
1.77
Ours
DDO
777M
1,26
1,211
SaM (Zheng al.., 2023a)
777M
1.29
1.21
(I) Effectiveness. With multi-round refinement, DDO achieves record-breaking guidance-free FID scores of 1.38/1.30 on CIFAR-10, 0.97 on ImageNet-64, and 1.26 on JmageNet. 512x512, significantly improving upon the EDM and EDM2 base models by 30%, 40% and 36%, respec- tively. Additionally, DDO outperforms all guidance-based and GAN-based methods requiring complex GAN-specific tuning or increasing inference costs.
5.3. Results on Autoregressive Models
The VAR models rely heavily on CFG to enhance gener- ation quality. A distinctive feature of CFG is its ability to balance diversity and fidelity by adjusting the guidance scale, which is essential for creating the FID-IS off Consequently, we need to accommodate DDO to ensure that the finetuned models remain compatible with CFG. To this end, we choose the reference and target distributions pe Pe as the guidance-free free model corresponding to w 0. To pre- serve the model's ability for unconditional generation, we incorporate random label dropout during DDO fine- mun-tuning. We set a for the unconditional part to prevent it from receiving negative signals from reference samples æ Pe.a
(2) Efficiency. Although we employ substantial training over dozens of rounds to maximize the performance and explore the upper bound of DDO, FID improves significantly within just few rounds. Notably, DDO in single round achieves FIDs of 1. 1.72/1 .58 on CIFAR-10 with the EDM base model, surpassing DG. On ImageNet-64, the compact EDM2-S attains an FID of 1.31 after only 3 rounds, surpassing the 4x larger EDM2-XL. On ImageNet 512 112, guidance- free EDM2-L matches CFG-enhanced EDM2-XXL. (2x
8

---

Direct Discriminative Optimization
VAR-di6
VAR 420
VAR-d1ó trick
VAR-130 trick
100
VAR-d16 - DDO
VAR-530 DIO
guidmce-fres
guidmne-ffee
300
250
4-
A0D wortoo
1:30
130
3
2.54
30
200
300
250
300
350
15
Guidance scale
Gnidince scale
(a) VAR-d16
(b) VAR-d30
(c) FID and IS w.r.f. guidance scale w
Figure 7. Illustrations of DDO on autoregressive models. (a)(h) FID-IS trade-off curves and (c) the impact of a under 3 = 0.02.
Table 4. Results class-conditional ImageNet 256x 256.
(2) Guidance-free performance. DDO significantly reduces the guidance-free FID from 11.33/4.74 to 3.12/1.79 for VAR-d16 and VAR-d30, achieving 3.6x and 2.6x improve- ment. Notably, the guidance-free FIDs even outperform CFG-enhanced FIDs (3.30/1.90) of the original VAR results, indicating that higher-quality samples can be generated at half the inference cost with DDO. This is also superior to methods like CCA (Chen et al., 2024a) which only aim to remove the CFG but harm the model performance.
"Samples are generated at 512 x 512 resolution and resized to 256 x 256 using OpenCV's LANCZLA method.
Type
Model
wio
*Params
NPE
FIDI
FID.
BigGAN deep (Brock. 2018)
112M
6.95
GAN
GigoGAN (Kang ol. 2023)
569M
3.45
StylsGAN-XL XL. (Sauer et 2022)
166M
2.30
ADM (Dbariwal& & Nichol, 2021)
S54M
250
10.94
4.59
ADM-U
608M
500
7.49
3.94
LDM4 (Rambach et ol., 2022)
400M
250
10.56
3,60
(Peobles Xic, 2023)
675M
250
9.62
2.27
Diffusion
ST--L (Ma et al., 2024)
675M
250
8.3
2,06
(3) CFG-enhanced performance. When combined with
RHPA (Yu
2024b)
675M
250
5.90
1.42
RIN (Jabri et al., 2022)
410M
1000
3.42
CFG, the finetuned VAR models achieve significantly better
U-VIT, (Hoogshooo
2023)
2B
512
2.77
2,44
FID-IS trade-offs than the pretrained counterparts, even when the latter employ sampling tricks. The lowest FID improves from 3.30/1.90 2.54/1.79. Furthermore, the finetuned VAR-d16 outperforms the 2x larger VAR-d20 (600M parameters) which has a CFG-enhanced FID of 2.57.
VDM++ (Kingma Gao, 2024)
512
2.40
2.12
LighmingDiT (Ya et al. 2025)
675M
250
2.17
1.35
MaskGIT (Chsing et
227M
6.18
MAGVIT-22 (Yu et 2023)
307M
64
3.65
1.78
Masked
MAR-H (Li 2024)
943M
256
2.35
1.55
MaskBa (Weber et al., 2024)
305M
256
1.52
VQGAN (Faser ol., 2021)
148
256
15.78
VTTVVGAN et 2021)
4.17
Effects of a Figure 7(c) visualizes FID and IS varying the CFG scale, where we finetune VAR-d16 under 3 = 0.02 and different a for 60 iterations. The results indicate that all a E [10, 100] consistently achieve a CFG-enhanced FID lower than that of the base model. Larger values of a tend to yield lower guidance-free FIDs but may slightly weaken performance when combined with CFG.
RQ- Transsffoomer (Lee et al., 2022)
3.8B
68
7.55
LlamaGen-38
2024)
3.1B
256
13.58
3,05
AR
Open MAGVIT2-XLL (Luo al., 2024)
1.5B
256
9.63
2.33
VAR-J16 (Tian 2024)
310M
10
3.62
3,30
VAR-130
2,08
10
2.17
1,90
RAR-XXI (No et al. 2024.)
1.5B
256
3.91
1.48
XAR-H (Res al., 2025)
1.24
VAR-J16 (w/o sumpling tricks)
310M
10
11:35
3.71
DDO
310M
10
3,12
2.54
VAR-430 (wlo sampling nicks)
2.0B
10
4.74
1.92
Ours
CCA (Chen al., 202(2)
2.0B
10
2.54
2.0B
EDM2-L a DDO downsompied")
777M
63
1.26
1.21
6. Conclusion
+ Solveee (Zheng al., 2023a)
777M
25
1.29
1.21
In this work, we introduce Direct Discriminative Optimiza- tion (DDO), universal enhancement technique designed for visual likelihood- generative models. Inspired by the GAN framework and the parameterization insights from DPO, DDO breaks the curse of forward KI. and substan- tially improves generation quality. Extensive experiments demonstrate its remarkable effectiveness and efficiency, sur- passing state-of-the t-thh-art diffusion and autoregressive models and achieving acord-breaking FID scores on standard im- age benchmarks. There remain promising directions for future exploration, such as eliminating the need for hyperpa- rameter searching, improving inference efficiency through distillation, and scaling to tasks like text-to text-to-image genera- tion. We leave such avenues for future work.
Main Results Table 4 presents the quantitative results on ImageNet 256x Figure (ba)(b) illustrates the FID- IS trade-off varying the CFG scale. We summarize the advantages of DDO as follows:
(1) Eliminating sampling tricks. is worth noting that the original VAR results are based on top-k and top-p sampling strategies, which artificially lower the temperature. These heuristics introduce training-inferent gap and fail to re- flect the genuine capability of pretrained models. In contrast, when evaluating models finetuned with DDO, we discard all such tricks, ensuring a more principled assessment of generative performance.

---

Direct Discriminative Optimization
Impact Statement
Chang, H., Zhang, H., Jiang, L.,, Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Chen, H., Su, H., Sun, P., and Zhu, J. Toward guidance-free ar visual generation via condition contrastive alignment. arXiv preprint arXiv:2410.09347 2024a.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Chen, J., Cai, H., Chen, J., Xie, E.,, Yang, S., Tang, H., Li, M., Lu, Y., and Han, S. Deep compression autoen- coder for efficient high-resolution diffusion models. arXiv preprint arXiv:2410. 10733, 2024b.
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gen- erative adversarial networks. In International conference on machine learning, pp. 223. PMLR, 2017.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17992 2021.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. arXiv preprint arXiv: 403.0133, 2024c.
Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, IEEE, 2009.
Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q.. Kreis, K., Aittala, M., Aila, T., Laine, S., et al. ediff- Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint ar arXiv:2211.01322 2022.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Bao, F. Xiang, C., Yue, G., He, G., Zhu, H., Zheng, K.,
Zhao, M., Liu, S., Wang. Y., and Zhu, J. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024.
Barratt, S. and Sharma, R. A note on the inception score. arXiv preprint 1 080197997, 2018
Dhariwal, P. and Nichol, A. Q. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume 4. Springer, 2006.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density esti- mation using real nvp. arXiv preprint 1Xiv:1605.088033 2016.
Brock, A. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: / 1096, 2018.
Du, Y. and Mordatch, I. Implicit generation and modeling with energy based models. Advances in Neural Informa- tion Processing Systems, 32, 2019.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luh- man, E., et al. Video generation models as world sim- ulators. 2024. URL https://oppeaii comfresearc/viddeo generation-modde www.olee-----wwrd--smmiieee
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12888 2021.
3, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are fhw-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
10

---

Direct Discriminative Optimization
Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint hitv:194409324. 2019.
Huang, Y., Gokaslan, A., Kuleshov, V., and Tompkin, J. The gan is dead; long live the gan! a modern gan baseline. 2025.
arXiv preprint ar..iv25001.08.55..
Jabri, A., Fleet, D., and Chen, T. Scalable adaptive computation for iterative generation. arXiv preprint 22..v:221..11 1972, 2022.
Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016. http: / /tl///www /www.
deeplearningbook
Kang. M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 10124- 10134. 2023.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C.,, and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pp. 2672- 2680, 2014.
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. Training generative adversarial networks with limited data. Advances in neural information pro- cessing systems, 33:121044122 14, 2020.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint XKi::2501.129453 2025.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022.
Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Fei-Fei, L.. Essa, I.. Jiang, L., and Lezama, J. Photorealistic video generation with diffusion models. arXiv preprint arXiv:23/2.06666 2023.
Karras, T., Aittala, M., Kynkäänniemi,, T., Lehtinen, J., Aila,
Gutmann, M. and Hyvärinen, A. Noise-contrastive estima- tion: A new estimation principle for unnormalized statisti- cal models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp- 297-304. JMLR Workshop and Conference Proceedings, 2010.
T., and Laine, S. Guiding diffusion model with a bad version of itself. arXiv preprint arXiv:2406.0255 02507, 2024a.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024b.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge local Nash equilibrium. In Guyon, I,, von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 6626 0000-6637722 2017.
Kim, D., Kim, Y., Kwon, S. J., Kang, W., and Moon, I.-C. Refining generative process with discriminator guidance in score-based diffusion models. In International Con- ference on Machine Learning, pp. 16567-16598. PMLR, 2023a.
Ho, J. and Salimans, T. Classifier-fres free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Ue- saka, T., He, Y., Mitsufuji, Y., and Ermon, S. Consistency trajectory models: Learning probability flow ode trajec- tory of diffusion. In The Twelfth International Conference on Learning Representations, 2023b.
Ho, J. Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Kim, D., Lai, C.-H., Liao, W.-H., Takida, Y., Murata, N., Ue- saka, T., Mitsufuji, Y., and Ermon, S. Pagoda: Progressive growing of one-step generator from a tow-resolution dif- fusion teacher. arXiv preprint arXiv:2405.14822 2024.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 10.00:22.0002303, 2022.
Kingma, D. and Gao, R. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024.
Hoogeboom, E., Heek, J., and Salimans, T. simple diffu- sion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp- 13213-13232, PMLR, 2023.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In International Conference on Learning Repre- sentations, 2014.

---

Direct Discriminative Optimization
Kingma, D. P., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. In Advances in Neural Informa- tion Processing Systems, 2021.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S Autore- gressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11523-11553 11532, 2022.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831, Pmir, 2021.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint ar arXiv:2406. /838, 2024.
Lipman, Y., Chen, R. T., Ben- Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv 2022.
Ren, S., Yu, Q.. He, J., Shen, X., Yuille, A., and Chen, L.-C. Beyond next-token: Next-x prediction for autoregres- sive visual generation. arXiv preprint arXiv:2502.20388 2025.
preprint
Lou, A., Meng, C., and Ermon, S. Discrete diffusion lan- guage modeling by estimating the ratios of the data distri- bution. arXiv preprint arXiv:2310. 112223111168311 16834, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Lu, C., Zheng, K., Bao, F., Chen, J., Li, C.,, and Zhu, J. Maximum likelihood training for score acore-based diffusion odes by high order denoising score matching. In Inter- national Conference on Machine Learning, pp. 14429- 14460. PMLR, 2022a.
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marro- quin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.0755: 2024.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances Neural Information Processing Systems, 2022b.
Sahoo, S. S., Deschenaux, J., Gokaslan, A., Wang, G., Chiu, J. T., and Kuleshov, V. The diffusion duality. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025.
Luo, Z., Shi, F. Ge, Y., Yang, Y., Wang, L., and Shan, Y. Open-magvit2: An open-source project toward democra- tizing rugo-segressvve visual generation. arXiv preprint arXiv:2409.044100 2024.
Salimans, T. and Ho, J. Progressive distillation for fast sam- pling of diffusion models. In International Conference on Learning Representations, 2022.
Ma. N., Goldstein, M.. Albergo, M. S., Boffi, N. M., Vanden- Eijnden, E., and Xie. S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. arXiv preprint arXi::2401.08744 2024.
Sauer, A., Schwarz, K., and Geiger, A. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International Conference Machine Learning, pp. 8162-8171. 8171. PMLR, 2021.
Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.00 04344.00049,,... 2024.
Nowozin, S., Cseke, B., and Tomioka, R. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Song, Y., Durkan, C., Murray, I., and Ermon, S. Maxi- mum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems, volume 34, pp. 1415-1428, 2021a.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
12

---

Direct Discriminative Optimization
Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint orxvv2406.006525. 2024.
Xu, Y., Corso, G., Jaakkola, T., Vahdat, A., and Kreis, K. Disco-diff: Enhancing continuous diffusion models with discrete latents. arXiv preprint arXiv:2407.03330 2024.
Yao, J., Yang, B., and Wang, X. Reconstruction vs. gener- ation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXiv:2501.0122, 2025.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2204. 022005, 2024.
Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Du- rand, F., and Freeman, W. T. Improved distribution match- ing distillation for fast image synthesis. arXiv preprint arXiv:2405. 14867, 2024.
Tschannen, M., Fastwood, C., and Mentzer, F. Givt: Gen- crative infinite- vocabulaay transformers. In European Conference on Computer Vision, pp. 292-309. Springer, 2024.
Yu, J., Li, X., Koh, Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector suctor-quantize image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021.
Vahdat, A., Kreis, K., and Kautz, Score-based generative modeling in latent space. In Advances in Neural Informa- tion Processing Systems, volume 34, pp. 11287-11302, 2021.
Yu, L.,, Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., et al. Language model beats fiffusiion-okkenizee is key to visual generation. arXiv preprint arXiv:23 2023.
Van Den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. In International conference on machine learning, pp. 1747-1756. PMLR, 2016.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Yu, Q.. He, J., Deng, X., Shen, X., and Chen, L.-C. Ran- domized autoregressive visual generation. arXiv preprint arXiv:2411.0077 2024a.
Wallace, B., Dang. M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C.,, Joty, S., and Naik, N. Diffusion model alignment using direct pref- erence optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228-22222 8238,823,,222 2024.
Yu, S., Kwak, S., Jang, H., Jeong, J., Huang, J., Shin, J., and Xie, S. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint 10. 2024b.
Zhang, J., Huang, H., Zhang, P., Wei, J., Zhu, and Chen, J. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In Internu- tional Conference on Machine Learning (ICML), 2025a.
Wang, Z., Zheng, H., He, P., Chen, W., and Zhou, M. Diffusion-gan: Training gans with diffusion. arXiv preprint ar arXiv:2206.0222 02262, 2022.
Zhang, J., Wei, J., Zhang, P., Xu, X., Huang, H., Wang, H., Jiang, K., Zhu, J., and Chen, J. Sageattention3: Microscal- ing fp4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv:2505. /594, 2025b.
Weber, M., Yu, L., Yu, Q. Deng, X., Shen, X., Cremers, D., and Chen, L.-C. Maskbit: Embedding-fir free image gen- eration via bit tokens. arXiv preprint arXiv:2409. 16211, 2024.
Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, J. Sageatten- tion: Accurate 8-bit attention for blug-and-play inference acceleration. In International Conference on Learning Representations (ICLR), 2025c.
Xiao, Z., Kreis, K., and Vahdat, A. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference Learning Representations, 2022.
Xie, J., Mao, W., Bai, Z., Zhang, J. Wang. W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint 240i 12528, 2024.
Zhang, J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., and Chen, Spargeattn: Accurate sparse attention accelerat- ing any model inference. In International Conference on Machine Learning (ICML), 2025d.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:23/2. /66682 2023.
Zheng, K., Lu, C., Chen, J., and Zhu, J. DPM-Solver- v3: Improved diffusion ode solver with empirical model statistics. In Thirty-sevent Conference on Neural Infor- mation Processing Systems, 2023a.
13

---

Direct Discriminative Optimization
Zheng, K., Lu, C., Chen, J., and Zhu, J. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference Machine Learning, pp. 42363-45 42389. PMLR, 2023b.
Zheng, K., Chen, Y., Mao, H., Liu, M.-Y., Zhu, and Zhang, Q. Masked diffusion models secretly time- agnostic masked models and exploit inaccurate categori- cal sampling. arXiv preprint arXiv:2409.02:
2024.
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:2408, 11039, 2024a.
Zhou, M., Zheng, H., Gu, Y., Wang, Z., and Huang, H. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. arXiv preprint arXiv:2410. 14919, 2024b.
14

---

Direct Discriminative Optimization
A. Related Work
Paradigms of Generative Models Beyond autoregressive (AR) and diffusion models introduced in Section 2.1, other likelihood-basee generative models have been explored in the literature. Variational autoencoders (VAEs) (Kingma & Welling, 2014), energy-based models (Du & Mordatch, 2019), and normalizing flows (Dinh et al., 2016) are once popular for generative modeling of continuous data, but have since fallen out of favor due their limited expressiveness and lack of scalability in modern large-scale generation tasks. In particular, VAEs are now primarily used as dimensionality reduction tools that compress data into latent spaces (Esser et al., 2021; Rombach et al., 2022), rather than generating samples from scratch. For discrete data generation, masked models, such as BERT (Devlin et al., 2019), CMLM (Ghazvininejad et al., 2019) for masked language modeling and MaskGIT (Chang et al., 2022) for masked image generation, offer an alternative likelihood- basddd paradigm to AR. While discrete diffusion models (Austin et al., 2021; Lou et al., 2023; Shi et al., 2024; Sahoo et al., 2024) have recently regained interest, they are largely equivalent to the simpler masked models and suffer from hidden numerical precision issues that lead to unfair evaluation when measured by the generative perplexity metric alone (Zheng et al., 2024), suggesting that the notion of "diffusion" is actually unnecessary or even harmful in these models There have also been pioneering efforts to combine different generative paradigms. MAR (Li et al., 2024) integrates masked modeling with the diffusion loss, enabling autoregressive image generation with continuous tokens. Transfusion (Zhou et al., 2024a) and Show-o (Xie et al., 2024) combine AR with diffusion/masked models for mulli--oddl generation, effectively synthesizing a mixture of text and image data. DDO is potentially applicable to these models for quality enhancement, and we leave such explorations for future work.
Improving Generation Quality with GAN Except for using GAN as auxiliary loss for enhancing one-step few-step generation in diffusion distillation (Kim et al., 2023b; Yin et al., 2024; Zhou et al., 2024b), as mentioned in Section 2.2, several works have explored directly integrating diffusion models with GANs. Notably, Xiao et al. (2022) replaces the reverse denoising steps in diffusion models with a sequence of conditional GAN generators, enabling few-step generation. Wang et al. (2022) modifies the GAN discriminator to distinguish between diffused real and generated samples in a time-aware manner. Kim et al. (2023a) leverages the gradient information from a trained discriminator to refine pretrained diffusion models. Chen et al. (2024a) adopts a binary classification loss with likelihood ratio parameterization similar to our objective, but its applicability is limited to removing CFG in autoregressive models and degrades the model performance. Apart from quality, quantized or sparse attention (Zhang et al., 2025c;a;b;d) can be employed in parallel to accelerate inference.
B. Theoretical Analyses of the DDO Objective
In this section, we investigate the theoretical properties of the DDO objective and provide informal proofs to Theorem 3.1, Theorem 3.2, and Theorem 3.3 in the main text.
B.1. Analyses of C(0)
Optimal Solution is straightforward to show that the optimal minimizing C()) satisfies pa- = Paam following the Pe (x)
common GAN literature (Goodfellow et al., 2014). Specifically, let ro( == log between the learnable and reference distribution. The objective L(0) can be expressed as an integral form:
(m)) denote the log-likelihood ratio
[(0) f ((0)mde
(20)
where
[()),= = Paaaaaa)) log (ro(2))) - Po... (x).
(ro(2))) 0
(21)
is the pointwise loss, and we only consider x in the valid range where Plata and Pa.,, have nonzero support. For any (a, b) E (R,) 0)}. the function y + -a a log y b log(1 - v). y E [0, 1] achieves its minimum at aa
a+5. Applying this to
the pointwise loss (()) z.. the minimizer satisfies
Po- (x)
Pdatu (x)
(ro- (x))
pg- (xx)
Pdata (x)
(22)
po-
(x)
+ pa (x) Pdata (x)) + pr (x)
Since the global minimizer of L(0) is the pointwise minimizer of L(0),, for all æ, it follows that pg- Pdata-
$ Apart from the masked case, other discrete diffusion variants, such uniform, have a more genuine connection diffusion (Sahoo
al., 2025).
15

---

Direct Discriminative Optimization
Loss Gradient Using the derivative identity dr (x)) Applying these to the pointwise loss, derive the gradient w.r.t. ro(ar):
da(2) a (z) da)), we obtain doogg(()
o(x), d dlog( 1-o(a))
da
dir
dl(()) - po.. po.. (1))((()(() - Pama (
Po(ar)
Pdata (2)
(1)o((())))
(x + po (2)) = (1- (1-d((z))(oo(z) - Pauaa (x)) (23)
dro(z)
Thus, the full loss gradient is given by
VoL(O) VoL(O) = / (l((0)= dro(x)
(oro( f (1--d(()))(olla) - Paltta log
(24)
Divergence Bounds We aim to derive bounds for the divergence between Pa and Pdata when is not optimal, using the difference between the loss value (()) and its optimal counterpart L Without any assumptions, the forward KL divergence
DKL. (pima Po) is lower bounded by (0) L By definition, L is the minimum loss value achieved when po = Pduta-
The difference (()) L can be decomposed as follows:
(()) L- = f Pdata (z) log
Po (x)
Pome (x)
+ Da (2) log
dæ
Po (x) + Pa (a)
pa (x) + pa (a))
+ f Pdataa (x)) log
Peinta (x)
po,, (x))
Pdttaa (x:) + pr (xx) + pa (2x) log
dx
Pditaa (az) + pr (22)
(25)
/ Pauta (ax) log
Pditta (x)
Po + p0 (
+ (Pamta (x) + Po (x))
log
Paatal (x) + Pons (x) cx
(x(x))
Pdata + Pdnt
po + pdmt
= (Péata |1 po) DKL 2
2
Therefore, DRL (Pasa Po) = (() - c + DRL( Pins Pa
PP
z [()) - L- While this result establishes a lower
2
bound for the divergence, additional assumptions are required to derive an upper bound. Specifically, we assume that
log
Peat and log are bounded, i.e., there exist constants M, M1, M2 such that [ro())] log piat
pe (ux) S M, M1 I
pe (xx)
Perr (ar)
log
< M2 for all x. The pointwise loss be expressed as a function of ra():):
Paau [z)
[(0) = f(ro(x)))
(26)
where
f(y) = Pamaa (x) log (8) Po., (x) log (1)) = (Puatu (x) + Pon. (x)) log (1 + e²) Paata
(27)
The first and second order derivatives of f are given by:
f' (v) = Pe.. (x)e"" Pdata
f (v) (Pdata (x) + pon (x)
(asm((x)) + pon (x)
(28)
1 eV
(1 + e2²2 eV)2
2 e + +e-
Applying Taylor's expansion at y rg- (xx
log (ur) pi (w)) we obtain:
f(ro(x)) = f(ro- (x)) + f'r (x))(ro(z) - ro. (x)) + ( (r)))(((t))rr) - -rg- (x)+
(29)
where & [min((r(ww) ro (m)), (rx (z)) Since fr (æ)) 0 and ro())) rr- (xx) log
Pan (x)) we get:
Po (a
log (x() )²
2
J"(() (C(0) L()))z)
(30)
Pdata (x))
Note that ("(x) is monotonically decreasing function w.r.t. ly/ and attains its maximum at the boundary of the given range, we have:
2
2
} S max { ('(ro((z))
2
< max
("(ro(x))) (" (x))
(31)
2+c M + M
Pdata (x) + (x)
= 2 max
Pdaaaa (x) + pen (x))
Pdnaa (x) po
16

---

Direct Discriminative Optimization
Therefore,
Pdata (2) (log Po (ax)
2 + C + e.M
Pdatta (a)) + Do (a)
- 2 max
(L(0) 2()))a))
Prdata (a:)
pa (2:) (a:)/Pmnaaa))
pa (20)
(32)
2 + e M +e
2 max
e -11ee eeee-a -M
(C(()) 2(00)=))
Applying Jensen's 's inequality, we derive an upper bound for the forward KL divergence:
Pdatal (2z) pa (x:
Pdltaa (xz) (logg
2 = S Pdota( (x)
log pel po(x) 2
DKL (Panta || po) = E Epena (ax)
log
(x)
dx
Po (x)
Pdltta (x:
(33)
I Civ ((0)- L.
where Ci
V
2 max
2maxx
24e My-- 1tctl
1 + e-Mii] related to the lower bound of log Plut, Similarly, Piasa
log
po(x))
)
2
Po (o
2 e- + e.d
Pdata (x) + Pa (x))
(o())
1n8x
(2(0)== - (((0)
Pdutal (2r) (x))
po (xx)
1 Pauaa (ar)/Po
Petata (zz)
(34)
2 + e -M + M
1 e e-MMMa ) [C((()=)))))))))) - L(( c
< M max
By integrating over æ, we obtain DKL (po Pdtaa) S C2)D(() L", where C2 =1/20 max
{
2tc M 140
1 + e-M2
related to the upper bound log Pfu
B.2. Analyses of (0)
Once we introduce additional coefficients a, B, the generalized DDO objective (f))(f) may become intractable and no longer admit pg- Pdata as the optimal solution. Specifically, the pointwise loss with a, is
La -daataaae (x) log (Bro(w)) ape (x) (((og())))))
(35)
The optimal should satisfy
dL),,(0)m a.ppa (Brr - 3pdata (z) (Bro(x))) = dro(w)
Pdtta ( (x)
(log
Pattal (x)
Do (Bro((x))
(36)
Pdttaa (2:) + (ap)
opp (x))
Pdata (x:
1/3
-x)(x=) Pe (az)
a.pe.. (x:)
However, since Pe parameterized as likelihood-band based generative model, must have self-normalized density. This optimality condition only achieved when a is proper normalizing constant satisfying
f
=1 1 = [o(w))d=1==-==
(ʃ
1- 1-1/3 1111/(())))))) (x)) (ar)dar) Pow
1/8 (x)
(37)
Under this specific choice a, the optimal solution pa
1-1/8 1/8 Pánta Otherwise, the optimization subject to the constraint f dx 1. To enforce this, we introduce a Lagrange multiplier À and define the Lagrangian:
Pou
C / (a.,(()) + A (1 (1- / po(x))da)
(38)
17

---

Direct Discriminative Optimization
To find the optimal po. we take the functional derivative of L w.r.f. Pa and set zero:
8C
dl), A = (z)) - Bp (z) - a (aro())))) dpp(xc)
dro(z)
X = 0
(39)
(x)(x;)
dpo(z)
which can be simplified to
po (2x) pr (x:)
8
A pa (2))
(o(a)) Po (2:)
)}
Pdatal (az)
a
(40)
panr (x)
pone (x)
This equation, combined with the constraint f po (z) (zz)da 1, determines both and the optimal po- However, no closed- form solution exists for this problem. Despite this, we can expect certain ranges of a to skew the optimal solution away from Paata toward the direction of Powe 1-1/8 Pauta even the exact equality does not hold.
C. Experiment Details
Throughout all experiments, each training run under given set of configurations (certain reference model and hyperparame- ters a, () is conducted on a single node with 8 NVIDIA A100 (SXM4-80GB) GPUs.
We follow the parameterization and noise schedule of EDM (Karras al., 2022) and EDM2 (Karras et al., 2024b). Specifically, EDM introduces a time-dependent skip connection that preconditions the denoiser D, (which predicts clean data an)) using a free-form network Fo, allowing Fo to predict an adaptive mixture of signal and noise:
Diffusion Models
Do 1) = Cskip (1) + Cou (1) (Cin(t)) Cuoise (t)
(41)
where
daaat
Cskip (t)
Coul (t) d +
Ctn(t)
Cnoise (t) = log
(42)
data
(2
V +
4
EDM employs simple variance exploding (VE) noise schedule satisfying a₁ = 1, = t. It's worth noting that the preconditioning used in EDM actually transforms it into v-prediction under the variance preserving (VP) noise schedule, owing to the normalizing factor (t) and the skip connection coefficients Cakip (t), Cour(t) (Zheng et al., 2023b). The EDM models are pretrained by a F-prediction MSE loss:
CEDM (0) Poterrppp...ttp((eeeeee)
Fo (ci(()))) Caoise (1)) F (x00 æz, 1)12
(43)
(aalf) where vt 2o te, Flooo x., 20 (t) is the prediction target, and p(t) is time distribution satisfying
log/ L ~ ((Pmean P2 where Pmeaa, Pad are hyperparameters.
We adopt similar settings for DDO finetuning. Specifically, we use the approximation in Eqn. (16) and set the weighting wt) satisfy c(t)llco c1l2 Fg F12,, leading the following objective (1 (xx) (-=)))
CEDM-DDO a
(0) -Bap)()) Epna (2o) log (-3 (11Fg
F12 Fo F 1
(44)
+aB (mo) log (B ||Fa F1B
|
112
where we use the same form of time distribution logt ~ NNPmean: Pl and Pincan Pau are typically the same as pertaining. See Appendix for the specific implementation. For each finetuning round, we launch ~0 nodes to sweep over the hyperparameters a, 9 in [0.5, 6.0] [0.01, 0.1.. We disable all dropout layers in the network ensure steady improvement. We also find numerical precision crucial for the diffusion DDO loss and disable mixed precision training. For each round, 50k images are generated offline from the reference model as the reference dataset.
For EDM on CIFAR-10, we finetune the unconditional and class-conditional model for 12 and 16 rounds, respectively. We set P Pmeaa = 112, Pana = 1.2 throughout all rounds, which is the same as pretraining. Each round has a duration of 1.5M images (30 epochs, 0.75% of pretraining) with a batch size of 512. The learning rate warms up linearly from 0 to 1.5e during each round, and the data augmentation probability is set 12% as pretraining. We find the exponential
18

---

Direct Discriminative Optimization
moving average (EMA) beneficial for stabilizing the model performance and choose a relatively small EMA half-life (0.25M
images) as we finetune less duration than pretraining. We evaluate the FID each time trained with 50k images and save the best model for the next round. Each round takes ~3h including both training and evaluation.
For EDM2-S on ImageNet-64, we finetune the model for 24 rounds, where we set Pmman = 0.8, = 1.6 for the first 16 rounds following pretraining, and increase Pud to 3.0 in the last rounds. Each round has a duration of 6.4M images (5 epochs, 0.6% pretraining) with batch size of 512. We use a learning rate 5e 5 for the first 16 rounds and 2e 5 for the last 8 rounds, along with the learning rate scheduler in EDM2 which is mix of linear warmup and inverse square root decay, where we set the ramp-up to IM images and the decay reference 2000 iterations. We follow the power function EMA introduced in EDM2 and set the EMA length to 0.05. We evaluate the FID each time trained with 2 (= 131k) images and save the best model for the next round. Each round takes ~ld including both training and evaluation.
For EDM2-L on ImageNet 512x512, we finetune the model for 28 rounds. Following the pretraining setup, we fix Piean = 0.4, while progressively increasing the variance Pau to 1.6, 2.0, and 3.0 starting from the 1st, 18th, and 22nd round, respectively. Each round has a duration 6.4M images (5 epochs, 0.34% of pretraining) with a batch size of 2048. We use a learning rate of 1e- 4 for the first 24 rounds and 5e 5 for the last rounds, along with the learning rate scheduler in EDM2 which is mix of linear warmup and inverse square root decay, where we set the ramp-up to 1M images and the decay reference 800 iterations. We follow the power function EMA introduced in EDM2 and set the EMA length to 0.05, We evaluate the FID each time trained with 28 ( 262k) images and save the best model for the next round. Each round takes 2d including both training and evaluation. We further boost the finetuned model with autoguidance (Karras et al., 2024a). We choose edm2-img5 - 111-0-11111---1 65. pk1 from Karras et al. (2024a) as the bad version model and use a small guidance scale of 1.1, which slightly improves the FID from 1.26 to 1.21.
Autoregressive Models We finetune VAR-d16 and VAR-d30 (Tian et al., 2024) both for only rounds. VAR is highly efficient at inference, enabling to sample from the reference distribution online during training by generating random latent tokens with the reference model conditioned on the same class labels those in the dataset batch. We disable all dropout layers in the network and set the label dropout probability to 50% for unconditional training. Unlike diffusion DDO, we enable mixed-precision when finetuning VAR. For each round, we launch ~~1 nodes to sweep over the hyperparameters 3 3 in [10.0, 100.0) x (0.02). Each round has a duration of 80 iterations (0.064 epoch, less than 0.03% of pretraining) with a batch size of 1024. We follow the learning rate scheduler in VAR pretraining and set the peak learning rate to smaller value of 4e 6. We evaluate the FIDs (corresponding to guidance- moderate CFG scale) every 4 iterations and save the best model for the next round. Each round takes ~5h/7.5h for VAR-d16/d30 including both training and evaluation.
D. Code Example
import torch
import torch.nn nn. functional as
Original diffusion loss of EDM
class EDMLoSS:
def
init_ (self, F F_mcan=-0.4, P_std=1.0, sigma_data=0.0 :) Self.P_mest = P_mean
= P_std
self self.aigma_dart sigma_data
def
net, images, labels=None)
net: the target denoiser network
images: real samples from the dataset, shape (B, C, H, w)
m*n
Sample diffusion time
and_norraal = torch. randn (limages. shape[0], 1, 1, 1], device=images device)
sigma (rnd mor__normal-- * sels.P_std self.P self.F_mean) (rean.ess ex
Diffusion loss weighting
weight (sigma**2 + sclf..igga_dataree
(sigma * self, sigma_datttttt data) **
sample Gaussian noise
noise
like (images) sigma
19

---

Direct Discriminative Optimization
Denoise
denoised net (images noise, sigma, labels)
# Compute loss
loss torch. sum (weight * (denoised images) ** 2, dim- (1, 2, 3)) return moss.meart ()
Di Diffusion 990 loss of EDM
class EDMLoss_DDO: def
_ini:_ P_mean=-0.4, P_std=1.0, sigma_data-0.5 alpha-1.0, beta=0. 02)
Self.P_mes P mean
self.P_std = P_std
self.sigma_datt _aatma_d_tt___ = sigma_dar data
sclf.alpha alpha
sclf.beta beta
def _call_ net, ref_net, images, make_images, labels-None, f (onne))
net: the target denoiser network
ref net: the reference denoiser network (frozen)
images: real sampies from the dataset, shape (B, C, Н, W)
fake images: fake sampies generated by the reference model, shape (B, C, H, W)
sample diffusion time
end_normal torch. randn ([images chape[0], 1, 1, 11, dovice-images.devae device) sigma = (rnd_normal + self.P_std + self.P_mean) ()
Diffusion loss weighting
weight (sigmav*2 self. salff.iigaaddataaae
(sigma self. sigma_dataat
Sample Gaussian noise
hoise = ranch_ (images) sigma
Denoise by the target model
net (images noise, sigma, labels)
net.eval
D_fake net (fake_images noise, sigma, fake_labels) tetitrain()
D D_oo -torch. sum (weight (D images) ** 2, dim= (1, 2, 3))
fake logp -torch. sum (weight fake fake images) 2, dim= (1, 2, 31)
Denoise by the reference model with torch (o
ref_D ref_net (images + noise, sigma, labels)
ref_D_fake ref_net (sake_images noise, sigma, fake_
ref ref_D_logp -torch. (weight (ref_D images) ** 2, dim= (1, 2, 3))
-torch.sum (weight (ref_D_fake fake_images) ** 2, dim= (1, 2, 3)
# Compute loss
Loss = -F 1s..ogiggoii (self. beta (D_ ret {_logpp)
-self alpha F Logssigmoid (eoggiigoid(-sel..eee (-self. (D. fake logp ref (akke_oogp)
return loss.mear
E. Additional Results
20

---

Direct Discriminative Optimization
Figure 8. Random samples of EDM (CIFAR-10, 10, Unconditional), FID 1.97.
Figure 9, Random samples of EDM + DDO (CIFAR-10, Unconditional), FID 1.38.
21

---

Direct Discriminative Optimization
Figure 10. Random samples of EDM (CIFAR-10, 10, Class- conditionn..,,.. FID 1.85.
Figure Random samples of EDM DDO (CIFAR-10, Class-conditional) FID 1.30.
22

---

Direct Discriminative Optimization
rscort
Figure 12. Random samples of EDM2-S (ImageNet-64, Class-conditional), FID 1.60.
Figure 13, Random samples EDM2-S DDO (ImageNet-64, Class-conditional), FID 0.97.
23

---

Direct Discriminative Optimization
Figure Illustration of multi-round refinement pricess on EDM2-S (ImageNet-64).
24

---

Direct Discriminative Optimization
VAR-d16 trick (FID 11.33)
VAR-d16 w/ trick (FID& 3.62)
VAR-d16 DDO (FID 3.12)
SUGA
Figure 15. Guidance- free samples by pretrained and finetuned VAR-d16.
25

---

Direct Discriminative Optimization
A/D. WAT WeAREEReeeee
VAR-d16 trick (FID 3.71)
VAR-d16 w/ trick (FID 3.30)
VAR-d16+ DDO (FID 2.54)
Figure 16. CFG-enhanced samples by pretrained and finetuned VAR-d16.
26

---

Direct Discriminative Optimization
ORS BG
Key aille
VAR-d30 trick (FID 4.74)
WEG CRA
CAMTO!
Yeul
VAR-d30 w/ trick (FID 2.17)
Buuk !
J
V DDO (FID 1.79)
2nblooy
Figure 17. Guidance-free samples by pretrained and finetuned VAR-d30.
27

---

Direct Discriminative Optimization
reekee
VAR-d30 trick (FID 1.92)
LOFONN
VAR-d30 w/ trick (FID 1.90)
OIO tamn
VAR-450 DDO (FID 1.73)
Figure 18. CFG-enhanced samples by pretrained and finetuned VAR-d30.
28


Reviews:
[{"review_id": "f36U2ChFFd", "reviewer": "Reviewer_wtcU", "summary": "This paper introduces Direct Discriminative Optimization (DDO), a novel finetuning framework designed to enhance the generation quality of likelihood-based generative models, such as diffusion and autoregressive models. Likelihood-based generative models are inherently limited by the mode-covering tendency of maximum likelihood estimation (MLE), which restricts their performance under limited model capacity. The key innovation of DDO lies in its parameterization of a discriminator using the likelihood ratio between a learnable target model and a fixed reference model, drawing inspiration from Direct Preference Optimization (DPO).\n\n\n## Update After Rebuttal\n\nI have reviewed the authors' rebuttal. I appreciate their response addressing my questions regarding the lack of Recall metric analysis and the specific hyperparameter settings (large alpha, small beta) required for DDO.\n\nOverall, the authors have satisfactorily addressed my concerns. Therefore, I maintain my score of 4 (Accept).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments. Below, we provide detailed responses to your concerns.\n\n> One potential weakness of the paper is the lack of discussion on diversity metrics Recall during the DDO finetuning process. Analyzing how recall evolves during the DDO distillation process could provide deeper insights into whether the method maintains or improves diversity while enhancing sample quality.\n\nWe did not report the Recall metric because (1) it is measured only on relatively large-scale datasets like ImageNet 256x256, while not on CIFAR-10/ImageNet-64 (2) it is not as importance as FID/IS metrics, and good generative models often have indistinguishable Recall scores (as reported in the VAR paper). Moreover, the FID metric itself is sensitive to the generation diversity. A model with low diversity will yield degenerated FID scores. \n\nAdditionally, we would like to provide some new results on higher-resolution datasets. After submission, we further use DDO to finetune EDM2-L on ImageNet 512x512, successfully advancing the FID from 2.11 to **1.36**, without any guidance. Some visualizations are provided in the [anonymous link](https://anonymous.4open.science/r/DDO/). We find DDO significantly enhancing the image quality/details without affecting the overall content/diversity. We will release all model checkpoints and code that reproduce the reported results upon acceptance.\n\n> Why does DDO require such specific hyperparameter settings (large alpha and small beta) to weight fake samples, while GANs do not seem to need this additional weighting?\n\nThough we show that a discriminator can be implicitly parameterized by a generative model to utilize the GAN loss, an implicit discriminator and an explicit discriminator (used in GANs) can have different expressiveness and inherent output space structure. As mentioned in the paper, the log-likelihood of a generative model scales linearly with the data dimention, so does the implicit discriminator. Therefore, we need a small beta to manually scale it. A large alpha is an empirically choice to oppose the fake samples and accelerate the FID decrease. For an explicit discriminator, the output is directly produced by a separate network (more expressive) and can be freely scaled internally, so fewer manual operations outside are needed.\n\nThank you again for your consideration, and for giving positive rating. We hope that our response can resolve your concerns, and we are happy to answer further questions.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. The authors demonstrate the effectiveness of Direct Discriminative Optimization (DDO) through extensive experiments, showing significant improvements in generation quality across multiple benchmarks. Additionally, they provide consistent improvements in FID scores for both guidance-free and CFG-enhanced autoregressive models on ImageNet 256×256.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the problem and application at hand. The use of benchmark datasets such as CIFAR-10, ImageNet-64, and ImageNet 256×256 is appropriate, as these are widely recognized and challenging benchmarks for evaluating generative models. The evaluation metric, Fréchet Inception Distance (FID), is a standard and reliable measure for assessing the quality and diversity of generated images.", "theoretical_claims": "I have reviewed the theoretical proofs in the paper to a reasonable extent, and while I did not perform a 100% detailed verification, the proofs appear to be correct and well-justified.", "experimental_designs_or_analyses": "The experimental setups are well-structured and comprehensive. The authors evaluate their proposed Direct Discriminative Optimization (DDO) method on standard image benchmarks, including CIFAR-10, ImageNet-64, and ImageNet 256×256, which are widely used and respected in the field of generative modeling. They apply DDO to finetune state-of-the-art models such as EDM, EDM2, and VAR, and compare their results against advanced generative baselines, including GAN-based approaches. This ensures a fair and rigorous evaluation of DDO's effectiveness.", "relation_to_broader_scientific_literature": "The key contributions of this paper, particularly the introduction of Direct Discriminative Optimization (DDO), are closely related to broader advancements in generative modeling literature. DDO bridges the gap between likelihood-based generative models (e.g., diffusion and autoregressive models) and adversarial training frameworks like GANs, addressing the mode-covering limitation of maximum likelihood estimation (MLE).", "essential_references_not_discussed": "After reviewing the paper, I did not find any particularly critical or essential references that were missing from the discussion.", "other_strengths_and_weaknesses": "Strength: The proposed Direct Discriminative Optimization (DDO) creatively combines ideas from likelihood-based generative models (e.g., diffusion and autoregressive models) and adversarial training frameworks (e.g., GANs), addressing the fundamental limitation of maximum likelihood estimation (MLE) in mode-covering.\n\nWeakness: One potential weakness of the paper is the lack of discussion on diversity metrics Recall during the DDO finetuning process. Analyzing how recall evolves during the DDO distillation process could provide deeper insights into whether the method maintains or improves diversity while enhancing sample quality.", "other_comments_or_suggestions": "I do not have any additional comments or suggestions for the paper.", "questions_for_authors": "The theoretical framework of DDO is elegant, but the paper mentions that a relatively large alpha value (e.g., alpha = 4.0) and a small beta value are required for practical effectiveness. Why does DDO require such specific hyperparameter settings (large alpha and small beta) to weight fake samples, while GANs do not seem to need this additional weighting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bYhQTcBsre", "reviewer": "Reviewer_GoSA", "summary": "This paper tackles the issue of the predisposition of likelihood-based generative models to cover modes of the data distribution, yielding unrealistic or blurry results. This paper introduces a solution to this problem that is complementary to the usual but impractical guidance methods via a fine-tuning method that integrates self-guidance-like mechanisms during training.\n\nThis fine-tuning, named DDO, relies on parameterizing a discriminator by the log probability ratio between current and reference generated distributions, that is optimized to fit the true discriminator between the data and reference distributions. This provably makes the current generated distribution converge towards the data distribution by improving density concentration on the modes.\n\nBy adopting tricks to make this method tractable, notably by estimating log likelihoods with Monte-Carlo samples of the ELBO and applying it iteratively on the reference model, the authors manage to show its significant benefits and obtain state-of-the-art generation performance on standard image datasets with and without guidance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments. Below, we provide detailed responses to your concerns.\n\n> Can the few debatable claims be reformulated along with the title?\n\n- I disagree with the title...it is not a discriminator per se, but can be used to parameterize a discriminator.\n\nWe agree with \"it is not a discriminator per se, but a parameterization\". Our use of the expression \"your xx is secretly xx\" mirrors the famous DPO [1]. In DPO, a language model is also not a reward model per se, but can be used to parameterized a reward model. We think the expression is a convention in the community and will not be misleading. Otherwise, we can also change the title to \"your ... can implicitly parameterize ...\".\n\n[1] Direct preference optimization: Your language model is secretly a reward model\n\n> Other improper claims in abstract/intro/Section 3.1... Formatting issues...\n\n we partially agree your comments but have to post discussion later due to character limitation of rebuttal stage\n\n> comparison to further regular training of the base model?\n\nOn unconditional CIFAR-10 with the EDM base model:\n\n|Iterations|0|500|750|1000|1250|1500|\n|:--------|:-------:|:-------:|:------:|:------:|:-------:|:-------:|\n|MLE Continued|1.97|1.961|1.969|1.972|1.988|2.011|\n|DDO ($\\alpha=6.0,\\beta=0.05$)|1.97|1.896|1.852|1.809|1.733|1.769|\n\nBy continuing training with the MLE objective, the FID just fluctuates within a small range and never advances the performance like DDO. This phenomenon can be expected, not only because the baselines are already extremely optimized, but also for the traditional MLE's inherent nature. As stressed in the paper's abstract and introduction, MLE is mode-covering and will be penalized seriously if the model underestimates the likelihood of any training samples. This limits the MLE objective's upper bound performance under limited model capacity. Therefore, we can never expect significant generation quality improvement by continuing training with MLE, while DDO is a fundamentally different way. We will add this comparison in the revised paper.\n\n> Improvements on related work\n\n- (Kingma & Gao, 2024)\"\n\nWe think this is common sense for diffusion models: the minimizer of denoising score matching is the ground-truth score function at every $t$ regardless of specific choices of $p(t),w(t)$. We cite (Kingma & Gao, 2024) because it is a systematic discussion on how to understand different $p(t),w(t)$ and how they are theoretically equivalent. \n\n- More stable GAN settings have been found\n\nWe will temper the \"unstable\" statements for GANs. The instability is not the only factor why we favor diffusion/autoregressive models over GANs. Diffusion/autoregressive models are proven scalable to high dimensional data. Besides, they have an iterative generation process, which intuitively provides a larger potential generation capability compared to GANs.\n\n- discussion of Chen et al. (2024)\n\nWe acknowledge the similarities and will add a section in the appendix to discuss the relationship. In our opinion, DDO is distinguished from CCA philosophically. On a high level, DDO's utilization of self-generated data can be seen as a form of on-policy optimization that roots deeply in reinforcement learning (RL) in today's language models. Similar to RL, DDO can fundamentally improve the base model's ability, while CCA cannot.\n\n- $f$-divergence GAN\n\nThank you for introducing the $f$-divergence GANs! We think these generalizations are applicable to DDO, while they are unable to directly address the numerical issues as the likelihood ratio $\\frac{p(x)}{q(x)}=\\exp(\\log\\frac{p(x)}{q(x)})$ can be explosive ($\\log\\frac{p(x)}{q(x)}$ scales with data dimension). Nevertheless, we will add a section in the appendix to discuss this possible extension and leave more detailed exploration to future work.\n\n> No code is provided\n\nDDO is simple to implement. To show reproducibility, we now release some code for the EDM2 codebase at the [anonymous link](https://anonymous.4open.science/r/DDO/). The loss function is simple (`DDO_EDMLoss` in `training_loop.py`), and we only need to add an additional dataloader to load model-generated fake samples.\n\nAdditionally, we provide some new results on higher-resolution datasets. After submission, we further finetune EDM2-L on ImageNet 512x512, successfully advancing the FID from 2.11 to **1.36**, without any guidance. Some visualizations are provided in [anonymous link](https://anonymous.4open.science/r/DDO/). We find DDO significantly enhancing the image quality/details without affecting the overall content/diversity. We will release all model checkpoints and code that reproduce the reported results upon acceptance.\n\nThank you again for your consideration, and for giving positive rating. We appreciate your responsibility in carefully reviewing our work and providing very contrustive suggestions that help us improve. We hope that our response can resolve your concerns, and we are happy to answer further questions.", "claims_and_evidence": "The central claims of the paper, summarized above, are **clearly presented and well supported** either by theoretical or empirical evidence.\n- The theoretical analysis is particularly appreciated as it provides **informative insights into the method** and helps design its practical implementation. It also makes the method simple to understand and **facilitates its reusability**. Up to my limited assessment (cf. below), theoretical results appear to be sound.\n- Experimental results **clearly show the benefits of the proposed fine-tuning compared to the base model and usual guidance methods**, notably the one of Chen et al. (2024) who also attempted to bypass the need for extra network/inference in guidance. The icing on the cake is that it enables the proposed method to surpass the state of the art in image generation.\n- To my knowledge, the idea of parameterizing a discriminator with the log likelihood of the model itself and training the latter by optimizing the same discriminator w.r.t. a reference, non-data, generated distribution, **is novel and has the potential to be widely reused**.\n\nNonetheless, a **few secondary claims need to be amended**.\n- I disagree with the title and the paper stating that \"your likelihood-based [...] model is secretly a GAN discriminator\": it is not a discriminator per se, but can be used to parameterize a discriminator. I would advise the authors to reformulate this claim and the title to a more factual statement.\n- The paper states in the abstract and the introduction that DDO is a \"unified framework that bridges likelihood-based generative training and the GAN objective\". This is misleading as it is rather a framework integrating (not bridging) likelihood-based models in a GAN discriminator training (not overall GAN objective, i.e. without the generator objective).\n- In Section 3.1, it is stated that \"stable convergence requires stronger initial conditions\" based on Section 3.2. While the choice of strong model initialization is generally well motivated, this assertion is too strong. Section 3.2, particularly in Theorem 3.2's assumptions, simply shows that strong model initialization is sufficient for stable training. Therefore, this claim should be toned down.", "methods_and_evaluation_criteria": "The evaluation of the proposed method is **in line with the standards of the literature**, both in terms of datasets (CIFAR-10 and two resolutions of ImageNet) and criteria (FID and IS). While other, more modern and less biased evaluation metrics could be considered, this is a shared shortcoming with almost all papers in this area of research.\n\nComparisons include most recent state-of-the-art methods. However, like in most other papers again, it is not clear whether the reported results were reproduced by the authors or simply reported from the original papers. I would recommend this to be specified somewhere in the paper.", "theoretical_claims": "I checked the correctness of all theoretical results by skimming through proofs in the appendix. **Up to my limited assessment, these theoretical results are sound.** This should be sufficient given that most derivations use standard tools from the literature.", "experimental_designs_or_analyses": "Conducted experiments are also done **within the standards of the literature** and based on the reference implementation of score-based diffusion models (EDM). There are, however, two important misses.\n- The evaluation **misses a comparison with an extended regular training of the baseline** (EDM) with comparable execution time to definitely decide on experimental relevance, although I have few reasonable doubts as most baselines are already extremely optimized.\n- **No code is provided** in the submission and no promise is made to release it for publication. This greatly hinders the reproducibility of the proposed method.", "relation_to_broader_scientific_literature": "The literature is overall well covered and discussed in the paper. It is a clear motivation for the introduction of the method as a means to alleviate mode covering behaviors of likelihood models. Nonetheless, two minor improvements should be considered.\n\n**ELBOs for diffusion models.** The authors state that \"alternative choices [of $p(t)$ and $w(t)$] share the same optimum as the true ELBO and can serve as surrogate objectives (Kingma & Gao, 2024)\". Can they elaborate this claim? I am not sure this is the result outlined in the referenced paper (which shows that alternative choices result in ELBOs with data augmentation).\n\n**GANs.** The paper discusses characteristics of GANs from the common knowledge in the literature, typically that their training is \"unstable\" or that their optimization if based on the Jensen-Shannon divergence or Wasserstein distance. While all this is supported by literature from the last decade, it should be tempered. More stable GAN settings have been found e.g. with StyleGAN (Karras et al., 2019) or FastGAN (Liu et al., 2021), and more comprehensive analyses of GAN optimization now deviate from the Jensen-Shannon / Wasserstein paradigm (Franceschi et al., 2022; Yi et al., 2023).\n\nKarras et al. A Style-Based Generator Architecture for Generative Adversarial Networks. CVPR 2019.\\\nLiu et al. Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. ICLR 2021.\\\nFranceschi et al. A Neural Tangent Kernel Perspective of GANs. ICML 2022.\\\nYi et al. MonoFlow: Rethinking Divergence GANs via the Perspective of Wasserstein Gradient Flows. ICML 2023.", "essential_references_not_discussed": "All strongly related works were discussed in the paper. Still, I would advise the authors to develop the discussion of Chen et al. (2024), which is only quickly mentioned in the appendix. While it is a concurrent work by ICML's guidelines, it does share similar techniques to the ones introduced in the submission. It would be beneficial for the community to acknowledge more precisely the similarities between both papers.", "other_strengths_and_weaknesses": "The method is simple and effective. The paper is overall very clear and well written. Overall, I believe this submission may have a **substantial impact on this area of research** both for the proposed fine-tuning and the underlying techniques.\n\nA small caveat is the need to tune additional hyperparameters, as is the case of most new techniques. Nevertheless, the sensitivity analysis included in the paper suggests that their optimization is not a limiting factor, especially compared to the greater complexity of direct competitors.", "other_comments_or_suggestions": "- I would like to ask the authors whether they considered other $f$-divergence GAN discriminator losses (Nowozin et al., 2016). To my understanding, the proposed method could be adapted to theses losses as their optimal discriminators involve density ratios. Some of them may even solve gradient vanishing issues more directly than the generalized objective introduced in Section 3.3.\n- The paper states that \"autoregressive models [...] learn discrete data distributions\" (Section 2.1), but they can also model continuous ones as shown e.g. by Tschannen et al. (2024).\n- The paper has a number of formatting issues that need to be solved for future versions.\n  - Many equations go over the columns' edges.\n  - Proper space should be left between captions and figures/tables.\n  - The dataset corresponding to experiments in Figure 5(b)(c) is not specified.\n  - The notation $s$ for the guidance scale is confusing as it share the same notation as the score network.\n  - Authors should check whether all figures are colorblind-friendly.\n\nNowozin et al. $f$-GAN: Training Generative Neural Samplers using Variational Divergence Minimization. NIPS 2016.\\\nTschannen et al. GIVT: Generative Infinite-Vocabulary Transformers. ECCV 2024.", "questions_for_authors": "The paper successfully presents a novel, high-potential, simple, and effective fine-tuning method for likelihood-based models. As such, it is valuable and I think it should be accepted. However, I only recommend a \"Weak accept\" as the paper is lacking in a few aspects, detailed above and summarized below. As I believe all of them can be addressed in the discussion period, I am willing to increase my score depending on the authors' answers.\n\n1. Can the few debatable claims be reformulated along with the title?\n2. How do the proposed fine-tuning compares to further regular training of the base model?\n3. Can the authors promise to release the code upon publication?\n\nI still expect other, more minor, weaknesses of the paper mentioned in my review to be addressed either in the author's response or in future versions of the paper.\n\n---\n\n## Post-rebuttal update\n\nGiven the satisfying answer provided by the authors below, I now recommend an \"Accept' provided that the paper is improved accordingly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zmO9smf0Ze", "reviewer": "Reviewer_622T", "summary": "This paper proposes a new optimizing method for likelihood-based models. The idea comes from GANs, but the method is not to introduce a new discriminator, which is more efficient and easy to apply. Theoretical analysis and experiments prove the effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments. Though you do not have notable concerns, we would like to provide some new results on higher-resolution datasets.\n\nAfter submission, we further use DDO to finetune EDM2-L on ImageNet 512x512, successfully advancing the FID from 2.11 to **1.36**, without any guidance. Some visualizations are provided in the [anonymous link](https://anonymous.4open.science/r/DDO/). We find DDO significantly enhancing the image quality/details without affecting the overall content/diversity. We will release all model checkpoints and code that reproduce the reported results upon acceptance.\n\nThank you again for your consideration, and for giving positive rating. We hope that our additional results can further support our method, and we are happy to answer further questions.", "claims_and_evidence": "The authors claim that their method will not complicate the training or increase inference costs.\n\nIn my opinion, we can also introduce a traditional GAN discriminator for the optimization of likelihood-based models. Thus, we need a direct comparison, which shows that utilizing the proposed method can achieve better, or at least comparable performance when compared with methods that utilize the traditional discriminator. Based on this kind of results, the proposed method can be valuable when it is efficient and easy to apply.", "methods_and_evaluation_criteria": "The proposed method makes strong sense for the current generative models.", "theoretical_claims": "I do not check all theoretical results carefully but take a quick look. I do not find any obvious error.", "experimental_designs_or_analyses": "I have checked the experiments. The experiments involve proper baselines, benchmarks, and evaluation metrics. I think there should be an additional ablation, as mentioned in \"Claims And Evidence.\"", "relation_to_broader_scientific_literature": "This paper essentially reports a work of combining GANs and likelihood-based models. Though it is not the first one, the idea of implicitly defining the discriminator is new, at least has never been utilized for improving likelihood-based models, to my knowledge.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "71AcmoeF0s", "reviewer": "Reviewer_bcz2", "summary": "The paper proposes fine-tuning a likelihood-based generative model to steer the generated distribution towards the real distribution from a new perspective. Specifically, the authors parameterize pretrained likelihood-based generative models as the discriminator within the GAN framework. Using a reference generative model, they unify the discriminator and generator as variants of the same generative model with different weights. The paper provides theoretical derivations and analyses of the method’s design, leading to a simple yet effective formulation for parameterizing the discriminator. The effectiveness of the proposed approach is demonstrated by fine-tuning EDM and VAR on the CIFAR-10 and ImageNet datasets. Additionally, ablation studies are conducted to analyze the choice of hyperparameters $\\alpha, \\beta$, and the image generation quality progressively improves during the refinement iterations.\n\n## Update after Rebuttal\nThe authors have addressed most of my concerns. Although the added hyperparameters require additional tuning, I maintain my weak accept to support the paper’s acceptance given its novelty.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments. Below, we provide detailed responses to your concerns.\n\n> The motivation for introducing $\\alpha$ and $\\beta$ to address gradient-vanishing/numerical issues is not well-supported. It would be helpful if the authors could provide experimental evidence to show the problem and demonstrate how $\\alpha$ and $\\beta$ help mitigate these issues.\n\nThe training will simply collapse without introducing a small $\\beta$. For example, on CIFAR-10, the data dimension is 3072, and the bits per dimension (BPD) is around 2.5, meaning the log-likelihood $\\log p_\\theta(x)$ is around $3072\\times2.5\\times\\log 2\\approx 5000$. The log-likelihood ratio $\\log\\frac{p_\\theta}{p_{\\theta_{ref}}}$ also scales linearly with the data dimension and can be as large as hundreds, making $\\sigma(\\log\\frac{p_\\theta}{p_{\\theta_{ref}}})$ suffering from gradient-vanishing. If we use $\\beta=1$, the gradient will vanish when $p_\\theta$ only deviates a bit from $p_{\\theta_{ref}}$, and the finetuning almost does nothing to the base model. We also observe in our experiments that the FID remains unchanged with a large $\\beta$. As for $\\alpha$, it is an empirical hyperparameter to oppose the fake samples and accelerate the FID decrease, which we have already ablated in Figure 5(c).\n\n> The ablation study for Multi-Round Refinement is missing. Could the authors provide experiments where the reference model remains fixed throughout the entire fine-tuning process? Meanwhile, could the authors provide ablation studies on the number of iterations per round? Is a smaller or larger number of iterations more effective?\n\nWe would like to clarify that (1) we already provided the ablation for multi-round refinement in Figure 5(a) and (2) Figure 5(b)(c) already provides ablation studies on the number of iterations per round, as we continuously monitor the FID score during the whole finetuning process in a single round (with the reference model remaining fixed), demonstrating performance improvement at different numbers of iterations.\n\n> Could the authors provide experiments on fine-tuning the EDM, EDM2, and VAR using a standard fine-tuning method to assess how much the proposed approach improves over it?\n\nGood question! Reviewer GoSA also mentioned this question, but he/she also says \"I have few reasonable doubts as most baselines are already extremely optimized\". We first show our empirical results and explain the fundamental reason behind it.\n\nThe \"standard fine-tuning method\" of likelihood-based models is to continue training with MLE, which can be seen as extending the pretraining period. On unconditional CIFAR-10 with the EDM base model:\n\n|Iterations|0|500|750|1000|1250|1500|\n|:--------|:-------:|:-------:|:------:|:------:|:-------:|:-------:|\n|MLE Continued|1.97|1.961|1.969|1.972|1.988|2.011|\n|DDO ($\\alpha=6.0,\\beta=0.05$)|1.97|1.896|1.852|1.809|1.733|1.769|\n\nBy continuing training with the MLE objective, the FID just fluctuates within a small range and never advances the performance like DDO. This phenomenon can be expected, not only because the baselines are already extremely optimized, but also for the traditional MLE's inherent nature. As stressed in the paper's abstract and introduction, MLE is mode-covering and will be penalized seriously if the model underestimates the likelihood of any training samples. This limits the MLE objective's upper bound performance under limited model capacity. Therefore, we can never expect significant generation quality improvement by continuing training with MLE, while DDO is a fundamental different way that breaks the limitation.\n\n> It would be useful to show whether the proposed method also benefits training the generative models from scratch.\n\nWe did not expect the method to training generative models from scratch. As stated in Section 3.1 and 3.2, DDO replies on good initializations for stable optimization. Intuitively (e.g., as seen in Figure 1), DDO mainly refines a well-trained density and concentrate it on main modes. Moreover, the MLE objective has proven to be a stable and scalable approach for pretraining of likelihood-based generative models. Therefore, the best practice is pretraining with MLE and refining with DDO.\n\nAdditionally, we would like to provide some new results on higher-resolution datasets. After submission, we further use DDO to finetune EDM2-L on ImageNet 512x512, successfully advancing the FID from 2.11 to **1.36**, without any guidance. Some visualizations are provided in the [anonymous link](https://anonymous.4open.science/r/DDO/). We find DDO significantly enhancing the image quality/details without affecting the overall content/diversity. We will release all model checkpoints and code that reproduce the reported results upon acceptance.\n\nThank you again for your consideration, and for giving positive rating. We hope that our response can resolve your concerns, and we are happy to answer further questions.", "claims_and_evidence": "Please see the weaknesses.", "methods_and_evaluation_criteria": "Yes, that makes sense.", "theoretical_claims": "The proofs are logical.", "experimental_designs_or_analyses": "The experiments are well designed.", "relation_to_broader_scientific_literature": "The paper offers a new perspective on parameterizing a likelihood-based visual generative model as a discriminator in the GAN framework and may provide insights for fine-tuning likelihood-based generative models beyond vision.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**:\n\n1. The idea of parameterizing a likelihood-based generative model as a discriminator is novel.\n\n2. The authors provide a solid theoretical analysis and derivation of the method's design.\n\n3. The authors validate the proposed method on existing pre-trained diffusion models, demonstrating its effectiveness.\n\n4. The paper is well-written and easy to follow.\n\n**Weaknesses**:\n\n1. The introduced $\\alpha$ and $\\beta$ require additional effort for hyperparameter search.\n\n2.  The motivation for introducing $\\alpha$ and $\\beta$ to address gradient-vanishing/numerical issues is not well-supported. It would be helpful if the authors could provide experimental evidence to show the problem and demonstrate how $\\alpha$ and $\\beta$ help mitigate these issues.\n\n3. The ablation study for Multi-Round Refinement is missing. Could the authors provide experiments where the reference model remains fixed throughout the entire fine-tuning process? Meanwhile, could the authors provide ablation studies on the number of iterations per round? Is a smaller or larger number of iterations more effective?\n\n4. Could the authors provide experiments on fine-tuning the EDM, EDM2, and VAR using a standard fine-tuning method to assess how much the proposed approach improves over it?\n\n5. It would be useful to show whether the proposed method also benefits training the generative models from scratch.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a fine-tuning method for generative models where a discriminator is parameterized through the likelihood ratio of the target model and a reference model. The reviewers praised the clarity of writing, novelty of the method, and empirical results. This paper is a clear strong accept.
Decision: Accept (spotlight poster)