Noise Conditional Variational Score Distillation
Authors: Xinyu Peng
Ziyang Zheng
Yaoming Wang
Han Li
Nuowen Kan
Wenrui Dai
Chenglin Li
Junni Zou
Hongkai Xiong
Venue: ICML 2025 poster (2025)
Keywords: generative models
diffusion distillation
inverse problems
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a scalable training method to distill a pretrained diffusion models into generative denoisers.

Abstract: We propose Noise Conditional Variational Score Distillation (NCVSD), a novel method for distilling pretrained diffusion models into generative denoisers. We achieve this by revealing that the unconditional score function implicitly characterizes the score function of denoising posterior distributions. By integrating this insight into the Variational Score Distillation (VSD) framework, we enable scalable learning of generative denoisers capable of approximating samples from the denoising posterior distribution across a wide range of noise levels. The proposed generative denoisers exhibit desirable properties that allow fast generation while preserve the benefit of iterative refinement: (1) fast one-step generation through sampling from pure Gaussian noise at high noise levels; (2) improved sample quality by scaling the test-time compute with multi-step sampling; and (3) zero-shot probabilistic inference for flexible and controllable sampling. 
We evaluate NCVSD through extensive experiments, including class-conditional image generation and inverse problem solving. By scaling the test-time compute, our method outperforms teacher diffusion models and is on par with consistency models of larger sizes. Additionally, with significantly fewer NFEs than diffusion-based methods, we achieve record-breaking LPIPS on inverse problems.

Noise Conditional Variational Score Distillation
Xinyu Peng" Ziyang Zheng²
Yaoming Wang 3 Han Li Nuowen Kan Wenrui Dai Chenglin Li Junni Zou' Hongkai Xiong?
(xypeng9903, zhangziyang, qi gingshi 9974, kannw_1230, dai daiwenrui, 1c11985, zcujunni, Jes edu. en; wangyacming0 3emeituan com
Abstract
1. Introduction
We propose Noise Conditional Variational Score Distillation (NCVSD), a novel method for dis- tilling pretrained diffusion models into genera- tive denoisers. We achieve this by revealing that the unconditional score function implicitly char- acterizes the score function of denoising poste- rior distributions. By integrating this insight into the Variational Score Distillation (VSD) frame- work, we enable scalable learning of generative denoisers capable of approximating samples from the denoising posterior distribution across a wide range of noise levels. The proposed generative denoisers exhibit desirable properties that allow fast generation while preserve the benefit of it- erative refinement: (1) fast one-step generation through sampling from pure Gaussian noise at high noise levels; (2) improved sample quality by scaling the test-time compute with multi-step sampling: and (3) pero-shot probabilistic infer- ence for flexible and controllable sampling. We evaluate NCVSD through extensive experiments, including class-conditional image generation and inverse problem solving. By scaling the test-time compute, our method outperforms teacher diffu- sion models and is on par with consistency mod- els of larger sizes. Additionally, with significantly fewer NFEs than diffusion-based methods, we achieve record-breakinn LPIPS inverse prob- lems. The source code is available at https: / //github. com/ http///yppgggnn////////
Diffusion models (Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021b; Karras al., 2022), also known as score-based generative models, have emerged as dom- inant paradigm for high-dimensional data generation. A defining characteristic lies in their inherently iterative sam- pling mechanism, offering unprecedented flexibility for inference-time control. This characteristic facilitates sev- eral advantages, including the flexible trade-off between computational complexity and sample quality, as well as en- abling zero-shot controllable sampling across wide range of downstream tasks (Chung et al., 2023; Yu et al., 2023; Song et al., 2023b; Uehara et al., 2025). Nevertheless, the it- erative process suffers from significant limitations regarding sampling efficiency and real-time applications. To address this issue, recent years have witnessed a surge of interest in distilling teacher diffusion models into fast generators (Luo et al., 2023; Yin et al., 2024b; Zhou al., 2024; Sauer et al., 2024). However, these generators support only one or few sampling steps, thereby discarding the capacity for iterative refinement of generated samples, which is especially crucial for imperfectly trained generators and zero-shot controllable sampling. This raises fundamental question: Can we de- velop a generative model that enables fast generation while preserving the desirable attributes of iterative refinement?
To tackle this challenge, we develop generative denoisers class of models specifically designed to generate approx- imate samples from the denoising posterior distributions. Well-trained generative denoisers demonstrate a unique com- bination of advantages that effectively balance sampling ef- ficiency and flexibility. Specifically, i) they enable efficient one-step unconditional generation by sampling from pure Gaussian noise; ii) they inherently support multi-step sam- pling with improved sample quality, offering paradigm shift from traditional training- time computation to flexible test-time computation (OpenAI, 2024; Geng et al., 2025); iii) they facilitate asymptotic exact probabilistic inference through seamless integration with the Split Gibbs Sampler framework (Vono al., 2019), illustrated in Figure 1.
'Department of Computer Science and Engineering, Shang- hai Jiao Tong University, Shanghai, China 'Department of Elec- tronic Engineering. Shanghai Jiao Tong University, Shanghai, China "Meituan Inc. China. Correspondence <zhennziiyng sjtu.edu.cn>> Yaoming Wang <wangyaom- ing03@meituan meituan.com com>, Wenrui Dai <daiwenrui @ @sw...edu.cn>>........
Ziyang Zheng
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To train generative denoisers, we first establish funda-

---

Noise Conditional Variational Score Distillation
Noley Image
Sample 0
Sample 1
Sample 2
Clasy-Conditional Generation
Motion deburring
Super resolution
Inpainting
(a) Generative Denoising
(b) Few-Step Generation
(c) Probabilistic Inference
Figure 1. The proposed generative denoisers, distilled from pretrained diffusion models, support a variety of tasks. In (a), the generative denoiser demonstrates the ability generate diverse samples that approximate the denoising posterior distribution at arbitrary noise levels. In (b), we present the 4-step class-conditional generation results on the ImageNet-51225 dataset. In (c), we showcase the plug-and-play prohabilistic inference capability of the generative denoiser.
mental theoretical connection by demonstrating that the unconditional score function inherently characterizes the score function of denoising posterior distributions. This insight enables us to extend conventional Variational Score Distillation (VSD) ( (Wang et al., 2024) by explicitly condi- tioning on noisy data, emerging as novel diffusion distilla- tion approach, namely Noise Conditional VSD (NCVSD).
for solving inverse problems using our generative denoisers, achieves competitive results compared to state-of- the-art diffusion-based inverse problem solvers (Chung et al., 2023; Wu et al., 2024; Zhang et al., 2024), while reducing the re- quired number function evaluations (NFE) by an order of magnitude. In particular, we achieve record- breaking LPIPS performance on a range of linear inverse problems. For chal- lenging nonlinear inverse problems, current diffusion-based solvers typically require 1k NFE to produce reasonable re- sults, whereas the proposed method only requires 50 NFE.
Furthermore, introduce an auxiliary adversarial loss to facilitate learning from real data, thereby overcoming the performance upper bound imposed by the teacher diffusion models. Finally, we meticulously engineer the parameteriza- tion of generative denoisers, which not only enables efficient knowledge transfer from the teacher diffusion model but also leverages the inductive bias of preconditioning, as elu- cidated by Karras et al. (2022).
2. Backgrounds
2.1. Diffusion Models
Diffusion models aim to generate samples that approximate the target data distribution (xax(Xx). To achieve this, they employ family of Gaussian perturbation kernels defined as (xx, xx) NN(x, [ If which gradually perturb the original distribution into sequence of noisy distributions (xx,) = E (x.,) [g(x, xo)] The noise scale a, is designed to monotonically increase with the diffusion time step t. Through this progressive perturbation process, the final dis- tribution (xx)) at terminal time T becomes sufficiently close to the isotropic Gaussian distribution N/00, I). In this paper, adopt o, t following the approach in (Kar- ras et al., 2022) for simplicity.
To demonstrate the effectiveness of NCVSD, evaluate the distilled generative denoisers via extensive experiments, including class conditionn image generation on ImageNet- 64x64 and ImageNet-51 5125112 datasets, as well as solving a wide range of linear and nonlinear inverse problems. In the class-conditional image generation task, we observed that generative denoisers beat consistency models (CM) (Song al., 2023c) with state-of-the-ar training method sCM (Lu & Song, 2025) of similar sizes. In addition, we observed that by scaling the test-time compute, generative denoisers can achieve performance comparable to sCM of larger sizes. For example, on ImageNet-5112xx x x512 dataset, the 4-step FID of generative denoiser (1.73), distilled from EDM2-L. (Kar- ras al., 2024), surpasses the FID of sCM (1.88) distilled from EDM2- XXL. In inverse problem solving tasks, we observed that PnP-GD, the proposed plug-and-play method
The main idea of diffusion models is to find a way to sample from (xx:) with annealing decreasing noise levels, such that at the end of the sampling process the distribution of the samples, g(x,..). will be close to the original data distri- bution Qdana (xp). One notable example is the Probability
2

---

Noise Conditional Variational Score Distillation
Flow Ordinary Differential Equation (PF-ODE) (Song et al., 2021b; Karras al., 2022), which takes the following form:
great interest. Diffusion methods have recently been widely adopted for posterior sampling, but existing approaches often involve trade-off's between flexibility, posterior exact- ness, and computational efficiency. Supervised methods (Sa- haria et al., 2022; Rombach et al., 2022) lack flexibility, as they require retraining for each specific task. Zero-shot methods offer greater flexibility by approximating the con- ditional score from a pretrained unconditional one, but they introduce irreducible errors by approximating the denoising posterior with Dirac (Chung et al., 2023) or Gaussian dis- tributions (Song et al., 2023b; Peng et al., 2024). Recently, asymptotically exact methods, such as PnP-DM (Wu et al., 2024), ensure exact posterior sampling in the asymptotic limit but are computationally expensive, relying on reverse diffusion simulations for sampling from denoising posterior distributions that requires a large number of NFEs.
dx.= = -tVx x log (xx))dt, >
~ q(xT),
(1)
where the unknown gradient of the log density V log (xx)), also known as the score function, can be linked to the condi- tional expectation through Tweedie's 's formula (Efron, 2011);
Vx, log ((x,) t-² (E[[x]/xa) - x().
(2)
Therefore, the score function can be estimated by train- ing a neural network (x,,)) x [[xo|xe] with in- put X and and parameters 6. referred to as the score model, through simple regression objective as ming Er, (xn)))))((( (x)) ||x (o((x,t)ll
2.2. Variational Score Distillation
In this paper, we propose a posterior sampling method that strikes a balance between flexibility, posterior exactness, and computational efficiency. To achieve this, we first es- tablish a fundamental theoretical connection, demonstrating that the conditional score function conditioned on noisy data has a tractable closed-form solution that can be precisely expressed using the unconditional score function. Building this result, we distill one-step generative denoiser ca- pable of efficiently sampling from the denoising posterior distribution across a wide range of noise levels. Leverag- ing this efficient generative denoiser, we address the slow reverse diffusion simulations in PnP-DM through a simple plug-in in replacement of the generative denoiser. This results in an efficient and flexible posterior sampling method with asymptotically exact guarantees.
To address the slow inference speed of diffusion mod- els, recent studies have explored methods for distilling diffusion models into GAN-like generators Xy = Go(z), z ~ (O,,,), Notably, Variational Score Distillation (VSD), initially developed for 3D generation in Prolific- Dreamer (Wang et al., 2024), has been successfully adapted to accelerate image generation through diffusion model dis- tillation (Luo et al., 2023; Yin et al., 2024b;a; Nguyen & Tran, 2024). The objective of VSD is minimize the re- versed KL divergence of the diffused model distribution and the diffused data distribution':
min Livad (0) (vDD
where (xx) : Ex Ge (q xo)] and (xx.) Eggn (xx)) (xx [xo)] are defined by adding Gaussian noise (0, (2T) on the generated data X = (z) and real data XD ~ Pdaus respectively. The fundamental result of VSD is that the gradient of the VSD objective can linked to the
3. Noise Conditional Variational Score Distillation
3.1. Conditioning VSD on Noisy Data
score functions (Wang et al., 2024; Luo et al., 2023):
The core objective of NCVSD is to learn conditional gen- erative model 1u0 (xol that generates samples approxi- mating the denoising posterior distribution g(xply=) xx across a wide range of noise levels a > O, where yo denotes the noisy data generated by adding Gaussian noise on Xp. To this end, we propose to solve the following reverse KL minimization problem:
V Ve = (e.
(V x: log (xx,) - Vx, log )) (Go(z)
(4)
88
gdura (Xo) (x
where Vx, log x(x)) can be estimated using a pretrained score model, and Vx, log (xx) can be estimated via auxiliary score model for the generated data X9 = (z)),, with the training of the score model being conducted online with the training of the generator (z)z).
min E. Egty.) [Drr (Xo ly (oooll))lllxxool))))) 0
(5)
where a is drawn from a predefined distribution of noise levels, g(y=) E (xo) [N Xu, o2]] is the marginal distribution of y.. and na (xolya) is defined using implicit distribution induced from conditional one-step generator as X = Gelyo, 2), I),
2.3. Diffusion- Posterior Sampling
In practical applications, sampling from posterior distribu- tion y(xoly) & Gdata (daan(Xo) (x ((yg))))))))))))))))))(())) given conditions y in of
Without loss of generality, omit the weighting functions for KI. divergences across different f, as they can he transformed into the density of f or considered as the importance weight.
However, directly optimizing Equation (5) is challenging, as the high-density regions of g( yo) may be extremely
3

---

Noise Conditional Variational Score Distillation
sparse in high-dimensional space (Song & Ermon, 2019; Wang al., 2024). Inspired by VSD, we diffuse the original distributions pa (x(x)ly>) and g(xo yo) using Gaussian ker- nels (xx) =
Estimate Model Score: V log pe (x,ly))) can be estimated
by training a conditional score model Da (Xe, Lo, o) using common diffusion objective:
(xxx Xx. t))), to construct an alternative optimization problem. Specifically, we define
min E (xt ly (g(())/())))))
[[|xo- - [[xxo-Doxx,n No [)]]]] (13)
po (x (xo(x)lye) (((x((xo)] g(xx1Vo)= == Bx(xoly=)) [g(xx|x))]
(6)
It is worth mentioning that V log po (x cannot be esti- mated in the same manner as Vx, log (xxry)>) The joint distribution of (y-, Xo) is ((o=))o from the assumption in Proposition 1. As a result, the distri- bution of yo given Xy is no longer Gaussian.
(7)
which differs
We then minimize the reverse KL divergence between (y)) and ((x+)ya) for all t:
min Laova (0) == E. (poxxxll
(8)
9
3.2. Auxiliary Adversarial Loss
Similar to Equation (4) for VSD, the gradient of the NCVSD loss relates to conditional score functions:
The effectiveness of the NCVSD gradient in Equation (9) critically depends on accurate estimation of both the data score and the model score. However, in practice, the data score provided by the pre-trained teacher score model is often imperfect, and accurately estimating the model score is challenging due to the evolving model parameter 0 dur- ing training. To address these issues, prior works have shown that incorporating an auxiliary adversarial loss can improve performance by leveraging real data in addition to the teacher model (Kim et al., 2024; Yin et al., 2024b; Zhou et al., 2025; Sauer al., 2024). Inspired from these approaches, we propose minimizing the Jensen-Shannon di- vergence (JSD) alongside the KL divergence minimization in Equation (8):
(0 (V log po (x - Vxi 1x, ((x y)))
20
We denote Vx, log Lo (Yo) and Vx, log (x, (No) ) as the model score and data score, where the former represents the score function of the generative denoiser's 's output distribu- tion and the latter corresponds to the data distribution, The derivation of Equation (9) is provided in Appendix A.1.
Estimate Data Score: The pretrained score model, which solely estimates Vx, log ((x),, cannot be direclty utilized to predict Vx, log ((x))ya). We address this by show- ing that Vx, log (xx lya) has a tractable closed-form so- lution, which can be exactly represented by Vx, log q(xx), as demonstrated in Proposition 1.
min Lad (0) == Er
(14)
To optimize Equation (14), we introduce classification neural network. C.., referred as the discriminator, to con- vert the JSD minimization (Equation (14)) into tractable adversaria optimization problem (Goodfellow et al., 2014):
Proposition 1. Suppose (Xo. Yo, xx) follow the joint dis- tribution (dmu( Xx. I'I) (xx, Xx. ( For any p > O, define the denoising posterior of xo with noise level p as ((x0 vp) x Idono (xo) (y //x0,,00000. xxo,p= I21). We obtain that
min max EL,.,y- Bx (x) flog (xt, t, t.yo,
8
q(xo|xt, No) (xo Naol
(10)
+ (oooxolyo xx) log(1 - (x,, (15)
Vx, log (xx(\xx))) = 2 [xo [x xi);
(11)
where Yog 3r, 0-141-2 and aeh (a-2 + 2-
Although sampling from (xxly>>) is intractable, we can ap- ply the bidirectional Monte Carlo method (Grosse et al., 2016; Zhao et al., 2024) by leveraging the fact that ))) = Adam (xo) (xx))((rr Xn, o²I). This enables us to compute an unbiased estimate of Equation (15) using samples from the training data distribution qdma (Xo)-
the noise level of q(xolyoa) which is referred to as the effective noise level.
Please refer to Appendix A.2 for the proof. Note that the unconditonal score function can be linked with IE [xoly..dl according to Equation (2). By Proposition 1, estimating Vx, log yx) can be reduced to leveraging a pretrained unconditional score model Doly,, p) se E[xxolyp). which is readily available in many settings. This leads to our proposed estimator for V log gxxely.) using Do:
Accordingly, we employ a weighted sum of Equations (8) and (15) as the training loss to implement NCVSD. To ef- fectively balance the contributions of Lnevst and Luiv. we employ uncertainty weighting (Kendall et al., 2018; Karras et al., 2024; Lu & Song. 2025) to transform the optimiza- tion of Cnevad into a form that resembles a Gaussian log- likelihood. Additionally, we scale Ludy by the number of
1-2 (Do
+4 2x, (0- + -)-)) x.) (12) x-2-1-
4

---

Noise Conditional Variational Score Distillation
the data dimensions, ensuring that both losses are approx- imately on the same scale. For further details and pseudo code or single iteration of the NCVSD training iteration, please refer to Appendix B.2 and Algorithm 3.
Algorithm Probablistic inference with PnP-GD
Input: generative denoiser uo(xolye)) energy function }8(), noise annealing schedule ON 01 O, u ~ (0, II
for i = N..... do
3.3. Multi-Step Sampling
x ~ (Xo Yo; = =') or multi-step sampling
The proposed generative denoiser also support multi-step sampling, allowing a trade-off between sample quality and inference cost. To this end, we introduce latent vari- ables X1:N² following the Denoising Diffusion Implicit Model (DDIM) (Song et al., 2021a) to extend g(xo Ya)) into q(Xo.N Yo) as g(Xo:Nly == g(xulyo (oXuly))q(x1.)) Xo). and q(X1:N (xo) == q(xN|xx) IT q(;;-1 X,, Xo). Ac- cordingly, the reverse process defined as p(Xo:N) = p(xN) x (xx). To enable a gradual approxima- tion of ((Xoly=) using an annealing decreasing noise sched- ule {or]/== as i decreases during the multi-step sampling, we construct the forward and reverse processes such that the marginal distributions p(x;) equal to (xx,ly,) for all as formalized in Proposition 2.
o < Jem then
Xo p Xn (1 n1) xb
else
Xo x end if
u ~ exp (- (-gE(n²²1)
1 | x51
- 20
end for Output: X0
3.4. Parameterization
In practice, we carefully design the parameterization of the required models, including the one-step generator for generative denoiser Go o, z), the score model to, 0) in Equation (13), and the discriminator (x, in Equation (15). The configurations provided in Appendix B.I. The careful design not only effec- tively reuses knowledge from the teacher diffusion model but also leverages the inductive bias of preconditioning from (Karras et al., 2022), which significantly accelerates training while maintaining performance.
Proposition 2. By constructing the following distributions:
((xx) xo) = N I)),, q(x;-1|xx Xx, xo) = NNxxx Ç. xx-X0 of
p(xN)= = E (((xoly))(((xxxx)))
p(x;-1 x.) = Ex(xo (q X,,
4. Plug-and-Plla Probabilistic Inference with Generative Denoiser
we have that any Ç E (0,1), q(x; (xo) = Nxx (xx]ee Xxo 1) and p(x;)= (yx: for i = 1,2, ..., N. In addition, the following equality holds:
This section focuses on sampling from an unnormalized target distribution defined by
(o. = =q (xo Yog = r oo 3xi
(16)
240,"
1 T (xx)) X (Xo) exp (-1
(17)
and the effective noise level Deff (0- + ox 2) b
where E(xo) is a given energy function, and B > 0 con- trols the influence of the energy. Efficient sampling from T mxxg) is crucial for various machine learning tasks, includ- ing classical Bayesian inference where [xxx) represents the negative log-likelihood, determining optimal policies in offline reinforcement learning (Peters et al., 2010; Lu et al., 2023), modeling desired sample distributions aligned with human preferences (Korbak et al., 2022; Uehara et al., 2025), among others. To address this, we propose a plug-and-play (PnP) probabilistic inference method, dubbed PnP-GD, us- ing well-trained generative denoiser (xoxx)l>>) that can achieve asymptotic exact sampling from π (xxo) under the ideal scenario (x(xx))>>) {(xoly>), as detailed below.
Please refer to Appendix A.3 for the proof. According to Proposition 2, multi-step sampling for the generative denoiser can be achieved by sampling recursively from p(x; as iteratively perform the following two steps:
1. sampling Xo from pe (Xolyom) according to Equa- tion (16);
2. sampling x from ((x (|x,, Xn);
The pseudocode for multi-step sampling is presented in Algorithm 2.
"Note that {x. J. and (Xt differ. The discrete- timer latent variable x, is introduced exclusively during the sampling phase, whereas x, denotes continuous-time, noise-corupted data variable that exists solely during the training process. Besides, the noise level corresponding to x; is o, while that of x; is t.
Split Gibbs Sampler: Motivated by asymptotic exact pos- terior sampling for diffusion models (Wu et al., 2024; Xu Chi, 2024), we leverage Split Gibbs Sampler (SGS) (Vono
5

---

Noise Conditional Variational Score Distillation
et al., 2019; Coeurdoux et al., 2024) to achieve plug-and- play probabilistic inference with po Xx ya). Specifically, we first introduce an auxiliary random variable u and de- fine the joint distribution of Xo and u as (Xo, u) == exp(log (dma(xo) 3 [(() 2- ||u xol where o gov- erns the strength of the penalty of the difference between Xo and u. Notably, the marginal distribution of TTo (Xo: u), To (xx), converges to our target distribution .t ((x)) as o - in terms of the total variation distance (Vono et al., 2019), resulting in asymptotically exact sampling from π (Xo)..
of PnP-GD produces generated samples with excessively sharp details. We hypothesize that this issue arises from the amplification of fine-grained details during the final steps of the PnP-GD process. To mitigate this problem, we propose averaging over multiple samples in the last MCMC chain of the PnP-GD process using an exponential moving average (EMA). This approach approximately computes an average of a mode of the posterior distribution I T(x)). bringing the result closer to the posterior mean and thus reducing the sharpness of the generated samples.
To sample from To (Xo. u). SGS alternately implements sampling from To (Xp) u) and To (u|x)) with an annealing
The pseudocode for the PnP-GD procedure is provided in Al- gorithm 1. Additionally, the multi-step sampling approach discussed in Section 3.3 can also be applied to approximate the prior step.
decreasing value of o, as follows:
The (xo/u) x exp (log Qdataa (alna(xo)) (xo)
- ju xol (18)
-0
Iu- TT(u Xo) x cxp (-}E(u) 20t
Xp
(19)
5. Experiments
where Equations (18) and (19) are referred to as prior step and likelihood step, respectively. As the prior step is equiv- alent to sampling from (xxoly, = u). it can be approxi- mated by a well-trained generative denoiser with input u and noise level o, i.c., p u). The likelihood step is generally straightforward to sample, for instance, by implementing the Unudjusted Langevin Algorithm (ULA) (Welling& & Teh, 2011), provided that & is differentiable.
In this section, we evaluate proposed method by testing the distilled generative denoisers on i) class-conditional image generation on ImageNet-64x66 64 and ImageNet-512x35 datasets (Deng et al., 2009), and ii) plug-and-play in- verse problem solving with PnP-GG on FFHQ 256 dataset (Karras et al., 2019). In Appendix B, we provide fur- ther details regarding the NCVSD training, class-conditional generation and inverse problem solving using PnP-GD. We include additional experimental results in Appendix C.
Note that PnP-DM and PnP-DM are both built upon the foundation of SGS. The primary distinction lies in how the prior step is approximated. In PnP-DM, simulating the reverse diffusion process is required, which not only computationally inefficient but also prone to irreducible discretization errors. In contrast, our approach significantly improves computational efficiency by requiring only one or a few NFEs for the prior step, while being free from any errors beyond those introduced by imperfect model training.
5.1. Image Generation
Setup: To test the generation performance of the proposed NCVSD, we follow the settings of EDM2 (Karras et al.,
2024), ECM (Geng et al., 2025), and sCM (Lu & Song,
2025), to train and scale different sizes of models on Ima- geNet 64x64 and ImageNet 512x512 datasets (Deng et al., 2009). Specifically, we distill models of different sizes from EDM2 teachers. including NCVSD-S, NCVSD-M NCVSD- L from EDM2-S, EDM2-M and EDM2-L, repectively. We standardize to Fréchet Inception Distance (FID) (Heusel et al., 2017) to measure the generation performance to com- pare different methods, following EDM2.
Theoretically, when the potential function in ULA is L-gradient Lipschitz, ULA provides performance guarantees if its step size is smaller than constant proportioned to -- (Durmus et al., 2019; Balasubramanian et al., 2022). For the likelihood step in Equation (19), it can be shown that the potential function, +E(+)+ 2 || - -xol 1| is (B-1L+ L o "g-gradient Lipschitz, provided that E-) is L-gradient Lipschitz. Based on this, we propose an adaptive step size " for ULA that adapts to the current noise level a of PnP-GD as % == C1. FL-(8-1C) + - - where Ci and C2 are hyperparameters. As can be seen, the step size decreases monotonically as a is annealed towards zero, which aligns with intuition. Note that Zhang et al. (2024) also suggested reducing the step size as o decreases, and empirical results indicate the effectiveness of this approach. In the experiments, C₁ is fixed to 1, while C2 is tuned for different probabilistic inference tasks.
ULA with Adaptive Step Size:
Baselines: We select consistency models (CM) (Song et al., 2023c) and its subsequent improvements (Geng et al., 2025; Lu & Song, 2025) as our primary comparisons. This is because models trained using NCVSD and CM exhibit very similar behaviors. Both approaches generate clean data from its noisy version of arbitrary noise levels, support multi-step generation to balance sample quality and sampling cost, and allow zero-shot controllable sampling. For instance, the original version of CM (Song et al., 2023c) already supports zero-shot image editing, and recently Tian et al. (2024) and Xu et al. (2024) have developed methods for zero-shot controllable sampling with consistency models to address a broader range of inverse problems.
EMA Samples: We observe that the vanilla implementation
6

---

Noise Conditional Variational Score Distillation
Table Sample quality on class-conditional ImageNet-64 and ImageNat-512x512, We report the number of function evaluations (NFE), Fréchet Inception Distance (FID), and the number of training iterations (#lter).
Class-Conditional ImageNet-64x00 64
Class-Conditional ImageNet--112x51
Method
NFEL FID
#Iter
Method
NFEJ
FID
#Iter]
Teacher Diffusion Models
Teacher Diffusion Models
EDM2-S (Karras al., 2024)
63
1.58
1024k
EDM2-S (Karras et al., 2024)
63x2
2.23
2048k
EDM2-M (Karras al., 2024)
63
1.43
2048k
EDM2-M (Karras et al., 2024)
63x2
2.01
2048k
EDM2-L (Karras al., 2024)
63
1.33
1024k
EDM2-L (Karras et al., 2024)
63x2
1.88
1792k
EDM2-XL (Karras et al., 2024)
63
1.33
640k
EDM2-XL (Karras et al., 2024)
63x2
1.85
1280k
EDM2-XXL (Karras et al., 2024)
63x2
1.81
896k
Consistency Models
Consistency Models
CD (Song et al., 2023c)
2
4.70
600k
ECM-S (Geng et al., 2025)
1
5.51
100k
sCD-S (Lu & Song, 2025)
3.07
200k
2
3.18
100k
2
2.50
200k
ECM-M (Geng et al., 2025)
1
3.67
100k
sCD-M (Lu Song, 2025)
1
2.75
200k
2
2.35
100k
2
2.26
200k
ECM-L (Geng et al., 2025)
1
3.55
100k
sCD-L (Lu & Song, 2025)
1
2.55
200k
2
2.14
100k
2
2.04
200k
ECM-XL (Geng et al., 2025)
1
3.35
100k
sCD-XL (Lu & Song, 2025)
1
2.40
200k
2
1.96
100k
2
1.93
200k
sCD-XL (Lu & Song. 2025)
1
2.44
400k
sCD-XXL (Lu & Song, 2025)
2.28
200k
2
1.66
400k
2
1.88
200k
Noise Conditional Variational Score Distillation
Noise Conditional Variational Score Distillation
NCVSD-S (Proposed)
1
3.13
32kx3
NCVSD-S (Proposed)
2.95
32kx3
2
2.66
32kx3
2 2.60
32kx3
4
2.14
32kx3
4
2.00
32kx3
NCVSD-M (Proposed)
1
3.04
32kx3
NCVSD-M (Proposed)
1
2.85
32kx3
2
2.47
32kx3
2
2.08
32kx3
4
1.92
32kx3
4
1.92
32kx3
NCVSD-L (Proposed)
1
2.96
32kx3
NCVSD-L (Proposed)
2.56
32kx3
2
2.35
32kx3
2
2.03
32kx3
4
1.53
32kx3
1.76
32kx3
Table 1 compares various methods on class- conditional image generation using Fréchet Inception Dis- tance (FID), Number of Function Evaluations (NFH), and training iterations. As can be seen, our methods outperform CMs of comparable sizes while requiring significantly fewer training iterations x32kx3 versus 200k, when accounting for score model and discriminator iterations for fair compari- son), demonstrating computational efficiency. Our methods also exhibit predictable performance gains with increased model size; the FID of NCVSD consistently decreases as model sizes grow, indicating training time scalability similar that of teacher EDM2 models and CMs. Furthermore, by increasing test-time computation (i.e., NFE), our methods match exceed the performance of larger diffusion models or CMs. For instance, on the ImageNet-512x551 dataset, the FID of 4-step NCVSD-L (1.76) surpasses that of 2-step SCD-XXL (1.88) and the diffusion model EDM2-
Results:
XXL (1.81). Similar results are observed on the ImageNet- 64 x64 dataset, where the FID of 4-step NCVSD-L (1.53) is better than the FID of 2-step sCD-XL (1.66).
5.2. Inverse Problem Solving
Sctup: To evaluate the zero-shot probabilistic inference ability, we test PnP-GD (Section 4) on several inverse prob- lems on the FFHQ 256 x 256 dataset (Karras et al., 2019). To obtain generative denoiser, we first pretrain a XS size EDM2 model FFHQ 256 256 and then distill using NCVSD (see Appendix B.3 for details). The performance is evaluated using Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 201 8) for perceptual quality. and Peak Signal-to-Noie Ratio (PSNR) for distortion quality.
We conduct comparisons across a variety noisy linear and nonlinear inverse problems. For linear inverse problems, we

---

Noise Conditional Variational Score Distillation
Table Quantitative results on noisy inverse problems. The results are averaged over 100 images. We use hold and underline when the proposed method (PnP-GD) achieves the best and the second best, respectively.
Inpaint (box)
Deblur (Gaussian) Deblur (motion) Super resolution Phase retrieval
Method
NFEL
LPIPS., PSNR1
LPIPS!
PSNR1
LPIPS PSNR1 LPIPS,, PSNR LPIPSJ PSNR1
DDRM (Kawar et al., 2022)
100
0.159
22.37
0.236
23.36
0.210
27.65
DPS (Chung et al., 2023)
1000
0.198
23.32
0.211
25.52
0.270
23.14
0.260
24.38
0.410 17.64
DiftPIR (Zhu et al., 2023)
100
0.186
25.02
0.236
27.36
0.255
26.57
0.260
26.64
IIGDM (Song et al., 2023a)
99
0.284
21.76
0.245
25.72
0.240
26.29
0.245
25.69
DWT-Var (Peng et al., 2024)
99
0.158
21.26
0.186
27.70
0.189
28.06
0.187
27.78
DAPS (Zhang et al., 2024)
1000
0.133
24.07
0.165
29.19
0.157
29.66
0.177
29.07
0.140
29.94
PnP-DM (Wu et al., 2024)
2483
0.191
27.81
0.183
28.23
0.190
27.77
0.364
22.39
PnP-GD (Proposed)
50
0.128
21.75
0.155
27.06
0.160
28.02
0.151
27.93
0.186
27.82
Measurement
Sample 0
Sample 1
Sample
Sample 3
guided diffusion models (IGDDM) (Song et al., 2023a), (5) diffusion posterior sampling (DPS) (Chung et al., 2023), diffusion models for plug-and-play image restoration (Diff- PIR) (Zhu et al., 2023), and denoising diffusion restoration models (DDRM) (Kawar et al., 2022).
The quantitative results for noisy inverse problems are presented in Table 2. Our method achieves the best or second best LPIPS across all inverse problems, demon- strating a superior perceptual quality compared to diffusion- based solvers. Notably, the state-of-the-ar diffusion- based method, DAPS, requires 1k NFE, whereas our method uses only 50 NFE. We observed that the PSNR performance of our method does not achieve the best performance as LPIPS, which can be attributed to the Distortion-perceplle off (Blau & Michaeli, 2018), indicating that our method tends to generate results that retain more high- frequency details rather than approximate the mean of all possible so- lutions. This approach typically leads to higher MSE (lower PSNR) but aligns more closely with perceptual quality met- rics such LPIPS (Chung et al., 2023; Zhang et al., 2024). In Figure 2, we show that PnP-GD can generate images with diversity as well as fined details for addressing ill ill-poseness in inverse problems. Additionally, our approach efficiently and stably handles challenging nonlinear phase retrieval problem. While current diffusion-based solvers typically re- quire over 1k NFE to achieve reasonable reconstructions for phase retrieval, our method attains competitive performance with just 50 NFE, representing roughly 20x acceleration.
Results:
trade-
Figure 2. Sample diversity for addressing ill-poseness. Top: box inpainting with 1 128x 128 mask. Bottom: super resolution from 16x downsampled images.
consider (1) box inpainting using center box mask of size 128x 128, (2) Gaussian deblurring with kernels sized 61 x 61 and a standard deviation of 3.0, and (3) motion deblurring with kernels sized 61 x 61 and std of 0.5. For nonlinear inverse problems, we consider (1) super-resolution from 4x- bicubic downscaled images and (2) the challenging phase retrieval problem with 4x oversampling. Following (Chung al., 2023; Zhang al., 2024), we report the best result from four independent samples.
6. Discussion
Our method integrates insights from multiple fields, refin- ing and advancing existing approaches. Conceptually, the key distinction between our method and recent generative poodels-particulatt diffusion models (DMs) and consis- tency models (CMs) (Song et al., 2023c) lies in how clean data is predicted from its noisy counterpart. Our method
We select the following baselines: (1) plug-and- play diffusion models (PnP-DM) (Wu et al., 2024), (2) de- coupled annealing posterior sampling (DAPS) (Zhang et al., 2024), (3) guided diffusion models with learned wavelet variances (DWT-Var) (Peng et al., 2024), (4) pseudoinverse-
Baselines:
8

---

Noise Conditional Variational Score Distillation
directly models the full posterior distribution over clean data, whereas DMs learn the MMSE prediction, and CMs solve for the initial conditions of PF-ODEs. This design choice offers notable advantages. Compared to DMs, our ap- proach enables more efficient data generation. Meanwhile, it surpasses CMs in facilitating plug-and-pla probabilistic inference via SGS, offering asymptotically exact guarantee.
to ensure stable convergence. As result, we have not been able validate the effectiveness of NCVSD beyond its current scale due to computational constraints. We also note that the proposed noise conditional score estimator (Equation (12)) is not limited to the VSD framework; rather, it can be integrated with any score distillation method to distill generative denoisers from pretrained score models. Improving stability of distillation, developing pretraining techniques for generative denoisers, and applying the noise conditional score estimator to other score distillation meth- ods all represent promising avenues for future research.
Regarding multi-step sampling, our method requires signifi- cantly fewer NFEs to match the performance of DMs. This efficiency stems from modeling reverse transitions using multi-modal implicit distributions rather than single-modal Gaussian distributions. Our approach shares similarities with denoising diffusion GANs (Xiao et al., 2022) and mo- ment matching (Salimans et al., 2024). However, denoising diffusion GANs are trained on fixed, limited set of noise levels, restricting flexibility in inference-time control. Mo- ment matching, on the other hand, does not function as true denoising posterior sampler since its output remains deterministic with respect to the noisy input.
Impact Statement
This work contributes to synthetic data generation and data augmentation by enabling the efficient production of high-quality samples. It also addresses key challenge in diffusion-bar maased inverse problem solvers by providing an efficient and accurate posterior sampling method, facilitat- ing fast solutions across various scenarios. However, the potential misuse of synthetic data generation, such as creat- ing harmful or misleading content, raises ethical concerns. Responsible deployment and adherence ethical guidelines are essential ensure the technology is used for societal benefit.
Another closely related line of work involves the distilla- tion of diffusion models into amortized posterior samplers (Mammadov al., 2024; Lee al., 2025). Our approach offers significant advantages in inference-tine time flexibility, extending beyond the constraints of solving single inverse problem defined at training. These advantages include gen- erating high-quality unconditional samples and tackling a broader range of inverse problems. Furthermore, since prior terms are optimized using proxy objectives these works (see Equation (14) in (Mammadov et al., 2024) and Equa- tion (9) in (Lee et al., 2025)), they do not guarantee that the posterior distribution is the unique minimizer of their loss functions. Consequently, these methods cannot be directly applied to train generative denoisers that support marginal- preserving multi-step sampling (Section 3.3) and the SGS sampler (Section 4), as our method does.
A Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 62320106003, Grant U24A20251, Grant 62401357, Grant 62401366, Grant 62431017, Grant 62125109, Grant 02371288, Grant 62301 299, Grant 62120106007, and in part by the Program of Shanghai Science and Technology Innovation Project under Grant 24BC3200800. The computations in this paper were partially run on the Baiyulan Al for Science Platform supported by the Artificial Intelligence Institute at Shanghai Jiao Tong University.
7. Conclusion
In conclusion, we propose NCVSD, a novel method for distilling pretrained diffusion models into generative de- noisers. NCVSD is grounded in the theoretical insight that the unconditional score function implicitly characterizes the score function of denoising posterior distributions. Em- pirically, our method exhibits outstanding performance in both few-step image generation and zero-shot inverse prob- lem solving tasks, proving its potential as an efficient and flexible generative model.
References
Balasubramanian, K., Chewi, S., Erdogdu, M. A., Salim, A.. and Zhang, S. Towards a theory of non-log-coneave sam-
pling: First-order stationarity guarantees for Langevin Monte Carlo. In Proceedings Thirty Fifth Conference on Learning Theory, pp. 2896-2923, PMLR, 2022.
Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and Machine Learning. Springer, 2006.
The proposed method relies on pre-trained diffusion models to distill a generative denoiser, which lim- its the possibility of training denoiser from scratch. Ad- ditionally, achieving state-of-the-ard performance requires adversarial training, which involves careful manual tuning
Limitations:
Blau, Y. and Michaeli, T. The perception-distorion tradeoff. In Proceedings 2018 IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. IEEE, 2018
9

---

Noise Conditional Variational Score Distillation
Chung, H., Kim, J., Mccann, M. T., Klasky, M. L.,, and Ye, J. C. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: // //openreview net forum? ? d=OnD 9ZGAGTOk.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems 35, Curran Associates, Inc., 2022.
pp.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T. and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 24174-24188 IEEE, 2024.
Coeurdoux, F., Dobigeon, N., and Chainais, P. Plug-and- play split Gibbs sampler: Embedding deep generative priors in Bayesian inference. IEEE Transactions on Image Processing, 33:3496- 3507, 2024.
Deng, J., Dong, W., Socher, R., Li, I..J.,, Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In Proceedings of 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255. IEEE, 2009.
Kawar, B., Flad, M.,, Ermon, S., and Song, J. Denoising diffusion restoration models. In Advances in Neural Infor- mation Processing Systems 35, pp. 23593-23606. Curran Associates, 2022.
Durmus, A., Majewski, S., and Miasojedow, B. Analysis of Langevin Monte Carlo via convex optimization. Journal of Machine Learning Research, 20(73):1-46, 2019.
Kendall, A.,
Gal, Y., and Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7482- 7491. IEEE, 2018.
Efron, B. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):6602- 1614, 2011.
Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Ue- saka, T. He, Y., Mitsufuji, Y., and Ermon, S. Consistency trajectory models: Learning probability flow ODE trajec- tory of diffusion. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //opereeiew... net forum? watrruumiidyymiip dd==jjj8feDD
Geng, Z., Pokle, A., Luo, W., Lin, J., and Kolter, J. Z. Consistency models made easy. In The Thirleenth In- ternational Conference on Learning Representations, 2025. URL htttps://pperrwe id=xOVxo9dSIn
n forum?
Korbak, T., Perez, E., and Buckley, C. L. RL with KL penal- ties is better viewed as Bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083-1091. ACL, 2022.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014.
Langley, Crafting papers on machine learning. Proceed- ings of the 17th International Conference on Machine Learning, pp. 1207-1216. Morgan Kaufmann, 2000.
Grosse, R. B., Ancha, S., and Roy, D. M. Measuring the reliability of MCMC inference with bidirectional Monte Carlo. In Advances in Neural Information Processing Sys- tems 29, pp. 2451-2459. Curran Associates, Inc., 2016.
Lee, S., Park, D., Kong, I., and Kim, J. Diffusion prior- based amortized variational inference for noisy inverse problems. In Proceedings of 18th European Conference on Computer Vision, pp. 288-304. Springer, 2025.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge local Nash equilibrium. In Advances in Neural Information Processing Systems 30, pp. 6626- 6637. Curran Associates, Inc., 2017.
Lu, C. and Song, Y. Simplifying, stabilizing and scaling continuous- time consistency models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: http///ppnneeeeeeeeeeeeeeeeeeeea net forum? id=LyJi5ugyJx.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In Advances in Neural Information Pro- cessing Systems 33, pp. 8840-6851 Curran Associates, Inc., 2020.
Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Contrastive energy prediction for exact energy- diffuusion sampling in offline reinforcement learning. In Proceedings of the 40th International Conference Ma- chine Learning, pp. 22825 22855. PMLR, 2023.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Pro- ceedings of 2019 JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4396-4405. IEFE, 2019.
Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang, universal approach for transferring
Z.
10

---

Noise Conditional Variational Score Distillation
knowledge from pre-trained diffusion models. In Ad- vances in Neural Information Processing Systems 36, pp. 76525-76546. Curran Associates, Inc., 2023.
Song. J., Vahdat, A., Mardani, M., and Kautz, J. Pseudoinverse-guiee gurrsee-guieddddeee diffusion models for inverse prob- lems. In The Eleventh International Conference Learning Representations, 2023a. URL https:// openreview net forum? i 1 [d9___MMMMMRK
Mammadov, A., Chung, H., and Ye, C. Amortized pos- terior sampling with diffusion prior distillation. arXiv preprint urXiv:2407.17907, 2024. URL https:// arxiv. org. abw.abss///0 179077.
Song, J., Zhang, Q. Yin, H., Mardani, M.. Liu, M.-Y.. Kautz, J., Chen, Y., and Vahdat, A. Loss-guided diffu- sion models for plug-and-play controllable generation. In Proceedings of the 40th International Conference on Machine Learning, pp. 32483-32498, PMLR, 2023b.
Nguyen, T. H. and Tran, A. SwiftBrush: One-step text- to-image diffusion model with variational score distilla- tion. In Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 7807-7816. IEEE, 2024.
Song, and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems 32, pp. 11918-11930. Curran Associates, Inc., 2019.
OpenAl. OpenAl ol system card. arXiv preprint axXiv:2412.16720, 2024. URL https://arxiv. brg/abs/2412.185
Song, Y. Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In The Ninth International Conference Learning Representations, 2021b. http:::::// /oppnreviewwweeeeeeeeepeeeeeewee net net/forum? id=PxTIG12RRHS.
Peng, X., Zheng, 7., Dai, W. Xiao, N., Li, C., Zou, J., and Xiong, H. Improving diffusion models for inverse problems using optimal posterior covariance. In Proceed- ings of the 4lst International Conference on Machine Learning, pp. 40370. PMLR, 2024.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. In Proceedings of the 40th International Conference on Machine Learning, pp. 32211-32253 PMLR, 2023c.
Peters, J., Mulling, K., and Altun, Y. Relative entropy policy search. In Proceedings of the 24th AAAI Conference on Artificial Intelligence, pp. 1607-1612. AAAI Press, 2010.
Tian, J., Zheng, Z., Peng. X., Li, Y., Dai, W., and Xiong, H. Deem: Dual data consistency guided consistency model for inverse problems. 2024 IEEE International Conference on Image Processing (ICIP), pp. 1507-1513, IEEE, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 10674-10685. IEEE, 2022.
Uehara, M. Zhao, Y., Wang, C., Li, X., Regev, A.. Levine, S., and Biancalani, T. Reward-guides con- trolled generation for inference-time alignment in dif- fusion models: Tutorial and review. arXiv preprint arXiv:2501.09685, 2025. URL https:/ //rriii///xvv... hts/ab//250000 org/abss2201.006680 09685v...
Saharia, C., Ho, J., Chan, W. Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713 4726, 2022.
Salimans, T., Mensink, T.. Heek, J., and Hoogeboom, E. Multistep distillation of diffusion models via moment matching. Advances in Neural Information Processing Systems 37, pp. 36046-36070. Curran Associates, Inc., 2024.
Vono, M., Dobigeon, N., and Chainais, P. Split-and- augmented Gibbs sampler Application to large-scale inference problems. IEEE Transactions on Signal Pro- cessing, 67(6)) 6648)1668811661 2019.
Wang. Z., Lu, C., Wang, Y.. Bao. F., Li, C., Su, H., and Zhu, J. ProlificDreamer: High-fidelity and diverse text-
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation. In Proceedings of 18th European Conference on Computer Vision, pp. 87-103. Springer, 2024.
to-3D generation with variational score distillation. In Advances in Neural Information Processing Systems 36, pp. 8406-8441. Curran Associates, Inc., 2024.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In The Ninth International Conference on Learning Representations, 2021a. URL https:// openrevis
Welling, M. and Tch, Y. W. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 681 668- Omnipress, 2011.

---

Noise Conditional Variational Score Distillation
Wu, Z., Sun, Y., Chen, Y., Zhang, B., Yue, Y., and Bouman, K. Principled probabilisticc imaging using diffusion mod- els plug-and-play priors. In Advances in Neural In- formation Processing Systems 37, pp. 118389-118427. Curran Associates, Inc., 2024.
Zhao, S., Brekelmans, R., Makhzani, A., and Grosse, R. B. Probabilistic inference in language models via twisted sequential Monte Carlo. In Proceedings of the 41st In- terntiionll Conference on Machine Learning, pp. 60704- 60748. PMLR, 2024.
Xiao, Z., Kreis, K., and Vahdat, A. Tackling the generative learning trilemma with denoising diffusion GANs. In The Tenth International Conference
Zhou, M., Zheng, H., Wang. 7.,, Yin, M., and Huang, H. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Proceedings of the 41 International Conference on Machine Learning, pp. 62307-6233 PMLR, 2024.
Learning Represen- lations, 2022. URL https:/ haappp:://ppereee /pppneeeeeeeeeeeeeeeeeeeeeeeeeee werevieeweeeee forum? -i q0Co.
Xu, T., Zhu, Z., He, D., Wang, Y., Sun, M., Li, N., Qin, H., Wang, Y., Liu, J., and Zhang, Y.-Q. Consis- tency models improve diffusion inverse solvers. arXiv preprint 12X0:22403//2063 2024. URL https:// arxiv. org/ 0tss/2403.1200
Zhou, M., Zheng, H., Gu, Y., Wang. Z., and Huang, H. Adversarial score identity distillation: Rapidly sur- passing the teacher in one step. In The Thirteenth In- ternational Conference on Learning Representations, /ttpottttt//oopeee//eeeeee /ppp//eeeeeeeeeeeeeeee net/ forum? id=1s2sGfWizd.
2025. URL https:/
Xu, X. and Chi, Y. Provably robust score-based diffusion posterior sampling for plug-and-play image reconstruc- tion. In Advances in Neural Information Processing Sys- Curran Associates, Inc., 2024.
Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R., and Van Gool, L. Denoising diffusion models for plug-and-play image restoration. In Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1229-1229, IEEE, 2023.
tems 37, pp. 36148-36184
Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F,, and Freeman, W. T. Improved distribution matching distillation for fast image synthesis. In Ad- vances in Neural Information Processing Systems 37, pp- 47455-47487. Curran Associates, Inc., 2024a.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6613-6623. IEEE, 2024b.
Yu, J., Wang, Y., Zhao, C., Ghanem, B., and Zhang, J. Freedom: Training-free free energy-guided conditional dif- fusion model. In Proceedings of 2023 IEEF/CVF In- ternational Conference on Computer Vision (ICCV), pp. 23174-23 23144. IEEE, 2023.
Zhang, B., Chu, W., Berner, J., Meng, C., Anandkumar, A., and Song, Y. Improving diffusion inverse prob- lem solving with decoupled noise annealing. arXiv preprint arXiv:2407.01521, 2024. URL https:// arxiv. org abs /2407 01521.
Zhang, L., Rao, A., and Agrawala. M. Adding conditional control to text-to-image diffusion models. In Proceedings of 2023 IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. 3813-3824. IEEE, 2023.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E.,, and Wang, O. The unreasonable effectiivnness of deep features as a perceptual metric. In Proceedings of 2018 IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 586 595. IEEE, 2018.
12

---

Noise Conditional Variational Score Distillation
A. Proofs
A.1. Derivation NCVSD Gradient as in Equation (9)
Denote x, = GGlYo.,,22 + e.
we have
V FLE D ( (x)
y
Vo po log (x(())))
0x.
y. un,
log pa (x + Xx, (x: log (xx
30
30
+Beyyoian (Vss log Vx, log
@Go 0, 2)
yo. u.
log po (x)lya)
(20)
00
00
suffices to show that ① equals zero:
y po(xxllo)) log pe (xxlyo )dx:
E,,=,,,
log po (x)
ELyy
00
00
1
po (xely))) po(xelyo)
og (x xxllyo)dx
/ po po(xxlyo)dxx
EL,y
08
Fr,y
O.
(21)
|0
A.2. Noise Conditional Score Estimator
The Tweedie's formula in Equation (2) plays central role in deriving the conditional score estimator for NCVSD. For completeness, provide the proof of its conditional version here.
Lemma (Conditional Tweedie's formula). xo: y, x follow the joint distribution q(xa, y, x.) g(xx)g((/xx))((xx xa) with (x)) N (xxxxxo. (T). Then
log (xelx)) = (-2(E(xolxxe.)) - x1).
x.
Proof.
(xe((
Vx, (x.l()
g(x:ly)
9(xx(y) Vx, / ((x:/x00 ()g(xoly)dss
((xely) Vx, / ((xx1220))(xx)l)))
(by conditional independence x; y xo)
(xx1l)) / ((xo y) Vx, o(xx X dxxo
a(xxly) / q(xx) ((xul))g(x, X Vx, log q (Co dxxo
=f((xx)12 ((xx0xxx Vx, log xx)x (by Bayes' rule: x(xo(xx,y) Eg(xolxr.)) [1 (x= xx)] =B Xx. x.).
((xoly)9(x, |xo]
9(x; yy)
(22)
13

---

Noise Conditional Variational Score Distillation
Then, provide the proof of Proposition I in the main paper follows. Proposition 1. Suppose (xo, Ya, Xx) follow the joint distribution qdoa WV |x0, o I Xo, 11). For any p 0, define the denoising posterior of Xo with noise level as p)x)lyp) & Adaia (x)) (yp/xo0 p²I). We obtain that
g(xolx;. No) = (x0 You +1 No
V x. loga(xe(Vo)) t (E X Yoor
Noti a-44==
xt);
where oef 20-2 & is the noise level of g(xolYas)) which is referred to as the effective noise level.
Proof.
g(xolxx.No) &
Adata (x))g(x (x()))(x//)))((()))()) (x0))((())))))))))))))
Adata (xo) exp
1 2/ |x0 - -xxl13 exp - 20 1|x0 - voll²) 112)
Qdata (xo) exp
-
1 2/2
+
2002
1
||x0| + (xo, 2x; + a -3y))
2 25-7 (1/x||' 1 - 2(x0, 12xtoo
x1 xx2) 1 xx
Adata (xo) exp
Qdata (x)) exp
x
2 11²) 200
(completing the square)
2( 2 2)
yy xx 41-2
X Yoa
(23)
Both sides are normalized, so Equation (10) holds.
By Lemma 1, we have Vx, log = 1-2 (E - -x.). Additionally, Equation (23) implies E[xo Xc,
E[xe Your
By 2-241-2
]x.]. Hence, Equation (11) holds.
A.3. Multi-Step Sampling
NCVSD implements multi-step sampling based on DDIM-like latent variable model, and we provide the pseudo code in Algorithm 2. In this section, we provide a detailed definition of the latent variable model and demonstrate that correctly preserves desired marginals. For ease of reading, we rewrite the definition as follows:
q(xo.Nlyo) = ((xolya)q(H)) |xx))
(24)
N
g(xIn\Xx)) xo) q(xx (x)) II
1xxx--1Xx,
(25)
1=2
where ([x)|Xx)) is defined N(xx, I2. To ensure that the marginal distribution (xx:/xx) equals N oxx) for all i 1,2, .... N, construct q(x;-1 X2, xo) as follows:
VI- c~NO, 8= x; Xo
x Xo + -i-1
(26)
σi
where Ç E [0, 1). Or equivalently,
((xi-1|xi,x0 := N + X; Xo
of
(27)
ai
Then, we provide the proof of Proposition 2 in the main paper as follows. Proposition 2. By constructing the following distributions:
g(xx xo) Nixx,, 02
g(x; Xx; xo) N NN x1-x0 02-000-111)
p(xN) E Ex(xolyy) (x(xN// (xo)]:
p(x (x.) Eg(xolx.. ligggigii 1 xo)];
14

---

Noise Conditional Variational Score Distillation
Algorithm 2 Multi-step generative denoising
Input: noisy data Yo. imput noise level a, generative denoiser μil (Xo) Yo). noise annealing schedule oN ON-1 > ...
Ø1 Ri 0, random factor Ç E [0, 1]
XN ~ (N(00 x1)
for i = N, ...., 2 do
Jef [a
J 2)
You = dal (o
Yo + 0, x;)
-x,))
Xp ~ pa (Xolyaal
X,-1 q( xx Xxi X0)
end for Output: Xo
where Ç [0, 1]. Then, have that q(x;) NNxx: T) and p(x;) (xx:(yn)) for 1,2,.., N. In addition, the following equality holds:
((Xo)x;.No) = 9 XD Yoy a 2 -y
0-240,
and the effective noise level og = 20- + 3--
Proof. We divide the proof into three parts, respectively devoted for proving g(x,/xo) = NN o,1), p(x,) = gxx,(yo).
and g(xo|x; yo) Xo Yowr "y.yto to;
3x,
Part I: Similar to (Lemma 1, Song et al. (2021a)), since ((x: = -xo I already hold for i = N, we can prove that
((x//x) = (xx, I. holds for i = 1,2,... - 1 by induction. Specifically, suppose q(x;/xn) = NN I), then
((x:-1 xo) = f q(xi-1 )x))
(28)
a Gaussian distribution. Its mean and variance can be determined by Bayes' theorem for Gaussian variables (2.115, Bishop Nasrabadi (2006) as
g(x=-1(x0) = N (xo (xo+ +0--Ii
Xo Xa-xo ,0 (5-Ivi-S-2 o² = -N(xo, (x0,,0__1)
(29)
Therefore, (xx N(xo, 1) for 1,2,
Part II: Since p(xN) Eg(xoly.))) (onely.))((xx))) (g(xN ( q(xx lyo) already hold for i N, also prove the statement p(x:) ((x:ly.) holds for = 0, 1,.,, N by induction. Specifically, suppose p(x;) = q(x,y=)) and we consider the following marginalization representation of q(x)=
/
g(x:-1l8o)
/ q(xx1yo (xxol:)))((()/xx.x))))> ) x No )xx
/ g(xi(186)
(f | (x )xxo
dx,
p(x.)
1(xx-1/x;)
-))
(30)
Therefore, pxx) = ((x:\y>) for = 1,.,,.,,N N.
Part III: By (xx, (xo) = Nixo, I2),, the joint distribution of Xo, Yo X; is qaata (aaaa(Xa)) (xon)(X)N((y O2IN of1). Equa- tion (16) holds according to Proposition 1. Thus, we conclude the proof.
15

---

Noise Conditional Variational Score Distillation
Algorithm 3 One gradient optimization step of the generative denoiser Go
Input: generative denoiser Go and EMA parameter 0 score model Do for estimating data score V log (xx score model Da for estimating model score log po (x+ly>). uncertainty weighting model Wa, discriminator Cg: learning rate 7), and EMA rate function B
Sample X from the dataset
Sample t, a from predefined distributions
Sample yo ~ (xx, d²I)
Sample Xg Golyo, a, z), where I0,
Sample X (Xo, (T)
Compute effective noisy sample as
Oat < (o 2 3 Nom t oen (o (o- + xx0)
Compute estimation for the data and model scores (rescale and omit x,)
So + Do(Yo Teft); Sp D2((e,
Compute loss and do gradient update for 0
C(B, - (x |X stopgrad(so - Ss + xa)11² + dim(xo) (t) diin(xo) log
00, - [0.1] - [V,C(O,, VxC(O,
8 + 8(t) - +
B. Experimental Details
In this section, we introduce parameterization of the neural networks used in the experiments, the training and inference details for NCVSD, and the details for inverse problem solving using PnP-GD.
B.1. Parameterization
Score model
We utilize a score model D to estimate the conditional score function V log (x y.)) in Equation (9):
V (xg t -2 (Da(xe, l, (Ya; o)) xt),
(31)
where the parameters & of D& are initialized from the teacher model Do and then fine-tuned via optimization of Equation (13) during the training process of the generator Gg. The additional condition inputs (y,o) is injected into D3 using a trainable control net (Zhang et al.. 2023) like architecture, and we use {.} emphasize its input. Specifically, we copy the encoder of the UNet model (does not share weights) and add the outputs of the copied encoder the outputs of the original encoders before input into the UNet decoder. We discard the zero convolutions in the original control net since it will break the magnitude preserving property introduced by EDM2 (Karras et al., 2024). Instead, we propose learnable magnitude preserving addition layer to achieve the same goal as the zero-convolutions. Specifically, denote a as the output of the original encoder, and denote b as the output of the copied encoder, we obtain the new output according to:
MP-Sum (a, b) == (1 w )a + wb
(32)
w²
where w E [0, 1] is a learnable weight, and is initialized to to prevent disrupting knowledge of the pretrained model, which function similarly to the zero convolutions in the original control net (Zhang et al., 2023). We force w lies in [o, 1] using similar implementation the foreed weight normalization in EDM2.
Generative denoiser In practice, use model Golyo, a, to implement the generative denoiser no (xxxllyo xxoly)). We parameterize Golyo, z) by adapting the network architecture of the pretrained score model Do(x,, t), given by
Golyo, 2) Doly (2)), z NO..
(33)
16

---

Noise Conditional Variational Score Distillation
where z is an additional noise input to introduce stochasticity for generating random samples. The parameters 0 of Dg are initialized from the teacher model Da, which enables efficient transfer of knowledge from the teacher model Do. well as reusing the inductive bias of preconditioning (Karras et al., 2022). However, we observed that directly using Equation (33) leads to severe mode collapse. To address this issue, we propose introducing stochasticity directly into Yo by adding random noise z. Specifically, we add z to ya using scaling factor ry to achieve a higher noise level â = a + ya, resulting in Ya = Yo + V@² - o²z² 7. Additionally, the original conditions Ya and a are fed into a trainable ControlNet, similar to the score model Dø. to preserve critical information. Therefore, the generative denoiser finally defined as:
Golya: 0, 2. = Do(ya, ã, ()..
(34)
Discriminator To parameterize Cy (x,, 0) for the adversarial loss in Equation (15), we employ two UNet encoders, EØr and Egg, to extract features from (x,, 1) and (Ya,o), respectively. The outputs of the final layers of these encoders are concatenated along the channel dimension, followed by an average pooling layer applied to the spatial dimensions, a linear layer, and sigmoid layer to produce a scalar output in [0, I]:
(, σ) == Proj Evr, (xt, t), Ev (v).
(35)
where Proj := Sigmoid o Linear o AvgPool o Concat, a == U2], and E Eng are initialized from the encoder of the teacher model Do-
B.2. Training and Inference
Uncertainty weighting We employ uncertainty weighting (Kendall et al., 2018; Karras et al., 2024; Lu & Song, 2025) to balance the loss contributions across different t. Specifically, since the NCVSD gradient is a vector-Jacobian product, we can convert it to gradient of a L2 loss, as follows:
Locaaa (0) = E (Vx, (x:l)( Xx, log ((x())o)) (Go(y-, 09
(36)
R 2 (Do(xt, - Doly a: Jeti) @Guly..o.x. m.
(37)
80
= = (IG stopgrad (G (Yo, o,
Do (8, + Doly oer Gat
(38)
L2 target
where stopgrad is the stop gradient operator, and the expectation is taken over (,, o, 2, Xx) where X, N(Go(y,, a, z),t"r) II). In Equation (37), we leverage the teacher model Do and the score model D& to approximate the score functions as Vx, (xglo(xxly>) & i (Dx (D,,(x,, - xt) and Vx, logg(x:lyo) 2 (D Gen) - xt). The scale of Equation (38) varies considerably across noise levels. To address this, we introduce an uncertainty weighting network ux ensure that the losses have unit variance accross noise levels. Specifically, we define the following two losses C, and Lz. which respectively provide unbiased estimates of the gradients of Equation (38) and Lady up scaling factor:
o)
Li := e (t) [ stopgrad So x0)]1( + dim(xe) wx (t)
(39)
Lg = - log Co (38,
(40)
where x, := Go(y), xo- := x,, So == Dolyoar Tea). and So == (,,x,,t,yo.0 o) for notation simplicity and to emphasize the dependency on parameters. Note that Li can be viewed as the sum of independent negative Gaussian log- ligeeiioissods over the data dimensions, while L2 is also negative log- likeliood..odd. To balance the contributions of the two losses, it is natural to scale L2 by the number of data dimensions. Therefore, we define the final loss as
L: Li dim(xx) L2.
(41)
To summarize, we provide pseudo code for the NCVSD training algorithm in Algorithm 3. Note that before optimizing Go using Algorithm 3, the score model D& and the discriminator Cs. should also be optimized for the distribution of Xy Golyo, 0, 2), will be elaborated below.
'A similar approach has been proposed in EDM (Karras et al., 2022) for introducing stochasticity in the diffusion sampler.
17

---

Noise Conditional Variational Score Distillation
Table 3. Hyperparameter details for training and inference.
ImageNet 64x64
ImageNet 512x512
FFHQ 256
S
M
L
S
M
XS
Model details
Channel multiplier
192
256
320
192
256
320
128
Dropout probability
0%
0%
0%
0%
0%
0%
0%
Stochasticity strength Y
0.414
0.414
0.414
0.414
0.414
0.414
0.414
Model capacity of Go (Mparams)
368.2
653.5
1020.1
368.2
653.5
1020.1
146.2
Training details
Effective batch size Learning rate max (atet) Learning rate decay (fret) Learning rate warm K images Adversarial warm up images
2048
2048
2048
2048
2048
2048
128
0.0100
0.0090
0.0080
0.0100
0.0090
0.0080
0.0120
35000
35000
70000
70000
70000
70000
35000
1000
1000
1000
1000
1000
1000
100
16778
16778
16778
16778
16778
16778
2097
Learning scaling for C. Learning rate scaling for Ga
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
Adam 3
0.9
0.9
0.9
0.9
0.9
0.9
0,9
Adam |32
0.99
0.99
0.99
0.99
0.99
0.99
0.99
Training samples (Mi, 2
64
64
64
64
64
64
Noise distribution P mean for
-0.8
-0.4
-0.4
-0.4
-0.8
Noise distribution Pau for t
1.6
1.6
1.6
1.0
1.0
1.0
1.6
Training cost
Mixed precision
fp16
fp16
fp16
fp16
fp16
fp16
fp16
Loss scaling to prevent overflows
1.0
1.0
1.0
1.0
1.0
1.0
0.1
Batch size per GPU
16
64
32
A100 GPU hours
650
~1100
1450
~650
1110
~ 1450
~300
Inference details
Random factor {
1.0
1.0
1.0
1.0
1.0
1.0
1-step sampling timesteps
10
10
10
12
12
12
2-step sampling timesteps
10,22
10,22
10,22
10,22
10,22
10,22
4-step sampling timesteps
0,10,20,30
0,10,20,30
0, 10,20.03
0, 10,0200,30
0,10,20,30
0,10,20,30
Training hyperparameters We perform one gradient descent optimization for the score model De (optimizing Equa- tion (13)) and the discriminator Cs (optimizing Equation (15)) before one gradient descent optimization for the generator Dg- To stabilize training, we employ adversarial loss warmup by disabling adversarial loss at the beginning of training. The distribution for the noisy data condition yo is defined by sampling uniformly on EDM (Karras et al., 2022) inference time noise schedule, given by
0,
( miax
i N- (o )maxx) i- i, N
(42)
where we select Omax
80.0, 0min = 0.002, p 7.0, and 1000. The distribution of defined using LogNormal distribution (Karras al., 2022) logt ([Pmaan P2d) where P Preann Pad are hyperparameters. We provide detailed training hyperparameters in Table 3.
Class-conditional generation
The class-conditional image generation ImageNet datasets is achieved by performing
denoising posterior sampling from pure Gaussian noise at a sufficiently high noise level Oinit We set annit = 80.0 for all experiments. To specifying the noise schedule o, for multi-step sampling proposed in Section 3.3, we select time steps and decide the noise level o, using the EDM inference time noise schedule given in Equation (42) with = 40, omin = 0.002, Fmax 80, and p = 7.0. Detailed time steps for each experiment can found Table
18

---

Noise Conditional Variational Score Distillation
Table 4. Hyperparameter details for PnP-GD.
Inpaint (Box)
Deblur (Gaussian)
Deblur (Motion)
Super resolution
Phase retrieval
Annealing schedule details
Steps (N)
50
50
50
50
max
80.0
80.0
80.0
80.0
80.0
min
0,002
0.002
0,002
0,002
0.002
P
2.0
2.0
2.0
2.0
2.0
EMA schedule details
EMA threshold (Fema)
00
00
00
00
0.2
EMA decay (1))
0.2
0.6
0.6
0.6
0.6
Likelihood step details
Energy strength B
le-4
2e-3
4e-3
le-3
1e-3
ULA step
100
100
ULA C
0.1
0.1
ULA C2
0.1
0.1
B.3. Inverse Problem Solving
Model Training To train generative denoiser on FFHQ dataset using NCVSD, we first pretrain a XS size diffusion model using EDM2 codebase. The hyperparameters setup mostly following XS size EDM2 model for ImageNet-64 x664 dataset, except that we use batch size of 128 and learning rate warmup of 1M images. Additionally, we implement loss scaling of 0.1 to prevent fpl6 overflows. We train the model until FID plateaus, which takes roughly 32M training images. Training generative denoiser on FFHQ dataset is the same as on ImageNet dataset. The training hyperparameters can be found in Table 3.
Likelihood Step
The general model for inverse problems given as follows:
y A(xx) + n, n of
(43)
where A the degradation operator, which is possibly nonlinear, and is an additive white Gaussian noise with std of oy. Using Bayesian framework for solving inverse problems by formulating the posterior distribution ((xo) qaam (Xo)(x) (x)), the likelihood function is given by Gaussian likelihood as q(y(xx) .V(ylAA(x)),) ozI). With the energy function formulation used in PnP-GD (Equation (17)), it is equivalent to define the energy as
E {xx) |yy A(xo) 12 , = 20
(44)
However, for better empirical performance, we also consider 33 as a hyperparameter to tune, following DAPS (Zhang et al., 2024). To accelerate the likelihood step, we use fast closed-form solvers implemented in PnP-DM (Wu et al., 2024) for linear inverse problems. For nonlinear inverse problems, we use ULA with adaptive step size presented in Section 4.
Hyperparameters For the annealing schedule o; in PnP-GD (Algorithm 1), we use EDM inference time noise schedule given in Equation (42). We provide the detailed hyperparameters of PnP-GD in Table 4.
B.4. Baseline Details
DDRM
We borrow the results reported in DAPS (Zhang et al., 2024).
DPS
We borrow the results reported in DAPS (Zhang al., 2024).
DiffPIR For noisy super-resolution, Gaussian deblurring, and motion deblurring, use the default setting in the original paper. For noisy inpainting, we use = 7.0, Ç = 1.0, and 100 NFE.
We use the IIGM implementation provided the codebase Peng et al. (2024) since the original code base does not support noisy linear inverse problems.
HIGDM
19

---

Noise Conditional Variational Score Distillation
Table SSIM comparisons noisy inverse problems. The results averaged 100 images. We bold and underline when the proposed method (PnP-GD) achieves the best and the second best, respectively.
Method
Inpaint (box)
Deblur (Gaussian)
Deblur (motion)
Super resolution
Phase retrieval
DDRM (Kawar et al., 2022)
0.801
0.732
0.512
0.782
DPS (Chung et al., 2023)
0.792
0.764
0.801
0.753
0.441
IIGDM (Song et al., 2023a)
0.663
0.720
0.733
0.720
DWT-Va (Peng et al., 2024)
0.796
0.795
0.798
0.802
DAPS (Zhang et al., 2024)
0.814
0.817
0.847
0.818
0.851
PnP-DM (Wu al., 2024)
0.780
0.795
0.787
0.628
PnP-GD (Proposed)
0.814
0.777
0.801
0.805
0.797
We report the results based on the original codebase of Peng al. (2024)* with the default settings. For box inpainting, use Type II guidance with the DWT-Var only used when the std of the diffusion noise is below 0.5.
DWT-Var
DAPS We report the results based on the DAPS codebase" with the default settings.
PnP-DM We compare with PnP-DM using EDM priors, i.e., PnP-DM (EDM) in the original paper. For linear inverse problems, we use the default setting and report the metrics based single sample instead of the mean over 20 samples for more direct comparison to PnP-GD. For phase retrieval, we use By = 0.05 instead of σy 0.01 of the original paper for fair comparisons.
C. Additional Results
Training FID curves In Figure 3, plot the training FID the number of training images, demonstrating classic training time scaling law and the effectiveness of test time scaling.
Qualitative samples for class-conditional image generation on ImageNet 512x512 In Figures 4-5, we show qualitative results of 4-step samples generated by NCVSD-L for class-conditional image generation on ImageNet-512x512 dataset.
SSIM comparisons for inverse problem solving In Table 5, we provide additional SSIM comparisons of different methods.
Effectiveness of EMA rate in PnP-GD In Figure 6, we plot the LPIPS and PSNR values under different EMA decay rates z used in PnP-GD. As shown, PSNR tends to favor larger values of μl, as they give more weight to historical samples, making the final results closer the posterior mean. This approach tends produce blurrier images but with less distortion. In contrast, LPIPS favors smaller values of μ., making the final results closer the posterior samples, which, while reducing blur, can lead to higher distortion. For visualization, please refer to Figure
Qualitative samples for inverse problem solving In Figure 8, present visual comparisons with different methods for inverse problem solving. As can be seen, our method (PnP-GD) generate samples with more high frequency details compared to baselines.
'https://gu wttp::ttccccmooppgggg/////ddddd otw- http:::///ittie htw....c.m/haaggnnnggli..ee....e 9/DPP///APPPSS
20

---

Noisc Conditional Variational Score Distillation
50
50
NCVSD-5(1
NCVSD-M
NCVSD-L stop)
30
NCVSD-5 (2 step)
NCVSD-M M (2 step)
NCVSD-L (2 step)
NCVSD-5 step)
NCVSD-M step)
NCVSD-L step)
20
20
20
10
19
5
3.06
2,96
2.47
2.14
1.92
2-
1.53
10000 20000 30000 40000 50000 60000 70000
10000 20000 30000 40000
50000 60000 70000 Training images
10000 20000 40000 50000 60000 70000
Training images
Training images
(a) Training FID ImageNet 64
50
50
50
NCVSD-5 step)
NCVSD-M step)
NCVSD-L (1 step)
NCVSD-5 (2 step)
NCVSD-M (2 step)
NCVSD-L (2 step)
30
NCVSD-5 (4
30
NCVSD stop)
30
NCVSD step)
20
20
20
RD
2.60
3-
2-85
3-
2.56
2
2.99
2
2.03
1.76
10000
20000
30000
40000
50000
10000
20000
30000
40000
50000
10000
20000
30000
40000
50000
Training images
Training images
Training
images
(b) Training FID on ImageNet 512x512
Figure 3. FID v.s the number training images.
21

---

Noise Conditional Variational Score Distillation
Figure 4. Uncurated 4-step samples generated by NCVSD-L for class-conditional ImageNet 512x512 generation. Top: class 88 (macaw);
middle: class 89 (cokatoo); bottom: class 388 (giant panda).
22

---

Noise Conditional Variational Score Distillation
Clei
Figure 5. Uncurated 4-step samples generated by NCVSD-L for class-conditional ImageNet 512x512 generation. Top: class 425 (barn);
middle: class 933 (cheeseburger); bottom: class 980 (volcano).
23

---

Noisc Conditional Variational Score Distillation
180
0.375
0.300
0.175
0.379
0.175
0.263
0.160
0.155
0.150
dedny
decay
71.29)
28.2
28.0
27.0
27.0
27.6
27.4
26.4
27.2
26.2
27.2
BMA
EMA
(a) Gaussian deblurring
(b) Motion deburring
(c) Super resolution
Figure 6. Effectiveness of EMA rate in PnP-GD.
Figure 7. Samples of PnP-GD under different EMA decay rate From left to right, equals to 0.0. 0.2, 0.4, 0.6, 0.8.
24

---

Noise Conditional Variational Score Distillation
Messurement
DifrPIR
DWT-VVr
DAPS
Pri--DD (Ours)
Reference
(a) Inpaint (box)
(Ours)
(b) Deblur (Gaussian)
Measurement
DWT-Var
DAP5
PnP-DM
PriP-DD (Ours)
Reference
(c) Deblur (motion)
Messurement
DWT-Var
DAPS
PnP-DM
PnP-GD (Ours)
Reference
(d) Super resolution
Messurement
DAPS
PMP.DM
PyP-GD (Ours)
Referance
(e) Phase retrieval
Figure Visual comparisons for inverse problem solving.
25


Reviews:
[{"review_id": "U1JLGlNH6Q", "reviewer": "Reviewer_15K4", "summary": "This paper proposes a new distillation scheme to learn a few-step posterior sampler of a diffusion process.  Unlike existing methods that achieve rich posterior sampling using exhaustive function evaluations (e.g., diffusion) or invertible neural networks (e.g., normalizing flows), the proposed method achieves high-fidelity stochastic sampling from the distribution of clean samples gives noisy samples with significantly less inference compute.  Experiments show that the proposed method is competitive with existing few-step samplers with less than quarter of the training budget and furthermore using a split Gibbs sampling technique similar to PnP-DM the proposed method outperforms all baselines on several  image inverse problem tasks.\n\n## update after rebuttal\n\nI thank the author(s) for their responses during the rebuttal period.  With the changes promised and results provided during the rebuttal phase I increased my score towards acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*W1：Choice of adaptive ...* \n\nWe thank the reviewer for pointing out the ambiguity in our claim. A function $f(\\cdot)$ is called L-gradient Lipschitz if it satisfies:\n$$\n\\lVert \\nabla f(x\\_1) - \\nabla f(x\\_2) \\rVert\\_2 \\leq L \\lVert x\\_1 - x\\_2 \\rVert\\_2, \\quad \\forall x\\_1, x\\_2.\n$$\nProvided that $\\mathcal{E}$ is L-gradient Lipschitz, the gradient difference of  $\\tfrac{1}{\\beta}\\mathcal{E}(\\cdot) + \\tfrac{1}{2 \\sigma^2} \\lVert \\cdot - u \\rVert\\_2^2$\n$$\n= \\lVert \\beta^{-1} (\\nabla \\mathcal{E}(x\\_1) - \\nabla \\mathcal{E}(x\\_2)) + \\sigma^{-2} (x\\_1 - x\\_2) \\rVert\\_2\n$$\n$$\n\\leq \\beta^{-1} \\lVert \\nabla \\mathcal{E}(x\\_1) - \\nabla \\mathcal{E}(x\\_2) \\rVert\\_2 + \\sigma^{-2} \\lVert x\\_1 - x\\_2 \\rVert\\_2\n$$\n$$\n\\leq \\beta^{-1} L \\lVert x\\_1 - x\\_2 \\rVert\\_2 + \\sigma^{-2} \\lVert x\\_1 - x\\_2 \\rVert\\_2\n$$\nTherefore, the postential function is $(\\beta^{-1} L + \\sigma^{-2})$-gradient Lipschitz.\n\n*W2: Optimality of Existing ...* \n\nThe claim holds in the non-parametric limit. This limitation arises because the loss functions of existing methods do not ensure that $q(x\\_0|y)$ is the unique minimizer. For instance, the objective in (Lee et al., 2025) is defined as:\n$$\n\\min\\_{\\theta} -\\mathbb{E}\\_{\\mu\\_{\\theta}(x\\_0|y)} [q(y|x\\_0)] + \\int w(t) D\\_{KL}(p\\_{\\theta}(x\\_t|y) || q(x\\_t)) dt,\n$$\nwhich can only be interpreted as a regularized optimization problem, with no guarantee that $\\mu\\_{\\theta}(x\\_0|y) = q(x\\_0|y)$ at convergence. Similarily, (Mammadov et. al., 2024) employ ELBO for the prior term (Eq.(14) in the paper) that is also no guarantee. In contrast, NCVSD achieves the optimal iff $\\mu\\_{\\theta}(x\\_0|y) = q(x\\_0|y)$ (Luo et al., 2023). This property allows marginal-preserving multi-step sampling (Sec. 3.3), and enables the application of the SGS (Sec. 4).\n\n*W3: Authors have cited ...*\n\nWe thank the reviewer for bringing this relevant work to our attention. We are look forward to contributing further to the field of posterior sampling.\n\n*W4: The main weakness lies ... & Existing methods that this work ...*\n\nWe thank the reviewer valuable suggestions. Please refer to W6,W8 for comparisons with PnP-DM and W6,W9 for one-step posterior sampler.\n\n*W5: All loss functions ...*\n\nWe thank the reviewer for the valuable suggestion. We will explicitly include the parameters to be optimized in all loss function to further enhance clarity. \n\n*W6: The authors should ...*\n\nWe appreciate the reviewer’s insightful suggestion. Below, we provide a dedicated background on posterior sampling, which will be included in the revised version by extending Section 2.1:\n\n*“Existing posterior sampling methods often involve trade-offs among flexibility, posterior exactness, and computational efficiency. Supervised approaches (e.g., Saharia et al., 2022) lack flexibility as they require retraining for each specific task. Zero-shot methods provide greater adaptability but introduce irreducible errors by approximating the denoising posterior with Dirac (Chung et al., 2022) or Gaussian distributions (Song et al., 2023). Asymptotically exact methods, such as PnP-DM (Wu et al., 2024), ensure asymptotically exact posterior sampling but are computationally intensive, relying on reverse diffusion simulations that demand a large number of NFEs.”*\n\n*W7: The impact statement ...*\n\nWe thank the reviewer for valuable suggestion. We will enhance the impact statement to emphasize the advancements in efficient generative modeling and inverse problem solving, and ethical considerations.\n\n*W8: What are the main ...*\n\nWe appreciate the reviewer’s valuable suggestion. The primary distinction between the proposed method and PnP-DM lies exactly in how the prior step is approximated. To address this, we will include the following clarification after L277 in the revised version:\n\n*\"The proposed method and PnP-DM are both built upon the foundation of SGS. The primary distinction lies in how the prior step is approximated. In PnP-DM, simulating the reverse diffusion process is required, which is not only computationally inefficient but also prone to irreducible discretization errors. In contrast, our approach significantly improves computational efficiency by requiring only one or a few NFEs for the prior step, while being free from any errors beyond those introduced by imperfect model training.\"*\n\n*W9: The two prior works ...*\n\nWe appreciate the reviewer’s valuable suggestion. High-quality sample generation is indeed a key strength of our method compared to prior works. Additionally, our approach offers two significant advantages: accurate posterior sampling (W2), and the flexibility to address a wide range of problems (W6).\n\nIn summary, we will incorporate the reviewers' insightful suggestions to further enhance the contributions of our paper. Specifically, we will:\n- Highlight the fast sampling (W8).\n- Include a dedicated background on posterior sampling (W6).\n- Elaborate novelty (W2,W6,W9).\n- Emphasize computational advantages (W8).\n- Improve impact statement (W7).", "claims_and_evidence": "Most of the claims are supported by evidence.  Below are two claims that should be supported further:\n\n1. *Choice of adaptive step size:*  It is not obvious why the function is $(\\beta^{-1}L + \\sigma^{-2})$-gradient Lipschitz.  A proof or simple derivation of this would be helpful.\n2. *Optimality of existing posterior sampling methods:* The authors claim that existing amortized posterior sampling schemes do not necessarily result in samples from the posterior at convergence.  Is this true in the non-parametric limit as well or is this claim based on empirical evidence justified by estimation/approximation errors? It would be useful to have some proof or elaborate technical argument about the convergence properties of the proposed method in comparison to existing methods.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are satisfactory.", "theoretical_claims": "Yes I checked all the proofs and they appear to be correct.", "experimental_designs_or_analyses": "Yes, the experimental design and analysis for both generation and inverse problem are satisfactory,", "relation_to_broader_scientific_literature": "The key contributions in this paper could be relevant to several different areas of scientific research.\n\n1. *Synthetic Data Generation and Data Augmentation:* The ability to produce high-quality samples from the data distribution could aid in synthetic data generation or data augmentation especially in data scarce settings. \n2. *Efficient Inverse Problem Solvers*: In many existing score-based inverse problem solvers, sampling from the posterior is crucial.  However this has generally been difficult and instead approximations leveraging the conditional expectation are made.  The contributions in this paper regarding efficient posterior sampling could help enable fast inverse solvers across a variety of modalities.", "essential_references_not_discussed": "Authors have cited revelant work in the paper.  Several new methods for posterior sampling methods using engression based on scoring rules have been introduced recently.  I have added one as a reference below but the authors are not expected to discuss or compare against this method as it is considered concurrent work.\n\n[1] De Bortoli, V., Galashov, A., Guntupalli, J. S., Zhou, G., Murphy, K., Gretton, A., & Doucet, A. (2025). Distributional Diffusion Models with Scoring Rules. arXiv preprint arXiv:2502.02483.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The approach to parametrize a stochastic few-step generative sampler is novel.  Many existing methods often leverage deterministic generators. The authors have done a good job describing how to design the posterior sampler conditioned on a noisy observation with Gaussian additive noise.\n2. The approach of leveraging a pre-trained unconditional score model to compute the score of the posterior is very interesting and could be very applicable  in practice.\n3.  The experiments are well designed and the results are competitive.\n\n**Weaknesses**\n\n1. The main weakness lies in the exposition of the method.  It is difficult to appreciate the proposed method in its current form as it is clear where it stands and how it compares to existing works in literature.\n2. Existing methods that this work builds upon are not discussed at length.  PnP-DM (Wu et al. 2024) is the foundation of the inverse problem algorithm but it is only cited.  It is not clear what the existing framework is and how the proposed framework is different upon initial reading.  Similarly distillation methods for learning a one-step posterior sampler are discussed in (Mammadov et. al., 2024) and (Lee et. al., 2024).  Beyond being cited there is no section dedicated to discussing these existing works and mentioning differences (and more importantly similarities).", "other_comments_or_suggestions": "Here are some comments regarding syntax and typos:\n\n1. All loss functions are typically a function of the parameters.  Furthermore, more care should be taken when defining optimization objectives.  For example, in equation (3) it would be more if the loss function was defined as $\\mathcal{L}(\\theta)$ and the optimization objective was $\\min_\\theta \\mathcal{L}(\\theta) \\triangleq \\min_\\theta \\mathbb{E}[\\dots]$. Please also look at equation (8) and equations (60)-(64).\n2. The authors should seriously consider devoting more time to discussing existing works on posterior sampling (that I mentioned in the weaknesses).  The paper would be stronger if there was a dedicated background section to these methods beyond just VSD.\n3. The impact statement should highlight some broader impacts and is not satisfactory in its current form.", "questions_for_authors": "1. What are the main differences between the proposed method and PnP-DM? It seems to be that the difference lies in the posterior sampler being used.  PnP-DM simulated the reverse diffusion process to generate a sample whereas the proposed the method use the few-step model.  Everything else with regards to the Gibbs sampler is the same.  Is this correct?  If so, this is not clearly described at all.  I would suggest adding a background section on PnP-DM, describing their algorithm with the Gibbs sampler.  Then point out the inefficient of the prior sampling step and mention how the proposed method solves the issue. \n2. The two prior works in (Mammadov et. al., 2024) and (Lee et. al., 2024) both describe schemes for one-step posterior sampling.  The key conceptual difference seems to be that the existing works sample from the posterior conditioned on a general noise observation (e.g., defined by a linear forward process) whereas the proposed method conditions on a noisy additive-Gaussian noise sample.  This seems to induce a key technical use case beyond just inverse problem solving as the proposed method can be used for high-quality sampling as well.  Coming to this realization took several cycles of reading the existing papers and then contrasting with the proposed method.  The authors need to detail the benefits of the method much more clearly.  Stress on fast sampling, have a detailed section on these existing works, describe technical novelty by *building on existing methods* and then mention key training/test-time computational advantages.\n\nTo summarize, I am prepared to raise my score if the authors are able to work on the exposition of the method by make a more just effort in discussing existing works, highlighting their pros/cons and then describing the advantages (and limitations) of the proposed method.  I believe the proposed method is interesting but it not ready for publication in its current form.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4alE4wHfYY", "reviewer": "Reviewer_7Tng", "summary": "This paper introduces Noise Conditional Variational Score Distillation, which distills a pre-trained diffusion model into a generative denoiser. The generative denoiser enables fast one-step generation while preserving the ability for iterative refinement. Experiments on image generation tasks and various inverse problems demonstrate the effectiveness of the proposed method.\n\n**update after rebuttal**\n\nThank you for your rebuttal. I will keep my score unchanged and remain positive about this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*W1: After reviewing this paper ...*\n\nConceptually, the key distinction between diffusion models with iterative refinement and our approach lies in how clean data $x\\_0$ is predicted from its noisy counterpart $y\\_{\\sigma} \\sim \\mathcal{N}(x\\_0, \\sigma^2 I)$. Diffusion models primarily focus on learning the MMSE prediction, whereas generative denoisers aim to model the full posterior distribution over $x\\_0$.\n\nRegarding the reviewer's concern about whether certain acceleration techniques for diffusion models could achieve similar goals as our proposed method, we argue that significant gaps remain where our method demonstrates clear advantages:\n\n- **Image Generation**: While acceleration techniques for diffusion models, such as DDIM (Song et al., 2020) or advanced numerical integrators (Lu et al., 2022; Karras et al., 2022), can reduce the required NFEs from 1k (original DDPM) to 10–100, the generative quality degrades significantly when NFEs drop below 10. In contrast, our method achieves state-of-the-art results with just 1–4 NFEs, with performance at 4 NFEs even surpassing that of the teacher diffusion model. Moreover, generative denoisers can theoretically match the data distribution using only 1 NFE, whereas diffusion models require an infinite number of time steps to achieve the same.\n\n- **Inverse Problem Solving**: Current diffusion-based methods either suffer from irreducible approximation errors by using Dirac (Chung et al., 2022) or Gaussian (Song et al., 2023; Peng et al., 2024) approximations for the denoising posterior, or achieve asymptotically exact sampling at the cost of expensive reverse diffusion simulations (Wu et al., 2024). The former lacks theoretical guarantees, while the latter requires a significant number of NFEs (e.g., 2483 for PnP-DM) and remains affected by discretization errors. In contrast, our method achieves superior results with 20x fewer NFEs, offering both computational efficiency and theoretical robustness by avoiding prior-step errors beyond those introduced by imperfect model training.\n\n*W2: Regarding the experimental ...*\n\nWe evaluate our method on the subset (100 images) of FFHQ dataset, following the standard practices established in diffusion-based restoration methods, e.g., DiffPIR (Zhu et al., 2023), DAPS (Zhang et al., 2024), PnP-DM (Wu et al., 2024), among others. Morevover, test on full validation set is computationally costly for methods like DPS, DAPS, PnP-DM (over 1000 NFE). This alignment also allows us to make a fair a comparison to existing methods with standard evaluation protocols.\n\nFor super-resolution tasks, both our method and the baseline diffusion models are trained and evaluated on 256×256 resolution datasets such as FFHQ, following common practice in recent literature. Since existing approaches—including EDM2—have not been trained on 2K-resolution data, we do not include evaluations on high-resolution benchmarks like DIV2K or Flick2K. Nevertheless, our results are consistent with prior work and effectively demonstrate the strength of our approach. Extending the framework to higher-resolution settings remains a promising direction for future research.\n\n*W3: I think SSIM ...*\n\nWe thank the reviewer for the suggestion. The SSIM performance is provided in the table below. \n\n*Table. SSIM performance of inverse problem solving on FFHQ dataset.*\n\n|Method|NFE|Inpaint(box)|Deblur(Gaussian)|Deblur(motion)|Super resolution|Phase retrieval|\n|-|-|-|-|-|-|-|\n| DDRM          |100|0.801| 0.732 | 0.512 | 0.782 | N/A   |\n| DPS           |1000|0.792| 0.764 | 0.801 | 0.753 | 0.441 | \n| PiGDM         |99|0.663| 0.720 | 0.733 | 0.720 | N/A   |\n| DWT-Var       |99|0.796| 0.795 | 0.798 | 0.802 | N/A   |\n| DAPS          |1000|0.814| 0.817 | 0.847 | 0.818 | 0.851 |\n| PnP-DM        |2483|N/A| 0.780 | 0.795 | 0.787 | 0.628 |\n| PnP-GD (Ours) |50| $\\mathbf{0.814}$ | 0.777 | $\\underline{\\text{0.801}}$ | $\\underline{0.805}$ | $\\underline{0.797}$ |\n\nSimilar to PSNR, our method demonstrates competitive performance, being only outperformed by DAPS. However, DAPS achieves this by employing significantly more NFEs and extensive hyperparameter tuning. In contrast, PnP-DM, the most relevant baseline, delivers inferior performance despite utilizing more NFEs than our approach. This table will be included in the revised version to provide a more comprehensive evaluation.", "claims_and_evidence": "Please refer to Strengths And Weaknesses.", "methods_and_evaluation_criteria": "Please refer to Strengths And Weaknesses.", "theoretical_claims": "Please refer to Strengths And Weaknesses.", "experimental_designs_or_analyses": "Please refer to Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "Please refer to Strengths And Weaknesses.", "essential_references_not_discussed": "Please refer to Strengths And Weaknesses.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-structured, and the overall storytelling is good, making it easy to follow.\n2. The idea of distilling a pre-trained diffusion model into a generative denoiser is straightforward.\n3. The quantitative and qualitive results appear promising.\n\n**Weaknesses**\n\n1. After reviewing this paper, I have a question regarding the fundamental motivation mentioned in the introduction (L24–L26). What is the key difference between a diffusion model with iterative refinement capabilities and the proposed generative denoiser? I understand the advantage of one/few-step sampling methods, but does extending them to a multi-step process truly constitute a meaningful idea? In other words, can we just apply certain techniques to shorten the sampling steps of a standard diffusion model?\n\n2. Regarding the experimental section, actually I haven't done too much work on \"diffusion for restoration\", so I am particularly curious about the testset selection process. It seems that you did not use the full test sets but instead selected a subset. Could you provide more details on this? Additionally, I would like to see results on standard benchmark testsets (e.g., SR ×4 results on DIV2K and Flick2K).\n\n3. I think SSIM also makes sense. Including SSIM results in Table 2 would enhance the persuasiveness of the paper.", "other_comments_or_suggestions": "Please refer to Strengths And Weaknesses.", "questions_for_authors": "Please refer to Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D3EdG2LppJ", "reviewer": "Reviewer_s1LA", "summary": "This paper proposes a novel method for distilling diffusion models into a generative image denoiser at any noise level. The proposed method is based on a theoretical result showing that the unconditional score function implicitly characterizes the score function of denoising posterior distributions at all noise levels. The method further incorporates adversarial training to overcome the performance upper bound imposed by the teacher diffusion model (the one which is being distilled).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*W1: ... additional datasets ...*\n\nPlease refer to W7.\n\n*W2: ... visual comparison  ...*\n\nPlease refer to W8.\n\n*W3: For example ...*\n\nPlease refer to W9.\n\n*W4: In L369 ... & W5: The authors say in L369 ...*\n\nWe respectfully disagree. The tradeoff between PSNR and LPIPS has been extensively demonstrated in prior works. For instance:\n\n- Figure 14 in DAPS and Figure 5 in RED-diff (Mardani et al., 2024) illustrate this tradeoff.\n- Section C.6 in DPS explicitly discusses this phenomenon.\n\nGenerative image restoration methods often prioritize perceptual quality, which emphasizes preserving high-frequency details. This focus can lead to lower PSNR, as PSNR penalize deviations from the mean / MMSE solution. However, these deviations often result in better perceptual quality, as captured by metrics like LPIPS.\n\nThe claim that our method produces results \"closer\" to posterior samples refers to its tendency to avoid regressing to the mean solution that maximizes PSNR. To address the reviewer's concern, we will revise the statement as follows:\n\n*\"Our method tends to generate results that retain more high-frequency details rather than approximate the mean of all possible solutions. This approach typically leads to higher MSE (lower PSNR) but aligns more closely with perceptual quality metrics such as LPIPS.\"*\n\n*W6: I think that evaluating ...*\n\nTo the best of our knowledge, related works such as CD, ECM, EDM2, and sCM primarily use FID as the main evaluation metric, while KID is not commonly reported. Additionally, FD is typically considered an auxiliary metric and is often included in the Appendix. To address reviewer's concern, we provide FD_DINOv2 below:\n\n*Table: FD_DINOv2 on ImageNet-512x512.*\n\n|Method|NFE|FD_DINOv2|\n|-|-|-|\n|sCD-M|2|55.70| \n|sCD-L |2|50.63|\n|NCVSD-M (Ours)|2|59.14|\n||4|48.83|\n\nFD_DINOv2 exhibit similar trends to FID: scaling test-time compute enables our method matches performance of larger-sized sCM models.\n\n*W7: The method is evaluated ...*\n\nFor image generation, we follow the EDM2, focusing on ImageNet-64x64 and ImageNet-512x512, and pretrained EDM2 models for other datasets are unavailable. Moreover, ImageNet-64x64 and ImageNet-512x512 serve as challenging benchmarks for image and latent domain, respectively. So we believe they sufficiently demonstrate our method's effectiveness and generalizability.\n\nFor inverse problem solving, pretraining an EDM2 model on ImageNet-256×256 is computationally costly. For example, training an S-size model at 64×64 resolution takes over 5 days on 32 A100 GPUs (Figure 11(a), EDM2). Additionally, the baseline PnP-DM focuses solely on the FFHQ dataset. While our current results already demonstrate the effectiveness of our method, we have also started running experiments on the ImageNet-256, and we expect to report preliminary results by the discussion stage.\n\n*W8: There are no visual ...*\n\nWe thank the reviewer for the suggestion. For image generation, visual samples are in Appendix D, with the classes align to those used in sCM, enabling visual comparisons.\n\nFor inverse problems, we include additional visual comparisons (https://anonymous.4open.science/r/ncvsd-D396) in the revised version. These visual examples further demonstrate that our method produce much better perceptual quality and preserve more high frequency details compared to baseline.\n\n*W9: The quantitative results ...*\n\nWe emphasize that achieving SOTA on inverse problems is not the primary goal of this paper, nor do we claim that LPIPS is more important than PSNR. Instead, inverse problem solving serves as a proof-of-concept to showcase the plug-and-play probabilistic inference capabilities of our method. Moreover, we already demonstrate competitive performance against well-established diffusion-based methods with significantly fewer NFEs. In contrast, prior works like CM (Song et al., 2023) on image editing provide only visual results without quantitative comparisons. The focus should be on the novelty and conceptual contributions of our approach rather than solely on performance metrics.\n\n*W10: Important quantitative ...*\n\nWe standardize FID for comparing different methods, following EDM2. This metric is consistently reported in baseline methods, making it a convenient choice for benchmarking. In contrast, metrics like Precision or Recall are either not reported, e.g., ECM, or reported only for indicating the performance tradeoffs, e.g., EDM2 and sCM. To address the reviewer’s concern, we include additional Precision and Recall in the Table below. For additional metrics for inverse problems, please refer to our response to W3 from Reviewer 7Tng.\n\n*Table: Precision and Recall on ImageNet-64x64.*\n\n|Method|NFE|Precision|Recall|\n|-|-|-|-|\n|CD|1|0.68|0.63|\n||2|0.69|0.64|\n|NCVSD-M (Ours)|1|0.72|0.60|\n||2|0.73|0.62|\n||4|0.74|0.62|\n\nPrecision is superior, reflecting better quality of generated samples, while the slightly lower recall score reflects the mode-seeking behaviour of reverse KL minimization.", "claims_and_evidence": "1. The authors claim in the abstract \"We evaluate NCVSD through extensive experiments, including class conditional image generation and inverse problem solving.\" But I don't find the experiments to be \"extensive\" at all. For image generation, the authors only consider ImageNet 64x64 and 512x512, and for inverse problems they consider FFHQ 256x256 and distill zero-shot posterior sampling methods.\nI expected to see additional datasets for both image generation and inverse problems.\n\n2. The authors claim in the abstract \"our method outperforms teacher diffusion models and is on par with consistency models of larger sizes.\" While this may seem true from Table 1, only FID is evaluated (which is a highly problematic measure of generative models [1]), and I can't find a visual comparison (both quality and variation) between the methods, not even in the appendix. Measures like FID only serve as indicators of performance, while visual examples matter the most.\n\n[1] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. George Stein et al. NeurIPS 2023.", "methods_and_evaluation_criteria": "Some evaluation criteria are problematic.\n1. For example, the authors claim in the abstract: \"We achieve record breaking LPIPS on inverse problems.\" But LPIPS is only one distortion measure. What is it about LPIPS that make it such a desirable measure to optimize? They authors don't explain this.\n2. In L369 (right) the authors claim: \"We observed that the PSNR performance of our method does not achieve the best performance as LPIPS, which can be attributed to the distortion-perception trade-off (Blau & Michaeli, 2018), indicating that our method tends to produce results that is closed to the posterior samples rather than the mean of all possible solutions.\" To my understanding, there is no tradeoff between PSNR and LPIPS: If the PSNR is equal to infinity (namely, MSE=0), then the LPIPS is equal to 0 (minimal, i.e. optimal). Both PSNR and LPIPS are distortion measures. The perception-distortion tradeoff talks about a tradeoff between any distortion measure (e.g., PSNR, LPIPS, SSIM) and the statistical distance between the distributions $p_X$ and $p_\\hat{X}$, not about some tradeoff between PSNR and LPIPS.\n3. The authors say in L369 (right) that achieving lower LPIPS indicates that their method produce samples that are \"closer\" to posterior samples. I don't understand this claim. There is no link between achieving lower LPIPS and producing results that are closer to posterior samples. I find this argument to be wrong and misleading.\n4. I think that evaluating generative methods only with FID is insufficient. I'd expect incorporating additional evaluations, such as KID, FD in feature space of self-supervised methods [1], etc.\n\n[1] Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. George Stein et al. NeurIPS 2023.", "theoretical_claims": "The math and theoretical claims in the paper seems okay to me.", "experimental_designs_or_analyses": "No code is attached. Couldn't verify the experiments. But the paper describes the experimental settings quite well.", "relation_to_broader_scientific_literature": "The main contribution of this paper is a distillation method for diffusion models, allowing few-step image generation. This task has been addressed in prior work, such as by consistency models (which the authors compare with).", "essential_references_not_discussed": "I can't think of any essential references which are not discussed.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is overall well written and easy to follow.\n2. I find the idea to distill the diffusion at different noise levels to be interesting and useful (e.g., for solving inverse problems).\n\nWeaknesses:\nThe experimental results are not very convincing:\n1. The method is evaluated on limited datasets (ImageNet for image generation, and FFHQ for inverse problems). \n2. There are no visual comparisons at all with other methods in both image generation and inverse problems. Visual results and comparisons are more important than quantitative results - since the paper is dealing with image processing.\n3. The quantitative results on inverse problems seem inferior: The method achieves better LPIPS but worse PSNR, and it's not clear to me why LPIPS is necessarily more appropriate or more important than PSNR. If optimizing LPIPS is that important, why do we need a diffusion model or to distill one? Why not just training a model to minimize LPIPS?\n4. Important quantitative evaluations are missing. E.g., evaluating image generation with additional measures such as KID, Precision and Recall, and evaluating the perceptual quality of image restoration algorithms with divergences (e.g., FID) and no-reference quality measures (e.g., NIQE). Considering additional distortion measures such as SSIM could also show better whether the proposed method is truly superior or not.", "other_comments_or_suggestions": "I recommend to add a figure depicting the method, both training and inference, to improve the paper's readability and clarity even further.", "questions_for_authors": "I have no particular questions for the authors. But they are welcome to address the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Noise Conditional Variational Score Distillation for distilling pretrained diffusion models into generative denoisers. The method's key insight—that unconditional score functions implicitly characterize denoising posterior distributions—is theoretically sound and enables computational efficiency while maintaining quality. While reviewers expressed varying opinions about evaluation methodology, with concerns about metric choices and dataset limitations, the authors provided rebuttals with additional metrics and visual comparisons. Reviewers acknowledged the work's strong theoretical foundations, and the demonstrated benefits of fast one-step generation with iterative refinement capability represent meaningful contributions to the field. Though some experimental limitations remain, particularly in the fairness of certain comparisons (Reviewer s1LA remained unconvinced about fairness in comparison), the theoretical rigor, and practical utility of the approach outweigh these concerns. I recommend weak accept, as this work advances our understanding of efficient score distillation for diffusion models.
Decision: Accept (poster)