A Mixture-Based Framework for Guiding Diffusion Models
Authors: Yazid Janati
Badr MOUFAD
Mehdi Abou El Qassime
Alain Oliviero Durmus
Eric Moulines
Jimmy Olsson
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Models
Guidance
Inverse Problems
Monte Carlo methods
Primary area: probabilistic_methods

Abstract: Denoising diffusion models have driven significant progress in the field of Bayesian inverse problems. Recent approaches use pre-trained diffusion models as priors to solve a wide range of such problems, only leveraging inference-time compute and thereby eliminating the need to retrain task-specific models on the same dataset. To approximate the posterior of a Bayesian inverse problem, a diffusion model samples from a sequence of intermediate posterior distributions, each with an intractable likelihood function. This work proposes a novel mixture approximation of these intermediate distributions. Since direct gradient-based sampling of these mixtures is infeasible due to intractable terms, we propose a practical method based on Gibbs sampling. We validate our approach through extensive experiments on image inverse problems, utilizing both pixel- and latent-space diffusion priors, as well as on source separation with an audio diffusion model. The code is available at \url{https://www.github.com/badr-moufad/mgdm}.

A Hixture-Based Framework for Guiding Diffusion Models
Yazid Janati" Badr Moufad
Mehdi Abou El Qassime' Jimmy Olsson
Alain Durmus'
Eric Moulines'
Abstract
associated with inverse problems. Prior knowledge about the signal of interest, often represented through samples from its underlying distribution po, can be leveraged to train a generative model p that acts as a prior. By combining it with the conditional density go (y (x) of the observation given the signal, deduced from the form of the inverse prob- lem hand, can compute the posterior distribution. Samples drawn from this posterior encapsulate plausible solutions that harmonize prior knowledge with the observed data. One straightforward approach to approximate sam- pling from the posterior distribution involves constructing a paired dataset i.i.d. signals and observations, (X;, Yi); where X; ~ Po and Y; ~ go(- and learning a direct mapping (Dong et al., 2015) or generative model (Ledig et al., 2017: Isola al., 2017). The latter, when queried with multiple independent noise samples alongside an obser- vation, generates diverse set of potential reconstructions. However, this approach is inherently task-specific, deliv- ering reliable reconstructions only when the conditional distribution of the observation remains unchanged at test time. As a result, it cannot straightforwardly adapt to unseen tasks with the same prior. Adaptation to new task can only be achieved by retraining new generative model.
Denoising diffusion models have driven signif- icant progress in the field of Bayesian inverse problems. Recent approaches use pre-trained dif- fusion models as priors to solve a wide range of such problems, only leveraging inference-time compute and thereby eliminating the need train task-specific models on the same dataset. To approximate the posterior of Bayesian in- verse problem, diffusion model samples from a sequence of intermediate posterior distributions, each with an intractable likelihood function. This work proposes novel mixture approximation of these intermediate distributions. Since direct gradient-basee based sampling of these mixtures is in- feasible due to intractable terms, we propose a practical method based on Gibbs sampling. We validate our approach through extensive experi- ments on image inverse problems, utilizing both pixel- and latent-space diffusion priors, as well as on source separation with an audio diffusion model.
An increasingly popular approach consists in learning a generative model only for the prior po, and then leveraging inference-time compute to solve any inverse problem for which the likelihood function x +> go x)) provided in closed form. This strategy eliminates the need for expen- sive and inefficient task-specific training. Initially explored with generative models such variational autoencoders and generative adversarial networks (Xia et al., 2022), this framework has recently been extended to denoising diffu- sion models (DDMs) (Song et al., 2021; Kadkhodaic & Simoncelli, 2020; Kawar et al., 2021; 2022; Chung et al., 2023; Song al., 2023a; Daras al.), which are the focus of the present paper.
1 Introduction
Inverse problems occur when a signal X of interest must be inferred from an incomplete and noisy observation Y, challenge frequently encountered in diverse fields such weather forecasting, image reconstruction (e.g., tomogra- phy or black-hole imaging), and speech processing. Such problems are typically ill-posed, making it essential to in- corporate additional constraints, regularization techniques, or prior knowledge to arrive at meaningful and realistic solutions.
The Bayesian framework, in conjunction with generative modeling, offers a systematic approach to the challenges
DDMs (Sohl-Dickstein al., 2015; Song & Ermon, 2019; Ho et al., 2020) achieve state-of-the-ard generative performance across a wide range of domains. At their core a forward noising process that transforms the data distribution po into a Gaussian distribution. A generative model then learned by reversing this noising process. With specific parameterization of the backward process,
Equal contribution 'Ecole polytechnique *KTH Royal Institute of Technology. Correspondence to: Yazid Janati, Badr Moufad <first.last <dirs:lass@pooyyccnnique.dd polytechnique..du.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Mixture-Based Framework for Guiding Diffusion Models
which converts noise into data samples, training the generative model reduces approximating denoisers for each noise level introduced during the forward process. Recent methods for training-free posterior sampling aim to approximate the denoisers for the posterior distribution, enabling the use of diffusion models for sampling (Ho et al., 2022: Chung et al., 2023; Song et al., 2023a). A posterior distribution denoiser can be decomposed into two terms: the prior denoiser the same noise level (provided by pre-trained diffusion model) and the gradient of the log-likelihood of the observation conditioned on the current noisy sample. The latter term, which is intractable, is what guides the samples during the denoising process towards the posterior distribution. Various approximations for this gradient term have been proposed. However, they are often crude and require significant adjustments and heuristies to ensure stability and satisfactory performance. When applied to latent diffusion models, they often demand additional, model-specific adjustments (Rout et al., 2024).
sively less smoothed distributions (p:)=_r- starting from a highly smoothed prior pr and ending the data distribution Po. For all s, E [] with s < t. define the noising Markov transition kernels
St s ( [x,= N(xii (as/as) Hidld)
(I)
where (au) is monotonically decreasing with og = 1, oT & 0, and of.- = 1 - (az/ (o.. Each smoothed distri- bution is a noised version of po and has density pr(x)) == I Grjo (x. (x)) dxp. The final distribution pr is close to N (Od; Iz). Moreover, define the backward Markov tran- sition kernels Paje (x,|xx (xx) x (x,)) (x|[x,) with 8 < t. Note that for all & < 8, the backward transitions satisfy
Pxje(xe(x.) = / (xx]
dx.
(2)
Consecutive distributions Pt and Pc+1 are linked through
the identity pt+1 (X = =J 90+1je(Xe+1 (x))pe (x,) dx,- Hence, given sample X1+1 ~ Pi+1, Xt Paa+1 ( (+1)))) is an exact sample from pt. This procedure defines a genera- tive model, in the sense that the last state Xo of the Markov chain (x+)_TT where the initial state Xy is sampled from pr, is a sample from po-
Our contribution. this paper, we present a principled method that circumvents these issues by introducing new approximation of the likelihood term, paired with sam- pling scheme based on Gibbs sampling (Geman & Geman, 1984). Our key observation is that multiple approximations can be derived for each likelihood term at a fixed noise level using a simple identity that it satisfies. However, the scores of these new likelihood approximations are not available in closed form, preventing us from deriving a direct posterior denoiser approximation by combining, through a mixture, the different likelihood approximations. We overcome this limitation by constructing a mixture approximation of the intermediate posterior distributions defined by the diffusion model for the original posterior. Our algorithm, MIXTURE- GUIDED DIFFUSION MODEL (MGDM), proceeds by se- quentially sampling from these mixtures using Gibbs sam- pling. This is enabled by a carefully designed data augmen- tation scheme that ensures straightforward Gibbs updates. A key advantage of our approach is its adaptability to available computational resources. Specifically, the number of Gibbs iterations acts as a tunable parameter, allowing substan- tial improvements with increased inference- time compute. MGDM demonstrates strong empirical performance across 10 image-restoration tasks involving both pixel-space and latent-space diffusion models, well as in musical source separation, even matching the performance of supervised methods.
However, simulating the backward transitions is impractica- ble in most applications, so the following Gaussian approxi- mation is used in practice. First, for E [1, II. define the conditional density of X, given Xy and X₁:
(x, Xx)
(3)
= -N(xs:7H| + - xe, [,, [a),
where Taja == 07 / and 0 == == o2) //// 1/00 Next, define by D == fxo Puje+1 P conditional expectation of Xo given X = xx-1 (referred to as the denoiser). Denote by D2+1 a parametric approxi- mation of D++1- Following Ho et al. (2020) and given an approximate sample X from pt+1, sampling from the bridge kernel 1 (Xe+1), where xo is re- placed by the estimate sample from pt- The complete sampling process proceeds as follows: first, Xy ~ .N(0,,, La); then, recursively, for every 2 1, X.
dxo the
yields an approximate
pe P ( (-1X+41), where for all < 4,
(x. == o (x, (D.())), x().
(4)
The final sample is defined as X, :- Df( and serves as an approximate sample from po- The parametric approxi- mations of the denoisers are trained by minimizing, with re- spect to the parameter 8, Lę denoising loss across all time steps. Finally, using the Tweedie formula (Robbins, 1956), we obtain the identity D, = a, (xx+oVV log (xx))). Consequently, the trained denoisers not only serve as gener- ative models but also provide parametric approximations of the score functions V log (x:)..
2 Background
2.1 Diffusion models
DDMs define a generative process for a data distribution Po on I2d by sequentially sampling from a series of progres-
2

---

A Mixture-Based Framework for Guiding Diffusion Models
2.2 Training-free guidance.
This shows that in order estimate Dÿ we only need to estimate V log ge (yl-), we already have access pre- trained parametric approximation of D.. A widely used ap- proximation of this likelihood term (Ho et al., 2022; Chung et al., 2023), which we will also use in the next section, is
After training diffusion model for the data distribution Po. it can be leveraged through guidance to address vari- ous downstream tasks without the need for additional fine- tuning. This line of research was pioneered in the seminal works of Song& & Ermon (2019), Kadkhodaie & Simoncelli (2020), Song et al. (2021), and Kawar et al. (2021), where the sampling process described in the previous section is adapted on-the-fly fly to address Bayesian inverse problems. In this setting, the user observes a realization y of random variable Y E IRdy, assumed to be drawn from the distri- bution with density (y) := - ( xx) po (x) dx, where
(y|x,) g(y
(9)
and amounts to approximating the posterior distribution Poj{([(xs) with a Dirac mass at D2 (xx). which we express as Pojo(- (xxx = (xf(x.) with a slight abuse of notation. To im- prove the quality of the sample, V log of (y/xx) is rescaled with a suitable weight (possibly depending on x:); see (Ho et al., 2022, Equation 8) and (Chung et al., 2023, Algorithm 1). We emphasize that the rescalings generally used are only heuristic. Compared to previous works, methods that per- form guidance using the approximation (9) incur additional computational overhead due to the calculation of vector- Jacobian product when evaluating V log 9" (//x). Never- theless, subsequent works using this approximation have shown remarkable improvements in performance across var- ious applications; see for example (Song et al., 2023a; Rozet & Louppe, 2023; Yu et al., 2023; Wu al., 2023; Jiang et al., 2023; Rozet al., 2024; Moufad al., 2024).
9o (x)xx is a likelihood term that encapsulates the knowledge of the forward model. A typical example is inverse prob- lems with Gaussian noise, i.e. go x) = N(y; (x), My), where A Rd - Rd and Σy a covariance matrix. The objective is to recover plausible underlying signals x, for which prior information is encoded in po- This recovery is achieved by sampling from the posterior distribution
T (xo) xx go (x) (xo)
common approach to constructing a sampler for this pos- terior distribution is adopt the diffusion model framework by sequentially sampling from the smoothed distributions ......,,? nY, duced in the previous section:
which are defined analogously to those intro-
3 Guidance with mixtures
We now present our main contribution: a novel density approximation of the smoothed posteriors qÝ.. Since their scores are intractable, gradient-based samplers cannot be directly applied. Thus, we develop Gibbs sampling scheme targeting data augmentation of our smoothed posterior approximation, marking our second key contribution.
क (xx) / 9000 (xx((xx (xo) dxn-
(5)
Following the derivations above, sampling these distribu- tions backwards in time is feasible provided that the condi- tional denoisers (Dt 1 are accessible. Each conditional denoiser is defined by
In the next two sections we develop an algorithm for the ideal generative model, i.e., we assume that we have at hand the true marginals Pe and backward transitions pstei then, in Section 3.2, we provide a practical implementation involving the learned model.
(x,) := / Xo T (x)lxx) dxo;
where the conditional posterior Th (Xo (x))xx) is given by (xolxx) x nh (xo) (xe xx)). By analogy with the
smoothed distributions defined for the prior, we obtain that
3.1 Guidance approximation
n. (xx) x f go (yooy/o))ee o(xe|x0))o dxo x ge (x())x))pxxxx))
We begin by extending the likelihood approximation in (9) introduced by Ho et al. (2022); Chung et al. (2023). First, note that by combining (2) with (7), we find that g (yl.) satisfies
(6)
where
ge (y(xxx) = / ( (x))s/ (xx) dx..
f90() ge (y(xx))) = f go(y) xo (oo)Po((xxlxxx) dxo:
for all t E [1, T and s E I0, -o -1. Thus, we obtain 1 different approximations of gx(xl-) by simply setting, for 11,2-1]. 1].
and we used that (o(x))99o(xx xx))= = Poje (x) (x... Next, using the Tweedie formula, the posterior and prior denoisers can be related as
:= / (x(1))
D. (x.) = (x.) + ar of olglogg(yy)))))
(8)
g ff(x)x)) ==
(x(y)x))Pos xx)) (xa (xx) dxe, t t2 (10)
3

---

A Mixture-Based Framework for Guiding Diffusion Models
where 0. (yl-) denotes the counterpart of (9), with the learned denoiser D replaced by the true denoiser D-- In contrast to o8 (yl-) in (9), the scores of these approxima- tions remain intractable even when the approximate model is used, as they involve an intractable integral. Instead, we take a different approach and use of (yl-) to define density approximations
Specifically, we use Gibbs sampler (GS) (Geman & Ge-
man, 1984; Casella & George, 1992; Gelfand, 2000), which, in this context, constructs a Markov chain (Xg XT, Xf)x() ree having T as its stationary distribution. Denote by T 00,0 Tyy "joo.s
its three full conditionals given by and * "Os,t
0,(y (x
T T (Xs (20: xx)
99 (,,,,,,))) 12, (X/
Gf (y (x)) J O7 (y xx Pr (x4) dx;
T 600 (Xr (xo Xs) = -Qt|s( (x|)s),
(x.)
(11)
Toss (Xo) x xe) = (xo
of the smoothed posteriors ... Since we have f such approximations, we consider a weighted mixture approxi- mation of ..े" defined, for 2 as
The proof of this fact is postponed to Appendix A.2. Then, one step of the associated (deterministic scan) GS is de- scribed in Algorithm 1.
m (xx)
Σ (xi),
(12)
Algorithm Gibbs sampler targeting (13)
1: Input: (Xt, Xy, X;)
2: draw X ~ TÝ (- (1x6;
where (wf) are time-dependent weights and x-1 1 with wf 2 0. Then, to sample approximately from "o we can use a sequential sampling procedure that runs through the intermediate distributions ..,,.., AY. Similar sequen- tial sampling procedures from posterior sequences differ- ent from (mī) ), have also been utilized in previous works. For instance, Wu et al. (2023); Rozet & Louppe (2023) use #% (xx)
wi m.
draw X; q. (xx+tt)
/ //noising //dencising
4: draw X ~ Poj (1)+1))
Since (13) admits मं as marginal, the process (XT) IEE will, at stationarity of (X%, )rE),,, have #े marginal distribution. We provide basic background on Gibbs sam- pling in Appendix A. and refer the reader to (Casella & George, 1992).
Our approach differs from these prior works by employing a mixture- based formula- tion with non-standard approximations of n.- However, sampling from n. remains a non-trivial challenge. In- deed, a naive procedure would consist in sampling an index Categorical (f [=])(=) and then use an approximate sampler only for Rς. However, we must address the in- tractability of both (-) and its score. In the next section, we propose method that fully overcomes these challenges. The discussion on selecting the weight sequence (wi))=] is postponed until after presenting the algorithm, more specifi- cally at the beginning of Section 5.
It is clear from Algorithm 1 that only the update of Xς depends the observation y, while the updates of the remaining components are sampled via (i) a noising step involving the forward transition (I), which can be performed exactly, and (ii) denoising step involving the prior diffusion model, which can be approximated using the pre-trained model.
Finally, to target the mixture (12), we first sample the mix- ture index s Categorical (wf)(=]); which determines the component of the mixture #/ (xe). Next, we apply Al- gorithm 1 R times to update the remaining coordinates, treating s as fixed, and output the result X²S". Note that an alternative to our method would be to consider a Gibbs sampler for which one of its marginal is directly the mix- ture (12) incorporating also the mixture index s as a state. However, this would then require sweeping over all states (Xo, X,), rendering it computationally expensive and impractical. We discuss other possible data augmentations and their limitations in Appendix A.4.
3.2 Data augmentation and Gibbs sampling
We first detail how to sample from a single component #t of the mixture (12) for given E [22 7] and 8 E 11, Consider first the extended distribution
0.s.t Xx. xt)
x &Poo (x xx, (xx
(13)
From the definitions in (12) and (10) it follows that #e is the x--marginal of (13), i.e.,
3.3 Practical implementation
For simplicity, we present the algorithm in the case where we progressively sample from each ny for E [2, 2J. In practice however, we subsample a small number K of timesteps (t)) with t > 1 and tk = T, and apply the algorithm only to (# )
(xx)) = (Xo, xx) dx,, dx,.
To sample approximately from मÿ. we employ a sampler tar- geting ni nÿ,,, and retain only the co-oonidie of its output.
4

---

A Mixture-Baser Framework for Guiding Diffusion Models
Algorithm 2 MIXTURE- DIFFUSION MODEL
defined in (3). This step corresponds the Gauss VI rou- tine in Algorithm 2 and is detailed in Appendix A.3. Re- garding the initialization of the GS for nt, we use the out- put of the previous GS targeting # see Lines 6 and in Algorithm 2. We maintain a running variable Xi. which is iteratively updated and serves as the initializa- tion for the other variables at the beginning of cach loop iteration. It is also the output of the algorithm. Indeed, note that the last distribution to which we apply the GS is 1º.1. (Xo, X1, X2) of which the marmmnginal is proportional to Po(Xx) f 9' (ylx1 (o (xo) dx1. Since the Gaussian density (110(1|xo) has a very small variance and Ø1 (yl.) P (-))). we may assume that f 91 (y/x)) 911o (x1 [xo]dx; R go (y (Xo) and hence that the posterior Ty of interest is ap- proximately the Xy- -o-marginal of the last extended distribu- tion. As result, we can take the Xn-coordinate of the output of the last GS, which is Xo and hence X5. mate sample from TÝ. In the first row of Figure I we display the evolution of X throughout the iterations with DDM pre-trained on the FFHQ dataset. It is seen that the algo- rithm reaches a plausible reconstruction y rather fast, at = 800 with T = 1000, and then spends the remaining iterations refining the details. As a comparison, the DAPS algorithm proposed by Zhang et al. (2024), which displayed in the second row, also maintains running variable at time 0 that serves as output the algorithm.
1: Input: Timesteps (t:) with t1 > and f = T.
Gibbs repetitions R. DDPM steps M, gradient steps G, probabilities { 1iK,t=1
2,6c-1
2: Xex =O, Ia)
3: Xo f D (Xex): Xi= + Xo
4: for i= = K to 2 do
5: $ ~ Categorical ful, )
6: Xo Xx
7:
Xt. ~ 11,00,t (118 X X
8:
for r- = =1 to R do
9:
X. + Gauss_ VI(Xo, Xu, 8, G) //see A.3
10:
Xo t DDDMDDP s, M)
11:
X (,,10(1XX,
12:
end for
13:
X5 < Xo
approxi-
14: end for
15: Output: X5
DAPS MGDM
Figure 1: Evolution of Xo throughout the iterations for MGDM and DAPS (Zhang et al., 2024).
4 Related works
Alternative likelihood approximations. In addition to this work, several other papers introduce alternative approxi- mations of g: (yl-). (Song et al., 2023a) proposes Gaussian approximation of Poje with mean given by the denoiser Dř and covariance being left hyperparameter. For linear in- verse problems with Gaussian noise, the likelihood 90 (yll) can be integrated exactly against this Gaussian approxi- mation, providing an alternative approximation of (y(y)-). Finzi et al. (2023); Stevens et al. (2023); Boys et al. (2023) use that the covariance of ( is proportional to the Jacobian of the denoiser (Meng et al., 2021). Computing the score of the resulting likelihood approximation, for linear inverse problems, is prohibitively expensive. To mitigate this, these works and subsequent ones assume that the Jaco- bian of the denoiser is constant with respect to x,. Despite this simplification, the score approximation still involves an expensive matrix inversion. Boys et al. (2023) use diago- nal approximation of the covariance based on its row sums. Rozet et al. (2024) use conjugate gradient to perform the matrix inversion efficiently. For general likelihoods go (y(y) (yl-), Song et al. (2023b) use Gaussian approximations of Song et al. (2023a) to estimate (yl-) using a standard Monte Carlo approach. For latent diffusion models, Rout et al. (2024) apply the approximation (9) together with a regu- larization term that penalizes latent variables deviating from
The denoising step in Algorithm 1 can be approximated by sampling from the learned diffusion model. To re- duce runtime, we again subsample a small number of timesteps {s, ) c [0, - 1]. ensuring that 80 = 0 and SM = s. We then generate (X,,), by sampling iteratively Xx, ~ p.. (+Ix,.,) ,) and retaining only X This oper- ation is referred to as DDPM(- s, .M) ) on Line 10 in Algo- rithm? 2. As for the step involving #N we follow Moufad et al. (2024) and sample approximately by fitting a Gaus- sian variational approximation. More specifically, given (Xo, xt), we draw from the Gaussian variational approxi- mation A //o := A (Hojooe, diag(efri..)) where the param- eters &sjo,t := (Pajo,t; pa) E TRdd x R.d are obtained by optimizing the right-hand side of
KL slo.c (1 xx))
E. [log (y) {})] KL(A9) aio, 1=)o,e x:)),
where X ~ A "0.r" The gradient of this quantity can be estimated straightforwardly using the reparameterization trick (Kingma& & Welling, 2013). The initial parameters Pe are set to the mean and covariance of ( (: [xo: xx)
5

---

A Mixture- Based Framework for Guiding Diffusion Models
Olocsation
Gromdirath
Sample 1
Sumpl
Sungle
Sumple
Olscr-ation
Grounitent
Sumple 1
Sauple 2
Sample 3
Sumpe:
Top maakk
Mosian Gansin Conter
Ucater musk
Hail musk:
SR (x10)
Batton
Figure 2: MGDM sample images for various tasks on ImageNet (left) and øF#2 (right) datasets.
fixed points of the decoder-encoder composition. Moufad al. (2024) propose general method for both vanilla and latent space diffuusinn models. At step of the diffusion process they first sample, an intermediate timestep < t, a state conditionally on y with the upproximation (9), be- fore returning back to the timestep f. In Appendix A.5 we explain in more details how the present work differs from this method.
On the other hand, sampling from #7 (1xxo) can be per- formed exactly when go(y)-) is the likelihood of a linear inverse problem with Gaussian noise, since (r (x)) is a Gaussian distribution. For more general problems, this step can be implemented using MCMC methods; see e.g. (Xu Chi, 2024, Algorithms & 4). Compared our algorithm, PNP-DM has a lower memory footprint because it does not require vector-Jacobian product as it uses the likelihood go (yl-) instead of 9, (yl-). However, as we show in the next section, this comes at the cost of performance, especially when using latent diffusion models. The REPAINT algo- rithm (Lugmayr al., 2022), which applies to noiseless linear inverse problems, uses noising and denoising steps repeatedly and can also be viewed cific Gibbs sampler. Finally, the recently proposed DAPS (Zhang et al., 2024) can also be related to a Gibbs sampler targeting specific sequence of distributions. Further details and comparisons to MGDM are provided in Appendix A.5.
Asymptotically exact methods. Trippe al. (2023); Wu al. (2023); Cardoso al. (2024); Dou & Song (2024); Corenflos et al. (2024); Li et al. (2024) use the sequential Monte Carlo (SMC) framework to construct an empirical approximation of the posterior distribution represented by N samples. The samples undergo transitions guided by user-defined updates, are reweighted using an appropriate importance weight, and are subsequently resampled fo- computational effort the most promising candidates. The performance of these methods improves by scaling the number of samples N, which impacts both the mem- ory requirement and compute time. As evidenced by the experiments in the next section, our method improves by increasing the number of Gibbs steps, which impacts only the runtime.
variant of spe-
5 Experiments
We evaluate MGDM on image inverse problems using both pixel-space and latent-space diffusion, as well as musical source separation tasks. For the pixel-space diffusion and the audio diffusion model, we compare MGDM against seven competitors: DPS (Chung et al., 2023), PGDM (Song et al., 2023a). DDNM (Wang et al., 2023), DIFFPIR (Zhu et al., 2023), REDDIFF (Mardani et al., 2024), DAPS (Zhang et al., 2024), and PNP-DM (Wu al., 2024). In the latent space setting, we benchmark against, four competitors: PSLD (Rout et al., 2024), RESAMPLE (Song et al., 2024). DAPS (Zhang et al., 2024), and PNP-DX DM (Wu et al., 2024). In Appendixes B.2-B.4, we provide a complete formal de- scription of the parameters of our algorithm as well as the implementation details of each competitor and its hyperpa- rameters. We emphasize that have tuned the parameters
Gibbs sampling approaches. The recent works (Wu et al., 2024; Xu & Chi, 2024) on PNP-DM also propose Gibbs compling-inspired algorithm. They consider (within the variance exploding framework) the distribution sequence (#+)) TT= where each distribution #t (xx) & go (x) the m--manginal of the extended distribution
* (Xe, x+) x go (y x (1)x)))ooo (o (xx)oxxx))))(xx, (xo)
As its full conditionals are #y (Xo #1jo (x, (xo) & go(y|x, xx 1q (x)), the GS targeting this
[x,) = (x)lx) and
joint distribution also proceeds with prior denoising step.
6

---

A Mixture- Barer Framework for Guiding Diffusion Models
Table Mean LPIPS for linear/nonlinear imaging tasks on the FF#Q and ImageNet datasets with σy 0.05. Lower metrics are better.
ImageNet
Task
MGDM
DPS
PGDM
DDNM
DISEPIR
REDDIFF
DAPS
PNP-DM
MGDM
DPS
PGDM
DDNM
DIFEPIR
REDDIFF
DAPS
PNP-DM
SR (x:4)
0.09
0.09
0.30
0.15
0.10
0.39
0.16
0.10
0.26
0:25
0.56
0.34
0.31
0.57
0.37
0.66
SR (x166
0.24
0.23
0.35
0.23
0.40
0.29
0.62
0.50
0.85
0.75
1,05
Dox, inpainting
0.10
0.17
0.17
0.12
0.34
0.19
0.13
0.18
0.23
0.35
0.29
0.26
0.30
0.36
0.30
0.42
Half mask
0.20
0.24
0.25
0.25
0.28
0.225
0.32
0.34
0.38
0.40
0.46
0.40
0.54
Canssion Deblar
0.12
0.17
0.87
0.20
0.12
0.24
0.24
0:14
0.50
0.37
1.00
0.45
0.53
0.59
0.76
Motion Deblur
0.0P
0.17
0.22
0.19
0.21
0.22
0.40
0.39
0.42
0.52
JPEG (QF- 2)
0.14
0.34
1,12
0.22
0.29
1.32
0.49
0.45
0.56
Phase retrieval
0.11
0.40
0.26
0:14
0.34
0.55
0.62
0.61
0.50
0.66
Noulinear deblur
0.51
0.68
0.51
0.66
0..41
0.49
HDR
0
0.40
0.20
0.10
0.19
0.21
0.84
0.19
0.14
0.31
of our algorithm per dataset and not per task.
evaluation is done on subset of 300 validation images per dataset. For FF we use the first 300 images, while for ImageNet, we randomly sample 300 images to avoid class bias. We report the LPIPS metric (Zhang et al., 2018) in Ta- bles and and defer the complete tables with FID, PSNR and SSIM along side 95% confidence interval to Table 6, Table 7, and Table 8. For the phase retrieval task specifically, we draw 4 samples for each algorithm and keep only the best scoring one in terms of LPIPS. A similar strategy is used in (Chung et al., 2023; Zhang et al., 2024; Wu et al., 2024). Across table rows, we highlight the best value in the 2nd best in and 3rd best in We provide a large gallery of exemplar reconstructions in Appendix B.9, Aside, we also extend our evaluation to higher-noise setup and Poisson- noise likelihood in Appendix B.8 and Appendix B.9.
Index sampling and Gibbs steps. During the first 75% of the diffusion process, at timestep 44 we sample the index s from Uniform(r: [T, [1=1] with r = 10 to mitigate instabili- ties. In the final 25% of the steps we set = f,-1 as this yields slightly improved results. On the image inverse prob- lems we use 100 diffusion steps with R = 1 Gibbs step. On the source separation task we use 20 diffusion steps with Rt = 6 Gibbs steps. The choice of weight sequence {wf [t= plays an important role for the algorithm's 's per- formance. Intuitively, it holds that that of (yl-)) R (yl+)) when s Rs O, suggesting that for all & [2. 2]. the weights should be set to 0 beyond certain threshold to ensure that is sampled near O. We found, however, that this strategy does not yield good performance for our algorithm. Instead, sampling the index uniformly leads to faster mixing. On tigh-dimensional image datasets, we observe that when s is consistently sampled near 0 at all iterations, the algorithm struggles to overcome the errors that accumulate at initial- ization, leading to suboptimal reconstructions. We provide both quantitative and qualitative evidence in Appendix B.I.
Results. Our method with a single Gibbs step consistently achieves competitive performance, ranking first on most tasks and standing out as the only approach to maintain ro- bust performance across all tasks. On latent FF#Q, we out- perform RESAMPLE and PSLD, both of which are specif- ically designed for latent problems, while our method is applied seamlessly off-the-shelf without any adaptation to latent diffusion. Qualitative comparisons in Figure 2 and in Appendix B.9 reveal that our method provides diverse, visu- ally coherent and sharp reconstructions. In contrast, DAPS, DDNM and DIFFPIR, despite scoring higher in PSNR and SSIM on some tasks, provide less coherent reconstructions; see Appendix B.6 for discussion and examples. Finally, a key strength of our algorithm is its ability to improve perfor- mance by increasing the number R of Gibbs steps. This is demonstrated for the most challenging task, phase retrieval, in Figure 3. In this experiment, we compute the LPIPS using a single sample per image (instead of four) and achieve a threefold reduction in average LPIPS simply by inereasing the compute time in the right direction. Indeed, increasing the number of gradient steps brings only marginal gains in this case whereas increasing the number of Gibbs steps leads to significant performance gains.
Images. We evaluate our method on diverse set of six linear inverse problems and four nonlinear inverse problems with three different image priors with 256 x 256 resolution: the pixel-space FFH model Choi et al. (2021), the latent- space FFHO of Rombach et al. (2022), and the ImageNet model of Dhariwal& & Nichol (2021). We use the noise level Ty 0.05 for all tasks. The linear problems include image inpainting with two masking configurations: a 150 x 150 central box mask and a malk- covering the right side of the image; Super Resolution (SR) tasks with upscaling fac- tors of x4 and x 16; Gaussian and motion deblurring, both using a kernel size of 61 61 following the experimental setup described by Chung al. (2023, Section 4). For the nonlinear setting, we consider JPEG dequantization with quality factor of 2%, implemented using the differentiable operator proposed by Shin & Song (2017); phase retrieval with an oversampling factor of x2; non-uniform deblur- ring using the operator introduced by Tran et al. (2021); High Dynamic Range (HDR) reconstruction following the setup detailed in Mardani et al. (2024, Section 5.2). The
Source separation. We now consider a linear inverse prob- lem with an audio diffusion prior that generates four de- pendent instrument soundtracks: bass, drums, guitar, and
7

---

A Mixture- Based Framework for Guiding Diffusion Models
Table Mean LPIPS for finear/nonlinear imaging tasks on F?HQ dataset with LDM prior and By = 0.05, Lower metrics are better.
Table 3: Mean SI-SDR, on s slakh2100 test dataset. The last row displays the mean over the four stems. Higher metrics are better.
Task
MGDM
RESAMPLE
PSLD
DAPS
PNP-DM
Stems
MGDM
PGDM
DDNM
MSDM
ISDM
DEMUCS312
(x4)
0.22
0.21
0.40
16.41
14.94
19.36
17.16
SR (X:1)
0.30
0.38
0.36
0.52
0.71
Drums
18.07
18.29
18.14
19.05
18.68
20.90
19:61
Box inpainting
0.18
0.22
0.27
0.37
0.31
Guitur
16.68
9.90
12.84
14.38
15.38
14.70
17.82
Half mask
0.26
0.30
0.32
0.49
0.44
Piano 16.17
10.41
12.31
11,46
14.73
14.13
16.32
Gonssian Deblur
0.18
0.16
0.59
0.32
0.32
All
13.77
14.92
14.96
17.27
17.73
Motion Dehlur
0.22
0.70
0.36
JPEG =
0.36
Phase retrioval
0.29
0.39
0.25
0.50
Nonlineur dehlur
0.29
0.33
0.37
0.37
is, to the best of our knowledge, considered to be state-of- the-art. We refer to it DEMUCS512- Finally, since the inverse problem is noiseless, we smooth it by using the likelihood (y)x) N(y; A with oy 10-4, This smoothing is applied consistently across all competitors except the dest-performing versions of MSDM and ISDM, which are tailored for noiseless problems, and DEMUCS$12- The results are reported in Table 3. Due to space constraints, we only show the best performing competitors and defer the complete table to Appendix B.6.
High dynamic rango
0.16
0.12
0.24
0.24
R
R=2
R
4
R::66 6
R 1,6 20-1
6.3
R 1,G
15.45
18.07
18:33
19.88
19989
UPIPS
17.93
18:18
1807
Guitr
12.55
14.73
16.26
1688
16000
Pizzo
11.82
14.34
15:38
18.37
All
14007
16.27
17.00
13.35
BEA8
61
2 # Gibbs steps R
Figure 3: Performance of MGDM as a function of the number of Gibbs steps Rt. The setup R = 1, G > 1 represents MGDM with Ii = 1 and number of gradient steps resulting in runtime equivalent to using
Results. We outperform, average, the other training-free competitors that use the same pre-trained model by a sub- stantial margin. In particular, we outperform the MSDM algorithm of Mariani et al. (2023) as well as ISDM which uses a different model. With R = Gibbs steps MGDM falls short of matching the performance DEMUCSSI2- We found instead that setting R = 1 and using a number of gradient steps ensuring equivalent runtime, as we did for the phase retrieval example, allows to achieve superior perfor- mance; see Figure 3. It is also seen that the average SI-SDR, increases monotonically with the number of Gibbs steps.
Left: Mcan SI-SDR, for multisource- audio separation task on s slakh2100 test dataset. Right: Mcan LPIPS for the phase retrieval task on FFHQ.
piano. The task involves separating the individual sources from mixture y of these four instruments; i.e. denoting by d' the dimension of one instrument soundtrack, the lin- operator is A assume no noise in the measurement and use the audio diffusion model of Mariani et al. (2023). The evaluation is conducted on the publicly available a a kh2 00 test dataset (Manilow et al., 2019) with the scale-invariant SDR improvement (SI-SDR,) metric (Roux et al., 2019). The SI-SDR, metric measures the improvement between the original audio source x; and the generated source x;, rela- tive to the mixture baseline y, i.e. it computes the difference SI-SDR(x, x,)) - SI-SDR y) where
E 12 + Ei=1 X, E Rd', We
6 Conclusion
We have developed novel posterior sampling scheme for denoising diffusion priors. The proposed algorithm pro- ceeds by sequentially sampling, using a Gibbs sampler, from a sequence of mixture approximations of the smoothed posteriors. Our experiments show that MGDM not only matches but often surpasses thee-offthhe-rt performance and reconstruction quality across various tasks. Further- more, we have demonstrated that the Gibbs sampling per- spective allows favorable performance improvement with inference-time compute scaling.
|xx;| 1| +
SI-SDR(X; 8,) = 10 logii
|ax; &; 1|2 +c²
where x _ and e = 10 Following Mariani et al. [x: |-c (2023, Section 5.2), tracks from the test dataset are evaluated using a sliding window approach with 4-second chunks and a 2-second overlap. We report the SI-SDR, metric in Table 3. For this task we compare against three other competing algorithms. First, the best version of the MSDM algorithm in (Mariani al., 2023) which uses the same pre-trained model and is directly comparable ISDM algorithm from the same paper and which relies on separate pre-trained models for each instrument, as well as the Demucs model (Défossez et al., 2019), trained with su- pervision to specifically solve source separation, augmented with 512 Gibbs sampling steps (Manilow et al., 2022) and
This work has certain limitations that open avenues for fur- ther exploration. While we outperform the state-of-the-art on most tasks and remains competitive overall on latent diffusion, we still fall short of what we achieve with pixel- space diffusion. We believe that bridging this gap requires more careful selection of the weight sequence. More broadly, an observation-drivet approach to sampling the index could further enhance MGDM. A second limitation is that our methodology does not extend to ODE-based sam- plers or DDIM, and adapting related ideas to these methods
method. Then, the
8

---

A Mixture-Bascr Framework for Guiding Diffusion Models
an interesting research direction. Finally, like all existing methods relying (9), approach incurs higher mem- ory cost compared to unconditional diffusion. It remains open question whether the vector- vector-Jacobian product can be eliminated without compromising performance.
Corenflos, A., Zhao, Z., Särkkä, S., Sjölund, J., and Schön, T. B. Conditioning diffusion models by explicit forward- backward bridging. arXiv preprint arXiv:2405.13794 2024.
Daras, G., Chung, H., Lai, C.-H., Mitsufuji, Y., Mi-
lanfar, P., Dimakis, A. G., Ye, C., and Delbra- cio, M. survey on diffusion models for inverse problems. 2024. URL hups://giannisdarass github.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
iofpublicatonss/dffttiinn survony___reee pdf.
Défosse7, A., Usunier, N., Bottou, I.., and Bach, F. Music source separation in the waveform domain. arXiv preprint arXiv:1911.1322 2019.
Acknowledgements
Dhariwal, P. and Nichol, Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
The work of Y.J. and B.M. has been supported by Tech- nology Innovation Institute (TII), project Fed2Leamn The work is supported by the Swedish Rescarch Council, project 2024-05680. The work of Eric Moulines has been partly funded by the European Union (ERC-2022-SYG-OCEE.) 101071601). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD01 made by GENCI.
Dong, C., Loy, C. C., He, K., and Tang, X. Image super- resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295-307, 2015.
Dou, Z. and Song. Y. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference Learning Representations, 2024. URL https: /tpp:://///eeeeeeeeeee net wowoommmdd-ppptet::ee0
References
Finzi, M. A., Boral, A., Wilson, A. G., Sha, F., and Zepeda- Núñez, L. User-defined event sampling and uncertainty quantification in diffusion models for physical dynam- ical systems. In International Conference Machine Learning, pp. 10136-10152. PMLR, 2023.
Boys, B., Girolami, M., Pidstrigach, J., Reich, S., Mosca, A., and Akyildiz, O. D. Tweedie moment projected diffusions for inverse problems. arXiv preprint urXiv:2310.06721, 2023.
Gelfand, A. E. Gibbs sampling. Journal of the American statistical Association, 100454) 2000.
Cardoso, G., el idrissi, Y. J., Corff, S. L., and Moulines, E. Monte carlo guided denoising diffusion models for bayesian linear inverse problems. In The Twelfth In- ternational Conference on Learning Representations, net/ forum?
Geman, S. and Geman, D. Stochastic relaxation, gibbs dis- tributions, and the bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelli- gence, (6): 741, 1984.
2024. URL
id=nHESSXvxXWK
Casella, G. and George, E. I. Explaining the gibbs sampler. The American Statistician, 46(3):167-174, 1992.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840-683 2020.
Choi, J., Kim, S., Jeong, Y., Gwon, Y., and Yoon, S. Ilvr: Conditioning method for denoising diffusion probabilistic models. in 2021 ieee. In CVF international conference on computer vision (ICCV), volume 1, pp. 2021.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633 8646, 2022.
Chung, H., Kim, J., Mecann, M. T., Klasky, M. L., and Ye, J. C. Diffusson posterior sampling for general noisy inverse problems. In The Eleventh International Confer- ence on Learning Representations, 2023. URL ht https: //openreview, net/ fwt.m/oormmmm/m dd=OnD 92GAGTOK.
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image-to- image translation with conditional adversarial networks. Proceedings of the IEEE conference on computer vi- sion and pattern recognition, pp. 1125-1134, 2017.

---

A Mixture-Bascr Framework for Guiding Diffusion Models
Jiang, C., Cormman, A., Park, C., Sapp, B., Zhou, Y., Anguelov, D., et al. Motiondiffuser: Controllable multi- agent motion prediction using diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9644-9653, 2023.
Mariani, G., Tallini, I., Postolache, E., Mancusi, M., Cosmo, L., and Rodolå, E. Multi-source diffusion models for simultaneous music generation and separation. arXiv preprint arXiv:2302.022577 2023.
Meng, C., Song. Y., Li, W., and Ermon, Estimating high order gradients of the data distribution by denoising. Advances in Neural Information Processing Systems, 34: 25359- 2021.
Kadkhodaie, Z. and Simoncelli, E. P. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007. 1344:000..1340, 2020.
Kawar, B., Vaksman, G., and Elad, M. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-217699 2021.
Moufad, B., Janati, Y., Bedin, L.., Durmus, A., Douc, R., Moulines, E., and Olsson, J. Variational diffusion pos- terior sampling with midpoint guidance. arXiv preprint arXiv: arXiv:2410.09945, 2024.
Kawar, B., Elad, M., Ermon, S., and Song, J. Denoising diffusion restoration models. Advances in Neural Infor-
mation Processing Systems, 35:23593-23606, 2022.
Robbins, H. E. An empirical bayes approach to statis- tics. In Proceedings of the Third Berkeley Sympo- sium Mathematical Statisties and Probability, Vol- ume 1: Contributions to the Theory of Statistics, 1956. URL https: / pettp:://api.semanttt /tss/CooppsDDD261148811 www.stiesccettooeeeeeeeeeeetttte
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61114 2013.
Ledig, C., Theis, L.,, Huszár, F., Caballero, J., Cunningham, A. Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4681 1-4690, 2017.
Roberts, G. O. and Smith, A. F. Simple conditions for the convergence of the gibbs sampler and metropolis-hastinge algorithms. Stochastic processes and their applications, 49(2):207-216, 1994.
Li, X., Zhao, Y., Wang, C., Scalia, G., Eraslan, G., Nair, S., Biancalani, T., Ji, S., Regev, A., Levine, S., et al. Derivative-free free guidance in continuous and discrete dif- fusion models with soft value-based decoding. arXiv preprint ar Xiv: 08252, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEFE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Gool, L.. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 11461-11471, 2022.
Rout, L., Raoof, N., Daras, G., Caramanis, C., Dimakis, A., and Shakkottai, S. Solving linear inverse problems prov- ably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Manilow, E., Wichern, G., Seetharaman, P., and Le Roux, Cutting music source separation some slakh; A dataset to study the impact of training data quality and quantity. In 2019 IEEE Workshop Applications of Signal Process- ing to Audio and Acoustics ( (WASPAA), pp. 45-49, 2019. doi: 10.100/AASPAA.20.... 99979198877
Roux, J. L., Wisdom, S., Erdogan, H., and Hershey, J. R. Sdr - half-baked or well done? In ICASSP 2019- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630, 2019. doi: 8683855
10. 10.100/CASSSP22000 8888888555....
Manilow, E., Hawthome, C., Huang, C.-Z. A., Pardo, B., and Engel, J. Improving source separation by explicitly modeling dependencies between sources. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 291-295. IEEE, 2022.
Rozet, F. and Louppe, G. Score-based data assimilation. Advances in Neural Information Processing Systems, 36: 40521-40541, 2023.
Rozet, F., Andry, G., Lanusse, F., and Louppe, G. Learning diffusion priors from observations by expec- tation maximization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /tss:///pppeeeeeee net/ forum? id=7v8 00488Fh61550
Mardani, M., Song, J., Kautz, J., and Vahdat, A. A vari- ational perspective on solving inverse problems with diffusion models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview net. htt.:forrumm1111110303333050
10

---

A Mixture-Bacs -aaere Framework for Guiding Diffusion Models
Schneider, F., Kamal, O., Jin, Z., and Schölkopf, B. Musai: text-to-music generation with long-context latent diffu- sion. arxiv preprint. arXiv preprint arXiv:2301.11757, 2023.
Wang, Y., Yu, J., and Zhang, J. Zero-shot image restora- tion using denoising diffusion null-space model. In The Eleventh International Conference on Learning Represen- tations, 2023, URL https: / openreview .nt. forum? i d= eQgMtNTQ.
Shin, R. and Song, D. Jpeg-resistant adversarial images. In NIPS 2017 workshop on machine learning and computer security, volume 1. pp. 8, 2017.
Wu, L.., Trippe, B. L.,, Naesseth, C. A., Cunningham, J. P., and Blei, D. Practical and asymptotically exact condi- tional sampling in diffusion models. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023, URL https: pettp::/opennee /ttp:///pppeeeeeeeeeemme net forum? id=eWKgrlzoRs
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference Machine Learning, pp. 2256-2265. PMLR, 2015.
Song, B., Kwon, S. M., Zhang, 7., Hu, X., Qu, Q., and Shen, L. Solving inverse problems with latent diffu- sion models via hard data consistency. In The Twelfth International Conference on Learning Representations, 2024. URL https:://////..t.. // http://ppprreiieeeee net/ forum? id= johd
Wu, Z., Sun, Y., Chen, Y., Zhang, B., Yue, Y., and Bouman, K. Principled probabilistic imaging using diffusion mod- els as plug-and auu-and--lay priors. Advances in Neural Informa- tion Processing Systems, 37:1 18389-11847777 2024.
Xia, W., Zhang, Y., Yang, Y., Xue, J.-H., Zhou, B., and Yang, M.-H. Gan inversion: A survey, IEEE transactions on pattern analysis and machine intelligence, 45(3):3121- 3138, 2022.
Song, J., Vahdat, A., Mardani, M., and Kautz, J. Pseudoinverse-guiee guided diffusion models for inverse prob- lems. In International Conference on Learning Represen- tations, 2023a. URL https: pettps::/openeee openreview / forum?id=20 1ai---9_sMMMMMMM
Xu, X. and Chi, Y. Provably robust score-based diffusion posterior sampling for plug-and-play image reconstruc- tion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: / //oenrevveew net/ forum? d=SLnnsaaYuul....
Song. J., Zhang, Q.. Yin, H., Mardani, M., Liu, M.-Y., Kautz, J., Chen, Y., and Vahdat, A. Loss-guided diffu- sion models for plug-and and-glay controllable generation. In International Conference on Machine Learning, pp. PMLR, 2023b.
Yu, J.. Wang, Y., Zhao, C., Ghanem, B., and Zhang. J. Free- dom: Training- free energy- mergy-guided guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23174-23184, 2023.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Zhang, B., Chu, W., Berner, J., Meng, C., Anandkumar, A., and Song, Y. Improving diffusion inverse problem solving with decoupled noise annealing. arXiv preprint vrXiv:2407.01522 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Stevens, T. S., van Gorp, H., Meral, F. C., Shin, J., Yu, J. Robert, J.-L., and van Sloun, R. J. Removing structured noise with diffusion models. arXiv preprini arXiv:2302.052900 2023.
Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R., and Van Gool, L. Denoising diffusion models for ald-pand image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219-1229, 2023.
Tran, P., Tran, A. T., Phung, Q., and Hoai, M. Explore image deblurring via encoded blur kernel space. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 11956-11965. 2021.
Trippe, B. L., Yim, J., Tischer, D., Baker, D., Broder- ick, T., Barzilay, R., and Jaakkola, T. S. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL https:// /ttpaa///ppeeeeeeee h/pp///eeeeee..c///o//./ n Forum? dd= 6T..XNME111.

---

A Mixture-Based Framework for Guiding Diffusion Models
A Methodology details
A.1 Primer on Gibbs sampling
In this section we lay out the basic properties of Gibbs sampling. We use thesure- notation for conciseness.
Let (d x1) be probability measure on I x RR. We denote by Monn (dxo (x1) and (1jo(dx/(x0) the associated full conditionals and we write Noo, Yu for its marginals. Define the transition kernels
Po(d(x(= xx) x1) == Paj1 (dx( x )ox, (dx() )); Pi (d(x), xx) xi) == [cx xx0)xx, (dx²)-
Each transition kernel updates only one coordinate at time. A full update of the coordinates is obtained by composition of the kernels, i.e.
Po (d(x), x() x1) := Pi (d((x x x Po) x1) x1).
Each transition admits the joint distribution 1oo1 as stationary distribution, meaning that (o,1(d(xo,
J Po(d(xo, x1)1x,, x1) No,1 (d(x), x()). Indeed, this seen by noting that
Po Po(d(xo: x1 |x x1) (d(x): x())
Hoj (dxo (x] )6x; ) xx)xxxxxx))
Mojo (dxo )6x;
Hojo (dxo (x )/ (dx) )oo (dx) xx) (dx():
and then integrating both sides w.r.t. (xg. x(). It then follows immediately that also PoP admits fo,1 as stationary distribution. Letting ((x), xf)) keNs be Markov chain with transition kernel P1, the law of (XP, x converges to po.1 as k: + do under mild conditions; see (Roberts & Smith, 1994).
A.2 Full Gibbs conditionals
In the main paper we consider the following data augmentation of the mixture म. (12)
9e ( (x)x))))sseee (xx (x)))
Tύ... (Xo, Xs, x,) =
(14)
I Ty (xx
dx',,
From this definition it is straightforward to see that / (xol Xs, xx) = Poj (x0 x.). In order to compute the full conditional デÝ o, (x, xx) we use the identity
Po (xo xx (x,
= (pt)))))))))(())000000000 (z.
(15)
from which it follows that
Pols (xo 19, (y
Y 10, (xx1X00, X1)
J Poys (y) (x dx
9.10 (xnoxx//o0)_,00000 (, (x.)
J (410(x()X00)
Xo
J [x Xo) (x) dx
Then, by noting that the bridge transition (3) satisfies 2=10. (x. X x1) = 9:00 Xx
(x) (x/x/ we find that
9. (y) X. ), Xxo xx)
n 00 (x, Xxo xx) = J [ x )0.j (xo, x,) dx
Finally, for the third conditional, using again the identity (15), we find that
Poj (Xo X, ). (y /x, )P (x (x. xx (x+) [ Pojs (xo X. )9, (y x. )p. (x. xx) (xx) dx
TY
7/0,0 (x| xx)
= = (xs)
12

---

A Mixture-Based Framework for Guiding Diffusion Models
A.3
Variational approximation
In this section we describe the variational approach of Moufad al. (2024), which we use fit a Gaussian variational approximation to Th (- x,) for fixed (Xo, x,). Similarly to the main paper consider the variational approximation
x 8/0,
(16)
and let (sjo,t (Mojo,es (M)oo,)PP E 12d F denote the variational parameters. The reverse KL divergence writes, following definition (3),
KL 5/0,r
ll T T0/0, xx))
- / log
2/10 (xs)
2 (x,) dxg + C
9, (y 1s110 (x. Xi)
|X8 (7t) + + -11-71)) ) xi
1 T
E
log G. iog (ylx))
Pa C.
(17)
20 20
2
Using the reparameterization trick (Kingma & Welling, 2013) and plugging-in in the neural network approximation ( -) of 9. (y),), we obtain the gradient estimator
Vili Xp: x1, 2) = Vep log 9 (y Ml diag(c2:1o..)1/22
Majo. + diag(eP=lo. )1/2
(71)a + - )a, {,)x),xx))1 x11
1 2 P. 0, 1d
20 s/o.t
where 7 NoO,, Ia). We initialize the variational parameters with the mean and covariance of the bridge kernel (3), i.e. at initialization, 10 Ooooo (1 " and Pa log -d. The Gauss_ VI routine is summarized in Algorithm 3.
Algorithm 3 Gauss_Vi VI routine
1: Input: vectors (Xo. x2), timesteps (s, t), gradient steps G
2: f é Yr. (o + (1 Mj,)),
3: p f log 20
to G do 4: for
5: Z NN,, La)
6: (u, f OptimizerSteepVvppCCC
2))
7: end for
8: 7 (O(0a, L2)
9: Output: μ + diag(e//2)2
Remark A.1. While the expectation the squared norm in (17) can be computed exactly, we found that, in practice, doing so degraded the algorithm's performance, producing blurrier images compared simply using Monte Carlo estimator for the full expectation.
Remark A.2. The fact that the density of our target distribution can be computed approximately by plugging the denoiser approximation allows us to add Mettopolis-Hastings (MH) correction with approximate acceptance ratio. Indeed, once we fit the Gaussian approximation, we can improve the accuracy of our sampler by simulating Markov chain (X²) ) where, given X.
8h+1 (.(dx.182)) (x (x( [ra 2)) + (1 =))ā
x)( (dxx) dz,
with
min (1, 9. 0, (x xi )1.0
g00 (xx)
(x(,,, x?) =min
0. ( No.
), 110,c (x:))
13

---

A Mixture-Bant Based Framework for Guiding Diffusion Models
A.4 Alternative data augmentation and sequence
Data augmentation. Our algorithm is based on one data-angmentation approach, but alternative augmentations could also be considered. Let [1,t- 1]]. Then the most obvious and natural data augmentation involves simply marginalizing out the Xo variable in (14), yielding
Th (X,, X1) x (2/yy, (xx,
Its full conditionals are (x, (y ( (xx) and (x, (xx(xx) qda (xr(xx). The first conditional is intractable for sampling, and could approximate it with Gaussian variational distribution, similar to our ap- proach for (-bx,
Indeed, this is possible since Vx, log ( (x,/ xx) Vx, log (y/x.) + Vx, lx..)(x.) +
Xx, log (x,(x)) which can then be approximated using the parametric approximations V log (y and V Vlogp. Ps (x,) = (-x, + a., (xx))) (1 a.).
The first drawback of this approach is that, in practice, it tends to degrade reconstruction quality-e.g. introducing blurriness-as as tends to 0, due to the poor approximation of the score near the data distribution. Additionally, beyond the loss of quality, we observe that it produces more incoherent reconstructions with noticeable artifacts. We hypothesize that this issue arises because the distribution we aim to approximately sample involves the prior transition Pa|t. which can be highly multi-modal when < t. This multi-modality may make the posterior <= (-Ix;) more challenging approximately sample from. On the other hand, when further conditioning Xo, the sampling problem becomes more well well-behaved, as we then target the posterior of a Gaussian distribution. Finally, while the score of T2i1 (x, (xx,) can be easily approximated, its density cannot, preventing the use of a Metropolis- Hastings correction, unless we use the independent proposal Paje (-1x;). However, this approach is suboptimal, as it does not incorporate any information from the observation. This is not the case of the data-augmentation approach we use in MGDM as we highlight Remark A.2.
Alternative sequence. An alternative to the mixture of posterior approximations (12), on which MGDM is based, the posterior formed as a mixture of likelihoods:
Of (y
ny
(xi)
Wi X xx))p( dx!
being the x; X--marginal of the extended distribution
(s, Xo, 2, x1) x
(yl2pp, (z/x)))))
(18)
Now, let (s, Xo, X,) b r then, conditionally s, the distribution of (Xo, Xx) is d. whereas
(1 ((== wt Gr 2 2y Xo, Xx)
Xx)
s| X, ~ Categorical
(=1
A Gibbs sampler targeting (18) is described in Algorithm 4. It allows updating the index in an observation-driven fashion, but is unfortunately computationally expensive as we need to evaluate the denoiser at Z in parallel for t - timesteps. A cheaper alternative could be to block the variables (s, Z) and use an independent MH step to target their joint conditional distribution. Denoting by A the joint proposal distribution on [1, L - I] x Rd used this independent MH step, the probability of accepting a candidate (s",z*) is
(1, w/ 9. (y )g. 10. Xo, (ss,) g.
re ((s, 2), (s", z²) = min
( ( 1..0 (z)x0 x, )A(s* [(sss")z) z²)
Remark A.3. Note that we could have used a similar data augmentation (18) for the mixture used in MGDM. This would yield the full conditional
{ wF (2)8o, Xz) mY 1
No, X, ~ Categorical
(2 Xu, X,) E
which is, however, intractable due to the normalizing constant involved in each $10.0*
14

---

A Mixture-Based Framework for Guiding Diffusion Models
Algorithm 4 Gibbs sampler targeting (13)
1: Input: ,x; 2", (x))
{
[-1 k=1
2: draw sh+1 Categorical
3: draw 7 n =0r+110.cl (1xx; (x)
4: draw X 0t|ort) (- 1
5: draw X ~ Polart1 (- 1))
A.5 Related algorithms
Comparison with Zhang et al. (2024) In this section we clarify the difference between MGDM and the DAPS algorithm (Zhang et al., 2024), which shares some similarities with our approach. The sampling procedure DAPS relies on sequential approximate sampling from the joint distribution
(Xn:() = mh (xT) II m (x
where
2 1 (x: xx+1) = f /1/0 u (xolx<+1) dxo
(19)
and Toje+1 (001)++1)== ny (x++1 (x))/ m² (x(+1). From this definition it follows that
क> (xx) - के "f|cti (x((xe+1) (x141) dx
and hence that the marginals of the joint distribution ने. are (m/ Yizo The canonical backward transition T.. (x, (x+1) oÿ (xx))++1j((xx++) (x,) has the alternative form
(x/xx(1))++1 / 100.411(x//ee x of (xo())+1) dxo:
which differs from (19) in the use of the bridge transition 000.++ instead the forward transition 4:/0-
In order to sample from # 1)), one needs first sample Xo T0 (-1x++1) and then X, (tj DAPS performs the former step using Langevin dynamics on an approximation of Toj4+1 (-1xe+1). More specifically, the authors use the approximation
go (y x No)NN(x D (x r2 Ia) J go x Nx De41 (x++1) ), r241 1 Ia) dx!
oop41 (x)
where r is hyperparameter. This approximation follows by noting that Th - (xo xx+1) & go ( (oy)xx))ooe11 (xo X and using the Gaussian approximation of Pojr+1 (-(x+)+ proposed by Song al. (2023a). The Langevin step is initialized with sample obtained by discretizing the probability flow ODE (Song et al., 2021) between t and 0.
Both MGDM and DAPS perform full noising and denoising steps and operate in a similar manner in this respect (with the distinction that we use DDPM instead of the probability flow ODF). The first fundamental difference is that we sample, conditionally on y and at a random timestep s, by drawing from (- (-1x,, x4) x 9. (y|x,, (xx,) (x, x:). Unlike DAPS, our method does not rely on a density approximation prior to applying an approximate sampler. The second main difference is the fact that within each denoising step, we can increase the number of Gibbs iterations improve the overall performance, as demonstrated in Figure 3. This is top of the number of gradient steps that we use to fit the variational approximation and which enhance the performance when we increase them.
On the other hand, DAPS does not require the computation of vector-Jacobian products of the denoiser and is thus more efficient in terms of memory. However it requires many calls to the likelihood function, which can substantially increase the runtime if expensive to evaluate. For example, with latent diffusion model, the runtime of DAPS is at least three times larger than that of MGDM, RESAMPLE, and PSL.D.
15

---

A Mixture-Based Framework for Guiding Diffusion Models
Comparison with Moufad et al. (2024) The more recent MGPS algorithm of Moufad et al. (2024) is also related to MGDM. Similarly to DAPS (Zhang al., 2024), their methodology relies sampling approximately from the posterior transition #d/++1 (+1x++) at each step of the backward denoising process. It builds on the following decomposition, which holds for all E [0. [0,t-1]:
T *Aje+1 (xee(x++)== - - Xs, xx+1)= (Xa (x++) dxs.
One step of MGPS proceeds by first sampling from an approximation of the posterior transition T (-1x++1) and then sampling from the bridge transition to return back to time t. The approximation of the posterior transition used in the MGPS is
9% ( (Xe)xe+1) (8,(y1x))r s|+ (x]]xe+1) dx,
(20)
Here one can then choose s to be sufficiently small to enhance the likelihood approximation, while still having an accurate Gaussian approximation of the transition Pstr+1 (-1x++1). The authors demonstrate, using solvable toy example, that this trade-off indeed exists; see (Moufad et al., 2024, Example 3.2). The approximate sampling step is then performed by fitting a Gaussian variational approximation to the approximation on the r.h.s. of (20), similarly to what we do in Algorithm 2.
Both MGDM and MGPS leverage the same idea of using, at step time t, likelihood approximations at earlier steps s < t. While in MGPS the time is set deterministically as a function of L, we sample it randomly. However, the main difference lies in the step where we sample conditionally on the observation y. Once the index s is sampled we proceed with R rounds of reverse KL minimization w.r.t. to different target distribution. Indeed, following Algorithm 2, in the first round we seek to fit a distribution with density proportional to x, + 5 (xx)) XX, X.). where X is output from the previous step of the algorithm. At step r, we fit Xs + S. (y xx. j.. (x. ( "x;")) where Xi-1 is sampled using a few DDPM steps starting from Xx-1 at time and X; - 9tls (11-; ). On the other hand, MGPS fits in a single round the distribution with density proportional to x, + 3 (y x. g. 1 (X ((+1). where Xe+1 is the output of the previous step. Finally, the authors report that the performance of MGPS improves when the number of gradient steps
increased. In our case, we have two axes, Gibbs iterations I and gradient steps, that allow us to improve the performance when more compute is available.
B Experiments details
B.1 Choice of weight sequence
In all our experiments we draw the index 8, time 4, from Uniform[r,44-i] with r = 10. The main motivation behind setting = 10 and not 1, which is more natural, is that we have found that otherwise it may lead to instabilities. This arises typically when an index s is sampled very close to O when t = T. To avoid such behavior we use a smaller learning rate in Algorithm 3 for the first few iterations and set 7 > 1. For the last 25% diffusion steps we set 8 deterninistically to 4-11 as we have found that this slightly improves the reconstructions quality. We also ramp up the number of gradient steps as this significantly sharpens the details in the images.
While more intuitive to sample close to as provides the best approximation error for the likelihood, we have found that this can significantly slow the mixing of the Gibbs sampler in very large dimensions and provides rather poor results when used with a small number of Gibbs steps. Practically speaking, significant artifacts arise during the initial iterations of the algorithm due to the optimization procedure, and they tend to persist in subsequent iterations when s is sampled close to O. To see why this is the case consider the following empirical discussion on a simplified scenario. We write x = [x, x where x E Ry and x E dd We assume that (x) = N(y: x, ozIa,): ). i.e., we observe only the first dy coordinates of the hidden state. Since s sampled near 0 we may assume that 9. (y) ()) go (y)-). Then, sampling Z ~ a xxooc( ( Xx, xx) is equivalent to sampling
2 N~N
(
2
to,t 02
of oz + o a0.0t
10p0 + (1 - mnwaaa -xx1 0300 Ity 0z + 00
y +
Z ~ NN + (1 - / 020 dy)
16

---

A Mixture-Basse Framework for Guiding Diffusion Models
Table 4: LPIPS on the FFHQ dataset for the two time-sampling distributions given in (21) and (22). We use f = 4 Gibbs steps for the phase retrieval task.
Distribution Phase retrieval (R 4)
JPEG2
Gaussian deblurring
Motion deblurring
0.10
0.14
0.12
0.09
0.53
0.19
0.16
0.19
200
600
900
1000
(pt)
Figure 4: Evolution of the rumning state X3 in Algorithm 2 for the two time- sampligg distributions given in (21) and (22).
setting Z 1 Z] and then concatenating both vectors. It is thus seen that the observed part of the state is updated with the observation whereas the bottom part is simply drawn from the prior. Moreover, if 02j0.e e RJ 0 then "ajo Rs I and Z is almost the same as Xp- In Algorithm 2, once we have sampled X, ~ Jo (Xo,, Xt), we first denoise it to obtain the new Xa and then noise it to obtain the new X,. As is sampled near 0, the denoising step will merely modify X, whereas the noising step will add sigmificant noise X, and may help with removing the artifacts. This noised sampled has however only small impact the next samples Xo, X, since (1 )aaa) O. In short, the first dy coordinates of the running
state Xg will be quickly replaced by the observation whereas the last d dy coordinates will be stuck at their initialization We illustrate this situation on a concrete example in Figure 4 where we consider half mask inpainting task. The first and second rows show the evolution of the running state Xi with the same-sanpling distributions
and will evolve only by a small amount throughout the iterations of the algorithm.
(onform[T,4-11 4-11
if i> [K/4] else
14
(21)
p Uniform [1, [t,/5]]
(22)
ie. the time-sampling distribution we in all our experiments, where K is the number of diffusion steps, and the one that use sample only close to 0, respectively. In Table 4 compute the LPIPS for both distributions subset of the tasks consider in the main paper. It is clear that μ outperforms u.. even when increase the number of Gibbs steps (see phase retrieval task).
B.2 Hyperparameters setup of MGDM
The details about the hyperparameters of MGDM are reported in Table 5. We adjust the optimization of the Gaussian Variational approximation in Algorithm 3 during the first and last diffusion steps. We ramp up the number of gradient steps during the final diffusion steps. This allows us to substantially improve the fine grained details of the reconstructions. Similarly, we reduce the learning rate in the early step to alleviate potential instabilities.
B.3
Audio source separation
In our experiment, the diffusion model employed provided by (Mariani et al., 2023) is trained on the slakh2100 training dataset', using only the four abundant instruments (bass, drums, guitar and piano) downsampled to 22 kHz. The denoiser network is based on latela,ent, time-domain unconditional variant of (Schneider al., 2023).
Its architecture follows U-Net design, comprising an encoder, bottleneck, and decoder. The encoder consists of six layers
'ttp::/ //www//wwwwwww/ a akh. com/
17

---

A Mixture-Bant Based Framework for Guiding Diffusion Models
Table 5: The hyperparameters used in MGDM for the considered datasets. The index of the timesteps {4]] & is taken in reverse order. The symbol # stands for "number of".
repetitions
Difiusion stops X
Dencising stegs Af
sampling distribution
Learnngg rate
Gradient stops
0.01
i [3K/4]
it K/44 otherwise
PPHQ
R
K 100
M 20
16] as in (21)
0.03
atherwise
20 if S [K74]
00.01
if [aK74]
PPRO LDM
R
K - 100
M 20
16 in (21)
20
if mid 10 =0=00
0.003
otherwise
otherwise
0.01
if [3K/4]
20 if K/4] otherwise
Emagelist
R=
K' 100
14 (21)
0.003
atherwise
5
20
if K3/4] otherwise
Andio-source separation
R-6
K-20
M 15
14] in (21)
0.005
3
Audio separation [Best result Table
R=1
==
15
14 (21)
0.005
90
with channel numbers [256, 512, 1024, 1024, 1024, 1024], where each layer includes two convolutional ResNet blocks, and multihead attention is applied the last three layers. The decoder mirrors the encoder structure in reverse. The bottleneck contains ResNet block, followed by self attentttenton mechanism, and then another ResNet block. Training performed on the four stacked instruments using the publicly available trainer from repository?.
B.4 Implementation of the competitors
In this section, we provide implementation details of the competitors. We adopt the hyperparameters recommended by the authors tune them on each dataset if they are not provided. We emphasize that we use the total number of diffusion steps available (1000 steps) for DPS, PGDM, DDNM, DIFFPIR, and PSLD. For the other algorithms,, steps until performance plateaued. The complete set of hyperparameters and
tuned the compute time
by increasing Laanevinddenoisinn/oppiiiii their values for both image experiments and audio-sound separation can be found in the supplementary material under the folders con f i eewpseeeeeiieeeee interrmmettss poantss/smmppllm sammplmlee er and configg gssii////ee exp_eoundddd/eeeeeeeeeeeeeeeeeee wwwooudd//mppleeeeeeeeeeee/ sampler.
DPS. We implemented Chung et al. (2023, Algorithm 1) and selected the hyperparameters of each considered task based on Chung et al. (2023, App. D). We tuned the algorithm for the other tasks, namely, we use 7 = 0.2 for JPEG 2%, 7 = 0.07 for High Dynamic Range tasks, and 7 for audio-source separation.
DiNPIRR We implemented Zhu et al. (2023, Algorithm 1) to make it compatible with our existing code base. We adopt the hyperparameters recommended in the official, released version?. We followed the guidelines in (Zhu et al., 2023, Eqn. (13)) to extend the algorithm to nonlinear problems. However, we noticed that the algorithm diverges in these cases and we could not follow up as the paper and the released code lack examples of nonlinear problems. Zhu et al. (2023) provides an FFT-based solution for the motion blur tasks which is only valid in the case of circular convolution. Hence, and since we adapted the experimental setup of Chung et al. (2023), we do not run the algorithm on motion blur task as it uses convolution with reflect padding. For audio-source separation, we found that X = μ = 1 works best.
DDNM. We adapted the implementation provided in the released code Namely, the authors provide classes, in the module funct inns/ trrd__pperttor.c py that implement the logic of the algorithm on each degradation operator separately. The adaptation includes factorizing these classes to single class to support all SVD linear degradation operators. On the other hand, we notice DDNM is unstable for operators whose SVD decomposition is prone to numerical errors, such as Gaussian Blur with wide convolution kernel. This results from the algorithm using the pseudo-inversy of the operator.
RedDiff. We used the implementation of REDDIFF available in the released code., For linear problems, we use the pseudo-inverse of the observation as an initialization of the variational optimization problem. On nonlinear problems, for which the pseudo-inverse of the observation is not available, we initialized the optimization with a sample from the standard Gaussian distribution.
Phtpps: ://github 'https: : ///gi thub. www.uuccomaa.a/
fFPIR
*https: ://i thub. com www/wyhuai//oocoocooooocwp.w / /DDNM "https://gt thub. wttpccooo//abbbeee
18

---

A Mixture-Bant Based Framework for Guiding Diffusion Models
0.4
LPIPS
0.3
10 # Denoising steps M
# Gibbs steps R
# Gradient steps G
# Diffusion steps K
Figure 5: Effect of individually increasing each parameter on the performance of our algorithm for the phase retrieval task. In each case, we vary one parameter while keeping the others fixed to the values specified in the original manuscript.
PGDM. We opted for the implementation available the REDDIFFÁ repository as some the authors are co-authors of PGDMas well. Notably, the implementation introduces a subtle deviation from Song et al. (2023a, Algorithm 1): in the algorithm's final step, the guidance term y is scaled by ar (yoe in their notation) whereas the implementation scales it by ar-10r, This adjustment improves the algorithm for most tasks except for JPEG dequantization. We found that the original scaling by a₂ is better in this case.
PSLD. We implemented the PSLD algorithm provided in Rout et al. (2024, Algorithm and referred to the publicly available implementation" to set the hyperparameters of the algorithm for the different tasks.
ReSample. We modified the original code? provided by the authors to make its hyperparameters directly adjustable, namely, the tolerance & and the maximum number of iterations N for solving the optimization problems related to hard data consistency, and the scaling factor for the variance of the stochastic resampling distribution y. We found the algorithm be sensitive to and that setting it to the noise level of the inverse problem yields the best reconstructions across tasks and noise levels. On the other hand, we noticed that has less impact on the quality of the reconstructions. Finally, we set threshold N = 200 on the maximum number of gradient iterations to make the algorithm less computationally intensive.
DAPS. We have the official codebase 8 We referred to Zhang al. (2024, Table. 7) to set the hyperparameters. For audio-source separation, set max and min to match those of the sound model and adapted the Langevin stepsize 1r and the standard deviation tau to the audio-separation task.
PNP-DM. We adapted the implementation provided in the released code". Specifically, we exposed the coupling parameter p including its initial value, minimum value, and decay rate, well the number of Langevin steps and its step size. The hyperparameters were set based on Wu et al. (2024, Table 3 and Table 4). For inpainting tasks, while it is theoretically possible to perform the likelihood steps using Gaussian conjugacy (Wu et al., 2024, Sec. 3.1), we found that using Langevin produced better results in practice. For example, the reconstructions in the left figure of Figure are obtained by sampling exactly from the posterior whereas on the r.h.s. we use Langevin dynamics. Although the audio separation task is linear and hence the likelihood steps can be implemented exactly, we encountered similar challenges as inpainting and therefore we used Langevin here as well.
B.5 Experiments reproducibility
Our code will be made available upon acceptange of the paper. the anonymous codebase provided as companion of the paper we use va, instead of ag match the conventions of existing codebases. All experiments were conducted on Nvidia Tesla V100 SXM2 GPUs. For the image experiments, and ImageNet 256 x 256 that numbered from 0 to 299. The image number used to seed the randomness of the experiments on that image. For the audio source separation experiments, the s slakkhl akh21 00 test dataset has tracks named following the pattern TrackOXXXX, where X represents digit in 0 - 9. The number XXXX was used as the seed for the
used 300 images from the validation sets of FFHQ
"https://gh" htt///tttuu./.t/ http:///ttttttt//eeee.e
Thttps: ttttps://giia ///gi thggtthub... http://....ooooooo//////////////
Phttps://gf thub. wtt.ob...ccmcaaaaaeeee / 9/DAPS Phttps://gt
19

---

A Mixture-Based Framework for Guiding Diffusion Models
experiments conducted each track.
B.6 Extended results
We present the complete results with FII,, LPIPS, PSNR, and SSIM metrics for the image inverse problems experiment in: Table 6 for FFHQ pixel-space, Table 7 for ImageNet, and in Table 8 for FFHQ LDM. Figure 5 we provided an extension of the ablation in Figure 3. Similarly, the complete results for the audio source separation experiments that include provided in Table 9.
all competitors
From Table 6 and Table 7, one can note that DDNM, DIFFPIR and DAPS score better in PSNR and SSIM compared to MGDM but score lower in LPIPS. For most the tasks we considered, one does not expect to recover an image very close to the reference and thus, metrics that perform pixel-wise comparisons are less relevant and favor images that are overly smooth. We provide evidence for this in the gallery of images below where we compare qualitatively the outputs of our algorithm with those of the competitors. can be seen that our method provides reconstructions with fine-grained details that MGDM in terms of PSNR and SSIM on the half mask task on ImageNet while failing reconstruct the missing r.s., of the images.
coherent with the reference image. Note for example that DDNM, DIFFPIR and DAPS outperform
20

---

A Mixture-Baser Framework for Guiding Diffusion Models
Table FID and mean LPIPS/PSNR/SSIM metrics along side 95%--confidenc interval on FFEQ 256 dataset with dy 0.05.
Task
MGDM
DPS
PGDM
DDNM
DIFEPIR
REDDIFF
DAPS
PNP-DM
LPIPS
SR (x4)
0.09 40.00
0.09 10.00
0.30 10.01
0.15 +0.00
0.10 40.00
0.39 10.01
0,16 10.01
0.10 10.00
(x16)
0.24 10.01
0.23 10.01
0.42
0.33 10.01
0.23 10.01
0.55 10.01
a0.01
10.01
Box inpointing
0.10 =0.00
0.17
0.17
0.00
0.00
0.19
0.13 =0.00
#0.01
Half mask
0.20 =0.01
0.23 #0.01
0.24 =0.01
0.23 #0.01
0,25 #0.01
0.28 #0.01
0.23 #0.01
0.32 #0.01
Gaussian Dehlur
0.12 40.00
0.17 +0.01
0.87 40.02
0,20 +0.01
0.12 40.00
0.24 +0.01
0.24 +0.01
0.14 +0.01
Motion Deblur
0.09 40.00
0.17 +0.01
0.22 +0.01
0.19 #0.01
0.21 40.01
(QF=s =
1.12
0.22
Phase retrieval
0.11 =0.02
0.40 #0.02
0.26 =0.02
0,14 20.01
#0.02
Nonlinear deblur
0.27 20.01
0.51 #0.04
0.68 #0.02
0.28 #0.01
0.31 #0.01
High dynamic range
0.12 +0.01
0.40 +0.06
0.20 +0.03
0.10 #0.01
0.19 #0.01
FID
(x4)
55.02
69.75
54.26
SR (x
62.12
61.24
161.64
104.10
58.12
141.76
117.83
63.49
Box inpainting
52. 08
89.13
60.99
64,06
68.34
64.24
60.29
75.67
Half mask
58.97
79.31
61.48
64,96
71.86
63.92
63.88
84.16
Gaussian Deblur
55.37
60.55
295.19
73.92
$4.29
71.82
56.69
Motion Deblur
52.65
61.80 10
102.88
70.76
JPEG (QF 2)
56.40
107.54
305.00
132.81
79.09
108.89
Phase retrievul
83.58
146.79
123.91
58.53
172.20
Nonlinear deblur
87.89
198.33
113.38
92.12
91.30
High dynamic range
54.56
165.70
75.63
SR (x4)
27.66 40.22
28.05 +0.24
24.57 10.14
29.45 a0.23
27.72 +0.22
26.75 10.13
28.44 x0.21
27.44 40,20
SR (x16)
21.01 #0.20
20.71 =0.21
18.51 10.14
22.32 #0.21
20.96 #0.20
21.46 10.17
19.75 #0.17
20.88 10.19
Box inpainting
22.38 +0.27
18.81 20.28
21.05 +0.25
22.34 #0.31
22.39 +0.32
21.46 #0.28
22.06 06.06666 +0.30
20.42 20.28
Half mask
15.39 #0.27
15.03 =0.27
15.29 #0.28
16.38 #0.35
16.04 #0.36
15.68 =0.34
16.25 30.3
14.35 +0.30
Gaussian Dehlur
25.64 #0.24
24.03 #0.22
13.34 #0.10
26.62 +0.23
25.78 +0.23
26.68 +0.22
26. 12 +0.23
25 89 #0.21
Motion Deblur
27.82 +0.20
24.12 #0.21
27 .4 +0. 13
27.07 +0.21
24.91 +0,24
JPEG (QF =
25.57 10.19
19.56 +0.60
12.57 10.10
24.53 10.13
25.72 10.18
22.42 10.18
Phase setrieval
27.55 10.65
16.56 20.63
24.58 20.67
27.84
21.63 10.70
Nonlinear deblur
23.55 10.27
16.08 10.87
94 10.25
24.56 10036
24.08 10.32
High dynamic runge
24.79 #0.42
18.71 #0.32
21. 21.69 =0.20
26.60 #0.38
21.59 #0.22
SSIM f
(x4)
0.80 #0.01
0.81 +0.01
0.56 +0.00
0.85 +0.00
0.78 #0.01
0.68 +0.00
0.81 +0.00
0.77 +0.01
(x16)
=0.01
0.58 +0.01
00.01
0.67 =0.01
0.59
0.60
0.58 =0.01
+0.01
Box inpainting
0.80 20.00
0.77 +0.00
0.70 20.00
0.83 +0.00
0.82 +0.00
0.70 #0.00
0.80 +0.00
0.75 +0.00
Half mask
0.67 10.01
0.67 10.01
0.59 10.01
0.74 10.01
0.72 40.01
0.63 10.01
0.71 20.01
0.65 10.01
Gaussian Dehlur
0.73 10.01
0.68 10.01
0.14 10.01
0.77 40.01
0.72 40.01
0.76 10.01
0.75 a0.01
0.72 40.01
Motion Deblur
0.80
0.70
0.71 #0.01
0.78 #0.01
0.75 +0.01
JPEG (QF=)) 2)
0.74
0.56
0.10
=0.01
0.76 =0.01
0.70 #0.01
Phase retrieval
0.78 20.02
0.49 #0.02
0.61 #0.02
0.81 =0.01
0.57 #0.02
Nonlinear dehlur
0.67 +0.01
0,44 +0.03
0.42 +0.01
0.71 #0.01
0.70 +0.01
High dynamic range
0.76 +0.02
0.55 +0.06
0.72 +0.04
0.85 +0.01
0.69 +0.01
21

---

A Mixture-Basce Framework for Guiding Diffusion Models
Table FID mean LPIPS//SNR/SSIN metrics along side 95%-ccnnieence interval on ImageNet 256 256 dataset with dy 0.05.
Task
MGDM
DPS
PGDM
DDNM
DIFFPIR
REDDIFF
DAPS
PNP-DM DMP-DM
LPIPS
SR (x4)
0.26 10.01
0.25 10.01
0.56 40.02
0.34 10.02
0.31 10.01
0.57 =0.02
0.37 a0.02
0.66 10.02
(x16)
0.55 =0.02
0.44 +0.01
0.62 #0.01
0.71 #0.02
0.50 #0.02
0.85 =0.02
0.75 #0.02
1.03 #0.02
Box inpeinting
0.23 =0.01
0.35 #0.01
0.29 =0.01
0.28 #0.01
0.30 #0.01
0.36
0,30 =0.01
0.42 #0.01
Half mask
0.31 20001
0.40 40.03
00.34 40.03
0.38 +0.03
0.40 40.03
0.46 20.03
0.40 +0.01
0.54 +0.01
Gaussian Dehlur
0.30 +0.01
0.37 +0.02
1,00 +0.01
0.45 +0.02
0.30 +0.01
0.53 #0.02
0.59 #0.02
0.76 +0.02
Motion Dehlur
0.22 +0.01
0.40 +0.04
0.39 #0.04
0.42 +0.02
0.52 #0.02
JPEG (QF=
0.38 =0.02
0.60 #0.06
1.32 20.01
0.49 =0.04
0.45 #0.02
0..02
Phase retrieval
0.55 =0.02
0.62 +0.02
0.61 =0.02
0,50 00.02
#0.01
Nonlinear dehlur
0.41 #0.01
0.82 +0.05
0.66 =0.05
0.41 +0.02
0.49 #0.02
High dynamic range
0.21 +0.02
0.84 +0.05
0.19 =0.04
0.14 +0.01
0.31 +0.02
FID
(x4)
94.77
131,97
109.83
104.44
139.67
113.54
(x16)
142.09
124.72
181.29
223.92
135.37
228.78
224.69
269.45
Box inpainting
113.49
173.77
123.99
133.41
145.42
157.76
138.02
178.59
Half mask
106.
144,08
112.73
111.63
118.21
133.44
114,60
147.14
Gaussian Deblur
105.92
110.73
287. 63
140.06
106 64
148.75
158.50
198.22
Motion Dehlur
94.74
114.03
144.47
133.07
166.30
JPEG (QF= = 2)
117.15
186.13
340.95
151.58
145.65
178.50
Phase retrieval
170.71
148.61
215.18
148.06
201.66
Nonlinear deblur
175.50
298.36
171.00
175.44
172.58
High dynamic range
104.01
353.01
99.81
94.38
126.53
PSNR
SR (x-)
23.88 #0.44
24.37 #0.49
18.45 #0.26
24.99 +0.50
23.43 =0.42
23.33 #0.35
24.38 #0.46
16.40 =0.24
SR (x 16)
18.12 #0.31
17.66 +0.39
15.27 +0.23
19.93 +0.40
18.40 =0.37
19.06 #0.33
18.18 +0.32
14.00 #0,17
Box inpainting
16.82 40.33
13.92 +0.30
16.73 +0.31
19.18 +0.44
19.05 #0.47
18.21 40.40
19.11 +0.40
18.03 a0.36
Half mask
13.77 +0.29
12.15 19.19
14.05 10.19
15.97 #0.21
15.64 =0.22
14.84 +0.20
16.00 =0.37
14.88 20.26
Gaussian Deblur
21.57 +0.43
20.65 #0.43
9.92 10.08
22.89 10.47
21.80 =0.44
22.72 +0.45
22.41 +0.45
15.85 =0.22
Motion Deblur
24.46 #0.42
21.38 #0.21
24.06 19.19
23.64 +0.44
22.47 #0.40
JPEG (QF
21.42 #0.32
16.33 +0.27
5.27 +0.04
22.07 +0.
22.68 +0.36
20,74 +0,30
Phase retrieval
16.01 +0.71
14.12 #0.49
15.41 +0.59
18.44 +0.72
15.02 #0.50
Nonlinear deblur
10.39
10.13 40.28
20.57 10.18
68 10.44
22.20 a0.43
High dynamic range
22.90 10.55
9.56 10.26
22.12 40.23
24.69 10.49
22.23 10.46
SSIM 1
SR (x4)
0.65 10.02
0.68 10.02
0.30 10.01
0.71 a0.02
0.60 10.01
0.57 10.01
0.66 40.02
0.25 10.01
(x16)
0.31 =0.01
0.39 +0.02
0.21 #0.01
0.49 #0.02
0.41 #0.02
0.44
0.44 +0.02
0.10 00.00
Box inpainting
0.71 =0.01
0.70
0.62 +0.00
0.77 #0.01
0.76 #0.01
0.67 =0.00
0.74 =0.01
0.64 +0.01
Half mask
0.59 10.01
0.58 40.03
0.52 40.02
0.68 40.03
0.67 40.03
0.59 20.03
0.66 40.01
0.57 40.01
Gaussian Dehlur
0.50 #0.02
0.50 40.02
0.08 40.00
0.59 40.02
0.51 +0.02
0.57 #0.02
0.56 #0.02
0.20 40.01
Motion Deblur
0.67 =0.01
0.55 +0.05
0.61 =0.03
0.63 #0.02
0.57 #0.02
JPEG (QF=
0.51 10.01
0.40 20.06
0.02 20.00
0.59 20.04
0.62 #0.02
0.57 #0.02
Phase retrieval
+0.02
0.25
Nonlinear dehlur
0.58 +0.01
0.25 #0.06
0.41 20.04
0.61 #0.02
0.58 #0.02
High dynamic range
0.72 +0.02
0.23 +0.06
0.72 =0.04
0.82 +0.01
0.66 #0.02
22

---

A Mixture-Bant Framework for Guiding Diffusion Models
Table FID mean LPIPS/PSNR/SSIN metrics along side 95%-confidence interval on FFHQ dataset with LDM prior with dy 0.05.
Task
MGDM
RESAMPLE
PSLD
DAPS
PNP-DM
LPIPS
SR (x4)
0.14 #0.01
0.22 #0.01
0.21 =0.01
0.28 #0.01
+0.01
SR (x16)
0.30 40.01
0.38 10.01
0.36 40.01
0.52 40.01
0.71 40.01
inpainting
0.18
0.22
0.37
0.31
Half mask
0.26 #0.01
0.30 =0.03
0.32 =0.03
0.44
Gaussian Dehlur
0.18 +0.01
0.16 +0.11
0.59 #0.01
0.32 #0.01
0.32 +0.01
Motion Dehlur
0.22 +0.01
0.20 +0.33
0.70 #0.03
0.36 +0.01
0.36 +0.01
JPEG (QF 2)
0.23 #0.01
0.26 +0.0
0.32 #0.01
20.61
Phase retrieval
0.29 +0.02
0.39 +0.02
0.25 +0.01
0..0
Nonlinear
0.29
0.37
High dy namic range
0.16 10.01
0.12 20.03
0.24 10.01
0.24 40.01
FII 4
SR (x4)
63.91
78.74
78.85
110.21
131.06
SR (x16)
69.26
100.44
92. 16
176.67
281.53
inpainting
76.22
126.49
83.53
162.11
129.27
Half mask
78.32
91.76
80.00
131.68
116.16
Gaussian Deblur
69 44
69.77
150.43
101.88 88
103.16
Motion Dehlur
74.43
77.97
158.40
120.74
122.12
JPRG (QF=
72.78
90.39
123.93
131.08
Phase retrieval
128.57
230.42
105.
169.94
Nonlinear
150.54
135.81
High dynamic range
75.74
65.79
102.
89.32
PSNR
SR (x4)
27.39 #0.21
25.85 10.25
25.80 +0.33
27:45 40.20
23.81 10.21
SR (x16)
20.60 +0.18
20.97 +0.
21.42 +0. 19
19.91 #0.16
17.07 #0.14
inpainting
21.81 +0.28
18.56
11.77 #0.26
19.57 #0.31
Half mask
15.71 10.30
14.89 10.17
14.62 19
9.13 40.25
14.15 20.28
Gaussian Deblur
26.79 #0.22
27.28 #0.22
17.99 10.13
26.86 #0.26
26.11 #0.19
Motion Deblur
25.27 +0.20
26.73 +0.
17.71 #0.12 12
25.37 #0.21
24.65 =0.18
JPBG (QF
24.27 40.18
24.77 40.15
25.22 40.18
23,86 10.17
Phase retrieval
22.54 #0.66
20. #0.61
27.05 +0.35
20.03 20.61
Nonlinear deblur
23.71 =0.27
10 10.19
22.03 =0.23
23.28 #0.26
High dynamic range
25.59 +0.32
25.91 #0.21
20.95 +0.38
20.21 #0.23
SSIM 1
SR
0.79 40.01
0.68 #0.01
0.71 #0.01
0.79 +0.01
0.70 +0.01
SR (x16)
0.58
0.56 10.01
20.01
0.59
0.52 20.01
inpainting
#0.01
0.73
Half mask
0.69 +0.01
0.67 #0.02
0.60 +0.03
0.55 +0.01
0.65 +0.01
Gaussian Deblur
0.77 10.01
0.75 10.01
0.27 10.01
0.78 40.01
0.77 10.01
Motion Deblur
0.73 #0.01
0.72 =0.03
0.24 =0.02
0.74 =0.01
0.72 +0.01
JPBG
0.71 #0.01
0.66 =0.03
0.75
0.001
retrieval
40.02
0.60
Nonlinear deblur
0.69 =0.01
0.67 20.03
=0.01
0.70 =0.01
High dynamic range
0.80 +0.01
0.83 20.03
0.74
0.73 #0.01
Table 9: Mean SI-SDR, along side 95% confidence interval on s1akh2100 test datasct. The last row "All" displays the mean over the four stems. Higher metrics are better.
Stems
MGDM
DPS
PGDM
DDNM
DIFTPIR
REDDIF
DAPS
PNP-DM
MSDM
ISDM
DEMUCSS13
Bass
18.49 +1.92
16.50 40.98
16.41 +1.10
14.94 +1 57
-2.34 +0.67
-0.40 #0.49
11:76 =1.61
2.90
17.12
19.36
17.16
Drums
18.07 +0.48
18.29 40.62
18.14 40.76
19.05 +0.51
9.47 +0.60
.0.98 #0.54
15.62 #0.42
7.89 40.95
18.68
20.90
19.61
Guitar
16.68 11.41
9.90 11.39
11.99
14.38 21.57
-1.01 =0.53
5.68 20.88
11.75 =1.55
4.51 11.19
15.38
14.70
17.82
Piano
9.52
All
17.35 21.01
14.33
14.96
20.69
12.16.1.23
17.73
23

---

A Mixture-Based Framework for Guiding Diffusion Models
B.7 Runtime and memory requirement comparison
We evaluate the runtime and GPU memory consumption for image experiments on the three considered diffusion model priors. Since not all algorithms support every task, we restrict the evaluation commune tasks. Figure 6 presents the average runtime and GPU memory requirement over both samples and tasks.
MGDM has memory requirements similar to DPSand PGDMin pixel space and aligns closely with other methods in latent space. Importantly, in latent diffusion which highly relevant scenario given the prevalence of latent-space models, MGDM is notably faster than all competitors while consistently achieving strong performance across benchmarks. Conversely, when operating directly in pixel space, exhibits relatively slower runtimes compared some alternatives. However, this increase in computational overhead is consistently balanced by improved and stable reconstruction quality across all considered tasks evidenced by the gallery of examples in Appendix B.9. Thus, position our method as offering beneficial trade-off, especially in scenarios where quality and consistency of results are paramount.
On the otherband, we highlight several important points regarding the competitors' runtime: 1) for latent diffusion, DAPS and PNP-DM perform significant amount of Langevin steps using the gradient of the likelihood. Since the latter involves a vector jacobian product the decoder, the runtime increases significantly. More generally, when the likelihood function expensive to evaluate, DAPS and PNP-DM are expected to be much slower than DPS, PGDMand MGDM. For PNP-DM on FFHQ have implemented the likelihood step exactly the linear tasks. On TmageNet however, using Langevin steps provided better results and this explains the significant increase in runtime.
Rantitue in acconcls
Memory uxago in GB
1500
1000
500
MGDM
RESAMPLE
PSLD
DAPS
PNP-DM
Runtie in soconds
Memory usagg in CB
200
100
MGDM
PGDM
DDNM
DIFFPIR
REDDIFF
DAPS
PNP-DM
H DDNM
Runtine in socondss
800
Metnory usigg in CB
600
400
200
0-
MGDM
DPS
PGDM
DIFFPIR REDDIFF
DAPS
PNP-DM
Figure 6: Comparison of the runtime (red bars - left axis) and memory requirement (blue bars right axis) between the considered algorithms on FFHQ latent space (1x row), FFHQ pixel space (2nd row), and ImageNet (3nd row).
B.8 Experiments with high noise setup
To ensure the performance of the MGDM extend to higher noise setups, we evaluat it on both Half-mask (linear task) and JPEG QF=2% (nonlinear task) while increasing the noise level to By = 0.3, increased of 0.05. The experiments were
24

---

A Mixture-Based Framework for Guiding Diffusion Models
conducted the FFHQ dataset in pixel-space, with MGDM being compared against DPS and DAPS. As evidenced by Table 10, MGDM consistently outperforms the other algorithms, namely in terms of LPIPS.
Table FID mean LPIPS/PSNR/SSII metrics along side 95%-confidence interval on FFHQ 256 256 dataset with By 0.3.
MGDM
DAPS
MGDM
DPS
DAPS
MGDM
DAPS
MGDM
DPS
DAPS
Task
LPIPS 1
FID 1.
PSNR
SSIM
Hall
0.23
0.67
56.77
15.59
14,86
50.27
10.01
JPBG JPBG-QQF=
0.21 #0.01
0.57 4003
0.24 -0,01
59.78
110.0
65.11
22.70 10.16
18.69 =0.54
23.62 +0.15
0.67 +001
0.54 +0.02
0.09 +0.01
B.9 Experiments with Poisson-noise likelihood
Pervious experiments were performed with Gaussian-noise likelihood, gu(y/x) = N(y; Ax),, oîT). Here, we extend our evaluation to Poisson-noise likelihood
(ylx) exp AA(x)) (-AA(x))
y!
where is the Poisson rate. We compare MGDM against DPS on several tasks: denoising, super-resolution (x4), Gaussian deblurring. and motion deblurring. These experiments were conducted on the FFHQ dataset pixel space with a Poisson rate of 0.05. Notably, while DPS relies on Gaussian approximation (Chung et al., 2023, Eqn. (19)) due the inherent challenges of Poisson likelihoods, we directly implement the Poisson likelihood without approximation. As detailed in Table 11, MGDM consistently outperforms DPS across all evaluated metrics.
Table 11: FID and mean LPIPS/PSNR/SSI metrics along side 95%-confidence interval FFEQ 256 256 dataset with Poisson noise
with Poisson rate À 0.05.
MGDM
DPS
MGDM
DPS
MGDM
DPS
MGDM
DPS
Task
LPIPS
FID
PSNR
SSIM
Denoising
+0.00
54.73
+0.01
SR (x4)
0.25 +0.01
0.25 #0.01
74.75
69.49
21.65 #0.21
17.62 #0.12
0.65 =0.01
0.56 #0.01
Gaussian Deblur
0.20 +0.01
0.30 =0.01
64.83
100.63
23.09 =0.22
16.49 10.16
0.66 =0.01
0.49 =0.01
Motion Deblur
0.21 +0.01
0.27 20.01
60.53
69.73
22.52 +0.20
16.91 20.15
0.66 =0.01
0.52 20.01
25

---

A Mixture-Based Framework for Guiding Diffusion Models
Observation
Reference
Observation
Reference
MCDM
MGDM
DPS
DPS
PCDM
PGDM
DDNM
DDNM
DIFFPIR
DIFFPIR
RED
REDDIFF
DAPS
DAPS
PNP-DM
PNP-DM
Figure 7: Reconstructions for half mask inpainting on FFHQ dataset.
26

---

A Mixture-Baser Framework for Guiding Diffusion Models
Observation
Reference
Observation
Reference
MGDM
MGDM
DPS
DPS
PCDM
PGDM
DDNM
DDNM
DIFFPIR
DIFFPIR
RED
REDDIFF
DAPS
DAPS
PNP-DM
PNP-DM
Figure 8: Reconstructions for box inpainting on FFHQ dataset.
27

---

A Mixture-Bant Framework for Guiding Diffusion Models
Observation
Reference
Observation
Reference
MGDM
MGDM
DPS
DPS
REDDIFF
REDDIFF
DAPS
DAPS
PNP-DM
PNP-DM
Figure Reconstructions for JPEG dequantization QF=2% on FFHQ dataset.
28

---

A Mixture-Baser Framework for Guiding Diffusion Models
Observation
Reference
Observation
Reference
MGDA
MGDM
DPS
DPS
PCDM
PGDM
DDNM
DDNM
DIFFPIR
DIFFPIR
REDDIFF
REDDIFF
DAPS
DAPS
PNP-DM
PNP-DM
Figure 10: Reconstructions Half mask inpainting on ImageNet dataset.
29

---

A Mixture-Based Framework for Guiding Diffusion Models
Observation
Reference
Observation
Reference
MGDM
MGDM
DPS
DPS
PCDM
PGDM
DDNM
DDNM
DIFFPIR
DIFFPIR
RED
REDDIFF
DAPS
DAPS
PNP-DM
PNP-DM
Figure 11: Reconstructions for Gaussian deblurring on ImageNet dataset.
30

---

A Mixture-Bans Framework for Guiding Diffusion Models
Observation
Reference
Observation Reference
MCDM
MGDM
DPS
DPS
REDDIFF
REDDIFF
DAPS
DAPS
PNP-DM
PNP-DM
Figure 12: Reconstructions for motion deblurring on FFHQ dataset.
31

---

A Mixture-Bant Framework for Guiding Diffusion Models
Observation
Reference
Observation Reference
MCDM
MGDM
RESAMPLE
RESAMPLE
PSLD
PSLD
DAPS
DAPS
PNP-DM
PNP-DM
Figure 13: Reconstructions for half mask inpainting on FFHQ dataset with LDM prior.
32

---

A Mixture-Bant Framework for Guiding Diffusion Models
Observation
Reference
Observation Reference
MCDM
MGDM
RESAMPLE
RESAMPLE
PSLD
PSLD
DAPS
DAPS
PNP-DM
PNP-DM
Figure 14: Reconstructions for SR x16 FFHQ dataset with LDM prior.
33

---

A Mixture-Bascd Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Observatio
Sumple
3 GANDER
9A
GGAL
Figure 15: Half mask inpainting FFHQ dataset.
34

---

A Mixture-Basce Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obsrvation
Sumple
Catalyst
Figure 16: Half mask inpainting on ImageNet dataset.
35

---

A Mixture-Basce Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obervation
Sumple
ors
orsi
Figure 17: Box inpainting on FFHQ dataset.
36

---

A Mixture-Band Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obsrvvtion
Sumple
R
Figure 18: Box inpainting on ImageNet dataset.
37

---

A Mixture-Basee Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obsrrvatio
Sumple
Figure 19; JPEG dequantization with QF = 2 FFHO dataset.
38

---

A Mixture-Basse Framework for Guiding Diffusion Models
Reference
Obsrvvatio
Sumple
Reference
Obervation
Sumple
S RS
RS
R
PER AKU STADIU PRR AKU STADIU
AKU STADIU
0
0
0
0
0
0
21
21
Figure 20: JPEG dequantization with QF = 2 on ImageNet dataset.
39

---

A Mixture-Basce Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obervation
Sumple
Figure 21: Motion deblurring on FFHQ dataset.
40

---

A Mixture-Basee Based Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obsrvation
Sumple
COGNAR
COGOC
Figure 22: Motion deblurring on ImageNet dataset.

---

A Mixture-Based Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obervation
Sumple
TRE
STRE
unn
Figure 23: SR(16x) on FFHC dataset.
42

---

A Mixture-Bascl Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obsrvation
Sumple
Figure 24: SR(16x) on ImageNet dataset.
43

---

A Mixture-Bass Based Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obervation
Sumple
Figure 25: High dynamic range on ImageNet dataset.
44

---

A Mexture-Basce Framework for Guiding Diffusion Models
Reserence
Observation
Sumple
Reference
Obervation
Sumple
OO
Figure 26: SR(4x) on FFHQ dataset with latent diffusion.
45

---

A Mixture-Baser Framework for Guiding Diffusion Models
Reference
Observation
Sumple
Reference
Obsrvvtion
Sumple
Figure 27: SR(16x) FFHQ dataset with latent diffusion.
46

---

A Mixture-Basce Framework for Guiding Diffusion Models
Refereuce
Observation
Sumple
Reference
Obsrvvaion
Sumple
Figure 28: Half mask on FFHQ dataset with latent diffusion.
47


Reviews:
[{"review_id": "wYvcGpAjaP", "reviewer": "Reviewer_WNdt", "summary": "The paper introduces Mixture-Guided Diffusion Model (MGDM) algorithm to improve likelihood approximation in diffusion models using Gibbs sampling. The approach constructs a mixture approximation of intermediate posterior distributions to address lack of closed-form likelihood scores. Data augmentation scheme uses Gibbs updates. MGDM is adaptable to computational resources by adjusting the number of Gibbs iterations, with higher number of iterations for improved performance. The method shows improved results across different image-restoration tasks and musical source separation.\nThis manuscript is well written!", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and helpful suggestions. Below, we directly address the points raised:\n\n>**1. Gibbs sampling can have convergence issues, even if you run more iterations. How do you address it?**\n\nWe acknowledge that Gibbs sampling can indeed exhibit convergence challenges, especially in high-dimensional or strongly correlated distributions. In fact, in Appendix B.1 we show that for one specific choice of weight sequence, convergence issues arise. More specifically, when the weight sequence is designed to sample the index $s$ close to $0$ at all iterations, MGDM mixes very slowly due to the high correlation between $x_0$ and $x_s$, since $s \\approx 0$. This motivates our approach of using intermediate uniform mixture posteriors, which bypasses these correlation issues and reduce dependence on the precise convergence of each Gibbs iteration, thereby enhancing overall robustness.\n\n>**2. Is there diversity in generated samples?**\n\nOur empirical results confirm that MGDM generates diverse samples. This is qualitatively evident in the visual samples provided in our supplementary materials; see, for example, Figure 2 and Figures 7–12.\n\n>**3. How to address the scaling issues of the algorithm?**\n\nWe interpret the reviewer’s question as referring to memory scaling. Our algorithm’s memory footprint is comparable to DPS and PGDM, but higher than methods like DiffPIR and DDNM. This increased memory usage enables superior reconstruction quality, particularly for inpainting and outpainting (Figures 5, 6, and 8 of the original manuscript). Achieving similar quality with lower memory requirements remains an open research question. Importantly, on latent diffusion, our memory footprint is the same as all the other methods.\n\nThank you very much for your feedback. We kindly refer you to our responses to Reviewers awaZ and SCZR, where we provide additional experiments and metrics that further support our claims.", "claims_and_evidence": "Please see the summary section.", "methods_and_evaluation_criteria": "The methods and evaluation datasets and criteria are sufficiently represented. Manuscript closely follows previously published works.", "theoretical_claims": "Gibbs sampler is used as an approximate posterior for the data augmentation of the mixture to represent the stationary distribution. This is used as conditionals to obtain the posterior distribution.  The proof is provided in the Appendix A.", "experimental_designs_or_analyses": "The experimental designs are appropriate for the defined problem, and are similar to previously published results.", "relation_to_broader_scientific_literature": "The use denoising diffusion models for the Bayesian inverse problems is well studied problem. The authors have done an excellent job in comparing with SotA approaches in this domain.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "Few topics that authors could discuss:\n1.\tGibbs sampling can have convergence issues, even if you run more iterations. How do you address it?\n2.     Is there diversity in generated samples?\n3.\tHow to address the scaling issues of the algorithm?", "other_comments_or_suggestions": "None.", "questions_for_authors": "Discussed above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O0MuQ0kmgq", "reviewer": "Reviewer_E9Pa", "summary": "To resolve the error in approximated likelihood gradient of diffusion posterior sampling (DPS) and relevant works, the paper defines a novel posterior density $p(x_t|y)$ as mixture of normalized $\\hat p_s(x_t|y)= \\hat p_s(y|x_t)p(x_t)$ where $\\hat p_s(y|x_t)= \\int \\hat p(y|x_s)p_{s|t}(x_s|x_t) dx_s$ , $\\hat p(y|x_s)= p(y|\\mathbb{E}[x_0|x_s])$ and $0<s\\leq t-1$. To sample from this intractable density, the paper also leverage a Gibbs sampling with Gaussian variational approximation.\n\n## update after rebuttal\n\nRegarding the first drawback in the evaluation: (2) Heavy reliance on a single metric (LPIPS) — the authors emphasize the inadequacy of pixel-wise metrics (PSNR and SSIM) for evaluating the specific use case of inverse problems. While the reviewer agrees that pixel-wise metrics can favor blurry or overly smooth outputs, these metrics remain important for assessing low-frequency information, such as color accuracy. Consequently, inverse problem solvers [1,2] typically report both pixel-wise and perceptual metrics, rather than relying on a single type. From this perspective, the reviewer believes that the evaluation in the paper places excessive emphasis on perceptual metrics.\n\nFor the other points, the major concerns have been resolved convincingly.\n\nIn summary, the reviewer thanks the authors for their efforts in the rebuttal and maintains the score of 3.\n\n[1] Direct Diffusion Bridge using Data Consistency for Inverse Problems, NeurIPS 2023\n\n[2] Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing, CVPR 2025", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your acknowledgment of the paper's clarity and the promising nature of our method. Below, we directly address your key points and questions, supported by additional results. The supplementary tables and figures mentioned can be accessed here: https://anonymous.4open.science/r/rebuttal-F9B0/rebuttal_tables.pdf.\n\n> **Drawbacks in evaluation: 1) Missing evaluation metric FID, and 2) Heavy reliance on a single metric (LPIPS).**\n\nFirst, we refer the reviewer to Appendix B.6, where we show that LPIPS is particularly suitable for our tasks, where reconstructions naturally deviate significantly from the reference images. On the other hand, pixel-based metrics (such as PSNR or SSIM) are inadequate for capturing meaningful perceptual differences in such scenarios, potentially leading to misleading conclusions. For instance, Figure 8 in the original manuscript clearly illustrates this limitation. Although reconstructions from methods like DAPS and DiffPIR appear overly smooth and lack coherence, they nevertheless achieve superior PSNR and SSIM scores, highlighting the inadequacy of these pixel-wise metrics for evaluating our specific use case.\n\nNevertheless, we acknowledge the importance of evaluating our method with diverse metrics, and therefore we have included FID scores in the revised evaluation. Please refer to Tables 1, 2, and 3 in the provided PDF link for detailed results. These additional results confirm that our method remains highly competitive in terms of FID scores across all three models evaluated.\n\n> **Could authors provide an analysis on efficiency of the proposed method compared with baselines? For example, runtime and memory cost.**\n\nPlease see Figure 1 in the linked PDF, which summarizes both runtime and memory consumption across our method and the relevant baselines. This figure will be integrated into the revised manuscript. \n\nOur method has memory requirements similar to DPS and PGDM in pixel space and aligns closely with other methods in latent space. Importantly, in latent diffusion—a highly relevant scenario given the prevalence of latent-space models—our method is notably faster than all competitors while consistently achieving strong performance across benchmarks.\nConversely, when operating directly in pixel space, our method exhibits somewhat slower runtimes compared to some alternatives. However, this increase in computational overhead is consistently balanced by improved and stable reconstruction quality across all considered tasks. Thus, we position our method as offering a beneficial trade-off, especially in scenarios where quality and consistency of results are paramount.\n\nWe highlight several important points regarding the competitors’ runtime: \n\n- For latent diffusion, DAPS and PnP-DM perform a significant amount of Langevin steps using the gradient of the likelihood. Since the latter involves a vector jacobian product of the decoder, the runtime increases significantly. More generally, when the likelihood function is expensive to evaluate, DAPS and PnP-DM are expected to be much slower than DPS, PGDM and MGDM. \n\n- For PnP-DM on FFHQ, we have implemented the likelihood step exactly on the linear tasks. On ImageNet however, using Langevin steps provided better results and this explains the significant increase in runtime. \n\nThank you very much for your feedback. Please let us if you have any further question. We kindly refer you to our responses to Reviewers awaZ and SCZR, where we’ve provided additional experiments and metrics that further support our claims.", "claims_and_evidence": "Main claim is that proposed mixture approximation of $p_t(x|y)$ is better than using likelihood approximation of $p_t(y|x)$, which are proposed by prior works, in two perspectives: principled way and adaptability to computational budget. Paper provides evidence via extensive experiments.", "methods_and_evaluation_criteria": "The paper evaluates their method on widely used benchmark datasets: FFHQ and ImageNet. Furthermore, authors provide extensive types of linear and non-linear problems. Drawbacks in evaluation is that 1) missing evaluation metric FID and 2) heavy dependency of analysis on a single metric LPIPS.", "theoretical_claims": "The reviewer has checked mathematical details given in the appendix.", "experimental_designs_or_analyses": "The reviewer checked the validity of experimental design. It follows the prior works in diffusion-based inverse problem solvers. However, the paper does not provide an analysis on its efficiency while it introduces Gibbs sampling with variational optimization.", "relation_to_broader_scientific_literature": "Inverse problem is applicable to side range of scientific problems. Thus, improving the performance for inverse problem leads to further impact to scientific literature too.", "essential_references_not_discussed": "The paper includes essential citations and related works.", "other_strengths_and_weaknesses": "Strength\n- Paper is written clearly and its motivation\n- Performance of the proposed method is promising\n\nWeakness\n - Analysis on efficiency is missing.\n - Reported metric in the main paper only focuses on a single perceptual metric, LPIPS.", "other_comments_or_suggestions": "No minor comments.", "questions_for_authors": "- Could authors provide an analysis on efficiency of the proposed method compared with baselines? For example. runtime and memory cost.\n- Could authors evaluate the FID for provided experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mgSkm5OeeF", "reviewer": "Reviewer_awaZ", "summary": "The paper presents a novel training-free guidance method that allows to samples from g(y|x_0)p(x_0) where p(x_0) is a pre-trained diffusion model distribution and g(y|x_0) is a likelihood function on the clean data. To do this, they come up with a novel approach to approximate the conditioned noisy distributions p(x_t|y) given the unconditional model, and on top of that, a new method to calculate the scores from these density approximations. The core of the method is that when sampling at diffusion noise level x_t, there is an inner loop that samples some s<t, and does Gibbs sampling from p(x_0, x_s, x_t)g(y|D(x_s)). This process defines a specific distribution over x_t, and the outer loop sampling process consists of moving through this sequence of distributions, until we hit p(x_0)g(y|x_0) at the end. To be more precise, the target distribution at each level is a mixture of the p(x_0, x_s, x_t)g(y|D(x_s)) distributions, with different probabilities for different s, and this is where the paper derives its name. The method achieves strong performance on a variety of linear and nonlinear inverse problem tasks on image data and multi-source audio separation. The method is also applicable for use with latent diffusion models. The authors also find that the Gibbs sampling procedure provides new ways to improve performance by applying more inference time compute.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We address the main weak points/questions below. The additional tables and figures mentioned below can be found here: https://anonymous.4open.science/r/rebuttal-F9B0/rebuttal_tables.pdf\n\n>  **comparison of runtime[...]**\n\nAlthough initially not mentioned (now corrected), we used the maximum diffusion steps (1000) for DPS, PGDM, DDNM, DiffPIR, and PSLD. For methods like RedDiff, DAPS, ReSample, and PnP-DM, we tuned the compute time by increasing Langevin/denoising/optimization steps until performance plateaued, ensuring optimal competitor performance. Hence, we are confident that we have pushed each competitor to their optimal performance limit. Please see Figure 1 for a figure summarizing the runtime and memory costs of each method. This figure will also be included in the revised version of our paper. Note that our method has similar memory demands as DPS and PGDM on pixel space, while in latent space its memory usage aligns with the other methods. Notably, in the latent-diffusion setting—which is particularly relevant given the prevalence of latent-space models—our method outperforms all others in speed while maintaining consistently strong performance across all benchmarks. In pixel space, our method is slower, but this comes with the benefit of consistent and strong performance. Please see our comment to Reviewer E9Pa regarding the runtime of the competitors. \n\n> **evaluation of scaling the inference-time [..]**\n\nWe thank the reviewer for the valuable suggestion. We conducted additional experiments analyzing performance evolution with hyperparameters $K, M, G$—representing regular diffusion steps, Gibbs repetitions, denoising steps in Gibbs for \\\\(p(x_0|x_s)\\\\), and gradient steps for approximating $q(x_s|x_0,x_t) p(y|g(D(x_s))$, respectively. Specifically, we extended Figure 3 to explore increased compute along different axes for the phase retrieval task (see Figure 2). We concluded that scaling diffusion, denoising, or gradient steps have less impact for this task than increasing Gibbs steps.\n\n> **model description [...]**  \n\nThank you for your suggestion. We acknowledge some parts of our presentation could be more apparent. To improve accessibility, we've reorganized Section 3, which begins with a concise conceptual overview before detailing the algorithm step-by-step. We have also added the main intuition behind latent variables $x_0, x_s$, and $x_t$ and their iterative evolution through denoising, noising, and variational updates.\n\n> **Why do we introduce the intermediate s-timestep [...]**\n\nOur method introduces two latent variables $x_s, x_0$, enabling updates without relying on unrealistic approximations of the denoising distributions. This contrasts with approaches like DAPS, which approximate the posterior distribution $x_0 \\mid x_t$ using a Gaussian distribution parameterized by a tunable covariance. Furthermore, our method leverages likelihood approximations derived from earlier diffusion steps $s < t$, a perspective that initially motivated our algorithm. Additionally, framing our approach through the lens of Gibbs sampling provides a novel dimension for performance enhancement. These three aspects—accurate latent variable integration, leveraging historical likelihood approximations, and exploiting Gibbs sampling insights—are currently underexplored and could significantly benefit the research community.\n\n> **Why is sampling s close to 0 is a source of instabilities?” + Why do we use a mixture of timesteps s instead of a fixed s schedule?**\n\nAs we explain in Appendix B.1, sampling $s$ close to 0 leads to an accurate likelihood approximation, which is particularly important for latent diffusion. This allows us to fit the observation quickly; however, the unobserved part of the state evolves very slowly, leading to a poor reconstruction of the unobserved part of the state; see Figure 4. This occurs because the prior $q_{s|0, t}(\\cdot|x_0, x_t)$ is concentrated around $x_0$ when s is close to 0. Conversely, when $s$ is far from 0, the likelihood approximation is less accurate but the prior becomes less constrained. Using a uniform mixture balances these opposing behaviors effectively, with minimal hyperparameters.\n\n> **[...]Boys et al. do use the network Jacobian, and as such the covariance is not constant w.r.t. $x_t$** \n\nIn Boys et al., the likelihood approximation (Eqn. (11), arXiv version) explicitly assumes that the covariance—and consequently the Jacobian of the denoiser—is constant w.r.t $ x_t $. Without this assumption, differentiating Eqn. (10) would not lead directly to Eqn. (11). Indeed, Boys et al. explicitly mention: ‘For this reason, we treat the matrix $C_{0|t} $ as constant w.r.t. $x_t$ when computing the gradient.’ Similarly, [2] employs the same assumption (see Eqn. (23), OpenReview version, and the corresponding commentary).\n\nAgain, we thank the reviewer for the valuable feedback. Please let us now if you have any further questions.", "claims_and_evidence": "I think most of the claims are supported by evidence.", "methods_and_evaluation_criteria": "The evaluation criteria makes sense for the application at hand, and the method is sensible as well. The method does, however, have some complexity, and the motivation for the particular choices in the method are not entirely clear for me.", "theoretical_claims": "I didn’t go through the details of the method very closely in the Appendix (although the main calculation-wise involved new part seems to be Appendix A.2.). I think I have understood the mathematical definition of the model, and it is sensible to me.", "experimental_designs_or_analyses": "The main experiments on imaging inverse problems are standard benchmarks in the field, and are sound applications to focus on. One issue is that I did not find a comparison of runtime or neural function evaluation count for the different methods. As pointed out in the paper, they are able to improve results with more inference-time compute, but this also holds for the other methods (at least by increasing the amount of diffusion steps for methods like DPS, PGDM, DDNM and DiffPIR, and by increasing the amount of optimization steps in methods like Reddiff). It is useful to be able to push the results further than previous methods in absolute terms, but ideally we would also have an analysis of the different methods at different levels of compute to clearly distinguish the regime in which the method provides improvements over prior work. In case an extensive evaluation of all of the competing methods at different compute requirements is difficult, I think that it would at least be important to include the compute requirements for producing the results. Based on Table 5 in the Appendix and my understanding of the method, in pixel-space FFHQ, it seems that the method is using on the order of >2700 forward passes and >700 backward passes of the denoiser? \n\n\nAnother part I found a bit lacking was the discussion on the hyperparameters K, R, M and G (regular diffusion steps, Gibbs repetitions, denoising steps in the Gibbs inner loop for p(x_0|x_s) and gradient steps for approximating q(x_s|x_0,x_t)p(y|g(D(x_s))). Would it be possible to provide a bit more thorough evaluation of scaling the inference-time compute along these different axes? This would be especially interesting considering that the authors found increasing R in the imaging inverse problem case to be more useful than increasing G, and vice versa for the audio source separation case.", "relation_to_broader_scientific_literature": "The paper continues exploring new methods for applying inference-time conditions on diffusion models without changing the denoiser network. Many works in the area have focused on directly approximating the modifications needed to the diffusion model score function, e.g., through Tweedie’s formula (e.g., [1], [2]), allowing the use of standard diffusion samplers. Other works, including this one, remove the requirement of using standard diffusion sampling processes, and instead redefine the sequence of marginal distributions, using MCMC-like methods to move to lower noise levels. This paper is in the latter category, and, to the best of my knowledge, proposes a novel method in this area. The central idea of introducing a triplet of timesteps (0,s,t) to update x_t to match with the constraint, using approximations at the s stage, is new, although  related to earlier some earlier work (as detailed in Appendix A.5 and the related works section “Gibbs sampling approaches”). \n\n[1] Chung, Hyungjin, et al. \"Diffusion Posterior Sampling for General Noisy Inverse Problems.\" The Eleventh International Conference on Learning Representations.\n[2] Song, Jiaming, et al. \"Pseudoinverse-guided diffusion models for inverse problems.\" International Conference on Learning Representations. 2023.", "essential_references_not_discussed": "I am not aware of key contributions in this domain that are missing, although I was not very familiar with the DAPS and MGPS algorithms cited as the closest related work.", "other_strengths_and_weaknesses": "I think that the idea of enlarging the design space of conditional samplers by considering triplets (x_0,x_s,x_t), performing approximations in strategic locations of the algorithm, and defining custom Gibbs samplers, appears sensible and is original. The quantitative results against baselines are good as well, and the method appears promising. \n\nA negative (aside from the experiments mentioned before) is that the model description was somewhat difficult to read, and the paper would benefit from improving the writing. For instance, having an overview and stating the key approximations up front before going to the weeds of the algorithm description could be useful. A more clear explanation of the method and the problems it tackles in the introduction would be useful as well. \n\nOn the actual algorithm side, the motivation for the particular design choices was left a bit lacking (see questions). The details of how the s distribution is chosen is also not particularly elegant, which is not a major issue since there is some analysis on it. Due to the concerns raised, I am starting out with a weak reject, but I am open to adjust after the rebuttal.", "other_comments_or_suggestions": "This is not a key concern, but it is mentioned in the Related Works that “Finzi et al., Stevens et al., Boys et al.” use that the covariance of p(x_0|x_t) is proportional to the Jacobian of the denoiser… To mitigate this, these works and subsequent ones assume that the Jacobian of the denoiser is constant with respect to x_t.” I don’t think this is exactly true: at least Finzi et al. and Boys et al. do use the network Jacobian, and as such the covariance is not constant w.r.t. x_t.  This is also true in subsequent work that does not use the network Jacobian for the covariance approximation ([1] and [2]).\n\n[1] Peng. et al. \"Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance.\" ICML 2024\n\n[2] Rissanen. et al. “ Free Hunch: Denoiser Covariance Estimation for Diffusion Models Without Extra Costs.” ICLR 2025", "questions_for_authors": "The method is quite complex, and it is a priori somewhat unclear why would we use this, instead of, e.g., the methods mentioned in Appendix A.5. Why do we introduce the intermediate s-timestep in the first place? Why do we use a mixture of timesteps s instead of a fixed s schedule? \n\nDo you have an intuition on why is sampling s close to 0 is a source of instabilities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2adqBcWSbG", "reviewer": "Reviewer_SCZR", "summary": "This paper explores solving linear and nonlinear inverse problems—sampling from $p(\\mathbf{x}_0|\\mathbf{y})$—using pre-trained unconditional diffusion models in a Bayesian framework. To approximate the posterior, the authors iteratively sample from intermediate distributions $p(\\mathbf{x}_t|\\mathbf{y})$, where the prior is given by the unconditional score network, but the likelihood is intractable. Their key contribution is a Gibbs sampling-based method to approximate and sample from $p(\\mathbf{x}_t|\\mathbf{y})$. At each denoising step, they perform $R$ Gibbs iterations, each consisting of: (1) $G$ gradient steps to fit a variational distribution, (2) sampling from the unconditional diffusion model using $M$ DDPM steps, and (3) a closed-form sampling step from the noising process. \n\nThe method is evaluated on linear and nonlinear inverse problems in both pixel and latent space, as well as on a linear audio source separation task. It performs well on pixel-space tasks (though sometimes underperforms competitors) and generally surpasses benchmarks in latent-space tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our paper. We address your main weak points/questions below. The additional tables we discuss below can be found here: https://anonymous.4open.science/r/rebuttal-F9B0/rebuttal_tables.pdf\n\n> **[...]  at least one example with a non-Gaussian likelihood [..].** \n\nIn the initial version, we prioritized extensive comparisons against multiple methods across two modalities. Following your suggestion, we have conducted preliminary experiments with Poisson noise, comparing our method  against DPS due to their similar experimental setup. For MGDM, we directly employed the Poisson likelihood without resorting to the Gaussian approximation and maintained the original hyperparameters. In contrast, we used the Gaussian approximation for DPS as outlined in Eqn. (19)[arXiv version], since the Poisson likelihood proved challenging to implement effectively and the original DPS paper lacks guidelines for this scenario. The results are detailed in Table 6, clearly showing that MGDM outperforms DPS across all considered metrics. Additionally, to address potential concerns regarding MGDM’s performance under higher noise conditions, we have included an extra benchmark with a noise standard deviation of 0.3, presented in Table 5.\n\n> **Report runtime [...]  Add FID as a metric** \n\nWe have implemented all of your recommendations. Please see Fig. 1 and Tables 1, 2, and 3 in the attached document. We refer to our response to R. E9Pa for comments on our runtime and that of the competitors. Our method consistently achieves competitive FID scores across all three evaluated models. \n\n> **better hyperparameter choices**\n\nPGDM has no official implementation, and directly implementing Algorithm 1 yielded suboptimal results. To ensure a fair comparison, we instead utilized the authors' official RedDiff implementation, where the guidance term is scaled by $\\\\alpha_{t-1}\\\\alpha_t$ rather than solely by $\\alpha_t$ (lines 70 and 73 in https://github.com/NVlabs/RED-diff/blob/master/algos/pgdm.py, with grad_term_weight=1). This modification improved PGDM's performance across most tasks, except for JPEG2, where the original formulation was superior (line 71 in our pgdm.py). Supporting this observation, Boys et al. (2023, Table 2) and Rozet et al. (2024, Table 4) also reported that PGDM generally underperforms DPS. For a fair comparison, we carefully tuned PGDM at 1000 steps to match compute budgets, despite Rozet et al.'s suggestion that fewer steps (around 100) might yield better results. Even under these conditions, PGDM at best matches DPS performance, reinforcing our conclusion that MGDM consistently outperforms both methods. We stress that we promote the reproduction of all benchmarks (as detailed in Appendix B.4 and our openly accessible codebase), to ease verification by the community.\n\n> **[...] number of Gibbs sampling steps [...]**\n \nIn  Fig. 3 of the paper, we demonstrate that performance consistently improves as the number of Gibbs steps increases for the audio separation task, surpassing all training-free posterior sampling methods. Furthermore, increasing the number of gradient steps allows our approach to exceed Demucs' performance. Across all experiments, we've consistently observed benefits from increasing Gibbs steps, whereas additional gradient steps yield diminishing returns beyond a certain threshold. Specifically, we selected $R=1$ for the image experiments as it provides the optimal balance between computational efficiency and competitive performance relative to other baselines.\n\n> **effect of $M$, the number of DDPM steps [...]** \n\nWe selected $M=20$ as it offered a trade-off between computational cost and image quality. While increasing $M$  enhances image quality, it does not significantly improve posterior exploration, in contrast to Gibbs and gradient steps. However, in response to your recommendation, we conducted additional experiments to investigate how varying $M$ and the number of diffusion steps affect performance; results can be found in Figure 2. Furthermore, regarding the choice of the weight sequence, we already provide in Apdx B.1 an empirical analysis of the strategy of sampling $s$ close to $0$ at every step. \n\n- **MC estimate of the KL:** We also found this result surprising, given the performance gains observed. Specifically, we discovered that estimating the KL divergence using the same noise as for the likelihood expectation effectively reduces the variance of the gradient estimator, leading to improved results.\n- **Comparison to MGPS** We have prioritized comparing well-established methods and recent contenders. Nevertheless, we commit to comparing with MGPS in the revised version of the paper.\n-  **non-diagonal covariance** We have already tested adding a low-rank perturbation to the diagonal covariance matrix but found that it doesn’t significantly improve the result. The most significant improvement comes from using a diagonal instead of a scalar matrix.", "claims_and_evidence": "The paper claims that the proposed method achieves performance on inverse problems that is either comparable to or better than related approaches. While the study includes a broad and relevant set of baselines, a few improvements could strengthen the validity of these claims:\n- **Report runtime**: Performance metrics alone are insufficient without the corresponding runtime. Including runtime (and memory requirements, if relevant) would provide a more complete comparison.\n- **Include standard deviations / confidence intervals**: Reporting only the mean values of LPIPS, PSNR, and SSIM does not fully convey the statistical significance of the results. Adding standard deviations / confidence intervals would help assess statistical significance.\n- **Add FID as a metric**: This is commonly used in inverse problems solved with pre-trained diffusion models, so I would have expected to see it in the image tasks.\n\nFinally, the authors claim that a strength of the method is the possibility to adjust the number of Gibbs sampling steps $R$, besides the number of gradient steps $G$ from the variational approximation, to enhance performance.  However, allocating more compute to Gibbs sampling rather than gradient steps appears beneficial only in the phase retrieval task, while for source separation, increasing $G$ seems to be the better strategy. In the other image-based experiments (aside from phase retrieval), only results for $R=1$ are reported, likely because this provided the best tradeoff between performance and runtime. While this is not necessarily an issue, the paper should more clearly specify in which tasks increasing Gibbs sampling steps leads to improvements and when prioritising gradient steps is the preferable approach.", "methods_and_evaluation_criteria": "The benchmark datasets do make sense, and I appreciate the fact that the authors provide experiments using both pixel- as well as latent-space models, and on both linear and nonlinear inverse problems. The audio task is also a good addition.\n\nHowever, I would also suggest evaluating FID to provide a more complete assessment of generative quality. Additionally, as mentioned earlier, reporting standard deviations or confidence intervals alongside the mean metrics would help assess the statistical significance of the results.\n\nFor a more comprehensive performance analysis across diverse inverse problems, it would have been useful to include at least one example with a non-Gaussian likelihood and perhaps a setting with higher noise. However, I appreciate that computational constraints may have limit the amount of experiments that can be conducted.", "theoretical_claims": "Yes, I did go through the proofs and, as far as I am concerned, they are correct. However, I do believe that introducing $s$ as a state in the Gibbs sampling procedure is better motivated theoretically, but I agree that it is computationally infeasible.\n\nThe only equations I disagree with are:\n- Equation (14) where the integration in the denominator should be over $\\mathrm{d} \\mathbf{x}_{s', t'}$\n- The one in the appendix (line 800 in A.5), though I believe it is likely a typo.", "experimental_designs_or_analyses": "The authors provide details on the sources of the baseline implementations, which generally seem reasonable. The tasks considered are fairly standard and have been explored in previous works, including those the authors compare against.\n\nI do wonder, however, whether better hyperparameter choices for the baselines could lead to improved performance. While tuning baselines extensively may be beyond the scope of the paper, I was surprised by the underperformance of PGDM vs. DPS. In the linear setting, I would have expected PGDM to perform comparably to or better than DPS. If this was not observed, it might have been because of suboptimal hyperparameter settings for PGDM.\n\nThe paper also makes several heuristic choices, which I feel are decently motivated, but might benefit from some extra investigation. Some of these are mentioned in the paper, but they include: the weight sequence, the number of DDPM steps $M$, the interaction between $R$ and $G$. However, there are so many hyperparameters, that I appreciate it is hard to gain a comprehensive of how each one of these affects performance, and whether this is task-dependent or not.", "relation_to_broader_scientific_literature": "The paper contains an extensive overview of the relevant literature, highlighting the most relevant works that tackle Bayesian inverse modelling leveraging pre-trained diffusion models. The alternative likelihood approaches are well captured, the authors mention the relevant SMC approaches, and also compare to the most closely related methods based on Gibbs sampling. The detailed comparisons in Appendix A.5. are particularly useful in clarifying the distinctions between MGDM and the closely related DAPS and MGPS methods.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "**Strengths**\n- S1: Able to handle pixel- and latent-space diffusion out of the box.\n- S2: Able to tune performance by tweaking two different hyperparameters: $R$ and $G$, although also see weakness 3.\n- S3: Clear comparison to related methods.\n- S4: Strong empirical performance in the majority of cases, although this should be analysed alongside runtime metrics and also include standard deviations besides mean metrics.\n\n**Weaknesses**\n- W1: One of the main weaknesses is the lack of computational time analysis when comparing to the baselines. In my perspective, this is crucial to holistically compare different related posterior sampling methods, especially when some metrics are so similar.\n- W2: Another weakness from the evaluation side is, as mentioned above, the lack of standard deviations / confidence intervals in the results.\n- W3: Although the authors stress that the ability to increase performance through increasing $R$ is a strength of the algorithm, this doesn’t seem to be the best strategy in all cases (i.e. directing compute to gradient steps is more lucrative in source separation). In general having too many hyperparameters can become overwhelming, especially if they are task- and domain- dependent. I am not convinced that currently the paper contains enough settings for the authors to make some clear recommendations.\n- W4: There is also limited exploration of the effect of $M$, the number of DDPM steps. Why did the authors go with $M=20$ and have other choices been explored too? Is it clear that having a fixed $M$ value is the optimal choice, rather than potentially having it depend somehow on $s$?\n- W5: All tasks consider either no noise or Gaussian noise with fixed $\\sigma_y=0.05$. This does not make it clear whether the method would perform well under non-Gaussian likelihoods.", "other_comments_or_suggestions": "- Line 201 right: “Treating s **as** fixed”\n- Line 315 right - you only compare to seven competitors.\n- The reference to Wu et al. [2024] [1] is repeated twice (2024a and 2024b). Same with Zhang et al. [2024] [2]\n- Line 303 right: **“repeatedly”** instead of **“repeatidly”**\n- Perhaps when mentioning the scaling from works such as DPS (L122 right) it is also worth highlighting that the scaling factors are heuristic, rather than very well theoretically underpinned.\n- Appendix A.2. Line 635 Equation (14) - the integration in the denominator should be over $\\mathrm{d} \\mathbf{x}_{s', t'}$ I think\n- Typo on line 800 - I think it should be $\\pi^{\\mathbf{y}}_{0|t+1}(\\mathbf{x}_0|\\mathbf{x}_{t+1})$\n- Typo on line 956: “their\" instead of \"there variables”\t\n- Line 1009: What do you mean by you “exposed” the coupling parameter $\\rho$?\n- Typo 1036: “fine-grained” instead of in-grained” + that **are** more coherent\n\n[1] Wu, Z., Sun, Y., Chen, Y., Zhang, B., Yue, Y., and Bouman, K. L. Principled probabilistic imaging using diffu- sion models as plug-and-play priors. arXiv preprint arXiv:2405.18782, 2024a \n[2] Zhang, B., Chu, W., Berner, J., Meng, C., Anandkumar, A., and Song, Y. Improving diffusion inverse problem solving with decoupled noise annealing. arXiv preprint arXiv:2407.01521, 2024a", "questions_for_authors": "Besides the already mentioned questions from the previous sections (mainly from the **Weaknesses** section), I have a few more questions:\n- Q1: Remark A.1. from Appendix A.3 - I find it surprising that computing the MC estimate for the squared norm in (17) gives better results than computing it analytically. Any insight as to why this might be the case?\n- Q2: Why don't you provide comparisons to MGPS?\n- Q3: This might be hard to estimate, but do you think a variational approximation with a non-diagonal covariance matrix would give significant improvements in the method? Or do you think that would be too complicated for relatively small improvements?\n- Q4: Do you have any intuition of how accurate the Gaussian approximation for $\\hat{\\pi}^{\\mathbf{y}}_{s|0, t}$ is for (possibly nonlinear) cases with non-Gaussian noise? I realise this might actually depend on how close $s$ is to $t$ and $0$, but just curious to see if you have any intuition about this.\n- Q5: For the Gibbs sampler procedure (Algorithm 1) - could you do the steps in another order? I agree that the one you chose seems to be the most natural one, but just wondering whether this is a possibility and how you think it might affect the results. In the limit, it should converge to the same distribution, right?\n\n## Update after rebuttal\nI am satisfied with how the rebuttal has addressed my concerns. In particular, I found the addition of the runtime and memory comparisons with competitors valuable additions to the paper. As a consequence, I increased my score to 4 (Accept).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has received all acceptances in the final recommendations. It presents a novel contribution by introducing the Mixture-Guided Diffusion Model, which aims to enhance likelihood approximation in diffusion models through the application of Gibbs sampling. While the reviewers initially raised concerns regarding the clarity of the method's explanation and the lack of efficiency analysis, the authors' rebuttal successfully addressed many of these issues. This ultimately led to a consensus among the reviewers in favor of acceptance. As a result, the Area Chair has decided to accept this paper.
Decision: Accept (poster)