Repulsive Latent Score Distillation for Solving Inverse Problems
Authors: Nicolas Zilberstein
Morteza Mardani
Santiago Segarra
Venue: ICLR 2025 Poster (2025)
Keywords: Inverse problems
Score distillation
Diffusion models
Mode collapse
Variational inference
Primary area: generative models
TL;DR: We propose Repulsive Latent Score Distillation, a particle-based variational method to solve inverse problem  at 512 x 512 resolution using latent diffusion models as prior.

Abstract: Score Distillation Sampling (SDS) has been pivotal for leveraging pre-trained diffusion models in downstream tasks such as inverse problems, but it faces two major challenges: $(i)$ mode collapse and $(ii)$ latent space inversion, which become more pronounced in high-dimensional data. 
To address mode collapse, we introduce a novel variational framework for posterior sampling. 
Utilizing the Wasserstein gradient flow interpretation of SDS, we propose a multimodal variational approximation with a \emph{repulsion} mechanism that promotes diversity among particles by penalizing pairwise kernel-based similarity. 
This repulsion acts as a simple regularizer, encouraging a more diverse set of solutions. 
To mitigate latent space ambiguity, we extend this framework with an \emph{augmented} variational distribution that disentangles the latent and data. 
This repulsive augmented formulation balances computational efficiency, quality, and diversity. 
Extensive experiments on linear and nonlinear inverse tasks with high-resolution images ($512 \times 512$) using pre-trained Stable Diffusion models demonstrate the effectiveness of our approach.

Published as a conference paper at ICLR 2025
REPULSIVE LATENT SCORE DISTILLATION FOR SOLVING INVERSE PROBLEMS
Nicolas Zilberstein
Morteza Mardani NVIDIA Inc rdan @nvidia.........
Rice University nzi Reileerseeine wew.eiiieeiieeeeee.
Santiago Segarra"
Rice University segarraerice.edi
ABSTRACT
Score Distillation Sampling (SDS) has been pivotal for leveraging pre-trained dif- fusion models in downstream tasks such as inverse problems, but it faces two major challenges: (t) mode collapse and (it) latent space inversion, which become more pronounced in high-dimensional data. To address mode collapse, we introduce novel variational framework for posterior sampling. Utilizing the Wasserstein gradient flow interpretation of SDS, we propose multimodal variational approx- imation with repulsion mechanism that promotes diversity among particles by penalizing pairwise kernel-base similarity. This repulsion acts as a simple reg- ularizer, encouraging a more diverse set of solutions. To mitigate latent space ambiguity, we extend this framework with augmented variational distribution that disentangles the latent and data. This repulsive augmented formulation bal- ances computational efficiency, quality, and diversity. Extensive experiments on linear and nonlinear inverse tasks with high high-resolution images (512 x 512) using pre-trained Stable Diffusion models demonstrate the effectiveness of our approach. The code is available at GittHub
INTRODUCTION
Diffusion models have recently achieved remarkable success in visual domains. key application of these models is solving various inverse problems in plug-and-play manner, where diffusion models act as rich priors to regularize the search space, ensuring the generation of plausible solutions. Variational samplers (Poole al. 2022: Mardani et al., 2024) approach sampling an optimization problem, providing a high degree of control and fidelity in generation. However, they encounter two significant challenges, particularly when dealing with high-dimensional data that requires diverse outputs: (cl) mode collapse, and (c2) inversion of the latent space, such as that seen in the adversarial autoencoder of Stable Diffusion (Rombach et al. 2021).
There have been few recent attempts to address these challenges separately in the context of text-to-image/3D generation and inverse problems. To mitigate (cl), for text-to-3D generation, ProlificDreamee (VSD) (Wang et al. 2024) introduces data-driven dispersion with independent particles. Still, the combination of independence and unimodal approximation per particle renders an optimization that collapses to the same local minimum, limiting diversity. Collaborative Score Distillation (CSD) (Kim al. 2023) seeks to diversify the variational approximation using Stein Variational Gradient Descent (SVGD) (Liu and Wang 2016), but smoothing particle gradients with SVGD is problematic in high- nigh-dimensional spaces (D' Angelo and Fortuin 2202 102la
Ba et al. 202I). To et al. [0244] Song ct al. 2024) remain computationally demanding, similar to earlier baxel-based methods (Chung et al. 2022a Song et al. 2022), due to multiple correction steps required for deviations from the image manifold, a challenge arising from adversarial training of autoencoders, akin to GAN inversion (Xia al. 2022 Daras al. 2021). J. Thus, no current solution effectively handles both mode collapse and
address (c2), recent samplers using latent diffusion models (Rout et al.
latent space issues in inverse problems.
"Equal advising.

---

Published as a conference paper at ICLR 2025
wilk nqrabiis
Trajectory of parsáiles
repulaion
p(xal>)
Repulsive Latent Score Distillation
(1)
Repulsion
X
Zig
ZO
750
(3)
Duffusion dessotser
Xg
(2)
(3)
Particles
y
0 1 760
Meustrement
Augmentation
Regularization
|ly 1(x6'111 + | D(x 2
c(a," .t) c Vannoo Σ log k(m,"" 2()
Figure 1: Illustration of Repulsive Latent Score Distillation (RLSD): propagates set particles by adding noise and applying two levels regularization: (4) Denoising, via score-matching regularization, which directs particles toward modes of the distribution p(xuly) (blue arrows); and (ii) Repulsion, which pushes particles apart (red arrows) to explore other regions of the posterior density. During sampling, the repulsion gradient ensures particles remain separated, leading to different modes, shown in the upper- right box.
We hypothesize that the primary issue with (cl) arises from collapse in high-dimensional spaces. To address this, we employ an ensemble of interactive particles with repulsion to prevent collapse. Inspired by kernel-density estimation (D Angelo and Fortuin 2021b), we introduce a particle-based multimodal variational approximation that incorporates repuisive forces. These forces are defined through pairwise interactions based on similarity, such as using radial basis kernel of DINO features Caron et al. 2021).
To address (c2), we propose a variational augmented distribution that jointly optimizes the latent and data variables, similar to half-quadratic splitting QGeman and Yang. 1995). This method disentangles the latent (prior) from the data (measurement), yielding solutions with sharper details. We show that KL minimization of our augmented interactive particle approximation leads to score-matching regularization with two gradient terms: (a) denoising regularization along the entire diffused trajectory, and (b) repulsion regularization to encourage diversity in the latent diffusion's 's trajectory; Fig. We refer to our method as Repulsive Latent Score Distillation (RLSD).
We validate the advantages of RLSD through extensive experiments both linear and nonlinear tasks, using Stable Diffusion as the prior. In diversity-crittcs cases such as inpainting and phase retrieval, our method provides solid trade-off between diversity and quality. For tasks where diversity is less essential, like deblurring, our augmented formulation offers a fast solver by avoiding score Jacobian computations and performs efficiently on high nigh-resolution (512 512) images. Overall, RLSD combines the strengths of variational samplers (memory and compute efficiency) and posterior samplers (diversity). enabling control over speed and diversity by adjusting the scalar weights between denoising and repulsion regularizations. A detailed comparison of RLSSD's properties is summarized in Table
All in all, the main contributions of this paper are summarized follows:
We propose Repulsive Latent Score Distillation (RLSD), a variational posterior sampler for general inverse problems with high-resolution images (e.g., 512 512), that trades-off diversity for quality in a controllable fashion simply via regularization weights.
We introduce a repulsion regularization to boost the diversity via an interactive particle-bar based variational approximation inspired by Wasserstein gradient flow.
To handle the latent space inversion, we propose distribution augmentation that decouples the latent and pixel space, rendering two-step optimization problem.
We perform extensive experiments for various (non)linear inverse tasks using Stable Diffusion.
The results indicate the superior performance of RLSD over existing alternatives such as
PLSD (Rout al. 2024), DPS (Chung al. 2022a and RED-Diff (Mardani et al., (024).
2

---

Published as a conference paper at ICLR 2025
Posterior samplers
Variational samplers
DPS
PSLD
RED-Diff
VSD
CSD
RLSD (Ours)
Diversity (Low-dim)
X
Diversity (High-dim)
/
X
X
x
Latent Diffusion
X
V
Linear Inv. Problems
J
V
Nonlinear Inv. Problems
No Score Jacobian
X X V
/
Table I: Comparison of our work with DPS, RED-Dift, PSLD, VSD, and CSD. Our method combines the strengths of variational samplers (no score Jacobian, computational efficienccy and posterior sampling algorithms (diversity high dimensions). In addition, formulation enables to solve nonlinear inverse problems 512 512.
RELATED WORKS
This paper is primarily related diffusion models at its core, and two related lines of work: inverse problems and score distillation sampling.
Diffusion models for inverse problems. Several works have used diffusion models as priors to solve inverse problems in various domains (Daras al. 2024 Kong et al.) 2020). A recent approach termed RED-Diff (Mardani et al., (024) uses variational inference for solving inverse problems with diffusion priors, similar to plug-an and--ala methods /Venkatakrishnan al. (013): see
also Zhu et al. (2023) Zhang et al. 2021). This method employs the diffusion model as denoisers at different scales, akin to the RED framework (Romano al.. 2017). Despite successfully balancing quality and runtime, it suffers from mode collapse due to the unimodal approximation. Furthermore, optimizing directly the pixel domain restricts the ability to leverage latent diffusion models like Stable Diffusion (Rombach et al. 202 I) for solving inverse problems at high-resolution. Recent works incorporate latent diffusion models as prior (Rout al., 2024] Chung al. 2024 Song et al. 2024). While they partially alleviate the computational demands of pixel-domain solvers, they introduce additional steps to correct the deviations from the image manifold, which arise from adversarial training of the autoencoder. Thus, it is still an open problem to develop methods that are fast, promote diversity, and optimize the latent space of the diffusion model.
diversity and mode collapse. Recently, SDS enabled the of pretrained diffusion models for text-to-3D generation (Poole et al. 2022). Although SDS provides an efficient mechanism for the aforementioned task, it often suffers from mode collapse and saturated images; see details about SDS and its formulation in Appendix ProlificDreamer (Wang et al., 2024) aims to fix the mode collapse using data-driven dispersion fine-tuned each iteration via LoRA (Hu et al. 2021). It is, however, costly, and the independence of particles hinders diversity. Recently, the authors in (Kim et al. 2023) propose to use the well-known Stein variational gradient descent (SVGD) as an update direction, which yields an interactive particle system. However, it is known that SVGD suffers from the curse of dimensionality (D' Angelo and Fortuin (201aa). Other related works are Armandpour et al. (2023). where the authors leverage the negative prompt eliminate undesired perspectives, and Wang et al. (2023). where an entropic regularization proposed.
Score distillation:
BACKGROUND
We review latent diffusion models in Section and we briefly discuss how they are incorporated as priors to solve inverse problems in Section 3.2
3.1 DIFFUSION MODELS IN THE LATENT SPACE
Diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020 Song et al. (021bb consist of two
processes modeled using stochastic differential equations: 1) forward process that gradually adds noise to a clean image, and 2) a reverse process that learns to generate images by iteratively denoising the diffused data. In latent diffusion models /Vahdat al. 2021 Rombach al. 2021), the data Xo
3

---

Published as a conference paper at ICLR 2025
is encoded into latent space through encoder E(xo) Zu. and the forward process follows the pariance-preservve SIDE (Song et al. 2021b) in the latent space: dz, 2 (t)z, V (())dW t. for é E [0, T]. Here, ((t) a function that defines a step size for each é from 0 to T, and is defined as {(t) == Smin + (Bmax - Smin) +. and W, is the standard Brownian motion. The forward process is designed in such a way that the distribution of Zr converges to a standard Gaussian distribution. Given the forward process, the reverse process is defined as dz,
-2)(t)
((t)V,, 2, p(zz) + / Wt), where Zz, p(z;) is the score function, which is unknown. To map back to the ambient space, we pass the generated sample Zg through a decoder D(zo) = Xu-
Therefore, to solve the reverse process and use it for sampling, the score function (V log p(z))) encoder (E), and decoder (D) are learned by minimizing the denoising score- more-mnaching loss (Vincent 2011). Diffused samples are generated as Z; 0n + O,E, where Zo encodes X. Pdata (x). and
Ot 1-ee e -i B(a)ds and or = /1 o.. The score function is approximated by Eg (zc. t) s -o Vx, logp(z)), and the score-matching loss is minimized. After training, samples are generated using samplers like DDPM (Ho et al. 2020 and DDIM (Song al. 2020).
3.2 INVERSE PROBLEMS WITH DIFFUSION PRIORS
In general, inverse problem aims to find an unknown signal Xa given some noisy measurement y. related via some forward model f()).
y (xx) +v, V V~~ ((0,00
where the forward model is domain dependent. In a Bayesian framework, the solution boils down to sample from the posterior p(xoly) ly) x p(y) p(x)) where p(y(xo) is the measurement model and p(xo) is the prior imposed by the diffusion model.
Diffusion posterior sampling approaches. These methods generate a sample from the posterior by running the reverse process (see Section 3.1 using conditional score at obtained via Bayes' rule as
Vx, log p(x:ly) = Vx, logp(y)x.) x.) + Vx, log p(x))
While the second term uses pre-trained diffusion model, the first is intractable, seen from ply xx) p(y Xx bp(xx Xo) xx )dx,. Prior works (Chung et al. 2022a Song et al. 2022 Kadkhodaie
and Simoncelli 2021 Song et al. 2023) address this with a Gaussian approximation of p(x, xx) using Tweedie's s formula E [x
(xi (x, t)). Still, this requires the computation of the score Jucobian, which is compuiationally expensive, especially for pixel-based models at high-resolution. This can be partially addressed by using suitable latent space (Rombach et al.) 2021), alleviating the computational demands of pixel-domain solvers for high-resolution images. However, discussed in the previous section, it introduces additional steps which arise from the adversarial training of the autoencoder (Rout et al.) 2024). We defer more details to Appendix B
Variational inference approaches. Recently, RED-diff was introduced in Mardani et al. (0024). which avoids computing the score Jacobiar RED-diff frames the sampling problem as stochastic optimization by minimizing the KL divergence
q(zoly) = argmin (Lgg(zol))lp(zol>>
q(2o y)
where X,, D(zo). When q(zoly) N(Hz,of1) the KL minimization (3) boils down maximuu--a-posteree optimization that leverages the diffusion model's trajectory regularizer, resulting in a simple and tractable method. However, this approach shares the same limitations as score distillation regarding diversity and mode collapse. Additionally, applying this formulation directly with latent diffusion models produces blurry results (see Appendix D.8.3
4 REPULSIVE VARIATIONAL DIFFUSION SAMPLING
In Section we address (cl) by introducing a repulsion mechanism, promoting diversity through
a multimodal variational approximation using interactive particles. Then, in Section 4.2 we tackle
RRFD-Diff was proposed in the pixel-domain. However, for the sake of clarity, here express it with respect the latent diffusion models.
4

---

Published as a conference paper at ICLR 2025
(c2) by proposing augmented variational formulation. Finally, in Section 4.3 combine both techniques to derive Repulsive Latent Score Distillation (RLSD), our proposed solver for inverse problems using latent diffusion models.
4.1 TACKLING MODE COLLAPSE: ENIIANCING DIVERSITY VIA REPULSION
We aim to solve inverse problems characterized by the forward model in by minimizing the reverse KL divergence B). However, explained in Section 3.2, minimizing (3) with a Gaussian variational distribution leads to a unimodal approximation of a multimodal posterior, which is problematic for highly ill-posed problems like inpainting. To circumvent this, we propose a particle approximation for defining a multimodal variational distribution. More precisely, we incorporate repulsion force within the particles to encourage the exploration of multiple modes.
Particle interpretation of SDS. To facilitate the presentation, throughout this section we consider the unconditional case of B i.e. without measurements:
((z) = argmin (T(((Zz)) (l)()))))ll()))
g(2o)
Following Song et al. (2021a) rewrite 4 in terms of the diffused trajectory
(z)) argmin [0 ()()))(())))
(5)
g(2n)
where (t) is a weighting function and q(z,) = f (zz (z) q dzo depends on the diffused trajec-
tory q(z zo) ~ [ax 1) and the variational approximation q(Zo). The optimization (5)
corresponds to score distillation, which can be formulated as a Wasserstein gradient flow (details of WGF can be found in Appendix E.T. At the particle level, the WGF is described by the following ODE
dz E, (tt))
log P
( Zi.T
(3) log 4-
22,,7 { dr,
(1)
(6)
where (1)
is the marginal distribution of a generic particle i at time-step т, and é is the noise level of the diffusion model. In a nutshell, the WGF of zn in (6) is computed expectation
(2,77 is the target distribution, gr
(t.7
(4)
its diffused trajectory (noise levels f), involving the gradients of 2g. This formulation shields light on how the particles are propagated when optimizing (5) More precisely, becomes evident that for an initial Gaussian variational approximation marginal for all T is also Gaussian. The dynamic in (6) yields a deterministic trajectory where the mode of the Gaussian variational approximation will match one of the modes of p(zu). Consequently, assuming the same initial position, all particles will converge to the same mode. Although this can be mitigated ad hoc by changing the particles' initial positions, we seek a more principled method.
0, the
In this context, we can enhance diversity by considering 1) multimodal (but most likely intractable) variational distribution or 2) interactive particle systems; this work, we focus on the second one due to its tractabilityy. When considering an interactive set of particles, the key design factor is the coupling term, which prevents the ensemble from collapsing to the same mode. In particular, we propose using a repulsion term.
Repulsive variational distribution. Inspired by D Angelo and Fortuin (2021b), we consider an ensemble of interacting particles coupled via repulsive force that pushes particles away from collapsing to the same solution. In nutshell, we introduce a repulsive force that yields the following
modification of the gradient flow in (6)
d E, (1)
logp
2
(2)
V
(3)) log OF (2(.7
(a)
R(z
(N)
dr, (7)
2
where N is the number of particles and R( its gradient is the repulsive f Notice that the marginal distribution in (7) each time-step is given by (where Z is a normalizing constant)
(N) is the coupling between particles such that
2
N
gr Z,,7
(N)
R (
4- i=1
II
(8)
We consider here repulsive force because seek diversity. However, attractive force can be considered within this same framework.
5

---

Published as a conference paper at ICLR 2025
Throughout this work, we consider pairwise kernel function & such that the repulsive force adopts the form V (4) R
(t) see the numerical experiments for the particular instances of the kernel k. The repulsive force allows us to consider simple and flexible variational distributions that discover multiple modes. For simple illustration in the Gaussian case, see Appendix D.10.1 Finally, notice that when = 0, we recover the i.i.d.
St.- (Z, 26.
(1)
(N)
- V Σソ= log k 21,77 Z
St.r
inon-repuliive case.
4.2 TACKLING LATENT INVERSION: AUGMENTATION OF THE VARIATIONAL DISTRIBUTION
As discussed in Section 3.2 solving directly in the latent spaces yields blurry solutions. To tackle this, we propose to solve an augmented version this problem, allowing us to decouple the data and the latent space of the diffusion model. Formally, we introduce an auxiliary variable Xg defined in the data (pixel) space, which yields an augmented variational distribution (Zo, y) and an augmented posterior as
p xoly) x exp
1 - -ي2 20 lly f(xx)
2pd x D(zo))l))
1
where P controls the correlation between the variables Xy and Zo. and exp (-Ag(zo)) represents the prior distribution parameterized by the latent diffusion model. Notice that the definition in (9) implies that Xo|20 22 ~ N(D(za), I")), i.e., the conditional distribution of the data point (Xo) is centered at the value of the decoder applied to the latent point (2n). However, this is not delta but has some variance given by p². It can be shown that p(xoly y, X. converges in total variational to the true posterior p(xoly, A) when p - 0 (Van Dyk and Meng, 2001 Vono al. 2020) (details can be found in Appendix A.I. We can reformulate the optimization problem in (3) as
((2o, xoly) = argmin (L(g(Zo. xoly) (pll)) Xoly)). x(zo.xoly)
(10)
When considering a diffusion model as data prior, our problem boils down to minimizing the variational lower bound, formalized in Proposition
Proposition 1 Assuming we have access to a diffusion model Vz, log p (ze) for the prior on Zo, then the KL minimization w.r.f q in (10) is equivalent minimizing the variational bound, which can be done by solving the following optimization problem
min Eg(zoly) lv) (g( })]] + By(xo.xoly) (00.
2 lly -(xx)]]1
(11)
+ 1// ((t)
[x00
D(
[IV log (ze y) Ve, log (z))l
dt.
2p
The proof is in Appendix A.2
When p - 0, then X = D(zo) and the augmented KL optimization
boils down the objective 3 To solve the problem in Proposition [ we need to specify the
variational distribution of Xo: 2 zo]y); we now incorporate our result from Section 4.1
4.3 REPULSIVE LATENT SCORE DISTILLATION FOR SOLVING INVERSE PROBLEMS
We now derive RLSD, which integrates the techniques from Sections 4.1 and 4.2 Specifically, apply the repulsive variational distribution introduced in Section 4.1 to instantiate the augmented variational formulation detailed in Proposition] [ By employing the particle approximation defined in (8) we define a multimodal distribution that enables better exploration of the posterior's 's search space, facilitating the discovery of multiple modes and addressing (c1). Notably, this variational approximation yields a tractable gradient, formalized in Proposition
When considering the variational distribution defined in 88 the KL. minimization w.r. ((x0, zuly) defined in Proposition can be approximated with ensemble of N particles and admits the following gradient
Proposition 2
u(+)
20 ly
1 1xx - D22 1(x() 2 2p2
+ 20.
(
(1)
(N)
(12)

---

Published as a conference paper at ICLR 2025
for 1,..,, N and where ($) =(x0
1x0 20 =. ). The regularization term is given by
N
2 (N)) = Eest de t) - - -V xi,
(6) Ger
(a) uur
rog (2,7."
you log
3
(13)
where > == Ta, da((t) and 7 2 O.
ae de
The proof can found in Appendix
terms: measurement matching term, The gradient defined in Proposition comprises three term measuring the discrepancy between the variable in the pixel space and the decoded latent, and a regularization temm that combines a score-matching regularizer with pivvesity-rromoiing component. This repulsion term acts as second regularizer, enhancing diversity during the sampling process. Our approach is not limited to any specific latent diffusion model.
Practical algorithm for sampling using RLSD. The underlying optimization of the gradient update in Proposition 2 a particular case of Propositon[] is highly non-convex (diffusion denoiser and the repulsion term) and challenging to solve. To alleviate this, we adopt a half quadratic splitting technique (Geman and Yang, 1995). The algorithm is shown in Algorithm T define sg(.] as stopped-grate gradient operator to emphasize that the term inside it is not differentiated during
the optimization step. We denote أ - For the weighting function X (w(t) is embedded) and timesteps, we follow the strategy introduced in Mardani et al. (2024), where X = A(ax/ox)) and the timesteps follow decreasing order (from fmax minn); we fix "max T and Emiin 0. Regarding computational burden, our final algorithm only performs one backpropagation through the decoder in the z-step and N backpropagations to compute the repulsive kernel (lines and 9 in Alg. [ are with respect to all particles). The complexity of the repulsive force depends on the number of particles as well as the domain of the kernel. Notice that the amount of particles a hyperparameter, allowing us to control the trade-off between diversity and speed. Importantly, in contrast to previous works, we do not backpropagate through the score network.
Algorithm 1 RLSD for solving inverse problems
Require: (.), L, Eo f), D(), 7, 4 I: Initialize [x],,)==,,, [2]o]
2: for f = 1 to L do
3: t == T {,T and I(0,
A \(oe/ae)
5:
z = aczi, + oiE
6: C. Exxl D(z +he (sg (o(()) - E- of log 2j=1 (z. 21 )1
zi.0
7: Z OptinizerSteppg br.)
C. En lly - fxx))] + D(zl.o)
9:
X OptimizerSlep,p (Lr, lr,)
10: end for 1N
11: return (x
EXPERIMENTS
In this section, we compare RLSD against state-of-the-art (SoTA) methods for solving inverse prob- lems using latent diffusion models. We consider 100 samples from the validation set of FFHQ (Karras) al.) 2019) used in Chung et al. (2022a). We compute PSNR [dB], LPIPS, and FID as metrics. Throughout the experiments, we seek to show the following:
Our method generates more diverse solutions, in particular for tasks like inpainting and phase retrieval,
When diversity is not relevant, our augmented formulation generates high-quality samples and outperforms baseline methods.
7

---

Published as a conference paper at ICLR 2025
Sampling setup. Unless we state otherwise, we consider 1000 steps (the full denoising trajectory) for all the cases. We denote by NonAug-RLan RLSD our repulsive method without augmentation (see Alg. 2 in Appendix D.I), and by KonnRepul--RLSD our method with augmentation but without repulsion. We consider Adam (Kingma 2014) in the optimization steps (lines and 9 in Alg. D and set the momentum pair (0.9, 0.99). We randomly initialize variables and z and generate batch of N = 4 particles per noisy measurement. Regarding the pre-trained model, we consider Stable diffusion v2.1, although other latent diffusion models can be used. As diversity metric, we evaluate the pairwise diversity as the 1 cosine similarity between the N particles. Lastly, for the kernel function,
we consider a RBF: k(zz, 2,) = exp(- (2.) -DDIO )ll") where h1 = m /log N, my is the h. median particle distance (Liu and Wang 2016) and IDINO is pre-trained neural network (Caron et al. 202I Details about implementation are in Appendix D.I. and ablation analysis in D.8
Baselines methods. As we focus on methods that leverage large pre-trained models such as Stable diffusion, we compare with the recent PSLD (Rout et al. (024)) and Latent RED-Diff. For completeness, we also include a comparison with SoTA methods in the pixel-domain, namely DPS (Chung et al.) 2022a) and RED-diff (Mardani al. 2024). Details about the implementation of each method are in Appendix D.I]] Given that pixel-based diffusion solvers generate images at 256 x 256, follow the strategy from Rout al. (2024) and downsample the results generated by our sampler, which have a 512 x 512 resolution, to do a fair comparison.
5.1 INPAINTING
Inpainting, with its inherent ambiguity, provides suitable benehmark to showcase two key aspects of RLSD: 1) high-quality reconstruction and 2) enhanced diversity achieved through the repulsion term. Additional linear inverse problems such as super resolution and deblurring are detailed in
Appendix[] D) Specifically, we consider box hiding half of the faces (see Fig. 2 and Appendix D.7p. For the hyperparameters, we set À 0.14, م = 0.075, 4- = 0.4 and 4,- 0.8. Results Table 2 show that RLSD) outperforms their baselines in image quality (PSNR and FID); in particular, it outperforms PSLDD, the other sampler a resolution of 512 x 512. Moreover, it demonstrates that our method can trade-off diversity for quality by modifying the weight while RLSD ( = 50) achieves higher diversity than MonRepuls-RLSD the later achieves better performance. In addition, in Table 14 we demonstrate that RLSD achieves the best performance when selecting the sample with best performance.
This highlights RLSD's ability combine superior reconstruction with higher diversity, suggesting mixed strategy where some particles interact and others do not. Indeed, this strategy EHybrid-RLSDD where three particles interact and one particle is propagated independently of the ensemble, combines the best of both worlds, achieving the best balance between quality and diversity.
piversity-qualits trade off. Fig.2 showcasses the diversity-quality trade-off. While PSLD generates four diverse samples of lower quality, Non-Repuls RLSD tends to fill the four images with very similar solutions. On the other hand, when considering RLSD (7- = 50), the results are more diverse while maintaining high quality: in images 1 and 3, the woman has the left eye hidden, while none of the generated images with NonRepuls-RIe, RL.SD show this. We defer to Appendix D.10 a more trade off.
exhaustive analysis of
5.2 NON-LINEAR INVERSE PROBLEMS
We consider in this case nonlinear inverse problems. Given that PSLD only works on linear inverse problems, we compare against latent DPS and latent RED-Diff.
Phase retrieval. We first consider phase retrieval, which deals with reconstructing the phase from only magnitude observations in the Fourier domain. Phase retrieval known as highly ill ill-posed problem, given that it is invariant to 180 rotation, which yields two equally probable modes. Thus, its posterior has multiple modes, which are discrete and isolated. We follow the strategy from DPS (Chung et al.. 2022ah, where an oversampling of rate 2 is used. We consider 6 particles for the particle variational approximation, and 6 independent particles for the non repulsivee case. Furthermore, we set 7 = 30, and we only consider repulsion between t E [0.4T, T). Results are shown in Table 3 First, observe that for this experiment, the augmented variational approximation (NonRepuls-RLL) RLSD and RLSD) entails more unstable algorithm, and thus, not converging to good modes.
8

---

Published as a conference paper at ICLR 2025
PSLD
NonRepub-RUL
RLSD(Y
Figure 2: Ipainting half a face (top bottom): Ground truth, measurement, PSLD, Non and RLSD (7 = 50). Each method generates four samples from different initializations, with RLSD incorporating repulsion. Both RLSD and NonRepuls-RLL -LSSD outperform PSLD. RLSD produces diverse results (e.g.. images and 3 hide the left eye). whereas NonRepuls-RLLn RLSD samples appear similar.
Table 2: Box inpainting (half face) with = 0.001 FFHQ 512. For evaluation, we downsample the estimated images of RLSD and PSLD 256. The best method for each metric bolded.
Sampler
PSNR [dB]
LPIPS 4
FID 4
Diversity
PSLD
22.72
0.082
57.7
0.03
Latent RED-diff
23.5
0.15
92.59
0.009
ReSample
19.44
0.2
146.68
RED-Diff (Pixel)
23.1
0.067
29.79
0.004
DPS (Pixel)
23.4
0.14
78.88
0.04
NonAuu-RLLSD (y = 50)
23.34
0.164
98.65
0.035
NonRepuls-RRl RLSD
24.98
0.079
29.18
0.004
RLSD (y = 50)
24.69
0.111
31.41
0.015
Hybrid-RISD
24.72
0.096
30.48
0.018
Second, the results show that our proposed method NonSug-RI.DD is more stable, and that the particle approximation effectively captures more modes. particular, Fig. 3 we show an example where the latent RED-diff generates 5 images that look similar, while NonAug-RLSD generates samples that corresponds different modes. This showcases that our methods indeed promote diversity, even for a nonlinear inverse problems and at resolution of 512 x 512.
High dynamic range (HDR). We try HDR, which performs the clipping function f(x) clip(2x, 1) the normalized RGB pixels. HDR is known to be simpler than phase retrieval, and where diversity not fundamental. Again, we consider Latent DPS as PSI.D does not work for nonlinear inverse problems. The results are in Table where RLSD outperforms all other baselines (when choosing the best sample among the group). This showcases that repulsion promotes better exploration, and reaching better modes.

---

FID Published as a conference paper at ICLR 2025
Table 3: Phase-retrieval with de 0.001 on FFHQ 512. best method for each metric bolded.
Sampler
PSNR [dB]
LPIPS 4
FID
Latent-DPS
14.98
0.618
291.68
Latent-RED--diff (7 0)
19.65
0.458
173.18
NonAug-RLSD (7 = 30)
24.21
0.359
130.09
NonRepuls-RL.DD
18.33
0.495
223.14
RLSD (7 30)
20.43
0.449
207
(a) NonAug-Ro RaSaug-RRSDDAAA 30).
(b) Latent RED-Diff.
Figure 3: Results for Phase Retrieval. Adding repulsion between particles allows sample from different
modes (top row).
Table 4: HDR with a, 0.001 on FFHQ 512. The best method for each metric is bolded.
Sampler
PSNR [dB]
LPIPS |
Latent-DPS
15.77
0.449
181.19
Satent-RED-diff
25.68
0.200
93.71
NonAug- NonAug-RLSD (7 30)
24.84
0.210
94.15
Non
27.72
0.087
37.57
RLSD (7 50)
28.13
0.089
37.32
CONCLUSIONS AND LIMITATIONS
In this paper, we introduce Repulsive Latent Score Distillation (RLSD), a plug-and-play variational sampler that leverages pre-trained latent diffusion models solve inverse problems, balancing quality, diversity, and computational efficiency. Inspired by the Wasserstein gradient flow of score distillation, RLSD mitigates mode collapse and latent diffusion inversion.
To tackle (cl) mode collapse, we introduce particle-based variational distribution with a repulsion mechanism based on kernel similarity. To handle (c2) latent inversion (from adversarial training), we propose distribution augmentation to decouple latent and pixel spaces. The algorithm applies two regularizations: denoising to enforce the prior and repulsion to promote diversity.
Numerical experiments demonstrate that RLSD merges the benefits of variational samplers (memory and compute efficiency) with posterior samplers (diversity), allowing control over speed and diversity through simple regularization weights. The repulsion force significantly boosts diversity in ill-posed problems like inpainting and phase retrieval.
Our method has some limitations. Including the repulsion term increases computational demands, complicating real-time use. The chosen repulsion kernel may not be optimal under high noise levels, suggesting need for adaptive kernel learning. Moreover, the method introduces additional hyperparameters, requiring better coupling for noise levels and deriving repulsion weights based on the forward operator.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
Research was sponsored by National Science Foundation (CCF 2340481) and the Army Research Office (Grant Number W91 INF-I7-S-0002), NZ was partially supported by Ken Kennedy Institute 2024-25 Ken Kennedy-Hie HAPe Cray Graduate Fellowship. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Army or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstandding any copyright notation herein.
REPRODUCIBILITY STATEMENT
In this work, we have taken several steps to ensure the reproducibility of our results. We provide a comprehensive description of the methodology in Section of of the main text and the full algorithm in pseudocode are in Algorithms D we give details of all the hyperparameters for each experiment. Additionally, we have included all necessary proofs for theoretical claims in Appendix A For the experiments, we gave details of all the datasets that we used and how we compute the metrics. We include link to our repository in the abstract. the README.nd file in the repository is explained the steps to run the code. Lastly, in Appendix 10. we included four gif files, which might require a pdf reader that can reproduce gif's
and In addition to this, Section 3 as well in Appendix D.
ETHICS STATEMENT
Our method has the potential to cause unintended negative consequences if not handled responsibly. Key ethical and societal risks include the amplification of biases, difficulties in verifying the au- thenticity of generated content, which could contribute to misinformation, and the economic impact on creative professionals. Additionally, there are concerns over the misuse of this technology for harmful purposes. privacy issues related to the datasets used, cultural insensitivity, and potential intellectual property conflicts surrounding Al-generated creations. Addressing these risks necessitates the development of strong ethical standards, regulatory frameworks, and safeguards to ensure fairness, privacy protection, and respect for cultural and intellectual property rights. Therefore, it is essential that RLSD and other generative models are applied with clear understanding of their limitations, and that outcomes are validated carefully to reduce these risks.
REFERENCES
Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2D diffusion into 3D, alleviate janus problem and beyond. arXiv preprint arXiv:2304.0499 2023.
Jimmy Ba, Murat A Erdogdu, Marzyeh Ghassemi, Shengyang Sun, Taiji Suzuki, Denny Wu, and Tianzong Zhang. Understanding the variance collapse of SVGD high dimensions. In Intl. Conf. Learn. Repr. (ICLR), 2021.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association. 112(518):859 877, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 9650-9660, 2021.
Yifan Chen, Daniel Zhengyu Huang, Jiaoyang Huang, Sebastian Reich, and Andrew M Stuart. Gradient flows for sampling: mean-field models, Gaussian approximations and affine invariance. arXiv preprint arXiv:2302.1 1024, 2023.
Hyungjin Chung. Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusson posterior sampling for general noisy inverse problems. In Intl. Conf. Learn. Repr. (ICLR), 2022a.
11

---

Published as a conference paper at ICLR 2025
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Inf. Process. Syst. (NIPS), 35: 25683-25690 2022b.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-iifus--faats Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), pages 12413-12422, 2022c
Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. Intl. Conf. on Machine Learning (ICML), 2024.
Gabriele Corso, Yilun Xu, Valentin De Bortoli, Regina Barzilay, and Tommi Jaakkola. Particle guidance: non-lID diverse sampling with diffusion models. Intl. Conf. Learn. Repr. (ICLR), 2024.
Francesco D' Angelo and Vincent Fortuin. Annealed Stein variational gradient descent. arXiv preprint wwXiv:2101.09815. 2021a.
Francesco D' Angelo and Vincent Fortuin. Repulsive deep ensembles are Bayesian. Advances in Neural Inf. Process. Syst. (NIPS), 34:3451-34444 2021b.
Giannis Daras, Joseph Dean, Ajil Jalal, and Alex Dimakis. Intermediate layer optimization for inverse problems using deep generative models. In Intl. Conf. on Machine Learning (ICML), pages 2421-2432. PMLR, 2021.
Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Jong Chul Ye, Peyman Milanfar, Alexandros G. Dimakis, and Mauricio Delbracio. A survey on diffusion models for inverse problems. arXiv preprint arXiv:2410.00083 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Inf. Process. Syst. (NIPS), 34:8780-8794, 2021.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. Intl. Conf. Learn. Repr. (ICLR), 2017.
Zchao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. Intl. Conf. Learn. Repr. (ICLR), 2024.
Andrew Duncan, Nikolas Nüsken, and Lukasz Szpruch. On the geometry of Stein variational gradient descent. J. Mach. Learn. Res., 24(56):1-39, 2023.
Elhadji C Faye, Mame Diarra Fall, and Nicolas Dobigeon. Regularization by denoising: Bayesian model and Langevin-within-sphe Gibbs sampling. arXiv preprint arXiv:2402.. 12292, 2024.
Berthy T Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L Bouman, and
William T Freeman. Score-based diffusion models as principled priors for inverse imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10520-10531, 2023.
Donald Geman and Chengda Yang. Nonlinear image recovery with half-quadratic regularization. IEEE transactions on Image Processing, 4(7):932-94 9466, 1995.
Yuan-Chen Guo, Ying-Tiar Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang. Yan-Pei Cao, and Hong-Hai Zhang. threestudio: A unified framework for 3d content generation. threestudio) 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Inf. Process. Syst. (NIPS), 33:6840-6851, 2020.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Inf. Process. Syst. (NIPS), 15:8633-8646 2022.
12

---

Published as a conference paper at ICLR 2025
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In Intl. Conf. Learn. Repr. (ICLR), 2021.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Int. Conf. Comput. Vis. Puttern Recogn. (CVPR), pages 867-876, 2022.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the Fokker-Plande equation. SIAM J. Math. Anal., 29(1):1-17, 1998.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. Advances in Neural Inf. Process. Syst. (NIPS), 34:13222-13254, 2021.
Ulugbek S Kamilov, Charles Bouman, Gregery Buzzard, and Brendt Wohlberg, Plug-and-play methods for integrating physical and learned models in computational imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 40:5))855-3 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), pages 4401-4410, 2019.
Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. In Intl. Conf. Learn. Repr. (ICLR), 2024.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. Advances in Neural Inf. Process. Syst. (NIPS), 34:21757-21769, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Inf. Process. Syst. (NIPS), 35:23593 35:23593-23666, 2022.
Jeongsol Kim, Geon Yeong Park, and Jong Chul Ye. Dreamsampler: Unifying diffusion sampling and score distillation for image manipulation. arXiv preprint a:X1v:2403.11415, 2024.
Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual editing. In Advances in Neural Inf. Process. Syst. (NIPS), volume 36, pages 73232-73257, 2023.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.698. 2014.
Florian Knoll, Kristian Bredies, Thomas Pock, and Rudolf Stollberger. Second order total generalized variation (tgv) for mri. Magnetic resonance in medicine, 65(2):480-491, 2011.
Erich Kobler, Teresa Klatzer, Kerstin Hammernik, and Thomas Pock. Variational networks: con-
necting variational methods and deep learning. In Pattern Recognition: 39th German Conference, GCPR 2017, Basel, Switzerland, September 12-15, 2017, Proceedings 39, pages 281-293. Springer, 2017.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In Intl. Conf. Learn. Repr. (ICLR), 2020.
Mare Lambert, Sinho Chewi, Francis Bach, Silvère Bonnabel, and Philippe Rigollet. Variational inference via Wasserstein gradient flows. Advances in Neural Inf. Process. Syst. (NIPS), 35: 14434-14447, 2022.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when Langevin meets Tweedie. SIAM J. Imag. Sciences, 75(2):701-737, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conf. Comp. Vision (ECCV), pages 740-755. Springer, 2014.
13

---

Published as a conference paper at ICLR 2025
Qiang Liu. Stein variational gradient descent as gradient flow. Advances in Neural Inf. Process. Syst. (NIPS), 30, 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. Advances Neural Inf. Process. Syst. (NIPS), 29, 2016.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Inf. Process. Syst. (NIPS), 36, 2024.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective solving inverse problems with diffusion models. Intl. Conf. Learn. Repr. (ICLR), 2024.
Roger B Nelsen. An introduction to copulas. Springer, 2006.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3D using 21)
diffuuion. In Intl. Conf. Learn. Repr. (ICLR), 2022.
Javier Portilla, Vasily Strela, Martin J Wainwright, and Eero P Simoncelli. Image denoising using scale mixtures of gaussians in the wavelet domain. IEEE Transactions on Image processing, 12 100,1338-13511 2003.
Yaniv Romano, Michael Flad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). SIAM J. Imag. Sciences, 10(4):1804-)) 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. 2022 iece. In Proceedings the IEEE/CVF Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), pages 10674-10685, 2021.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Inf. Process. Syst. (NIPS), 36, 2024.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. Int. J. Comp. Vision, 1 115:211-299 2015.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. LAION-SB: An open large-scale dataset for training next generation image-text models. Advances in Neural Inf. Process. Syst. (NIPS), 35:25278 22229424222, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Intl. Conf. Machine Learning (ICML), pages 2256-2265. PMLR, 2015.
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. Intl. Conf. Learn. Repr. (ICLR), 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Intl. Conf. Learn. Repr. (ICLR), 2020.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoonveess-gguided diffusion models for inverse problems. In Intl. Conf. Learn. Repr. (ICLR), 2022.
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming Yu Liu, Jan Kautz, Yongxin Loss-guided
Chen, and Arash Vahdat. Loss- guided diffusion models for alug-and-play controllable generation. In Intl. Conf. Machine Learning (ICML.), pages 32483-32499 PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances Neural Inf. Process. Syst. (NIPS), 34:1415-1428, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Intl. Conf. Learn. Repr. (ICLR), 2021b.
Francesco Tonolini, Jack Radford, Alex Turpin, Daniele Faccio, and Roderick Murray-Smilte Vari- ational inference for computational imaging inverse problems. Journal of Machine Learning Research, 21(179):1-46, 2020.
Dustin Tran, David Blei, and Edo M Airoldi. Copula variational inference. Advances in Neural Inf. Process. Syst. (NIPS), 28, 2015.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Inf. Process. Syst. (NIPS), 14:11187-11302, 2021.
David Van Dyk and Xiao-Li Meng. The of data augmentation. Comp. Graph. Stat., 10(1): 1-50, 2001.
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. IEEE Global Conf. Signal and Info. Process. (GlobalSIP), pages 945-948. IEEE, 2013.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 2367):1661-116744 2011.
Maxime Vono, Nicolas Dobigeon, and Pierre Chainais. Asymptotically exact data augmentation: Models, properties, and algorithms. J. Comp. Graph. Stat., 30(2):335 348, 2020.
Peihao Wang, Dejia Xu, Zhiwen Fan, Dilin Wang, Sreyas Mohan, Forrest landola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, et al. Taming mode collapse in score distillation for text-to-3D generation. arXiv preprint arXiv:2401.00900 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. Advances in Neural Inf. Process. Syst. (NIPS), 36, 2024.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming -ssan Yang. Gan inversion: A survey, IEEE Trans. Pattern Anal. Mach. Intell., 45(3):3121-3138 2022.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Lue Van Gool, and Radu Timofte. Plug-and-play image restoration with deep denoiser prior. IEEE Trans. Pattern Anal. Mach. Intell., 44(10): 6360- 2021.
Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo. HIFA: advanced diffusion guidance. In Intl. Conf. Learn. Repr. (ICLR), 2024.
High-fidelity text-to-3d generation with
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Lue Van Gool. Denoising diffusion models for plug-and and-play image restoration. In Proceedings of the IEEE/CVF Int. Conf. Comput. Vis. Pattern Recogn. (CVPR), pages 1219-1229, 2023.
Nicolas Zilberstein, Chris Dick, Rahman Doost-Mohammady, Ashutosh Sabharwal, and Santiago Segarra. Annealed Langevin dynamics for massive MIMO detection. IEEE Trans. Wireless Commun., 2022.
Nicolas Zilberstein, Ashutosh Sabharwal, and Santiago Segarra. Solving linear inverse problems using higher-order annealed Langevin diffusion. IEEE Trans. Signal Process., 2024.
15

---

Published as a conference paper at ICLR 2025
A TECHNICAL PROOFS
A.1 DATA ACGMENTATION
In Section 4.2 consider an augmented variational distribution go(Xo: zoly) such that
/ go (Xo, zoly yzzo.
go (xoooy))
(14)
Therefore, seek joint distribution such that Property holds.
Property For all X E IR-, it holds limp- (Xo) = π (xn))
Notice that our approach using data augmentation resembles some recent methods introduced in the bibliography. In Zhu et al. (2023), the authors propose to decouple data and diffusion model. More recently, a Bayesian version the RED method was proposed in Faye et al. (2024). This method shares similarities with our method, in the sense that both are augmented versions that resemble RED. However, our method has three main differences: 1) leverages latent diffusion models, which allows us to solve large-scale inverse problems (512 x 512 and beyond), 2) it uses the diffused trajectory to regularize the solution, and promotes diversity via the coupling term.
A.2 PROOF OF PROPOSITION
We expand the KI. objective as follows
(T,(g(Zo, X xoly))lp(zo: xoly)) = /
g(z0, xa dzn
q(zo, xoly) log
(15)
p(zu, Xo ly)
(xxn Zo y ) pp())
q(2o, x xoly) log
ply xo) Xo /2 ( dzodxo
g(zo, xo xoly) ) log (xxo zo: Jdzodxo
/ q(zo. xoly) log ply x xo)dz)dx
(15)
/ g(zo: xoly) log p (z) ddnd
(Ai)
q(zo q(zooxoly> xoly) q(zu ly) dzndxo + log p(y).
p(zo)
(tv)
Based on the augmented posterior, we have for (di) and (iii) that
(ii) / q(z0 xoly) logply xx)d2gdxx Ey(xx.xol))
1 lly f(xo)11
(16)
200
and
(ita) = ff /(20, xoly) log zo d2d2ddxx [R((xo.20l))
2p2 I|x D(20)112
(17)
Regarding the first term, we can write as
10=/ ( y [g(xo Zu. y) log (xx0 Zo. y) dzndxxo
E Ex((ool))
[g
Zu. y)].
(18)
Finally, the last term can be obtained by following theorem 2 in [Song al. (2021a). assuming that the score is learned exactly, namely (ze; () = -o, Vz, log (Zr). and under some mild assumptions on growth log q (z, y) and p (2x) at infinity, we have
[ B(1)
(L.(g(Zol)) [l((z)))
2 ( (t) |/ log (Ze yy Vz. log (2))l dt (19)
16

---

Published as a conference paper at ICLR 2025
over the denoising diffusion trajectory {z,} for positive values (3(t)). This essentially implies that a weighted mctre-natching over the continuous denoising diffusion trajectory is equal to the KL divergence.
A.3 PROOF OF PROPOSITION
The first two terms are straightforward. We focus here on the regularization term. The regularization term in Proposition [ corresponds to the score matching loss defined in Song al. (2021a) For general weighting schemes wil), we have the following Lemma from Song et al. 202 la
Lemma 1 The time-derivative of the KL divergence at timestep obeys
KL (g (z: y)) 1pp (2+)) ((()
[V n, log q (Ze yy) - V. logy (2.)1
y)
dt
2
Then, under the condition w0)) 0, the integral in Proposition can be written as (see Song et al. (222
1."
B(t) w0 (() 20,
log q (ze yy) - Vz, log (ze) [12] dt
l/ w(t) dKL (1 (2: yy) 11p (2+)) dt
dt
(()KL (z 1 v) 11 (2e) + /" ('())L (2xx ly) (2:)) dt
J0 w' ((tea)y Toog 9 ((z y)
dl,
p (2+)
where wt()) dw())) de The equality holds because w(t)LL (g (z: y) |lp (2.))l is zero at t = 0 and t t= T. This is because w() = 0 by assumption at = 0, and Fy becomes pure Gaussian noise at the
end of the diffusion process which makes p (zr) 9 (zr y) and thus KL (g (zz y) |l (zz)) 0.
Now, we consider our proposed variational distribution defined in (8) with N particles and the pairwise kernel. For each particle i, we apply the forward diffusion zi" aiz!" + o,e, which yields the distribution (2,)) ly) N (arz
(t)
(i) .ff).. and thus Vz, log q: (z. y) =
(z) aszi") /0t (0) By applying the re-paramettrizzation trick, obtain
(1) V (0 reg(zi
,,.,,, 2z (1)))
(20)
So w d((t),
(
N Σ
dz
Ma ( log
(1)
+ v
log (1) y) - Vz. log log
dt
x
day
==1
so(2())()) T
fr w/ (((t)Eens((.))
N
-V 1
log
2(
' +
Ix dt
j=1
J0 T
[ Yo, log
N Σ
2(0)) - co(a(")))) ) di
0' w' Eens(111) σt
(6)
2
j=1
We can rearrange terms to arrive at the following compact form
reg(z(")
z (N))-
(21)
2 x2() 2(0)) - co(a.") 00)
d, -
yor log
(
==1
17

---

Published as a conference paper at ICLR 2025
for A := (t)uua When considering the measurement matching term and the error between the ambient and the augmented variable, we obtain X == Tw' (tu(tto//,,
B DIFFUSION FOR INVERSE PROBLEMS
Diffusion models are powerful generative models. Therefore, they have been used as deep generative priors to solve inverse problems. Given pre-trained diffusion model, this involves running the backward process using a guidance (likelihood) term that incorporates the measurement information. Formally, can sample from the posterior p(xoly) by running 22
1 3()), 3(1) XVx log lxg) Vx, logp(y/x()) + VB(I)aW.-
dx.
(22)
Early studies used Langevin dynamics for linear problems Kadkhodaic and Simoncelli 2021 Kawar al.
202T Laumont al. 2022 Zilberstein et 2024 2022). while others used DDPM Kawar et al.
2022, Chung et al. 2022c b Ho et al. (022). However, approximating the guidance term remains challenge. Previous works addressed this with Gaussian approximation of p(x)) X around the MMSE estimator via Tweedie's ' formula, increasing computational burden (Chung et al., 2022a Song et al. 2022, Kadkhodaie and Simoncelli] 2021, [Snn et al. 2023). These methods crudely approximate the posterior score, especially for non-small noise levels. One of the most effective methods is DPS (Chung et al. 2022a), which assumes:
p (y(x;) = (y|&o := E xx)) = N (ylf (E (x.(xx)).
Essentially, DPS approximates the likelihood with unimodal Gaussian distribution center around the MMSE estimator IE [xn x=). Under this approximation, the term p (y) xx) boils down to the gradient of multivariate Gaussian. Although it achieves impressive results, the unimodal approximation is far from optimal. Furthermore, its adaptation to use a latent diffusion model is not straightforward, as
explained in (Rout et al. 2024). Recent works (Rout al. 2024 Song al. 2024 Kim et al. 2024] Chung et al. 2024) extended this by sampling from the latent space of diffusion models but still face limitations due to the intractable model likelihood. In particular, PSLD incorporates an additional
term to guide the reconstruction towards fixed point of the autoencoder process. This yields the following guidance score, where gluing term is added circumvent the discontinuity issues at the boundary.
Vz, 2, logp(y z:) = Vz, log (y/30 = D ( (ZE [z))= +
(23)
V. [z0o2z]] & (AT y + AI- A D (E
|z:)))]
where
p ( N N (D (z)/z,))),
Notice that while the gluing term is effective for linear inverse problems, it cannot handle non-linear cases.
C DISCUSSION ON VARIATIONAL. AUGMENTED DISTRIBUTION
In Section 4.2 we introduced an augmented variational distribution instead a variational formulation in the latent space directly. This decision stems from the observation that optimizing in the latent space often produces blurry reconstructions We hypothesize that this due to the nonlinearity of the decoder D..) and the adversarial training of the encoder-decodee pair. Specifically, the autoencoder tends to compressfine details, resulting reconstructions that capture high-level semantics but fail to reproduce the fine- me-ggraieed features.
To address this issue, we correct deviations from the image manifold during optimization by using an augmented variational formulation. This introduces coupling term between x and z to account for these deviations. First, we define the true augmented posterior distribution. We write X, >Dzn)) + a26, where a is the variance of the posterior of the decoder. Intuitively, when optimizing (10),
18

---

Published as a conference paper at ICLR 2025
optimizing the following joint target posterior p(x, z which has the following two conditionals associated
p(x|z,y) (N(uuz,)) E(z)) p(z|x) x p(x(z)p(z)
(24)
(25)
where E(z) A Dz) and μlz,y) = E(z) y and p(z)
the diffusion prior. Therefore, the variational inference the augmented formulation aims to approximate the first Gaussian with another Gaussian q(x y) ~N (H,,, T²I), with oR - 0, i.e., MAP estimate, and the second one with particle approximation, promoting diversity throughout the diffused trajectory.
This analysis also helps explain why NonAug-RLe RLSD outperforms its augmented variant in phase retrieval. Phase retrieval is nonlinear inverse problem, making alters (24) intractable: we cannot express the mean and covariance es explained above. Consequently, the augmentation renders a more difficult problem, which might explain why it is unstable.
D ADDITIONAL EXPERIMENTS
D.1 IMPLEMETATION DETAILS OF BASELINES
To facilitate reproducibility, we share anonymous link of our source codehttps: httptttss:///fft le. io/ FQNg3J5GPSY$ the paper is accepted, we will publish in public repository.
We the origianl code from Rout al. (2024). We use the version with Stable Diffusion, and we select hyperparameters as detailed in the paper.
PSL.D/Latent-DPS.
RED-Diff. We use the implementation from the original paper (Mardani et al. 2024 We follow the same weighting scheme, and we use X 0.25 and 1,- = 0.1. As pretrained models, for FFHQ we use the model from Chung et al. (2022a), while for ImageNet we use the one from Dhariwal and Nichol )
Latent RED-DifY. This method is the same as RED-Diff but using a latent diffusion model (as explain in Section
DPS. We use the implementation from the original paper (Chung et al. 2022a). We follow their configuration of hyperparameters. We use the same pretrained models as RED-Diff.
FPS-SMC and Song, 2024). We use the implementation from the original paper. We follow their configuration of hyperparameters. We use the same pretrained models as RED-Diff.
IIGMM.
We use the implementation from the original paper (Song et al. 2022). We follow their configuration of hyperparameters. We use the same pretrained models RED-Diff.
ReSample We use the implementation from the original paper (Song et al. 2022 We follow
their configuration of hyperparameters. We use LDM-VQ-4 trained on FFHQ. We tried using Stable Diffusion (the original implementation does not support it), but we got worst results.
NonAug-RLan RLSD. This method corresponds to our variant using the particle- based variational approximation 8 and without augmentation. For clarity, show it in Alg. 2
19

---

Published as a conference paper at ICLR 2025
Algorithm 2 Non-augmented RLSD for solving inverse problems
Require: y. y.f L, co(z,,)), (,.r, for / = to do
Initialize {2 1.
1 T T.T and E ~ NO(,I)
A (
_ = 0z2,, + OGE
C. E Ily ( (sg (o) - yV 100 log Ex=1 k 22)) 2
20 - OptimizerStep, (Czz (..)
end for return fx D(z
For augmented formulation, we Stable diffusion trained in the LAION (Schuhmann et al. 2022) dataset as its pre-trained model. For the kernel function, we consider a RBF k(zi, 2,)
RLSD.
[=D (DINO) where h4 = m flog N, my is the median particle distance LLiu and
exp(
he Wang 2016) and 9DINO is a pre-trained neural network (Caron et al.) 2021). Notice that NonRepuls-
RLSD corresponds to 7
Regarding the similarity metric, we consider the cosine similarity in the range of DINO defined as follow
9DINO (x,)² IDINO (x₃)
Sim(x xN) n 1)
(26)
1SDINO (x:)ll2 IIDDINO (x,)]l2²
Base on this metric, we define diversity as
Div
"im
(27)
Lastly, we consider an decreasing annealing schedule. It has been notice in previous works (Mardani al. 2024, Zhu al.] 2024) that decreasing timestep works better than sampling uniformly at random. As a consequence, we consider this scheme.
D.2 SUPER RESOLUTION
We consider super resolution from X 8 downsampled images. In this case we use À 0.2, أ = 0.05,
4. 0.4 and 4,, 0.6. The results are shown in Table For additional comparison, we compare also with solvers that generate samples at 256 256. For this comparison, similar to Section 5.1 we downsample the result RLSD to 256 and compare at that resolution. The results are in Table This example illustrates the key difference between RLSD and PSLD for solving inverse problems. To be more specific, PSLD generates images of faces that have the typical artifacts when sampling with Stable Diffusion. On the other hand, RLSD leverages Stable Diffusion multiple denoisers at different scale.
Table 5: SR x 8 with a, 0.001 FFHQ 512. The best method for each metric and experiment is bolded.
Sampler
PSNR [dB]
LPIPS 1
FID |
PSLD
24.82
0.314
81.31
Latent RED-diff
26.07
0.439
76.07
NonRepuls-RL.n RLSD
28.39
0.286
65.42
For completeness, we also compare with RED-Diff when solving SR x 4, as has the same resolution input than RLSD with SR x (RED-Diff handles images of size 256x256). The results is Table 7
20

---

Published as a conference paper at ICLR 2025
Table SR. with o. 0.001 FFHQQ-FFHQ/ 256. The best method for each metric and experiment is bolded.
Sampler
PSNR [dB] f
LPIPS
FID +
NonRepuls-RLSD
28.4
0.149
65.42
RED-Diff
25.69
0.264
104.59
DPS
23.83
0.175
90,45
IIDM
24.44
0.128
82.01
Table 7: SR x 8 with - 256. The best method for each metric and experiment is bolded.
Sampler
PSNR [dB]
LPIPS
FID +
NonRepuls-RLa -RSD
28.4
0.149
65.42
RED-Diff (SR x 4)
28.91
0.157
78.41
DPS (SR
27.14
0.128
71.8
FPS-SMC (SR x 4)
27.36
0.21
120.49
D.3 MOTION DEBLURRING
We consider Motion Blurring. In this case we use À 0.007, 0.01, = 0.4 and L, = 0.3, and 500. In particular, we follow Chung et al. (2022a), where we convolve the image with a 61 61 motion kernel that is randomly sampled with intensity 0.3². The results are shown in Table 8 For additional comparison, we compare also with solvers that generate samples at 256 x 256. For this comparison, similar to Section 5.I downsample the result of RLSD to 256 and compare that resolution. The results are in Table 9
Table 8: Motion Blurring with o, 0.001 FFHQ 512. The best method for each metric and experiment is bolded.
Sampler
PSNR [dB] f
LPIPS +
FID +
PSLD
25.17
0.389
135.22
Latent RED-diff
27.85
0.329
118.09
NonRepuls-RL.SD
30.4
0.23
56.79
Table 9: Motion Blurring with o, 0.001 FFHQ 256. experiment is bolded.
The best method for each metric and
Sampler
PSNR [dB] f
LPIPS +
FID |
NonRepuls-RLSD
30.47
0.095
56.79
RED-Diff
30.27
0.15
103.17
DPS
24.7
0.22
90.45
ReSample
26.82
0.115
72.74
D.4 INPAINTING WITH MASKED BOX (HALF-FACE)
We consider here additional baselines for box inpainting (half-face). See results in Table T0
21

---

Published as a conference paper at ICLR 2025
Table 10: Box inpainting (half face) with oy 0.001 FFHQ 256. The best method for each metric and experiment is bolded.
Sampler
PSNR [dB] f LPIPS
FID
NonAug-RLan RLSD (y 50)
23.34
0.164
98.65
NonRepuls-RLSD
24.98
0.079
29.18
RLSD (- 50)
24.69
0.111
31.41
Hybrid-RLSD
24.72
0.096
30.48
IIGDM
23.74
0.077
33.8
FPS-SMC
24.91
0.086
59.59
D.5 INPAINTING WITH RANDOM MASK
We consider random inpainting where drop 80% of the pixels. In this case we use À 0.009, I 0.08, 4-- 0.4 and 4, = 0.8. The numerical results are shown in Table I1 and visual examples are shown Fig.
Table Random inpainting with (80%) mask and with (a, = 0.001) FFHQ 512. bold the best method for each metric and experiment.
Sampler PSLD NonRepuls-RISD
PSNR [dB] f LPIPS +
FID |
28.53
0.212
65.14
30.56
0.145
41.11
Table 12: Random inpainting with (80%) mask and with (a, = 0.001) FFHQ 256. In bold is the best method for each metric and experiment.
Sampler
PSNR [dB]
LPIPS 1
FID |
NonRepuls-RLLe RLSD
30.56
0.073
41.11
RED-Diff
28.55
0.074
62.87
DPS
26.48
0.15
95.44
ReSample
27.49
0.062
54.42
D.6 INPAINTING FREE MASK
We use the free masks (10% 20%) from Saharia al. 2022). For this experiment, we consider
ImageNet (Russakovsky al. 2015 to demonstrate that our method outperforms its baselines other datasets.
consider the X 1..15.7 = 0.15, 4r = 0.8 and 4_ 0.4, and we consider 500 steps (instead of the full trajectory of 1000 steps) for both RLSD and PSLD. For PSLD, we the parameters from their experiments with box inpainting (similar to the case of half face). The quantitative results shown in Table 13 and qualitative results in Fig. 14
Table 13:
Free mask (Saharia al. 2022) with (a,, 0.001) ImageNet 512. In bold is the best method for each metric and experiment.
Sampler PSLD NonRepuls-RLSD
PSNR [dB]
LPIPS 1
FID |
22.56
0.154
69.43
26.77
0.075
60.53
Results when considering the best sample across particles. instead of focusing on the average performance, we focus on the performance achieved by the best image among the particle ensemble,
22

---

Published as a conference paper at ICLR 2025
then the conclusion is different in favor of the RLSD; see Table 14 This can be explained follows: while some of the modes obtained with RLSD might not be as good as NonRepuls-RLan RLSD, others might be better.
Table 14: Box inpainting (half face) with 0.001 - -FIIQ 512. We consider the best particle across each batch of them.
Sampler
PSNR [dB]
LPIPS +
FID |
PSLD (mean)
21.34
0.10
57.7
PSLD (max)
22.72
0.082
57.7
JonRepuls-RLSV (mean)
24.98
0.079
29.18
NonRepuls-RLi RR.SD (max)
25.82
0.071
29.18
RLSD (7 50) (mean)
24.69
0.111
31,41
RLSD (y 50) (max)
25.84
0.069
27.84
D.7 QUALITATIVE RESULTS
(a) NonAug-RI.n RLSD 30).
(b) Latent RED-Diff.
Figure Phase Retrieval, Adding repulsion allows to sample from different modes.
(a) ConAug-RLSS) (> 30),
(b) Latent RED-Diff.
Figure 5: Phase Retrieval. Adding repulsion allows to sample from different modes
(a)
(b)
Figure 6: Two qualitative examples of HDR. Both a) and h) corresponds to NonRepuls-RI. and each has Ground-trutte truth: Mcasurement; Estimation. NonRepuls-RLas RLSD generates an image of high fideefiiiityy in a nonlinear problem at 512 512.
23

---

Published as a conference paper at ICLR 2025
Figure 7: Inpainting half face using (from top to bottom): Ground truth Mcasurement, PSLD, NonRepuls- RLSD, and RLSD 50). We generate four samples for each method from different initialization; for RLSD, they interact through the repulsion term. First, NonRepuls RLSS, and RLSD outperforms PSLD for all four images. Second, while images and from RLSD (last row) look different, the images 1 and of NonRepuls-RLSD are similar, this illustrates that RLSD promotes diversity.
Figure 8: Inpainting half face using (from to bottom): Ground truth and Measurement, PSLD, NonRepuls- RLSD, and RLSD (> = 50). We generate four samples for each method from a different initialization; for RLSD, they interact through the repulsion term. First, NonRepuls RLSD. and RLSD outperforms PSLD for all four images. Second, image 4 from RLSD (last row) looks different (has brown eye), while the four samples of NonRepuls-RLSDD have blue eye; this illustrates that RLSD promotes diversity.
24

---

Published as a conference paper at ICLR 2025
Figure 9: Random inpainting (80%): PSLD (top) and NonRepuls-RL.N (down), four different samples for
cach method. Notice that NonRepuls-RL.S RLSD generates better reconstruction of the background.
Figure 10: Random inpainting (80%): PSLD (top) and NonRepuls- (down), and four different samples for each method. Notice that NonRepuls-RRan RLSD generates a sharper reconstruction of the face.
Ground truth
Input
PLSD
RLSD
Figure 11: Additional examples for random inpainting. Notice that RLSD have more details in the background for the first row, while the second row it reconstruct details the white label.
25

---

Published as a conference paper at ICLR 2025
Ground truth
Input
PLSD
RLSD
Figure 12: Qualitative examples different inverse problems for PSLD (third column) NonRepuls-RLe RLSD (forth column): Rows and 2 are motion blurring, Rows 3 and 4 are SR x For motion deblurring, RLSD clearly outperforms PLSD (PSLD reconstruct broken image, SR. x PSLD introduces artifacts, which are typical when sampling faces with Stable Diffusion.
Ground truth
Input
DPS
RED-Diff
RLSD
Figure 13: Qualitative examples of different inverse problems for DPS (third column). RED- Diff' (forth column) and NonRepuls-RLAe RLSD (fifth column): Rows 1 and 2 are half face inpainting. Rows and 4 are SR x Notice that for SR 8, both DPS and RED-Diff generates images that look different than the original. This expected considering that both operate at a resolution of 256 x 256. This demostrates the advantage of using high-resolution model.
26

---

Published as a conference paper at ICLR 2025
Ground truth
Input
PLSD
RLSD
Figure 14: Qualitative examples of Inpainting with free mask Imagenet. We consider PSLD (third column) and NonRepuls-RLSD (forth column). For rows 2, 4-7, RLSD outperforms PSLD by a large margin. Furthermore, notice that RLSD can reconstruct backgrounds that might difficult, such as the one row On the other hand, RLSD struggles to reconstruct fine-details as it is shown in row 3. We expand on this in Appendix D.9
27

---

Published as a conference paper at ICLR 2025
D.8 ABLATION OF RLSD
D.8.1 RUNNING TIME/NUMBER OF STEPS
We ablate the running time between RLSD and PSLD when generating two samples for super resolution x8. We ran the experiments the same NVIDIA A100 GPU with 80GB. When running the full trajectory (1000 steps), the running time of RLSD without particles is 8.8 minutes, while PSLD is 9.2 minutes. When running 200 steps, the running time of RLSD without particles is 1.75 minutes, while PSLD is 1.9 minutes. The results for this case are shown in Fig. [5] Notice that although the running time difference between RLSD and PSLD is large, our formulation leverages the diffused trajectory as denoiser. Therefore, we require fewer steps to obtain high-fidelity estimation; for instance, for super resolution we need just 200 steps. This showcases that our method works fine with just few number of steps; see also Fig.
16
(a)
(b)
Figure 15: Reconstruction for different number of steps. a) Ground-truth (left) and Measurement (right). b) The the left the reconstruction using PSLD with 200 steps (running time 1.9 min), while the two of the right are with RLSD (y = 0) (running time = 1.75 min). c) The two on the left are the reconstruction using PSLD with 1000 steps (running time 9.2 min), while the of the right with RL.SD O) (running time = 8.8 min).
Compared to ReSample Song et al. (2024). RLSD is also computationally more efficient. Specifically, ReSample involves an inner optimization within the sampling. Consequently, when using LDM (256x256 resolution), it takes 4 minutes in one A100 GPU to generate 1 image. Moreover, when we tried Stable Diffusion, took 16 minutes (considering the same hyperparameter configuration). We hypothesize that ReSample takes long time with Stable Diffusion due to the optimization step, where it might struggle to converge at every step. While this might improved, it demonstrates that the method needs more fine-tuning than ours. On the other hand, method, using Stable Diffusion as prior, can generate 4 samples with just 200 steps in less than 4 minutes (without repulsion, image per minute) and minutes (with full repulsion).
28

---

Published as a conference paper at ICLR 2025
100
L 200
300
L 500
999
LPIPS 0.065
LPIPS 0.063
LPIPS 0.062
LPIPS 0.059
LPIPS 0.058
function of (number of steps the optimization). Increasing modifies the number of denoisers that are used (the limits of the interval are the same, and it changes the step-size). Also, we need decrease 4, (from to 0.8) well the coupling term from 0.15 0.1.
Figure 16: Reconstruction
D.8.2 REPULSION TERM
Effect of 7/ We showcase an example of the effect of 7. We consider N 4 particles and 200
steps, = 0.1, A = 0.15, 4, = 0.4 and 4, = 1. It is important to remark that when considering fewer steps, we need higher 7 (compared to the example
Section [.1] the reason why this
happens is due to the annealing factor in the repulsion term (3),, given by (t)). The example for
y 0, 100, 150, 200 is shown in Fig. 17 Also, in Table we present quantiative results.
Table 15: Ablation y for inpainting with free-mask.
PSNR [dB] f
LPIPS +
FID
Diversity 1
0
30.03
0.064
22.62
0.004
25
30.04
0.064
22.21
0.005
50
30
0.064
22.31
0.005
Effect of repulsion on a fixed interval. Depending on the downstream task, we can include repulsion only fixed interval. For instance, this is the case of Phase Retrieval, where modes discrete and isolated. Therefore, once each particle get around one of the mode (different), we can turn-off the repulsion, which yields faster solver. For the case of inpainting, we show in Fig. 18 an ablation changing frepul where E [0, [zepui] and in Fig. 19 when E [/ropul, T"] (with T 200).
We observe that for diversity, it more important have repulsion the beginnig of the sampling, which corresponds to the higher noise levels. Intuitively. it is more important to impose repulsion in the high-level semantics of the image instead of the fine-details. However, adding some repulsion later in the sampling process might improve some details (see the case frepul E [100, 200] in Fig. 19p.
Effect of number of particles. Lastly, we do an ablation when considering more particles in the repulsion term. Given four particles, consider three settings: N = 0 (four independent particles), N 2 (two independent particles and two interacting particles) and = 4 (four interacting particles). We again the free mask (Saharia et al., 2022). and L. 200. In Table 16 we show quantitative results, while in Figs. 20 we show two qualitative results. In both cases for the last columns correspond to the i.i.d. particles. Consequently, those two images can change when considering N 4, while the first two columns can change when moving from N = 0
2,
two. In particular, in Fig. 20 with N 2 incerase diversity, while Fig. 21 need least
N 4.
29

---

Published as a conference paper at ICLR 2025
== 0
Y = 100
Y = 150
Figure 17: Reconstruction as function of (number of repulsion), for L = 200 and N 4. Increasing enhances diversity: for 150, out of samples show mouth, while y samples have mouth hidden.
Table 16: Ablation when changing the number of particles in the repulsion term with 0.001 FFHQ 512.
Sampler
PSNR [dB] 1
LPIPS &
FID |
Diversity
NonRepuls-RISD (N = 0)
30.03
0.064
65.42
0.002
RLSD (N = 2))
29.99
0.064
71.8
0.004
RLSD (N 4)
29.99
0.063
69.91
0.005
30

---

Published as a conference paper at ICLR 2025
trepul 20
trepul 50
trepul 100
trepul 200
Figure 18: Reconstruction for y 150, for 200, N and repulsion between and Exepul: when
brepul = T. we have repulsion in all the trajectory, while Leeuu = 0 corresponds to MonRepuls-RLSD.
NonRepnls-RLSh
trepul [150, 200]
trepul [100, 200]
trepul E [50, 200]
Figure 19: Reconstruction for 150, for = 200, N 4 and repulsion for [frepul, 200]; notice that
Erppll = T' corresponds to NonRepuls-RLSD.
31

---

Published as a conference paper at ICLR 2025
N = 0
N N= 2
N = 4
Figure 20: Reconstruction when increasing the number of particles the repulsion, for = 0, and Notice
that N = 0 corresponds to the NonRepuls BesReuul--LLDD.... N = 2 to Hybrid case, and N = 4 to RLSD.
32

---

Published as a conference paper at ICLR 2025
N
N= 2
N 4
Figure 21: Reconstruction when increasing the number of particles in the repulsion, for N = 0, and 4. Notice
that N 0 corresponds to the NonRepuls- WonRepul-RRLSD. RLSD. N 2 Hybrid case, and N = 4 RLSD. In this need full repulsion to increase diversity w.r.t the Non RLSS case.
33

---

Published as a conference paper at ICLR 2025
D.8.3 COMPARISON BETWEEN NONAUG-RLSD VS RLSD
We compare in Fig. 22 NonAug-RLaS RLSS vs RLSD. Notice that RLSD achieves a sharper reconstructed image; in particular, the shirt for the case of NonAug-RLSD RLSD has not fine-detail, while RLSD has.
(a)
(b)
Figure 22: Comparison between RLSD (top row of Fig. b) and NonAug- RLSD (last row of Fig. b). RLSD achieves sharper reconstructed image; in particular, the shirt for the case of NonAug-RLan RLSD has not fine-detail, while RLSD has.
D.8.4 COMPARISON WITH IMAGES FROM IMAGENET BETWEEN RLSD AND RED-DIFF WITH DIFFUSION PRIOR TRAINED ON FFHQ
Here show some examples of reconstruction using out-of-distribution images; we samples from ImageNet (Russakovsky et al. 2015). In Fig. 23 we show an example from ImageNet, where compare RLSD with RED-diff using FFHQ Clearly, the performance of RLSD is better than RED-diff. This is expected given that the diffusion model RED-diff with FFHQ. However, this demonstrates that using more powerful diffusion model prior enables to deploy model for different type of images.
D.8.5 COUPLING PARAMETER
We do an ablation the coupling parameter p. We consider three values: ò = {0.03, 0.07, 0.12). The qualitative results for this case shown in Fig. 24 and the quantiative in Table I7 As expected, increasing the coupling parameter yields a more diverse set of images. The reason why this happens because the prior (through 2) has weight when estimating the image However, this increase in diversity is penalized by a lower performance. On the other hand, lower p generates four images that look very similar. It important note that our repulsion term is better trade-off in terms of diversity/quality.
Table Ablation p for inpainting with free-mask 7
p
PSNR [dB] f
LPIPS 1
FID
Diversity
0.03
29.94
0.069
25.57
0.01
0.07
30.03
0.064
22.62
0.002
0.12
28.6
0.065
23.25
0.006
34

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
Figure 23: Results of the reconstruction of an image from the validation set of ImageNet using RLSD (b) and RED-diff with diffusion prior trained on FFHQ (c). This example showchases that using a large-pre trained model such as Stable Diffusion enables our method with images from very different classes (Imagenet and FFHQ).
p 0.03
p 0.07
p = 0.12
Figure 24: Reconstruction as function of p (coupling parameter). Increasing p enhances diversity but reduces
quality: for p 0.12, the 4 samples looks different but with a low quality, while with a lower the diversity is lower performance better.
35

---

Published as a conference paper at ICLR 2025
D.9 LIMITATIONS AND FAILURE SETTINGS
Throughout the experiments, we demonstrated that RLSD outperforms other baselines, particularly PSLD. Additionally, we show that ConAug-RLSSD capable of solving nonlinear inverse problems at high resolutions. However, RLSD still faces challenges in certain settings and inverse problems. Below, we highlight some of these cases, which we aim to address in future work.
Lack of fine details in ImageNet. When performing inpainting on ImageNet, we observe that our method struggles to reconstruct fine details.
MMM E
MM
Figure 25: Example of setting where method struggles to have high-quality reconstruction. In particular, RLSD generate images without small details: for instance, the building does not have windows in the walls. While this might be circumvent by increasing the weight of the prior, our simulations did not work.
When seek to reconstruct the face of a person, observe that our method might struggle based on the background. For instance, the samples from Fig. 26 look blurry and that has more natural reconstruction.
Full-face on FFHQ.
unnatural. On the other hand, in Fig. 27 we observe Still, it has a lack of details in some parts of the face (eyebrow for example).
Figure 26: Example of setting where our method struggles to have high-quality reconstruction. In particular, RLSD generate a face that looks unnatural.
When applying the same box to ImageNet, we observe the same as it is shown in Fig. 28
Repulsion too high. Lastly, important setting where RLSD fails is when 7 is too high. As an example, we show in Fig. 29 case where 7 = 200, and the same setting as in Appendix [.8] While in Appendix D.8.2 we illustrated that increasing 7 enhances diversity, if ry is too high, then the reconstruction fails.
Fixed resolution given by the diffusion prior. The dimension of the images generated with the diffusion prior is constrained by the existing pretrained diffusion priors (currently, Stable Diffusion provides resolutions up to 512 x 512). This still considred a high dimensional data, compared with existing methods that work with resolutions up to 256 x 256. Extending our framework handle higher dimensions than the one of the diffusion prior is an interesting research direction, where ideas from compositional generation might be helpful. We leave this future work.
36

---

Published as a conference paper at ICLR 2025
Figure 27: Example of a more natural reconstruction for full-face example. In particular, RLSD generate a face that looks natural than Fig. 26 Still, the eyebrow looks blurry.
Figure 28: Example from ImageNet, where we see there a lack details.
Figure 29: Example of failure when considering too high. We consider N 4 (the second row), L = 200, and the setting as in Appendix D.8
D.10 TRADE-OFF BETWEEN DIVERSITY AND QUALITY IN UNCONSTRAINED GENERATION
To demonstrate the generality of RLSD and our particle-based variational approximation, we also include additional experiments when considering unconstrained sampling: we include results for the unconstrained case. i.c., text-to-image and text-to-3D. Details about Score distillation and the methods used in this section can be found in Appendix B
D.10.1 TOY EXAMPLE WITH BIMODAL GAUSSIAN DISTRIBUTION
We consider here toy example to showcases how the parameter (the amount of repulsion) dictates the trade-off between diversity and quality. We consider a mixture of Gaussians of parameters AG ([1, 0.005I) and [2 0.005T), and we consider two settings:
1. Two independent Gaussians with
0 where fit the parameters
37

---

Published as a conference paper at ICLR 2025
2. Two dependent Gaussians with 0 where fit the mean parameters and coupled via an Euclidean RBF kemel.
For each setting, we compute 200 realizations with the same seed. In Fig. 30a we show how many realizations suffers from mode collapse (? 0 corresponds to the first setring), while in Figs. (3bb and we show realization for y and 7 = 2000, where we observe that the "quality" of the estimation for this high value is poorer compared to 7 1.
Nom. moode-collas
60
00-
00-
200
903
600
S00
1000
(a)
(b)
(c)
Figure 30: a) Number of realizations that has a particle collapse to the same mode as a function of the amount of repulsion that consider. Estimation when considering b) 7 1) 2000.
10.0.2 TEXT-TO-IMAGE GENERATION: TRADE-OFF PLOT AND ABLATION OF KERNEL FUNCTION
Implementation details. Here we describe the details of the experiments on 2D images with ProlifieDreamer (VSD) (Wang et al., 2024). which is described Appendix VSD is particular case of score distillation for unconstrained sampling. We optimize 500 optimization steps, follow their setup, and we train a U-Net from scratch to estimate the variational score: we used the code from the public repositoryhttpp... consider ADAM optimizer and set the learning rate of particle images 0.03 and the learning rate of U-Net is 0.0001. We consider only 4 particles as it is enough promote diversity (in particular when considering DINO as feature extractor). The images and parameters are initializad at random. We run all the experiments in single NVIDIA A100 GPU of 80GB.
git.ub..bb...m com/ yuwnzhi zht/ htt.:prolifiii.i www.ddreeeer...mm
Trade-off plot. We study the trade-off between diversity and quality when increasing the amount of repulsion. We consider 75 images from the COCO dataset and compute average qualities and diversities across all images. We also include comparison with stochastic sampling using the Euler discretization the backward process, with 30 steps (denoted by 'Ancestral'), ). This serves an upper bound on the performance of distillation techniques. We consider 4 particles for both the distillation optimization and the sampling via Buler. The results are shown in Fig. 3T where we compute FID (lower is better), Aesthetic (higher is better) and CLIP (higher is better) scores function of diversity score (27); a higher diversity score corresponds to more diverse image generation. For the plot, we fix all the hyperparameters, and sweep y between (ProlificDreamme) and 40, and we consider Ancestral sampling as an upper bound in terms of trade-off. Clearly, adding repulsion increases the diversity, while slightly decreasing the quality metries for low values of 7 However, for these lower values of 7, the diversity is still far from the stochastic sampling. When increasing the values of above 30, close the gap in terms of diversity the cost of decreasing the quality (FID and Aesthetic) as well as the text-aligne aliigmmeteet (CLIP).
Using different domains/distances for the repulsion force different domains, namely Euclidean, DINO and LPIPS.
We compare the RBF when considering
1. RBF in the Euclidean domain. We compare between DINO and RBF for repulsion with ? = 10 with two qualitative examples in Figs. 32 from COCO validation set (Lin et al., 2014) constructed in (Jain et al. 2022).
38

---

Published as a conference paper at ICLR 2025
Prilife mppission
Aneetral
0,94
-y
Aestheta repulconn
6.35
216-
Anuestral Prolife-
5.30
Clap
Paoliee pesubion.
214
5.20
Ancesical
Prolifo
6.15
0.58
0.30
0.35
0.35
0:38
0.25
0.30
01:33
0.30
6:38
0.56
0.30
0.35
0.35
0.38
Doveniity
Divanity
Divenity
Figure 31: We consider the trade-off between diversity and quality for text-to-2D image generation using score distillation. We consider {0. 10, 20, 30, 40}. From left to right: Diversity (27) vs left) FID, middle) Aesthetic score and right) CLIP score. We consistently observe that adding repulsion increases diversity, while slightly decreasing the quality of the images. Notice that we need repulsion with y > 30 to reach the diversity level of stochastic sampling using Euler.
Figure 32: Comparison between RBF with Euclidean distance and DINO. We generate samples using the
prompt a) "a gold tied under brown dress shirt with stripes.." b) baseball bat with a batting helmet upsidedown." ". From the top to bottom: {0, 10 L0(uuclidean), 10(DINO)) with CFG = 7.5 and 500 steps.
2. RBF using LPIPS. We also tried LPIPS metric in RBF. However, we observe that when combined with ProlifiDrreamer, the generated images have some artifacts or are not well aligned with the text prompt; two exampes are shown in Figs. 33
Comparison between RBF with LPIPS (top row) and DINO (bottom row). We generate samples using the prompt a) "a gold tied under brown dress shirt with stripes..". b) "a baseball bat with a batting helmet upsidedown." ". From the top to bottom: (10(LPIPS), 10(DINO)) with CFG = 7.5 and 500 steps.
Figure 33:
D.10.3 TEXT-TO-3D GENERATION
Lastly, demonstrate how method improves the mode collapse phenomena in text-to-3D generation. Our main motivation is to show that including the repulsion entails more diverse set of scenes when changing the seed.
39

---

Published as a conference paper at ICLR 2025
Implementation details. We consider DreamFusion (Poole et al. 2022 base method and the implementation from the Threestudio framework (Guo et al. )023). All 3D models are optimized for 10000 iterations using Adam optimizer with learning rate of 0.01. and use the same configuration as DreamFusion for the rendering. For the NeRF architecture we use an MLP, and we use the same configuration from the setting in Threestudio. For the diffusion model, with guidance scale of 20. For the repulsion, we use repulsion weight of y = 2000;, We use one Nvidía A A10 of 80GB, and consider a batch of 20 particles; this presents limitation for using more computationally -emand methods such as ProlificDreamer. For the RBF, we use DINO,
DeepFloyd-IF-XI--V
Results. In Figs. 34 and 39 we show two qualitative examples. For each case, show two different views of the generated object. Furthermore, we include the corresponding gif files for the prompt "An ice cream sundae" in Figs. 31 34. Notice that with repulsion colors of glass for the ice cream (dark and white), something that we could not achieve without repulsion. Furthermore, while the case without repulsion generates perspective, our method generates two scenes that show the ice cream at different distances. Lastly, notice that the quality of the generated scene is bounded by the performance of the base method (in this case, DreamFusion). Therefore, the results look saturated, similar to what happen with SDS.
generate two different
scenes that have similar
(a) Without repulsion Seed (
(b) Without repulsion Seed 15
(c) With repulsion Seed
(d) With repulsion Seed 15
Figure 34: Text-to-3 generation using DreamFusion with the prompt "An ice cream sundae" and considering batch of 20 samples (particles) a, without repulsion with seed and seed 15, with repulsion with seed and 15 respectively. For cach case, we show two different views of the object. Clearly, the two cases without repulsion look very similar, generating the same type of ice cream sundae. On the other hand, adding repulsion increase diversity of the scene (a different glass, and color). However, this comes the cost of less details in the ice cream.
Figure 35: Without repulsion Seed 0
Figure 36: With repulsion seed 0
40

---

Published as a conference paper at ICLR 2025
Figure 37: Without repulsion seed 15
Figure 38: With repulsion seed 15
(a) Without repulsion Seed
(b) Without repulsion Seed 35
(c) With repulsion SSedd
(d) With repulsion SSed 35
Figure 39: Text-to-3D generation using DreamFusion with the prompt bulldozer made of toy bricks", and considering batch of 20 samples (particles) b) without repulsion with seed I and seed 35, c. d) with repulsion with seed and respectively. For each case, we show two different views.
E SCORE DISTILLATION
Score distillation sampling (Poole et al. 2022 was the first proposed method to optimize a generator by using distillation. Consider pretrained score function Eg (X,,t) - log ((x,( representing distribution of interest Po- The idea of score distillation sampling (SDS) is to train generator Gs. such that the output of the generator given an input m sample X I (m(m) ~ po- This is achieved by optimizing the distillation loss (Poole et al. (022)
(xp = 96 (-) = Fo..240..) w(t) Tt KL (q(x,/x0 = 9o(-)) 11 po(xx))
(28)
where w(t) weighting function and ((x; Xg = (-)) is the variational distribution; when in the case our proposed method in Section 4
(.).) Φ. i.e., identity mapping, then
Although its remarkable success in generating 3D scenes, these methods suffer from a mode collapse problem. This is mainly driven by the choice of the (reverse) KI. divergence the loss in (28) and the fact that a unimodal variational distribution q is used. In particular, they need to consider a high CFG in the classifier-fres ffree guidance score (Ho and Salimans) 2021
60 (Zr; c, () = (z:; Ø3, + w (ep (2,,, (Z 23, ())
(29)
where Ø indicates null condition and the w is the CFG weight; they consider w 100.
The mode collapse phenomenon is clear when observing SDS through the lens of gradient flows 4.1 which entails a seeking optimization. Therefore, if we want to avoid mode collapse, we need to modify either the loss function (the divergence) the variational approximation. While the former renders intractable formulations, the latter can be modified easily. Therefore, previous works focused on modifying the variational distribution. See, for instance, PromificDreamer (Wang al.
41

---

Published as a conference paper at ICLR 2025
2024), and more recently, in Luo al. (2024); also other works (Katzir al. 2024 Wang al.)
2023) have studied this problem, and proposed alternative approaches to circumvent this problem.
In ProlificDreamer, the authors consider particle approximation of the variational distribution by randomizing the parameters 0, which yields the following update rule
0xo
VoLvsn = Er,c,c ((()) (xo(Xi; c, - or Vx, log (x+(())
(30)
00
Notice that o Xx, log (xx(c) unknown, so they fine-tune pre-trained diffusion model for each particle (which represents each rendered image). Therefore, they incorporate a second optimization problem that minimizes DSM loss; they parametrize the score network using LoRA (Hu et al. 2021. However, adding an auxiliary neural network does not necessarily promote diversity.
E.
GRADIENT FLOW PERSPECTIVE OF SCORE DISTILLATION
We can interepret the variational diffusion sampling optimization procedure as a Wasserstein gradient flow when constraint to
manifold.
Wasserstein gradient flow. WGF describes how probability distributions change over time by minimizing a functional on P(R"), representing the space of probability distributions over IR" with finite second moments. Denoted as (P(R"), W2). this space employs the Wasserstein-2 distance as its metric, termed the Wasserstein space. Before delving into the WGF, we define the Wasserstein gradient of a functional F(4) as
Vw, Vx 8F(())
(31)
8g
where &F(() limm-+0
F(a+ea)-F(a) the first variation defined for any direction in the tangent space of P. Given this definition and two boundary conditions pa = go(x) and P = p(x), we can define path of densities q where its evolution is described by the Liouville equation (also known as continuity equation)
bg
Ögr div (q: Vw, (q+)) Òr
(32)
At the particle level, for given particle x, g. inl the gradient flows defines dynamical system drive vector field {+r}-20 in the Euclidean space IR" given by
dx, = Uy (x,) = Vw, (q:) (x,) dr.
Therefore, this ODE describes the evolution of the particle x, where the associated marginal g- evolves to decrease F (qr) along the direction of steepest descent according the continuity equation (32)
Notice that the WGF is defined in continuous domain for r. We can discretized via the following movement minimization scheme with step size h, also known as the Kinaannnnnnndeeerr-Orre (JKO) scheme (Jordan al. 1998).
Ck+A = argmin { [5()
1 W2 (q. (m. 2h
+
(33)
(EP(R")
Notice that the JKO scheme has two terms: the first seeks minimize the functional F(g) and the second is a regularization term, that penalize to stay close to in Wasserstein-2 distance as much as possible. can be shown that as -> the limiting solution of (33) coincides with the path (ar)r20 defined by the continuity equation 32
Throughout this work, we consider the KT. divergence the function, i.e, Faa(g) = KL(glpp). Then, the Liouville equation boils down to the Fokker-Plance equation
dq. div (gz (Vx log 9- Vx logp)) δT
(34)
where the Wasserstein gradient is V FR. (q.) Vx log (g//p) and the probability flow ODE follows (6). Lastly, although here we focus on the Wasserstein metric, we can consider other metrics that yield different flows (Chen al. 2023). This is future avenue to explore.
42

---

Published as a conference paper at ICLR 2025
Bures- Wrresssessrrtnin gradient flow. We now show that RED-diff (Mardani al. 2024 can be de- rived from the gradient flow perspective when considering the flow in in the Bures- Wasserstein space (BW (RR), W2), i..., the subspace of the Wasserstein space consisting of Gaussian distributions.
The Wasserstein-2 distance between two Gaussian distributions = N (Mgg Egg) and p = =N (Aps Mp))) has closed form,
W3 (2 p) 11 ppll2 B8 (Egz Ep);
where B Ep) = E ( ( is the squared Bures distance. By restricting
the JKO scheme in (33) to the Warres-WWssesstein space, the authors in Lambert et al. 2022) showed
that the discretization entails solution given by the limiting curve (qt . (14, E.) 1200 Je20 Therefore,
the gradient flow of the KL divergence the Bures- Wasserstein space boils down the evolution of
the means and covariance matrices of Gaussians, described by the following ODEs:
d1l dr
p(x)
Ex-ar
V log
(35)
2 (x)
d2' d5
p(x) (x(x)) ) (x ur)) + Ex
p(x)
Ex--gr
V log
(x pT)) Vx log
(36)
Cr (x)
In essence, RED-diff (and DreamFusion as well) follows the ODE corresponding to the mean with the diffusion as regularizer and the likelihood as measurement term. The key of the proposed methods is that the variational distribution uses the diffused trajectory (and therefore, the denoiser) as regularizer.
F COMPARISON WITH OTHER PARTICLE METHODS USING DIFFUSION MODELS
F.1 STEIN VARIATIONAL GRADIENT DESCENT (SVGD)
SVGD is deterministic particle-based variational inference method (Liu 2017), 7, and is the core
method of (Kim et al. 2023). Through the lens of gradient flow, it optimizes the same functional (KI. divergence) but considers a different metric induced by the Stein operator (Duncan al. (023). In
particular, given a pairwise repulsion as RZ (N) = k 2 2tr
(
the update direction in
SVGD for the particle zt, given by
Σ ks B1,T
log 6) 2 (4,7
( k(z
(37)
When comparing the gradient update 37 with (7).. the main difference resides in how is computed the gradient of the log target: while in SVGD, all the ensembles follow the same averaged gradient direction weighted by the similarity kernel function, direction. This difference in the score has important implications: SVGD suffers from the curse of dimensionality and poor exploration of the space in high-dimensional multimodal distributions, suffering the mode collapse phenomenon. Some alternatives have been proposed, like adding an annealing schedules (D Angelo and Fortuin, 2021a Ba et al. [02T], but the method still discovers only few modes in multimodal distributions. Conversely, the repulsive method achieves better exploration, and therefore, it finds more modes.
our method each particle uses its score
F.2 PARTICLE GUIDANCE
In the context of sampling, a recent work termed Particle guidance (Corso et al., 2024) proposed method to incorporate an interactive particle sampler based on diffusion. They proposed a guidance term that couples a set of particles to guide the backwards diffusion process towards different modes of the target distribution. Although they leverage a similar idea, the scope of our work is different. Here focus diffusion model in the context of distillation, and therefore, which allows to deploy in constrained problems like inverse problems, or unconstrained cases like text-to-3D.
regularizer,
43

---

Published as a conference paper at ICLR 2025
F.3 COPULA MODELS
Copula models Nelsen 2006); Tran et al. (0115) are methods for defining variational distributions that incorporate dependencies between variables. While typically this is done for modeling the relationship between each dimension, in this case, we use the copula model to define dependencies between batch of particles and promote diversity via repulsion.
G VARIATIONAL SAMPLERS FOR INVERSE PROBLEMS
Variational inference methods Blei et al. 2017) are one of the most important techniques for sampling from intractable distributions. In a nutshell, VI defines parametric distribution where the parameters are learned via stochastic gradient methods. key difference between variational samplers and others, such as those based on the Monte Carlo Markov Chain, is that VI defines a parametric distribution. Thus, once the parameters are learned, we can easily generate samples.
In inverse problems, different variational samplers have been proposed in Knoll et al. [201] Portilla jet al. (2003); Kobler al. (2017). The difference between these approaches lies the dehnition of the variational distribution. For instance, Kobler et al. (2017) leverages the algorithm-unrolln framework, while Portilla et al. (2003) uses mixture Gaussians in the wavelet domain. More recently, the authors in Tonolini al. (2020) proposed two-stage semi-supervised method for learning variational samplers: first stage that learns the forward model and a second stage that trains conditional variational auto-encoder that learns solve the inverse problem.
Related to our work, both Feng et al. (023) and Mardani et al. (024) combine variational inference with diffusion priors. Our method has three main differences with Mardani et al, (2024): 1) we define our variational inference problem a latent space, allowing us to exploit the latent diffusion model, 2) we consider multi-modal variational distribution with a repulsion term couple the particles, enabling a better exploration, and 3) we decouple the data and prior term to handle the challenges of latent inversion. On the other hand, Feng et (2023) also considers a variational perspective using diffusion priors. First, they consider based diffusion models, which simplifies the formulation. Second, they incorporate the diffusion prior by computing the log probabilities: this requires solving the underlying ODE and estimating a divergence, which is computationally expensive. Lastly, they consider normalizing flow as variational distribution, i.e., they define a distribution 4s (x l parameterized by Φ; in particular, they consider RealNVP (Dinh et al. 2017). While this differs from our definition in (8), our framework allows incorporating a normalizing flow as a variational distribution. Exploring this type of parametric distribution opens interesting research direction, where we define an amortized distribution that can generate solutions only a few steps; we leave this as future work A pseudo-code version of this method is shown in Alg 3
Algorithm 3 Score-Based Diffusion Models as Principled Priors for Inverse Imaging Feng al. (2023)
Require: y, f.),, so(xe, t) for Niter do
Compute log po (Xo) log (xT)+ V-(-38(()zdd (xt) t)) dt via ODE solver and divergence estimation Minimize DKL. (gellpo( y))
end for return (xly)
Lastly, it is important to note the connection of RLSD with plug-and-play methods Kamilov et al. 2023); Zhang et al. 202I). However, none of these works incorporate latent diffusion model as denoisers.
44


Reviews:
[{"review_id": "joyelqMId8", "reviewer": "Reviewer_DJhq", "summary": "The paper introduces Repulsive Latent Score Distillation (RLSD), a novel variational posterior sampling framework designed to address key challenges faced by Score Distillation Sampling (SDS) in high-dimensional data, specifically mode collapse and latent space inversion. By leveraging an interactive particle-based variational approach with a repulsion mechanism, RLSD encourages diversity among solutions through a kernel-based regularization strategy. Additionally, the framework incorporates an augmented variational distribution that decouples the latent space from the data space, optimizing computational efficiency while preserving detail and quality. The authors validate RLSD through extensive experiments on high-resolution (512 × 512) inverse tasks, showcasing improvements in linear inverse problems like inpainting, deblurring and nonlinear inverse problems like phase retrieval using Stable Diffusion models.", "questions": "Please refer to the \"weaknesses\" section for questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In Table 1, the authors compared RLSD with all the relevant prior works in a very clear way, emphasizing the scalability over dimensions and linear / nonlinear inverse problems as well as the avoidance of score Jacobian computations. \n2. The introduction of a repulsion regularization mechanism that leverages kernel-based similarity to prevent particles from collapsing into a single mode is a step forward. This approach promotes exploration of diverse solutions, a common challenge in high-dimensional generative tasks.\n3. The presentation of this paper is good, the reference is relatively complete. \n4. The paper provides robust experimental results across different types of inverse problems, including linear and nonlinear tasks. The diversity and quality trade-off demonstrated by RLSD is well-supported by quantitative metrics and visual comparisons.", "weaknesses": "1. While the proposed approach is well-executed, its novelty compared to existing methods could be more explicitly justified. I think using repulsive forces for diversity and variational formulations for latent space decoupling are not truly unique and novel, compared the existing framework of variational formulations. \n2. For the experiment section, is it possible for you to add more discussion or results on the performance of RLSD with larger-scale datasets and higher-dimensional spaces to clarify practical usage limits?\n3. For the related works section, can you include a part (or put in appendix) with the comparison to other variational samplers as well as the difference between their pseudo-codes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpFUEOJXRO", "reviewer": "Reviewer_GZBS", "summary": "This paper presents Repulsive Latent Score Distillation (RLSD), a method that addresses the challenges of mode collapse and latent space inversion in Score Distillation Sampling (SDS) for high-dimensional data. RLSD introduces a repulsion mechanism to promote diversity among solutions and an augmented variational distribution to disentangle latent and data spaces. Inspired by the Wasserstein gradient flow, RLSD balances computational efficiency, quality, and diversity in inverse problems like inpainting phase retrieval, and HDR.", "questions": "1. A recent baseline published in ICLR 2024 [1], which also uses a latent-diffusion-based approach for inverse problem-solving, is missing. Can you clarify the advantages of your method over this baseline? I suggest that a direct comparison would be beneficial.\n\n2. The variety of linear inverse problems in the main text is limted. I reviewed the appendix for additional experiments on other tasks. Why not include these results in the main text? A fair and complete comparison with PSLD, Latent RED-Diff, RED-Diff, DPS, and [1] across various linear inverse problems would strengthen the analysis.\n\n3. Since Algorithm 1 involves two optimizer steps, could you clarify the computational efficiency of the proposed method?\n\n\n[1] Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency. Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, Liyue Shen. ICLR 2024.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I appreciate the overall strong contribution of this paper. It is well-written and easy to follow. The proposed method introduces a repulsion mechanism to enhance diversity among solutions and an augmented variational distribution to separate latent and data spaces. Experiments on both linear and nonlinear inverse problems demonstrate the method’s effectiveness.", "weaknesses": "1. A recent baseline published in ICLR 2024 [1], which also uses a latent-diffusion-based approach for inverse problem-solving, is missing. \n\n2. I found the coverage of linear inverse problems to be limited. Previous works like DPS, PLSD and RED-DIFF typically include four types of linear inverse problems—such as inpainting, super-resolution, Gaussian deblurring, and motion deblurring—which would strengthen the analysis.\n\n\n[1] Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency. Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, Liyue Shen. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OZTmxMJuwZ", "reviewer": "Reviewer_XHxy", "summary": "This paper focus on solving two challenges in inverse problem solving with latent score distillation: mode collapse and latent space inversion. To solve mode collapse, the authors add a repulsive regularization loss to push different samples away from each other. To improve the latent space inversion, the authors define an augmented joint distribution $p(z_0, x_0 | y)$ and propose a two-step optimization algorithm. The authors demonstrate their model on FFHQ 512 dataset using the inpainting, phase retrieval and HDR tasks.", "questions": "1. In the Sampling setup, the proposed method employs Stable Diffusion v2.1. Do the baseline methods (PLSD, latent RED-Diff, latent DPS) also use Stable Diffusion v2.1? It appears that in the original paper, PLSD used Stable Diffusion v1.5. Additionally, it may not be fair to directly compare methods using Stable Diffusion as a prior with those using pixel-level diffusion models as a prior, as the pixel-level diffusion models are smaller in size and trained on less data.\n\n2. In the RLSD, N=4 particles are generated per noisy measurement. How are these four different results used to calculate the metrics reported in Tables 2, 3, and 4? Are the metrics calculated by selecting the best result among the four, or are they averaged across all four results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper presents a clear and well-structured introduction to the concepts of repulsive regularization and augmented variational distribution. The motivation behind these methods is well-founded, and the theoretical derivation is both detailed and easy to understand.", "weaknesses": "My major concern is that the effectiveness of the two proposed techniques in this paper does not seem to be fully supported by the experiments. The proposed techniques do not consistently yield good results under all conditions. In more detail:\n\n1. Table 2,3,4 show that there is no single setting that can consistently perform the best (or almost the best) across all the three tasks in this paper. For example, NonRepuls-RLSD works the best in HDR, but it performs badly  (second-to-worst) in Phase retrieval. NonAug-RLSD shows best results in Phase-retrieval, but it performs bad in HDR and inpainting. And the full RLSD model has never achieved the best results across the three tasks.\n\n2. While directly enforcing dissimilarity between particles can intuitively enhance their diversity, pushing particles apart may also drive them away from the true data manifold and affect the condition $y = f(x) + v$. Therefore, the strength of this term should be highly influenced  the landscape of the underlying distribution—specifically, whether it is indeed highly multimodal. This likely explains why incorporating repulsive regularization does not always yield improved results.\n\n3. The two proposed techniques introduce several new hyperparameters that require tuning (for example, $N, \\gamma, \\rho, l$). While some quantitative results are presented in D7, a comprehensive qualitative ablation study examining the sensitivity of performance to different parameter selections might be needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7ZqBLsOrNl", "reviewer": "Reviewer_Xthg", "summary": "The paper proposes (Repulsive Latent Score Distillation) RLSD, a new method leveraging pre-trained diffusion models for inverse problems. RLSD uses a repulsive term in its variational sampling process, which aims to address mode collapse by promoting diversity. Experimental results suggest that RLSD may offer improvements in both quality and diversity for tasks like image inpainting.", "questions": "How sensitive is RLSD to the initialization of particle positions? \n\nIn cases where diversity is essential, what strategies do you suggest to improve convergence to distinct modes?\n\nCan RLSD achieve similar performance when the number of particles is reduced? If so, what modifications, if any, would be required?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The RLSD approach is novel, with a theoretically sound foundation in Wasserstein gradient flow and variational sampling. Additionally the repulsion term does appear to promote diversity.", "weaknesses": "While comparisons with a few models are presented, the paper does not address how RLSD performs against the broader set of models for inverse problems using diffusion prior methods. For example how about comparisons with DDNM, Pi-GDM, FPS, FPS-SMC, ect.? Without these comparisons it makes it difficult to gauge just how much of an improvement this approach is compared to similar approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)