Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior
Authors: Tongda Xu
Xiyan Cai
Xinjie Zhang
Xingtong Ge
Dailan He
Ming Sun
Jingjing Liu
Ya-Qin Zhang
Jian Li
Yan Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion models
Inverse problem
Primary area: generative models
TL;DR: We show that diffusion posterior sampling is in fact maximizing a posterior, and propose two improvements based on that.

Abstract: Recent advancements in diffusion models have been leveraged to address inverse problems without additional training, and Diffusion Posterior Sampling (DPS) (Chung et al., 2022a) is among the most popular approaches. Previous analyses suggest that DPS accomplishes posterior sampling by approximating the conditional score. While in this paper, we demonstrate that the conditional score approximation employed by DPS is not as effective as previously assumed, but rather aligns more closely with the principle of maximizing a posterior (MAP). This assertion is substantiated through an examination of DPS on 512$\times$512 ImageNet images, revealing that: 1) DPS’s conditional score estimation significantly diverges from the score of a well-trained conditional diffusion model and is even inferior to the unconditional score; 2) The mean of DPS’s conditional score estimation deviates significantly from zero, rendering it an invalid score estimation; 3) DPS generates high-quality samples with significantly lower diversity. In light of the above findings, we posit that DPS more closely resembles MAP than a conditional score estimator, and accordingly propose the following enhancements to DPS: 1) we explicitly maximize the posterior through multi-step gradient ascent and projection; 2) we utilize a light-weighted conditional score estimator trained with only 100 images and 8 GPU hours. Extensive experimental results indicate that these proposed improvements significantly enhance DPS's performance. The source code for these improvements is provided in https://github.com/tongdaxu/Rethinking-Diffusion-Posterior-Sampling-From-Conditional-Score-Estimator-to-Maximizing-a-Posterior.

Published as a conference paper at ICLR 2025
RETHINKING DIFFUSION POSTERIOR SAMPLING: FROM CONDITIONAL SCORE ESTIMATOR TO MAXI-
MIZING A POSTERIOR
Tongda X Xiyan Cait.s, Xinjie Zhang', Xingtong Ge Dailan Hc l Institute for Al Industry Research, Tsinghua University Department of Computer Science and Technology, Tsinghua University
Ming Sun?, Jingjing Liu', Ya-Qin Zhang', Jian Li Yan Wang
3 New York University, 'Hong Kong University of Science and Technology, SSenseTime Research TThe Chinese University Hong Kong, Kuaaishou Technology
Institute for Interdisciplinary Information Sciences, Tsinghua University
x. tettnnnanny.eeeee edu, lapordgel gma 1. com, Langyan@air..sinnhuua tsinnguu....n.. edu. cn
ABSTRACT
Recent advancements in diffusion models have been leveraged to address inverse problems without additional training, and Diffusion Posterior Sampling (DPS) (Chung et al., 2022a) is among the most popular approaches. Previous analy- ses suggest that DPS accomplishes posterior sampling by approximating the con- ditional score. While in this paper, we demonstrate that the conditional score approximation employed by DPS is not as effective as previously assumed, but rather aligns more closely with the principle of maximizing posterior (MAP). This assertion is substantiated through an examination of DPS on 512x512 Ima- geNet images, revealing that: DPS's conditional score estimation significantly diverges from the score of well-trained conditional diffusion model and is even inferior to the unconditional score; The mean of DPS's conditional score esti- mation deviates significantly from zero, rendering it an invalid score estimation; 3) DPS generates high-quality samples with significantly lower diversity. In light of the above findings, we posit that DPS more closely resembles MAP than con- ditional score estimator, and accordingly propose the following enhancements to DPS: 1) explicitly maximize the posterior through multi-step gradient ascent and projection; 2) we utilize weighte conditional score estimator trained with only 100 images and 8 GPU hours. Extensive experimental results indicate that these proposed improvements significantly enhance DPS's performance. The source code for these improvements is provided in this link
INTRODUCTION
In recent years, diffusion models have emerged as a powerful tool for solving inverse problems without additional training (Song et al.) 2020 Chung al. 2022a Rout al. 2024). One prominent approach leveraging diffusion models to tackle inverse problems is Diffusion Posterior Sampling
(DPS) (Chung al., 2022a). DPS has gained significant attention as it effectively produce high quality samples for various image restoration problems, such as super resolution and deblurring.
The conventional understanding of DPS posits that it approximates the conditional score achieve posterior sampling (Chung et al. 2022a Song et al. 2023c). Various subsequent works follow this explanation (Yu al., 2023 Boys al. 2023) Rout al. 2023 Chung et al.) 2023 Yang al.) 2024]. However, recent theoretical study has revealed that this approximation has large error lower bound (Yang et al. 2024). In this paper, we challenge the prevailing view by presenting a numerical
examination of DPS in practical scenarios, particularly for 512x512 ImageNet images. Our analysis
"To whom the correspondence should be addressed.

---

Published as a conference paper at ICLR 2025
reveals that DPS aligns more closely with the principles of maximizing posterior (MAP) rather than conditional score estimation.
From empirical study, make three major observations: 1). The conditional score estimation of DPS considerably diverges from the score of a properly trained conditional diffusion model and even outperformed by unconditional score; 2). The mean of DPS's conditional score estimate significantly deviates from zero, thus failing to qualify as a valid score; 3). The samples generated by DPS, although of high quality, exhibit markedly lower diversity. These findings collectively argue that DPS more closely aligns with the MAP framework than with conditional score estimation.
Given these insights, we propose enhancements DPS better align it with the concept of MAP. Our proposed modifications include: 1) Explicit maximization of the posterior through multi-step gradient ascent and projection: 2) The employment of a lightweight conditional score estimator trained with 100 images and 8 GPU hours. Extensive experimental evaluations demonstrate that these improvements notably boost the performance DPS.
Our technical contributions can be summarized as follows:
(Section We demonstrate that DPS aligns more closely with the principles of maximizing a posterior (MAP) than the conditional score estimation, by showing it exhibits significant score estimation errors, high score mean, and low sample diversity.
We introduced multi-step gradient ascent algorithm to explicitly maximize the posterior and lightweight conditional score estimator trained with 100 images and GPU hours, and both of them significantly boost DPS's performance.
(Section 4
(Section 5) Our extensive experimental results substantiate the effectiveness of the pro- posed enhancements, significantly improving the performance metrics of DPS.
2 PRELIMINARIES
Denoising Diffusion Probability Model Diffusion models represent an important class of genera- tive models, which utilizes a T'-step Gaussian Markov chain (Sohl-Dickstein al. 2015). One of the most widely adopted diffusion models the Denoising Diffusion Probability Model (DDPM) We denote the source image Xu, and the forward Markov chain of DDPM is:
(Ho et al. 2020).
q(xr. X,) = II x(x,(Xx-1), where ((x,((x-11 = NWI.IN(IIXX. -x,,,,1 (1) t=1
where 3r are hyperparameters. The reverse process of DDPM is a Markov chain with the transition Po(Xi-1|X.)) with the score function Vx, log p(X:) approximated by a neural network sy (Xe, t):
Po .... X1) p(Xx)
II [=1
Pe (X(Xx_1XX1),
where po Po (X -N(
(X, + 3i5o (Xe, (),, o;
(2)
Var
and O2,, of are parameters determined by 3e (oo et al. 2020).
Diffusion Posterior Sampling On the other hand, Diffusion Posterior Sampling (DPS) (Chung et al., 2022a) extends DDPM by enabling conditional sampling. Given an operator f(.) and an observation y = (X() from some unknown X/, DPS can approximately sample from the posterior (Nool))). utilizing the pre-trained DDPM model (Xo).
Specifically, For each DDPM step Eq. 2 DPS includes an additional offset that penalizes the distance between the transformed posterior mean and the measurement. More specifically, after X,-1 obtained from DDPM, DPS updates additionally with:
X,-1 X,-1 G x. [E[X(EE]]])
vll.
(3)
where & are hyperparameters and E.X, X,) the posterior mean estimated by Tweedie's formula:
x =
Xx + (1 - oz (X,,
(4)
Vā:
In this paper, adopt the notation of DPS in pixel space. For DPS in latent space, Appendix,
2

---

Published as a conference paper at ICLR 2025
3 DPS IS CLOSER TO MAXIMIZING A POSTERIOR
3.1 DPS AS CONDITIONAL SCORE ESTIMATOR
key theoretical justification for DPS is that it can be interpreted as a conditional score estimator. Chung et al. (0222)) and [Song et al. (023c) argue that to sample from the posterior po (Xoly), one can integrate an estimate of the conditional score xx log po (X(llly) into the DDPM update in Eq. and replace the unconditional score model so so(X,, t). Then, to sample from pe (Xoly)) we can
perform ancestral sampling through a new Markov chain with the transition distribution:
Pe(Xi-1|X, y) = N (X, + 3e V. log (X of
(5)
Vate
To estimate the conditional score Vx, log pa (X, (X,yy). Chung et al. (2022a) and Song et al. show that can be decomposed into the unconditional score model and a likelihood term:
(2023c)
Vx, log Po = so (Xe, t) + Vx, log po (v)
(6)
As Y Xo X, forms a Markov chain, the term (a X,) can be estimated via Monte Carlo:
L...k K
Pa X2) = PPpo =
Σ
p(yla));
K
(Xoox)
where p(y]Xo) & (op(-A(((X))
and (....) is some distance metric. When the posterior
sample oó is approximated by the posterior mean E/ X X,). and the distance A(..)) is the 12 norm weighted by Çr. the above Monte Carlo estimation becomes DPS (Chung et al. 2022a):
Po (o(y ply Xu- B[X, x,)) exp (- (x,) vll).
Many subsequent works follow this theory to explain why DPS works (Chung et al.
2022a Song
al. 2023c Yu et al., 2023] Boos et al. 2023 Rout et al. 2023 Zhang al. 2024 Chung et al.
Yang al. 2024). Chung et al. (2022a) and Song et al. (023c) support this explanation by showing that the approximation error to p(y) X.) is upper-bounded. However, Yang et al. (2024)
2023
challenge this theory by demonstrating that the approximation error has a large lower bound.
To verify whether DPS approximates the conditional score, we first represent the conditional score estimation of DPS using the unconditional score and the 12 norm:
Proposition 1. DPS is equivalent to DDPM with estimated conditional score:
spps (X,s((X,t,g)) = (x,
04 B,
G.
x.)) yll.
In the following sections, will examine sDPs t, y) in practical scenarios, specifically for 512x512 images, to determine whether it is good estimation of the conditional score.
3.2 OBSERVATION DPS HAS LARGE SCORE ERROR
Our first observation is that the conditional estimation of DPS has large error, when compared with the unconditional score. Furthermore, when tuning the hyperparameter G,, a larger score error often results in better image quality.
More specifically, we consider the scenario where f(.) is x8 bicubic down-sampling, We use the score of StableSR (Wang al., 2024), a state-of-the-ant Stable Diffusion- based super-resolution method, as the baseline (See details in Appendix. B.T.. We observe that starting from the score of StableSR, the distance to the score of DPS is sigmitcantly larger than the distance to the uncondi- tional score. Denote the score of DPS as sDPs(X; X,,t,y), the score of StableSR as so (x,, (, v) and the unconditional score as so( X,, t), we empirically observe:
|| X,, , t, y) sy (X,,,,, w) ||so(X,,
(X,, t.
(10)
In Figure [ we show the score error of different methods compared to StableSR. The results indicate that only when G. 0.05 and > 500 is the score error of DPS marginally smaller than that of the
3

---

Published as a conference paper at ICLR 2025
unconditional score. However, when 0.05, DPS does not work well (See Figure 2 and Table Пp. For all other Çe. the score error of DPS is significantly larger than that of the unconditional score. Additionally, Table [ shows that Çe decreases, the image quality of DPS deteriorates. This leads to an unexpected phenomenon: for DPS, a larger score error correlates with better image quality.
To verify that StableSR is reliable baseline for conditional score, train super-resolution Con- troINet Zhang et al., 2023), which uses a different neural network. However, as shown in Figure the score of the ControlNet significantly smaller than unconditional score. Besides, StableSR outperforms DPS in FID (Fréchet inception distance) (Heusel et al. 2017). KID (Kernel Inception Distance) (Binkowski al. 2018) and LPIPS (Learned Perceptual Image Patch Similarity) (Zhang et al. 2018). confirming that it is reliable estimate of the conditional score.
downcampless
DPS ((-0n5)
DPS (C6
DPS (0=0.05)
ors (t-12)
prs (Q
unconditonal Contro
10-4
DPS (c-4.5)
StableR
10
10
10-4
600
Figure 2: Qualitative results of different ap- Figure 1: The score to StableSR. DPS proaches for SRx8. DPS only works with has a larger error than unconditional score. Se = 4.8.
Table I:
Quantitative results for S R.bble 2: The mean of score for SRx8.
DPS only works with & 4.8.
The mean of DPS (G 4.8) is large.
LPIPS
FID
KID)
Score's ' Mean (k=1000)
DPS (G 0.05)
0.7833
131.0
62.1c-3
DPS (C, 0.05)
0.4058
DPS (G. 0.3)
0.7349
150.5
87.6e-3
DPS (G, 0.3)
0.5078
(G 1.2)
0.6056
133.0
86.3e-3
DPS (G: 1.2)
1.4713
DPS (G = 1.8)
0.4137
58.48
14.6e-3
DPS (G 4.8)
5,8568
ControlNet
0.4657
67.38
19.3e-3
Unconditional
0.4026
StableSR 0,2855
29.12
0.9e-3
StableSR
0,3939
3.3 OBSERVATION II: DPS HAS LARGE SCORE MEAN
Our second observation is that the conditional score estimation of DPS has larger non-zero mean. Additionally, when tuning the hyperparameter Ç., we find that larger score mean correlates with better image quality. More specifically, we empirically observe:
EispPs(X,, v)]l
E [Xsg(X,, t)] = [E[so (X,, v)]
(11)
We examine the mean of the score function as any valid score function has zero mean. This zero- mean property is widely adopted in reinforcement learning and gradient estimators (Williams 1992
Mnih & Rezende 2016 Mohamed et al. 2020):
E V x (x))
/ X(X)Vx log p()xxX Vxp(X)dx= V
= ((X)dx= - 0. (12)
present the absolute value of the empirical mean of the score for different methods. Each score is computed at the initial step Xyr for 1000 samples. The results show that the uncon- ditional score and StableSR have score means close to 0.4. For DPS with &, 0.05 and the score mean also close to 0.4, but DPS does not perform well for small Ç. In contrast, for DPS with good practical performance (G: 4.8), the score mean is close 5.8, which is far from zero
In Table 2
0.3,
4

---

Published as a conference paper at ICLR 2025
3.4 OBSERVATION III: DPS HAS Low SAMPLE DIVERSITY
From these observations, we can conclude that DPS is not estimating the conditional score accu- rately. So why does DPS perform well in practice? To shed light this, we additionally show that the sample diversity of DPS is much lower than that of well-trained conditional diffusion models such StableSR. Denote the sample variance of DPS V[X [y] and the sample variance of StableSR as V[X y). we empirically observe:
Vixo DPS Ml V[Xolu]
(13)
We draw k = 50 samples from DPS and StableSR respectively and compute the per-pixel standard deviation. The results shown in Figure and Table 3 We observe that DPS with S. 4.8 has much lower standard deviation than StableSR. In Figure|4 we show that the samples from StableSR contain more visual variations than DPS, which is also observed by Cohen al. (2023)
DPS (g-488)
StableSR
Table 3: The average per-pixel standard deviation of DPS is much lower than that of Sta-
-946
-939
030
bleSR.
-22
Pixel's STD. (k=50)
DPS (G 4.8) StableSR
0.0453
0.3939
Figure 3: The per-pixel standard deviation of DPS is much lower than that of StableSR.
downsampless x8
DPS (g-4.)
StableSR
Figure 4: The image samples from DPS has much lower visual diversity than that of StableSR.
3.5 HYPOTHESIS: DPS ALIGNS MORE CLOSELY TO MAXIMIZING POSTERIOR
With Observation I & II, one can conclude that DPS is not an effective conditional score estimator for image restoration. However, DPS does work well and produces high-quality samples, despite being poor conditional score estimator. Along with Observation III that DPS has low sample diversity, we hypothesize that DPS is closer to another paradigm of image restoration other than posterior sampling: the maximization of posterior (MAP) estimate.
Hypothesis Instead of approximating posterior sampling
X ~ po PX Xx,
(14)
DPS is in fact attempting to maximizing a posterior
X arg max Po X Xr,, v)-
(15)
The MAP hypothesis is the simplest one that can explain all observations:
Observation I & II: Since DPS is not estimating the conditional score, its estimated score might have large error and mean.
5

---

Published as a conference paper at ICLR 2025
Observation III: As MAP estimators produce deterministic samples, the low sample diver- sity of DPS is also explained.
Besides, the MAP hypothesis also explains previous works that are not explainable with conditional score estimation theory, such as why DPS has high score error lower-bound (Yang et al.) 2024)
and why Adam helps DPS (He et al. 2023; Chung et al. 2023) (See Appendix C.I).
4 IMPROVING DPS WITH MAP HYPOTHESIS
4.1 IMPROVEMENT I: EXPLICIT MAP IMPLEMENTATION
In light the MAP hypothesis, we modify DPS to explicitly maximize the posterior. order to achieve this, we first reformulate the MAP hypothesis into a constrained optimization:
Proposition 3. MAP in Eq. 75 is equivalent the following in probability as d oo:
X < arg max log po (y) x)) s.t. X po po((X-1lXx))
(16)
To implement the constrained optimization in Eq. 16 we leverage an observation made in Yang al. (2024). that dimension + 0o, the isotropic Gaussian distribution (=( con- centrates to sphere surface
with radius vdd, and center | Then, can adopt multiple steps of gradient ascent and project the result onto the sphere surface ((po(X, ())) following Menon et al. (2020) and Yang et al. (2024). The detailed implementa- tion is shown in Algorithm[] It has two differences from DPS: 1). We implement the maximization of log po(y) x,-1) with multiple steps of gradient ascent. 2). We project the result onto the sphere surface S (ppo ( X ())) where po po(X (X,- (X,) concentrates around.
Algorithm DPS
Algorithm DMAP
input T. V.
input T, K, f(.)..((CC
(0, for / T' to do
2r N(O0
for T to do
#4-1 ~ po (X
#t-1 ~ po Po( (2t) H4-1 1[X
2t-1 FI-1 (x -il
for j = 1 K
return Zg
J(- = -1 - 4. (B|x -vll
= ft doo
1
return
argmax log P(y|Xr-1) S(pa(Xr-I(xi)
argmax log p(y|Xt-1) S(pa(Xr-((o)) argmax log p(y(Xr-)) S(po(Xr-1(Xo)
Xt-1
X1
#:
Xi-1
[[X
[[X
[[XI[[[i]]
Xi-1
DPS
DSG
DMAP
DDPM step
Gradient step
Projection step
Figure 5: Conceptual illustration of the update procedure of DPS, DSG and DMAP. The green circle S(p(X,-1XX)) is the sphere surface that x(X,-1(X,) concentrates to. The blueline arg max log p(yl X,-1) is the manifold of X that maximizes log X(
As Algorithm 2 a faithful implementation of the MAP in Eq. 15 a Posterior (DMAP). To better understand why DMAP works, Figure 5 we illustrate the update procedure DMAP, and compare it with DPS (Chung et al.; (022a) and DSG (Yang et al. 2024). It is shown that for DPS, the resulting X,-1 might not live on S(po (0po Xx-1 lX.)), and might not
we name Diffusion Maximize

---

Published as a conference paper at ICLR 2025
maximize log (- For DSG, the resulting X₃-1 must live ((po(X(-1|x.)) but might not maximize log pe ((-1), While for DMAP, the resulting X₁-1 must live on (( and can better achieve maximizing log po(y)XX-1) due to multi-step update.
4.2 IMPROVEMENT II: A LIGITT-EIGGIN CONDITIONAL SCORE ESTIMATOR
Furthermore, the MAP hypothesis allows us to improve DPS with a reasonable but not-so- accurate conditional score estimator (CSE). Denote the backward transition distribution of CSE as 4o (Xx y), we can use it as the initialization point for solving the constrained optimization problem (i.e., use estimated qo x y) replace unconditional po (xx,- - (Xx) in Eq. [6
Xx:
In other word, we iteratively optimize the following posterior using multi-step gradient ascent:
X f- arg max log pa (y) X)- with init. X,-1 qo (X._11XX.
(17)
To improve DPS, the approximate posterior qe (o(x,11 y) can be sot-so-accurate. fact, long as the cross-entropy between the approximate posterior and the true posterior is smaller than the cross-entropy between the unconditional distribution and the true posterior, the above optimization problem has a better initialization than Eq. [T]iin expectation:
Proposition 4. Denote the cross entropy H(...), .), then
H(qs (X Po (X 7 (X, ( Po (v (x) > Exo(X. flog Po (u(xx-1))
< H(po (x, v))
(18)
In practice, adopt ControlNet (Zhag et al. 2023) CSE. We surprised find that CSE is very efficient in terms of both data and temporal complexity. Specifically, only takes GPU hours and 100 images to train, which is roughly equivalent to the time of running DPS for 100 images. For DPS evaluated with 1000 images, the additional training time of CSE is around 28s for each image, while the DPS itself costs several minutes. Alternatively, can also use fully synthetic images generated by Stable Diffusion, in which case our CSE does not require any new data.
Table 4: Quantitative Results on ImageNet 512 Dataset. Bold: best. Underline: second best. Our proposed approaches improve DPS significantly.
SRx8
Gaussian Deblur
Non-lineas Deblur
Time(s)
PSNR
LPIPS
FID.
PSNR
LPIPS
FID.
PSNR
LPIPS,
FIDI
Training Free
DPS
162
22.33
0.4137
58.48
23.05
0.4267
59.31
23.34
0,4173
62.10
PSLD
279
22.28
0.4163
59.08
23.06
0.4305
60.73
Fre:DOM
195
22.64
0.3961
52.94
23.33
0.4104
54.26
23.39
0.4023
58.98
ReSample
433
22.78
0.3949
52.18
23.50
0.4076
$3.00
23.47
0.4026
59.24
DSG
23.15
0.3912
51.15
23.70
0.3977
52.01
23.41
0.3861
57.81
DMAP (sume)
172
23.37
0.3770
44.37
24.42
0.3752
44.15
24.41
0,3437
47.65
DMAP (full)
517
23,52
0.3494
39.56
24.86
0.3407
38.33
24.99
0,3135
39,93
Conditional Score Estimator Trained with & GPU Hours
CSE (n=100)
36 28
16.54
0.4841
85.58
17.54
0.4042
58.89
16,20
0.5172
94.68
CSE (n=1000)
35 28
16.68
0.4657
17.18
0.3946
49.50
15.69
0.5028
77.52
CSE (Self-gen)
36 28
17.15
0.4794
82.32
18.78
0.3700
45.21
16.45
0,5213
106.2
DPS + Conditional Score Estimator Thained with
GPU
Houry
DPS
162
22.33
0.4137
58.48
23.05
0.4267
59.31
23.34
0.4173
62.10
(m=100)
186 +
22.98
0.3229
25.20
0.2306
23.39
24.08
0.3351
40.25
DPS CSE (n=1000)
184 +
23.35
0,3142
36.23
25.47
0.2150
18.96
24.17
0.3384
36.90
DPS + CSB (Self- gen)
192 + 28
23.09
0.3082
38.60
25.41
0.2222
20.88
24.01
0,3250
40.64
DMAP + Conditional Score Estimator Trained with GPU
Hours
DMAP (full)
517
23.52
0.3494
06.56
24.86
0.3407
38.33
24.99
0.3135
39.93
DMAP= CSE (n=100)
603 +
23.58
0.3357
39.56
26.17
0.2580
22.66
25.19
0,3048
33.08
DMAP CSE (n=1000)
602 + 28
23.71
0.3254
36.64
26.39
0.2550
20..49
25.27
0,3114
32.97
DMAP CSE (Self-gen)
602 + 28
23.33
0.3107
38.29
26.34
0.2438
21.17
25.19
0.2776
32.04
EXPERIMENTAL RESULTS
5.1
EXPERIMENTAL. SETUP
Dataset & Diffusion Model We utilize the first 1000 images from the ImageNet validation split. All
images resized and center-cropped 5122 pixels. We use Stable Diffusion 2.0 as the base diffu-
7

---

Published as a conference paper at ICLR 2025
sion model and employ 500-step ancestral sampling solver (i.e., DDPM) to align with assump-
tions. There are some works using ancestral sampling solvers (Yu et al. 2023 Yang et al. 2024)
while some other works (Chung et al. 2023 Rout et al. 2023) using PF- ODE solvers (t.e., D DDM).
We discuss the impact of base diffusion models, solvers and detailed setup in Appendix, B.4
Operators & Metries For the operator f-), adopt x downsampling, Gaussian blurring with a kernel size 61 and an intensity 3.0, and non-linear blurring as described by |Chung et al. (202aa)
Following previous works (Chung al. 2022a), we utilize PSNR (Peak Signal-to-Noise Ratio), LPIPS, and FID to evaluate the performance of all methods.
improvements against several DPS algorithms in the latent space. These include DPS (Chung et al. 2022a) implemented in latent space as described in Rout et al. (2024, PSLD (Rout et 2024p, RaSple (Song et al. 2023ap. FreeDOM (Yu al.
Baselines We compare
2023). and
DSG Yang et al. 2024). We acknow ledge the existence of other very competitive works Song et al.,
2023c Rout al. [2023] Chung et al.) 2023] Mardani al. 2023b (Song et al.) 2023b). However,
these approaches are either not open-sourced have not been evaluated in the latent space, and hence, are not included our comparison.
Conditional Score Estimator The Conditional Score Estimator (CSE) utilizes ControlNet neural network and is trained for 5000 steps with batch size of 64, taking approximately 8 hours on one A100 GPU. We train the CSE using various datasets, including subsets of 100 and 1000 images from the ImageNet training set, as well as eelf-generated data from Stable Diffusion 2.0, For detailed information about the training setup and datasets, please refer to Appendix, B.1
DMAP (Improvement
DPS + CSE (μ-100) (Improvement TT)
y
Souree
DPS
SRx8
SR388
Gaussion Debtur
Non-linee -inear Deblur
Figure 6: Qualitative results 512 x 512 ImageNet images. Our proposed approaches improve DPS significantly.
8

---

Published as a conference paper at ICLR 2025
5.2 MAIN RESULTS
Improvement I We evaluate DMAP in two different settings: "same complexity" and "best per- formance", to test DMAP as a practical algorithm and verify our MAP hypothesis. In the "same complexity" setting, set the gradient ascent steps of DMAP K 2 and reduce the diffusion steps by 2. This results DMAP (same), which has almost the same runtime as DPS. In the "best performance" setting, increase the gradient ascent steps of DMAP to K
without reducing the diffusion steps. This results in DMAP (full), which significantly slower than DPS. As shown in Table 4 and Figure T0 12 DMAP (same) outperforms all other methods across most metrics while maintaining almost identical complexity to DPS. Furthermore, DMAP (full) outperforms all other methods by large margin, albeit at the cost of increased complexity. Therefore, we conclude that DMAP is an efficient and practical algorithm, and our MAP hypothesis is reasonable.
Improvement We evaluate our Improvement II across three different dataset settings. Specifi- cally, we train a Conditional Score Estimator (CSE) using 100 and 1000 images from ImageNet, as well as self-generated images from Stable Diffusion 2.0 itself. Each training session takes approxi- mately 8 GPU hours. As shown in Tableeaand Figure 6 10 12 the CSE on its own does not provide substantial results. However, when integrated with DPS and DMAP algorithms, the CSE signif- icantly enhances their performance. Besides, the additional complexity of CSE is also marginal. More specifically, the training cost amortized by 1000 images is 28s, which much smaller than 162s of DPS itself. Overall, DPS CSE is only 25% slower than DPS, including the training time.
(dowasample x8)
Source
CSE (a=100)
CSE (n=1000)
CSE (self-2en)
(n 25)
(n-100)
100000)
DPS
CSE (scll-gen)
Figure 7: Effect of CSE and number of training images. CSE does not work well without DPS.
Table 5: Impact of steps parameter T, K.
Table 6: Impact of dataset size and type.
T
Time(s)
PSNR
LPIPS
FID
of Image
Dataset
PSNR
LPIPS
FID
DPS
100
42
20.50
0.5980
132.8
(DPS)
22.33
0.4137
58.48
DSG
100
42
22.12
0.4739
76.24
25
ImageNet
22.26
0.4050
75.64
DMAP
100
48
22.69
0.4763
80.50
50
ImageNet
22.84
0.3527
55.78
DMAP
50
22.60
0.4591
72.05
100
ImageNet
22.98
0.3229
44.59
DMAP
25
22.65
0.4681
76.63
1000
ImageNet
23.35
0.3142
36.23
100000
ImageNet
23.46
0.3147
35.96
Self-gen
23.09
0.3082
38.60
Table 7: Impact of GPU hours.
# of GPU hours
PSNR
LPIPS
FID
(DPS)
22.33
0.4137
58.48
23.04
0.3716
49.18
4
23.15
0.3310
40.34
23.46
0.3147
35.96

---

Published as a conference paper at ICLR 2025
5.3 ABLATION STUDIES
Improvement I In Table 5 we present an ablation study examine how balancing the diffusion
steps T' and gradient steps K affects the performance of DMAP, under the constraint T x K = 100. The setting of 100 diffusion steps is commonly used in fast DPS (Yang et al. 2024 He et al. 2024). Our findings indicate that for K = 1,2, 4, the PSNR of DMAP does not vary significantly. However, we observe that when K 2, the LPIPS and FID scores are better than those of other settings, and outperform DPS and DSG.
Improvement In Table 6 and Figure 7 we present the impact of dataset size and type on our Improvement II. The results show that CSE trained with only 100 images can significantly enhance the performance of DPS, while a CSE trained with just 25 images fails to do so, likely due to overfitting. Moreover, the benefit of increasing the dataset size becomes minimal once the dataset contains 2 1000 images. Additionally, a CSE trained with self-generated images also leads to notable improvements. In Table Results indicate that a CSE trained for 2 4 hours can significantly enhance the performance of DPS.
present the impact of training time our Improvement II.
RELATED WORK
Recently, zero-shot conditional sampling from diffusion models has drawn great attention. Various
approaches have been developed, including linear projection (Wang et al. 2022 Kawar et al. 2022
Chung al. 2022b Lugmayr al. 2022; Song al.. [022] Dou & Song
2023 Pokle et al.
2024 Cardoso et al. 2024), Monte Carlo sampling ( [Wu et al. 2024; Phillips et al.
2024
Dou & 2024).
Song 2024), and variational inference (Feng et al.) 2023 Mardani et al. [2023] Janati al.
Among these paradigms, Diffusion Posterior Sampling (DPS) and its variants are the most popular
and widely adopted (Chung et al., 2022a Song et al. 2023c Yu et al., 2023 Boys et al.
2023
Rout et al. 2023 Zhang et al. [024] Chung et al. [223] Bansal et al.,
2023 Yang et al.
2024
This popularity is due to the DPS family's 's ability to handle high-resolution images, accommodate
non-linear operators, and operate with reasonable efficiency.
On the other hand, unlike Monte Carlo based approaches (Wu et al. 2024 Dou & Song 2024), DPS is less explainable. Chung et al. (2022a) and [Song et al. 2023c) propose that DPS works as a conditional score estimator, and provide an upper-bound on the estimation error. However, Yang al. (2024) show that the estimation error has a non-trivial lower-bound for high-dimension data. To the hest of our knowledge, we are the first to check the estimation error of DPS as a conditional score estimator in refe-life scenario, and the first argue that DPS is closer maximize a posterior (MAP). Besides the observations in this paper, our MAP assumption also explains previous observa- tions that are contradicted to conditional score estimation assumption, such as why DPS works well despite it has large estimation eтror lower-bound (Yang et al.) (024), why DPS works with Adam (He et al. 2023 Chung et al. 2023), and why DPS has low sample diversity (Cohen et al. 2023).
7 DISCUSSION & CONCLUSION
One limitation of the conditional score estimator used in Improvement II is that it employs Control- Net (Zhang et al. 2023), which is not optimized for sample and computational efficiency. would be particularly interesting if we could improve DPS using just handful of examples and shorter time, extending its applicability from zero-shot to few-shot scenarios.
To conclude, we demonstrate that DPS does not function three key observations. Instead, we hypothesize that DPS is closer to maximizing a posterior dis- tribution. Based on this new hypothesis. we propose two improvements to DPS. Empirical results show that both of our proposed improvements significantly enhance the performance of DPS.
conditional score estimator through
ACKNOWLEDGMENTS
This work is supported by Wuxi Research Institute of Applied Technologies, Tsinghua University under Grant 20242001 120. No Patent or IP Claims.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 843-852, 2023.
URL wttpthttss:aappp. sew.re..cssmcccccccccccccccc htt.://........rooooooooii2i2222
Mikolaj Binkowski, Danica Sutherland, Michal Arbel, and Arthur Gretton. Demystifying mmd gans. ArXiv, 0ess/8001.1101,, 2018.
USID:3531855
Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and Deniz Akyildiz. Tweedic moment projected diffusions for inverse problems. arXiv preprint arXiv:2310.06721, 2023.
Gabriel Cardoso, Yazid Janati el idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided denoising diffusion models for bayesian linear inverse problems. In The Twelfth International Conference on Learning Representations, 2024. URL hnttip:://openreiieeweeee
Hyungjin Chung, Jeongsol Kim, Michael T. McCann, Mare Louis Klasky, and C. Ye. Diffusion posterior sampling for general noisy inverse problems. ArXiv, 146877 2022a. URL
https://api. sema ieschol ar. otrg//orrrss:::5222222
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and J. C. Ye. Improving diffusion models for inverse problems using manifold constraints, ArXiv, abs/2206.0088 009411, 2022b. URL BtLhhtts:: /
manticscholar wrt/CorrpsssD---....o... 244922222222
Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent
diffusion models for inverse problems. ArXiv, abs/2310.011100, 2023. URL https://api.c.
manticscholar
26:3605774
Noa Cohen, Hila Manor, Yuval Bahat, and Tomer Michaeli. From posterior sampling to meaningful diversity in image restoration, ArXiv, abs/2310.160411 2023, URL https://ppp semantic
scholar. http:/aar..crrrooops102235533332
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Teo Deveney, Jan Stanczuk, Lisa Maria Kreusser, Chris Budd, and Carola-Bibian Bibiane Schönlieb. Clos- ing the ode-sde gap score-based diffusion models through the fokker-planck equation, arXiv preprint arXiv:2311 159966, 2023.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. abs/2105.05233, 2021. URL httppss/appp..ennnticccooo.....
ArXiv,
234357997
Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference on Learning Representations, 2023.
Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference on Learning Representations, 2024.
Berthy Feng. Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L Bouman, and William T Freeman. Score-based based diffusion models as principled priors for inverse imaging. arXiv preprint ar344:2304. /751, 2023.
Linchao He, Hongyu Yan, Mengting Luo, Kunming Luo, Wang Wang, Wenchao Du, Hu Chen, Hong ling Yang, and Yan Zhang. Fast and stable diffusion inverse solver with history gradient update. 2023. URL htt.ts:://aatt. http::/aap...maatt... semantics www.ttiischhhhchoocooccoccacceee http:////w.pp...aaaaeeeeeeaaaaaa wtt.:://ww.......ooooooooooaaaaa
Yutong He, Naoki Murata, Chieh-Haee Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji, J Zico Kolter. Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. In The Tweifth International Conference on Learning Representations, 2024. URL chttpps://opprreeeeeee net./ http://eeeeeeeeeeefeooooooooooon

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Neural Information Processing Systems, 2017. https::/: semanticscholaat /rg/Co
rpus : TP2611:326772
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Yazid Janati, Alain Durmus, Eric Moulines, and Jimmy Olsson. Divide-and-conquer posterior sam- pling for denoising diffusion priors. arXiv preprint arXiv:2403. /407, 2024.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. ArXiv, 003644 2022. URL https://api.... swm.nniismmmniiiiimam
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration
models. ArXiv, abs/2201.11793, 2022. URL URL httph:tpp//// /appppap.... www.neicscooollrr........... ott.attttes//pp...eennt.tcccoolo Crrg org/Co
rous ID: 24:441161116
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR,
abs/1412.60 2es//442.6980, 2014. URL https: http:://w...oaaattcccccoarrrrrrr
6628106
Hung-Jin Lin, Sheng-Wei Huang, Shang-Hong Lai, and Chen-Kuo Chiang. Indoor scene layout
estimation from single image. 2018 24th International Conference on Pattern Recognition (ICPR), pp. 842-847, 2018. URL htthht/s:/// petps://api... www.aeticccccooor..oorooooooaooa 54212984
Yaron Lipman, Ricky TQ Chen, Heli Ben Gen-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11451-11461, 2022. URL https:
7/api. wtt.tttcccccoocooooooooooooo ID:
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective solving inverse problems with diffusion models. ArXiv, abs/2305 04391, 2023a. URL https://app.
semanticscho.
http://orpusuuu ID: :5: 258577287
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. variational perspective on solving inverse problems with diffusion models. arXiv preprint 2xXi::2305.0439,,, 2023b.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2434-2442, 2020. URL
ootp/oorootD: 21163416226
Andriy Mnih and Danilo Rezende. Variational inference for monte carlo objectives. In International Conference on Machine Learning, pp. 2188-2196. PMLR, 2016.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient esti- mation machine learning, Journal of Machine Learning Research, 21(132):1-62 2020.
Angus Phillips, Hai-Dang Dau, Michael John Hutchinson, Valentin De Bortoli, George Deligian- nidis, and Arnaud Doucet. Particle denoising diffusion sampler. ArXiv, abs/2402.06320, 2024. URL https: :/ap::://a..... http://ppp..a..a.aa.eeeea semant semanticscholla.
26:6666161111
Ashwini Pokle, Matthew J Muckley, Ricky TQ Chen, and Brian Karrer. Training ffeni lincar image inversion via flows. arXiv preprint arXiv:23 2023.
Ashwini Pokle, Matthew J. Muckley, Ricky T. Q. Chen, and Brian Karrer. Training free linear image inversion via flows, 2024.
12

---

Published as a conference paper at ICLR 2025
Litu Rout, Yujia Chen, Abhishek Kumar, Constantine Caramanis. Sanjay Shakkottai, and Wen- Sheng Chu. Beyond first-order tweedie: Solving inverse problems using latent diffusion. ArXiv, abs/2312.008825 2023. URL
265609906
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkot- tai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265. PMLR, 2015.
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solv- ing inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123, 2023a.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guien diffusion models for inverse problems. In International Conference on Learning Representations, 2022.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse guided diffusion models for inverse problems. In International Conference on Learning Representations, 2023b.
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning, 2023c. URL httpt:://// //api//ppppp semantic scholar, wttp:ooorosS::::200555011
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. International Journal of Computer Vision, pp. 1-21, 2024.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. ArXiv, abs/2212.00490, 2022. URL
org/CorpusID:2200 ID:1ss222225455
Ronald J Williams.
Simple statistical gradient- algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Lingxiao Yang, Shutong Ding, Yifan Cai, Jingyi Yu, Jingya Wang, and Ye Shi. Guidance with spherical gaussian constraint for conditional diffusion. arXiv preprint arXiv:2402.03201 2024.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free
energy-guided conditional diffusion model. ArXiv, 0bs/2303.09833 2023. hRLLht:::://
w.semantt. www.ciccchhhllcccoocc
org /orpp ID: 20D:222629955
Bingliang Zhang, Wenda Chu, Julius Berner, Chenlin Meng, Anima Anandkumar, and Yang Song.
Improving diffusion inverse problem solving with decoupled noise annealing. arXiv preprint arXiv:2407.0152. 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ArXiv, 0ww/23302.0553.. 2023. Utt.tttss:///ag wwwp:essseaaaiiccchheioo
Drc/CorpusID:2266277
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018. URL https:/ //ap:///pp.p semanticschol.
F.org/CorrpuaID:446655
13

---

Published as a conference paper at ICLR 2025
A NOTATIONS AND PROOF OF MAIN RESULTS
A.1 NOTATIONS
In the main text of this paper, we follow the notation of diffusion in pixel space without explicitly emphasizing the existence of the auto-encoder in latent diffusion model. Some previous works (Song et al., 2023a Chung et al. 2023) explicitly emphasize the existence of the auto-encoder, as they address the approximation errors caused by it. However, our paper and some other works Yang et al. 2024) omit the auto-encoder, as it is not the subject of study and is not considered very important.
For completeness, we provide the notation for latent diffusion with the auto-encoder here. More specifically, denote the decoder as (.) and the diffusion state in latent space Zz. Conse- quently, the DPS update can be expressed as:
Pixel diffusion notation: X,-1 = - V. (()o/()) - yll.
(19)
Latent diffusion notation: Z Z ( yll-
All other formulas can be scamlessly transferred into the latent space by replacing (IE[ yll with (D 2ol (.)) - y/l. For instance, we provide the algorithms for DPS and DMAP the latent space as shown in Algorithm 3 and Algorithm respectively.
Algorithm 3: Latent DPS
Algorithm 4: Latent DMAP
input T, M.
input K. V...
2T (0, for do
2T (U.
for 7 to do
21-1 Po(2z
Rt-1 ~ -1 12))), M E[Z,-1121
-t-1 21-1 (V - wll
for j = to do
return (22))
2/-1 = 2-1 & (VD1 vll
2/-1 = ft-1 ddo,
-113 -
return D(zg)
A.2 PROOF OF MAIN RESULTS
Proposition DPS is equivalent DDPM with estimated conditional score:
Var
spps(X(,t,y)) (X(,
Be (Vxx.)((( yll.
(9)
Proof. Take Eq. into the DDPM update in Eq. 2 have
Pe(Xi-1 = N
(X, + (ss
Va
= =N( Va
(X, + (X,, of G.
vll.
(20)
which is equivalent to the DDPM sampling followed by a DPS update.
Proposition 3. MAP in Eq. 15 is equivalent to the following in probability as - oo:
X < arg max log po X,-1), S.I. X,-1 ~ po(Xx-1lX.)..
(16)
Proof. We can rewrite the optimization target in Eq. 15 as
arg max log po (X Xe, v) arg max log po (Xt-1, X,, Xx)p///
arg max log (X v/,))
(21)
arg max log (y X
(22)
14

---

Published as a conference paper at ICLR 2025
where (b) holds since po(y is fixed and (a)(c) hold due to the Bayesian rule. As y Xo
X1-1 - -X, forms a Markov Chain, we have
Po Xx) - -
(23)
Then we have
arg max log X Xx,1 = arg max Po (ul X (X,1(()))
arg max log po (w/ X )po X Xx))
(24)
Following Theorem 3.1.1 Cover 1999). we know that as the dimension d + the log likelihood
of the isotropic Gaussian distribution p(X,, - converge to a constant in probability:
lo ( & d In (n(2Tea²)
(25)
Therefore Vz (o(X)-1(Xx), the likelihood is constant in the limit. Then, the optimization target can be rewritten into
arg max mo (y(X s.t.X,-1 ~ (o(()-11X))).
(26)
This completes the proof.
Proposition 4. Denote the cross entropy H(...). then
(Xo-1xXl) pe Pe 3)) < (X)) v))
1 Pogglloopooooooo (y) X+-) >
(18)
Proof. As XL-1- X, forms Markov chain, we have
po (0
Xx, 1 = po
(27)
Then we can rewrite cross entropy as
Po ( < H(po (X v)) + (X (log po (y/ x1) + log (X x po (yl Xx-1) + log (X
(28)
As cross entropy is always larger than entropy have
(ooerr 1999), for any distribution # (X
E, log po (x- Eps (x, log (X-1
F log pe (x)) < Ep XX
(29)
Taking Eq. 29 into Eq. 28 we have our result
E Exo
Epa(X, (x)
(30)
A.3 ADDITIONAL THEORETICAL RESULTS
For now, we can not derive MAP directly from DPS. However, with additional assumptions, it is possible to derive MAP directly from variant of DPS: DSG (Yang al. 2024). More specifically, DSG improves DPS with a spherical projection. The update of DSG is as follows:
Vx, (EE y|
X E[X.-1|X,) Viiar V (I
(31)
Assumption 5. We need some additional assumptions:
1. The posterior mean E[X,|X,]] is perfect approximation to posterior samples from ((.o)..)) In that case, the approximation error Eq. diminishes, i.e.:
po(y)X.) p(y) X. E()o exp (Eloo vll-
(32)
15

---

Published as a conference paper at ICLR 2025
2. The of in Eq. is small that log p xx-1) is locally linear in range of vno, The Xt, X1-1 is so close that Vx.-, log ply x ss Vx, log p(y)
Under those assumptions, we can show that DSG is also MAP: Proposition
The DSG solves the MAP problem X + arg max po (Xt-1 (Xt,1)Xt,1) in Hypothe- sis when dimension n is high.
Proof. We first consider a gradient ascent with learning rate dz::
x = [[X + az Vx, logp(y)
As we have assumed in Assumption Vx.-, x() = V log p(y(x)),, As gradient is the direct of steepest descent, when changes, the X E[ X + ag Vx, v, log p(y] is the maximizer of log p(y) (X,)), given the distance from the point X;-1 to center E[X,-1 Xx] is fixed. On the other hand, the DSG can be
the logp(y)X,,) is locally linear and
written as the above gradient ascent
V x. (f(x v/1
X E[X
not
(33)
x. If ylll
Vni
- E| |
Vx, log p())())).).
(34)
Vx. (E(X - xlll
where (a) holds due to Assumption gradient ascent. Further, from the update procedure DSG, the distance of X,-1 [[] is Vno, Therefore, the result of DSG is the minimizer of Vx, ,p(u) Xt-1), on the sphere surface centred at E[X,-1|XX Xx] with radius vnay
We further notice that DSG is equivalent to the above
X 1 + arg max loyp(y XI_1), E S(B(X,-1|X.), VTia.),
where [X,]), v
is the sphere surface centred at E[X;-1[X,)] with radius v no,, Be- sides, from (Yang al., 2024), we know that dimension
co, X p(X,-1 X,) is equivalent to X,-1 E ([X(Xx /no,). Then as dimension n + co, DSG is equivalent to:
X,-1 f- arg max log (())) X (x))
which is again equivalent to Hypothesis by Proposition
Despite DSG being already MAP under Assumption our method remains to be MAP without it. Therefore, DSG alone can not replace DMAP, and its performance is inferior to DMAP as shown in Table
5 1, the original DPS is equivalent to classifier-based guidance DDhaiiwll & Nichol 2021). What is classifier-bass guidance on earth theoretically remains an open problem. For now, we leave the strict theoretical relationship between original DPS and MAP to future works. Once this problem is solved, the nature classifier-bis based guidance can be revealed. We believe this problem is beyond the scope of paper focused on DPS.
With Assumption
We can think DPS as a simplification of DMAP, with one step gradient ascent and without spherical projection. As long as the step size of diffusion model is small, the log p( X,,,) i) can be locally linear (Assumption 2).
that case, one step of gradient ascent enough. As long as DPS guidance step is much smaller than the DDPM step, the distance from X,-1 to the sphere surface ((E[X,-1|X) / v is small. In that case, spherical projection is no longer necessary. And when those conditions approximately satisfied, DPS also works well.
A.4 A TOY EXAMPLE
To better understand the relationship between DPS and MAP, we provide a toy example:
The source distribution p(X)) is 2D 2-GMM (Gaussian Mixture Model). The centres are (-1,-1) (11,+1) and the diagonal do 0.3.
16

---

Published as a conference paper at ICLR 2025
The operator f is inpainting. More specifically, the second dimension of the random vari- able is set to O, the measurement = (0.5, 0).
The diffusion is 100 steps Karras diffusion (Karras al. 2022) with max 4 and p = 7.
The diffusion scheduler is Euler Ancestral, and the DPS C. 0.05.
In Figure 8 it is shown that the true posterior is two mode distribution. However, the samples of DPS cover only one mode of true posterior. In that example, the behaviour of DPS closer to MAP.
prior p(XO)
posterior p(XO
DPS samples
2.0
2.0
2.0
1.5
0.5
0.5
0.0
0.0
0.0
0.5
005
005
GMM center
sample X observation
-2.0
-2.
-2.
-2
-2
Figure 8: toy example with 2D 2-GMM and inpainting operator. The true posterior a modal distribution, while the samples of DPS concentrate to one mode.
B ADDITIONAL EXPERIMENTS
B.1 ADDITIONAL EXPERIMENTAL SETUP
All experiments were conducted on a computer equipped with an Nvidia A100 GPU, CUDA 12.1, and PyTorch 2.0. For the base diffusion model, we used the official Stable Diffussio 2.0 base model. The checkpoint for this model can be found at htttps://hugginnffaac. cо st abilityall
The score emrr plot in Figure. is computed using the image Figure. 2 val 00000013, png of ImageNet dataset. Similarly, the pixel variance in Table. using the image in Figure. 3 which is L.SVRC2012 1elll2_vaa__0000001 png of ImageNet cataset.
which is ILSVRC2012
B.2 ADDITIONAL. DETAILS ON CSE
For the conditional score estimator (CSE), we followed the diffusers'' implementatiion of ControlNet (Zhang et al. 2023). This implementation is available at httttss://ithhbbb.m
ng face/ fflffuueee Fusersss /bl 1lblll. ob/ ma n e ammp les cont wotootooonnt/ttaninccoom wtwprrraannnoooroolloeoooooooooe net. py Specifically, we used Adam (Kingma Ba 20144) optimizer with batch size of 64 and learning rate of x 10-6, We trained the ControlNet with an e-prediction target for 5000 steps, which took approximately 8 hours on single Nvidia A 100 GPU. Data augmentation techniques used include random resizing with a scale range of 0.8 to 2.0, random horizontal flipping with a probability of 0.5, and random color jittering with a strength of 0.1.
B.3 ADDITIONAL DETAILS ON BASELINES AND HYPER-PARE PARAMETERS
PSLD (Rout et al. 2024 includes additional gluing term the original DPS for better perfor- mance in the latent space. Specifically, denoting the encoder of latent diffusion as E(.) and the decoder D.), the PSLD update is given by:
Z 2-1 GVZ, (E - vll.
(35)
74 2. E ZZ 8(T Z Z ((((E(Z//))))))
17

---

Published as a conference paper at ICLR 2025
Table 8: Detailed hlyper-paammeee for different methods and operators.
SR
Gaussian Deblur, lineur Deblur
DPS
T 500, G. 4.8
T 500, G 0.6
PSLD
= 50D, G 1.8, "v 0.1
= 500, 9 D.6,
0.1
FreeDOM
T= 500, = 4.8, K [(,0)
100, 250
T 500, Ç 0.6, K [(,
100, 250
ReSample
T 500, 4.8, K 200, N 50, [2)
100, 250
T 500, 4 0.6, K 200, N 50, 01.
100, 250
DSG
= 500, 0.08
= 500, 0.02
DMAP (same)
250, K 2. 9.6
250, K 2.
DMAP (full)
500, K 3.
500, K 3,
DPS CSF
Same as DPS Same DMAP
DMAP CSE
where (*) is the transpose of the operator f()) Therefore, PSLD is applicable only to linear operators. Table we present how the parameters Çe and " are chosen for different tasks.
FrecDOM Yuu et al. 2023) enhances DPS by incorporating additional step where the algorithm moves back and forth along the backward Markov chain. This approach is termed efficient time- travel. Specifically, for tort E [c1, C2], FreeDOM moves back to time + 1 for K iterations by adding
noise back.
= - N
(36)
In this way, FreeDOM effectively performs K steps of gradient ascent. Given small range for [ci, C2], FreeDOM does not significantly increase computational complexity. In Table g we present how the parameters Çe, K, C1, and 02 are chosen for different tasks.
ReSample (Song et al., 2023a) builds upon the efficient time-travel framework of FreeDOM. In addition, it further optimizes the image or latent representation directly and projects them back by adding noise. Specifically, for every N steps, ReSample solves the optimization problem by performing K steps of gradient ascent the pixel space or latent space:
Pixel space optimization: Z" E(arg min ;Ilu f(x)1),
(37)
1 Latent space optimization: Z" arg min
7 (DD()))
with initialization by Tweedie's formula:
Pixel space initialization: X D D (x))
(38)
Latent space initialization: Z [X]]
After this update, ReSample projects z" back by adding noise back with hyper-parameter np:
2 77c 0 EE
Zn Z,| + (1 Zz) + N(O, The (1 (1-a,) -0)-)
n (1 (1-1a))
(39)
1) + (1 (x.)
In practice, ReSample does not significantly increase the complexity of DPS, thanks the skipping parameter N. In Table we present how the parameters Ç., K, and N are chosen for different tasks.
DSG (Yang et al., 2024) identifies that the distribution p(Xx,1|XXa) concentrates on the surface of sphere. Therefore, DSG proposes to project the result of DPS onto this sphere. DSG does not perform the DPS update directly. Instead, it computes the scaled norm of the update:
V Vx,ll (E (X,)) yl u" -ooo V (E[[o yylll
(40)
so that the norm of u² matches the norm of the DDPM noise (/ o?I). Then, DSG performs the update by controlling the strength parameter G::
-0, 01 nT' = - + Ge(u" -
(41)
m
X E(Xx-1XX + Vda,
1
18

---

Published as a conference paper at ICLR 2025
DSG does not increase the complexity of DPS. In Table 8 we present how the parameters Ç chosen for different tasks.
DMAP For DMAP with similar complexity to DPS, set K 2 and reduce the number of diffusion steps T by half, ensuring that DMAP (same) has comparable complexity to DPS. Conse- quently, double to compensate for the reduced number of diffusion steps. For DMAP with full performance, we set K while keeping other parameters the same as in DPS.
Methods with CSE For methods that used with the Conditional Score Estimator (CSE), keep the parameters the same as those used in the methods without CSE.
B.4 ABLATION STUDY ON DIFFUSION MODELS AND SOLVERS
Table 9: Ablation study on base diffusion model and solvers.
Diffusion
Solver
PSNR
LPIPS
FID
Stable Diffusion 2.0
Ancestral Sampling (DDPM)
22.33
0.4137
58.48
Stable Diffusion 2.0
PF-ODE (DDIM)
22.08
0,4229
63.82
Stable Diffusion 1.5
Ancestral Sampling (DDPM)
22.59
0,4074
69.74
Stable Diffusion 1.5 vs. Stable Diffusion 2.0 Several previous works (Song et al., 2023a Rout
et al., (224)) have adopted Stable Diffusion 1.5 as the base model. Other works have not specified the version of Stable Diffusion used (Chung et al. 2023 Rout al.) 2023). To better understand the effect of the Stable Diffusion version, we compare the performance of Stable Diffusion 1.5 and
Stable Diffusion 2.0. In our comparison, the operator is downsampling x8, the
are T 500 and Ç 4.8, and the algorithm chosen is simply DPS.
(downsample x8)
Source
DPS with SD1.5
DPS with SD2.0
Figure 9: Visual comparison of DPS reconstruction with Stable Diffusion 1.5 and 2.0.
In Table it is shown that adopting Stable Diffusion 2.0 improves the performance of DPS signifi- cantly in terms of FID, while causing slight performance degradation in terms of PSNR and I.PPS. Furthermore, as shown in Figure / Stablc Diffusion 2.0 looks much better visually. We have not included Stable Diffusion 3.0 in the comparison because is now-matching model (Lipman al., 2022) rather than VP/VE diffusion. For such models, flow-matching specific inversion methods, such as those proposed by Pokle et al. (2023), might be more appropriate.
Ancestral Sampling vs. PF-ODE The original DPS (Chung et al. 2022a) for pixel diffusion adopts ancestral sampling solvers such as DDPM. Some later works on latent diffusion adopt Probability Flow Ordinary Differential Equation (PF-ODE) solvers, such as DDIM (Song et al. 2023a Chung et al. 2023, Rout al. 2023). while other works on latent diffusion continue to use ancestral sampling solvers (Yu et al. 2023 Yang al. 2024). To better understand the effect of solvers, compared the performance of ancestral sampling and PF-ODE solvers. For this comparison, the base diffusion model is Stable Diffusion 2.0, the operator is downsampling by factor of 8, the hyper-pane param-parameters are = 500 and Ce = 4.8, and the chosen algorithm is simply DPS.
In Table 9 it is shown that adopting ancestral sampling marginally improves the performance of DPS. This observation aligns with previous findings that ancestral sampling outperforms PF-ODE when the number of diffusion steps is relatively large (Deveney et al. 2023).
19

---

Published as a conference paper at ICLR 2025
B.5 ADDITIONAL QUALITATIVE RESULTS
In Figure 10
we present additional qualitative results for different methods.
B.6 ADDITIONAL QUANTITATIVE RESULTS
In additional super- reseurresolttoon and deblurring, we present experimental results for another two operators: image inpainting and segmentation. For inpainting, we choose fix box of size 128 x 128 centered at the image. For segmentation, (2018). For inpainting. we adopt first 100 images of ImageNet. For segmentation, we adopt first 100 images of LSUN bedroom. For segmentation, image restoration metries are no longer meaningful. Therefore, we use another set of metries, including MIoU (Mean Intersection over Union) and FID. In Table I0 we show that for inpaining and segmentation, our proposed approaches (DMAP and DPS+CSE) also improve DPS significantly.
choose bedroom segmentation model by [in et al.
Table 10: Additional quantitative results for inpainting and segmentation.
Inpainting
Bedroom Segmentation
Time(s)
PSNRT
LPIPS..
FID
MloUt
FID,
173
23.72
0.3189
115.4
0.3358
116.7
PSLD
230
23.92
0.3279
117.6
DSG
23.73
0.3395
112.9
0.3790
114.7
DMAP (same)
187
24.50
0.2958
111.8
0.3883
102.7
DMAP (full)
484
24.68
0,2828
106.3
0.4188
104.9
(n=100)
189 + 28
23.98
0,2618
74.7
0.3832
110.3
DPS CSE (Self-gen)
189 28
24.03
0,2523
76.4
0.4159
107.8
C ADDITIONAL DISCUSSIONS
C.I ADDITIONAL EXPLANATIONS TO PREVIOUS WORKS
Besides the three observations, our MAP hypothesis can also explain previous works that are hard to explain using conditional score estimation assumption:
He et (2023) and Chhng et al. (203)) observe that Adam (Kingma Ba 2014) helps DPS, which is not reasonable if DPS is conditional score estimator, because Adam is an adaptive learning rate optimizer. In other words, Adam scales the score Vx, log P(y unevenly in different dimensions. This scaling, theoretically, makes the estimation of Vx, log logp(y) X,) inaccurate, and should deteriorate the performance of DPS. However, we assume DPS is solving the MAP in Eq. [6]] then Adam, general method of gradient ascent, may not harm the performance DPS.
Yang et al. (2024) prove that the conditional score estimation of DPS has a large approxima- tion lowerbound, which increase data dimension increase. If we follow the conditional score estimation assumption, DPS will not work at all for high dimension data such as images.
Cohen et al. (2023) observe that the reconstruction of DPS lacks perceptually meaningful variants, and propose greedy sampling technique to improve DPS's 's sample diversity. This visual observation is quantified in our Observation III.
ETHICS STATEMENT
The approach proposed in this paper allows for conditional generation without the need to train new model. This not only saves the energy that would typically be used to train a conditional generative diffusion model but also decreases the related carbon emissions. Nevertheless, the possible negative effects are similar to those of other conditional generative models, including issues related to trust and the ethical considerations of producing fake images.
20

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For theoretical results, the proofs for all theorems are presented in Appendix For the experiments, we used four publicly accessible datasets. Additional implementation details are provided in Ap- pendix B.I Moreover, we include the source code for reproducing the experimental results in the supplementary material.
21

---

Published as a conference paper at ICLR 2025
y (downsample x8)
Source
DPS
PSLD
FreeDOM
ReSample
DSG
DMAP (same)
DMAP (full)
CSE (n 100)
DPS + CSE (n 100)
DPS + CSE (n 1000)
Figure 10: Additional qualitative results for SRx8.
22

---

Published as a conference paper at ICLR 2025
y (Gaussian Blur)
Source
DPS
PSLD
FrecDOM
ReSample
DSG
DMAP (same)
DMAP (full)
CSE (n- 100)
DPS CSE (n 100)
DPS CSE (n 1000)
Figure 11: Additional qualitative results for Gaussian deblurring.
23

---

Published as a conference paper at ICLR 2025
(Non-linear Blur)
Souree
DPS
PSLD
FreeDOM
ReSample
Not Not Applicable
DSG
DMAP (same)
DMAP (full)
CSE (n=100)
DPS + CSE (n= 100)
DPS + CSE (n= 1000)
Figure 12: Additional qualitative results for non con-irnear deblurring.
24


Reviews:
[{"review_id": "jELrn0Ek0i", "reviewer": "Reviewer_7J3w", "summary": "This paper points out the ineffectiveness of Diffusion Posterior Sampling(DPS), revealing that DPS aligns more closely with maximum a posteriori(MAP) estimation rather than conditional score estimation. This paper demonstrates that DPS’s score estimation diverges from an accurate conditional model and produces low-diversity samples. The authors enhance DPS by explicitly maximizing the posterior with gradient ascent and incorporating a lightweight conditional score estimator.", "questions": "When distribution is multi-modal, is it possible to say that working as MAP estimate is the main reason for low-diversity generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well organized. I enjoy reading this paper. \n- This work suggests MAP hypothesis that can explain the recent mysterious observations regarding DPS. I think this is clear and novel contribution.\n- Significant performance gain by being more faithful to MAP estimate well supports their hypothesis.\n-", "weaknesses": "- Although I think insights from this paper is beneficial, this paper heavily relies on the empirical observations to develop and support their claims. It is okay, but theoretical insights will be strengthen their claim and more naturally lead to their MAP hypothesis.\n- This work is limited to show that MAP hypothesis can explain the weird part of DPS rather than clarifying the reason why it works in MAP estimate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ziGPBCBBaB", "reviewer": "Reviewer_omzH", "summary": "This paper challenges the prevailing view that Diffusion Posterior Sampling (DPS) works by approximating conditional score, hypothesizing that DPS is closer to maximizing a posterior (MAP) and proposes two improvements for DPS based on the new hypothesis.", "questions": "* To further investigate the hypothesis that DPS is closer to MAP than conditional score estimation, one suggestion is to experiment with toy distributions where the ground truth posterior is known. This approach would allow for a direct numerical comparison between the results of DPS and the true posterior.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper presents compelling evidence through empirical analysis on 512x512 ImageNet images that DPS does not effectively approximate conditional score, contradicting prior understanding.\n* The proposed improvements, explicit MAP implementation (DMAP) and a light-weighted conditional score estimator, are shown to significantly enhance the performance of DPS.\n* The paper is well-organized and provides clear explanations of the technical details, including the mathematical formulations and algorithms.", "weaknesses": "* The paper mainly focuses on image restoration tasks and the generalizability of the MAP hypothesis to other inverse problems is not explored.\n* While the light-weighted conditional score estimator is efficient, its performance still lags behind well-trained conditional diffusion models like StableSR.\n* The authors claim that the MAP hypothesis can explain why Adam helps DPS, but this claim is not sufficiently substantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "alPjKbKqQn", "reviewer": "Reviewer_3ddK", "summary": "The authors challenge the traditional view that DPS functions as a conditional score estimator, arguing instead that it aligns more closely with MAP. Through empirical analysis, they find that DPS exhibits significant score estimation errors, a high score mean, and low sample diversity, supporting this reinterpretation. To improve DPS, they propose explicitly maximizing the posterior using gradient ascent and introduce a lightweight conditional score estimator. These methods demonstrate improved results in terms of image quality, particularly for deblurring tasks.", "questions": "Is DPS not used for in painting as well? You seem to only provide results for deblurring.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper has the potential for significant impact as this is a popular field at the moment.\n2. It introduces new ideas and reinterprets existing arguments, supported by data that validates this new perspective.\n3. Building on their reinterpretation, the authors develop a new method that outperforms traditional DPS.", "weaknesses": "1. The paper has several grammatical flaws:\n   - Captions for tables should be placed on top, as I believe this is required formatting by ICLR.\n   - Add an arrow to indicate that lower FID is better, and do the same for other metrics.\n   - There are many undefined acronyms, such as FID, KID, and LPIPS.\n   - The captions for figures and tables could be improved for clarity, adding the takeaway from each table and image as in the text would benefit the reader.\n   - In line 196, the notation for different math symbols is redundant with what is already explained on line 153.\n   - The claim \"With Observation I & II, one can conclude that DPS is far away from a reasonable conditional score estimator in practical scenarios\" is quite extreme, especially since it is shown on just one problem.\n   - In line 255, \"we hypothesis\" should be corrected to \"we hypothesize.\"\n   - In line 273, \"(See Appendix C.1)\" should not be placed after a sentence.\n   - The first quotation marks throughout the paper are incorrect; in LaTeX, use `` for the opening quotation mark.\n   - Citations like \"Chung et al. (2022a); Song et al. (2023c)\" are not grammatically correct and appear throughout the paper. Correct the citation style to improve readability.\n\n2. There are no results for inpainting, unlike the original DPS paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)