Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems
Authors: Jiawei Zhang
Jiaxin Zhuang
Cheng Jin
Gen Li
Yuantao Gu
Venue: NeurIPS 2024 poster (2024)
Keywords: Inverse problem
diffusion model
optimization
generative modeling
Primary area: diffusion_based_models

Abstract: The recent emergence of diffusion models has significantly advanced the precision of learnable priors, presenting innovative avenues for addressing inverse problems. Previous works have endeavored to integrate diffusion priors into the maximum a posteriori estimation (MAP) framework and design optimization methods to solve the inverse problem. However, prevailing optimization-based rithms primarily exploit the prior information within the diffusion models while neglecting their denoising capability. To bridge this gap, this work leverages the diffusion process to reframe noisy inverse problems as a two-variable constrained optimization task by introducing an auxiliary optimization variable that represents a 'noisy' sample at an equivalent denoising step. The projection gradient descent method is efficiently utilized to solve the corresponding optimization problem by truncating the gradient through the $\mu$-predictor. The proposed algorithm, termed ProjDiff, effectively harnesses the prior information and the denoising capability of a pre-trained diffusion model within the optimization framework. Extensive experiments on the image restoration tasks and source separation and partial generation tasks demonstrate that ProjDiff exhibits superior performance across various linear and nonlinear inverse problems, highlighting its potential for practical applications. Code is available at https://github.com/weigerzan/ProjDiff/.

Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems
Jiawei Zhang
Jiaxin Zhuang Tsinghua University
Tsinghua University
tsinghua.edus edw.auuaeddd...m
zhuangjx230mailss tsinghua. edu. en
Cheng Jin
Gen Li
Yuantao Gu Tsinghua University gyt@tsinghua edu, cn
Tsinghua University
CUHK
j jinc21 caaals..mll..... tsinghua.edde edu. cn
genli@cuhk. edu. hk
Abstract
The recent emergence of diffusion models has significantly advanced the precision learnable priors, presenting innovative avenues for addressing inverse problems. Previous works have endeavored to integrate diffusion priors into the maximum posterioni estimation (MAP) framework and design optimization methods to solve the inverse problem. However, prevailing optimization-basse exploit the prior information within the diffusion models while neglecting their denoising capability. To bridge this gap, this work leverages the diffusion process to reframe noisy inverse problems as a two-variable constrained optimization auxiliary optimization variable that represents 'noisy'
rithms primarily
task by introducing sample at an equivalent denoising step. The projection gradient descent method is efficiently utilized solve the corresponding optimization problem by truncating the gradient through the j-predictor. The proposed algorithm, termed ProjDiff, effectively harnesses the prior information and the denoising capability of pre- trained diffusion model within the optimization framework. Extensive experiments on the image restoration tasks and source separation and partial generation tasks demonstrate that ProjDiff exhibits superior performance across various linear and nonlinear inverse problems, highlighting potential for practical applications. wtt.//oooo/c/eeeee Code is available at http://wwww....e/////////////ppp
1 Introduction
Denoising diffusion models have achieved tremendous success the field of generative modeling [1-5]. Their remarkable ability to capture data priors provides promising avenues for solving inverse problems [6], which are widely exploited image restoration [7-12], medical image processing [13, 14), 3D vision [15], audio processing [16, 17] and beyond.
Numerous endeavors have sought to harness diffusion models to address inverse problems [18- 27J. Since the sampling process of diffusion models is a reverse Markov chain, most approaches attempt integrate the guidance provided by the observation equation into the sampling chain. For instance, DDRM [20] achieves favorable results for linear inverse problems with low complexity by introducing a new variational distribution. DDNM [21] capitalizes on the concept of null- range decomposition. effectively rectifying the saage component of the intermediate steps by
Jiawei Zhang, Jiaxin Zhuang, Cheng Jin, and Yuantao Gu with the Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China. Gen Li is with the Department of Statistics, The Chinese University of Hong Kong, Hong Kong, China. The corresponding author is Yuantao Gu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Observation function:
Diffusson
y'= A You p a..
TT-1
Optimization problem:
Inverse
L(xo, xx,) s.t. y Ax
Observation function
Constrained optimization
Problem
Update
Of
(%cr
Pta
Kg
Update
Process
V. log
Projection
y'
Ax
PDF-contox
Fa log ((6l))))
PDF-conlour
Figure 1: Framework of ProjDiff. We introduce an auxiliary variable x,, and transform the inverse problem into two-variabe wariable constrained optimization problem which can be solved using the projec- tion gradient method.
leveraging the observation information. DPS [23] guides the generation process using the gradients of intermediate steps with respect to the observation error Moreover, methods based on Monte Carlo Particle Filtering [25-27] are employed approximate the posterior.
Since inverse problems are often modeled as maximum a posteriori (MAP) estimations, recent efforts have been made to explicitly integrate diffusion models the prior term in the optimization framework [28, 24]. [28] proposed to employ the Evidence Lower Bound (ELBO) to approximate the prior, facilitating the application of diffusion models in inverse problems in plug-anb foug-and-plyy manner. [24] achieved analogous results from a variational perspective. In these optimization- based methods, due the presence of the observation noise, common practice is to adopt Gaussian likelihood. However, it's worth noting that, since diffusion models are inherently effective denoising, considering the observation noise in the likelihood term fails fully leverage diffusion models' denoising capability.
To fully utilize both the prior information and the denoising capability inherent in diffusion models within the optimization framework, we introduce an auxiliary optimization variable to accommodate the influence of the observation noise. We derive novel two-variable objective based the properties of the diffusion process and transform the inverse problem into a constrained optimization practical approximation of the stochastic gradient
task. Through gradient truncation, we obtain of the objective which sidesteps significant computational overhead. The proposed algorithm, termed Proj tackles the inverse problem by employing the concept of projection gradient descent to solve the corresponding optimization task. We also discuss the noise-free version of ProjDiff as special case, which, compared other intimization-based methods, ensures superior consistency between the generated results and observations. ProjDiff's applicability also extends beyond linear observations encompass nonlinear functions, which enhances its competitiveness and versatility.
We demonstrate the outstanding performance ProjDiff through comprehensive experiments on various benchmarks. In both linear and nonlinear image restoration tasks, ProjDiff exhibites superior performance among existing ofte-of-the-arr (SOTA) algorithms. In the music separation task, ProjDiff shows for the first time, the best of our knowledge, that diffusion-basee based separation algorithm can surpass previous SOTA, which further demonstrates the powerful potential of diffusion models and provides insights better harness their capabilities in inverse problems for future research.
Backgrounds
2.1 Denoising diffusion models
Denoising diffusion models [1-3] are class of latent variable generative models tailored to capture targeted data distribution gxx). Diffusion models typically predefine forward process characterized Markov chain, wherein the transition probability is stipulated as Gaussian distribution:

---

T ((xO:T) = ((xo) II (xx (
(I)
where a and by represent the scale and variance parameters, respectively. The sampling process of diffusion models aims to invert the forward chain via a parameterized reverse process, which is also modeled as a Markov chain with learnable parameter 0:
T
(Xo.T) (xT) II (x, (xx;), Pa Pa(x;-1 = (xx oi),
(2)
where fo denotes the mean function and the variance of predefined. In this work, we primarily focus on the Variance Preservation (VP) diffusion [19]', characterized by parameters
/ by =1
with ag = and IT= Ct s2 O. With these parameters, we have (x,, xo) N(xx;; / Vaxx ae) II) where a2 = II i= dg. The initial distribution of the reverse process is set to match the forward process, i.e., (x()) NO, I). while the mean function is chosen as
fx () vā- + V 0 of X1 (OL
(3)
a4
Here Po serves as an estimation of X₂ time typically implemented through a parameterized neural network referred to as "u-ppredictor" The variance of E [0, 1 [c-1] can take various
values. When
Tr the reverse process aligns with DDPM [2]; while when प.
retaining o as an adjustable parameter, one obtains DDIM [3]. We refer to [19, 29-31] for more theoretical guarantees of diffusion models.
2.2 Diffusion models as data prior
As stated in [32, 28], for well-trained diffusion model, the Evidence Lower Bound (ELBO) can effectively approximate the log- loglliielihood of the samples, i.e.
log (xx) N C Σ A(1)E Xo (x, 1)l1?
where X(t) represents the coefficient within the ELBO and C' is a constant. Utilizing the ELBO as the log-prior term enables diffusion models to serve as a plug-and-play prior for general inverse problems. [28] handled the ELBO via stochastic gradient descent and the reparameterization method.
By sampling 1 (1, .,, T} and e ~ (O, I), a stochastic gradient the ELBO can be derived as
(5) where Dx, (-) denotes the Jocabian of the μ-predictor. [24] derived similar proxy objective from the perspective of variational inference, and proposed a more computationally efficient approximation of the stochastic gradient by reweighting the objective. Notably, when the variational distribution is a delta function, variational inference aligns with the MAP estimation. In this work, we introduce novel two-variable ELBO by constructing an auxiliary variable that accounts for the observation noise, thereby utilizing both the prior information and the denoising capability in diffusion models simultaneously.
dr = (Dx, Me (VTxx + -I- ai, t) - 1) (xo ue + vI t)e,t))
2.3 Equivalent noisy samples
As stated in [26], the noisy observation of a sample can be regarded as a noise-free observation of an equivalent noisy sample certain steps in the diffusion process. For linear observations y Axp where E IR' y E Rmy,, and n E NO, I, common practice [20, 21, 26] involves the Singular Value Decomposition (SVD) to attain a decoupled form. Without loss of generality, assume A has full row rank. Utilizing the SVD A = USV? yields UTy S(VT xo) + on, thus
This work is also effective for Variance Exploding (VE) diffusion. defer discussion to Appendix B. Thh case of e-predictor is similar, with the conversion given by (x((x:) (xe vI axco(x:)) /Vo..
3

---

transforming the observation equation to y = SXy + on, where y Y and Xo V'x,, Xo, which can be further written element-wise as
y1 X0, ni: 1 S myy
(6)
8;
8;
where X and y, represent the ith component of x. and y, respectively, and s; denotes the ith singular value A. Assuming there exist 4 E 21, ... 7), 1 S S my such that as, 1/(1 (a//s))²) let y Vor,,y and then the observation function can be rewritten
y Xo, +
Ct n,.
(7)
va,, Xo.i + /1 Cz., n, can interpreted as the ith component of sample at time step 1,, namely Xt,5. Thus [26] proposed to employ Monte Carlo Particle Filtering to restore the samples up to the equivalent noise level, and then apply the backward transition to map these noisy samples back the clean samples. In this work, we introduce this equivalent noisy sample as auxiliary variable, thus better handling the observation noise in the optimization framework.
Method
In this section, we introduce the ProjDiff algorithm. The main idea behind Proj lies in constructing an auxiliary variable to align the noisy observations to specific steps of the diffusion process, thereby forming two-variable constrained optimization task, which can be tackled by approximating the stochastic gradients and employing the projection gradient descent method. We begin by deriving the Pifj algorithm for noisy linear observations and then discuss its noise-free version special case and the extension of ProjDiff to accommodate nonlinear observations.
3.1 ProjDiff
Consider the Gaussian observation equation y Axo + on. The goal of the inverse problem is to recover the original data Xg given y, A, and σ. The decoupled observation equation in (7) indicates that one apply SVD to reduce a linear observation into the simple form with A [my 0] in the spectral domain, which implics the observation y represents the first my components of x. Consequently, we focus this simplest observation function declare algorithm. Under this condition, assuming there exists some ta E the observation function
,T such that Ct,, = 1/ (1 + o²). we rewrite
y = :m ['=[ (Var, xo + v ar, n')
where = y/v and n' Nm.,,mm)). The noisy observation y' can be interpreted noise-free observation of vār_xo + VT- ar,n', n', which turn can be viewed as a sample in the manifold of qxx,,). Thus, we introduce auxiliary variable xt, which denotes noisy sample at time step ta. and propose to optimize the log- log-posterior term of the two variables as
max log ((x,, Xt, ly) max log q No, Xt.) + log (Xo Xt.) log (y)) Xo.Xi,
Xo.Xic
2 max log g(y Xx. Xt. + log po (Xo. xx,) log q
(10)
Xo.Nia
where in (10) we use pe to approximate the true prior q. By the construction the auxiliary variable Xt,, the likelihood term embodies a stringent consistency between this auxiliary variable and the noisy observation, which serves as a constraint. Note that log q(y) is independent of the optimization variables, thus (10) is transformed into the following constrained optimization task:
max log po (xo, X No.Xt,
s.t.
1 0] Xt.
(11)
Now we seek the ELBO as a proxy of the joint log-prior prior term of xy and xt,, to render it tractable. Leveraging Jensen's 's inequality and the transition probability of the diffusion model, reach the following proposition.
In practice, t. may not always be an integer, while the derivation remains similar. We concentrate on the scenario where la assumes integer in the main text. More discussions are deferred to the Appendix.

---

Proposition 1. Considering the DDIM reference distribution qo, we have the variational lower bound of the log-prior term as
log popoo Xx. xt, C
(x (o(t)l(xo [o((
(I) denoising mutching term
(. ((())
Ho (Vxx, ar, E)l (u(t))ooxxx))))
(2) noisy prior term
(12)
where C is constant independent of and Xt,, the weight g(t), w(t) are functions of ar and the DDIM variance os, and
The complete derivation of the lower bound is provided in Appendix A. Note that the summation on the right-hand side of (12) is partitioned into two parts. The first part, the denoising matching term, pertains to the approximation of log pe (x xx signifying the consistency between the noise-free variable Xo and the auxiliary variable Xt. The second part, the noisy prior term, corresponds to the approximation of the pog-prior of the noisy auxiliary variable x Conceptually, the workflow of utilizing this ELBO for inverse problems can be delineated as follows: the noisy prior term, coupled with the constraint on the auxiliary variable, exploits the prior information of the diffusion model and the observation information to recover a noisy sample Xt, that satisfies the observation equation, while the denoising matching term leverages the denoising capability of the diffusion model to restore the clean sample Xg from the noisy sample Xt..
Now, we address the ELBO outlined in Proposition 1 utilizzin the principle of the stochastic gradient method. The constant C' is independent of the optimization variables and can therefore be disregarded. The following proposition is obtained through reparameterization.
Proposition Randomly sampling t ~ u (1,2..... T} e, é' ~ N I), the stochastic gradients of the proxy objective (for minimization) with respect to Xy and xt, are, respectively,
V
||x0 pro (11
1 $ bai
dx
(13)
0
t 1> lai
Vxen
|1x0 - po (he, t)ll
t tSt
dxig.
V Xia
||xt_ - -VVor, 110 (he, 1)
+0)ll(((())((0) (wo (he,)),))
1> las
(14)
where
Varxo+ + Te(x, - Va Xo x0)/ /ai/o1..0, Xt. + /11 Ti ai, and 7/6 St are determined by and the DDIM variance öt.
aio + GIE t i tai
h;
(15)
t> ta-
The coefficients are omitted they can be scaled into the step sizes. Note that in such stochastic gradients, the Jacobian of (+)()) is involved, which necessitates backpropagation through the neural network and incurs significant computational and storage overhead. Therefore,
approximate yet effective and practical method by truncating the gradients in (-) i.c. Dx) Ro and Dx., n4 (.) O. Intuitively, the p-predictor of the diffusion model should be resilient to small perturbations in the input. For instance, when we feed a noisy image into a pre-trained diffusion model and introduce minor perturbations to the original image, anticipate the predicted image to maintain consistency with the original one. Thus the approximation for X, is acceptable, and similarly for Xt, if the noise level of xt, is not too large. By such gradient truncation, the approximation of the stochastic gradients for Xa and Xt, are, respectively,
d =
Xp
1e (h,,))
S bui
(16)
0
> lar
dx 1
1 1 Lai
(17)
Xt. vor (be, 8) -I- or,E'
f >>> to
5

---

Note that is not subject to the constraint and can be updated directly by stochastic gradient descent. x is involved in the equality constraint, so we apply the projection gradient descent method. Considering the projection operator of the observation in (11), an iteration step of the proposed algorithm is outlined as follows
Xn f- Xg
1/) dxp.
(18)
Xx. + Imy, O Omy x myy) y' + diag (Omy Xmy Im momm (xr. m2dx..
(19)
where and 172 are selected step sizes and diag(-) denotes the operation of arranging entries into diagonal matrix. We term this algorithm ProjDiff.
Algorithm 1 PifjDiff for VP diffusion (noisy observation).
Require: Observation y', pre-trained diffusion model μe, step sizes M) 72, total steps T, noise
schedule σ1 dr,, equivalent noise level Tt..
Sample ET, 67 ~ (O,
2: Initialize Xt,, f- Va,,, (er, vI Ci ETi
to to to + do
4: Sample 6' (O,
5: Calculate the approximate stochastic gradient dx,,; as (17);
6: Update x,, as (19);
7: end for
8: Initialize Xy + , ta):
9: for = to to 1 do
10: Sample e ~ (N0,,) I);
11: Calculate the approximate stochastic gradient dx as (16);
Update Xo (18);
13: end for
return Xo
3.2
Noise-free observations: special case
In the noise-free scenario, the observation equation becomes Axo and the auxiliary variable Xt. degrades to la O, thus the constraint is directly applied on Xg- The corresponding optimization problem can be expressed as
min Σ x(I)E ||x Xg
1eg (x,, 1)1
S.t. y= Axo-
(20)
Utilizing reparameterization and gradient truncation, the approximate stochastic gradient of the objective is
dx,; = Xy - 140 (VaxX + vi 116, 1). Then the iteration Pifj in the noise-free scenario given by:
(21)
X - A'y (1- (xo r)de.xo))
(22)
where At is the Moore-Penross pseudo-inverse of matrix A and vj is the selected step size. The principle of the ProjDiff algorithm in noise-free scenarios resides in treating the observation as constraint. In contrast [24] where the authors used Gaussian likelihood for noise-free observations, ProjDiff guarantees better consistency between the generated results and the observations.
3.3 Extension to nonlinear observation
Nonlinear inverse problems inherently more ill-posed than linear ones. For instance, in experiments, we consider the phase retrieval task, which aims to recover the original image given only the Fourier magnitude spectrum. The observation equation is:
y DFF
(23)

---

where P is the zero zero-eerrdeeee padding matrix. Effectively addressing nonlinear inverse problems can signifi- cantly enhance the algorithm's 's versatility. Here, we discuss about the ProjDiff algorithm for nonlinear observations concerning noise-free and noisy scenarios.
For noise-free nonlinear observation functions, the key point of ProjDiff lies in identifying an appropriate projection operator for the nonlinear constraint y A(x)). For some nonlinear functions A(-), such projection operator is analytic or can be approximated. For more intricate functions, we can resort to gradient descent or sub-gradient methods to minimize ||y - A(x)]l² starting from the initial point Xy to approximate the projection operator. Thus ProjDiff is applicable such class of nonlinear observations.
For noisy inverse problems, ProjDiff necessitates mapping the noise applied the observation to the noise applied on the original data Xg. Such a mapping is strict in the linear case. For nonlinear observations, the exact equivalent noise might not be accessible but may be estimated through some appropriate approach. In cases where the observation function is overly complex, one may leave the determination of the equivalent noise variance hyperparameter be manually tuned. This renders ProjDiff' applicable to noisy nonlinear observations.
3.4 Time steps, initialization, and Restricted Encoding
Here we elaborate on crucial details and address the weak observation problem.
Time steps. In practice, we observe that selecting time steps in a decreasing manner from T to 1 yields the optimal performance and provides more stable results with reduced oscillations, which aligns with the findings reported in [28, 24]. Therefore, we adopt this time step schedule as the default setting in this work. Furthermore, step multiple times (say N times) leads to enhanced performance, which bears a resemblance to the correction step proposed in [19] and the time-travel technique in [21].
note that in certain scenarios, repeating the same time
Initialization. We propose to use random initialization the ProjDiff algorithm. The core concept entails initializing Xo and xt, to be close to the manifold of q(xo) and q(x:,). respectively, which is achievable via the μe- d-prrdictor (or the Tweedie's s formula [33]) and the forward process. Randomly sampling Gaussian noise Cq ~ (O, I), we initialize the optimization variables by X po(er, T) and X + /r, Xy + -1-2a ã., Ep for some e/ ~ (O, Nonetheless, given the time step schedule and the form of the stochastic gradients discussed above, we note that the Xo remains unaltered until x, is fully optimized. Therefore, for noisy observations, would be more efficient to re-initialize Xy so (xt,, (a) once Xt, has been optimized.
Weak observation problem. In most image restoration tasks, the constraints imposed by observations typically strong enough to lead to unique original data point, rendering Proj effective in yielding satisfactory results. However, in scenarios where observations are weak, particularly in situations with significant degrees of freedom in the inverse problem, the performance of ProjDiff tends to decline. We term these scenarios the weak observation problem. Drawing inspiration from [34], where the authors found that including larger variance when remapping Xy to x, leads to noisy results, we attempt to adjust the noise schedule to diminish the variance of the sampled x,. We propose Restricted Encoding to tackle the weak observation problem, which entails fixing initial noise 6g ~ NO0, I) and reparameterizing the sampling of x, as
X
Xo + E + v - C22 E2E
(24)
where e I), and E [0, v1 043 regulates the level of randomness.
The complete implementation of ProjDiff for noisy observations outlined in Algorithm 1. More variations of Difj used in our experiments can be found in Appendix H.
4 Experiments.
We present the comparison of the proposed ProjDiff algorithm with several SOTA algorithms in two primary scenarios: (1) image restoration, and (2) source (music) separation and partial generation. All experiments are performed on single NVIDIA 3090ti GPU.
7

---

Table 1: Noisy restoration on ImageNet with 0.05. The LPIPS metrics are multiplied by 100.
Super- Resolution
Inpainting
Gaussian Deblurring
NFEs
Method
PSNR1 SSIMT LPIPS| FIDI
PSNR SSIM LPIPSU FID.
PSNRT SSIM LPIPS| FID
Aty
21.85 / 0.45 / 65.34 183.32
14.21 00244 67.42 176.52
17.78/ 0.29 / 60.25 100.05
1000
DPS
24.44 / 0.67 31811 / 33.17
30.15 / 0.86 17.76 22.03
24.26/ / 0.64 37.03 50.17
DDRM
25.66 / 0.72/ 34.88 55.71
29.99 0.87 17.11/ 19.88
27.82/ 0.79 28.62 45.9%
100
DDNM+
25.61 / /..2 34.45 54.10
30.00 /.87 17.09 20.08
27.90/ 0.79 28.63 46.54
100
RED-diff
22.74/ / 0.49 53.24 96.26
9.85/0.17 83.92 / 281.65
23.74/ / 0.50 /4. 68.23
100
ProjDiff
25.73 / 0.73/ 34.12 55.03
31.09 0.89/ 13.65 13.69
27.91 0.79 / 25.11 /3..27
Table Phase retrieval results. The LPIPS metrics are multiplied by 100.
Phase Retrieval a 0
Phase Retrieval a = 0.1
NFEs
Method
PSNRT SSIM LPIPS.), FID
PSNRT SSIM LPIPS., FIDJ
ER
11.15/0.19 84.48 409.91
11.16/0.19 84.43 412.59
HIO
11.97 82.06 / 342.52
11.87 / 0.24 82.18 339.13
OSS
81.08 360098
12.55 / 0.25 81.20 364.52
1000
DPS
13.19 /2.10//7.// 172.56
12.58 0.18 / 134.64
1000
RED-diff
14.19 0.28 / 63.42/ 173.52
12.36 0.14/75.98 221.11
1000
ProjDiff
33.39 / 0.74/ 20.42/ / 35.91
23.84/ / 0.60 38.55 76.20
4.1 Image restoration
Linear tasks. We demonstrate the performance of ProjDiff across three linear image restoration tasks on ImageNet [35]: 4x super-resolution (with average pooling), 50% random inpainting, and Gaussian deblurring. Comparisons are conducted against prominent diffusion-basee image restoration algorithms in recent years, including DDRM [20], DDNM+ [21], DPS [23], and RED-diff [24]. The performance of the least square solution (&o A Aty) is reported as a baseline. For fair comparison, DDRM, DDNM, RED-diff, and ProjDiff utilize 100 function evaluations, while DPS uses 1000 function evaluations its performance significantly declines with 100 steps. The pre-trained model on ImageNet is sourced from [36], specifically the 256 256 model without classifier guidance. Testing is conducted on a 1k sub-testset of ImageNet consistent with [2 Metrics include PSNR, SSIM [37], LPIPS [38], and FID [39], with all L.IPS values multiplied by 100 for clarity.
Table presents the restoration metrics for noisy observations with standard deviation a 0.05 (doubled when pixels are rescaled to [-1, 1)) on ImageNet. ProjDiff demonstrated highly competitive performance compared to other algorithms. Some visualization results are shown in Figure 2. More experiments on ImageNet and CelebA [40] alongside ablation studies are shown in Appendix C.
Nonlinear tasks. We evaluate the effectiveness of Pifj for nonlinear observations through the
challenging phase retrieval task on the FFHQ dataset [41]. We compare ProjDiff against DPS [23] and RED-diff [24] as they are applicable to nonlinear observations. Traditional baseline algorithms also reported, including the Error Reduction (ER) algorithm [42], Hybrid Input-Output (HIO) [43], and Oversampling Smoothness (OSS) [44]. Table 2 presents the results for both noise-free free and noisy scenarios. One sample is generated per image from each algorithm. Given the inherently ill posedness of phase retrieval, Pioj requires more steps to achieve superior results. We set the number of function evaluations to 1000 for all three algorithms. The results demonstrate that ProjDiff excels in addressing nonlinear inverse problems, regardless of whether the observation is noise-free free or noisy. Some visualization results are shown Figure 3. Further experiments on the high dynamic range (HDR) task can be found in Appendix C.

---

GT
Baseline DDRM
DDNM
DPS
RED-diff
Ours
Debiar
Topsitiog,
SR4
Figure Linear restoration on CelebA (a 0.05). Baseline means X = A y.
GT
DPS
RED-diE
Ones
GT
DPS
RED-diff
Onra
Plase
Figure 3: Nonlinear restoration on FFHQ (noise-free).
4.2 Source separation and partial generation
We evaluate ProjDiff on source separation and partial generation tasks following [45, 46]. The SLACK2100 dataset [47] is employed and the pre-trau pre-trained diffusion models are sourced from [45].
The objective of the source separation task is to separate the audio tracks of four instruments (piano, bass, guitar, and drums) from mixed audio sequence. We compare ProjDiff against the methods reported in [45], along with RED-diff [24] due to its relevance to this work. Performance is assessed using the scale-invariant SDR improvement (SI-SDR;) [48] for each instrument well as their average. Tabel presents the performance on the ISDM model [45] of different algorithms. ProjDiff' demonstrates significant performance improvements compared to other diffusion-based algorithms. The particularly noteworthy result is that ProjDiff surpasses the Demucs method [49], marking the first instance, the best of our knowledge, where diffusion-based separation algorithm significantly outperforms previous SOTA, not to mention that the diffusion model is not specifically designed or trained for the separation task.
The partial generation task aims to generate tracks of other instruments given partial of the instruments tracks while ensuring harmony. We employ the sub-FAD metrics [50, 45] for different partial generation tasks. Note that partial generation poses a weak observation problem as discussed in Section 3.4, thus the proposed Restricted Encoding method is applied for ProjDiff. The results shown in Table 4. ProjDiff outperforms other algorithms across different partial generation tasks. Additional results and in-depth analysis including ablation studies be found in Appendix D.
5 Conclusion
In this work, we introduce ProjDifff, a versatile inverse problem solver that harnesses the power of diffusion models to capture intricate data prior and denoise simultancously. By deriving novel two-variable ELBO as proxy for the log-prior, we reframe the inverse problems as constrained optimization tasks and address them via the projection gradient method. We meticulously explore the implementation details of ProjDiff and propose refined initialization methods and optional noise schedules to enhance performance. Through extensive experiments across image restoration tasks and source separation and partial generation tasks, we demonstrate the competitive performance and versatility of ProjDiff in linear, nonlinear, noise-free, and noisy problems. ProjDiff provides insights into better leveraging diffusion priors for inverse problems for future work.

---

Table 3: SI-SDR; for source separation task (higher is better).
Method
Bass
Drums
Guitar
Piano
Average
Demucs+Gibbs'
17.16
19.61
17.82
16.32
17.73
ISDM-Gaussian
14.27
19.10
12.74
12.20
14.58
ISDM-Dirac
19.36
20.90
14.70
14.13
17.27
RED-diff
17.96
21.60
15.99
15.33
17.72
ProjDiff 20.09
22.91
17.29
16.62
19.23
Previous SOTA.
Table 4: The sub-FAD metries different partial generation tasks (lower is better). The capital letters in the table header represent the initial letters of the generated instruments. For example, BD represents the task of generating bass and drums given piano and guitar.
Method
B
D
G
P
BD
BG
BP
DG
DP
GP
BDG
BDP
BGP
DGP
MSDM
0.43
1.30
0.12
0.73
1.79
0.79
1.60
1.59
2.00
1.47
2.19
2.63
3.02
3.02
RED-diff
0.44
2.26
0.18
0.71
3.66
1.14
2.56
3.10
3.26
1.74
5.65
7.73
6.14
4.87
ProjDiff
0.42
1.15
0.31
0.60
1.37
0.69
1.06
1.41
1.60
1.17
1.66
1.79
1.85
2.25
Limitations and future work. One of the possible limitations of ProjDiff is the potential challenge of effectively handling noisy observations with highly intricate functions. Also, its reliance on Gaussian observation noise may limit its applicability to other noise types like Poisson multiplicative noise. Additionally, ProjDiff needs manual tuning for the step size. Future research could focus extending ProjDiff to tackle intricate noisy nonlinear observations more effectively, well as developing adaptive step size strategies to further enhance the performance while reducing the need for manual intervention.
Acknowledgement
This work was supported by NSAF (Grant No. U2230201) and Grant from the Guoqiang Institute, Tsinghua University.
References
[1] Jascha Sohl-Dich Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[3] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[4] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5788 2022.
[5] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[6] Andrey Nikolayevich Tikhonov al. On the stability of inverse problems. In Dokl, akad. nauk sssr, volume 39, pages 195-198, 1943.
[ Deepa Kundur and Dimitrios Hatzinakos, Blind image deconvolution. IFEE signal processing magazine, 13(3):43-64, 1996.
10

---

[8] Sung Cheol Park, Min Kyu Park, and Moon Gi Kang. Super-resolution image reconstruction: technical overview. IEEE signal processing magazine, 20(3):21-36, 2003.
[9] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th anmual conference on Computer graphics and interactive techniques, pages
2000.
[10] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. A review of image denoising algorithms. with new one. Multiscale modeling & simulation, 4(2):490-530, 2005.
[11] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Computer Vision-ECCC 2016: 14th European Conference, Amstenlam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 649-666. Springer, 2016.
[12] Yoav Shechtman, Yonina C Eldar, Oren Cohen, Henry Nicholas Chapman, Jianwei Miao, and Mordechai Segev. Phase retrieval with application to optical imaging: a contemporary overview. IEEE signal processing magazine, 32(3):87-109, 2015.
[13] Morteza Mardani, Enhao Gong, Joseph Y. Cheng, Shreyas S. Vasanawala, Greg Zaharchuk, Lei Xing, and John M. Pauly. Deep generative adversarial neural networks for compressive sensing mri. IEEE Transactions on Medical Imaging, 38(1): (111:1677-17 2019.
[14] Yang Song. Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medi- cal imaging with score-based generative models. In International Conference Learning Representations, 2022.
[15] Hyungjin Chung, Dohoon Ryu, Michael T McCann, Mare Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22542-22551, 2023.
[16] Andreas Schuhmacher, Jørgen Hald, Karsten Bo Rasmussen, and Per Christian Hansen. Sound source reconstruction using inverse boundary element calculations. The Journal of the Acoustical Society of America, 113(1):114-12) 2003.
[17] Antonio Pereira, Jérôme Antoni, and Quentin Leclere, Empirical bayesian regularization of the inverse acoustic problem. Applied Acoustics, 97:11-29, 2015.
[18] Babjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
[19] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[20] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models, Advances in Neural Information Processing Systems, 35:23593-23606, 2022.
[21] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion mull-space model. The Eleventh International Conference on Learning Representations, 2023.
[22] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse Seeudoinverse-guidee guided diffusion models for inverse problems. In International Conference on Learning Representations, 2022.
[23] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffussin posterior sampling for general noisy inverse problems. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
[24] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference Learning Representations, 2024.
[25] Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023.

---

[26] Gabriel Cardoso, Yazid Janati el idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided denoising diffusion models for bayesian linear inverse problems. In The Twelfth Interna- lional Conference on Learning Representations, 2024.
[27] Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: filtering perspective. In The Twelfth International Conference on Learning Representations, 2024.
[28] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as blug-and-play priors. Advances in Neural Information Processing Systems, 35:14715-14728, 2022.
[29] Gen Li and Yuling Yan. O(d/T) convergence theory for diffusion probabilistic models under minimal assumptions. arXiv preprint arXiv:2409. 18959, 2024.
[30] Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards non-asympiotic convergence for diffusion-based generative models. In The Twelfth International Conference on Learning Representations, 2024.
[31] Gen Li and Changxiao Cai. Provable acceleration for diffusion models under minimal assump- tions. arXiv preprini urXiv:2410.23285 2024.
[32] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208. 19700, 2022.
[33] Bradley Efron. Tweedie's s formula and selection bias. Journal of the American Statistical Association, 106(496):1602-166)) 2011.
[34] Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations, 2024.
[35] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
[36] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[37] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
[38] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[39] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[40] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[41] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[42] James R Fienup. Phase retrieval algorithms: a comparison. Applied optics, (1555)) 1982.
[43] Fienup and Dainty. Phase retrieval and image reconstruction for astronomy. Image recovery: theory and application, 231:275, 1987.
12

---

[44] Jose A Rodriguez, Rui Xu, Chen, Yunfei Zou, and Jianwei Miao. Oversampling smoothness: an effective algorithm for phase retrieval noisy diffraction intensities. Journal of applied crystallography, 46(2):312-318, 2013.
[45] Giorgio Mariani, Irene Tallini, Emilian Postolache, Michele Mancusi, Luca Cosmo, and Emanuele Rodolà. Multi-source diffusion models for simultaneous music generation and separation. In The Twelfth International Conference on Learning Representations, 2024.
[46] Emilian Postolache, Giorgio Mariani, Luca Cosmo, Emmanouil Benetos, and Emanuele Rodolà. Generalized multi-source inference for text conditioned music diffusion models. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6980-6984. IEEE, 2024.
[47] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting music source separation some slakh; A dataset to study the impact of training data quality and quantity. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics ( (WASPAA), pages 45-49. IEEE, 2019.
[48] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey, Sdr-half-baked or well done? In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 626 630. IEEE, 2019.
[49] Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, and Jesse En- gel. Improving source separation by explicitly modeling dependencies between sources. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 291-295, IEEE, 2022.
[50] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet Audio Distance: A Reference-Frer Free Metric for Evaluating Music Enhancement Algorithms. In Proc. Interspeech 2019, pages 2350- 2350-2354, 2019.
[51] Charles Stein. Estimation of the mean of multivariate normal distribution. The annals of Statistics, pages 1135-1151, 1981.
[52] Xiangming Meng and Yoshiyuki Kabashima. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12343, 2022.
[53] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1219-1229, 2023.
[54] Yu Bruck and LG Sodin. On the ambiguity of the image reconstruction problem. Optics communications, 30(3):304-308, 1979.
[55] MHMH Hayes. The reconstruction of a multidimensional sequence from the phase or magnitude of its fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing, 30(2):140-154 1982.
[56] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
13

---

Appendix
In the appendix, we provide further details and additional experimental results omitted in the main text. First, we present the complete derivations of the propositions for the ProjDiff algorithm (Appendix A), and then provide the derivations for ProjDiff in the variance exploding setting (Appendix B). Next, we present additional numerical results. For the image restoration tasks, we report the results of noise- free scenarios on the ImageNet dataset, both noise-free and noisy scenarios on the CelebA dataset, the nonlinear HDR task on the FFHQ dataset, and the ablation studies (Appendix C). For source separation and partial generation tasks, results on the MSDM model, and ablation experiments with waveform visualizations are shown in Appendix D. Then we provide the details of the experiments, including the mathematical form of the inverse problems, and the details for calculating metrics (Appendix E). The implementation details of ProjDiff for various inverse problems are discussed in Appendix F, especially the details for the phase retrieval task and high dynamic range task. Discussion about the performance gap between ProjDiff and the MAP framework is presented in Appendix G. Algorithm blocks for all kinds of Proj used in this work are outlined in Appendix H. Finally, we include more visualizations of the restoration results to present the performance of ProjDiff intuitively in Appendix I.
A Derivation for ProjDiff in VP diffusion
A.1 Preliminaries and lemmas
Assume y Ax + n = Jmy 0]xo + n. For general linear observations, SVD can be used to reach this decoupled form in the spectral domain. We rewrite the observation equation as
y' [my: 0] (Vār, Xo + /1
Te. Te, n²).
(25)
where Ot., 11((1 + 2 for some la 1,... 7), y'= = Vou,y and n' (O, Ima). This makes the noise-free observation for sample at time step ta namely the auxiliary variable x
We consider the DDIM reference distribution as
(x Xo) = yo (x7/X0)
Or (x+- (x1, X0): II 2=2
(26)
where the transition probability follows
(x(x|(x0) (0,
go (x:11 xo) = .V Xx--li V + / a= - o3 Xx VaxXo
(27)
VI ai
where ō. is the variance hyperparameter in DDIM. We further choose ō.
Th fort >> t.r which makes the stochastic process between t, and Markov. For 1 < las we leave ã; hyperparameter. Thus the reference distribution can be rewritten as
1- -
I-a: V a-1
In
go (x1: = go (XT (XT )o (xt, (xo)
II
do (x: Xx, t=
Lo (xt-1 1 Xe. Xt). tmtu+2
II
(28)
Thus
yo (x1:aTya xo) go (xT II
go (x.-1 Xo) II yo (xt- 1x,,, Xx).
(29)
The following Lemma regarding transition probabilities can be proved by induction. Lemma 1. The conditional distribution xf conditioning on Xt, and xo is
JV (xis v ar/or, Xt. (1 [x/ar.)))
t > tai
Lo (xx xx))
(30)
N (xxi Vaixo + to (xe, var. /VI at, (Z1)
5 tas
14

---

where
7t V1 at o
ta
ax 02
k+1
k+1
(31)
ax+1
hatt
Proof. When é tus (30) holds as the chain between tu and Markov.
When t < ta (30) and (31) holds when = to as it is equal to the DDIM reference distribution. We assume (30) and (31) holds when to E {[,,2.... t.. then when t to have
xo) / dx (Xto X Xt., Xo)
4o (Xto (xto-1/Xe, xxx,,,
= dx (xto-1 X to Ao (Xto [t,, X0).
(32)
9o (Xto-1 Xtg Xo) is a Gaussian distribution according the DDIM refernece distribution, and Yo (Xt, Nt., Xo) is a Gaussian distribution due to the inductive hypothesis. Then by the convolution law of two Gaussian distributions, (30) and (31) hold when to 1. By induction, Lemma concludes.
The following lemma can be proved using Bayes's formula. Lemma 2. The conditional distribution of x conditioned on Xt,, and Xt xt 22ttt to 2is
X1-1 ai /a1 + v1 00-1/au. 0. Xt Var/ol. Xt.
Lo (x(x)-11xx,x,,)))
0.I
V1 - /t/
(33)
where
o7 = a./0/-1) /.-1/ata) a./a,.).
(34)
Proof. By Bayes's formula,
Jo (Xt-1 Xx, Xx..) (x,1 Xt, (x1 (x (xxx(xx-1190 (x (Xia) qa (Xx xx..)
(35)
Yo (x+ x...)
Substituting the transition probability fort 2 tu. we obtain the conclusion of Lemma 2.
The transition probability of pe follows 4o as (x xx) go gx (xxxx xt) for 2 and po (x xx) = (Xo: μn (x IT)..
A.2 Proof of Proposition
Proof. Based on the above preliminaries, we derive the ELBO of log x.,)) as follows:
log (x xx,)) log / dx log (x 4o (XITVV No. xt..)
Ja (1.T (xo: Xt.)
T
lo (XITVV Xoo Xt.))
2 dxirrve log (x((T) II
Po Po 1xxe) to (x II to (x, X2, Xo) I +2 do (x,-1 X x,)
1=1
= =Eg (xoxx..) (x
B qo (xes br, xo) DEL //
(I)
E Bys log pog,o( (x)
D Px X1 x (x
t=
- Ke (XT(Xe, (xoxxx))
(36)
15

---

The KL divergence between 9- (xr]xxx)) xx,) and po (x())) is zero as they are both (O, I) by the setting of diffusion models. Now we handle (I) and (2) respectively.
(1) popoox (x)(x1) is Gaussian and thus we have
(x (xx,xxx))), log po
po (xo = c1
R (x.)xxx...) Xo (x((11, 1)ll?
(37)
2
where by writing a we denote the term independent of xo and Xt, 9o (x x|x,, Xo) and po (xx-1(Xx)
also Gaussian, so by the KL divergence of the Gaussian distributions, we have (for t (a)
ye (xe: (x. DK (90 (xt-1 1x,,, xt. (x (x,)
1 =02 + 20;
Var-ixo v at of X1 V vā,X0
(x,
V (us-1M0(Xe1 t)
v1 ā
V at-1- -0
X1 (x1:1(((1:1)
(38)
VI az.
2
V 04-1 oi
= +
Va--1
vot
By (x
110 1)1
2
v1 -T- T2.
(2) Similarly,
to (xe, (x..))
log pe (Xt, ((o+t)
Cg
20 By (xx)))
X1, v 1o (x 1)
(39)
Xla+1 Vor, 10
(40 Xt, +1, to + +1)
Tta - -0, -hat1
and for t 2 to 2
Egg (x.1xee
DEL L (go (xxxx11/xx Xt, (ll.llopxx)11 xt))
=c4 +
1
ai a0-1//1, ar, Xta + / a.-1/a
0 X1 v Oi ai Xta
20,
V ar/
(40)
1 of Vaipoo(X),)) ()
-)) 11- 04-1
Ce.
Here we apply the reparameterizzttion trick in advance. We write
X = xx(()))== / /.//R,, Xt,, + V Te/TTr, e,
(41)
where (N, I), since do (xx) (x,)x N(xi v ae//ur, Xta, (1 Te/ II for t 2 to + 1. Substitute this into (39) and we have
E (x )x,) log po (x,, x(a+1)
at.+1 V C2 - -ou+1
1 =C3 + BE.N((.)))
(x._ 1oxx +1))
20t
Di.
VI Ctat1
0t. 0
o2
2
at,t1 Ci1
Clot1
c/ + Egg Nea ). + +1][[xt, - -(Var, 1) + VI - --ar 6') 112
no(x +1), Xt,+1 v Cr, +1 /00, Xta
+ E Bg 1x, wu(t + 1)
V1 - 0e.+1/00 Ti.
(42)
16

---

where
1
+1
1t
0 1a7
g(la +)
1
(43)
200 20
ata
01 +1
al+t1
02
T
at 0tatl
ac+11
w(aa + 1)
(44)
0lat1
Otta
+ +1
0ta+1
Ota
And also (40) is (for t 2 La + 2)
Ra (xi (xx,)))))))))))))))))) DKL
X XL, (1Po(((-1xxx)
=Cu + gt (xx (x -VVar, 1) /1 az,
c 112
(45)
X1 Voy X1a
+ (t) (xx(xxe)
(o(xx,1) V1 Tee
with
2
1
a:-1
1 0--1
Ctt
g(t)
(46)
2
ato
02.
Clt
w(()
1
Vai-1
1
1t-
oi 2
Val
ar
of
ar
o7
a2
at
Ct
Ct
(47)
Thus by integrate (37), (38), (42), and (45), we obtain the Variantional Lower bound of po Xt,) as follows:
log Po(xo, (xo, x..)
20
B (Xe Xox (a(t)l1x0 - mo(x,, 2)1
Σ
By (xx
g(() |xx..
(V(x)))( + v an
(48)
11
-os /is Xsa
+ w(t)
(o(x);t),
V1 Fe//Tee
Here we reach the result of Proposition 1., with the coefficient g() given by
1 1= 1;
287
(Va--1 /1 CT-1 0
2 $ tuj
287
s(l)=
atat1 ate
- Dia -
t=
(49)
20
D1at1
1
De-1
1 1-07 0
de
f. +2
207
Tha
Tha
and wt) is give by
1
Bea-1
Dig tott
Big 2
t=
Dea
1-
1- Be
w(t)=
1
Va,11
Va,
[
v Cre
10
ae Dta
222
t t2t,
Dea
(50)
0
17

---

A.3 Proof of Proposition 2
Proof. We continue the derivation by using the reparameterization trick. The reparameterization of (42) and (45) has been given above, we focus on (37) and (38) now. Applying Lemma 1, we can reparameterize them as
E (xe, (x,))
[xo (x(xx;))) t)ll
(51)
(Varxo + (xta - Vor, xo) /V1 or, + Gec, t
= Be
We fürther disregard all the constants. follows
The proxy objective for minimization can be rewritten as
L L(x0, (xx,)X,))
_ Be-N(o,))
to
|x0- uo (waixo + 7t-
Xr, var, Xo
Sie,t) 11
vI aro
E. Ee (1 [xx | (Vāu
Σ t=t+t
ar.
Xio + +V1
+ /1 - Tre at et
Tta
w(t) + g(t) no (Va
X1, + V1- Te/ Te/Tee e, -,) .c)
(52)
Now we seck for stochastic gradients L. Sampling (1,2,.... T and e, E NN,, the stochastic gradient with respect to Xy and Xt, are, respectively,
V xo
Ixx0 no (VorXo + (xxe, Voi,, ar, + Gec, t),
t i ta:
dx
(53)
t to:
and
Vx., || Xo pa (VTiXo +x(e, - Var, xo)/ (ol/vV== I ate + GGE, 111
1 15 bai
dx,,t
Vxe, Xta
Xin (VoraHo Ha (Var/ar.... + v1 - az/an, e,t + vI or - 11?
+ any (Vailar,,, + vT - -4////// ar,, e, :))
1 1> lay
(54)
where the coefficient g() is omitted as can be scaled into the step size. Here we reach the result of Proposition 2 in the main text.
A.4 Gradient truncation and remarks
In practice, we truncate the gradient through μ4o i.e., we let
Mo(+ = Dx,, (.)()) = O.
(55)
Thus the approximate stochastic gradients are, respectively,
Xx uo (Vo,x0 + (xe. /o), II + Ge,t)
t < taj
d
(56)
00
> las
and
0
x taj
dx
(57)
Xt_, (Vor, (o../o, Xt, 11 - e, + VT-
1> la-
At this point, we obtain the approximate stochastic gradients of the objective function with respect to the two variables Xa and x,,- We can perform gradient descent on Xy and the projection gradient method on Xt, to solve the problem.
Remark:
The idea introducing new noise to ensure that the update direction of x,, remains as much possible within the manifold at time step t.. The expression (xt.. Vār. (o()) vI Cie E'))
18

---

can be viewed as vector pointing from Xt, to vār, (a(+) + - As (-) is close to the manifold of Xg, the introduction of e' allows the endpoint the gradient vector be close to the manifold at time la.
2. For a general linear observation equation, the derivation remains similar when transforming the observations into the spectral domain using SVD. Due to potentially differing noise levels for each component, the objective function will take on an element-wise form. This based on the fact that in the diffusion model, we assume that the elements of x are independent when conditioning on X
3. In practice, find that setting the DDIM variance o, = 0 for t S ta yields the best results, which results in 7 = vI - 0g and Se 0. But this poses a problem as it leads to g(t) - co. In this work, we still scale all the coefficients (t) into the step sizes as it works well in the experiments. We leave this issue for future research.
4. Note that one may find la is not an integer in practice, as Ott, may not be in the discrete series To. This won't affect our algorithm, as it can be derived similarly that the approximate stochastic gradients in (56) and (57) still hold. The only difference lies in the initialization of xo which will be discussed in Appendix F.
5. We use the p-predictor in the derivation. The case of e-predictor is similar, with the conversion given by ( (x, vI (x))(xx)) /vāg. We refer to [19, 32] for more details about the relation between the u-predictor, the e-predictor and the Stein score [51].
B ProjDiff for VE diffusion
In this section, we derive the Proj algorithm for the Variance Exploding (VE) diffusion used in the source separation and partial generation tasks. The experiments using VE diffusion in this work are all noise-free, thus we only focus on the noise-free scenarios and leave the noisy scenarios of VE diffusion for future work.
Consider the forward transition probability q(xx[x=-1) N(x;;
oi-1) with or > Ør-1 > 01 > dg + O. When or is large enough, g(xx) NN I The initial
distribution of the reverse process is set to be (x() = NN 0f = q(xT) and the reverse transition the DDPM manner is
I (xt mo(xi, 1)), --100]- - - 1)
pe (x 1xx)) .V[ (x 1)
(58)
o
fort 2 2, which matches the condition probability
0 (x: xo),
-
q(x(-1 xo) N Xx +
(59)
o²
o
And we assume the last sampling step is set to be
po x1) = (xx:: (x1, (x1, 1), IT),
(60)
for some small 8p.
Thus the ELBO is derived as the same in DDPM [2].
(x = =log /"
T
(xxrr (x0)
po (xu:))
g(x1:1 (x)) dx1:7
1=1
2c - DKL(g(X1) (xo) (xr)) + Ex(x) log po (xo (x)lxx)
(61)
No)
2m2
(xxxT()00))) ss 0, and log po = C1 2 100-1u((1110 1)112. The KL divergence
between q(x,-1 1 Xo) and po (x(xxxlxx)) is
0, - -/-1
DKL 10, =
|xo
()(x
(62)
2 o
19

---

Table Noise-free restoration on ImageNet dataset. LPIPS metries are multiplied by 100.
Super-Resolution
Inpainting
Gaussian Deblurring
NFEs
Method
PSNR1 SSIM LPIPS) FID.,
PSNRT SSIM1 LPIPS., FID.
PSNR1 SSIMT LPIPS., FID
Aty
24.22 0.70/ 45.14/ 130.30
14.54 / 0.30/ 65.90 / 169.65
37.19 /0.95 / 12.15 14.77
1000
DPS
26.14/ 24.17 28.46
32.97 /0.92/ 9.74 14.08
27.21 /0.75 / 29.59) 39.93
100
DDRM
27.08 0.79 24.97 38.36
31.33 0.91 11.23 13.03
41.52 / 0.98 / /4.47 2,98
100
DDNM
27.07 0.79 23.87 /3.45
31.63 / 0.91 9.19 9.57
43.47 0.99 2.52 1.59
100
RED-diff
27.23/ / 0.79/26. 42.23
9.81 / 0.17 83.58 268.93
28.65 / 0.83 / 26.04 / 35.76
100
ProjDiff 27.09/ / 0.79/ / 23.42 / 32.95
33.19 / 0.94 7.16/ 7.60
44.17 0.99 2.22 / 1.35
for 2 2. Then by reparameterization, we get the Evidence Lower Bound as
ELBOvE C
Σ s (roDelxo -x0 o.e, 1)11
(63)
with
t=
25 J.
s(t)
(64)
2 25TST.
20
Sampling (1,2..... TJ. NNO,,) I), and truncating the gradient through μe in the VP setting, stochastic gradient for - -EBOOv (for minimization) is
dx Xy mx T)e,
(65)
Then the iteration step of the ProjDiff (noise-free) in the VE setting is
(ProjDiff for VE) Xo PA,Y (xo -nxxx.)).
(66)
where 1) is the step size.
The RED-diff [24] algorithm the VE setting can be derived similarly. In the source separation and
partial generation tasks, we use these forms as the implementation of our ProjDiff and RED-diff [24] algorithms.
Remark: For VE diffusion, the Restricted Encoding method involves fixing the initial noise (O.. I) and reparameterizing the sampling of as + EEO V f for 10,
Specifically, we set & J.-1 in partial generation tasks.
C
Additional experiments for image restoration
C.1 Additional results
Here, we present the omitted experiments for image restoration tasks. The results for noise-free restoration tasks on ImageNet are shown in Table 5, the results for noise-free and noisy restoration tasks on CelebA are shown in Table 6 and 7, respectively, and the results for nonlinear high dynamic range (HDR) task are in Table On the CelebA dataset, we further compare ProjDiff with more diffusion based rithms, including ITGDM [22], DMPS [52], Resample [34], and DiffPIR [53]. Pifj demonstrates highly competitive performance on all these tasks.
C.2 Ablation study
Metrics balance in the super-resolution task. An interesting observation is the trade-off between objective metries and perceptual metries the super-resolution task. DPS achieves the best perceptual metrics but performs poorly objective metrics, while RED-diff performs well in objective metrics
20

---

Table 6: Noise-free restoration on CelebA dataset. LPIPS metrics are multiplied by 100.
Super-Resolution
Inpainting
Gaussian Deblurring
NFEs
Method
PSNR f SSIM 1 LPIPS &. FID 4
PSNR 1 SSIM 1 LPIPS FID
PSNR SSIM LPIPS J FID
y'y
2..23/0.80/ 47.03/ 110.11
13.96 /..25 / 75.544 230.61
18.85 / 0.63 34.51 34.77
1000
DPS
30.25 / 0.86/ 16.88 35.65
36.02/ / 0.95 8.79 24.73
35.55 0.94 11.27 25.45
100
DDRM
31.38/ /.888/ 15.48 34.12
35.77 0.95 9.03 21.62
42.41 0.98 5.03 7773
100
DDNM
31.39 / 0.88/ 14.45 26.17
36.32/ / 0.96 / 6.86 12.45
45,36 0.99 3.01 / 2.08
100
RED-diff
32.44 /0.90/ 16.58 30.49
7.94/0.18 /78.23 192.07
33.07 0.90 17.02 20.54
100
HIGDM
30.47 / 0.87 16.01 / 344.27
36.31 / 0.96 8.71 24.27
40.69 / 0.98 5.92 17.38
100
DMPS
30.94/ 0.87 17.25 31.87
/ 18.58
42.84/0.99 /4.27 3.86
100
Resample
31,62 / 0.88 20.11 44.18
35.00 / 0.93 13.03/ 30.87
33.60 / 0.91 / 17.45 37.70
100
DifPIR
31.30/ 0.88 15.47 32.68
35.96 / 0.95 7.85 14.67
27.87 0.73 32.33 50.18
100
ProjDiff
32.57 / 0.90 16.08 33.74
36.84/ / 0.96/ 6.66 12.27
45.57 0.99 2.99 2.05
Table 7: Noisy restoration CelebA dataset with 0.05. LPIPS metries multiplied by 100.
Super-Resolute Resolution
Inpainting
Gaussian Deblurring
NFEs
Method
PSNR 1 SSIM 1 LPIPS FID
PSNR 1 SSIM 1 LPIPS FID
PSNR SSIM LPIPS FID 18.06 / 0.33 / 61.51 99.90
A'y
23.64/0.4P / 68.72/ 147.89
13.70/ 0.19 76.07 226.28
1000
DPS
27.98 / 0.78 23.10/ 39.91
32.80/ 0.90 16.32 32.80
29.46 / 0.81 / 21.19 38.54
100
DDRM
29.20/0.883 0.82 21.92 14
32.81 /0.90 16.78 35.28
30.51 0.85 / 19.89 38.24
100
DDNM
2..19/0.82 21.89 40.20
32.80/ 0.90 / 16.80 35.13
30.60 / 0.85 / 19.79 38.23
100
RED-diff
24.98 00.55 50.59 73.89
7.96 / 0.18 / 78.27 / 192.32
26.32 0.56 41.23 56.59
100
IGDM
28.25 0.79 22.73 38.70
32.68 0.90 16.02 31.54
27.57 0.76 23.76 38.61
100
DMPS
29.01 / /0811 / 22.87 37.80
32.18 0.87 / 18.77 23.88
30.51 0.84 20.32 33.63
100
Resample
29.58 0.83 14.18/ 45.51
33.11 0.90 15.97 31,38
30.89 0.85 21.91 11.91/388. 38.61
100
DifPPIR
27.87 / 0,73 32.33 50.18
29.71 0.74 27.14/////
26.93 / 0.62 38.07 53.01
100
ProjDiff
29.49 20.86 36.87
33.43/ 0.91 15.33/ /1.44
31.41
18.12 34.59
Table 8: HDR results. For noisy observation, the standard deviation 0.1. The LPIPS metric is multiplied by 100.
HDR =
HDR = 0.1
NFEs
Method
PSNR SSIM LPIPS) FID.
PSNR SSIMI LPIPS FID. 15.84 0.43 51.26 188.68 22.64 0.66 38.488 55.31 25.71 0.84 25.91 32.87
1000
DPS
15.69 0.43 51.22/ 193.15 193.15 26.01 0.87 23.60/ 34.03
1000
RED-diff
1000
ProjDiff
28.65 0.87 16.26 / 18.44
Table 9: Noise-free restoration with 20 steps on ImageNet. The LPIPS metrics are multiplied by 100.
Ressuuuiss PSNR1 SSIM1 LPIPS, FID,
Inpainting
Gaussian deblur
NFEs
Method
PSNRT SSIM1 LPIPS., FID.
PSNR SSIMI LPIPS., FID
20
DDRM
26.54 0.77 25.88 40.80 26.49 0.77 24.27 33.99
28.63 / 0.86 / 20.85 30.34
40.46 / 0.98 / 5.68 4.46
20
DDNM
29.50 0.88 16.09 / 21.01
42.11 0.98 3.72 2.87
20
ProjDifr
26.83 0.78 25.19 37.59
30.46 / 0.90/ 15.41 / 20.09
42.24 0.98 / 3,778 2.99
Table 10: Noisy restoration with 20 steps on ImageNet (a 0.05). The LPIPS metrics are multiplied
by 100.
Huper-Resolune PSNR1 SSIM LPIPS FIDJ,
Inpainting
Gaussian deblur
NFEs
Method
PSNR1 SSIM LPIPS FID),
PSNR1 SSIM LPIPS FIDJ
20
DDRM
25.20/ 0.69 / 35.59 53.45
27.55 /0.00/ 24.76 34.15
29.35 455.24 27.70 / 0.77 30.78 50.63 27.87 0.79 24.49 30.53
20
DDNM+
25.23 0.70/ 35.06/ 52.12
27.56 / 0.80 24.71 34.24
ProjDiff
25.76/ 0.72/ 34.88 55.71
29.27 19.43 23.89
21

---

ProDit
Prop
27.2
ReDDofff
27.3
8ONM
DONM
26.8
20.0
20.6
26.4
26
26.2
Trs
DPn
20
28
70
34
36
31.5
22
22.5
23
23.5
24
24.5
25.5
26
26.5
27
3113
107)
Figure 4: Noise-free super-resolution results on ImageNet. The red lines show the variation of PSNR v.s. FID and LPIPS of ProjDifff algorithm.
Table 11: Phase retrieval results. The LPIPS metrics are multiplied by 100.
Phase Retrieval a = 0
Phase Retrieval o = 0.1
NFEs
Method
PSNR1 SSIM LPIPS., FID
PSNR1 SSIM LPIPS,, FID.
ER-4trials
11.36 / 0.20 83.95 410.75
11.32 83.87 410.53
HIO-4trials
12.60 / 0.26 / 79.78 336.94
12.62 / 0.26 / 79.88 333.90
OSS-4trials
12.75 / 0.26 79.95 / 342.36
12.69 / 0.25 80.37 344.84
1000
DPS--trials
19.36 / 0.45 / 48..48/ / 104.70
17.25 / 0.38 51.83/ / 18.10
1000
ProjDiff-4trials
41.58 / 0.94/ /.49 7.199
28.11 0.76 / 28.43 53.38
for noise-free super-resolution but declines on perceptual metrics. We find that in the noise-free super-resolution task. the step size of ProjDiff' controls the balance between objective and perceptual metrics. Figure 4 presents Proj FID-PSNR curve and LPIPS-PSNR curve the super-resolution task on ImageNet, with step sizes ranging from [1.2, 1.9]. Note that lower values of LPIPS and FID and higher values PSNR indicate better restoration results, thus the top left corner represents the ideal performance. The red lines demonstrate the variation in objective and perceptual metries of Diff by adjusting the step size. Also note that ProjDiff exhibits superior performance compared DDRM, DDNM, DPS and RED-diff.
Fewer steps. DDRM [20] and DDNM [21] can both address image restoration with fewer steps (20 steps). Here test the performance of the proposed ProjDiff with fewer steps ImageNet. Table and Table show comparisons of Pifj with DDRM and DDNM three linear restoration tasks under noise-free and noisy observations, respectively. ProjDiff also demonstrates highly competitive performance across multiple metrics, indicating its potential to perform image restoration with fewer steps.
Four trials for the phase retrieval task. It is noted in [23] that in the phase retrieval task, DPS requires sampling four times for each image and selecting the best result to achieve satisfactory performance. Therefore, we compare the performance samplings. We also report the performance of the ER, HIO, and OSS algorithms with four samplings as basclines. The best image is selected as the sample with the highed PSNR metric. The results shown in Table 11. Both DPS and ProjDiff show performance improvements with four trials, with Diff particular exhibiting much better performance, reaching a PSNR of 41.58dB in the noise-free case.
DPS and Proj with four independent
The impact of the gradient truncation method used in ProjDiff. We compared ProjDiff with and without the gradient truncation method on the CelebA dataset, as shown in Table 12. Proj without gradient truncation is denoted as ProjDiff-FG. The results indicates that there some performance loss when using gradient truncation compared to not using it, while the efficiency increases by nearly three times. Considering that using gradient truncation has already achieved satisfactory performance, accept the trade-off of some performance loss for the gain in efficiency.
22

---

Table 12: Ablation study of the gradient truncation method in ProjDiff on noise-free tasks CelebA.
Super-Resolution
Inpainting
Gaussian Debluring
Time (8)
Method
PSNR 1 SSIM 1 LPIPS FID
PSNR 1 SSIM LPPPS & FID
PSNR SSIM LPIPS & FID
4.42
ProjDiff
32.57 0.900 16.08 33.74
36.84/0.96 12.27
/..99/ 2.05 45,98 0,99 2.72 1,94
11.13
ProjDif--PG
3 16.87 36,39
37.30/0.96 7.12/ 11,65
Table 13: Using gradient descent to approximate the projection operator.
Phase Retrieval = PSNRT SSIM LPIPS| FID
Phase Retrieval a 0.1
NFEs
Method
PSNRT SSIM LPIPS| FID 23.84 0.60 38.55 76.20 23.08 0.56 44.86 97.13
1000
ProjDiff
33.39/ 0.74 / 20.42 35.91 33.55 / 0.71 / 23.04 / 45.44
1000
ProjDiff-GD
Using gradient descent to approximate the projection operation for nonlinear tasks. We claim that for general nonlinear tasks, ProjDiff can be applied by using gradient descent to approximate the projection operation. Here we verify this method in the phase retrieval task (ProjDiff- DD in Table 13). The results indicate that approximating the projection operator of X with respect the observation equation by minimizing |yy A( (()) starting from can also achieve satisfactory performance, which suggests that ProjDiff has generalizability for general nonlinear inverse problems.
Sensitivity tests for the hyperparameters. We conduct the sensitivity tests for the step size m and noise level dg in ProjDiff on noisy super-resolution task on CelebA, as shown in Table 14. "xa" denotes that we perturb the input standard deviation of ProjDiff by multiplying a. The results indicate that ProjDiff exhibits certain degree of robustness the step size and noise level. This also demonstrates that retaining the equivalent noise level an adjustable hyperparameter is feasible when it cannot be directly calculated.
RED-diff with random initialization. The RED-diff algorithm observed to accidentally fail in the 50% random inpainting task. Here we demonstrate that by adopting the random initialization method proposed in this work, RED-diff achieves more reasonable results. We compare RED- diff's performance on ImageNet for both noise-free and noisy inpainting tasks when using random initialization (denoted as 'RED-diff with random init') ') and using degraded images for initialization as proposed in [24]. The results are shown in Table 15. RED-diff's performance is significantly improved when using random initialization.
D Additional results for source separation and partial generation
D.1
Results MSDM model
Two models were trained in [45]: one is MSDM which models the joint distribution among the four instruments, and the other is ISDM which models the distribution of each instrument independently. ISDM performs better than MSDN in the source separation task as reported in [45]. We have presented Pifj's performance on ISDM in the main text, and here we supplement the experiment on MSDM. The SI-SDR, metries of ProjDiff, RED-diff, MEDD--Gussinn, and MSDM-Dii Dirac shown in Table 16. ProjDiff also demonstrates the best performance on the MSDM model.
Table 14: Ablation study of the hyperparameters in Proj noisy super-resolution task CelebA.
PSNR
LPIPS
7 lo
x0.7
x0.9
x1.0
x1.1
x1.3
x0.7
x0.9
x1.0
x1.1
x1.3
0.1
29.92
30.16
30.19
30.18
30.11
21.96
21.71
22.25
22.70
23.29
0.2
29.69
29.94
29.96
29.94
29.85
22.48
20.92
21.23
21.60
22.12
0.4
29.16
29.47
29.49
29.47
29.37
24.25
21.04
20.86
21.06
21.51
0.8
28.45
28.85
28.89
28.87
28.75
27.67
22.82
21.85
21.63
21.74
1.6
27.69
28.10
28.14
28.09
27.88
32.23
26.62
24.88
24.07
23.69
23

---

Table 15: Comparison of RED-diff with and without random initialization on ImageNet inpainting task. LPIPS metrics are multiplied by 100.
Inpainting (a 0)
Inpainting (a 0.05)
PSNR t SSIM t LPIPS FID L 9.81 0.17 83.58 268.93 29.86 / 0.89/ 14.55 / 15.68
PSNR SSIM 1 LPIPS & FID +
RED-diff RED-diff with random init
9.85 0.17 83.92 28.92/281.66
25.98 0,66 29.38 28.02
Table 16: SI-SDR, metrics on MSDM (higher is better).
Method
Bass
Drums
Guitar
Piano
Mean
MSDM- Gaussian
13.93
17.92
14.19
12.11
14.54
MSDM-Dirac
17.12
18.68
15.38
14.73
16.48
RED-diff
16.00
19.98
16.15
13.67
16.45
ProjDiff
17.65
20.76
17.47
15.17
17.76
D.2 Ablation study
Repetition steps. ProjDiff repeats N 5 times on each time step as we state in Section 3.3 in the source separation task, while MSDM-Gaussian and MSDM-Dirac [45] only repeat twice using the correction steps method in [19]. A natural doubt is whether the performance improvement arises from more function evaluations. We test MSDM-Dire algorithm with more steps, and the results are shown in Table 17. When increasing the correction steps from 2 to 5 which matches the function evaluations of ProjDiff, the performance of MSDM-Dirac declines. This verifies that ProjDiff''s advantage does not simply arise from more function evaluations. Moreover, we test Pioj performance with more repetitions using the MSDM model, namely 10, 25, and 50 times with the step size adjusting accordingly to stabilize convergence. The step size is set to 0.5/N, where N the number of repetitions. The results are shown Table 18. Further increasing the number of repetitions leads to even better metrics for ProjDiff.
Table 17: More correction steps for MSDM-Dirac (higher is better).
Bass
Drums
Guitar
Piano
Mean
MSDM-Dirac (2 correction steps)
17.12
18.68
15.38
14.73
16.48
MSDM-Dirac (5 correction steps)
15.68
17.49
15.05
14.01
15.56
Momentum mechanism. We use Polyak momentum in the source separation task. Here we conduct experiments validate the effectiveness of the momentum mechanism. The results with different momentum hyperparameter ranging from 0 to 0.9 are shown in Table 19. Setting / to 0.5 yields the best performance. We infer that certain level of momentum over large momentum may cause oscillations at small time steps and thus affect the results. This explains why the performance is relatively poor when 3 0.9 in the source separation task.
accelerate convergence, while
Weak observation problem and Restricted Encoding. Here we further explain why the partial generation task is a weak observation problem. The partial generation task aims to generate the tracks of other instruments based on some instruments, c.g., generating drums, piano, and guitar based on bass. The constraint provided by the observations is very weak, which is because in musical ensemble, the correlation between different instruments exists but is not significant, and composer can freely create the melodies of other instruments based on certain instruments' melodies. Therefore, this is more of a generation task than a restoration task, and we refer to this type of problem as the "weak observation problem" where the constraints provided by the observations are weak.
Now we verify the effectiveness of the proposed Restricted Encoding method the partial generation task. While keeping other parameters unchanged, we switch the sampling method of x, in Proj from Restricted Encoding to sampling directly from the forward process q(x,|xx (denoted as "wo Restricted Encoding"). ). The results are shown in Table 20. Without Restricted Encoding, the
24

---

Table 18: More repetition steps for ProjDiff (higher is better).
Bass
Drums
Guitar
Piano
Mcan
ProjDiff's steps
17.65
20.76
17.47
15.17
17.76
ProjDiff 10 steps
17.67
20.78
17.47
15.22
17.79
ProjDiff 25 steps
17.71
20.80
17.55
15.30
17.84
ProjDiff 50 steps
17.74
20.83
17.57
15.32
17.86
Table 19: Different momentum for ProjDiff (higher is better).
Bass
Drums
Guitar
Piano
Mean
ProjDifff 3 =
17.56
20.75
17.41
15.06
17.70
ProjDiff $ 0.3
17.47
20.75
17.44
15.10
17.69
ProjDiff 8 = 0.5
17.65
20.76
17.47
15.17
17.76
ProjDiff 8 = 0.7
17.64
20.75
17.33
15.03
17.69
ProjDiff B =099
17.82
20.74
17.16
15.00
17.67
performance of ProjDiff significantly declines. This indicates the importance of Restricted Encoding for the weak observation problem.
Visualization of the waveforms. In the partial generation tasks, observation is that the number of generated instruments decreases, the advantage of ProjDiff becomes smaller. We point out this is because the sub-FAD metric considers the combination of the generated and given tracks. Therefore, in cases where more instruments are provided, the combined track is closer to the ground truth, and the difference among different algorithms would be smaller. This can also explain why ProjDiff' performs slightly worse than MSDM and RED-diff when generating only the tracks of guitar. This may be because ProjDiff tends aggressively create new musical segments, thus resulting in lower similarity to the ground truth set. We present some waveform samples in Figure 5 to verify this claim. The samples are randomly selected and the same row shares the same ID. Difj' results have larger amplitudes while RED-diff and MSDM's s results tend to remain silent.
Moreover, following the conventions in acoustics researches, include some audio samples of ProjDiff in the supplementary materials for subjective evaluation.
E Experimental details
E.1 Inverse problem settings
Source separation. Denote the tracks of piano, drums, guitar, and bass as X1, X2, X3, E R. where n represents the length the sequences. The observation function for source separation is y (,,, x2, x3. x1))
x1 + X2 X3 X1.
Partial generation. The observation matrix for partial generation is diagonal matrix with elements of either zero one, i.e., A = diag @11,, 422, 033, 044) with @11, 422, 033, 044 {0. 1}. 0 represents that the corresponding instruments are to be generated while represents that the instrument is provided as the condition. The observation is (A(x,, X2, X3, x..)))"
The super-resolution task uses x 4 average pooling as the degradation function, which is quite similar to the source separation task. Consider x image block, with elements
Table 20: Comparison of Diff with and without Restricted Encoding (lower is better).
BD
BG
BP
DG
DP
GP
BDG
BDP
BGP
DGP
Diff
0.42
1.15
0.31
0.60
1.37
0.69
1.06
1.41
1.60
1.17
1.66
1.79
1.85
2.25
with Restricted Encoding
Dift
Restricted Encoding
0.44
2.26
0.18
0.70
3,67
1.14
2.55
3.10
3.24
1.74
5.66
7.26
6.13
4.85
25

---

ProjDiff'
RED-diff
MSDM
Figure Samples of the partial generation results. The results from ProjDiff (left) have larger amplitude, while results from RED-diff (middle) and MSDM (right) have more silent periods.
XI E [R = 1,2,,,,,,,,444 The dimension represents the RGB channels. The observation
equation is thus given by y = 16 Ei,= Xz,j. For the entire image, the calculation can be paralleled for all 4 x 4 blocks using matrix operations in Py PoToch.
1
Random inpainting. The observations matrix for random inpainting is diagonal matrix of zeros and ones, = diag [xx) for < I n where n is the total number of pixels in an image, specifically 256 x 256 in this work. The diagonal elements agk have 50% chance to be O and a 50% chance to be
Gaussian deblurring. We follow the settings from [20, 21] for Gaussian deblurring, with 1D Gaussian kernel of size 5 and standard deviation of 10. The observation equation is in the form of convolution. Efficient SVD proposed in [20] can be used for calculating the projection operator.
Phase retrieval. Our experimental setup for phase retrieval follows [23]. The observation is the amplitude spectrum of the original image. However, as the phase retrieval task is highly ill-posed, is nearly impossible to recover the image directly from the amplitude spectrum. Therefore, we follow the standard practices [54, 55, 23] by first padding the image with zeros and then computing the amplitude spectrum. This increases the spectral resolution and provides more information for retrieval. Thus the observation equation for phase retrieval is y = [DF[(P x)]. where P represents the padding operation, which is linear operator. The original image is 256 256, and the padding is done with size 64 in four directions, resulting in padded image of 384 x 384. Note that one significant difference between the phase retrieval experiment in this paper and in [23] lies that [23] reported the best results from 4 independent runs for each image, while we perform all the algorithms once for each image.
High dynamic range. The observation for the high dynamic range task can be written element-wise as
2x; < 1; -1 < 2x; 1;
f(x,)
2x;
(67)
2x, 2 1,
i.c., first multiplying all the pixels by 2 and then clipping them within [-1,
26

---

E.2 Hyperparameters for the compared algorithms
DDRM. DDRM uses the recommended parameters from [20], i.e. 17 = 0.85 and % 1.0.
DDNM. DDNM uses the recommended parameter - 0.85 from [21]. For noisy tasks, we report the performance of the DDNM+ algorithm [21].
DPS. Following the settings in [23]. the step size is set to & - E/yly (x)))x))) On ImageNet, & is set to 1 for super-resolution and inpainting tasks, and 0.4 for Gaussian deblurring. Since [23] did not conduct experiments on CelebA, we conducted fask-by-task parameter tuning for the learning rate in DPS and reported the results, ensuring fair comparison. For phase retrieval and high dynamic range tasks, Ç = 0.4.
First, in the image restoration tasks, we follow the settings in [24] using the Adam optimizer with momentum pairs of (0.9, 0.99). RED-diff requires a balancing parameter and a step size Ir. À is set to 0.25 for all image restoration tasks, and the step size Ir is set to 0.25 for the super-resolution and 0.5 for all other tasks. However, the source separation and partial generation tasks, we find that the Adam optimizer yields poor results. Therefore, we run RED-diff with the same parameters used in Proj and set À = 1.0 which is tuned to the best. Thus in the source separation task, the momentum is set to 0.5 and the step size is set to 0.1 with 5 repetitions. In the partial generation task, the momentum is set to 0.9 and the step size is set to 0.05 with no repetitions.
RED-diff.
TIGM. We carefully conducted hyperparameter searches for each task using the first eight images with the average PSNR as the metric. Except for Noisy Gaussian Deblur task, we avoided using other algorithms to initialize IIDM to ensure a fair comparison. Specifically, we first use the forward transition to map the degraded image to the noise level corresponding to the 500th step, and then use 100 steps of IIGDM to solve the inverse problem. We find that this approach is much better than starting directly from white noise (i.e. 1000th step). For Noisy Gaussian Deblur task, IIGDM without initialization from other algorithms struggled to achieve satisfactory results. Therefore, use DDNM+ to obtain the sample at the 500th step and then switch to IIGDM to complete the solution.
DMPS. The step size hyperparameter was searched for each task using the first eight images with the average PSNR as the metric.
Resample. The weight hyperparameter 7 was searched for each task using the first eight images with the average PSNR as the metric. ReSample was originally designed for Latent Diffusion, thus in our experiments, we consider the encoder-decoder as identity mappings.
DifIPIR. The weight hyperparameter À was searched for each task using the first eight images with the average PSNR as the metric.
NSDM/SSMM
for source separation. Following [45], the algorithms use the Sehum mechanism [56], with Schum 20 for the MSDM model and 40 for the ISDM model. Correction steps [19] are set to
MSDM for partial generation. Similarly, Schum 10 and no correction steps are used.
E.3 Details for metries calculation
Here we provide the details of how we calculate the metrics for replication and fair comparison.
For image restoration, we first save the generated results in the png format. When all re- sults are generated, we read them back in to calculate the metrics. For PSNR and SSIM, we use wil__sinnnnooiii ratio and crructual_similariii
methods from the skimage library". mmxg)
When calculating SSIM, the parameter 'data range' is set to maattt__nn=ggeeraattt generated image.min()) For LPIPS, we use the LPIPS library'. Note that before calculating LPIPS, pixel values are normalized to |-1, 1]. And for FID, we use the torch-fidelity library".
"htpp://gituu..com//e/e//
"htpps///ithuub.co/// com/rdbrhanng/reeeptaaliiiiiiii hhtps://githubbc http:////ttt...o//////////hhche
27

---

For the SI-SDR, metric used in source separation, the calculation is as described in [45]:
SI-SDR; SI-SDR xn) 31-SDR(xn:33))
||ax,11 112 +
(68)
SI-SDR(X,, x.)) = 10 logio ||ax, - &n 12 t
where Xn is the ground truth sequence, represents the ground truth summation of the four instru- ments, and Xn is the estimated track from the algorithm. a = [x,.1 numerical errors.
tc and e = 10- to prevent
For the sub-FAD metric in the partial generation tasks, we first calculate the summation of the generated tracks and the corresponding conditional tracks and save the mixtures the wav format. Then the frechet audio distance library' to compute the FAD metrics between the generated mixtures and the ground truth mixtures.
Implemention details of ProjDiff
F.1 Initialization for noisy observation with non-integer la
In the case of noisy observations, we aim to initialize x, using the optimized xt,, as (x fa). However, note that to may not correspond exactly to any discretized time step in practice. In such instance, we assume to E 21, such that to I to S to 1, and consequently can first perform a forward transition & /ag+ 1/0 Xt. /1 t for some 6 ~ NO, I), and then initialize X < (xt +
F.2
Details for linear observations
The projection operator for source separation is
P(x1, X1X1,X2,Xx X3. = (x1, x2, X3, X4) 4 (y
xx, (1,1,1,1
(69)
The projection operator for partial generation is
P(x1, X2, X3, X4)
X4) yA.
(70)
Super-resolution and random inpainting are similar to source separation and partial generation, respectively. The projection operators of these problems can obtained without SVD. Regarding equivalent variance, for 4 X 4 super-resolution, based on the average of independent Gaussian variables, the equivalent variance is 16a² with a² being the variance of the noise adding to the observation. For inpainting, the equivalent noise should enjoy the same variance as the noise on the observations.
Gaussian deblurring task is a bit more complex. The projection operator is
P(xo) (I A'A)x + AAy,
(71)
where At the Moore-Penrose pseudo-inverse of A. The element-wing equivalent noise can be handled using the SVD.
F.3 Details for nonlinear observations
Phase retrieval. For the phase retrieval task, consider the observation y [DFT(Px)) Given input Xo and observations y, first we calculate
DFT (Pxn) and compute the projection operator of y |x| with respect z, which & = y 3 z/ |zz] where the division is element-win wise and @ denotes the Hadamard product. Next, we map 2 back to the original pixel space. As DFT is invertible and the the padding matrix, we define the inverse transformation matrix corresponding to P P that extract the central 256 x 256 pixels from 384 x 384 image. Thus the inverse mapping can be written as x = PF (2). Finally, as the resulting x is in the complex space, we project it onto
htanss//gghbbb.oomm/guuuu///fbee distee distance
28

---

Table 21: Optimal step sizes for PiojDiff.
Dataset
ImaggeNet
ColebA
FFHQ
Rppe--eesottiinn
Inpainting
Goussian Deblurring Super-Resolutien Resolution Inpainting Gaussian Deblurring Phase Retrieval
HDR
Noise-free
20V100steps
20/100steps
20/100steps
100steps
100steps
100steps
1000steps
100steps
1.5/ 1.7
1.1
0.9
0.7
1.1
0.8
1.5
2.0
Supor-Resolution
Inpainting Gaussian Deblurring sepor-Resolution Inpainting Gaussian Deblurring
Phase Retrieval
HDR
Noisy
2W110sseps
20/100sseps
20/100steps
100steps
100sstpps
100steps
1000ssteps
100steps
7h
0.5
4.0/
0.5 0.1
0.4
0.1
1.9
the real space, namely Xproj rxa where the real(-) operator takes the real part of a complex tensor.
Regarding equivalent noise, we use the principle of energy equality in the space domain and in the frequency domain. The energy of observation noise is nio where n1 = 384 is the size of the spectrum image. Assuming the variance of the effective noise is aen. the noise energy in the space domain is n² a with 256. Equating both energy yields del n1a/ /n 1.5a.
High dynamic range. The regular projection operator the HDR task can be expressed element-wise as follows
y:/2
-1 < Y; 1;
0.5
y, = and x; S 0.5;
yi), =
(72)
0.5
y; = - and x; 2 -0.5;
else.
This formulation applies to the noise-free case. Nonetheless, we discover an adjusted version of the projection operator that can further accommodate noisy observations, which is
y:/2
- 0.5 y; < 0.5;
y1/2
y, 2 1 and x; S 0.5;
Pawisy (i)
(73)
y:/2 y, S -1 and x; 2 -0.5;
X,
else.
This adjusted form is more adept handling uncertainty the observation values within the intervals [-1, 0.5] and [0.5, 1] induced by the observation noise. We employ this modified projection operator the noisy HDR task. The equivalent noise variance can simply approximated as o = (a/2)²
F.4 Hyperparameters.
In all experiments, PiojDiff uses SGDM as the optimizer. The momentum set to 0.5 for the source separation task and 0.9 for the partial generation task. The step size set to 0.1 for source separation with 5 repetition steps and 0.05 for partial generation with no repetition. For all image restoration tasks, the momentum is set and 1/2 for the noisy observations is fixed to 1.0. The DDIM variance
for & S ta and o, a = - /1 11- ā for ta- No repetitions are applied. Other tuned optimal step sizes are shown in Table 21. The time step schedule follows [45] for the source separation and partial generation tasks, and follows [20, 21] for the image restoration tasks.
o,
G Discussion about the MAP framework
ProjDiff shares similarities with the Maximum Posteriori (MAP) estimation method. Theoretically, in noisy situations, it is only necessary to Gaussian likelihood, i.c., p(y(xx) = >N(y: X,, o I). to solve the inverse problems. However, Pioj introduces new auxiliary variable, which leads to better experimental outcomes than the MAP framework. Here, present some qualitative discussion towards this experimental result.
On one hand, if we are given perfectly accurate priors and likelihoods, and have sufficient com- exact posterior and also its gradients. Within the MAP framework, this should lead to the ideal results. However, in practice, the priors provided by diffusion
putational capability, we could obtain the
29

---

models are not entirely accurate, and the weight coefficients between the priors and the likelihood can- always be perfectly set. Moreover, dealing with the priors in diffusion models relies on stochastic optimization methods. These factors imply that the MAP framework often fails to achieve the desired solution. In such cases, further exploring the information or capabilities within the diffusion model can provide substantial assistance in solving inverse problems and enhance performance. This is one of the reasons why Proj may have a performance advantage over the original MAP framework.
On the other hand, MAP methods utilize gradient descent to leverage the information from the observation, while ProjDiff transforms this into a projection operation by introducing noisy auxiliary variables. This approach offers numerous advantages: there is no need to consider the step size of gradient descent (at least for the likelihood term); there is no need to consider the weight coeffi- cients between the likelihood and prior terms; and it ensures consistency between noisy samples and observations (the role of this consistency has also been confirmed in the noise-free scenario). Moreover, the introduction of this auxiliary variable indicates that ProjDiff' can actually be viewed as simultaneously recovering clean data and the noise added to the observations, which can yield more accurate results than merely characterizing the noise prior with a Gaussian distribution. These transformations are all thanks to ProjDifff's s utilization of both the clean prior and noisy prior modeled by diffusion models (i.e., the prior modeled by the diffusion model and its denoising capability).
H Algorithm blocks
Here, we present additional algorithm blocks for Proj used the experiments. Algorithm is used for noise-free restoration tasks. Algorithm 3 and Algorithm 4 used in the source separation task and partial generation task, respectively.
30

---

Algorithm 2 ProjDiff for VP diffusion (noise-free).
Require: Observation y, observation function A, pre-trained diffusion model μe,, step size total steps T, noise schedule o1.
1: Sample eT ~ (O. I);
2: Initialize Xg < (E T)::
3: fort= fo do
4: Sample 6 ~ (o, I);
5: Calculate the approximate stochastic gradient: dxp,t = Xo - -ullVaro + v1 0GEE, 1);
6: Update Xo: Xg < (x r7dxo.t);
7: end for
8: return Xo
Algorithm 3 ProjDiff for VE diffusion (noise-free).
Require: Observation y, observation function A, pre-trained diffusion model μi., step size momentum 3, total steps T, iterations per step N, noise schedule 0 ....0TT.
1: Sample ÉT ~ NO. I);
2: Initialize X < Ho(oTre,TT)
3: V + 0:
4: for T do
5: for k = to N do
6: Sample ε, N(O,));
7:
Calculate the approximate stochastic gradient: dxo,t = Xo -oo(x0 + o.C,,,tt); Update momentum: v +v (1 3)dx..
8:
9: Update Xg: x t- (x. - rjv);
10: end for
end for
12:
return XO
Algorithm 4 ProjDiff for VE diffusion with restricted encoding (noise-free).
Require: Observation y. observation function A, pre-trained diffusion model μea, step size momentum 8, total steps T, iterations per step N, noise schedule 0 o....TT.
1: Sample éT; 6a A0, I);
2: Initialize Xg + ne (T(TTeT,T);
3: v V+ 0:
4: for =
do 5: for k == 0 N 1 do
6:
Sample 6t ~ (O, Calculate the approximate stochastic gradient: dx = No - -oe( + J.-1E0
7:
-i,,Et,,,t):
8:
Update momentum: Bv (1- 9) 3) Update Xo: X + PA,y (Xo - rTv);
9:
end for
11: end for
12: return Xo
I Visualization results of image restoration
31

---

y
Baseline
DDPS
RED-diff
Debiur
Impeiting
SIA
Figure 6: Noise-free results on ImageNet. Baseline means &o Aty.
GT
y
Bascline
DDRM
DDNM
DPS
RED-diff
Deblur
Impeiting
Figure 7: Noisy results on ImageNet Æ = 0.05. Baseline means &n = tty.
32

---

GT
y
Baseline
DDNM
DPS
RED-diff
Deblur
Impeiting
SRA
Figure 8: Noise-free results on CelebA. Baseline means En A y.
GT
y
Bascline
DDRM
DDNM
DPS
RED-diff
Doblur
Impeiting
SKAI
Figure 9: Noisy results on CelebA a = 0.05. Baseline means Xy = Aty.
33

---

GT
y
DPS
RED-diff
Ours
EDX
Phase retrieval
Figure 10: Noise-free nonlinear restoration on FFHQ (a 0).
34

---

GT
y
DPS
RED-diff
Ours
EDX
Phase retrieval
Figure 11: Noisy nonlinear restoration on FFHQ (a 0.05).
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are consistent with this paper's main contributions and scope, and have been verified through extensive experiments and ablation studies.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
See section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
36

---

Answer: [Yes]
Justification:
See Appendix A and B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The code is included in the supplemental materials and the experiment details are presented. See Appendix E and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
37

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and instructions are included in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//wpp...c//////// wttpcccccppppuuucceeeec gwwts//gggs/////eee / wwwssssiiiiieeeeeeeooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All of the experiment details and implemention details are presented. See Appendix and F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Following the convention of inverse problem research, we report the perfor- mance comparison of all algorithms with fixed random seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
38

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All of the experiments are conducted on single NVIDIA 3090ti GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://wnrrrssc.ppupeeeeeeeeeeee wttppppllc///aeccocccoccceocctct htwpo//iii/hhiisuuidieoeee ines?
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics in detail and conformed with it in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper focuses on designing algorithms for inverse problems, without underlying social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
39

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No datasets models proposed and there is such risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All databases, models, and toolkits used in this paper are cited from the original papers or URLs.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
40

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode, cow /ataattss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code provided in the supplementary materials is anonymized and well documented with the running details.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
41

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "gqUSL1jHRO", "reviewer": "Reviewer_g7gD", "summary": "This paper proposes ProjDiff for solving inverse problems with pre-trained diffusion models. By deriving a two-variable ELBO as a proxy for the log-prior, this paper reframes the inverse problems as constrained optimization tasks and address them via the projection gradient method.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper writing is clear and easy to follow.\n2. The derivation of some formulas in this paper is solid.", "weaknesses": "1. The assumption of this method for the measurement noise in Section 3.1 is doubtable. (1) As the authors claim in the limitation section, assuming the Gaussian noise will limit its applicability to other noise types like Poisson or multiplicative noise; (2) The whole method design heavily relies on know the exact standard deviation of the Gaussian noise \\sigma, which is impractical and can lead to robustnesses issues when facing unknown noise level. Actually, the noise estimation itself is a challenging problem, particularly on the degraded measurement A(x). Please check the related works [1,2]. In general, this method is built on an impractical scenario, so I am worried about the practical usage of this method.\n\n2. The approximation for implementing the update rules just after Proposition 2 is not convincing. (1) One of the reasons for this approximation is that \"the μ-predictor of the diffusion model should be resilient to small perturbations in the input\", but this hope usually is not the truth, particularly when t is large, i.e., at the beginning of the reverse sampling procedure; (2) There is not much evidence for this important approximation, no matter theoretical or empirical evidence.\n\n3. The proposed method is too \"delicate\", as shown in Algorithm 1. (1) Again, this method needs to know the measurement noise level; (2)\nthis method contains several hyper-parameters. In Table 17, it is clear that the hyper-parameters are heavily tuned, and there is no specific way to guide the hyper-parameter tuning.\n\n4. No ablation studies for the hyper-parameters. For this \"delicate\" method, a systematical ablation study is needed. For example, it would be good to report the results for different combinations of some important hyper-parameters.\n\n5. The extension to nonlinear inverse problems is not convincing. This paper proposes to use min_x || y - A(x) ||^2 to approximate the projection operator. However, it is almost impossible to solve nonlinear inverse problems by using this formulation because it will be easy to be stuck in local minimizers. Also, if this formulation can solve nonlinear inverse problems well, then there is no need to write this paper.\n\n6. The experiment settings for phase retrieval are doubtable. (1) I highly suspect that this paper downplays DPS in some implicit ways. I ran DPS for phase retrieval on FFHQ before, and remembered DPS could achieve over 30dB for PSNR after trying different initializations. In the original DPS paper, the authors claim that DPS need different initializations but it seems that this paper omits this; (2) there is no comparison with the golden standard method for phase retrieval, i.e., HIO+ER.\n\n7. Some recent SOTA methods are missing for comparison, including ReSample[3] (ICLR'24 spotlight), DiffPIR[4] (CVPR'23), DMPS[5]. Please check the recent survey for more related works [6].\n\n\n\n[1] Liu, X., Tanaka, M. and Okutomi, M., 2013. Single-image noise level estimation for blind denoising. IEEE transactions on image processing, 22(12), pp.5226-5237.\n\n[2] Li, F., Fang, F., Li, Z. and Zeng, T., 2023. Single image noise level estimation by artificial noise. Signal Processing, 213, p.109215.\n\n[3] Song, B., Kwon, S.M., Zhang, Z., Hu, X., Qu, Q. and Shen, L., 2023. Solving inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123.\n\n[4] Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R. and Van Gool, L., 2023. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1219-1229).\n\n[5] Meng, X. and Kabashima, Y., 2022. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12343.\n\n[6] Li, X., Ren, Y., Jin, X., Lan, C., Wang, X., Zeng, W., Wang, X. and Chen, Z., 2023. Diffusion Models for Image Restoration and Enhancement--A Comprehensive Survey. arXiv preprint arXiv:2308.09388.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's suggestions, and here are our responses.\n\n1. ''The assumption for the measurement noise is doubtable.''\n\nWe respectfully disagree. Firstly, in practical applications, noise-free inverse problems or inverse problems with Gaussian noise are ubiquitous, and the standard deviation of the Gaussian noise can be estimated using the method mentioned by the reviewer or retained as an adjustable hyperparameter. Secondly, a series of works  (including but not limited to [18, 20, 21, 22, 26, 27] in the main paper) that use diffusion models to solve the inverse problems assume additive Gaussian noise and known variance (DMPS mentioned by the reviewer also assumes additive Gaussian noise with known variance, and the DiffPIR mentioned by the reviewer also assumes the variance is known). We believe that these works, as well as ours, have practical value. Lastly, as shown in the ablation study in response to point 4, ProjDiff is robust to some perturbation of the standard deviation. Therefore, our work holds practical significance.\n\n2. ''The approximation after Proposition 2 is not convincing.''\n\nWe respectfully disagree. Firstly, this approximation is acceptable when $t$ is small, as the $\\\\mu$-predictor generally learns well to recover the clean image from a slightly noisy image. Secondly, for large $t$, in the case of VP diffusion, note that the coefficient $\\\\overline\\\\alpha_t$ in front of $x_0$ or $x_{t_a}$ is small, thus the gradients corresponding to $x_0$ and $x_{t_a}$ are also small. For VE diffusion, similarly, the dominant term in $\\\\mu$ will also be the noise term when $t$ is large, making it insensitive to the input $x_0$. Our experimental results have demonstrated the effectiveness of this approximation. Furthermore, we supplement a new experiment that uses the stochastic gradient without this approximation, as shown in Table 6 of the attached PDF (denoted as ProjDiff-FG). It can be observed that using this approximation results in a certain loss of performance, but the efficiency is nearly tripled. Considering that applying this approximation has already achieved satisfactory performance, we are willing to accept a certain performance loss in exchange for efficiency.\n\n3. ''The proposed method is too 'delicate'.''\n\nWe respectfully disagree. Once again, assuming knowledge of the variance of Gaussian noise is a default setting in many works within the field. Secondly, as we explain in Appendix F, we fix $\\\\eta_2=1.0$, thus only $\\\\eta_1$ needs to be tuned (as well as the equivalent noise level $\\\\overline \\\\alpha_{t_a}$ if we leave it as a hyperparameter to be tuned). We conduct hyperparameter tuning on the first eight images for each task. Given that the complexity of ProjDiff is not too high, tuning hyperparameters is not time-consuming. In the ablation study in response to point 4, we also demonstrate that ProjDiff is robust to some perturbation of the step size and the standard deviation (i.e. the equivalent noise level). To the best of our knowledge, many related works have at least one hyperparameter that needs to be adjusted, including the algorithms mentioned by the reviewer.\n\n4. Ablation studies for hyper-parameters \n\nWe supplement the ablation study concerning $\\\\eta_1$ and ${\\\\sigma}_0$ in Table 7 of the attached PDF. ProjDiff exhibits robustness to some perturbation of the hyperparameters of the step size and the noise standard deviation.\n\n5. ''The extension to nonlinear inverse problems is not convincing''.\n\nThe reviewer has misunderstood our approach. One of the core aspects of ProjDiff is to project a sample $x_0$ onto the manifold corresponding to $y=\\\\mathcal{A}(x)$. When $\\\\mathcal{A}$ is nonlinear whose projection operator is unavailable, one can instead find a solution that is close to $x_0$ and close to the observation manifold by solving $\\\\min_x||y-\\\\mathcal{A}(x)||^2$ starting from $x_0$. This solution can serve as an approximation for the projection operation corresponding to a nonlinear observation equation. We also supplement an experiment to validate the effectiveness of this method in the Phase Retrieval task (ProjDiff-GD in Table 4 of the attached PDF).\n\n6. ''The experiment settings for phase retrieval are doubtable.''\n\nWe respectfully disagree. Our experimental settings for phase retrieval are correct, fair, and reasonable. All of our experimental settings are detailed in Appendix E. We generate one sample per image for each algorithm. The good performance of ProjDiff under this setup further demonstrates its robustness. We also supplement the results of allowing ProjDiff and DPS to generate 4 samples for each image, as shown in Table 3 of the PDF (denoted as ProjDiff-4trials and DPS-4trials). Both DPS and ProjDiff show performance improvements, with ProjDiff in particular exhibiting better performance, reaching a PSNR of 41.58 in the noise-free case. The performance of HIO, ER, and OSS is also presented in Table 3 and the global response, where we report the results with both one repetition and four repetitions. \n\nNote that DPS does not achieve the PSNR of 30 as mentioned by the reviewer. To dispel further doubts about the correctness of our reproduction, we independently run the official code of DPS four times without any modifications (i.e., noise variance set to 0.05, with oversampling consistent with our experiments) and take the best result per sample, obtaining a PSNR of 17.45, indicating that our reproduction is also correct.\n\n7. Comparison with the latest SOTA. \n\nWe supplement comparisons of ProjDiff with ReSample, DiffPIR, DMPS, and $\\\\Pi$GDM on the CelebA dataset, as shown in Table 1 & 2 in the attached PDF. The hyperparameters are tuned task by task for fairness. ReSample is designed for Latent Diffusion, thus in our experiments, we consider the encoder-decoder as identity mappings. ProjDiff remains highly competitive when compared with these algorithms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LiGKaiGkbx", "reviewer": "Reviewer_Wp65", "summary": "The authors present a new way of solving inverse problems using a diffusion model based prior.\nThe key idea is that the forward process, which is to be undone, usually involves Gaussian noise.\nThe authors utilize this by viewing the noisy observation as a as random variable at an intermediate time step t of the diffusion forward process.\nThis allows them to formulate the inverse problem as a joint optimisation problem for x_0 and x_t given the noisy observation.\nAs a by product, their method is able to solve inverse problems which do not involve noise as well.\nThe authors evaluate their method and achieve superior results compared to various baselines.", "questions": "Regarding the SVD composition, I believe I am missing an important aspect about how the noisy observation can be considered as a step in the diffusion process when it involves A.\nLet's say A corresponds to a convolution with a blur kernel. Would this not mean that we are considering a blurred noisy image to be part of the diffusion process. This would be incorrect, because the blurring would not be part of the training data statistics which would only include clean images, right?", "limitations": "My understanding is that the method is limited to either image degradation processes that involve Gaissian noise, or are noise-free.\nIs this the case?\nIf yes, this would be a significant limitation with respect to practical applicability, e.g. in settings with Poisson shot noise.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* I believe this is a principled new perspective on diffusion models for inverse problems and I am sure it can be potentially useful for many practical applications.\n\n* The method achieves remarkable results.\n\n* I very much appreciate the fact that the method can be applied to noise-free inverse problems, without having to artificially include non-existent Gaussian noise in the model.", "weaknesses": "* The motivation, comparing the method to the more established MAP approach, could be a bit more clear.\nThe authors write: \n\"In this work, we introduce a novel two-variable ELBO by constructing an auxiliary variable that accounts for the observation noise, thereby utilizing both the prior information and the denoising capability in diffusion models simultaneously.\"\nWhy exactly is is describable to utilize the denoising capability of the model? Is the something wrong with the normal way of including the log of the forward model in the optimisation task? Why can we expect an improvement? Is it a problem of optimisation? Surely the way task itself is normally (e.g. [28]) formulated is correctly, or not?\n\n* Some aspects of the method were difficult for me to follow. This may be due to my inexperience with some of the involved mathematics.\nIn particular, I wonder if the SVD based decoupling could be explained or illustrated differently to facilitate understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's recognition of the innovation, effectiveness, and practicality of our work. Below are our responses to the reviewer's questions.\n\n1. Clarification on the motivation and comparison to the established MAP approach. \n\nThe core motivation of this paper stems from the fact that diffusion models not only model the distribution of clean data $p(x_0)$, but also simultaneously model the distribution of noisy data with varying variances of additive Gaussian noise $p(x_t)$. When observing a degraded data with Gaussian noise, the noisy data distribution can bring more usable information for solving the inverse problem. Therefore, compared to using only the prior information modeled by the diffusion model for $x_0$, we choose to use the prior information of both $x_0$ and $x_{t_a}$. Our algorithm can be viewed as two steps: first, recover the noisy sample $x_{t_a}$, and then use the consistency constraints of $x_{t_a}$ and the prior of $x_0$ to recover $x_0$ (which can be viewed as a denoising process). In other words, we utilize both the prior modeled by the diffusion model and its denoising capabilities.\n\nThe conventional approach of including the logarithm of the forward model in the optimization task is also feasible, as seen in [24, 28] in the main paper, and we have compared with it in our experiments (RED-diff). The improvement of our algorithm compared to these methods can be expected because our method utilizes more information modeled by the diffusion model, i.e., it uses both the prior and denoising capabilities (or both the clean data prior and the noisy data prior). We believe this is the essential difference between our algorithm and the methods that include the log of the forward model and $\\\\log p(x_0)$ modeled by diffusion models in the MAP framework.\n\n2. Further explanation on SVD-based decoupling. \n\nHere we provide additional clarification regarding the description in section 2.3. Firstly, since $V$ is orthogonal, we acknowledge that knowing the prior of $x_0$ is equivalent to knowing the prior of $\\\\overline x_0=V^T x_0$. Secondly, since the covariance matrix of both the forward transition and a single-step backward transition of the diffusion model is the diagonal matrix, each coordinate can be processed individually. Equation (7) presents the observation equation in a coordinate-wise form, where for each coordinate index $i$, the $i$-th coordinate of the noisy observation $\\\\overline y_i$ can be considered as a noise-free observation of the $i$-th coordinate of the sample $\\\\overline x_{t_i}$ at step $t_i$. In other words, each coordinate of the noisy observation (after spectral decomposition) can be considered as a noise-free observation of the coordinate at a certain step in the diffusion process. \n\nRegarding the reviewer's concern about a convolution with a blur kernel, it is not about viewing the noisy blurred image as a step in the diffusion process, but rather, after SVD, each coordinate is treated as the noise-free observation of the coordinate corresponding to a certain step in the diffusion model. We hope the above explanation could describe our method more clearly. Thanks for the reviewer's suggestions, and we will make further modifications and explanations regarding the SVD part of the paper.\n\n3. Regarding the limitations.\n\nYes, the limitation of this method is that it can only handle noise-free inverse problems or inverse problems with additive Gaussian noise. Addressing other types of noise, such as Poisson noise or multiplicative noise, is a promising direction for further research. In practical applications, noise-free inverse problems or inverse problems with additive Gaussian noise are widespread, hence there is a series of work in the field of solving inverse problems with diffusion models that assumes Gaussian noise (including but not limited to [18, 20, 21, 22, 26, 27] in the main paper). Therefore, our work holds practical significance.\n\nWe would like to express our gratitude once again for the reviewer's suggestions for our work. Should there be any further questions, we welcome continued discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgSUEBpFp1", "reviewer": "Reviewer_dXRm", "summary": "This paper proposed a new sampling strategy for solving noisy inverse problems using diffusion models. The proposed method is called ProjDiff, which is based on two-step minimization of log-posterior using gradient descent. The sampling procedure is derived by (1) Reparametrization of the noisy measurement as a noiseless measurement of an intermediate noisy sample in the diffusion process, and (2) obtaining a proxy objective from the variational lower bound.", "questions": "What does the author mean by weak observations? (Line 214)\n\nHow does the algorithm look when used for nonlinear inverse problems? Can equation (8) be written for nonlinear measurements?\n\n\nHow do the authors set the $\\eta_1$, $\\eta_2$, and equivalent noise level in the algorithms?\nAre the solutions sensitive to these parameters?", "limitations": "Partially addressed by the authors.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The paper's approach to addressing the noisy inverse problem using the reparamterization of measurement is interesting. \n\n+ Deriving the sampling procedure by directly minimizing the evidence lower bound is innovative. \n\n+ The performance is competitive with the SOTA methods in solving inverse problems.", "weaknesses": "+ The paper is not well-written and it is hard to follow. Justification: \n\n     1. Theoretical analysis results lack organization. Example: Lemma 1 and Lemma 2 should be separated from the main proof. The proofs for the propositions should also be separated and distinctly expressed. \n\n     2. Theoretical results should be explicitly included. The proofs for Lemma 1 and Lemma 2 are not included in the text. If taken from a source, then the reference should be mentioned, otherwise, the proofs should be included. \n\n    3. Abstract includes some concerning statements.\n        >Since inverse problems inherently entail maximum a posteriori estimation, previous works have endeavored to integrate diffusion priors into the optimization frameworks.\n\n        Inverse problems could be solved using MMSE estimation. It is unclear what kind of optimization algorithm the authors refer to. \n       \n        > by introducing an auxiliary optimization variable. By employing gradient truncation, the projection gradient descent method is efficiently utilized to solve the corresponding optimization problem. \n\n        The abstract does not reflect the method used.  The statement is vague and could be inferred in many different ways. \n\n       Revising the abstract to better reflect the proposed method can strengthen the paper.\n\n  4. The second and third paragraph of the introduction needs to be revised. Instead of briefly mentioning what each existing method does, the authors could state how their method is different from the literature and how it contributes to the literature. \n\n\n\n* The truncation of the gradient is based on an assumption and leads to an approximation in the solutions. Thus, the reduced computational cost should not be mentioned as a contribution of the paper, specifically when the authors have not included any experimental results regarding this. \n > 44: Through gradient truncation, we obtain an approximation of the stochastic gradient of the objective, effectively sidestepping significant computational overhead.\n\n\nOverall, the weaknesses of the paper are mostly related to the presentation of the paper and the blurred message, not the proposed method and effectiveness of it. \n\nMinor issues: \n\n> Line 19: Their remarkable ability to capture data priors enables effective guidance for solving inverse problems [6], which are widely exploited in image restoration. \n\nGuidance in inverse problems usually refers to enforcing data consistency with the measurement and relies on the forward model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's recognition of the innovation and effectiveness of our work. Below are our responses.\n\n1. Reorganizing the theoretical analysis and the proofs of the lemmas. \n\nThanks for the reviewer's suggestion. We will move Lemma 1 and 2 ahead of the proofs of the Propositions, and we will present the proofs of the two Propositions in a clear and independent manner. The proofs of Lemma 1 and 2 are based on Bayes' theorem and mathematical induction, and we will supplement the proofs of this part.\n\n2. Abstract includes some concerning statements. \n\nThanks for the reviewer's feedback. We will revise the abstract accordingly.\n\nThe first sentence will be revised as:\n\n''Previous works have endeavored to integrate diffusion priors into the maximum a posteriori estimation (MAP) framework and design optimization methods to solve the inverse problem.''\n\nThe second sentence will be revised as:\n\n''... by introducing an auxiliary optimization variable that represents a 'noisy' sample at an equivalent denoising step. The projection gradient descent method is efficiently utilized to solve the corresponding optimization problem by truncating the gradient through the $\\\\mu$-predictor.''\n\nWe will further refine the abstract to make it more fluent and reflective of our approach.\n\n3. The authors could state how their method is different from the literature and how it contributes to the literature. \n\nCompared to previous works that apply diffusion in the MAP framework for solving inverse problems, our core observation is that diffusion models not only model the distribution of clean data but also implicitly model the distribution of noisy data. This allows us to leverage both the clean data prior and the noisy data prior to solve noisy inverse problems (in other words, both the prior modeled by the diffusion model and its denoising capabilities). Thanks for the reviewer's suggestions, and we will further improve the expression of paragraph 3 and paragraph 4.\n\n4. The reduced computational cost\n\nThanks for the reviewer's comments. We did not intend to present the reduction of complexity as a contribution of this paper. Line 44 will be revised to:\n\n''We obtain a more practical approximation of the stochastic gradient of the objective through gradient truncation.''\n\nAdditionally, we have added an experiment comparing the use of our gradient approximation with not using it, as shown in Table 6 of the attached PDF. It can be observed that using the approximation results in a certain loss of performance, but the efficiency is nearly tripled. Considering that the method using this approximation has already achieved satisfactory performance, we accept a certain performance loss in exchange for efficiency.\n\n5. Line 19.\n\nThanks for the reviewer's suggestions. Line 19 will be revised to\n\n''Their remarkable ability to capture data priors provides promising avenues for solving inverse problems [6]...''\n\n6. The weak observations problems.\n\nWe refer to inverse problems where the constraints provided by the observation are relatively loose as ''the weak observation problem''. For instance, the partial generation task in this paper aims to generate tracks for other instruments based on certain tracks (e.g., generating tracks of drums, bass, and guitar given a track of piano). This problem is highly flexible as a composer can create various different tracks for other instruments from the same piano track while ensuring harmony. This is a case belonging to the ''the weak observation problems''. In contrast, tasks such as super-resolution, inpainting, and deblurring in image restoration are cases where the observation strongly constrains the original data. For example, given a low-resolution image, its corresponding high-resolution image is almost unique, with lower degrees of freedom, and thus it does not belong to the ''weak observation problems''.\n\n7. Regarding nonlinear measurements\n\nThe algorithm for nonlinear measurements is almost identical, except that the projection operations in equations (19) and (22) are replaced with the projection operations corresponding to nonlinear measurements, or by minimizing $||y-\\\\mathcal{A}(x))||^2$ from the initial point to approximate the projection operation.\n\nFor general nonlinear measurements, equation (8) cannot be strictly written. However, we can still apply a similar idea to handle nonlinear measurements, which is to simply find an equivalent noise level to deal with noisy observations. This equivalent noise level can either be calculated using rules or retained as a hyperparameter to adjust. Our Phase Retrieval experiments and HDR experiments have verified the effectiveness of our method.\n\n8. The hyperparameters\n\nAll of the details of our hyperparameters are provided in Appendix E and F. Regarding the sensitivity of hyperparameters, we supplement a new experiment in Table 7 of the attached PDF. It can be observed that our algorithm exhibits robustness to the step size and the noise variance.\n\nWe would like to express our gratitude once again for the reviewer's suggestions for our work. Should there be any further questions, we welcome continued discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lWeev3WTkM", "reviewer": "Reviewer_krnF", "summary": "Authors propose ProjDiff which reframes noisy inverse problems with diffusion models as a two-variable constrained optimization by introducing an auxiliary optimization variable. Authors derive a two-variable ELBO as a proxy for the log-prior and solve the optimization problem via projection gradient descent. Authors conduct comprehensive experiments on several image restoration tasks (super-resolution, inpainting and deblurring), source separation, and partial generations tasks.", "questions": "* I have several questions regarding DDNM numbers. First of all, DDNM [1] is designed for noise-free image reconstruction problems. I believe that is the reason why DDNM on Gaussian deblurring performs bad (as in Table 1: 7db PSNR, 0.03 SSIM, etc.). I would suggest the authors to switch to the DDNM+ variant as described in [1] for noisy inverse problems. If authors are already using DDNM+, could you clarify why the performance is bad on Gaussian deblurring?\n* line 37-39: \"However, it’s worth noting that, since diffusion models are inherently effective at denoising, considering the observation noise in the likelihood term fails to fully leverage diffusion models’ denoising capability.\" Could the authors clarify what it means to not fully leverage diffusion models here?\n* I would recommend the authors to include $\\Pi$GDM [2] in their comparisons especially since it performs much better than DDRM.\n* Some other missing citations on solving inverse problems with diffusion models: CCDF [3], latent diffusion models: PSLD [4]\n* line 663-665: \"Since [23] did not conduct experiments on CelebA, we use the parameters on FFHQ for the CelebA dataset as both FFHQ and CelebA are datasets of faces.\". I don't think this is a good practice. In my experience, even though both CelebA and FFHQ are face datasets, DPS is not robust to the choice of step size. I believe some hyper-parameter search on a small set of images is necessary for fair comparison.\n* Do you think ProjDiff can be extended for using latent diffusion models as a prior?\n---\nReferences:\n\n[1] Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-shot image restoration using denoising diffusion null-space model.\" arXiv preprint arXiv:2212.00490 (2022).\n\n[2] Song, Jiaming et al. “Pseudoinverse-Guided Diffusion Models for Inverse Problems.” International Conference on Learning Representations (2023).\n\n[3] Chung, Hyungjin, Byeongsu Sim, and Jong Chul Ye. \"Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[4] Rout, Litu, et al. \"Solving linear inverse problems provably via posterior sampling with latent diffusion models.\" Advances in Neural Information Processing Systems 36 (2024).", "limitations": "Limitations are adequately addressed by the authors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Overall, paper is easy to follow and structured nicely.\n* I appreciate extensive numerical results that are presented in the paper.\n* Numbers for source separation look significantly better than the baselines (I have to note that I'm not an expert in that area)", "weaknesses": "* Prior work section is not comprehensive.\n* The gains in performance compared to baseline methods (especially for noisy restoration tasks) is not convincing. Moreover, hyperparameter search for baseline models were not conducted and wrong baseline method was used (DDNM for measurements with noise).\n* See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's affirmation of the effectiveness of our method, especially the outstanding performance in music separation and partial generation tasks. Below are our responses.\n\n1. DDNM+ numbers for noisy Deblurring tasks\n\nWe are very grateful to the reviewer for pointing out this issue. We used the official code of DDNM in the noisy linear inverse problem. We double-checked the official code and confirmed that it indeed calls the DDNM+ algorithm. However, we found that the implementation of Gaussian Deblurring in the official code seems to be incomplete, which is the reason for the abnormal results in our paper. \n\nWe re-implement the part of Gaussian Deblurring and test it on both ImageNet and CelebA. The corrected results are shown in Table 5 of the attached PDF, where DDNM+ achieved reasonable results. We also checked Super-resolution and Inpainting and confirmed that their implementations are correct. We will correct the experimental results in our paper and annotate that we used the DDNM+ algorithm for noisy tasks. We would like to express our gratitude once again to the reviewer for helping us correct this error.\n\n2. ''What it means to not fully leverage diffusion models''\n\nOur observation is that diffusion models not only model the distribution of clean data $p(x_0)$, but they also implicitly model the distribution of noisy data with different variances of Gaussian noise $p(x_t)$ within the diffusion process. This means that compared to other types of data priors, diffusion models can provide more usable information for solving inverse problems with Gaussian noisy observations. Therefore, if solved within a conventional MAP framework, only the clean data prior modeled by the diffusion model would be utilized while the noisy data prior within the diffusion model is neglected.\n\nIn contrast, we introduce an auxiliary variable to transform the noisy observation into an equivalent noise-free observation of the noisy sample, thereby utilizing both the clean data prior and the noise data prior. The process of recovering clean data from the noisy auxiliary variable can be regarded as a denoising process, hence our method actually utilizes both the data prior modeled by the diffusion model and its denoising capability.\n\n3. More comparative algorithms and related works \n\nThanks for the reviewer's suggestion. We supplement a comparison of ProjDiff with $\\\\Pi$GDM, together with ReSample, DiffPIR, and DMPS on the CelebA dataset. The results are shown in Table 1 & 2 of the attached PDF. ProjDiff still demonstrates competitive performance.\n\nFor $\\\\Pi$GDM, we carefully conducted hyperparameter searches for each task using the first eight images with the average PSNR as the metric. Except for Noisy Gaussian Deblur task, we avoided using other algorithms to initialize $\\\\Pi$GDM to ensure a fair comparison. Specifically, we first use the forward transition to map the degraded image to the noise level corresponding to the 500th step, and then use 100 steps of $\\\\Pi$GDM to solve the inverse problem. We find that this approach is much better than starting directly from white noise (i.e. 1000th step). For Noisy Gaussian Deblur task, $\\\\Pi$GDM without initialization from other algorithms struggled to achieve satisfactory results. Therefore, we use DDNM+ to obtain the sample at the 500th step and then switch to $\\\\Pi$GDM to complete the solution.\n\nWe notice that $\\\\Pi$GDM does not always outperform the performance of our reproduced DDRM. This may be because we report the performance of DDRM using 100 steps, which is much better than the default parameters of DDRM (20 steps).\n\nWe will further refine the explanation of related work according to the reviewer's suggestions.\n\n4. DPS on CelebA\n\nThanks for the reviewer's suggestion. We have re-adjusted the learning rate hyperparameter $\\\\eta$ for DPS on the CelebA dataset task by task. Using the average PSNR of the first eight images as the criterion (consistent with the method we used to adjust the hyperparameter of ProjDiff), we carefully scanned the learning rate hyperparameter with a precision of 0.1. The latest results are shown in Table 1&2 of the attached PDF (denoted as DPS-s). We will update these results in the paper and include an explanation.\n\n5. ProjDiff for latent diffusion\n\nYes, we believe ProjDiff can be applied in latent diffusion. The main challenge in solving inverse problems with latent diffusion priors is the high nonlinearity of the observation equation introduced by the neural encoder-decoder. ProjDiff has the capability to handle nonlinear observations. Firstly, for noise-free observations, ProjDiff can approximate the projection operation by taking $x_0$ as the initial point and minimizing $||y-\\\\mathcal{A}(\\\\mathcal{D}(x))||^2$ (where $\\\\mathcal{D}$ represents the neural Decoder). Secondly, for noisy observations, ProjDiff also needs to set an equivalent noise level, which can be left as an adjustable hyperparameter. Our supplemental experiments in Table 7 of the attached PDF have verified that ProjDiff is robust to some perturbations in the noise variance (i.e. equivalent noise level), making manual adjustment of the equivalent noise level feasible. We believe this is a promising direction for future work.\n\nWe would like to express our gratitude once again for the reviewer's suggestions for our work. Should there be any further questions, we welcome continued discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Please refer to the attached one-page PDF for the supplementary experimental results.

We appreciate the valuable feedback provided by all the reviewers on our paper, which has helped us to further refine our work. We are particularly encouraged by the following comments from the reviewers:

(1) ''This is a principled new perspective on diffusion models for inverse problems'' and ''it can be potentially useful for many practical applications'' (Reviewer Wp65)

(2) ''The performance is competitive'' (Reviewer dXRm)

(3) ''extensive numerical results'' and ''Numbers for source separation look significantly better than the baselines'' (Reviewer krnF)

Based on the reviewers' suggestions and advice, we have diligently conducted a substantial number of additional experiments. Please refer to the attached one-page PDF for the supplementary experimental results. Below, we provide a summary and explanation of the supplementary results.

Supplementary experiments include (all LPIPS metrics have been multiplied by 100 for clarity):

(1) New comparisons with $\\Pi$GDM, Resample, DMPS, and DiffPIR on the CelebA dataset (Tables 1 & 2), as suggested by reviewers g7gD and krnF. We have carefully adjusted the parameters task by task for the four algorithms to ensure a fair comparison. The results indicate that ProjDiff is highly competitive compared to these algorithms.

(2) Results of DPS with a task-by-task parameter tuning on the CelebA dataset (DPS-s in Tables 1 & 2). We are grateful for the suggestion from reviewer krnF. We have conducted task-by-task parameter tuning for the learning rate in DPS on the CelebA dataset and reported the results, ensuring a fair comparison. The results show that ProjDiff still holds an advantage over DPS after parameter tuning.

(3) The results for DPS and ProjDiff in the Phase Retrieval task by taking the best one from 4 independent trials for each image (DPS-4trials and ProjDiff-4trials in Table 3), in response to the question raised by reviewer g7gD. With 4 repetitions, ProjDiff shows a very significant improvement in performance, achieves a PSNR of 41.58 in the noise-free scenario, and still holds a considerable advantage over DPS.

(4) Comparisons with ER, HIO, and OSS algorithms on the Phase Retrieval task (Table 3), as suggested by reviewer g7gD. Due to space constraints, only the results with four repetitions are reported in Table 3. We have also tested the results of these three algorithms with one repetition, as shown in the following table.


|            |                Phase Retrieval ($\sigma_0=0$)                |               Phase Retrieval ($\sigma_0=0.1$)               |
| :--------: | :----------------------------------------------------------: | :----------------------------------------------------------: |
|            | PSNR $\uparrow$ SSIM $\uparrow$ LPIPS $\downarrow$ FID $\downarrow$ | PSNR $\uparrow$ SSIM $\uparrow$ LPIPS $\downarrow$ FID $\downarrow$ |
| ER-1trial  |                11.15 / 0.19 / 84.48 / 409.91                 |                11.16 / 0.19 / 84.43 / 412.59                 |
| HIO-1trial |                11.97 / 0.25 / 82.06 / 342.52                 |                11.87 / 0.24 / 82.18 / 339.13                 |
| OSS-1trial |                12.57 / 0.35 / 81.08 / 360.98                 |                12.55 / 0.25 / 81.20 / 364.52                 |

(5) The performance of ProjDiff in the Phase Retrieval task using gradient descent to approximate the projection operation (ProjDiff-GD in Table 4), in response to the question from reviewer g7gD. The results indicate that approximating the projection operator of $x_0$ with respect to the observation equation by minimizing $||y-\\mathcal{A}(x)||^2$ starting from $x_0$ can also achieve satisfactory performance, which suggests that ProjDiff has generalizability for nonlinear inverse problems.

(6) Corrected metrics for the DDNM+ algorithm on the Noisy Gaussian Deblurring task (Table 5). We appreciate the reviewer krnF for pointing out the issue with the metrics. We have re-implemented the DDNM+ algorithm on the Noisy Gaussian Deblurring task and reported the corrected performance.

(7)  Ablation study on the gradient truncation method used in ProjDiff (Table 6), in response to the questions from reviewers g7gD and dXRm. ProjDiff without gradient truncation is denoted as ProjDiff-FG. The results show that there is some performance loss when using gradient truncation compared to not using it, while the efficiency increases by nearly three times. Considering that using gradient truncation has already achieved satisfactory performance, we accept the trade-off of some performance loss for the gain in efficiency.

(8) Sensitivity tests for the hyperparameters $\\eta_1$ and noise level $\\sigma_0$ in ProjDiff (Table 7), as suggested by reviewers g7gD and dXRm. '$\\times a$' denotes that we perturb the input standard deviation of ProjDiff by multiplying $a$ (this will affect the equivalent noise level). The results indicate that ProjDiff exhibits a certain degree of robustness to the step size and noise level. This also demonstrates that retaining the equivalent noise level as an adjustable hyperparameter is feasible when it cannot be directly calculated.

We would like to express our gratitude to all the reviewers once again, and we believe that these additional experiments would significantly contribute to the improvement of our manuscript.

Meta-review: The paper propose a method for solving inverse problems using diffusion models. They reformulate the log-prior using a two-variable method and use projected gradients to solve the resulting constrained optimization problem. The method is applied to super-resolution, inpainting and deblurring tasks. Three reviewers positively rated the paper, with one reviewer providing a very negative response. During the discussion period, two of the positive reviews increased their rating, while the negative reviewer didn't engage in the discussion phase. It is not clear how significant or real the negative reviewers' concerns are, and I'm inclined to give the three combined positive responses more weight. They found the quantitative results to be convincing and the method to be in line with state-of-the-art approaches and so something that could be useful to the NeurIPS audience.
Decision: Accept (poster)