Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

Authors: jiawei  zhang
jiaxin  zhuang
jin  cheng
gen  li
yuantao  gu
cheng  jin
china gen li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 08:24:26

Abstract: The recent emergence of diffusion models has significantly advanced the precision of learnable priors, presenting innovative avenues for addressing inverse problems. Previous works have endeavored to integrate diffusion priors into the maximum a posteriori estimation (MAP) framework and design optimization methods to solve the inverse problem. However, prevailing optimization-based rithms primarily exploit the prior information within the diffusion models while neglecting their denoising capability. To bridge this gap, this work leverages the diffusion process to reframe noisy inverse problems as a two-variable constrained optimization task by introducing an auxiliary optimization variable that represents a 'noisy' sample at an equivalent denoising step. The projection gradient descent method is efficiently utilized to solve the corresponding optimization problem by truncating the gradient through the $\mu$-predictor. The proposed algorithm, termed ProjDiff, effectively harnesses the prior information and the denoising capability of a pre-trained diffusion model within the optimization framework. Extensive experiments on the image restoration tasks and source separation and partial generation tasks demonstrate that ProjDiff exhibits superior performance across various linear and nonlinear inverse problems, highlighting its potential for practical applications. Code is available at https://github.com/weigerzan/ProjDiff/.

Main paper: 
 introduction: 
 Denoising diffusion models have achieved tremendous success in the field of generative modeling [1] [2] [3] [4] [5] . Their remarkable ability to capture data priors provides promising avenues for solving inverse problems [6] , which are widely exploited in image restoration [7] [8] [9] [10] [11] [12] , medical image processing [13, 14] , 3D vision [15] , audio processing [16, 17] and beyond.Numerous endeavors have sought to harness diffusion models to address inverse problems [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] . Since the sampling process of diffusion models is a reverse Markov chain, most approaches attempt to integrate the guidance provided by the observation equation into the sampling chain. For instance, DDRM [20] achieves favorable results for linear inverse problems with low complexity by introducing a new variational distribution. DDNM [21] capitalizes on the concept of nullrange decomposition, effectively rectifying the range-space component of the intermediate steps by Figure 1 : Framework of ProjDiff. We introduce an auxiliary variable x ta and transform the inverse problem into a two-variable constrained optimization problem which can be solved using the projection gradient method. leveraging the observation information. DPS [23] guides the generation process using the gradients of intermediate steps with respect to the observation error. Moreover, methods based on Monte Carlo Particle Filtering [25] [26] [27] are employed to approximate the true posterior.Since inverse problems are often modeled as maximum a posteriori (MAP) estimations, recent efforts have been made to explicitly integrate diffusion models as the prior term in the optimization framework [28, 24] . [28] proposed to employ the Evidence Lower Bound (ELBO) to approximate the prior, facilitating the application of diffusion models in inverse problems in a plug-and-play manner. [24] achieved analogous results from a variational perspective. In these optimizationbased methods, due to the presence of the observation noise, a common practice is to adopt a Gaussian likelihood. However, it's worth noting that, since diffusion models are inherently effective at denoising, considering the observation noise in the likelihood term fails to fully leverage diffusion models' denoising capability.To fully utilize both the prior information and the denoising capability inherent in diffusion models within the optimization framework, we introduce an auxiliary optimization variable to accommodate the influence of the observation noise. We derive a novel two-variable objective based on the properties of the diffusion process and transform the inverse problem into a constrained optimization task. Through gradient truncation, we obtain a more practical approximation of the stochastic gradient of the objective which sidesteps significant computational overhead. The proposed algorithm, termed ProjDiff, tackles the inverse problem by employing the concept of projection gradient descent to solve the corresponding optimization task. We also discuss the noise-free version of ProjDiff as a special case, which, compared to other optimization-based methods, ensures superior consistency between the generated results and observations. ProjDiff's applicability also extends beyond linear observations to encompass nonlinear functions, which enhances its competitiveness and versatility.We demonstrate the outstanding performance of ProjDiff through comprehensive experiments on various benchmarks. In both linear and nonlinear image restoration tasks, ProjDiff exhibites superior performance among existing state-of-the-art (SOTA) algorithms. In the music separation task, ProjDiff shows for the first time, to the best of our knowledge, that a diffusion-based separation algorithm can surpass previous SOTA, which further demonstrates the powerful potential of diffusion models and provides insights to better harness their capabilities in inverse problems for future research. 
denoising diffusion models: 
 Denoising diffusion models [1] [2] [3] are a class of latent variable generative models tailored to capture a targeted data distribution q(x). Diffusion models typically predefine a forward process characterized as a Markov chain, wherein the transition probability is stipulated as Gaussian distribution: q(x 0:T ) = q(x 0 ) T t=1 q(x t |x t-1 ), q(x t |x t-1 ) = N (x t ; a t x t-1 , b t I),where a t and b t represent the scale and variance parameters, respectively. The sampling process of diffusion models aims to invert the forward chain via a parameterized reverse process, which is also modeled as a Markov chain with learnable parameter θ:EQUATIONwhere f θ denotes the mean function and the variance σ2 t is predefined. In this work, we primarily focus on the Variance Preservation (VP) diffusion [19] 1 , characterized by parameters a t = √ α t , b t = 1-α t with α 0 ≈ 1 and T t=0 α t ≈ 0. With these parameters, we have q(x t |x 0 ) = N (x t ; √ α t x 0 , (1α t )I), where α t = t i=0 α t . The initial distribution of the reverse process is set to match the forward process, i.e., p θ (x T ) = N (0, I), while the mean function is chosen asEQUATIONHere µ θ serves as an estimation of x 0 at time t, typically implemented through a parameterized neural network referred to as a "µ-predictor" 2 . The variance σ 2 t ∈ [0, 1 -α t-1 ] can take various values. When σ t = 1-αt-1 1-αt 1 -αt αt-1 , the reverse process aligns with DDPM [2] ; while when retaining σ t = σt as an adjustable parameter, one obtains DDIM [3] . We refer to [19, [29] [30] [31] for more theoretical guarantees of diffusion models.As stated in [32, 28] , for a well-trained diffusion model, the Evidence Lower Bound (ELBO) can effectively approximate the log-likelihood of the samples, i.e. log q(x 0 ) ≈ C -EQUATIONwhere λ(t) represents the coefficient within the ELBO and C is a constant. Utilizing the ELBO as the log-prior term enables diffusion models to serve as a plug-and-play prior for general inverse problems. [28] handled the ELBO via stochastic gradient descent and the reparameterization method. By sampling t ∼ U {1, 2, . . . , T } and ϵ ∼ N (0, I), a stochastic gradient of the ELBO can be derived asEQUATIONwhere D x0 µ θ (•) denotes the Jocabian of the µ-predictor. [24] derived a similar proxy objective from the perspective of variational inference, and proposed a more computationally efficient approximation of the stochastic gradient by reweighting the objective. Notably, when the variational distribution is a delta function, variational inference aligns with the MAP estimation. In this work, we introduce a novel two-variable ELBO by constructing an auxiliary variable that accounts for the observation noise, thereby utilizing both the prior information and the denoising capability in diffusion models simultaneously.As stated in [26] , the noisy observation of a sample can be regarded as a noise-free observation of an equivalent noisy sample at certain steps in the diffusion process. For linear observations y = Ax 0 +σn, where x 0 ∈ R mx , y ∈ R my , and n ∈ N (0, I my ), a common practice [20, 21, 26] involves the Singular Value Decomposition (SVD) to attain a decoupled form. Without loss of generality, assume A has full row rank. Utilizing the SVD A = USV T yields U T y = S(V T x 0 ) + σn, thus transforming the observation equation to y = Sx 0 + σn, where y = U T y and x 0 = V T x 0 , which can be further written element-wise asEQUATIONwhere x 0,i and y i represent the ith component of x 0 and y, respectively, and s i denotes the ith singular value of A. Assuming there existt i ∈ {1, 2, . . . , T } , 1 ≤ i ≤ m y such that α ti = 1/(1 + (σ/s i ) 2 ), let y ′ i =√ α ti y i /s i and then the observation function can be rewritten asy ′ i = α ti x 0,i + 1 -α ti n i . (7) √ α ti x 0,i + √ 1 -α ti n ican be interpreted as the ith component of a sample at time step t i , namely x ti,i . Thus [26] proposed to employ Monte Carlo Particle Filtering to restore the samples up to the equivalent noise level, and then apply the backward transition to map these noisy samples back to the clean samples. In this work, we introduce this equivalent noisy sample as an auxiliary variable, thus better handling the observation noise in the optimization framework.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we introduce the ProjDiff algorithm. The main idea behind ProjDiff lies in constructing an auxiliary variable to align the noisy observations to specific steps of the diffusion process, thereby forming a two-variable constrained optimization task, which can be tackled by approximating the stochastic gradients and employing the projection gradient descent method. We begin by deriving the ProjDiff algorithm for noisy linear observations and then discuss its noise-free version as a special case and the extension of ProjDiff to accommodate nonlinear observations.Consider the Gaussian observation equation y = Ax 0 + σn. The goal of the inverse problem is to recover the original data x 0 given y, A, and σ. The decoupled observation equation in (7) indicates that one can apply SVD to reduce a linear observation into the simple form with A = [I my , 0] in the spectral domain, which implies the observation y represents the first m y components of x. Consequently, we can focus on this simplest observation function to declare our algorithm. Under this condition, assuming there exists some t a ∈ [0, 1, . . . , T ]3 such that α ta = 1/ 1 + σ 2 , we rewrite the observation function asEQUATIONwhere y ′ = y/ √ 1 + σ 2 and n ′ ∼ N (0, I mx ). The noisy observation y ′ can be interpreted as a noise-free observation of √ α ta x 0 + √ 1 -α ta n ′ , which in turn can be viewed as a sample in the manifold of q(x ta ). Thus, we introduce an auxiliary variable x ta which denotes a noisy sample at time step t a , and propose to optimize the log-posterior term of the two variables asEQUATIONEQUATIONwhere in (10) we use p θ to approximate the true prior q. By the construction of the auxiliary variable x ta , the likelihood term embodies a stringent consistency between this auxiliary variable and the noisy observation, which serves as a constraint. Note that log q(y) is independent of the optimization variables, thus (10) is transformed into the following constrained optimization task:EQUATIONNow we seek the ELBO as a proxy of the joint log-prior term of x 0 and x ta to render it tractable. Leveraging Jensen's inequality and the transition probability of the diffusion model, we reach the following proposition.Proposition 1. Considering the DDIM reference distribution q σ , we have the variational lower bound of the log-prior term aslog p θ (x 0 , x ta ) ≥ C - ta t=1 E qσ(xt|x0,xt a ) g(t)||x 0 -µ θ (x t , t)|| 2(1) denoising matching termEQUATIONnoisy prior termEQUATION)where C is a constant independent of x 0 and x ta , the weight g(t), w(t) are functions of α t and the DDIM variance σt , and ν = x t -αt αt a x ta / 1 -αt αt a .The complete derivation of the lower bound is provided in Appendix A. Note that the summation on the right-hand side of ( 12) is partitioned into two parts. The first part, the denoising matching term, pertains to the approximation of log p θ (x 0 |x ta ), signifying the consistency between the noise-free variable x 0 and the auxiliary variable x ta . The second part, the noisy prior term, corresponds to the approximation of the log-prior of the noisy auxiliary variable x ta . Conceptually, the workflow of utilizing this ELBO for inverse problems can be delineated as follows: the noisy prior term, coupled with the constraint on the auxiliary variable, exploits the prior information of the diffusion model and the observation information to recover a noisy sample x ta that satisfies the observation equation, while the denoising matching term leverages the denoising capability of the diffusion model to restore the clean sample x 0 from the noisy sample x ta . Now, we address the ELBO outlined in Proposition 1 utilizing the principle of the stochastic gradient method. The constant C is independent of the optimization variables and can therefore be disregarded. The following proposition is obtained through reparameterization. Proposition 2. Randomly sampling t ∼ U {1, 2, . . . , T } , ϵ, ϵ ′ ∼ N (0, I), the stochastic gradients of the proxy objective (for minimization) with respect to x 0 and x ta are, respectively,EQUATIONEQUATIONwhereEQUATIONand γ t , ζ t are determined by α t and the DDIM variance σt .The coefficients are omitted as they can be scaled into the step sizes. Note that in such stochastic gradients, the Jacobian of µ θ (•) is involved, which necessitates backpropagation through the neural network and incurs significant computational and storage overhead. Therefore, we use an approximate yet effective and practical method by truncating the gradients inµ θ (•), i.e. D x0 µ θ (•) ≈ 0 and D xt a µ θ (•) ≈ 0.Intuitively, the µ-predictor of the diffusion model should be resilient to small perturbations in the input. For instance, when we feed a noisy image into a pre-trained diffusion model and introduce minor perturbations to the original image, we anticipate the predicted image to maintain consistency with the original one. Thus the approximation for x 0 is acceptable, and similarly for x ta if the noise level of x ta is not too large. By such gradient truncation, the approximation of the stochastic gradients for x 0 and x ta are, respectively,EQUATIONEQUATIONNote that x 0 is not subject to the constraint and can be updated directly by stochastic gradient descent. x ta is involved in the equality constraint, so we apply the projection gradient descent method.Considering the projection operator of the observation in (11) , an iteration step of the proposed algorithm is outlined as followsEQUATIONx ta ← I my , 0 my×(mx-my) T y ′ + diag 0 my×my , I mx-my x ta -η 2 dxt a ,t ,where η 1 and η 2 are selected step sizes and diag(•) denotes the operation of arranging entries into a diagonal matrix. We term this algorithm ProjDiff.Algorithm 1 ProjDiff for VP diffusion (noisy observation).Require: Observation y ′ , pre-trained diffusion model µ θ , step sizes η 1 , η 2 , total steps T , noise schedule α 1 . . . α T , equivalent noise level α ta . 1: Sample ϵ T , ϵ ′ T ∼ N (0, I);2: Initialize x ta ← √ α ta µ θ (ϵ T , T ) + √ 1 -α ta ϵ ′ T ; 3: for t = T to t a + 1 do 4:Sample ϵ, ϵ ′ ∼ N (0, I); Update x 0 as (18); 13: end for 14: return x 0In the noise-free scenario, the observation equation becomes y = Ax 0 and the auxiliary variable x ta degrades to t a = 0, thus the constraint is directly applied on x 0 . The corresponding optimization problem can be expressed asEQUATIONUtilizing reparameterization and gradient truncation, the approximate stochastic gradient of the objective isEQUATIONThen the iteration of ProjDiff in the noise-free scenario is given by:EQUATIONwhere A † is the Moore-Penrose pseudo-inverse of matrix A and η is the selected step size. The principle of the ProjDiff algorithm in noise-free scenarios resides in treating the observation as a constraint. In contrast to [24] where the authors used Gaussian likelihood for noise-free observations, ProjDiff guarantees better consistency between the generated results and the observations.Nonlinear inverse problems are inherently more ill-posed than linear ones. For instance, in our experiments, we consider the phase retrieval task, which aims to recover the original image given only the Fourier magnitude spectrum. The observation equation is:EQUATION)where P is the zero-padding matrix. Effectively addressing nonlinear inverse problems can significantly enhance the algorithm's versatility. Here, we discuss about the ProjDiff algorithm for nonlinear observations concerning noise-free and noisy scenarios.For noise-free nonlinear observation functions, the key point of ProjDiff lies in identifying an appropriate projection operator for the nonlinear constraint y = A(x 0 ). For some nonlinear functions A(•), such a projection operator is analytic or can be approximated. For more intricate functions, we can resort to gradient descent or sub-gradient methods to minimize ||y -A(x)|| 2 starting from the initial point x 0 to approximate the projection operator. Thus ProjDiff is applicable to such class of nonlinear observations.For noisy inverse problems, ProjDiff necessitates mapping the noise applied on the observation to the noise applied on the original data x 0 . Such a mapping is strict in the linear case. For nonlinear observations, the exact equivalent noise might not be accessible but may be estimated through some appropriate approach. In cases where the observation function is overly complex, one may leave the determination of the equivalent noise variance as a hyperparameter to be manually tuned. This renders ProjDiff applicable to noisy nonlinear observations.Here we elaborate on crucial details and address the weak observation problem.Time steps. In practice, we observe that selecting time steps in a decreasing manner from T to 1 yields the optimal performance and provides more stable results with reduced oscillations, which aligns with the findings reported in [28, 24] . Therefore, we adopt this time step schedule as the default setting in this work. Furthermore, we note that in certain scenarios, repeating the same time step multiple times (say N times) leads to enhanced performance, which bears a resemblance to the correction step proposed in [19] and the time-travel technique in [21] .Initialization. We propose to use random initialization in the ProjDiff algorithm. The core concept entails initializing x 0 and x ta to be close to the manifold of q(x 0 ) and q(x ta ), respectively, which is achievable via the µ-predictor (or the Tweedie's formula [33] ) and the forward process. Randomly sampling a Gaussian noise ϵ T ∼ N (0, I), we initialize the optimization variables by x 0 ← µ θ (ϵ T , T ) and x ta ← √ α ta x 0 + √ 1 -α ta ϵ ′ T for some ϵ ′ T ∼ N (0, I). Nonetheless, given the time step schedule and the form of the stochastic gradients discussed above, we note that the x 0 remains unaltered until x ta is fully optimized. Therefore, for noisy observations, it would be more efficient to re-initialize x 0 as µ θ (x ta , t a ) once x ta has been optimized.Weak observation problem. In most image restoration tasks, the constraints imposed by observations are typically strong enough to lead to a unique original data point, rendering ProjDiff effective in yielding satisfactory results. However, in scenarios where observations are weak, particularly in situations with significant degrees of freedom in the inverse problem, the performance of ProjDiff tends to decline. We term these scenarios the weak observation problem. Drawing inspiration from [34] , where the authors found that including a larger variance when remapping x 0 to x t leads to noisy results, we attempt to adjust the noise schedule to diminish the variance of the sampled x t . We propose Restricted Encoding to tackle the weak observation problem, which entails fixing an initial noise ϵ 0 ∼ N (0, I) and reparameterizing the sampling of x t asEQUATIONwhere ϵ ∼ N (0, I), and ξ ∈ [0, √ 1 -α t ] regulates the level of randomness.The complete implementation of ProjDiff for noisy observations is outlined in Algorithm 1. More variations of ProjDiff used in our experiments can be found in Appendix H.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments.: 
 We present the comparison of the proposed ProjDiff algorithm with several SOTA algorithms in two primary scenarios: (1) image restoration, and (2) source (music) separation and partial generation.All experiments are performed on a single NVIDIA 3090ti GPU. Linear tasks. We demonstrate the performance of ProjDiff across three linear image restoration tasks on ImageNet [35] : 4× super-resolution (with average pooling), 50% random inpainting, and Gaussian deblurring. Comparisons are conducted against prominent diffusion-based image restoration algorithms in recent years, including DDRM [20] , DDNM+ [21] , DPS [23] , and RED-diff [24] . The performance of the least square solution (x 0 = A † y) is reported as a baseline. For a fair comparison, DDRM, DDNM, RED-diff, and ProjDiff utilize 100 function evaluations, while DPS uses 1000 function evaluations as its performance significantly declines with 100 steps. The pre-trained model on ImageNet is sourced from [36] , specifically the 256 × 256 model without classifier guidance.Testing is conducted on a 1k sub-testset of ImageNet consistent with [21] . Metrics include PSNR, SSIM [37] , LPIPS [38] , and FID [39] , with all LPIPS values multiplied by 100 for clarity.Table 1 presents the restoration metrics for noisy observations with standard deviation σ = 0.05 (doubled when pixels are rescaled to [-1, 1]) on ImageNet. ProjDiff demonstrated highly competitive performance compared to other algorithms. Some visualization results are shown in Figure 2 . More experiments on ImageNet and CelebA [40] alongside ablation studies are shown in Appendix C.Nonlinear tasks. We evaluate the effectiveness of ProjDiff for nonlinear observations through the challenging phase retrieval task on the FFHQ dataset [41] . We compare ProjDiff against DPS [23] and RED-diff [24] as they are applicable to nonlinear observations. Traditional baseline algorithms are also reported, including the Error Reduction (ER) algorithm [42] , Hybrid Input-Output (HIO) [43] , and Oversampling Smoothness (OSS) [44] . Table 2 presents the results for both noise-free and noisy scenarios. One sample is generated per image from each algorithm. Given the inherently ill-posedness of phase retrieval, ProjDiff requires more steps to achieve superior results. We set the number of function evaluations to 1000 for all three algorithms. The results demonstrate that ProjDiff excels in addressing nonlinear inverse problems, regardless of whether the observation is noise-free or noisy. Some visualization results are shown in Figure 3 . Further experiments on the high dynamic range (HDR) task can be found in Appendix C. We evaluate ProjDiff on source separation and partial generation tasks following [45, 46] . The SLACK2100 dataset [47] is employed and the pre-trained diffusion models are sourced from [45] .The objective of the source separation task is to separate the audio tracks of four instruments (piano, bass, guitar, and drums) from a mixed audio sequence. We compare ProjDiff against the methods reported in [45] , along with RED-diff [24] due to its relevance to this work. Performance is assessed using the scale-invariant SDR improvement (SI-SDR i ) [48] for each instrument as well as their average. Tabel 3 presents the performance on the ISDM model [45] of different algorithms. ProjDiff demonstrates significant performance improvements compared to other diffusion-based algorithms.The particularly noteworthy result is that ProjDiff surpasses the Demucs method [49] , marking the first instance, to the best of our knowledge, where a diffusion-based separation algorithm significantly outperforms previous SOTA, not to mention that the diffusion model is not specifically designed or trained for the separation task.The partial generation task aims to generate tracks of other instruments given partial of the instruments tracks while ensuring harmony. We employ the sub-FAD metrics [50, 45] for different partial generation tasks. Note that partial generation poses a weak observation problem as discussed in Section 3.4, thus the proposed Restricted Encoding method is applied for ProjDiff. The results are shown in Table 4 . ProjDiff outperforms other algorithms across different partial generation tasks. Additional results and in-depth analysis including ablation studies can be found in Appendix D. 
conclusion: 
 In this work, we introduce ProjDiff, a versatile inverse problem solver that harnesses the power of diffusion models to capture intricate data prior and denoise simultaneously. By deriving a novel two-variable ELBO as a proxy for the log-prior, we reframe the inverse problems as constrained optimization tasks and address them via the projection gradient method. We meticulously explore the implementation details of ProjDiff and propose refined initialization methods and optional noise schedules to enhance its performance. Through extensive experiments across image restoration tasks and source separation and partial generation tasks, we demonstrate the competitive performance and versatility of ProjDiff in linear, nonlinear, noise-free, and noisy problems. ProjDiff provides insights into better leveraging diffusion priors for inverse problems for future work. Limitations and future work. One of the possible limitations of ProjDiff is the potential challenge of effectively handling noisy observations with highly intricate functions. Also, its reliance on Gaussian observation noise may limit its applicability to other noise types like Poisson or multiplicative noise. Additionally, ProjDiff needs manual tuning for the step size. Future research could focus on extending ProjDiff to tackle intricate noisy nonlinear observations more effectively, as well as developing adaptive step size strategies to further enhance the performance while reducing the need for manual intervention.EQUATIONwhere α ta = 1/ 1 + σ 2 for some t a ∈ [0, 1, . . . , T ], y ′ = √ α ta y and n ′ ∼ N (0, I mx ). This makes the y ′ a noise-free observation for a sample at time step t a , namely the auxiliary variable x ta .We consider the DDIM reference distribution asEQUATIONwhere the transition probability followsEQUATION)where σt is the variance hyperparameter in DDIM. We further choose σt = σ t = 1-αt-1 1-αt 1 -αt αt-1 for t > t a , which makes the stochastic process between t a and T Markov. For t ≤ t a , we leave σt a hyperparameter. Thus the reference distribution can be rewritten asEQUATIONThusEQUATIONThe following Lemma regarding transition probabilities can be proved by induction. Lemma 1. The conditional distribution of x t conditioning on x ta and x 0 isEQUATIONwhereEQUATIONProof. When t > t a , (30) holds as the chain between t a and T Markov.When t < t a , ( 30) and ( 31) holds when t = t a as it is equal to the DDIM reference distribution. We assume (30) and ( 31) holds when t = t 0 ∈ {1, 2, . . . , t a }, then when t = t 0 -1, we haveEQUATIONq σ (x t0-1 |x t0 , x 0 ) is a Gaussian distribution according to the DDIM refernece distribution, and q σ (x t0 |x ta , x 0 ) is a Gaussian distribution due to the inductive hypothesis. Then by the convolution law of two Gaussian distributions, ( 30) and ( 31) hold when t = t 0 -1. By induction, Lemma 1 concludes.The following lemma can be proved using Bayes's formula. Lemma 2. The conditional distribution of x t-1 conditioned on x ta and x t for t ≥ t a + 2 isEQUATIONEQUATIONProof. By Bayes's formula,EQUATION)Substituting the transition probability for t ≥ t a , we obtain the conclusion of Lemma 2.The transition probability of p θ follows q σ as p θ (x t-1 |x t ) = q σ (x t-1 |µ θ (x t , t), x t ) for t ≥ 2, and p θ (x 0 |x 1 ) = N (x 0 ; µ θ (x 1 , 1), σ2 1 I). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No datasets or models are proposed and there is no such risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All databases, models, and toolkits used in this paper are cited from the original papers or URLs.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The code provided in the supplementary materials is anonymized and well documented with the running details. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof. Based on the above preliminaries, we derive the ELBO of log p θ (x 0 , x ta ) as follows:EQUATIONEQUATIONEQUATIONThe KL divergence between q σ (x T |x ta ) and p θ (x T ) is zero as they are both N (0, I) by the setting of diffusion models. Now we handle (1) and ( 2) respectively.(1) p θ (x 0 |x 1 ) is Gaussian and thus we haveEQUATIONwhere by writing c i we denote the term independent of x 0 and x ta . q σ (x t-1 |x t , x 0 ) and p θ (x t-1 |x t ) are also Gaussian, so by the KL divergence of the Gaussian distributions, we have (for t ≤ t a )EQUATION)(2) Similarly,EQUATIONand for t ≥ t a + 2EQUATION)Here we apply the reparameterization trick in advance. We writeEQUATIONwhere ϵ ∼ N (0, I), since q σ (xt |x ta ) = N (x t ; α t /α ta x ta , (1 -α t /α ta )I) for t ≥ t a + 1.Substitute this into (39) and we haveEQUATIONwhereEQUATIONEQUATIONAnd also (40) is (for t ≥ t a + 2)EQUATIONwithEQUATIONEQUATIONThus by integrate ( 37), ( 38), (42) , and (45), we obtain the Variantional Lower bound of p θ (x 0 , x ta ) as follows:EQUATION)Here we reach the result of Proposition 1., with the coefficient g(t) is given byEQUATIONand w(t) is give byEQUATIONA.3 Proof of Proposition 2Proof. We continue the derivation by using the reparameterization trick. The reparameterization of ( 42) and ( 45) has been given above, we focus on (37) and ( 38) now. Applying Lemma 1, we can reparameterize them asEQUATION)We further disregard all the constants. The proxy objective for minimization can be rewritten as followsEQUATIONNow we seek for stochastic gradients of L. Sampling t ∼ U {1, 2, . . . , T } and ϵ, ϵ ′ ∈ N (0, I), the stochastic gradient with respect to x 0 and x ta are, respectively,EQUATIONandEQUATIONwhere the coefficient g(t) is omitted as it can be scaled into the step size. Here we reach the result of Proposition 2 in the main text.In practice, we truncate the gradient through µ θ , i.e., we letEQUATION)Thus the approximate stochastic gradients are, respectively,EQUATIONandEQUATION)At this point, we obtain the approximate stochastic gradients of the objective function with respect to the two variables x 0 and x ta . We can perform gradient descent on x 0 and the projection gradient method on x ta to solve the problem.1. The idea of introducing a new noise ϵ ′ is to ensure that the update direction of x ta remains as much as possible within the manifold at time step t a . The expression -(x ta - √ α ta µ θ (•) - √ 1 -α ta ϵ ′ ))can be viewed as a vector pointing from x ta to √ α ta µ θ (•) + √ 1 -α ta ϵ ′ . As µ θ (•) is close to the manifold of x 0 , the introduction of ϵ ′ allows the endpoint of the gradient vector to be close to the manifold at time t a .2. For a general linear observation equation, the derivation remains similar when transforming the observations into the spectral domain using SVD. Due to potentially differing noise levels for each component, the objective function will take on an element-wise form. This is based on the fact that in the diffusion model, we assume that the elements of x t-1 are independent when conditioning on x t .3. In practice, we find that setting the DDIM variance σt = 0 for t ≤ t a yields the best results, which results in γ t = √ 1 -α t and ζ t = 0. But this poses a problem as it leads to g(t) → ∞. In this work, we still scale all the coefficients g(t) into the step sizes as it works well in the experiments. We leave this issue for future research.4. Note that one may find t a is not an integer in practice, as α ta may not be in the discrete series α 0:T . This won't affect our algorithm, as it can be derived similarly that the approximate stochastic gradients in ( 56) and (57) still hold. The only difference lies in the initialization of x 0 which will be discussed in Appendix F. 5 . We use the µ-predictor in the derivation. The case of ϵ-predictor is similar, with the conversion given by µθ (x t ) = x t - √ 1 -α t ϵ θ (x t ) / √ α t .We refer to [19, 32] for more details about the relation between the µ-predictor, the ϵ-predictor and the Stein score [51] .In this section, we derive the ProjDiff algorithm for the Variance Exploding (VE) diffusion used in the source separation and partial generation tasks. The experiments using VE diffusion in this work are all noise-free, thus we only focus on the noise-free scenarios and leave the noisy scenarios of VE diffusion for future work.q(x t |x t-1 ) = N (x t ; x t-1 , (σ 2 t -σ 2 t-1 )I), with σ T > σ T -1 > • • • > σ 1 > σ 0 → 0. When σ T is large enough, q(x T ) ≈ N (0, σ 2 T I).The initial distribution of the reverse process is set to be p θ (x T ) = N (0, σ 2 T I) ≈ q(x T ) and the reverse transition in the DDPM manner isEQUATIONfor t ≥ 2, which matches the condition probabilityEQUATION)And we assume the last sampling step is set to beEQUATIONfor some small δ 0 .Thus the ELBO is derived as the same in DDPM [2] .EQUATIOND KL (q(x T |x 0 )||p θ (x T )) ≈ 0, and log p θ (x EQUATIONEQUATIONwithEQUATION)Sampling t ∼ U {1, 2, . . . , T }, ϵ ∼ N (0, I), and truncating the gradient through µ θ as in the VP setting, a stochastic gradient for -ELBO VE (for minimization) isEQUATIONThen the iteration step of the ProjDiff (noise-free) in the VE setting isEQUATIONwhere η is the step size.The RED-diff [24] algorithm in the VE setting can be derived similarly. In the source separation and partial generation tasks, we use these forms as the implementation of our ProjDiff and RED-diff [24] algorithms.Remark: For VE diffusion, the Restricted Encoding method involves fixing the initial noise ϵ 0 ∼ N (0, I) and reparameterizing the sampling of x t as x t = x 0 + ξϵ 0 + σ 2 t -ξ 2 ϵ for ϵ ∼ N (0, I). Specifically, we set ξ = σ t-1 in partial generation tasks.Here, we present the omitted experiments for image restoration tasks. The results for noise-free restoration tasks on ImageNet are shown in Table 5 , the results for noise-free and noisy restoration tasks on CelebA are shown in Table 6 and 7 , respectively, and the results for nonlinear high dynamic range (HDR) task are in Table 8 . On the CelebA dataset, we further compare ProjDiff with more diffusion based rithms, including ΠGDM [22] , DMPS [52] , Resample [34] , and DiffPIR [53] . ProjDiff demonstrates highly competitive performance on all these tasks.Metrics balance in the super-resolution task. An interesting observation is the trade-off between objective metrics and perceptual metrics in the super-resolution task. DPS achieves the best perceptual metrics but performs poorly on objective metrics, while RED-diff performs well in objective metrics Fewer steps. DDRM [20] and DDNM [21] can both address image restoration with fewer steps (20 steps). Here we test the performance of the proposed ProjDiff with fewer steps on ImageNet. Table 9 and Table 10 show comparisons of ProjDiff with DDRM and DDNM in three linear restoration tasks under noise-free and noisy observations, respectively. ProjDiff also demonstrates highly competitive performance across multiple metrics, indicating its potential to perform image restoration with fewer steps.Four trials for the phase retrieval task. It is noted in [23] that in the phase retrieval task, DPS requires sampling four times for each image and selecting the best result to achieve satisfactory performance. Therefore, we compare the performance of DPS and ProjDiff with four independent samplings. We also report the performance of the ER, HIO, and OSS algorithms with four samplings as baselines. The best image is selected as the sample with the highed PSNR metric. The results are shown in Table 11 . Both DPS and ProjDiff show performance improvements with four trials, with ProjDiff in particular exhibiting much better performance, reaching a PSNR of 41.58dB in the noise-free case.The impact of the gradient truncation method used in ProjDiff. We compared ProjDiff with and without the gradient truncation method on the CelebA dataset, as shown in Table 12 . ProjDiff without gradient truncation is denoted as ProjDiff-FG. The results indicates that there is some performance loss when using gradient truncation compared to not using it, while the efficiency increases by nearly three times. Considering that using gradient truncation has already achieved satisfactory performance, we accept the trade-off of some performance loss for the gain in efficiency. Using gradient descent to approximate the projection operation for nonlinear tasks. We claim that for general nonlinear tasks, ProjDiff can be applied by using gradient descent to approximate the projection operation. Here we verify this method in the phase retrieval task (ProjDiff-GD in Table 13 ).The results indicate that approximating the projection operator of x 0 with respect to the observation equation by minimizing ||y -A(x)|| 2 starting from x 0 can also achieve satisfactory performance, which suggests that ProjDiff has generalizability for general nonlinear inverse problems.Sensitivity tests for the hyperparameters. We conduct the sensitivity tests for the step size η 1 and noise level σ 0 in ProjDiff on noisy super-resolution task on CelebA, as shown in Table 14 .'×a' denotes that we perturb the input standard deviation of ProjDiff by multiplying a. The results indicate that ProjDiff exhibits a certain degree of robustness to the step size and noise level. This also demonstrates that retaining the equivalent noise level as an adjustable hyperparameter is feasible when it cannot be directly calculated.RED-diff with random initialization. The RED-diff algorithm is observed to accidentally fail in the 50% random inpainting task. Here we demonstrate that by adopting the random initialization method proposed in this work, RED-diff achieves more reasonable results. We compare REDdiff's performance on ImageNet for both noise-free and noisy inpainting tasks when using random initialization (denoted as 'RED-diff with random init') and using degraded images for initialization as proposed in [24] . The results are shown in Table 15 . RED-diff's performance is significantly improved when using random initialization.Two models were trained in [45] : one is MSDM which models the joint distribution among the four instruments, and the other is ISDM which models the distribution of each instrument independently. ISDM performs better than MSDN in the source separation task as reported in [45] . We have presented ProjDiff's performance on ISDM in the main text, and here we supplement the experiment on MSDM. The SI-SDR i metrics of ProjDiff, RED-diff, MSDM-Gaussian, and MSDM-Dirac are shown in Table 16 . ProjDiff also demonstrates the best performance on the MSDM model. [45] only repeat twice using the correction steps method in [19] . A natural doubt is whether the performance improvement arises from more function evaluations. We test MSDM-Dirac algorithm with more steps, and the results are shown in Table 17 . When increasing the correction steps from 2 to 5 which matches the function evaluations of ProjDiff, the performance of MSDM-Dirac declines. This verifies that ProjDiff's advantage does not simply arise from more function evaluations. Moreover, we test ProjDiff's performance with more repetitions using the MSDM model, namely 10, 25, and 50 times with the step size adjusting accordingly to stabilize convergence. The step size is set to 0.5/N , where N is the number of repetitions. The results are shown in Table 18 . Further increasing the number of repetitions leads to even better metrics for ProjDiff. Momentum mechanism. We use Polyak momentum in the source separation task. Here we conduct experiments to validate the effectiveness of the momentum mechanism. The results with different momentum hyperparameter β ranging from 0 to 0.9 are shown in Table 19 . Setting β to 0.5 yields the best performance. We infer that a certain level of momentum can accelerate convergence, while over large momentum may cause oscillations at small time steps and thus affect the results. This explains why the performance is relatively poor when β = 0.9 in the source separation task.Weak observation problem and Restricted Encoding. Here we further explain why the partial generation task is a weak observation problem. The partial generation task aims to generate the tracks of other instruments based on some instruments, e.g., generating drums, piano, and guitar based on bass. The constraint provided by the observations is very weak, which is because in a musical ensemble, the correlation between different instruments exists but is not significant, and a composer can freely create the melodies of other instruments based on certain instruments' melodies. Therefore, this is more of a generation task than a restoration task, and we refer to this type of problem as the "weak observation problem" where the constraints provided by the observations are weak. Now we verify the effectiveness of the proposed Restricted Encoding method in the partial generation task. While keeping other parameters unchanged, we switch the sampling method of x t in ProjDiff from Restricted Encoding to sampling directly from the forward process q(x t |x 0 ) (denoted as 'w/o Restricted Encoding'). The results are shown in Table 20 . Without Restricted Encoding, the Visualization of the waveforms. In the partial generation tasks, an observation is that as the of generated instruments decreases, the advantage of ProjDiff becomes smaller. We point out this is because the sub-FAD metric considers the combination of the generated and given tracks. Therefore, in cases where more instruments are provided, the combined track is closer to the ground truth, and the difference among different algorithms would be smaller. This can also explain why ProjDiff performs slightly worse than MSDM and RED-diff when generating only the tracks of guitar. This may be because ProjDiff tends to aggressively create new musical segments, thus resulting in a lower similarity to the ground truth set. We present some waveform samples in Figure 5 to verify this claim. The samples are randomly selected and the same row shares the same ID. ProjDiff's results have larger amplitudes while RED-diff and MSDM's results tend to remain silent.Moreover, following the conventions in acoustics researches, we include some audio samples of ProjDiff in the supplementary materials for subjective evaluation.Source separation. Denote the tracks of piano, drums, guitar, and bass asx 1 , x 2 , x 3 , x 4 ∈ R n ,where n represents the length of the sequences. The observation function for source separation isy = (1, 1, 1, 1)(x 1 , x 2 , x 3 , x 4 ) T T = x 1 + x 2 + x 3 + x 4 .Partial generation. The observation matrix for partial generation is a diagonal matrix with elements of either zero or one, i.e., A = diag(a 11 , a 22 , a 33 , a 44 ) with a 11 , a 22 , a 33 , a 44 ∈ {0, 1}. 0 represents that the corresponding instruments are to be generated while 1 represents that the instrument is provided as the condition. The observation is y= A(x 1 , x 2 , x 3 , x 4 ) T T .Super-Resolution. The super-resolution task uses 4 × 4 average pooling as the degradation function, which is quite similar to the source separation task. Consider a 4 × 4 image block, with elements x i,j ∈ R 3 , i, j = 1, 2, 3, 4. The dimension of 3 represents the RGB channels. The observation equation is thus given by y = 1 16 4 i,j=1 x i,j . For the entire image, the calculation can be paralleled for all 4 × 4 blocks using matrix operations in PyTorch.Random inpainting. The observations matrix for random inpainting is a diagonal matrix of zeros and ones, A = diag{a kk } for 1 ≤ k ≤ n where n is the total number of pixels in an image, specifically 256 × 256 in this work. The diagonal elements a kk have a 50% chance to be 0 and a 50% chance to be 1.Gaussian deblurring. We follow the settings from [20, 21] for Gaussian deblurring, with a 1D Gaussian kernel of size 5 and a standard deviation of 10. The observation equation is in the form of convolution. Efficient SVD proposed in [20] can be used for calculating the projection operator.Phase retrieval. Our experimental setup for phase retrieval follows [23] . The observation is the amplitude spectrum of the original image. However, as the phase retrieval task is highly ill-posed, it is nearly impossible to recover the image directly from the amplitude spectrum. Therefore, we follow the standard practices [54, 55, 23] by first padding the image with zeros and then computing the amplitude spectrum. This increases the spectral resolution and provides more information for retrieval. Thus the observation equation for phase retrieval is y = |DFT(Px)|, where P represents the padding operation, which is a linear operator. The original image is 256 × 256, and the padding is done with size 64 in four directions, resulting in a padded image of 384 × 384. Note that one significant difference between the phase retrieval experiment in this paper and in [23] lies in that [23] reported the best results from 4 independent runs for each image, while we perform all the algorithms once for each image.High dynamic range. The observation for the high dynamic range task can be written element-wise asEQUATIONi.e., first multiplying all the pixels by 2 and then clipping them within [-1, 1].DDRM. DDRM uses the recommended parameters from [20] , i.e. η = 0.85 and η b = 1.0.DDNM. DDNM uses the recommended parameter η = 0.85 from [21] . For noisy tasks, we report the performance of the DDNM+ algorithm [21] .DPS. Following the settings in [23] , the step size is set to ξ t = ξ/ ||y -A(µ θ (x t ))|| 2 . On ImageNet, ξ is set to 1 for super-resolution and inpainting tasks, and 0.4 for Gaussian deblurring. Since [23] did not conduct experiments on CelebA, we conducted task-by-task parameter tuning for the learning rate in DPS and reported the results, ensuring a fair comparison. For phase retrieval and high dynamic range tasks, ξ = 0.4.First, in the image restoration tasks, we follow the settings in [24] using the Adam optimizer with momentum pairs of (0.9, 0.99). RED-diff requires a balancing parameter λ and a step size lr. λ is set to 0.25 for all image restoration tasks, and the step size lr is set to 0.25 for the super-resolution and 0.5 for all other tasks. However, in the source separation and partial generation tasks, we find that the Adam optimizer yields poor results. Therefore, we run RED-diff with the same parameters used in ProjDiff and set λ = 1.0 which is tuned to the best. Thus in the source separation task, the momentum is set to 0.5 and the step size is set to 0.1 with 5 repetitions. In the partial generation task, the momentum is set to 0.9 and the step size is set to 0.05 with no repetitions.ΠGDM. We carefully conducted hyperparameter searches for each task using the first eight images with the average PSNR as the metric. Except for Noisy Gaussian Deblur task, we avoided using other algorithms to initialize ΠGDM to ensure a fair comparison. Specifically, we first use the forward transition to map the degraded image to the noise level corresponding to the 500th step, and then use 100 steps of ΠGDM to solve the inverse problem. We find that this approach is much better than starting directly from white noise (i.e. 1000th step). For Noisy Gaussian Deblur task, ΠGDM without initialization from other algorithms struggled to achieve satisfactory results. Therefore, we use DDNM+ to obtain the sample at the 500th step and then switch to ΠGDM to complete the solution.DMPS. The step size hyperparameter λ was searched for each task using the first eight images with the average PSNR as the metric.Resample. The weight hyperparameter γ was searched for each task using the first eight images with the average PSNR as the metric. ReSample was originally designed for Latent Diffusion, thus in our experiments, we consider the encoder-decoder as identity mappings.DiffPIR. The weight hyperparameter λ was searched for each task using the first eight images with the average PSNR as the metric.MSDM/ISDM-Gaussian/Dirac for source separation. Following [45] , the algorithms use the S churn mechanism [56] , with S churn = 20 for the MSDM model and 40 for the ISDM model. Correction steps [19] are set to 2.MSDM for partial generation. Similarly, S churn = 10 and no correction steps are used.Here we provide the details of how we calculate the metrics for replication and fair comparison.For image restoration, we first save the generated results in the png format. When all results are generated, we read them back in to calculate the metrics. For PSNR and SSIM, we use peak_signal_noise_ratio and structural_similarity methods from the skimage library 4 . When calculating SSIM, the parameter 'data_range' is set to 'data_range=generated_image.max()-generated_image.min()'. For LPIPS, we use the LPIPS library 5 . Note that before calculating LPIPS, pixel values are normalized to [-1, 1]. And for FID, we use the torch-fidelity library 6 .For the SI-SDR i metric used in source separation, the calculation is as described in [45] :EQUATIONwhere x n is the ground truth sequence, y represents the ground truth summation of the four instruments, and xn is the estimated track from the algorithm. α =x T n xn+ϵ ||xn|| 2 +ϵ and ϵ = 10 -8 to prevent numerical errors.For the sub-FAD metric in the partial generation tasks, we first calculate the summation of the generated tracks and the corresponding conditional tracks and save the mixtures in the wav format. Then we use the frechet_audio_distance library 7 to compute the FAD metrics between the generated mixtures and the ground truth mixtures.F.1 Initialization for noisy observation with non-integer t aIn the case of noisy observations, we aim to initialize x 0 using the optimized x ta as µ θ (x ta , t a ). However, note that t a may not correspond exactly to any discretized time step in practice. In such instance, we assume t 0 ∈ [1, 2, . . . , T ] such that t 0 ≤ t a ≤ t 0 + 1, and consequently we can first perform a forward transition as xt0+1 = α t0+1 /α ta x ta + 1 -α t0+1 /α ta ϵ for some ϵ ∼ N (0, I), and then initialize x 0 ← µ θ (x t0+1 , t 0 + 1).The projection operator for source separation isEQUATION)The projection operator for partial generation isEQUATIONSuper-resolution and random inpainting are similar to source separation and partial generation, respectively. The projection operators of these problems can be obtained without SVD. Regarding equivalent variance, for 4 × 4 super-resolution, based on the average of independent Gaussian variables, the equivalent variance is 16σ 2 with σ 2 being the variance of the noise adding to the observation. For inpainting, the equivalent noise should enjoy the same variance as the noise on the observations.Gaussian deblurring task is a bit more complex. The projection operator isEQUATIONwhere A † is the Moore-Penrose pseudo-inverse of A. The element-wise equivalent noise can be handled using the SVD.Phase retrieval. For the phase retrieval task, consider the observation y = |DFT(Px)|. Given input x 0 and observations y, first we calculate z = DFT(Px 0 ), and compute the projection operator of y = |z| with respect to z, which is z = y ⊙ z/|z| where the division is element-wise and ⊙ denotes the Hadamard product. Next, we map z back to the original pixel space. As DFT is invertible and the P is the padding matrix, we define the inverse transformation matrix corresponding to P as P 1 that extract the central 256 × 256 pixels from a 384 × 384 image. Thus the inverse mapping can be written as x = P 1 DFT -1 (z). Finally, as the resulting x is in the complex space, we project it onto the real space, namely x proj = real(x), where the real(•) operator takes the real part of a complex tensor.Regarding equivalent noise, we use the principle of energy equality in the space domain and in the frequency domain. The energy of observation noise is n 2 1 σ 2 where n 1 = 384 is the size of the spectrum image. Assuming the variance of the effective noise is σ eff , the noise energy in the space domain is n 2 σ 2 eff with n = 256. Equating both energy yields σ eff = n 1 σ/n = 1.5σ. High dynamic range. The regular projection operator of the HDR task can be expressed element-wise as followsP(x i , y i ) =        y i /2 -1 < y i < 1; 0.5y i = 1 and x i ≤ 0.5; -0.5 y i = -1 and x i ≥ -0.5; x i else.(EQUATIONThis formulation applies to the noise-free case. Nonetheless, we discover an adjusted version of the projection operator that can further accommodate noisy observations, which isP noisy (x i , y i ) =       y i /2 -0.5 < y i < 0.5; y i /2 y i ≥ 1 and x i ≤ 0.5; y i /2 y i ≤ -1 and x i ≥ -0.5;EQUATIONThis adjusted form is more adept at handling uncertainty in the observation values within the intervals [-1, -0.5] and [0.5, 1] induced by the observation noise. We employ this modified projection operator in the noisy HDR task. The equivalent noise variance can be simply approximated as σ 2 eff = (σ/2) 2 .F.4 Hyperparameters.In all experiments, ProjDiff uses SGDM as the optimizer. The momentum is set to 0.5 for the source separation task and 0.9 for the partial generation task. The step size is set to 0.1 for source separation with 5 repetition steps and 0.05 for partial generation with no repetition. For all image restoration tasks, the momentum is set to 0, and η 2 for the noisy observations is fixed to 1.0. The DDIM variance σt is set to 0 for t ≤ t a and to σt = σ t =1-αt-1 1-αt 1 -αt αt-1 for t > t a .No repetitions are applied. Other tuned optimal step sizes are shown in Table 21 . The time step schedule follows [45] for the source separation and partial generation tasks, and follows [20, 21] for the image restoration tasks.ProjDiff shares similarities with the Maximum A Posteriori (MAP) estimation method. Theoretically, in noisy situations, it is only necessary to use a Gaussian likelihood, i.e., p(y|x 0 ) = N (y; x 0 , σ 2 I), to solve the inverse problems. However, ProjDiff introduces a new auxiliary variable, which leads to better experimental outcomes than the MAP framework. Here, we present some qualitative discussion towards this experimental result.On one hand, if we are given perfectly accurate priors and likelihoods, and have sufficient computational capability, we could obtain the exact posterior and also its gradients. Within the MAP framework, this should lead to the ideal results. However, in practice, the priors provided by diffusion models are not entirely accurate, and the weight coefficients between the priors and the likelihood cannot always be perfectly set. Moreover, dealing with the priors in diffusion models relies on stochastic optimization methods. These factors imply that the MAP framework often fails to achieve the desired solution. In such cases, further exploring the information or capabilities within the diffusion model can provide substantial assistance in solving inverse problems and enhance performance. This is one of the reasons why ProjDiff may have a performance advantage over the original MAP framework.On the other hand, MAP methods utilize gradient descent to leverage the information from the observation, while ProjDiff transforms this into a projection operation by introducing noisy auxiliary variables. This approach offers numerous advantages: there is no need to consider the step size of gradient descent (at least for the likelihood term); there is no need to consider the weight coefficients between the likelihood and prior terms; and it ensures consistency between noisy samples and observations (the role of this consistency has also been confirmed in the noise-free scenario). Moreover, the introduction of this auxiliary variable indicates that ProjDiff can actually be viewed as simultaneously recovering clean data and the noise added to the observations, which can yield more accurate results than merely characterizing the noise prior with a Gaussian distribution. These transformations are all thanks to ProjDiff's utilization of both the clean prior and noisy prior modeled by diffusion models (i.e., the prior modeled by the diffusion model and its denoising capability).Here, we present additional algorithm blocks for ProjDiff used in the experiments. Algorithm 2 is used for noise-free restoration tasks. Algorithm 3 and Algorithm 4 are used in the source separation task and partial generation task, respectively.Require: Observation y, observation function A, pre-trained diffusion model µ θ , step size η, total steps T , noise schedule α 1 . . . α T . 1: Sample ϵ T ∼ N (0, I); 2: Initialize x 0 ← µ θ (ϵ T , T ); 3: for t = T to 1 do Calculate the approximate stochastic gradient: dx0,t = x 0 -µ θ ( √ α t x 0 + √ 1 -α t ϵ t , t); Calculate the approximate stochastic gradient: dx0,t = x 0 -µ θ (x 0 + σ t ϵ t,k , t); Calculate the approximate stochastic gradient: dx0,t = x 0 -µ θ (x 0 + σ t-1 ϵ 0 + σ 2 t -σ 2 t-1 ϵ t,k , t); 8: Update momentum: v ← βv + (1 -β) dx0,t ; 9:Update x 0 : x 0 ← P A,y (x 0 -ηv); Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.This work is also effective for Variance Exploding (VE) diffusion. We defer the discussion to Appendix B.The case of ϵ-predictor is similar, with the conversion given by µθ (xt) = xt -√ 1 -αtϵ θ (xt) / √ αt.In practice, ta may not always be an integer, while the derivation remains similar. We concentrate on the scenario where ta assumes an integer in the main text. More discussions are deferred to the Appendix.https://github.com/scikit-image/scikit-imagehttps://github.com/richzhang/PerceptualSimilarityhttps://github.com/toshas/torch-fidelityhttps://github.com/gudgud96/frechet-audio-distance 


Decision: The paper propose a method for solving inverse problems using diffusion models. They reformulate the log-prior using a two-variable method and use projected gradients to solve the resulting constrained optimization problem. The method is applied to super-resolution, inpainting and deblurring tasks. Three reviewers positively rated the paper, with one reviewer providing a very negative response. During the discussion period, two of the positive reviews increased their rating, while the negative reviewer didn't engage in the discussion phase. It is not clear how significant or real the negative reviewers' concerns are, and I'm inclined to give the three combined positive responses more weight. They found the quantitative results to be convincing and the method to be in line with state-of-the-art approaches and so something that could be useful to the NeurIPS audience.

Reviews: review: paper_summary: paper_summary: This paper proposes ProjDiff for solving inverse problems with pre-trained diffusion models. By deriving a two-variable ELBO as a proxy for the log-prior, this paper reframes the inverse problems as constrained optimization tasks and address them via the projection gradient method.
strength_weakness: strength_weakness: 1. The assumption of this method for the measurement noise in Section 3.1 is doubtable. (1) As the authors claim in the limitation section, assuming the Gaussian noise will limit its applicability to other noise types like Poisson or multiplicative noise; (2) The whole method design heavily relies on know the exact standard deviation of the Gaussian noise \sigma, which is impractical and can lead to robustnesses issues when facing unknown noise level. Actually, the noise estimation itself is a challenging problem, particularly on the degraded measurement A(x). Please check the related works [1,2]. In general, this method is built on an impractical scenario, so I am worried about the practical usage of this method.

2. The approximation for implementing the update rules just after Proposition 2 is not convincing. (1) One of the reasons for this approximation is that "the μ-predictor of the diffusion model should be resilient to small perturbations in the input", but this hope usually is not the truth, particularly when t is large, i.e., at the beginning of the reverse sampling procedure; (2) There is not much evidence for this important approximation, no matter theoretical or empirical evidence.

3. The proposed method is too "delicate", as shown in Algorithm 1. (1) Again, this method needs to know the measurement noise level; (2)
this method contains several hyper-parameters. In Table 17, it is clear that the hyper-parameters are heavily tuned, and there is no specific way to guide the hyper-parameter tuning.

4. No ablation studies for the hyper-parameters. For this "delicate" method, a systematical ablation study is needed. For example, it would be good to report the results for different combinations of some important hyper-parameters.

5. The extension to nonlinear inverse problems is not convincing. This paper proposes to use min_x || y - A(x) ||^2 to approximate the projection operator. However, it is almost impossible to solve nonlinear inverse problems by using this formulation because it will be easy to be stuck in local minimizers. Also, if this formulation can solve nonlinear inverse problems well, then there is no need to write this paper.

6. The experiment settings for phase retrieval are doubtable. (1) I highly suspect that this paper downplays DPS in some implicit ways. I ran DPS for phase retrieval on FFHQ before, and remembered DPS could achieve over 30dB for PSNR after trying different initializations. In the original DPS paper, the authors claim that DPS need different initializations but it seems that this paper omits this; (2) there is no comparison with the golden standard method for phase retrieval, i.e., HIO+ER.

7. Some recent SOTA methods are missing for comparison, including ReSample[3] (ICLR'24 spotlight), DiffPIR[4] (CVPR'23), DMPS[5]. Please check the recent survey for more related works [6].



[1] Liu, X., Tanaka, M. and Okutomi, M., 2013. Single-image noise level estimation for blind denoising. IEEE transactions on image processing, 22(12), pp.5226-5237.

[2] Li, F., Fang, F., Li, Z. and Zeng, T., 2023. Single image noise level estimation by artificial noise. Signal Processing, 213, p.109215.

[3] Song, B., Kwon, S.M., Zhang, Z., Hu, X., Qu, Q. and Shen, L., 2023. Solving inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123.

[4] Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R. and Van Gool, L., 2023. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1219-1229).

[5] Meng, X. and Kabashima, Y., 2022. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12343.

[6] Li, X., Ren, Y., Jin, X., Lan, C., Wang, X., Zeng, W., Wang, X. and Chen, Z., 2023. Diffusion Models for Image Restoration and Enhancement--A Comprehensive Survey. arXiv preprint arXiv:2308.09388.
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The authors present a new way of solving inverse problems using a diffusion model based prior.
The key idea is that the forward process, which is to be undone, usually involves Gaussian noise.
The authors utilize this by viewing the noisy observation as a as random variable at an intermediate time step t of the diffusion forward process.
This allows them to formulate the inverse problem as a joint optimisation problem for x_0 and x_t given the noisy observation.
As a by product, their method is able to solve inverse problems which do not involve noise as well.
The authors evaluate their method and achieve superior results compared to various baselines.
strength_weakness: strength_weakness: * The motivation, comparing the method to the more established MAP approach, could be a bit more clear.
The authors write: 
"In this work, we introduce a novel two-variable ELBO by constructing an auxiliary variable that accounts for the observation noise, thereby utilizing both the prior information and the denoising capability in diffusion models simultaneously."
Why exactly is is describable to utilize the denoising capability of the model? Is the something wrong with the normal way of including the log of the forward model in the optimisation task? Why can we expect an improvement? Is it a problem of optimisation? Surely the way task itself is normally (e.g. [28]) formulated is correctly, or not?

* Some aspects of the method were difficult for me to follow. This may be due to my inexperience with some of the involved mathematics.
In particular, I wonder if the SVD based decoupling could be explained or illustrated differently to facilitate understanding.
questions: questions: Regarding the SVD composition, I believe I am missing an important aspect about how the noisy observation can be considered as a step in the diffusion process when it involves A.
Let's say A corresponds to a convolution with a blur kernel. Would this not mean that we are considering a blurred noisy image to be part of the diffusion process. This would be incorrect, because the blurring would not be part of the training data statistics which would only include clean images, right?
limitations: limitations: My understanding is that the method is limited to either image degradation processes that involve Gaissian noise, or are noise-free.
Is this the case?
If yes, this would be a significant limitation with respect to practical applicability, e.g. in settings with Poisson shot noise.
paper_summary: paper_summary: This paper proposed a new sampling strategy for solving noisy inverse problems using diffusion models. The proposed method is called ProjDiff, which is based on two-step minimization of log-posterior using gradient descent. The sampling procedure is derived by (1) Reparametrization of the noisy measurement as a noiseless measurement of an intermediate noisy sample in the diffusion process, and (2) obtaining a proxy objective from the variational lower bound.
strength_weakness: strength_weakness: + The paper is not well-written and it is hard to follow. Justification: 

     1. Theoretical analysis results lack organization. Example: Lemma 1 and Lemma 2 should be separated from the main proof. The proofs for the propositions should also be separated and distinctly expressed. 

     2. Theoretical results should be explicitly included. The proofs for Lemma 1 and Lemma 2 are not included in the text. If taken from a source, then the reference should be mentioned, otherwise, the proofs should be included. 

    3. Abstract includes some concerning statements.
        >Since inverse problems inherently entail maximum a posteriori estimation, previous works have endeavored to integrate diffusion priors into the optimization frameworks.

        Inverse problems could be solved using MMSE estimation. It is unclear what kind of optimization algorithm the authors refer to. 
       
        > by introducing an auxiliary optimization variable. By employing gradient truncation, the projection gradient descent method is efficiently utilized to solve the corresponding optimization problem. 

        The abstract does not reflect the method used.  The statement is vague and could be inferred in many different ways. 

       Revising the abstract to better reflect the proposed method can strengthen the paper.

  4. The second and third paragraph of the introduction needs to be revised. Instead of briefly mentioning what each existing method does, the authors could state how their method is different from the literature and how it contributes to the literature. 



* The truncation of the gradient is based on an assumption and leads to an approximation in the solutions. Thus, the reduced computational cost should not be mentioned as a contribution of the paper, specifically when the authors have not included any experimental results regarding this. 
 > 44: Through gradient truncation, we obtain an approximation of the stochastic gradient of the objective, effectively sidestepping significant computational overhead.


Overall, the weaknesses of the paper are mostly related to the presentation of the paper and the blurred message, not the proposed method and effectiveness of it. 

Minor issues: 

> Line 19: Their remarkable ability to capture data priors enables effective guidance for solving inverse problems [6], which are widely exploited in image restoration. 

Guidance in inverse problems usually refers to enforcing data consistency with the measurement and relies on the forward model.
questions: questions: What does the author mean by weak observations? (Line 214)

How does the algorithm look when used for nonlinear inverse problems? Can equation (8) be written for nonlinear measurements?


How do the authors set the $\eta_1$, $\eta_2$, and equivalent noise level in the algorithms?
Are the solutions sensitive to these parameters?
limitations: limitations: Partially addressed by the authors.
paper_summary: paper_summary: Authors propose ProjDiff which reframes noisy inverse problems with diffusion models as a two-variable constrained optimization by introducing an auxiliary optimization variable. Authors derive a two-variable ELBO as a proxy for the log-prior and solve the optimization problem via projection gradient descent. Authors conduct comprehensive experiments on several image restoration tasks (super-resolution, inpainting and deblurring), source separation, and partial generations tasks.
strength_weakness: strength_weakness: * Prior work section is not comprehensive.
* The gains in performance compared to baseline methods (especially for noisy restoration tasks) is not convincing. Moreover, hyperparameter search for baseline models were not conducted and wrong baseline method was used (DDNM for measurements with noise).
* See the questions below.
questions: questions: * I have several questions regarding DDNM numbers. First of all, DDNM [1] is designed for noise-free image reconstruction problems. I believe that is the reason why DDNM on Gaussian deblurring performs bad (as in Table 1: 7db PSNR, 0.03 SSIM, etc.). I would suggest the authors to switch to the DDNM+ variant as described in [1] for noisy inverse problems. If authors are already using DDNM+, could you clarify why the performance is bad on Gaussian deblurring?
* line 37-39: "However, it’s worth noting that, since diffusion models are inherently effective at denoising, considering the observation noise in the likelihood term fails to fully leverage diffusion models’ denoising capability." Could the authors clarify what it means to not fully leverage diffusion models here?
* I would recommend the authors to include $\Pi$GDM [2] in their comparisons especially since it performs much better than DDRM.
* Some other missing citations on solving inverse problems with diffusion models: CCDF [3], latent diffusion models: PSLD [4]
* line 663-665: "Since [23] did not conduct experiments on CelebA, we use the parameters on FFHQ for the CelebA dataset as both FFHQ and CelebA are datasets of faces.". I don't think this is a good practice. In my experience, even though both CelebA and FFHQ are face datasets, DPS is not robust to the choice of step size. I believe some hyper-parameter search on a small set of images is necessary for fair comparison.
* Do you think ProjDiff can be extended for using latent diffusion models as a prior?
---
References:

[1] Wang, Yinhuai, Jiwen Yu, and Jian Zhang. "Zero-shot image restoration using denoising diffusion null-space model." arXiv preprint arXiv:2212.00490 (2022).

[2] Song, Jiaming et al. “Pseudoinverse-Guided Diffusion Models for Inverse Problems.” International Conference on Learning Representations (2023).

[3] Chung, Hyungjin, Byeongsu Sim, and Jong Chul Ye. "Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[4] Rout, Litu, et al. "Solving linear inverse problems provably via posterior sampling with latent diffusion models." Advances in Neural Information Processing Systems 36 (2024).
limitations: limitations: Limitations are adequately addressed by the authors.

Comments: title: Thanks
Looking forward to your reply!
Response to the rebuttal
Thanks for the reply.
Thank you for the rebuttal.
comment: Absolutely! We are willing to incorporate these arguments into our final version of the paper to make our motivation and the advantages of our algorithm more clear. Thank you for the suggestion!
Thanks, that makes sense.
Would you include this line of argument in a final version of the paper?
Dear Reviewer g7gD:

We would like to express our sincere gratitude once again for your valuable comments and suggestions.

The discussion phase is due to conclude in 30 hours, and we have already received responses from the other three reviewers. We are eagerly awaiting your feedback. Should there be any further questions, we would be more than happy to engage in continued discussion if time permits.

Best regards.
We would like to extend our sincerest gratitude for raising the score! We are greatly encouraged!
Thank you for providing the rebuttal. 
After reading the reviews from all the reviewers and rebuttals, I would update my score to 6.
We greatly appreciate the reviewers' careful consideration of our rebuttal! The issue can be explained from two perspectives:

On one hand, ideally, if we are given perfectly accurate priors and likelihoods, and have sufficient computational capability, we could obtain the exact posterior and also its gradients. Within the MAP framework, this should lead to the ideal results. However, in practice, the priors provided by diffusion models are not entirely accurate, and the weight coefficients between the priors and the likelihood cannot always be perfectly set. Moreover, dealing with the priors in diffusion models relies on stochastic optimization methods. These factors imply that the MAP framework often fails to achieve the desired solution. In such cases, further exploring the information or capabilities within the diffusion model can provide substantial assistance in solving inverse problems and enhance performance. This is one of the reasons why our algorithm has a performance advantage over the original MAP framework.

On the other hand, MAP methods utilize gradient descent to leverage the information from the observation, while ProjDiff transforms this into a projection operation by introducing noisy auxiliary variables. This approach offers numerous advantages: there is no need to consider the step size of gradient descent (at least for the likelihood term); there is no need to consider the weight coefficients between the likelihood and prior terms; and it ensures consistency between noisy samples and observations (the role of this consistency has also been confirmed in the noise-free scenario). Moreover, the introduction of this auxiliary variable indicates that ProjDiff can actually be viewed as simultaneously recovering clean data and the noise added to the observations, which can yield more accurate results than merely characterizing the noise prior with a Gaussian distribution. These transformations are all thanks to ProjDiff's utilization of both the clean prior and noisy prior modeled by diffusion models (i.e., the prior modeled by the diffusion model and its denoising capability). This is why we claim that we "have utilized more information from the diffusion model than the original MAP framework".

We would like to express our gratitude once again to the reviewer for the response! We hope that these additional answers will make our motivation and the reasons for the performance advantages of our algorithm clearer.
Thank you for the clarifications!

**Regarding the motivation:**
*"The conventional approach of including the logarithm of the forward model in the optimization task is also feasible, as seen in [24, 28] in the main paper, and we have compared with it in our experiments (RED-diff). The improvement of our algorithm compared to these methods can be expected because our method utilizes more information modeled by the diffusion model, i.e., it uses both the prior and denoising capabilities (or both the clean data prior and the noisy data prior). We believe this is the essential difference between our algorithm and the methods that include the log of the forward model and modeled by diffusion models in the MAP framework."*

I am afraid I still can't completely follow here. Surely the correct prior  together with the correct forward model should provide (or be proportional to) the correct posterior. Maximising the posterior will provide the true MAP estimate. How can *"additional information"* beyond this be useful?
Are you saying your method is better at finding the MAP estimate (compared to the conventional approach) or is your method finding something better than the MAP estimate?
We would like to express our heartfelt gratitude for your recognition and raising the score! Thank you once again for your kind consideration!
I would like to thank the time and effort authors put into their rebuttal. I appreciate that they have incorporated all the feedback into providing a fairer comparison (fixing DDNM+, tuning DPS step size, etc.) against baseline methods especially in a short window of time.

Most of my concerns are alleviated and my questions are answered. I've also read the comments of other reviewers (and authors' response to them). I believe the results are more convincing with the updated numbers and the proposed changes. Therefore, I'm happily updating my score to $6$ and increasing soundness to $3$.