Title: Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems
Venue: NeurIPS 2024
Accepted: yes
Keywords: Inverse problem

diffusion model

optimization

generative modeling
Abstract: The recent emergence of diffusion models has significantly advanced the precision of learnable priors, presenting innovative avenues for addressing inverse problems. Previous works have endeavored to integrate diffusion priors into the maximum a posteriori estimation (MAP) framework and design optimization methods to solve the inverse problem. However, prevailing optimization-based rithms primarily exploit the prior information within the diffusion models while neglecting their denoising capability. To bridge this gap, this work leverages the diffusion process to reframe noisy inverse problems as a two-variable constrained optimization task by introducing an auxiliary optimization variable that represents a 'noisy' sample at an equivalent denoising step. The projection gradient descent method is efficiently utilized to solve the corresponding optimization problem by truncating the gradient through the $\mu$-predictor. The proposed algorithm, termed ProjDiff, effectively harnesses the prior information and the denoising capability of a pre-trained diffusion model within the optimization framework. Extensive experiments on the image restoration tasks and source separation and partial generation tasks demonstrate that ProjDiff exhibits superior performance across various linear and nonlinear inverse problems, highlighting its potential for practical applications. Code is available at https://github.com/weigerzan/ProjDiff/.

Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

Abstract

The recent emergence of diffusion models has significantly advanced the precision of learnable priors, presenting innovative avenues for addressing inverse problems. Previous works have endeavored to integrate diffusion priors into the maximum a posteriori estimation (MAP) framework and design optimization methods to solve the inverse problem. However, prevailing optimization-based rithms primarily exploit the prior information within the diffusion models while neglecting their denoising capability. To bridge this gap, this work leverages the diffusion process to reframe noisy inverse problems as a two-variable constrained optimization task by introducing an auxiliary optimization variable that represents a 'noisy' sample at an equivalent denoising step. The projection gradient descent method is efficiently utilized to solve the corresponding optimization problem by truncating the gradient through the µ -predictor. The proposed algorithm, termed ProjDiff, effectively harnesses the prior information and the denoising capability of a pretrained diffusion model within the optimization framework. Extensive experiments on the image restoration tasks and source separation and partial generation tasks demonstrate that ProjDiff exhibits superior performance across various linear and nonlinear inverse problems, highlighting its potential for practical applications. Code is available at https://github.com/weigerzan/ProjDiff/.

1 Introduction

Denoising diffusion models have achieved tremendous success in the field of generative modeling [1-5]. Their remarkable ability to capture data priors provides promising avenues for solving inverse problems [6], which are widely exploited in image restoration [7-12], medical image processing [13, 14], 3D vision [15], audio processing [16, 17] and beyond.

Numerous endeavors have sought to harness diffusion models to address inverse problems [1827]. Since the sampling process of diffusion models is a reverse Markov chain, most approaches attempt to integrate the guidance provided by the observation equation into the sampling chain. For instance, DDRM [20] achieves favorable results for linear inverse problems with low complexity by introducing a new variational distribution. DDNM [21] capitalizes on the concept of nullrange decomposition, effectively rectifying the range-space component of the intermediate steps by

Jiawei Zhang, Jiaxin Zhuang, Cheng Jin, and Yuantao Gu are with the Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China. Gen Li is with the Department of Statistics, The Chinese University of Hong Kong, Hong Kong, China. The corresponding author is Yuantao Gu.

𝑥

!

𝑥

"

!

𝑥

"

!

𝑥

#

𝑥

$

Diffusion

𝑥

%

𝑥

%&#

Inverse

Problem

Observation

function

𝑦′

Constrained

optimization

𝑥 $

$

Observation function :

Optimization problem:

…

…

#

"

)

Projection

∇

! "

"

"

"

)

y

#

= Ax

" $

!

|x

#

"

)

∇

! %

%

|x

"

"

)

PDF-contour

PDF-contour

Update

Process

Update of

x

%

Update of

x

"

"

leveraging the observation information. DPS [23] guides the generation process using the gradients of intermediate steps with respect to the observation error. Moreover, methods based on Monte Carlo Particle Filtering [25-27] are employed to approximate the true posterior.

Since inverse problems are often modeled as maximum a posteriori (MAP) estimations, recent efforts have been made to explicitly integrate diffusion models as the prior term in the optimization framework [28, 24]. [28] proposed to employ the Evidence Lower Bound (ELBO) to approximate the prior, facilitating the application of diffusion models in inverse problems in a plug-and-play manner. [24] achieved analogous results from a variational perspective. In these optimizationbased methods, due to the presence of the observation noise, a common practice is to adopt a Gaussian likelihood. However, it's worth noting that, since diffusion models are inherently effective at denoising, considering the observation noise in the likelihood term fails to fully leverage diffusion models' denoising capability.

To fully utilize both the prior information and the denoising capability inherent in diffusion models within the optimization framework, we introduce an auxiliary optimization variable to accommodate the influence of the observation noise. We derive a novel two-variable objective based on the properties of the diffusion process and transform the inverse problem into a constrained optimization task. Through gradient truncation, we obtain a more practical approximation of the stochastic gradient of the objective which sidesteps significant computational overhead. The proposed algorithm, termed ProjDiff, tackles the inverse problem by employing the concept of projection gradient descent to solve the corresponding optimization task. We also discuss the noise-free version of ProjDiff as a special case, which, compared to other optimization-based methods, ensures superior consistency between the generated results and observations. ProjDiff's applicability also extends beyond linear observations to encompass nonlinear functions, which enhances its competitiveness and versatility.

We demonstrate the outstanding performance of ProjDiff through comprehensive experiments on various benchmarks. In both linear and nonlinear image restoration tasks, ProjDiff exhibites superior performance among existing state-of-the-art (SOTA) algorithms. In the music separation task, ProjDiff shows for the first time, to the best of our knowledge, that a diffusion-based separation algorithm can surpass previous SOTA, which further demonstrates the powerful potential of diffusion models and provides insights to better harness their capabilities in inverse problems for future research.

2 Backgrounds

2.1 Denoising diffusion models

Denoising diffusion models [1-3] are a class of latent variable generative models tailored to capture a targeted data distribution q ( x ) . Diffusion models typically predefine a forward process characterized as a Markov chain, wherein the transition probability is stipulated as Gaussian distribution:

where a t and b t represent the scale and variance parameters, respectively. The sampling process of diffusion models aims to invert the forward chain via a parameterized reverse process, which is also modeled as a Markov chain with learnable parameter θ :

where f θ denotes the mean function and the variance σ 2 t is predefined. In this work, we primarily focus on the Variance Preservation (VP) diffusion [19] 1 , characterized by parameters a t = √ α t , b t = 1 -α t with α 0 ≈ 1 and ∏ T t =0 α t ≈ 0 . With these parameters, we have q ( x t | x 0 ) = N ( x t ; √ α t x 0 , (1 -α t ) I ) , where α t = ∏ t i =0 α t . The initial distribution of the reverse process is set to match the forward process, i.e., p θ ( x T ) = N ( 0 , I ) , while the mean function is chosen as

Here µ θ serves as an estimation of x 0 at time t , typically implemented through a parameterized neural network referred to as a ' µ -predictor' 2 . The variance σ 2 t ∈ [0 , 1 -α t -1 ] can take various values. When σ t = √ 1 -α t -1 1 -α t √ 1 -α t α t -1 , the reverse process aligns with DDPM [2]; while when retaining σ t = ˜ σ t as an adjustable parameter, one obtains DDIM [3]. We refer to [19, 29-31] for more theoretical guarantees of diffusion models.

2.2 Diffusion models as data prior

As stated in [32, 28], for a well-trained diffusion model, the Evidence Lower Bound (ELBO) can effectively approximate the log-likelihood of the samples, i.e.

where λ ( t ) represents the coefficient within the ELBO and C is a constant. Utilizing the ELBO as the log-prior term enables diffusion models to serve as a plug-and-play prior for general inverse problems. [28] handled the ELBO via stochastic gradient descent and the reparameterization method. By sampling t ∼ U { 1 , 2 , . . . , T } and ϵ ∼ N ( 0 , I ) , a stochastic gradient of the ELBO can be derived as

where D x 0 µ θ ( · ) denotes the Jocabian of the µ -predictor. [24] derived a similar proxy objective from the perspective of variational inference, and proposed a more computationally efficient approximation of the stochastic gradient by reweighting the objective. Notably, when the variational distribution is a delta function, variational inference aligns with the MAP estimation. In this work, we introduce a novel two-variable ELBO by constructing an auxiliary variable that accounts for the observation noise, thereby utilizing both the prior information and the denoising capability in diffusion models simultaneously.

2.3 Equivalent noisy samples

As stated in [26], the noisy observation of a sample can be regarded as a noise-free observation of an equivalent noisy sample at certain steps in the diffusion process. For linear observations y = Ax 0 + σ n , where x 0 ∈ R m x , y ∈ R m y , and n ∈ N ( 0 , I m y ) , a common practice [20, 21, 26] involves the Singular Value Decomposition (SVD) to attain a decoupled form. Without loss of generality, assume A has full row rank. Utilizing the SVD A = USV T yields U T y = S ( V T x 0 ) + σ n , thus

transforming the observation equation to y = Sx 0 + σ n , where y = U T y and x 0 = V T x 0 , which can be further written element-wise as

where x 0 ,i and y i represent the i th component of x 0 and y , respectively, and s i denotes the i th singular value of A . Assuming there exist t i ∈ { 1 , 2 , . . . , T } , 1 ≤ i ≤ m y such that α t i = 1 / (1 + ( σ/s i ) 2 ) , let y ′ i = √ α t i y i /s i and then the observation function can be rewritten as

√ α t i x 0 ,i + √ 1 -α t i n i can be interpreted as the i th component of a sample at time step t i , namely x t i ,i . Thus [26] proposed to employ Monte Carlo Particle Filtering to restore the samples up to the equivalent noise level, and then apply the backward transition to map these noisy samples back to the clean samples. In this work, we introduce this equivalent noisy sample as an auxiliary variable, thus better handling the observation noise in the optimization framework.

3 Method

In this section, we introduce the ProjDiff algorithm. The main idea behind ProjDiff lies in constructing an auxiliary variable to align the noisy observations to specific steps of the diffusion process, thereby forming a two-variable constrained optimization task, which can be tackled by approximating the stochastic gradients and employing the projection gradient descent method. We begin by deriving the ProjDiff algorithm for noisy linear observations and then discuss its noise-free version as a special case and the extension of ProjDiff to accommodate nonlinear observations.

3.1 ProjDiff

Consider the Gaussian observation equation y = Ax 0 + σ n . The goal of the inverse problem is to recover the original data x 0 given y , A , and σ . The decoupled observation equation in (7) indicates that one can apply SVD to reduce a linear observation into the simple form with A = [ I m y , 0 ] in the spectral domain, which implies the observation y represents the first m y components of x . Consequently, we can focus on this simplest observation function to declare our algorithm. Under this condition, assuming there exists some t a ∈ [0 , 1 , . . . , T ] 3 such that α t a = 1 / ( 1 + σ 2 ) , we rewrite the observation function as

where y ′ = y / √ 1 + σ 2 and n ′ ∼ N ( 0 , I m x ) . The noisy observation y ′ can be interpreted as a noise-free observation of √ α t a x 0 + √ 1 -α t a n ′ , which in turn can be viewed as a sample in the manifold of q ( x t a ) . Thus, we introduce an auxiliary variable x t a which denotes a noisy sample at time step t a , and propose to optimize the log-posterior term of the two variables as

where in (10) we use p θ to approximate the true prior q . By the construction of the auxiliary variable x t a , the likelihood term embodies a stringent consistency between this auxiliary variable and the noisy observation, which serves as a constraint. Note that log q ( y ) is independent of the optimization variables, thus (10) is transformed into the following constrained optimization task:

Now we seek the ELBO as a proxy of the joint log-prior term of x 0 and x t a to render it tractable. Leveraging Jensen's inequality and the transition probability of the diffusion model, we reach the following proposition.

Proposition 1. Considering the DDIM reference distribution q σ , we have the variational lower bound of the log-prior term as

where C is a constant independent of x 0 and x t a , the weight g ( t ) , w ( t ) are functions of α t and the DDIM variance ˜ σ t , and ν = ( x t -√ α t α t a x t a ) / √ 1 -α t α t a .

The complete derivation of the lower bound is provided in Appendix A. Note that the summation on the right-hand side of (12) is partitioned into two parts. The first part, the denoising matching term , pertains to the approximation of log p θ ( x 0 | x t a ) , signifying the consistency between the noise-free variable x 0 and the auxiliary variable x t a . The second part, the noisy prior term , corresponds to the approximation of the log-prior of the noisy auxiliary variable x t a . Conceptually, the workflow of utilizing this ELBO for inverse problems can be delineated as follows: the noisy prior term, coupled with the constraint on the auxiliary variable, exploits the prior information of the diffusion model and the observation information to recover a noisy sample x t a that satisfies the observation equation, while the denoising matching term leverages the denoising capability of the diffusion model to restore the clean sample x 0 from the noisy sample x t a .

Now, we address the ELBO outlined in Proposition 1 utilizing the principle of the stochastic gradient method. The constant C is independent of the optimization variables and can therefore be disregarded. The following proposition is obtained through reparameterization.

Proposition 2. Randomly sampling t ∼ U { 1 , 2 , . . . , T } , ϵ , ϵ ′ ∼ N ( 0 , I ) , the stochastic gradients of the proxy objective (for minimization) with respect to x 0 and x t a are, respectively,

where

and γ t , ζ t are determined by α t and the DDIM variance ˜ σ t .

The coefficients are omitted as they can be scaled into the step sizes. Note that in such stochastic gradients, the Jacobian of µ θ ( · ) is involved, which necessitates backpropagation through the neural network and incurs significant computational and storage overhead. Therefore, we use an approximate yet effective and practical method by truncating the gradients in µ θ ( · ) , i.e. D x 0 µ θ ( · ) ≈ 0 and D x t a µ θ ( · ) ≈ 0 . Intuitively, the µ -predictor of the diffusion model should be resilient to small perturbations in the input. For instance, when we feed a noisy image into a pre-trained diffusion model and introduce minor perturbations to the original image, we anticipate the predicted image to maintain consistency with the original one. Thus the approximation for x 0 is acceptable, and similarly for x t a if the noise level of x t a is not too large. By such gradient truncation, the approximation of the stochastic gradients for x 0 and x t a are, respectively,

Note that x 0 is not subject to the constraint and can be updated directly by stochastic gradient descent. x t a is involved in the equality constraint, so we apply the projection gradient descent method. Considering the projection operator of the observation in (11), an iteration step of the proposed algorithm is outlined as follows

where η 1 and η 2 are selected step sizes and diag ( · ) denotes the operation of arranging entries into a diagonal matrix. We term this algorithm ProjDiff.

Algorithm 1 ProjDiff for VP diffusion (noisy observation).

Require: Observation y ′ , pre-trained diffusion model µ θ , step sizes η 1 , η 2 , total steps T , noise schedule α 1 . . . α T , equivalent noise level α t a .

2: Initialize x t a ← √ α t a µ θ ( ϵ T , T ) + √ 1 -α t a ϵ ′ T ;

1: Sample ϵ T , ϵ ′ T ∼ N ( 0 , I ) ;

3: for t = T to t a +1 do

4: Sample ϵ , ϵ ′ ∼ N ( 0 , I ) ;

5: Calculate the approximate stochastic gradient ˜ d x t a ,t as (17);

6: Update x t a as (19);

7: end for

8: Initialize x 0 ← µ θ ( x t a , t a ) ;

9: for t = t a to 1 do

10: Sample ϵ ∼ N ( 0 , I ) ;

11: Calculate the approximate stochastic gradient ˜ d x 0 ,t as (16);

12: Update x 0 as (18);

13: end for

14: return x 0

3.2 Noise-free observations: a special case

In the noise-free scenario, the observation equation becomes y = Ax 0 and the auxiliary variable x t a degrades to t a = 0 , thus the constraint is directly applied on x 0 . The corresponding optimization problem can be expressed as

Utilizing reparameterization and gradient truncation, the approximate stochastic gradient of the objective is

Then the iteration of ProjDiff in the noise-free scenario is given by:

where A † is the Moore-Penrose pseudo-inverse of matrix A and η is the selected step size. The principle of the ProjDiff algorithm in noise-free scenarios resides in treating the observation as a constraint. In contrast to [24] where the authors used Gaussian likelihood for noise-free observations, ProjDiff guarantees better consistency between the generated results and the observations.

3.3 Extension to nonlinear observation

Nonlinear inverse problems are inherently more ill-posed than linear ones. For instance, in our experiments, we consider the phase retrieval task, which aims to recover the original image given only the Fourier magnitude spectrum. The observation equation is:

where P is the zero-padding matrix. Effectively addressing nonlinear inverse problems can significantly enhance the algorithm's versatility. Here, we discuss about the ProjDiff algorithm for nonlinear observations concerning noise-free and noisy scenarios.

For noise-free nonlinear observation functions, the key point of ProjDiff lies in identifying an appropriate projection operator for the nonlinear constraint y = A ( x 0 ) . For some nonlinear functions A ( · ) , such a projection operator is analytic or can be approximated. For more intricate functions, we can resort to gradient descent or sub-gradient methods to minimize || y -A ( x ) || 2 starting from the initial point x 0 to approximate the projection operator. Thus ProjDiff is applicable to such class of nonlinear observations.

For noisy inverse problems, ProjDiff necessitates mapping the noise applied on the observation to the noise applied on the original data x 0 . Such a mapping is strict in the linear case. For nonlinear observations, the exact equivalent noise might not be accessible but may be estimated through some appropriate approach. In cases where the observation function is overly complex, one may leave the determination of the equivalent noise variance as a hyperparameter to be manually tuned. This renders ProjDiff applicable to noisy nonlinear observations.

3.4 Time steps, initialization, and Restricted Encoding

Here we elaborate on crucial details and address the weak observation problem.

Time steps. In practice, we observe that selecting time steps in a decreasing manner from T to 1 yields the optimal performance and provides more stable results with reduced oscillations, which aligns with the findings reported in [28, 24]. Therefore, we adopt this time step schedule as the default setting in this work. Furthermore, we note that in certain scenarios, repeating the same time step multiple times (say N times) leads to enhanced performance, which bears a resemblance to the correction step proposed in [19] and the time-travel technique in [21].

Initialization. We propose to use random initialization in the ProjDiff algorithm. The core concept entails initializing x 0 and x t a to be close to the manifold of q ( x 0 ) and q ( x t a ) , respectively, which is achievable via the µ -predictor (or the Tweedie's formula [33]) and the forward process. Randomly sampling a Gaussian noise ϵ T ∼ N ( 0 , I ) , we initialize the optimization variables by x 0 ← µ θ ( ϵ T , T ) and x t a ← √ α t a x 0 + √ 1 -α t a ϵ ′ T for some ϵ ′ T ∼ N ( 0 , I ) . Nonetheless, given the time step schedule and the form of the stochastic gradients discussed above, we note that the x 0 remains unaltered until x t a is fully optimized. Therefore, for noisy observations, it would be more efficient to re-initialize x 0 as µ θ ( x t a , t a ) once x t a has been optimized.

Weak observation problem. In most image restoration tasks, the constraints imposed by observations are typically strong enough to lead to a unique original data point, rendering ProjDiff effective in yielding satisfactory results. However, in scenarios where observations are weak, particularly in situations with significant degrees of freedom in the inverse problem, the performance of ProjDiff tends to decline. We term these scenarios the weak observation problem. Drawing inspiration from [34], where the authors found that including a larger variance when remapping x 0 to x t leads to noisy results, we attempt to adjust the noise schedule to diminish the variance of the sampled x t . We propose Restricted Encoding to tackle the weak observation problem, which entails fixing an initial noise ϵ 0 ∼ N ( 0 , I ) and reparameterizing the sampling of x t as

where ϵ ∼ N ( 0 , I ) , and ξ ∈ [0 , √ 1 -α t ] regulates the level of randomness.

The complete implementation of ProjDiff for noisy observations is outlined in Algorithm 1. More variations of ProjDiff used in our experiments can be found in Appendix H.

4 Experiments.

We present the comparison of the proposed ProjDiff algorithm with several SOTA algorithms in two primary scenarios: (1) image restoration, and (2) source (music) separation and partial generation. All experiments are performed on a single NVIDIA 3090ti GPU.

4.1 Image restoration

Linear tasks. We demonstrate the performance of ProjDiff across three linear image restoration tasks on ImageNet [35]: 4 × super-resolution (with average pooling), 50 %random inpainting, and Gaussian deblurring. Comparisons are conducted against prominent diffusion-based image restoration algorithms in recent years, including DDRM [20], DDNM+ [21], DPS [23], and RED-diff [24]. The performance of the least square solution ( ˆ x 0 = A † y ) is reported as a baseline. For a fair comparison, DDRM, DDNM, RED-diff, and ProjDiff utilize 100 function evaluations, while DPS uses 1000 function evaluations as its performance significantly declines with 100 steps. The pre-trained model on ImageNet is sourced from [36], specifically the 256 × 256 model without classifier guidance. Testing is conducted on a 1k sub-testset of ImageNet consistent with [21]. Metrics include PSNR, SSIM [37], LPIPS [38], and FID [39], with all LPIPS values multiplied by 100 for clarity.

Table 1 presents the restoration metrics for noisy observations with standard deviation σ = 0 . 05 (doubled when pixels are rescaled to [ -1 , 1] ) on ImageNet. ProjDiff demonstrated highly competitive performance compared to other algorithms. Some visualization results are shown in Figure 2. More experiments on ImageNet and CelebA [40] alongside ablation studies are shown in Appendix C.

Nonlinear tasks. We evaluate the effectiveness of ProjDiff for nonlinear observations through the challenging phase retrieval task on the FFHQ dataset [41]. We compare ProjDiff against DPS [23] and RED-diff [24] as they are applicable to nonlinear observations. Traditional baseline algorithms are also reported, including the Error Reduction (ER) algorithm [42], Hybrid Input-Output (HIO) [43], and Oversampling Smoothness (OSS) [44]. Table 2 presents the results for both noise-free and noisy scenarios. One sample is generated per image from each algorithm. Given the inherently ill-posedness of phase retrieval, ProjDiff requires more steps to achieve superior results. We set the number of function evaluations to 1000 for all three algorithms. The results demonstrate that ProjDiff excels in addressing nonlinear inverse problems, regardless of whether the observation is noise-free or noisy. Some visualization results are shown in Figure 3. Further experiments on the high dynamic range (HDR) task can be found in Appendix C.

GT

y

Baseline

DDRM

DDNM

DPS

RED-diff

Ours

Deblur

Inpaiting

SR4

GT

y

DPS

RED-diff

Ours

GT

y

DPS

RED-diff

Ours

Phase retrieval

HDR

4.2 Source separation and partial generation

We evaluate ProjDiff on source separation and partial generation tasks following [45, 46]. The SLACK2100 dataset [47] is employed and the pre-trained diffusion models are sourced from [45].

The objective of the source separation task is to separate the audio tracks of four instruments (piano, bass, guitar, and drums) from a mixed audio sequence. We compare ProjDiff against the methods reported in [45], along with RED-diff [24] due to its relevance to this work. Performance is assessed using the scale-invariant SDR improvement (SI-SDRi) [48] for each instrument as well as their average. Tabel 3 presents the performance on the ISDM model [45] of different algorithms. ProjDiff demonstrates significant performance improvements compared to other diffusion-based algorithms. The particularly noteworthy result is that ProjDiff surpasses the Demucs method [49], marking the first instance, to the best of our knowledge, where a diffusion-based separation algorithm significantly outperforms previous SOTA, not to mention that the diffusion model is not specifically designed or trained for the separation task.

The partial generation task aims to generate tracks of other instruments given partial of the instruments tracks while ensuring harmony. We employ the sub-FAD metrics [50, 45] for different partial generation tasks. Note that partial generation poses a weak observation problem as discussed in Section 3.4, thus the proposed Restricted Encoding method is applied for ProjDiff. The results are shown in Table 4. ProjDiff outperforms other algorithms across different partial generation tasks. Additional results and in-depth analysis including ablation studies can be found in Appendix D.

5 Conclusion

In this work, we introduce ProjDiff, a versatile inverse problem solver that harnesses the power of diffusion models to capture intricate data prior and denoise simultaneously. By deriving a novel two-variable ELBO as a proxy for the log-prior, we reframe the inverse problems as constrained optimization tasks and address them via the projection gradient method. We meticulously explore the implementation details of ProjDiff and propose refined initialization methods and optional noise schedules to enhance its performance. Through extensive experiments across image restoration tasks and source separation and partial generation tasks, we demonstrate the competitive performance and versatility of ProjDiff in linear, nonlinear, noise-free, and noisy problems. ProjDiff provides insights into better leveraging diffusion priors for inverse problems for future work.

*

Previous SOTA.

Limitations and future work. One of the possible limitations of ProjDiff is the potential challenge of effectively handling noisy observations with highly intricate functions. Also, its reliance on Gaussian observation noise may limit its applicability to other noise types like Poisson or multiplicative noise. Additionally, ProjDiff needs manual tuning for the step size. Future research could focus on extending ProjDiff to tackle intricate noisy nonlinear observations more effectively, as well as developing adaptive step size strategies to further enhance the performance while reducing the need for manual intervention.

References

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35:5775-5787, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Andrey Nikolayevich Tikhonov et al. On the stability of inverse problems. In Dokl. akad. nauk sssr , volume 39, pages 195-198, 1943.

Deepa Kundur and Dimitrios Hatzinakos. Blind image deconvolution. IEEE signal processing magazine , 13(3):43-64, 1996.

Sung Cheol Park, Min Kyu Park, and Moon Gi Kang. Super-resolution image reconstruction: a technical overview. IEEE signal processing magazine , 20(3):21-36, 2003.

Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques , pages 417-424, 2000.

Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. A review of image denoising algorithms, with a new one. Multiscale modeling & simulation , 4(2):490-530, 2005.

Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 , pages 649-666. Springer, 2016.

Yoav Shechtman, Yonina C Eldar, Oren Cohen, Henry Nicholas Chapman, Jianwei Miao, and Mordechai Segev. Phase retrieval with application to optical imaging: a contemporary overview. IEEE signal processing magazine , 32(3):87-109, 2015.

Morteza Mardani, Enhao Gong, Joseph Y. Cheng, Shreyas S. Vasanawala, Greg Zaharchuk, Lei Xing, and John M. Pauly. Deep generative adversarial neural networks for compressive sensing mri. IEEE Transactions on Medical Imaging , 38(1):167-179, 2019.

Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representations , 2022.

Hyungjin Chung, Dohoon Ryu, Michael T McCann, Marc L Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22542-22551, 2023.

Andreas Schuhmacher, Jørgen Hald, Karsten Bo Rasmussen, and Per Christian Hansen. Sound source reconstruction using inverse boundary element calculations. The Journal of the Acoustical Society of America , 113(1):114-127, 2003.

Antonio Pereira, Jérôme Antoni, and Quentin Leclere. Empirical bayesian regularization of the inverse acoustic problem. Applied Acoustics , 97:11-29, 2015.

Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems , 34:21757-21769, 2021.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems , 35:23593-23606, 2022.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. The Eleventh International Conference on Learning Representations , 2023.

Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations , 2022.

Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations , 2023.

Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference on Learning Representations , 2024.

Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motifscaffolding problem. In The Eleventh International Conference on Learning Representations , 2023.

Gabriel Cardoso, Yazid Janati el idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided denoising diffusion models for bayesian linear inverse problems. In The Twelfth International Conference on Learning Representations , 2024.

Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference on Learning Representations , 2024.

Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. Advances in Neural Information Processing Systems , 35:14715-14728, 2022.

Gen Li and Yuling Yan. O ( d /T ) convergence theory for diffusion probabilistic models under minimal assumptions. arXiv preprint arXiv:2409.18959 , 2024.

Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards non-asymptotic convergence for diffusion-based generative models. In The Twelfth International Conference on Learning Representations , 2024.

Gen Li and Changxiao Cai. Provable acceleration for diffusion models under minimal assumptions. arXiv preprint arXiv:2410.23285 , 2024.

Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970 , 2022.

Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association , 106(496):1602-1614, 2011.

Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations , 2024.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

James R Fienup. Phase retrieval algorithms: a comparison. Applied optics , 21(15):2758-2769, 1982.

C Fienup and J Dainty. Phase retrieval and image reconstruction for astronomy. Image recovery: theory and application , 231:275, 1987.

Jose A Rodriguez, Rui Xu, C-C Chen, Yunfei Zou, and Jianwei Miao. Oversampling smoothness: an effective algorithm for phase retrieval of noisy diffraction intensities. Journal of applied crystallography , 46(2):312-318, 2013.

Giorgio Mariani, Irene Tallini, Emilian Postolache, Michele Mancusi, Luca Cosmo, and Emanuele Rodolà. Multi-source diffusion models for simultaneous music generation and separation. In The Twelfth International Conference on Learning Representations , 2024.

Emilian Postolache, Giorgio Mariani, Luca Cosmo, Emmanouil Benetos, and Emanuele Rodolà. Generalized multi-source inference for text conditioned music diffusion models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6980-6984. IEEE, 2024.

Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting music source separation some slakh: A dataset to study the impact of training data quality and quantity. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , pages 45-49. IEEE, 2019.

Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 626-630. IEEE, 2019.

Ethan Manilow, Curtis Hawthorne, Cheng-Zhi Anna Huang, Bryan Pardo, and Jesse Engel. Improving source separation by explicitly modeling dependencies between sources. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 291-295. IEEE, 2022.

Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms. In Proc. Interspeech 2019 , pages 2350-2354, 2019.

Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics , pages 1135-1151, 1981.

Xiangming Meng and Yoshiyuki Kabashima. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12343 , 2022.

Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1219-1229, 2023.

Yu M Bruck and LG Sodin. On the ambiguity of the image reconstruction problem. Optics communications , 30(3):304-308, 1979.

MHMHHayes. The reconstruction of a multidimensional sequence from the phase or magnitude of its fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing , 30(2):140-154, 1982.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

Appendix

In the appendix, we provide further details and additional experimental results omitted in the main text. First, we present the complete derivations of the propositions for the ProjDiff algorithm (Appendix A), and then provide the derivations for ProjDiff in the variance exploding setting (Appendix B). Next, we present additional numerical results. For the image restoration tasks, we report the results of noisefree scenarios on the ImageNet dataset, both noise-free and noisy scenarios on the CelebA dataset, the nonlinear HDR task on the FFHQ dataset, and the ablation studies (Appendix C). For source separation and partial generation tasks, results on the MSDM model, and ablation experiments with waveform visualizations are shown in Appendix D. Then we provide the details of the experiments, including the mathematical form of the inverse problems, and the details for calculating metrics (Appendix E). The implementation details of ProjDiff for various inverse problems are discussed in Appendix F, especially the details for the phase retrieval task and high dynamic range task. Discussion about the performance gap between ProjDiff and the MAP framework is presented in Appendix G. Algorithm blocks for all kinds of ProjDiff used in this work are outlined in Appendix H. Finally, we include more visualizations of the restoration results to present the performance of ProjDiff intuitively in Appendix I.

A Derivation for ProjDiff in VP diffusion

A.1 Preliminaries and lemmas

Assume y = Ax 0 + n = [ I m y , 0 ] x 0 + n . For general linear observations, SVD can be used to reach this decoupled form in the spectral domain. We rewrite the observation equation as

where α t a = 1 / ( 1 + σ 2 ) for some t a ∈ [0 , 1 , . . . , T ] , y ′ = √ α t a y and n ′ ∼ N ( 0 , I m x ) . This makes the y ′ a noise-free observation for a sample at time step t a , namely the auxiliary variable x t a . We consider the DDIM reference distribution as

where the transition probability follows

where ˜ σ t is the variance hyperparameter in DDIM. We further choose ˜ σ t = σ t = √ 1 -α t -1 1 -α t √ 1 -α t α t -1 for t > t a , which makes the stochastic process between t a and T Markov. For t ≤ t a , we leave ˜ σ t a hyperparameter. Thus the reference distribution can be rewritten as

Thus

The following Lemma regarding transition probabilities can be proved by induction.

Lemma 1. The conditional distribution of x t conditioning on x t a and x 0 is

where

Proof. When t > t a , (30) holds as the chain between t a and T Markov.

When t < t a , (30) and (31) holds when t = t a as it is equal to the DDIM reference distribution. We assume (30) and (31) holds when t = t 0 ∈ { 1 , 2 , . . . , t a } , then when t = t 0 -1 , we have

q σ ( x t 0 -1 | x t 0 , x 0 ) is a Gaussian distribution according to the DDIM refernece distribution, and q σ ( x t 0 | x t a , x 0 ) is a Gaussian distribution due to the inductive hypothesis. Then by the convolution law of two Gaussian distributions, (30) and (31) hold when t = t 0 -1 . By induction, Lemma 1 concludes.

The following lemma can be proved using Bayes's formula.

Lemma 2. The conditional distribution of x t -1 conditioned on x t a and x t for t ≥ t a +2 is

where

Proof. By Bayes's formula,

Substituting the transition probability for t ≥ t a , we obtain the conclusion of Lemma 2.

The transition probability of p θ follows q σ as p θ ( x t -1 | x t ) = q σ ( x t -1 | µ θ ( x t , t ) , x t ) for t ≥ 2 , and p θ ( x 0 | x 1 ) = N ( x 0 ; µ θ ( x 1 , 1) , ˜ σ 2 1 I ) .

A.2 Proof of Proposition 1

Proof. Based on the above preliminaries, we derive the ELBO of log p θ ( x 0 , x t a ) as follows:

The KL divergence between q σ ( x T | x t a ) and p θ ( x T ) is zero as they are both N ( 0 , I ) by the setting of diffusion models. Now we handle (1) and (2) respectively.

(1) p θ ( x 0 | x 1 ) is Gaussian and thus we have

where by writing c i we denote the term independent of x 0 and x t a . q σ ( x t -1 | x t , x 0 ) and p θ ( x t -1 | x t ) are also Gaussian, so by the KL divergence of the Gaussian distributions, we have (for t ≤ t a )

(2) Similarly,

and for t ≥ t a +2

Here we apply the reparameterization trick in advance. We write

where ϵ ∼ N ( 0 , I ) , since q σ ( x t | x t a ) = N ( x t ; √ α t /α t a x t a , (1 -α t /α t a ) I ) for t ≥ t a + 1 . Substitute this into (39) and we have

E q σ ( x t a +1 | x t a ) log p θ ( x t a | x t a +1 )

where

And also (40) is (for t ≥ t a +2 )

with

Thus by integrate (37), (38), (42), and (45), we obtain the Variantional Lower bound of p θ ( x 0 , x t a ) as follows:

Here we reach the result of Proposition 1. , with the coefficient g ( t ) is given by

and w ( t ) is give by

A.3 Proof of Proposition 2

Proof. We continue the derivation by using the reparameterization trick. The reparameterization of (42) and (45) has been given above, we focus on (37) and (38) now. Applying Lemma 1, we can reparameterize them as

We further disregard all the constants. The proxy objective for minimization can be rewritten as follows

Now we seek for stochastic gradients of L . Sampling t ∼ U { 1 , 2 , . . . , T } and ϵ , ϵ ′ ∈ N ( 0 , I ) , the stochastic gradient with respect to x 0 and x t a are, respectively,

and

where the coefficient g ( t ) is omitted as it can be scaled into the step size. Here we reach the result of Proposition 2 in the main text.

A.4 Gradient truncation and remarks

In practice, we truncate the gradient through µ θ , i.e., we let

Thus the approximate stochastic gradients are, respectively,

and

At this point, we obtain the approximate stochastic gradients of the objective function with respect to the two variables x 0 and x t a . We can perform gradient descent on x 0 and the projection gradient method on x t a to solve the problem.

Remark:

1. The idea of introducing a new noise ϵ ′ is to ensure that the update direction of x t a remains as much as possible within the manifold at time step t a . The expression -( x t a - √ α t a µ θ ( · ) - √ 1 -α t a ϵ ′ ))

can be viewed as a vector pointing from x t a to √ α t a µ θ ( · ) + √ 1 -α t a ϵ ′ . As µ θ ( · ) is close to the manifold of x 0 , the introduction of ϵ ′ allows the endpoint of the gradient vector to be close to the manifold at time t a .

For a general linear observation equation, the derivation remains similar when transforming the observations into the spectral domain using SVD. Due to potentially differing noise levels for each component, the objective function will take on an element-wise form. This is based on the fact that in the diffusion model, we assume that the elements of x t -1 are independent when conditioning on x t .

In practice, we find that setting the DDIM variance ˜ σ t = 0 for t ≤ t a yields the best results, which results in γ t = √ 1 -α t and ζ t = 0 . But this poses a problem as it leads to g ( t ) →∞ . In this work, we still scale all the coefficients g ( t ) into the step sizes as it works well in the experiments. We leave this issue for future research.

Note that one may find t a is not an integer in practice, as α t a may not be in the discrete series α 0: T . This won't affect our algorithm, as it can be derived similarly that the approximate stochastic gradients in (56) and (57) still hold. The only difference lies in the initialization of x 0 which will be discussed in Appendix F.

We use the µ -predictor in the derivation. The case of ϵ -predictor is similar, with the conversion given by µ θ ( x t ) = ( x t - √ 1 -α t ϵ θ ( x t ) ) / √ α t . We refer to [19, 32] for more details about the relation between the µ -predictor, the ϵ -predictor and the Stein score [51].

B ProjDiff for VE diffusion

In this section, we derive the ProjDiff algorithm for the Variance Exploding (VE) diffusion used in the source separation and partial generation tasks. The experiments using VE diffusion in this work are all noise-free, thus we only focus on the noise-free scenarios and leave the noisy scenarios of VE diffusion for future work.

Consider the forward transition probability q ( x t | x t -1 ) = N ( x t ; x t -1 , ( σ 2 t -σ 2 t -1 ) I ) , with σ T > σ T -1 > · · · > σ 1 > σ 0 → 0 . When σ T is large enough, q ( x T ) ≈ N ( 0 , σ 2 T I ) . The initial distribution of the reverse process is set to be p θ ( x T ) = N ( 0 , σ 2 T I ) ≈ q ( x T ) and the reverse transition in the DDPM manner is

for t ≥ 2 , which matches the condition probability

And we assume the last sampling step is set to be

for some small δ 0 .

Thus the ELBO is derived as the same in DDPM [2].

D KL ( q ( x T | x 0 ) || p θ ( x T )) ≈ 0 , and log p θ ( x 0 | x 1 ) = c 1 -1 2 δ 2 0 || x 0 -µ θ ( x 1 , 1) || 2 . The KL divergence between q ( x t -1 | x t , x 0 ) and p θ ( x t -1 | x t ) is

for t ≥ 2 . Then by reparameterization, we get the Evidence Lower Bound as

with

Sampling t ∼ U { 1 , 2 , . . . , T } , ϵ ∼ N ( 0 , I ) , and truncating the gradient through µ θ as in the VP setting, a stochastic gradient for -ELBOVE (for minimization) is

Then the iteration step of the ProjDiff (noise-free) in the VE setting is

where η is the step size.

The RED-diff [24] algorithm in the VE setting can be derived similarly. In the source separation and partial generation tasks, we use these forms as the implementation of our ProjDiff and RED-diff [24] algorithms.

Remark: For VE diffusion, the Restricted Encoding method involves fixing the initial noise ϵ 0 ∼ N ( 0 , I ) and reparameterizing the sampling of x t as x t = x 0 + ξ ϵ 0 + √ σ 2 t -ξ 2 ϵ for ϵ ∼ N ( 0 , I ) . Specifically, we set ξ = σ t -1 in partial generation tasks.

C Additional experiments for image restoration

C.1 Additional results

Here, we present the omitted experiments for image restoration tasks. The results for noise-free restoration tasks on ImageNet are shown in Table 5, the results for noise-free and noisy restoration tasks on CelebA are shown in Table 6 and 7, respectively, and the results for nonlinear high dynamic range (HDR) task are in Table 8. On the CelebA dataset, we further compare ProjDiff with more diffusion based rithms, including Π GDM [22], DMPS [52], Resample [34], and DiffPIR [53]. ProjDiff demonstrates highly competitive performance on all these tasks.

C.2 Ablation study

Metrics balance in the super-resolution task. An interesting observation is the trade-off between objective metrics and perceptual metrics in the super-resolution task. DPS achieves the best perceptual metrics but performs poorly on objective metrics, while RED-diff performs well in objective metrics

27.4

27.4

ProjDiff

ProjDiff

27.2

F

Q

RED-diff

27.2

RED-diff

27

DDNM

O

DDRM

27

0

DDNM

DDRM

26.8

26.8

PSNR

PSNR

26.6

F

26.6

26.4

26.4

F

26.2

26.2

DPS

DPS

26

26

26

28

30

32

34

36

38

40

42

44

21.5

22

22.5

23

23.5

24

24.5

25

25.5

26

26.5

27

FID

LPIPS (× 10²)

for noise-free super-resolution but declines on perceptual metrics. We find that in the noise-free super-resolution task, the step size of ProjDiff controls the balance between objective and perceptual metrics. Figure 4 presents ProjDiff's FID-PSNR curve and LPIPS-PSNR curve in the super-resolution task on ImageNet, with step sizes ranging from [1 . 2 , 1 . 9] . Note that lower values of LPIPS and FID and higher values of PSNR indicate better restoration results, thus the top left corner represents the ideal performance. The red lines demonstrate the variation in objective and perceptual metrics of ProjDiff by adjusting the step size. Also note that ProjDiff exhibits superior performance compared to DDRM, DDNM, DPS and RED-diff.

Fewer steps. DDRM [20] and DDNM [21] can both address image restoration with fewer steps ( 20 steps). Here we test the performance of the proposed ProjDiff with fewer steps on ImageNet. Table 9 and Table 10 show comparisons of ProjDiff with DDRM and DDNM in three linear restoration tasks under noise-free and noisy observations, respectively. ProjDiff also demonstrates highly competitive performance across multiple metrics, indicating its potential to perform image restoration with fewer steps.

Four trials for the phase retrieval task. It is noted in [23] that in the phase retrieval task, DPS requires sampling four times for each image and selecting the best result to achieve satisfactory performance. Therefore, we compare the performance of DPS and ProjDiff with four independent samplings. We also report the performance of the ER, HIO, and OSS algorithms with four samplings as baselines. The best image is selected as the sample with the highed PSNR metric. The results are shown in Table 11. Both DPS and ProjDiff show performance improvements with four trials, with ProjDiff in particular exhibiting much better performance, reaching a PSNR of 41.58dB in the noise-free case.

The impact of the gradient truncation method used in ProjDiff. We compared ProjDiff with and without the gradient truncation method on the CelebA dataset, as shown in Table 12. ProjDiff without gradient truncation is denoted as ProjDiff-FG. The results indicates that there is some performance loss when using gradient truncation compared to not using it, while the efficiency increases by nearly three times. Considering that using gradient truncation has already achieved satisfactory performance, we accept the trade-off of some performance loss for the gain in efficiency.

Using gradient descent to approximate the projection operation for nonlinear tasks. We claim that for general nonlinear tasks, ProjDiff can be applied by using gradient descent to approximate the projection operation. Here we verify this method in the phase retrieval task (ProjDiff-GD in Table 13). The results indicate that approximating the projection operator of x 0 with respect to the observation equation by minimizing || y -A ( x ) || 2 starting from x 0 can also achieve satisfactory performance, which suggests that ProjDiff has generalizability for general nonlinear inverse problems.

Sensitivity tests for the hyperparameters. We conduct the sensitivity tests for the step size η 1 and noise level σ 0 in ProjDiff on noisy super-resolution task on CelebA, as shown in Table 14. ' × a ' denotes that we perturb the input standard deviation of ProjDiff by multiplying a . The results indicate that ProjDiff exhibits a certain degree of robustness to the step size and noise level. This also demonstrates that retaining the equivalent noise level as an adjustable hyperparameter is feasible when it cannot be directly calculated.

RED-diff with random initialization. The RED-diff algorithm is observed to accidentally fail in the 50% random inpainting task. Here we demonstrate that by adopting the random initialization method proposed in this work, RED-diff achieves more reasonable results. We compare REDdiff's performance on ImageNet for both noise-free and noisy inpainting tasks when using random initialization (denoted as 'RED-diff with random init') and using degraded images for initialization as proposed in [24]. The results are shown in Table 15. RED-diff's performance is significantly improved when using random initialization.

D Additional results for source separation and partial generation

D.1 Results on MSDM model

Two models were trained in [45]: one is MSDM which models the joint distribution among the four instruments, and the other is ISDM which models the distribution of each instrument independently. ISDM performs better than MSDN in the source separation task as reported in [45]. We have presented ProjDiff's performance on ISDM in the main text, and here we supplement the experiment on MSDM. The SI-SDRi metrics of ProjDiff, RED-diff, MSDM-Gaussian, and MSDM-Dirac are shown in Table 16. ProjDiff also demonstrates the best performance on the MSDM model.

D.2 Ablation study

Repetition steps. ProjDiff repeats N = 5 times on each time step as we state in Section 3.3 in the source separation task, while MSDM-Gaussian and MSDM-Dirac [45] only repeat twice using the correction steps method in [19]. A natural doubt is whether the performance improvement arises from more function evaluations. We test MSDM-Dirac algorithm with more steps, and the results are shown in Table 17. When increasing the correction steps from 2 to 5 which matches the function evaluations of ProjDiff, the performance of MSDM-Dirac declines. This verifies that ProjDiff's advantage does not simply arise from more function evaluations. Moreover, we test ProjDiff's performance with more repetitions using the MSDM model, namely 10, 25, and 50 times with the step size adjusting accordingly to stabilize convergence. The step size is set to 0 . 5 /N , where N is the number of repetitions. The results are shown in Table 18. Further increasing the number of repetitions leads to even better metrics for ProjDiff.

Momentum mechanism. We use Polyak momentum in the source separation task. Here we conduct experiments to validate the effectiveness of the momentum mechanism. The results with different momentum hyperparameter β ranging from 0 to 0 . 9 are shown in Table 19. Setting β to 0 . 5 yields the best performance. We infer that a certain level of momentum can accelerate convergence, while over large momentum may cause oscillations at small time steps and thus affect the results. This explains why the performance is relatively poor when β = 0 . 9 in the source separation task.

Weak observation problem and Restricted Encoding. Here we further explain why the partial generation task is a weak observation problem. The partial generation task aims to generate the tracks of other instruments based on some instruments, e.g., generating drums, piano, and guitar based on bass. The constraint provided by the observations is very weak, which is because in a musical ensemble, the correlation between different instruments exists but is not significant, and a composer can freely create the melodies of other instruments based on certain instruments' melodies. Therefore, this is more of a generation task than a restoration task, and we refer to this type of problem as the 'weak observation problem' where the constraints provided by the observations are weak.

Now we verify the effectiveness of the proposed Restricted Encoding method in the partial generation task. While keeping other parameters unchanged, we switch the sampling method of x t in ProjDiff from Restricted Encoding to sampling directly from the forward process q ( x t | x 0 ) (denoted as 'w/o Restricted Encoding'). The results are shown in Table 20. Without Restricted Encoding, the

performance of ProjDiff significantly declines. This indicates the importance of Restricted Encoding for the weak observation problem.

Visualization of the waveforms. In the partial generation tasks, an observation is that as the number of generated instruments decreases, the advantage of ProjDiff becomes smaller. We point out this is because the sub-FAD metric considers the combination of the generated and given tracks. Therefore, in cases where more instruments are provided, the combined track is closer to the ground truth, and the difference among different algorithms would be smaller. This can also explain why ProjDiff performs slightly worse than MSDM and RED-diff when generating only the tracks of guitar. This may be because ProjDiff tends to aggressively create new musical segments, thus resulting in a lower similarity to the ground truth set. We present some waveform samples in Figure 5 to verify this claim. The samples are randomly selected and the same row shares the same ID. ProjDiff's results have larger amplitudes while RED-diff and MSDM's results tend to remain silent.

Moreover, following the conventions in acoustics researches, we include some audio samples of ProjDiff in the supplementary materials for subjective evaluation.

E Experimental details

E.1 Inverse problem settings

Source separation. Denote the tracks of piano, drums, guitar, and bass as x 1 , x 2 , x 3 , x 4 ∈ R n , where n represents the length of the sequences. The observation function for source separation is y = ( (1 , 1 , 1 , 1)( x 1 , x 2 , x 3 , x 4 ) T ) T = x 1 + x 2 + x 3 + x 4 .

Partial generation. The observation matrix for partial generation is a diagonal matrix with elements of either zero or one, i.e., A = diag ( a 11 , a 22 , a 33 , a 44 ) with a 11 , a 22 , a 33 , a 44 ∈ { 0 , 1 } . 0 represents that the corresponding instruments are to be generated while 1 represents that the instrument is provided as the condition. The observation is y = ( A ( x 1 , x 2 , x 3 , x 4 ) T ) T .

Super-Resolution. The super-resolution task uses 4 × 4 average pooling as the degradation function, which is quite similar to the source separation task. Consider a 4 × 4 image block, with elements

ProjDiff

RED-diff

MSDM

x i,j ∈ R 3 , i, j = 1 , 2 , 3 , 4 . The dimension of 3 represents the RGB channels. The observation equation is thus given by y = 1 16 ∑ 4 i,j =1 x i,j . For the entire image, the calculation can be paralleled for all 4 × 4 blocks using matrix operations in PyTorch.

Random inpainting. The observations matrix for random inpainting is a diagonal matrix of zeros and ones, A = diag { a kk } for 1 ≤ k ≤ n where n is the total number of pixels in an image, specifically 256 × 256 in this work. The diagonal elements a kk have a 50% chance to be 0 and a 50% chance to be 1.

Gaussian deblurring. We follow the settings from [20, 21] for Gaussian deblurring, with a 1D Gaussian kernel of size 5 and a standard deviation of 10 . The observation equation is in the form of convolution. Efficient SVD proposed in [20] can be used for calculating the projection operator.

Phase retrieval. Our experimental setup for phase retrieval follows [23]. The observation is the amplitude spectrum of the original image. However, as the phase retrieval task is highly ill-posed, it is nearly impossible to recover the image directly from the amplitude spectrum. Therefore, we follow the standard practices [54, 55, 23] by first padding the image with zeros and then computing the amplitude spectrum. This increases the spectral resolution and provides more information for retrieval. Thus the observation equation for phase retrieval is y = | DFT ( Px ) | , where P represents the padding operation, which is a linear operator. The original image is 256 × 256 , and the padding is done with size 64 in four directions, resulting in a padded image of 384 × 384 . Note that one significant difference between the phase retrieval experiment in this paper and in [23] lies in that [23] reported the best results from 4 independent runs for each image, while we perform all the algorithms once for each image.

High dynamic range. The observation for the high dynamic range task can be written element-wise as

i.e., first multiplying all the pixels by 2 and then clipping them within [ -1 , 1] .

E.2 Hyperparameters for the compared algorithms

DDRM. DDRM uses the recommended parameters from [20], i.e. η = 0 . 85 and η b = 1 . 0 .

DDNM. DDNMuses the recommended parameter η = 0 . 85 from [21]. For noisy tasks, we report the performance of the DDNM+ algorithm [21].

DPS. Following the settings in [23], the step size is set to ξ t = ξ/ || y -A ( µ θ ( x t )) || 2 . On ImageNet, ξ is set to 1 for super-resolution and inpainting tasks, and 0 . 4 for Gaussian deblurring. Since [23] did not conduct experiments on CelebA, we conducted task-by-task parameter tuning for the learning rate in DPS and reported the results, ensuring a fair comparison. For phase retrieval and high dynamic range tasks, ξ = 0 . 4 .

RED-diff. First, in the image restoration tasks, we follow the settings in [24] using the Adam optimizer with momentum pairs of (0 . 9 , 0 . 99) . RED-diff requires a balancing parameter λ and a step size lr. λ is set to 0.25 for all image restoration tasks, and the step size lr is set to 0 . 25 for the super-resolution and 0 . 5 for all other tasks. However, in the source separation and partial generation tasks, we find that the Adam optimizer yields poor results. Therefore, we run RED-diff with the same parameters used in ProjDiff and set λ = 1 . 0 which is tuned to the best. Thus in the source separation task, the momentum is set to 0 . 5 and the step size is set to 0 . 1 with 5 repetitions. In the partial generation task, the momentum is set to 0 . 9 and the step size is set to 0 . 05 with no repetitions.

Π GDM. We carefully conducted hyperparameter searches for each task using the first eight images with the average PSNR as the metric. Except for Noisy Gaussian Deblur task, we avoided using other algorithms to initialize Π GDM to ensure a fair comparison. Specifically, we first use the forward transition to map the degraded image to the noise level corresponding to the 500th step, and then use 100 steps of Π GDM to solve the inverse problem. We find that this approach is much better than starting directly from white noise (i.e. 1000th step). For Noisy Gaussian Deblur task, Π GDM without initialization from other algorithms struggled to achieve satisfactory results. Therefore, we use DDNM+ to obtain the sample at the 500th step and then switch to Π GDM to complete the solution.

DMPS. The step size hyperparameter λ was searched for each task using the first eight images with the average PSNR as the metric.

Resample. The weight hyperparameter γ was searched for each task using the first eight images with the average PSNR as the metric. ReSample was originally designed for Latent Diffusion, thus in our experiments, we consider the encoder-decoder as identity mappings.

DiffPIR. The weight hyperparameter λ was searched for each task using the first eight images with the average PSNR as the metric.

MSDM/ISDM-Gaussian/Dirac for source separation. Following [45], the algorithms use the S churn mechanism [56], with S churn = 20 for the MSDM model and 40 for the ISDM model. Correction steps [19] are set to 2 .

MSDMfor partial generation. Similarly, S churn = 10 and no correction steps are used.

E.3 Details for metrics calculation

Here we provide the details of how we calculate the metrics for replication and fair comparison.

For image restoration, we first save the generated results in the png format. When all results are generated, we read them back in to calculate the metrics. For PSNR and SSIM, we use peak_signal_noise_ratio and structural_similarity methods from the skimage library 4 . When calculating SSIM, the parameter 'data_range' is set to 'data_range=generated_image.max()-generated_image.min()'. For LPIPS, we use the LPIPS library 5 . Note that before calculating LPIPS, pixel values are normalized to [ -1 , 1] . And for FID, we use the torch-fidelity library 6 .

For the SI-SDRi metric used in source separation, the calculation is as described in [45]:

where x n is the ground truth sequence, y represents the ground truth summation of the four instruments, and ˆ x n is the estimated track from the algorithm. α = x T n ˆ x n + ϵ || x n || 2 + ϵ and ϵ = 10 -8 to prevent numerical errors.

For the sub-FAD metric in the partial generation tasks, we first calculate the summation of the generated tracks and the corresponding conditional tracks and save the mixtures in the wav format. Then we use the frechet_audio_distance library 7 to compute the FAD metrics between the generated mixtures and the ground truth mixtures.

F Implemention details of ProjDiff

F.1 Initialization for noisy observation with non-integer t a

In the case of noisy observations, we aim to initialize x 0 using the optimized x t a as µ θ ( x t a , t a ) . However, note that t a may not correspond exactly to any discretized time step in practice. In such instance, we assume t 0 ∈ [1 , 2 , . . . , T ] such that t 0 ≤ t a ≤ t 0 + 1 , and consequently we can first perform a forward transition as ˆ x t 0 +1 = √ α t 0 +1 /α t a x t a + √ 1 -α t 0 +1 /α t a ϵ for some ϵ ∼ N ( 0 , I ) , and then initialize x 0 ← µ θ (ˆ x t 0 +1 , t 0 +1) .

F.2 Details for linear observations

The projection operator for source separation is

The projection operator for partial generation is

Super-resolution and random inpainting are similar to source separation and partial generation, respectively. The projection operators of these problems can be obtained without SVD. Regarding equivalent variance, for 4 × 4 super-resolution, based on the average of independent Gaussian variables, the equivalent variance is 16 σ 2 with σ 2 being the variance of the noise adding to the observation. For inpainting, the equivalent noise should enjoy the same variance as the noise on the observations.

Gaussian deblurring task is a bit more complex. The projection operator is

where A † is the Moore-Penrose pseudo-inverse of A . The element-wise equivalent noise can be handled using the SVD.

F.3 Details for nonlinear observations

Phase retrieval. For the phase retrieval task, consider the observation y = | DFT ( Px ) | . Given input x 0 and observations y , first we calculate z = DFT ( Px 0 ) , and compute the projection operator of y = | z | with respect to z , which is ˜ z = y ⊙ z / | z | where the division is element-wise and ⊙ denotes the Hadamard product. Next, we map ˜ z back to the original pixel space. As DFT is invertible and the P is the padding matrix, we define the inverse transformation matrix corresponding to P as P 1 that extract the central 256 × 256 pixels from a 384 × 384 image. Thus the inverse mapping can be written as ˜ x = P 1 DFT -1 (˜ z ) . Finally, as the resulting ˜ x is in the complex space, we project it onto

the real space, namely x proj = real (˜ x ) , where the real ( · ) operator takes the real part of a complex tensor.

Regarding equivalent noise, we use the principle of energy equality in the space domain and in the frequency domain. The energy of observation noise is n 2 1 σ 2 where n 1 = 384 is the size of the spectrum image. Assuming the variance of the effective noise is σ eff, the noise energy in the space domain is n 2 σ 2 eff with n = 256 . Equating both energy yields σ eff = n 1 σ/ n = 1 . 5 σ .

High dynamic range. The regular projection operator of the HDR task can be expressed element-wise as follows

This formulation applies to the noise-free case. Nonetheless, we discover an adjusted version of the projection operator that can further accommodate noisy observations, which is

This adjusted form is more adept at handling uncertainty in the observation values within the intervals [ -1 , -0 . 5] and [0 . 5 , 1] induced by the observation noise. We employ this modified projection operator in the noisy HDR task. The equivalent noise variance can be simply approximated as σ 2 eff = ( σ/ 2) 2 .

F.4 Hyperparameters.

In all experiments, ProjDiff uses SGDM as the optimizer. The momentum is set to 0 . 5 for the source separation task and 0 . 9 for the partial generation task. The step size is set to 0 . 1 for source separation with 5 repetition steps and 0 . 05 for partial generation with no repetition. For all image restoration tasks, the momentum is set to 0 , and η 2 for the noisy observations is fixed to 1 . 0 . The DDIM variance

˜ σ t is set to 0 for t ≤ t a and to ˜ σ t = σ t = √ 1 -α t -1 1 -α t √ 1 -α t α t -1 for t > t a . No repetitions are applied. Other tuned optimal step sizes are shown in Table 21. The time step schedule follows [45] for the source separation and partial generation tasks, and follows [20, 21] for the image restoration tasks.

G Discussion about the MAP framework

ProjDiff shares similarities with the Maximum A Posteriori (MAP) estimation method. Theoretically, in noisy situations, it is only necessary to use a Gaussian likelihood, i.e., p ( y | x 0 ) = N ( y ; x 0 , σ 2 I ) , to solve the inverse problems. However, ProjDiff introduces a new auxiliary variable, which leads to better experimental outcomes than the MAP framework. Here, we present some qualitative discussion towards this experimental result.

On one hand, if we are given perfectly accurate priors and likelihoods, and have sufficient computational capability, we could obtain the exact posterior and also its gradients. Within the MAP framework, this should lead to the ideal results. However, in practice, the priors provided by diffusion

models are not entirely accurate, and the weight coefficients between the priors and the likelihood cannot always be perfectly set. Moreover, dealing with the priors in diffusion models relies on stochastic optimization methods. These factors imply that the MAP framework often fails to achieve the desired solution. In such cases, further exploring the information or capabilities within the diffusion model can provide substantial assistance in solving inverse problems and enhance performance. This is one of the reasons why ProjDiff may have a performance advantage over the original MAP framework.

On the other hand, MAP methods utilize gradient descent to leverage the information from the observation, while ProjDiff transforms this into a projection operation by introducing noisy auxiliary variables. This approach offers numerous advantages: there is no need to consider the step size of gradient descent (at least for the likelihood term); there is no need to consider the weight coefficients between the likelihood and prior terms; and it ensures consistency between noisy samples and observations (the role of this consistency has also been confirmed in the noise-free scenario). Moreover, the introduction of this auxiliary variable indicates that ProjDiff can actually be viewed as simultaneously recovering clean data and the noise added to the observations, which can yield more accurate results than merely characterizing the noise prior with a Gaussian distribution. These transformations are all thanks to ProjDiff's utilization of both the clean prior and noisy prior modeled by diffusion models (i.e., the prior modeled by the diffusion model and its denoising capability).

H Algorithm blocks

Here, we present additional algorithm blocks for ProjDiff used in the experiments. Algorithm 2 is used for noise-free restoration tasks. Algorithm 3 and Algorithm 4 are used in the source separation task and partial generation task, respectively.

Algorithm 2 ProjDiff for VP diffusion (noise-free).

Require: Observation y , observation function A , pre-trained diffusion model µ θ , step size η , total steps T , noise schedule α . . . α .

1 T 1: Sample ϵ T ∼ N ( 0 , I ) ; 2: Initialize x 0 ← µ θ ( ϵ T , T ) ; 3: for t = T to 1 do 4: Sample ϵ t ∼ N ( 0 , I ) ; 5: Calculate the approximate stochastic gradient: ˜ d x 0 ,t = x 0 -µ θ ( √ α t x 0 + √ 1 -α t ϵ t , t ) ; 6: Update x 0 : x 0 ←P A , y ( x 0 -η ˜ d x 0 ,t ) ; 7: end for 8: return x 0

Algorithm 3 ProjDiff for VE diffusion (noise-free).

Require: Observation y , observation function A , pre-trained diffusion model µ θ , step size η , momentum β , total steps T , iterations per step N , noise schedule σ . . . σ .

1 T 1: Sample ϵ T ∼ N ( 0 , I ) ; 2: Initialize x 0 ← µ θ ( σ T ϵ T , T ) ; 3: v ← 0 ; 4: for t = T to 1 do 5: for k = 0 to N -1 do 6: Sample ϵ t,k ∼ N ( 0 , I ) ; 7: Calculate the approximate stochastic gradient: ˜ d x 0 ,t = x 0 -µ θ ( x 0 + σ t ϵ t,k , t ) ; 8: Update momentum: v ← β v +(1 -β ) ˜ d x 0 ,t ; 9: Update x 0 : x 0 ←P A , y ( x 0 -η v ) ; 10: end for 11: end for 12: return x 0

Algorithm 4 ProjDiff for VE diffusion with restricted encoding (noise-free).

Require: Observation y , observation function A , pre-trained diffusion model µ θ , step size η , momentum β , total steps T , iterations per step N , noise schedule σ . . . σ .

+

I Visualization results of image restoration

GT

y

Baseline

DDRM

DDNM

DPS

RED-diff

Ours

Deblur

Inpaiting

SR4

GT

y

Baseline

DDRM

DDNM

DPS

RED-diff

Ours

Deblur

Inpaiting

SR4

GT

y

Baseline

DDRM

DDNM

DPS

RED-diff

Ours

Deblur

Inpaiting

SR4

GT

y

Baseline

DDRM

DDNM

DPS

RED-diff

Ours

Deblur

Inpaiting

SR4

HDR

Phase retrieval

GT

DPS

RED-diff

Ours

y

HDR

Phase retrieval

GT

DPS

RED-diff

Ours

y

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction are consistent with this paper's main contributions and scope, and have been verified through extensive experiments and ablation studies.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Appendix A and B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code is included in the supplemental materials and the experiment details are presented. See Appendix E and F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and instructions are included in the supplemental materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All of the experiment details and implemention details are presented. See Appendix E and F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Following the convention of inverse problem research, we report the performance comparison of all algorithms with a fixed random seed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All of the experiments are conducted on a single NVIDIA 3090ti GPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics in detail and conformed with it in this paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper focuses on designing algorithms for inverse problems, without underlying social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No datasets or models are proposed and there is no such risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All databases, models, and toolkits used in this paper are cited from the original papers or URLs.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code provided in the supplementary materials is anonymized and well documented with the running details.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose ProjDiff which reframes noisy inverse problems with diffusion models as a two-variable constrained optimization by introducing an auxiliary optimization variable. Authors derive a two-variable ELBO as a proxy for the log-prior and solve the optimization problem via projection gradient descent. Authors conduct comprehensive experiments on several image restoration tasks (super-resolution, inpainting and deblurring), source separation, and partial generations tasks.\n\nSTRENGTHS:\n* Overall, paper is easy to follow and structured nicely.\n* I appreciate extensive numerical results that are presented in the paper.\n* Numbers for source separation look significantly better than the baselines (I have to note that I'm not an expert in that area)\n\nWEAKNESSES:\n* Prior work section is not comprehensive.\n* The gains in performance compared to baseline methods (especially for noisy restoration tasks) is not convincing. Moreover, hyperparameter search for baseline models were not conducted and wrong baseline method was used (DDNM for measurements with noise).\n* See the questions below.\n\nQUESTIONS:\n* I have several questions regarding DDNM numbers. First of all, DDNM [1] is designed for noise-free image reconstruction problems. I believe that is the reason why DDNM on Gaussian deblurring performs bad (as in Table 1: 7db PSNR, 0.03 SSIM, etc.). I would suggest the authors to switch to the DDNM+ variant as described in [1] for noisy inverse problems. If authors are already using DDNM+, could you clarify why the performance is bad on Gaussian deblurring?\n* line 37-39: \"However, it’s worth noting that, since diffusion models are inherently effective at denoising, considering the observation noise in the likelihood term fails to fully leverage diffusion models’ denoising capability.\" Could the authors clarify what it means to not fully leverage diffusion models here?\n* I would recommend the authors to include $\\Pi$GDM [2] in their comparisons especially since it performs much better than DDRM.\n* Some other missing citations on solving inverse problems with diffusion models: CCDF [3], latent diffusion models: PSLD [4]\n* line 663-665: \"Since [23] did not conduct experiments on CelebA, we use the parameters on FFHQ for the CelebA dataset as both FFHQ and CelebA are datasets of faces.\". I don't think this is a good practice. In my experience, even though both CelebA and FFHQ are face datasets, DPS is not robust to the choice of step size. I believe some hyper-parameter search on a small set of images is necessary for fair comparison.\n* Do you think ProjDiff can be extended for using latent diffusion models as a prior?\n---\nReferences:\n\n[1] Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-shot image restoration using denoising diffusion null-space model.\" arXiv preprint arXiv:2212.00490 (2022).\n\n[2] Song, Jiaming et al. “Pseudoinverse-Guided Diffusion Models for Inverse Problems.” International Conference on Learning Representations (2023).\n\n[3] Chung, Hyungjin, Byeongsu Sim, and Jong Chul Ye. \"Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[4] Rout, Litu, et al. \"Solving linear inverse problems provably via posterior sampling with latent diffusion models.\" Advances in Neural Information Processing Systems 36 (2024).\n\nLIMITATIONS:\nLimitations are adequately addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new sampling strategy for solving noisy inverse problems using diffusion models. The proposed method is called ProjDiff, which is based on two-step minimization of log-posterior using gradient descent. The sampling procedure is derived by (1) Reparametrization of the noisy measurement as a noiseless measurement of an intermediate noisy sample in the diffusion process, and (2) obtaining a proxy objective from the variational lower bound.\n\nSTRENGTHS:\n+ The paper's approach to addressing the noisy inverse problem using the reparamterization of measurement is interesting. \n\n+ Deriving the sampling procedure by directly minimizing the evidence lower bound is innovative. \n\n+ The performance is competitive with the SOTA methods in solving inverse problems.\n\nWEAKNESSES:\n+ The paper is not well-written and it is hard to follow. Justification: \n\n     1. Theoretical analysis results lack organization. Example: Lemma 1 and Lemma 2 should be separated from the main proof. The proofs for the propositions should also be separated and distinctly expressed. \n\n     2. Theoretical results should be explicitly included. The proofs for Lemma 1 and Lemma 2 are not included in the text. If taken from a source, then the reference should be mentioned, otherwise, the proofs should be included. \n\n    3. Abstract includes some concerning statements.\n        >Since inverse problems inherently entail maximum a posteriori estimation, previous works have endeavored to integrate diffusion priors into the optimization frameworks.\n\n        Inverse problems could be solved using MMSE estimation. It is unclear what kind of optimization algorithm the authors refer to. \n       \n        > by introducing an auxiliary optimization variable. By employing gradient truncation, the projection gradient descent method is efficiently utilized to solve the corresponding optimization problem. \n\n        The abstract does not reflect the method used.  The statement is vague and could be inferred in many different ways. \n\n       Revising the abstract to better reflect the proposed method can strengthen the paper.\n\n  4. The second and third paragraph of the introduction needs to be revised. Instead of briefly mentioning what each existing method does, the authors could state how their method is different from the literature and how it contributes to the literature. \n\n\n\n* The truncation of the gradient is based on an assumption and leads to an approximation in the solutions. Thus, the reduced computational cost should not be mentioned as a contribution of the paper, specifically when the authors have not included any experimental results regarding this. \n > 44: Through gradient truncation, we obtain an approximation of the stochastic gradient of the objective, effectively sidestepping significant computational overhead.\n\n\nOverall, the weaknesses of the paper are mostly related to the presentation of the paper and the blurred message, not the proposed method and effectiveness of it. \n\nMinor issues: \n\n> Line 19: Their remarkable ability to capture data priors enables effective guidance for solving inverse problems [6], which are widely exploited in image restoration. \n\nGuidance in inverse problems usually refers to enforcing data consistency with the measurement and relies on the forward model.\n\nQUESTIONS:\nWhat does the author mean by weak observations? (Line 214)\n\nHow does the algorithm look when used for nonlinear inverse problems? Can equation (8) be written for nonlinear measurements?\n\n\nHow do the authors set the $\\eta_1$, $\\eta_2$, and equivalent noise level in the algorithms?\nAre the solutions sensitive to these parameters?\n\nLIMITATIONS:\nPartially addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new way of solving inverse problems using a diffusion model based prior.\nThe key idea is that the forward process, which is to be undone, usually involves Gaussian noise.\nThe authors utilize this by viewing the noisy observation as a as random variable at an intermediate time step t of the diffusion forward process.\nThis allows them to formulate the inverse problem as a joint optimisation problem for x_0 and x_t given the noisy observation.\nAs a by product, their method is able to solve inverse problems which do not involve noise as well.\nThe authors evaluate their method and achieve superior results compared to various baselines.\n\nSTRENGTHS:\n* I believe this is a principled new perspective on diffusion models for inverse problems and I am sure it can be potentially useful for many practical applications.\n\n* The method achieves remarkable results.\n\n* I very much appreciate the fact that the method can be applied to noise-free inverse problems, without having to artificially include non-existent Gaussian noise in the model.\n\nWEAKNESSES:\n* The motivation, comparing the method to the more established MAP approach, could be a bit more clear.\nThe authors write: \n\"In this work, we introduce a novel two-variable ELBO by constructing an auxiliary variable that accounts for the observation noise, thereby utilizing both the prior information and the denoising capability in diffusion models simultaneously.\"\nWhy exactly is is describable to utilize the denoising capability of the model? Is the something wrong with the normal way of including the log of the forward model in the optimisation task? Why can we expect an improvement? Is it a problem of optimisation? Surely the way task itself is normally (e.g. [28]) formulated is correctly, or not?\n\n* Some aspects of the method were difficult for me to follow. This may be due to my inexperience with some of the involved mathematics.\nIn particular, I wonder if the SVD based decoupling could be explained or illustrated differently to facilitate understanding.\n\nQUESTIONS:\nRegarding the SVD composition, I believe I am missing an important aspect about how the noisy observation can be considered as a step in the diffusion process when it involves A.\nLet's say A corresponds to a convolution with a blur kernel. Would this not mean that we are considering a blurred noisy image to be part of the diffusion process. This would be incorrect, because the blurring would not be part of the training data statistics which would only include clean images, right?\n\nLIMITATIONS:\nMy understanding is that the method is limited to either image degradation processes that involve Gaissian noise, or are noise-free.\nIs this the case?\nIf yes, this would be a significant limitation with respect to practical applicability, e.g. in settings with Poisson shot noise.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ProjDiff for solving inverse problems with pre-trained diffusion models. By deriving a two-variable ELBO as a proxy for the log-prior, this paper reframes the inverse problems as constrained optimization tasks and address them via the projection gradient method.\n\nSTRENGTHS:\n1. The paper writing is clear and easy to follow.\n2. The derivation of some formulas in this paper is solid.\n\nWEAKNESSES:\n1. The assumption of this method for the measurement noise in Section 3.1 is doubtable. (1) As the authors claim in the limitation section, assuming the Gaussian noise will limit its applicability to other noise types like Poisson or multiplicative noise; (2) The whole method design heavily relies on know the exact standard deviation of the Gaussian noise \\sigma, which is impractical and can lead to robustnesses issues when facing unknown noise level. Actually, the noise estimation itself is a challenging problem, particularly on the degraded measurement A(x). Please check the related works [1,2]. In general, this method is built on an impractical scenario, so I am worried about the practical usage of this method.\n\n2. The approximation for implementing the update rules just after Proposition 2 is not convincing. (1) One of the reasons for this approximation is that \"the μ-predictor of the diffusion model should be resilient to small perturbations in the input\", but this hope usually is not the truth, particularly when t is large, i.e., at the beginning of the reverse sampling procedure; (2) There is not much evidence for this important approximation, no matter theoretical or empirical evidence.\n\n3. The proposed method is too \"delicate\", as shown in Algorithm 1. (1) Again, this method needs to know the measurement noise level; (2)\nthis method contains several hyper-parameters. In Table 17, it is clear that the hyper-parameters are heavily tuned, and there is no specific way to guide the hyper-parameter tuning.\n\n4. No ablation studies for the hyper-parameters. For this \"delicate\" method, a systematical ablation study is needed. For example, it would be good to report the results for different combinations of some important hyper-parameters.\n\n5. The extension to nonlinear inverse problems is not convincing. This paper proposes to use min_x || y - A(x) ||^2 to approximate the projection operator. However, it is almost impossible to solve nonlinear inverse problems by using this formulation because it will be easy to be stuck in local minimizers. Also, if this formulation can solve nonlinear inverse problems well, then there is no need to write this paper.\n\n6. The experiment settings for phase retrieval are doubtable. (1) I highly suspect that this paper downplays DPS in some implicit ways. I ran DPS for phase retrieval on FFHQ before, and remembered DPS could achieve over 30dB for PSNR after trying different initializations. In the original DPS paper, the authors claim that DPS need different initializations but it seems that this paper omits this; (2) there is no comparison with the golden standard method for phase retrieval, i.e., HIO+ER.\n\n7. Some recent SOTA methods are missing for comparison, including ReSample[3] (ICLR'24 spotlight), DiffPIR[4] (CVPR'23), DMPS[5]. Please check the recent survey for more related works [6].\n\n\n\n[1] Liu, X., Tanaka, M. and Okutomi, M., 2013. Single-image noise level estimation for blind denoising. IEEE transactions on image processing, 22(12), pp.5226-5237.\n\n[2] Li, F., Fang, F., Li, Z. and Zeng, T., 2023. Single image noise level estimation by artificial noise. Signal Processing, 213, p.109215.\n\n[3] Song, B., Kwon, S.M., Zhang, Z., Hu, X., Qu, Q. and Shen, L., 2023. Solving inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123.\n\n[4] Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R. and Van Gool, L., 2023. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1219-1229).\n\n[5] Meng, X. and Kabashima, Y., 2022. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12343.\n\n[6] Li, X., Ren, Y., Jin, X., Lan, C., Wang, X., Zeng, W., Wang, X. and Chen, Z., 2023. Diffusion Models for Image Restoration and Enhancement--A Comprehensive Survey. arXiv preprint arXiv:2308.09388.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-18"}