Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper deals with diffusion models for inverse problems. An important drawback of this approach is that to produce satisfactory results, it is necessary to iterate the forward and the reverse diffusions several times. Based on the manifold hypothesis, the authors show that this occurs because the score function only acts on the normal direction of the manifold, and therefore a drift occurs pushing the inference path out of the manifold. To solve this, besides the measurement constraint equation, they propose to include a constraint on the reverse diffusion equation given by the gradient of the $L^2$ residual of the inverse problem for the Bayes optimal denoising step from Noise2Noise (Kim and Ye, Neurips 2021). The authors prove that this constraint forces the diffusion to lie on the data manifold. These claims are supported by theoretical proofs and a clear geometric interpretation. 

strengths_and_weaknesses: ## Strengths

(+) The paper is very well written  
(+) The idea of combining Noise2Score and diffusion models is novel, interesting, and completely sound.   
(+) The proposed strategy is fully backed up by the theortical arguments and proofs provided by the authors.   
(+) Thorough comparison with state-of-the-art diffusion model-based approaches and supervised learning-based baselines confirm the superiority of the proposed approach.   

## Weaknesses

(-) It would be nice to show and discuss failure cases, or situations when the proposed approach does not outperform the others.  

Minor comments:

- table X, figure Y, section Z, etc. --> Table X, Figure Y, Section Z, etc.
- Eq. 3: $x_t$ --> $x(t)$
- Fix punctuation at the end of Eqs. 6 and 9
- L71: $\mathbb{R}^n$ --> $\mathbb{R}^m$
- L76: utilize
- L77: rely
- Eqs. 13 and 14: for consistency, write them in the discrete setting
- L144: uses
- L154: recall the definition of $p_0$ (it was only defined in Section 2)
- SuppMat, L465-466: check the sentence
- SuppMat, Fig. 6: there are two lines in red that should be in green
 - SuppMat, L502: $\epsilon_\theta$ --> $z_\theta$
 - SuppMat,L507: (4) --> Table 4
 - SuppMat, L509: (1) --> Algorithm 1

questions: None
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a simple augmentation to existing score-based diffusion solvers for inverse problems by applying an extra correction term in the diffusion step. This term, in theory, captures the on-manifold component of the current estimation error.

-------------

After rebuttal: The authors have convinced me that the empirical experiments are sound, and have also tempered their theoretical claims. I am happy with the paper in its present state, as I am convinced that it presents a novel technique for inverse imaging, and achieves a solid improvement on the prior state of the art.
strengths_and_weaknesses: Strengths:
- The proposed step is elegant and computationally simple.
- The results, if correct (see below), suggest massive improvements in signal recovery.

Weaknesses:
- The main theorem does not state its assumptions clearly. Moreover, I don't believe they hold, and this is quite problematic for the proposed algorithm. (See overview.)
- The empirical experiments are somewhat problematic. (See overview.)
- (Minor) The use of Bayes rule in Section 3.1 to obtain Eqs. 11 and 12 is somewhat misleading; this is a general technique for conditional sampling in score-based diffusion models, see for e.g. [1, 2, 3].

Overview:
Overall, I am very excited by the direction proposed here, but I think there are currently major issues in the reasoning and execution of the paper. I believe the paper will be much stronger after addressing these problems. Ultimately, my concerns are two-fold:

Theoretically, I am not convinced that the main result of the paper, upon which the proposed algorithm rests, holds in practice. Theorem 1 requires Proposition 1 to hold, and Proposition 1 assumes that the score function is a *global minimizer* of the denoising loss (Eq. 9). This is nearly impossible in a diffusion model, since the denoising function at t=T (or t=0 in the reverse process) must "recover" an image from pure noise --- this is a highly ill-defined problem. Another way to see that this cannot hold: If the score function was a global minimizer, the generative process for unconditional diffusion models would take 1 inference step (in practice, it takes thousands of steps in the vanilla diffusion model). The suboptimality of the score function has consequences that extend past theory, as the central manifold constraint gradient (MCG) term also relies on the global optimality of the score function to define manifold projection (Eq. 13). Thus, this key assumption is quite central to the entire paper.

Empirically, the visual quality of competing work seems like it could be somewhat misrepresented, especially in image inpainting. For example, in Figure 3, the visual quality of RePaint [4] and LaMa [5] both differ greatly from that reported in their respective papers (see Figure 1 for each paper). Moreover, looking more closely at the experimental setup of the competing works RePAINT and LaMa, the two papers share the same experiments for inpainting, and establish a baseline. Namely, their inpainting task involves recovering an original image corrupted by "Wide" and "Narrow" masks. This paper uses a different set of masks, and reports significantly different LPIP scores for RePAINT and LaMa for this task. To be clear, I think it is fair to change the experimental setup. However, I wonder if the authors did not spend enough time tuning the competing models in this new setup, and consequently misrepresented the competing works.

[1] Dhariwal, P., & Nichol, A. (2021). Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 8780-8794.
[2] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. (2015, June). Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (pp. 2256-2265). PMLR.
[3] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.
[4] Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. (2022). Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11461-11471).
[5] Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., ... & Lempitsky, V. (2022). Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2149-2159).

questions: See (Overview) in Strengths and Weaknesses for questions / points to address.
limitations: No.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a framework for solving inverse problems using diffusion models as priors. The idea is to introduce another constraint in the formulation of the posterior distribution that enforces the generated sample to be on the manifold of natural images. The introduced constraint is motivated by the Tweedie formula, that relates the score function (of the noisy data distribution) and the best MMSE estimator (denoising).
Several experimental results on different imaging inverse problems show that the proposed method outperforms existing methods in FID, LPIPS, PSNR and SSIM metrics.
strengths_and_weaknesses: Strengths:
- The paper introduces an interesting adaptation to traditional sampling methods motivated by the Tweedie formula. 
- The paper is generally well written and the idea is well motivated.
- Several experimental results on different imaging inverse problems show that the proposed method outperforms existing methods in FID, LPIPS, PSNR and SSIM metrics.

Weakness:
- The paper exposition could be better:
- The paper relates the idea a lot to noise2score, but the key motivation is the Tweedie formula. With that in mind it would be better to introduce the Tweedie formula and relevant work better (see below)
- Motivation for the additional constraint is not clear (Section 3.1). In particular Eq (14) deduction is unclear (\mathcal{C} is a set, so how should one interpret p(\mathcal{C} | x) ?
- How is obtained W for each application.
- Theoretical findings should be better connected to the proposed algorithm. 

In general I like the paper, the idea is interesting, and the experimental results show a clear win in performance. The main current issue to me is that the motivation and exposition is not clear enough. Most of the pieces are already there but a reorganization and better discussion is needed. 

questions: For completeness, I re-list here some of the questions raised above:

1. Tweedie formula vs Noise2score.The key motivation seems to be coming from the Tweedie formula. With that in mind it would be better to introduce the Tweedie formula and relevant work better. See for example Section IIB in [Ong2019]:

[Ong2019] Ong, F., Milanfar, P. and Getreuer, P., 2019. Local kernels that approximate Bayesian regularization and proximal operators. IEEE Transactions on Image Processing, 28(6), pp.3007-3019.

2. Motivation for the additional constraint is not clear (Section 3.1). In particular Eq (14) deduction is unclear (\mathcal{C} is a set, so how should one interpret p(\mathcal{C} | x) ? Could you please elaborate more on how Eq (14) is introduced (motivation/deduction). 

3. How is obtained W for each application, it is unclear where this W is coming from. Explaining better this term in (14) will help a lot to the understanding of the algorithm.

Other minor comments:
- l59 -  p_t(x) should be log p_t(x)
- Eq (3) - \epsilon is undefined maybe just add "where \epsilon <<1".
- Eq (18), Eq (20),  Eq (22). In all these equations it says "We chose W…" This implies that W is not deducted from the model but it's something that we need to choose. It is unclear how to do this and how it's related to the introduced motivation where everything seem to be coming naturally from the Tweedie formula. 

I'm more than happy to increase my final rating if during the rebuttal we manage to clarify some of the aspects regarding this work.

limitations: Limitations are correctly discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose a new methodology to perform conditional sampling using denoising diffusion models. Namely, they combine existing conditional score-matching techniques using a stochastic contraction with a slight change of the score. Indeed, instead of considering the unconditional score functional they consider the score function associated with the conditional distribution given by some manifold constraint. The authors show that this term is in fact complementary to the existing unconditional score term as the score term only (locally) project on the data manifold, whereas the additional term moves locally on the tangent space of this manifold. They evaluate their methodologies against some baselines and show promising results for inpainting, colorization and CT-scan reconstruction.
strengths_and_weaknesses: STRENGTHS:

* The paper introduces a new methodology for inpainting which makes sense from a theoretical point of view, is easy to implement and seems to yield good results.

WEAKNESSES:

* I think that a deeper discussion with recent works such as [1] is missing. It is not clear from reading the main document what is the state of the art conditional sampling method using diffusion models (at least previous to this work).

* In terms of novelty it seems that the proposed correction was already introduced in [2]. As a consequence, the method proposed by the authors can be see as a generalization to arbitrary linear problems (as stated by the authors) and therefore has limited novely.

* With the stochastic contraction term and the gradient term proposed by the authors it is not clear at all if we are sampling from the right posterior distribution. I understand that the method yields good visual results but in fields like CT-reconstruction the visual quality should not be the only metrics as we also need a models which provide good uncertainty quantification. Such considerations are missing here. As a first question: can you characterize what is the invariant measure of the proposed diffusion model?

[1] Kawar et al. (2022) -- Denoising diffusion restoration models
[2] Ho et al. (2022) -- Video diffusion models
questions: * The authors state that "Accordingly, the score function ∇x log pt (x) in (2) should be replaced by the conditional score $\nabla_x \log p_t(x|y)$. Unfortunately, this strictly restricts the generalization capability of the neural network since the conditional score should be retrained whenever the conditions change." This is not necessarily true as the score could be trained using some amortization w.r.t. y. In this case $y$ can be simply passed as another input to the diffusion model.

* I don't really understand how the authors derive (14) from (13) and (6). From what I see, the authors are simply plugging $\nabla \log p(y|\hat{x}_0)$ in the drift of the backward Markov chain. This makes a lot of sense, but I find the explanation provided by the authors to be quite convoluted.

* Assumption 1 is not the manifold assumption as understood in the literature. Usually the manifold assumption ensures that the data lives on a lower dimensional manifold but do not impose any local linear structure.  It is quite hard to believe that the data of any reasonable image distribution is linear locally. I understand that the authors need this assumption to make their results rigorous but this should be clearly stated.

* I really enjoyed reading about Proposition 2 and Theorem 1. I think that together with Figure 2 they bring a lot of insight about the method. However, the authors do not discuss at all the effect of the projection step $A x_i' + b$. I would have really appreciated an ablation study comparing the effects of these two terms which both introduce some information about the measurement.

* I am not an expert in this domain but I have to admit that I am a bit surprised by the FID scores obtained by the authors which seem to be dramatically better than the rest of the methods. Could the authors provide details about the experimental process and in particular how they train the different models? For example do you perform hyperparameters tuning.

* I think that the comparison with DDRM is a bit biased because DDRM only use 20 steps to recover the data whereas in the appendix the authors indicate that they use between 1000 and 2000 steps depending on the task. The trade-off between sampling quality and NFE is not investigated. Similarly, when comparing with RePaint the authors chose N = 200.
limitations: The limitations are discussed in the conclusion.
I found this discussion to be appropriate.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper suggests to improve the performance a diffusion-based reconstruction scheme by adding to its updates the gradient of a log-likelihood term that uses the link between $x_i$ to $x_0$.
Some mathematical motivation is provided.
strengths_and_weaknesses: (1). The introduction part is lacking. You should mention other methods that use the same pre-trained priors for solving many (almost arbitrary) reconstruction tasks (i.e., in an unsupervised fashion).
For example, GAN-based reconstruction methods [a],[b] and the powerful plug-and-play (P&P) denoisers approach [c],[d],[e].
The P&P denoisers approach [c], which is much older than diffusion-based reconstruction methods, commonly uses general-porpose CNN denoisers to impose the prior within iterative schemes [d], [e]. 
So, in fact, diffusion-based reconstruction methods can be understood as P&P variant with class-specific denoisers (determined by the training data) that can handle extreme noise levels (which makes them generative models).
I expect these topics to be discussed in the introduction.

(2). The continuous formulation of the diffusion process in the introduction part seems redundant.

(3). Fix the notation: in Eq. 15 you mean y rather than y0, right? 
Also, the "argument" of $\hat{x}_0(\cdot)$ is $x_i$, right? this should be clearly written.

(4). The presentation of the MCG ingredient needs to be improved. Essentially, the "manifold constraint" is the information on the prior of the signal: the transformation from $x_i$ to $x_0$ (e.g.,the constraint in Eq. 13 is independent of $y$).
Yet, in the term that is added to Eq. 15 (compared to Eq. 7) you already connect it with the log-likelihood term (data fidelity term).
Thus, it should be clearly stated throughout the paper that this new term combines data fidelity term and the prior information of the transformation from $x_i$ to $x_0$.
Furthermore, I tend to believe that there exist other works that also use (perhaps in a similar way) the log-likelihood function in their iterative diffusion reconstruction schemes. Please make sure that you cover the exiting literature. 

(5). Note that for $\alpha=0$ your additional term is canceled. Thus, your experiments should include ablation study and also some discussion and examination of the effect of different values of alpha.

(6). Please verify that there is no mismatch between Eq. 15 and the algorithms that you actually use in the experiments.
For example, in Algorithm 1 in Appendix C, line 10 is not aligned with Eq. 15 (alpha is multiplied with a matrix).
Moreover, observe that your MCG component in line 10 of Algorithm 1 vanishes(!) when you compute (I-P'P) * d/dxi ||P'*(y0-P*x0)||:

(I-P'P) * d/dxi ||P'*(y0-P*x0)|| = (I-P'P) * d/dx0 ||P'*(y0-P*x0)|| * dx0/dxi = (I-P'P) * P'P*(P*x0-y0) * dx0/dxi = 0,

because (I-P'P) * P'P=0, since for inpainting P is m rows of the identity matrix I_n so P'P is a projection matrix.

(7). In your algorithm, *all* the choices of $A$ and $b$ for the different tasks (including CT) makes Eq. 16 (and Eq. 8) coincide with the "back-projection step" in [e].
The iterative usage of such a data consistency step with pre-trained denoisers (which, as mentioned above, is very similar to diffusion-based reconstruction) has been proposed in [e] (and theoretically analyzed in follow-up works). Such very related works should be mentioned.

(8). The competitors in the experiments seem somewhat weak. For example, the recovery that is presented to some of them changes even the known pixels in inpainting. Recoveries that do not utilize such known information are obviously weak. Hence, it would have been interesting and informative to see comparison with strong (non-naive) GAN-based reconstruction methods, such as [b] (obviously, with GANs that are trained on the same training data as the other methods, e.g., by using common datasets and GANs).
The method in [b] is a clear example of a method that does not fall into the authors' statement that "the ability to maintain perfect measurement consistency ... is often not satisfied for unsupervised GAN-based solutions".

(9). In the experiments section, you must include a discussion on the number of iterations and run-time of the different methods that are examined.


[a] Bora, A., Jalal, A., Price, E. and Dimakis, A.G., 2017, July. Compressed sensing using generative models. In International Conference on Machine Learning (pp. 537-546). PMLR.

[b] Hussein, S.A., Tirer, T. and Giryes, R., 2020, April. Image-adaptive GAN based reconstruction. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 3121-3129).

[c] Venkatakrishnan, S.V., Bouman, C.A. and Wohlberg, B., 2013, December. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing (pp. 945-948). IEEE.

[d] Zhang, K., Zuo, W., Gu, S. and Zhang, L., 2017. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3929-3938).

[e] Tirer, T. and Giryes, R., 2018. Image restoration by iterative denoising and backward projections. IEEE Transactions on Image Processing, 28(3), pp.1220-1234.

questions: My questions and comments are listed above.
limitations: My questions and comments are listed above.
review_title: null

Meta-review: metareview: This paper proposes to incorporate manifold constraints in diffusion model (for inverse problems). The general consensus after rebuttal is that this submission is worthy of acceptance to NeurIPS. Please incorporate the remaining reviewers' feedback for the camera-ready version.
confidence: Certain
Decision: Accept