Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the efficiency problem of the unsupervised approach in image restoration. The authors proposed DDRM, a denoising diffusion generative model that uses variational inference to learn the the posterior distribution of a linear inverse problem for image restoration. Specifically, the authors define a diffusion process of image restoration using variational distributions, where the trainable parameters of the distributions can be learnt by maximizing an ELBO objective. The experiments on ImageNet show that DDRM has good performance and outperforms previous methods. 
strengths_and_weaknesses: I like the paper in that it presents a solid and sound technical approach with clear theorems and proofs for using diffusion generative models for image restoration. The focus of the paper is on defining the variational objective and the relevant distributions in the diffusion process, which allows efficient inference. The contribution is novel and significant. 

The results of the paper appear to be convincing enough. Both quantitative and qualitative results demonstrate strong performance compared to previous methods. 

The writing is great. The paper reads well and relatively easy to understand. 

On the downside, the experiments shown in the paper can be more extensive, and some analyses on the degradation model could be added. As the proposed method is claimed as an unsupervised method for image restoration with priors learnt on ImageNet, it is a pity that only synthetic degradation models are shown in the results. As ImageNet is a dataset with real images, it would be nice to understand further how well the real degradation can be handled with the proposed model. 

questions: (1) From the visual results, it seems that sometimes the results are oversmooth, e.g., see the hand in Figure 4. It would be nice to evaluate the output images with a metric on image structure similarity, e.g., SSIM. Could the authors consider adding this metric to Table 1 or 2? 

(2) In Table 1 and 2, how many NFEs are required for DDRM to outperform previous methods? Could the authors also provide the runtime statistics (memory, time) for each NFE? These numbers might support the efficiency claim better.

(3) Section 3.3: from a practical point of view, what happens if we learn a different model for each inverse problem? Would that give even better performance?
limitations: 
- There are a few typos in the manuscript, e.g., delubrring, diffsuion. Please use a spell checker to double check.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes Denoising Diffusion Restoration Models (DDRM), which is the first general sampling-based inverse problem solver for various unsupervised image restoration tasks. DDRM outperforms existing baselines and is efficient in terms of NFEs. Also, DDRM is more beneficial when there is noise in the input images.
strengths_and_weaknesses: Strength

- The proposed method requires much fewer iterations compared to existing diffusion-based methods.

- The proposed method was experimented on various image restoration tasks including super-resolution, inpainting and colorization.

Weakness

- As mentioned in the Conclusions, the proposed method is focused on linear inverse problems, so there is likely to be a weakness in the non-linear inverse problem that exists in real data. It would be good to suggest the possible limitations and the analysis when the proposed method is applied to the non-linear inverse problem setup.

- Also, there are many cases where the degradation operator is unknown at the inference stage. Therefore, it would be good to have a mention on how to apply the proposed method in a practical real-world situation (degradation operator is unknown).

- Although the above two limits seem out-of-scope of this paper, it  would be good to analyze the proposed method from various angles by experimenting with various data and setups. For example, it is possible to consider various data sets, image resolution and quality, as well as non-linear inverse problem cases and unknown degradation operators.

questions: - It is preferred to report quantitative results according to the number of iterations for the proposed method. What happens if more than 20 iterations are applied for the proposed method in Table1,2?

- Most of the experiments were performed on ImageNet. Does the proposed method work well for frequently used datasets for each task? For instance, Set5, Set14, B100, Urban100 and Manga109 for super-resolution. What happens to verify the proposed method by applying it to real-world data such as medical imaging mentioned in the introduction?

limitations: - PSNR and KID were used as evaluation metrics. I am curious about quantitative results when other evaluation metrics are utilized for image reconstruction problems, such as BRISQUE, NIQE, PIQE, RankIQA, MetaIQA. 

BRISQUE: Mittal, Anish, Anush Krishna Moorthy, and Alan Conrad Bovik. "No-reference image quality assessment in the spatial domain." IEEE Transactions on Image Processing (TIP) 21.12 (2012): 4695-4708.

NIQE: Mittal, Anish, Rajiv Soundararajan, and Alan C. Bovik. "Making a “completely blind” image quality analyzer." IEEE Signal Processing Letters (SPL) 20.3 (2012): 209-212.

PIQE: Venkatanath, N., et al. "Blind image quality evaluation using perception based features." 2015 Twenty First National Conference on Communications (NCC). IEEE, 2015.

RankIQA: Liu, Xialei, Joost Van De Weijer, and Andrew D. Bagdanov. "Rankiqa: Learning from rankings for no-reference image quality assessment." Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017.

MetaIQA: Zhu, Hancheng, et al. "MetaIQA: Deep meta-learning for no-reference image quality assessment." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method for the inverse problem based on diffusion model. Attributed to the unsupervised mechanism, it is able to 
generalize well in several tasks as shown in the paper.
strengths_and_weaknesses: Strengths:
1. As shown in Theorem C.2, the ELBO objective is equivalent to the form of that in DDPM/DDIM. Thus, the proposed method can readily employ the well-trained DDPM/DDIM model, which avoids the expensive training phase.

2. The proposed method is with fine generalization capability since it does not rely on specific degradation assumpution.

Weaknesses:
The main weakness of this work mainly focuses on the experiments. The experimental is too simple compared with related research works. Specifically,

1. In super-resolution, it only consider the block averaging filter. To the best of my knowledge, this is rarely used in super-resolution community.  Bicubic downsampling is the commonly used to evaluate the performance of different methods. I suggest the author should provide more experimental results under such setting, and compared with more recent methods.

2. For deblurring, aside from the more challenged motion blur case, the simple anisotropic Gaussian deblurring should be at least considered.

3. Even though announced to be left in future work, the proposed method relies on the pre-known degradation operator H, which is inaccessible in most of scenarios. Generally, this is a beautiful but unpractical work.
questions: See Weaknesses above.
limitations: This work is a non-blind method, and relies on the pre-provide degradation operator, which largely limits its practical value.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies image restoration problem and develops an efficient solution called denoising diffusion restoration model (DDRM). The key idea is to borrow principles from variational inference and exploit a pre-trained denoising diffusion generative model. This is conceptually similar to the popular pre-trained models in high-level vision (i.e., one pre-trained model supports different down-stream tasks). The DDRM has been tested on several image restoration tasks including SR, deblurring, inpainting, and colorization. The fast running speed is one of the salient features of the proposed DDRM.
strengths_and_weaknesses: Strengths:
+ The paper is well written and easy to follow.
+ DDRM is appealing because it makes little assumption about the degradation model (i.e., unsupervised approach).
+ Computational efficiency of DDRM is attractive for real-world applications.
+ DDRM has achieved improved restoration performance over other competing approaches including DGP (ECCV'2020) and SNIPS (NeurIPS'2021)

Weaknesses:
- It seems to me that the convergence issue of the DDRM is not discussed in the paper. Will DDRM-based image restoration always converge after 20 or 100 steps?
questions: 1. When DDRM produce a sequence of samples (restored images) like SNIPS, can you estimate their MSE like SURE does? If Yes, it might be possible to obtain a weighted (instead of unweighted) average in Fig. 4 of Supplementary Material.
2.  What assumption do you make about the degradation model H? Since Eq. (3) does SVD on H, H cannot be arbitrary, right? From Fig. 6, the current deblurring experiments assume optical blur. Can DDRM be applied to motion blur (but still satisfy the separable condition in D.5 of Supplementary)?
3. I am a bit confused by Fig. 1 in the Supplementary Material. What is the difference between "DDRM (20)" and "Denoised"? I assume "Denoised" means an MMSE denoiser. But the text can be more clear. Meanwhile, does DDRM reach a fixed-point iteration - i.e., x=f(x), where f is the denoiser? The motivation here is that we might assume clean images do satisfy such fixed-point condition. 
limitations: n/a
review_title: null

Meta-review: metareview: In this paper, the authors present a variational inference approach based on a diffusion model to learning the posterior distribution of an unsupervised linear inverse problem.  The presented method is evaluated on multiple linear inverse problems and beats state of the art unsupervised inversion methods while running significantly faster than them.  The algorithm has strong theoretical support, in the form of a rigorously proven equivalence with unconditional denoising with diffusion models.  The paper could be improved by a more extensive set of experiments, including investigation of stability against incorrect degradation models.  Overall, the algorithmic advance with a strong theoretical foundation outweighs these weaknesses, and the paper is recommended for acceptance.
confidence: Less certain
Decision: Accept