Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to use a pre-trained diffusion model as a generative prior for BFR. With a diffused estimator, the LQ images with different degradation can be transformed into the latent space of the diffusion model. A weighted L2 loss has been proposed to train the diffused estimator. Generally, the paper is well-written with good performance.
strength_and_weaknesses: Strength: 
1. The idea of bringing the diffusion model into BFR task is somewhat new but the contribution of this paper is not significant. 
2. The results of DifFace are better than current state-of-the-art methods and extensive experiments and analysis have been provided to evaluate the proposed method.
Weaknesses: 
1. The main contribution of this paper is designing a diffused estimator to transform the degraded facial images into the latent space of diffusion model, which is not new [1] and this contribution is very limited.
2. The proposed DifFace model has the advantage of dealing with severe degradation. How about the real degradation face images with fewer degradation? Can the DifFace still restore the identity well?
3. It was claimed that this paper theoretically shows that the Markov chain can compress the predicted error by a factor of less than 1. However, no theoretical analysis can be found in this manuscript.
4. It was claimed that the proposed approach avoids using GAN and perceptual loss. However, it is not difficult to use such loss and stable results can also be obtained by those works such as GFPGAN, VQFR. In the proposed method, Eq.(9) introduce a weighted factor k_N, which strictly decreases monotonically with the timestep N. How to choose k_N and a_N? The number of inference sampling times N is also manually chosen. Compared with previous methods, the proposed method requires a more manual and empirical setting.
5. It’s still unclear why p(x_N |y_0) is a Gaussian distribution as shown in Eq.(8). How about the Laplace distribution? 
6. Since the diffusion model requires a large number of sampling, i.e., 400 in this work, the inference time of DifFace is too slow compared with other methods. The number of sampling need further ablation study. 

[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020.

clarity,_quality,_novelty_and_reproducibility: 
The paper is well-written and easy to follow. However, the novelty of the proposed method is limited. 

summary_of_the_review: Although this paper achieves good performance, the contribution and novelty of this paper are not enough.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a method for Blind Face Restoration (BFR) using pre-trained diffusion models for faces. The main novelty is a design of a transition model from a low quality image that estimates an intermediate state of the diffusion process of a high quality image. This allows the method to use a pre-trained diffusion model as the image restoration back-bone. The paper is equipped with extensive analysis to motivate the approach. The paper also provided qualitative and quantitative comparisons on BFR task with several previous art. The results are generally in favor of the proposed method demonstrating robust face restoration quality. 
strength_and_weaknesses: Strength:

1. The paper is well written and a pleasure to read. The paper posed a very simple yet effective method for face restoration. The method is well motivated and explained, with extensive analysis of the method some design choices, e.g., choice of starting timestep N.

2. The idea is novel to me. The paper proposed a method to leverage a pre-trained diffusion model for BFR. The method is novel in the sense that it constructs a transition distribution to infer an intermediate state of the diffusion process of a high quality image from a low quality counterpart. 

3. The results are generally convincing. The proposed method seems to be more robust than most previous methods that are used as baselines in the paper, with less artifact and higher fidelity in visual comparisons. Quantitative evaluations also demonstrated better performance than previous art.

Weakness:

I don't have much complaints or concerns as the author has already mentioned that one limitation of the method is that, by having to go through the diffusion process, the performance is slow. 
One question that I have is the choice of the training data for the generative model. Since variants of diffusion models has been used in training image generation models for diverse images, would training on data beyond human faces provide better prior for face restoration? In Figure 13, the hand restoration is clearly failing. Is this due to the lack of training data diversity of the diffusion model since they are only trained on FFHQ?



clarity,_quality,_novelty_and_reproducibility: The paper has good quality and clarity. It is overall a good read and the work is original in my viewpoint. 
summary_of_the_review: I think the paper proposed a simple and original method for blind face restoration. The paper is well written. The experimental results are convincing. I hold a positive view of the paper and I recommend accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method DifFace, being able to cope with unseen and complex degradations more gracefully without complicated loss designs. Comprehensive experiments show that DifFace is better than current state-of-the-art methods, especially in cases with severe degradations. 
strength_and_weaknesses: Pros:
1. The motivation is clear. 
2. The paper is well-written and organized.
Cons:
1. More results on more challenging datasets are needed to verify the superiority of the proposed method.
2. Some related works are missing, e.g., Toward High-Quality Face-Mask Occluded Restoration, Joint Face Image Restoration and Frontalization for Recognition
clarity,_quality,_novelty_and_reproducibility: The authors devise a new diffusion-based BFR approach to cope with severe and unknown degradations. The notion of formulating the posterior distribution as a Markov chain that starts from the LQ image and ends at the desirable HQ image is novel. We theoretically show that the Markov chain can compress the predicted error by a factor of less than 1.
The authors show that the image prior captured in a pretrained diffusion model can be harnessed by having the Markov chain built partially on the reverse diffusion process. Such a unique design also allows us to explicitly control the restoration’s fidelity and realism by changing the Markov chain’s length.
The authors show that BFR can be achieved without complicated losses as in existing methods. The proposed method only needs to train a neural network with the basic L2 loss, simplifying the training pipeline.
summary_of_the_review: See Strength And Weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of blind face restoration. To overcome the two limitations (lack of generalization property and requirement of multiple constraints), this paper proposes a new approach based on the latest diffusion model in the latent space. The key idea is to learn a posterior distribution from LQ to HQ with the help of a transition distribution from LQ to an intermediate state of pre-trained diffusion model. The design of transition distribution takes into account both the efficiency of training process and robustness to unknown degradation. Extensive experimental results are reported to demonstrate the superiority of the proposed DifFace to other competing methods.
strength_and_weaknesses: + The motivation behind DifFace is clear. In view of the wide impact of probabilistic diffusion models on image synthesis, it is not surprising to see that LDM can be applied to low-level vision tasks such as blind face restoration.
+ The design of transition distribution in Sec. 3.2 and its discussion in Sec. 3.3 are easy to follow. Model analysis in Sec. 4 offers some good insight about the interpretability of model parameters (starting timestep N and diffused estimator f).
+ The reported subjective quality comparison results are convincing (e.g., Fig. 2, Fig. 7 and Fig. 8). Objective quality comparison in Table 1 also clearly favors the proposed method DifFace.

- I had thought this work will work in the latent space such as latent diffusion model (LDM, CVPR'2022). Based on the discussion in Sec. 5.4, this work has not been extended into the latent space. I am just curious about the authors' further thought about this line of extension. Maybe StyleGAN will serve as a natural candidate for the latent space?
- "All models are wrong; some are useful". It will be nice to show some failure examples for the proposed DifFace. How about its performance in the presence of occlusion or severe pose variation or atmospheric turbulence (e.g., IARPA BRIAR scenario)? 
- If you supply Fig. 2(g) as the input image to DifFace, will it output an undistorted image? In other words, have you verified that the restored face image is a fixed point for the proposed restoration operator?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The issue of reproducibility is not addressed in the paper. But based on its detailed discussion in Sec. 4, I tend to give authors some benefit of doubt. It should not be difficult to reproduce the results reported in this paper.
summary_of_the_review: I am mostly impressed by the reported experimental results of this paper. Its theoretical contribution is OK but incremental (the derived diffusion model still works in the pixel instead of the latent space). The computational complexity and the model parameters of this work do not compare favorably over existing methods such as GLEAN and DFDNet.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of blind face restoration. The paper proposes a method called difface based on an application of diffusion models. The method performs very well for face restoration problems. The method is relatively slow owing to its dependence on diffusion models. 

The paper provides an extensive evaluation, specifically, it evaluates on three real-world datasets and compares the performance to several reasonable baseline methods.

Three critiques that remain are the following. Those critiques and the paper were discussed amongst the AC and the reviewers during a virtual meeting and through an online discussion. 
- The paper is a relatively straightforward application of diffusion models to image (face) restoration, and the paper focuses on the relatively narrow problem of face restoration. While the paper provides excellent for this setup, some consensus emerged that in light of serval other works studying image reconstruction with diffusion models, the contribution is somewhat limited.
- The paper writes 'We theoretically show that the Markov chain can compress the predicted error by a factor of less than 1'. It was unclear to several reviewers where this statement comes from, and this is somewhat important, since the authors write that 'the main contribution of this work is to design a novel and theoretically sound posterior distribution for blind face restoration.', so I and the reviewer expected a concrete analytical statement, which is not provided. The authors explained on openreview where this statement emerges, but the paper still doesn't contain an explanation. Also, the error contraction is more an observation based on one equation rather than a proof or a statement, which the abstract suggests. 
- No supplement to re-produce the results is provided.

justification_for_why_not_higher_score:  The paper is a relatively straightforward application of diffusion models to image (face) restoration, and the paper focuses on the relatively narrow problem of face restoration. While the paper provides excellent for this setup, some consensus emerged that in light of serval other works studying image reconstruction with diffusion models, the contribution is somewhat limited.
justification_for_why_not_lower_score: The method in the paper performs very well, and the evaluation is nearly flawless in that it considers several suitable datasets for face restoration and compares to well-chosen baselines.
Decision: Reject