DifFace: Blind Face Restoration with Diffused Error Contraction
Authors: Zongsheng Yue
Chen Change Loy
Venue: Submitted to ICLR 2023 (2023)
Keywords: Face Restoration
Diffusion Model
Super-resolution

Abstract: While deep learning-based methods for blind face restoration have achieved unprecedented success, they still suffer from two major limitations. First, most of them deteriorate when facing complex degradations out of their training data. Second, these methods require multiple constraints, e.g., fidelity, perceptual, and adversarial losses, which requires laborious hyper-parameters tuning to stabilize and balance their influences. In this work, we propose a novel method named DifFace, being able to cope with unseen and complex degradations more gracefully without complicated loss designs. The key of our method is to establish a posterior distribution from the observed low-quality (LQ) image to its high-quality (HQ) counterpart. In particular, we design a transition distribution from the LQ image to the intermediate state of a pre-trained diffusion model and then gradually transmit from this intermediate state to the HQ target by recursively applying a pre-trained diffusion model. The transition distribution only relies on a restoration backbone that is trained with L2 loss on some synthetic data, which favorably avoids the cumbersome training process in existing methods. Moreover, the transition distribution is capable of contracting the error of the restoration backbone and thus makes our method more robust to unknown degradations. Comprehensive experiments show that DifFace is superior to current state-of-the-art methods, especially in cases with severe degradations. Code and model will be released.

Under review as a conference paper at ICLR 2023
DIFFACE: BLIND FACE RESTORATION WITH DIFFUSED ERROR CONTRACTION
Anonymous authors Paper under double-blind review
ABSTRACT
While deep learning-based methods for blind face restoration have achieved un- precedented success, they still suffer from two major limitations. First, most of them deteriorate seriously when facing complex degradations out of their training data. Second, these methods require multiple constraints, e.g., fidelity, perceptual, and adversarial losses, which requires laborious syper-parametted tuning to stabi- lize and balance their influences. In this work, we propose novel method named DifFace, being able to cope with unseen and complex degradations more grace- fully without complicated loss designs. The key of our method to establish a posterior distribution from the observed low-quality (LQ) image to its high-quality (HQ) counterpart. In particular, we design transition distribution from the I. image to the intermediate state of pre-trained diffusion model and then gradu- ally transmit from this intermediate state to the HQ target by recursively applying pre- traitra diffusion model. The transition distribution only relies restora- tion backbone that is trained with L2 loss on some synthetic data, which favorably avoids the cumbersome training process in existing methods. Morcover, the tran- sition distribution is capable of contracting the error of the restoration backbone and thus makes method more robust to unknown degradations. Comprehen- sive experiments show that DifFace is superior to current state-of-the-as methods, especially in cases with severe degradations. Code and model will be released.
INTRODUCTION
Blind face restoration (BFR) aims at recovering a high-quality (HQ) image from its low-quality (LQ) counterpart, which usually suffers from complex degradations, such as noise, blurring, and downsampling. BFR is an extremely ill-posed inverse problem as multiple HQ solutions may exist for any given LQ image.
Approaches for BFR have been dominated by deep learning-based methods (Wang et al., 2021; Tu et al., 2021; Feihong et al., 2022; Gu et al., 2022). The main idea of them is to learn a mapping, usually parametrized large amount of pre-collected L.Q/QQ image pairs. In most cases, these image pairs are synthesized by assuming a degradation model that often deviates from the real one. Most existing methods are sensitive to such a deviation and thus suffer a dramatic performance drop when encountering mismatched degradations in real scenarios.
deep neural network, from the LQ images the HQ ones based on
Various constraints have been designed to mitigate the influence of such a deviation and improve the restoration quality. The L2 (or L1) loss is commonly used ensure fidelity, although these pixel-wise losses are known to favor the prediction of an average (or a median) over the plausible solutions. Recent BFR methods also introduce the adversarial loss (Goodfellow et al., 2014) and the perceptual loss (Johnson et al., 2016; Zhang et al., 2018) to achieve more realistic results. Besides, some existing methods also exploit face-specific priors to further constrain the restored solution, e.g., face landmarks (Chen et al., 2018), facial components (Li et al., 2020), and generative priors (Chan et al., 2022; Pan et al., 2021; Wang et al., 2021; Yang et al., 2021). Considering so many constraints together makes the training unnecessarily complicated, often requiring laborious dyper-parameters tuning to make a trade-off among these constraints. Worse, the notorious instability of adversarial loss makes the training more challenging.

---

Under review as a conference paper at ICLR 2023
Pa 1x<
Xr
XN
XN-1
X0
Xx (xxpoo)
HQ Iniage
H+
Yo
((:w)
Diffused Estimator
LQ Image
Figure I: Overview of the proposed method. The solid lines denote the whole inference pipeline of our method. For ease of comparison, we also mark out the forward and reverse processes of the diffusion model by dotted lines.
(a) LQ
(b) DFDNet
(c) PSERGAN
(d) GLEAN
(c) GEPGAN
(1) VQFR
(g) Ours
Figure 2: Comparative results of recent state-of-the-art methods and the proposed method typical real example. From left to right: (a) low-quality image, (b)-(g) restored results of DFDNet (Li al., 2020), GLEAN (Chan et al., 2022), PSFRGAN (Chen al., 2021), GFPGAN (Wang al., 2021), VQFR (Gu et al., 2022), and our proposed method.
In this work, we devise a novel BFR approach DifFace, inspired by recent success of diffusion model (Sohl-Dickstein et al., 2015) in image generation (Dhariwal & Nichol, 2021). Our method does not require multiple constraints in training, and yet capable of dealing with unknown and complex degradations. Importantly, we leverage the rich image priors and strong generative capa- bility pretrained diffusion model without retraining on any manually assumed degradations.
To this end, we establish posterior distribution p( aiming to infer the HQ image no con- ditioned on its IQQ counterpart yo- Due to the complex degradations, solving for this posterior non-trivial in blind restoration. Our solution to this problem, as depicted in Fig. 1, is to approximate this posterior distribution by a transition distribution N (Boo), where xN a diffused version of the desirable HQ image æp. followed with reverse Markov chain that estimates from æv. To construct this transition distribution p( p (yo), we introduce deep neural network which can be simply trained just using L2 loss.
Such a transition distribution is appealing in that it is well motivated by an important observation in DDPM (Ho et al., 2020), where data is destroyed by re-scaling it with factor of less than and adding noise the diffusion process. Bringing this notion to our context, the residual between 2o and yo is also contracted by this factor after diffusion. Our framework uniquely leverages this property by inferring the intermediate diffused variable N (where N < '') from the LQ image yo- of which the residual HQ image Wy is reduced. And then from this intermediate state infer the desirable Zg. There are several advantages of doing so: i) our solution is more efficient than the full reverse diffusion process from Tr 2g, ii) we do not need retrain the diffusion model from scratch, and iii) we can still take advantage of the pre-trained diffusion model via the reverse Markov chain from x. to mo.
SR3 (Saharia et al., 2022b) also exploits the potentials of diffusion model for blind restoration. It feeds the LQ image into the diffusion model as a condition to guide the restoration at each timestep. This requires one to retrain the diffusion model from scratch on pre-collected training data. Hence, would still suffer from the issue of degradation mismatch when dealing with real-world data. Dif- ferent from SR3, our method does not need to train the diffusion model from scratch but sufficiently leverages the prior knowledge contained the pretrained diffusion model. The unique design on transition distribution p( (yo) further allows cope with unknown degradations.
2

---

Under review as a conference paper at ICLR 2023
In summary, the contributions of this work are follows:
We devise a new diffusion- based BFR approach to cope with severe and unknown degradations. The notion of formulating the posterior distribution a Markov chain that starts from the LQ image and ends at the desirable HQ image is novel. We theoretically show that the Markov chain can compress the predicted error by factor of less than 1.
We show that the image prior captured having the Markov chain built partially on the reverse diffusion process. Such unique design also allows us to explicitly control the restoration's 's fidelity and realism by changing the Markov chain's 's length.
pretrained diffusion model can harnessed by
We show that BFR can be achieved without complicated losses as in existing methods, Our method only needs to train neural network with the basic L2 loss, simplifying the training pipeline. preview of our restored results compared with existing methods are shown in Fig. 2.
PRELIMINARIES
We provide a brief introduction to the diffusion probabilistic model (Sohl-Dickstein et al., 2015) (known diffusion model for brevity) ease the subsequent presentation of our method.
Diffusion model was proposed in Sohl-Dickstein et al. (2015), mainly inspired by the non- equilibrium statistical physies. Ho al. (2020) built up novel connection between diffusion model and denoising scoring matching. Furthermore, Song et al. (2020) proposed unified framework formulate the diffusion model from the perspective of stochastic differential equation (SDE). Attributed to its sound and perfect theoretical support, diffusion model has achieved impressive suc- cess in generation of images (Ho et al., 2020; Dhariwal & Nichol, 2021), audio (Chen al., 2020), graph (Niu et al., 2020) and shapes (Cai et al., 2020).
Diffusion model consists of forward process (or diffusion process) and a reverse process. Given data point with probability distribution g(#n), the forward process gradually destroys its data structure by repeated application of the following Markov diffusion kernel:
q = N(x;; v1 1, t t=
1.2......
where BI...., Br is pre-defined or learned noise variance schedule. With rational design on the variance schedule, theoretically guarantees that q(x;) converges to the unit spherical Gaussian distribution. It is noteworthy that the marginal distribution at arbitraty timestep has the following analytical form:
q( mo) (aze Vaizo, (1 -o)
where C₁ [1 -B)).
As for the reverse process, it aims to learn a transition kernel from æ; to æ;-1, which is defined as the following Gaussian distribution:
po po( M (x;-1; t). Ex t)),
(3)
where 0 is the learnable parameter. With such learned transition kernel, approximate the data distribution ((Zn) via the following marginal distribution:
po(xo) f p(xT)
II [=1
pe xz, xz, dx1.2
(4)
where p(œT) NaT:; 0, I).
PROPOSED METHOD
In this section, we present our BFR method, DifFace, and discuss how to exploit the image priors encapsulated in pretrained diffusion model. To keep the notations consistent with Sec. we denote the LQ image and HQ image as yo and Øg- To restore the desirable HQ image from its degraded counterpart, need to design rational posterior distribution of o p( (y)).
3

---

Under review as a conference paper at ICLR 2023
(b) N--10
(c) N-200
(d) N-100
(6) N-400
( Ni-500
(g) N--600
Ubi N-700
Figure 3: Illustration of the diffused N (top row) and the reconstructed results (bottom row) by the pre-trained diffusion model from different starting timesteps. Note that the employed diffusion model trained with 1000 discrete steps following Dhariwal & Nichol (2021).
3.1 MOTIVATION
Considering a diffusion model with T' discrete steps, provides transition function from a; to Tt-1. With the aid of this transition, we can construct the posterior distribution p(za Wo) as follows:
p p(æNl10)) II (0 (x))1
(5)
where 1 S. N < T is arbitraty timestep. To restore zo from you we can leverage the ancestral sampling (Bishop & Nasrabadi, 2006) as follows:
IN p(zN (o). I:-1 x (11)) N, I.....1.
(6)
Since the transition kernel po (0;-11 x.2) our goal thus turns to design the transition distribution //o)
be readily borrowed from the pretrained diffusion model,
We have an important observation by diving into Eq. (6). If replacing p(æN 13b) with the marginal distribution x (0) defined in Eq. (2), it degenerates into the diffusion and reconstruction process for æ6 via the pretrained diffusion model, i.e.,
EN q( (00)
Pt- [ pe (xt-11 N. 1...,,
Diffusion
Reconstruction
In Figure 3, we show some diffused and reconstructed results under different settings for the starting timestep N. One can observe that when N lies in reasonable range (e.g., N < 500), meaning that æo is slightly "de- stroied", it is possible to accurately reconstruct æg using the pretrained diffusion model.
-15
This observation indicates that N is an ideal choice for the desired p(æv|3n My) by reasonable N. Since the HQ image Zo is inaccessible in the task of BFR, we thus explore how to design plausible p(æNl (y)) to approximate x(xN)
100
Figure 4: The change curves of KN and aN with the starting timestep N.
3.2 DESIGN
Recall that our goal is to design transition distribution p(a, Mo) to approximate x For- tunately, the target distribution g( n (o) has analytical form as shown in Eq. (2). This inspires us to formulate N as a Gaussian distribution as follows:
p (30) A' (æN; VaN flyo: w), ax))))
where f(;: w) is neural network with parameter w, aiming provide initial prediction for øu- should be noted that the final restored result by our method is achieved by sampling from the whole Markov chain of Eq. (6) (see Fig. 1), and not directly predicted by [(;w)) As for ((:w), it
4

---

Under review as a conference paper at ICLR 2023
is only used to construct the marginal distribution of æN, diffused version of zu, and thus named as "diffused estimator" in this work.
Next consider the Kullback-Leibler (KL) divergence between the designed p(æN (yo) and its target g( . By denoting the predicted error of f(-; w) as e 2g - f(yo: w), we have
DKL. [p(æN (a mn)
2 Kw llll
where KiN oN As shown in Fig. 4, AN strictly decreases monotonically with the timestep N. Hence, larger N will offer better approximattion to q(æN a2)) and further achieve more realistic image via the reverse process from N to Zo of the pretrained diffusion model. However, 2N will contain more noises when is getting larger, as shown in Fig. 3. Thus, an overly large N will inevitably deviate the restored result from the ground truth to. Therefore, the choice of N induces a cualism-fidelity trade-off for the restored HQ image. We provide the ablation study in Sec. 4.
3.3 DISCUSSION
Instead of directly learning mapping from yo the desirable æn under multiple constraints like current deep searning-based methods (Chen et al., 2021; Yang et al., 2021; Wang et al., 2021), the proposed method turns to predict æN. diffused version of zo. from you Such new learning paradigm brings several significant ad advantages compared to current approaches. Specifically,
According to the designed p(æN (B) in Eq. (8), we can access the diffused æN as follows:
xN = VaN f(Vo w) + / aN)c = VaNzO Vane + v(1 ax)c.
(10)
where e = zu - f(yo: w). C ~ (N(i,, It can be seen that the predicted error e is contracted by factor of vaN that is less than 1 as shown in Fig. 4.
Attributed to such an error contracttion, our method has a greater error tolerance to the diffused estimator f(: w). Hence, f(; w) can be simply trained with L2 loss some synthetic data in this work. This unique flexibility circumvents the sophisticated training process compared with most recent methods (Li et al., 2020; Wang al., 2021; Gu al., 2022). Our experiments (Sec. 5) show that the proposed method consistently obtains satisfactory restoration results over existing methods, especially when dealing with severe and complex degradations.
After obtaining the diffused æN via Eq. (10), our method gradually generates the desirable HQ result by sampling recursively from (2 /2r)) starting from = N and ending at t = 1. Through this sampling procedure, we effectively leverage the rich image priors and powerful generation capability of the pretrained diffusion model to help the restoration task. Unlike existing methods, since the diffusion model is completely trained on the HQ images in unsupervised manner, it thus reduces the dependence of our method the manually synthesized training data, of which the distribution may deviate from the true degradation.
In summary, designed posterior distribution p(z yo) in Eq. (5) renders a more robust learning paradigm for BFR.
4 MODEL ANALYSIS
In this section, we analyze the influence of the starting timestep N and the diffused estimator [(;; our model. We consider two elassical network architectures the backbone for [(; i.e., SRCNN (Dong et al., 2015) and SwinIR (Liang et al.. 2021).
Training Settings. We train the diffused estimator f(; w) on the FFHQ dataset (Karras et al., 2019) that contains 70000 HQ face images. We firstly resize the HQ images into a resolution of 512 x 512, and then synthesize the LQ images following a typical degradation model used in recent literature (Wang et al., 2021):
y ー= f1(=
[[<x> * ki) t. +n In- T..
(11)
where and x are the LQ and HQ image, k; is the Gaussian kernel with kernel width 1, n, is Gaussian noise with standard deviation a, is 2D convolutional operator, d.s and ts are the Bicubic
5

---

Under review as a conference paper at ICLR 2023
LQ
(11) (211
N060
(02)
(42)
(12)
(22)
NH600
Figure 5: An example restored by Difface under different settings of the starting timestep N and the diffused estimator f(: ()) (a) LQ image, (b) HQ image, (c1)-(c2) restored results by SRCNN and SwinIR, (dI)-(il) restored results by Difface that takes SRCNN as diffused estimator, (d2)-(12) restored results by DifFace that takes SwinIR as diffused estimator.
O OFPOAN
VOFR
4AV2RR
+
12705
Scale
Stale
(6)
(c)
Figure 6: From left right: (a) FID and LPIPS with respect the starting timestep N the validation dataset, (b)-(c) LPIPS and IDS with respect to the downsampling scale factor (i.e., 8 in Eq. (11)) the testing dataset.
downsampling upsampling operators with scale and [:]PEG, represents the JPEG compression process with quality factor q. And the hyper-parameters I, 8, a, and q are uniformly sampled from [0.1, 15). [0.8, 32). jo, 20). and [30, 100] respectively. To evaluate the performances of DifFace under different settings for and f(: we selected 3500 HQ images from CelebA-HQ (Karras et al., 2018) and generated the LQ images via Eq. (11) our validation dataset.
We adopt the Adam (Kingma & Ba, 2015) algorithm to optimize the network parameters w under L2 loss. The batch size is set as 16, and other settings of Adam follow the default configurations of Pytorch (Paszke et al., 2019). We train the model for 600k iterations, and the learning rate is decayed gradually from le-4 to le-6 with the cosine annealing schedule (Loshchilov & Hutter, 2017). As for the diffusion model, it contains 1000 discrete steps with linear noise schedule, and is trained on FFHQ (Karras et al., 2018) dataset based the official code of Dhariwal & Nichol (2021).
Starting Timestep. In Fig. 5, we show some restored results by DifFace under different settings for the starting timesteps N. More real-world visual examples can be found in Fig. of Appendix. The results show that one can make a trade-off off between realism and fidelity through different choices of N. particular, we larger N, the restored results would appear more realistic but with lower fidelity in comparison to the ground truth HQ images. The phenomenon is reflected by the average FID (Heusel et al., 2017) and LPIPS (Zhang et al., 2018) scores with respect to N in Fig. 6 (a). The proposed DifFace performs very well in the range of [400, 450), and thus set N as 400 throughout the whole experiments in this work. In practice. we speedup the inference process four times following Nichol & Dhariwal (2021), and thus sample 100 steps for each testing image.
Diffused Estimator. Figure 5 displays an example restored by DifFace, which either takes SR- CNN (Dong al., 2015) SwinIR (Liang et al., 2021) the diffused estimator f(: w). More de- tails and comparisons on these two backbones are provided in Appendix C. Even with the simplest SRCNN that only contains several plain convolutional layers, DifFace is able restore plausible
'https://gi thub. htt.b..coooppea.. /gui ded-di ffusiio

---

Under review as a conference paper at ICLR 2023
Table 1: The quantitative comparisons of different methods CelebA-Test. The best and second
best results are highlighted in bold and underline, respectively.
Methods
Metries
Input
DFDNet
PULSE
PSFRGAN
GLEAN
GFPGAN
VQFR
DifFace
PSNRT
23.41
23.15
22.14
22.74
23,41
22.18
21.94
24.08
SSIMT
0.702
0.629
0,682
0,630
0.666
0.631
0.585
0.703
LPIPS
0.604
0.554
0,508
0,500
0.469
0.495
0.471
0.435
FID
283.48
64.65
48.27
52.08
59.84
26.90
25.74
18.20
IDS
78,32
86.21
74.97
68.14
67.13
66.76
65.87
62.39
PSFRGAN
GRPGAN
Figure 7: Visual comparisons of different methods on the synthetic dataset CelebA-Test.
HQ image under severe degradations. Using elaborated architecture like SwinIR results in more apparent details (c.g., hairs). The results suggest the versatility of DifFace in the choices of the diffused estimators. In the following experiments, we use SwinIR as our diffused estimator for f(;w).
EXPERIMENTAL RESULTS
In this section, we conducted extensive experiments to verify the effectiveness of DifFace on both synthetic and real-world datasets. The training follows the settings detailed in Sec. 4.
5.1 EVALUATIONAL SETUP
Testing Datasets. We evaluate DifFace on one synthetic dataset and three real- world datasets. The synthetic dataset, denoted as CelebA-Tess A-Test, contains 3500 HQ images from CelebA-HQ (Karras al., 2018), and the corresponding LQ images are synthesized via Eq. (1 (). The specific settings on the
the degradation can be found in Appendix B. As for the rorl-world datasets, consider three typical ones with different degrees of degradation, namely LFW-Test, WebPhoto- Test (Wang et al., 2021), and WIDER- Test (Zhou et al., 2022). I.FW-Test consists of 1711 mildly degraded face images in the wild, which contains et al., 2008). WebPhoto-Tess is made up of 407 face images crawled from the internet. Some of them old photos with severe degradations. WIDER-TE -est selects 970 face images with very heavy degradations from the WIDER Face dataset (Yang et al., 2016), and is thus suitable to test the robustness of different methods under severe degradations.
image for each person in LFW dataset (Huang
Comparison Methods. We compare DifFace with six recent BFR methods, including DFDNet (Li al.. 2020), PULSE (Menon al., 2020), PSFRGAN (Chen al.. 2021), GLEAN (Chan al., 2022), GFPGAN (Wang et al., 2021), and VQFR (Gu et al., 2022).
5.2 EVALUATION ON SYNTHETIC DATA
To evaluate different methods comprehensively, we adopt five quantitative metrics, namely PSNR, SSIM (Wang et al., 2004), I.PPS (Zhang al., 2018), FID (Heusel et al., 2017), and Identity Score

---

Under review as a conference paper at ICLR 2023
Table 2: FID scores of different methods on three real-world testing datasets. The best and second best results are highlighted in bold and underline, respectively.
Methods
Datasets
# Images
DFDNet
PULSE
PSERGAN
GLEAN
GFPGAN
VQFR
DifFace
WIDER-Test
970
58.23
69.55
49.85
70.12
39.76
44.54
37,49
LFW-Test
1711
59.83
65.17
49.80
46.39
50.13
50.88
45.23
WebPhoto- Test
407
92.82
86.40
85.45
94.34
87.86
75.46
85.52
0
200
(b) DTDNet
PCLSC
(d) PSURGAN
GLEAN
GIPGAN
VQFR
(h) DyfFace
Figure 8: Visual comparisons of different methods the real-world examples from LFW-Test (first. row), WebPhoto-Test (second row), and WIDER-Test (third row).
(IDS). Specifically, LPIPS is learned perceptual similarity metric calculated based on deep features of VGG (Simonyan & Zisserman, 2015) network. IDS is the embedding angle of ArcFace (Deng et al., 2019) between two images, and mainly reflects identity preservation. As for FID, it is the KL divergence between the feature distributions (assumed as Gaussian) of the restored images and the ground truth images.
We summarize the comparative results CelebA-Test in Table 1. The proposed DifFace achieves the best performance across all five metrics, indicating its effectiveness and superiority in the task of BFR. To further verify the robustness of DifFace, compare it with GFPGAN and VQFR using LPIPS and IDS under different degrees of degradation in Fig. 6(b) and (c). We generate different degrees of degradation by introducing five downsampling scale factors, i.e., 4, 16, 24, and 32, to the images in CelebA-Test A total of 700 image pairs are generated for each scale. Figure 6(b) and (c) record the averaged performance of different methods over these 700 images with respect to the five downsampling scale factors. While DifFace is slightly inferior to VQFR and GFPGAN under the scale factors 4 and 8 (mild degradations), surpasses other methods and its performance drops more gracefully in the cases of larger factors (severe degradations). This results substantiate the robustness of our DifFace in scenarios with very severe degradations. The outcome consistent with the analysis in Sec. 3.3.
For easy visulization, three typical examples the CelebA- Test are shown in Fig. 7. In the first example with mild degradation, all methods except PULSE are able restore realistic-looking image. PULSE fails to preserve the identity because its optimization cannot find the correct latent code in GAN through inversion. In the second and third examples that exhibit more severe degra- dation, most compared methods produce obvious artifacts. PULSE returns a plausible face but fails preserve the identity. Under this challenging degradation setting, DifFace still achieves pleasant results. Such robustness complies with our observation in Fig. 6(b) and (c). For more quantitative results, please refer Fig. 15 in Appendix.
5.3 EVALUATION ON REAL-WORLD DATA
In the experiments on real-world datasets, we mainly adopt FID as the quantitative metric since the ground truths are inaccessible. We firstly estimate the feature statisties the restored images and the HQ images in FFHQ (Karras et al., 2019) dataset, respectively, and then calculate the KL divergence FID. We attempted other non-reference metrics, namely NIQE (Mittal al., 2012), NRQM (Ma
8

---

Under review as a conference paper at ICLR 2023
LQ
Varied Random Seed
Figure 9:
Unlike most existing BFR methods, DifFace can generate multiple diverse and plausible results given LQ image, by setting different random seeds for the diffusion model. This example extracted from real-world dataset FIDER-Test
Table 3: Comparisons of different methods in terms of model size (M) and runtime (s). For DifFace,
"A+B" denotes the number of parameters in f(; w) (A) and the pre-trained diffusion model (B).
Methods
DFDNet
PULSE
PSERGAN
GLEAN
GFPGAN
VQFR
DifFace
Running Time (s)
1.404
3.928
0.062
0.176
0.039
0.243
4.32
# Parameters (M)
240.11
24.11
67.26
210.84
76.21
76.56
15.79+159.59
et al., 2017), and PI (Blau et al., 2018). These metrics are not adopt here as empirically found that they behave unusually pessimistic for diffusion- methods. For completeness, we provide the experimental results non-reference metrics in Appendix F.
The comparative results are summarized in Table 2. We can observe that DifFace achieves the best performance both WIDER-Test and LeW-Test. On the WebPhoto-Tese it also surpasses most of recent BFR methods except VQFR and PSERGAN. It should be noted that the FID values on WebPhoto-TH Test may not be representative estimate the feature distribution of the restored images. To supplement the analysis, we show three typical examples of these datasets in Fig. 8, and more visual results given in Fig. 16 of Appendix. is again observed that DifFace provides hetter restored results, especially in the second and third examples with severe unknown degradations.
the dataset contains too few images (totally 407) to
Most existing BFR methods produce only one result for each LQ input, although there may be many reasonable possibilities. This is because they only learn deterministic mapping between the LQ and HQ images. It is interesting to note that DifFace, as shown in Fig. 9, is capable of producing multiple diverse and plausible HQ solutions for any given LQ image by setting different random seeds for the pretrained diffusion model. More results are shown in Fig. 17 and 18 in Appendix. This "pluralistic" property is favorable BFR, as there exist many different HQ images that can generate the same LQ counterpart.
5.4 LIMITATION
Despite the good performance of DifFace, the inference efficiency of our method is limited by the iterative sampling process in the diffusion model. Table 3 lists the comparison of different methods on model size, i.c., number of learnable parameters, and running time. It is observed that the model size of DifFace (inclusive the pretrained diffusion model)
comparable to that of GLEAN and DFDNet. As for the runing time, it takes 4.32s to deal with a testing example under current setting (100 sampling steps for inference). which is slower than most of competing methods. is possible to further reduce the sampling process to 20 steps without evident performance drop, and the detailed performance comparisons on different acceleration settings are analyzed in Appendix G. This cuts the running time to 0.92s, lying between DFDNet and VQFR.
6 CONCLUSION
We have proposed a new BFR method called DifFace in this work. DifFace is appealing as it only relies on a restoration backbone that trained with L2 loss. This vastly simplifies the complicated training objectives in most current approaches. Importantly, we have proposed posterior distri- bution that is well-suited for BFR. It consists of a transition kernel and a Markov chain partially borrowed from pre-trained diffusion model. The former acts error compressor, and thus makes our method more robust to severe degradations. The latter effectively leverages the powerful diffusion model to facilitate BFR. Extensive experiments have demonstrated the effectiveness and robustness of our method both on synthetic and real-worle datasets. We hope that this work could inspire more robust diffusion- restoration methods in the future.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor. The 2018 pirm challenge perceptual image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Ruojin Cai, Guandao Yang, Hadar Averbuch-Eiie Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 364-381. Springer, 2020.
Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. Glean: Generative latent bank for farge-factor image super-resolution IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2022.
Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan-Yee K Wong. Progressive semantic-aware style transformation for blind face restoration. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11896-11905, 2021.
Chaofeng Chen, Xinyu Shi, Yipeng Qin, Xiaoming Li, Xiaoguang Han, Tao Yang, and Shihui Guo. Real-world blind super-resolution via feature matching with implicit high-resolution priors, 2022.
URL httpsttp//// /tt.iaavvvv//////////ee///aatava org/ abs / 2202. 13142.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In Proceedings of the International Confer- ence on Learning Representations (ICLR), 2020.
Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. Fsrnet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 2492-2501, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, Icee, 2009,
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4690-4699, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis, Advances in Neural Information Processing Systems (NeurIPS), 34:8780-8794 2021.
Berk Dogan, Shuhang Gu, and Radu Timofte. Exemplar guided face image super-resolution without facial landmarks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 0-0, 2019.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 38(2):295-307, 2015.
Patrick Esser, Robin Rombach, and Bjor Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 12873-12883, 2021.
Lu Feihong, Chen Hang. Li Kang. Deng Qiliang, Zhao jian, Zhang Kaipeng, and Han Hong* To- ward high-quality face-mask occluded restoration. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2022.
Ian Goodfellow, Jean Pbadet- Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- fion Processing Systems (NeurIPS), 27, 2014.
10

---

Under review as a conference paper at ICLR 2023
Jinjin Gu, Yujum Shen, and Bolei Zhou. Image processing using multi- gan prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3012- 3021, 2020.
Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng- Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33:6840-683 6851, 2020.
Xiaobin Hu, Wenqi Ren, John LaMaster, Xiaochun Cao, Xiaoming Li, Zechao Li, Bjoern Menze, and Wei Liu. Face super-resolution guided by facial priors. In Proceedings of the European Conference Computer Vision (ECCV), pp. 763-780. Springer, 2020.
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in 'Real-Life' Images: detection, alignment, and recognition, 2008.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In Proceedings of the International Conference on Learn- ing Representations (ICLR), 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4401-4410, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng. Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-59, 2022.
Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang. Learning warped guidance for blind face restoration. In Proceedings of the European conference on computer vision (ECCV), pp. 272-289, 2018.
Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang. Blind face restoration via deep multi-scale component dictionaries. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pp. 399 399-415. Springer, 2020.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE//VV International Confer- ence on Computer Vision (ICCV) Workshops, pp. 1933-1844,, 2021.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In Pro- ceedings of the International Conference on Learning Representations (ICLR). OpenReview.net, 2017. URL https: http//ppenevviieeeeeeeeeeeeeeeet
net/ forum?iduudroo http://ooooonn-----eoeeeee Skq89Scxx.
Chao Ma. Chih- Yuan Yang, Xiaokang Yang, and Ming- Hisng-Hsuan Yang. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding, 158:1-16, 2017.
Kangfu Mei and Vishal M. Patel. LTT-GAN: looking through turbulence by inverting gans. CoRR, abs/21 02.0211,,, 2021. URL httpt:t//t/:/xi...////// /abg/bss////ooo..o.o.oo....sss http///ss///1// 22122 02379.
11

---

Under review as a conference paper at ICLR 2023
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp- 2437-2445, 2020.
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-212, 2012.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference Machine Learning (ICML), pp. 8162-8171. PMLR, 2021.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Per- mutation invariant graph generation via score-based generative modeling. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the International Conference gence and Statistics (AISTATS), volume 108 of Proceedings of Machine Learning Research, pp. 4474-4484. PMLR, 26-28 Aug 2020. URL httph:/ httpttttpppoceeeeedeeee http:///rocceeeeeeeeeeeee ngs mlr wwt.:mlww.p....ee...e.eeeeeeses. niu20a html.
Artificial Intelli-
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions Puttern Analysis and Machine Intelligence (TPAMI), 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Wenqi Ren, Jiaolong Yang, Senyou Deng, David Wipf, Xiaochun Cao, and Xin Tong. Face video deblurring using 3d facial priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9388 9397, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Patlern Recognition (CVPR), pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In Proceedings the ACM SIGGRAPH Conference, pp. 1-10, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super resouutin via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022b.
Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang. Deep semantic face deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3260-8269 2018.
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874-1883, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.), Proceedings of the International Con- ference Leurning Representations (ICLR), 2015. URL http:://axxia wowvvorrg/aaaa abg/abs/1400 1556.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference Machine Learning (ICMI.), pp. 2256-2265. PMLR, 2015.
12

---

Under review as a conference paper at ICLR 2023
Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
Xiaoguang Tu, Jian Zhao, Qiankun Liu, Wenjie Ai, Guodong Guo, Zhifeng Li, Wei Liu, and Jiashi Feng. Joint face image restoration and frontalization for recognition. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 32(3): -298, 2021.
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 9168-9178, 2021.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing (TIP), 13 (4):600-612, 2004.
Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. Restoreformer: High- quality blind face restoration from undegraded key-value pairs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17512-17521, 2022.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming- Hsuan Yang. Gan inversion: survey, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: A face detection bench- mark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5525-5533, 2016.
Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 672-681, 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586-595, 2018.
Shangehen Zhou, Kelvin C. K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer, 2022. URL https: aaaaaaivvvv..oo org/ /bs/
2206. 1166..1253.
Feida Zhu, Junwei Zhu, Wenqing Chu, Xinyi Zhang, Xiaozhong Ji, Chengjie Wang, and Ying Tai. Blind face restoration via integrating face shape and generative priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 7662-7671, 2022.
13

---

Under review as a conference paper at ICLR 2023
A RELATED WORK
We provide a more detailed discussion here on related work. The research on BFR mainly focused on exploring better face priors obtain more rational results. In the following, we review a few popular priors for BFR, namely geometric priors, reference priors, and generative priors. Diffusion prior can be considered
type of generative prior.
Geometric Priors. Face images are highly structured compared with natural images. The structural information, such as facial landmarks (Chen al., 2018), 8), face parsing maps (Chen al., 2021; Shen et al., 2018; Chen et al., 2021), facial component heatmaps (Li et al., 2020), and 3D shapes (Ren et al., 2019; Hu et al., 2020; Zhu et al., 2022), can used guidance to facilitate the restoration process. The geometric face priors estimated from degraded inputs can be unreliable, affecting the performance of the subsequent BFR task.
Reference Priors. Some existing methods (Li al., 2018; Dogan et al., 2019) guide the restoration with an additional HQ reference image that owns the same identity as the degraded input. The main limitations of these methods stem from their dependence on the HQ reference images, which inaccessible in some scenarios. Li et al. (2020) address this limitation by constructing an offline facial component dictionary based on the features extracted from HQ images. Then, it searches for the closest facial components in this dictionary for the given LQ images during restoration.
Generative Priors. Unlike Li al. (2020), more recent approaches directly exploit the rich priors encapsulated in generative models for BFR. Following the paradigm of GAN inversion (Xia et al., 2022), the earliest explorations (Menon et al., 2020; Gu et al., 2020; Pan al., 2021) iteratively optimize the latent code of pre-trained GAN for the desirable HQ target. To circumvent the time- consuming optimization, some studies (Chan et al., 2022; Wang et al., 2021; Yang et al., 2021) directly embed the decoder of the pre-trained StyleGAN (Karras et al., 2019) into the BFR network, and evidently improve the restoration performance. The success of VQGAN (Esser et al., 2021) in image generation also inspires several BFR methods. These methods mainly design different strategies, e.g., cross-attention (Wang et al., 2022) and parallel decoder (Gu et al., 2022), to improve the matching between the codebook elements of the degraded input and the underlying HQ image. Even though these methods have performed well in most scenarios, they still failed to handle some examples with complicated degradations out of their training data.
Attributed to the powerful capability of diffusion model in image generation, a flurry of works based on diffusion model have been proposed recently. Typically. SR3 (Saharia al., 2022b) and SRDiff (Li et al., 2022) both feed the observed LQ image into the diffusion model as condition to guide the restoration in training. Following a similar modeling manner, Rombach al. (2022) proposed to train the diffusion model in the latent space to accelerate the inference speed. These methods all require one to retrain the diffusion model from scratch on some pre-collected data. The learned model would be still very sensitive to the degradation mismatch when generalizing to other datasets. Therefore, this paper devises a new learning paradigm based on a pretrianed diffusion model address these issues.
B SETUP ON VALIDATION AND TESTING DATASETS
In the main text, we synthesized one validation dataset and one testing dataset, each of which con- tains 3,500 HQ images randomly selected from CelebA-HQ (Karras et al., 2018). The former is used to select reasonable starting timestep N for DifFace, and the latter evaluate the performance of different methods. To synthesize the LQ images, we employ the degradation in Eq. (11) with the following settings:
E 14,8,16,24,22 32), E {1,5, 10, 15, 20}, 9 E {30, 40, 50, 60, 70}.
1
L,, E (2,4,6, 8, 10, 12, 14), 0 (0, "....."..7
where lz. Ly. and control the generation of the blur kernel k. Specifically, the covariance matrix Σ of k is defined as follows:
[cos 0
sin 0 cos 0
12 A
0
U =
= UAUT.
(12)
sin 0
12
14

---

Under review as a conference paper at ICLR 2023
Table Quantitative comparisons of SRCNN, SwinlR and DifFace on CelebA-Test "DifFace(X)"
means that it takes the backbone "X" as the diffused estimator f(: The parameters in "Dif-
Face(X)" includes both of that the backbone "X" and the pre-trained diffusion model M.).
Metrics
Methods
PSNRT
SSIMÍ
LPIPST
FID
IDS
Parameters(M)
SRCNN
24.72
0.736
0.528
102.89
67.39
1.03
DifFace(SRCNN)
23.72
0.695
0.461
25.37
65.78
1.03+ 159.59
SwinIR
25.48
0.749
0,482
87.17
58.26
15.79
SifFace(SwinIRR)
24.08
0.703
0.435
18.20
62.39
15.79+159.59
(Q)
(b1) SRUNN
(21) N00
(411 N-0
N00
(T) N-450
(211 N-000
(bi) NY100
(b2) SwintR
(62) 200
(42) N-000
No-00
N5-
(22) N-000
N N0-60
SRCNS
(ol) N-200
(d1) N-300
Nm000
N-450
[gl) N-500
(hi) N-600
(b2) SwintR
N-200
N-300
(e2) Nm100
N-450
N-500
(h2) N-600
Figure 10: Qualitative comparisons of DifFace LFW-Test (top row) and WIDER-Test (bottom row) under different settings for the starting timestep N and the diffused estimator [(; w)). (a) LQ image, (bb)- restored results by SRCNN and SwinlR, (c1)-(h1) restored results by DifFace that takes SRCNN as diffused estimator, (c2)-(h2) restored results by DifFace that takes SwinIR as diffused estimator.
C DIFFUSED ESTIMATOR
C.1 NETWORK ARCHITECTURES
We consider two typical restoration backbones for the diffused estimator f(;w,, namely SR- CNN (Dong et al., 2015) and SwinIR (Liang et al., 2021). To apply them in our method, we slightly adjust their settings. The LQ/HQ image pairs in method both with size 512 512, we add two (or three) PixelUnshuffle (Shi et al., 2016) layers with downscale factor to reduce the input size 128 128 64 x 64) for SRCNN (or SwinIR), After each PixelUnshuffle layers except the last one, one convolutional and LeakyReLU layers are followed to fuse the features. Similarly, two (or three) PixelShuffle (Shi al., 2016) layers are also added the tail of SRCNN (or SwinIR) to upsample the size back to 512 x 512.
As for SRCNN, we adopt nine convolutional layers with kernel size 5 between the PixelUnshuffle and PixelShuffle layers, and each convolutional layer has 64 channels. As for SwinIR, we follow the official settings for real-world image super-resolution task with scale factor
Phttps://gi http://gggtttbb.ooc Joniinguu ang/ Sw///wwn
15

---

Under review as a conference paper at ICLR 2023
Zoomed LQ
IIQ
SR3
DyPace
Zoomed LIQ
IQQ
SKL
DyFace
(a) x8 super-resolutio
(b) super-resouttio
Figure 11: Visual comparisons of SR3 and DifFace 8x (left) and 16x (right) face image super- resolution.
C.2 PERFORMANCE COMPARISON
To quantitatively evaluate these two restoration backbones and their influences to DifFace, we con- duct experiments CelebA-Test and summarize the results in Table 4. We also provide some quantitative comparisons in Figure 10 on the real datasets LFW-Test (top row) and WIDER- (bottom row). The results offer a few observations follows:
Even though taking the simplest backbone SRCNN as our diffused estimator, DifFace is still able to produce highly natural-looking results with realistic image details, such as hairs and stubbles. This delivers that the proposed DifFace does not rely on specific designs on the dif- fused estimator.
Owning to more advanced network architectures as well as more learnable parameters, SwinIR outperforms SRCNN in both qualitative and quantitative comparisons.
When the starting timestep N gets larger, DifFace is capable of achieving more realistic results. However, larger N will introduce more randomness and thus reduce the fidelity.
Based on these observations, superior performance. As for the starting timestep N, the proposed DifFace performs stably and well when fixing it in the range [400, 450).
suggest taking SwinIR the diffused estimator attributed to its
D BICUBIC REPEERRESSOLTION
Diffusion model can be used for image restoration by directly introducing the LQ image yo condition guide the generation of æ6 (Saharia al., 2022a:b; Song al., 2020), namely
T.
p(æolyo) = p((r)) II pa (w.- 12,, yo 1.7.
(13)
where p(xT)= N(rr(0, I), po(x;-11e 1 Mo) is the learned transition kernel from æ; #t-1 conditioned on Yo- To access such transition kernel, an intuitive way to retrain the diffusion model from scratch. Specifically, in each timestep t, 2; and yo concatenated together input to predict æ;-1 in training. Following this paradigm, SR3 (Saharia et al., 2022b) trained a diffusion model for image super-resolution based on bicubic degradation.
We omit the comparison against SR3 (Saharia et al., 2022b) in the main text as it can only deal with bicubic degradation. Here, extend the proposed DifFace bicubic super-resolution and make
16

---

Under review as a conference paper at ICLR 2023
Table
Quantitative results of SR3 and DifFace on bicubic face image super-resolution.
Metrics
Methods
Scale factors
PSNRT
SSIM
LPIPS_
IDS]
SR3
22.74
0.585
0.505
25.66
x8
DifFace
27.24
0.754
0.375
38.89
SR3
20.50
0.506
0.645
64.19
x16
DifFace
25.03
0.712
0.412
51.36
Table 6: Non-reference metrics different methods on the real-world dataset WIDER-Test. "Dif- fusion" denotes the average results 3,000 images randomly generated by the pretrained diffusion model, which can be regarded as the upper bound DifFace.
Methods
Metries
DFDNet
PULSE
PSERGAN
GLEAN
GFPGAN
VQFR
DifFace
Diffusion
NIQEJ
5.67
5.27
3.89
5.13
3.81
3.02
4.24
4.11
NRQM
6.80
4.05
8.01
7.03
8.07
8.78
6.11
6.60
PI4
4.81
5.88
3.33
4.45
3.08
2.17
4.46
4.17
fair comparison with SR3. We do not make any changes apart from upsampling an LQ image with bicubic interpolation and feeding it into DifFace to restore the high-resolution target. Although the degradation model we use (Eq. (1 (11)) does contain the bicubic degradation, we do retrain or finetune our diffused estimator such degradation.
Since the code of SR3 is not released, we thus adopt an unofficially re-implemented version instead. This model is specifically trained for 8x face image super-resolution from size 64 x 64 to 512 x 512. To further evaluate the generalization capability to different degradation, we also test its performance on the task of 16x super-resolution from 32 32 to 512 x 512. As for the testing dataset, we randomly select 200 images from CelebA-HQ (Karras et al., 2018).
Table 5 lists the quantitative results of SR3 and DifFace the task of bicubic super-resolution and the corresponding visual comparisons are shown in Fig. 11. DifFace achieves better or at least com- parable performance 8x super-resolution, even though SR3 is specifically trained for this bicubic upsampling task. When generalized to 16x super-resolution, DifFace outperforms SR3 on both qual- itative and qualitative comparisons, indicating the robustness of DifFace unknown degradations. On the other hand, DifFace is more efficient than SR3, because SR3 has to pass through the whole reverse process of the diffusion model while DifFace starts from the intermediate state (i.e., x of the reverse Markov chain.
Another popular diffusion based image super-resolution model is LDM (Rombach et al., 2022). trains a diffusion model to approximate the conditional distribution in Eq. (13) in the latent space of VQGAN (Esser et al., 2021), which evidently accelerates inference speed. Since LDM only releases pretrained model trained on natural images, we thus cannot provide comparisons with on the task BFR.
E ATMOSPHERIC TURBULENCE RESTORATION
In this section, we consider a more challenging degradation type. namely atmospheric turbulence. causes geometric distortion and blur effects on the captured images, and negatively affects the face-related downstreaming tasks. Note that none of the DifFace and the other comparison methods contains any atmospheric turbulence or similar degradation types. Hence, it is suitable to compare the robustness of different methods on such a degradation type.
is difficult to collect large real-world dataset degraded by atmospheric turbulence, since most of the relevant datasets are privately protected due their military use. We cropped eight real-world image pairs from the manuscript Mei & Patel (2021), and resize them to the resolution 512. Each pair contains image degraded by atmospheric turbulence and one corresponding reference image. We call this dataset as Turb8- Test this work.
Phttps://gi thub. hwwb..com/Jaasp ry/ magee
www.-aupee--aaaaee Super-upee--rree Reso luti onttiin-viii vvia Iterati Ref nement
17

---

Under review as a conference paper at ICLR 2023
Table 7: Performance comparisons of DFDNet, VQFR, and DifFace under different acceleration settings. "Dif(A/B) means that the whole reverse process of the pretrained diffusion model contains sampling steps after acceleration, and the starting timestep N in DifFace is B. In the main text of this paper, reported the performance of Dif(250/ 100) our result.
Methods
Metrics
Di((1000/400
(2500/000
(i0(10)/40)
Dif(50/20)
Dif(20/8)
DFDNet
VQFR
PSNR
24.01
24.08
24.09
24.00
22.87
23.15
21.94
SSIMt
0.692
0.703
0.707
0.711
0.673
0.629
0.585
LPIPS
0.441
0.434
0.435
0.440
0.497
0.554
0.471
FID
17.51
18.20
21.00
24.45
26.97
64.65
25.74
IDS
62.56
62.39
62.08
61.77
60.95
86.21
65.87
Runtime
17.09
4.32
1.77
0.92
0.41
1.40
0.24
Figure 12 shows the qualitative comparison results of different methods on Turb8-Test dataset. Dif- Face is able to achieve photo-realistic results and evidently outperforms other methods in visual quality. In particular, the first three examples represent the same person but with large pose vari- ations, DifFace still performs well in this scenario. Unfortunately, DifFace sometimes cannot pre- serve the identity very well when the input loses too much image information, e.g., the last example in Fig. 12. We leave this problem to future research.
F DISCUSSION ON NON-REFERENCE METRICS
In Sec. 5.3 of our main text, evaluate the performance of different methods on real-world data using FID score. In this section, we further consider three commonly used non-reference metrics, namely NIQE (Mittal et al., 2012), NRQM (Ma et al., 2017), and PI (Blau et al., 2018). Table 6 summarizes the detailed comparisons on WIDER-Test. Despite the strong superiority of DifFace, shows surprisingly weak performance against GAN based methods (e.g.. VQFR, GFPGAN, and PSFRGAN) across these three non-reference metrics.
To explore the underlying reasons behind this phenomenon, randomly generate 3,000 HQ face images using the pre-trained diffusion model, and then evaluate their image quality based on these non-reference metries. The average results denoted "Diffusion" and marked by gray color in Table 6. One can regard this performance as the upper bound of our proposed DifFace.
It is interesting observe that even this upper bound is inferior GAN based restoration methods. Such conclusion is unreasonably pessimistic considering the various positive results of diffusion models reported againtst GANs (Song et al., 2020; Dhariwal & Nichol, 2021). More investigations are called to study this gap introduced by the aforementioned non-reference metrics.
G INFERENCE ACCELERATION
The main limitation of DifFace is the slow inference speed due to the gradual sampling mechanism inherited from diffusion model. In this work, the employed diffusion model is trained with 1000 discrete diffusion steps, and the starting timestep N is set as 400. To shorten the inference time, we reduce the sampling steps 250 from 1000 for the diffusion model using the technique Nichol & Dhariwal (2021), and the starting timestep N is thus reduced to 100 from 400.
Table 7 details the performance DifFace of reducing the inference process to different steps. We can see that DifFace performs very stably under different acceleration setup. Under the extreme setting where only 20 sampling steps is used in DifFace, its inference efficiency is significantly improved, having its range of rumning time lying between DFDNet and VQFR. Even though speedup the inference by 20 times, DifFace still superior to VQFR. Several qualitative compar- isons are displayed in Fig. 13.
18

---

Under review as a conference paper at ICLR 2023
H EXTENSION TO LATENT SPACE
Instead of learning the data distribution the image space, Rombach et al. (2022) propose LDM model the data distribution in the low dimentional latent space of VQGAN (Esser et al., 2021), alleviating the requirement on huge computational resources. Specifically, LDM first trains a VQ- GAN model which consists of an encoder E and a decoder on the training dataset S = (x (1 M N==1
Given any x(() E S, the encoder maps it to the latent space via 2(1) = E (ax(),, and the decoder aims to reconstruct this image by (f)) = D (z(+). Then, diffusion model is trained on the latent space Z {2 of such VQGAN.
Extending the proposed Difface to latent space can be easily achieved by moving the posterior p(xol3e) of Eq. (5) latent space, namely
N II pe (Zz Eo E (3/0)
p(zol0o) / p(zN ε0)
(14)
where yo is the degraded image, E is the encoder of VQGAN. To restore the high-quality image æo from you we first get sample from p(zo| (0), and then decode it to the image space, i.e.,
20 p
To = D (zo).
(15)
Similar to that in image space, the transition distribution p(zN E is also formulated as a Gaussian distribution but built in latent space:
p ē0)= NV -V Van (2o: w), v(1 ax))).
(16)
As for the diffused estimator f(; ()), is trained on the synthetic image pairs (xx(), ( (MM gen- erated via Eq. (11) follows:
Σ L (u((
(17)
where L(-,.) is pre-defined feature matching loss in latent space. With the formulation above, we extend DifFace to the latent space, and denote it as DifFace-Latent.
We make some initial exploration along this line by simply setting L(...) of Eq. (17) as L. loss, and some visual comparisons on CelebA-Test are displayed in Fig. 14. It can be observed that DifFace-LLaent is also able to successfully recover realistic-looking result with clear facial details. However, its identity preservation is inferior to DifFace. We speculate that this drawback can be overcame by employing a more effective latent feature matching loss -(.-). In fact, the design of feature matching in latent space is still open research topic, two representative works Zhou et al. (2022) and Chen et al. (2022). We leave this to our future work.
19

---

Under review as a conference paper at ICLR 2023
(6) LQ
(b) PULSE
PSERGAN
GLAN
(e) GEPGAN
(9 VQFR
DifPace
(h) Reference
Figure 12: Qualitative comparisons of different methods the real-world Turb8- Test dataset.
LQQ
(000
(250/10)100
(100044))
(i
Figure 13: Qualitative comparisons of VQFR and DifFace under different acceleration settings. "Dif(A/B)" means that the whole reverse length of the pretrained diffusion model is A, and the corresponding starting timestep N in DifFace is set B.
20

---

Under review as a conference paper at ICLR 2023
DifFace
Difacce Latent
DiFace
Figure 14: Qualitative comparisons of DifFace and its extended version (DifFace-Latent in the latent space.
(b)
(J)
(g) VQFR
DyFaee
Figure 15: Visual comparisons of different method on the synthetic dataset CelebA-Test
21

---

Under review as a conference paper at ICLR 2023
LFW-Test
PeaPPhonii Test
WIDER TEDER-Tes Test
DFINE:
PSERGIN
GLEAN
Figure 16: Visual comparisons of different method three real-world datasets.
LO
HQ
Varied Random Seed
Figure 17: Three restoration examples of DifFace on the synthetic dataset CelebA-Tess by setting different random seeds for the diffusion model.
22

---

Under review as a conference paper at ICLR 2023
LQ
Varied Random Seed
Figure 18: Three restoration examples of DifFace on the real-world dataset WIDER-Test by setting different random seeds for the diffusion model.
Variel
Varied Random Seed (ImageNet)
600
(b)
Figure 19: Qualitative comparisons of DifFace with pretrained diffusion models trained on different datasets. The top row employs diffusion model trained FFHQ (Karras et al., 2019), while the bottom row on ImageNet (Deng et al., 2009).
(u) DifFace (-1)
(b) Difface (0)
(c) DifFace (1)
(d) DifFace (2)
(e) DifFace (3)
(I) DifFace (4)
Iterations
Figure 20: Visual results by iteratively applying the proposed DifFace on one real-world example. "DifTace (i)" denotes the restored result that takes "DifFace (i-1)" as the degraded input, and "Dif- Face (-1)" the low-quality image.
23


Reviews:
[{"review_id": "ZSQIkDd5E0P", "reviewer": "Reviewer_SR7T", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a method for Blind Face Restoration (BFR) using pre-trained diffusion models for faces. The main novelty is a design of a transition model from a low quality image that estimates an intermediate state of the diffusion process of a high quality image. This allows the method to use a pre-trained diffusion model as the image restoration back-bone. The paper is equipped with extensive analysis to motivate the approach. The paper also provided qualitative and quantitative comparisons on BFR task with several previous art. The results are generally in favor of the proposed method demonstrating robust face restoration quality. ", "summary_of_the_review": "I think the paper proposed a simple and original method for blind face restoration. The paper is well written. The experimental results are convincing. I hold a positive view of the paper and I recommend accept.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well written and a pleasure to read. The paper posed a very simple yet effective method for face restoration. The method is well motivated and explained, with extensive analysis of the method some design choices, e.g., choice of starting timestep N.\n\n2. The idea is novel to me. The paper proposed a method to leverage a pre-trained diffusion model for BFR. The method is novel in the sense that it constructs a transition distribution to infer an intermediate state of the diffusion process of a high quality image from a low quality counterpart. \n\n3. The results are generally convincing. The proposed method seems to be more robust than most previous methods that are used as baselines in the paper, with less artifact and higher fidelity in visual comparisons. Quantitative evaluations also demonstrated better performance than previous art.\n\nWeakness:\n\nI don't have much complaints or concerns as the author has already mentioned that one limitation of the method is that, by having to go through the diffusion process, the performance is slow. \nOne question that I have is the choice of the training data for the generative model. Since variants of diffusion models has been used in training image generation models for diverse images, would training on data beyond human faces provide better prior for face restoration? In Figure 13, the hand restoration is clearly failing. Is this due to the lack of training data diversity of the diffusion model since they are only trained on FFHQ?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RVNwpoyxYex", "reviewer": "Reviewer_yge8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method DifFace, being able to cope with unseen and complex degradations more gracefully without complicated loss designs. Comprehensive experiments show that DifFace is better than current state-of-the-art methods, especially in cases with severe degradations. ", "summary_of_the_review": "See Strength And Weaknesses.", "strength_and_weaknesses": "Pros:\n1. The motivation is clear. \n2. The paper is well-written and organized.\nCons:\n1. More results on more challenging datasets are needed to verify the superiority of the proposed method.\n2. Some related works are missing, e.g., Toward High-Quality Face-Mask Occluded Restoration, Joint Face Image Restoration and Frontalization for Recognition", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7LfYcSxd7O", "reviewer": "Reviewer_FutU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of blind face restoration. To overcome the two limitations (lack of generalization property and requirement of multiple constraints), this paper proposes a new approach based on the latest diffusion model in the latent space. The key idea is to learn a posterior distribution from LQ to HQ with the help of a transition distribution from LQ to an intermediate state of pre-trained diffusion model. The design of transition distribution takes into account both the efficiency of training process and robustness to unknown degradation. Extensive experimental results are reported to demonstrate the superiority of the proposed DifFace to other competing methods.", "summary_of_the_review": "I am mostly impressed by the reported experimental results of this paper. Its theoretical contribution is OK but incremental (the derived diffusion model still works in the pixel instead of the latent space). The computational complexity and the model parameters of this work do not compare favorably over existing methods such as GLEAN and DFDNet.", "strength_and_weaknesses": "+ The motivation behind DifFace is clear. In view of the wide impact of probabilistic diffusion models on image synthesis, it is not surprising to see that LDM can be applied to low-level vision tasks such as blind face restoration.\n+ The design of transition distribution in Sec. 3.2 and its discussion in Sec. 3.3 are easy to follow. Model analysis in Sec. 4 offers some good insight about the interpretability of model parameters (starting timestep N and diffused estimator f).\n+ The reported subjective quality comparison results are convincing (e.g., Fig. 2, Fig. 7 and Fig. 8). Objective quality comparison in Table 1 also clearly favors the proposed method DifFace.\n\n- I had thought this work will work in the latent space such as latent diffusion model (LDM, CVPR'2022). Based on the discussion in Sec. 5.4, this work has not been extended into the latent space. I am just curious about the authors' further thought about this line of extension. Maybe StyleGAN will serve as a natural candidate for the latent space?\n- \"All models are wrong; some are useful\". It will be nice to show some failure examples for the proposed DifFace. How about its performance in the presence of occlusion or severe pose variation or atmospheric turbulence (e.g., IARPA BRIAR scenario)? \n- If you supply Fig. 2(g) as the input image to DifFace, will it output an undistorted image? In other words, have you verified that the restored face image is a fixed point for the proposed restoration operator?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-CwK1oVP7qE", "reviewer": "Reviewer_sp9a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to use a pre-trained diffusion model as a generative prior for BFR. With a diffused estimator, the LQ images with different degradation can be transformed into the latent space of the diffusion model. A weighted L2 loss has been proposed to train the diffused estimator. Generally, the paper is well-written with good performance.", "summary_of_the_review": "Although this paper achieves good performance, the contribution and novelty of this paper are not enough.", "strength_and_weaknesses": "Strength: \n1. The idea of bringing the diffusion model into BFR task is somewhat new but the contribution of this paper is not significant. \n2. The results of DifFace are better than current state-of-the-art methods and extensive experiments and analysis have been provided to evaluate the proposed method.\nWeaknesses: \n1. The main contribution of this paper is designing a diffused estimator to transform the degraded facial images into the latent space of diffusion model, which is not new [1] and this contribution is very limited.\n2. The proposed DifFace model has the advantage of dealing with severe degradation. How about the real degradation face images with fewer degradation? Can the DifFace still restore the identity well?\n3. It was claimed that this paper theoretically shows that the Markov chain can compress the predicted error by a factor of less than 1. However, no theoretical analysis can be found in this manuscript.\n4. It was claimed that the proposed approach avoids using GAN and perceptual loss. However, it is not difficult to use such loss and stable results can also be obtained by those works such as GFPGAN, VQFR. In the proposed method, Eq.(9) introduce a weighted factor k_N, which strictly decreases monotonically with the timestep N. How to choose k_N and a_N? The number of inference sampling times N is also manually chosen. Compared with previous methods, the proposed method requires a more manual and empirical setting.\n5. It’s still unclear why p(x_N |y_0) is a Gaussian distribution as shown in Eq.(8). How about the Laplace distribution? \n6. Since the diffusion model requires a large number of sampling, i.e., 400 in this work, the inference time of DifFace is too slow compared with other methods. The number of sampling need further ablation study. \n\n[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33:6840–6851, 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject