PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance
Authors: Peiqing Yang
Shangchen Zhou
Qingyi Tao
Chen Change Loy
Venue: NeurIPS 2023 poster (2023)
Keywords: Face Restoration
Diffusion

Abstract: Exploiting pre-trained diffusion models for restoration has recently become a favored alternative to the traditional task-specific training approach. Previous works have achieved noteworthy success by limiting the solution space using explicit degradation models. However, these methods often fall short when faced with complex degradations as they generally cannot be precisely modeled. In this paper, we introduce $\textit{partial guidance}$, a fresh perspective that is more adaptable to real-world degradations compared to existing works. Rather than specifically defining the degradation process, our approach models the desired properties, such as image structure and color statistics of high-quality images, and applies this guidance during the reverse diffusion process. These properties are readily available and make no assumptions about the degradation process. When combined with a diffusion prior, this partial guidance can deliver appealing results across a range of restoration tasks. Additionally, our method can be extended to handle composite tasks by consolidating multiple high-quality image properties, achieved by integrating the guidance from respective tasks. Experimental results demonstrate that our method not only outperforms existing diffusion-prior-based approaches but also competes favorably with task-specific models.

PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance
Peiqing Yang'
Shangchen Zhou'
Qingyi Tao?
Chen Change Loy'
S Lab, Nanyang Technological University
"Sense Research, Singapore https://github com/ httpppppyaa aog-yang/PGDin.
Abstract
Exploiting pre-trained diffusion models for restoration has recently become favored alternative to the traditional task-specific training approach. Previous works have achieved noteworthy success by limiting the solution space using explicit degradation models. However, these methods often fall short when faced with complex degradations as they generally cannot be precisely modeled. In this paper, propose PGDiff' by introducing partial guidance, fresh perspective that is more adaptable to real-world degradations compared to existing works. Rather than specifically defining the degradation process, our approach models the desired properties, such as image structure and color statistics of high-quality images, and applies this guidance during the reverse diffusion process. These properties readily available and make no assumptions about the degradation process. When combined with diffusion prior, this partial guidance can deliver appealing results across range of restoration tasks. Additionally, PGDiff' can be extended to handle composite tasks by consolidating multiple high-quality image properties, achieved by integrating the guidance from respective tasks. Experimental results demonstrate that our method not only outperforms existing diffusion-pii cerisiioo--iior---e -ea-----ooo-eeeedeee based approaches but also competes favorably with task-specific models.
Introduction
Recent years have seen diffusion models achieve outstanding results in synthesizing realistic details across various content [33, 29, 7, 15, 34]. The rich generative prior inherent in these models opens up a vast array of possibilities for tasks like super-resolution, inpainting, and colorization. Consequently, there has been a growing interest in formulating efficient guidance strategies for pre-trained diffusion models, enabling their successful adaptation to various restoration tasks [9, 42, 20, 37].
A common approach [9, 42, 20] is to constrain the solution space of intermediate outputs during the denoising process' At each iteration, the intermediate output is modified such that its degraded counterpart is guided towards the input low-quality (LQ) image. Existing works achieve this goal either by using a closed-form solution [42, 20] or kack-propagating simple losses [9]. These methods versatile in the sense that the pre-trained diffusion model can be adapted to various tasks without fine-tuning, as long as the degradation process is known advance.
While possessing great versatility, the aforementioned methods are inevitably limited in generaliz- ability due to the need for prior knowledge of the degradation process. In particular, a closed-form solution generally does not exist except for special cases such as linear operators. In addition, back-prec peck-propaggtin losses demand differentiabilityy of the degradation process, which is violated for many degradations such as JPEG compression. Importantly, degradations in the wild often consist
refers the reverse diffusion process, not the image denoising task.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Property
Chasiffee:: F+
Luve
Target: y
Ispat
Lightou
rabkansyitiaa
Blind Face Restoration
Pigeissas
Colar State
Piteoss (ri
AdalNika)
< Σ
Ytater
Caker Seate
Viol
Partial Guidance
Face Ingaintiing
X,
(a
Inconditional Diflasion
Figure 1: Overview of Our PGDiff Framework for Versatile Face Restoration. Here, we take the colorization task as example to illustrate our inference pipeline. One may refer to Table for the corresponding details (e.g., property, classifier, and target) other tasks. We show that our method can handle wide range of tasks, including (a) blind face restoration, (b) face colorization, (c) face inpainting, and also composite tasks such as (d) old photo restoration.
of a mixture of degradations [41], and hence, is difficult, if not impossible, to model them accu- rately. As result, existing works generally limit the scope simplified cases, such fixed-kernel downsampling. The generalization to real-world degradations remains a formidable challenge.
Motivated by the above, instead of modeling the degradation process, we propose to model the desired properties of high-quality (HQ) images. The merit of such guidance is the agnosticity the degradation process. However, it remains unclear what properties are desired and how appropriate guidance can be constructed. Through our extensive experiments, we find that with diffusion prior acting natural image regularization, one could simply guide the denoising process with easily accessible properties, such as image structure and color statistics. For example, as shown in Fig. 1, one could generate plausible outputs simply by providing guidance on the lightness and the statistics (i.e., mean and variance) of each color channel, without knowing the exact decolorization process. By constraining the HQ image space, our idea bypasses the difficulty of knowing the prior relation between IQQ and HQ images, thus improving generalizability.
In this work, devise simple yet effective instantiation named PGDiff by introducing partial guidace. PGDiff adopts classifier guidance [7] to constrain the denoising process. Each image property corresponds a classifier, and the intermediate outputs are updated by paok-propgggting the gradient computed on the loss between the classifier output and the target property. Since our partial guidance is agnostic to the degradation process, it can be easily extended to complex tasks by compositing multiple properties. For instance, the task of old photo restoration can be regarded as combination of restoration, inpainting, and colorization, and the resultant guidance is represented as weighted sum of the guidance in the respective task. We also demonstrate that common losses such as perceptual loss [2, 16] and adversarial loss [22] can be incorporated for further performance gain.
Contributions. Our main contributions include i) a new concept of adapting diffusion models to restoration without presumptions of the degradation process. We show that suffices to guide the denoising process with easily accessible properties in the HQ image space, with diffusion prior acting as regularization, and ii) partial guidance, a versatile approach that is applicable to a broad range of image restoration and enhancement tasks. Furthermore, it allows flexible combinations of guidance for intricate tasks. We conduct extensive experiments to demonstrate the effectiveness of PGDiff on a variety of challenging tasks including blind face restoration and old photo restoration. We also demonstrate interesting applications, such as reference- based restoration. The results confirm the superiority PGDiff over previous state-of-the-arr methods.
2 Related Work
Generative Prior for Restoration. Generative prior has been widely adopted for range image restoration tasks, including super-resolution, inpainting, and colorization. One prominent approach in

---

this field is the use of pre-treate pre-trained generative adversarial networks (GANs) [10, 18, 1]. For instance, GAN-inversion [26, 11, 28] inverts a corrupted image to a latent code, which is then used for generating a clean image. Another direction is to incorporate the prior into an encoder-decoder architecture [3, 4, 40, 44], bypassing the lengthy optimization during inference. VQVAE [35] is also commonly used generative prior. Existing works [48, 13, 43, 47] generally first train VQVAE with a reconstruction objective, followed by a fine-tuning stage to adapt to the subsequent restoration task. Recently, diffusion models have gained increasing attention due to their unprecedented performance in various generation tasks [33, 29, 7, 15, 34], and such attention has led to interest in leveraging them as a prior for restoration.
Diffusion Prior. There has been a growing interest in formulating efficient guidance strategies for pre-trained diffusion models, enabling their successful adaptation various restoration tasks [9, 42, 20, 37, 39]. Among them, DDRM [20]. DDNM [42], and GDP [9] adopt a zero-shot approach to adapt a pre-trained diffusion model for restoration without the need of task-specific training. At each iteration, the intermediate output is modified such that its degraded counterpart is guided towards the input low-quality image. This is achieved under an assumed degradation process, either in the form of fixed linear matrix [42, 20] or parameterized degradation model [9], with learnable parameters representing degradation extents. In this work, we also exploit the generative prior of a pre-trained diffusion model by formulating efficient guidance for it, but unlike existing works that limit the solution space using explicit degradations [9, 42, 20), we propose to model the desired properties of high-quality images. Such design is agnostic the degradation process, circumventing the difficulty of modeling the degradation process.
3 Methodology
PGDiff is based on diffusion models. In this section, we first introduce the background related to our method in Sec. 3.1, and the details of our method presented in Sec. 3.2.
3.1 Preliminary
Diffusion Models. The diffusion model [33] is a class of generative models that learn to model a data distribution p(z)). In particular, the forward process is a process that iteratively adds Gaussian noise to an input 2g p(x), and the reverse process progressively converts the data from the noise distribution back to the data distribution, often known as the denoising process.
For an unconditional diffusion model with 7 discrete steps, at each step I,, there exists transition distribution ( (1)1) with variance schedule Be [15]:
q( = Nax vI B3 21-1,
Under the reparameterization trick, I, can be written as:
Ft = Vas 2:-1 + vI ai
(2)
where ar 1 3r and e ((e 0, I). Recursively, let ā I[ 0% have
xr Di zig + v1 az. C...
During sampling, the process starts with pure Gaussian noise 2y Nar;; 0, and iteratively performs the denoising step. In practice, the ground-truth denoising step is approximated [7] by po(x.-1(ar) (xr) as:
po (aa) (No t), Eo t)).
(4)
where Ex(x,, is constant depending on pre-defined 3, and ma(xe, t) is generally parameterized by a network eø (2,, t):
Br
(0( Va, (x+
co 1))-
(5)
-I- at
From Eq. (3), one also directly approximate vo from eg:
āt (o(
=
Ft
(6)
04
3

---

Algorithm 1 Sampling with partial guidunce, given diffusion model (na (), Eo ()), classifier (o())))), target y, gradient scale s, range for multiple gradient steps Sttort and Send, and the number of gradient steps N. The dynamic guidance weight Snorm is omitted for brevity.
Input: a luw-quality image yo < sample from (0, for t from to 1 do
n. t Mel (Zz, t), (zr,t)
to - Vā. 21
1-
co (2t,
ar
Sstrrt I S
Send then
Di Multiple Gradient Steps
repeat
2t sample from N( sL.V 2olllly byy po(u(
to -- va." Z2
- col
until N times
end if
V2VVolle Tz-1 sample from N(p sEV Rolly 11 -oolyl0o))
end for return 20
Table Examples of Partial Guidance. Each image property corresponds to classifier, and each task involves one or multiple properties as guidance. The target value of each property is generally obtained either from the input image yo or the denoised intermediate output åo. For a composite task, we simply decompose it into multiple tasks and combine the respective guidance. Here, Clean denotes pre-trained restorer detailed See. 4.1, Identity refers to an identity mapping, and yref represents a reference image containing entity with the same identity yo-
Task
Property
Target: y
Classifter: Pe (y)()))
Inpainting
Unmasked Region
Mask
Mank
Lightness
rgb2gray (16)
rgb2gray
Colorization
Homogeneous Task
Color Statisties
AdaIN [18]
Identity
Restoration
Smooth Semantics
Clean (16)
Identity
Smooth Semantics
Clean (3n)
Identity
Ref-Based Restoration
Identity Reference
ArcFace (Hrer)
ArcFace [6]
Task
Composition
Composite Task
Old Photo Restoration
Restoration Inpainting + Colorization
(w/ scratches)
Classifier Guidance. Classifier guidance is used to guide unconditional diffusion model that conditional generation is achieved. Let be the target and P& (y((x) (z) be a classifier, the conditional distribution is approximated as a Gaussian similar the unconditional counterpart, but with the mean shifted by Ex t)g [7]:
(I y) No N (N t) + Eg (Rt, t)g. Eg (o( (I),
where g V x log (o(y)=) (==pp(t))))))) The gradient g acts as a guidance that leads the unconditional sampling distribution towards the condition target
3.2 Partial Guidance
Our partial guidance does not assume any prior knowledge of the degradation process. Instead, with diffusion prior acting as a regularization, we provide guidance only on the desired properties of high-quality images. The key to PGDiff is to construct proper guidance for each task. In this section, we will discuss the overall framework, and the formulation of the guidance for each task is presented in Sec. 4. The overview is summarized in Fig. and Algorithm I.
Property and Classifier. The first step of PGDiff high-quality output possesses. As summarized in Table 1, each image property corresponds to classifier ( ( and the intermediate outputs D, are updated by back-propagating the gradient computed the loss between the classifier output and the target y.
determine the desired properties which the

---

CNWTransformerr-ased Methods
Diffusion-prior-bsee Methods
Input
PULSE
GFP-GAN
CodeForner
GDP
DDNM
DifFace
Ours
Comparison on Blind Face Restoration, Input faces are corrupted by real-world degrada- tions. Our PGDiff produces high-quality faces with faithful details. (Zoom in for best view.)
Figure 2:
Given specific property (e.g.. lightness), construct the corresponding classifier (e.g.. rgb2gray), and apply classifier guidance during the reverse diffusion process as shown Fig. 1. Although our PGDiff conceptually similar classifier guidance, find that the conventional guidance scheme often leads to suboptimal performance. In this work, we borrow ideas from existing works [37, 5] and adopt a dynamic guidance scheme, which introduces adjustments to the guidance weight and number of gradient steps for enhanced quality and controllability.
Dynamic Guidance Scheme. Our dynamic guidance scheme consists of two components. First, we observe that the conventional classifier guidance, which adopts constant gradient scale s, often fails in guiding the output towards the target value. This is especially unfavourable in tasks where high similarity to the target is desired, such as inpainting and colorization. To alleviate this problem, calculate the gradient scale based on the magnitude change of the intermediate image [37]]
|| 21-11l2
SnoTM
119112
(8)
where 2/-1 ~N(no Eo). In this way, the dynamic guidance weight Snorm varies along iterations, more effectively guiding the output towards the target, thus improving the output quality.
Second, the conventional classifier guidance typically executes single gradient step at each denoising step. However, a single gradient step may not sufficiently steer the output toward the intended target, particularly when the intermediate outputs are laden with noise in the early phases of the denoising process. To address this, we allow multiple gradient steps at each denoising step [5) to improve flexibility. Specifically, one can improve the guidance strength of specific property by increasing the number of gradient steps. The process degenerates to the conventional classifier guidance when the number of gradient steps is set to 1. During inference, users have the flexibility modulate the strength of guidance for each property as per their requirements, thus boosting overall controllability.
Composite Guidance. Our partial guidance controls only the properties of high-quality outputs, and therefore can be easily extended to complex degradations by stacking respective properties. This is achieved by compositing the classifiers and summing the loss corresponding to each property. An example of composite tasks is shown Table 1. In addition, we also demonstrate that additional losses such perceptual loss [2, 16] and adversarial loss [22] can be incorporated for further quality improvement. Experiments demonstrate that our PGIDfff achieves better performance than existing works in complex tasks, where accurate modeling of the degradation process is impossible.
5

---

Table 2: Quantitative comparison on the real- world LeW-Test, WebPhoto-Tests and WIDER-Test datasets, Red and blue indicate the best and the second best performance, respectively.
CNWTransforrmer-basee Methods
Diffusion-prior-baseed Methods
Dataset
Metric
GFP-GAN [40]
CodeFormer [48]
DifFace [45]
Ours
FID
72.45
74.10
67.98
71.62
LFW-Test
NIQEL
3,90
4.52
5.47
4.15
FID1
91.43
86.19
90.58
86.18
WebPhoto-Test
NIQEL
4.13
4.65
4.48
4,34
FID
40.93
40.26
38.54
39.17
WIDER-Test
NIQEL
3.77
4.12
4.44
3.93
Table 3: Quantitative comparison on the synthetic CelebRef-HQ dataset. Red and blue indicate the best and the second best performance, respectively.
Can/ Methods
Diflusion-prior-bawee Methods
Metric
GFP-GAN [40] CodeFormer [48]
DifFace [45]
Ours (w/o ref)
Ours (w/ ref)
FID4 186.88
129.17
123.18
119.98
121.25
MUSIQU
63.33
69.62
60.98
67.26
64.67
LPIPS
0.49
0.36
0.35
0.34
0.35
IDSt
0.36
0.55
0,56
0.44
0.76
4
Applications
By exploiting the diffusion prior, our PGDiff applies to a wide range of restoration tasks by selecting appropriate guidance. In this section, we will introduce the guidance formulation and provide experimental results.
4.1 Blind Face Restoration
Partial Guidance Formulation. The objective of blind face restoration is to reconstruct a high- quality face image given low-quality imput corrupted by unknown degradations. In this task, the most straightforward approach is to train network with the MSE loss using synthetic pairs. However, while these methods are able to remove the degradations in the input, it well-known [26] that the MSE loss alone results in over-smoothed outputs. Therefore, extensive efforts have been devoted to improving the perceptual quality, such as incorporating addition losses (e.g., GAN loss) [22, 10, 16, 46, 8] and components (e.g., codebook [48, 13, 43, 47, 35] and dictionary [23, 24, 12, 8J). These approaches often require multi-stage training and experience training instability.
In our framework, we decompose high-quality face image into smooth semantics and high-freyuency details, and provide guidance solely on the smooth semantics. this way, the output 2o in each diffu- sion step is guided towards degradation-fiet solution space, and the diffusion prior is responsible for detail synthesis. Given an input low-quality image yo. we adopt pre-trained face restoration model predict smooth semantics as partial guidance. Our approach alleviates the training pressure of the previous models by optimizing model solely with the MSE loss. This is because our goal is to obtain smooth semantics without hallucinating unnecessary high-frequency details. Nevertheless, one can also provide guidance of various forms by selecting different restorers, such as CodeFormer [48]. The loss for classifier guidance is computed as: Lres
||ã0 f(vo)
Qualitative Results. We evaluate the proposed PGDiff on three real-world datasets, namely LFW- Test [40], WebPhoto-Test [40], and WIDER- Test [48]. We compare our method with both task-specific WWWTrrnnffrrmer-aasee restoration models [48, 26, 40] and ceaffusioo-prior-beee
models? [9, 42, 45]. methods such as GDP [9] and DDNM [42] are unable to generalize to real- world degradations, producing outputs with notable artifacts. In contrast, our PGDiff successfully removes the degradations and restores the facial details invisible in the input
As shown Fig. 2, existing diffusion--prorr-ban
2 Among them, GDP [9] and DDNM [42] support only 4x Exeed-eenel downsampling, while DifFace [45] is task-ipecific model for blind face restoration.

---

images. Moreover, our PGDiff performs favorably over task-specific methods even without extensive training on this task.
Quantitative Results on Real-world Datasets. To compare our performance with other methods quantitatively on real-world datasets, we adopt FID [14] and NIQE [27] as the evaluation metrics and test on three real-world datasets: LFW-Test [40], WebPhoto- Test [40], and WIDER-Tes -est [48]. LFW- Test consists of the first image from each person whose name starts with in the LFW dataset [40], which are 431 images in total. WebPhoto-Tees is dataset comprising 407 images with medium degradations collected from the Internet. WIDER- Test contains 970 severely degraded images from the WIDER Face dataset [40]. As shown in Table 2, method achieves the best second-best scores across all three datasets for both metrics. A Although GFP-GAN achieves the best NIQE scores across datasets, notable artifacts can be observed, shown in Fig. 2. Meanwhile, our method shows exceptional robustness and produces visually pleasing outputs without artifacts.
Quantitative Results Synthetic Dataset. We present a quantitative evaluation the synthetic CelebRef-HO dataset [24] in Table 3. Considering the importance of
in blind face restoration, we introduce reference-based restoration in Sec. 4.5 in addition the general restoration in Sec. 4.1. Table 3 shows that our methods achieve best or second best scores across both no-reference (NR) metrics for image quality (i.e., FID and MUSIQ) and full-reference (FR) metries for identity preservation (i.e. LPIPS and IDS). Since we employ heavy degradation settings when synthesizing CelebRef-HQ. is noteworthy that identity features are largely distorted severely corrupted input images. Thus, it is almost impossible to predict any additional identity information. Nevertheless, with our reference- based restoration, we observe that high-quality reference image of the same person helps generate personal characteristics that highly similar to the ground truth. The large enhancement of identity preservation is also indicated in Table 3, where our reference-basee method achieves the highest IDS, increasing by 0.32.
preetity-pressrving face without
4.2 Face Colorization
Partial Guidance Formulation. Motivated by color space decomposition (e.g., YCbCr, YUV), we decompose our guidance into lightness and color, and provide respective guidance on the two aspects. For lightness, the input image acts as a natural target since is a homogeneous-color image. Specifically, guide the output lightness towards that of the input using the simple rgb2gray operation. Equivalently, the loss is formulated as follows: L₁ = ||xgb2gray( (g) - rgb2gray (yo) The lightness guidance also be regarded dense structure guidance. This is essential in preserving image content.
With the lightness guidance constraining the structure of the output, we could guide the color synthesis process with a lenient constraint color statistics (i.e., mean and variance of each color channel). In particular, we construct the target by applying AdaIN [18] to âg. using pre-determined set of color statistics for each R, G. B channel. Then we push do towards the color-normalized output: L. مااا sg (AdaIN ( P)) I12, where IP refers the set of color statistics and sg(-) denotes the gradrrra operation [35]. The overall loss is formulated as: Leator = Li + Le, where a is a constant that controls the relative importance of the structure and color guidance. To construct a universal color tone, we compute the average color statistics from selected subset of the CelehA-HQ dataset [17]. We find that this simple strategy suffices to produce faithful results. Furthermore, our PGDiff can produce outputs with diverse color styles by computing the color statistics from different reference images.
Experimental Results. As shown in Fig. 3, GDP [9] and DDNM [42] lack the capability to produce vibrant colors. In contrast, our PGDiff produces colorized outputs simply by modeling the lightness and color statistics. Furthermore, our method is able to generate outputs with diverse color styles by calculating color statistics from various reference sets.
4.3 Face Inpainting
Partial Guidance Formulation. Since diffusion models have demonstrated remarkable capability in synthesizing realistic content [33, 29, 7, 15, 34], we apply guidance only on the unmasked regions, and rely on the synthesizing power of diffusion models to generate details in the masked regions. Let be a binary mask where 0 and denote the masked and unmasked regions, respectively. We confine the solution by ensuring that the resulting image closely resembles the input image
7

---

Various Color Styles
Ground Truth
Input
GDP
DDNM
Ours
Figure 3: Comparison on Face Colorization. Our PGDiff produces diverse colorized output with various color statistics given as guidance. The first column of our results is guided by the average color statistics of subset of the CelebA-HQ dataset [17], and the guiding statistics for the remaining three columns represented as image in the top right corner.
< Various Random Seeds ->
Ground Truth
Input
GPEN
CodeFormer
GDP
DDNM
Ours
Comparison on Face Inpainting on Challenging Cases. Our PGDiff produces natural outputs with pleasant details coherent with the unmasked regions. Moreover, different random seeds give various contents of high quality.
Figure 4:
within the unmasked regions: Linpaint ||B @ òg &0ll.. where represents the pixel-wise multiplication.
Experimental Results. We conduct experiments CelebRef-HQ [24]. As depicted in Fig. GPEN [44] and GDP [9] are unable to produce natural outputs, whereas CodeFormer [48] and DDNM [42] generate outputs with artifacts, such as color incoherence visual flaws. In contrast, our PGDiff successfully generates outputs with pleasant details coherent to the unmasked regions.
4.4 Old Photo Restoration
Partial Guidance Formulation. Quality degradations (e.g., blur, noise, downsampling, and JPEG
compression), color homogeneity, and scratches are three commonly artifacts in old photos. Therefore, we cast this problem as a joint task of restoration, colorization, and inpainting'. Similar to face colorization that composites the loss for each property, we composite the respective loss each task, and the overall loss is written as: Lotd = Lres + "color Leclor + "iippaint Linpaint, where Tinpaint and Yootor are constants controlling the relative importance of the different losses.
Experimental Results. We compare our PGDiff with BOPB [38]. GFP-GAN [40] and DDNM [42]. Among them, BOPB is model specifically for old photo restoration, GFP-GAN (v1) is able to restore and colorize faces, and DDNM is a cofffusion-prior-baae old photos with scratches. As shown in Fig. 5, BOPB, GFP-GAN, and DDNM all fail to give natural color in such composite task. While DDNM is able to complete scratches given proper scratch map, it fails to give a high-quality face restoration result. On the contrary, PGDiff generates sharp colorized faces without scratches and artifacts.
method that also claims to restore
We locate the scratches using automated algorithm [38]. and then inpaint the scratched regions.

---

Various Color Styles
Input
ВOPB
GEP-GAN
DDNM
Ours
Figure 5: Comparison on Old Photo Restoration on Challenging Cases. For a severely damaged old photo, with one eye masked with scratch, while only DDNM [42] is able to complete the missing eye, its restoration quality is significantly low. In contrast, our PGDiff produces high-quality restored outputs with natural color and complete faces.
Input
Eyaaf
Truth
Inpuu
Lyes
Lreff
Ground
Figure 6: Snference-Basedd Face Restoration. Our PGDiff, using Lref with identity loss guidance, produces personal characteristies that are hard to recover without reference, i.e, using Lrees only. (Zoom in for details)
4.5 Reference-Basee Restoration
Partial Guidance Formulation. In reference-basee restoration, reference image from the same identity is given to improve the resemblance of personal details in the output image. Most existing works exploiting diffusion prior [9, 42, 20, 45] are not applicable this task as there is direct transformation between the reference and the target. In contrast, our partial guidance is extensible to more complex tasks simply by compositing multiple losses. In particular, our PGDiff can incorporate personal identity as partial attribute of facial image. By utilizing a reference image and incor- porating the identity loss into the partial guidance, our framework can achieve improved personal details. We extract the identity features from the reference image using pre-trained face recognition network, such as ArcFace [6]. We then include the negative cosine similarity to the loss term Lres in blind face restoration (Sec. 4.1): Cref = Lres B sim( U2o, Uyy), where B controls the relative weight of the two losses. Here sim(-) represents the cosine similarity, and Us. and U, denote the ArcFace features of the predicted denoised image and the reference, respectively.
Experimental Results. We use the CelebRef-HQ dataset [24], which contains 1, 005 entities and each person has 3 to 21 high-quality images. To build testing pairs, for each entity, we choose one image and apply heavy degradations as the input, and then we select another image from the same identity the reference. In Fig. 6, we observe that without the identity loss term sim (,mu2,,, 0r), some of the personal details such as facial wrinkles and eye color cannot be recovered from the distorted inputs. With the additional identity loss guidance, such fine details can be restored. In addition, our PGDiff can be used to improve identity preservation of arbitrary face restorers. For instance, as shown in Fig. (a), by using CodeFormer [48] as our restorer and incorporating the identity loss, the fine details that CodeFormer alone cannot restore can now be recovered. Quantitative results are included Table and discussed in Sec. 4.1.
4.6 Quality Enhancement
Partial Guidance Formulation. Perceptual loss [16] and adversarial loss [10] are two common training losses used to improve quality. Motivated by this, we are interested in whether such losses can also be used as the guidance for additional quality gain. We demonstrate this possibility in the task of blind face restoration using the following loss: Lquality Lres Xper ||VGG(2o) VGG(y) 1 AGAN D(āu), where Aper and ÀGAN are the relative weights. Here VGG and D represent pre-trained VGG16 [32] and the GAN discriminator [19]. respectively.
Experimental Results. We demonstrate Fig. (b) that perceptual loss and adversarial loss can boost the blind restoration performance in terms of higher fidelity with photo-realism details.

---

Input
CodeFomer w 0
Reference
Ref-based
Ground Truth
Input
Lres
Lquality
CodeFormer
Referenced-basee CodeFomer
(b) Quality enhancement
Figure 7: (a) Using CoreFrrmer the restorer with our identity guidance improves the reconstruc- tion of fine details similar the ground truth. (b) The comparison results show that the quality enhancement loss is able to enhance fidelity with photo-realism details.
Input
Constant
Dynamic Sporm
Input
N
N
(a) Effect of dynamic guidance weight
(b) Effect multiple guidance steps
Figure 8: Ablation Study of Dynamic Guidance. The comparison results on the dynamic guidance scheme verify its effectiveness over the conventional classifier guidance scheme.
5 Ablation Studies
In this section, we perform ablation studies on the dynamic guidance scheme mentioned in Sec. 3.2 verify its effectiveness over the conventional classifier guidance scheme.
Effectiveness of Dynamic Guidance Weight. We first investigate the effectiveness of dynamic guidance weight Snorm in the face inpainting task, where the unmasked regions of the output image should be of high similarity to that of the input. As shown in Fig. 8 (a), without the dynamic guidance weight, although plausible content can still be generated in the masked area, the similarity and sharpness of the unmasked regions are remarkably decreased compared with the input. With Snorm replacing the constant s, the output is of high quality with unmasked regions nicely preserved. The results indicate that our dynamic guidance weight is the key to ensuring high similarity to the target during the guidance process.
Effectiveness of Multiple Gradient Steps. To verify the effectiveness of multiple gradient steps, we compare the blind restoration results with the number of guidance steps N set to 1, and While N is just the conventional classifier guidance, we set N 2 during the first 0.57 steps and set N = 3 during the first 0.3I' steps. As shown in Fig. (b), artifacts are removed and finer details generated N increases. These results suggest that multiple gradient steps serve to improve the strength of guiding the output toward the intended target, particularly when the intermediate outputs laden with noise in the early phases of the denoising process.
Conclusion
The generalizability of existing cattssion--rior-baee restoration approaches is limited by their reliance on prior knowledge of the degradation process. This study aims to offer a solution that alleviates this constraint, thereby broadening its applicability the real-world degradations. We find that through directly modeling high-quality image properties, one can reconstruct faithful outputs without knowing the exact degradation process. We exploit the synthesizing power diffusion models and provide guidance only properties that are easily accessible. Our proposed PGDiff with partial guidance is not only effective but is also extensible to composite tasks through aggregating multiple properties. Experiments demonstrate that PGDiff outperforms diffusion--pior-beee braio--bsorddd approaches in both homogeneous and composite tasks and matches the performance task-specific methods.
Acknowledgement. This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry ps)t
10

---

References
[1] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
[2] Joan Bruna, Pablo Sprechmann, and Yann LeCun. Super-resolution with deep convolutional sufficient statistics. In ICLR, 2016.
[3] Kelvin C.K. Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. GLEAN: Generative latent bank for large-factor image super-resolttion. In CVPR, 2021.
[4] Kelvin C.K. Chan, Xiangyu Xu, Xintao Wang. Jinwei Gu, and Chen Change Loy. GLEAN: Generative latent bank for image super-resolution and beyond. TPAMI, 2022.
[5] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen- -r. Attend-ang and-Excite: Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH, 2023,
[6] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurlPS, 2021.
[8] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[9] Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. CVPR, 2023.
[10] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
[11] Jinjin Gu, Yujun Shen, and Bolei Zhou, Image processing using multi-code GAN prior. In CVPR, 2020.
[12] Shuhang Gu, Wangmeng Zuo, Qi Xie, Deyu Meng, Xiangehu Feng, and Lei Zhang. Convolu- tional sparse coding for image super-resolution. In ICCV, 2015.
[13] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. VQFR: Blind face restoration with vector-quantized dictionary and parallel decoder. In ECCV, 2022.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrIPS, 2020.
[16] Justin Johnson, Alexandre Alahi, and Fei-Fei Li. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[17] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
[18] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
[20] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NewrIPS, 2022.

---

[21] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: Multi-scale image quality transformer. In ICCV, 2021.
[22] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, 2017.
[23] Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang. Blind face restoration via deep multi-scale component dictionaries. In ECCV, 2020.
[24] Xiaoming Li, Shiguang Zhang. Shangchen Zhou, Lei Zhang, and Wangmeng Zuo. Learning dual memory dictionaries for blind face restoration. TPAMI, 2022.
[25] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. In ICCV, 2021.
[26] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PULSE: Self- supervised photo upsampling via latent space exploration of generative models. In CVPR, 2020.
[27] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-212, 2012.
[28] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In ECCV, 2020.
[29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. ICC, 115:211-252. 2015.
[31] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop. Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[33] Jascha Sohll-Dckstedd Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICLR, 2015.
[34] Yang Song, Jascha Gohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[35] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurlPS, 2017.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS. 2017.
[37] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. arXiv preprint arXiv:2211.1377 2022.
[38] Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen. Bringing old photos back to life. In CVPR, 2020.
[39] Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin C.K. Chan, and Chen Change Loy. Ex- ploiting diffusion prior for real-world image super-resolution. arXiv preprint 00X0022305.070/5 2023.
[40] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-worke blind face restoration with generative facial prior. In CVPR, 2021.
12

---

[41] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data. ICCV, 2021.
[42] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2023.
[43] Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. RestoreFormer: High-quality blind face restoration from undegraded key-value pairs. In CVPR, 2022.
[44] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. GAN prior embedded network for blind face restoration in the wild. In CVPR, 2021.
[45] Zongsheng Yue and Chen Change Loy. DifFace: Blind face restoration with diffused error contraction. arXiv preprint arXiv:2212.065/2, 2022.
[46] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[47] Yang Zhao, Yu-Chuan Su, Chun-Te Chu, Yandong Li, Marius Renn, Yukun Zhu, Changyou Chen, and Xuhui Jia. Rethinking deep face restoration. In CVPR, 2022.
[48] Shangehen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022.
13

---

Appendix
In this supplementary material, we provide additional discussions and results. In Sec. A, we present additional implementation details including the inference requirements, choice of hyperparameters involved in the inference process, and discussions on the pre-trained restorer for blind face restoration. In Sec. B, we provide more results on various tasks, i.e., blind face restoration, old photo restoration, reference-based restoration, face colorization, and inpainting. Sec. C and Sec. D discuss the limitations and potential negative societal impacts of our work, respectively.
A Implementation Details
A.1 Inference Requirements
The pre-trained diffusion model we employ is a 512 x 512 denoising network trained on the FFHQ
dataset [18] provided by [45]. The inference process is carried out on NVIDIA RTX A5000 GPU,
A.2 Inference Hyperparameters
During the inference process, there involves hyperparameters belonging to three categories. (1) Sampling Parameters: The parameters in the sampling process (e.g., gradient scale s). (2) Par- tial Guidance Parameters: Additional parameters introduced by our partial guidance, which are mainly relative weights for properties involved in a certain task (e.g., a that controls the relative importance between the structure and color guidance in face colorization). (3) Optional Parameters: Parameters for optional quality enhancement (e.g., the range for multiple gradient steps take place [Satart, Send)). While it is principally flexible to tune the hyperparameters case by case, we provide a set of default parameter choices for each homogeneous task Table 4.
Table 4: Default hyperparameter settings in our experiments.
Task
Sampling
Partial Guidance
Optional
Suara
Unmasked Region
Lightness
Statistics
Color
Smooth Semantics
Mestity Reference
Perceptial Loss
Loss
Restoration
0.1
Cres
Tw5t 7 0.77
16:2
1e-2
Colotization
/
0.01
0.0.C.
Ispainting
Cinpaint
Ref-Bused Resioration
Crcs
10ein( Caa Vel
Tw005
T 0.7
le2
le-2
A.3 Restorer Design
Network Structure. In the blind face restoration task, given an input low-quality (LQ) image yo- we adopt pre-trained face restoration model f to predict smooth semantics as partial guidance. In this work, we employ the x generator of Real-ESRGAN [41] as our restoration backbone. The network follows the basic structure of SRResNet [22]. with RRDB being its basic blocks. In a x1 generator, the input image is first downsampled 4 times by a pixel unshuffling [31] layer before any convolution operations. In work, we deal with 512 512 input/output pairs, which means that most computation is done only in a 128 x 128 resolution scale. To employ it as the restorer f. we modify some of its settings. Empirically we find that adding 2, and as the input alongside Yo can enhance the sample quality in terms of sharpness. Consequently, the input to f is concatenation of Vo. Ft, and 4, with embedded with the sinusoidal timestep embeddings [36].
Training Details. f is implemented with the PyTorch framework and trained using four NVIDIA Tesla V100 GPUs 200K iterations. We train f with the FFHQ [18] and CelebA-HQ [17] datasets and form training pairs by synthesizing LQ images I from their HQ counterparts Tn, following a common pipeline with second-order degradation model [41, 23, 40, 44]. Since our goal is to obtain smooth semantics without hallucinating unnecessary high-frequency details, it is sufficient to optimize the model f solely with the MSE loss.
Model Analysis. To investigate the most effective restorer for blind face restoration, we compare the sample quality with restorer being (1o) and f Vt, 1), respectively. Here, ([o, It, 1) is the one trained by ourselves discussed above, and f(yo) is SwinIR [25] from DifFace [45], which is also
14

---

Table 5: Quantitative comparison on the synthetic CelebA-Test dataset on inpainting and colorization tasks. Red indicates the best performance.
Task
Inpainting
Colorization
Metric
FID
NIQEJ
MUSIQ-KonIQT
FID
NIQEL MUSIQ-AVAT
DDNM [42]
137.57
5.35
59.38
146.66
5.11
4.07
CodeFormer [48]
120.93
4.22
72.48
126.91
4.43
4.91
Ours
115.99
3.65
73.20
119.31
4.71
5.23
trained with MSE loss only. As shown in Fig. 9, when all the other inference settings are the same, we find that the sample quality with restorer f(yo: 24, t) higher in terms of sharpness compared with that of (wo). One may sacrifice certain degree of sharpness achieve higher inference speed by substituting the restorer with f (30), whose output is constant throughout T timesteps.
Input
Output with Restorer f(r))
Output with Restorer f(yo.
Input
Output with Restorer f f(r)
Output with Restorer f
Figure 9: Visual comparison of the restoration outputs with different restorers f in blind restoration. We observe that including 2t and as the input to f enhances the sharpness of the restored images.
15

---

B
More Results
B.1 More Results on Blind Face Restoration
In this section, we provide more qualitative comparisons with state-of-the-art methods, including
(1) task-specific CNN/Taanformerr-base restoration methods: PULSE [26], GFP-GAN [40], and CodeFormer [48] and (2) diffusion-pinn prioo-ppoooooiiee prior-based methods: GDP [9]. DDNM [42] and DifFace [45].
CNN/Tranfformer-Be Methods
Diffusion-prior-based Methods
Input
GFP-GAN
CodeFoomerr
GDP
DifFace
Figure 10: Comparison on Blind Face Restoration. Input faces are corrupted by real-world degradations. Our PGDiff produces high-quality faces with faithful details. (Zoom in for best view)
16

---

B.2 More Results on Old Photo Restoration
We provide more visual results of old photo restoration on challenging cases both with and without scratches, shown in Fig. 11. The test images come from both the CelebChild-Tess dataset [40] and the Internet. We compare our method with GFP-GAN (v1) [40] and DDNM [42]. Our method demonstrates obvious advantage in sample quality, especially in terms of vibrant colors, fine details, and sharpness.
Input
GFP-GAN
DDNM
Ours
Input
GFP-GAN
DDNM
Ours
Figure 11: Comparison on Old Photo Restoration on Challenging Cases. Our PGDiff able to produce high-quality restored outputs with natural color and complete faces.
17

---

B.3 More Results on Reference-Based Restoration
We provide more visual results on the reference- based restoration in Fig. 12, which is our exploratory extension based on blind face restoration. Test images come from the CelebRef-HQ dataset [24], which contains 1, 005 entities and each person has 3 to high-quality images. With identity loss added, observe that our method is able to produce personal characteristics similar to those of the ground truth.
rk ã
Input
Lrss
Reference
Lref
Ground Truth
Figure 12: Comparison on Reference- Face Restoration, Our PGDiff produces personal characteristics which are hard to recover without reference.
18

---

B.4 More Results on Face Inpainting
In this section, we provide quantitative and more qualitative comparisons with faae-of-thhe-as art methods in Fig. 13, including (1) task-specific methods: GPEN [44] and CodeFormer [48] and (2) diffusion- prior-based methods: GDP [9] and DDNM [42]. shown Fig. and Fig. 13, since DDNM and CodeFormer relatively more competitive than others, make quantitative comparisons of our methods against them.
We can observe from Fig. 13 that our method is able to recover challenging structures such as glasses. Moreover, diverse and photo-realism outputs can obtained by setting different random seeds. As for quantitative comparisons in Table 5, we believe that the ability to produce diverse results is also crucial in this task, and the evaluation should not be constrained to the similarity to the ground truth. Thus, opt to employ NR metrics including FID, NIQE, and MUSIQ instead of FR metrics. Regarding the MUSIQ metric, it has been trained on different datasets featuring various purposes [21] J. For inpainting, we employ MUSIQ- Q which focuses on quality assessment. Our method is able to achieve the highest score across all metrics.
< Various Random Seeds ~>
B
L
E
N
E E
N E
V VE E
A E
E
As
Ground Truth
Input
GPEN
CodeFormer
GDP
DDNM
Ours
Figure 13: Comparison on Face Inpainting on Challenging Cases. Our PGDiff produces natural outputs with pleasant details coherent to the unmasked regions. Moreover, different random seeds give various contents of high quality.
19

---

B.5 More Results on Face Colorization
In this section, we provide quantitative and more qualitative comparisons with state-of-the-art methods in Fig. 14, including (I) task-specific methods: CodeFormer [48] and (2) diffusion-prior based methods: GDP [9] and DDNM [42]. shown in Fig. 3 and Fig. 14, since DDNM and CodeFormer relatively more competitive than others, make quantitative comparisons of our methods against them.
We can observe from Fig. 14 that our method produces more vibrant colors and finer details than DDNM and CodeFormer. Moreover, our method demonstrates a desirable diversity by guiding with various color statistics. As for quantitative comparisons in Table 5, we believe that the ability to produce diverse results also crucial in this task, and the evaluation should not constrained to the similarity to the ground truth. Thus, we opt to employ NR metrics including FID, NIQE, and MUSIQ instead FR metrics. Regarding the MUSIQ metric, it has been trained on different datasets featuring various purposes [21]. For colorization,
choose MUSIQ-AVA that puts more emphasis on aesthetic assessment. Although CodeFormer has a better score in NIQE, it clearly alters the input identity (see Fig. 14) and requires training separate model for each task. the contrary, our method requires only pre-trained diffusion model for both inpainting and colorization, and is able to achieve best scores across almost all metrics.
Various Color Styles
Ground Truth
Input
CodeFormer
GDP
DDNM
Ours
Figure 14: Comparison on Face Colorization. Our PGDiff produces diverse colorized outputs with various color statistics given as guidance.
20

---

C Limitations
As our PGDiff based on a pre-trained diffusion model, our performance largely depends on the capability of the model in use. In addition, since face-specific diffusion model is adopted in this work, our method applicable only to faces in its current form. Nevertheless, this problem can be resolved by adopting stronger models trained for generic objects. For example, shown in Fig. 15, we employ an unconditional 256 256 diffusion model trained on the ImageNet dataset [30] provided by [7], and achieve promising results on inpainting and colorization. Further exploration on natural scene restoration will be left as our future work.
Input
Ours
Ground Truth
Imput
Ours
Ground Truth
(a) Inpainting
(b) Colorization
Figure 15: Extension on natural images for the inpainting and colorization tasks. By employing an unconditional 256 x 256 diffusion model trained the ImageNet dataset [30] provided by [7], our method achieves promising results.
D Broader Impacts
This work focuses restoring images corrupted by various forms of degradations. On the one hand, our method is capable enhancing the quality of images and improving user experiences. On the other hand, our method could generate inaccurate outputs, especially when the input is heavily corrupted. This could potentially lead to deceptive information, such as incorrect identity recognition. In addition, similar to other restoration algorithms, our method could be used by malicious users for data falsification. We advise the public to use our method with care.
21


Reviews:
[{"review_id": "J25AqZ6ybE", "reviewer": "Reviewer_GVHx", "summary": "This paper proposes a novel solution for blind face restoration. Instead of modeling the degradation process, the authors propose to model the desired properties of high-quality images as classifiers. Similar to guided diffusion, the authors guide the diffusion generation process with specific classifiers to achieve image restoration. Visual results well proved the effectiveness of the proposed methods.", "questions": "1. Please see the weaknesses.\n2. I am curious about the effectiveness of PG on natural image restoration. For example, what about applying PG on stable diffusion?\n3. For the colorization task, the lightness constraint is linear and can be analytically solved in a similar way to DDNM. I wonder if it works to simply apply Color Statistics to DDNM? ", "limitations": "Though the evaluation is poor, I still give a relatively positive score for the time being and hope that the authors can supplement quantitative experiments (and subjective questionnaires for blind restoration) and objectively discuss the problems existing in practical applications. After all, this work is about blind restoration. If there is a new method that works well, it will be very helpful to the community.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of modeling the desired properties of high-quality images as classifiers is novel and interesting.\n2. The proposed method can solve blind image restoration and can restore images following the reference properties, e.g., color and identity, which is novel and practical.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. There seems no quantitative evaluation. It is not persuasive enough with only visual comparisons. \n2. The setting of hyperparameters may be difficult.\n3. Some overclaims. For example, the author said in line 115, \"Our partial guidance does not assume any prior knowledge of the degradation process.\" However, PG needs to know the degradation type. For inpainting, the mask is also needed. (2) ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Quantitative evaluation and user study.** Please refer to the global response in \"Author Rebuttal\" for detailed elaboration.\n\n**Setting of hyperparameters.** As mentioned in A.2 of the supplementary material, while it is principally flexible to tune the hyperparameters case by case, we provided a set of default parameter choices for each homogeneous task in Table 1.\n\n**Statement of our agnosticism to the degradation process.** We intended to emphasize the agnosticism of degradation process on blind restoration. Existing works that employ guided diffusion for image restoration (DDRM, DDNM, GDP) require prior knowledge on the degradation process (e.g., downsampling kernels). By modeling the desired properties of high-quality images, we bypass this requirement. Thank you for pointing out the inpainting case where a mask is needed, and we will revise our claim to exclude the inpainting task.\n\n**Applying partial guidance on natural image restoration.** Due to the different nature of human faces and natural images, we plan to explore partial guidance on natural images in our future work. Nevertheless, we have made some early explorations demonstrating the feasibility. We provide some natural image restoration results in Fig. 1 of the rebuttal material. In addition, we also provided natural image inpainting and colorization results in Fig. 7 of the supplementary material. Since stable diffusion performs diffusion process in the latent space, it requires careful design of appropriate properties in cases where existing ones are not applicable. This aspect will be further investigated in our future work.\n\n**Applying color statistics on DDNM.** As shown in Fig. 2 of the rebuttal material, we apply our partial guidance on DDNM by guiding on $\\hat{x} _{0|t}$ (computed by DDNM) with AdaIN($\\hat{x} _{0|t}$). DDNM produces more vivid and natural colorized outputs with various color statistics given as guidance. It shows that our partial guidance can be flexibly applied in the DDNM framework for further improvement.\n\n**Limitations.** Please refer to the global response in \"Author Rebuttal\" for detailed elaboration.\n"}, {"review_id": "fnN7Nnje8O", "reviewer": "Reviewer_FgM3", "summary": "\tThis paper proposes partial guidance, an approach exploiting pre-trained diffusion models for face restoration. Instead of making assumption about the specific degradation process, partial guidance models properties of high-quality images such as structure and color statistics to implement classifier-guidance during the reverse diffusion process. This approach suits a range of restoration task and can be extended to composite tasks. Experiments demonstrate the effectiveness of the proposed method.", "questions": "-\tIn blind face restoration task, in addition to real-world data, why synthetic data (like degraded CelebA/CelebA-HQ) is not used for evaluation of more quantitative metrics (like PSNR, SSIM, LPIPS), which is adopted in previous work like DifFace.\n-\tOther concerns have already been mentioned in Weakness.\n", "limitations": "Yes, the authors have addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-\tThis paper considers exploiting image properties irrelevant to specific degradation process to tackle versatile face restoration problems, which has demonstrated effectiveness in experiments.\n-\tThe qualitative results of the proposed method provided in the paper and the supplementary material are sufficient and impressive.\n", "weaknesses": "-\tExcept blind face restoration task, more quantitative results are expected in the remaining tasks like face colorization and face inpainting.\n-\tIn blind face restoration task, the pretrained restorer (such as Real-ESRGAN stated in the supplementary material) is employed and finetuned to predict smooth semantics, but these intermediate results are missing, and the improvement of the proposed method compared to the pretrained/finetuned restorer is not demonstrated or analyzed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Quantitative evaluation and user study.** We believe that PSNR and SSIM fail to reflect the true image quality. For example, in Tab.1 of CodeFormer, input images achieve the third highest PSNR scores and the highest SSIM scores across all methods. Thus, we exclude them from full-reference metrics in our evaluation and employ LPIPS and IDS instead. Please refer to the global response in \"Author Rebuttal\" for detailed elaboration.\n\n**Visualization of intermediate results.** As shown in Fig. 3 of the rebuttal material, given an input low-quality image, we visualize intermediate results during the sampling process. $f(y_0,x_t,t)$ shows the smooth semantics predicted by a pretrained restorer at time $t$, which serves as guidance for sampling $x_{t-1}$. We observe that the pretrained restorer can only predict faces without rich details and sharpness at the early stage. Thanks to our design of including $x_t$ alongside $y_0$ as the input to $f$, the prediction results of the pretrained restorer benefit from $\\hat{x} _0$ which is growing sharper, and in turn enhance the sharpness of the $x _{t-1}$ sampled.\n\n"}, {"review_id": "ig5Jbwdg38", "reviewer": "Reviewer_hvsJ", "summary": "This paper proposes to use some simple properties to guide the reverse diffusion process. The proposed approach makes no assumptions about the degradation process. This paper also shows many different face restoration visual results to demonstrate the superiority of the proposed method.\n", "questions": "Please see Weakness.", "limitations": "This paper proposes to use some simple properties to guide the reverse diffusion process. But this paper has not discussed the limitations of the proposed method. I suggest the author discuss the potential and limitations of the proposed method in more detail such as showing some failure cases, which will make the contributions of the proposed method more significant.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed method has been experimented with in many different face restoration tasks, which demonstrates that the proposed method can be easily adapted to other face restoration tasks.\n2.\tOverall, this paper is well-writing and easy to understand.\n", "weaknesses": "1.\tThe main contribution of this paper is using some simple properties to guide the reverse diffusion process, which is inherited from classifier guidance [7]. What is the technical and original contribution of this paper? Please discuss the technical differences between the proposed work and classifier guidance [7].\n2.\tHow to choose the properties that are used in different tasks? This deserves an ablation study. Please show the influence of some different properties.\n3.\tThis paper claims one of the advantages of the proposed method is no assumption of the degradation process. As far as I know, the DifFace also does not need to know the degradation process. What is the advantage of the proposed method against DifFace?\n4.\tNo quantitative results are found in the manuscript. Only visual results are not enough to evaluate the proposed method. Please provide a fair quantitative comparison against other comparing methods such as commonly used datasets CelebA-Test, and three real-world testing datasets proposed in GFPGAN.\n5.\tThe inference time is also important to evaluate the proposed method. Please provide the inference time comparison again with other comparing methods.\n6.\tSome failure cases can contribute to understanding the limitation of the proposed method. Please show some failure cases.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Technical differences between our method and classifier guidance.** To model the desired properties of high-quality images, we devise an instantiation named partial guidance by adapting classifier guidance in image restoration (IR) tasks. While the rough idea is inherited from the classifier guidance framework, we extend the classifier from a label predictor to a dense predictor corresponding to a specific image property in the image restoration task. More importantly, we propose special design for the classifier network (see A.3 in the supplementary), dynamic guidance scheme (see Sec. 3.2 of the manuscript), and composite guidance (see Sec. 3.2 of the manuscript), which are all crucial to allow flexible property combinations solving versatile image restoration tasks. To our knowledge, we are the first to extend classifier guidance to an IR paradigm where the degradation process is not needed.\n\n**Choice of Properties.** The choice of properties for each task is listed at Tab. 1 of the manuscript. The selected properties are common ones involved in each task. For example, lightness and color values make up a colorized image. In colorization tasks, it is natural for us to guide on these two properties. For blind face restoration, guidance may only come from the input image. We find that using a clean version of the input image gives better performance than using the original input image (see Fig. 4 of the rebuttal material). As for extending general face restoration to reference-based restoration, identity features (e.g., ArcFace embedding) are widely used to extract personal characteristics. Thus, we propose to guide on the identity features of a reference image of the same identity as the input.\n\n**Advantages of the proposed method against DifFace.** Our method proposes a more general framework than DifFace, and DifFace is in fact a special case of our method when tackling blind face restoration. Specifically, DifFace is equivalent to applying partial guidance at only one iteration step multiple times until convergence. As for the application scope, while DifFace is confined to only blind face restoration, our method is versatile at multiple tasks thanks to the flexibility of our framework. Furthermore, extensive qualitative and quantitative results together show that our method achieves better quality and fidelity in the blind face restoration task than DifFace.\n\n**Quantitative evaluation and user study.** Please refer to the global response in \"Author Rebuttal\" for detailed elaboration.\n\n**Inference time.** Table below shows the inference time comparison with other methods. The results are tested on GeForce RTX 3090, with $512 \\times 512$ input images. Although the submitted version is based on DDPM, which takes 1000 steps during sampling, we also provide an accelerated version by using DDIM without a major performance drop (FID: [DDPM] 115.99 vs [DDIM] 117.99 on inpainting for example). Further acceleration such as distillation is left as our future work. The inference time can thus be about 10 times faster, which is comparable to almost all the other diffusion-based methods. \n\n| *Inference Time* | **Ours (DDPM, 1000 steps)** | **Ours (DDIM, 100 steps)** | **GDP** | **DDNM** | **DifFace** | **CodeFormer** | **GFPGAN** |\n|----------|----------|-----------|---------|----------|-------------|----------------|------------|\n| Time (s/img) | 118.18   |   12.02    \t| 149.62  | 15.78\t| 3.96    \t| 0.05       \t| 0.03   \t|\n\n**Limitations.** Please refer to the global response in \"Author Rebuttal\" for detailed elaboration.\n"}, {"review_id": "r6xXkmFwoX", "reviewer": "Reviewer_ggEw", "summary": "The current restoration approaches based on diffusion prior rely on prior knowledge of the degradation process,  and thus fail to seamlessly adapt to different scenarios. Motivated by this, the paper proposed a \" partial guidance\" approach to directly modeling the distribution properties of high-quality images and then exploit it to serve as guidance for the diffusion process. The extensive experiments demonstrate the advantage of the proposed approach.", "questions": "How would the proposed method compare to the pre-trained text2img diffusion model?\n\n- Let me explain for further clarification: with the current pre-trained text2img diffusion model (stable diffusion, for example), one can perform image enhancement using img2img pipeline,  where you start with an inversed image (plus a small amount of noise) and guide the diffusion process with text prompt like \"a high-resolution high-quality photo\".  It would be interesting to add such an experiment for comparison.\n\n- for you reference, by \"img2img\" pipeline I mean: https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/img2img", "limitations": "No major limitation in terms of societal impact ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method diverges from traditional practices of modeling the degradation process, and instead, focuses on modeling the desired visual properties of high-quality images. The learned vision cues are then served as diffusion guidance for the generation process. This motivation and the proposed model are technically sound to me. As demonstrated in the paper, it showcases adaptability to different degradation situations, yielding outstanding results.\n\n- The experiments conducted in the study seems solid, featuring comprehensive comparisons with various baseline approaches.", "weaknesses": "- For face restoration, it's important to make sure the restoration is performed in an identity-preserving way, and quantitative evaluation regarding this is crucial. However, such quantitative evaluation seems to be missing in the paper. \n\n- To clarify further,  I have no doubt the image quality of the proposed approach outperforms the baseline, but I'm concerned if the quality is improved at the cost of degradation in identity. So it's important to get a quantitative evaluation regarding this", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Identity-preserving evaluation on blind face restoration.** We provide a quantitative evaluation regarding identity-preserving issues in the table below. Considering the importance of identity-preserving in blind face restoration, we introduced reference-based restoration in Sec. 4.5 of the manuscript in addition to the general restoration in Sec. 4.1. Please note that existing methods do not have a mechanism to accept additional identity or reference guidance.\n\nSince we employ heavy degradation settings when synthesizing CelebRef-HQ, it is noteworthy that identity features are largely distorted in severely corrupted input images (see Fig. 6 of the rebuttal material). Thus, it is almost impossible to predict an identity-preserving face without any additional identity information. However, with our reference-based restoration, we observe that the restored face with a high-quality reference image of the same person helps generate personal characteristics which are very similar to the ground truth. The large enhancement of identity preservation can also be seen in the table below, where our reference-based methods achieve the highest IDS, increasing by 0.32 and significantly higher than the other methods. Thanks to the flexibility of our framework, high-quality reference images effectively contribute to restoring low-quality inputs especially in terms of identity preservation.\n\n|   *Blind Restoration*  | **Ours (w/ ref)** | **Ours (w/o ref)** | **DifFace** | **GFPGAN** | **CodeFormer** |\n|-----------|:------------:|:--------:|:-----------:|:----------:|:--------------:|\n| **IDS**$\\uparrow$   | \t**0.76** \t|   0.44   | \t0.56*\t|\t0.36\t|  \t0.55  \t|\n\n**Comparison between our method and text2img diffusion model.** As suggested by the reviewer ggEw, we compare the results generated by img2img stable diffusion model guided by different text prompts in Fig. 5 of the rebuttal material. We observe that a general text prompt like “A high-resolution high-quality photo” fails to generate a truly high-quality restored output given a severely corrupted input. Adding more detailed descriptions (e.g., “boy”, “smiling boy”, and “grinning boy”) of the input image helps enhance the output quality, but at a cost of significant loss of fidelity and identity. By comparison, our method obviously outperforms the text2img diffusion model in terms of both quality and fidelity.\n"}]

Author rebuttal:
We are encouraged that the reviewers find our work novel and interesting [Reviewer GVHx, ggEw]; practical and versatile in multiple image restoration tasks [Reviewer GVHx, FgM3, hvsJ, ggEw]; presenting impressive and outstanding visual results [Reviewer GVHx, FgM3, ggEw]; well-written and easy to follow [Reviewer GVHx, hvsJ].

In light of the common concerns about more quantitative evaluation and discussion on limitations of our method, we supplement with a detailed discourse on these matters here.

**Quantitative evaluation and user study.** 

**For blind face restoration.** We provided a quantitative comparison in Tab. 2 of the supplementary, evaluating on three *real-world* datasets LFW-Test, WebPhoto-Test, and WIDER-Test. Our method achieves best or second-best scores across all three datasets for both FID and NIQE metrics. 

We also include a quantitative evaluation on the *synthetic* CelebRef-HQ dataset in the table below. Considering the importance of identity-preserving in blind face restoration, we introduced reference-based restoration in Sec. 4.5 of the manuscript in addition to the general restoration in Sec. 4.1. As shown in Fig. 2 of both the manuscript and supplementary materials, since DifFace, GFPGAN, and CodeFormer are relatively more competitive than others, we make quantitative comparisons of our methods against them. The table below shows that our methods achieve best or second best scores across both no-reference (NR) metrics for image quality (i.e., FID and MUSIQ) and full-reference (FR) metrics for identity preservation (i.e., LPIPS and IDS). 

Since we employ heavy degradation settings when synthesizing CelebRef-HQ, it is noteworthy that identity features are largely distorted in severely corrupted input images (see Fig. 6 of the rebuttal material). Thus, it is almost impossible to predict an identity-preserving face without any additional identity information. Nevertheless, with our reference-based restoration, we observe that the restored face with a high-quality reference image of the same person helps generate personal characteristics that are highly similar to the ground truth. The large enhancement of identity preservation is also indicated in the table below, where our reference-based method achieves the highest IDS, increasing by 0.32.

|   *Blind Restoration*  | **Ours (w/ ref)** | **Ours (w/o ref)** | **DifFace** | **GFPGAN** | **CodeFormer** |
|-----------|:------------:|:--------:|:-----------:|:----------:|:--------------:|
| **FID**$\downarrow$   |	121.25*	|  **119.98**  |	123.18   |   186.88   | 	129.17 	|
| **MUSIQ-KonIQ**$\uparrow$ | 	64.67	|   67.26*  |	60.98	|	63.33   |	  **69.62** 	|
| **LPIPS**$\downarrow$ | 	0.35* 	|   **0.34**   | 	0.35*	|	0.49	|  	0.36  	|
| **IDS**$\uparrow$   | 	**0.76** 	|   0.44   | 	0.56*	|	0.36	|  	0.55  	|

In addition to the qualitative and quantitative evaluations on blind face restoration, the results of a user study are shown in the table below, with an involvement of 30 participants. Each participant is shown 30 randomly sampled image triplets and asked to select a visually better restored output. Each triplet is composed of one corrupted input, one from our method, and one from a randomly chosen method. It is observed that our results are preferred by participants compared to DifFace and GFPGAN. Considering CodeFormer is a task-specific method that requires careful multi-stage training, our method is able to achieve a comparable performance with CodeFormer even without extensive training on the task.

|           *User Study*      | **DifFace** | **GFPGAN** | **CodeFormer** |
|-----------------|:--------:|:-------:|:--------------:|
| **Favoring Ours**$\uparrow$ |  72.60%  |  72.01% |     51.22%     |

**For face inpainting and colorization.** We provide a quantitative evaluation on the *synthetic* CelebA-Test dataset in the table below. We believe that the ability to produce diverse results is also crucial in these two tasks, and the evaluation should not be constrained to the similarity to the ground truth. Thus, we opt to employ NR metrics including FID, NIQE, and MUSIQ instead of FR metrics. Regarding the MUSIQ metric, it has been trained on different datasets featuring various purposes. For inpainting, we employ MUSIQ-KonIQ which focuses on quality assessment, while for colorization, we choose MUSIQ-AVA that puts more emphasis on aesthetic assessment. Although CodeFormer has a better score in NIQE for the colorization task, it clearly alters the input identity (see Fig. 6 of the supplementary) and requires training a separate model for each task. On the contrary, our method requires only a pre-trained diffusion model for both inpainting and colorization, and is able to achieve best scores across almost all metrics. The qualitative results in Fig. 3, 4 of the manuscript and Fig. 5, 6 of the supplementary also prove the superiority of our approach.

| *Inpainting*  	| **Ours** | **CodeFormer** | **DDNM** |
|-----------------|----------|----------------|----------|
| **FID**$\downarrow$     	|  **115.99**  | 	120.93 	|  137.57  |
| **NIQE**$\downarrow$    	|   **3.65**   |  	4.22  	|   5.35   |
| **MUSIQ-KonIQ**$\uparrow$ |   **73.20**  |  	72.48 	|   59.38  |

| *Colorization* | **Ours** | **CF** | **DDNM** |
|------------------|----------|--------|----------|
| **FID**$\downarrow$      	|  **119.31**  | 126.91 |  146.66  |
| **NIQE**$\downarrow$     	|   4.71   |  **4.43**  |   5.11   |
| **MUSIQ-AVA**$\uparrow$|   **5.23**   |  4.91  |   4.07   |

**Limitations.**

As our partial guidance is based on a pre-trained diffusion model, our performance largely depends on the capability of the model in use. The pre-trained face diffusion model we employed is trained on the FFHQ dataset, where side faces seldom exist. As a result, failures tend to occur on side faces (see Fig. 7 in the rebuttal material). Since CodeFormer and DifFace are also trained on the FFHQ dataset, we share similar issues on side faces.


Meta-review: The paper presents a method for face restoration using a mechanism called partial guidance, wherein intermediate outputs of the diffusion model are supervised by a classifier to promote, e.g., consistency with the grayscale input for colorization or identity similarity. Initially the paper received borderline reviews, with reviewers raising concerns about the lack of quantitative evaluation and need for additional comparisons to baseline methods, especially DifFace. The authors responded with a quantitative evaluation of the method compared to baselines for blind restoration, inpainting, and colorization. Three of the four reviewers upgraded their scores, and, after considering the rebuttal and discussion, the AC finds the evaluation and quality of results sufficient to merit acceptance. The authors should include the quantitative evaluation and the additional results requested by reviewers in the camera ready version.
Decision: Accept (poster)