SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Partial Guidance Formulation. Motivated by color space decomposition (e.g., YCbCr, YUV), we decompose our guidance into lightness and color, and provide respective guidance on the two aspects. For lightness, the input image acts as a natural target since it is a homogeneous-color image. Specifically, we guide the output lightness towards that of the input using the simple rgb2gray operation. Equivalently, the loss is formulated as follows:  $\mathcal{L}_l = ||\mathrm{rgb}2\mathrm{gray}(\hat{x}_0) - \mathrm{rgb}2\mathrm{gray}(y_0)||_2^2$ . The lightness guidance can also be regarded as a dense structure guidance. This is essential in preserving image content.
With the lightness guidance constraining the structure of the output, we could guide the color synthesis process with a lenient constraint - color statistics (i.e., mean and variance of each color channel). In particular, we construct the target by applying AdaIN [18] to  $\hat{x}_0$ , using a pre-determined set of color statistics for each R, G, B channel. Then we push  $\hat{x}_0$  towards the color-normalized output:  $\mathcal{L}_c = ||\hat{x}_0 - \mathrm{sg}(\mathrm{AdaIN}(\hat{x}_0,\mathbb{P}))||_2^2$ , where  $\mathbb{P}$  refers to the set of color statistics and  $\mathrm{sg}(\cdot)$  denotes the stop-gradient operation [35]. The overall loss is formulated as:  $\mathcal{L}_{color} = \mathcal{L}_l + \alpha \cdot \mathcal{L}_c$ , where  $\alpha$  is a constant that controls the relative importance of the structure and color guidance. To construct a universal color tone, we compute the average color statistics from a selected subset of the CelebA-HQ dataset [17]. We find that this simple strategy suffices to produce faithful results. Furthermore, our PGDiff can produce outputs with diverse color styles by computing the color statistics from different reference images.
Experimental Results. As shown in Fig. 3, GDP [9] and DDNM [42] lack the capability to produce vibrant colors. In contrast, our PGDiff produces colorized outputs simply by modeling the lightness and color statistics. Furthermore, our method is able to generate outputs with diverse color styles by calculating color statistics from various reference sets.
4.3 Face Inpainting
Partial Guidance Formulation. Since diffusion models have demonstrated remarkable capability in synthesizing realistic content [33, 29, 7, 15, 34], we apply guidance only on the unmasked regions, and rely on the synthesizing power of diffusion models to generate details in the masked regions. Let  $B$  be a binary mask where 0 and 1 denote the masked and unmasked regions, respectively. We confine the solution by ensuring that the resulting image closely resembles the input image
GDP
Figure 3: Comparison on Face Colorization. Our PGDiff produces diverse colorized output with various color statistics given as guidance. The first column of our results is guided by the average color statistics of a subset of the CelebA-HQ dataset [17], and the guiding statistics for the remaining three columns are represented as an image in the top right corner.
Ground Truth
Figure 4: Comparison on Face Inpainting on Challenging Cases. Our PGDiff produces natural outputs with pleasant details coherent with the unmasked regions. Moreover, different random seeds give various contents of high quality.
$\leftarrow$  Various Random Seeds
GDP
within the unmasked regions:  $\mathcal{L}_{inpaint} = ||B\otimes \hat{x}_0 - B\otimes y_0||_2^2$ , where  $\otimes$  represents the pixel-wise multiplication.
Experimental Results. We conduct experiments on CelebRef-HQ [24]. As depicted in Fig. 4, GPEN [44] and GDP [9] are unable to produce natural outputs, whereas CodeFormer [48] and DDNM [42] generate outputs with artifacts, such as color incoherence or visual flaws. In contrast, our PGDiff successfully generates outputs with pleasant details coherent to the unmasked regions.
4.4 Old Photo Restoration
Partial Guidance Formulation. Quality degradations (e.g., blur, noise, downsampling, and JPEG compression), color homogeneity, and scratches are three commonly seen artifacts in old photos. Therefore, we cast this problem as a joint task of restoration, colorization, and inpainting<sup>3</sup>. Similar to face colorization that composes the loss for each property, we compose the respective loss in each task, and the overall loss is written as:  $\mathcal{L}_{old} = \mathcal{L}_{res} + \gamma_{color} \cdot \mathcal{L}_{color} + \gamma_{inpaint} \cdot \mathcal{L}_{inpaint}$ , where  $\gamma_{inpaint}$  and  $\gamma_{color}$  are constants controlling the relative importance of the different losses.
Experimental Results. We compare our PGDiff with BOPB [38], GFP-GAN [40] and DDNM [42]. Among them, BOPB is a model specifically for old photo restoration, GFP-GAN (v1) is able to restore and colorize faces, and DDNM is a diffusion-prior-based method that also claims to restore old photos with scratches. As shown in Fig. 5, BOPB, GFP-GAN, and DDNM all fail to give natural color in such a composite task. While DDNM is able to complete scratches given a proper scratch map, it fails to give a high-quality face restoration result. On the contrary, PGDiff generates sharp colorized faces without scratches and artifacts.
3We locate the scratches using an automated algorithm [38], and then inpaint the scratched regions.
BOPB
GFP-GAN
Figure 5: Comparison on Old Photo Restoration on Challenging Cases. For a severely damaged old photo, with one eye masked with scratch, while only DDNM [42] is able to complete the missing eye, its restoration quality is significantly low. In contrast, our PGDiff produces high-quality restored outputs with natural color and complete faces.
Input
Figure 6: Reference-Based Face Restoration. Our PGDiff, using  $\mathcal{L}_{ref}$  with identity loss as guidance, produces personal characteristics that are hard to recover without reference, i.e., using  $\mathcal{L}_{res}$  only. (Zoom in for details)
$\mathcal{L}_{res}$
$\mathcal{L}_{ref}$
4.5 Reference-Based Restoration
Partial Guidance Formulation. In reference-based restoration, a reference image from the same identity is given to improve the resemblance of personal details in the output image. Most existing works exploiting diffusion prior [9, 42, 20, 45] are not applicable to this task as there is no direct transformation between the reference and the target. In contrast, our partial guidance is extensible to more complex tasks simply by compositing multiple losses. In particular, our PGDiff can incorporate personal identity as a partial attribute of a facial image. By utilizing a reference image and incorporating the identity loss into the partial guidance, our framework can achieve improved personal details. We extract the identity features from the reference image using a pre-trained face recognition network, such as ArcFace [6]. We then include the negative cosine similarity to the loss term  $\mathcal{L}_{res}$  in blind face restoration (Sec. 4.1):  $\mathcal{L}_{ref} = \mathcal{L}_{res} - \beta \cdot \mathrm{sim}(v_{\hat{x}_0}, v_r)$ , where  $\beta$  controls the relative weight of the two losses. Here  $\mathrm{sim}(\cdot)$  represents the cosine similarity, and  $v_{\hat{x}_0}$  and  $v_r$  denote the ArcFace features of the predicted denoised image and the reference, respectively.
Experimental Results. We use the CelebRef-HQ dataset [24], which contains 1,005 entities and each person has 3 to 21 high-quality images. To build testing pairs, for each entity, we choose one image and apply heavy degradations as the input, and then we select another image from the same identity as the reference. In Fig. 6, we observe that without the identity loss term  $\sin(v_{\hat{x}_0}, v_r)$ , some of the personal details such as facial wrinkles and eye color cannot be recovered from the distorted inputs. With the additional identity loss as guidance, such fine details can be restored. In addition, our PGDiff can be used to improve identity preservation of arbitrary face restorers. For instance, as shown in Fig. 7 (a), by using CodeFormer [48] as our restorer and incorporating the identity loss, the fine details that CodeFormer alone cannot restore can now be recovered. Quantitative results are included in Table 3 and discussed in Sec. 4.1.
4.6 Quality Enhancement
Partial Guidance Formulation. Perceptual loss [16] and adversarial loss [10] are two common training losses used to improve quality. Motivated by this, we are interested in whether such losses can also be used as the guidance for additional quality gain. We demonstrate this possibility in the task of blind face restoration using the following loss:  $\mathcal{L}_{\text{quality}} = \mathcal{L}_{\text{res}} + \lambda_{per} \cdot ||\nabla G(G(\hat{x}_0) - VGG(y)||_2^2 + \lambda_{GAN} \cdot D(\hat{x}_0)$ , where  $\lambda_{per}$  and  $\lambda_{GAN}$  are the relative weights. Here VGG and  $D$  represent pre-trained VGG16 [32] and the GAN discriminator [19], respectively.
Experimental Results. We demonstrate in Fig. 7 (b) that perceptual loss and adversarial loss can boost the blind restoration performance in terms of higher fidelity with photo-realism details.
Figure 7: (a) Using CodeFormer as the restorer with our identity guidance improves the reconstruction of fine details similar to the ground truth. (b) The comparison results show that the quality enhancement loss is able to enhance fidelity with photo-realism details.
(a) Effect of dynamic guidance weight
Figure 8: Ablation Study of Dynamic Guidance. The comparison results on the dynamic guidance scheme verify its effectiveness over the conventional classifier guidance scheme.
(b) Effect of multiple guidance steps
In this section, we perform ablation studies on the dynamic guidance scheme mentioned in Sec. 3.2 to verify its effectiveness over the conventional classifier guidance scheme.
Effectiveness of Dynamic Guidance Weight. We first investigate the effectiveness of dynamic guidance weight  $s_{norm}$  in the face inpainting task, where the unmasked regions of the output image should be of high similarity to that of the input. As shown in Fig. 8 (a), without the dynamic guidance weight, although plausible content can still be generated in the masked area, the similarity and sharpness of the unmasked regions are remarkably decreased compared with the input. With  $s_{norm}$  replacing the constant  $s$ , the output is of high quality with unmasked regions nicely preserved. The results indicate that our dynamic guidance weight is the key to ensuring high similarity to the target during the guidance process.
Effectiveness of Multiple Gradient Steps. To verify the effectiveness of multiple gradient steps, we compare the blind restoration results with the number of guidance steps  $N$  set to be 1, 2, and 3. While  $N = 1$  is just the conventional classifier guidance, we set  $N = 2$  during the first  $0.5T$  steps and set  $N = 3$  during the first  $0.3T$  steps. As shown in Fig. 8 (b), artifacts are removed and finer details are generated as  $N$  increases. These results suggest that multiple gradient steps serve to improve the strength of guiding the output toward the intended target, particularly when the intermediate outputs are laden with noise in the early phases of the denoising process.
The generalizability of existing diffusion-prior-based restoration approaches is limited by their reliance on prior knowledge of the degradation process. This study aims to offer a solution that alleviates this constraint, thereby broadening its applicability to the real-world degradations. We find that through directly modeling high-quality image properties, one can reconstruct faithful outputs without knowing the exact degradation process. We exploit the synthesizing power of diffusion models and provide guidance only on properties that are easily accessible. Our proposed PGDiff with partial guidance is not only effective but is also extensible to composite tasks through aggregating multiple properties. Experiments demonstrate that PGDiff outperforms diffusion-prior-based approaches in both homogeneous and composite tasks and matches the performance of task-specific methods.
Acknowledgement. This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
[1] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
[2] Joan Bruna, Pablo Sprechmann, and Yann LeCun. Super-resolution with deep convolutional sufficient statistics. In ICLR, 2016.
[3] Kelvin C.K. Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. GLEAN: Generative latent bank for large-factor image super-resolution. In CVPR, 2021.
[4] Kelvin C.K. Chan, Xiangyu Xu, Xintao Wang, Jinwei Gu, and Chen Change Loy. GLEAN: Generative latent bank for image super-resolution and beyond. TPAMI, 2022.
[5] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-Excite: Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH, 2023.
[6] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021.
[8] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[9] Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. In CVPR, 2023.
[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
[11] Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code GAN prior. In CVPR, 2020.
[12] Shuhang Gu, Wangmeng Zuo, Qi Xie, Deyu Meng, Xiangchu Feng, and Lei Zhang. Convolutional sparse coding for image super-resolution. In ICCV, 2015.
[13] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. VQFR: Blind face restoration with vector-quantized dictionary and parallel decoder. In ECCV, 2022.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[16] Justin Johnson, Alexandre Alahi, and Fei-Fei Li. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[17] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
[18] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
[20] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In NeurIPS, 2022.
[41] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data. In ICCV, 2021.
[42] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2023.
[43] Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. RestoreFormer: High-quality blind face restoration from undegraded key-value pairs. In CVPR, 2022.
[44] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. GAN prior embedded network for blind face restoration in the wild. In CVPR, 2021.
[45] Zongsheng Yue and Chen Change Loy. DiffFace: Blind face restoration with diffused error contraction. arXiv preprint arXiv:2212.06512, 2022.
[46] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[47] Yang Zhao, Yu-Chuan Su, Chun-Te Chu, Yandong Li, Marius Renn, Yukun Zhu, Changyou Chen, and Xuhui Jia. Rethinking deep face restoration. In CVPR, 2022.
[48] Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022.
In this supplementary material, we provide additional discussions and results. In Sec. A, we present additional implementation details including the inference requirements, choice of hyperparameters involved in the inference process, and discussions on the pre-trained restorer for blind face restoration. In Sec. B, we provide more results on various tasks, i.e., blind face restoration, old photo restoration, reference-based restoration, face colorization, and inpainting. Sec. C and Sec. D discuss the limitations and potential negative societal impacts of our work, respectively.
A.1 Inference Requirements
The pre-trained diffusion model we employ is a  $512 \times 512$  denoising network trained on the FFHQ dataset [18] provided by [45]. The inference process is carried out on NVIDIA RTX A5000 GPU.
A.2 Inference Hyperparameters
During the inference process, there involves hyperparameters belonging to three categories. (1) Sampling Parameters: The parameters in the sampling process (e.g., gradient scale  $s$ ). (2) Partial Guidance Parameters: Additional parameters introduced by our partial guidance, which are mainly relative weights for properties involved in a certain task (e.g.,  $\alpha$  that controls the relative importance between the structure and color guidance in face colorization). (3) Optional Parameters: Parameters for optional quality enhancement (e.g., the range for multiple gradient steps to take place  $[S_{start}, S_{end}]$ ). While it is principally flexible to tune the hyperparameters case by case, we provide a set of default parameter choices for each homogeneous task in Table 4.
Table 4: Default hyperparameter settings in our experiments.
Task	Sampling	Unmasked Region	Partial Guidance	N=2	Optional
snorm	s	Lightness	Color Statistics	Smooth Semantics	Identity Reference	N=3	Perceptual Loss	GAN Loss
Restoration	0.1	-	-	-	Lres	-	T~0.5T	T~0.7T	1e-2	1e-2
Colorization	✓	0.01	-	Ll	0.01Lc	-	-	-	-	-	-
Inpainting	✓	0.01	Linpaint	-	-	-	-	-	-	-	-
Ref-Based Restoration	0.1	-	-	-	Lres	10sim(vx0,vr)	T~0.5T	T~0.7T	1e-2	1e-2
A.3 Restorer Design
Network Structure. In the blind face restoration task, given an input low-quality (LQ) image  $y_0$ , we adopt a pre-trained face restoration model  $f$  to predict smooth semantics as partial guidance. In this work, we employ the  $\times 1$  generator of Real-ESRGAN [41] as our restoration backbone. The network follows the basic structure of SRResNet [22], with RRDB being its basic blocks. In a  $\times 1$  generator, the input image is first downsampled 4 times by a pixel unshuffling [31] layer before any convolution operations. In our work, we deal with  $512 \times 512$  input/output pairs, which means that most computation is done only in a  $128 \times 128$  resolution scale. To employ it as the restorer  $f$ , we modify some of its settings. Empirically we find that adding  $x_t$  and  $t$  as the input alongside  $y_0$  can enhance the sample quality in terms of sharpness. Consequently, the input to  $f$  is a concatenation of  $y_0$ ,  $x_t$ , and  $t$ , with  $t$  embedded with the sinusoidal timestep embeddings [36].
Training Details.  $f$  is implemented with the PyTorch framework and trained using four NVIDIA Tesla V100 GPUs at 200K iterations. We train  $f$  with the FFHQ [18] and CelebA-HQ [17] datasets and form training pairs by synthesizing LQ images  $I_{l}$  from their HQ counterparts  $I_{h}$ , following a common pipeline with a second-order degradation model [41, 23, 40, 44]. Since our goal is to obtain smooth semantics without hallucinating unnecessary high-frequency details, it is sufficient to optimize the model  $f$  solely with the MSE loss.
Model Analysis. To investigate the most effective restorer for blind face restoration, we compare the sample quality with restorer being  $f(y_0)$  and  $f(y_0,x_t,t)$ , respectively. Here,  $f(y_0,x_t,t)$  is the one trained by ourselves as discussed above, and  $f(y_0)$  is SwinIR [25] from Difface [45], which is also
Table 5: Quantitative comparison on the synthetic CelebA-Test dataset on inpainting and colorization tasks. Red indicates the best performance.
Task
Metric	Inpainting	Colorization
FID↓	NIQE↓	MUSIQ-KonIQ↑	FID↓	NIQE↓	MUSIQ-AVA↑
DDNM [42]	137.57	5.35	59.38	146.66	5.11	4.07
CodeFormer [48]	120.93	4.22	72.48	126.91	4.43	4.91
Ours	115.99	3.65	73.20	119.31	4.71	5.23
trained with MSE loss only. As shown in Fig. 9, when all the other inference settings are the same, we find that the sample quality with restorer  $f(y_0, x_t, t)$  is higher in terms of sharpness compared with that of  $f(y_0)$ . One may sacrifice a certain degree of sharpness to achieve higher inference speed by substituting the restorer with  $f(y_0)$ , whose output is constant throughout  $T$  timesteps.
Output with Restorer  $f(y_0)$
Figure 9: Visual comparison of the restoration outputs with different restorers  $f$  in blind restoration. We observe that including  $x_{t}$  and  $t$  as the input to  $f$  enhances the sharpness of the restored images.
Output with Restorer  $f(y_0,x_t,t)$
Input
Output with Restorer  $f(y_0,x_t,t)$
B.1 More Results on Blind Face Restoration
In this section, we provide more qualitative comparisons with state-of-the-art methods, including (1) task-specific CNN/Transformer-based restoration methods: PULSE [26], GFP-GAN [40], and CodeFormer [48] and (2) diffusion-prior-based methods: GDP [9], DDNM [42] and DiffFace [45].
Figure 10: Comparison on Blind Face Restoration. Input faces are corrupted by real-world degradations. Our PGDiff produces high-quality faces with faithful details. (Zoom in for best view)
B.2 More Results on Old Photo Restoration
We provide more visual results of old photo restoration on challenging cases both with and without scratches, as shown in Fig. 11. The test images come from both the CelebChild-Test dataset [40] and the Internet. We compare our method with GFP-GAN (v1) [40] and DDNM [42]. Our method demonstrates an obvious advantage in sample quality, especially in terms of vibrant colors, fine details, and sharpness.
Figure 11: Comparison on Old Photo Restoration on Challenging Cases. Our PGDiff is able to produce high-quality restored outputs with natural color and complete faces.
B.3 More Results on Reference-Based Restoration
We provide more visual results on the reference-based restoration in Fig. 12, which is our exploratory extension based on blind face restoration. Test images come from the CelebRef-HQ dataset [24], which contains 1,005 entities and each person has 3 to 21 high-quality images. With identity loss added, we observe that our method is able to produce personal characteristics similar to those of the ground truth.
Figure 12: Comparison on Reference-Based Face Restoration. Our PGDiff produces personal characteristics which are hard to recover without reference.
B.4 More Results on Face Inpainting
In this section, we provide quantitative and more qualitative comparisons with state-of-the-art methods in Fig. 13, including (1) task-specific methods: GPEN [44] and CodeFormer [48] and (2) diffusion-prior-based methods: GDP [9] and DDNM [42]. As shown in Fig. 4 and Fig. 13, since DDNM and CodeFormer are relatively more competitive than others, we make quantitative comparisons of our methods against them.