SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can observe from Fig. 13 that our method is able to recover challenging structures such as glasses. Moreover, diverse and photo-realism outputs can be obtained by setting different random seeds. As for quantitative comparisons in Table 5, we believe that the ability to produce diverse results is also crucial in this task, and the evaluation should not be constrained to the similarity to the ground truth. Thus, we opt to employ NR metrics including FID, NIQE, and MUSIQ instead of FR metrics. Regarding the MUSIQ metric, it has been trained on different datasets featuring various purposes [21]. For inpainting, we employ MUSIQ-KonIQ which focuses on quality assessment. Our method is able to achieve the highest score across all metrics.
Figure 13: Comparison on Face Inpainting on Challenging Cases. Our PGDiff produces natural outputs with pleasant details coherent to the unmasked regions. Moreover, different random seeds give various contents of high quality.
B.5 More Results on Face Colorization
In this section, we provide quantitative and more qualitative comparisons with state-of-the-art methods in Fig. 14, including (1) task-specific methods: CodeFormer [48] and (2) diffusion-prior-based methods: GDP [9] and DDNM [42]. As shown in Fig. 3 and Fig. 14, since DDNM and CodeFormer are relatively more competitive than others, we make quantitative comparisons of our methods against them.
We can observe from Fig. 14 that our method produces more vibrant colors and finer details than DDNM and CodeFormer. Moreover, our method demonstrates a desirable diversity by guiding with various color statistics. As for quantitative comparisons in Table 5, we believe that the ability to produce diverse results is also crucial in this task, and the evaluation should not be constrained to the similarity to the ground truth. Thus, we opt to employ NR metrics including FID, NIQE, and MUSIQ instead of FR metrics. Regarding the MUSIQ metric, it has been trained on different datasets featuring various purposes [21]. For colorization, we choose MUSIQ-AVA that puts more emphasis on aesthetic assessment. Although CodeFormer has a better score in NIQE, it clearly alters the input identity (see Fig. 14) and requires training a separate model for each task. On the contrary, our method requires only a pre-trained diffusion model for both inpainting and colorization, and is able to achieve best scores across almost all metrics.
Figure 14: Comparison on Face Colorization. Our PGDiff produces diverse colorized outputs with various color statistics given as guidance.
As our PGDiff is based on a pre-trained diffusion model, our performance largely depends on the capability of the model in use. In addition, since a face-specific diffusion model is adopted in this work, our method is applicable only to faces in its current form. Nevertheless, this problem can be resolved by adopting stronger models trained for generic objects. For example, as shown in Fig. 15, we employ an unconditional  $256 \times 256$  diffusion model trained on the ImageNet dataset [30] provided by [7], and achieve promising results on inpainting and colorization. Further exploration on natural scene restoration will be left as our future work.
Input
Figure 15: Extension on natural images for the inpainting and colorization tasks. By employing an unconditional  $256 \times 256$  diffusion model trained on the ImageNet dataset [30] provided by [7], our method achieves promising results.
Ours
(a) Inpainting
Ours
(b) Colorization
This work focuses on restoring images corrupted by various forms of degradations. On the one hand, our method is capable of enhancing the quality of images and improving user experiences. On the other hand, our method could generate inaccurate outputs, especially when the input is heavily corrupted. This could potentially lead to deceptive information, such as incorrect identity recognition. In addition, similar to other restoration algorithms, our method could be used by malicious users for data falsification. We advise the public to use our method with care.