Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model
Authors: Keda TAO
Jinjin Gu
Yulun Zhang
Xiucheng Wang
Nan Cheng
Venue: ICLR 2025 Spotlight (2025)
Keywords: Face image restoration
diffusion model
Primary area: generative models

Abstract: We introduce a novel Multi-modal Guided Real-World Face Restoration (MGFR) technique designed to improve the quality of facial image restoration from low-quality inputs. Leveraging a blend of attribute text prompts, high-quality reference images, and identity information, MGFR can mitigate the generation of false facial attributes and identities often associated with generative face restoration methods. By incorporating a dual-control adapter and a two-stage training strategy, our method effectively utilizes multi-modal prior information for targeted restoration tasks. We also present the Reface-HQ dataset, comprising over 21,000 high-resolution facial images across 4800 identities, to address the need for reference face training images. Our approach achieves superior visual quality in restoring facial details under severe degradation and allows for controlled restoration processes, enhancing the accuracy of identity preservation and attribute correction. Including negative quality samples and attribute prompts in the training further refines the model's ability to generate detailed and perceptually accurate images.

Published as a conference paper at ICLR 2025
OVERCOMING FALSE ILLUSIONS IN REAL-WORLD FACE RESTORATION WITH MULTI-MODAL GUIDED DIFFUSION MODEL
Keda Tao', Jinjin Gu²; Yulun Zhang', Xiucheng Wang', Nan Chengl:4+
Xidian University. 'The University of Sydney, 'Shanghai Jiao Tong University State Key Laboratory of Integrated Services Networks
KD www.Tutttttooooooooo com, jinjin in. w.n.guussse suuesydne..edee edu. zulzhanges jtt .du. cn xcwang 1 utuu.. xidian. edu. en, dr nan chengSleee.org
[s] Referance-basad Face Restoration Results
CodeFormer
Referesce Image
LR
DR2
Codeformer
DIMBIR
wio Reforance Image
Rehrence
Oura si Relerence Image
Ground Truth
Controlling
Prompts
Prompit... woman
Prompt: waman
old
No Prompt
Prompl: who his big eyes...
Prompe young woman who has...
Prompt: who has eyeglasses...
Rose
Figure 1: The proposed MGFR model demonstrates exceptional ability in restoring low-quality face images, yielding outstanding visual effects with the addition of reference images, particularly in situations of extreme degradation, shown (a). Furthermore,
model capable of carget-specifts restoration (b), directed by facial attribute prompts. This encompasses defining facial age characteristics (Case 1), adjusting the restoration pracess hased on attribute prompts (Case 2), and executing precise modifications to facial elements (Case 3). w/o Reference Image means the results of our model without introducing reference image.
ABSTRACT
We introduce novel Multi-modal Guided Real-Worlh Face Restoration (MGFR) technique designed improve the quality of facial image restoration from low- quality inputs. Leveraging a blend of attribute text prompts, high-quality reference images, and identity information, MGFR can mitigate the generation of false facial attributes and identities often associated with generative face restoration methods. By incorporating dual-control adapter and two-stage training strategy, our method effectively utilizes multi-modal prior information for targeted restoration tasks. We also present the Reface-HQ dataset, comprising over 21,000 high- resolution facial images across 4800 identities, to address the need for reference face training images. Our approach achieves superior visual quality in restoring facial details under severe degradation and allows for controlled restoration pro-
"Corresponding authors

---

Published as a conference paper at ICLR 2025
cesses, enhancing the accuracy of identity preservation and attribute correction. Including negative quality samples and attribute prompts in the training further refines the model's ' ability to generate detailed and perceptually accurate images.
INTRODUCTION
Weal-World Face Restoration (FR) aims to reconstruct high-resolution. high-quality (HQ) facial im- ages from their degraded, low-resolution observations. Recent works, leveraging powerful generative priors and diffusion models, have achieved significant progress (Menon et al.) [2020] Yang et al.) 2021b; Wang et al. 202 Lin et al. 2023 Wang et al. (023b), particularly in addressing severely degraded facial images. However, the information contained in the low-quality (LQ) inputs is limited.
FR inevitably introduces the illusion of generation, producing results with different facial attributes or even different identities from the target image. For example, in Figure T (a) and Figure 2 we cannot effectively predict the eye colour and skin characteristics of the person in the LQ input, resulting in the output results even the quality can be improved having an apparent perceptual distance from the target image. Many applications find this unacceptable, as humans can readily identify these flaws. Achieving optimal facial image recovery requires effectively tackling false hallucinations.
Practically, we find that for the restoration of specific face images, we can obtain a lot of prior information. For example, we may know this person's 's various attributes and identity, and there may even be other clear images of this person in the photo album. Suppose we can use this information as additional guidance to guide the restoration. In that case, we can alleviate the impact of false illusions on key issues, thus helping to generate facial details that better suit our needs. For example, in F (a), when we provide an additional key description of gender and age, we can correct the illusion. In Figure and Figure (b), additional high-quality images are used as reference, and the details of the eyes and skin texture can be accurately generated. What is even more gratifying is that this kind of prior information can be widely obtained, making this problem of application significant.
This work proposes a method called Multi-modal Guided Real-World Face Restoration (MGFR). We aim to use multiple control methods to consider diverse multi-modal prior information in FR to restore face images in a targeted manner. Specifically, MGFR uses attribute text prompts, HQ reference images, and identity information as priors for collaborative guidance during restoration. We designed a dual-control adapter with a two-stage training strategy to balance the complex multi- and multi- source prior information. This dual controller is compatible with pre-trained generative diffusion models (Rombach et al. (022) and prioritizes restoration tasks while incorporating additional multi- modal guidance. In addition, image samples containing over 4800 identities and 21000 high-resolution facial images. Based on the FFHQ (Karras al., 2019a) and the proposed Reface-HQ datasets, we develop a high-quality synthetic dataset for model training enriched with attribute text prompts. Furthermore, we adopt a counterintuitive strategy to integrate negative-quality samples with negative-quality prompts and attribte prompts into training to enhance perceptual quality and detail generation.
collect the Reface-HQ dataset address the scarcity of reference
The proposed MGFR model shows exemplary performance in the FR task, achieving superior visual quality in facial details, especially under severe degradation conditions. MGFR can take high- resolution reference image as prior information and restore important details based on the reference image that cannot be displayed in the LQ input. The identity information provided by the reference image will also be considered in FR to ensure that the restoration does not change the identity characteristies. In addition, MGFR can also provide certain degree of control over the restoration process through attribute text prompts, significantly enhancing the feasibility of FR. As shown in Figure (b)) textual prompts fulfil dual function: they significantly reduce facial attribute illusions, such as "iig eyes" or "old", and also guide the restoration of specific facial features, such as "wearing glasses" and "young".
RELATED WORKS
Real-Worke Face Image Restoration Real-world face restoration (FR) concentrates on the chal-
lenging task of reconstructing HQ face images from LQ inputs. These LQ inputs often blemished by various forms of quality degradation, such as low-resolution (Chen et al.) 2018 Dong et al. 2014 Lim al. 2017), blur (Kupyn et al. 2018 Shen al.] 2018), noise (Zhang et al. 201 1)
2

---

Published as a conference paper at ICLR 2025
S
S
WORKin, no Figure 2: Motivation, In conditions of severe degradation, the loss of facial identity information becomes profoundly pronounced without a reference image. During the face restoration process, distortions of facial attributes, including gender and age. commonly encountered. Appropriate attribute prompts offer additional reference points and exert control in the recovery process.
(a)
LQ
No Prompt
young.
GT
(b) LQ
w/o Reference
w/ Reference
GT
and JPEG compression artifacts (Dong et al. 2015), etc. FR heavily relies on facial priors, such as facial landmarks (Chen et al. 20188) parsing maps (Chen et al., 20118 (02T), and facial component
Rombach et al. 2022 Gu et al. 2020 [hhen et al. (2020) have also emerged fundamental elements in providing vibrant textures and
heatmaps Yu et al. (018). Generative priors (Krraa et al. 2020
details in FR (Menon et al. 2020 Hu et al. 2023 Zhu et al. 2022). Advanced techniques like GPEN
(Yang et al. a) GFP-GAN Waag al. 202 and ELENN (Chan et al.) 2021) are recognized for more effectively incorporating these priors within encoder-decoder structures. There are also works that considerably reduce the uncertainty commonly associated with generative priors (Gu et al.) 2022, Zhou et al., 2022. Wang et al. 2022), which are trained on discrete feature codebooks for high-quality facial images. Recently, diffusion models like DiffBIR (Lin et al., 2023) have revitalized interest in this area, leveraging the generative power of pre-trained LDM as a prior. DR2 (Wang et al.) 2023b) also contributes by transforming input images into noisy states and then denoising them to capture the essential semantic information.
Reference- Face Image Restoration Reference- face restoration utilizes HQ images of the same identity as references. This concept was first introduced in (Li et al. 2018a). To address discrepancies in poses and expressions, GWAInet (Dogan et al. 2019) and the later work of Li et al. (Li et al. 2020b; 2018b) focused on more effectively directing deformations or choosing the optimal reference image for reconstruction. MyStyle (Nitzan et al. (022) adopts unique approach by refining StyleGAN (Karras et al. 2019a) with numerous reference images based on personal appearance. DMDNet (Li et al. 2023) employs a dictionary constructed from diverse, high-quality facial images to rehabilitate degraded images using its high-quality components. In the MGFR framework, incorporating a single reference image is vital for tailoring the restoration to individual faces. Unlike conventional methods, MGFR does not require strict alignment constraints on expressions or postures.
Multi-modal Guided Generation Diffusion models have shown significant effectiveness in a broad range of image processing tasks. Current methods (Chen et al. 2023b; Zhang al.. 2023) Yu et al.
2024: Chen et al., 2023a) employ pre-trained text-to-image diffusion models (Rombach et al., 2022) for image processing, demonstrating the potential of language as comprehensive input for image reconstruction tasks. Concurrently, approaches like ControlNet (Zhang et al. 2023), T21-adapter (Mou et al. 2023), and ControlNet-XS (Zavadski et al. 2023) have further developed the integration of more intricate condition controls within the text-to-image framework, facilitating more precise and tailored image generation. Nevertheless, the field of FR, particularly in the utilization of natural language prompts, continues to be an area of untapped potential.
METHODOLOGY
The proposed MGFR method is able to take face attribute text prompts, reference images, and identity information as input to alleviate illusions and improve visual effects. MGFR involves controlling information from multiple modalities. However, we found that if the model is directly trained to process control information from multiple sources and of different importance, is not easy to utilize all the information effectively. The model may ignore the more complex information to utilize. This causes some of the controls to fail and reduces image quality. In our method, text prompts are the most complex control information because they involve understanding text and the correspondence between text and face attributes. Therefore, we divide the training into two stages. In the first stage, the training focuses on the basic text-guided restoration model (Section B.1.. This allows the model to restore high-que gigh-quality images and understand facial attributes. Then, we introduce other control information this basis. The second stage introduces the HQ reference image and face identity
3

---

Published as a conference paper at ICLR 2025
Dual-contral control Adapter
RLF
Identity Extractor
RedBlock
Znr/
ROA
Image
3
Compleey
Refarenca input
LG Input
Zic
VAE Encoder
VAE Desoder
Big
Z
Protpe: the divcrpption of's man big noxe,
Pre-trinned Stable Diflusion
Frazon
Figure 3: Framework Overview, This figure illustrates the overall workflow of the proposed MGFR model.
information as the control means (Section 3.2 To improve the image effect further, Section 3.3
describes negative examples and the adopted prompting strategy.
3.1 STAGE ONE: TEXT-GUIDED FACE RESTORATION
This stage trains the face image restoration model that accepts text prompt as input, as shown in Figure We use the pre-trained Stable Diffusion (SD) (Rombach al. 2022) model our generative prior and train an additional adapter to extend it to the face restoration applications. The pre-trained SD generative prior has the ability to understand face image attributes and text and generate high- quality face images. In this stage, model restores the image according to the condition {y. h. where y represents the degraded LQ image, and Co constitutes the facial attribute prompts describing the face attributes. We first the CLIP text encoder (Radford et al., 2021 to calculate the text embedding e, = CC.P(c.). The LQ input y is also mapped to a latent representation 2ig using the VAE encoder in SD (Rombach al. 2022). We then perform diffusion generation on this latent representation. In the framework of SD, the model uses UNet (Ronneberger et al., 2015) denoising model Eg (ze, f, e,) to perform the diffusion generation process, where is the time stamp in diffusion model and is the intermediate results at time t. Based on the ControlNet (Zhang et al. 2023) framework, we use an external adapter that takes the LQ input y and text prompts embedding e, as input to provide guidance for the fixed UNet Eg. We call this adapter the LQ Control Adapter (LCA). Specifically, the UNet model contains the encoder, intermediate blocks, and the decoder. The decoder receives features from the encoder and fuses them at each corresponding scale. The LCA contains the same encoder and intermediate blocks as the UNet model. The feature output each scale in LCA is integrated with the corresponding scale of the UNet decoder to achieve the effect of output control. However, we found that simply using the above ControlNet framework has key limitation the lack of information exchange from the UNet encoder to the LCA. This gap means that the LCA is unaware of the processes that are performed in the UNet encoder, thereby limiting its ability to add the feature output of each
generate effective control features. In order to solve this problem, scale in the UNet encoder to the corresponding scale in the LCA. The LQ controller part of Figure 5 illustrates this operation. In this way, the capability of the LCA greatly enhanced, so better visual effects and control results can be achieved.
3.2 STAGE Two: MULTI-MODALLGUIDER FACE RESTORATION
After the first stage of training, the model can already reconstruct high-quality images from the LQ inputs guided by text prompts. Next, we further enrich the guidance and introduce high-quality reference images and face identity information as additional control means based on the first-stage model. We design a new Dual-Control Adapter (DCA), as shown Figure 3 DCA, we introduce Reference Control Adapter (RCA) specifically for reference image processing. RCA has the same architecture as LCA, and its role to extract related and useful information from reference images and identity information and provide additional details to LCA. The input of RCA includes HQ reference face image r containing the same identity as the LQ input and its identity information embedding ey. For the reference image r, we first model for feature extraction to obtain 2ref- Next, we fuse the LQ latent representation 2ig with Fref using reference and IQQ feature fusion module (RLF). This module allows RCA to identify
the VAE encoder consistent with the SD
4

---

Published as a conference paper at ICLR 2025
Bref
LCA
RCA
Frezse
LQ-blook
+
Ref.block
LG Input
+
LO-klock
Ref Nef-bllck
Crooder
LQ.block
Ref-block
of sanredk kig be beers...
Discader
LQ-block
Ref-block
Figure 4: The model architecture employed during the initial training stage is discussed. In the article, "Ours w/o Reference Image' refers to the outcome of the model trained following this stage.
Figure 5: Dual-control Adapter. I.Q-llncks from the LQ control adapter (LCA), and Ref.-blocks are from the reference control adapter (RCA). @ repre- sents the element-wiss add operation.
the high-frequent details missing in the LQ input and perform targeted information extraction for restoration guidance. For identity information embedding. we calculate & Proj(Arcface(r)) where A Areface(r) is the face recognition model Arcface (Deng et al.) 2019) to extract the identity feature from the reference image r. We align to the space that RCA can handle through a trainable linear projection layer Proj(-). Due to the function of the RCA extracting information from the reference image according to the LQ input, the RCA requires the information of the LCA branch as input. At the same time, RCA needs to provide the extracted information back to LCA in reverse. Therefore, we designed a dual-way interaction mechanism for RCA and LCA, as shown in Figure In this design, RCA runs in parallel with LCA. At each scale, the LQ block in LCA first processes the fused information of both two branches and then hands the intermediate features to RCA. RCA performs feature extraction and processing based on these intermediate results and reference conditions and finally uses the same operation to apply the processing results to the next layer of LCA processing. Finally, the output of each scale of LCA is applied to the corresponding position of the UNet decoder. RCA directly affects LCA and, therefore, also affects the calculation of UNet. Since then, we have had a dual-control adapter that can accept multiple control inputs.
3.3 NEGATIVE SAMPLES AND PROMPT
Classifier-Free Guidance (CFG) (Ho & Salimans 2022) introduces novel control mechanism utilizing negative prompts to delineate unwanted content for the model. This feature can be leveraged inhibit the generation of low-quality images by the model and enhance the precision of facial detail reconstruction. Throughout the inference phase, at each step of diffusion, three distinct predictions are generated: one employing the positive prompt pos, another using the negative quality prompt ngg, and third via the negative attribute prompt na (the negation sentence described by pos). We combine the results generated from these different prompts to form the final output:
2- 2,09 + Ang x (2f-1 2(-1) 1) + Ana X (zf-0
where Ana and And is the hyperparameters, 2,20 Eo(2,, Rigs Zref, L, nq), 2/-1 = Eo( Zig. žref, 1, na). pos represents a standard description of a facial attribute. ng is the negative words of quality, e.g., "oil painting, cartoon, blur, dirty, messy, low quality, deformation, low resolution, over-smooth", na is used for a negative description of a facial attribute, implying complete negation. Accurate prediction in both positive and negative directions is essential for the CFG technique. The lack of negative-quality samples and prompts in our training might cause the model to misinterpret negative prompts, leading to artifacts. To resolve this, we generated 16K images with negative-qualic prompts using the original SD generative model and included these low-quality images in our training to enable the model to learn the concept of negative quality. (a) shows an example of the negative quality sample and prompt.
Eo(2e, 24g. Zref t,, pos). 2
4
EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Datasets. Our two-stage training method requires different datasets for training. For the first stage, we mainly train the model's ability to restore HQ images and process text prompts. Therefore, we need HQ images with text annotations for training. We synthesize training image pairs using the FFHQ dataset (Karras et al. 2019b). FFHQ contains 70,000 high-resolution face images, and resize these images to 512 x 512 for training. In the second stage, in addition to requiring HQ face images to create training image pairs, we also need to assign HQ reference images with consistent identities but different details to each image. Although there are some datasets proposed
5

---

Published as a conference paper at ICLR 2025
DR2
Image
Image
CodeFomerr
DIMBIR
Image
Reference
Ours Raference Image
Ground Truth
Figure 6: The MGFR model demonstrates a remarkable capacity for restoring LQ images. Upon integrating the reference image, particularly in instances of severe degradation, the model significantly enhances the restoration of facial details and overall image quality.
Figure 7: In the qualita- tive comparison of real- world low-quality (LQ) images, MGFR demon- strates success in recover- ing facial details without false illusion and preserv- ing identity from. Please zoom in for a better view.
Input
DifIBI
CodeFormer
w/o Reference
Ours
Reference
for reference face restoration (Liu et al., 2015) Yi al. 2014). the resolution and quality these datasets cannot meet the current requirements. In this work, we collect a new dataset for referenced face restoration called Reface-HQ. Reface-HH HQ contains 21,500 high-quality and diverse images of over 4800 identities. Additional details of Reface-HO can be found in the Appendix A To synthesize LQ images, we follow the degradation model and setting used in (Wang al.) al.] 2023b). Our test data also involves multiple sources, including CelebA-Test (Liu et al. 2015), Reface- Test, and real-world LQ images collected from Internet. Specifically, CelebA- Test contains 3,000 testing images from the CelebA-HO dataset. Reface- Test contains 1,300 images of 280 identities split from the proposed
Reface-Ht -efacce-HQuae dataset. The LQ images for testing are synthesized within the same degradation range as the training setting.
Attribute Prompt. Text prompts are impor- tant for us control face attributes and improve quality. In our method, a total of three types of prompts are introduced. Two attribute prompts describe the face attributes, and the last one de- scribes the negative quality of the image. For attribute prompts. pos contains positive descrip- tions of face attributes, while na describes at- tributes that do not exist in this image to provide negative prompts of attributes. To obtain these descriptions, we first use pre-trained face at- tribute detector (He et al. 2017) to extract the presence of each attribute in the face. We con- sidered 28 different attributes in this work. For attributes that have high confidence to exist, we add them to the pos positive attributes. For the remaining attributes with low confidence, classify
HugilienQQully Sumplee
Sacii Arvilaut Pronect
"A old man who has Big nose, hail...
rusoitoon cve Brocth
(N]
Facial Atribuut Detector LLM
Figure 9: Training Data Composition. Initially, neg- ative quality samples are incorporated into the training to enhance the clarity and quality of the restored image. Furthermore, large language models, coupled with fa- cial attribute classifier, are employed extract attribute texts for integration into the training.

---

Published as a conference paper at ICLR 2025
Input
No Prompe
Add: 'ipptick"
Inpur
No Prompt
Add: "earrings"
Input
No Prompt
Add: 'big eyes
Figure 8: MGFR demonstrates capability of face image restoration facilitated by text prompts. It possesses the capacity to artificially modulate specific aspects of the restoration outcomes, such as determining the presence of accessories like lipstick or glasses (Cases & 2). and orchestrating the restoration process alignment with facial attributes (Case 3).
Real-SR(x4)
Real-SR(xx5
Real-SR(x16)
Mathod
LPIIS
MarIQA
ClipiQA
MUSIQ
1.PIPS
ManIQA
CligIQA
MUSIQ
I.PISS L
ManIQA
ClipQQ
MUSIQ
PSERGAN
0.2928
0.5927
0,5702
23.39
0.3315
0.6015
0.5956
73.08
0.3788
0.5739
0.6274
71.76
GPEN
0.2838
0.6596
0.643
69.25
0.5217
0.6754
0.6299)
68.63
0.3831
0.5618
0.5897
66.61
VQFR
0.2951
0.2875
0.2490
62.95
0.5277
04163
0.236
61.92
0.3761
0.6513
0.2148
CodeFormer
0.2927
0.5803
0.5179
0.3191
0.5970
0.6235
75.09
0.3821
DR2
0.5749
0.441
63.43
0.3580
0.5346
0.4494
0.3796
0.5160
0.5035
DITBRR
0.2611
0.6068
0.7681
74.27
0.3017
0.6088
0.7439
73.87
0.1238
0.536]
0.7161
67.41
BFRIiisiion
0.3258
0.5477
0.5572
45.32
0.3739
0.4404
0.5298
42.84
0,3735
0.4204
0,5098
43.16
Dars w/f Referrncc
0.2925
0.6854
0.8344
76.22
0.5227
0.6776
0,8083
75.94
0,3760
0.6729
0,7944
75.76
Table 1: Quantitative Comparison in CelebA-Test. Results in red and blue signify the highest and second highest, respectively. The indicates metries whereby lower values constitute improved outcomes, with higher values preferred for all other metrics.
them as no negative attributes. At this time, these attributes are still separate words. We use large language model to organize the separated words into natural language to facilitate the understanding of the CLIP text encoder. Thus, each face image is associated with two attribute prompts detailing existing and non-exixe con-xxittnng attributes. For the negative quality prompt, ng involves "low quality, low resolution, over-smoothed and distorted images", as shown in Figure 9 (a). See Appendix B for comprehensive details on the training and inference procedures involving attribute prompts.
Implementation. The training involved fine-tuning based Stable Diffusion v2.1 (Rombach et al.
2022), with the control adapter structure adhering to (Zhang et al. (023). The Adam optimizer (Kingma & Ba (014)) was employed, featuring a learning rate of e 5. The initial training stage spanned 15 days, while the subsequent stage lasted 5 days, utilizing 4 Nvidia A A100 GPUs with a
batch size of 4. For testing purposes, the hyperparameters were set as T 500, Ana = 0.5 and Xng 0.5.
Metries. For quantitative comparison, followed by many previous works (Lin et al. 2023 Yu et al. 2024), the selected metrics include full-reference metrics PSNR, SSIM, and LPIPS (Zhang et al.
2018), as well as non-reference metrics ManIQA (Yang et al. 2022), ClipIQA (Wang al. 2023a). and MUSIQ (Ke et al. 2021). Furthermore, the Areface identity distance (Deng et al. 2019 (ID) is utilized to assess the similarity of identity information.
4.2
COMPARISONS WITH OF METHODS
MGFR is qualitatively and quantitatively compared with state-of-the-art methods in FR. Notably, the model trained in the initial stage, which is a restoration model solely guided by attribute prompts, already achieves superior visual results. The non-reference prior-based methods selected include
PSFRGAN (Chen et al.) 2021). GPEN (Yang et al. 2021a), VQFR (Gu et al. (022)) CodeFormer (Zhou al. 2022). DR2 (Wang al. 2023b), BlRfuusionn (Chen et al.)
(024) and DifIBIR (Lin et al.. 2023). along with reference basor-based methods ASFFNet (Li et al. 2020b) and DMDNet (Li
ct al. 2023). Particularly, to ensure contrastive fairness during the inference stage, the description text, containing restricted attributes, is obtained through low-resolution processing. In practical applications, however, users can freely set attribute prompts, enabling more precise and comprehensive guidance. For qualitative results comparing ASFFNet and DMDNet, please refer to the Appendix.
Comparison on Synthetic Degradations. Firstly, a quantitative comparison of our model without reference images on the synthetically degraded CelebA-Test dataset is conducted without reference image guidance. According to Table [ our model achieves the best results on all non-reference metrics, indicative of the superior image quality of the results. Due to space limitation, the values of SSIM and PSNR of Table are shown in Appendix [C.]] Additionally, the method's ' limitations on full-reference metrics are also noted. This phenomenon, preliminarily demonstrated by experiments
7

---

Published as a conference paper at ICLR 2025
"young, woman blonde
Input
High Blur
Face Swap
Reference
Input
No Prompt
'No beard, young"
Figure 10: Attribute prompts that manifestly contra- Figure 11: Face Swapping: MGFR is capable of lever- vene low-rcsolution inputs prove ineffectual and result aging the reference map to alter the comprehensive in distortions and artifacts within the restored image.
components of the face.
The results of study. We domly select face images under multiple test datasets for user study. Our model achieves excellent recovery quality, which can further enhanced with high-quality reference image and identity information guidance.
Figure 12:
Proportian Seletion
60
Propationn Scleciio
60
.00
20-
20
VOFR CodeFFrme: DiMBIR MGBER (a) User Study on compared methods
win Reference (b) User study on using Referenee image
w/ Reference
in Yu et al. 2024 Jinjin et al. 2020), necessitates a reevaluation of the reference value of indicators like PSNR, SSIM, LPIPS, and the proposal more effective methods to assess advanced FR methods, particularly as quality improves. More qualitative comparison results of our model can be found in Appendix Subsequently, Figure 6 and Figure (a) present a qualitative comparison of the MGFR method applied to the Reface- Test dataset. Even in cases of severe degradation, successfully produces highly superior facial details guided by the reference image. In addition, we provide comparison between model without reference images and MGFR, with particular focus on FR tasks involving features like double eyelids, pupil color, and finer facial details, such as wrinkles and moles, which cannot be accurately captured without reference image guidance. This further demonstrates the superiority of utilizing reference image guidance in the FR task. Finally, Table offers quantitative comparison results, indicating that our method significantly surpasses other state-of- of-the-art methods in perceived quality.
C
method
We also conducted user study with a total of 40 participants, comparing MGFR other approaches. Participants were asked to select the best quality recovery result from these test techniques for each pair of comparison images, or no reference image was provided, the result that came closest the Ground Truth. Section 4.2 presents the results, which demonstrate that method outperforms the state-of-the-art methods terms of recovery quality. Furthermore, the reconstruction effect can be further enhanced by using the reference image guidance.
Comparison on Real-world Degradations. Additionally, our method was tested on real-world LQ images, which involved col- lecting degraded face images of publicly available images along- side reference images. The qual- itative results, presented in Fig- ure [2] demonstrate that the re- sulting images possess realistic visual effects with minimal facial illusions. More results are pre-
Degradarion
Method
I.PISS
ManIQA
ClipQQ
ASFFNar
23.43
0.5685
0,6215
71.66
0.7051
DMDNer
72.31
DR2
23,58
0.6581
0.2532
0.5340
0.5956
69.00
0.7957
CodeFormer
23.88
0.6905
0.2912
0.4929
0.5823
74.80
0.6379
DifBBR
24.12
0.6717
0.2785
0.5547
0.7474
73.73
0.6379
MGBFR(CUrs)
23.10
0.6248
0.2688
0.6355
03147
75,51
0.5166
ASETNet
21.70
0.6472
0.3013
0.5503
0,6221
71.57
0.931
DMDNNr
22.37
0.3179
0.479
0.4727
67.27
0.9270
0.3400
0.5547
71.30
DiMBIR
21.51
0.5939
0.3956
0.4937
0.7114
67.42
0.5876
MGBFR(Ours)
21.75
0.6033
0.2989
0.6534
0.5046
75.06
0.7401
Table Quantitative Comparison Reface- Test. Quantitative com-
parison of guided recovery results based on reference images. DR2, CodeFormer, and DitIBRR do not use reference images.
sented in Appendix C
4.3 CONTROLLING RESTORATION WITH ATTRIBUTES PROMPTS
Our method facilitates targeted image restoration guided by attribute prompts. As illustrated Fig- ure the comparison between the first and second cases reveals that the integration of supplementary attribute prompts facilitates the manipulation of subtle facial attributes absent the original image. This includes the addition of glasses, earrings, and accessories. In scenarios of severe degradation,
8

---

Published as a conference paper at ICLR 2025
Figure 13: Negative quality prompts engender restoration outcomes characterized by high definition, whereas negative at- tribute prompts yield results with enhanced detail.
Use Nogative Quality Prompt
Use Negative Attribtee Prompt
Input
w/o Negative Prompt
GT
Real-SRoc))
SSIM
PSNK
LPIPS
ManKQA
ClipQQ
MUSIQ
Procnpts
LPIPS
SSIM
PSNR
MasIQA
ClipIQA
MUSIQ
Link
22.17
pos
who Link-LR
0.6699
23.86
0,2873
0.6152
0,6645
65,70
25.05
0.4782
0.2568
49.97
Ours)
0.6248
23.10
0.2088
0.0335
0.8147
75.51
0.6844
24.43
0.6421
0.7008
0.2930
0.0066
23:27
0.6656
0.7999
75.34
Table 3: Ablation study of additional information ex- change in the MGBFR model. "wo Link-LR' means that the upward flow of information from LCA to RCA is removed.
0.2202
0.3227
0.611
0.5904
24.49
22:34
0.6437
0.6776
0.2029
0.8083
73.11
75.91
Table 4: Ablation study of attribute prompts and nega- tive prompts
exemplified by the third case, reconstructing facial features like eyes poses a significant challenge without external prompts. More results are shown Appendix E.1
However, it is imperative to acknowledge that attribute prompts do not invariably yield efficacy. As demonstrated in Figure IO] our model is capable of control tuning through attribute prompts. However, prompts that starkly contradict LQ inputs. like "blonde hair", are found to be ineffective. This ensures the model's adherence the provided LQ inputs. Furthermore, as illustrated by the input of an LQ male face in Figure 10 when the input attribute is "Female", the model subtly incorporates the attribute label "Female" into the image. This is achieved through modifications like the addition of an earring and the removal of the beard while remaining faithful to the LQ input. Such modifications further underscore the efficacy of attribute text in guiding the restoration process. This outcome is not unexpected. On the contrary, excessive control capability might lead to reduction in the restoration effectiveness, countering the fundamental intent of image reconstruction efforts and therehy demonstrating the robustness of the proposed method.
4.4 ABLATION STUDY
Attribute Prompt and Negative Samples. Figure 13 displays qualitative results under various settings, aligning with the strategies outlined in Section 333 It can be seen that incorporating negative quality prompt significantly enhances restoration quality. while the addition of a negative attribute prompt yields images with finer details. Quantitative results under various settings are also presented T Adding either positive attribute prompts or negative quality prompts is observed to improve the perceived quality of the images significantly. Utilizing both types of prompts in conjunction with the negative attribute prompt achieves the most favourable perceived effect. The impact of hyperparameters
the results was also explored, revealing that settings of Ana 0.5 and Ang = 0.5 yield the best perceptual outcomes, balancing sharpness and definition. Please refer to Appendix G for detailed qualitative results in different hyperparameters.
Face Swapping. MGFR can facilitate face-swapping operations involving the processing of highly degraded LQQ images to obscure identities, shown in Figure [1] Face images and identity informa- tion from diffcrent identities are utilized as guides to achieve face swapping and identity replacement. Besides proposing an additional application for the model, this experiment further illustrates the method's 's efficacy in utilizing identity information and reference images for guidance.
Additional Information Exchange. Unlike (Zhang et al., 2023 we have integrated an additional information flow exchange link (Link-UL) from the U-net model to LCA, and a bidirectional information flow link (Link-LR) between LCA and RCA. Table dispplas the quantitative test results for the presence of the aforementioned information exchange links. Notably, "w/o Link-UI.' refers to results obtained with a single information flow from LCA to U-net model. It evident that additional information flow exchanges result in improved perceived quality.
Areface Identity Embedding. Our model is able to leverage identity information to guide the image restoration process, aiming mitigate the deficit in facial identity information substantially. As demonstrated in Figure 15 our model employs the identity encoding formulated by the identity information extractor to mitigate the deficiency of facial identity information in the restored image. After losing arcface identity embedding, the recovered results still have high quality but there is a false illusion of face identity information.

---

Published as a conference paper at ICLR 2025
(a)
LQ
Ours
GT
Reference
(b)
LQ
Ours
GT
Reference
Figure 14: Face restoration results with reference images with different expressions and poses. When there are differences in pose and expression between the reference image and the low-quality input image, our model still achieve good restoration effect without the generation of artifacts.
1QQ
Ours wlo
Ours w/ ID)
GT
Ours w/o
Oars w/ ID
GT
Figure 15: Ablation experiments about areface identity embedding (ID). The additional identity embedding can greatly reduce the false illusion identity information the recovery results.
Different expressions and poses reference images. In previous studies, reference image-based face restoration has been widely explored, but its efficacy is constrained by the need for strict alignment between the reference image and the low-quality (LQ) input. As shown in Figure 21 the recovery results of ASFFNet and MDMNet exhibit severe distortions when the reference image and the LQ input are slightly misaligned. However, our method completely resolves this issue, as imposes strict requirements the expression, pose, other variations of the reference image. As demonstrated in Figure [4| even when there are discrepancies between the reference image and the LQ input, such as face orientation, labeling, makeup, or pose, our model consistently achieves high-qualit, restoration without any artifacts.
4.5 LIMITATIONS AND DISCUSSION
Although this represents an initial foray into attribute text-guided face image restoration, the flexibility of its text input is somewhat constrained due to the nature of the training samples. The model struggles to fully comprehend freely composed attribute description sentences, tending instead to rely on attribute labels embedded within fixed template text prompts, which limits its applicability in broader contexts. Furthermore, when users input attribute labels unseen during training, these do not effectively guide the recovery process. These limitations highlight the importance and necessity of utilizing high-quality data on larger scale.
Moreover, believe that excessive flexibility in controlling facial attributes poses risk of model misuse. Our model demonstrates robust recovery even in the absence attribute hints. Therefore, for the first time, we incorporate additional multimodal information to enhance recovery quality without increasing task complexity. The reference image serves solely to enhance facial details and does not alter the recovered results. In summary, task: ensuring that the output remains faithful to the low-quality input. Furthermore, leveraging multiple reference images enables more precise detail recovery, while the exploration of utilizing multiple low low-quality reference images for collaborative guidance is left for future research.
remain committed to the objective of the recovery
CONCLUSION
We introduce MGFR as pioneering method in real-world face restoration at the cutting edge of face image restoration technology, capable using multi-modal information for guidance to achieve realistic visual effects. Simultaneously, MGFR extends the possibilities of face restoration by controlling text prompts with attributes. The proposed Reface-HO dataset also offers significant potential for advancing the development of face restoration models based reference images. As the first technological advancements.
face image restoration model, MGFR establishes a new benchmark for future
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGE
This work was supported by the National Key Research and Development Program of China (2044YFB200500).
REFERENCES
Fadi Boutros, Jonas Henry Grebe, Arjan Kuijper, and Naser Damer. Idiff-face: Synthetic- face recognition through fizzy identity-onditioned diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp. 67-74. IEEE, 2018.
Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. Glean: Generative latent bank for large-factor image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14245- 14254, 2021.
Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan- Yee K Wong. Progressive semantic-aware style transformation for blind face restoration. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 11896-11905, 2021.
Xiaoxu Chen, Jingfan Tan, Tao Wang, Kaihao Zhang, Wenhan Luo, and Xiaochun Cao. Towards real-world blind face restoration with generative diffusion prior. IEEE Transactions Circuits and Systems for Video Technology, 2024.
Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. Fsree:: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2492-2501 2018
Zheng Chen, Yulun Zhang, Jinjin Gu, Xin Yuan, Linghe Kong, Guihai Chen, and Xiaokang Yang. Image super-resolution with text prompt diffusion. arXiv preprint arXiv22311.1428,, 2023a.
Zheng Chen, Yulun Zhang, Ding Liu, Bin Xia, Jinjin Gu, Linghe Kong, and Xin Yuan. Hierarchical integration diffusion model for realistic image deblurring. arXiv preprint arXiv:2305. 12966, 2023b.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Berk Dogan, Shuhang Gu, and Radu Timofte. Exemplar guided face image super-resolution without facial landmarks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, pp. 184-199. Springer, 2014.
Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. Compression artifacts reduction by deep convolutional network. In Proceedings of the IEEE international conference computer vision, pp. 576-584, 2015.
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3012-3021, 2020.
Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng- Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder. European Conference Computer Vision, pp. 126-143. Springer, 2022.
Keke He, Zhanxiong Wang, Yanwei Fu, Rui Feng, Yu-Gang Jiang, and Xiangyang Xue. Adaptively weighted multi-task deep network for person attribute classification. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1636-1644, 2017.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Yujie Hu, Yinhuai Wang, and Jian Zhang. Dear-gan: Degradation-aware face restoration with gan prior. IEEE Transactions on Circuits and Systems for Video Technology. 2023.
Gu Jinjin, Cai Haoming, Chen Haoyu, Ye Xiaoxing, Jimmy Ren, and Dong Chao. Pipal: a large- scale image quality assessment dataset for perceptual image restoration. In Computer Visi 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 633-651. Springer, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019a.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4401 4410, 2019b.
Tero Karras, Samuli Laine, Miika A Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. Proceedings of the IEEE/CVF conference on computer vision and paitern recognition, pp. 8110-8119, 2020.
Junjie Ke, Qifei Wang, Yilin Wang. Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5148-5157, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 14/2.6980 2014.
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jifí Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8183-8192, 2018
Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang. Learning warped guidance for blind face restoration. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018a
Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang. Learning warped guidance for blind face restoration. In Proceedingy of the European conference on computer vision (ECCV), pp. 272-289, 2018b.
Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang. Blind face restoration via deep multi-scale component dictionaries. In European conference on computer vision, pp. 395-415. Springer, 2020a.
Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo. En- hanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020b.
Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo. Enhanced blind face restoration with multi- exeeple images and adaptive spatial feature fusion. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2706-2715 2020c.
Xiaoming Li, Shiguang Zhang, Shangchen Zhou, Lei Zhang, and Wangmeng Zuo. Learning dual memory dictionaries for blind face restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, (5(5))5904-5911 2022.
Xiaoming Li, Shiguang Zhang, Shangchen Zhou, Lei Zhang, and Wangmeng Zuo. Learning dual memory dictionaries for blind face restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5904-591 2023. doi: 10.
12

---

Published as a conference paper at ICLR 2025
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136-144, 2017.
Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint ar08v 15070, 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp.
2015.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 2437-2445, 2020.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. arXiv preprint 230.:::3300000553,, 2023.
Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-or. Mystyle: personalized generative prior, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9243-9252, 2020.
Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang. Deep semantic face deblurring, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8260-8269, 2018.
Zi Teng, Xiaosheng Yu, and Chengdong Wu. Blind face restoration via multi-prion collaboration and adaptive feature fusion. Frontiers in Neurorobotics, 16:797231, 2022.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2555-2563, 2023a.
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pp. 9168-9178 2021a.
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9168-9178, 2021b.
Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjie Zheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. Dr2: Diffusion-based robust degradation remover for blind face restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1704-1713, 2023b.
13

---

Published as a conference paper at ICLR 2025
Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. Restoreformer: High- quality blind face restoration from undegraded key-value pairs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17512-17521, 2022.
Sidi Yang. Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujuu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
1191-1200, 2022.
Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 672-681 2021a.
Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 672-681, 2021b.
Dong Yi, Zhen Lei, Shengeai Liao, and Stan Li. Learning face representation from scratch. arXiv preprint 1eXiv::411.79900 2014.
Fanghua Yu, Jinjin Gu, Zheyuan Li. Jinfan Hu, Xiangtao Kong, Xintao Wang, Jingwen Hc, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint arXiv:2401 130.2227....67,,, 2024.
Xin Yu, Basura Fernando, Bemard Ghanem, Fatih Porikli, and Richard Hartley. Face super-resolution guided by facial component heatmaps. In Proceedings of the European conference on computer vision (ECCV), pp. 217-233, 2018.
Denis Zavadski, Johann-Friedriae Feiden, and Carsten Rother. Controlnet-xs: Designing an effi- cient and effective architecture for controlling text-to-image diffusion models. arXiv preprint arXiv:2312.06573. 2023.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions on image processing, 26(7): 3152-3155, 2017.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 186-595,77 2018.
Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. Advances in Neural Information Processing Systems, 35:30599-30611, 2022.
Feida Zhu, Junwei Zhu, Wenqing Chu, Xinyi Zhang, Xiaozhong Ji, Chengjie Wang, and Ying Tai. Blind face restoration via integrating face shape and generative priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7662-7671, 2022.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A REFACE-HQ DATASET
The bulk of prior reference-based face restoration methodologies commonly fo- cus on training and testing with 256 256 images. This is primarily due to the limitations of existing datasets such CelebA Liu et al. (2015), VggFace2 Cao et al. (2018), and Fecbace Yi al. (2014), which offer reference images mainly for face or attribute recognition but do not include high-quality images suitable
Dataset
Number of
Image
Size
Synthesized
CASIA-WebFace
10575
494414
256
x
Celoba
10.177
202599
178x218
x
IDif Face
128x
VggFace2 HQ
1200
24000
512x512
X
CelebRef-HQ
1000
10000
512x512
X
Reface-Ho
4800
21500
512x512
x
Table 5: Datasets Comparison.
for training at higher resolutions, like 512 x 512 or 1024 x 1024, thereby limiting their practical applications. Additionally, high-definition datasets recently introduced, such as CelebRef-HQ/Li et al. (2022) and VggFace2-HQQ face challenges in maximizing the potential of models due to their limited number images and narrow range of identities.
To address this challenge, we have created a new real-world dataset named Reface-HO, as shown in Figure 16 The Reface-HO dataset encompasses high-definition facial images of celebrities, which have been collected from the Internet. Initially, images with inadequate resolution (minimum 512), low quality and outliers lacking facial features were eliminated. Subsequently, identities represented by fewer than two images were excluded, and face image crop alignment was conducted. Each identity was also manually inspected to eliminate discrepancies age and makeup. Additionally, to enhance the fairness and inclusiveness of the algorithm, includes samples from all races and skin colors. We strive to ensure the diversity of the training data, thereby minimizing algorithmic bias and discrimination, and further enhancing the algorithm's 's faimness and inclusiveness. In summary. Reface-HQ encompasses 4,800 identities, totaling 21500 images with a resolution of 512, subsequently partitioned into three segments: 4520 identities for the training set and 280 for the Reface-Test. The comparison of datasets available for special face Boutros
meticulously review the dataset to ensure
restoration tasks is shown in Table 5 IDiff-Face Boutros et al. (2023 is composite dataset with an indefinite number of images.
Idently
Mendiy
Figure 16: Demonstration of the Refacc-HQ datasct.
A. ABLATION EXPERIMENT
For diffusion models and adapter structures, both the quality and quantity of training data are critical factors affecting the model's final performance. Table presents the quantitative comparison results of our proposed model under various training data volumes. is evident that the model's 's performance significantly decreases with only 10,000 training data samples.
Real-SR(x8)
SSIM
PSNR
LPIPS
ManIQA ClipIQA
MUSIQ
10K Training Samples
0.6254
23.46
0.2535
0.6388 0.6535
0.7424
72.23
20K Training Samples
0.6248
23.10
0.2688
0.8147
75.51
Table Ablation Experiment about training.
15

---

Published as a conference paper at ICLR 2025
B ATTRIBUTE PROMPT
This section provides a supplementary note on the attribute text prompts utilized in MGFR. For the training data, attribute la- bels are first extracted from the FFHQ Reface-HO dataset's s face images using a facial attribute classifier. The 28 types of tributes included are listed in Table while labels with binomial characteristics (such as Male and Female, beard and beard, etc.) are not repetitively shown. Regarding the classification threshold, attributes with a probability greater than 0.6 are considered positive, those with probability less than 0.4 as negative, and the rest as uncertain in describing facial features. LLM is utilized to embed the attribute labels into a descriptive sentence template, thereby enhancing the model's understanding. To augment the model's grasp of negative attribute descriptions, two sentences of prompt text are provided for each image, illustrated in Figure 17 Both descriptions offer a positive portrayal of the face, with Prompt B specifically focusing on the negative attributes.
Black Hair
Blond Hair
Blurry
Brown Hair
Eyeglasses
Hair
Heavy Makeup
Mouth Slightly Open
Mustache
Big Eyes
No Beard
Receding Hairline
Sidehums
Smiling
Sinight Hair
Wearing Earrings Big Nose
Weuring Hat
Male
Wearing Necklace
Wearing Lipstick
Young
Wavy
Big Lips
Bangy
Table Face Attribute.
we apply positive attribute prompts (pos), negative quality prompts (nq)) and negative attribute prompts (na) in each iteration. For example, in restoring LQ image, it assumed contain attributes like 'smiling, man, black hair, eyeglasses,' the corresponding text for image restoration can be generated as follows:
In the inference stage, following the approach detailed in Section 3.3
Positive Prompt: A high quality, high resolution, realistic and extremely detailed image in the description of a smiling man who has black hair and eyeglasses.
Negative Attribute Prompt: A high quality, high resolution, realistic and extremely detailed image not in the description of a smiling man who has black hair and eyeglasses.
Negative Quality Prompt: low quality, low resolution, over smooth and deformation
image.
The underlying premise prevent our model from generating low-quality images and images with mismatched facial attributes. Extensive experiments demonstrate the effectiveness of our proposed attribute prompts.
Prompt high quality, high resolution, realistic and extremely detailed image the description smiling young woman who has big nose. she wearing lipstick and she no beard.
Prompt high quality, high resolution, realistic and extremely detailed image not the description of old man who has bangs, big lips, black hair, blond hair, brownhair, oyeglasses, gray hair, straight hair, wavy hair.
Figure 17: Attribute prompts composition in training.
16

---

Published as a conference paper at ICLR 2025
C MORE QUALITATIVE COMPARISONS FOR OUR MODEL WITHOUT REFERENCE IMAGES.
This section presents qualitative comparisons experimen- tal results of our model without reference images, focusing on attribute text-guided face recovery. Importantly, for fair comparison, the attribute during the inference phase are derived from the LQ input, which means the model's maximum potential is not fully realized. We assert that in practical scenarios, users will be able to supply more precise attribute text for enhanced recovery guidance. Al- though, our model demonstrates the most superior visual effects and details when compared to other state-of-the- of-thh--rtttttttt methods.
WebPhoto-Test
ManIQA
ClipQQA
MUSIQ
DR2 DiIBIR
0.4868
0.6184
64.36
0.4068
0.6858
55.73
Ours
0.5901
0.8397
72.52
Table Quantitative comparison with other diffusion model- based methods on real-world degradations in WebPhoto-Tees
Figure 18 and Figure 19 display the qualitative comparison results of our model against other advanced models under conditions of mild and moderate degradation of LQ input, respectively. It is evident that the previous methods exhibit severe facial illusion, whereas our model attains the best visual outcomes. Notably, as shown in Figure 20 and Figure[21] to recover severely degraded input images with high quality and fidelity. Finally, Figure [2 shows the effect of restoration real-world LQ inputs and Table [ presents the quantitative comparison results between our model and the principal comparison methods using real-world LQ inputs from the WebPhoto-Tess dataset.
model demonstrates a remarkable ability
O
DR2 VQER Figure 18: More qualitative comparisons for text-guided baseline model synthetic dataset under mild degradation in CelebA-Tes dataset. Zoom for best view.
LR
PSERGAN
GPEN
DifTBIR
CodeFormer Ours w/o Ref. GT
Ours wlo Ref. Figure 19: More qualitative comparisons for text-guided baseline model on synthetic dataset under moderate degradation in ColebA-Te -Test dataset. Zoom for best view.
LR
GPEN
DR2
VQFR
DimBI
CodeFormer
GT
LR
GPEN
DR2
VQFR
DiMBIR
CodeFormer Ours wlo Ref.
GT
Figure 20: More qualitative comparisons for our text-guided baseline model synthetic dataset under severe degradation in CelebA-Test dataset. Zoom for best view.
17

---

Published as a conference paper at ICLR 2025
DR2 CodeFormer Figure 21: More qualitative comparisons for our text- text-guided baseline model on synthetic dataset under severe degradation in Reface-Tes. dataset. Zoom in for best view.
ASFFNet
MDMNet
DiAIBI
Ours Ref.
GT
Ours w/o Reference Figure 22: More qualitative comparisons for our model without reference images real world images. 7oom in for best view.
TR
DR2
CodeFormer
DifTAIR
C.1 ADDITIONAL SUPPLEMENT TO TABLE
Due to space limitation, for the quantitative comparison results of our model without reference images. Table does not show the numerical values of PSNR and SSIM, and supplement them in T Table 10 and Table
Real-SR (4) ManIQA
Method
LPIPS
PSNR
SSIM
ClipIQA
MUSIQ
PSERGAN
0.2938
23.72
0.6522
0,5927
0.5702
73.39
GPEN
0.2828
24.78
0.7056
0.6596
0.6430
69.25
VQFR
0.2951
23.81
0.6878
0.2875
0.2490
62.95
CodeFormer
0.2927
24.56
0.6809
0,5803
0.5179
75.47
DR2
0.3264
23.74
0.6827
0,5749
0.4441
63.43
DiMBIR
0.2611
24.49
0.6778
0,6068
0.7681
74.27
BFRffusion
0.3258
24.87
0.7014
0.5477
0.5572
45.32
MGFR(Ours)
0.2925
23.25
0.6104
0.6854
0.8244
76.22
Table 9: Quantitative Comparison in CelebA-Test. Results in red and blue signify the highest and second highest, respectively. The indicates metries whereby lower values constitute improved outcomes, with higher values preferred for all other metries.
D TRAINING AND INFERENCE CONSUMING ANALYSIS
In terms of training consumption, the proposed MGFR model employs two-stage training strategy for the dual-control adapter, leading to a moderate increase in training cost. However, for the diffusion- based image restoration model, this additional training time remains relatively short. Nonetheless, this investment is justified, as the proposed MGFR model demonstrates excellent recovery performance. Additionally, the dual-control adapter's specialized design enables superior restoration results depend on the guidance multimodal information. Our experiments (Figure 32) confirm that employing single traditional adapter structure for multimodal input often results redundancy between the
18

---

Published as a conference paper at ICLR 2025
Real-SR( x8)
Method
LPIPS
PSNR
SSIM
ManIQA
ClipiQA
MUSIQ
PSERGAN
0.3315
22.85
0.6232
0.6015
0.5956
73.08
GPEN
0.3217
23.88
0.6822
0.6754
0.6299
68.63
VQFR
0.3277
23.16
0.6683
0.4163
0.2363
61.92
CodeFormer
0.3193
21.81
0.5799
0.5970
0.6235
75.09
DR2
0.3580
23.26
0.6725
0.5246
0.4494
59.46
DirBIR
0.3017
23.47
0.6442
0.6058
0.7439
73.87
BFRffusion
0.3739
23.72
0.6718
0.4404
0,5298
42.84
MGFR(Ours)
0.3227
22.34
0,5904
0.6776
0,8083
75.94
Table 10: Quantitative Comparison in CelebA-Test Results in red and blue signify the highest and second
highest, respectively. The 4 indicates metrics whereby lower values constitute improved outcomes, with higher values preferred for all other metrics.
Real-SR(x
Method
LPIPS
PSNR
SSIM
ManIQA
ClipIQA
MUSIQ
PSERGAN
0.3788
21.27
0.5899
0.5739
0.6274
71.76
GPEN
0.3831
22.22
0.6541
0.6618
0.5897
66.61
VQFR
0.3761
21.72
0.6413
0,6513
0.2148
60.49
CodeFormer
0.3821
21.19
0.5717
0.5803
0.5877
70.85
DR2
0.3796
21.06
0.6225
0.5160
0.5035
70.31
DifBBIR
0.4238
21.21
0.5654
0.5361
0.7164
67.41
BFRffusion
0.3735
23.67
0.6716
0.4204
0.5098
43.16
MGFR(Ours)
0.3760
20.54
0,5452
0.6729
0,7944
75.76
Table 11: Quantitative Comparison in CelebA-Test Results in red and blue signify the highest and second highest, respectively. The indicates metrics whereby lower values constitute improved outcomes, with higher values preferred for all other metrics.
reference image and the low-quality input, as well as color inconsistencies in the recovered output. This observation, however, does not preclude further exploration in this area. Our future work will focus on employing specially designed single-transformer adapter to replace the dual-control adapter, aiming reduce the model's complexity.
In addition, Table 12 presents the average inference time, memory consumption, parameter count, and FLOPs statisties. Notably, the CFG strategy compatible with all LDM-based recovery models. Results are presented separately to reflect the CFG strategy's 's influence during inference. Without the CFG strategy, our model exhibits slightly higher time and memory consumption compared to DiffBIR Lin et al. (2023). DR2 Wang et al. (2023b) and BFRfusion Chen et al. (2024) exhibit faster inference times; however, their recovery performance is suboptimal. Furthermore, SUPIR's large-scale model design results in significantly higher training and testing costs compared to other methods, including MGFR. However, MGFR outperforms SUPIR Yu et al. (2024) on the face image restoration task while incurring lower costs (see Appendix M. It should be noted that efficiency is not the primary focus of this work. Moreover, we believe that the development of efficient lightweight models is grounded in the superior performance of large-scale models. Our future iterations will explore model compression techniques, such as quantization and pruning, to enhance inference speed and reduce parameter counts while maintaining MGFR's superior performance the face recovery task.
Method
Average time (s)
Memory consuming (M)
#Params (M)
FLOPs (G)
DiffBIR
11260
1716.7
897.5
DR2
2.6
3144
93.56
388.94
BFRífusion
3.2
8338
1197.4
784.5
SUPIR
47.6
54318
3870.0
11950
Ours(w/o CFG)
6.9
15351
2029.3
890.5
Ours (w/ CFG)
12.5
15351
2029.3
2672.4
Table 12: Inference consuming compared with other diffusion model-based methods.
19

---

Published as a conference paper at ICLR 2025
Input
Prompt
old, woman'
Input
No Prompe
eevellasee'
Input
No Prompt
'narrings'
Input
No Prompt
Ripsick
Ingut
No Prompt
'mouth slighdl open'
Inpul
No Prompp
"big eyes'
Figure 23: Influences of attribute prompts.
E CONTROLLING WITH ATTRIBUTES PROMPTS
E CONTROLLING RESTORATION
Our model facilitates guidance through user-defined attribute prompts during testing. Figure 23 exemplifies this with demonstration of attribute prompt-controlled recovery. Notably, 'No Prompt refers to the initial prompt input, 'A high quality, high resolution, realistic, and extremely detailed image.' As illustrates, users can employ prompts like 'old' to define the approximate age in the restored image, or "yyellasess' and "earrings' to add accessories to the image. Furthermore, users can provide additional attribute prompts to refine unsatisfactory results. For instance, 'lipstick' can be used to add lipstick, 'mouth slightly open' to adjust the mouth's 's appearance. More significantly. severe illusions, particularly in the eye area, are common in previous methods due to insufficient information in LQ inputs. This observation underscores the importance of attribute prompts in method, as using 'big eyes' leads to more realistic eye effects. Therefore, we posit that attribute text holds potential versatile tool for controlling face recovery.
E.2 SENSITIVITY ANALYSIS
Moreover, as depicted in Figure 24 case and case 2, with increasing levels of degradation, the model's 's reliance on attribute prompts for control becomes more apparent, leading to greater flexibility. This observation, a logical experimental outcome, confirms the model's fidelity to LQ inputs during recovery. Specifically, attribute prompts that starkly contradict the LQ input do not influence the effect, which aligns with our expectations. The primary function of attribute labels, we contend, is to facilitate more efficient and effective image restoration, rather than focus image editing and control. This is intrinsic to the core objective real-world face restoration. our method, all attribute labels listed in Table 7 including 'black hair', "brown hair', and others, do possess the ability to control recovery but rather aid the model in interpreting the IQQ input. These insights robustly underscore the effectiveness of approach.
F USER STUDY
Currently, the relevance and efficacy of metrics such as PSNR, SSIM, and LPIPS require evaluation. In this study, User study conducted quality. The study concentrated on two primary questions: (1) How does our model without reference images perform in terms of restoring image quality versus reducing facial illusions compared to
alternative metric for assessing image restoration
20

---

Published as a conference paper at ICLR 2025
Input
Prompt
'no plasser'
Input 8)
Prompt
glasser"
Input
No Prompt
'ilue eyer'
Figure 24: We investigate the following options for attribute prompt control. First of all, the model becomes increasingly dependent on attribute prompt as input deterioration inereases (case & case 2). Second, the input attribute tag does not have a control role it not present in Table (case 3).
previous methods? (2) Does the addition of reference image and identity information in guiding restoration result in images that are closer the Ground Truth compared the model without reference images? Two sets of questionnaires were prepared, and the study was conducted with 50 participants. Participants were presented with random, anonymous options for their selection. For question (1), our model was compared with DiffBIR Lin et al. (023)), VQFR Gu et al. (2022), and CodeFormer Zhou et al. (2022), focusing on selecting images with better quality and fewer hallucinations, without providing Ground Truth images. This comparison involved 50 sets of images. For question (2), a self-comparison approach was adopted. Specifically, ground truth images were provided, and participants
asked choose between restoration results with and without reference images, assessing them based on their proximity and realism to the ground truth. In this experiment, 50 pairs of synthetically degraded images were compared.
Subsequently, the first part of the user study, focusing on the improvement of our model in terms of image quality and the reduction of facial illusion, is discussed. The results and detailed information of this study segment are presented in Figure 25 and Figure 26] It was observed that the majority of the 50 participants favored our model for superior image quality and minimal facial illusions. Reflecting on the recovery results of the advanced method CodeFormer, illustrated in Appendix C is noted that while CodeFormer achieves relatively good quality in restored images, considerable facial illusions persist, particularly around the mouth and eyes. In contrast, our method consistently produces high-quality, realistic facial images with minimal facial illusion. These findings underscore the our model's 's capability to reduce illusion and enhance image quality through negative prompts. Specifically, supported by the diffusion model and LR control adapter, our model is adept at generating realistic high-quality restorations influenced by negative prompts, and it effectively minimizes facial illusions by utilizing an optimal amount of attribute prompts. The synergy of these elements paves the way for further exploration in MGFR.
is noteworthy that our two-part User study also corresponds to the two-stage development process of the MGFR model. For the second part, the first User study has demonstrated the superior performance of our model, as shown the figure. Participants generally agreed that adding a guide to the reference image would further achieve superior visual effects.
DiffBIR
fnliuting speddine chrow the Image lightr tmage qurilty renang optinex aroway comporiion.
6%
CodeFormer 19%
Imagge quallty tn follniing J. tadal detal textuee
3. Imagee relies
Plaae soloct the mage with higher image quality zmong four ostions
VQFR. 6%
MGBFR 69%
Figure 25: Results and question details of user study.
G ABLATION STUDY FOR NEGATIVE PROMPT
For negative prompts, we introduce two hyperparameters, Ana and Ang- However, we find that the changes of the values tend to have the same effect the restored images. Thus, keep
21

---

Published as a conference paper at ICLR 2025
cloose bost
Plese sdkt ding to the is bellving attari, in orde of profenanse:
Montty (the secalinss fit of the reference imoge should parson)
Image
Referance,
w/ Reference
hont mature umder the netemnnce
quiliy
seaarch, wrriilie other
Please compatisnn
88%
Reforence
Figure 26: Results and question details of study.
Ana 0.5 and Ang 0.5 during reasoning. Here we will represent the values of Ana and Xng with to show the qualitative comparison results under different hyperparameters in Figure
H IMPACT STATEMENTS
Controlled generation technology, as a pivotal innovation in the field of diffusion models, exerts significant impact across multiple sectors of society. In the creative industries, it enables artists and designers to realize complex visions with unprecedented precision and flexibility, fostering innovation in digital art, design, and multimedia content creation. In commercial applications, controlled generation technology enhances marketing strategies by offering more targeted and dynamic advertising visuals, effectively engaging consumers. Additionally, its influence extends to education and training, where it can revolutionize teaching methods and materials, especially in deeeellyde disciplines, by generating customized educational content and simulations.
The work presented in this paper aims to advance machine learning and computer vision. This method provide the public with better face processing effects and has greater social value. However, the technique is designed to process facial information, inevitably involving facial attributes such as race and privacy risks. We are accompanied by captions. We are also wary of potentially discriminatory attribute descriptions in our research. Our method also provides control over face restoration, which reduces the possibility of our method outputting harmful information.
of these risks. Our research uses publicly available data and images
I MORE QUALITATIVE COMPARISONS FOR MGFR MODEL
Figure 28 displlas the qualitative comparison results between the proposed MGFR model and other advanced methods. The "w/o Reference Image" represents the restoration results of our model after initial training. The use of the negative intuition strategy and attribute prompts significantly reduces the false illusions in face images and substantially enhances overall quality. Subsequently, the inclusion of additional multi-modal information, such as reference images and identity information, achieve superior visual effects.
J SCALABILITY OF MGFR FOR REAL-WORLDD VIDEO FACE RESTORATION
The proposed MGFR framework shows significant potential for real-world video-based face recovery tasks. Unlike single-image restoration, video restoration poses the unique challenge of ensuring temporal consistency. To address this, our method leverages the recovered output of the previous frame as a reference for the current frame. This approach aligns seamlessly with our model architecture. which integrates high-quality continuous frame references into guided restoration. Additionally, as our model does not require strict alignment between the reference and low-quality inputs, effectively handles natural variations in pose and expression commonly found in consecutive video frames, surpassing previous reference- based face restoration models. By leveraging temporal dependencies between frames, the proposed method ensures identity consistency and high-quality recovery in video sequences. Future work could enhance this approach by integrating explicit temporal models or constraints, such optical flow guidance, to better handle motion artifacts and dynamic variations
22

---

Published as a conference paper at ICLR 2025
Blury
Sharp
Input
A= 0.9
Figure 27: Influence of hyperparameters on recovery effect in CFG. The smaller À does not get clear recovery result and the huge À causes the recovered image to be over sharp.
video data. Unlike single-image restoration based on reference images, video data offers more diverse and abundant training samples, which believe will further unlock the potential of our proposed model. This will be a key focus of our future work.
w/o Reference Image
Refurence
Raterence Image
wio Reforence
Reference
Ground Truth
who Reforence Image
Referonoe
Image
GodeForner
wis Reference Image
Ground Truth
Figure 28: More qualitative comparisons for MGFR with reference image and guidance on synthetic dataset in Reface- Test dataset. Zoom in for best view.
23

---

Published as a conference paper at ICLR 2025
K
MODEL STABILITY
Sced 2982
Seed 78
11281
424
GT
Figure 29: Model Stability Analysis. The recovery results MGFR remain consistent across different random seeds, climinating the need for selection among multiple input outcomes.
L BRIEF OVERVIEW OF EVALUATION METRICS
For quantitative comparison, the selected image quality evaluation metrics include full-reference metrics PSNR, SSIM, and LPIPS Zhang et al. (2018). Yu al. (2024): Jinji al. (2020) experiment initially confirmed that image restoration quality improves, the reference utility of metrics such as PSNR, SSIM, and LPIPS needs be re-evaluated, necessitating the selection more effective evaluation indicators. Therefore, introduce three non-reference metrics ManIQA Yang et al. (2022), ClipIQA Wang et al.] (2023a), and MUSIQ Ke et al. (2021) in this work.
A summary of each evaluation metric is provided below.
SSIM is a key metric for assessing image restoration quality, measuring the similarity between the restored and original images based on brightness, contrast, and structural information. It has been widely used in previous face image restoration tasks Lin et al.
(2023);
Wang et al. (2023b 202 la): Yang et al.
(021a); Zhou et al. (2022); ; Gu al. (2022): 202 Li et al. (2023) 2020b a): Wang et al.
Yu et al. (2024) Chan et al. [2021) ); Chen et al.
2021b):
Li et al. (2020c) : Teng et al. (2022).
PSNR is metric derived from the mean square error (MSE), calculated as the logarithmic ratio of the maximum possible pixel value the error. The results are expressed in decibels (dB), where higher values signify better image quality. It has been widely used in previous
face image restoration tasks [in et al. (2023); Wang et al. (2023b; 2021a) Yang et al.
(2021a); Zhou et al. (2022); Gu et al. (2022); Dogan et al. (2019): Yu ct al. 2024); Chan
ct al. 2021): Chen et al. (2021); Li et al. (2023 2020b a): Wang et al. (2021b) Li al.
(2020e) Teng et al. (l.22222))
LPIPS quantifies image differences by extracting features from deep neural networks and measuring the distances between these features. This metric better captures perceptual changes image details and textures. Previous studies have emphasized image similarity
metrics aligned with human visual perception Lin et al. (2023) Wang al. (2023b 2021a lay;
Yang et al. (2021a): Zhou et al. (2022); Gu et al. (2022); Yu et al. (2024) Chan et al, (202
Chen et al. (2021 Li et al. (2023 2020b
Wang et al. (202 et al. (2020c):
Teng
et al. (2022).
ManIQA maps images into a low-dimensional manifold space and analyzes their feature distribution and location assess image quality. This approach demonstrates high correlation with perceived quality, and its effectiveness has been validated in Yu et al. (024).
MUSIQ implements a multi-scale feature extraction mechanism designed to capture the quality characteristics of images across varying resolutions and perceptual scales for effective image quality evaluation, and its effectiveness has been validated in Yu et al. (2024).
ClipIQA leverages the robust vision-languag priors embedded within the CLIP model. The focus on enhancing the capability to evaluate both quality perception (seeing) and abstract perception (feeling) of visual content. This approach's effectiveness has been demonstrated in Yu et al. (2024).
M QUALITATIVE COMPARISON WITH SUPIR
24

---

Published as a conference paper at ICLR 2025
I.R
SUPIR
Ours wlo Ref.
GT
Figure Qualitative comparisons with SUPIR et al. 2024) for our text-guided baseline model on synthetic dataset under moderate degradation CelebA- Test dataset. Zoom in for the best view.
LR
SUPIR
Ours Reference
Reference
Figure 31: Qualitative comparisons with SUPIR Yu et al. (2024) for MGFR on synthetic dataset under moderate degradation in Reface-Tes Test dataset. Zoom in for best view.
Ours DCA Figure 32: Ablation experiments comparing the reception of mnulti-modal information using a single control adapter (SCA) versus dual control adapter (DCA) revealed that SCA led to reduced recovery performance and increased chromatic aberration.
Ours SCA
GT
Reference
25


Reviews:
[{"review_id": "BjAwQr579P", "reviewer": "Reviewer_1C8W", "summary": "The paper propose to utilize diffusion prior, as long as multi-modality input to perform real world face restoration. The method can receive text or reference image for face restoration. \n\nThey also present a dataset that contains identity-image pair. \n\nThere is both quantitative  and qualitative metric, which demonstrate the effectiveness of this method.", "questions": "1. The design of the Dual control Adapter is a little but tricky. Can one just replace it by a transformer?\n    2. There can be different  result for every sample. How diverse is the model? And how to decide which sample to use?\n    3. Will be dataset be realised?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation that using multi-modality input to assist face restoration is nice. As shown in the paper, there could be ambiguity during restoration, and the additional input could be helpful. \n2 .  The use of diffusion prior for this work also make sense, and it models a probability given the input. \n3. The dataset could be beneficial to the community.", "weaknesses": "1. The method used in the paper is not new. The main contribution of this works seems to be on the dataset and using expositing method on a newer task. \n2. Ablation of the effectiveness of using  trained diffusion prior is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1XH5zj5gfL", "reviewer": "Reviewer_2kxB", "summary": "This work addresses the challenge of real-world face restoration. The authors introduce the Multi-modal Guided Real-World Face Restoration (MGFR) method, which improves facial image restoration from low-quality inputs using attribute prompts and reference images. Specifically, MGFR employs a dual-control adapter and a two-stage training strategy, leveraging a dataset of over 23,000 high-resolution images to enhance visual quality, identity preservation, and attribute correction. Experimental results demonstrate that the proposed method achieves superior performance.", "questions": "1. Not super important, but how do the authors feel the proposed method can be extended for real-world video face restoration? Can they please add some discussion on the scalability of the proposed method?\n2. Overall the paper looks promising and makes meaningful contributions, however, it lacks some important experiments and details. The authors can refer to the weaknesses section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The topic of real-world face restoration is interesting. \n2. The overall writing of the manuscript is clear and easy to follow.\n3. This work introduces the Reface-HQ dataset, which comprises over 23,000 high-resolution images. This dataset can provide a foundation for training and evaluating the model.", "weaknesses": "1. The proposed method appears complex, and its running time and memory consumption are not superior to existing methods like DiffBIR and DR2. Since running time and memory usage are crucial for real-world applications, this raises concerns about its practical feasibility.\n2. There is a lack of comparison in FLOPs and Params in the main comparison and ablation study.\n3. Lacking comparison to recent works (SUPIR, BFRffusion) in Table 1. \na. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild\nb. Towards real-world blind face restoration with generative diffusion prior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1cGiZvdPC6", "reviewer": "Reviewer_bNtm", "summary": "The paper introduces a Multi-modal Guided Real-World Face Restoration (MGFR) approach, which aims to enhance facial image restoration from low-quality inputs by leveraging multi-modal prior information. This includes attribute text prompts, high-quality reference images, and identity information to mitigate false facial attributes and identities typically generated by current restoration methods. MGFR employs a dual-control adapter and a two-stage training strategy to effectively utilize these priors for targeted restoration tasks. The paper also introduces the Reface-HQ dataset, containing over 23K high-resolution facial images from 5K identities, to support the training of reference-based restoration models. The proposed approach demonstrates superior performance in restoring facial details, particularly under severe degradation, while ensuring identity preservation and attribute correction.", "questions": "1) How will the performance vary if different versions of diffusion models are used? That is, would diffusion models be the strong foundation of achieving the good performance?\n\n2) Some figures, say fig.4 and 5, are with too small fonts and unclear color denotions, making it less readable.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novel framework:\n\nThe MGFR approach introduces a novel dual-control adapter and two-stage training strategy, effectively combining multi-modal priors for enhanced face restoration. The use of attribute text prompts alongside high-quality reference images and identity information is a significant advancement, offering more control over the restoration process.\n\n* Dataset Contribution:  \nThe introduction of the Reface-HQ dataset addresses a critical gap in the availability of high-resolution reference images, providing a valuable resource for the community.\n\n* Good Performance: \nThe method achieves superior visual quality in restoring facial details under severe degradation, demonstrating its practical applicability in real-world scenarios. The ability to control restoration through textual prompts enhances the flexibility and precision of the restoration process.\n\n* Mitigation of False Illusions: MGFR effectively addresses the problem of false facial attributes and identities, a common issue in existing generative face restoration methods.", "weaknesses": "* Complexity of Implementation:\nThe integration of multiple modalities and the dual-control adapter may introduce complexity in implementation. Therefore, I recommend that this paper should consider a more detailed information on computational requirements and scalability would be beneficial.\n\n* Evaluation Metrics:\n\nWhile the paper demonstrates superior performance, a more thorough discussion of the evaluation metrics used to assess visual quality, identity preservation, and attribute correction would strengthen the claims.\n\n* Ablation Studies:\nThe paper would benefit from additional ablation studies to isolate the impact of each component (e.g., attribute prompts, reference images) on the overall performance, providing deeper insights into the effectiveness of each modality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)