Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model

Authors: Keda TAO, Jinjin Gu, Yulun Zhang, Xiucheng Wang, Nan Cheng
Primary area: generative models
Keywords: Face image restoration, diffusion model

Abstract:
We introduce a novel Multi-modal Guided Real-World Face Restoration (MGFR) technique designed to improve the quality of facial image restoration from low-quality inputs. Leveraging a blend of attribute text prompts, high-quality reference images, and identity information, MGFR can mitigate the generation of false facial attributes and identities often associated with generative face restoration methods. By incorporating a dual-control adapter and a two-stage training strategy, our method effectively utilizes multi-modal prior information for targeted restoration tasks. We also present the Reface-HQ dataset, comprising over 21,000 high-resolution facial images across 4800 identities, to address the need for reference face training images. Our approach achieves superior visual quality in restoring facial details under severe degradation and allows for controlled restoration processes, enhancing the accuracy of identity preservation and attribute correction. Including negative quality samples and attribute prompts in the training further refines the model's ability to generate detailed and perceptually accurate images.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730388268804

thread_type:
official_review

<thread_object>
creator: reviewer_bntm

<content>
summary:
The paper introduces a Multi-modal Guided Real-World Face Restoration (MGFR) approach, which aims to enhance facial image restoration from low-quality inputs by leveraging multi-modal prior information. This includes attribute text prompts, high-quality reference images, and identity information to mitigate false facial attributes and identities typically generated by current restoration methods. MGFR employs a dual-control adapter and a two-stage training strategy to effectively utilize these priors for targeted restoration tasks. The paper also introduces the Reface-HQ dataset, containing over 23K high-resolution facial images from 5K identities, to support the training of reference-based restoration models. The proposed approach demonstrates superior performance in restoring facial details, particularly under severe degradation, while ensuring identity preservation and attribute correction.

strengths:
* Novel framework:

The MGFR approach introduces a novel dual-control adapter and two-stage training strategy, effectively combining multi-modal priors for enhanced face restoration. The use of attribute text prompts alongside high-quality reference images and identity information is a significant advancement, offering more control over the restoration process.

* Dataset Contribution:  
The introduction of the Reface-HQ dataset addresses a critical gap in the availability of high-resolution reference images, providing a valuable resource for the community.

* Good Performance: 
The method achieves superior visual quality in restoring facial details under severe degradation, demonstrating its practical applicability in real-world scenarios. The ability to control restoration through textual prompts enhances the flexibility and precision of the restoration process.

* Mitigation of False Illusions: MGFR effectively addresses the problem of false facial attributes and identities, a common issue in existing generative face restoration methods.

weaknesses:
* Complexity of Implementation:
The integration of multiple modalities and the dual-control adapter may introduce complexity in implementation. Therefore, I recommend that this paper should consider a more detailed information on computational requirements and scalability would be beneficial.

* Evaluation Metrics:

While the paper demonstrates superior performance, a more thorough discussion of the evaluation metrics used to assess visual quality, identity preservation, and attribute correction would strengthen the claims.

* Ablation Studies:
The paper would benefit from additional ablation studies to isolate the impact of each component (e.g., attribute prompts, reference images) on the overall performance, providing deeper insights into the effectiveness of each modality.

questions:
1) How will the performance vary if different versions of diffusion models are used? That is, would diffusion models be the strong foundation of achieving the good performance?

2) Some figures, say fig.4 and 5, are with too small fonts and unclear color denotions, making it less readable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535353323

thread_type:
official_review

<thread_object>
creator: reviewer_2kxb

<content>
summary:
This work addresses the challenge of real-world face restoration. The authors introduce the Multi-modal Guided Real-World Face Restoration (MGFR) method, which improves facial image restoration from low-quality inputs using attribute prompts and reference images. Specifically, MGFR employs a dual-control adapter and a two-stage training strategy, leveraging a dataset of over 23,000 high-resolution images to enhance visual quality, identity preservation, and attribute correction. Experimental results demonstrate that the proposed method achieves superior performance.

strengths:
1. The topic of real-world face restoration is interesting. 
2. The overall writing of the manuscript is clear and easy to follow.
3. This work introduces the Reface-HQ dataset, which comprises over 23,000 high-resolution images. This dataset can provide a foundation for training and evaluating the model.

weaknesses:
1. The proposed method appears complex, and its running time and memory consumption are not superior to existing methods like DiffBIR and DR2. Since running time and memory usage are crucial for real-world applications, this raises concerns about its practical feasibility.
2. There is a lack of comparison in FLOPs and Params in the main comparison and ablation study.
3. Lacking comparison to recent works (SUPIR, BFRffusion) in Table 1. 
a. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild
b. Towards real-world blind face restoration with generative diffusion prior.

questions:
1. Not super important, but how do the authors feel the proposed method can be extended for real-world video face restoration? Can they please add some discussion on the scalability of the proposed method?
2. Overall the paper looks promising and makes meaningful contributions, however, it lacks some important experiments and details. The authors can refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685448545

thread_type:
official_review

<thread_object>
creator: reviewer_1c8w

<content>
summary:
The paper propose to utilize diffusion prior, as long as multi-modality input to perform real world face restoration. The method can receive text or reference image for face restoration. 

They also present a dataset that contains identity-image pair. 

There is both quantitative  and qualitative metric, which demonstrate the effectiveness of this method.

strengths:
1. The motivation that using multi-modality input to assist face restoration is nice. As shown in the paper, there could be ambiguity during restoration, and the additional input could be helpful. 
2 .  The use of diffusion prior for this work also make sense, and it models a probability given the input. 
3. The dataset could be beneficial to the community.

weaknesses:
1. The method used in the paper is not new. The main contribution of this works seems to be on the dataset and using expositing method on a newer task. 
2. Ablation of the effectiveness of using  trained diffusion prior is missing.

questions:
1. The design of the Dual control Adapter is a little but tricky. Can one just replace it by a transformer?
    2. There can be different  result for every sample. How diverse is the model? And how to decide which sample to use?
    3. Will be dataset be realised?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732624777081

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers and area chairs for their valuable comments and recognition of our work, and we are pleased that all reviewers have responded positively to our paper, particularly their acknowledgment of the novel MGFR framework, the contribution of the Reface-HQ dataset, and the method’s outstanding performance. We also appreciate the reviewers’ recognition of our efforts in integrating multimodal priors, addressing real-world degradation, and reducing the generation of false facial attributes. 

Reviewers 1C8W (reviewer 1), 2kxB (reviewer 2) and bNtm (reviewer 3) provided insightful comments. In response to their feedback, we have provided detailed clarifications, conducted further analyses, and included additional experiments and data where necessary. Our response is summarized as follows:

1. In response to the concerns raised by reviewers 1, 2, and 3 regarding the complexity of dual-control adapters in the model and the feasibility of applying MGFR’s inference consumption in real-world scenarios, we added a discussion in Appendix D on the necessity of dual-control adapters and identified simplifying their design as a key direction for future work. Table 12 was updated to include the FLOPs and total model parameters for each diffusion-based face restoration method during inference. When compared to state-of-the-art methods, our model demonstrates comparable resource consumption. Finally, we believe that the development of efficient lightweight models is grounded in the superior performance of large-scale models, and future work will incorporate model compression techniques, including quantization and pruning, into MGFR.

2. In response to reviewer 1’s question regarding the selection of samples for MGFR after repeated inference, we included an evaluation of model stability in the revised manuscript, detailed in Appendix K, demonstrating that recovery results remain consistent across different random seeds.

3. In response to reviewer 2’s mention of recent face restoration models (BFRffusion and SUPIR), we incorporated a quantitative comparison of BFRffusion into the revised manuscript and, given time constraints, included a qualitative comparison of SUPIR in Appendix M. The results demonstrate that the proposed MGFR achieves superior recovery performance, particularly in preserving face identity information.

4. In response to reviewer 3’s concerns regarding evaluation metrics and suggestions for improving Figures 4 and 5, we have provided a summary of the evaluation indicators in Appendix L of the revised manuscript and enhanced Figures 4 and 5 to improve their readability.

5. Motivated by reviewer 2’s suggestion to extend MGFR to video face restoration, we included a detailed discussion of this application in the revised manuscript and identified it as a key direction for future research, as detailed in Appendix J.

In conclusion, we are the first attempt to use multimodal information to guide real world face image restoration. We are the first reference image-based face image restoration method that does not require strict face alignment. Qualitative and quantitative evaluations demonstrate that the proposed MGFR model delivers state-of-the-art performance. Future efforts will focus on developing lightweight model variants and streamlined designs, while MGFR has already established a new baseline for real-world face restoration tasks.

Finally, thanks again to all reviewers for their positive feedback and recognition of our work. In the new version of the manuscript, the revisions are marked in blue. If there are any other questions, we would be more than happy to engage in further discussions.

The best,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents the Multi-modal guided face restoration method, which significantly improves face restoration from low-quality inputs using attribute prompts and reference images. Using the Reface-HQ dataset of over 23,000 images, it enhances visual quality and identity accuracy, demonstrating good performance through rigorous experiments.

***Strengths:*** 
-  Propose a novel dual-control adapter for facial image restoration.
-  Introduce a large-scale, diverse dataset to train and validate the model. The authors promise to release this one for community.
-  Outperform existing methods in visual quality and identity preservation.

***Weaknesses:***
The model's complexity is a concern, although addressed in the authors' rebuttal.
More evidence of generalization  performance on real cases  would strengthen the claims.

After careful consideration and discussion, we are pleased to accept this submission. The decision to accept this paper is based on the innovative approach, strong experimental results, and positive reviewer feedback. The authors have partially addressed the concerns about model complexity, enhancing the paper's contribution to the field of face image restoration.