3D Priors-Guided Diffusion for Blind Face Restoration

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 19:22:58

Abstract: Blind face restoration aims to restore a sharp face image from a degraded counterpart. Recent methods using GANs as priors have achieved many successful stories in this domain. However, these methods still struggle to balance realism and fidelity when facing complex degradation scenarios. In this paper, we propose a novel framework by embedding 3D facial priors into a denoising diffusion model, enabling the extraction of facial structure and identity information from 3D facial images. Specifically, the downgraded image undergoes initial processing through a pre-trained restoration network to obtain an incompletely restored face image. This image is then fed into the 3D Morphable Model (3DMM) to reconstruct a 3D facial image. During the denoising process, the structural and identity information is extracted from the 3D prior image using a multi-level feature extraction module. Given that the denoising process of the diffusion model primarily involves initial structure refinement followed by texture detail enhancement, we propose a time-aware fusion block (TAFB). This module can provide more effective fusion information for denoising as the time step changes. Extensive experiments demonstrate that our network performs favorably against state-of-the-art algorithms on synthetic and real-world datasets for blind face restoration.

Main paper: 
 related work 2.1 blind face restoration: 
 As a fundamental task in computer vision, blind face restoration (BFR) aims to recover a high-quality face image from its degraded counterpart in the presence of unknown degraded types and parameters. CNN-based approaches can learn mapping relationships between low-/high-quality image pairs from large-scale collected datasets, but they often fail to restore high-frequency texture details on the face. In recent years, significant progress has been made for BFR by using facial geometry priors and generative priors. Harnessing the specific structure and details of faces, techniques grounded in facial geometry priors utilize prior information such as parsing maps, landmarks, and reference images to enhance the restoration of facial images [3, 4, 7] . On the other hand, generative priors-based methods employ powerful generative models like StyleGAN [20] and incorporate adversarial training to enhance the visual quality of the restored images. Despite the fact that generative prior methods have the ability to restore facial details more effectively when compared to CNN-based restoration approaches, they may encounter significant challenges pertaining to training difficulties and model collapse problems [45] .In the past few years, the majority of generative tasks in the field of computer vision have been dominated by GAN-based methods [17, 33] , which generate decent images through adversarial training. However, these methods may encounter challenges in training difficulties and model collapse [45] . With the application of diffusion models [14, 28] in the generative task domain, these models have demonstrated unprecedented generative capabilities in terms of image quality and diversity. Diffusion models have also been widely applied to various computer vision tasks, including image super-resolution [21, 30, 32] , image inpainting [27] , image segmentation [1, 2] , image-to-image translation [31] , text-to-image translation [10] , and more. In the context of BFR, DiffBIR [25] Utilizes the pre-trained text-to-image diffusion model and adopts a multi-stage pipeline approach for image restoration. However, it is characterized by a substantial number of parameters and relies solely on addition for feature fusion. To address the challenge of incorporating diverse degradations for real-world scenarios in training data, the Diffusion-based Robust Degradation Remover (DR2) [40] introduces a method that transforms degraded images into degradation-agnostic predictions before utilizing an enhancement module for high-fidelity image restoration. Denoising diffusion probabilistic models (DDPM) consist of forward and backward Markov processes. The forward process gradually adds random noise to the image, and we denote these latent variables as 𝑥 1 , ..., 𝑥 𝑇 , where 𝑥 𝑇 becomes a completely noisy image. The backward process of DDPM is a denoising process, where it learns a Markov chain that gradually transforms a simple noise distribution (such as isotropic Gaussian distribution) into the target data distribution. Throughout the entire forward and backward processes of DDPM, the dimensions of the image remain consistent. Each step of the forward process can be represented by the following equation:EQUATIONwhere 𝛽 1 , ..., 𝛽 𝑇 are fixed variance values. At each step, Gaussian noise with variance 𝛽 𝑡 is added, resulting in the final 𝑥 𝑇 being mapped to pure Gaussian noise. Let 𝑥 0 be the original image, and it is possible to obtain the noisy image at any step 𝑡 based on 𝑥 0 :EQUATIONwhere 𝛼 𝑡 := 1 -𝛽 𝑡 and ᾱ𝑡 := 𝑡 𝑛=1 𝛼 𝑛 . These parameters are predefined prior to training. DDPM achieves the denoising process by predicting the mean of the noise added from step 𝑥 𝑡 to 𝑥 𝑡 -1 :EQUATIONwhere 𝑝 𝜃 (𝑥 𝑡 -1 | 𝑥 𝑡 ) represents the backward process from 𝑥 𝑡 to 𝑥 𝑡 -1 , while 𝜇 𝜃 (𝑥 𝑡 , 𝑡) denotes the diffusion model with parameter 𝜃 . At step 𝑡, 𝑥 𝑡 -1 can be expressed by the predicted mean and 𝑥 𝑡 :EQUATIONwhere z ∼ 𝑁 (0, I) is a standard Gaussian noise and has the same dimensionality as noisy image 𝑥 1 , ..., 𝑥 𝑇 . By performing denoising for T steps, the pure noisy image can be transformed into the target data distribution. 
proposed method: 
 The overall framework for face restoration by incorporating 3D priors into a diffusion model is illustrated in Fig. 4 . Our overall framework mainly consists of two branches, including the 3D reconstruction branch and the diffusion branch. The 3D reconstruction branch includes the SwinIR model and the 3DMM model, while the denoising diffusion branch mainly includes a U-net, a multi-level feature extraction module, and a Time-Aware Fusion Block (TAFB).The low-quality image is initially restored using the pre-trained restoration module, SwinIR, resulting in an initial face restoration result denoted as x 𝑖𝑛𝑖𝑡 . The 3D facial image is reconstructed using the D3DFR method [6] . The multi-level feature extraction module extracts identity and structural information features across different scales of the 3D facial image. These features are then input into the TAFB, in conjunction with features extracted from the noisy image and the current timestep 𝑡. Subsequently, the time-aware block fuses the features across various time steps and passes them to the subsequent feature extraction block. We will introduce the 3D reconstruction branch and the diffusion branch in detail.Although current diffusion-based blind face restoration methods have shown promising results in terms of image quality restoration, they often fail to ensure the identity consistency of the restored faces, as illustrated in Fig. 2 . It is mainly due to the conflict and balance of realism and fidelity, particularly in complex degradation scenarios. To mitigate this problem, 3D facial priors as structure and identity constraints are embedded into a denoising diffusion process to keep the high fidelity while generating high-realism images. However, considering the dynamic nature of the denoising process in the diffusion model, which involves initial structure refinement followed by texture detail enhancement, we design ingenious modules to incorporate the 3D priors into the diffusion model. 1.) A customized multi-level feature extraction method is designed to fully exploit both structural and identity information of 3D facial images, which are then mapped into the noise estimation process. 2.) A Time-Aware Fusion Block (TAFB) is proposed to enhance the fusion of identity information into the noise estimation and offer a more efficient and adaptive fusion of weights for the denoising process. 3.) The features of 3D facial priors are multiiteratively embedded into the denoising process at each step and avoid inadequate excavation of 3D facial priors.3D face priors encompass a wealth of hierarchical features, including low-level details such as sharp edges and lighting, as well as perceptual-level information related to identity. However, Lowquality images for blind face restoration often suffer from multiple complex types of degradation (e.g., blur, noise, JPEG compression artifacts, low resolution, etc.). So the input low-quality image 𝑥 𝐿𝑞 is first processed by the pre-trained restoration module, SwinIR [24] , to obtain the initial face restoration result 𝑥 𝐼𝑛𝑖𝑡 .EQUATIONThe outcomes obtained will be utilized as inputs for the diffusion model and the 3D Morphable Model (3DMM) [6] module. The lowquality image is processed using ResNet-50 [12] to obtain the 3DMM coefficients 𝑧 3𝑑 .EQUATIONwhere 𝑧 3𝑑 is a 257-dimensional vector, denoted as 𝑧 3𝑑 = (𝜶 , 𝜷, 𝜹, 𝜸, p), where 𝜶 , 𝜷, 𝜹, 𝜸 and p respectively represent the identity information, facial expression, texture, illumination [29] , and facial pose.We define the 3D shape S and texture T of a face as follows:EQUATIONS and T respectively represent the average facial shape and facial texture. The variables B 𝑖𝑑 , B 𝑒𝑥𝑝 and B 𝑡 represent the principal component analysis (PCA) bases for identity, expression, and texture, respectively. The color information of 3D face, denoted as C, can be represented as:EQUATIONThe 3D image is reconstructed using the D3DFR method, with coefficients S, T, and C. The preliminary restoration module and the 3D reconstruction module are not involved in the training process. As illustrated in Fig. 3 , 3D faces reconstructed directly from lowquality images exhibit significant deviations in facial expressions, structure, facial features, mouth shape, and illumination. In contrast, the 3D faces we reconstructed appear more realistic and faithful to the original subject.In the denoising diffusion branch, we initially subject 𝑥 ℎ𝑞 to the forward process to derive the noise image as per Eq. 2, (where t ∈ [1, T], and T represents the total number of denoising diffusion steps). Subsequently, the reconstructed facial 3D image is fed into the multi-level feature extraction module to extract features ranging from coarse to fine, thereby capturing both structural and identity information within the facial 3D image.EQUATIONResBlock uses the same structure as SR3 [32] , and the obtained features will be sent to the next level for feature extraction.EQUATIONDownsampling is performed through a convolution layer, where 𝐹 𝑖 3𝑑 represents the facial 3D image features extracted from the 𝑖-th layer. We also use ResBlock for feature extraction on the noisy image 𝑥 𝑡 . 𝐹 𝑖 𝑖𝑛 , 𝐹 𝑖 3𝑑 , and timestep 𝑡 is input into the Time Aware Fusion Block, as depicted in Fig. 4 . Since the guidance information required by the denoising diffusion model is different at different time steps, we first input t into the Multilayer Perceptron (MLP) to learn the weight parameters.𝛼1, 𝛽1, 𝛾1, 𝛾2, 𝛾3 = 𝑀𝐿𝑃 (𝑡),We apply Spatial Feature Transform (SFT) to modulate the facial 3D image features processed by the LayerNorm layer.EQUATIONwhere ⊙ represents Element-wise Multiplication, passing 𝐹 1 through different Channel Squeezes (CS) can obtain different weights in the channel dimension. For the sake of simplicity, we only draw four channels in Fig. 4 , two sets of different weights. Multiply with 𝐹 1 and 𝐹 𝑖 𝑖𝑛 passing through the LayerNorm layer, respectively. Through channel attention, the model can focus more on important structural information in facial images.EQUATIONAfter splicing 𝐹 3 and 𝐹 4 through concat, we use 1X1 convolution to transform the number of channels, and then pass through the scale add module. This module uses the weights learned at time step 𝑡 to combine the output features, denoising image features, and 3D Features.EQUATIONThen 𝐹 5 is input to the Feedforward Neural Network (FFN) and then passes through the Gate layer. The gate layer is mainly used as a gating mechanism in the feedforward network, and the weights are learned by 𝑡.EQUATIONWe optimize the conditional denoising diffusion model through the following equation:EQUATIONIn the inference stage, we use the same truncated sampling method as [43] for inference. We set the denoising steps to 100, and the specific network architecture layers will be presented in the supplementary materials. 
experiments 4.1 experimental settings: 
 Datasets. Following the methods [22, 37, 42] , we also selected the FFHQ [19] dataset as our training dataset, which consists of 70,000 high-resolution face images with a resolution of 1024×1024. Initially, we used simple downsampling to resize the images in the dataset from 1024 × 1024 to 512 × 512, which served as the high-quality (HQ) images in our training dataset. Following the methods [22, 37, 42] , we also employed the same random degradation approach to synthesize the LQ images:EQUATIONThe corresponding LQ images were synthesized via Eq. ( 18), where 𝑦 represents the HQ image, 𝑘 𝜎 denotes the Gaussian blur kernel, 𝑟 signifies the downsampling factor, and 𝑞 represents JPEG compressed images with a quality factor of 𝑞. We randomly sampled the parameters 𝜎, 𝑟 , 𝛿, 𝑞 from {0.1 : 10}, {4 : 20}, {1 : 20}, {30 : 70}. We utilized three datasets for evaluation: CelebA-Test [18] , LFW-Test [16] , WIDER-Test [41] , and WebPhoto [37] . The CelebA-Test dataset consists of 3000 synthetic images randomly sampled from CelebA-HQ [18] , a high-resolution image dataset. The LQ images represent degraded images with an unknown degradation model and parameters via Eq. ( 18). The LFW-Test dataset comprises 1711 face images collected from the internet, representing real-world data with a certain level of complexity and diversity. The WebPhoto-Test dataset consists of 407 face images gathered from various online sources. The WIDER-Test dataset comprises 970 severely degraded facial images sourced from the WIDER Face dataset [41] . Implementation details. In the experiment, we first validated the effectiveness of the proposed method on the blind restoration task. Then, we further demonstrated its superiority by testing on synthetic and real datasets. Our method employed the Adam optimizer. The default initial learning rate is set to 0.0001, and the learning rate do not decay during training. The experiment is conducted on the A100 GPUs with a batch size of 16. Quantitative comparisons of CelebA-test dataset. The results obtained from our experiments, as presented in Tab. 1, demonstrate that our approach outperforms other methods in terms of quantitative measures when evaluated on the CelebA-Test dataset. Our approach achieves the highest scores in FID-F, FID-G, and IDS, indicating that our restoration results closely resemble both the [18] . Our method performs better in both identity consistency and structure consistency.PSFRGAN [3] GFPGAN [37] Codeformer [46] VQFR [11] DiffBIR [25] DR2 [40] Ours For the performance of blind face restoration, we conducted tests on the real datasets LFW-Test [16] , WebPhoto [37] , and WIDER-Test [41] using the Fréchet Inception Distance (FID) as the evaluation metric. The best and second-best results are highlighted in red and blue, respectively.distribution of real face images and natural images, while also maintaining the structural consistency and identity consistency of the restored face. Besides, the pixel-level metrics SSIM and PSNR are metrics used to evaluate structural similarity and image quality. In these two indicators, the performance of our method also reaches the top scores among all methods, indicating that the restored face image is the best in terms of structural similarity and image quality.Quantitative comparisons of the real-world dataset. Furthermore, our study encompasses a comprehensive quantitative analysis of the real-world datasets LFW-Test, WebPhoto, and WIDER-Test as delineated in Tab. 2. Notably, our method emerged as the top performer on the WebPhoto dataset, showcasing its exceptional efficacy in facial restoration. Conversely, our method secured the second-highest performance on the LFW-Test dataset, as evidenced by the FID evaluation metric. However, real-world images typically The results in the first row are from the LFW-Test dataset [16] , the results in the second row are from the WebPhoto dataset [37] , and the results in the third and fourth rows are from the WIDER-Test dataset [41] . exhibit lesser degradation compared to synthetically altered images. Consequently, the full extent of our method's optimal facial repair capabilities may not be fully realized in such scenarios. Qualitative comparisons of CelebA-test dataset. We conduct a qualitative analysis of six blind face restoration methods. As shown in Fig. 5 , the highlighted regions in red boxes indicate significant differences among the methods in terms of facial detail restoration. GFPGAN [37] , a GAN-based approach, fails to recover in severe degradation cases. On the other hand, CodeFormer [46] learns discrete codebooks and decoders to store high-quality visual parts of face images through self-reconstructive learning. Our method, by incorporating 3D facial prior information into the diffusion model, better ensures the preservation of facial identity. Our approach demonstrates excellent fidelity in facial contours, nose, eyes, and mouth, approaching the ground truth. Qualitative comparisons of Real-World dataset. The qualitative outcomes of the real-world dataset are depicted in Fig. 6 . Existing methodologies struggle to compensate adequately for information when the input image experiences extensive degradation. In contrast, our methodology introduces 3D facial prior information, resulting in visually more appealing outputs, particularly in cases of severe degradation in the input image. Analysis of the weight in Scale Add Block. As shown in Fig. 7 , the weight of 3D image features in Scale Add Block begins to increase as denoising proceeds. The first stage is mainly the recovery of structural information, and the subsequent stages are the recovery of detailed texture information [35] . The process will destroy certain structural information, and the weight of 3D prior information will be reduced. The second stage mainly relies on denoising image features, and the weight continues to increase. Analysis of computational complexity and inference time. The comparative computational complexity analysis between our method and existing diffusion-based techniques is outlined in Tab. 4. Parameter count and Multiply-Accumulate Operations (MACs) are calculated using the profile function sourced. Inference time measurements are performed using a single NVIDIA A100 GPU.Metrics. We evaluate our method using three distinct perceptual metrics: LPIPS [44] , and FID (Fréchet Inception Distance) [13] . For completeness, we also include two distortion-based metrics: PSNR Quantitative and qualitative analyses of ablation settings. We conduct both quantitative and qualitative analyses on the CelebA-Test dataset [18] . As depicted in Fig. 8 , Our method shows better recovery effects on individual glasses, eyes, mouth, eyebrows, and facial structures. Since the 3D prior information can reconstruct the area under the glasses, when the 3D prior information is missing, the eyes will fail to reconstruct. If the TAFB module fusion function or time embedding is not integrated, the generated image is prone to over-smoothing because conditional guidance cannot provide different guidance information at different time steps.Based on the findings from the quantitative analysis presented in Tab. 3, it is evident that the absence of facial 3D prior information as a guide results in the most significant decline across all indicators. This is attributed to the crucial role of facial 3D prior informationParameters MACs Time(s/image) DR2 [40] 179.31M 918.843G 0.694 DiffBIR [25] in offering a clear facial structure for face restoration and identity preservation. Notably, the omission of the temporal embedding block in the fusion method leads to a substantial decrease in the quality of the restored image. This is due to the inability to provide more precise guidance throughout the entire denoising process.Directly guide the diffusion model through the initial restored image for face restoration. Since there is no 3D facial reconstruction of the eye area, the eye area will encounter serious artifacts. 
conclusion: 
 We propose a blind degraded face image restoration model based on a 3D facial prior diffusion model, which is inspired by the fact that the 3D prior information not only contains facial details but also includes identity information. To ensure realism and fidelity, 3D priors can be regarded as identity and structure constraints in the denoising diffusion process to ensure identity consistency while generating high-quality images. Specifically, the structural features and identity features in the 3D prior information are extracted through the multi-level feature extraction module. Given that the denoising process of the diffusion model primarily involves initial structure refinement followed by texture detail enhancement, we propose a Time-Aware Fusion Block (TFAB). TAFB is used to effectively and weight-adaptively fuse the features with the features extracted from the noise image to more accurately predict the noise and restore the identity and structure consistent with face images. Extensive experiments demonstrate that the proposed model performs favorably against state-of-the-art algorithms.  
: 
 The restoration results in the first-row show that our method is better consistent with GroundTruth(GT) in the restoration of facial features.As an ill-posed problem, this task is highly challenging since lowquality face images can suffer from multiple degradations, such as downsampling, blurring, noise, and compression. Multifarious traditional end-to-end methods have been proposed based on convolutional neural networks (CNNs) to learn mapping relationships between low-quality and high-quality image pairs, but they fail to restore fine details on the face [5, 26] . Recently, due to the powerful ability of Generative Adversarial Networks (GANs) [8] to generate realistic face images, some methods [9, 42, 47] have utilized GANs as a prior for generating high-quality faces. These methods extract low-quality face features using CNNs and encode them into the latent space of the GAN. However, methods employing GANs as priors may encounter training collapse [45] , a consequence of challenges in optimizing the objective function during training. To improve the identity consistency of the restored images, some methods incorporate prior information during the face restoration process, such as facial geometric priors [4, 15] and reference priors [22, 23] . For example, Gu et al. use vector quantization face Restoration (VQFR) [11] to guide the model for generating more realistic facial details by storing the features extracted from highresolution images in a dictionary. Some methods [39, 46] leverage the powerful feature extraction capability of Vision Transformer (ViT) [36] to achieve better restoration results. However, as shown in Fig. 2 , the restoration results of these models cannot achieve a balance between fidelity and authenticity when dealing with complex scenes or under specific conditions. These approaches continue to face challenges in accurately restoring intricate facial features while simultaneously capturing realistic textural qualities, as exemplified in Fig. 1 .More recently, denoising diffusion models have been introduced into image restoration. For example, [32] feeds the combination of low-quality and noisy images into the denoising diffusion model for noise prediction. [30] encodes the low-quality image once and passes it through a cross-attention mechanism, feeding the features into a diffusion model. Additionally, [25] utilizes a parallel encoding module to encode the condition information and inputs it into the decoder of the denoising diffusion model at each step of the LQ HQ 3DMM Ours denoising process. SR3 [32] presents a method for achieving image super-resolution through iterative refinement, involving the generation of conditional images by concat low-resolution images and noise images. Latent diffusion [30] learns the data distribution in the latent space by gradually diffusing the noise signal and completes the guidance by applying cross attention to the conditions in the denoising network. However, throughout the denoising diffusion process, the low-frequency components exhibit gradual changes as the time step progresses, whereas the high-frequency components undergo more pronounced alterations [34] . Besides, in the initial phase of the denoising process, the primary emphasis is on refining the structure of the restored image, while in the later stages, the main focus shifts to refining the intricate textures of the restored image [35] . To improve image restoration quality, it is not sufficient to perform feature extraction on the guide image only once, nor is it adequate to rely solely on simple addition or cross-attention to fuse the features. Different from aforementioned methods, to ensure the fidelity, authenticity, and identity consistency of restored images in complex degradation scenarios, we propose a 3D prior-guided diffusion model by incorporating 3D facial prior information as constraints. Besides, we propose a multi-level feature extraction module to extract structural and identity information from 3D prior information at each time step and then weight-adaptively fuse this conditional guidance information with noisy image feature information through a Time-Aware Fusion Block. Overall, the main contributions of this paper can be summarized as follows:• A novel diffusion-based face restoration network is proposed, which integrates 3D facial structure into the noise estimation process. In our approach, a multi-level feature extraction method is tailored to extract both structural and identity information, which is then projected into the latent noise space. • To well adapt to the denoising process of the diffusion model that begins with structure refinement and progresses to texture detail enhancement, a Time-Aware Fusion Block (TAFB) is proposed to effectively and weight-adaptively fuse facial prior features and noisy image features at different time steps. • Comprehensive experiments demonstrate that the proposed method performs favorably against state-of-the-art algorithms on synthetic and real-world datasets in terms of image quality restoration and identity consistency. 
 
 Appendix:  


Decision: All 3 reviewers suggested acceptance for this paper (1 Weak Accept and 2 Borderline Accept). The AC has read the paper, reviews, and rebuttal, and concurs that this paper should be accepted, given the technical contribution of injecting 3D prior into face restoration diffusion model and the good performance the proposed method. Congratulations and please revise the paper according to the reviewer's suggestions. The author should also consider citing a relevant work:
Wen et al. Face Video Deblurring using 3D Facial Priors. ICCV 2019.

Reviews: review: paper_summary: paper_summary: The paper titled "3D Priors-Guided Diffusion for Blind Face Restoration" introduces a novel framework for restoring clear face images from degraded counterparts. The approach leverages a diffusion-based model that incorporates 3D facial priors as structural and identity constraints into a denoising diffusion process. The authors propose a Time-Aware Fusion Block (TAFB) to adaptively fuse facial prior features with noisy image features at different time steps of the denoising process. The method is evaluated on both synthetic and real-world datasets, demonstrating improved performance over state-of-the-art algorithms in terms of image quality restoration and identity consistency.
strength_weakness: strength_weakness: 1.The paper presents a unique approach to blind face restoration by integrating 3D facial structure and identity information into a diffusion model, which is a novel contribution to the field. The proposed method's technical correctness is evident in its detailed explanation of the diffusion process and the innovative Time-Aware Fusion Block, which shows a deep understanding of the problem space.  The paper provides a comprehensive evaluation using both synthetic and real-world datasets, which strengthens the credibility of the proposed method's effectiveness.
2.The paper is well-structured, with clear explanations and figures that effectively illustrate the proposed method and its results.
3. The paper is supported by a solid theoretical foundation and backed by empirical evidence through extensive experiments.
limitations: limitations: 1. While the paper discusses the computational complexity, a deeper analysis comparing the proposed method with other state-of-the-art techniques in terms of runtime and resource efficiency could be beneficial.
2. The author presented five different experimental groups in the ablation experiment section to evaluate the proposed method. Near P534, the author stated that during the inference stage, the same truncation sampling method as in reference [43] was used. I don't know if the author is considering increasing the use of reasoning methods other than truncation sampling, perhaps the experimental results will be better.
3. The paper could benefit from additional discussion on the model's robustness to different types and degrees of degradations.
4. P556, should "We utilized three datasets..." be changed to "We utilized four datasets..."? 
5. Should the full names of FID-F and FID-G evaluation indicators be provided in section 4.1.
6. The paper primarily focuses on face restoration. It would be useful to see how the proposed method generalizes to other image restoration tasks.
paper_summary: paper_summary: This paper proposes a novel blind face restoration network for recovering high-resolution face images from degraded face images. The main contributions include innovatively integrating 3D face structure into the noise estimation process and designing a multi-level feature extraction method to extract structural and identity information. Additionally, a temporal-aware fusion block is proposed to adapt the denoising process of the diffusion model from structure refinement to texture detail enhancement. The paper has good innovation and meets the requirements of the conference.
strength_weakness: strength_weakness: This paper clearly articulates the key issues that need to be addressed in blind face restoration and gradually introduces the proposed method, providing a novel perspective for the research field of blind face restoration.
limitations: limitations: (1) There are some issues with the sentences in the paper, for example, in Section 4.1: ‘We utilized three datasets for evaluation: CelebA-Test [18], LFW-Test [16], WIDER-Test [41], and WebPhoto [37].’ Here, the authors actually used four test datasets, but the statement says three.
(2) The proposed method in this paper performs exceptionally well in preserving facial structure. Do the authors have further ideas regarding the details of facial features, such as eyes and makeup?
paper_summary: paper_summary: This paper uses 3DMM and Diffusion Model for Blind Face Restoration. Specifically, this work uses DiffIR to pre-restore noisy images for better 3DMM, and then fuses 3DMM to diffusion model using a SFT-like method.
strength_weakness: strength_weakness: 1. Recovering 2D imgs with 3D priors is a difficult task, especially when the 3D info is not just accurate.
2. As far as I know, the first Diffusion-based BFR method ultilizing 3D priors.
limitations: limitations: 1. This paper mentions ensuring id consistency. However, I haven't seen any innovative explicit ID enhancement operations, as fusing 2D feature maps with a SFT module is a quite common operation.
2. It seems that other methods have better results when the image is not so noisy. eg. the first 2 rows in Fig. 6. 
3. Is the downsampling/blurring factor the same as previous methods during training? If different, will it lead to unfair issues?
4. Will the information provided by the 3D part cause interference for images that are not so blurry?