AuthFace: Towards Authentic Blind Face Restoration with Face-oriented Generative Diffusion Prior

## Abstract

Abstract Blind face restoration (BFR) is a fundamental and challenging problem in computer vision.
To faithfully restore high-quality (HQ) photos from poor-quality ones, recent research endeavors predominantly rely on facial image priors from the powerful pretrained text-to-image (T2I) diffusion models.
However, such priors often lead to the incorrect generation of non-facial features and insufficient facial details, thus rendering them less practical for real-world applications.
In this paper, we propose a novel framework, namely AuthFace that achieves highly authentic face restoration results by exploring a face-oriented generative diffusion prior.
To learn such a prior, we first collect a dataset of 1.5K high-quality images, with resolutions exceeding 8K, captured by professional photographers.
Based on the dataset, we then introduce a novel face-oriented restoration-tuning pipeline that fine-tunes a pretrained T2I model.
Identifying key criteria of quality-first and
photography-guided annotation, we involve the retouching and reviewing process under the guidance of photographers for high-quality images that show rich facial features.
The photography-guided annotation system fully explores the potential of these high-quality photographic images.
In this way, the potent natural image priors from pretrained T2I diffusion models can be subtly harnessed, specifically enhancing their capability in facial detail restoration.
Moreover, to minimize artifacts in critical facial areas, such as eyes and mouth, we propose a time-aware latent facial feature loss to learn the authentic face restoration process.
Extensive experiments on the synthetic and real-world BFR datasets demonstrate the superiority of our approach. Codes and datasets will be available upon acceptance.

## 1 Introduction

Face images captured in natural settings often exhibit various forms of degradation, including compression, blur, and noise ; .
Capturing high-quality (HQ) face images is crucial, as humans are highly sensitive to subtle facial details. Blind face restoration (BFR) aims to reconstruct HQ images from degraded inputs and has rapidly progressed in recent years due to significant research interest.
However, BFR remains an ill-posed problem due to the unknown degradation and the loss of valuable information resulting from these complex conditions .

Sufficient prior information is critical for HQ reconstruction. Researchers have used geometric and reference priors from sources like ; ; ; ; ; ; ; ; ; ; ; ; to guide face restoration. These priors, however, are limited by their sensitivity to degradation and inability to capture fine facial details.
With advancements in generative models, such as StyleGAN  and VQVAE , recent works ; ; ; ; ; ; ; have leveraged pretrained networks to derive facial priors, achieving superior results compared to earlier methods. Nonetheless, these approaches still face significant performance declines in handling unseen cases.
Denoising diffusion probabilistic models (DDPMs)  have shown promise as an alternative to generative adversarial networks (GANs)  in image generation.
Some approaches ; use pretrained DDPMs to diffuse and then denoise degraded inputs.
However, their practical application is hindered by the loss of original identity and detailed facial features , with pretrained DDPMs also facing limitations in representational capacity.

The remarkable success of large-scale pretrained text-to-image (T2I) models ; has provided another promising prior.
Many researchers explore the potential of StableDiffusion (SD) models  as the powerful prior in challenging low-level vision tasks, including real-world image super-resolution ; ; ; and BFR ; .
Since the face details are often lost due to the degradation and down-sampling processes of VAE  in SD models, BFRffusion  and DiffMAC  rely on the facial priors within SD models to recreate these details.
However, being designed for general text-to-image tasks, SD models often fail to retain essential facial details, like skin texture (see Fig. 2 (b)).
Therefore, these methods typically produce overly smooth images in the T2I task.
Moreover, their extensive image priors can lead to the incorrect generation of non-facial features, resulting in artifacts, especially for images with ambiguous degradation.
These specific limitations – incorrect generation of non-facial features (blue box at \engordnumber1 row of Fig. 1) and missing facial details (red box at \engordnumber3 row of Fig. 1)– severely limit the practical deployment of these models in real-world applications.

Figure: Figure 1: Compared with the results from a state-of-the-art (SOTA) method SUPIR using StableDiffusion-XL (SDXL) as prior, our approach excels in capturing and rendering intricate facial details. For instance, our result has a more distinct jawline (see blue arrow) in the \engordnumber2 row, effectively distinguishing the jaw from the neck. Zoom in for more details.
Refer to caption: x1.png

To tackle these problems, we propose Authface, a novel BFR method with face-oriented generative diffusion prior, designed to restore highly authentic face images.
The highlight of our Authface is that it brings a paradigm shift for BFR – with a two-stage training pipeline: 1) Face-oriented Fine-tuning on Pretrained T2I Model, and 2) Highly Authentic Face Restoration.
The underlying premise for Stage I is that pretrained T2I models e.g., SD models, can serve as effective generative diffusion priors for restoration tasks (Sec. 3.1). They can be customized for face-centric applications via fine-tuning while retaining their generation capabilities.

Figure: Figure 2: (a) A HQ face image with its paired tags generated through photography-guided image annotation. Specifically, we provide an additional photographic tag (blue box) beyond the semantic tags used in previous methods (gray box). (b) Qualitative comparison between StableDiffusion-XL (SDXL) and our fine-tuned model, which is exclusively trained on the collected high-quality dataset, in the T2I task. Notably, SDXL tends to generate over-smooth skin even when given prompts specifying sharp details and sharp focus. Zoom in for more details.
Refer to caption: x2.png

In analyzing key factors for fine-tuning pretrained T2I models to meet human preferences for authentic facial images, we identify two key criteria as our face-oriented generative diffusion prior:
1) Quality-first image collection. Contrary to training T2I base models with large datasets like LAION-5B , the quality of the dataset, rather than its size, dictates the generation quality in the fine-tuning process.
2) Photography-guided image annotation. Fine-tuning the pretrained T2I models for HQ facial tasks requires more than just basic annotations like human accessories, especially for HQ face images with a pronounced stylistic orientation (see Fig. 2 (a)).
In line with our established criteria, we collect a curated dataset of 1.5K HQ face images – each enriched with detailed photographic annotations – to fine-tune the pretrained T2I models for the first stage. With the curated dataset, we are able to fine-tune the T2I models following their original optimization strategies, as illustrated in Fig. 3. With fine-tuning, the pretrained T2I model is required with the detailed facial prior, which can be demonstrated with the T2I task as shown in Fig. 2 (b).
To achieve the goal of highly authentic face restoration in Stage II, we leverage the ControlNet  for training (Sec. 4).
However, directly following the protocol of training ControlNet with the MSE loss tends to contribute to the loss of key facial details, such as eyes and mouths.
To resolve this issue, we propose a time-aware latent facial feature loss to directly constrain the regions where humans are sensitive in the latent space. Our extensive experiments demonstrate the superior authentic detail generation performance on synthetic and real-world datasets.

In summary, our major contributions are three-fold: I) Novel Research Direction: Our work pioneers a new approach by enhancing the generative capabilities of pretrained T2I models for authentic face restoration, moving beyond traditional model design.
II) New Methodology: Our AuthFace, a novel framework, enhances the detail handling of pretrained T2I models through a unique face-oriented restoration tuning pipeline. Our method significantly sharpens fine facial details and includes a time-aware latent facial feature loss, which effectively reduces artifacts in critical areas like the eyes and mouth.
III) New High-quality Dataset: We have compiled a dataset of 1.5K high-resolution images. We expect it can serve as a foundational and important resource to further advance the field of high-fidelity authentic face restoration.

## 2 Related Works

Prior-based Blind Face Restoration
Blind face restoration (BFR) employs a variety of priors, classified into geometric, reference, and generative categories. Geometric priors, such as facial landmarks ; , face parsing maps ; ; , facial component heatmaps , and 3D face shapes ; ; , provide crucial structural information for restoring degraded faces. Reference-based methods use images to deliver identity information, enhancing the fidelity of the restored faces ; ; ; . Moreover, some researchers have implemented generative facial priors, like StyleGAN , to refine facial details ; ; ; ; .
Another approach involves using pretrained Vector-Quantize codebooks that contain detailed facial information ; ; .
Given their remarkable performance in image generation, denoising diffusion probabilistic models  have become increasingly popular in BFR.
Notable examples, such as DifFace , DR2 , and PGDiff , utilize denoising U-Nets pretrained on HQ face datasets to achieve face restoration at pixel level.
Specifically, Zhao et al.  attempts to improve the authentic performance via feeding network with enhanced ground-truth images.
Recently, large-scale pretrained text-to-image models like StableDiffusion (SD) have been employed to address the BFR problem. DiffBIR  leverages SD priors for real-world image super-resolution and BFR by incorporating degraded input image information in the latent space.
Specifically targeting BFR, BFRfusion  extracts multi-scale facial features in the latent space from low-quality face images.
However, achieving authentic BFR with pretrained T2I models in the latent space remains underexplored.

Fine-Tuning
Fine-tuning is widely used to align pretrained large language models (LLMs) with human preferences, improving their effectiveness .
This technique, successful in LLMs with small, HQ datasets ; , has been adapted to text-to-image models to enhance text-image alignment ; ; .
For example, Emu  improves aesthetic alignment using fine-tuned HQ image-text pairs.
Playground v2.5  enhances human features using a quality-controlled dataset, and CosmicMan  generates superior human-centric content with large, refined datasets.
However, these methods often produce overly smooth images, which may not be ideal for BFR tasks where authentic and realistic images are essential.

## 3 Methodology

The goal of our work is to achieve authentic face restoration by minimizing unrealistic outcomes and enhancing the rendition of human-preferred features. It is structured into two distinct stages:
1) Face-oriented Tuning on Pre-trained T2I Model. We integrate supervised fine-tuning  and quality-tuning  strategies to refine StableDiffusion-XL (SDXL), enhancing it with detailed facial features as our face-oriented generative diffusion prior (Sec. 3.1);
2) Highly Authentic Face Restoration. Utilizing the face-oriented generative diffusion prior, we implement ControlNet  to direct the restoration process based on the quality of input degradation (Sec.4).
Moreover, we introduce a time-aware latent facial feature loss to improve key facial features during restoration (Sec.4).

### 3.1 Stage I: Face-oriented Fine-tuning on Pre-trained T2I Model

The face-oriented tuning procedure for a pre-trained T2I model consists of two main parts: 1) a quality-first dataset preparation process to obtain and filter HQ face images, and 2) photography-guided data annotation to move beyond basic labels that only convey semantic information.

Quality-first Image Collection.
Training T2I models typically requires large datasets.
However, akin to the significant performance improvements seen in large language models fine-tuned with just 1K HQ examples , it has been demonstrated that enhancing the aesthetic quality of generated results can be achieved with only a few thousand extremely HQ images .
This highlights that dataset quality is more important than size in the fine-tuning process.
Inspired by this, we apply the quality-first principle to prepare our fine-tuning dataset.

Collecting HQ real-world face images is challenging due to privacy and copyright concerns, and existing datasets like FFHQ  suffer from issues such as JPEG degradation, blur, and Gaussian noise .
To overcome these challenges, we source extremely HQ face images from the professional photography website Unsplash , which offers a license supporting both commercial and non-commercial use.
Although the collected images are captured and post-processed by professional photographers, not all images prominently feature faces.
To address this, we implement a set of data filtering strategies to create an HQ, face-centric subset.
These strategies include face detection to remove images without faces or with small faces, and image quality assessment to filter out images with excessive artifacts such as pepper noise.
Also, we use face landmark detection to locate eyes and mouth, enabling us to follow the alignment process used in FFHQ ; , better suited for the BFR task.

Recognizing the racial imbalance in our dataset (predominantly Caucasian and African descent), we collaborate with professional photographers to build an HQ dataset featuring individuals of Asian descent, using top-level studio settings.
All facial images are manually filtered to ensure they present clear skin texture and hair details, resulting in a fine-tuning dataset of 1,500 extremely HQ images.

Figure: Figure 3: The framework of face-oriented tuning.
Refer to caption: x3.png

Photography-guided Image Annotation.
The quality of prompts is essential for both training  and fine-tuning  pretrained T2I models.
For example, CosmicMan  fine-tunes SDXL for human-centric content generation by breaking human parsing maps into several parts to provide detailed annotations.
However, for face-oriented tuning tasks, densely annotated images are less effective.
After cropping and alignment, the semantic information in facial images is limited, often capturing only overall human attributes.
This differs significantly from other tasks where densely packed semantic information is prevalent.
For face-oriented tuning, capturing stylistic information beyond basic semantics is crucial.
In portrait photography, this includes expressions, skin texture, makeup, and lighting, essential for authentic face restoration.

Therefore, we apply photography-guided data annotation to generate prompts for our fine-tuning dataset, especially given that our dataset consists of HQ portraits by professional photographers with strong stylistic tendencies.
We follow previous methods ; to realize automatic captioning tasks with Vision-Language Models (VLMs).
Specifically, we leverage the pretrained LLaVA-1.6  as the automatic caption to generate a tag-style prompt to avoid redundant prepositions and adverbs .
Fig. 2 (a) illustrates some examples of photography-guided data annotation.
Based on the dataset, we can fine-tune the pre-trained T2I model, SDXL, as shown in Fig. 3.
Different from the training of SDXL, we fix the resolution of training images instead of multi-aspect training.

### 3.2 Stage II: Highly Authentic Face Restoration

Fig. 4 illustrates the structure of stage II. Given the fine-tuned SDXL model as our face-oriented generative diffusion prior, we need an adaptor that can control the fine-tuned SDXL to generate high-quality facial images based on its degraded input. With the successful application of ControlNet  in real-world image super-resolution ; , we apply it as the controller for the BFR task.

The training of stage II is as follows. The latent representation of an HQ facial image is obtained by the encoder of a pretrained VAE, denoted as $\mathbf{z}_{0}$. The diffusion process progressively introduces noise to $\mathbf{z}_{0}$, resulting in a noisy latent $\mathbf{z}_{t}$, where $t$ represents the randomly sampled diffusion step. The restoration is conditioned on the additional input $\mathbf{c}$, which is the degraded face image, guiding the generation process. For each diffusion step $t$, the noisy latent $\mathbf{z}_{t}$ is processed together with the control condition $\mathbf{c}$, and null prompts [""].
We train the ControlNet by minimizing the $L_{2}$ loss between the predicted noise $\epsilon_{\theta}$ and the added noise $\epsilon$ ($\epsilon\sim\mathcal{N}(0,I)$).
The optimization objective is:

$$ $\mathcal{L}_{noise}=\mathbb{E}_{\mathbf{z}_{0},t,\mathbf{c},\epsilon\sim \mathcal{N}(0,I)}\left[\|\epsilon-\epsilon_{\theta}(\mathbf{z}_{t},t,[""], \mathbf{c})\|_{2}^{2}\right].$ (1) $$

Specifically, we freeze the parameters of our fine-tuned SDXL model to preserve the enhanced facial priors and its original natural image priors. We initialize ControlNet with the encoder from our fine-tuned SDXL model while solely training ControlNet.

Figure: Figure 4: An overview of Stage II. Denoising UNet, carried over from Stage I, maintains its facial priors by freezing its parameters, while ControlNet acts as an adapter for handling degraded inputs.
Refer to caption: x4.png

Time-aware Latent Facial Feature Loss.
Reducing incorrect generation is crucial for authentic face restoration, as humans are sensitive to key facial features like eyes and mouths.
However, the MSE loss (Eq. 1) used to train the ControlNet only provides a holistic constraint, where both the background and face of the degraded image equally influence optimization.
Thanks to the spatially invariant features of the conditioning embedding module in ControlNet, the latent space retains spatial dimensions . This allows for pixel-level constraints in the latent space.

To enhance key facial features, we propose a time-aware latent facial feature loss that provides additional constraints on the eyes and mouth. Inspired by GFP-GAN , we train separate facial feature discriminators to ensure these regions in the restored results match natural distributions.
Unlike GFP-GAN, our method incorporates the diffusion and denoising process of DDPMs, considering time as a variable.
Previous studies; ; show that during denoising, the generated results evolve from rough shapes to high-resolution images.
Therefore, using shared model weights for various time steps is not ideal.
Given the predicted noise $\epsilon_{\theta}$, sampled diffusion step $t$, and noisy latent $\mathbf{z}_{t}$, we can estimate the predicted latent $\mathbf{z}_{0}^{\text{pred}}$ according to the closed form formulation in DDIM  as:

$$ $\footnotesize\mathbf{z}_{0}^{\text{pred}}=\frac{\mathbf{z}_{t}-\sqrt{\beta_{ \text{prod}_{t}}}\cdot\epsilon_{\theta}}{\sqrt{\alpha_{\text{prod}_{t}}}}, \alpha_{\text{prod}_{t}}=\prod_{i=1}^{t}\alpha_{i}\quad,\quad\beta_{\text{prod }_{t}}=\prod_{i=1}^{t}\beta_{i}$ (2) $$

where $\alpha_{i}$ is the noise decay factor at each diffusion step and $\beta_{i}$ is the noise variance schedule.
We then locate these two regions with pertrained face landmark detection network  from the ground-truth (GT) image, and transform these pixel level’s location to the latent space ones ($POS_{eyes}$, $POS_{mouth}$) by downsampling with a factor of eight.
With the latent space position of eyes and the mouth, we crop these regions from the predicted latent $\mathbf{z}_{0}^{\text{pred}}$ and the latent of HQ image $z_{0}$ respectively to obtain the facial feature patches, $P_{eye}=\{p_{eye},p_{eye}^{pred}\}$ and $P_{mouth}=\{p_{mouth},p_{mouth}^{pred}\}$.
Inspired by the logit-normal sampling in StableDiffusion 3   and the finding in Fig. 5, our time-aware latent facial feature loss focus on the intermediate steps when the major shape of eyes and mouth arise via assigning higher weight, as follow:

$$ $\text{Weight}P=\frac{1}{s\sqrt{2\pi}}\frac{1}{t(1-t)}\exp\left(-\frac{(\mathrm {logit}(t)-m)^{2}}{2s^{2}}\right),$ (3) $$

where $\mathrm{logit}(t)=log\frac{1}{t(1-t)}$, $m$ and $s$ are the location parameter and scale parameter, respectively.
The time-aware latent facial feature loss is defined as follows:

$$ $\displaystyle\mathcal{L}_{facial}=\sum_{P\in{P_{\text{eye}},P_{\text{mouth}}}} \text{Weight}P\Big{(}$ $\displaystyle\lambda_{d}\mathbb{E}_{p^{\text{pred}}}\left[\log(1-\mathbb{D}_{P }(p^{\text{pred}}))\right]$ (4) $\displaystyle+\lambda_{s}\left|\text{Gram}(\psi(p^{\text{pred}}))-\text{Gram}( \psi(p))\right|\Big{)},$ $$

where $\mathbb{D}_{P}$ refers to the discriminators for different facial regions, specifically $\mathbb{D}_{eyes}$ and $\mathbb{D}_{mouth}$. $\psi$ represents multi-scale features of the regional facial feature discriminator. Gram() operation refers to calculating the Gram matrix static
$\lambda_{d}$ and $\lambda_{s}$ are the weights of the discriminative loss and the style loss, respectively.
The total loss function of AuthFace is defined as $\mathcal{L}_{\text{total}}=\mathcal{L}_{noise}+\mathcal{L}_{facial}$.

Figure: Figure 5: Visualization of the diffusion process at different steps. In the early steps (t = 999 - 599), the main content of the images is predominantly noise, with key facial features obscured. In the later steps (t = 61 - 0), the shapes of key facial features become fixed, with minimal changes.
Refer to caption: x5.png

Figure: Figure 6: Qualitative results on CelebA-Test dataset. Red box areas in \engordnumber1 row highlight the detailed skin texture and eyebrows achieved by our method. Zoom in for details.
Refer to caption: x6.png

## 4 Experiments

### 4.1 Experimental Settings

Implementation Details:
Our base model is initialized from StableDiffusion-XL (SDXL) , and we fine-tune the entire U-Net from this base model.
We employ AdamW  optimizer with the learning rate of $5e-7$ during the fintuing process, where the batch size and the training iteration are set to 96 and 50k, respectively.
We apply the same optimizer with the learning rate of $2e-5$ with the batch size 48 for training ControlNet.
All experiments are conducted on four NVIDIA L40s GPUs in the resolution of 1024 $\times$ 1024 for fintuning model and 512 $\times$ 512 for training ControlNet.

Training and Test Dataset:
The training dataset for the fine-tuning process of our face-oriented model comprises 1.5K high-quality face images, each enriched with detailed photographic annotations.
For training our AuthFace network, we resize the FFHQ dataset  from a resolution of 1024×1024 to 512×512.
To form training pairs, we follow the settings, including degradation types and degrees, as outlined in previous methods ; .
Following ; ; , we evaluate our method on a synthetic dataset, CelebA-Test , and three real-world datasets: LFW-Test , WebPhoto-Test , and WIDER-Test .

Metrics:
To evaluate our method’s performance on the Celeb-A dataset with ground truth, we use PSNR , SSIM , and LPIPS .
Besides, we follow SUPIR  introducing non-reference image quality assessment metrics, MUSIQ , ManIQA  ,ClipIQA , and FID .

### 4.2 Comparison and Evaluation

We compare our method with SOTA BFR methods in three different categories: (I) GAN-based methods, including GFP-GAN  and PSFR-GAN ;
(II) Codebook-based method, including CodeFormer ;
(III) Diffusion-based methods, including DR2  and BFRffusion ;
Notably, we compare our method with SOTA IR method, SUPIR , which is also based on SDXL .
All methods are tested with official codes.

Figure: Figure 7: Qualitative results on real-world datasets. Results in \engordnumber1 row are from LFW-Test dataset . Results in \engordnumber2 row come from WebPhoto-Test dataset . Results in \engordnumber3 row are from WIDER-Test dataset including a zoomed-in view of the skin highlighted in red box areas. Zoom in for details.
Refer to caption: x7.png

**Table 1: Quantitative results for blind face restoration on both synthetic and real-world datasets. The highest result is highlighted in red while the second highest result is highlighted in blue.**
| Datasets | Metrics | GFPGAN | PSFRGAN | CodeFormer | DR2 | BFRffusion | SUPIR | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CelebA | PSNR$\uparrow$ | 24.65 | 24.68 | 25.15 | 21.43 | 26.19 | 25.00 | 25.57 |
| SSIM$\uparrow$ | 0.6669 | 0.6322 | 0.6647 | 0.5943 | 0.6829 | 0.6487 | 0.6768 |  |
| LPIPS$\downarrow$ | 0.2308 | 0.2943 | 0.2269 | 0.3443 | 0.2272 | 0.2716 | 0.2143 |  |
| MANIQA$\uparrow$ | 0.5633 | 0.5103 | 0.5546 | 0.5397 | 0.5964 | 0.5233 | 0.6624 |  |
| MUSIQ$\uparrow$ | 73.91 | 73.32 | 75.56 | 70.42 | 71.90 | 72.92 | 75.76 |  |
| FID$\downarrow$ | 42.62 | 47.59 | 52.43 | 56.59 | 40.74 | 35.01 | 50.93 |  |
| CLIPIQA$\uparrow$ | 0.6790 | 0.6310 | 0.6716 | 0.5770 | 0.6863 | 0.6103 | 0.7065 |  |
| LFW | MANIQA$\uparrow$ | 0.5514 | 0.5176 | 0.5415 | 0.5326 | 0.5528 | 0.4768 | 0.6431 |
| MUSIQ$\uparrow$ | 73.58 | 73.60 | 75.49 | 71.04 | 69.86 | 69.90 | 75.87 |  |
| FID$\downarrow$ | 49.96 | 51.89 | 52.36 | 47.14 | 49.92 | 41.98 | 45.29 |  |
| CLIPIQA$\uparrow$ | 0.6994 | 0.6471 | 0.6893 | 0.6069 | 0.6969 | 0.5931 | 0.7350 |  |
| WebPhoto | MANIQA$\uparrow$ | 0.5351 | 0.4793 | 0.5241 | 0.4843 | 0.4721 | 0.4394 | 0.5860 |
| MUSIQ$\uparrow$ | 72.13 | 71.67 | 74.01 | 67.19 | 61.78 | 65.67 | 74.11 |  |
| FID$\downarrow$ | 87.35 | 88.45 | 83.19 | 107.86 | 84.29 | 73.44 | 90.04 |  |
| CLIPIQA$\uparrow$ | 0.6888 | 0.6366 | 0.6922 | 0.5690 | 0.6308 | 0.5767 | 0.6964 |  |
| WIDER | MANIQA$\uparrow$ | 0.5289 | 0.4925 | 0.5119 | 0.4989 | 0.4923 | 0.4522 | 0.5941 |
| MUSIQ$\uparrow$ | 72.80 | 71.50 | 73.40 | 67.18 | 61.87 | 67.19 | 74.59 |  |
| FID$\downarrow$ | 39.49 | 49.84 | 38.78 | 45.27 | 55.22 | 42.61 | 36.10 |  |
| CLIPIQA$\uparrow$ | 0.7101 | 0.6482 | 0.6990 | 0.5943 | 0.6789 | 0.6093 | 0.7306 |  |

Comparison on Synthetic Dataset:
Quantitative results in Tab. 1 showcase our method’s superior performance on the CelebA-Test dataset, outperforming baselines in all non-reference image quality assessment metrics except FID.
Notably, we achieve SOTA performance in terms of the LPIPS score.
This marks a significant validation of our approach for the BFR task.
Qualitatively, as depicted in Fig. 6, our AuthFace achieves authentic face restoration. Specifically, all methods except ours fail to recover the face paint in the first example, and our method results in the best hair detail. The second example involves recovering the side face, which is one of the most challenging cases in BFR . GFP-GAN, CodeFormer, and BFRfusion fail to restore authentic mouth details (yellow circle), while the results from GFP-GAN, CodeFormer, DR2, and SUPIR lose the right eye (red box). Only our method produces realistic results in these key regions, with best details in the eyebrow and skin texture.

Comparison on Real-world Dataset:
The robustness of our method is demonstrated by its SOTA performance in all metrics and real-world datasets, except for the FID score in the LFW-Test and WebPhoto-Test datasets, as shown in Tab. 1. Notably, the MANIQA score in the LFW-Test dataset exceeds the baselines by 0.09.
In the LFW-Test dataset, GFP-GAN, CodeFormer, and DR2 fail to reconstruct realistic results in the eye regions due to incorrect generation at the edges of glasses (see the red box in the first row of Fig. 7).
In the second row of Fig. 7, our method outperforms others by accurately reconstructing both the upper and lower teeth without the artifacts around the hands, highlighted in a yellow circle.
In the WebPhoto-Test dataset, our approach not only precisely reconstructs details such as helmets and goatees but also delivers the best skin texture, as showcased in the red box areas. More visualization results are in the appendix and the supplmat.

### 4.3 Ablation Study

**Table 2: Ablation studies of variant generative diffusion prior and time-aware latent facial feature loss. The highest result is highlighted in red while the second highest result is highlighted in blue.**
| Dataset | Exp. | Diffusion Prior | $\mathcal{L}_{facial}$ | Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SDXL | Ours | Const. | Time-aware | PSNR$\uparrow$ | MANIQA$\uparrow$ | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$ |  |  |
| CelebA | (a) | ✓ |  |  |  | 24.39 | 0.5781 | 69.25 | 0.6465 |
| (b) |  | ✓ |  |  | 25.59 | 0.5057 | 74.42 | 0.7088 |  |
| (c) |  | ✓ | ✓ |  | 23.95 | 0.6449 | 73.66 | 0.6821 |  |
| (d) |  | ✓ |  | ✓ | 25.57 | 0.6624 | 75.76 | 0.7065 |  |
| WebPhoto | (a) | ✓ |  |  |  | - | 0.5252 | 67.01 | 0.6276 |
| (b) |  | ✓ |  |  | - | 0.5810 | 72.35 | 0.6833 |  |
| (c) |  | ✓ | ✓ |  | - | 0.5767 | 68.52 | 0.6861 |  |
| (d) |  | ✓ |  | ✓ | - | 0.5860 | 74.11 | 0.6964 |  |

We conduct ablation studies on CelebA-Test and WebPhoto-Test to assess the effectiveness of every part of our method.

Figure: Figure 8: Visualization of ablation results. \engordnumber1 row and \engordnumber2 row are the examples from CelebA-Test and WebPhoto-Test datasets, respectively. Please zoom in for more details.
Refer to caption: x8.png

Effectiveness of Face-oriented Fine-Tuning:
We conducted an ablation study to evaluate the effectiveness of face-oriented tuning, as shown in Tab. 2 (a) and (b). In experiment (a), the original SDXL is used as the base model, and ControlNet is initialized with it. In experiment (b), the fine-tuned SDXL is used as the base model, and ControlNet is initialized with this fine-tuned version.
Except for the MANIQA score on the CelebA-Test dataset, experiment (b) consistently outperforms experiment (a), highlighting the necessity of face-oriented tuning for the generative diffusion prior. Notably, in the WebPhoto-Test dataset, experiment (b) excels across all metrics, including CLIPIQA (0.6833 vs. 0.6276), MUSIQ (72.35 vs. 67.01), and MANIQA (0.5810 vs. 0.5252).
Experiment (b) enhances facial details such as eyebrows and skin texture (red box, first row in Fig. 8) and eyelashes (red box, second row). Additionally, it reduces errors in key facial features, resulting in clearer eyes (red box, first row) and better restoration of teeth (blue box, second row).

Effectiveness of Time-aware Latent Facial Feature Loss:
To evaluate the effectiveness of the time-aware latent facial feature loss, we conducted experiments as shown in Tab. 2 (b), (c), and (d). Using constant weights for various time steps (experiment (c)) negatively impacts optimization, resulting in performance drops across most metrics, except for the MANIQA score on the CelebA-Test dataset compared to experiment (b).
By focusing on steps when the major shapes of eyes and mouth emerge and assigning higher weights during these steps, our time-aware loss achieves the best performance on both synthetic and real-world datasets, except for the PSNR and CLIPIQA scores on the CelebA dataset. As shown in Fig. 8, using latent space facial feature loss in experiments (c) and (d) improves the restoration of eyes and mouth (see the red box in the first row and the blue box in the second row).
Notably, the time-aware strategy in experiment (d) not only reduces artifacts (as indicated by the blue and red arrows in Fig. 8) but also enhances details (e.g., the sharp edge of the glasses in the first row and the delicate skin texture and eyebrows in the second row).

## 5 Conclusion

This paper presented a new approach for achieving authentic face restoration by avoiding incorrect generations and enhancing facial details.
Specifically, we proposed a face-oriented restoration-tuning paradigm to fine-tune the pretrained T2I model with high-quality face images, enabling the pretrained T2I model, SDXL, to develop a prior for facial details.
Utilizing this face-oriented generative diffusion prior, we introduced AuthFace for the blind face restoration task, achieving authentic face restoration.
Additionally, we introduced the time-aware latent facial feature loss to further improve the robustness of restoration in key facial features.
Experimental results demonstrate the superiority and effectiveness of our method.

Limitation and Future Work:
The process of collecting high-quality images requires significant human resources to filter out low-quality images.
We plan to develop an aesthetic-oriented image quality assessment network to reduce labor costs.

## Appendix A Details of Face-oriented Tuning Datasets

Existing datasets like FFHQ  suffer from issues such as JPEG degradation, blur, and Gaussian noise .
We have compiled a collection of 1,500 high-quality images, exceeding resolutions of 8K, captured by professional photographers. These improvements address the quality limitations of traditional datasets, with examples showcased in Fig. 9 and Fig. 10.

We detail our dataset collection and annotation process in Fig. 11, which depicts the entire pipeline, including data collection, reviewing, and tagging. Apart from sourcing from Unsplash, we collaborate with professional photographers to collect studio-based images of Asian descent. These professionals also retouch each image to enhance skin texture while removing blemishes. All images undergo manual screening to ensure they are neither over-smoothed nor contain pepper noise, preserving detailed facial features.

Figure: Figure 9: Example of FFHQ datasets with noticeable blur and noise. Zoom in for more details.
Refer to caption: x9.png

Figure: Figure 10: Example of our HQ datasets with details of skin texture. Zoom in for more details.
Refer to caption: x10.png

Figure: Figure 11: Illustration of collecting high-quality datasets for face-oriented fine-tuning.
Refer to caption: x11.png

Figure: Figure 12: More qualitative comparison of the text-to-image task between (a) using the pretrained SDXL; (b) fine-tuning the SDXL with only semantic tags; and (c) fine-tuning with both semantic and our proposed photography-guided tags.
Refer to caption: x12.png

## Appendix B More ablation studies

Effectiveness of Face-oriented Fine-Tuning:
To demonstrate the effectiveness of face-oriented fine-tuning, we conduct additional experiments on the task of text-to-image and provide more qualitative results on LFW-Test for the task of blind face restoration.

First, to underscore the value of our photography-guided annotation, we conducted ablation studies under three settings: (a) using the pretrained SDXL; (b) fine-tuning the SDXL with only semantic tags; and (c) fine-tuning with both semantic and our proposed photography-guided tags. We evaluated authenticity with FID and Human Preference Score v2 (Hpsv2)  and through a user study with 20 participants who assessed images from 10 prompts. According to Tab. 3, setting (c) not only scored the highest on FID and Hpsv2 but also received the best average ranking, indicating that users consistently preferred images produced by models fine-tuned with both semantic and our proposed photography-guided tags.
Besides, Fig. 12 demonstrates that our face-oriented fine-tuning successfully equips SDXL with dedicated facial details.

To further evaluate the effectiveness of face-oriented fine-tuning
We provide more qualitative results on LFW-Test as shown in Fig. 13.
In experiment (a), the original SDXL is used as the base model, and ControlNet is initialized with it. In experiment (b), the fine-tuned SDXL is used as the base model, and ControlNet is initialized with this fine-tuned version.
Notably, the results of Exp. b has the best visual experience enjoying authentic facial details, such as the dedicated skin texture and hair, which demonstrates the importance of face-oriented fine-tuning.

**Table 3: Ablation studies of variant tags used in fine-tuning. The highest result is highlighted in The highest result is highlighted in red, while the second highest result is are blue for clarity.**
| Exp | FID $\downarrow$ | Hpsv2  $\uparrow$ | User study rank $\downarrow$ |
| --- | --- | --- | --- |
| (a) | 95.13 | 0.2637 | 2.53 |
| (b) | 62.90 | 0.2712 | 2.17 |
| (c) | 51.09 | 0.2903 | 1.33 |

Figure: Figure 13: Visualization of ablation results on LFW-Test dataset. Zoom in for more details.
Refer to caption: x13.png

Different hyper meter of time-aware latent facial feature loss:
Through our face detection experiments, we identified that the timestep where the average confidence score reaches 0.5 is 0.32 (normalized) in the FFHQ dataset. We designated this timestep as the most critical, assigning it the highest weight. Therefore, we set the m as -0.5 and s as 1.0, where Eq. 3 in the main paper peaks at t=0.37.
We conduct an ablation study on these hyperparameters m and s as detailed in Tab. 4 and we also provide Fig. 14 showing the weight distributions of different hypermeter.
This study confirms that our chosen settings yield the best outcomes, thus validating the robustness of our experimental approach.

**Table 4: Ablation studies of variant generative diffusion prior and time-aware latent facial feature loss. The highest result is highlighted in red while the second highest result is highlighted in blue.**
| Dataset | Exp. | Location<br>parameter m | Metrics |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | MANIQA$\uparrow$ | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$ |  |  |  |
| CelebA | (a) | m = -0.5 | 25.57 | 0.6624 | 75.76 | 0.7065 |
| (b) | m = 0.0 | 25.40 | 0.6399 | 74.92 | 0.6786 |  |
| (c) | m = 0.5 | 25.37 | 0.6462 | 74.67 | 0.6882 |  |
| (d) | s = 0.5 | 25.42 | 0.6440 | 74.72 | 0.6794 |  |
| WebPhoto | (a) | m = -0.5 | - | 0.5860 | 74.11 | 0.6964 |
| (b) | m = 0.0 | - | 0.5760 | 73.51 | 0.6657 |  |
| (c) | m = 0.5 | - | 0.5829 | 73.40 | 0.6755 |  |
|  | (d) | s = 0.5 | - | 0.5756 | 73.06 | 0.6686 |

Figure: Figure 14: The weight distributions of different hyper meters of time-aware latent facial feature loss.
Refer to caption: x14.png

## Appendix C User Study:

We implemented an AB-test with 20 participants using 20 facial images from our datasets to gauge human perception of our method compared to six baselines. Participants were shown two images, labeled A and B, and asked to choose from three responses: "A is better," "B is better," or "Both are equally good," with image positions randomized.
As detailed in Tab. 5, our method was preferred, indicating it produces results that are both more authentic and visually appealing.

**Table 5: Results of user study. "Ours" is the percentage that our result is preferred, "Others" is the percentage that some other method is preferred, "Same" is the percentage that the users have no preference.**
| Methods | Others | Same | Ours |
| --- | --- | --- | --- |
| GFPGAN | 26.75% | 2.5% | 70.75% |
| PSFRGAN | 7.5% | 0.75% | 91.75% |
| CodeFormer | 23.5% | 1.25% | 75.25% |
| DR2 | 10.5% | 0.5% | 89% |
| BFRffusion | 25.25% | 2% | 72.75% |
| SUPIR | 22.5% | 1% | 76.5% |

## Appendix D Running time

We evaluated our method’s runtime on a single NVIDIA L40s GPU, as detailed in Tab. 6. Notably, although both our method and SUPIR utilize SDXL, SUPIR demands significantly more time due to its initial enhancement phase and the use of LLaVA for text prompts. This supports our decision to omit details prompts in stage 2.

**Table 6: Running time of different networks. Please note that all methods are evaluated in 512$\times$ 512 input images, while DR2 reconstructs high-quality face images at 256$\times$ 256 and upscale to 512$\times$512 with an enchantment module according to its official setting.**
| Method | PSFRGAN | GFP-GAN | CodeFormer | DR2 | BFRffusion | SUPIR | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Time (s) | 0.06 | 0.17 | 0.01 | 0.49 | 2.89 | 10.36 | 5.25 |

## Appendix E More visualization results

In this section, we provide more visual comparisons with state-of-the-art methods in CelebA-Test, LFT-Test, WebPhoto-Test, and WIDER-Test datasets as shown in Fig. 15 and Fig. 16.

Figure: Figure 15: Visualization results in CelebA-Test and LFW-Test dataset including a zoomed-in view of the skin highlighted in red box areas. Zoom in for more details.
Refer to caption: x15.png

Figure: Figure 16: Visualization results in WebPhoto-Test and WIDER-Test dataset. Results in WIDER-Test dataset (\engordnumber2 case) include a zoomed-in view of the skin highlighted in red box areas. Zoom in for more details.
Refer to caption: x16.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. As mentioned in Weaknesses, more qualitative and quantitative comparisons with Sota methods are needed.
2. It is recommended to replace the image in Figure 1 to make this challenge more convincing.

It was a bit of work to collect this dataset, and I would raise the score if the author could fix the mentioned problems.
soundness: 3
strengths: 1. This paper proposes a New High-quality Dataset, with a resolution exceeding 8K.
2. This paper proposes a novel loss to enhance mouth and eye regions, specifically designed for face restoration tasks.
3. Quantitative experiment results show excellent performance for blind face restoration on synthetic and real-world datasets.
summary: To tackle the problem of missing details and artifacts, this paper proposes Authface, a novel BFR method with face-oriented generative diffusion prior, designed to restore highly authentic face images. This paper proposes an HQ dataset containing 1.5K images with a resolution exceeding 8K. Based on it, the pretrained T2I model is fine-tuned with semantic and photographic prompts. After tuning, authors leverage ControlNet and a novel time-aware latent facial feature loss to achieve highly authentic face restoration.
weaknesses: 1. The visual image in Figure 1 does not convince me that Authface performs better than SUPIR in the task of missing details and incorrect details. For example, in the red box in the first row, Authface lacks more details than SUPIR.
2. In qualitative experiments, Authface does not perform as well as BFRffusion. For example, in the fourth row of Figure 6, the eyes of BFRffusion are closer to GT than Aythface, and in the first two rows of Figure 7, there is little difference between the two methods. In fact, the image quality of the teeth of BFRffusion is better.
3. More Sota compared methods. Please show more comparisons on more Sota methods, like DiffBIR and StableSR, which also leverage ControlNet as a part of the framework.
4. Contribution. As mentioned above, DiffBIR, StableSR, and so on, have already enhanced the generative capabilities of pretrained T2I models for BFR, which is not a novel research direction.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Do the authors analyze how the number of high-quality face images enables the pretrained T2I model to produce a more accurate face prior? I think that 1.5K high-quality face images may not be sufficient for effectively fine-tuning the T2I model.
2. Can the authors provide more universal evaluations (refer to the weaknesses section)?
soundness: 3
strengths: 1. The idea of using a restoration-tuning pipeline guided by quality-first annotations to enhance facial features to guide the  BFR is interesting.
2. Overall, the writing of this paper is good and easy to follow.
3. The experimental results look good on some benchmark datasets.
summary: This work focuses on the problem of blind face restoration (BFR). The authors aim to overcome the limitations of existing methods using pre-trained text-to-image diffusion models and propose AuthFace for BFR. Specifically, AuthFace leverages a dataset of 1,500 high-quality images taken by professional photographers to create a face-oriented generative diffusion prior. It employs a restoration-tuning pipeline guided by quality-first annotations to enhance facial features and introduces a time-aware latent facial feature loss to minimize artifacts in critical areas. Experimental results demonstrate its superior performance on synthetic and real-world BFR datasets.
weaknesses: 1.  While using high-quality face images to fine-tune the pre-trained model is likely important for generating better face priors, the proposed method seems to rely on an additional 1,500 high-quality images, which may be only a trick compared to methods like BFRfusion and SUPIR.
2. In Fig. 6 and Fig. 7, the restored face images exhibit artifacts. For instance, the left eye is closed in the ground truth (GT) image, but in the restoration produced by AuthFace, the eye appears partially open.
3. Some universal evaluations should be conducted. The authors do not use the pre-trained T2I models (fine-tuned after incorporating the 1,500 high-quality images) on existing methods, which could validate the effectiveness of the proposed approach in comparison to those methods.
4. The proposed method seems very heavy. Some complexity results should be provided, such as parameters, FLOPs, and runtime. They can help to more fairly assess the true effectiveness of the method.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: please see Weaknesses
soundness: 3
strengths: 1. A dataset of 1.5K HQ face images is proposed.
2. The generated faces look good and closer to a real face.
summary: The paper introduces AuthFace, a framework that achieves highly authentic face restoration by leveraging a face-oriented generative diffusion prior. This work first proposes a dataset of 1.5K HQ face images to fine-tune the pretrained T2I models. Then, the ControlNet is used for training a restoration model for LQ to HQ, where a time-aware latent facial feature loss is proposed to train this restoration model.
weaknesses: My concerns are as follows:
1. About the proposed dataset. After a careful reading, I observe that the proposed dataset is only used in stage one, i.e., used to fine-tune the prestrained T2I model, which means that the dataset seems to be solving the problem of generating faces only, not for the BFR task. This can also be seen in Figure 4, where the authors are using an example from the FFHQ dataset to plot.
2. Concerns about methodological innovativeness. The methodological contribution of this work is simply to fine-tune the existing T2I model, i.e., SDXL and ControlNet, and the novelty may be inadequate.
3. For the proposed time-aware latent facial feature loss, why consider only the eyes and mouth? For the facial structures, the nose is very important.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: See the Weaknesses.
soundness: 2
strengths: 1. This paper introduces a high-quality face dataset tailored for restoration tasks, enhancing model training with detailed facial features.
2. This method achieves strong results in restoring realistic facial details, outperforming existing methods in key areas like eyes and mouth across various datasets.
summary: The paper proposes a method for blind face restoration. It fine-tunes generative models with high-quality face data and uses a time-aware facial feature loss, improving detail restoration in key areas like eyes and mouth. AuthFace outperforms current methods in both synthetic and real-world tests.
weaknesses: 1. In Figure 2(b), the fine-tuned model shows only marginal improvements over the baseline SDXL model, which brings into question whether the fine-tuning process effectively enhances facial detail restoration.
2. While the study emphasizes the use of photographic prompts during fine-tuning, it lacks specific experiments to verify if these prompts genuinely contribute to improved model performance.
3. The authors did not incorporate photographic prompts in the second stage of training or during testing. Exploring whether adding these prompts could further enhance the model’s restoration quality would be beneficial.
4. Since AuthFace uses a specially collected, high-quality face dataset, the comparison with other methods might not be entirely fair. Competing methods didn’t have access to this dataset, which could make it hard to tell if AuthFace’s improvements come from the method itself or just from having better data.

Meta-review: The paper proposes a method for blind face restoration by fine-tuning a pretrained text-to-image (T2I) model on high-quality facial data and subsequently training a ControlNet for face restoration. To support this approach, the authors collected a new high-quality face dataset with resolutions exceeding 8K, along with photographic focus captions.

Strengths
* The paper introduces a high-quality facial dataset with exceptionally large resolution, which could benefit face restoration research.
* The method demonstrates solid quantitative performance on established metrics and benchmarks.

Weaknesses
* The qualitative results are not consistently convincing, with some reviewers noting that the restored faces lack sufficient details or contain artifacts.
* The comparison with state-of-the-art methods is incomplete, and the fairness of the comparisons is questionable due to the additional training data used by the proposed method.
* The technical contribution is limited. The work relies heavily on existing methods that has been well-studied.

Despite the introduction of a high-quality dataset and quantitative improvements, multiple reviewers expressed concerns about the limited technical novelty, unfair comparisons, and less-than-convincing qualitative improvements. The authors’ rebuttal did not fully address these concerns, particularly regarding the fairness of the evaluation and the practical effectiveness of the approach. As a result, the paper's overall contribution is considered insufficient to warrant acceptance.

Decision: Reject