Dual Associated Encoder for Face Restoration
Authors: YU-JU TSAI
Yu-Lun Liu
Lu Qi
Kelvin C.K. Chan
Ming-Hsuan Yang
Venue: ICLR 2024 poster (2024)
Keywords: Blind Face Restoration
Vector Quantization
Codebook Prior
Feature Encoding
Feature Representation
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Restoring facial details from low-quality (LQ) images has remained challenging due to the nature of the problem caused by various degradations in the wild. 
The codebook prior has been proposed to address the ill-posed problems by leveraging an autoencoder and learned codebook of high-quality (HQ) features, achieving remarkable quality.
However, existing approaches in this paradigm frequently depend on a single encoder pre-trained on HQ data for restoring HQ images, disregarding the domain gap and distinct feature representations between LQ and HQ images.
As a result, encoding LQ inputs with the same encoder could be insufficient, resulting in imprecise feature representation and leading to suboptimal performance.
To tackle this problem, we propose a novel dual-branch framework named $\textit{DAEFR}$. Our method introduces an auxiliary LQ branch that extracts domain-specific information from the LQ inputs. 
Additionally, we incorporate association training to promote effective synergy between the two branches, enhancing code prediction and restoration quality.
We evaluate the effectiveness of DAEFR on both synthetic and real-world datasets, demonstrating its superior performance in restoring facial details.
Project page: https://liagm.github.io/DAEFR/

Published as a conference paper at ICLR 2024
DUAL ASSOCIATED ENCODER FOR FACE RESTORATION
Yu-Ju Tsai"
Yu-Lun Liu
Lu Qi
Kelvin C.K. Chan
Ming-Hsuan Yang
UC Merced
National Yang Ming Chiao Tung University
Google Research
ABSTRACT
Restoring facial details from low-quality (LQ) images has remained challenging due to the nature of the problem caused by various degradations in the wild. The codebook prior has been proposed to address the ill-posed problems by leveraging an autoencoder and learned codebook of high-quality (HQ) features, achieving remarkable quality. However, existing approaches in this paradigm frequently depend on a single encoder pre-trained on HQ data for restoring HQ images, disregarding the domain gap and distinct feature representations between LQ and HQ images. As a result, encoding LQ inputs with the same encoder could be insufficient, resulting in imprecise feature representation and leading to suboptimal performance. To tackle this problem, we propose a novel dual- branch framework named DAEFR. Our method introduces an auxiliary LQ branch that extracts domain- specific information from the LQ inputs. Additionally, we incorporate association training promote effective synergy between the two branches, enhancing code prediction and restoration quality. We evaluate the effectiveness of DAEFR on both synthetic and real-world datasets, demonstrating its superior performance in restoring facial details. Project page: https://iagg.. https///iagmmgithuui////// github.io/DAEFR.
INTRODUCTION
Blind face restoration presents a formidable challenge as it entails restoring facial images that are degraded by complex and unknown sources of degradation. This degradation process often results in the loss of valuable information. It introduces a significant domain gap, making it arduous to restore the facial image to its original quality with high accuracy. The task itself is inherently ill- posed, and prior works rely on leveraging different priors to enhance the performance of restoration algorithms. Notably, the codebook prior emerges as promising solution, showcasing its effectiveness in generating satisfactory results within such challenging scenarios. By incorporating codebook prior, these approaches demonstrate improved performance in blind face restoration tasks.
Existing codebook methods (Zhou al., 2022; Gu al., 2022; Wang et al., 2022b) address the inclusion of low-quality (LQ) images by adjusting the encoder, which is pre-trained on high-quality (HQ) data, depicted in Fig. (a)) However, this approach introduces domain bias due domain gap and overlooks the distinct feature representations between the encoder and LQ images. Consequently, employing the pre-trained encoder encode LQ information may potentially result in imprecise feature representation and lead to suboptimal performance in the restoration process. Furthermore, these approaches neglect the L.Q domain's 's inherent visual characteristies and statistical properties, which provide valuable information for enhancing the restoration process.
To overcome these limitations, we present novel framework called DAEFR, which incorporates dedicated auxiliary branch for LQ information encoding (Fig. (b)).. This auxiliary branch is exclusively trained on LQQ data, alleviating domain bias and acquiring a precise feature representation of the LQ domain. By integrating the auxiliary branch into framework, enhanced LQ representation identity and content information from the original LQ images, which supplement the lost information. DAEFR utilizes both HQ and auxiliary LQ encoders to capture domain-specific information, thereby enhancing the representation of image content.
effectively harness the
The core idea of our method is effectively fuse visual information from two branches. If we naively combine the HQ and LQ features, the existing domain gap causes misalignment in feature
"Corresponding author

---

Published as a conference paper at ICLR 2024
Cookbook Leanin
Codabock
HQ
BQQ tnnecer
11Q
HQ
HQ
Encoder
Docoder
HQ
Association
LQ
LG Ebooder
LQ
U2 Oadkaaak
Rarteetiin Stage
Pretesined Weight
RiO Enooder
LQ
Enooder
Restoration Pipeline
HQ
LQ
Festurre Fusion
Restonation Pipeline
HQ
IO Ebcodan
(a) Framework of existing methods
(b) Our proposed framework
Figure 1: Comparison to existing framework. (a) Existing codebook prior approaches learn an encoder in the first stage. During the restoration stage, these approaches utilize LQ images to fine-tune the encoder using pre-trained weights obtained from HQ images. However, this approach introduces domain bias due domain gap and overlooks the distinct feature representations between the encoder and LQ input images. (b) In the codebook learning stage, we propose the integration of an auxiliary branch specifically designed for encoding LQ information. This auxiliary branch is trained exclusively using IQ data to address domain bias and obtain precise feature representation. Furthermore, we introduce an association stage and feature fusion module enhance the integration of information from both encoders and assist our restoration pipeline.
representation, rendering them challenging to utilize effectively. Similar to CLIPP (Radford et al., 2021), incorporate association stage after extracting features from both branches. This stage aligns the features to a shared domain, effectively bridging the gap between LQ and HQ features and facillitating more comprehensive and integrated representation of feature information. To ensure the effective fusion of information, we employ a multi-head cross-attentinn associated encoders that can adequately represent both the HQ and LQ domains. This module enables us to merge the features from these associated encoders and generate fused features. Through the fusion process, where the features from the associated encoders are combined with the LQ domain information, our approach effectively mitigates the challenges of domain gap and information loss and leverages the complementary aspects of the HQ and LQ domains, leading to improved restoration results. The main contributions of this work are:
module after acquiring the
We introduce an auxiliary LQ encoder to construct more precise feature representation that adeptly captures the unique visual characteristics and statistical properties inherent to the LQ domain. By incorporating information from a hybrid domain, our association and feature fusion methods effectively use the representation from the LQ domain and address the challenge of domain gap and information loss in image restoration, resulting in enhanced outcomes.
We propose novel approach, DAEFR, to address the challenging face restoration problem under severe degradation. We evaluate our method with extensive experiments and ablation studies and demonstrate its effectiveness with superior quantitative and qualitative performances.
2 RELATED WORK
Face Restoration. Blind face restoration techniques commonly exploit the structured characteristies of facial features and incorporate geometric priors to achieve desirable outcomes. Numerous methods propose the use of facial landmarks (Chen et al., 2018; Kim et al., 2019; Ma al., 2020; Zhang & Wu, 2022), face parsing maps (Chen et al., 2021; Shen et al., 2018: Yang et al., 2020), facial component heatmaps (Wang et al., 2019; Chen al., 2021; 2020; Kalarot al., 2020), 3D shapes (Hu et al., 2020; Ren et al., 2019; Zhu et al., 2022; Hu et al., 2021). However, accurately acquiring prior information from degraded faces poses a significant challenge, and relying solely on geometrie priors may not yield adequate details for HQ face restoration.
Various reference-basee methods have been developed to address these limitations (Dogan et al., 2019; Li et al., 2020; 2018). These methods typically rely on having reference images that share the same identity as the degraded input face. However, obtaining such reference images often impractical not readily available. Other approaches, such DFDNet (Li al., 2020), construct HQ facial component features dictionaries. However, these component-specinee dictionaries may lack the necessary information to restore certain facial regions, such as skin and hair.
2

---

Published as a conference paper at ICLR 2024
To address this issue, approaches utilize generative facial priors from pre-trained generators. such as StyleGAN2 (Karras et al.. 2019). These priors are employed through iterative latent optimization for GAN inversion (Gu et al., 2020; Menon et al., 2020) or direct latent encoding of degraded faces (Richardson et al., 2021). However, preserving high fidelity in the restored faces becomes challenging when projecting degraded faces into the continuous infinite latent space.
On the other hand, GLEAN (Chan et al., 2021; 2022), GPEN (Yang et al., 2021), GFPGAN (Wang al., 2021), GCFSR (He al., 2022), and Panini-Net (Wang al., 2022a) incorporate generative priors into encoder-decoder models, utilizing additional structural information from input images as guidance. Although these methods improve fidelity, they heavily rely on the guidance of the inputs. which can introduce artifacts when the images are severely corrupted.
Most recently, diffusion models for generating HQ content. Several approaches (Yue & Loy, 2022; Wang et al., 2023) exploit the effectiveness of the diffusion prior to restore LQ face images. However, these methods do not preserve the identity information present in the LQ images well.
et al., 2015; Ho et al., 2020) have been developed
Vector Quantized Codebook Prior. A vector-quantizee codebook is introduced in the VQ-VAE framework (Van Den Oord et al., 2017). Unlike continuous outputs, the encoder network this approach generates discrete outputs, and the codebook prior is learned rather than static. Subsequent research proposes various enhancements to codebook learning. VQVAE2 (Razavi et al., 2019) introduces a multi-scale codebook to improve image generation capabilities. On the other hand, VQGAN (Esser et al., 2021) 1) trains the codebook using an adversarial objective, enabling the codebook to achieve high perceptual quality.
Recently, codebook- based methods (Wang et al., 2022b; Gu et al., 2022; Zhou et al., 2022; Zhao et al., 2022) explore the of learned HQ dictionaries or codehooks that contain more generic and detailed information for face restoration. CodeFormer (Zhou et al., 2022) employs a transformer to establish the appropriate mapping between LQ features and code indices. Subsequently, it uses the code index to retrieve the corresponding feature in the codebook for image restoration. RestoreFormer (Wang al., 2022b) and VQFR (Gu al., 2022) attempt to directly incorporate LQ information with the codebook information based on the codebook prior. However, these methods may encounter severe degradation limitations, as the LQ information can negatively impact the HQ information derived from the codebook.
3 METHOD
Our primary objective is to mitigate the domain gap and information loss that emerge while restoring HQ images from I.Q images. This challenge has a substantial impact on the accuracy and effectiveness of the restoration process. To address this issue, we propose an innovative framework with an auxiliary LQ encoder incorporating domain- speciffc information from the LQ domain. Furthermore, we utilize feature association techniques between the HQ and LQ encoders to enhance restoration.
In our framework, we first create discrete codebooks for both the HQ and LQ domains and utilize vector quantization (Esser et al., 2021; Van Den Oord et al., 2017) to train quantized autoencoder via self-reconstruction (Sec. 3.1). Then, we introduce a feature association technique in a way similar to the CLIP model (Radford et al., 2021) to associate two encoders and aim to reduce the domain gap between the two domains (Sec. 3.2). In the next stage, a feature fusion module is trained using multi-head coossattenttion (MHCA) technique to combine the features extracted from the two associated encoders (EA and EAA). We employ the transformer to perform the code prediction process using the integrated information from both encoders, which predicts the relevant code elements in the HQ codebook. Subsequently, the decoder utilizes the restored code features to generate HQ images (Sec. 3.3). Our framework is illustrated in Fig. 2.
3.1 DISCRETE CODEBOOK LEARNING STAGE
Similar to VQGAN (Esser et al., 2021), approach involves encoding domain-specific information through an autoencoder and codebook, enabling the capture of domain characteristics during the training phase. Both HQ and LQ paths trained using the same settings ensure consistency in feature representation. Here, we present the HQ reconstruction path as an illustrative example, noting that the LQ reconstruction path follows identical procedure.
3

---

Published as a conference paper at ICLR 2024
Eateder
Decoder
PP
PP
Lg Enceder
LO Decoder
P1
TT
(a) Discrete Codebook L caming Stage (Sec. 3.1)
(b) A ssociation Stage (Sec. 3.2)
Trazefanc
X0.6 staud Modile NHHA
f
lves
Eicodde
E(
Code
(c) Feature Fusion & Code Prediction Stage (Sec. 3.3)
Figure 2: Proposed DAEFR framework. (a) Initially, we train the autoencoder and discrete codebook for both HQ and LQ image domains through self-reconstruction.. (b) Once we obtain both encoders (Eи and Ez). we divide the feature (Zn, and Z₁) into patches (P." and P.F) and construct a similarity matrix Massee that associates HQ and LQ features while incorporating spatial information. To promote maximum similarity between patch features, we employ cross-entropy loss function to maximize the diagonal of the matrix. (c) After obtaining the associated encoders (E and E Z(") from the associated encoders, generating fused features ZÍ. We then input the fused feature Z to the transformer T, which predicts the corresponding code index s for the HQ codebook Cn. Finally, use the predicted code index to retrieve the features and feed them the HQ decoder D to restore the image.
multi-heas rross-attention module (MHCA) merge the features (Z₁, and
The process shown in Fig. 2(a) involves the encoding of the HQ face image In E IR Hxxx'x into compressed feature Zn E Rmaxxxd by the encoder Eн. This step is carried out by replacing each feature vector of Zn, with the nearest code item in the learnable codebook C₁ Ck Rd--- Consequently, obtain the quantized feature Zi, IR mxxdd
k=D
(i-)) z arg min CCC,
1z cull2,
ke e (0..., -11,
1],
(1)
where Znh zo((.3) and 2/20 ((6))) are the feature vectors on the position (i, of Zí, and Zn. || -1-16 is the L2-norm. After obtaining the quantized feature Zg the decoder DHr then proceeds reconstruct the HQ face image Ti E HR x3 using Zi,-
Training Objectives. Similar to the prior arts (Gu et al., 2022; Zhou et al., 2022; Wang al., 2022b; Esser et al., 2021) setting, We incorporate four distinct losses for training, which includes three image-level reconstruction losses (i.e., L1 loss L,, perceptual loss Lper (Zhang et al., 2018; Johnson et al., 2016), and adversarial loss Ladv (Isola et al., 2017)) and one code-level loss Loode (Esser et al., 2021; Van Den Oord et al., 2017):
L |
I 11.
Cper = [1b( - 2b (1, Ludv = [log D(I)) + log(1 - D(I
Lande ||sg( Zn. 2 Zl| sg( 11
where the feature extractor of VGG19 (Simonyan Zisserman, 2014) is represented by 4. D is patch-based discriminator (Isola et al., 2017), and sg(-) denotes the stop-gradient operator. The value of 3 is set at 0.25. The final loss Ccodebook is:
Loadsbook f1 + Aper Lper + >adv Ladv + Leode:
(3)
where set Aper 1.0 and Anay 0.8 in setting.
4

---

Published as a conference paper at ICLR 2024
3.2 ASSOCIATION STAGE
In this work, we reduce the domain gap between the HQ and LQ domains, allowing the two encoders encompass a greater range of information from both domains. Once we obtain the domain encoders from the codebook learning stage, we take inspiration from the CLIP model (Radford et al., 2021) and propose feature patch association algorithm. This technique involves applying the feature patch association on the output from the two encoders. By utilizing this approach, we aim to further minimize the domain gap between the HQ and LQ domains.
As shown in Fig. 2(b), after obtaining the HQ and LQ domain encoder (En and EL) from the previous stage, we proceed to flatten the output features (Z,, and Zi E Rmxxxx into corresponding patches (P and P, i [I....,,m, X ni). This flattening enables us to construct similarity matrix (Masse E RNNN, N = m X n), which we use to quantify the similarity between different patch features. Specifically, we calculate the cosine similarity for each patch feature and constrain them to maximize the similarity along the diagonal of the matrix. By applying this constraint, the two encoders are prompted to connect patch features close in both spatial location and feature level, preserving their spatial relationship throughout the association process. Combining the patch features from both encoders results in two associated encoders, denoted as E and Eý, that integrate specific domain information, which will be utilized in the subsequent stage.
Training Objectives. We perform joint training on the HQ and I.Q reconstruction paths, incorpo- rating the association part. To facilitate the feature association process, we adopt the cross-entropy loss (C& and ((F) to effectively constrain the similarity matrix Massoc:
N CH = N EEΣLN ye, log(ph)), Ct
N C
ΣΣ Yi. log(ph))); N
1
where N denotes the patch size, and C' represents the number of classes, which, in our specific case, is equal to N. The ground truth label is denoted as yii,j whereas the cosine similarity score in the similarity matrix M in the HQ axis is represented as ph,j- Conversely, the score in the LQ axis is defined as ph..- The final objective of the feature association part is:
Lasone = L1 + Aper Lper + Anav Ludv + Leode + (cH + /22
(5)
where we integrate the same losses and weights used in the codebook learning stage to maintain the representation of features.
3.3 FEATURE FUSION & CODE PREDICTION STAGE
After obtaining the two associated encoders EA and E from the feature association stage, both encoders to encode the LQ image It, as shown in Fig. 2(c). Specifically, we extract feature information Z E Rmmxnxd and Z Rmxx from the I.Q image 1 using each encoder E and F separately. Similar (Vaswani et al., 2017: Wang et al., 2022b), cross-attention (MHCA) module to merge the feature information from both encoders and generate a fused feature 2 E Ruxuxx that incorporates the LQ domain information. This fused feature Z is expected to contain useful information from both HQ and LQ domains: A MHCA(Z ( 2,²). The MHCA mechanism lets the module focus on different aspects of the feature space and better capture the relevant information from both encoders.
multi-head
Once the fused feature Z is obtained using the feature fusion technique with MHCA, we utilize a transformer-basee classification approach (Zhou et al., 2022), to predict the corresponding class as code index s. Initially, we flatten the fused feature Z E jRmxxxd as 2 E (m) and input the flattened fused feature 2 into the transformer and obtain the predicted code index (0. N 1)mn. During this process, the HQ codebook C₁ and HQ decoder DH from the codebook learning stage are frozen. We then use the predicted code index s to locate the corresponding feature in the HQ codebook C₁ and feed the resulting feature Zł to the decoder DH to generate HQ images Ires. as depicted in Fig. 2(c). This step efficiently enhances the image restoration by incorporating information from the hybrid domain.
Training Objectives. We utilize two losses to train the MHCA and transformer module effectively to ensure proper feature fusion and code index prediction learning. The first loss is an 1.2 loss sade
5

---

Published as a conference paper at ICLR 2024
which encourages the fused feature Z to closely resemble the quantized feature Zí from the HQ codebook Ca. This loss helps ensure that the features are properly combined and maintains the relevant information from both the HQ and I.Q domains. The second loss is a cross-entropy loss Conde for code index prediction, enabling the model to accurately predict the corresponding code index S in the HQ codebook Cr.
TA
code
1Z sg( (2)))
code
Σ
& log(s:),
where we obtain the ground truth feature Zf, and code index & from the codebook learning stage, which we retrieve the quantized feature Zí, from the HQ codebook Cn using the code index S. The final objective of the feature fusion and code prediction is:
Speddict Area: Lleat + feode
where we set the 1.2 loss weight Aeat = 10 in our experiments.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Implementation Details. In our implementation, the size of the input face image is 512 x 512 x 3, and the size of the quantized feature is 16 16 x 256. The codebooks contain N = 1,024 code items, and the channel of each item is 256. Throughout the entire training process, we employ the Adam optimizer (Kingma& & Ba, 2014) with batch size 32 and set the learning rate to 1.44 x 10-1 The HQ and LQQ reconstruction codebook priors are trained for 700K and 400K iterations, respectively, and the feature association part is trained for 70K iterations. Finally, the feature fusion and code prediction stage is trained for 100K iterations. The proposed method is implemented in Pytorch and trained with eight NVIDIA Tesla A100 GPUs.
Training Dataset. We train our model on the FFHQ dataset (Karras et al., 2019), which contains 70,000 high-quality face images. For training, we resize all images from 1024 x 1024 to 512 x 512. To generate the paired data, synthesize the degraded images the FFHQ dataset using the same procedure as the compared methods (Li et al., 2018; 8; 2020; Wang et al., ;021; 2022b; Gu et al., 2022; Zhou et al., 2022). First, the HQ image Anigh is blurred (convolution operator @) by a Gaussian kernel ka. Subsequently, a downsampling operation J with a scaling factor r is performed to reduce the image's 's resolution. Next, additive Gaussian noise nı added the downsampled image. JPEG compression with a quality factor q is applied to further degrade image quality. The resulting image then upsampled 1 with a scaling factor r to a resolution of 512 x 512 to obtain the degraded how image. In our experiment setting, we randomly sample a. r, 6, and from 10 15), [0.8, 30), [0, 20], and [30, 100], respectively. The procedure can be formulated as follows:
(8) Testing Dataset. Our evaluation follows the settings in prior literature (Wang al., 2022b; Gu et al., 2022; Zhou et al., 2022), and includes four datasets: the synthetic dataset CelebA-Test and three real-world datasets, namely. LFW-Test, WIDER- Test, and BRIAR-Test. CelebA-Test comprises 3,000 images selected from the CelebA-HQ testing partition (Karras et al., 2018). LFW-Test consists of 1,711 images representing the first image of each identity in the validation part of the LFW dataset (Huang et al., 2008). Zhou et al. (Zhou et al., 2022) collected the WIDER- from the WIDER Face dataset (Yang et al., 2016), which comprises 970 face images. Lastly, the BRIAR-Tew contains 2,120 face images selected from the BRIAR dataset (Cornett et al., 2023). The BRIAR- dataset provides a wider range of challenging and diverse degradation levels that enable the evaluation of the generalization and robustness of face restoration methods (All subjects shown in the paper have consented to publication.).
how = [[(Thigh @ ki) 4. + na] JPEG,) fr.
Metries. In evaluating our method's 's performance on the CelebA- dataset with ground truth, we employ PSNR, SSIM, and LPIPS (Zhang et al., 2018) as evaluation metrics. We utilize the commonly used non-reference perceptual metrics, FID (Heusel et al., 2017) and NIQE (Mittal et al., 2012) to evaluate real-world datasets without ground truth. Similar to prior work (Gu et al., 2022; Wang et al., 2022b; Zhou et al., 2022), we measure the identity of the generated images by using the embedding angle of ArcFace (Deng et al., 2019), referred to "IDA". To better measure the fidelity of generated facial images with accurate facial positions and expressions, we additionally adopt landmark distance (LMD) as the fidelity metric.

---

Published as a conference paper at ICLR 2024
Table Quantitative comparisons. Red and blue indicate the best and second-best, respectively.
(a) Real-world datascts.
(b) The syntheric CelebA- AelebA-Test datasct.
Dhisett
L.W-Tet
WiDEB--be
ERIARR-aa
Methoals
FID.
LPRSS
NIQE,
[DA]
LMD]
PSNRR
SSIM
Matteds
2111
[S3]]
Inpatt
$37.013
0.528
19.267
1.455
17.016
20.833
0.638
GEP-GAN
PSFRGAN
137,557
49 15.51
30,057
4.054
3.966
47:257
39:230
4.055
3.585
196.734
97.350
3.979
5.281
PSPRGAN
GEP-GAN
GPEN
55,008
66.367
06.130
0.635
0.425
0.650
3913
4.063
3.311
1.361
1.258
1.200
9.301
7.313
7.259
20.303
20.545
19.594
0.536
0.522
0.553
RecorfFomer
4.013
1.238
8383
30
0.494
Conavomm
50.
4.422
38:20
6.06
98.314
5005
CodeFoomner
5.381
0.575
VQUR
50:712
1.399
04.139
3.00
92.012
4993
VQFR
9.128
19484
0.422
3.830
45.235
5,188
3-417
5.104
1.215
7,890
0.595
5,034
LEW
WIDER-Ted
ARRAARTON.
Imput
PSERGAN
GFP-GAN
RestorFFomer
CodeFormer
VQFR
DR2
DAEFR (Ours)
Figure 3: Qualitative comparison on real-world datasets. The BRIAR- Test dataset contains original identity clean images, allowing us ascertain that the individual this image not wearing glasses. Our DAEFR method exhibits robustness in restoring high-qualit, faces even under heavy degradation.
4.2 COMPARISONS WITH STATE-OF-THE-AAR METHODS
We compare the proposed method, DAEFR, with soute-of-the-and methods: PSFRGAN (Chen et al., 2021), GFP-GAN (Wang et al., 2021), GPEN (Yang al., 2021), RestoreFormer (Wang al., 2022b), CodeFormer (Zhou et al., 2022), VQFR (Gu al., 2022), and DR2 (Wang et al., 2023).
Evaluation on Real-world Datasets. As shown in Table I(a), our proposed DAEFR outperforms other methods regarding perceptual quality metrics, evidenced by the lowest FID scores on the WIDER-TesT and BRIAR-TesT dataset and second-best scores the LFW-Testt dataset, indicating the statistical similarity between the distributions of real and generated images. Regarding NIQE scores, which assess the perceptual quality of images, our method achieves the highest score on the LFW-Test dataset and the hiihhet on the WIDER-Ter and BRIAR-Te datasets.
Visual comparisons in Fig. 3 further demonstrate the robustness of our method against severe degradation, resulting in visually appealing outcomes. In contrast, RestoreFormer (Wang et al., 2022b) and VQFR (Gu al.. 2022) exhibit noticeable artifacts in their results, while CodeFormer (Zhou et al., 2022) and DR2 (Wang et al., 2023) tend to produce smoothed outcomes, losing intricate facial details. DAEFR, however, effectively preserves the identity of the restored images, producing natural results with fine details. This preservation of identity can be attributed to the design of our model, which emphasizes the retention of original facial features during the restoration process. These observations underscore the strong generalization ability of our method.
Evaluation the Synthetic Dataset. We compare DAEFR quantitatively with existing approaches on the synthetic CelebA -est dataset in Table (bb). Our method demonstrates competitive performance in image quality metrics, namely FID and LPIPS, achieving the second- scores among the evaluated methods. These metrics represent the distribution and visual similarity between the restored and original images. Moreover, our approach effectively preserves identity information, as evidenced by comparable IDA and LMD scores. These metrics assess how well the model preserves the identity and facial structure of the original image.
To further illustrate these points, we provide qualitative comparison in Fig. 4. Our method demonstrates the ability to generate high-quality restoration results with fine facial details, surpassing the performance of the other methods. This ability to restore fine facial details results from model's capacity to extract and utilize high-frequency information from the degraded images, contributing to its overall performance. We provide more visual comparisons in the appendix and supplementary.
7

---

Published as a conference paper at ICLR 2024
Tngut
PSERGAN
GFP-GAN
GPEN
RestoreFommer
CodeForner
VQFR
DRZ
DAFFR (Ours)
GT
Figure 4: Qualitative comparison on the synthetic CelebA-Test dataset. Our DAEFR method
exhibits robustness in restoring high-quality faces even under heavy degradation.
Ablation studies of variant networks and association methods on the CelebA-Test. The terms "HQ encode" and "LQ encoder" refer to the encoders used for encoding LQ images. "w/o" and "with" in the association part indicate whether the encoder undergoes the feature association process. The fusion types "Linear" and "MHCA" the feature fusion module represent linear projection with three fully connected layers or multi-head cross attention, respectively.
Table 2:
Encoder (Sec. 3.1) HQ encoder LQ encoder
Association (Sec. 3.2) with
Feature Fusion (Sec. 3.3)
Metries LMD
Exp.
w/o
Linear
MHCA
LPIPS]
NIQEL
(a)
0.344
4.170
4,394
(h)
0.343
4.265
4.510
/
V
0.349
4.258
4.297
(d)
V
0.343
4.197
4.290
(e) (Ours)
0.351
4.019
3.815
4.3 ABLATION STUDY
Number of Encoders. In our research, we conduct an initial investigation to examine the impact of the number of encoders on the overall performance. We present this analysis in Exp. (a) to (c) as detailed in Table 2. The results of our investigation indicate that utilizing two encoders as the input source yields better performance than using a single encoder. This conclusion supported by the superior NIQE score achieved in the experiments. This ablation study confirms that including an additional LQ encoder can provide domain- -emain--pecific information, aiding image restoration. To provide further insights, show the visual results of Exp. (a) to (c) in Fig. 5.
Association Stage. Furthermore, we evaluate the effectiveness of our association stage through Exp. (c) and (d), as depicted in Table 2. The results indicate that utilizing associated encoders as our input source leads superior performance across all evaluation metries compared to ass-associ encoders. This empirical evidence validates that our association stage effectively enhances the encoder's capability to retrieve information from the alternative domain and effectively reduces the domain gap between HQ and L. To provide visual evidence of the improvements achieved by our approach. we present the visual results of Exp. (c) and (d) in Fig. 5. These visual comparisons further support the superiority of our method.
Feature Fusion Module. Finally, assess the effectiveness of our feature fusion module, com- paring two different approaches: a baseline method employing linear projection with three fully connected layers and our proposed multi-head cross-attention module (MHCA). The experimental results are presented in Table 2, specifically under Exp. (d) and Exp. (c). We aim to determine which approach yields better performance regarding both LMD and NIQE scores. The results demonstrate a notable improvement when utilizing the MHCA compared to the straightforward linear projection method for feature fusion. This indicates that our MHCA can effectively fuse nomain-specinf information from both the HQ and LQ domains while preserving crucial identity information. To provide a visual representation of the results, we showcase the outcomes of Exp. (d) and Exp. (e) in Fig. 5. These visual representations further support our findings and demonstrate the superior performance of our proposed MHCA in enhancing the image restoration process.
Effectiveness of Low-Quality Feature from Auxiliary Branch. To demonstrate the effectiveness of our auxiliary LQ branch, we conduct validated experiments of fusing LQ features with feature Zţ in the feature fusion and code prediction stage. These experiments involve extracting LQ features from the LQ codebook and adding a control module (Wang et al., 2018; Zhou et al., 2022) to fuse the LQ feature and feature Z9 before feeding to the HQ decoder.
8

---

Published as a conference paper at ICLR 2024
Input
Exp. (a)
Exp, (b)
Exp (c)
Exp, (c)
Exp. Ours
GT
Figure 5: Ablation studies. The experimental index accordance with the Table 2 configuration is utilized. Our method successfully produces intricate facial details and closely resembles the ground truth, even when the input undergoes severe degradation. Importantly, we effectively retain the identity information from the degraded input.
Table 3: Quantitative evaluation for the effectiveness of LQ feature on the CelebA-Test, The LQ feature scale indicates the fusion scalar with feature Zř. The performance becomes better when we increase the scalar Sty, and the experimental results prove the effectiveness of our LQ feature, which encodes essential visual attributes and speeifc statistical characteristies.
LQ Feature Scale
sty=0
Sty
Sig 00.
$1,=0.6
$1g=0.8
$t
LPIPS
0.399
0,389
0.378
0,370
0.364
0.363
PSNR f SSIM
20.040
20.367
20.727
21.074
21.348
21.488
0.558
0.571
0.585
0.600
0.612
0.616
Our quantitative evaluation employs quality metrics, including LPIPS, PSNR, and SSIM. We conduct these experiments the CelebA-Tec Test dataset. Our quantitative results clearly prove the positive impact when we increase the scale of LQ feature scalar Sig, as shown in Table. 3. This experiment underscores the practical advantages of our LQ branch and shows the LQ features effectively encode essential visual attributes and domain-specific characteristics, enhancing the image restoration process. We place the detailed experiment setting and visual comparison in the appendix and supplementary.
Table 4: Quantitative evaluation on the face recognition task. We conduct the quantitative experiments on the LFW dataset (Huang et al., 2008) of the face recognition task with the official ArcFace (Deng et al., 2019) model with Verification performance (%) as our evaluation metric. The degradation parameters ranging from 10,000 to 40,000 correspond to varying levels of degradation.
Methods
Origin
10000 (Slight)
20000
30000
40000 (Severe)
CodeFormer (Zhou et al., 2022)
98.23
97.35
91.28
81.25
71.67
DAEFR (Ours)
98.51
97.86
92.15
82.69
73.78
Validation on Downstream Face Recognition Task. We conduct downstream face recognition task using the proposed restoration method on the LFW (Huang et al., 2008) face recognition validation split set (with 12,000 images). We use the unseen atmospheric turbulence degradation to simulate diverse degradation levels, employing the methodology outlined in (Chimitt & Chan, 2020) for degradation generation. The degradation parameters ranging from 10,000 to 40,000 correspond to varying levels of degradation, spanning from slight to severe. We provide the dataset samples in the appendix and supplementary material.
We employ the official ArcFace (Deng et al., 2019) model with Verification performance (%) as our evaluation metric to evaluate the restored images. The experimental results, presented in Table 4, demonstrate the superior performance of our method across various degradation levels. Particularly noteworthy is the widening performance gap between CodeFormer and our method as the degradation severity escalates. These findings validate the efficacy of our additional I.Q encoder, which captures valuable information from the LQ domain, thus significantly augmenting the restoration process.
5 CONCLUSION
In this paper, we propose DAEFR to effectively tackle the challenge of blind face restoration, generating high-quality facial images from low-quality ones despite the domain gap and information loss between HQ and LQ image types. We introduce an auxiliary LQ encoder that captures the LQ domain's specific visual characteristics and statistical properties. We also employ feature association techniques between the HQ and LQ encoders to alleviate the domain gap. Furthermore, we leverage attention mechanisms to fuse the features extracted from the two associated encoders, enabling them to contribute to the code prediction process and produce high-quality results. Our experiments show that DAEFR produces promising results in both synthetic and real-world datasets with severe degradation, demonstrating its effectiveness in blind face restoration.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2022-21 102100001. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 13
Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. Glean: Generative latent bank for large-factor image super-resolution. In CVPR, 2021. 3
Kelvin CK Chan, Xiangyu Xu, Xintao Wang, Jinwei Gu, and Chen Change Loy. Glean: Generative latent bank for image super-resolution and beyond. TPAMI, 2022.
Chaofeng Chen, Dihong Gong, Hao Wang. Zhifeng Li, and Kwan- Yee K Wong. Learning spatial attention for face super-resolution. TIP, 2020. 2
Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan- Yee K Wong. Progressive semantic-aware style transformation for blind face restoration. In CVPR, 2021. 2, 17
Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. Fsrnet: End-to-end learning face super-resolution with facial priors. CVPR, 2018.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 13
Nicholas Chimitt and Stanley H Chan. Simulating anisoplanatic turbulence by sampling intermodal and spatially correlated zemike coefficients. Optical Engineering, 2020. 9, 17
David Cornett, Joel Brogan, Nell Barber, Deniz Aykac, Seth Baird, Nicholas Burchfield, Carl Dukes, Andrew Duncan, Regina Ferrell, Jim Goddard, et al. Expanding accurate person recognition to new altitudes and ranges: The BRIAR dataset. In WACV, 2023.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin
loss for deep face recognition. In CVPR, 2019, 6,
Berk Dogan, Shuhang Gu, and Radu Timofte. Exemplar guided face image super-resolution without facial landmarks. In CVPRW, 2019. 2
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image
synthesis. In CVPR, 2021. 3,4 4
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. CVPR, 2020. 3
Yuchao Gu, Xintao Wang, Lianghin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder. In ECCV, 2022. 3, 4, 6,
Jingwen He, Wu Shi. Kai Chen, Lean Fu, and Chao Dong. Gefsr: a generative and controllable face super resolution method without facial and gan priors. In CVPR, 2022. 3
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In NeurIPS, 2017.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 3
Xiaobin Hu, Wenqi Ren, John LaMaster, Xiaochun Cao, Xiaoming Li, Zechao Li, Bjoern Menze, and Wei Liu. Face super-resolution guided by 3d facial priors. In ECCV, 2020.
Xiaobin Hu, Wenqi Ren, Jiaolong Yang. Xiaochun Cao, David Wipf, Bjoemn Menze, Xin Tong, and Hongbin Zha. Face restoration via plug-and-play 3d facial priors. TPAMI, 2021. 2
Gary B Huang. Marwan Mattar, Tamara Berg, and Eric Learned- Miller. Labeled faces in the wild: A database for studying face recognition unconstrained environments. Workshop on Faces in 'Real-Life' Images: Detection, Alignment, and Recognition, 2008. 6, 16
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017. 4
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 4
Ratheesh Kalarot, Tao Li, and Fatih Porikli. Component attention guided face super-resolution network: Cagface. In WACV, 2020. 2
Tero Karras, Timo Aila. Samuli Laine. and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018. 6
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 3, 6
Deokyun Kim, Minseon Kim, Gihyun Kwon, and Dae-Shik Kim. Progressive face super-resolution via attention to facial landmark. arXiv preprint arXiv: 08239, 2019. 2
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1r12v::412.69888 2014.
Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang. Learning warped guidance for blind face restoration. In ECCV, 2018. 2, 6
Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang. Blind face restoration via deep multi-scale component dictionaries. ECCV, 2020. 2, 6
Cheng Ma, Zhenyu Jiang, Yongming Rao, Jiwen Lu, and Jie Zhou. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In CVPR, 2020. 2
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In CVPR, 2020. 3
Anish Mittal, Rajiv Soundararajan, and Alan Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 2012. 6
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML,, 2021. 2, 5, 15
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurlPS, 2019. 3
Wenqi Ren, Jiaolong Yang. Senyou Deng, David Wipf, Xiaochun Cao, and Xin Tong. Face video deblurring using 3d facial priors. In ICCV, 2019.
Flad Richardson, Yuval Alaluf, Or Patashník, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In CVPR, 2021.
Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuar Yang. Deep semantic face deblurring. In CVPR, 2018. 2
11

---

Published as a conference paper at ICLR 2024
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014. 4
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurlPS, 2017. 3,4
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017. 5, 13
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR, 2018. 8, 16
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In CVPR, 2021. 3, 6, 7,
Xinyao Wang, Liefeng Bo, and Li Fuxin. Adaptive wing loss for robust face alignment via heatmap regression. In ICCV, 2019.
Yinhuai Wang, Yujie Hu, and Jian Zhang. Panini-net: Gan prior based degradation-aware feature interpolation for face restoration. In AAAI, 2022a. 3
Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjie Zheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. Dr2: Diffusion- based robust degradation remover for blind face restoration. In CVPR, 2023. 3, 7,
Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. Restoreformer: High- quality blind face restoration from undegraded key-value pairs. In CVPR, 2022b. 1, 4, 5, 6, 7, 13, 17
Lingbo Yang, Shanshe Wang, Siwei Ma, Wen Gao, Chang Liu, Pan Wang, and Peiran Ren. Hifacegan: Face renovation via collaborative suppression and replenishment. In ACM MM, 2020. 2
Shuo Yang. Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: A face detection benchmark. In CVPR, 2016. 6
Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration the wild. In CVPR, 2021. 3, 7, 17
Zongsheng Yue and Chen Change Loy. Difface: Blind face restoration with diffused error contraction. arXiv preprint 2rXiv:22/2.065/2, 2022.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 4. 6
Xi Zhang and Xiaolin Wu. Multi-modality deep restoration of extremely compressed face videos. TPAMI, 2022.
Yang Zhao, Yu-Chuan Su, Chun-Te Chu, Yandong Li, Marius Renn, Yukun Zhu, Changyou Chen, and Xuhui Jia. Rethinking deep face restoration. In CVPR, 2022. 3
Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurlPS, 2022. 1, 3, 4, 5, 6, 7, 8, 9, 16, 17
Feida Zhu, Junwei Zhu, Wenqing Chu, Xinyi Zhang, Xiaozhong Ji, Chengjie Wang, and Ying Tai. Blind face restoration via integrating face shape and generative priors. CVPR, 2022.
12

---

Published as a conference paper at ICLR 2024
Appendix
A OVERVIEW
This supplementary material presents additional results to complement the main manuseript. First, we describe the detailed network architecture in Section B. Then, we conduct more ablation studies to demonstrate the effectiveness of the association stage and our auxiliary LQ branch in Section C. Finally, we show more visual comparisons with state-of-the-art methods in Section D.
B NETWORK ARCHITECTURE
The detailed structures of the encoder and decoder are shown in Table 5. Our restoration process employs an identical encoder and decoder structure for both the HQ and L.Q paths. We use transformer (Vaswani al., 2017) module consisting of nine self-attention blocks for the structure. Additionally, we enhance its expressiveness by incorporating sinusoidal positional embedding (Carion al., 2020; Cheng et al., 2022) on query Q and keys K. This modification aids in capturing positional information effectively within the transformer. In the feature fusion stage, we employ a multi-head cross-attention module (MHCA) effectively merge the HQ and LQ features extracted from the encoders, which is inspired by (Wang et al., 2022b; Vaswani et al., 2017). The implementation of MHCA is: Q 2 W + by,
K = K= W + bb. V= V=2,W, + b
(9)
where we utilize feature Z as queries Q and feature 2 as keys K and values V. These features are multiplied by their respective learnable weights W W//k/o E Rdxd and biased by ba/k/u E Rd.
z MHCA(Z, ZA = FFN(LN(Z, z7²),,
(10)
where LN is the layer normalization, and FFN is the hordard network composed of two convolu- tion layers.
Table 5: Detailed architecture of our encoder and decoder. GN: GroupNorm; channels; C is the length of the features in the HQ codebook. 1 and f mean the feature pass direction in the table.
Input size
Encoder 4
Decoder 1
(Residual block: 32-GN, 64-c} x x 2 { (Residual block: 32-GN, 64-c3 x 2 x 3
512x512
Bilinear downsampling 2x
Conv 3 3, 128-c
Conv 3 x 3, 64-c
Bilinear upsampling 2x
[Residual block: 32-GN, 128-c) 2 x
(Residual block: 32-GN, 128-e) 2 x
256
Bilinear downsampling 2x
Conv 3 3. 128-c
Conv 3, 128-c
Bilinear upsampling 2
(Residual block: 32-GN, 128-c) 2 x
(Residual block: 32-GN, 128-c) 2 x
128x128
Bilinear downsampling 2x
Conv 3 3, 256-c
Conv 3 3. 128-c
Bilinear upsampling 2x
RResidual block: 32-GN, 256-c) x 2
x
(Residual block: 32-GN, 256-c) x 2 } x3
64
Bilinear downsampling 2x
Cony 3 3, 256-c
Conv 3 3, 256-c
Bilinear upsampling 2
Residdul block: 32-GN, 256-c3 x 2 x
(Residual block: 32-GN, 256-c) 2 x
32x
Bilinear downsampling 2x
Conv 3 x 3. 512-c
Conv 3 3, 256-c
Bilinear upsampling 2
(Residual block: 32-GN, 512-c) 2 x
(Residual block: 32-GN, 512-c) 2
16
Conv_out Conv x 3, 512-c >> C-c
Conv 3 x 3, C-c + 512-c
C MORE ABLATION STUDIES
In this section, we delve into further ablation studies concerning the association stage and our auxiliary LQ branch, focusing architecture design and various loss designs. In proposed approach,
13

---

Published as a conference paper at ICLR 2024
aim to demonstrate the association stage's effectiveness and our auxiliary LQ branch. Through these studics, we provide detailed analysis and evidence to support the impact and contribution the association stage and our auxiliary LQ branch in improving the overall performance of the image restoration process.
Domain Gap before Association Stage We thoroughly investigate the domain gap in three distinct pathways: HQ path, LQ path, and Hybrid path. All three pathways involve the utilization of LQ images as input. Specifically, the LQ image is inputred into the LQ encoder in the Hybrid path, and the resulting features are subsequently forwarded the HQ codebook. The HQ codebook is employed to identify the closest feature, which is then utilized for reconstruction using the HQ decoder. As illustrated in Figure 6, both the HQ path and Hybrid path exhibit limitations in effectively reconstructing restoring the LQ image due to the domain gap. Notably, the Hybrid path fails to generate any facial features.
(a) HQ Image
(b) LQ Image
(c) HQ Path
(d) LQ Path
(e) Hybrid Path
Figure 6: Domain gap. We utilize LQ images as input to evaluate the reconstruction ability of various path types. The significant domain gap between HQ and LQ images significantly affects the ability to reconstruct the images. (Hybrid Path: LQ encoder + HQ codebook HQ decoder)
Different Association Architecture We explore various network architectures to enhance the capabilities of the LQ encoder, as depicted Fig.7. However, our experimental results demonstrated that solely enabling the LQ encoder to learn from the HQ encoder did not effectively address the domain gap issue. Furthermore, the less constrained HQ feature representation did not align well with LQ features, leading to the generation of non-face images, depicted in F(g. Therefore, we opt for joint optimization approach, incorporating both the HQ and LQ paths, to preserve the feature representation. This approach aims empower both encoders capture more comprehensive information from both domains and reduce the domain gap.
BQ Codebook
HQ
HQ Encoder
HQ
HBQ Encoder
HQ Decoder
HQ
Association
Association
LQ
L. Enooder
LQ
LQ
LQ
LQ Encoder
1.0 Decoder
LQ
Codebook
Codebook
(a) Learn information from HQ Encoder
(b) Our proposed framework
Figure 7: Different association architectures. (a) Force the LQ encoder to learn information from the HQ encoder. (b) Joint optimization for both the HQ and LQ paths.
14

---

Published as a conference paper at ICLR 2024
(a) HQ Image
(b) LQ Image
(c) Learn from HQ
(d) Joinly ssoccite
Figure 8: Visual results for different association architectures. We utilize LQ images as input. (c) is the result of only letting the LQ encoder learn the information from the HQ encoder. (d) is the result of jointly optimizing both paths.
(a) HQ Image
(b) LQ Image
(c) L1 Loss
(c) L2 Loss
CE Loss
Figure 9: Visual results for different loss settings. We utilize LQ images as input. (c) (d) The L1 and L2 loss fails to associate the features of the two domains effectively and loses certain original information. (e) The utilization of cross-entropy loss enhances the association between the features of the two domains, resulting in the preservation of greater amount of original information.
Different Loss Settings We explore various types loss functions address the similarity between HQ and LQ features. We investigate using the L1 and L2 loss to enforce an exact match between the HQ and LQ features. However, experimental results demonstrate that this approach does not yield satisfactory outcomes, as shown in Fig. 9. We also present the quantitative results of our CelebA validation dataset in Table 6.
Drawing inspiration from the CLIP model (Radford al., 2021), construct similarity matrix and utilize the cross-entropy loss to constrain the feature similarity. As depicted in Fig. we configure experiments to the LQ image input and employ the LQ encoder generate features. These features are then processed through the HQ codebook to identify the nearest matching feature and then decoded using the HQ decoder.
By incorporating the cross-entropy loss, we effectively establish associations between the HQ and L. features, enhancing the overall performance of approach.
15

---

Published as a conference paper at ICLR 2024
Table 6: Quantitative evaluation. We conduct the quantitative experiments with different association loss setting on our CelebA validation set.
Losses
FID
LPIPS
Input
143.991
0.444
LI Loss
70.717
0.403
L2 Loss
138.212
0.552
Cross-Enne Entropy Loss
41.230
0.361
24
1000000000
HQQ
Ercoder
Dut.bok
Cree Mudile
HQ Decoder
Coara
4
LO Encoder
MICA
Dj
lres
Feature Scalar Sug
E
29
Code index
arg 0n
Feature Fusion & Code Prediction Stage (Sec. 3.3)
Figure 10: Detailed network architecture for the experiment of controlling the scale of LQ features. We conduct validated experiments of fusing LQ features in the feature fusion and code prediction stage. Given feature control module and feature scalar Sty, control the scale of the LQ feature Zi to fuse with the feature Zł before feeding to the HQ decoder.
Effectiveness of Low-Qual Quu-Quulity Feature from Auxiliary Branch. To demonstrate the effectiveness
of our auxiliary LQ branch, we conduct validated experiments of fusing LQ features with feature Z in the feature fusion and code prediction stage. These experiments involve extracting LQ features Zi from the IQ codebook and adding control module (Wang et al., 2018; Zhou et al., 2022). Given feature control module and feature scalar Sig. we can control the scale of the LQ feature Zi to fuse with the feature Zł before feeding to the HQ decoder. The network architecture is shown in Fig. 10.
We conduct these experiments on the CelebA- Test dataset. Our visual results clearly prove the positive impact when increase the scale of LQ feature scalar 8ig, shown in Fig. 11. This experiment underscores the practical advantages of our IQ branch and shows the IQ features effectively encode essential visual attributes and domain -specffic characteristics, enhancing the image restoration process.
Ingut
Siq
Sig =.2
Sig =04
Suy =0,6
Stg =08
Sig L.
GT
Figure 11: Visual comparison with different LQQ feature scalar. We compare the visual results with different LQ feature scalar Sig, showing that the LQ features indeed encode essential visual attributes and
characteristics, even when the input is severely degraded.
Validation on Downstream Face Recognition Task We conduct a comprehensive downstream face recognition task with the following procedure. We utilize the L.FW (Huang et al., 2008) Face
16

---

Published as a conference paper at ICLR 2024
Origin
10000
20000
30000
40000
Figure 12: Visualization for face recognition validation dataset. We choose the unseen atmospheric turbulence degradation to simulate diverse degradation levels, employing the methodology outlined in (Chimitt & Chan, 2020) for degradation generation. The degradation parameters ranging from 10,000 to 40,000 correspond to varying levels of degradation, spanning from slight to severe.
recognition dataset's validation split, comprising 12,000 images. We choose the unseen atmospheric turbulence degradation to simulate diverse degradation levels, employing the methodology outlined (Chimitt & Chan, 2020) for degradation generation. The degradation parameters ranging from 10,000 to 40,000 correspond to varying levels of degradation, spanning from slight to severe. We provide the dataset samples in Fig. 12.
D MORE VISUAL COMPARISON
In this section, we present additional visual comparisons with state-of-the-art methods: PSFR- GAN (Chen et al., 2021), GFP-GAN (Wang et al., 2021), GPEN (Yang al., 2021), Restore- Former (Wang et al., 2022b), CodeFormer (Zhou et al., 2022), VQFR (Gu et al., 2022), and DR2 (Wang et al., 2023).
The qualitative comparisons on the LFW-Test are shown in Fig. 13. The qualitative comparisons on the WIDER- Test are shown in Fig. 14, Fig. 15 and Fig. 16. The qualitative comparisons the BRIAR-Tes! are shown Fig. 17. The qualitative comparisons on the CelebA-Test are shown in Fig. 18.
We also compare the large occlusion and pose situation with state-of-the-art methods in Fig. 19.
While method demonstrates robustness in most severe degradation scenarios, also observe instances where may fail, particularly in cases with large face poses. This can be expected as the FFHQ dataset contains few samples with large face poses, leading a scarcity of relevant codebook features to effectively address such situations, resulting in less satisfactory restoration and reconstruction outcomes. We show the failure cases in Fig. 20.
These comparisons demonstrate that our proposed DAEFR generates high-quality facial images and effectively preserves the identities, even under severe degradation the input faces. Furthermore, compared to the alternative methods, DAEFR performs better in recovering finer details and producing more realistic facial outputs.
17

---

Published as a conference paper at ICLR 2024
Input
PSERGAN
GFP-GAN
GPEN
RestoreFommer
CodeFormer
VQFR
DR2
DAEFR (Ours)
Input
PSFRGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEFR (Ours)
Input
PSFRGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEFR (Ours)
Figure 13: Qualitative comparison LFW-Test datasets. Our DAEFR method exhibits robustness
in restoring high-quality faces in detail part.
18

---

Published as a conference paper at ICLR 2024
Input
PSFRGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEFR Ouurs)
Input
PSFRGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEFR (Ours)
Figure 14: Qualitative comparison on WIDER-TesS datasets. Our DAEER method exhibits
robustness in restoring high-quality faces detail part.
19

---

Published as a conference paper at ICLR 2024
Input
PSFRGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEFR (Ours)
Input
PSFRGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEER (Ours)
Figure 15: Qualitative comparison on WIDER-Tes datasets. Our DAEFR method exhibits
robustness in restoring high-quality faces in detail part.
20

---

Published as a conference paper at ICLR 2024
Input
PSFRGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEFR (Ours)
Input
PSFRGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEFR (Ours)
Figure 16: Qualitative comparison WIDER-Test datasets. Our DAEFR method exhibits
robustness in restoring high-quality faces detail part.
21

---

Published as a conference paper at ICLR 2024
Gallery
Tapul
PSFRGAN
GEP-GAN
GPEN
RestoreFormer
CodeForerr
VQFR
DR2
DAEFR (OUrs)
Gallery
Input
PSERGAN
GFP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEER (Ours)
Figure 17: Qualitative comparison BRIAR-TH datasets. Our DAEFR method exhibits robust- ness in restoring high-quality faces detail part.
22

---

Published as a conference paper at ICLR 2024
Inpat
PSERGAN
GFP-GAN
GPEN
RestorFoorer
CodeFormer
VQFR
DR2
DAEPR (Our)
GT
Input
PSERGAN
GFP-GAN
GPEN
RestonFormer
CodeFormer
VOFR
DR2
DAEFR (Ours)
GT
Input
PSERGAN
GFP-GAN
GPEN
RestoreFormer
CodaFormer
VQFR
DR2
DAEER (Ours)
GT
Input
PSERGAN
GEP-GAN
GPEN
RestoreFormer
CodeFormer
VQFR
DR2
DAEFR (Ours)
CT
Figure 18: Qualitative comparison on CelebA-Test datasets. Our DAEFR method exhibits robust- ness in restoring high-quality faces in detail part.
23

---

Published as a conference paper at ICLR 2024
Lage Ocdusion
Large Pose
Input
RestomFormer
CodeFormer
VQER
DAEFR (Ours)
Figure 19: Qualitative comparison on large occlusion and pose situation. Our DAEFR method exhibits robustness in restoring high-quality faces in detail part.
Fuilure Case
Input
RestorFFomet
CodeFormer
VQFR
DR2
(Our
Figure 20: Failure case extreme pose situation. Our DAEER method fails in extremely pose situations.
24


Reviews:
[{"review_id": "duJ8fRFtIf", "reviewer": "Reviewer_UndL", "summary": "The paper presents \"DAEFR,\" a novel framework for blind face restoration, specifically focusing on restoring high-quality facial images from low-quality ones. This challenge arises due to complex and unknown sources of degradation in facial images. The key innovation in DAEFR is the introduction of an auxiliary low-quality (LQ) encoder, trained exclusively on LQ data. This branch captures domain-specific information from LQ inputs, addressing the domain gap and distinct feature representations between LQ and HQ images.\n\nDAEFR's methodology involves discrete codebook learning for both HQ and LQ domains, feature association between the HQ and LQ encoders, and a feature fusion stage that combines the information from both encoders. This approach is designed to overcome limitations in existing codebook methods, which often exhibit domain bias due to a reliance on encoders pre-trained solely on HQ data.\n\nThe paper's contributions are significant in several aspects:\n*  Introduction of the auxiliary LQ encoder for more accurate LQ domain feature representation.\n* Utilization of association and feature fusion methods to effectively bridge the domain gap between LQ and HQ images, enhancing restoration outcomes.\n* A comprehensive evaluation of DAEFR on synthetic and real-world datasets demonstrates superior performance in restoring facial details compared to existing state-of-the-art methods.\nOverall, the paper proposes a novel and effective approach to address the challenges of blind face restoration under severe degradation, emphasizing maintaining the fidelity and identity information present in the original LQ images.", "questions": "1. **Generalization to Other Image Types**: How well does the DAEFR method generalize to other images beyond facial images? Are there specific types of image degradation or different domains where DAEFR might not perform as effectively?\n\n2. **Handling Extreme Degradations**: Could you provide more insights into how DAEFR performs under extremely diverse or uncommon image degradations? Understanding its limitations in such scenarios would be crucial for practical applications.\n\n3. **Computational Efficiency**: Can you provide details regarding the computational efficiency of DAEFR, such as processing time and resource requirements? How scalable is the method for larger datasets or higher-resolution images?\n\n4. **Impact of Auxiliary Encoder**: Could you elaborate on the specific impact of the auxiliary LQ encoder in the restoration process? For instance, how does the restoration quality differ when the auxiliary encoder is not used?\n\n5. **Comparison with Diverse Restoration Methods**: The paper compares DAEFR with similar restoration approaches. Could you compare its performance with restoration techniques that use fundamentally different principles?\n\n6. **User Study for Qualitative Assessment**: Have you considered conducting a user study to evaluate the perceptual quality of the restored images? This could provide valuable insights into the real-world applicability of DAEFR.\n\n7. **Further Methodological Details**: Could you provide more technical details or insights on the feature association techniques used? How do they specifically contribute to bridging the domain gap between LQ and HQ images?\n\nResponses to these questions could greatly enhance the understanding of DAEFR's capabilities, limitations, and potential areas for future development.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's strengths are notable across various dimensions, including originality, quality, clarity, and significance:\n\n### Originality\n- **Innovative Approach**: Introducing an auxiliary LQ encoder specifically trained on LQ data is a creative solution to the domain gap problem in image restoration. This approach significantly differs from conventional methods, primarily relying on encoders pre-trained on HQ data.\n- **Feature Fusion and Association Techniques**: The application of feature association techniques and the subsequent fusion of features from both LQ and HQ encoders demonstrate a novel integration of ideas, enhancing the restoration process's effectiveness.\n\n### Quality\n- **Comprehensive Evaluation**: The method is thoroughly evaluated on synthetic and real-world datasets, providing substantial evidence of its effectiveness. The comparison with state-of-the-art methods further underlines the quality of the proposed approach.\n- **Robustness and Fidelity**: The paper demonstrates the robustness of the DAEFR method in preserving the identity and details in restored images, even under severe degradation conditions.\n\n### Clarity\n- **Well-Structured Presentation**: The paper is well-organized, with each section logically flowing into the next. The methodology, experimental setup, and results are clearly explained, making them accessible to readers.\n- **Effective Use of Visual Aids**: Including figures and tables aids in illustrating the methodology and showcasing the results, enhancing the overall clarity of the paper.\n\n### Significance\n- **Contribution to Blind Face Restoration**: The paper addresses a significant challenge in image restoration, particularly in restoring HQ images from LQ ones. The proposed solution can potentially influence future research directions in this area.\n- **Applicability and Impact**: The approach can greatly interest researchers and practitioners in computer vision, offering a novel tool for addressing a common problem in image restoration. Its applicability to real-world, severely degraded images enhances its significance.", "weaknesses": "The paper, while strong in many aspects, does have some areas that could be improved upon for a more comprehensive understanding and assessment of the proposed method:\n\n1. **Generalization to Other Domains**: The paper focuses on facial image restoration. It would be beneficial to see how the proposed DAEFR method performs in other domains of image restoration or different types of image degradation beyond facial images. This expansion could provide insights into the method's versatility and applicability in broader contexts.\n\n2. **Comparison with Varied Methods**: While the paper compares DAEFR with state-of-the-art methods, it primarily focuses on similar restoration approaches. Including comparisons with diverse methods, especially those using different underlying principles or restoration techniques could provide a more rounded evaluation of DAEFR's performance.\n\n3. **Limitation in Handling Extremely Diverse Degradations**: The paper could explore the limitations of the DAEFR method in handling extremely diverse or uncommon types of image degradations. Understanding these limitations would be critical for practical applications and could guide future improvements to the method.\n\n4. **Computational Efficiency and Scalability**: The paper does not delve deeply into the computational efficiency or scalability of the proposed method. Details on the computational resources required, processing time, and scalability to larger datasets or higher-resolution images would be valuable for assessing the practicality of DAEFR in real-world scenarios.\n\n5. **Detailed Analysis of Auxiliary Encoder's Impact**: While introducing an auxiliary LQ encoder is a key feature of the paper, a more detailed analysis of how this encoder improves the restoration process would be insightful. This could include comparisons between results with and without the auxiliary encoder or exploring its impact on different types of degradation.\n\n6. **User Study for Qualitative Assessment**: Incorporating a user study to supplement the quantitative evaluations could provide additional insights into the perceptual quality of the restored images. This would be particularly useful in understanding the real-world applicability of the method.\n\nAddressing these aspects could strengthen the paper's contributions and provide a more comprehensive understanding of the DAEFR method's capabilities and potential areas for further development.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yCTBYYuYoK", "reviewer": "Reviewer_huYM", "summary": "This paper presents a new approach called the Dual Associated Encoder for facial restoration. In this method, an auxiliary Low-Quality (LQ) branch is introduced to extract vital information from LQ inputs. Subsequently, it employs a structure similar to CLIP to establish connections between the LQ and High-Quality (HQ) encoders. This connection aims to reduce the domain gap and information loss when restoring HQ images from LQ inputs. The experimental outcomes illustrate the highly promising performance of this novel approach.", "questions": "1.\tWhile the paper predominantly highlights the advantages of the proposed method, could you offer instances where the method encountered shortcomings or limitations?\n2.\tCould you elaborate on the key distinction between DAEFR and CodeFormer?\n3.\tCan you provide further experimental details into the \"Effectiveness of Low-Quality Feature from Auxiliary Branch\" as examined in your ablation studies?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper offers a coherent and well-founded justification for the research, with a method design that closely aligns with the research objectives.\n2.\tThe paper effectively communicates the method, ensuring readers can easily comprehend the underlying concepts and techniques.\n3.\tThe experimental results showcase remarkable performance, affirming the method's efficacy in tackling the face restoration challenge.", "weaknesses": "1.\tAbsence of Future Research Guidance: The paper does not offer any recommendations or insights into potential future research directions or enhancements for the proposed method.\n2.\tOmission of Limitation Discourse: The paper lacks a discussion regarding its limitations and possible factors for analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HeqKDCFvhT", "reviewer": "Reviewer_qW23", "summary": "This paper introduces a dual-branch framework, named DAEFR, designed for the restoration of high-quality (HQ) facial details from low-quality (LQ) images. Within this framework, an auxiliary LQ encoder and an HQ encoder are employed in conjunction with feature association techniques to capture visual characteristics from LQ images. Subsequently, the features extracted from both encoders are combined to enhance their quality. Finally, the HQ decoder is utilized for the reconstruction of high-quality images. The effectiveness of DAEFR is evaluated using both real-world and synthetic datasets.", "questions": "Please discuss the concerns in the Weaknesses Section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The notion of incorporating an additional encoder with weight sharing is intriguing.\n\n2. The authors have extensively verified the significance of each component via thorough ablation studies.\n\n3. This approach adeptly addresses various common and severe degradations and maintains a high standard of writing quality.", "weaknesses": "1. Can you provide a detailed explanation of the primary differentiation between DAEFR and CodeFormer?\n\n2. The paper does not delve into its limitations or potential factors for analysis, which would greatly enrich its discussion.\n\n3. The paper outperforms baseline methods in the downstream face recognition task. Could you provide a comprehensive explanation of these results?\n\n4. The paper does not provide any suggestions or insights into potential avenues for future research or improvements to the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2RDZ0toPqb", "reviewer": "Reviewer_cKwn", "summary": "In this paper, the authors propose a framework, named dual associated encoder for face restoration (DAEFR), for face restoration. Specifically, different from the existing codebook based methods using only one autoencoder for high-resolution images, the authors propose to add another stream for low-resolution images. To fuse and align the features from both low and high resolution images, an association stage is designed. The associated features then will be extracted and utilized for face restoration.\n\nExperimental results have demonstrated the effectiveness of the proposed method.", "questions": "In Table 2, it seems like all the alter methods outperform the proposed method in terms of LPIPS. Please give discussions or visualizations to explain why this happens.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well written.\n2. The idea is well presented, explained, and demonstrated.\n3. The proposed method may inspire the researchers in this area.", "weaknesses": "1. The contribution looks marginal to me since all the methods used in different stage are well designed and demonstrated. Adding another stream for low-resolution might not be a major contribution for a top-tier venue like ICLR.\n2. I got some questions for the experimental results which can be seen in the questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i3mh9rqDnw", "reviewer": "Reviewer_X7AW", "summary": "To address the domain gap between low-quality and high-quality images and improve the performance of face restoration, the paper introduces a novel framework called DAEFR. This framework incorporates LQ (low-quality) image domain information by introducing an auxiliary branch that extracts unique LQ domain-specific features to complement the HQ (high-quality) domain information. To further align the features between the HQ and LQ domains, the paper employs a CLIP-like constraint to enhance the correlation between the two domains. Additionally, to facilitate better feature fusion between these two domains, the framework introduces a multihead cross-attention module. Evaluation results demonstrate the effectiveness of DAEFR.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper proposes a framework designed to incorporate distinctive features from low-quality (LQ) images, thereby enhancing the face restoration task.\n2.\tTo mitigate the domain gap between HQ and LQ images, the paper proposes an association strategy during training, and incorporates a multihead cross-attention module for better feature fusion between these two domains.\n3.\tThe experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed framework.", "weaknesses": "1.\tThe full name of the proposed framework, DAEFR, is missing. It should be mentioned on its first occurrence in the paper.\n2.\tThe proposed method requires training two sets of encoder and decoder for both HQ and LQ images. This will double the training resource requirements.\n3.\tI think it will be better if there is more elaboration on the domain gap issue that the current works exist, i.e., the motivation of the paper. Currently, it is not intuitive from figure 1 and from current discussion.\n4.\tCheck the spellings. For example, “recently” on the beginning of second paragraph in the “Vector Quantized Codebook Prior” of the related work.\n5.\tThere are some confusions about the training process of the network. In the first stage (section 3.1), you firstly train the two autoencoders of LQ and HQ using the codebook loss. After the first-stage training is complete, you train the two encoders using both the codebook loss and the association loss. Why not combine the two stages into one, or just apply the association loss in stage 2? Besides, in stage 3, you state in the Training Objectives that the MHCA and transformer module are trained in this stage. However, from figure 2(c), the two encoders seem not to be frozen during stage 3.\n6.\tThe results in Table 1 indicate that the proposed method does not significantly outperform other methods, especially for the synthetic CelebA-Test dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)