LD-BFR: Vector-Quantization-Based Face Restoration Model with Latent Diffusion Enhancement

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 08:35:04

Abstract: Blind Face Restoration (BFR) aims to restore high-quality face images from low-quality images with unknown degradation. Previous GAN-based or ViT-based methods have shown promising results, but have identity details loss once degradation is severe; while recent diffusion-based methods work on image level and take a lot of time to infer. To restore images in any degradation types with high quality and spend less time, we propose LD-BFR, a novel BFR framework that integrates both the strengths of vector quantization and latent diffusion. First, we employ a Dual Cross-Attention vector quantization to restore the degraded image in a global manner. Then we utilize the restored high-quality quantized feature as the guidance in our latent diffusion model to generate high-quality restored images with rich details. With the help of the proposed high-quality feature injection module, our LD-BFR effectively injects the high-quality feature as a condition to guide the generation of our latent diffusion model. Extensive experiments demonstrate the superior performance of our model over the state-of-the-art BFR methods.

Main paper: 
 introduction: 
 Blind face restoration (BFR) aims at recovering high-quality (HQ) face images from low-quality (LQ) face images with unknown degradation. Due to the ill-posed nature of this inverse problem and the diverse forms of degraded face images encountered in practice, it is highly desirable to develop a method that can faithfully restore degraded images into high-fidelity ones regardless of the type of degradation.In recent years, there have been significant improvements in restoration quality due to the rapid development of deep learningbased methods, which can be classfied into GAN-based, ViT-based, and diffusion-based methods. GAN-based and ViT-based methods utilize various priors to guide the restoration process, including geometric [1, 11, 19] , inference [3, 12, 13] , and generative priors [21, 25, 29] . Besides, [5, 22, 28] use high-quality codebooks to reconstruct high-quality faces with realness and fidelity. These methods show good generation quality in most scenarios, but may suffer from delicate identity features missing and hallucinating uncanny artifacts (as shown in Fig. 1 ). Moreover, they also suffer from the problem of training collapse and cannot deal with the dataset of long-tail distribution [15, 27] . Restored faces derived by these methods are prone to change person identities, and they are hard to achieve the balance between image restoration quality and character fidelity maintenance.Diffusion-based methods show good results in dealing with the dataset of long-tail distribution. Some restoration methods [18, 24] leverage the pretrained diffusion model to deal with face images of known degradation forms, but are unsuitable for BFR tasks. Other diffusion-based methods [2, 15, 23] can deal with unknown degradation forms and work for BFR tasks: ILVR [2] uses pixel-wise low-frequency guidance to remove degradation; DR2 [23] further combines the diffusion model and previous BFR methods; and Diff-BFR [15] combines the cascaded diffusion model and the unconditional model. However, none of these diffusion-based BFR methods work on latent space, have a high inference efficiency, and fully utilize the information in the input low-quality images.To achieve good performance by incorporating the advantages of both GAN-based and diffusion-based methods, we propose LD-BFR, a novel blind face restoration scheme based on the latent diffusion model and vector-quantized (VQ) codebook. 1) We propose a VQ-Restore and Diffusion-Enhancement pipeline which integrates VQ restoration with the conditioned latent diffusion model. It employs the VQ codebook for the first-stage restoration in a global manner, and then utilizes the latent diffusion model conditioned on the high-quality quantized feature for the secondstage to enhance the quality and identity details of the output image.2) In the VQ-Restoration stage, we design a Dual Cross-Attention VQ-Restoration module. It utilizes vector quantization by a highquality codebook and leverages the combination of Channel Cross-Attention and Spatial Cross-Attention to maintain good identity and texture information. 3) Then, we propose a Diffusion-based Quality Enhancement module. Different from previous methods which utilize conditions like geometries, 3D priors to guide the restoration process, we employ the high-quality quantized feature obtained from the VQ-restoration module as the condition to guide a pretrained diffusion model, to further restore the image details and boost the generation quality. To inject the high-quality quantized feature into the diffusion model, we propose a novel HQ Injection module, which injects based on self-attention in the Encoding stage and injects based on cross-attention in the Decoding stage. It first fuses the high-quality feature with the intermediate feature in the diffusion model and then injects the fused feature into the original intermediate feature, proven to be a more effective way to inject the high-quality feature.Extensive quantitative and qualitative experiments demonstrate that our model outperforms the existing GAN-based and diffusionbased BFR methods in different datasets. Moreover, comprehensive ablation studies validate the effectiveness of the proposed pipeline and each module. Our contributions can be summarized as the following three-fold:• We propose LD-BFR, a novel blind face restoration framework that integrates the Vector-Quantized restoration with the conditional latent diffusion model (with the high-quality quantized feature as a condition), which improves the restoration quality. Extensive quantitative and qualitative experiments demonstrate the superiority of our model. inference [3, 12, 13] , and generative priors [21, 25, 29] .(1) Geometric-prior-based methods typically leverage unique geometry and spatial distribution information of faces, e.g., facial heatmaps [11] , parsing maps [1, 19] , and component heatmaps [26] to help restore images. However, since the geometric priors are mainly generated from degraded faces, it's difficult to obtain accurate facial priors.(2) Inference-based methods [3, 13] guide the face restoration process by utilizing additional inference of the same identity as the degraded image. However, these highly requested inferences may not always be available. DFDNet [12] collects high-quality facial component features as inference priors to mitigate this problem.(3) Generative priors such as pre-trained StyleGAN [10] also have been utilized to further improve restoration quality. PULUS [14] utilizes latent optimization to optimize the latent code of pre-trained StyleGAN [9] . Furthermore, GPEN [21] , and GFP-GAN [25] embed generative priors into the encoder-decoder structure.(4) Among Non-prior based methods, the most effective methods are ViT-based, which employ pre-trained Vector-Quantize [4, 16, 20] codebooks. Restoreformer [22] , VQFR [5] , and CodeFormer [28] pre-train high-quality dictionaries on entire faces.The above methods can achieve good results, but most of them are GAN-based and ViT-based methods, they often suffer from problems of training collapse and a weak ability to deal with long-tail distribution. Diffusion Models. Denoising Diffusion Probabilistic Models (DD PM) [6] show amazing results in image generation. Leveraging the rich and diverse priors offered by the diffusion model, a diffusionbased image restoration method has been proposed. SR3 [18] modifies the structure of DDPM through channel-wise concatenation to make DDPM condition on low-resolution images. ILVR [2] leverages a low-pass filter to control the generative process of pre-trained DDPM for image-translation tasks. DDRM [24] assumes the degrading process is linearly reversible and utilizes SVD (Singular Value Decomposition) to help restore images with certain degradation. However, it cannot work once the degrading matrix is unknown. DR2 [23] proposes a two-stage image restoration method. After coarsely removing degradation by the ILVR process, it leverages an enhancement module to improve the restored results. However, previous diffusion-based methods only work for known degradation removal or conduct diffusion processes on image space, which makes the inference process slow. Our method is based on the latent diffusion model and injects the HQ-feature estimated by the Encoder of VQ-GAN to guide the restoration. Since the diffusion model is good at detail enhancement and coping with long-tail distribution, we combine the VQ-GAN model with the diffusion module, using the VQ-GAN model for HQ-feature extraction, and the diffusion module for further feature enhancement. Therefore, our LD-BFR can take into account both the efficiency of reasoning and the restoration of identity details. 
method 3.1 overview: 
 The existing blind face restoration methods mainly employ the GAN or Diffusion model as the main framework to restore the degraded images. However, GAN-based or ViT-based models suffer from the problem of identity details lost, and diffusion models have a low inference efficiency. Few methods try to combine these two types of methods or fully utilize the information in the input lowquality images. To address these issues, we propose LD-BFR, a novel BFR framework that combines Vector-Quantized Restoration and Diffusion models, using the restored high-quality quantized feature as the bridge. The proposed model leverages the strength of vector quantization to ensure perceptual compression without massive information loss and offers high-quality texture information. It also leverages the strength of the diffusion model to help deal with long-tail distribution and further restore the identity information.Specifically, our LD-BFR consists of two stages: Dual Cross-Attention VQ Restoration stage (DC-VQR), and Diffusion-Based Quality Enhancement stage (DQE). Stage 1: DC-VQR aims to extract HQ-feature by a VQ-GAN model trained on LQ-HQ pairs. The VQ-GAN takes a low-quality degraded image 𝑥 𝐿𝑄 as input, initially utilizing the encoder 𝐸 𝐿𝑄 to obtain the compressed low-quality feature 𝑓 𝐿𝑄 . Subsequently, we acquire an intermediate high-quality feature by fetching the closest feature from the high-quality codebook of the pre-trained VQ-GAN. Then, both the intermediate high-quality feature and the low-quality feature are input into our Dual Cross-Attention module, which combines Channel Cross-Attention and Spatial Cross-Attention, and outputs the enhanced high-quality feature 𝑓 𝐻𝑄 , which encapsulates comprehensive information from the input degraded image and enables an initial restoration.Stage 2: DQE aims to further boost the restoration efficacy and enhance the similarity between the identities of the input and restored images, by employing a latent diffusion model with the enhanced high-quality feature 𝑓 𝐻𝑄 as a condition. In the denoising process, we first sample a random noise 𝑧 𝑇 from N (0, 𝐼 ). Then, leveraging 𝑓 𝐻𝑄 obtained from Stage 1 as a condition, we integrate it into the UNet model using our proposed High-Quality Feature Injection module (HQI), which injects HQ-feature based on selfattention in the Encoding stage and injects based on cross-attention in the Decoding stage. Finally, we use the Decoder from the original VQGAN of the pretrained LDM to upsample the latent feature for a high-definition and high-quality output.In summary, the dual cross-attention Encoder first compresses the degraded face image. Then the diffusion model restores and enhances the quality and the identity details of the HQ feature. Finally, the vector-quantized Decoder upsamples the feature to get high-quality images.Vector Quantization (VQ) [4] has shown its effectiveness in image inpainting, image translation, and image restoration [5, 22, 28] . The quantized codebook usually can ensure a good generation quality, since the features in the codebook are learned from highquality real images. Therefore, we employ the vector-quantized (VQ) codebook as the basic module to restore the degraded image initially. Specifically, our Vector-Quantized Restoration module (VQR) is composed of: 1) a Low-Quality Encoder 𝐸 𝐿𝑄 that encodes the low-quality degraded image 𝑥 𝐿𝑄 , and quantized feature fetched from a HQ codebook 𝐿 𝐻𝑄 , and 2) a quality enhancing Dual Cross-Attention Module 𝐷𝐶𝐴𝑀 that boosts identity preservation and avoids color shift issues mentioned in previous works [28] . The Low-Quality Encoder 𝐸 𝐿𝑄 and Dual Cross-Attention 𝐷𝐶𝐴𝑀 are trained together as shown in Fig. 3 . HQ encoder, HQ decoder, HQ dictionary It should be pointed out that the codebook 𝐿 𝐻𝑄 is trained on high-quality face images form FFHQ, and the Decoder used here is different from the Decoder used in the final stage of the Diffusion-Enhancement module.Low-Quality Encoder and Quantized Feature Fetching. To process the input degraded image 𝑥 𝐿𝑄 , we introduce a perceptual encoder 𝐸 𝐿𝑄 to encode the degraded image 𝑥 𝐿𝑄 into low-quality feature 𝑓 𝐿𝑄 ∈ 𝑅 𝑐 ×ℎ×𝑤 , the training details is shown in Sec. 4.2. After the low-quality feature 𝑓 𝐿𝑄 is obtained, we fetch the closest feature 𝑓 ′ 𝐻𝑄 from the high-quality codebook 𝐿 𝐻𝑄 along dimension ℎ and 𝑤, i.e., fetching ℎ × 𝑤 vectors of dimension 𝑐 from the codebook. Since all the vectors from the codebook are learned from high-quality images, this quantized feature 𝑓 ′ 𝐻𝑄 captures the features of highquality images and can help improve the quality of the restored images.Dual Cross-Attention Module. The fetched quantized feature 𝑓 ′ 𝐻𝑄 would inevitably lose some identity information due to a large compression ratio and also have a color shift problem due to the quantization and fetching operation. To solve these issues, we introduce a Dual Cross-Attention module, which consists of the combination of Channel Cross-Attention and Spatial Cross-Attention. It takes the low-quality feature 𝑓 𝐿𝑄 (for identity information and color distribution) and the fetched high-quality quantized feature 𝑓 ′ 𝐻𝑄 (for high-quality texture information) as inputs, and outputs the refined high-quality feature 𝑓 𝐻𝑄 . Our dual cross-attention module is composed of a parallel spatial attention module and a channel attention module.Specifically, in the spatial cross attention module, 𝑓 𝐿𝑄 is mapped into query 𝑄 = 𝑊 𝑄 𝑓 𝐿𝑄 and 𝑓 ′ 𝐻𝑄 is mapped into key 𝐾 = 𝑊 𝐾 𝑓 ′ 𝐻𝑄 and value 𝑉 = 𝑊 𝑉 𝑓 ′ 𝐻𝑄 , and the refined feature is generated by𝑓 𝑆𝐶𝐴 = 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 ( 𝑄 ×𝐾 𝑇 √ 𝑑) × 𝑉 . In the channel cross-attention module, we get channel matrix 𝑀 𝐶 from 𝑓 𝐿𝑄 , and the refined feature is generated by 𝑓 𝐶𝐶𝐴 = 𝑀 𝐶 × 𝑓 ′ 𝐻𝑄 . Finally, we add the two features together to get the refined high-quality feature 𝑓 𝐻𝑄 = 𝑓 𝑆𝐶𝐴 + 𝑓 𝐶𝐶𝐴 .With the proposed Vector-Quantized Restoration (VQR) module, our model can compress the degraded images into high-quality features, though we don't use a large compression ratio, some identity details are missing due to degradation. To restore identity details like freckles or eyelashes and solve long-tail distribution datasets, we propose to combine VQ restoration with diffusion by innovatively utilizing restored high-quality feature as the condition for diffusion.Different from previous methods that utilize 3D information prior [7, 29] or identity information feature [3, 13] as the condition, our Diffusion-based Quality Enhancing module (DQE) is the first model that leverages the information restored high-quality quantized vector information (obtained in DC-VQR) as the condition, information which has the advantage of containing the features of high-quality facial details. Moreover, we propose a novel High-Quality Feature Injection module (HQI), which can better inject the high-quality feature into the latent diffusion model to restore the low-quality image compared to previous condition injection methods [17] . The main framework of our DQE is shown in Fig. 2 and the details are introduced as follows:HQ code-conditioned Latent Diffusion Model. Our DQE is built on a latent diffusion model with the condition as the restored high-quality quantized feature obtained from the Vector-Quantized Restoration module 𝑓 𝐻𝑄 = 𝑉 𝑄𝑅(𝑥 𝐿𝑄 ). Different from previous methods that utilize geometry or identity information as the condition, our restored HQ quantized feature 𝑓 𝐻𝑄 captures the information of high-quality facial details, which can better guide the diffusion model to generate high-quality restored images.Specifically, during training, we first encode ground truth highquality images x into high-quality feature f 𝐻𝑄 by 𝐸 𝐻𝑄 (from HQ VQGAN) and add 𝑡-step noise to f 𝐻𝑄 in the diffusion process. Then, in the denoise process, we utilize the restored HQ quantized feature 𝑓 𝐻𝑄 = 𝐷𝐶-𝑉 𝑄𝑅(𝑥 𝐿𝑄 )(from Compression Encoder) as the condition to guide the diffusion UNet 𝜇 𝜃 . We inject the restored HQ feature 𝑓 𝐻𝑄 into the diffusion UNet 𝜇 𝜃 with the HQ Injection module, keeping the output identity the same as the restored highquality feature 𝑓 𝐻𝑄 and boosting its quality. In the inference stage, we randomly sample a noise latent 𝑍 𝑇 from N (0, 𝐼 ) and denoise it with the condition of 𝑓 𝐻𝑄 by : where 𝜇 𝜃 (𝑧 𝑡 , 𝑓 𝐻𝑄 , 𝑡) is our HQcode-conditioned UNet model that takes both the noisy latent 𝑧 𝑡 and HQ feature 𝑓 𝐻𝑄 as inputs and outputs the mean of 𝑧 𝑡 -1 .EQUATIONThe HQ Injection module. To inject the restored HQ feature 𝑓 𝐻𝑄 into diffusion, we propose a High-Quality Feature Injection module (HQI), a simple yet effective module that can effectively inject the information into the diffusion model without influencing the original generation quality. Different from Latent Diffusion Model that solely relies on the cross-attention module to inject condition, our HQI module utilizes both cross-attention and selfattention modules for different merge purposes and better condition injection.Our HQI module can be divided into Encoder-HQI (E-HQI) and Decoder-HQI (D-HQI). 1) Self-Attention Based Encoder-HQI: During the encoding stage, the input HQI firstly uses pixel unshuffle modules adjust to the condition 𝑓 𝐻𝑄 and concatenate it with the intermediate feature 𝑓 𝑖 . Then it uses a convolution layer and a self-attention module to make sure that the texture information contained in the condition is properly merged. The output feature is 𝑓 𝑜 = 𝑓 𝑖 + 𝑆𝐴(𝐶𝑜𝑛𝑣 (𝐶𝑜𝑛𝑐𝑎𝑡 (𝑓 𝑖 , 𝑃𝑖𝑥𝑈 𝑆 (𝑓 𝐻𝑄 ))). 2) Cross-Attention based Decoder-HQI: Then in the decoding stage, we further use Decoder HQI to inject semantic and identity information from our condition 𝑓 𝐻𝑄 . We first use an Adaptive Layer Normalization (AdaLN) to adjust the input feature, where the scale and shift parameters of AdaLN are obtained from 𝑀𝐿𝑃 (𝑓 𝐻𝑄 ). Then it uses a Cross-Attention module to merge the semantic and identity information of the condition 𝑓 𝐻𝑄 . The output feature is 𝑓 𝑜 = 𝑓 𝑖 + 𝐶𝐴(𝐴𝑑𝑎𝐿𝑁 (𝑓 𝑖 , 𝑀𝐿𝑃 (𝑓 𝐻𝑄 )), 𝑓 𝐻𝑄 ). During training, the module of the original diffusion model is frozen. Therefore, we do not compromise the high-quality prior-generation capability of the original diffusion model.The reason for the design of SA-based E-HQI and CA-based D-HQI is that: During the Encoding stage, the model needs to compress the texture information and extract identity features, while the input is a noised feature initially, which necessitates us to obtain texture information from the HQ feature. Therefore, the Encoder-HQI first merges the HQ feature and the intermediate feature, and then conducts self-attention on the merged feature to make sure the texture information in HQ feature is properly injected. In the Decoding stage, we need to avoid color shifts caused by incorrect texture combinations. Therefore, the Decoder-HQI first uses an AdaLN module, and then leverages a cross-attention module to inject semantic and identity features from the HQ feature.High-Quality Decoder from Pretrained VQ-GAN. We employ the Decoder of the pretrained VQ-GAN from latent diffusion model (LDM) [17] as the upsample model after the diffusion stage. It should be noted that this VQ-GAN is different from the 𝐸 𝐻𝑄 and 𝐷 𝐻𝑄 in DC-VQR module.Dual Cross-Attention Vector-Quantized Restoration module.In the training process of our Vector-Quantized Restoration module, we first employ the pretrained VQGAN model [17] on the FFHQ dataset [9] , including the high-quality encoder 𝐸 𝑂,𝐻𝑄 , codebook 𝐵 𝐻𝑄 and the decoder 𝐷 𝐻𝑄 . The decoder is used as our upsample decoder which can generate high-resolution images from the feature restored by the diffusion model.Then, we train our low-quality encoder 𝐸 𝐿𝑄 (also it's high-quality codebook 𝐿 𝐻𝑄 ) and the dual cross-attention module 𝐷𝐶𝐴𝑀 on the paired data of degraded image 𝑥 𝐿𝑄 and the corresponding ground truth high-quality image x𝐻𝑄 . This VQ-GAN model has a different compression ratio from the aforementioned high-quality encoder 𝐸 𝑂,𝐻𝑄 , and it also has a corresponding high-quality encoder 𝐸 𝐻𝑄 and HQ codebooks𝐿 𝐻𝑄 . To ensure the high-quality quantized feature 𝑓 𝐻𝑄 output from 𝐷𝐶𝐴𝑀 is close to the ground truth highquality feature f 𝐻𝑄 = 𝐸 ′ 𝐻𝑄 ( x𝐻𝑄 ). Specifically, we compute the MSE distance between 𝑓 𝐻𝑄 and f 𝐻𝑄 by:EQUATIONMoreover, we additionally add a constraint in the pixel space by computing the MSE distance of the decoded output 𝑥 𝐻𝑄 = 𝐷 ′ 𝐻𝑄 (𝑓 𝐻𝑄 ) and the ground truth high-quality image x𝐻𝑄 by:EQUATIONFinally, our low-quality encoder 𝐸 * 𝐿𝑄 and the quality enhancing cross-attention module 𝐷𝐶𝐴𝑀 * (Compression Encoder module) are trained by:EQUATIONDiffusion-Based Quality Enhancing module. To enhance training stability, our latent diffusion model is trained with two stages, where the first stage is trained as an unconditional model, and we add the HQ Injection module in the second stage. In the first stage, we encode the ground truth high-quality images into high-quality feature f 𝐻𝑄 = 𝐸 𝐻𝑄 ( x𝐻𝑄 ), and randomly add 𝑡-step noise 𝜖 into f 𝐻𝑄 to get 𝑧 𝑡 , and denoise it with UNet. We minimize the distance between the added noise and the predicted noise to train the unconditional UNet model:EQUATIONwhere 𝜃 1 is the trainable parameters in the unconditional UNet model. With the unconditional UNet model being trained, our latent diffusion model can generate high-quality results by conducting the denoising process. Then, to enhance the identity similarity, we train the HQ Injection module, where we add 𝑡-step noise to the ground truth high-quality feature f 𝐻𝑄 to get 𝑧 𝑡 , and then denoise it conditioned on the restored high-quality feature 𝑓 𝐻𝑄 = 𝑉 𝑄𝑅(𝑥 𝐿𝑄 ). The HQ injection module 𝜇 𝜃 2 can be trained by:EQUATIONwhere 𝜃 2 is the parameters of our proposed HQ Injection module and 𝜃 = (𝜃 1 , 𝜃 2 ). 
experiment 4.1 datasets and implementation: 
 Datasets. For the training process, we use the FFHQ dataset [9] as our training dataset which consists of 70,000 high-quality face images. We follow eq (7) to synthesize the degraded images where 𝜎, 𝑟, 𝛿 and 𝑞 are randomly sampled from {0.2 : 10}, {1 : 8}, {0 : 15} and {60 : 100}. And for the testing process, we evaluate our LD-BFR on one synthetic dataset (CelebA-Test), and two real-world datasets (LFW-Test and WIDER-Test). A brief introduction of each dataset is shown as below:(1) CelebA-Test consists of 3,000 images and is synthesized by applying the degradation model which is commonly used in previous works [13, 21] on the testing set of CelebA-HQ images. The degrading model is as below:EQUATIONA high-quality image 𝑥 is firstly blurred by a blur kernel 𝑘 𝜎 . Then a scale factor 𝑟 is used to bicubically downsample the image and additive noise is added after downsampling. 𝑛 𝛿 is the added noise and is randomly chosen from Gaussian and Poisson. Finally, applying a JPEG compression with quality factor 𝑞 to generate the final degraded image 𝑦. For a fair comparison, We follow the (3) WIDER-Test. We use the WIDER-Test dataset offered by CodeFormer [28] . It comprises 970 severely degraded face images from the WIDER Face dataset, providing a more challenging dataset to evaluate the generalizability and robustness of blind face restoration methods. is shown in Fig. 5 , our approach is very good at maintaining some identity details, such as earrings and jewelry which are ignored by the GAN-based method. Due to the combination of VQ-GAN and Diffusion model via the restored HQ feature, our method not only improves the details of the restored face images but also is good at dealing with severe degradation. Comparison on Real-world Dataset. As is shown in Tab. 2, our method achieves the highest scores of FID on the LFW-Test and WIDER-Test datasets. This shows that LD-BFR can handle realworld datasets with different degradation. From the visual comparisons in Fig. 6 , it can be found that our method is robust to real severe degradation and produces visually pleasant results.Importance of dual Cross-Attention module. To verify the effectiveness of the dual Cross-Attention module, we train a LQ Encoder without dual cross-attention and a LQ encoder with dual cross-attention under the same settings. To avoid the influence of the Diffusion model, we test both encoders without the diffusion process and use the same decoder and codebook. The visual comparison is shown in Fig. 7 . The results indicate that the dual cross-attention module effectively enhances the restoration quality and identity preservation. It solves the color shifts problem of restored images. Effectiveness of Diffusion-based Quality Enhancing. We investigate the effectiveness of the Diffusion process on the image with severe degradation. We show the qualitative comparison in Fig. 8 , where the results indicate that diffusion enhancement is the key to ensuring the robustness and effectiveness of our method. It quality-enhancing process further removes degradation and enhances the restored feature. The restored face has severe artifacts without the diffusion process (DQE) once the degradation is severe.Effectiveness of our HQI settings. To show the effectiveness of our input HQI and output HQI, we trained four models under the same conditions. These four models use the same VQ model to compress features and upsample, they all use the U-Net structure but use different methods to inject features. Their structures are shown in Fig. 9 : model A uses concat module to inject feature, As shown in Fig. 10 , model A simply uses the conv layer to inject the condition. Its restored images have some unbearable artifacts and have identity details lost. Model B only uses self-attentionbased Encoder-HQI to inject the condition, its restored images suffer from the problem of color shift, which may be caused by unseasonable texture information merge. Model C only uses crossattention-based Decoder-HQI to merge the condition, it keeps the high-quality details but changes the fidelity. From the visualization, our HQI helps accurately merge and restore high-quality facial details to better match the degraded input. 
conclusion: 
 In this paper, we propose LD-BFR, a novel framework integrating both the strengths of vector quantization and latent diffusion, which can restore images with high quality. By leveraging vector quantization, we can ensure a good generation quality of the restored image. Moreover, we introduce a latent diffusion conditioned on the quantized feature by our high-quality injection module, which further improves the generation quality and boosts identity restoration. Extensive experiments demonstrate the superior performance of our LD-BFR over the existing methods. 
 
 Appendix:  


Decision: This work combines vector quantization and latent diffusion models for face restoration, which achieves state-of-the-art performance. Initially, this paper received two borderline accept, one borderline reject, and one weak reject. After the rebuttal, all the reviewers were happy with the authors' response, and the final ratings are three borderline accept and one weak accept. There are clear technical merits of this paper, and we would recommend acceptance to this paper.

Reviews: review: paper_summary: paper_summary: The paper proposes LD-BFR for Blind Face Restoration (BFR) task that combines vector quantization and latent diffusion to restore high-quality facial images from low-quality inputs with unknown degradation. LD-BFR employs a Dual Cross-Attention vector quantization module for global restoration, followed by a latent diffusion model enhanced with high-quality quantized features for detailed image generation. This approach aims to overcome the limitations of existing GAN-based and diffusion-based methods, such as identity detail loss and inefficiency. Extensive experiments demonstrate the superior performance of LD-BFR in maintaining fidelity and quality across various degradation types.
strength_weakness: strength_weakness: This paper is well-organized and straightforward. The quantitative and qualitative experiment results clearly demonstrate the advantages of the proposed pipeline, particularly showing the benefits of the Dual Cross-Attention VQRestoration (DC-VQR) module. The experiments are not only conducted on the common CelebA test dataset, but also on the LFW-Test and WIDER-Test datasets to show the robustness of this method. Furthermore, the sufficient ablation study of the HQI module shows the effectiveness of the structure design.
limitations: limitations: [1]. Why does the PSNR of GPEN reach 22.8 in Table 1? The qualitative results of GPEN on the CelebA-Test look not good enough.

[2]. Are there any quantitative results of the ablation study? The comprehensive quantitative result of each module can better prove the effectiveness of your structures.

[3]. Could you show the GT on the ablation study for better comparison?

[4]. What's the step number of the denoising process during the inference? How does the step number affect your final visual results? Could you show the inference time of different methods? Is your method much slower than the non-diffusion-based model?
paper_summary: paper_summary: The paper proposes a blind face restoration framework (LD-BFR) combining vector quantization and latent diffusion. Specifically, LD-BFR first restores degraded face images globally using the proposed dual cross-attention vector quantization, then generates high-quality images with rich details through a latent diffusion model guided by restored quantized features. The proposed high-quality feature injection module enhances restoration quality. The experiments show that the proposed method achieves competitive performance.
strength_weakness: strength_weakness: 1. The paper is overall well-written.
2. The proposed method achieves competitive performance in face restoration tasks.
limitations: limitations: 1. Unclear motivation and lack of insights. The proposed method is a combination of the vector quantization technique and the diffusion model. The vector quantization technique has been extensively studied in the CodeFormer and VQFR.
2. Novelty. In the proposed method, the core module is the proposed dual cross-attention. However, channel and spatial attention are very common in recent literature.
3. Lacking comparison methods. Recent diffusion methods like DiffBFR [1] and StableSR [2] are not used for comparison. 
   [1]. Lin et al., DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
   [2]. Wang et al., Exploiting Diffusion Prior for Real-World Image Super-Resolution.

4. As a diffusion-based method, the efficiency is important in applications. However, the authors do not provide the running efficiency comparison among different methods.
paper_summary: paper_summary: The paper introduces a novel Blind Face Restoration (BFR) framework called LD-BFR, which aims to restore high-quality face images from low-quality images with unknown degradation. The existing GAN-based or ViT-based methods have shown promising results but suffer from identity details loss when the degradation is severe, while recent diffusion-based methods are time-consuming. LD-BFR integrates vector quantization and latent diffusion to address these issues.
strength_weakness: strength_weakness: (+) The idea of Integration of vector quantization and latent diffusion for efficient face restoration.
limitations: limitations: (-) The author concludes this method can deal with long-Tari distribution and inference efficiency problems by combining VQ and LDM. However, this cannot support the paper's motivation convincedly. Some methods have already been introduced for BFR [1,2,3]. Besides. There are already numerous works in FR with VQ+LDM-based methods like [4].  

(-) Although the author provides some visual results of DCAM/DQE/HQI modules, they seem just brown from some existing works. Besides, no quantitative comparison results are provided to support its effectiveness. The results are not convincing. To me, this work simply replaces the previous VQFR with an LDM without any other insightful designations.

[1] Efficient Diffusion Model for Image Restoration by Residual Shifting

[2] Towards Real-World Blind Face Restoration with Generative Diffusion Prior

[3] CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using Score-Based Diffusion Models

[4] Image Super-Resolution via Latent Diffusion: A Sampling-Space Mixture of Experts and Frequency-Augmented Decoder Approach

Additional suggestions:
 Line#114-123. The author should not include diffusion-based methods (#Line110). 
Fig.9 and Fig.10 are confusing. It’s hard for reader to link the relations between two figures. 
Fig.10 Caption A-D
paper_summary: paper_summary: This is a work on BFR that utilizes VQGAN to do preliminary processing in the latent space and further repair in the latent space using LDM. In the preliminary processing of VQGAN, the training model of VQGAN is directly used, inheriting the Encoder and Codebook parts. The Dual Cross-Attention vector quantization is proposed to combine the details of the dictionary thrower and the ID information of the LQ as the final guidance of the LDM. In the process of further restoration of the latent space by LDM, compared with the conditional embedding approach in the past, the paper proposes the high-quality feature injection module, which uses different embedding methods in the downsampling and upsampling layers of U-Net. This article is clearly structured and descriptive, and puts forward its own ideas about both guidance content and embedding of DMs, and verifies the feasibility by comparing them in the experimental results.
strength_weakness: strength_weakness: (1) The content description of the paper is very clear, and the detailed illustrations of each stage are also very clear.

(2) In the content setting of guidance, it tries to use the way of dictionary-based preprocessing, which is more accurate for the restoration of excessively degraded face images.

(3) In the embedding part of conditional, it innovatively puts forward the scheme with different embedding forms in the up-sampling and down-sampling stages, but it does not specifically analyze and explain the feasible reasons in it.
limitations: limitations: (1) The abstract mentions “To restore images in any degradation types with high quality and spend less time”, may I ask which aspect does the less time here specifically refer to, and the article does not follow up with a corresponding find. 

(2) There is no comparison between Restoreformer and DiffBIR in experiments, which are the two more classical comparison methods.

(3) Methods such as GFPGAN, VQFR, etc. have been done on CelebChild, a real-world dataset, can you provide the experimental results on this dataset? 

(4) Can you provide the quantitative comparison results of the ablation experiments on the CelebA dataset to see the extent of the role of each module from a quantitative point of view？

(5) In Figure 9 and Figure 10 when the ablation comparison on the HQI module is given, is there any quantitative comparison of the results of this part, or the convergence plot of the training loss? 

(6) The channel cross-attention in DC-VQR, why is it set up like that in the article? What does $W*f$ mean?

(7) How is the whole model trained here? Because the model has two phases and the output of the first phase is conditional on the second phase.