ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration
Authors: Chi-Wei Hsiao
Yu-Lun Liu
Cheng-Kun Yang
Sheng-Po Kuo
Kevin Jou
Chia-Ping Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Blind face restoration
Diffusion models
Image restoration
Reference-based
Latent diffusion model
Primary area: diffusion_based_models

Abstract: While recent works on blind face image restoration have successfully produced impressive high-quality (HQ) images with abundant details from low-quality (LQ) input images, the generated content may not accurately reflect the real appearance of a person. To address this problem, incorporating well-shot personal images as additional reference inputs may be a promising strategy. Inspired by the recent success of the Latent Diffusion Model (LDM) in image generation, we propose ReF-LDM—an adaptation of LDM designed to generate HQ face images conditioned on one LQ image and multiple HQ reference images. Our LDM-based model incorporates an effective and efficient mechanism, CacheKV, for conditioning on reference images. Additionally, we design a timestep-scaled identity loss, enabling LDM to focus on learning the discriminating features of human faces. Lastly, we construct FFHQ-ref, a dataset consisting of 20,406 high-quality (HQ) face images with corresponding reference images, which can serve as both training and evaluation data for reference-based face restoration models.

ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration
Chi-Wei Hsiao!
Yu-Lun Liu²
Cheng-Kun Yang' Chia-Ping Chen'
Sheng-Po Kuo
Yucheun Kevin Jou'
MediaTTek
National Yang Ming Chiao Tung University
Abstract
While recent works on blind face image restoration have successfully produced impressive high-quality (HQ) images with abundant details from low-quality (LQ) input images, the generated content may accurately reflect the real appearance of a person. To address this problem, incorporating well-shot personal images as additional reference inputs could be promising strategy. Inspired by the recent success of the Latent Diffusion Model (LDM), we propose ReF-LDM an adapta- tion of LDM designed to generate HQ face images conditioned on one LQ image and multiple HQ reference images. Our model integrates an effective and efficient mechanism, CacheKV, to leverage the reference images during the generation process. Additionally, we design timestep-scaled identity loss, enabling our LDM-based model to focus on learning the discriminating features human faces. Lastly, we construct FFHQ-Ref, a dataset consisting of 20,405 high-quality (HQ) face images with corresponding reference images, which can serve as both training and evaluation data for eeference-based face restoration models.
Introduction
Recent works [5] 26 32 have achieved impressive results in generating realistic high-quality (HQ) face image from an input low qualitalty (LQ) image. However, the important features person's 's face may be corrupted in the LQ image, and thus the reconstructed image may look like different person. To tackle this problem, besides the LQ image, additional HQ images of this person can be used as reference input. Moreover, allowing multiple reference images may lead to better quality because they offer more comprehensive appearance of this person in different conditions, c.g., different poses, expressions, lighting.
previous work [66] has explored using multiple reference images for face restoration. Their method, however, depends face landmark model to detect facial components (i.e., eyes, nose, and mouse), which may become unreliable when the input LQ image is severely degraded. Besides, latent diffusion model (LDM) (22] has also been used in different image generating tasks with different input conditions, such as low-resolution images, semantic maps, or sketch images 22 29
Inspired by the recent success of LDM, we propose ReF-LDM for reference- face image restoration. Unlike previous conditional LDM methods where their input conditions are usually spatially aligned with the target image, the reference images are not aligned with the target HQ image in our case. Therefore, we design a CacheKV mechanism, which effectively and efficiently integrates the reference images, albeit with different poses and expressions. Furthermore, timestep-seadt identity loss to drive the reconstructed image to look like the same person of the LQ and reference images. Lastly, we also construet new large-scat scale dataset of face images with corresponding reference images, which can serve as both training and evaluation datasets for future reference-based face restoration research.
introduce
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Input LQ image
(b) LDM
(c) ReF-LDM
(d) Input reference images
Figure 1: Reference-based face image restoration. Given input low-quality face image (a), Latent Diffusion Model (LDM) can reconstruct a high-quality image (b); however, it may not be faithful the individual's 's facial identity. To address this problem, we propose ReF-LDM, which restores a high-quality image with faithful details (c) by utilizing additional reference images (d).
With the above components, ReF-LDDM outperforms recent state-of-the-art methods with significant improvement in face identity similarity. Extensive ablation studies for the proposed CacheKV mechanism and timestep-scaled identity loss are also conducted and reported. The main contributions of this work can be summarized as:
We propose ReF-LDM, which features effective and efficient CacheKV mechanism, for restoring an LQ face image using multiple reference images.
We introduce a timestep-scaled identity loss, which considers the characteristics diffusion models and helps ReF-LDM better learn the discriminating features of human identities.
We construct FFHQ-Ref, dataset comprising 20,406 high-quality face images and their corre- sponding reference images, facilitate the advance of reference-based face image restoration.
2
Related work
Face restoration without personal reference images Numerous studies have been proposed for blind face image restoration [28
Recent works such VQFR and CodeFormer [32) have achieved promising results by exploiting VQGAN, while DAEFR 26 further employs dual-branch encoder mitigate the domain gap between LQ and HQ images. Inspired by
I3
26
the success of diffusion models, several works 23 31
27 25 have adopted diffusion models for
face image restoration. However, as these methods not leverage reference images, the restored images may differ from the authentic facial appearance of person, especially when an input image severely degraded.
Face restoration with personal reference images Several methods 14 15] 16 19 have attempted
to utilize additional reference images to enhance personal fidelity in face restoration. GFRNet [14 warps single reference image match the face pose of the LQQ image, while ASFNet [15 selects the reference image with the closest matching facial landmarks to serve as the network input. Closer to the setting of this work, DMDNet [16] also utilizes multiple reference images. It detects facial

---

landmarks on the LQ image and the reference images to extract features of facial components, and then integrates these features into the model by querying the corresponding components. However, their method relies landmark detection, which may not be robust on severely degraded I.Q images. In contrast, our ReF-LDDM implicitly learns the correspondences between the features of the LQ image and the reference images, without the need for landmark detection. From different perspective, MyStyle [9] adopts per-person optimization setting, leveraging hundreds of images of an individual to define personalized subspace within the latent space of StyleGAN [2]. In comparison, approach offers greater flexibility, capable of utilizing one to several reference images without the need for personalized model optimization for each individual.
Latent diffusion models with image conditions Previous work demonstrates that LDM can generate an image from a low-resolution image by simple channel-axis concatenation [22]. However, reference images in our task are not spatially aligned with the target HQ image, thus requiring more sophisticated integration mechanism. MasaCtrl [ achieves text-to-image synthesis with single reference image by replacing the original keys and values tokens with those from the reference image. However, their solution requires passing the reference image through the denoising network for multiple timesteps, which increases computation and limits feasibility for extending to multiple reference images. In contrast, propose efficient CacheKV mechanism that leverages multiple reference images by eliminating the redundant network passes.
The proposed ReF-LDM model
In this section, we present the proposed ReF-LDM model. We introduce the network architecture in Sec. [.1] where CacheKV mechanism is designed leverage reference images. We illustrate how to train our model with the timestep-scaled identity loss in Sec. 3.2
Main denoising process
times
Denoising U-net
X HQ
V(o,1)
Q
Z
Zo
D
KV
X
Concat Channel
Concat KV Token
Train-time
L
CacheKV
RV
root
X
(x
3
Frozen Autoencoder
KV
KV
Zucl
Denoising
not
time
Figure 2: The proposed ReF-LDM pipeline. Our model accepts a low-quality image and multiple high-quality reference images input and generates high-quality image. The blue top panel alone represents a typical LDM (2 denoising process. For an I.Q image XLQ. we concatenate its latent Z.o with 2, along the channel axis serve the input for the denoising U-net. For the reference images (Xret) we design a CacheK mechanism, depicted in the red panel, to extract and cache their key and value tokens using the same denoising U-net for just one time. These cached KV tokens can then be utlized repeatedly in each of the 7 timesteps of the main denoising process. During training, we adopt the classic LDM loss (LLDM) and introduce timestep-scat scaled identity loss (Ctme ID).

---

3.1 Model architecture of ReF-LDM
The proposed ReF-LDM accepts an input LQ image and multiple reference images to generate a target HQ image. Its model architecture is based on the latent diffusion model [22], with additional designs to incorporate the input LQ image and the reference images.
3.1.1 Preliminaries on Latent Diffusion Model
To generate an image, an image diffusion model 8 starts from a noisy image Xy E R initialized with a Gaussian distribution, and gradually denoises it to clean image Xo with denoising network over 7 timesteps. A latent diffusion model [22] operates similarly, but the diffusion process takes place in more compact latent space of a pre-trained and frozen autoencoder (encoder & and decoder D). That is, it begins with a random latent 2r E R R. HH,xW,,CC, and progressively denoises it clean latent Zo- A clean image then generated by passing the clean latent through the decoder during the inference phase, i.e., Xo D(zo): conversely, a ground truth clean latent is obtained by encoding clean image with the encoder during the training phase, i.e., Znj E(x). A typical choice for the denoising network is a U-net with self self-attention layers at multiple scales.
3.1.2 CacheKV: mechanism for incorporating reference images
As illustrated in Fig. our ReF-LDM leverages input LQ image NLQ and multiple reference images { to generate a target HQ image XHQ- For an LQ image, we simply concatenate its latent encoded by the frozen encoder, ZLo El (xLQ), with the diffusion denoising latent z; along the channel axis to serve as the input of the denoising U-net. For reference images, we design a CacheKV mechanism. Essentially, we extract and cache the features of reference images using the same denoising U-net just once; these cached features can then be used repeatedly at each of the T timesteps in the main denoising process. Specifically, we pass the encoded latent of each reference image, Zee (Xxer). through the U-net to extract their keys and values (K)s) at each self-attention layer and store them in CacheKV. Subsequently, during the main diffusion process, within each self-attention layer of the U-net, we concatenate the reference KVs (from the corresponding self-attention layer) with the main KVs along the token axis. This mechanism enables the U-net to incorporate the additional KVs from the reference images into the main denoising process. When extracting KVs from the reference and pad Zet with zero tensor to accommodate the
images, we use a timestep embedding of additional channels introduced for the LQ image.
To summarize, for inference, we first run the U-net to extract CacheKV from the reference images; subsequently, we proceed through the main denoising process for T timesteps, during which the U-net integrates ZLQ and reference CacheKV. For training, in each iteration, we first run the U-net to extract CacheKV, and then we run the U-net again to estimate the target latent from a sampled noisy latent 2,, incorporating the conditions Zoo and reference CacheKV.
3.1.3 Comparing CacheKV with other designs
There are other intuitive designs for integrating the reference latents {Zeet} into the diffusion denoising process. However, they are either ineffective computationally inefficient compared to the proposed CacheKV. The quantitative evaluation and computational analysis is reported in Sec. 1.2.1 We depict these designs in Fig, and provide intuitive explanation follows:
Concatenating the condition with n, along the channel axis works well for IQQ images (and for other 2D) conditions such as semantic maps [22);; however, it is not effective for reference images. critical difference between these conditions is while the LQ image is spatially aligned with the target HQ image, the reference images are not. Therefore, it is challenging for the model to leverage reference images using simple channel-concatenation.
Chamne-concateenatio:::
Cross-attention Cross-attention layers have been proven useful for text conditions in text- to- to-image models 22). In our ablation experiment, we insert cross-attention layer after each self-attention layer and use the reference latents { to produce keys and values. While cross- cross-attention appears to have the potential to address the spatial misalignment problem, it still fails to effectively utilize the reference images. The difference between our CacheKV and the cross-attention setting is that CacheKV provides the reference images in more aligned feature space for the main denoising process to leverage. Specifically, the CacheKV is extracted using the same U-net and the corresponding self-attention layer in the main denoising process. In

---

contrast, the cross- cross- attention setting processes the reference images only with the frozen encoder, resulting in features that are less aligned with those in the U-net of the denoising process.
Spatial-concatenattorr: Concatenating (Zret) with 2e along the spatial dimension to serve as the input for U-net also effectively leverages the reference images. Conceptually, spatial- concatenation treats reference images in very similar way our CacheKV. In both mechanisms, [Zert are processed through the denoising U-net, allowing the reference KVs to be accessed by the queries (Qs) of the main diffusion latent Z However, spatial-concatenatiion requires significantly more computational resources compared to our CacheKV. It passes (Zer} with Z; to the U-net at each of the T denoising timesteps, whereas CacheKV only passes {Zerr through the U-net once. Moreover, spatial-concatenttiin also requires significantly more GPU memory. as the spatial size of the input for the U-net increases with the number of reference images. As for self-attention layer in the U-net, both mechanisms increase memory usage; C introduces additional reference Vss, while spatial-coneatenation introduces reference QK Vs.
KY
KV
ALo
Concat Channel
Channel-Concat
Cross-Attention
patchity
KV Cross- coss-attention
(a) Channel-concatenation
(b) Cross-attention
<
Senviding
x.u
Chane
Spatial
Concet KV Taken
Spatial-Concat
CacheKV
coche
n. time
(c) Spatial-coneatenation
(d) CacheKV (ours)
Figure 3: Different mechanisms for incorporating reference images into the main denoising process.
3.2
Timestep-scaled identity loss
3.2.1 Timestep-scaled identity loss
As this work aims for face image restoration, we employ the identity loss to enhance face similarity, which is adopted in many face-related tasks [9] 21 28). The identity loss minimizes the distance within the embedding space a face recognition model, thereby capturing the discriminating features of human faces more effectively than the plain RGB pixel space. In experiments, we use the ArcFace model with cosine distance between the ID embedding vectors as the identity loss.
However, naively adding identity loss to the training of ReF-LDM significantly worsens the image quality. One possible explanation might that, the one-step model prediction No D(zo) (2.) at very noisy timestep (e.g.. T) is very different from a natural face image and thus out of the distribution that the ArcFace model trained on; therefore, the identity loss provides ineffective supervision for diffusion models at large timesteps.

---

Based on this assumption, we propose timestep-scat scaleed identity loss, where scaling factor is introduced scale down the identity loss when larger timestep sampled in training step. Specifically, the timestep-scaled identity loss is defined as:
R(x) R(x') Lime ID V LID Vāz (1 | [[(x)]]|[((x]]]
(I)
where R is face recognition model, and v follows the definition in a typical diffusion process X 22 in which a noisy latent z, is sampled given a clean latent zi as:
q(z: zij N (VVaz (1 (1)
3.2.2
Training ReF-LDM with timestep-scaled identity loss
We train our ReF-LDM with the classic LDM loss and the proposed timestep-scaled identity loss:
Ltotal = LLDM + Atim Cime ID
Recall that the denoising U-net estimates the target latent in the latent space of the frozen autoencoder, and typical LLDM computed as the LI distance between the estimated latent and the target latent. To compute the identity loss with the face recognition model, which accepts image as input, decode the estimated latent into the image space using the frozen decoder, i.c, X = D(zo). The experiments in Sec. 5.2.2 show that timestep-scales identity loss can improve face similarity without degrading image quality, unlike the naive usage of identity loss.
FFHQ-Ref dataset
Recent works for non-reference-based face restoration commonly train their models with FFHQ dataset [12), which comprises 70,000 high-quality face images of wide appearance variety with appropriate licenses crawled from Flickr. These images originally; however, we find that a good portion of the images are of the same identities. Thus, construct reference-based dataset-FTHH--eff- consideration described as follows.
provided with reference labels
the FFHQ dataset, with careful
4.1 Finding reference images the same identity
To determine whether two images belong to the same identity, we utilize the face recognition model ArcFace B.] Specifically, we first extract the ID ArcFace embeddings for all images. Then, for each image, compute the cosine distances between its embedding and the embeddings of all other images. A distance less than a threshold r = 0.4 indicates that the images are valid references belonging the same person. Following this procedure, identify 20,405 images with corresponding reference images.
4.2 Splitting data according to identity
To enable the FFHQ-Red Ref dataset to serve both training and evaluation datasets for reference- bassed face restoration models, we divide the images into train, validation, and test splits. However, random data splitting may result in the train and test splits containing images of the same individual, which is not ideal for a fair evaluation. To ensure that all images of single identity are assigned to only one data split, group the images based on their identities. Specifically, we consider identity grouping as a graph problem, where each image acts as a vertex and any pair of images with a distance less than are connected by edges. We then apply the connected component algorithm from graph theory, where each connected component represents a group of images belonging to the same person. Finally, identified 6,523 identities and divided them into three splits: train split with 18,816 images of 6,073 identities, a validation split with 732 images of 300 identities. and test split with 857 images of 150 identities. We report more statistics in Appendix C
4.3 Constructing evaluation dataset with practical considerations
Practical Considerations For a fair and meaningful evaluation, the input reference images should not be excessively similar to the target image; hence, set minimum cosine distance threshold

---

of 0.1 for the test set. Additionally, we manually check the images in the test split to verify that all reference images indeed correspond the same identity. Furthermore. in the context of reference- based face restoration applications, it is preferable to select input reference images that capture more comprehensive representation of person's 's appearance, such as varying face poses or expressions. Although target image in the test split of our FFHQ-Ref may have two to nine reference images, different reference-basee methods may have their own constraints on the maximum number of input reference images. To emulate more representative set of reference images, we sort all available reference images target image using farthest point sampling on the ArcFace distance.
Degradation synthesis for input LQ images For synthesizing input LQ images from ground truth HQ images, follow the degradation model used in previous works 228 5 32
XL.Q ([(Xno ka) fr tPGG), tr.
where an HQ image is blurred with a Gaussian kernel k,, downsampled by scale, added with Gaussian noise nas compressed with JPEG quality level y, and upscaled to the original size.
We construct two evaluation datasets with different degradation levels:
FFHO-Ref-Moderatt: a, r, 8, and q are sampled from [0, 8). [1, 8], [0, 15), and [60, 100]. FFHQ-Ref-Seveee o, r, 8, and q are sampled from [8, 16). [8, 32). [o, 20], and [30, 100).
4.4 Comparison between FFHQ-Ref and existing datasets
Table summmrrize the differences between our proposed FFHQ-Ref and existing datasets. While the CelebRef-HQ dataset [16] has been constructed to train and evaluate reference-basee face restoration models, our FFHQ-Ret -ef dataset contains twice as many images and six times the number of identities compared to CelebRef-HQ. Moreover, built upon FFHQ (12), FFHQ-Ref provides superior image quality over CelebRef-HQ. as indicated by the lower NIQE score (3.68 vs. 3,97). Some ground- truth images in CelebRef-HQ are affected by watermarks and mirror padding artifacts, shown in Appendix B
Table I: Comparison between the proposed FFHQ-Reff and existing datasets.
Dataset
With reference
Licensed
Quality
Images
Identities
FFHQ
12
70,000
CelebRef-HQ [66
10,555
1,005
FFHQ-Ref
20,405
6,523
5
Experiments
In this section, we describe the experimental setup in Sec. 5.1 discuss ablation studies in Sec. 5.2 and provide the comparison between our ReF-LDM and the statc-of-the-an methods in Sec. 5.3
5.1
Experimental setup
5.1.1 Implementation details
To exploit more ground truth images without available reference images, we use 68,411 images in the FFHQ dataset train a VQGAN 4 the frozen autoencoder and an LDM with only LQ condition. We then finetune our ReF-LDM from the LQ-conditioned LDM with the 18,816 images in our FFHQ-Ref dataset. All models are trained excluding the test split images to ensure fair evaluation on our FFHQ Ref benchmark. In our experiments, we adopt a 512x5 12 image resolution, fix the number of reference images to five, and set loss scale Atime ID 0.1. During training, we synthesize input LQ images with r, 6, and sampled from 16). 1, 32). [0, 20), and [30, 100), respectively. For inference, we use 100 DDIM 24 steps and a towards reference images. We provides more implementation details in the Appendix G
fuee-nuidance [7] with scale of 1.5
7

---

5.1.2 Evaluation datasets and metrics
For evaluation datasets, we use the test split of our FFHQ- with two different degradation levels: severe and moderate. In addition, previous son-reference-basee methods commonly CelebA-Test [28] for evaluation, which comprises 3,000 LQ and HQ image pairs sampled from the CelebA-HQ dataset [T]] Therefore, follow the same procedures described in Sec. 4o construct a subset of 2,533 images with available reference images, termed CelebA-Testi Test-Ref.
For evaluation metrics, we adopt the identity similarity (IDS) B2). which is the cosine similarity calculated using the face recognition model ArcFace [3] We also use the widely used perceptual metrics LPIPS [30]. As face pixels are more of concern in the task of face restoration, we also measure the face-region LPIPS (fLPIPS), which the LPIPS calculated using only the pixels in face regions. For assessing no-reference image quality, adopt NIQE (18). Furthermore, measure the FID [6], using 70,000 images from the FFHQ dataset as the target distribution.
5.2 Ablation studies
We provide the ablation studies of the proposed CacheKV, timestep-scaled identity loss, and the number of input reference images. each ablation experiment, we fine-tune the model for 50,000 steps from the same LDM pre-trained without reference images. We compare the difference settings with the FFHQ-Ref-Severs dataset.
5.2.1 VacheKV and other mechanisms
The CacheKV is proposed for integrating the input reference images into the diffusion denoising process. We compare it with other mechanisms illustrated in Sec. 3.1.3] According to Table 2 clannel-concatenation and gross-attention fail to leverage reference images to improve the identity similarity (IDS). In contrast, both patial-concatenattion and CacheK significantly enhance IDS. Moreover, our CacheKV is more computationally efficient than spatial-comcatenation,, requiring only 20% of the inference time and 39% of the GPU memory.
Table 2: Comparison between CacheKV and other mechanisms for input reference images (run with five reference images on single GTX 1080).
Mechanism
IDSt
NIQEL
LPIPS]
Inference time.)
Memory
Chamnel-concatenation
0.23
4.49
0.46
4.17
1.77
Cross-attention 0.23
4.56
0.46
14.54
2.80
Spatial-concatenatioon
0.69
4.84
0.43
58.36
7.44
CacheKV
0.65
4.38
0.43
12.15
2.87
Table 3: Ablation results for the timestep- scaled identity loss.
Loss
IDSt
NIQEL
LLDM LLDM + LID
0.52
4.56
(c) Input references
0.69
6.56
LLDM + Ltime ID
0.65
4.38
Table 4: Design choices for ID loss scaling.
Scale for ID loss
IDSt
NIQEJ
( Input LQ
(g) LldM
(h) + +Li
(i) + Ltime ID
0.65
4.38
1 1<<100
0.52
4.55
Figure 4: Visual ablation results for the timestep- scaled identity loss.
1,<300
0.61
4.44

---

5.2.2 Timestep-scaled identity loss
To validate the benefits of the proposed timestep-scaled identity loss, we train ReF-LDM with three different loss settings: without identity loss (LLDDM) with naive identity loss (LIDMM + LTD), and with the proposed timestep-scaled identity loss (CIDDM Ltime ID). As show in Table and Fig. 4 while the naive identity loss can improve identity similarity (IDS), our timestep-scaled identity loss can do so without sacrificing the image quality (NIQE).
As explained in Sec. 3.2 we employ vāz to scale down the identity loss for larger and noisier timestep f. In Table 4 we compare this design choice with other alternative scaling factors, 1,0000 and 50000 which apply the identity loss only when the sampled timestep is smaller than 100 or 500, respectively. The results suggest that Vār is more effective than the alternatives.
5.2.3 Multiple input reference images
There are two to nine reference images for a target image in the test split of our FFHQ-Ref. While fix the number of reference images to five when training ReF-LDM, the proposed CacheKV mechanism has the flexibility to take varying number of reference images during inference. To validate the effectiveness of utilizing multiple reference images, we evaluate ReF-LDM with maximum of 1, 3, 5, and 8 reference images, respectively. As shown in Table 3 using more reference images significantly improves the identity similarity (from 0.52 to 0.66). However, increasing the number of reference images also increases the computation time, as shown in Table 6 Since using eight reference images encounters out-of-memory issue on a single GTX 1080, we use at most five reference images in our experiments for simplicity.
Table Image quality with different numbrs of reference images.
Table 6: Inference time with different numbers of reference images.
Max num refs
IDS
LPIPS
Num refs
Time 1080
Time @390L
1
0.52
0.45
1
4.86
3.00
3
0.62
0.44
3
7.09 5
3.54
5
0.65
0,43
12.03
4.51
8
0.66
0.43
8
out-of-memory
6.26
5.3 Comparison with state-of-the-art methods
Table Comparison of ReF-LDM with state-of-the-an methods across three benchmarks. Note the highlighting 1st, 2nd, and a gray cell indicating evaluation data leakage for prior methods
Method
FFHQ MenQ-RRe-SSee Severe
FFHQ Ref- Moderatde
CelebA-Test Test-Ref
IDSt
[T.IPS| LPIPS)
FID
IDSt
[T.IPS) IPPPS|
FID.
IDSt
(I.PPSS I.PPS.
CodeFormer
32
0.323
0.108
0.398
51.51
0.760
0.084
0.301
38.78
0.660
0.092
0.340
VQFR 5
0.308
0.112
0.415
52.96
0.659
0.089
0.324
36.77
0.558
0.096
0.352
DAEFR 26
0.294
0.118
0.435
49.08
0.614
0.093
0.333
33.86
0.491
0.101
0.367
LDM
0.231
0.125
0.453
34.40
0.753
0.095
0.344
32.16
0.663
0.093
0.368
DMDNet 16
0.185
0.162
0.511
72.66
0.810
0.096
0.348
36,60
0.752
0.097
0.362
ReF-LDDM
0.676
0.110
0.429
37.60
0.840
0.088
0.332
33.05
0.779
0.093
0.368
As DMDNer encounters landmark detection failures and fails to yield results for 214/837, henchmarks respectively, we compute metrics for DMDNNet using remaining images.
29/857, and 488/2,533 images on the three
5.3.1 Quantitative comparison
We compare our ReF-LDM with state-of state-of-the-an art methods on FFHQ-Ret Ref---RRf-Seeer. FFHQ-Ref PFHQO-Re-MModertee and -eee----est-Ref. Table the performance of competing methods in terms of IDS, fLPIPS, LPIPS, and FID (targeting the FFHQ image distribution). Without the information in reference images, the existing non-reference-base restoration methods (CodeFormer 32. VQFR (5), and DAEFR (26)
fail to preserve the facial identity, leading to significantly lower IDS. The reference-bass method,

---

DMDNet [16,, fails to restore the severely degraded images because it depends on unreliable facial landmark detection, reflected by higher fLPIPS. contrast, our ReF-LDM consistently outperforms DMDNet on identity similarity and other metrics, owing to the proposed CacheKV V mechanism and timestep-scaled identity loss, which effectively leverage the input reference images without the need for landmark detection. We also note that our method exhibits slightly inferior results in LPIPS metric. This is due to the difference in the background pixels, we provide further details the Appendix [ It is also worth mentioning that the competing methods benefit from data leakage on the FFHQ-Ret Ref benchmarks, as their models are trained with the entire FFHQ dataset or with different train split than the identity-base based one in the proposed FFHQ-Ret
5.3.2 Qualitative comparison
In Fig. 5 we present a qualitative comparison between our ReF-LDM, the pre-trained LDM with- out reference images, CodeFomer (a SOTA referencee-bssee method), and DMDNet (a SOTA reference- based method). Given the severely degraded image. DMDNet generates distorted face images based on incorreetly detected landmarks. While CodeFormer yields realistic face images, does not preserve the facial identity well. In contrast, our ReF-LDM produces results that are both realistic and faithful the individual's facial identity.
Input LQ
GT
CodeFormer
DMDNNet
LDM
ReF-LDM
Figure 5: Qualitative comparison. From left to right: input LQ, ground truth, other methods, and our ReF-LDM. From top bottom: Refo--RRf-Seeee,, FFHQ-Ref Ref-Moderate, and CelebA-Test-RRe
Limitations
When the face region is occluded by other objects, model may generate artifacts. For certain face poses (c.g, side face), the reconstructed eyes may appear unnatural. These problems are also commonly observed in other methods and might be caused due the lack of such training images. However, there are some examples showing that these problems can be alleviated if our model is provided reference images with similar face poses target image. Visual examples of these limitations are provided in Appendix
7 Conclusion
In summary, we propose ReF-LDM, which incorporates the CacheKY mechanism and the timestep- scaled identity loss, effectively utilize multiple reference images for face restoration. Additionally, we construct the FFHQ-Re -ef dataset. which surpasses the existing dataset both quantity and quality, to facilitate the research in reference-based face restoration. Evaluation results demonstrate that ReF-LDM achieves superior performance in face identity similarity over state-of af- methods.
10

---

Acknowledgments
The authors wish to express their gratitude Professor Wei-Chen Chiu for his valuable suggestion to exclude the reference images that are too similar to the target images when constructing the proposed FFHQ-Ref dataset.
References
[1] M. Cao, X. Wang, Z. Qi, Y. Shan, X. Qie, and Y. Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22560-22570, October 2023.
[2] C. Chen, X. Li, L. Yang, X. Lin, L. Zhang, and K.-Y. K. Wong. Progressive semantic-aware style transformation for blind face restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11896-1 1905, 2021.
[3] Deng, J. Guo, X. Niannan, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[4] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[5] Y. Gu, X. Wang, L. Xie, C. Dong, G. Li, Y. Shan, and M.-M. Cheng. Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder. In European Conference on Computer Vision, pages 126-143. Springer, 2022.
[6] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[7] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[8] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Ad- vances in neural information processing systems, 33:6840-6851, 2020.
[9] R. Huang, S. Zhang. T. Li, and R. He. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In Proceedings of the IEEE international conference on computer vision, pages 1239-2448, 2017.
[10] K. Karkkainen and J. Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages -1588, 2021.
[11] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10194 2017.
[12] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4411 2019.

---

[13] Y.-F. Lau, T. Zhang, Z. Rao, and Q. Chen. Ented: Enhanced neural texture extraction and distribution for reference-based blind face restoration. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5162-5171, 2024.
[14] X. Li, M. Liu, Y. Ye, W. Zuo, L. Lin, and R. Yang. Learning warped guidance for blind face restoration. In Proceedings of the European conference on computer vision (ECCV), pages 272-289, 2018.
[15] X. Li, W. Li, D. Ren, H. Zhang, M. Wang, and W. Zuo. Enhanced blind face restoration with multi-exemplaar images and adaptive spatial feature fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2706-2715, 2020.
[16] X. Li, S. Zhang, S. Zhou, L. Zhang, and W. Zuo. Learning dual memory dictionaries for blind face restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5904-5911 2022.
[17] X. Lin, J. He, Z. Chen, Z. Lyu, B. Fei, B. Dai, W. Ouyang, Y. Qiao, and C. Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308. 15070, 2023.
[18] A. Mittal, R. Soundararajan, and A. C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):20--21,, 2012.
[19] Y. Nitzan, K. Aberman, Q. He, O. Liba, M. Yarom, Y. Gandelsman, Mosseri, Y. Pritch, and D. Cohen-Or. Mystyle: A personalized generative prior. ACM Transactions on Graphics (TOG), 41 (1()): 2022.
[20] S. Pouyanfar, S. Sengupta, M. Mohammadi, E. Abraham, B. Bloomquist,
L. Dauterman, A. Parikh, S. Lim, and E. Sommerlade. Frr-net: A real-time blind face restoration and relighting network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1240-1250, 2023.
[21] E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287-2296 2021.
[22] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[23] C. Saharia, J. Ho, W. Chan, T. Salimans, J. Fleet, and M. Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713-4726, 2022.
[24] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[25] M. Suin, N. G. Nair, C. P. Lau, M. Patel, and R. Chellappa. Diffuse and restore: A region-adaptive diffusion model for prentity blind face restoration. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
12

---

[26] Y.-J. Tsai, Y.-L. Liu, L. Qi, K. C. Chan, and M.-H. Yang. Dual associated encoder for face restoration. In The Twelfth International Conference on Learning Representations, 2024.
[27] J. Wang, Z. Yue, S. Zhou, K. C. Chan, and C. C. Loy. Exploiting diffusion prior for real-world image super-resolution. 2024.
[28] X. Wang, Y. Li, H. Zhang, and Y. Shan. Towards real-world blind face restora- tion with generative facial prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9168-9177 2021.
[29] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text- to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[30] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[31] Y. Zhao, T. Hou, Y.-C. Su, X. Jia, Y. Li, and M. Grundmann. Towards authentic face restoration with iterative diffusion models and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7312-7322, 2023.
[32] S. Zhou, K. Chan, C. Li, and C. C. Loy. Towards robust blind face restoration with codebook lookup transformer. Advances in Neural Information Processing Systems, 35:30599-3006 2022.
13

---

A Broader Impacts
The ReF-LDM has the capability leverage personal appearances from reference images. This introduces potential risk of misuse, where it could be employed for malicious face editing by using a low-quality image in conjunction with reference images from different individual.
Reference
Input LQ
ReF-LDM
Reference
Input LQ
ReF-LDM
Figure 6: Examples of ReF-LDD using reference images from different individuals.
B Image quality issues in the previous dataset CelebRef-HO
As described in Sec. 4.4 the previous dataset for the reference-based face restoration task, CelebRef-HQ [6]. exhibits issues with image quality. We provide examples where the ground truth images in this dataset are corrupted by watermarks and mirror padding in Fig. 7
Figure 7: Example images with mirror padding and watermark artifacts in the CelebRef-HQ dataset.
C Statistics of FFHQ-Ref dataset
We analyze the statistics of the proposed FFHQ-Ref dataset, introduced in Sec. In Fig. we plot the distribution of the number of available reference images.
Furthermore, we assess the race, age, and gender distributions of the dataset using labels predicted by FairFace [10]. As depicted in Fig. 9 the race distribution within FFHQ-Ref is imbalanced, with predominance of the 'white' category. To mitigate this, we intentionally sampled greater number of images from other races to construct more balanced test set. Additionally, illustrated in Fig. TO FFHQ-Ref encompasses broad age range, from infants (0-2 years) to the elderly (70+ years). However, the distribution is not uniform across ages and genders. For example, there is a notably higher proportion of young females (29.2% of '20-29 female').
14

---

(a) Training set
(b) Validation set
(c) Test set
Figure Distribution of the number of available reference images per image in the FFHQ-Ref dataset of train, validation, and test splits.
376
64%
Med:: Eiiion
2,,5
T.0%
(a) FFHQ-Ref
(b) FFHQ-R Ref-Test
Figure 9: Race distribution within FFHQ-Red Ref dataset.
20-28
20.9
HARD
2.%
37%
2:4
20%
2629
96%
4344 Vade
(a) FFHQ-Ref
(b) FFHO-Ref-Test
Figure 10: Age and gender distribution within FFHQ-Rea dataset.
D Examples of differences in background regions
In Fig. we provide some examples where our ReF-LDM are more different to the ground truth in background pixels compared to prior methods. In the first example, the ReF-LDM attempts to restore another face in the background. In the second
15

---

example, our ReF-LDM restored the mirror padding in the CelebA-Test dataset as hairs.
Input LQ
GT
CodeFormer
DMDNet
LDM
ReF-LDM
Figure Examples where ReF-LDM generates background-region details that differ more from the ground truth.
E Examples of illumination change
Fig. 12 shows example where ReF-LDM exhibits warmer illumination compared to LDM. We conjecture that this may be due to the impact of the strong warm lighting in the input reference images. To address this issue, post-processing tricks, such as adjusting the means of the R, G, B channels to match those of the input LQ image. Another potential solution might be training ReF-LDM with data augmentation on the illuminations of input reference images, to encourage the model to disregard the illuminations of input references and maintain consistency with that of the input LQ image.
could employ
(a) Input LQ
(b) GT
(c) LDM
(d) RefLDM
(e) Input references
Figure 12: An example of (d) ReF-LDM demonstrating an illumination change, likely influenced by the strong warm lighting of the (e) input reference images.
16

---

Examples of failure cases
Here we provide visual examples for the limitation described in Sec. 6 As shown Fig. 13 when the face region is occluded, our ReF-LDM and the prior models tend to generate from unnatural artifacts. For side face images, the ReF-LDM may not work well when the input reference images do not contain faces of similar pose, as shown in Fig. 14, However, Fig. 15| suggests that our ReF-LDM can effectively exploit the reference images of similar face poses to improve the results.
Input LQ
GT
CodeFormer
DMDNet
LDM
Ref-LDM
Figure 13: Some failure cases when the input LQ images are occluded.
Input LQ
GT
CodeFormer
DMDNet
LDM
ReF-LDM
Figure 14: Some failure cases of side faces when side-face images absent in the references.
Input LQ
GT
CodeFormer
DMDNet
LDM
ReF-LDM
Figure 15: Some successful cases of side faces when side-face images are included in the references.
17

---

G More implementation details
G.1 Classifier-free guidance towards reference images
Classifier-free guidance 77 is technique widely used in diffusion models for
guiding the generated results towards a condition e with a controllable scale factor at inference time:
(z. = (z( Ø) + 8 (c)(zz,c) c0 <z Ø))
(5)
In our experiments, we use classifier-free guidance towards reference images with 1.5.
Eo Zr ZLQ (Zer)) = c0(Z,, Z.Q Ø) + s. (Ee(Z), ZLO, {zct)) - cs(2,. L.Q. Ø))
(6)
During the training phase, we randomly drop the conditions by setting them to zero tensors with a probability 0.I.
G.2 Data augmentation for input reference images
During the training phase, we use a fixed number of five input reference images. When target images with less than five reference images are sampled, we repeat the reference images to obtain five reference images. In addition, we apply image augmentation the input reference images with the following operations: color jitter (brightness + 0.2, contrast + 0.2, saturation + 0.2, hue + 0.02), affine transform (rotation + 2. translation + 0.05, scale + 0.05), perspective transform (scale + 0.2, probability 0.5), and horizontal flip (probability 0.5). Lastly, we randomly shuffle the order of available reference images for a target image, so that a different combination of reference images can be sampled at each training iteration. In Fig, 16 we provide an example where set of two reference images is augmented to a set of five reference images.
Figure 16: Data augmentation for reference images.
G.3 Training details
We trained the VQGAN for 200,000 iterations with batch size 32 on four A6000 GPUs for days. We trained the LDM with only LQ condition for 500,000 iterations with batch size 40 on four A6000 GPUs for 7 days. We finetuned the ReF-LDM for 150,000 iterations with batch size 8 on four 3090 GPUs for 6 days. For training losses, the LDM is trained using only the typical LDM loss LLDM, while the ReF-LDM is trained with both LLDM and the proposed Ltime ID.
18

---

G.4 Hyperparameters of networks
For the frozen autoencoder, we use VQGAN as in the LDM [2] with the following settings:
input image: 512x512x3
latent representation: 64x64x8
code booksize: 8192
network hyperparameters: base channel as 128, multiplier for each scale as [1, 1, 4] with 2 residual blocks.
For the denoising U-net, we use the following settings:
input latent: 64x64x16
output latent: 64x64x8
attention layer at resolutions: 32x32, 16x16, and 8x8
network hyperparameters: base channel as 160, multiplier for each scale as [1, 2, 2, 4] with 2 residual blocks.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accu- rately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: The method is described in Sec. Lec.Blad the ablation studies and evaluation results are provided in Sec. 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
We discuss the limitation in Sec. and provide visual examples in Appendix
Justification:
F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" " section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
20

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-ters system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Review- ers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to repro- duce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide implementation details in Sec. ?? and Appendix G
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for repro- ducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We plan to release our dataset and model after the paper's acceptance. They are not included in the current submission because we are waiting formal permission from the associated company.
22

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https} for more de-
hmapg...ccaaaiicauuussss/eeettss tails.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environ- ment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experi- ments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide details in Sec. ?? and Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
23

---

Answer: [No]
Justification: We do not report error bars, as the training our LDM-based model for multiple times costs too much time and computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed
form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed er- rors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric eπror bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We provide the computer resources for inference in Table. 2 and for training in Appendix G.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, in- ternal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make it into the paper).

---

9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://neurips... cttppp/ccccpb/bccccccccc EthicsQuidelines?
Answer: [Yes]
Justification: Our FFHQ-Ref dataset in this work is based on an exisiting dataset FFHQ where only images with appropriate licences are collected.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Question:
Answer:
[Yes]
Justification:
We discuss the broader impacts in Appedix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveil- lance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy con- siderations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
legitimate to point out that an improvement
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
25

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We haven't release the dataset or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks.
The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly states that our FFHQ-Ref is constructed based on the FFHQ dataset in Sec 4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possi- ble, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets,
cot.:/aseccommmtaatttttss has curated licenses for
26

---

some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We haven't release the dataset.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable).
You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This work does not contain crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data
collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study par- ticipants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
27

---

Answer:
[NA]
Justification:
This work does not contain crowdsoureing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "nsjzAxMpXH", "reviewer": "Reviewer_E2Jp", "summary": "This paper investigated the reference-based face restoration model based on latent diffusion model. It proposed a ReF-LDM method involving a CacheKV module and a timestep-scaled identity loss. A FFHQ-Ref dataset consisting of 20,405 HQ face images with corresponding reference images. Experiments are conducted to evaluate it.", "questions": "One major concern is that the novelty seems to be limited. For example, the authors are suggested to discuss CacheKV with MasaCtrl [1], and discuss timestep-scaled identity loss with the identity loss used in customized generation [2].  [2] is a concurrent work and can also be ignored.\n\n[1] MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, Arxiv 2023.\n[2] PuLID: Pure and Lightning ID Customization via Contrastive Alignment, Arxiv 2024.\n\n\nFor the other questions, please refer to the weakness.", "limitations": "Yes. The authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ A reference-based face restoration model based on latent diffusion model.\n+ A FFHQ-Ref dataset consisting of 20,405 HQ face images with corresponding reference images.\n+ A CacheKV module and a timestep-scaled identity loss.", "weaknesses": "- While the of reference-base face restoration is interesting, from Table 6, the improvement by ReF-LDM is limited in comparison to no-reference based methods. This issue may be alleviated by further improving the fusion mechanism  or introducing more plausible metrics.\n\n- In the ablation study (Table 3), the effect of timestep-scaled identity loss is given. However, more experiments are required to assess the results by other possible variants of timestep-scaled identity loss.\n\n- From the visual results, it seems that ReF-LDM may change the illumination of the degraded images (see Fig.s  8&10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1-1. Limited improvement over no-reference methods in Table 6.**\n\nReF-LDM achieves a significant boost in face similarity metric (IDS↑ 0.675 vs. 0.323 for the best no-reference method CodeFormer). As suggested by the reviewer, we evaluated an additional metric FID (with target distribution as FFHQ 70k images) and found that our ReF-LDM has better FID especially for severely degraded LQ images (FID↓ 37.4 vs. 49.1 for the best no-reference method DAEFR). We appreciate the reviewer’s suggestion and will include the FID evaluation in the revision.\n\n| FID-FFHQ↓ | FFHQ-Ref-Severe | FFHQ-Ref-Moderate |\n|-----------------|-------------------|---------------------|\n| CodeFormer | 51.5              | 38.8                |\n| VQFR           | 53.0              | 36.8                |\n| DAEFR         | 49.1              | 33.9                |\n| ReF-LDM (ours) | **37.4**  | **32.9**           |\n\nWe agree that we do not have the best LPIPS scores and that additional fusion modules (like CodeFormer’s CFT or VQFR’s dual decoders) may further improve LPIPS, while in this work we would like to focus on a simple network design to integrates an input LQ and multiple reference images into an LDM-based model.\n\n> **Q1-2. More ablation studies of variants of timestep-scaled identity loss.**\n\nWe conducted an additional experiment with variant setting $t<500$ (timestep-dependent scale = 1 when $t<500$ otherwise 0), which shares the same motivation as the default setting $\\sqrt{\\bar{\\alpha_t}}$ to downscale the identity loss when a larger and noisier timestep t is sampled. The example images (in PDF figure 1) and the metrics in the table suggest that $t<500$ also works but the default $\\sqrt{\\bar{\\alpha_t}}$ yields better identity similarity and quality. We appreciate the reviewer’s suggestion and will train and include more variants of timestep-dependent scales in the revision.\n\n|                         | IDS↑ | NIQE↓ |\n|---------------------|------|-------|\n| no ID loss          | 0.52 | 4.56  |\n| constant            | 0.69 | 6.56  |\n| default √āt          | 0.65 | 4.38  |\n| **variant t<500**   | 0.61 | 4.43  |\n\n> **Q1-3. Illumination Changes.**\n\nThank you for pointing this out. We checked the input reference images used in the examples where the illumination changed. In the case of Figure 8, the reference images have strong orange lightning and our output has a warmer illumination. Therefore, the cause may be the influence from the reference images. We will include case studies and discuss possible future work (such as illumination-preservation loss or different model architecture design) in the revision.\n\n> **Q1-4. Our CacheKV vs. related work MasaCtrl.**\n\nMasaCtrl proposes a training-free method to exploit StableDiffusion for reference-based text-to-image synthesis. They also utilize the denoising U-net to extract keys and features (KVs) from a reference image (termed as source image in their paper). However, we are different in several aspects:\n1. CacheKV passes the references through the U-net only once and reuses the saved KVs in each main diffusion denoising step without redundant computation, while MasaCtrl passes the reference through the U-net for every diffusion step.\n2. We have an ablation experiment “spatial-concatenation” that is closer to MasaCtrl, as it also passes references to U-net for each step (paper L125-135). As shown in paper Table2, our CacheKV (without the redundant U-net passes) is more efficient, especially when we have multiple reference images.\n3. Our model implicitly learns how to naturally integrate the references’ facial features into LQ’s image layout, while MasaCtrl explicitly uses object masks (computed with text condition’s cross-attention scores, ex. “dog”) to prevent the access of irrelevant reference features and messy results.\n\nWe appreciate the suggestion to discuss the related work MasaCtrl and will include it in the revision.\n\n> **Q1-5. Our timestep-scaled identity loss vs. concurrent work PuLID (Arxiv 2024).**\n\nThe concurrent work PuLID shares the same motivation with us, which is, to avoid using identity loss on a flawed one-step output from a larger and noisier timestep $t$. PuLID exploits a few-step distilled diffusion model and runs four denoising steps to obtain a better output for identity loss. They adopt off-the-shelf SDXL-lightning for their SDXl-based model. However, for a non-SD-based work (like our ReF-LDM), PuLID’s solution requires an additional few-step diffusion distillation training process. In contrast, we propose a simple and effective solution by applying a timestep-dependent scale to identity loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kJ4TyQqaOx", "reviewer": "Reviewer_5Xiy", "summary": "1. This paper proposed ReF-LDM for reference-based face image restoration with CacheKV to utilize the input reference images.\n\n2. This paper introduced a timestep-scaled identity loss, which considers the characteristics of diffusion models and helps ReF-LDM better learn the discriminating features of human identities.\n\n3. This paper constructed FFHQ-Ref, a dataset comprising 20,406 high-quality face images and their corresponding reference images, to facilitate the advance of reference-based face image restoration.", "questions": "Please refer to the weakness section and try to answer these five questions during rebuttal.", "limitations": "Limitations are discussed in this paper.\n\nWhen the face region is occluded by other objects, the proposed model may generate artifacts.\nFor certain face poses (e.g, side face), the reconstructed eyes may appear unnatural.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposed ReF-LDM, which incorporates the CacheKV mechanism and the timestep scaled identity loss, to effectively utilize multiple reference images for face restoration. \n\n2. This paper constructed the FFHQ-Ref dataset, which surpasses the existing dataset in both quantity and quality,  to facilitate the research in reference-based face restoration. \n\n3. Evaluation results demonstrate that ReF-LDM achieves superior performance in face identity similarity over state-of-the-art methods.", "weaknesses": "1. The idea of replacing the key/value in self-attention layers with that of reference images has been widely used in style transfer work. This work follows the same scheme - extracting style features (key/value) from multiple reference images and inserting them during denoising. However, from the results, it seems that the style of reference images affects less on the generation. I wonder the reason behind it.\n2. At the first stage of training, i.e., train an LDM with only LD condition, which loss function is adapted here?\n3. Since reference images aim to provide ID information during restoration, why not just inject face embeddings of multiple HQ faces as a condition? What are the advantages of the proposed method over the above strategy?\n4. For the ablation study of the timestep-dependent identity loss, have the authors tried to apply such a loss after T steps? \n5. The proposed dataset does not contain new data, which is basically a re-labeling and cleaning work on FFHQ. Overall, the novelty is not significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q2-1. Different impacts of reference images in comparison to style transfer works.**\n\nThe reference images affect model outputs differently because face image restoration and style transfer have different objectives. In our task, the model output is an HQ image, and the model inputs are reference images and an LQ image (which is synthesized by applying degradation to the HQ image). Therefore, our model is encouraged to follow the image layout and style of the input LQ image and to integrate the distinctive facial features (e.g., eye shape) of the input references by the LDM loss and timestep-scaled identity loss.\n> **Q2-2. Loss function used to pretrain LDM with only LQ input.**\n\nWe use only the typical LDM loss (without the proposed timestep-scaled identity loss) to pretrain LDM. We will state this clearly in the revision.\n> **Q2-3. Why not use face ID embeddings of reference images as network condition?**\n\nWe conducted an additional ablation experiment which uses pretrained ArcFace to encode input reference images. The example images (in PDF figure 2) and the lower face similarity metric (IDS↑ CacheKV 0.65 vs. ArcFace 0.24) suggest that it is more challenging for ReF-LDM to integrate the reference identity information with ArcFace than with CacheKV. This difficulty may arise because ArcFace embedding space is very different from that of the main diffusion U-net. In contrast, CacheKV uses the same diffusion U-net to provide reference-image keys and values with the corresponding attention layers, making it easier for the main diffusion U-net to integrate.\nAnother intuitive explanation is that ArcFace is trained to distinguish different identities, thus, it is a good discriminator and provides effective supervision as the identity loss. However, ArcFace is not trained to generate a face image, so it may not contain enough appearance information for the restoration task.\n> **Q2-4. Ablation study of timestep-scaled identity loss, apply loss after T steps.**\n\nIn a training iteration of LDM-based models, we typically sample a $z_t$ and pass it though a denoising network once then compute the loss. To compute identity loss on a multi-step prediction, we need to pass an input $z_t$ through the denoising network recurrently for at most $T=1000$ passes. This will greatly increase the required computational time and GPU memory, so we didn’t experiment with such a setting.\n> **Q2-5. The proposed FFHQ-Ref dataset contains no new images.**\n\nWhile FFHQ-Ref is indeed based on the images collected by FFHQ dataset, we believe its value lies in the boost of image quality and quantity over previous reference-based dataset (compared in paper Table 1). In addition, the careful curation and organization for the task of reference-based restoration (i.e., ID-based data splitting and FPS-sorted reference images) also enable fair and practical evaluation for future works in this field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AqfKS08YlL", "reviewer": "Reviewer_ZeBa", "summary": "To enhance the performance of blind face image restoration, this paper proposes a reference-based method named CacheKV, which adapts LDM to restore lowquality (LQ) conditioned on several high-quality (HQ) images. As there is no dataset available for reference-based image restoration, therefore, the authors construct a reference-based dataset FFHQ-Ref specifically tailored for this task.", "questions": "What will happen if we use face image from another identity as refrence image?", "limitations": "see weakness.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "-The authors collected a new dataset, which could facilitate related research topics.\n-The paper proposes CacheKV that effectively leverages the information provided by the reference images, significantly improving the performance of blind face image restoration. \n- The proposed method performs better than other referencebased methods.", "weaknesses": "-The paper does not explicitly explain how CacheKV works. \n-How does it use varying numbers of reference images when calculating qkv? \n-How does CacheKV provide a more aligned feature space compared to cross attention?\n-There are many GAN based blind face image restoration papers (reference based & non-reference based), however, the authors did not compare with them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q3-1. Explain how CacheKV works.**\n\nWe include an illustration in the PDF figure 3 (which is a simplified version of the paper figure 2) and explain as follows.\n\nDuring inference,\n- We first extract the CacheKV from reference images. Specifically, for a reference image, we encode the image $x_\\text{ref}$ to latent $z_\\text{ref}$ with VAE-encoder. Then, we pass the latent $z_\\text{ref}$ to the same U-net used by the main diffusion process, where we save the reference-image keys and values of each self-attention layer.\n- In the main diffusion process, the saved reference-image keys and values (i.e. CacheKV) can be accessed for each denoising timestep $t=T,…,1$. Specifically, in each self-attention layer of the main U-net, we concatenate the reference-image keys (of the corresponding attention layer) with the main keys along the token dimension, and the same goes for the values. By doing so, the main U-net can utilize keys and values of the reference images while denoising.\n\nWe appreciate your question and will refine the description (paper L90-105) in the revision.\n\n> **Q3-2. How does CacheKV use varying numbers of reference images when calculating QKV?**\n\nWe separately pass each reference image through the VAE-encoder and the U-net.\nFor each attention layer in the U-net, we append and save the keys/values from all the reference images along the token dimension, i.e., one reference ($N_\\text{token}, d_\\text{key}$) → multiple references ($N_\\text{ref}* N_\\text{token}, d _\\text{key}$).\nSimilarly, in the main diffusion process, we concatenate the reference keys/values with the main U-net keys/values along the token dimension.\nWe appreciate your question and will explain this in a clearer way in the revision.\n\n> **Q3-3. Why does CacheKV provide a more aligned feature space than ablation setting CrossAttention?**\n\nWe include an illustration of CacheKV and CrossAttention in PDF figure 3. \n-  CacheKV uses the same U-net as the main diffusion process. So in the main diffusion process,  each attention layer can access the reference-image keys and values of the corresponding attention layer (which is an aligned feature space).\n-  CrossAttention only encodes reference images with VAE-encoder and passes the same reference-image latent {$z_\\text{ref}$} to all cross-attention layers. The VAE’s latent space is less aligned with the U-net of the main diffusion process.\n\nWe appreciate your question. We will include the illustrations and refine the explanation in the revision.\n\n> **Q3-4. No comparison with GAN-based papers.**\n\nWe have compared with some recent GAN-based BFR methods in Table 6, where DMDNet, DAEFR, and VQFR use discriminator and GAN loss; VQFR and CodeFormer use pretrained VQGAN.\nWe didn't run some earlier GAN-based models on our benchmarks (ex. 2021 GFP-GAN, 2020 ASFNet), as the papers we have compared with (ex. 2024 DAEFR, 2022 CodeFormer, 2022 DMDNet) outperformed these earlier methods. Please let us know if the reviewer suggests that we compare with specific papers, we will try to provide additional comparison during the reviewer-author discussion period.\n\n> **Q3-5. What will happen if we use reference images from another identity?**\n\nWe included example results of using input reference images from two different people in PDF figure 4. ReF-LDM outputs reasonable high-quality images based on the LQ image with facial features affected by the reference identity (ex. wrinkles). We appreciate the reviewer’s interesting question, and we will include related results in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWrG4n0Fy2", "reviewer": "Reviewer_G6n4", "summary": "- This paper addresses face image restoration using multiple reference images, employing the CacheKV technique to utilize these images effectively. To overcome the issue of loss of appearance information in latent diffusion models, a timestep-scaled identity loss is proposed.\n- The design choices for CacheKV are explained with experimental results in Section 5.2.1, and extensively discussed in Section 3.2.2.\n- Additionally, the superiority of the proposed methodology is demonstrated through various experiments using datasets with different levels of degradation, such as FFHQ-Ref-Severe and FFHQ-Ref-Moderate.", "questions": "- How is the time embedding $O$ selected when extracting CacheKV from reference images?\n- I wonder the results of blind face image restoration of Ref-LDM which get the zero reference image.", "limitations": "Please refer to the weakness section.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The proposed framework effectively utilizes multiple reference images without information loss and provides a suitable framework for leveraging this information within latent diffusion models.\n- Sufficient experiments validate each proposed method, and both quantitative and qualitative results support the claims.", "weaknesses": "- The use of a pre-trained diffusion model redundantly during training and inference results in slower speeds and higher computational costs, posing an obstacle for applications such as video face restoration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q4-1. The redundant use of the diffusion model will cause slower speed and more computation, limiting applications like video face restoration.**\n\nDuring inference, the CacheKV mechanism pre-computes and saves the keys and values (KVs) for each reference image with only one pass of diffusion network; then, the saved reference KVs are reused in each step of the multi-step diffusion denoising process. In a similar way, for video face restoration, we can pre-compute and save the CacheKV and reuse the same CacheKV for all frames of a video without redundant computation.\n\n> **Q4-2. How is the time embedding selected when extracting CacheKV from reference images.**\n\nWe choose the time embedding of timestep 0 (the same one used by the main diffusion U-net) when extracting CacheKV. Our motivation is that the reference images are clean, so we use the last and cleanest diffusion timestep $t=0$. We have also experimented with a learnable time embedding, while the results are similar to using that of timestep 0. So we stick to the original choice. The motivation of using timestep 0 is mentioned in paper L93-94, we will explain it more clearly in the revision.\n\n> **Q4-3. The results with no reference image.**\n\nWe show the outputs of ReF-LDM without reference image in PDF figure 5, which suggests that ReF-LDM can restore an LQ image to a reasonable HQ image without reference images. The reason may be twofold:\n\n1. our denoising U-net has been pretrained with only LQ condition (paper L218-220) \n2. we drop the reference condition during training with probability 0.1 (paper L419).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Response to all reviewers and area chairs for a brief summary

We sincerely thank all reviewers for their careful reading and insightful comments. We are encouraged by the positive feedback and recognition of our work's key strengths noted by the reviewers:

1. A new reference-based face restoration model based on latent diffusion model. [E2Jp, 5Xiy, G6n4]
2. CacheKV mechanism, effectively integrating multiple reference images. [E2Jp, ZeBa]
3. An effective timestep-scaled identity loss. [E2Jp, 5Xiy]
4. New FFHQ-Ref dataset, surpassing existing reference-based dataset. [E2Jp, 5Xiy, ZeBa]
5. Superior performance in face identity similarity over previous methods. [5Xiy, ZeBa]
6. Experiments to validate each proposed component. [G6n4]

We have responded to each reviewer individually to address any comments. We would like to give a brief summary.

1. Improvements: We address the limited improvement over no-reference methods by highlighting the superior IDS and presenting an additional FID metric that shows better performance.
2. Ablation studies: We conduct additional experiments on a variant of the timestep-scaled identity loss and demonstrate CacheKV's advantage over face embeddings.
3. CacheKV explanation: We provide a more explicit explanation of how CacheKV works and its advantages over cross-attention.
4. Comparison to related work: We clarify the differences between our designs and those of MasaCtrl and PuLID.
5. Dataset value: We defend the value of FFHQ-Ref, despite it not containing new images, by highlighting its improved quality and organization for reference-based face tasks.
6. Additional experiments: We present results of using reference images from different identities and in no-reference scenarios.
7. Computational efficiency: We explain how CacheKV can be optimized for applications like video face restoration.

Again, we thank all reviewers and area chairs!

Best,

Authors

Meta-review: This paper proposes a latent diffusion model, named ReF-LDM, for reference-based face image restoration. The authors design a CacheKV mechanism to effectively and efficiently integrate information from mulitple reference images. They also proposes a timestep-scaled identity loss to better preserve the identity in the restored image. Finally, they construct a new dataset based on the FFHQ datatset for reference-based face restoration.

The authors have clearly explained the working principal of their CacheKV mechanism and its difference from other similar designs. They have also included further ablation study for their timestep-scaled identity loss. These have resolved most of the concerns raised by the reviewers. The major concern is the overall novelty of this work seems minor.
Decision: Accept (poster)