FUSION IS ALL YOU NEED : FACE FUSION FOR CUSTOMIZED IDENTITY-PRESERVING IMAGE SYNTHESIS

## Abstract

Abstract Text-to-image (T2I) models have significantly advanced the development of artificial intelligence, enabling the generation of high-quality images in diverse contexts based on specific text prompts. However, existing T2I-based methods often struggle to accurately reproduce the appearance of individuals from a reference image and to create novel representations of those individuals in various settings. To address this, we leverage the pre-trained UNet from Stable Diffusion to incorporate the target face image directly into the generation process. Our approach diverges from prior methods that depend on fixed encoders or static face embeddings, which often fail to bridge encoding gaps. Instead, we capitalize on UNet’s sophisticated encoding capabilities to process reference images across multiple scales. By innovatively altering the cross-attention layers of the UNet, we effectively fuse individual identities into the generative process. This strategic integration of facial features across various scales not only enhances the robustness and consistency of the generated images but also facilitates efficient multi-reference and multi-identity generation. Our method sets a new benchmark in identity-preserving image generation, delivering state-of-the-art results in similarity metrics while maintaining prompt alignment.

## 1 Introduction

The synthesis of images based on text prompts has been made possible by the extraordinary capabilities
of recently developed large text-to-image (T2I) models, the generated image has high quality
with diverse contexts, the strong semantic prior that is acquired from a vast collection of
image-caption pairs is one of the primary benefits of these models. However, the output domain of
such models is restricted by the bottleneck of the quality and the number of image-caption pairs, the generated contents are hard to control only based on text prompts, and the expressiveness of
text is limited, which further constrains the content generalization. Moreover, models that embed
the text in an integrated language-vision space are unable to precisely reconstruct the appearance of
specific subjects; they can only generate variations of the image content.
Existing work like DreamBooth (Ruiz et al., ) aim to tackle the issue of subject inconsistent generation of T2I models, DreamBooth can be applied to different T2I models, enabling the model to produce more personalized and finely calibrated outputs after training with class-specific prior preservation loss on three
to five images of a subject. However, DreamBooth need unique identifier for personalized training
with specific subject and is not dedicated for human subject. IP-Adapter (Ye et al., ) is proposed
for mitigating the restricted information expressiveness of text prompt by utilizing the image as the prompt, it is mainly based on decoupled cross-attention mechanism to separate image feature and
text feature and training these newly added layers, the input to the cross attention is coming
from encoders (CLIP (Radford et al., ) or Face embedding model) and reshaped by a projection model to match target shape if necessary. IP-Adapter can control the model generation for
various content generation based on specific image prompt while ensuring the content alignment to
the text, InstantID (Wang et al., ) is another method newly proposed for face-identity conditioned image generation also utilizing face embeddings, while previous methods such as IP-Adapter (Ye et al., ) and InstantID (Wang et al., )
achieve good image results, however, the generated images are lacking in terms of face fidelity with less
controllability. Moreover, they often generate cartoonish looking faces and not completely realistic.

As illustrated in Figure 1, existing ID-preserving methods struggle to
maintain consistent identity appearance across generated images. Additionally, the textual
information provided in prompts is often poorly reflected in the outputs of previous methods,
resulting in incomplete alignment between the prompt and the generated content.
For instance, when using “Cry” style as the text prompt, models like IPA-FaceID-Plus (Ye et al., )
fails in terms of face fidelity and InstantID (Wang et al., ) fails to accurately convey the appearance of a crying face and its’ results are often negatively influenced by the reference image’s appearance.

In contrast, our method addresses these limitations by generating facial
expressions that accurately align with the given text prompt. Specifically,
we propose a novel approach that not only ensures high-fidelity face generation but also enhances
identity similarity without sacrificing prompt alignment. Our method leverages the reference face image directly,
overcoming the limitations of encoders like CLIP (Radford et al., ) by integrating the reference image
with the generated one during the cross-attention process. New keys and values are introduced to capture the
reference image’s facial information, which is then fused with text features.
This design enables our method to produce high-quality, realistic images while remaining computationally
efficient for training.

Our main contributions are summarized as follows:

- $\bullet$
We introduce a new method for enhancing ID preserving ability of existing T2I models by
incorporating reference face image directly into diffusion process which achieves SOTA
result in terms of ID similarity, our method shows the superiority in face controllability
generation through the corresponding text prompt without degradation on face alignment.
- $\bullet$
Our approach can also be extended for multiple identity generation and supports multiple face references.
- $\bullet$
We conduct extensive experiments comparing our method with other SOTA identity-preserving techniques regarding facial similarity and text-image alignment, offering empirical validation for the efficacy of the proposed approach.

Figure: Figure 1: Comparison with other SOTA methods in terms of facial expression controllability and prompt alignment, first row are the input identities. It’s worth to notice that IPA-FaceId-Plus (Ye et al., ) loses the face fidelity while InstantID (Wang et al., ) loses a bit of the prompt alignment. On the other hand our method maintains ID preservation with accurate facial expression generation.
Refer to caption: extracted/5895063/assets/new/style.png

## 2 Related Work

### 2.1 TEXT-TO-IMAGE DIFFUSION (T2I) MODELS

In recent years, diffusion models (DM) have become the state-of-the-art in text-to-image (T2I) generation, attracting significant attention on social media. Numerous studies on T2I diffusion models have been published, with continued research expected in the near future. Diffusion models can be broadly categorized into two groups based on their operational space: pixel space or latent space.

Pixel-space models, such as GLIDE (Nichol et al., ) and Imagen (Saharia et al., ), generate images directly from high-dimensional pixel data using pixel-space diffusion priors. In contrast, latent-space models, such as Stable Diffusion (Rombach et al., ), VQ-diffusion (Gu et al., ), and DALL-E 2 (Ramesh et al., ), operate in a compressed, low-dimensional space after the image has been encoded.

GLIDE introduced classifier-free guidance for T2I, replacing the original class labels with text prompts, which human evaluators found more effective than CLIP-guided models (Radford et al., ). Imagen builds on this by using pre-trained language models as text encoders and introducing dynamic thresholding for classifier-free guidance.

Stable Diffusion takes a different approach by optimizing pre-trained autoencoders to map images and their text descriptions into vectors in latent space. Cross-attention layers are employed to guide the generation in this compressed space. Similar to GLIDE (Nichol et al., ), it follows a classifier-free guidance approach and uses a guidance scale hyperparameter, allowing for greater alignment with the text prompt as the scale increases.

### 2.2 SUBJECT-DRIVEN IMAGE GENERATION

Subject-specific text-to-image generation utilizes a collection of photos of a specific subject to produce tailored visuals derived from textual descriptions, DreamBooth (Ruiz et al., )
is one of subject-driven generation methods, that is based on fine-tuning T2I models to integrate
subject information into various output context domains. DreamBooth constructs a specific prompt
with a unique identifier for the pre-defined subject e.g., “A [V ] dog”. Besides, the class-specific prior
preservation loss is introduced for maintaining semantic information expressiveness of pre-trained
model and enabling diverse subject-specific image generation.
ProFusion (Zhou et al., ) and E4T (Gal et al., ) adopt a similar approach, fine-tuning a dedicated identity token for each individual. However, this necessitates separate fine-tuning for every identity, making them impractical in many scenarios.

### 2.3 ID PRESERVING IMAGE GENERATION

Current T2I models have limitations for ID-preserving image generation, the subject of the generated image is not being controlled and prompt engineering cannot promise such purpose. IP-Adapter
(Ye et al., ) utilizes the image prompt to facilitate customized image generation. The input image is encoded into image features by pre-trained CLIP model (Radford et al., ), then the image
features are further mapped by a linear mapping model before feeding into cross-attention layers.
IP-Adapter-FaceID is based on IP-Adapter which is dedicated for realistic face image synthesis by
using face ID embedding from face recognition model. For accurate portrait face generation, IP-Adapter-FaceID-Portrait is another variant which supports input multiple face images. For more realistic and
ID-preserving face generation, IP-Adapter-FaceID-Plus takes directly the reference image along with face embeddings as input
for enhanced face fidelity generation. Overall, all different IP-Adapter methods primarily rely on CLIP
image encoder. Compared with the IP-Adapter, InstantID (Wang et al., ) is proposed for instant personalized image generation and is compatible with community-pre-trained models. Similar to IP-Adapter-FaceID-Plus, it relies on face embeddings with decoupled cross-attention layers, allowing the use of visual prompts. Additionally, it introduces a module named IdentityNet, responsible for encoding detailed features from the reference face input image with spatial control using ControlNet (Zhang et al., ).
Arc2Face (Papantoniou et al., ) is an identity-conditioned foundation model for photorealistic face image synthesis based on only the ID embedding from pre-trained ArcFace networks
(Deng et al., ). It is proposed to generate high quality face dataset like Flickr-Faces-HQ Dataset
(FFHQ) (Karras et al., ) but with more identities. Arc2Face uses ID embedding that extracted
from a cropped face image after feeding into ArcFace model, then the ID embedding is mapped
into CLIP latent space by applying text encoder and being added into cross-attention layers of fine-tuned UNet together with text features from other words, However its’ primarily goal is to extract facialid features from face embeddings and not suitable for customized ID-preserving image generation.

## 3 Method

Figure: Figure 2: The overall pipeline of the proposed method. We first concatenate the target face image with the image to be generated for applying attention at different scales, we modify cross-attention layers to include new keys and values layers, their output is concatenated with the output of the original ones from text prompt.
Refer to caption: extracted/5895063/assets/new/method.png

### 3.1 Overview

We propose a novel method for customized, identity-preserving image generation with the following
objectives:

- $\bullet$
Maintain original prompt alignment: Ensure the generated images remain aligned with the
original prompts across diverse image generation scenarios.
- $\bullet$
Generalize facial features: Enable the model to generalize effectively
to different face views and expressions using just a single image sample as input.
- $\bullet$
Achieve realistic appearance and identity fidelity: Produce images that not only look realistic but also accurately preserve the facial identity of the reference subject.
- $\bullet$
Efficient training: Design the method requires limited training resources and time.

Unlike other approaches that rely on fixed encoders (e.g., IP-Face-Adapter-plus uses a CLIP image
encoder as described in (Radford et al., )) or face embeddings, our method leverages the visual
knowledge embedded in the pre-trained UNet of a Stable Diffusion model (Rombach et al., ).
We input the target face identity image into the UNet, concatenated with the image to be generated.
Cross-attention is then applied between these two inputs at various scales and layers within the
UNet. This approach facilitates more robust and consistent generation results. To allow the use of
face image in guidance, we modify the original cross-attention layers of the UNet. Specifically, we
introduce new keys and values layers that take the face image as input and generate new keys and
values encoding the face identity, these are then concatenated with the keys and values generated by
the text layers, as illustrated in Figure 2. By avoiding the use of fixed encoders and face embeddings,
which can suffer from encoding gaps, our method provides a more flexible and effective solution
for identity-preserving image generation. The integration of the target face identity directly into
the UNet’s cross-attention mechanism ensures that the generated images retain the desired facial
characteristics while maintaining alignment with the input prompts and generalizing well across
different face views and expressions.

### 3.2 Fused Attention

The cross-attention mechanism is applied at various layers of the UNet to integrate the input condition prompt c with the hidden state h, which represents the latent vector of the image being generated
at a specific layer within the UNet. The queries $Q_{h}$, keys $K_{c}$, and values $V_{c}$ are computed as follows:

$$ $Q_{h}=Linear_{q}(h),K_{c}=Linear_{k}(c),V_{c}=Linear_{v}(c)$ (1) $$

Here, $K_{c}$ and $V_{c}$ are of dimension $R^{N_{c}\times D}$, and $Q_{h}$ is of dimension $R^{N_{h}\times D}$.

The functions $Linear_{q}$, $Linear_{k}$, and $Linear_{v}$ represent the original linear layers from the Stable Diffusion (SD) model (Rombach et al., ).
The keys and values derived from the text prompt are used to compute the attention score, which is
subsequently multiplied with the values. This mechanism ensures that the guidance provided by the
text is effectively integrated into the generation process.
To facilitate the identity preservation and generalization capabilities, we modify this attention mechanism to
incorporate the target face identity. Specifically, we introduce additional layers that process the face
identity image and generate new keys and values that encode the facial features. These new keys
and values are then concatenated with the original keys and values derived from the text prompt. We
append two additional layers, $Linear_{k_{f}}$
and $Linear_{v_{f}}$, to the attention mechanism. These layers
represent the only trainable components of our proposed pipeline.
For a face image latent hidden state $h_{f}$ at a specific attention layer within the UNet, new keys and
values that encapsulate the face identity are computed as follows:

$$ $K_{f}=Linear_{k_{f}}(h_{f}),V_{f}=Linear_{v_{f}}(h_{f})$ (2) $$

Here, $K_{f}$ and $V_{f}$ are of dimension $R^{N_{h}\times D}$.

The final keys $K^{{}^{\prime}}$
and values $V^{{}^{\prime}}$
, which incorporate both the prompt and the face identity, are
obtained by concatenating the original keys and values with the newly computed face identity keys
and values:

$$ $K^{{}^{\prime}}=[K_{c};K_{f}],V^{{}^{\prime}}=[V_{c};V_{f}]$ (3) $$

In this equation, $K^{{}^{\prime}}$
and $V^{{}^{\prime}}$
are of dimension $R^{(Nc+Nh)\times D}$.

By integrating the face identity through these additional layers and concatenation, our method ensures that the generated images not only align with the input text prompts but also accurately preserve the facial characteristics of the target identity. This approach allows for more precise control
over the generated images, enabling the creation of high-quality, identity-preserving results.
The efficiency of our method is maintained by limiting the trainable components to just the two
additional layers, $Linear_{k_{f}}$
and $Linear_{v_{f}}$
. This design choice reduces the computational overhead
and training time required, making our approach practical for real-world applications.
The attention scores are computed as follows:

$$ $S=Softmax(M+\alpha\cdot Q^{{}^{\prime}}{K^{{}^{\prime}}}^{T})$ (4) $$

Here, M denotes the attention mask, $\alpha$ is a scale term, and S represents the attention scores
that incorporate both the text and the face image. The attention scores matrix S has the dimension
$R^{Nh\times(N_{c}+N_{h})}$
.
Finally, the output hidden state h is obtained by multiplying the attention scores with the concatenated values:

$$ $h=SV^{{}^{\prime}}$ (5) $$

The resulting hidden state h has dimensions $R^{N_{h}\times D}$.
As mentioned in Section 2, previous methods often separate the identity (ID) guidance into a separate
attention layer, where the result of the ID attention is added to the output of the original cross-attention layer guided by the text. In contrast, we find that our approach of concatenating the keys
and values from the face identity directly into the original cross-attention layer provides stronger and
more consistent ID guidance. This method avoids potential conflicts that can arise from combining
two separate guiding attention layers.
Additionally, our approach eliminates the need to project the face image to match a specific target
shape. Instead, the concatenated keys and values are seamlessly integrated during the multiplication
process, and the hidden state is restored to its original size without any additional projections.

### 3.3 Multiple References

Figure: Figure 3: Comparison between using n references to guide the generation by our methods, first column represents the references identities, second column is the guidance using only one image of the references, third is using two references And fourth using all three, we notice that generated face is a mix of the features in the references.
Refer to caption: extracted/5895063/assets/new/ref.png

Overall, we believe that a single image clearly showing the identity instance is sufficient for identity-preserving generation. However, there are cases where using multiple references can yield better
results, especially when the face in the reference input is occluded or different features are displayed across
different reference images. Our method can easily accommodate the use of multiple references.
For identities $f_{1},\dots,f_{n}$ with corresponding hidden states $h_{1},\dots,h_{n}$, the keys and values can be
obtained using Equation 2 and Equation 3 is then modified as follows:

$$ $K^{{}^{\prime}}=[K_{c};K_{f_{1}};\dots;K_{f_{n}}],V^{{}^{\prime}}=[V_{c};V_{f_ {1}};\dots;V_{f_{n}}]$ (6) $$

This modification allows for the integration of any number of identities without the need for further
training or fine-tuning. We showcase sample results in Figure 3, demonstrating the effectiveness of
using three identities for improved generation.
The same principle can be applied to generate an image with an identity that is a mixture of distinct
identities, a process commonly known as face morphing (Fu & Damer, ), It is important to note that the order of the identities matters, as the attention mechanism in SD (Rombach et al., ) does not account for this.

By extending our method to handle multiple references, we enhance the flexibility and robustness of
the identity-preserving generation process. This capability allows the model to leverage information
from multiple sources, leading to better handling of occlusions and variations in facial features
across different images.

### 3.4 Multiple Identities

Figure: Figure 4: Multiple face identities image generation using our proposed method, first row represents the three input distinct identities, following row contains sample results for different prompts, depthmap based ControlNet (Zhang et al., ) was used for the pose.
Refer to caption: extracted/5895063/assets/new/iden.png

Handling multiple identities in image generation remains an open challenge. In this work, we propose a solution, that combines our previously mentioned method to address this issue. The
goal is to generate an image containing (n) distinct identities $f_{1},\dots,f_{n}$ based on input references
and a given prompt.
Given masks $M_{1},\dots,M_{n}$ where $M_{i}$
indicates the location mask of the face for identity i, we scale
each mask and feed to each cross-attention layer in the UNet, the scaling is done based on
the scale of the hidden state input to the attention and for some layer with scales s the reshaped and scaled masks are
${M_{1}}^{(s)},\dots,{M_{n}}^{(s)}$,
for each of the face identities, we can get keys and values that are combined following Equation 6, if we applied attention like this we will obtain faces that are mix of all identities
features (i.e. face morphing), we instead use a customized attention mask, denoted as ${\bm{A}}$, where the mask is defined as:

$$ ${\bm{A}}_{q,k}=\begin{cases}-\inf&\text{if }isFaceKey(k)\wedge identity(k)=i \wedge\exists{M_{j}}^{(s)}:{M_{j_{q}}}^{(s)}=true\wedge j\neq i\\ 1&\text{otherwise }\end{cases}$ (7) $$

where q indicates the query and k is the key, this restricts the guidance for each face reference to a single, specific face, it also ensures that all identities are present in the generated image. We demonstrate the
effectiveness of our method with graphical results using distinct input identities in Figure 4.
These results highlight the success of our approach in generating images conditioned on multiple
face identities.

## 4 Experiments

Figure: Figure 5: Comparison with other SOTA methods, with prompt used as “astronaut in a garden” and the first row represents the input target faces identities, the following rows are results from IP-faceAdapter (Ye et al., ), IP-face-Adapter-plus (Ye et al., ) and ours respectively. Identities are selected from ffhq dataset (Karras et al., ).
Refer to caption: extracted/5895063/assets/new/gall.png

### 4.1 Experiment Settings

In our work, we utilize Stable Diffusion (Rombach et al., ) as the base model, specifically the Realistic Vision v4.0 checkpoint(^1^11https://huggingface.co/SG161222/Realistic_Vision_V4.0_noVAE). We train on a subset of the LAION-face dataset (Zheng et al., ), consisting of approximately 80,000 text-image pairs. All training and experiments are conducted on two Tesla V100 GPUs (32 GB), with a batch size of 1 per GPU. For comparison with other models, we use InstantID (Wang et al., ), which employs the SDXL model (providing an advantage over our method). For IPA-FaceId, we use IPA-FaceID-plusv2 (Ye et al., ), both utilizing the Realistic Vision v4.0 checkpoint as the base model. To evaluate the CLIP score, we generate a list of 100 identities conditioned on different prompts using LLMs. We provide two versions of our method, with the second version trained for more timesteps and on a larger dataset.

### 4.2 Evaluation Metrics

Previous research on identity-preserving imagine generation tasks (Wang et al., ; Ye et al., )
lacks a defined metric for assessing identity preservation. We
employ cosine distance to assess the ID-conditioning capability of various approaches in our study.
Face embeddings are acquired by feeding generated images into pre-trained face recognition models.
We compare the face embedding of the reference image with those from the created image; a reduced
distance signifies superior ID-consistent generating performance. Additionally, we calculate the
CLIP score using the CLIP model (Radford et al., ) to assess content alignment in face image
generation across several scenarios with varied text prompts, the higher CLIP score indicates high text-image alignment. For the evaluation of image face quality, we employ SSIM and PSNR to compare
the generated images with the reference images. In order to measure the face fidelity, we focus on
calculating those metrics based on the face region only by cropping the faces from the generated images.

Figure: Figure 6: Visual comparison with other SOTA methods in different generation styles. The first column are the input identities along with input styles, from left to right, we show the results of IPA (Ye et al., ), IPA-FaceID, IPA-FaceID-SDXL, IPA-FaceID-Plus, InstantID (Wang et al., ) and ours. It’s noticed that previous methods suffer from the lack of face fidelity or style degradation. Images are selected from InstantID (Wang et al., ).
Refer to caption: extracted/5895063/assets/2.png

### 4.3 Qualitative Results

In this section, we evaluate our proposed method for visual comparison in terms of identity (ID) preservation. As shown in Figure 5, we generate images based on multiple identity face images with the same text prompt “astronaut in a garden”. IPA-FaceID tends to generate face images with low ID preservation and sometimes fails to maintain the gender and age of the reference face image. On the other hand, IPA-FaceID-Plus performs better in maintaining the gender and age of the input identity; however, it still loses some original facial attributes, such as skin tone and hairstyles.

In contrast, our method generates face images with a high level of ID preservation, preserving facial expressions, gender, age, and detailed features such as hairstyles and skin color. Specifically, to investigate accurate facial expression generation, we generate images using different text prompt styles like “Anger”, “Cry”, and “Laugh” to control the facial emotion in the generated images. As depicted in Figure 6, InstantID generates images with less emotional alignment to the associated text prompts, while IPA-FaceID-Plus can generate the intended emotion. However, IPA-FaceID-Plus has less control over ID preservation, producing exaggerated facial expressions in random styles. Our method provides strong control of facial expressions based on the corresponding text prompt, especially in the case of the “Cry” prompt, even if the reference image already has a ‘smile’ or ‘laugh’ facial expression, our method still emphasizes the importance of the text prompt in the final image generation. Our method produces ID-consistent face images while delivering more natural-looking facial expression representations, without suppressing the contribution of the text prompt to the generated images.

In Figure 6, we aim to compare the ID preservation in different image styles generation. Same as
previous work InstantID (Wang et al., ), we use a set of text prompts to indicate various prompt styles like
“Line art”, “Faded Film” and “Watercolor”. Our method is able to generate comparable results with
other SOTA methods while maintaining high degree of face and style alignment together.

**Table 1: Quantative evaluations of ID preservation.**
|  | Cosine distance $\downarrow$ |  |  |
| --- | --- | --- | --- |
| Method | VGG-faces | Facenet512 | ArcFace |
| IPA-FaceID | 0.60 | 0.27 | 0.55 |
| IPA-FaceID-Plus | 0.36 | 0.25 | 0.39 |
| InstantID | 0.42 | 0.27 | 0.38 |
| $Ours_{v1}$ | 0.33 | 0.22 | 0.34 |
| $Ours_{v2}$ | 0.25 | 0.18 | 0.28 |

### 4.4 Quantitative Results

Besides visual comparisons, we also numerically evaluate our method based on CLIP score, SSIM,
PSNR and cosine distance using deepface (Serengil & Ozpinar, ; ) utilizing three models namely VGG-Face, Facenet512 and ArcFace. We
utilize a subset of FFHQ (Karras et al., ) dataset high quality face images as the input identities
in these experiments.

Table 1 shows the numerical results of cosine distance on different methods. Firstly, in terms of
the face alignment, it’s clear that our methods achieve significantly higher alignment than the other
methods on all face recognition models. The faces in our generated image maintain high level of
face features from original reference identity and can be easily used for deceiving different face
recognition systems.

In Table 2, we judge the face quality and prompt alignment offering different methods, IPA-FaceId
has the highest CLIP score, however its’ ID similarity is very low compared to ours and other
methods, ours v1 and IPA-FaceID-Plus maintains almost similar CLIP score with IPA-FaceID-Plus
being slightly better, ours v2 is worse in terms of CLIP, but it achieves significantly higher identity
similarity. We exclude InstantID (Wang et al., ) from CLIP score due to the fact that it requires
pose conditioning and the base model for the public checkpoint is SDXL based, which will make
the comparison not plausible. We also compare the face quality using PSNR and SSIM metrics,
our method achieves best result on both, further demonstrating that our method is capable of
generating high quality realistic faces with high ID fidelity.

## 5 limitations and future work

We have observed that fine facial features tend to degrade when the face occupies a small portion of
the overall image. This issue primarily stems from the limitations of Stable Diffusion (Rombach et al., ). Additionally, when our method is used in conjunction with ControlNet (Zhang et al., ),
it results in slight reduction in identity fidelity, which can be resolved by light fine-tuning or reducing the conditioning strength. In our future work,
we aim to address these challenges by enhancing the ability to maintain fine details and improving
control in scenarios involving the generation of multiple identities.

## 6 Conclusion

In this paper, we present a novel method for ID-preserving image generation that excels in maintaining high identity fidelity while producing realistic facial images. Our approach demonstrates
robust generalization across various facial views and expressions. Additionally, we propose innovative solutions for handling multiple references and multi-identity generation, effectively bridging
encoder domain gaps and leveraging multi-reference guidance for enhanced performance. Our method
achieves new state-of-the-art results in preserving identity similarity, as evidenced by our extensive
experimental evaluations. Furthermore, it maintains good alignment with given prompts, setting a
new benchmark in the field of identity-preserving image generation.

**Table 2: Quantative evaluations of face quality and prompt alignment.**
| Method | CLIP Score $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ |
| --- | --- | --- | --- |
| IPA-FaceID | 25.23 | 27.94 | 0.36 |
| IPA-FaceID-Plus | 23.76 | 28.04 | 0.44 |
| InstantID | - | 27.93 | 0.33 |
| $Ours_{v1}$ | 23.28 | 28.25 | 0.41 |
| $Ours_{v2}$ | 22.51 | 28.40 | 0.47 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See weekness.
soundness: 2
strengths: - This method surpasses previous approaches in identity fidelity, as evidenced by superior performance in relevant metrics.
- This method supports customizable multi-identity generation.
summary: This paper improves text-to-image generation by preserving individual identities in generated images. Unlike previous methods, it directly incorporates face images into the UNet of Stable Diffusion. By adjusting the cross-attention layers, the model better captures facial features across scales. This approach achieves high accuracy in both identity similarity and prompt alignment.
weaknesses: - The writing of the paper is quite rough, with at least 10 instances of incorrect punctuation usage and a generally disorganized flow of ideas.
- The paper lacks sufficient contribution, as the core method is still based on IP Adapter without original innovations. Additionally, it is challenging to observe any performance advantages.
- The results are highly limited; in Figure 4, the customized faces are barely recognizable, with significant loss of facial details. The authors need to further explain the reasons behind this. Additionally, the paper claims to address the issue of decoupling expressions from the reference image; however, in Figure 5, each result appears to be a copy-paste of the reference image’s expressions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: The main concerns about the paper are listed in the Weakness section. Here are some minor questions for the authors regarding unclear writing etc.:

- Equation 4: What does Q' represent? Q value should not be modified in the proposed method based on my understanding. Also, what does M here represent? Please provide the complete formula and how it differs from the standard attention mechanism for clear understanding.
- Line 304: If you mention this technique can be used for face morphing, it is better to provide several examples to illustrate.
- Figure 4: This figure appears to be unclear. Which identities are used for each output? Why do some outputs contain three identities and some two (are those with two missing one reference or do they only get two references as input)? What depth map is used for each output?
- Section 4.1: How long does it take for training? How large is the larger dataset used to train the second version?
- Line 376: How are the face regions obtained?
- Line 427: Fig.6 has nothing to do with emotion. I guess you mean Fig.1?
soundness: 2
strengths: The authors propose to pass the reference image to the diffusion model directly to avoid using an extra image encoder and potential encoding gap issues. The method can generalize to multiple reference images of the same identity and multiple identities in the same output image.
summary: This paper proposes an identity-preserving image synthesis approach that directly passes the reference identity image to the diffusion model and fuses the attention maps obtained by a new cross-attention module with the original text-guided ones. The authors provide qualitative and quantitative results to demonstrate the effectiveness of the method.
weaknesses: - **Incomplete experiments**: 
  - The paper ensembles a very similar approach to IP-Adapter (referred to as IPA-FaceID-Plus in paper) with two major differences: (1) no extra image encoder, and (2) cross-attention fusion at the attention mask stage. Both are claimed by the authors to improve the performance over previous approaches. However, the authors fail to provide an ablation study for the audience to know the exact effect of these two modifications and how much they contribute to the final performance.
  - Lack of quantitative comparison with important baseline InstantID. The authors claim they use SDXL which is a superior base model over the one they use, therefore eliminated for fair comparison. But for an important baseline, the authors should either upgrade the proposed method to SDXL or downgrade InstantID to the same base model to make the comparison.

- **Unsatisfying Quality**:
Qualitative results do not show clear advances compared to previous methods. Fig.3 in particular gives very bad-quality results. Besides, Fig.5 shows signs of overfitting the reference pose and facial expression in the generated results, which raises the concern that the improved ID preservation results (in Tab.1) and improved PSNR and SSIM (in Tab.2) may be caused by this overfitting. Fig.6 does not reveal clear improvements over IPA-FaceID-Plus and InstantID also.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: The paper presents a novel text-to-image synthesis method that excels in several key areas. By directly incorporating the target face image into the diffusion process, it significantly enhances the identity preservation capabilities of T2I models. This method consistently generates high-quality, realistic images that accurately maintain the fidelity of facial features and expressions aligned with text prompts. It also offers remarkable flexibility and scalability, supporting complex image generation scenarios involving multiple identities and reference images. Additionally, the method is designed for computational efficiency, requiring less training time and fewer resources, which makes it highly practical for real-world applications.
summary: This paper presents a novel text-to-image (T2I) synthesis method leveraging a modified UNet from Stable Diffusion to improve identity preservation in generated images. By altering UNet's cross-attention layers, it achieves robust multi-identity and multi-reference generation, setting new benchmarks in identity similarity metrics.
weaknesses: 1. The method can struggle with fine facial features, especially when the face occupies a small portion of the image due to the limitations of the underlying Stable Diffusion model.

2. Potential Overfitting: The method's reliance on direct face image integration might lead to overfitting specific facial features or identities, especially in a dataset with limited diversity.

3.Could we possibly tackle the issue of facial feature degradation by improving the core architecture of Stable Diffusion or ControlNet itself, instead of just relying on the fine-tuning or adjusting the condition strength mentioned in the article?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - What is the attention mask in formula (4)? It is not explained here how this mask is constructed and what its role is.
- In Figure (3), the improvement brought by more reference images is not obvious.
- In this experiment, the model was trained using only 80K images, and its diversity is questionable. It should be fair to use more non-celebrity (different gender, age, ethnicity) in testing.
- In Figure (5), the generated image has a copy-paste problem. Why does this happen?
- The credibility of the quantitative results is questionable. It is hard to believe that InstantID's similarity is worse, although it does suffer from text controllability.
- Why intermediate hidden state is a better representation of face information? More discussion will be useful.
soundness: 3
strengths: - Personalized generation is an important task in the field of images and has not been well solved.
- It reveals a good balance between face fidelity and prompt following ability.
- It is interesting to use UNet to handle reference image instead of an extra image encoder.
- This method is simple but achieves a new SOTA in identity-preserving image generation.
summary: This paper is dedicated to personalized text-to-image generation. Unlike previous works, this approach not uses frozen face encoder and static facial embedding to guide the generation. Instead, they utilize the hidden states within the pretrained UNet from reference facial image to replace static facial embedding, furthermore, they choose to concatenate K, V from text and facial image.
weaknesses: - The novelty is kind of limited.
- Although it may be a problem of base model (SD1.5), it is hard to claim that the facial fidelity is satisfied. This is my major concern.
- The experimental results are not convincing enough, the diversity may be a big problem.

Meta-review: The paper proposes an identity-preserving text-to-image generation method by modifying the UNet architecture in Stable Diffusion. Instead of using a separate image encoder, the method directly incorporates reference facial images into the cross-attention layers, aiming to improve identity fidelity and prompt alignment. However, there are several weaknesses of this paper. First, the method lacks novelty as it closely resembles IP-Adapter, experimental results are limited and unconvincing (e.g., poor facial detail and diversity). Second, key ablation studies and comparisons to strong baselines like InstantID are missing (the base model is not consistent). Additionally, qualitative results showed overfitting issues, such as copy-paste artifacts of expressions and poses. The lack of rigorous experiments across diverse datasets and incomplete analysis further weakens the submission. Last, the writing of this paper needs to be improved. While the method addresses a relevant problem and offers incremental improvements, the novelty concerns, incomplete experimental validation, and unconvincing results are significant. Therefore, I recommend rejecting the paper.

Decision: Reject