PuLID: Pure and Lightning ID Customization via Contrastive Alignment
Authors: Zinan Guo
Yanze Wu
Zhuowei Chen
Lang Chen
Peng Zhang
Qian HE
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion
controllable image generation
image customization
Primary area: diffusion_based_models
TL;DR: We introduce PuLID, a tuning-free ID customization approach. PuLID maintains high ID fidelity while effectively reducing interference with the original model's behavior

Abstract: We propose Pure and Lightning ID customization (PuLID), a novel tuning-free ID customization method for text-to-image generation. By incorporating a Lightning T2I branch with a standard diffusion one, PuLID introduces both contrastive alignment loss and accurate ID loss, minimizing disruption to the original model and ensuring high ID fidelity. Experiments show that PuLID achieves superior performance in both ID fidelity and editability. Another attractive property of PuLID is that the image elements (\eg, background, lighting, composition, and style) before and after the ID insertion are kept as consistent as possible. Codes and models are available at https://github.com/ToTheBeginning/PuLID

PuLID: Pure and Lightning ID Customization via Contrastive Alignment
Zinan Guo"
Yanze Wu*t
Zhuowei Chen
Lang Chen
Peng Zhang
Qian He
ByteDance Inc.
guozinan. 10bytedance.... com
muyanze1230gmal com
Input
Lagend
Ster Wers. lightsober
The
white cyborg
popercur silhouctte
dork haaded
Popmort Blind corteon
IPAdapter
InstantID
T21 w/o ITD
Figure 1: We introduce PuLID, a duming-fres ID customization approach. PuLID maintains high ID fidelity while effectively reducing interference with the original model's behavior.
Abstract
We propose Pure and Lightning ID customization (PuLID), a novel tuning-free ID customization method for text-to-image generation. By incorporating Lightning T2I branch with a standard diffusion one, PuLID introduces both contrastive alignment loss and accurate loss, minimizing disruption to the original model and ensuring high ID fidelity. Experiments show that PuLID achieves superior performance in both ID fidelity and editability. Another attractive property Pul is that the image elements (e.g., background, lighting, composition, and style) before and after the insertion are kept as consistent as possible. Codes and models available at http:://giihub..oomTTooo www.e///TTTeeegiinnnnuuu..u.
'Equal contributions. Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1 Introduction
As a special category of customized text-to-image (T2I) generation 7 36 19 46 50]
identity (ID)
customization allow users to adapt pre-trained T21 diffusion models to align with their personalized ID. One line of work 7 36 14 19 fine-tunes certain parameters on several images with the same ID provided by the user, thereby embedding the ID into the generative model. These methods have spawned many popular AI portrait applications, such as PhotoAl and EPIK.
While tuning-based solutions have achieved commendable results, customizing for each ID requires tens of minutes of fine-tuning, thus making the personalization process economically expensive. Another line of work [48, 550 3 42 22 21 44 forgoes the necessity of fine-tuning for each ID, instead resorting to pre-training an ID adapter
28 on an expansive portrait dataset. These methods typically utilize an encoder (e.g.. CLIP image encoder (32)) to extract the ID feature. The extracted feature is then integrated into the base diffusion model in a specific way (e.g., embedded into pross-attention layer). Although highly efficient, these tuning-free methods face two significant challenges.
Insertion of ID disrupts the original model's behavior. A pure ID information embedding should feature two characteristics. Firstly, an ideal ID insertion should alter only ID-related aspects, such as face, hairstyle, and skin color, while image elements not directly associated with the specific identity, such as background. lighting, composition, and style, should be consistent with the behavior of the original model. To our knowledge, this point has not been focused by previous works. While some research (50 44 22 has shown the ability for stylized ID generation, notable style degradation occurs when compared with images before ID insertion (as depicted in Fig. D. Methods with higher ID fidelity tend to induce more severe style degradation.
Secondly, after the ID insertion, it should still retain the ability of the original T2I model to follow prompts. In the context of ID customization, this generally implies the capacity to alter ID attributes (e.g., age, gender, expression, and hair), orientation, and accessories (e.g., glasses) via prompts. To achieve these features, current solutions generally fall into two categories. The first category involves enhancing the encoder. IPAdapter (50 I shifted from carly-version CLIP extraction of grid features to utilizing face recognition backbone 6 to extract more abstract and relevant ID information. Despite the improved editability, the ID fidelity is not high enough. InstantID 44 builds on this by including an additional LD&Lammarrk ControlNet [52] for more effective modulation. Even though the ID similarity improves significantly, it compromises some degree of editability and flexibility. The second category of methods (22] supports non-reconttr training to enhance editability by constructing datasets grouped by ID; each ID includes several images. However, creating such datasets demands significant effort. Also, most IDs correspond to a limited number of celebrities, which might limit their effectiveness on non-celebrities.
Lack of ID fidelity. Given our human sensitivity to faces, maintaining high degree of ID fidelity is crucial in ID customization tasks. Inspired by the successful experience of face generation (35] 45 tasks during the GAN era [9) a straightforward idea for improving ID fidelity is introduce ID loss within diffusion training. However, due to the iterative denoising nature of diffusion models 12] achieving an accurate Xg needs multiple steps. The resource consumption for training in this manner can be prohibitively high. Consequently, some methods (3 predict Xy directly from the current timestep and then calculate the ID loss. However, when the current timestep is large, the predicted x, is often noisy and flawed. Calculating ID loss under such conditions is obviously inaccurate, as the face recognition backbone (6) is trained on photo-realistic images. Although some workarounds have been proposed, such as calculating ID loss only at less noisy timesteps [29] or predicting Xo with an additional inference step [54], there still remains room for improvement.
In this work, to maintain high ID fidelity while reducing the influence on the original model's behavior, we propose PuLID, a pure and lighting ID customization method via contrastive alignment. Specifically, we introduce a Lightning T2I branch alongside the standard biffusion-denoisee training branch. Leveraging recent fast sampling methods (66 38 23), the lighting T2I branch can generate high-quality images from pure noise with a limited and manageable number of steps. With this additional branch, we can simultaneously address the two challenges mentioned above. Firstly, to minimize the influence on the original model's behavior, we construct contrastive pair with the same prompt and initial latent, with and without ID insertion. During the Lightning T2I process, we align the UNet features between the contrastive pair semantically, instructing the ID adapter how to insert
2

---

ID information without affecting the behavior of the original model. Secondly, as we now have the precise and high-quality generated X, after ID insertion, we can naturally extract its face embedding and calculate an accurate ID loss with the ground truth face embedding. It is worth mentioning that such Xo generation process aligns with the actual test setting. Our experiments demonstrate that optimizing the ID loss in this context can significantly increase ID similarity.
The contributions are summarized as follows. (1) We propose a tuning-free method, namely, PuLID, which preserves high ID similarity while mitigating the impact on the original model's behavior. (2) We introduce a Lightning T2I branch alongside the regular diffusion branch. Within this branch, we incorporate contrastive alignment loss and ID loss to minimize the contamination of ID information on the original model while ensuring fidelity. Compared to the current mainstream approaches that improve the ID encoder or datasets, we offer new perspective and training paradigm. (3) Experiments show that our method achieves SOTA performance in terms of both ID fidelity and editability. Moreover, compared to existing methods, our ID information is less invasive to the model, making our method more flexible for practical applications.
2 Related Work
Tuning-free Text-to-image ID Customization. ID Customization for text-to-image models aims to empower pre-trained models to generate images of specific identities while following the text descriptions. To case the resource demand necessitated by tuning- based methods 36 14 19 10
[1]. a series of tuning-free methods [42] 44 29 50 22 48 51 4 have emerged, which directly encode ID information into the generation process. The major challenge these methods encounter is
minimizing disruption to the original behavior of T21 models while still maintaining high ID fidelity.
In terms of minimizing the disruption, one plausible approach is to utilize a face recognition model [6] to extract more abstract and relevant facial -pmain- representations, as done by IP-Apdater- FacelD [1] and InstantID [4] A dataset comprising multiple images from the same ID can facilitate the learning of a common representation [22]. Despite the progress made by these approaches, they have yet to fundamentally solve the disruption issue. Notably, models with higher ID fidelity often cause more significant disruptions to the behavior of the original model. In this study, we propose a new perspective and training method to tackle this issue. Interestingly, the suggested method does not require laborious dataset collection grouped by ID, nor is it confined to a specific ID encoder.
To improve ID fidelity, ID loss is employed in previous works 18 3. motivated by its effectiveness in prior GAN-based works [35 45 However, in these methods, To is typically directly predicted from the current timestep using single step, often resulting in noisy and flawed images. Such images are not ideal for the face recognition models (6), as they are trained on real-world images. PortraitBooth (29 alleviates this issue by only applying ID loss at less noisy stages, which ignores such loss in the early steps, thereby limiting its overall effectiveness. Diffswap (44 obtains a better predicted To by employing two steps instead of just one, even though this estimation still contains noisy artifacts. In our work, with the introduced Lightning T2I training branch, we can calculate ID loss in more accurate setting.
We notice a concurrent work, LCM-Lookahead (,, which also uses fast sampling technology (i.e., LCM (26)) to achieve more precise prediction of za- However, there are several differences between this work and ours. Firstly, LCM-Lookahead makes a precise prediction of my during the conventional diffusion-denoising process, whereas we start from pure noise and iteratively denoise to zig Our approach, which aligns better with actual testing settings, makes the optimization of ID loss more direct. Secondly, to enhance prompt editing capability, L.CM-Lookaeele
capitalized on the mode collapse phenomenon of SDXL-Tume Turboo [38] to synthesis an ID-consistent dataset. However, the synthetic dataset might face diversity and consistency challenges, and the authors found that training with this dataset may lean towards stylized results more frequently than other methods. In contrast, our method does not need an ID ID-grouped dataset. Instead, we enhance prompt follow ability through a more fundamental and intuitive approach, namely, contrastive alignment.
Fast Sampling of Diffusion Models. In practice, diffusion models are typically trained under 1000 steps. During inference, such a lengthy process can be shortened to few dozen steps with the help of advanced sampling methods 39 25 [7 Recent distill-based works 23 26 38 further accelerate this generation process within 10 steps. The core motivation to guide the student network to align with points further from the base teacher model. In this study, the Lightnnigg T2I training branch we
3

---

introduce leverages the SDXL-Lightning (23] acceleration technology, thus enabling us to generate high-quality images from pure noise just steps.
3 Methods
Conventional Diffusion branch
ID Encoder
Dilzaionn Lasa
Arclace
VIT
Lightning T2I branch
Contrestive Pair
prompts list
#eatic enmmatccc
wethenoller
poth wio
Dal Sa144h
Crnas 4tm
Myyment Lo0e
Aigmment
Accurate ID Loss
Qsi
Qui
preect
ID LoGS
Cdll
Figure 2: Overview PuLID framework. The upper half of the framework illustrates the con- ventional diffusion training process. The face extracted from the same image is employed as the ID condition Cia- The lower half of the framework demonstrates the Lightning T2I training branch introduced in this study. It leverages the recent fast sampling methods to iteratively denoise from pure noise to high-quality images in few steps (4 in this paper). In this branch, we construct contrastive paths with and without IID injection and introduce alignment loss to instruct the model how to insert ID) condition without disrupting the original model's behavior. As this branch can produce photo-realistic images, implies that we can achieve more accurate ID loss for optimization.
3.1 Preliminary
Diffusion models [12] are class generative models capable of synthesizing desired data samples through iterative denoising. A conventional diffusion training encapsulates two procedures, the forward diffusion process
reverse denoising process. During the diffusion process, noise is sampled and added to the data sample To based on a predefined noise schedule. This process yields noisy sample I; at timestep t. Conversely, during the denoising process, denoisng model ey takes æt. 1, and optional additional conditions C' as inputs to predict the added noise, the optimization process can be articulated as:
Lan F.ro.c.t
(I - (a 4, C)
(1)
The denoising model eş in modern T21 diffusion models [37 33 30
is predominantly a UNET
composed of residual blocks (II), self-attention layers, and cross-attention 43) layers. The prompt, as a condition, is embedded into the cross-attention layers adhering to the attention mechanism, illustrated follows:
Attention(Q (,,K, K,V) = Softmax OKT V Va
(2)
K= W K WKTtze(Ca (Crnt) V W WyTee (Gtnt),
where Q is projected from the UNET image features, Ttzt denotes a pre-trained language model that converts prompt Ctat to textual features, W and Wy the learned linear layers.
ID Customization in T2I diffusion introduces ID images C,a as an additional condition, working together with the prompt control image generation. Tuning-free customization [16] 48 50 methods

---

typically employ an encoder to extract ID features from Cta. The encoder often includes frozen backbone, such as CLIP image encoder 32 or face recognition backbone [6], along with learnable head. A simple yet effective technique to embed the II) features to the pre-trained T21 model is to add parallel cross-attention layers to the original ones. these parallel layers, learnable linear layers are introduced to project the ID features into Ka and Via for calculating attention with Q. This technique, proposed by IP-Adapter [50], has been widely used, we also adopt it for embedding ID features in this study.
3.2 Basic Settings
We build our model based on the pre-trained SDXL [30]. which a SOTA T21 latent diffusion model. Our ID encoder employs two commonly used backbones within the ID customization domain: the face recognition model [6] and the CLIP image encoder [32], to extract ID features. Specifically, we concatenate the feature vectors from the last layer of both backbones (for the CLIP image encoder, we use the CLS token feature), and employ a Multilayer Perceptron (MLP) to map them into tokens
as the global ID features. Additionally, following ELITE's approach [46], use MLPs to map the multi-layer features of CLIP to another tokens, serving as the local ID features. It is worth noting that our method is not restricted specific encoder.
3.3 Discussion on Common Diffusion Training in ID Customization
Currently, tuning-free ID customization methods generally face a challenge: the embedding of the ID disrupts the behavior of the original model. The disruption manifests in two ways: firstly, the ID-irrelevant elements in the generated image (c.g.. background, lighting, composition, and style) have changed extensively compared to before the II) insertion; secondly, there is loss of prompt adherence, implying we can hardly edit the ID) attributes, orientations, and accessories with the prompt. Typically, models with higher ID fidelity suffer more severe disruptions. Before present our solutions, we first analyze why conventional diffusion training would cause this issue.
In conventional ID Customization diffusion training process, as formulated in Eq. 1 the ID condition Cia is usually cropped from the target image xo [50 44 In this scenario, the ID condition aligns completely with the prompt and UNET features, implying the ID condition does not constitute contamination to the T2I diffusion model during the training process. This essentially forms a reconstruction training task. So, to better reconstruct xo (or predict noise e), the model will make the utmost effort to use all the information from ID features (which may likely contain ID-irrelevant information), well as bias the training parameters towards the dataset distribution, typically in the realistic portrait domain. Consequently, during testing, when we provide a prompt that in conflict misaligned with the ID condition, such as altering ID attributes changing styles, these methods tend to fail. This is because there exists a disparity between the testing and training settings.
3.4 Uncontaminated ID Insertion via Contrastive Alignment
While is difficult ascertain whether the insertion of ID disrupts the original model's behavior during the conventional diffusion training, it is rather easy to recognize under the test settings. For instance, we can easily observe whether the elements of the image change after the ID is embedded, and whether it still possesses prompt follow ability. Thus, our solution is intuitive. We introduce a Lightning T2I training branch beyond the conventional diffussinn-deniisngg training branch. Just like in the test setting, the Lighting T21 branch starts from pure noise and goes through the full iterative denoising steps until reaching Zo. Leveraging recent fast sampling methods (26 38 23), the Lighting T21 branch can generate high-quality images from pure noise with a limited and manageable number of steps. Concretely, we employ SDXL-Lightning
2 with 4 denoising steps. We prepare list of challenging prompts that can easily reveal contamination, as shown in Table 8 During each training iteration, a random prompt from this list is chosen as the textual condition for the Lighttning T2I branch. Then, we construct contrastive paths that start from the same prompt and initial latent. One path is conditioned only by the prompt, while the other path employs both the ID and the prompt as conditions. By semantically aligning the UNET features on these two paths, the model will learn how to embed ID without impacting the behavior of the original model. The overview of our method shown in Fig. 2
5

---

Qui
mman'
portroii"
"cinematic"
"sweater"
12l w0
Laliga-layust
Laite sem
"a man. nndnng bike? sketch"
Softmax (l.//
8.l.max Kgg /v
prampi
Q2
'man'
"portrait"
"cinematic"
"sweater"
725 will ID
Laliya
(a) Illustration of Calign
(b) Effect of Calign
Figure 3: Illustration and Effect of the alignment loss.
We chose align the contrastive paths in their corresponding UNET's cross-attention layers. Specifi- cally, we denote the UNET features in the path without ID embedding as Qt, whereas the correspond- ing UNET features in the contrastive path with ID embedding Qtid- For simplicity, we omit the specific layers and denoising steps here. actuality, alignment conducted across all layers and time steps.
Our alignment loss consists of two components: the semantic alignment loss and the layout alignment loss. An illustration is presented in Fig. (a). We use textual features K to query the UNET features Q. For each token K, will calculate the correlation with Q, and further aggregate Q based on the correlation matrix. Analogous to Eq. [ the attention mechanism here can be expressed as Attention K, Q, Q). which can be interpreted as the response of the UNET features to the prompt. The insight behind our semantic alignment loss is simple: if the embedding of ID does not affect the original model's behavior, then the response the UNET features to the prompt should be similar in both paths. Therefore, our semantic alignment loss Lalign sem can be formulated as follows:
K KQ Vd
KQU
Lalign Softmax
)Qiid Softmax
1 Vd
As illustrated in Fig. (b), the introduction of Calign-sem significantly mitigates the issue of ID information contaminating the model's behavior. However, it cannot guarantee layout consistency, so add layout alignment loss Lalign-layout which defined
Calign-layout = [Quid Q+112
(4)
The full alignment loss is formulated as
Calign Aalign-sem Aaliga-semmallggoseen Calign-sem
Aalign-tayout Calign--yy
(5)
where Aalign-se sem and layeul serve as hyperparameters that determine the relative importance of each loss item. In practice, we set layout to relatively small value, as we found that a larger value compromises the ID fidelity.
3.5 Optimizing ID Loss in More Accurate Setting
In ID Customization tasks, ensuring a high degree of ID fidelity is essential, given our innate human sensitivity towards discerning facial features. To improve the ID fidelity, aside from enhancements on the ID encoder (50) 44 (53), another universal and parallel improvement is the introducing of an ID loss 6 29 during the training. However, these methods directly predict xy at the t-th timestep in the diffusion training process, only using single step. This will produce noisy and flawed predicted To. subsequently leading to inaccurate calculation ID loss. To ease this issue, recent work [29] proposes to only applying the ID loss on less noisy stages. However, since the ID loss only affects portion of timesteps, which may polentially limit the full effectiveness it. this study, thanks to the introduced Lightning T2I branch, the above issue can be fundamentally resolved. Firstly, we can swiftly generate an accurate Zo conditioned on the from pure noise within steps. Consequently, calculating the ID loss this To, which is very close to the real-world data distribution, is evidently

---

more precise. Secondly, optimizing ID loss in setting that aligns with the testing phase, is more direct and effective. Formally, the ID loss Lia is defined as;
(6) where æ7 denotes the pure noise, L-T2I represents the Lightning T2I branch, and & denotes the face recognition backbone [6]. To generate photo-realistic faces, we fix the prompt Ctar to "portrait, color, cinematic".
Lia - CosSim (á(Cia), (LLTT2I(TT Cia, Ctns))),
3.6 Full Objective
The full learning objective is defined as:
L Lain + Calign XigCus-
During training, only the newly introduced MLPs and the learnable linear layers Kid and Via in cross-attention layers are optimized with this objective, with the rest remaining frozen.
4 Experiments
4.1
Implementation Details
We build our PuLID model based on SDXL 30 and the 4-step SDXL-Lightning 23 For the ID encoder, we use antelopev2 6 as the face recognition model and EVA-CLIP 440 as the CLIP Image encoder. Our training dataset comprises 1.5 million high-quality human images collected from the Internet, with captions automatically generated by BLIP-2 [20]. Our training process consists of three stages. In the first stage, we use the conventional diffusion loss Caint to train the model. In the second stage, resume from the first stage model and train with the ID loss Li (we use arcface-50 [6] to calculate ID loss) and diffusion loss Ldift. This model strives for the maximum ID fidelity without considering the contamination the original model. In the third stage, we add the alignment loss Calign and use the full objective as shown in Eq. 7 to fine-tune the model. We set the Aalign-sem to 0.6, Aalign-layout 0.1, and A to 1.0. In the Lightning T2I training branch, set the resolution of the generated image to 768 x 768 to conserve memory. Training is performed with Py Torch and diffusers on 8 NVIDIA A100 GPUs in internal cluster.
4.2 Test Settings
For consistency comparison, unless otherwise specified, all results this paper are generated with the SDXL-Lightning [23 base model over 4 steps using the DPM++ 2M sampler [17]. The CFG-scale set to 1.2, as recommended by [23). Moreover, for each comparison sample, all methods utilize the same seed. We observe that the comparison methods, namely InstantID 44 and IPAdapter (more specifically. IPAdapter-Facee Face [1) are highly compatible with the SDXL-Lighhning model. As shown in appendix subsection 7.5 when compared to using SDXL-base 30 the base model, employing SDXLL-Lggtniig results in InstantID generating more natural and aesthetically pleasing images, and enables IPAdapter to achieve higher ID fidelity. Furthermore, we provide quantitative comparison with these methods on SDXL-base, with the conclusions remaining consistent with those on SDXL-Lightning.
To more effectively evaluate these methods, we collected a diverse portrait test set from internet. This set covers a variety of skin tones, ages, and genders, totaling 120 images, which we refer to as DivID-122 120. As a supplementary resource, we also used a recent open-source test set, Unsplash-50 (8), which comprises 50 portrait images uploaded to the Unsplash website between February and March 2024.
4.3 Qualitative Comparison
As shown in Fig. 4 when compared to SOTA methods such as IPAdapter and InstantID, our PuLID tends to achieve higher ID fidelity while creating less disruption to the original model. From columns 1, 2, 5, 6, and 7, it is clear that our method can attain high ID similarity in realistic portrait scenes and delivers better aesthetics. Conversely, other methods either fall short ID fidelity or show diminished aesthetics compared to the base model. Another distinct advantage of our approach is
7

---

Input
Playing wearing hot on the beach white dreas
portrait cinematic
portrait side view
anime pikachu
portrait paper art
portrait child
portrait
cartoony mario
mask
IPAdapter
T21 wio ID PuLiD(our)) InstantD
Figure 4: Qualitative comparisons. T2I w/o ID represents the output generated by the original T21 model without inserting ID, which reflects the behavior of the original model. Our PuLID achieves higher ID fidelity while causing less disruption to the original model. As the disruption to the model reduced, results generated by PuLID accurately reproduce the lighting (Ist row), style (4th row), and even layout (5th row) of the original model. This unique advantage broadens the scope for more flexible application of PuL.ID.
Table 1: Quantitative comparisons. *We observed that PhotoMaker shows limited compatibility with SDXL-Lightning, hence, we compare its performance on SDXL-base in this table.
DivID-120
Unsplash-50 CLIP-Tt
Face Sim.
CLIP-T1
CLIP-I
Face Sim.t
CLIP-I1
Photo 0.271
26.06
0.649
0.193
27.38
0.692
IPAdapter
0.619
28.36
0.703
0.615
28.71
0.701
InstantID 0.725
28.72
0.680
0.614
30.55
0.736
PuLID (ours)
0.733
31.31
0.812
0.659
32.16
0.840
that the disruption to the model decreases, the results produced by PuLID accurately replicate the lighting (Ist column), style (4th column), and even layout (5th column) of the original model. In contrast, although comparative methods can also perform stylization, notable style degradation can be noticed when compared to the original model. Finally, our model also possesses respectable prompt- editing capabilities, such as changing orientation (2nd column), altering attributes (6th column), and modifying accessories (7th column). More qualitative results can be found in subsection 7.1 and subsection 7.2 of the appendix.
4.4 Quantitative Comparison
The quantitative results are reported Table T Face Sim. represents the ID cosine similarity, with ID embeddings extracted using CurricularFace [15]. CurricularFace is different from the face recognition models we use the encoder and for calculating ID loss. CLIP-T 32 measures the ability to follow prompt. We also use CLIP-I to quantify the CLIP image similarity between two images before and after the ID insertion. A higher CLIP-I metrie indicates a smaller modification in image elements (such as the background, composition, style) after ID insertion, suggesting lower degree

---

Input
w/o & aalign
wi & align
T21 wio ID
Input
w/o 4. Lalignn
w/ L. align
Til ID
upper body
playing plano the beach dress
2d
playing bosketball wateroslor
Figure 5: Qualitative comparison for ablation study on alignment loss.
Table 2: Quantitative comparisons for ablation studies on ID loss and alignment loss.
DivID 120
Unsplash-50 CLIP-TT
Face Sim.1
CLIP-TY
CLIP-I1
Face Sim.
CLIP-I
Baseline (Stagel)
0.561
29.06
0.736
0.514
30.16
0.769
ID Loss naive
0.652
27.05
0.683
0.601
28.00
0.707
w/ ID Loss (Stage2)
0.761
24.91
0.624
0.708
25.83
0.646
PuLID (Stage3)
0.733
31.31
0.812
0.659
32.16
0.840
of disruption the original model's behavior. We provide the evaluation prompts in the appendix. As observed from Table our method, PuLID, surpasses comparison methods across all three metrics, achieves SOTA performance in terms of both ID fidelity and editability. Furthermore, method significantly outperforms others with respect to the CLIP-I metric, implying that our method incurs much less intrusion on model behavior compared to other methods.
Ablation
Alignment loss ablation. Fig. displays qualitative comparison between models trained with and without the alignment loss Calign- As observed, without Lalign- the embedding of ID severely disrupts the behavior of the original model. This disruption manifests modify style (the left two cases of Fig. 5 and orientation (the lower right case of Fig. 5). Also, the layout would collapse the extent that the face occupies the majority of the image area, resulting in a diminished diversification of the layout. However, with the introduction our alignment loss, this disruption can be significantly reduced. From Table 2 could also observe large improvement in CLIP-T and CLIP-I when equipped with Lalign (from Stage2 to Stage3).
inability for the prompt to precisely
ID loss ablation. Table iillustrates the improvement in ID fidelity using the naive ID loss (directly predicting Ty from current timestep) and the more accurate ID loss Ly introduced this paper, in comparison to the baseline. As observed, La accomplish greater improvement compared to the naive ID loss. We attribute this to the more precise Zo provided by the Lightning- -T2 branch, which also better aligns with the testing setting, thereby making the optimization of ID loss more direct and effective. Another worth mentioning point is that the baseline, naively trained on the internal dataset, underperforms in ID fidelity and editability. Conversely, the introduction of the PuLID training paradigms delivers significant enhancement. Therefore, this substantiates that the improvement mainly comes from the method, rather than the dataset.
Limitation
While our PuLID achieves superior fidelity and editability with minimal disruption to the base model's behavior, it still has limitations. Due to the incorporation of the Lightning T21 branch, the training speed per iteration slower than that conventional diffusion training, and also demands more CUDA memory. However, this issue can be significantly mitigated when future fast sampling

---

methods can generate satisfying results in single step (currently, we use 4 steps), Another limitation that although the accurate ID loss introduced in the Lightniig T2I branch markedly enhances ID fidelity, it also impacts image quality to some extent, such as causing blurriness in faces. This can be discerned Fig. (b) and Fig. 5 Nonetheless, this issue is not uniquely associated to our method, as similar phenomena has been observed in reward tuning methods [49 31, 47), with [5] attributing it to the reward- hacking problem. Although this issue can be largely alleviated by pairing with our proposed contrastive alignment loss, future work could explore more effective ID loss that does not negatively affect image quality.
Conclusion
This paper presents PuLID, novel tuning-free approach to ID customization in text-to-image generation. By incorporating Lightning T2I branch along with a contrastive alignment strategy, PuLID achieves superior ID fidelity and editability with minimal disruption to the base model's behavior. Experimental results demonstrate that Pul.ID surpasses current methods, showcasing its potential for ficxible and efficient personalized image generation. Future work could explore the application of this proposed training paradigm to other image customization tasks, like IP and style customization.
References
[1]
2024.
[2]
https://github.co cowpbuub-cccombbbacaa---oooeeeee
2024.
[3] Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding, Yangyang Song, Shizun Wang, Xu Wang, Hao Yang, Jing Liu, Kang Du, et al. Photoverse: Tuning-free image customization with text-to-image diffusion models. arXiv:2309.05793, 2023.
[4] Zhuowei Chen, Shancheng Fang, Wei Liu, Qian He, Mengqi Huang, and Zhendong Mao. Dreamidentity: Enhanced editability for efficient face-identity preserved image generation. In AAAI, 2024.
[5] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In ICLR, 2024.
[6] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos 7afeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[7] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
[8] Rinon Gal, Or Lichter, Elad Richardson, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Lem-lookahead for encoder-based text-to-image personalization. In ECCV, 2024.
[9] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farte Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
[10] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In ICCV, 2023.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrlPS, 2020.
[13] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In ICML, 2019.
[14] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
10

---

[15] Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In CVPR, 2020.
[16) Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingho Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv:2304.02642. 2023.
[17] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Neur/PS, 2022.
[18] Kihong Kim, Yunho Kim, Scokju Cho, Junyoung Seo, Jisu Nam, Kychul Lee, Seungryong Kim, and KwangHee Lee. Diffface: Diffusion-based face swapping with facial guidance. arXiv:2212. 13344, 2022.
[19] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of text-to-image diffusion. In CVPR, 2023.
[20] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In /CML, 2023.
[21] Xiaoming Li, Xinyu Hou, and Chen Change Loy. When stylegan meets stable diffusion: a w plus adapter for personalized image generation. In CVPR, 2024.
[22] Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding. In CVPR, 2024.
[23] Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv:2402. 13929, 2024.
[24] Shanchuan Lin and Xiao Yang. nhimatediff-lightning Cross-model diffusion distillation. arXiv:2403. 12706, 2024.
[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. vrXiv:2211.0f00 1.01095, 2022.
[26] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv:2310.04377 2023.
[27] Yuchen Luo, Yong Zhang, Junchi Yan, and Wei Liu. Generalizing face forgery detection with high-frequency features. In CVPR, 2021.
[28] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. In AAAI, 2024.
[29] Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji. Portraitbooth: A versatile portrait model for fast identity-preserved personalization. In CVPR, 2024.
[30] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
[31] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXhv:2310.03739 2023.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021.
[33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. WrXiv:204.006155, 2022.
[34] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hdper-sd: Trajectory segmented consistency model for efficient image synthesis. In NewrlPS, 2024.
[35] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In CVPR, 2021.
11

---

[36]
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
[37]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[38] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv:2311.17044 2023.
[39] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICIR, 2021.
[40] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv:2303.15389, 2023.
[41] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In SIGGRAPH, 2023.
[42] Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. FaceO: Instantaneously conditioning a text-to-image model on face. SIGGRAPH Asia, 2023.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[44] Qixun Wang. Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot preseity- generation in seconds. arXiv:2401.07519 2024.
[45] Xintao Wang. Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In CVPR, 2021.
[46] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In ICCV, 2023.
[47] Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Keqiang Sun, Zhaoyang Huang, Guanglu Song, Yu Liu, and Hongsheng Li. Deep reward supervisions for tuning text-to-image diffusion models. arXiv:2405.0072 00760, 2024.
[48] Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser:
Tuning-free free multi-subject image generation with localized attention. IJCV, 2024.
[49] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding. Jie Tang. and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In NeurlPS, 2024.
[50] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv:2308.0672/ 2023.
[51] Ge Yuan, Xiaodong Cun, Yong Zhang, Maomao Li, Chenyang Qi, Xintao Wang, Ying Shan, and Huicheng Zheng. Inserting anybody in diffusion models via celeb basis. In NeurIPS, 2023.
[52] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[53] Shilong Zhang. Lianghua Huang, Xi Chen, Yifei Zhang. Zhi-Fan Wu, Yutong Feng, Wei Wang, Yujun Shen, Yu Liu, and Ping Luo. Flashface: Human image personalization with high-fidelity identity preservation. arXiv:2403. 17008 2024.
[54] Wenliang Zhao, Yongming Rao, Weikang Shi, Zuyan Liu, Jie Zhou, and Jiwen Lu. Diffswwp:: High- High-fidelity and controllable face swapping via 3d-aware masked diffusion. In CVPR, 2023.
12

---

7 Appendix
7.1 More Applications of PuLID
We provide more applications of our PuLIDD in Fig- 6 encompassing style alterations (Ist row). IP fusion (2nd row), accessories modification (3rd row), recontextualization (4th row), attributes editing (5th row), transformation from rea-phooo-realisiic domains to photo-realistie ones (6th row), and ID mixing (7th row).
7.2 Generalization Ability of PuLID
Testing With Community Models. During training, keep the base model parameters fixed. We use SDXL. in the initial stage and SDXX-LLihhniig, which is distilled from SDXL. in the final two stages. SDXL-Lighting is effective in retaining the style and layout of SDXL., allowing our model to generalize to community models based on SDXL during testing. However, disruptions caused by ID insertion are more noticeable when testing other base models rather than SDXL-Lightning, our training base model. To combat this, we enhance compatibility by simultaneously aligning with multiple base models during training. This process involves loading different base models onto different GPU ranks, while single ID adapter is shared across all ranks (24). Using this strategy, our PuLID model attains robust compatibility with popular community models, both accelerated and accaecclerated ones. This includes models like RealVisXI. (Fig JuggermautxX--Lignnnnn (Fig T0. and DreamShaper-X--Liinnnnng (Fig. II
Training With Non-accelerated Base Models. We want to highlight that the core essence of our paper is introduce more accurate ID loss and alignment loss the T2I training branch to achieve better ID fidelity and editability. The fast sampling method (such as SDXL-Lightning) serves as an optional acceleration trick, but it is not indispensable. Without the fast sampling method, would need 30 inference steps with CFG on the T2I branch, compared to the current 4 required inference steps without CFG. Due to CUDA memory bottleneck (we exclude the use of gradient checkpointing due to its significant speed penalty). it is not feasible to perform backpropagation (BP) of the gradient all timesteps. Nonetheless, remains possible to make optimization viable with strategic techniques. Particularly, for the optimization of ID) loss, BP of the gradient happens only for the last few timesteps [5] For the optimization of alignment loss, a timestep is randomly selected for BP of the gradient. Table shows the differences in speed and memory consumption between training with and without acceleration. From the table, we can see that if we do not use fast sampling and take SDXL as the base model for the T2I training branch, efficiency will indeed be much lower. However, thanks to the carefully designed optimization strategies mentioned above, the training method presented in this paper can be effectively adapted to non-accelerated models, with performance being further improved, shown in Table Additional visual results can be found in Fig. 12 In summary, our method does not rely accelerated base models, thus reflecting the universality of our approach. We also successfully adapt PuL.DD to a much larger and non-accelerated base model, FLUX 2), and open-source it at https://github.com To BeBeginniin/PPu.I
Table 3: Speed and memory comparison between training with and without acceleration. ts denotes timestep.
BP last ts
BP last ts
BP last 3 ts
BP last 4 ts
BP last 20 ts
w/ fast sampling
2. 2.66tten4441BB
2. 2.9w/teer49GBB
3. 1.GG
3.35/tter(3GBB OOM
w/o fast sampling
6. 6.6G/)te((0GGB)
7.05/iter(65GBB
7. 7.3
OOM
Table 4: Quantitative comparison between training with and without acceleration.
DivID-120 CLIP-Tt
Unsplash-50 CLIP-Tt
Face Sim.t
CLIP-I1
Face Sim.
CLIP-I
w/ fast sampling
0.733
31.31
0.812
0.659
32.16
0.840
w/o fast sampling
0.743
31.75
0.842
0.687
32.58
0.865
13

---

Fantsay
sayon
acrowolf
CorTezE
ID
ID
WOTEON
conNert
comvert
connert
ID
Figure 6: More applications. Including style changes, IP fusion, accessory modification, recon- textualization, attribute editing, transformation from nenaphito domain to photo-realistic domain, and ID mixing. Note that all these high-quality images are generated in just steps with SDXL-Lightning model, without the need for additional Lora.
14

---

Table 5: Quantitative comparison of training with different fast sampling methods and inference
steps.
DivID-120 CLIP-TT
Unsplash-50 CLIP-TY
Face fim.
CLIP-I1
Face Sim.1
CLIP-It
Hyper-SD T=1
0.694
31.91
0.819
0.632
31.89
0.857
Hyper-SD T=2
0.720
32.08
0,810
0.653
32.35
0,840
Lightning T=4
0.733
31.31
0,812
0.659
32.16
0.840
Lightning T=8
0.734
31.66
0.818
0.668
32.19
0.850
cartoony manic
strong
PaLiibouur))
M
M
InstantD
M
M
M
M
IPAulageer
Figure 7: Results under different ID weights. The weights incrementally increase from left to right, culminating at the chosen weight at the far right.
7.3 Ablation Study on Fast Sampling Methods
Our selection criteria for a fast sampling method is that it can generate high-quality images within limited number of steps while well preserving the style and layout of the original model. SDXL- Lightning fulfills these requirements and was the SOTA option at that time, so we selected it to accelerate our T2I training branch. Moreover, we settle on steps as it achieves balance between generates
efficiency and quality. With fewer steps (e.g., 1 or 2), the likelihood that SDXL-Lightnin flawed faces increases. However, have noted a recently emerged fast sampling method, Hyper- on I and 2 steps. Therefore, we present the
SD 34, which performs better than SDXL-LLghtmin results of training on step and steps with Hyper-SD, as well as 8 steps with SDXLL-Lghtniig in Table[5] As shown in this table, training with or 2 steps leads to reduction in face similarity. Meanwhile, training with steps slightly enhances overall performance. Considering both efficiency and performance, steps is sound choice.
7.4 More Details about the Test Settings
Evaluation Prompts. Table 7 presents the complete list of prompts utilized in the calculation of CLIP-T and CLIP-I. For measuring Face Sim., employ the prompt "portrait, color, cinematic, in garden, soft light, detailed face" to guarantee that the generated face is photo-realistic and detectable. For each ID and each prompt, randomly generate four images for evaluation.
Hyperparameter Selection for the Comparison Methods. We employ the popular and widely used ComfyUl UI workflow P in the community to test IPAdapter and InstantID. For InstantD, the weight is set to 0.8. For IPAdapter, the lora scale is set to 0.8 and the IP weight is set to 1.5. Additionally, we present the results of these methods under different ID weights (as shown in Fig 7 It can be
hautss//gituubbco///uuu//CCoCoi InstantlD https:/ggiibbbbommmmoo/////m/mup /ataa///ooooaaoo///o///o//oo/ool
15

---

Table Quantitative comparisons on SDXL-base.
DivID-120
Unsplash-50 CLIP-TY
Face Sim.1
CLIP-TT
CLIP-I1
Face Sim.
CLIP-I1
PhotoMaker
0.271
26.06
0.649
0.193
27.38
0.692
IPAdapter InstantID
0.597
29.39
0.720
0.572
30.03
0.749
0.755
25.33
0.608
0.648
26.41
0.649
PuLID (max. ID sim.)
0.773
24.77
0.598
0.711
25.33
0.628
PuLID (ours)
0.722
30.69
0.781
0.654
31.23
0.813
ID
InstantID
IP Iddateer
SDXL-Lighning
SDXL-bbase
SDXL-Lighhing
SDXL-base
Figure 8: Visual comparison of different base model for InstantID and IPAdapter.
observed that for InstantID and IPAdapter, the ID embedding only has minimal impact on the original model when the ID weight set to a relatively small value. However, this point, the ID similarity decreases significantly. Since our method only influences the ID-relevant parts, it doesn't 't significantly disrupt the original model's behavior, even when increasing ID weight.
7.5 Comparisons on SDXL-base
The Influence of Changing Base Model on the Comparison Methods. Here, we present the visual differences associated with these methods when varying base models are used. Fig. 8 showcases the qualitative comparison results. As observable from the figure, InstantID attains better image quality and aesthetics when using SDXL-Lightning instead of SDXL-base, while the ID fidelity shows little visual difference. For IPAdapter, better ID fidelity can be achieved using SDXL--Lghhning compared to SDXL--bas.
Quantitative Comparison. We provide the quantitative comparison on SDXL-base in Table, 6 As evident, our final model, PuLID, still outperforms the comparative methods in most scenarios, with the exception of Face Sim. on DivID-120, where our method slightly falls behind InstantID, Nevertheless, our method significantly surpasses InstantID in terms of CLIP-T and CLIP-I metries, and our model (from Stage2) that strives for maximum ID similarity also manages to surpass InstantID on the Face Sim. metric. Another intriguing and noteworthy point that typically, an enhancement in ID similarity accompanies decrease in the CLIP-T and CLIP-I metrics. However, our method has the unique capability to not only achieve a high degree of ID similarity, but concurrently maintain high editability and low interference. This is facilitated by the novel training paradigm introduced in this study.
7.6 Prompt List for Contrastive Alignment
Table 8 presents the complete list of prompts used in contrastive alignment. Despite their simplicity, these prompts already ensure an effective methodology. Future exploration will involve determining if there exists more optimal list of prompts.
16

---

Nersts therss paaning perter
Vhi and
pormaat: chandiic Egitigg
vider woter JBiCE
arhawst sall with
spmratstcseee hearly
Apriial of Eoppang bid, luugh bourlly
serrach
with techcloyy sesepeer
and Elat
68 Africa
Figure 9; PuLID with RealVis-XT, as base model. Zoom in for best view
17

---

shat of glawing forest sprit
Aurona Artiet Caured Rese!
wickdd shut
RysherbLe diroastte NOTEE whth vill
blrmechunic cyberpur gone spkied byloiid
ethened TIREECOTT Spperiaaee
Conaatic Lighting aun Eight
Super
Figure 10: PuLID with Juggernaut-XLe XL-LtttX--LLLLLLin as base model. Zoom in for best view
18

---

SCRCe. full plate
Lord lings,
greee shinad Strek
pertral, kooree royol-ake sweoter
Astrenoui, planet carth
depictiss Enteligence
vecreedogy engeniz
Figure 11: PuLID with Dreamshaper-XL-LLighnnng as base model. Zoom in for best view
19

---

dayy
fraihed glos
Ganrry candy
Intle
mode
araficc shot. dark
romen,
parting
prst
drowings
20yber-s d Africsse
20-yaa--dh Curade
<55n von
091, Crylng
29t msh
28
Aomon covering mn fice with hards
wendo
carolc
Film Bimn Paahng
Ize
Figure 12:
Qualitative results PuLID trained without fast sampling methods. We test the model on a wide range styles, face materials, lighting conditions, and editing tasks, demonstrating the effectiveness of generalizability of PuLID. Zoom in for best view.
7.7 Broader Impacts
Thanks the superior ID fidelity and editability offered by PuLID, this technology holds the potential to enhance personalized content creation, offering benefits for various applications such as entertainment and virtual reality. However, could also be exploited create misleading or false representations of individuals, potentially posing privacy infringements or being used maliciously. such as in deepfakes. Although existing deepfake detection methods [27] can help mitigate these risks, we underscore the imperative role of developing and adhering to stringent ethical guidelines for ensuring responsible use of such technology.
20

---

Category
Prompt
Clothing
portrait, a person wearing spacesuit portrait, a person wearing a surgical mask
portrait, a person wearing eye mask
Accessory
portrait, a person wearing headphones with red hair portrait, a person wearing a doctoral cap portrait, a person cooking in the kitchen portrait, a person coding front of a computer portrait, a young child laughing at the camera portrait, am angry person,old portrait, color,side view
Action
Attribute
View
background
portrait, with beautiful purple sunset at the beach in the background portrait, pencil drawing
portrait,3ID Animation, Disney jyll,cute..oppaar blindbox
Style
portrait, kawaii style, cute, adorable, brightly colored, cheerful, anime influence, highly detailed portrait,latee art in a cup
portrait, side view, in papereraft style portrait, a garden gnome, in papercraft sketch, wearing glasses portrait, Madhubani, wearing mask portrait, Ukiyo-e Painting style, playing the violin portrait of green-skinned shrek, wearing lacoste purple sweater
Complex
Table 7: Evaluation prompts.
portrait, color, cinematic
portrait, anime artwork
portrait, comic art, graphic novel art
portrait, digital artwork, illustrative, painterly, matte painting
portrait, magnificent, celestial, ethereal, painterly, epic, majestic, magical, fantasy art, cover art, dreamy
portrait, vibrant, professional, sleek, modern, minimalist, graphic, line art, vector graphics
portrait, cyberpunk, vaporwave, neon, vibes, vibrant, stumningly beautiful, crisp, detailed, sleek, ultramodem, magenta highlights, dark purple shadows, high contrast, cinematic, ultra detailed, intricate, professional
portrait, energetic brushwork, bold colors, abstract forms, expressive, emotional
portrait, film noir style, monochrome, high contrast, dramatic shadows, 1940s style, mysterious, cinematic
portrait, papercut, mixed media, textured paper, overlapping, asymmetrical, abstract, vibrant
portrait, paper mache representation, 3D, sculptural, textured, handmade, vibrant, fun portrait, vaporwave style, retro aesthetic, eyberpunk, vibrant, neon colors, vintage 80s and 90s style, highly detailed
portrait, watercolor painting, vibrant, beautiful, painterly, detailed, textural, artistic
portrait, impressionist painting, loose brushwork, vibrant color, light and shadow play
portrait, expressionist, raw, emotional, dynamic, distortion for emotional effect, vibrant, use of unusual colors, detailed
Table 8: Prompt list for contrastive alignment.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the paper contains sufficient details for reproducing the experi- ments, and more implementation details can be referred to our open-source code httpgghhwwbb.ooooooTeeeeeeeiiiie Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We
provide
the codes
and
models
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
aublic//uudees/Codeebbbmsss
www.ssssinoooiyyylyooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttp:/cccpblccguuuiiceeee wtteee///oooSSeissss.ee.ee. www.ssssiicooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides the settings and details of the method, more details will be available in the codes.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following previous methods, we did not report the error bars. But in order to calculate the metrics, we used the same seed and averaged the generated results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide the sufficient information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:::///neuiiii httprpeurp...// httph:/ttt....a.p.......aaaalaat
Answer: [Yes]
Justification:
Yes, the research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss both potential positive societal impacts and negative societal impacts of the work in subsection 7.7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, they are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "hG6rPjvWXW", "reviewer": "Reviewer_bfEK", "summary": "The paper propose a tuning-free method for customization text-to-image diffusion model. Particularly, the author propose to utilize efficient diffusion model (SDXL-lightning in this case) to generate samples during training. Then, they adopt contrastive alignment loss to preserve the identity of subject in the input.", "questions": "- Since the author roll out full diffusion path (4 steps) during training, how much memory is required for training (e.g., compared to existing methods)? \n- How does the model perform if we set the number of diffusion step to (1, 2, 6)? (and how much memory footprint)\n- Does this method work for subject other than human as input (e.g., dog, cat, etc) ?", "limitations": "please refer to weakness section.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is fairly well-written and easy to read.\n- The proposed method is well-motivated and sound.\n- The ablation experiment demonstrates the effectiveness of each proposed components.\n- Reported results seem quite impressive.", "weaknesses": "- Since the model requires an efficient text-to-image diffusion model (in terms of number of inference steps), it can hinder the application of  the introduced method with other base models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and valuable comments. Our responses are as follows:\n\n**`Q1:` Since the model requires an efficient text-to-image diffusion model (in terms of number of inference steps), it can hinder the application of the introduced method with other base models.**\n\nPlease refer to the detailed explanation in `GQ1`. The fast sampling plugin is not indispensable for our method, which indicates that our method can be adapted to other base models without an acceleration plugin.\n\n---\n\n**`Q2:` How much memory is required for training?**\n\nWe provide comprehensive data in `GQ1` and `GQ2` and discuss the memory issue in the Limitations section of the paper (L464-L465). As most comparative methods did not make their training codes open-source or disclose their memory usage, we cannot make a direct comparison.\n\n---\n\n**`Q3:` How does the model perform if we set the number of diffusion step to (1, 2, 6)? (and how much memory footprint)**\n\nPlease see `GQ2` for a detailed comparison. Note that due to memory constraints, we only update the gradients at 6 of the 8 steps.\n\n---\n\n\n**`Q4:` Does this method work for subject other than human as input (e.g., dog, cat, etc)?**\n\nYes, it is theoretically feasible. Please refer to the discussion in `GQ3`.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LjE6xPDmcj", "reviewer": "Reviewer_GW6B", "summary": "This article presents a novel fine-tuning-free ID customization method called PuLID for text-to-image generation tasks. The method introduces Lightning T2I and contrast alignment loss, aiming to minimize the interference with the original model behavior while maintaining high ID fidelity. Experiments show that PuLID achieves excellent performance in terms of both ID fidelity and editability.", "questions": "1. Lines 133 and 195 both mention that Q is an image feature of UNet, could the authors please explain how that image feature was obtained.\n2. Is L_align-layout necessary? Is there any connection to your solution to the problem of ID information polluting the prompt?\n3. In addition I would like to know how the authors' approach relates to the CFG scale (Classifier-Free Guidance scale) and whether CFG is valid for this task.", "limitations": "YES", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Introducing Lightning T2I branching and contrast learning approaches to achieve better performing ID customization methods.\n2. The authors provide a wealth of ablation experiments to demonstrate the effectiveness of the method.", "weaknesses": "Here are some points and suggestions on how the authors should address them:\n\nClarification of Contributions:\nThe authors should clearly delineate their contributions in the manuscript. If the primary contribution is the use of contrastive alignment loss and ID loss, they should emphasize how these methods specifically address the problem of high ID similarity and reduce ID interference.\nThey should provide a more detailed explanation of how these losses are implemented and why they are effective.\n\nRelated Work:\nThe authors should expand their discussion on related work to include whether contrastive learning has been applied in similar contexts. This would help situate their work within the broader field and demonstrate its novelty.\n\nMethod Section:\nThere should be a more detailed explanation of how the method was realized. This includes a clear description of the steps taken, the rationale behind the chosen approach, and any unique aspects of the implementation.\n\nTheoretical Analysis:\nThe authors should provide a more in-depth theoretical analysis, particularly for components like L_align-sem. They should explain the theoretical underpinnings of their approach and how it contributes to the overall effectiveness of the method.\nDiscussion on Main Contributions:\n\nThe manuscript should include a richer discussion on the main contributions. For example, the authors should explore whether the proposed loss functions reduce the weight of ID in text-to-image generation tasks. They should provide evidence or reasoning to support their claims.\n\nAblation Studies:\nThe authors should conduct ablation experiments to validate the necessity and effectiveness of each component of their method. For instance, if L_align-layout is introduced without ablation studies, it's unclear how crucial this component is to the overall performance.\nExperimental Results:\n\nWhile the effectiveness is illustrated in the experimental results, the authors should ensure that these results are robust and that the experiments are designed to test the method under various conditions.\n\nLayout Consistency:\nThe authors should address the issue of layout consistency more thoroughly. If L_align-layout is mentioned but not rigorously tested, they should either conduct additional experiments or provide a theoretical justification for its inclusion.\nOverall Clarity and Depth:\n\nThe authors should ensure that the manuscript is clear and that the depth of the discussion matches the complexity of the problem. This includes explaining the limitations of their approach and how it compares to existing methods.\nBy addressing these points, the authors can provide a more comprehensive and convincing argument for their contributions, making their research more accessible and impactful to the academic community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful advices. We respond to your core questions as follows:\n\n\n**`Q1:` Lines 133 and 195 both mention that Q is an image feature of UNet, please explain how that image feature was obtained.**\n\nIn each cross-attention layer of UNET, the UNET image features are projected into query features via a linear layer. We denote these features as Q. This is reflected in L132-L134 of the main paper. We will revise the description to make this clearer.\n\n---\n\n**`Q2:` Is L_align-layout necessary? Is there any connection to your solution to the problem of ID information polluting the prompt?**\n\nYes, it is necessary. We consider layout and composition of the generated images as part of the original model's behavior. Therefore, if the layout and composition change after ID insertion, we perceive it as a disruption to the original model's behavior. The L_{align-layout} is helpful in maintaining the original model's layout and composition, as demonstrated in Figure 3 (b) of the paper, we also provide more qualitative results in **Figure 2 of the Rebuttal PDF**. For a quantitative ablation, we provide the table below:\n\n|                             | Face Sim. | CLIP-T | CLIP-I |\n| --------------------------- | --------- | ------ | ------ |\n| Stage2                      | 0.761     | 24.91  | 0.624  |\n| Stage3 w/o L_{align-layout} | 0.728     | 30.33  | 0.758  |\n| Stage3                      | 0.733     | 31.31  | 0.812  |\n\nAs observed from the table, if L_{align-layout} is removed in stage3, despite achieving the same level of face similarity as stage3 with all losses, both CLIP-T and CLIP-I decrease. While the decline in CLIP-T is relatively mild, the decrease in CLIP-I is more significant, as it tends to reflect consistency in layout and composition to a certain extent.\n\n\n---\n\n**`Q3:` How the approach relates to the CFG scale and whether CFG is valid for this task**\n\nDuring training, our Lightning T2I branch does not require CFG, hence the CFG scale is 1. However, our method can incorporate CFG during training. When the SDXL-Lightning acceleration is not employed (refer to GQ1), a CFG scale of 7.5 is used.\n\nAs for testing, various CFG scales are compatible with our approach. For the base models utilized in our paper, we adopt the recommended CFG scales, such as 1.2 for SDXL-Lightning, 7 for RealVisXL, and 2 for Juggernaut-XL-Lightning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W3py68QzUK", "reviewer": "Reviewer_BtJK", "summary": "This paper introduces Pure and Lightning ID customization (PuLID). It is a tuning-free approach for customizing identities in text-to-image generation. The model is trained over a huge dataset.\n\nTechnically, it integrates a Lightning T2I branch alongside a standard diffusion model, incorporating contrastive alignment loss and accurate ID loss to maintain fidelity to the original model. \n\nExperimental results show PuLID achieves superior performance in preserving identity fidelity and enhancing editability. Additionally, PuLID ensures consistency in image elements like background, lighting, composition, and style throughout the identity customization process.", "questions": "Check the weaknesses", "limitations": "Check the weaknesses", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well-written as it explores the shortcomings of previous approaches in realistic scenarios, highlighting issues such as fidelity and the impact of inserting IDs into original T2I diffusion models.\n\n2. The novelty of this paper lies in its new perspective on training paradigms with the introduction of PuLID. Unlike existing ID-preservation papers that focus on ID encoders, PuLID distinguishes itself by incorporating a Lightning T2I branch. This distinction is further enhanced through the integration of ID loss and alignment loss.\n\n3. The experiments conducted in this study demonstrate superior performance both quantitatively and qualitatively compared to current methods.", "weaknesses": "1. The primary limitation, in my opinion, is that the PuLID model is trained using an internal dataset. This raises questions about whether the observed improvements stem primarily from the introduction of this new dataset or from the efficacy of the method itself. Consequently, it's unclear whether comparisons with other methods on public datasets are fair or solely reflective of dataset differences.\n\n2. Another picky question for the authors is to include more quantitative comparison metrics as in PhotoMaker[1] and ConsistentID[2], including DINO, Face Diversity, FID, Speed, FGIS (fine-grained identity similarity), etc. Furthermore, can you explain why the experimental results in your paper are much lower than that showed in the original PhotoMaker paper?\n\n3. It seems the T2I lightning branch is pretty important for your method PuLID. Have you done any ablation study over the choices of the T2I lightning models, e.g. the LCM, SDXL-turbo, TCM, InstaFlow, UFOGen, etc? This is to verify your model can be based on SDXL or other backbones, and the lightning models are with diverse choices. Also you choose T=4, would it be influenced by T=1,2,8...?\n\n\n\n\n[1] PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding\n\n[2] ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive and positive comments. Our responses are as follows:\n\n**`Q1:` Concerns about the internal training dataset.**\n\n**1. Dataset or method – primary source of improvement?**\nTable 2 in the main paper illustrates that a baseline, naively trained on the internal dataset, underperforms in ID fidelity and editability. Conversely, the introduction of the PuLID training paradigms delivers significant enhancement. Therefore, this substantiates that the improvement mainly comes from the method, rather than the dataset. We will revise the table analysis to emphasize this point.\n\n\n**2. Fairness in comparing methods trained on different datasets?**\nExisting SOTA methods often utilize internal and private training datasets. For instance, InstantID uses a 60 million-entry dataset, compared to our 1.5 million, and PhotoMaker uses an internal ID-group dataset, more challenging to gather than a non-ID-group dataset.\n\n**3. Lower dataset quality requirements with our method.** We would like to highlight that our method is less dependent on high-quality training datasets. Particularly, the ID loss and alignment loss in the Lightning T2I branch only require uncaptioned facial regions. Further, the unique feature of PuLID, which minimize the contamination to the original model behavior, ensures that the results are unbiased towards the quality of the training dataset, effectively reducing dataset quality requirements.\n\n---\n\n**`Q2:` More quantitative metrics.**\n\nThanks for your suggestion, here are the results on additional metrics:\n\n|            | Speed↑   | FGIS↑ | FID↓ | Face Div.↑ |\n| ---------- | ---------- | ----- | ------ | ---------- |\n| PhotoMaker | 8.69iter/s | 0.596 | 147.62 | 0.531      |\n| IPAdapter  | 6.08iter/s | 0.571 | 139.33 | 0.536      |\n| InstantID  | 4.66iter/s | 0.592 | 152.28 | 0.502      |\n| PuLID      | 5.85iter/s | 0.619 | 89.80  | 0.527      |\n\n*Speed* is tested on an A100 GPU in the ComfyUI environment at a resolution of 1024x1024. Our method, PuLID, and IPAdapter have similar speeds, while InstantID is slower due to its inclusion of a ControlNet. PhotoMaker is the fastest because it uses LoRA in attention layers, rather than new to_k and to_v layers like adapter-based methods, thus facilitating faster inference speeds by fusing LoRA before testing.\n\n*FGIS* measures the similarity of DINO embeddings in the facial region. While our method outperforms other methods on this metric, we would also like to emphasize that the embeddings extracted by the backbone trained on face recognition are more aligned with human perception and are widely recognized.\n\n*FID* measures the distance between the distributions of generated images with and without ID insertion. Our method significantly outperforms others on this metric, indicating that our method causes less disruption to the base model after ID insertion.\n\n*Face Div.* measures the diversity of generated images by calculating the LPIPS scores between pairs of faces in the generated images. Our method is comparable to PhotoMaker and IPAdapter in this regard. However, it's worthy to note that we achieved such diversity under the condition of much higher face similarity.\n\nPlease note that we do not provide the DINO similarity for the full image region as we believe that measuring DINO similarity across the entire image is less accurate and reasonable compared to focusing on the facial region.\n\n\n---\n\n**`Q3:` Reasons for PhotoMaker's Underperformance in the Experimental Results.**\n\nThe observed underperformance of PhotoMaker is anticipated and aligns with findings from concurrent research, namely LCM-Lookahead. We speculate that this may be attributed to the narrow scope of celebrities in PhotoMaker's training dataset, limiting its effectiveness when applied to non-celebrities, as discussed in L41-L43 of the main paper.\n\n---\n\n**`Q4:` Ablation on different fast sampling methods and different steps.**\n\nPlease refer to `GQ2` for detailed comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DSKJfEAjCA", "reviewer": "Reviewer_pgk9", "summary": "In this paper, an ID customization method for text-to-image generation without tuning is studied. In practical applications, the method often encounters the problems of lack of ID fidelity and interference of the original pattern line by inserting ID. To solve this problem, the author proposes a new ID customization method for text-to-image generation without tuning. A large number of experiments, called Pure and Lightning ID customization (PuLID), on multiple benchmark datasets validate the superiority of the proposed PuLID.", "questions": "I am mainly concerned about the universality of the proposed method and theoretical analysis, whether the proposed Lightning T2I branch can be applied to other diffusion models, and how fast and efficient it is in the actual training and reasoning process.", "limitations": "In general, the content and theory of the article are very rich, but there are some limitations. 1) There are few experimental data, which is not well reflected in aspects such as universality and speed. 2) Is it possible to extend this method and replace ID with other things, such as (clothing)?", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well motivated, cleverly designed and clearly illustrated. The experimental results are obviously superior to other methods.", "weaknesses": "The author's work is very good, but I still have three questions I would like to discuss with them.\n\n1. Experimental results show that PuLID has achieved good performance on specific data sets, but the generalization ability of the model on different types and styles of data sets has not been discussed much in this paper. Can the author provide more experimental results on the generalization of the model? In addition, is there further analysis and discussion of possible limitations of the model, such as its performance in handling certain ID features or style transitions?\n\n2. The authors mentioned that in order to solve the two problems mentioned in their method, the final loss calculation is based on the combination of three kinds of losses. What impact do the different proportions of the three kinds of losses have on the experimental results, especially the impact of ID loss?\n\n3.The paper points out that high-quality images can be generated in fewer steps through the Lightning T2I branch, and ID losses can be calculated on this basis. Is there a significant improvement in computing efficiency with this method? Compared to the traditional multi-step diffusion model, what is the training and reasoning time of the Lightning T2I branch in practical applications?\n\n4.The PULID used by the author in the experiment is based on SDXL and the 4-step SDXL-Lightning. Is it possible to conduct experiments based on other diffusion models to judge its universality？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgment and careful reviews. Our responses are as follows:\n\n**`Q1:` Discuss the limitations and generalization ability of the method.**\n\nThe limitations have been discussed in Section 6.1 of the paper. In terms of generalizability of PuLID, the paper has validated its effectiveness on the following grounds:\n\n1. Section 6.5 presents the compatibility of the PuLID model on some widely-used community base models (both accelerated and non-accelerated models), exhibiting the broad applicability of our method.\n\n2. We evaluate our method on two datasets. DivID-120, proposed in this paper, encapsulates diverse skin tones, ages, and genders, while Upsplash-50, proposed by LCM-Lookahead, comprises non-celebrity photos newly uploaded this year, hence unlikely to be a part of the training dataset. These datasets offer a wide range of input types. We will release DivID-120 to facilitate the evaluations for ID customization methods.\n\n3. The qualitative results showcased in Figures 1, 4, 6, 9, 10, 11 demonstrate various style transition results, including 2D, 3D, anime, pixel, line art, cyberpunk, cg, fantasy, cartoon, etc., alongside a spectrum of facial edits. Most of these scenarios were not included in the training prompts, further underscoring our model's generalization abilities.\n\nWe also provide more qualitative results in **Figure 1 of the Rebuttal PDF**.\n\n---\n\n**`Q2:` How do the proportions of the three losses affect experimental results, especially ID loss?**\n\n- *Impact of diffusion loss*: In practice, we keep the weight of the diffusion loss constant (i.e., 1) while adjusting the weights for ID loss and alignment loss. In extreme experiments where we exclude the diffusion loss, the detailed facial fidelity slightly degrades, particularly with regard to eyewear and hairstyles. This is because the ID loss doesn't directly constrain these details, hence the reconstruction loss from the diffusion loss helps in preserving them.\n\n- *Impact of ID Loss and Alignment Loss*: The weights of these two losses determine the trade-off between ID similarity and editability. Increasing the weight of ID loss enhances ID similarity but reduces editability. Conversely, increasing the weight of alignment loss boosts editability but lowers ID similarity.\n\n---\n\n**`Q3:` Is there a significant improvement in computing efficiency with fast sampling method, compared to the traditional multi-step (e.g., 30) diffusion model.**\n\nPlease refer to `GQ1`. When compared to the traditional diffusion model's 30-step inference, using a 4-step fast sampling can make the training process twice as fast; if compared with a 50-step inference, it would be three times faster.\n\n---\n\n**`Q4:` Concerns about the universality of this method.**\n\nAs explained in `GQ1`, our method can be adapted to base models without acceleration plugins, demonstrating the method's generalizability and universality.\n\n---\n\n**`Q5:` Is it possible to extend this method and replace ID with other things, such as (clothing)?**\n\nIt is theoretically feasible; please refer to `GQ3` for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We sincerely appreciate all reviewers for their insightful and valuable feedback. We are delighted that the reviewers find the paper well-written, the method compellingly motivated, the idea innovative, and the experimental results superior and impressive. Below, we address their common concerns.

**`GQ1:` Is the method proposed in the paper heavily reliant on fast sampling techniques? If so, does this imply that the method lacks generalizability, such as being unable to adapt to a model without corresponding fast sampling methods?**

Great question! We want to clarify that the core essence of our paper is to introduce a more accurate ID loss and alignment loss in the T2I training branch to achieve better ID fidelity and editability. Lightning (the fast sampling method) serves as an optional acceleration trick, but it is not indispensable.

Without the fast sampling method, we'd need 30 inference steps with CFG on the T2I branch, compared to the current 4 necessary inference steps without CFG. Due to CUDA memory bottleneck (we exclude the use of gradient checkpointing due to its significant speed penalty), it's not possible to perform backpropagation (BP) of the gradient at all timesteps. However, it remains possible to make optimization viable with strategic techniques. Particularly, for the optimization of ID loss, BP of the gradient happens only for the last few timesteps. For the optimization of alignment loss, a timestep is randomly selected for BP of the gradient. A comparison table below shows the differences in speed and memory consumption between training with and without acceleration.

|     | BP last 1 timestep | BP last 2 timestep | BP last 3 timestep | BP last 4 timestep | BP last 20 timestep |
| -------- | ------- | ------- | ------- | ------- | ------- |
| w/ (4 steps) fast sampling | 2.6s/iter(41GB)    | 2.9s/iter(49GB)    | 3.1s/iter(56GB)    | 3.3s/iter(63GB)    | -    |
| w/o fast sampling   | 6.6s/iter(50GB)    | 7.0s/iter(65GB)    | 7.3s/iter(80GB)    | OOM    | OOM    |

From the table above, we see that if we do not use fast sampling and take SDXL-base as the base model for the T2I training branch, efficiency will indeed be much lower. However, thanks to the carefully designed optimization strategies mentioned above, the training method presented in this paper can be effectively adapted to non-accelerated models, with performance being further improved, shown in the next table.

|               | \|-------- | DivID-120 | --------\| | \|--------  | Unsplash-50 | --------\| |
| ------------- | --------- | --------- | --------- | --------- | -------- | -------- |
|               | Face Sim. | CLIP-T    | CLIP-I    | Face Sim. | CLIP-T   | CLIP-I   |
| w/ (4 steps) fast sampling  | 0.733     | 31.31     | 0.812     | 0.659     | 32.16    | 0.840    |
| w/o fast sampling | 0.743     | 31.75     | 0.842     | 0.687     | 32.58    | 0.865    |

Additional visual results can be found in **Figure 1 of the Rebuttal PDF**. In conclusion, our method does not rely on accelerated base models, thus reflecting the universality of our approach. We also have plans to adapt PuLID to stronger base models and make them open-source in the future, such as FLUX.1, which was recently released.

---

**`GQ2:` Have the authors explored alternative fast sampling methods beyond SDXL-Lightning? Additionally, how effective is the method trained with a differing number of steps (e.g., 1, 2, 8)?**

Firstly, our selection criteria for a fast sampling method is that it can generate high-quality images within a limited number of steps while well preserving the style and layout of the original model. SDXL-Lightning fulfills these requirements and was the SOTA at that time, so we chose it to speed up our T2I training branch. Secondly, we select 4 steps as it balances efficiency and quality. With fewer steps (e.g., 1 or 2), the likelihood that SDXL-Lightning generates flawed faces increases. However, we have noted a recently emerged fast sampling method, Hyper-SD, which performs better than SDXL-Lightning on 1 and 2 steps. Therefore, we present the results of training on 1 step and 2 steps with Hyper-SD, as well as on 8 steps with SDXL-Lightning in the subsequent table.

|               |        |           | \|-------- | DivID-120 | --------\| | \|--------  | Unsplash-50 | --------\| |
| ------------- | ------ | --------- | --------- | --------- | --------- | --------- | -------- | -------- |
|               | Memory | Speed     | Face Sim. | CLIP-T    | CLIP-I    | Face Sim. | CLIP-T   | CLIP-I   |
| Hyper-SD T=1  | 41GB   | 2.2s/iter | 0.694     | 31.91     | 0.819     | 0.632     | 31.89    | 0.857    |
| Hyper-SD T=2  | 49GB   | 2.5s/iter | 0.720     | 32.08     | 0.810     | 0.653     | 32.35    | 0.840    |
| Lightning T=4 | 63GB   | 3.3s/iter | 0.733     | 31.31     | 0.812     | 0.659     | 32.16    | 0.840    |
| Lightning T=8 | 77GB   | 4.1s/iter | 0.734     | 31.66     | 0.818     | 0.668     | 32.19    | 0.850    |

As shown in this table, training with 1 or 2 steps leads to a reduction in face similarity. Meanwhile, training with 8 steps slightly enhances overall performance. Considering both efficiency and performance, 4 steps is a sound choice.


---

**`GQ3:` Can the training method be extended to IP customization task (e.g., clothing, dog, cat)**

Yes, since both IP and ID can be regarded as subjects to some extent, our training method can be adapted to the IP customization task with minor changes. Specifically, the ID loss can be replaced by an objective that measures the similarity between IPs, like CLIP image similarity. The alignment loss does not need to change, but the list of training prompts should be customized for the specific task. Our preliminary experiments indicate that our training method can effectively enhance the prompt-following abilities for IP customization tasks. We will leave this to be explored in future work.

Meta-review: This paper studies tuning-free ID customization for T2I generation. Reviewers gave accept, accept, strong accept, and accept ratings. The ratings are overwhelmingly positive. Reviewers are happy with the good presentation, novelty, performance, and many experiments. The AC recommends accepting this paper as a spotlight presentation.
Decision: Accept (poster)