Foundation Cures Personalization: Improving Personalized Models’ Prompt Consistency via Hidden Foundation Knowledge
Authors: Yiyang Cai
Zhengkai Jiang
Yulong Liu
Chunyang Jiang
Wei Xue
Yike Guo
Wenhan Luo
Venue: NeurIPS 2025 poster ()
Keywords: Image Generation
Persoanalization
Primary area: deep_learning
TL;DR: Facial personalization models fail to control facial attributes accurately. FreeCure proposed in this paper fixes this problem with a training-free framework without harming these models' impressive ability in maintaining identity information.

Abstract: Facial personalization faces challenges to maintain identity fidelity without disrupting the foundation model's prompt consistency. The mainstream personalization models employ identity embedding to integrate identity information within the attention mechanisms. However, our preliminary findings reveal that identity embeddings compromise the effectiveness of other tokens in the prompt, thereby limiting high prompt consistency and attribute-level controllability. Moreover, by deactivating identity embedding, personalization models still demonstrate the underlying foundation models' ability to control facial attributes precisely. It suggests that such foundation models' knowledge can be leveraged to cure the ill-aligned prompt consistency of personalization models. Building upon these insights, we propose FreeCure, a framework that improves the prompt consistency of personalization models with their latent foundation models' knowledge. First, by setting a dual inference paradigm with/without identity embedding, we identify attributes (e.g., hair, accessories, etc.) for enhancements. Second, we introduce a novel foundation-aware self-attention module, coupled with an inversion-based process to bring well-aligned attribute information to the personalization process. Our approach is training-free, and can effectively enhance a wide array of facial attributes; and it can be seamlessly integrated into existing popular personalization models based on both Stable Diffusion and FLUX. FreeCure has consistently shown significant improvements in prompt consistency across these facial personalization models while maintaining the integrity of their original identity fidelity.

Foundation Cures Personalization: Improving Personalized Models' Prompt Consistency via Hidden Foundation Knowledge
Yiyang Cai', Zhengkai Jiang", Yulong Liu', Chunyang Jiang'
Wei Xue Yike Guo", Wenhan Luo
Hong Kong University of Science and Technology (HKUST)
Tencent Hunyuan
https://yiyangcall git.ugitthhu.... www.:eeccreeeeeeeeeeeeeeeee wiwpeeurrrrria...e... Aaggc.iithbb io/
Abstract
Facial personalization faces challenges to maintain identity fidelity without disrupt- ing the foundation model's prompt consistency. The mainstream personalization models employ identity embedding to integrate identity information within the attention mechanisms. However, our preliminary findings reveal that identity em- beddings compromise the effectiveness of other tokens in the prompt, thereby limiting high prompt consistency and attribute-level controllability. Moreover, by deactivating identity embedding, personalization models still demonstrate the underlying foundation models' ability to control facial attributes precisely. suggests that such foundation models" knowledge can be leveraged to cure the ill-aligned prompt consistency of personalization models. Building upon these insights, we propose FreeCure, framework that improves the prompt consistency personalization models with their latent foundation models' knowledge. First, setting a dual inference paradigm with/without identity embedding, attributes (e.g., hair, accessories, etc.) for enhancements. Second, we introduce novel foundation-awee
identify
self-attention module, coupled with an inversion- based process to bring well-aligned attribute information to the personalization process. Our approach is training-free, and can effectively enhance wide array of facial attributes: and it can be seamlessly integrated into existing popular personalization models based both Stable Diffusion and FLUX. FreeCure has consistently shown significant improvements in prompt consistency across these facial personalization models while maintaining the integrity of their original identity fidelity.
1 Introduction
Human face-centric personalization represents compelling downstream applications of art creation, advertising, and entertainment in the realm of text-to-image synthesis [23 50) Given a limited
number of images that depict particular identities, facial personalization models generate novel content that reflects these identities through diverse conditions [14] 46, 69 38]. However, this aspiration hindered by a persistent challenge: the necessity to maintain high fidelity to the identity while ensuring the controllability of the generated content, also referred to as prompt consistency [72] 26) This challenge is pronounced in facial personalization since any imperfections or misalignment in the generated faces are particularly salient to humans. Therefore, compared to common object personalization, human face-centric personalization mandates dedicated attention and research efforts.
"Correpponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pul ID) (NoudiS 2024)
PhonsMaker (CVP CreeCrne
Face Face-DiTise (CVPR 2024) -raoCrrn
FracDDillaiin (CVPR, 2024) FreCuee
Outputs with
Outputs w/o ID
FieeCure
lduliity
Radlity!
boods haif
with ongly loaking
with Frowiing wornedy
syss
Pronpt
Outputs with FreeCure
Figure I: Personalization models (a) demonstrate strong capability in preserving identity fidelity, albeit the cost of reduced prompt consistency. A prevalent feature in most personalization models is that when their identity embedding inputs are deactivated, they regain the ability to exhibit highly accurate prompt consistency with respect to facial attributes (b), a characteristic closely aligned to their foundation models. Our proposed FreeCure effectively leverages the latent foundational knowledge inherent in personalized models, enhancing prompt consistency in scenarios involving complex facial attribute control while preserving the identity fidelity (c).
Previous research of facial personalization aims to integrate ddentity-speciffi information into the cross-attention mechanisms, with either fine-tuning based strategy [46 14] or tuning-free paradigm
with an identity encoder [64] [34] 18 (59 (7). However, the dual objectives of maintaining prompt
consistency and identity fidelity remain unresolved. Despite the aforementioned advancements, state-of-th therthr-rt personalization techniques still struggle to enhance identity fidelity without sacrificing
prompt consistency.
Our preliminary experiments indicate that, under identical experimental conditions (e.g.. prompts and initial random noise), personalization models' outputs exhibit a significant decline in prompt consistency compared to generated results without identity embedding. For instance, as shown the first two examples in [ig personalization models fail to generate "blonde curly hair" and "angry" faces accurately, whereas their counterparts without identity embedding can handle these prompts in a highly faithful manner. Plus, when handling more complex prompts that consist of multiple attributes, the personalization model generates even poorer results. These findings show a fact overlooked by most previous works: while personalization models show their degradation in prompt consistency, the ability of their original foundation models is unharmed but overridden. Based on this observation, we further find that identity embeddings can significantly undermine prompt consistency by impeding normal representation of other attribute-related tokens within the prompt through cross-attent atttentttn mechanisms. This, in turn, adversely affects their effective expression in the latent space of the U-Net. However, given the compelling zero-shot identity extraction capability of identity embedding. directly modifying personalization models' well-trained cross-attentics modules can destroy their ability to capture precise identity information. Therefore, our core pursuit is clear: Is it feasible to mitigate the erosion of prompt consistency in personalization models while keeping their trained cross-attention modules unaffected?
Motivated by this objective, we propose FreeCure, a framework enhancing the prompt consistency of personalization models through the guidance of their latent foundation knowledge. While keeping cross-attention modules intact, we propose a novel foundation-aue attributes with high prompt consistency to replace those that are ill-aligned during personalization generation. To protect the identity unharmed, this strategy also leverages semantic segmentation models to generate the scaling masks of these attributes, therefore making such replacement happen highly localized and harmonious manner. Furthermore, we use a simple but effective approach called asymmetric prompt guidance (APG) to restore abstract attributes such as expression. Through comprehensive experiments, FreeCure has been verified to effectively restore a wide variety of misaligned attributes produced by various state-of-the-a personalization models.
self-attention (FASA), enabling

---

In summary, the contributions of our paper are threefold:
We identify the limitations in prompt consisteney prevalent across existing face-centric personalization models. Building upon this, we explore the negative effects of identity embedding and elucidate the fundamental reasons for its adverse impacts.
We propose training-free free framework that leverages high prompt consistency information from foundation models to enhance multiple weakened attributes generated by personaliza- tion models. The enhancement of various attributes is achieved without mutual interference.
Our framework can be seamlessly integrated into widely used personalization models and diverse foundational models, including Stable Diffusion and FLUX. Comprehensive experiments show that our approach improves prompt consistency while maintaining the well-trained ability for identity preservation.
Related Work
Phentity-preservingg Generation. Sentity-preserving generation can be broadly categorized into
fine-tuning- based and encoder-based approaches. Fine-tuning- trattunnnn---eedd methods [14] 57 19 73
67 40 either optimize vector within textual embedding to encode identity, modify specific weights
47 [52] or use LoRA A techniques [24]. However, these methods require training a distinct model for each identity, which compromises scalability and increases susceptibility to overfitting. In contrast, encoder-based methods utilize large-scale pretraining to automatically derive identity representations aligning with textual embeddings, enabling zero-shot
in the model [46] I 7 25 4
17
only train a mapping network or modify cross-attention weights to embed identity information, while others
personalization with novel identity references. Some methods |63 9
64
49
62
34
43
665
15, 60, 36 59 56 33 71 10] incorporate cross-altention adapters during training. Recent
[3] to reach more impressive performance. In this work, we focus primarily encoder- based methods, they represent the state- of-the-art paradigm for personalization and demonstrate particular efficacy in facial performance.
works 18 27 also leverages Diffusion Transformers (DiT) [[
42
Attention in Diffusion Models. Attention mechanisms serve as foundational components in text- to-image diffusion models. Prior research 20 16
41 has leveraged semantic information from cross aatss-attention maps to facilitate object-level editing, while other methods
74
30
55
G
35 29
37
53 51 58 have employed spatial features from self-attention layers to achieve more precise modifications or style transfer. However, the positional information derived from attention maps are inherently constrained to object-level manipulation, making them less robust for fine-gra fine-grained facial attribute generation. Plus, the function of face-centric embeddings within attention mechanisms remains insufficiently explored, especially in facial personalization models.
(54 39 S 61 21
II
3 Revisit ID Embedding in Personalization
We conduct comprehensive analysis to justify the limitations of current personalization methods in maintaining prompt consistency. This investigation highlights the challenges inherent in existing approaches provides a foundational basis and critical insights for our proposed methodology.
3.1 Dual Denoising to Study Prompt Consistency
To elucidate current personalization methods' limitations, devise comparative experiment an initial exploration. We maintain a fixed noisy latent code 27 and implement two parallel denoising procedures [5]. The only difference is that one denoising procedure incorporates textual embeddings e and identity embedding Gid. while the other set Cid into a zero tensor Cia For clarity, we will refer the denoising procedure with identity embedding (personalization denoising) as PD, and the denoising process that excludes identity embedding (foundation denoising) as FD hereafter:
PD 6p = c0 : 4 Cia): ; FD ef = 6a(2e, 1, Ga)
where Ep and e denote the predicted noise from PD and FD, respectively. We have conducted these experiments using two facial personalization methods 64 34 and results are presented in Column

---

0.4 0.6 0.8
Cromdiicaiies
Osspets
prompt: "a <man> image with black curly hair, loughing happily"
Figure 2: Analysis on cross- tross-attention maps of facial personalization models. Left: token-wise attention map visualization. Right: interpolation experiment on PD and FD's cross-attention maps.
(a) of Fig 2 It is evident that the output from PD exhibits reduced ability to match the specific facial attributes compared to the counterpart without identity embedding. For instance: 1) The hair features do not align with the prompt's specified "black curly" as expected; 2) The expression "laughing" is also restricted. Conversely, the results from FD show greater prompt consistency. These findings indicate that current identity embedding methods may not effectively address "the balance between identity fidelity and prompt consistency".
3.2 ID Embedding's Effect on Attention Layers
OBSERVATION 1: ID embeddings disrupt cross-attention layers. To investigate the underlying causes of this phenomenon, we conduct a visualization experiment inspired by 220 2 [54, because identity embedding is primarily fused into cross-attention layers. Columns (b-d) of Fig left part present the visualization results of cross-attention maps for both PD and FD processes, focusing on identity embeddings and the serribtte-seccffie token embeddings for "black curly hair" and "laughing". reveals that the identity embedding significantly amplifies activation in facial regions while simultaneously reducing activation for other tokens and disrupting their typical interactions within the cross-attention mechanism. Additionally, we observe that during the FD process, prompt consistency remains preserved. This observation underscores an insight: although most personalization models are well fine-tuned, their intrinsic capability to generate faces with high prompt consistency still preserved but unexpectedly overridden by external identity embeddings. Notably, this latent capability can be effectively activated via the FD procedure, as its name "Foundation Denoising" indicates.
OBSERVATION 2: Personalized cross-attention layers are highly susceptible. We conduct another experiment on cross- attssnttttntoon layers, which involves incorporating a portion of the cross- attention maps from the FD process into those of the PD process. The formula for this approach is
AP kim <- Softmar(a S.n + AP..))
where A represents the cross-attention map of PD's identity embedding, and Af represents the cross-attention map of the FD's zero-valued embedding. The parameter a controls the weight of the FD's cross-attention map that is injected. The results are shown on the right part of Fig It observed that, as the weight of the map from the FD process increases, the model quickly loses identity fidelity, even though it regains the facial attributes that were missing before. This experiment demonstrates that cross- cooss-tttention layers are rather susceptible. To preserve the models' capacity for identity preservation, it is better to leave these well-trained cross- cross-attention modules unaltered.
Our preliminary findings demonstrate that identity embeddings cross- cross-attention layers are disrupting personalization models' prompt consistency. In contrast, its strong ability of identity extraction makes rather challenging be modified. Given this "dilemma", we aim to explore a new approach from the perspective of self-attention, inspired by [16] [2]. Since most personalization models introduce minimal modifications to self-attention layers, it is reasonable to assume that the aforementioned hidden foundation knowledge is preserved within them. By enhancing the self-attention layers in personalization models while keeping cross-attention layers intact, we anticipate achieving better alignment of facial attributes in personalized generation.

---

Useeti deniity urbaddig
min lough happly
porpl
Trained
Trainea Partonal Eustiii
Horld
Hodid
[ML
cttribute razforation with gadayle (4850)
senglaseess lnigting hipppity
X
Sromet
Mantiry
Parenaltaatinn
celf-atherrion
62 dall infereece Sratch auertt unbedt
Ts lecrized attriberi restorstion with moddes
Poundde oith cel--eerre Sel rentien
presoned FAS4
Figure 3: Overview of FreeCure. For a personalization model cp. we first introduce (a): dual inference paradigm to generate faces with/without identity (Tp and Iz), where 1 without identity embedding shows better prompt consistency. Next, we leverage a segmentation model VP(() to derive related masks of target attributes with clear spatial information (hair, sunglasses, etc.) and merge them into a mask M. In (b): we modify the original self-attention modules with our proposed FASA (c), which concatenates key and value matrices of FD process and PD process, together with scaling mask to achieve the attribute injection. Finally, we utilize simple yet effective strategy (d): asymmetric prompt guidance (APG) restore abstract attributes (e.g., expressions).
Methodology
Building upon our investigation in Sec we propose FreeCure, a training- free framework designed improve the prompt consistency of facial personalization models (see Fig Initially, we develop a foundation-awart self-attention (FASA) mechanism to integrate localized attributes from the FD process into the PD process (Sec 4.1). Subsequently. we apply an asymmetric prompt guidance (APG) strategy to reconstruct more abstract attributes (Sec 4.2)
4.1 Foundation-Awarg Self-Attention
Given a reference image Tref that provides the target identity, a well-trained personalization model co, user-defined prompt that contains sequence of facial attributes A = [A1, A2, An). our goal is to employ knowledge FD to enhance prompt consistency PD's output.
Obtain Masks of Spatial Localized Attributes. As shown in 3 (a), we adapt the dual infer- ence branches which are identical to the FD/PD process introduced in See for personalization models to generate the personalized face I, with unsatisfactory prompt consistency as well the foundation face I with high prompt consistency. Next, we utilize an external face parsing model, denoted as (-), to extract the binary mask M; corresponding to A, from the foundation outputs Iy. Generally, when restoring multiple attributes, we simply need to compute the different masks respectively and merge them M U.). M contains the spatial information of attributes that align with the target prompt, which will play important role in the attribute restoration process that will be mentioned in the next part.
117
Cwick diace
Figure 4: Fine-grained attribute enhancement via masks. Extracting masks from the FD re- sults makes the FASA module only focus enhancement for target attributes, minimizing its negative effect on identity fidelity.
FASA Mechanism. FASA the core module that links the information between FD and PD processes and enables personalization models to restore the attributes via their foundation knowledge. As shown Fig Cc), we at first identify the self-attention modules the model. Specifically, in each timestep and attention layer I, we denote PD and FD's key, query, and value matrices as XQV!! ( Qi, Vill and KQV (K] Q² v Second, we concatenate the key and
5

---

value matrices of FD process to those of PD process: K2 [Kp, Kj), Vu = [V, VjJ. Thus, by omitting the labels of and I for simplicity, the operation of FASA is
P. = Softmax Opk T )
(3)
Fine-grained Restoration with Scaling Masks. The approach described in EqBlallows the PD process to obtain information from the FD process. However, we observe that the method also retains a substantial degree of unrelated features, leading to a notable loss of identity fidelity, as shown in 4 To constrain attribute restoration in a specific region without disrupting identity information, we apply pre-calculated masks M of different attributes to the similarity map. Additionally, to further enhance FASA's performance, we introduce additional scaling factor, denoted wi to control the magnitude of injecting attribute features from FD to PD. Therefore, the enhanced FASA mechanism can be written as
FAS, Softmax [ W.MM ① Q )
Vd
Where denotes a matrix with all elements equal to I, designed to preserve attention to the original features within the PD process. @ represents the Hadamard product.
In the full-attention layers of Diffusion Transformers (DiT) such as FLUX, visual and textual information is integrated into unified sequence representation [X; C?. Within this architecture, the FASA mechanism operates in similar manner, with a key distinction: the attribute mask is applied solely to the component derived from the visual queries of the PD branch (Q2) ) and the visual keys
of the FD branch (K²). This selective masking strategy preserves the original cross-modal attention patterns between visual and textual elements, resembling OminiControl [51]:
FASA flur (KQV,, ( Soottmx M(w) flur O OpKT V,
(5)
Vd 11xl2 1igx12
1lxch 11221
wMixx xli 01gx
M(w) flux
(6)
Where Op = =p 10 Q is the original query matrix of PD branch, and K = I KG K
[V, ve, V are FASA's enhanced key and value matrices which integrates information from both PD and FD branches. With such enhancement, attributes of the FD process can be successfully extracted and precisely injected into the PD process without disrupting the personalization model's ability of maintaining identity fidelity.
4.2 Asymmetric Prompt Guidance
Following attributes with clear location restored via FASA, enhance more abstract attributes such as expressions, we introduce a simple yet effective method called asymmetric prompt guidance (APG). This strategy is based on the diffusion model's inversion [50]. During the inversion phase, the model accepts only a template prompt (e.g. "a man") that does not include any target attributes. In the denoising process, add the target attributes' tokens back to this template prompt (e.g. "a man" " model, this
- "a man laughing "). By leveraging the pretrained controllability of the approach enhances such attributes, resulting in the final refined image Lout- Throughout the denoising process, we use only pure textual prompts without identity embeddings, thereby avoiding their potential influence on the tokens related to the target attributes, as discussed in See Furthermore, to better preserve the identity, we start the denoising process directly from an intermediate latent code 2Gr.7 7 E [0, 1), where the ligh-level identity information has already been established.
5 Experiments
Evaluation Datasets. We collect an extensive dataset including 50 identities, with 30 derived from the CelebA-HQ [32 and the other 20 non-celebrity identities curated by our team. Each identity is represented by single image, with spectrum of facial characteristics. The prompt set consists of 20 prompts containing different facial attributes. For each (identity, prompt) pair, we produce 20 images. Detailed information can be found in Appendix A.2

---

Baselines. We evaluate FreeCure using several representative facial personalization methods: FastComposer [64]. Face-diffuser 622, Face2Diffusion 49 InstantID [9 PhotoMaker [44.
Pul.ID 18] and Infinite You 277.
Among these, Face-diffuser, Face2Diffusion, and FastComposer
are implemented using SDv1-5 [45]. whereas InstantID, PhotoMaker, and PuL.I are based on
SD-XLL 44. PuLID and Infinite You employ FLUX. --vv [ as their foundation models.
Evaluation Metries. We adopt CLIP-T [22] calculate prompt consistency (PC). To calculate identity fidelity (IF), we use MTCNN
68 and FaceNet 48 to extract the embedding of the
generated/reffrrnce faces and compute the cosine similarity. Following PhotoMaker, we adopt the face diversity (Face Div.) metric which calculates LPIPS [70] between facial areas. Lastly, following Face2Diffusion, we adopt PC x IF score, since reflects the overall balance of prompt consistency and identity fidelity. We compute the harmonic mean (hMean) of PC and IF.
in FASA 2.0, to ensure that attribute information from FD can be sufficiently integrated into PD. The ry in APG is set to 0.5 to maintain the balance between identity fidelity and prompt consistency. For attribute segmentation, we leverage BiSeNet [66] and Segment-Anything (28) for different facial attributes.
FreeCure Settings. We
5.1
Main Results
Table 1: Main quantitative evaluation results. With FreeCure, the mainstream personalization
models' prompt consistency highly enhanced on critical quantitative metrics.
Method
PC(%)
IF(%) f
Face Div. (%)
PC x IF (hMean) f
FastComposer
18.14
43.19
38.92
25.55
FastComposer + FreeCure
21.02 (+15.91%)
41.02 (-5.02%)
41.01(+5.37%)
27.80 (+8.82%)
Face-Diffuser
20.67
58.34
40.82
30.52
Face-Diffuser FreeCure
22.48 (+8.76%)
57.51 (-1.42%)
41.95(+2.77%)
32.32 (+5.90%)
FiffuDiff
21.92
39.98
43.51
28.31
Face2Diffusion FreeCure
23.26 (46.12%)
39.23 (-1.88%)
44.29(+1.79%)
29.20 (+3.15%)
InstandD
21.89
63.94
48.98
32.61
InstantID + FreeCure
23.62 (+7.90%)
62.01(-3.02%)
51.82 (+5.80%)
34.21 (+4.91%%
PhotoMaker
23.04
51.84
47.29
31.90
PhotoMaker FreeCure
24.91 (+8.11%)
50.15 (-3.26%)
48.52 (+2.60%)
33.28 (+4.34%)
PuLID (SDXL) PuLID (SDXL) FreeCure
25.16
58.23
42.12
35.14
26.05 (+3.55%)
56.95 (-2.20%)
43.72 (+3.80%)
35.74 (+1.74%)
PuLID (FLUX) PuLID (FLUX) + FreeCure
22.42
74.97
43.91
34.52
24.78 (+10.53%)
72.61 (-3.15%)
46.09 (+4.96%)
36.95 (+7.04%)
InfiniteYou Infinite + FreeCure
23.77
79.71
44.28
36.62
25.25 (+6.23%)
77.13 (-3.24%)
46.82 (+5. 745.74%)
38.05 (43.90%)
Overall Performance Comparison. Table I and Fig. 5 6 present a comparative analysis of our proposed method with the baselines, examining both quantitative and qualitative aspects. It is easy to notice that baselines often fail to accurately reflect key facial attributes mentioned in the prompts. For instance, these baselines often generate faces with identical expression (row 2, 4 of Fig and row 2 of Fig 6) For attributes with large areas (e.g., hair, sunglasses), baselines often cannot generate them harmoniously (row 3, and of Fig 5 and row 1 of 6). Conversely, our approach shows a remarkable ability to enhance absent or faint attributes, significantly improving prompt consistency for these basclines. FreeCure can even tackle subtle facial attributes (c.g., eye color and earrings, see row 5, 6 of Fig 5). Notably, FreeCure achieves significant performance improvements across all foundation models' personalization methods, demonstrating its strong generalizability. Secondly, we notice that FreeCure leads to a slight decline in identity fidelity, which can be attributed to positively improved facial diversity. Since baselines tend to produce faces closely resembling their references, they inherently score higher in identity fidelity. Ultimately, in terms of the PC x IF metric, our method also shows considerable improvement over all baselines. conclusion, both quantitatively and qualitatively, FreeCure demonstrates positive balance by enhancing prompt consistency while keeping the reduction identity fidelity to minimum. Additional comparisons and results more references (including non-celebrities) are available in Appendix. A.3.I We also
7

---

FastComposer
Face-Difluser (CVPR 2024)
Faev2Diflosion
Instand (2024)
PhotoMaker (CVPR
Put (
Reference Prompt
(UCC 2024)
(CVPR
(NeurIPS 2024)
ongiral
FreeCurz
FreeCum
ongirnl
PreeCarm
original
PrexCarc
original
FreeCare
original
ForeClur
noa witt
PURK
Figure 5: Qualitative comparison with facial personalization baselines (including baselines built upon SDv1.5 and SDXL). Different attributes prompts are highlighted various colors. Comparison of corresponding FD outputs is provided in the Appendix A.3.2
Reference Prompt
(NeueIPS
PuLID (FLUX)
InfiniteYou
Reference
Prompt
PuLID (FLUX) (NeurIPS
InfiniteYon ou (ICCV 2025)
Orginll
FreCrre
Ociginal
PraeCume
Origiaal
PereCuune
Orisinal
mon 100 cuty
worth with BETSIT LooiT
Long acrrady
Figure 6: Qualitative comparison with FLUX-based facial personalization baselines. Different attributes in prompts are highlighted in various colors.
provide corresponding FD outputs for Fig to show the effectiveness of FreeCure to transfer correct attributes from the FD to PD process in Appendix, A.3.2
Prompt Consistency with Multiple Attributes. Prompts involving multiple facial attributes pose greater challenge personalization' 's prompt consistency but better reflect practical user needs. Table 2 illustrates FreeCure's improvements on baselines with prompts including multiple facial attributes. As the number of attributes increases, the PC values of baselines tend to decrease. In contrast, FreeCure's 's improvement in PC becomes more significant as the complexity of the prompt increases. More analysis of FreeCure's non-disruption manner multiple prompt personalization available in Appendix[A.44 1 In summary, through integration with FreeCure, personalization models can effectively address more complex and realistic prompt instructions.
5.2 Robustness Justification
Robust Performance with Different Initial Noises. We observe that, with identical initial noise, all baselines' FD and PD processes generate faces with similar attribute locations. It is important to validate that FASA's robust performance without these condition. Thus, we relax this condition and regenerate faces with different initial noises. shows that results under the settings comparable, which confirms that FASA can effectively enhance the generated results of PD, even when its FD counterpart produces faces with variable spatial structures.
Visualization of FASA maps. Fig visualizes the FASA map A E Hxxx2) whose size is doubled according to Eq 4 For a given query point gi pp- its corresponding scores At: are extracted from both K, and Ky. When 9; falls into associated with target attributes (e.g..

---

Table 2: Quantitative comparison of prompt consistency with different number attributes. After calculating the metrics for each method, we compute their mean values based on the corre-
sponding foundation model type. For metrics of each baseline, please refer to Appendix A.3.3
Foundation Model
PC (1 Attr.) t
PC (2 Attr.) t
PC (3 Attr.)
SDv1.5 SDv1.5 + FreeCure
21.01 22.70 (+8.04%)
20.34 22.34 (+9.87%)
18.49 21.16 (+14,44%)
SDXL
23,83
23.31
22.65
SDXL FreeCure
25.11 (+5.34%)
24.80 (+6.36%)
24.49 (+8.14%)
FLUX FLUX FreeCure
24.15
22.64
21.88
25.75 (+6.63%)
24.71 (49.15%)
24.16 (+10.45%)
bitck uuiy lar
Blook
Refonoo
DJ Phet surpare with identia mnttal netec
(5) Timal are ouppats with diffonent tnttal noine
Figure 7: Performance of FASA w/ and w/o identical initial noises. FASA can precisely enhance attributes even if PD and FD produce faces with different locations, sizes, and angles.
Q
x
K,
N,
Q.
K,
Figure 8: Visualization of the FASA maps for attribute related area (red points) and non- attribute related area (green points).
hair, sunglasses), the FASA map exhibits greater attention to information from Ky. corresponding to FD. Conversely, in regions unrelated to attributes, the FASA map retains its attention on PD. This visualization substantiates the role of FASA in transferring fine-graing ane-grained attribute information from FD to PD. Additionally, FASA preserves its original attention pattern in regions unrelated to attributes, thereby ensuring the performance of personalization models' identity fidelity. More detailed visualization analyses of FASA are available in Appendix A.4.6
5.3 Ablation Study
Overall Analysis of Each Component. Table presents the individual performance of FASA and APG in enhancing attributes across all baselines. Both components demonstrate positive effects compared to the baseline metrics in Ta with FASA showing more noticeable improvements. We attribute this FASA's ability to effectively handle attributes covering larger areas, such as hair and sunglasses, resulting in more observable enhancements.
penirnaloo cockee
pote aill codie
(a) Ablation of FASA scaling mask strategy.
(b) Ablation of APG intermediate timesteps.
Figure 9: Ablation Studies.
Scaling Mask Strategy. Figure PaureDDallln Paliltustatess the evaluation of the scaling mask strategy implemented in FASA. Low scaling values critically hinder FASA's capacity to effectively transfer attribute information to the personalization denoising. On the contrary, high scaling factors may negatively impact the overall quality of the generated faces. Generally, the optimal value of should be around 2.0. More ablation studies of FASA are available Appendix
Effect of Inversion's Intermediate Timesteps. Figure 9b demonstrates the effects of different starting timesteps APG, represented by the parameter y in Section 4.2 To achieve noticeable
improvement in attributes while maintaining identity fidelity, an optimal 7 should be set as 0.5.

---

Table 3: Quantitative ablation analysis for FASA and APG's independent effect. After calculating the metrics for each method, we compute their mean values based on the corresponding foundation model type. For metrics of each baseline, please refer Appendix A.4.4
Foundation Model
FASA
APG
PC( f
IF(%) f
PC x IF (hMean) f
/
x
21.83
46.39
29.69
SDv1.5
X
/
20.58
46.81
28.59
x
24.55
56.97
34.31
SDXL
/
23.97
56.87
33.73
24.50
75.66
37.01
FLUX
24.09
75.85
36.57
Hyperparameter Analysis. Our supplemental ablation analysis indicates that the hyperparameters w and 7 consistently fall within specific ranges across different baselines: w E [1.8, 2.4], [0.5, 0.6]. Baselines based on Stable Diffusion v1.5
49 64 62 require larger values (up to 2.4), while other 2.0. Since results remain stable within these ranges, we adopt unified setting (w = 2.0, 0.5) throughout our experiments.
methods 16 27 59 perform well around
5.4
User Study
To further validate FreeCure's superiority, conducted online user study with 30 participants. The study was designed as follows: for each baseline method, we randomly selected 10 samples. Each sample included reference image, text prompt, the baseline's ' output, and the result refined by FreeCure. Participants were asked to evaluate the prompt consistency and identity fidelity of each output. As shown in Fig TOa and Fig TOb the results demonstrate a clear preference for FreeCure on prompt consistency and equal preference on identity fidelity, indicating that FreeCure's potential negative impact on identity preservation is minimal.
Perference Prompt Consistency
Perference Identity Fidelty
TEATS
Perecitagg (5))
8U.%
Pereeitage (56)
31.2%
TEPN
4EAS
(8.%%
3.2%
$3.4%
161%
44%
12.5%
165%
15.%
16%%
8..%%
Naw Stfle
Anforrie
PastCopuerr
Fiee Dafbeer
NuD mm
(a) User preference on prompt consistency
(b) User preference on identity fidelity
Figure 10: User study of FreeCure. The preference ratio indicate that FreeCure can improve prompt consistency without undermining identity fidelity of different personalization models.
6 Limitation and Conclusion
Limitation. FreeCure is influenced by inherent bias and maximum achievable capabilities of personalization models, which is commonly observed in plug-and-play adaptation frameworks. Furthermore, it struggles with certain transparent objects, such as glass bottles. In some cases, we also observe slight attribute entanglement between FD and PD processes. Furthermore, FreeCure's application to personalization models based on auto-regressive architectures could be further explored.
Facing the challenge that personalization models employing identity embeddings frequently struggle to preserve prompt consistency, we introduce FreeCure, training- free framework that leverages the high prompt consistency inherent in foundation models to refine the output of personalization models, leading to remarkable improvement their prompt consistency and minimal disruption to their original identity fidelity Our experiments validate the effectiveness of FreeCure on popular baselines from different foundation models, particularly in scenarios with complex prompts that encompass multiple attributes.
Conclusion.
10

---

Acknowledgments
The research supported in part by Theme-based Research Scheme (T45-205/21-N) from Hong Kong RGC, Generative Al Research, Development Centre from InnoHK, in part by the National Natural Science Foundation of China (Grant No. 62372480), in part by HKUST-MetaX Joint Lab Fund (No. METAX24EG01-D. We also thank Yubai Wei, Moheng Li, and Meixin Zhou for their assistance with the user study.
References
[1]
2024.
[2] Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch Elor, and Daniel Cohen-Or. Cross- image attention for zero-shot appearance transfer. In ACM SIGGRAPH 2024 Conference Pupers, pages 1-12, 2024.
[3] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text-to-image personalization. ACM Transactions on Graphics (TOG), 42(6))1-10, 2023.
[4] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen Or, Ariel Shamir, and Amit
H. Bermano. aggaiic tuning-encoder for fast personalization of text-to-image models. In SIGGRAPH Asia 2023 Conference Papers, pages 1-10, 2023.
[5] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 18392-18444 18402, 2023.
[6] Mingdeng Cao. Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560- 22570, 2023.
[7] Daewon Chae, Nokyung Park, Jinkyu Kim, and Kimin Lee. Instructbooth: Instruction-follovin personalized text-to-image generation. arXiv preprint arXiv:2312.03011 2023.
[8] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-exte Attention-based semantic guidance for text-to-image diffusion models. ACM transactions on Graphics (TOG), 42(4))1-10, 2023.
[9] Zhuowei Chen, Shancheng Fang, Wei Liu, Qian He, Mengqi Huang, and Zhendong Mao. Dreamidentity: enhanced editability for efficient face-identity preserved image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1281-1289,
2024.
[10] Siying Cui, Jia Guo, Xiang An, Jiankang Deng, Yongle Zhao, Xinyu Wei, and Ziyong Feng. Idadapter: Learning mixed features for tuning-free personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 950-959. 2024.
[11] Ganggui Ding, Canyu Zhao, Wen Wang, Zhen Yang, Zide Liu, Hao Chen, and Chunhua Shen. Freecustom: Tuning-free customized image generation for multi-concept composition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9089-9098, 2024.
[12] Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text- to-image generation via positive-negative prompt-tuning. arXiv preprint 1.X1v:2211.11337,, 2022.
[13] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the s1 International Conference on Machine Learning, ICML:24. JMgR. 2024.

---

[14] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
[15] Rinon Gal, Or Lichter, Elad Richardson, Or Patashmik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. Lem-lookahead for encoder-based text-to-image personalization. In European Conference on Computer Vision, pages 322-340. Springer, 2024.
[16] Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, et al. Photoswap: Personalized subject swapping in images. Advances Neural Information Processing Systems, 36, 2024.
[17] Yuchao Gu, Xintao Wang. Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
(18) Zinan Guo, Yanze Wu, Chen Zhuowei, Lang chen, Peng Zhang, and Qian HE. Pulid: Pure and lightning id customization via contrastive aligmment. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak. and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 36777-36804. Curran Associates, Inc., 2024.
[19] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff:: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
[20] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2023.
[21] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4775-4785, 2024.
[22] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[24] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 2021.
[25] Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, and Qian He. Dreamtuner: Single image is enough for subject-driven generation. arXiv preprint arXiv:2312. 2023.
[26] Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusion model-based image editing: A survey, arXiv preprint 17X1v:2402.17525, 2024.
[27] Liming Jiang, Qing Yan, Yumin Jia, Zichuan Liu, Hao Kang, and Xin Lu. Infiniteyou: Flexible photo recrafting while preserving your identity. arXiv preprint arXiv:2503. 254. 2025.
[28] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEFE/CVE International Conference on Computer Vision, pages 2023.
12

---

[29] Lingjie Kong, Kai Wu, Xiaobin Hu, Wenhui Han, Jinlong Peng, Chengming Xu, Donghao Luo, Jiangning Zhang, Chengjie Wang, and Yanwei Fu. Anymaker: Zero-shot general object customization via decoupled dual-level id injection. arXiv preprint arXiv:2406. /643, 2024.
[30] Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. Omg: Occlusion-friendie personalized multi-concept generation in diffusion models. In European Conference on Computer Vision, 2024.
[31] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1931-1941, June 2023.
[32] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2020.
[33] Xiaoming Li, Xinyu Hou, and Chen Change Loy. When stylegan meets stable diffusion: a w+ adapter for personalized image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2187-2196, 2024.
[34] Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8640-8650, 2024.
[35] Bingyan Liu, Chengyu Wang. Tingfeng Cao, Kui Jia, and Jun Huang. Towards understanding cross and self-attention in stable diffusion for text-guided image editing. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7817-7826, 2024.
[36] Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion Open domain per- sonalized text-to-image generation without test-time fine-tuning. In ACM SIGGRAPH 2024 Conference Papers, pages 1-12, 2024.
[37] Yuhang Ma, Wenting Xu, Jiji Tang, Qinfeng Jin, Rongsheng Zhang, Zeng Zhao, Changjie Fan, and Zhipeng Hu. Character-adapter Prompt-guided region control for high-fidelity character customization. arXiv preprint arXiv:2406. 16537, 2024.
[38] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang. Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 4296 4304, 2024.
[39] Jisu Nam, Heesu Kim, DongJac Lee, Siyoon Jin, Seungryong Kim, and Seunggyu Chang. Dreammatcher: Appearance matching self-attention for semantically-consisee text-to-image personalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8100-8110, 2024.
[40] Lianyu Pang, Jian Yin, Haoran Xie, Qiping Wang, Qing Li, and Xudong Mao. Cross initial- ization for face personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8393-8403, 2024.
[41] Or Patashnik, Rinon Gal, Daniil Ostashev, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen- Or. Nested attention: Semantic-aware attention values for concept personalization. In Proceed- ings of the Special Interest Group Computer Graphics and Interactive Techniques Conference Conference Papers, pages 1-12, 2025.
[42] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205. 2023.
[43] Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji. Portraitbooth: A versatile portrait model for fast personalization. Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pages 27080-27090 2024.
13

---

[44] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprin! arXiv:2307.01952 2023.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[46] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-22510 2023.
[47] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingho Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personaliza- tion of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6527-6536, 2024.
[48] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[49] Kaede Shiohara and Toshihiko Yamasaki. Face2diffusion for fast and editable face personaliza- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6850-6859, 2024.
[50] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02200, 2020.
[51] Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer. arXiv preprint arX.v22411.15098, 2024.
[52] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPII 2023 conference proceedings, pages 1-11, 2023.
[53] Yoad Tewel, Rinon Gal, Dvir Samuel, Yuval Atzmon, Lior Wolf, and Gal Chechik. Add- it: Training-free object insertion in images with pretrained diffusion models. arXiv preprint arXiv:2411.07233 2024.
[54] Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free consistent text-to-image generation. ACM Transactions on Gruphics (TOG), 43(4):1-18, 2024.
[55] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[56] Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. FaceO: Instantaneously conditioning text-to-image model on face. In SIGGRAPH Asia 2023 Conference Papers, pages 1-10, 2023.
[57] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in eext-to-image generation. arXiv preprint arXiv:2303.09552. 2023.
[58] Kuan-Chieh Wang, Daniil Ostashev, Yuwei Fang, Sergey Tulyakov, and Kfir Aberman. Moa: Mixture-of-atteniion for subject-context disentanglement personalized image generation. In SIGGRAPH Asia 2024 Conference Papers, pages 1-12, 2024.
[59] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-reesering generation in seconds. arXiv preprint arXiv:2401.07519, 2024.
14

---

[60] Ye Wang, Xuping Xie, Lanjun Wang. Zili Yi, and Rui Ma. Dp-adapter: Dual- adapter for boosting fidelity and text consistency in customizable human image generation. arXiv preprini arXiv:2502. 13999, 2025.
[61] Yibin Wang, Weizhong Zhang, and Cheng Jin. Magicface: Training-free universal-style human image customized synthesis. arXiv preprint arXiv:2408.07433: 2024.
[62] Yibin Wang. Weizhong Zhang, Jianwei Zheng, and Cheng Jin. High-fidelity person-centric subject-to-imen -injec synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7675-7684, 2024.
[63] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for eustomized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943-
15953, 2023.
[64] Guangxuan Xiao, Tianwei Yin, William Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning- free multi-subject image generation with localized attention. International Journal of Computer Vision, pages 1-20, 2024.
[65] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint (rxXiv2308.06722) 2023.
[66] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 2018
[67] Ge Yuan, Xiaodong Cun, Yong Zhang, Maomao Li, Chenyang Qi, Xintao Wang, Ying Shan, and Huicheng Zheng. Inserting anybody in diffusion models via celeb basis. Advances in Neural Information Processing Systems, 36, 2024.
[68] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23())): 1499 1503, 2016.
[69] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836-3847, 2023.
[70] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
[71] Shilong Zhang, Lianghua Huang, Xi Chen, Yifei Zhang, Zhi-Fan Wu, Yutong Feng, Wei Wang, Yujun Shen, Yu Liu, and Ping Luo. Flashface: Human image personalization with high nigh-fidelity identity preservation. arXiv preprint arXiv:2403.1 17008, 2024.
[72] Xulu Zhang, Xiao-Yong Wei, Wengyu Zhang, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, and Qing Li. A survey on personalized content synthesis with diffusion models. arXiv preprint arXi::2405.05533 2024.
[73] Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Tong- Yee Lee, Oliver Deussen, and Changsheng Xu. Prospect: Prompt spectrum for attribute-aware personalization of diffusion models. ACM Transactions on Graphics (TOG), 42(6))1-14, 2023.
[74] Jing Zhao, Heliang Zheng, Chaoyue Wang, Long Lan, and Wenjing Yang. Magicfusion: Boosting text-to-image generation performance by fusing diffusion models. In Proceedings of the IEEF/CV International Conference on Computer Vision (ICCV), pages 22592-22602
October 2023.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction are supported by results of the experiments. See S
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
FreeCure can be influenced by original models' inherent biases and maximum
achievable capabilities.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
16

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
All assumptions should be clearly stated or referenced
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We report the experiment details and we will release related codes. See Sec
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
17

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our dataset consists of public part and self-collected part. We will release them together with codes. See Sec 5
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
swi.iiossiippcccc.oo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
18

---

Answer: [Yes]
Justification: Our method does not include any training process, and we have specified the hyperparameter of our framework. See Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All evaluation samples are randomly initialized and follows the identical evaluation process.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We run experiments on one H800 GPU (80GB).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
19

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The paper confirms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
20

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper cites the original papers, codes, and datasets which are used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessii thcode www.com//a/oo/oc datasat has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

A Appendix
The Appendix consists of three sections:
A.I includes ethics statement that addresses the privacy of our evaluation datasets and our methods for mitigating potential risks.
provides implementation details, including details of setting up the dual denoising paradigm for every baseline. Details of segmentation models, prompt datasets as well as evaluation metrics are also included.
See A.c
Sec provides additional results of FreeCure in a boarder range of references, including non-celebrities. It also includes complementary FD outputs corresponding to the results of the main submission (Fig 6), and broader attribute enhancement achieved through the integration of advanced models, such as Segment-Anything Model [28]
Sec A.4 provides a comprehensive analysis of FreeCure, including analysis for initial noise conditions, detailed visualization of FASA modules, more detailed ablation studies, runtime analysis and template of the user study.
A.1 Ethics Statements
We confirm that our human face datasets, featuring both celebrities and non-celebrities, are sourced from public datasets or Google Images, limited to CC-licensed content. We have made extensive efforts balance critical attributes like gender, race, and age mitigate significant bias. Furthermore, we acknowledge that FreeCure could be misused for malicious purposes, such as creating imperson- ating identities. To mitigate these risks, we are committed ethical governance and controlled release strategy. For instance, we release the method only to accredited researchers under a license prohibiting misuse, with mandatory ethics training.
A.2 More Implementation Details
A.2.1
Triggering Hidden Foundation Knowledge in Personalization Baselines
The core architecture of FreeCure is based on dual denoising paradigm of the FD and PD process, which triggers hidden foundation knowledge personalization models. This paradigm deactivates the identity embeddings which are integrated into the cross-attention layers. In Table we illustrate how we deactivate identity embedding for different baselines mentioned in the main paper since they have various identity embedding fusion strategies (e.g., LoRA, adapters) and prompt templates.
Table Strategies for implementing dual denoising paradigms on different baselines. "black curly hair" is used here as an example and can be replaced with other facial attributes.
Method
PD inputs
FD inputs
FastComposer
<P> <| </mmage>> with black curly hair
<P> image with black curly hair
Face-Diffuser
<P>> < </image|> >> with black curly hair
<P> image with black curly hair
Face2Diffusion
af 1 with black curly hair
<P> image with black curly hair
InstantID
a <P> with black curly hair
<P> image with black curly hair
PhotoMaker a <P> img with black curly hair a <P> image with black curly hair
PuLID
a <P> with black curly hair
a <P> image with black curly hair
Infinite
a <P> with black curly hair
<P> image with black curly hair
FastComposer (44 Face-Diffuser 62 and PhotoMaker [3]]share the similar prompting template, wherein their identity embeddings are integrated into the "<P>" word, which is initialized with class- specific words (e.g., mun, woman, boy, girl). To establish the FD process, we maintain the original plain text embedding for "<P>" without incorporating any identity information, effectively equivalent to applying a null identity embedding. Face2Diffusion [49] integrated the identity embedding into the trigger word "f". Therefore, replace the identity embedding with zero-valued tensor before its
23

---

Table 5: Information for facial attribute and label relationship of BiSeNet.
label
attribute
label
attribute
label
attribute
background
glasses
12
lower lip
1
facial skin
right ear
13
neck
right eyebrow
left ear
14
necklace
3
left eyebrow
nose
15
cloth
left eye
earrings
16
hair
5 right eye
11
upper lip
17
hat
Table Prompts for evaluation categorized by the number of included aftributes (range from to 3). <S> will be replaced with placeholder tokens such as man, woman, boy, etc.
Attribute
Prompt
<S> with black curly hair
with blonde curly hair
a <S> with red long straight hair
<S> with very angry looking
a <S> with frowning worriedly
a <$> laughing happily
<S> wearing a mask
a <S> wearing sunglasses
a <S> with white curly hair, frowning worriedly a <S> with black curly hair, laughing happily
<S> with blonde curly hair and blue eyes
<S> with blue eyes, laughing happily
2
a <S> wearing sunglasses, laughing happily <S> with black hair, wearing silver earrings a <S> with blonde hair and blue eyes
<S> with sunglasses, frowning worriedly
a <S> with red curly hair, wearing pearl earrings, unhappy looking <S> with blue eyes and blonde curly hair, smiling
3
<S> with white curly hair, wearing sunglasses, laughing happily <S> with black curly hair, wearing silver earrings, frowning worriedly
I and Infinite You (27 that employ residual cross-attention adapters for identity fusion, we initialize their identity embeddings as zero tensors with matching dimensionality while preserving the original textual embeddings unchanged.
merging process. For baselines such InstantID [59], PuLID 18]
A.2.2 Attribute Segmentation Models
As mentioned in Sec 5 we use two mainstream segmentation models: BiSeNet and Segment-Anythin models for attribute-awan aware mask generation. BiSeNet is popular framework for face parsing that capable of generating semantic masks corresponding to facial attributes. T BiSeNet's prediction label and corresponding facial attributes, demonstrating that it can address the majority of facial attributes, thereby verifying the robustness of FreeCure. Additionally, FreeCure is designed to be seamlessly integrated with various segmentation models, not limited to BiSeNet. We have showeased results that utilize
as face parsing model in See A.3.4
A.2.3 Facial Prompts for Evaluation
Table/B introduces the prompts for evaluating enhancement performance for facial attributes. Gener- ally, our prompts include multiple facial attributes, ensuring that previous baselines' weaknesses in prompt consistency can be fully uncovered and highlighting the enhanced performance of FreeCure.
24

---

A.2.4 Details of Metrics
Prompt Consistency (PC, also known as CLIP-T), We leverage the official implementation of the vision transforme provided by OpenAI.
Identity Fidelity (IF). We use the official implementation of MTCNN + FaceNet pipeling to conduct the processes of face detection and feature extraction from facial regions. Additionally, compute the cosine similarity between two face embeddings to evaluate their similarity.
Face Diversity (Face Div.). We utilize the official implementation of LPIPS to quantify the perceptual distance between two facial images.
A.3
More Results
A.3.1 More Results Celebrity & Non-Celebrity References
In 11 12 and 13 we provide additional experimental results more reference images to further validate the consistent performance FreeCure. This robust performance shows the potential of FreeCure to enhance various personalization models in practical applications, as the ability to handle non-celebrity personalization is critical requirement real-world scenarios.
Phups://github.c Phatps//gttuuh.co/oppaaa/// Co..I
"htps::/githuh.comm/// hyweh
25

---

FastComeeser (IJCV 2024)
Face-Difluser (CVPR 2024)
Fice2Diffusion (CVPR 2024)
Reference
Prompt
original
with FreeCure
original
with FreeCure
original
with FreeCure
man with blorde curly hair
man wearing sungl asoes
womsn with Mosde curly heir and peori earrings
man with Mondo curly hair and blue ayas
mon lecoring asnglasseet happy lookiig
weiring sarnings,
unhappy looking
Figure 11: More results for baselines implemented with SDv1.5, including FastComposer, Face- Diffuser and Face2Diffusion. We provide personalization results including non-celebrity identities.
26

---

InstantID (2024)
PhotoMaker (CVPR 2024)
PULID-SDXL (NeurIPS 2024)
Reference
Prompt
original
original
with
original
with
blonde with
frowning worriedly
pases
hair and sornings
blards carly hair slue
6 man weering sungle coouss happy looking
6 woman with red curty hair, weoring peor! eorrings. annpppy looking
Figure 12: More results for baselines implemented with SDXI., including InstantID, PhotoMaker and PuLID-SDXL. We provide personalization results including non-celebrity identities.
27

---

PuLID (FLUX)
Infinite
Reference
Prompt
(NeurlPS 2024)
(ICCV 2025)
Original
with FreeCure
Original
with
FreeCure
with black curly hair
woman Frowning worriedly
woman with blonde curly hair happy looking
man wearing sunglasses, asses, frowning worriedly
with curly hair, wearing pearl earrings, unhoppy looking
a man with eyes blonde curly hair,
smiling
Figure 13: More results for baselines implemented with FLUX, including PuLID-FLUX and Infiniteoo,, We provide personalization results including non-celebrity identities.
28

---

A.3.2 Supplementary Comparison of Results for Main Paper
To better demonstrate the efficacy of FreeCure, Fig 14 15 present comparative analysis of the results generated by the FD, PD, and enhanced outputs by FreeCure linked Fig The results indicate that FreeCure faithfully transfers target prompt-aligned attributes from the FD into the PD processes while preserving the original identity information.
FastComposer (UJV 2024)
Dafe-Diffuser (CVPR 2024)
Face2Diffusion (CVPR 2024)
man with black hair
wearing sunglasses, joyful looking
girl while curly hair, frowning worriedly
man with blue eyes and curly hair, smiling
Figure 14: Corresponding comparative FD & PD outputs for results in the main submission. (FastComposse, Cafe-Diffuser and FaceDiffusion) (a) refers original personalization outputs, (b) refers to foundation outputs, and (c) refers to outputs enhanced with proposed FreeCure.
29

---

Instantl (2024)
PhotoMaker (CVPR 2024)
PuLID (NeurIPS 2024)
man black curly hair
man wearing sunglasses, joyful looking
with white curly hair, Frowning worriedly
with blue eyes and blonde curly smillng
Figure 15: Corresponding comparative FD & PD outputs for results in the main paper. (Insan- IDD, PhotoMaker and PuLID) (a) refers to original personalization outputs, (b) refers to foundation outputs, and (c) refers to outputs enhanced with proposed FreeCure.
30

---

A.3.3 More Detailed Quantitative Results for Prompt Consistency with Multiple Attributes.
As supplementary results for 2 we provide source prompt consistency results of each baseline considering prompts with various attributes in Table shows that as the prompt becomes increas- ingly complicated, all baselines are facing more decreasing in prompt consistency, which can limit their application real scenarios. The results also support the fact that FreeCure's improvement in PC becomes more significant as the complexity of the prompt increases.
Table 7: Quantitative comparison of prompt consistency with different number of attributes.
Method
PC (1 Attr.) f
PC (2 Attr) f
PC (3 Attr.)
FastComposer
18.83
18.05
16.92 19.69 (+16.37%)
FastComposer FreeCure
21.42 (+13.75%)
21.29 (+17.95%)
Face-Diffuser
21.68
20.98
18.02
Face-Diffuser + FreeCure
22.89 (+5.58%)
22.72 (+8.29%)
21.17 (+17.48%)
Face2Diffusion
22.54
21.98
20.54
Face2Diffusion + FreeCure
23.81 (+5.63%)
23.02 (+4.73%)
22.63 (+10.18%)
InstantID InstantID + FreeCure
22.43
21.81
20.95
23.98 (+6.91%)
23.52 (+7.84%)
23.11
(+10.319)
PhotoMaker PhotoMaker + FreeCure
23.51
23.01
22.57
25.04 (+6.51%)
24.91 (+8.26%)
24.64 (+11.19%)
PuLID (SDXL) PuLID (SDXL) + FreeCure
25,56
25.12
24.43
26.30 (+3.34%)
25.96 (+5.32%)
25.73 (+3.55%)
PuLID (FLUX) PuLID (FLUX) + FreeCure
23.29
22.08
21.32
25.52 (+9.57%)
24.49 (+ (+10.91%)
23.87 (+11.96%)
Infinite Infinite FreeCure
25.01
23.19
22.43
25.98 (+3.88%)
24.92 (+7.46%
24.45 (49.01% %)
A.3.4 Integration with Segment-Anything for Attribute Extraction
Fig [6]shwcesess FreeCure's consistent performance when integrated with more advanced segmen- tation models such Segment-Anything For instance, the attribute mask cannot be handled by BiSeNet since its label absent. However, by replacing BiSeNet with Segment-Anythin and specifying the target prompt "mask," valid semantic masks be accurately generated, thereby guaranteeing effective attribute enhancement through FASA. Given that Begment-Anything extracts attributes based on flexible prompts, its seamless integration significantly enhances the versatility of FreeCure, enabling it to address a broader range scenarios.
Figure 16: Results for FreeCure's integration with Regment-Anything FreeCure can seamlessly integrate with more advanced segmentation models such as Segment-Ane -egthig-nything for extraction of boarder range attributes. Target prompt: "a person wearing mask".
A.3.5 Analysis on non-facial attributes enhancement
We have conducted supplementary experiments about attributes beyond facial elements. Following the prompt set of PuL.D, we evaluated prompts involving common objects, clothing, and other typical non-facial attributes, using Segment Anything for mask generation. The experimental results across several baselines are shown in Table
31

---

Table Quantitative comparison of on non non-facial attributes.
1
IF(%) t
Face Div. (%) t
Pace2Diffusion
22.87
41.08
41.34
Face2Diffusion + FreeCure
23.34 (+2.06%)
40.75 (-0.80%)
41.91 (+1.38%)
InstantID InstantlD + FreeCure
22.43
65,55
47.73
23.51 (+4.81%)
65.34 (-0.32%)
48.06 (+0.69%)
PhotoMaker PhotoMaker + FreeCure
24.67
52.14
46.02
25.37 (+2.84%)
51.77 (-0.71%)
46.58 (+1.21%
PuLID (SDXL)
27.43
59.41
41.97
PuLID (SDXL) + FreeCure
28.13 (+2.55%)
59.03 (-0.64%)
42.05 (+0.19%)
Experimental results indicate that the facial personalization model exhibits a weaker copy-paste effect in non-facial regions, leading to less pronounced improvements in prompt consistency compared to the facial attributes specifically addressed our submission. However, we observed that FreeCure still demonstrates significant restoration effects for attributes such as headphones and hats, which in close proximity to the face (see Fig Moreover, since the restoration occurs outside the facial region, FreeCure has negligible negative impact on identity fidelity.
man holding a bear cup
man wearing a headphone
woman wearing doctoral hat
Figure 17: Results on non-facial attributes. FreeCure can still improve prompt consistency on non-facial attributes, especially on objects whose location close face.
32

---

A.4 More Analysis
A.4.1
Non-interference Enhancement of Multiple Facial Attributes
We validate that the subsequent enhancement processes of FreeCure do not impact attributes that have already been enhanced. APG is used in the latter stages, attributes previously enhanced through FASA remain unaffected. This illustrates that FreeCure's improvement across multiple attributes is highly robust and consistent.
the intermediate results FreeCure's different stages. When
hair freweing
sueglessos happy
curly
+ Blonde
curly pesri carrings urhpppy
blonds curly blue syas happy
Demonstration of FreeCure's non-interference manner. top personalization models' outputs; middle row: intermediate enhanced results with FASA; bottom row: final results after APG enhancement.
Figure 18:
A.4.2 More Robustness Justification under Inaccurate Attribute Masks
Although FreeCure rely attribute segmentation process based pretrained semantic segmentation models. Its performance does not sensitive to the accuracy of these models. We design two-way validation experiments that introduce inaccuracy attribute masks. First, apply dilation operation on masks, which might include area which does not belong to the target attribute area. Second, we apply erosion operation, which might cause loss of some useful information about attributes. To create significant distortion of masks, we set the receptive field of both dilation and erosion to 25 pixels. We conduct experiments several baselines, and the Table 9 below reports the results for respective prompt consistency (PC) and identity fidelity (IF).
Table 9: PC and Performance for Inaccurate Masks
original IF
original PC
dialated IF
dialated PC
eroded IF
eroded PC
InstantID
62.01
23.62
62.23
23.45
62.23
23.21
PhotoMaker
50.15
24.91
50.01
24.87
50.44
24.34
PaLID-SDXI.
56.95
26.05
56.74
25.51
57.01
25.36
PaLID-FLUX
72.61
24.78
72.39
24.52
72.97
24.32
The results demonstrate that identity fidelity (IF) remains largely consistent across both dilated and eroded mask conditions, with slight improvements observed in eroded masks. This occurs because mask erosion preserves certain attributes similar to the reference image. As for prompt consistency, both dilation and erosion cause minimal degradation, though crosion exhibits marginally greater
33

---

impact. We consider that this stems from cases where fine- -ra-gerinn attribute restoration (e.g., iris color, earrings) may fail when the mask completely degenerates due to erosion.
Generally, the performance degradation of FreeCure due to inaccurate masks limited. Moreover, in practical applications, specialized segmentation models (e.g., BiSeNet, Segment Anything) exhibit high robustness, making failure cases caused by mask inaccuracies even rarer than observed in these deliberately designed experiments. This further validates the inherent robustness of the FreeCure framework.
A.4.3 More Comparison Results w/ and w/o Identical Initial Noise
We conduct extensive experiments evaluate the impact of initial noise conditions. Fig 19 demon- strates that under identical initial noise conditions, FreeCure consistently enhances target attributes across different input reference images. This finding highlights the robustness and practical applica- bility of FreeCure. Fig illlustrtass that FreeCure can successfully inject attribute information from the FD process to the PD process even their final generated faces have various locations, angles, and sizes. This finding particularly demonstrates the robustness of the FASA module.
Referenee
Pusoralizaioo wil Fizeet Cure
Persora lizationn + Freek ure
Referenee
Personalization Personalizatoo
wio FreeCure
+ FreeCure
Prompt: <person> with blonde curly hair
Figure 19: More studies for attribute enhancement start with identical initial noise. FreeCure exhibits robust performance with the single foundation output's guidance on personalized outputs from various references (generated based on Face-Diffuser and PhotoMaker).
34

---

Reference
+ sunglasses
PD
+ blonde curly hair
blonde curly hair
+ sunglasses
Figure 20: More validation on FreeCure's robust performance with different initial noises for PD and FD processes. Even if PD and FD's outputting faces have totally different locations, angles, and sizes, FreeCure can still exhibit stable enhancement performance.
35

---

A.4.4 More Detailed Quantitative Results for Ablation Studies
Table[T0 provides supplementary details for ablation studies mentioned in Table The results indicate that both FASA and APG consistently improve prompt consistency across all original personalization models. Notably, FASA yields more substantial improvements compared to APG, aligning with our findings in S.c
Table 10: Quantitative ablation analysis for FASA and APG's independent effect on overall performance. The table provides detailed ablation studies each baselines.
Method
FASA
APG
PC(%) f
IF(%) f
PC x IF (hMean) 1
/
X
20.91
41.94
27.91
FastComposer
X
v
19.03
42.84
26.35
V
x
21.86
57.92
31.74
Face-Diffuser
X
V
20.74
58.11
30.60
V
x
22.73
39.30
28.80
Face2Diffusion
X
v
21.98
39.47
28.24
V
x
23.02
62.45
33.64
InstantID
x
22.76
62.51
33.40
V
x
24.76
50.62
33.25
PhotoMaker
x
23.61
50.86
32.25
V
x
25.86
57.85
35.74
PuLID (SDXL)
x
/
25.54
57.24
35.33
24.12
72.98
36.26
PuLID (FLUX)
X
/
23.86
73.03
35.97
X
24.87
78.34
37.75
Infinite You
X
24.32
78.67
37.15
Stoole cufty' kot
sing casss
BLcck eatly nar
down-blecke
mil-ineds
up-hiidos
down-blocks minbbooke
midhlocks up-Nodks
doun-blacks spppllckk
Refornne
FD оzgp
Figure 21: Ablation study for applying FASA different positions of UNet. We find out that applying FASA modules to upsampling blocks alone will be sufficient enough to exhibit promising enhancement for target attributes.
A.4.5 Ablations of Applying FASA at Different Positions in the Denoising Model
We conduct detailed ablation studies to determine the optimal placement of the FASA module within
denoising model, as illustrated Fig 21 When FASA applied exclusively to the downsampling and middle blocks, its effectiveness is limited. In contrast, applying FASA to the upsampling blocks yields the most significant performance improvements. Furthermore, adding FASA the middle and downsampling blocks provides negligible additional benefits when the upsampling blocks already
36

---

utilized, as the upsampling blocks alone are sufficient to accurately transfer attribute information from the FD to the PD process. Based on these findings, we conclude that applying FASA to the denoising model's upsampling blocks represents the optimal configuration.
A.4.6 More Visualization of FASA
We provide more detailed visualization of FASA Fig which substantiates the claim of Sec. 5..2 that FASA enhances facial attributes in fine-grainee manner. For instance, FASA effectively captures and faithfully transfers information of attributes localized in small areas (e.g. eyes, pearl earrings). Furthermore, in regions unrelated to the target attributes, FASA maintains strong alignment with the original PD attention maps, demonstrating that preserves the core functionality of the personalization models.
#blonde curly hair blue eyes * pearl carnings
+blonds sunglosses
Qp
XI
Kp
Ky ]
Pp
X[
Kg
K
Figure 22: More visualization of FASA map attribule-relevant and attribute-irrelevan areas. Red points refer to the area target attributes and green points refer to the area which is not related to target attributes.
37

---

A.4.7 More Studies on Identity Embedding
23 illustrates the impact of identity embedding interpolation when integrated into the cross- attention layers of different blocks, as mentioned in Secß] When cross-attention maps of the FD process are injected into the downsampling blocks or middle blocks, the changes in output are minimal, even all identity embedding's ' cross-attention maps are replaced (a = 1). when the interpolation is applied to the upsampling blocks that significant degradation of identity information evident, while other facial attributes are effectively restored. summary, the identity embedding exerts its most significant influence within the upsampling blocks of the denoising model. However, a small value of a can cause significant identity information loss, supporting the argument that the well-trained cross-attention layers for identity information extraction is susceptible. in See 3
only
Jown-blecks
Nocks
up
a 00.0
a 0.1
0.2
a 0.3
0-04
a 00.5
-06
0.7
a 0.5
0.9
a 1.0
prompt: "a <man> image with black curly hair, laughing happily"
Figure 23: Identity embedding interpolation's influence on cross-attention layers of different blocks in the denoising model. It is applied on: downsampling blocks (top), bottleneck blocks (middle), and upsampling blocks (bottom).
We have conducted detailed analysis using finer-grained a values and their corresponding quantita- tive outcomes. Below are the results:
Table 11: Fined-graines Quantitative Metrics for Examples of
Interpolation in Sec
0
0.2
0.4
0.6
0.8
top
Identity Fidelity
50.23
24.53
10.34
6.42
3.43
2.1
Prompt Consistency
24.98
25.5
26.39
28.54
30.43
33.01
bottom
Identity Fidelity
52.34
49.52
29.12
15.75
6.32
3.2
Prompt Consistency
26.76
30.34
31.09
32.98
34.23
35.02
The results demonstrate that identity fidelity declines sharply even when only a small portion of the original cross-attention map is altered, suggesting that identifying a well-optimized sweet spof through cross-attention manipulation alone is highly challenging. In summary, difficulties in cross-attention manipulation provide with more solid motivation in the exploration of self-attention modules.
A.4.8 Inference Time Analysis
We conduct experiments on a single H800 GPU with 80 GB VRAM to measure each baseline's 's inference time before and after applying FreeCure. Table 12 below reports the results ("*" means 30-step denoising due to official guidance and 50-step denoising is applied for the rest baselines)
It is true that introducing extra calculation can lead to longer runtime, we would like to emphasize that most training-fre free methods introduce extra processes, including inversion operation, denoising processes, and extra attention calculation. Plus. comparing to the fact that most encoder- based personalization methods require a long range time of data collection, preprocessing and tuning (this may consume large array of GPUs and thousands of GPU-hours), our training-fres free method provides an innovative perspective for prompt consistency improvement by the knowledge in themselves, without the need for designing new model architecture and dataset curation.
38

---

Table 12: Runtime analysis
Baseline (seconds)
Baseline FreeCure (seconds)
Stage
Stage with FASA
APG
Total
FastComposer
1.79
1.96
2.19
1.83
5.98
Face-Diffuser
2.13
2.63
2.33
1.92
6.88
Face2Diffusion*
1.13
1.25
1.47
1.23
3.95
PhotoMaker
6.52
7.04
10.03
6.1
23.17
InstantID*
7.12
7.63
10.65
6.23
24.51
PuLID-SDXL
7.43
8.02
10.96
7.28
26.26
PuLID-FLUX*
12.84
14.89
16.86
12.95
44.7
Infinete You*
8.34
10.23
13.53
9.03
32.79
A.4.9
User Study Template
Fig 24 show an example of our proposed user study on the performance of FreeCure.
Personalization has been a popular application for Al tochnology. Given a reference portrait photo person and a prompt instruction, personalized model can generute novel image that includes this person conforms the description prompt. Two metrics are important:
Identity fidelity: The generated person must closely match the identity in the real portrait photo. Prompt consistency: The generated image must closely match the given description.
Next, you will evalunte the personalized model on these two aspects. In each test case, you will see reference portmait photo R, text prompt and two personalized generation results, A and Based on your own judgment, you assign preference scores (1-10) two images both identity fidelity and prompt consistency (the more you satisfied, the higher score you assign).
B
man with blonde curly hair and blue eyes, laughing
identity fidelity:
your evaluation
prompt consistency:
Figure 24: An example of user study. We use scoring strategy to collect user assessments about the samples with/without FreeCure.
39


Reviews:
[{"review_id": "Kxqw3E5cZg", "reviewer": "Reviewer_uVNV", "summary": "This paper addresses the problem of human face personalization in text guided image generation. First, to understand the problem modes, two image generation pipelines are explored: personalized denoising, with the identity embedding incorporated into the denoising/generation process, and foundation denoising, where the identity embedding is set to zero. It is found that the PD model encodes identity but loses consistency with other components of the prompt. On the other hand, the FD model does not have identity but allows for consistency to the rest of the prompt. Further, observing the cross attention layers indicates that the identity embedding reduces the intensity of the cross attention maps for other text prompts.\n\nTo solve this, FreeCure is proposed. This training-free method consists of two parts: (a) Foundation-Aware Self Attention: a dual-inference pipeline is used, where attention maps are generated for both the FD and PD pathways. Next, an external masking model like segment anything generates segmentation masks for attributes corresponding to various different parts of the prompt. These masked attention maps from the FD branch are then concatenated and combined together in the final attention branch. (b) Asymmetric prompt guidance: after attributes with clear location restored are generated via FASA, APG enhances attributes through inversion and denoising. First, during inversion, only a template prompt (a man) is given. Then, during denoising, the full prompt with attributes is included. No identity embeddings are used in this process, only text.\n\nAcross both celebrity and non-celebrity identities, compared with several recent baseline methods, the proposed method is found to improve all prior methods in terms of identity and prompt consistency, across a range of metrics. The method is also found to work with several attributes.", "questions": "Questions:\n1. I was not able to find the code in the submission: will that be included later?\n2. How sensitive is the method to hyper parameters such as \\omega and \\gamma: are these the same for all methods?\n3. In general, what types of models can the proposed method work with and what can it not work with? It would be good to include these in the limitations.", "limitations": "Yes, with some suggestions above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is largely well written, and method is easy to understand\n2. The method is well motivated, with analyses pointing out the tradeoff between prompt consistency and identity\n3. The method is simple, plug and play and easy to apply to most existing methods to improve prompt consistency\n4. Results show benefits afforded to existing methods by including the proposed method\n\nWeaknesses:\n1. I could not find a discussion on runtime and computational overhead. Naively, it seems like there would be a conisderable overhead to each method due to the proposed method, that should be discussed.\n2. It seems that the method would be quite sensitive to the accuracy of the mask generation method. How sensitive is the proposed method to inaccurate/partially accurate masks?\n3. For qualitative heavy results such as this, it would be good to include a user study to emphasize the improvements afforded by the proposed method.", "was_revised": true, "final_justification": "I would like the thank the authors for their response and the discussion. Based on this I would like to weight my score closer to acceptance, and recommend a weak accept score, taking into account both the strengths and the inherent weaknesses of this work.", "rebuttal": "We sincerely appreciate your valuable feedback and constructive suggestions, particularly the positive assessment of our paper's motivation, methodological simplicity, and comprehensive experimental analysis. In response to the raised concerns, we would like to address the following points for clarification.\n## 1. Runtime Analysis\nWe conduct experiments on a single H800 GPU with 80 GB VRAM to measure each baseline’s inference time before and after applying FreeCure. The table below reports the results (\"*\" means 30-step denoising due to official guidance and 50-step denoising is applied for the rest baselines):\n>**Table 1: Runtime Analysis**\n|     | Baseline (seconds) | Baseline + FreeCure (seconds) | |   |  |\n|:----:|:--------:|:---------:|:------:|:-------:|:--------:|\n|     |  **Total**| Stage 1| Stage 2 FASA | APG   | **Total**  |\n| FastComposer |**1.79**| 1.96| 2.19| 1.83  | **5.98** |\n| Face-Diffuser   |**2.13**| 2.63| 2.33 | 1.92  | **6.88**|\n| Face2Diffusion* |**1.13**| 1.25| 1.47| 1.23  | **3.95** |\n| PhotoMaker     |**6.52**| 7.04| 10.03| 6.1   | **23.17** |\n| InstantID*          |**7.12**| 7.63| 10.65| 6.23  |**24.51** |\n| PuLID-SDXL    |**7.43**| 8.02| 10.96| 7.28  | **26.26**  |\n| PuLID-FLUX*    |**12.84**| 14.89| 16.86| 12.95 | **44.7**  |\n| InfineteYou*      | **8.34**| 10.23| 13.53| 9.03  | **32.79**  |\n\nWhile we acknowledge that introducing extra calculation can lead to longer runtime, we also emphasize that most training-free methods introduce extra processes, including inversion operation, denoising processes, and extra attention calculation. Plus, comparing to the fact that most encoder-based personalization methods require a long range time of data collection, preprocessing and tuning (this may consume a large array of GPUs and thousands of GPU-hours), our training-free method provides an innovative perspective for prompt consistency improvement by the knowledge in themselves, without the need for designing new model architecture and dataset curation. \n\n## 2. Robustness under the Scenario of Inaccurate Masks\nWe reflect this concern by designing two-way validation experiments that introduce inaccuracy to attribute masks. First, we apply a **dilation** operation on masks, which might include area which does not belong to the target attribute area. Second, we apply an **erosion** operation, which might cause loss of some useful information about attributes. To create significant distortion of masks, we set the receptive field of both dilation and erosion to 25 pixels. We conduct experiments on several baselines, and the table below reports the results for respective prompt consistency (PC) and ideneity fidelity (IF):\n> **Table 2: PC and IF Performance for Inaccurate Masks**\n|            | PC with original masks | IF with original masks | PC with dilated masks | IF with dilated masks | PC with eroded masks | IF with eroded masks |\n|---|----|------|---|----|----|----|\n| InstantID  | 23.62    | 62.01   | 23.45  | 62.23  | 23.21  | 62.23  |\n| PhotoMaker | 24.91   | 50.15   | 24.87  | 50.01    | 24.34 | 50.44   |\n| PuLID-SDXL | 26.05  | 56.95  | 25.51  | 56.34 | 25.36  | 57.01 |\n| PuLID-FLUX | 24.78   | 72.61  | 24.52   | 72.39   | 24.32  | 72.97  |\n\nThe results demonstrate that identity fidelity (IF) remains largely consistent across both dilated and eroded mask conditions, with slight improvements observed in eroded masks. This occurs because mask erosion preserves certain attributes similar to the reference image. As for prompt consistency, both dilation and erosion cause minimal degradation, though erosion exhibits marginally greater impact. We consider that this stems from cases where fine-grained attribute restoration (e.g., iris color, earrings) may fail when the mask completely degenerates due to erosion.\n\nGenerally, the performance degradation of FreeCure due to inaccurate masks is limited. Moreover, in practical applications, specialized segmentation models (e.g., BiSeNet, Segment Anything) exhibit high robustness, making failure cases caused by mask inaccuracies even rarer than observed in these deliberately designed experiments. This further validates the inherent robustness of the FreeCure framework.\n\n## 3. User Study\nWe appreciate your attention to this point and agree that a user study can provide constructive support for the evaluation of our method. We conduct an online questionnaire with 30 participants. Regarding the questionnaire design: For each baseline, we randomly select 10 samples. Each sample consists of a reference image, a prompt, the baseline-generated result, and the result after applying FreeCure. Participants are asked to evaluate the identity fidelity (IF) and prompt consistency (PC) of each result. Specifically, they rate their satisfaction on a scale of 1 to 10, considering identity fidelity and prompt consistency. The survey samples from different baselines are presented randomly, and the order of the baseline results and FreeCure-applied results is also randomized to ensure that participants' judgments for each sample remained relatively independent. Based on scoring results, we provide both the scoring average values and preference ratio.\n> **Table 3: User Study Score**\n|     | Baseline      |    | Baseline + FreeCure |    |\n|--|----|-----|-----|---|\n|     | IF Score Avg. | PC Score Avg. | IF Score Avg.       | PC Score Avg. |\n| FastComposer   | 7.55  | 4.96   | 7.51 | **8.35** |\n| Face-Diffuser     | 7.56  | 5.08   | 7.41  | **7.62**   |\n| Face2Diffusion   | 7.83  | 5.02   | 7.71  | **7.83**  |\n| PhotoMaker       | 6.95  | 6.41   | 7.49   | **8.04**   |\n| InstantID            | 7.78  | 5.28   | 7.79   | **8.20**   |\n| PuLID-SDXL      | 7.76  | 6.38   | 8.08    | **8.26**   |\n| PuLID-FLUX      | 7.78  | 6.56   | 8.08   | **8.02**  |\n| InfineteYou        | 7.83  | 6.52   | 7.79   | **8.18**  |\n\n> **Table 4: User Preference on Identity Fidelity**\n|   | Prefer Baseline | Prefer Equally | Prefer Baseline + FreeCure |\n|---|----|---|-----|\n| FastComposer   | 28.88% | **60.54%**  | 10.58%  |\n| Face-Diffuser     | 25.33%  | **63.83%** | 10.83%|\n| Face2Diffusion   | 18.88%  | **68.00%** | 13.13%  |\n| PhotoMaker       | 12.54% | **78.83%** | 8.63%  |\n| InstantID            | 25.38% | **64.29%**  | 10.33%  |\n| PuLID-SDXL      | 10.67% | **81.38%**  | 7.96%  |\n| PuLID-FLUX      | 11.50% | **84.67%** | 3.83%  |\n| InfineteYou        | 10.67% | **79.25%**   | 10.08%   |\n\n> **Table 5: User Preference on Prompt Consistency**\n|                | Prefer Baseline | Prefer Equally | Prefer Baseline + FreeCure |\n|---|--|---|---|\n| FastComposer   | 4.29%  | 14.38% | **81.33%** |\n| Face-Diffuser  | 5.17%  | 16.71%   | **78.13%**  |\n| Face2Diffusion | 4.04%   | 16.04%| **79.92%**    |\n| PhotoMaker     | 4.38%   | 31.88%  | **63.75%**  |\n| InstantID      | 8.46%    | 12.58%  | **78.96%**  |\n| PuLID-SDXL     | 4.29%  | 16.92% | **78.79%** |\n| PuLID-FLUX     | 10.54%  | 16.46%  | **73.00%**  |\n| InfineteYou    | 9.75%   | 21.00% | **69.25%** |\n\nThe results show that, based on human evaluation, FreeCure consistently improves every baseline's prompt consistency. Furthermore, we observe that there are many cases where participant tends to give the same score for identity fidelity. Such performance supports the fact that FreeCure's negative effect on identity fidelity is quite limited in a practical manner and promises its practical use in the future. In our updated version, we plan to provide the template example for the questionnaire and more straightforward visualization techniques (e.g., column chart) to demonstrate the results of the user study.\n\n## 4. Hyper-parameter analysis\nWe acknowledge that the optimal configuration of FreeCure varies slightly for each baseline. However, this does not constitute a weakness: for each baseline, the fluctuation of their optimal hyperparameters is within an acceptable range:\n$$\n\\omega \\in [1.8, 2.4], \\gamma \\in [0.5, 0.6]\n$$\nBaselines built on Stable Diffusion v1.5 (Face2Diffusion, FastComposer, Face-Diffuser) require a larger $\\omega$ value up to 2.4, and other baselines remain around 2.0. Within this range, the overall results do not differ much, which allows us to determine a unified hyperparameter setting without any further complicated strategy ($\\omega = 2.0$ and $\\gamma = 0.5$ throughout all experiments in the submission), and still achieved consistent performance improvement in prompt consistency with the maintenance of identity fidelity.\n\nSecondly, we would like to point out that since the foundation models and architectures of each baseline are diverse, FreeCure—being a plug-and-play method—should be allowed for a reasonable adjustable range to better adapt to each baseline.\n\n## 5. Code Release\nThanks for pointing this out. We will release the code of FreeCure later.\n\n## 6. Limitations\nOur method can seamlessly apply to most diffusion-based personalization models (including those based on SD-v1.5, SDXL, and DiT), which dominate the personalization task. Since currently there is no influential personalization method based on auto-regressive models, it may cost some extra effort to adapt our methods to these methods in the future. We will add the discussion of this matter in the limitations."}, {"review_id": "LmksNaOBCq", "reviewer": "Reviewer_rzYz", "summary": "The authors introduce the tradeoff between identity preservation and prompt adherence in current human-face embedding-based personalization methods, provide an explanation and analyze the effect of key related factors - the aggressive effect of personalization embeddings on cross attention layers, and the relative sensitivity of these layers to any change. Then, they provide a method that avoids cross-attention manipulation by taking self attention layers into account. They present a method that harnesses the spatial and semantic information encapsulated in the non-personalized foundation models self attention layers to enforce the wanted changes in the personalized generation process, without harming the personalization itself.", "questions": "* The trade-off depicted in the right rows Figure 2 is clear, but a finer resolution is needed to contradict the existence of a well optimized sweet spot. For example, in the top-row between alpha=0.2 and alpha=0.4, and in the bottom-row between alpha=0.4 and alpha=0.6.\n* In section 4.1 (FASA Mechanism), there is a change in the matrices’ dimensionality due to concatenation. How does this change incorporate in the existing architecture? It might be implicitly inferred with previous domain knowledge, but I believe it requires explicit addressing.\n* The method details the manipulation it applies to the UNET inference process, but some of the presented results are derived from FLUX based methods, while FLUX does not contain UNET in its architecture. This part needs clarification.\n* The set of desired prompts is rather small - curly hair (in different colors), eye color, happiness, sunglasses, and pearl earrings. I would like to see a wider variety of prompts with non-trivial specifications , e.g. with an eye path, posture change, background environment, etc.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n* The paper includes a convincing description of the problem, with a well performed empirical analysis to support it.\n* The method is innovative and yet not too complicated.\n* The results, both quantitatively and qualitatively,  show consistent improvement compared to baselines.\n* The framework is applicable to multiple embedding-based methods, spanning different base models.\nWeaknesses:\n* The paper presents many different faces (mainly in the appendix), but the variety of prompts is low.\n* The analysis in section 3 could be more complete.\n* Some technical details are missing, or de-emphasized.", "was_revised": true, "final_justification": "Sorry for missing this field, please see my comments  in the author discussion", "rebuttal": "Thanks for your valuable comments and positive evaluation on our empirical analysis, method simplicity and experimental results. We propose the following points to address all your concerns:\n## 1. More Comprehensive Prompt Set\nThank you for raising this point. First, our method can leverage any attribute segmentation model to handle different types of features. In **Section A.2.4** of the supplementary materials, we discuss the performance of general semantic segmentation models (e.g., SAM) within our framework and address attributes that fall outside facial features. It is important to note that since most current encoder-based facial personalization methods extract and use the facial region as a conditional input, the copy-paste issue is most pronounced in facial attributes—which is the primary focus of our paper. In contrast, non-facial attributes, which are not subject to additional constraints during training, exhibit a less significant copy-paste effect. \n\nHowever, we fully understand your concern regarding the completeness of this discussion and have conducted supplementary experiments to address it. Following PuLID’s prompt set, we evaluated prompts involving common objects, clothing, and other typical non-facial attributes, using Segment Anything for mask generation. The experimental results across several baselines are as follows:\n\n> **Table 1: Additional Prompt Set with Non-facial Attributes**\n|   | new prompts                        |\n|---|------------------------------------|\n| 1 | a person wearing a *black headphone* |\n| 2 | a person wearing a *doctoral hat*    |\n| 3 | a person holding a *beer cup*        |\n| 4 | a person wearing a *spacesuit*       |\n| 5 | a person wearing a *yellow scarf*    |\n\n> **Table 2: Perfomance on Additional Prompt Set with Non-facial Attributes**\n|                           | PC    | IF    | Face Div. | IF x PC (hMean) |\n|---------------------------|-------|-------|-----------|-----------------|\n| Face2Diffusion            | 22.87 | 41.08 | 41.34     | 29.38           |\n| Face2Diffusion + FreeCure | **23.34 (+2.06%)** | 40.75 (-0.80%)| **41.91 (+1.38%)**    | **29.68(+1.01%)**|\n| InstantID                 | 22.43 | 65.55 | 47.73     | 33.42           |\n| InstantID + FreeCure      | **23.51(+4.81%)** | 65.34(-0.32%) | **48.06(+0.69%)**  | **34.58(+3.46%)**|\n| PhotoMaker                | 24.67 | 52.14 | 46.02     | 33.49           |\n| PhotoMaker + FreeCure     | **25.37(+2.84%)** | 51.77(-0.71%) | **46.58(+1.21%)**| **34.05(+1.67%)** |\n| PuLID (SDXL)              | 27.43 | 59.41 | 41.97     | 37.53           |\n| PuLID (SDXL) + FreeCure   | **28.13(+2.55%)** | 59.03(-0.64%) | **42.05(+0.19%)**| **38.10(+1.52%)**|\n\nExperimental results indicate that the facial personalization model exhibits a weaker copy-paste effect in non-facial regions, leading to less pronounced improvements in prompt consistency compared to the facial attributes specifically addressed in our submission. However, we observed that FreeCure still demonstrates significant restoration effects for attributes such as headphones and hats, which are in close proximity to the face (additional comparative images will be included in the updated version). Moreover, since the restoration occurs outside the facial region, FreeCure has a more negligible negative impact on identity fidelity. Regarding attributes such as body pose and background, we believe their relevance to our paper’s focus—resolving face copy-paste issues—is minimal, and thus they were not discussed. We appreciate your understanding in this regard.\n\nIn summary, while our method remains applicable to various non-facial attributes due to its generalizability, the optimization effects are less pronounced but also noticeable compared to facial features. \n\n## 2. Detailed discussion for Section 3\nThanks for pointing out this detail. We have conducted a detailed analysis using finer-grained $\\alpha$ values and their corresponding quantitative outcomes. Below are the results:\n\n> **Table 3: Fined-grained Quantitative Metrics for Examples of Cross-attention Interpolation in Section 3**\n|            | $\\alpha$              | 0     | 0.1   | 0.2   | 0.3   | 0.4   | 0.5   | 0.6   | 0.7   | 0.8   | 0.9   | 1     |\n|------------|--------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| top row    | IF  | 50.23 | 37.43 | **24.53** | **16.42** | **10.34** | 7.32  | 6.42  | 4.22  | 3.43  | 2.34  | 2.1   |\n|            | PC | 24.98 | 25.39 | **25.5**  | **25.61** | **26.39** | 27.53 | 28.54 | 29.43 | 30.43 | 32.34 | 33.01 |\n| bottom row | IF  | 52.34 | 50.23 | 49.52 | 40.01 | **29.12** | **23.89** | **15.75** | 10.34 | 6.32  | 4.11  | 3.2   |\n|            | PC | 26.76 | 28.43 | 30.34 | 31.12 | **31.09** | **31.87** | **32.98** | 33.43 | 34.23 | 34.79 | 35.02 |\n\nThe results demonstrate that identity fidelity declines sharply even when only a small portion of the original cross-attention map is altered, suggesting that identifying a '*well-optimized sweet spot*' through cross-attention manipulation alone is highly challenging. Plus, as you noted, the fine-grained performance varies considerably between different $\\alpha$ ranges (e.g., $\\alpha \\in [0.2, 0.4]$ for the top row and $\\alpha \\in [0.4, 0.6]$), further highlighting a key limitation of cross-attention manipulation: its optimal control parameter can exhibit substantial variability. We will include a more detailed version of Figure 2 in the updated submission to provide further clarity.\n\n## 3. Matrices Dimensionality Change of FASA\nWe reflect this question by providing the details of FASA. In the original self-attention mechanism with batch size 1, the dimensionality of $Q, K, V$ considering with $n$ heads is:\n$$\nQ, K, V \\in \\mathbf{R}^{1\\times n \\times l \\times (\\frac{h}{n})}\n$$\nThe calculation of attention map $A = Softmax(\\frac{QK^T}{\\sqrt{d}}) \\in \\mathbf{R}^{1\\times n \\times l \\times l}$ is straightforward. After we replace this self-attention layer with FASA module, typically we have batch size 2 due to dual inference design:\n$$\n    \\hat{Q}, \\hat{K}, \\hat{V} \\in \\mathbf{R}^{2\\times n \\times l \\times (\\frac{h}{n})}\n$$\nIn FASA inference process, the PD process is in the first batch while FD is in the second. We copy the FD batch and concatenate it to PD batch along the sequence dimension $l$:\n$$\n    \\hat{K}\\_{[0, :, :, :]} \\leftarrow \\mbox{concat}(\\hat{K}\\_{[0, :, :, :]}, \\hat{K}\\_{[1, :, :, :]}, \\mbox{dim=2})\n$$\n$$\n    \\hat{V}\\_{[0, :, :, :]} \\leftarrow \\mbox{concat}(\\hat{V}\\_{[0, :, :, :]}, \\hat{V}\\_{[1, :, :, :]}, \\mbox{dim=2})\n$$\nNow the PD process have an extended matrices of $\\hat{K}\\_{p}, \\hat{V}\\_{p} \\in \\mathbf{R}^{1\\times n \\times 2l \\times (\\frac{h}{n})}$. Therefore, FASA's attention map matrices $\\hat{A\\_p} \\in \\mathbf{R}^{1\\times n \\times l \\times 2l}$ should also be extended. As for the scaling mask strategy, we resize the extracted mask $\\mathcal{M}$ to the size $l\\times l$ and replicate it to target size $1\\times n \\times l\\times l$ after concatenating with a matrix with all elements equal to 1; this new mask can process $\\hat{A\\_p}$ accordingly. The entire process of PD process after applying FASA is as follows:\n$$\n    \\mbox{FASA}(\\hat{Q\\_p}, \\hat{K\\_p}, \\hat{V\\_p}) = \\mbox{Softmax}(\\frac{[\\mathbf{1},\\omega\\mathcal{M}]\\odot \\hat{Q\\_{p}}\\hat{K\\_p}^{T}}{\\sqrt{d}})\\hat{V\\_p}.\n$$\nThis process aligns with Eq. 4 in the submission. Afterwards, PD's output dimensionality aligns with the original FD process and can be gathered together for the inference process of subsequent layers (e.g., FFN). Please note that since the PD and FD's matrices of $K$ and $V$ have different dimensions, their attention scores are calculated separately. However, this is easy to implement because the communication happens between two batches, and the external mask $\\mathcal{M}$ can be defined during FASA modules' initialization. \n\n## 4. FASA's Implementation in FLUX\nThanks for your comments on this. We agree that this point should be clarified. The implementation of FASA on FLUX's full-attention is almost identical to U-Net's self-attention but with some slight differences in mask strategy. The input of full-attention is a joint sequence of noisy latent and text condition: $[X; C], X\\in \\mathbf{R}^{b\\times l\\_1 \\times h}, C \\in \\mathbf{R}^{b \\times l\\_2 \\times h}$. Since PD and FD share the identity text condition, we extract noisy latent relevant $K\\_f^X, V\\_f^X$ from the FD process and concatenate it to $K\\_p^X, V\\_p^X$ of the PD process, while $Q$ keeps consistent:\n$$\n\\hat{Q\\_p} = [Q\\_p^X; Q\\_p^C]; \\hat{K\\_p} = [K\\_p^X;K\\_p^C;K\\_f^X]; \\hat{V\\_p} = [V\\_p^X;V\\_p^C;V\\_f^X]\n$$\nThe main difference lies in the mask strategy. Aligning with U-Net's implementation, we apply the all-one matrix to PD's original attention part. Specially, we assign scaling masks $\\omega \\mathcal{M}$ to attention map of $Q\\_p^X and K\\_f^X$ and assign a all-zero mask to attention map of $Q\\_p^C and K\\_f^X$ to preserve PD's original full-attention patterns:\n$$\n\\mbox{FASA}(\\hat{Q\\_p}, \\hat{K\\_p}, \\hat{V\\_p}) = \\mbox{Softmax}(\\frac{\\mathcal{M}(\\omega)\\_{flux}\\odot \\hat{Q\\_{p}}\\hat{K\\_p}^{T}}{\\sqrt{d}})\\hat{V\\_p},\n$$\n$$\n\\mathcal{M}(\\omega)\\_{flux} = \\begin{pmatrix}\n  \\mathbf{1}\\_{l\\_1\\times l\\_1} & \\mathbf{1}\\_{l\\_1\\times l\\_2} & \\omega \\mathcal{M}\\_{l\\_1\\times l\\_1} \\\\\\\\\n  \\mathbf{1}\\_{l\\_2\\times l\\_1} & \\mathbf{1}\\_{l\\_2\\times l\\_2} & \\mathbf{0}\\_{l\\_2\\times l\\_1}\n\\end{pmatrix}\n$$\nThis mask pattern resembles FASA's implementation in U-Net which injects attribute-aware features from FD to PD while keeping the original full-attention's pattern. We will update this formula for FLUX in the updated version of the submission."}, {"review_id": "SswiTn85Y3", "reviewer": "Reviewer_nf3e", "summary": "This paper introduces a training-free framework called FreeCure attempted to address the common trade-off between identity fidelity and prompt consistency in personalized image generation. \nThe core of their method, Foundation-Aware Self-Attention (FASA), transfers features from the foundation diffusion (FD) process to the personalization diffusion (PD) process using segmentation masks to improve attribute control while aiming to preserve the subject's identity. Additionally, an Asymmetric Prompt Guidance (APG) technique is used to refine abstract attributes like expressions. The method is presented as a plug-and-play module for various existing personalization models.", "questions": "1. Novelty of the FASA Technique: The writing in the methodology section is misleading. The proposed Foundation-Aware Self-Attention (FASA) mechanism appears to be a rebranding of the \"mutual self-attention\" concept introduced in the MASACtrl paper. The core operation of concatenating key and value matrices from two different generation processes to control synthesis is not a novel contribution.\n\n2. Missing Mathematical Equation of APG: I am not sure if I understand this correctly. As I interpret it, you first run FD alone in the first pass to obtain intermediate denoised latents without the expression prompt, and then use these latents as the starting point for a second pass with FASA and the expression prompt. Is this correct? Why is inversion necessary? Why not run the second step directly? Is the choice of $\\gamma$ consistent across all your results? Do you use APG in all results shown in the paper?\n\n3. Attribute Entanglement: The shared attention mechanism is fundamentally based on KV injection. The prior identity information from the hidden states of the FD process will also be introduced into the PD process, leading to lower identity preservation, as observed in nearly all samples.\n\n4. Mask Dependency: The masking strategy must be tuned for each prompt and relies on detailed manual masking. How to eliminate manual masking while keeping the system stable should be further investigated.\n\n5. Missing Discussion of Critical Prior Art: The paper fails to discuss or compare against \"Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation\" (MoA), presented at SIGGRAPH Asia 2024. MoA can be considered a trainable counterpart addressing a similar problem as this work.\n\n6. Missing Baseline Comparison: A simple yet effective baseline would be to compare FreeCure against a state-of-the-art personalization method using a reduced inference weight (e.g., a scale of 0.6) combined with a post-processing face-swapping technique. This alternative also aims to balance identity preservation and prompt fidelity and would serve as a valuable comparison point given the complexity introduced by FreeCure.", "limitations": "The limitation discussion of this work requires more clarification. The authors state that FreeCure’s performance is influenced by the characteristics of the underlying personalization models, including their inherent biases and maximum achievable capabilities. This explanation is reasonable and expected. However, we would appreciate a more candid discussion of the general limitations of the proposed method — even when applied within a fixed foundation model. I provide some limitations are raised in the questions above and the authors might know more limitations and can showcase more failure cases.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe main strength of this work lies in its introduction of a training-free approach that addresses a known limitation of personalization methods — namely, the performance drop in prompt alignment.\n\nWeaknesses:\nHowever, in nearly all examples, the proposed method suffers from a clear trade-off between identity preservation and prompt alignment. Additionally, the shared attention mechanism used is not novel; similar techniques have been widely explored in prior work within the field. See Questions section for more details.", "was_revised": true, "final_justification": "At current, I would like to raise my score to a borderline accept. I appreciate the efforts of authors in clarifying some of my previous concerns. My only remaining concern is the Attribute Entanglement in Prompt Decomposition (PD). While the rebuttal argues that masking alleviates entanglement, my concern is that the key-value pairs from PD still contain features entangled with both target attributes and unwanted prior identity signals. These prior identity features are not explicitly excluded during shared attention. I believe that further disentangling and removing the prior identity features before injection would strengthen the proposed method.", "rebuttal": "## 1. Novelty Clarification\nWhile we agree that both FreeCure and MasaCtrl apply self-attention manipulation. We argue that our method is significantly different from MasaCtrl in several points that guarantee its novelty:\n+ **Different Generation Paradigm.** Masactrl converts an input image into a latent code through inversion and then copies it to its dual inference counterpart. In contrast, FreeCure exhibits less dependency on this. FreeCure's primary process generates images directly from noise, and the initial noise for its FD and PD processes can be sampled arbitrarily without requiring consistency. This fundamental difference in the generative paradigm enhances the diversity of restoration results in FreeCure (see **Section 5.2 and A.3.2** of the submission for details). By comparison, MasaCtrl's dual inference mechanism strictly depends on identical latent codes, lacking exploration into the effects of different initial latents.\n+ **Different Task Focus.** While MasaCtrl primarily investigates non-rigid image editing tasks such as changing pose or viewpoint, our work focuses on enhancing prompt consistency in facial personalization models. Moreover, whereas MasaCtrl performs editing on the entire subject (e.g., pose adjustment), our focus lies in refining internal features (regardless of their granularity) of the subject and can seamlessly integrate with various mainstream facial personalization models.\n+ **Better Adaptability.** The adaptability of the FreeCure extends beyond U-Net's self-attention mechanism. Its FASA algorithm can also be applied to the full-attention of FLUX. However, this aspect remains unexplored in MasaCtrl’s discussions. MasaCtrl also lacks discussion on the simultaneous editing of multiple object attributes. In contrast, our proposed FASA framework can effectively handle multiple facial attributes concurrently while maintaining high identity fidelity.\n+ **Masking Strategy.** Building upon the second point, while MasaCtrl’s mutual self-attention mechanism leverages cross-attention maps as guidance, its subsequent thresholding operation introduces great noise and fails to accurately localize fine-grained features. To further validate this, we modify FreeCure's mask by replacing segmentation-model-extracted masks with those derived from cross-attention maps. We aggregated cross-attention maps for the corresponding tokens in a way identical to MasaCtrl and applied Otsu’s thresholding method for binarization:\n\n> **Table 1: Performance with Cross-attention Mask**\n|| PC of Original FreeCure|IF of Original FreeCure|PC of FreeCure with cross-attention mask|IF of FreeCure with cross-attention mask|\n|-|-|-|-|-|\n|InstantID|**23.62**|**62.01**|19.23|59.23|\n|PhotoMaker|**24.91**|**50.15**|22.43|49.32|\n|PuLID-SDXL|**26.05**|**56.95**|24.52|53.43|\n|PuLID-FLUX|**24.78**|**72.61**|22.95|67.67|\n\nThe generated images will be updated in the new version. We can see that when using the cross-attention map as a mask, all baselines showed decreased performance in both prompt consistency and identity fidelity. This happens because compared to masks created by expert segmentation models, the cross-attention mask picks up too much irrelevant noise during the enhancement process. This noise undermines useful features and damages the original facial identity fidelity.\n\nTo sum up, we argue that although FASA of FreeCure and mutual self-attention of MasaCtrl share a similar concept of attention control, their difference lies in, but is not limited to, overall paradigm, problem focus, adaptability, as well as application complexity. Since other reviewers (uVNV,rzYz, and tTVJ) acknowledge the motivation, simplicity, and generalizability of FreeCure, its novelty and contribution should be guaranteed.\n\n## 2. Supplementary Explanation of APG\n+ **Formula and implementation of APG.** During APG, FASA is excluded since this stage only involves single-batch inference of the baseline model. We employ the original baseline's FD process independently and leverage its inherent consistency in handling abstract attributes to further refine the results improved by FASA. The process can be described via the following formula:\n$$\n\\hat{z_{\\gamma T}} = \\mbox{Inversion}(I_r, c_0); \nI_{out} = \\mbox{Denoise}(\\hat{z_{\\gamma T}}, c_1)\n$$\nwhere $I\\_r, I\\_{out}, \\hat{z\\_{\\gamma T}}$ matches the annotation in Figure 3 of submission, $c\\_0$ denotes a template prompt without any attribute description (e.g., \"*a man*\") and $c\\_1$ denotes prompt containing target attirbutes (e.g., \"*a man laughing*\").\n\n+ **Reason for separating APG from the second step.** Such separation allows distinct handling of spatially localized features and abstract attributes. Processing these features jointly within the same denoising stage would hinder their effective decoupling, ultimately compromising the enhancement of individual characteristics.\n\n+ **Value selection for $\\gamma$.** All experimental data and visual results in our main submission were obtained consistently using $\\gamma=0.5$. While we acknowledge that the optimal $\\gamma$ may vary slightly across different baselines, its value consistently falls within the range of $[0.5, 0.6]$. For a plug-and-play method, we argue that this minor variation is acceptable due to the diversity of baselines.\n\n+ **Usage of APG**. During the generation phase, if no prompt related to abstract features (e.g., facial expression) is matched, the APG module can be safely omitted.\n\n## 3. Attribute Entanglement\nWe argue that the comments of \"*trade-off in nearly all examples*\" do not have specific evidence. The problem of \"*...The prior identity information from the hidden states of the FD process will also be introduced...*\" can be directly answered by **Figure 4** of our submission. Our mask guidance mechanism provides an effective solution to protect the identity fidelity of PD from FD. Our visual experiments in **Section 5.2** also confirm that in non-target regions, the FD process exerts minimal influence on the PD denoising process. Plus, our method's superiority has been validated both qualitatively and quantitatively and acknowledged by other reviewers. A supplementary user study required by Reviewer uVNV also highly aligns with our consistent improvement of the submission. Therefore, we respectfully suggest that you reconsider your assessment.  \n\n## 4. Mask Dependency\nYou may have some misunderstandings which we would like to clarify: FreeCure’s mask extraction does not rely on any case-specific \"*tuning*\" or \"*manual processing*\". Instead, our system employs a lookup table encompassing various facial attributes to align with the input prompt. When target attributes are matched, the face parsing model (e.g., BiSeNet) or Segment Anything Model (SAM) automatically generates the corresponding masks. Specifically, BiSeNet inherently predicts masks for all facial attributes, so only the matched masks need to be extracted. In the Segment Anything, the model takes the matched attributes as input to produce the mask. Thanks to these models’ generalization capabilities in segmentation, the entire process operates without manual intervention.\n\n## 5. Discussion of MoA\nMoA's prior attention primarily addresses the generation of regions unrelated to the subject (e.g., background). This objective fundamentally differs from FreeCure, which is designed to enhance prompt consistency for features within the main subject. The only commonality between the two approaches may be their shared goal of integrating foundational model knowledge into personalized models. Additionally, FreeCure is a plug-and-play framework, compatible with diverse personalization methods. Its relationship with MoA is better characterized as complementary (i.e., upstream/downstream) rather than being a \"*trainable counterpart*\". Given FreeCure’s demonstrated adaptability across a wide range of personalization models, we anticipate its potential application to MoA as well. In the updated version of our submission, we will include this reference and discuss the feasibility of integrating FreeCure into MoA.\n\n## 6. Missing Baseline Comparison\nWe argue that the pipeline proposed by the reviewer cannot be comparable to our method. The reviewer suggests using a '*reduced inference weight*' to generate results, but this approach harms both identity fidelity and prompt consistency (see our discussion in **Section 3**). Instead, directly adopting our proposed FD process (equivalent to setting the 'inference weight' directly to 0) could be a more viable solution.\n\nPlus, this pipeline critically depends on face swapping models to disentangle identity and attributes, particularly when dealing with occlusions (e.g., sunglasses, masks) or spatial misalignments in foundation model outputs. Current art face-swapping methods (e.g., DreamID, DiffSwap) rarely address these cases, and their failures mirror the copy-paste issues in personalized models. In contrast, FreeCure provides a simpler, more effective solution, as detailed in **Section 5.2**.\n\nIn summary, there exists a significant gap between the baseline mentioned by the reviewer and the problem setting addressed in our work. We contend that, in the context of facial personalization, this pipeline does not constitute a comparable paradigm to our proposed method.\n\n## 7. Limitations\nAlthough FreeCure shows consistent improvements in wide range of baselines and attributes. We observe that when enhancing transparent objects (e.g., cups), the restored images may exhibit optical distortions in transparent regions. We will include a discussion of this failure case in the Limitations section and explore potential optimizations for such cases in future work.\n\n## 8. Ethical Concerns\nSince our evaluation dataset comes from public resources (e.g., CelebA-HQ, public webpages). It is safe to say that concerns on data privacy is minimal. \n\n## Reference\n[1] MoA, SIGGRAPH Asia 2024.\n\n[2] DreamID, arXiv 2025.\n\n[3] Diffswap, CVPR 2023."}, {"review_id": "lW2mthLqsM", "reviewer": "Reviewer_kayu", "summary": "This paper tries to tackle the common failure of personalization models to adhere to text prompts. It proposes FreeCure, a training-free framework that identifies a key cause: identity embeddings suppressing the foundation model's prompt-following ability. FreeCure uses a dual-inference process to leverage the \"cured\" output from the foundation model to guide and correct the personalized generation, improving prompt consistency while maintaining identity.", "questions": "1. How much does the cost of inference increase?\n2. How robust are the attributes parsing and segmentation models? Can complex user prompts be automatically processed?\n3. Figure  3 shows the implementation of Unet, how is the text-image full attention model like flux implemented?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. The argument that ID embeddings suppress the model's underlying prompt-following capabilities is reasonable and provides a strong motivation for the solution.\n2. The training-free \"curing\" approach is elegant and shows significant, consistent improvements across a wide range of popular personalization models and diverse foundation models.\n\nWeaknesses:\n1. Increased Inference Cost: The dual-inference design effectively doubles the computational cost per step, which is a significant trade-off for speed-sensitive applications.\n2. Relies on an external segmentation model for localizing attributes, which adds a layer of complexity and a potential point of failure.\n3. The two proposed points appear to be more of practical engineering techniques.", "was_revised": true, "final_justification": "Thanks for the rebuttal; it effectively solved my question， and I raise my score to weak accept", "rebuttal": "## 1. Inference Cost\n\nWe conduct experiments on a single H800 GPU with 80 GB VRAM to measure each baseline’s inference time before and after applying FreeCure. The table below reports the results (\"*\" means 30-step denoising due to official guidance and 50-step denoising is applied for the rest baselines):\n\n>**Table 1: Runtime Analysis**\n|     | Baseline (seconds) | Baseline + FreeCure (Seconds) | |   |  |\n|:----:|:--------:|:---------:|:------:|:-------:|:--------:|\n|     |  **Total**| Stage 1| Stage 2 FASA | APG   | **Total**  |\n| FastComposer |**1.79**| 1.96| 2.19| 1.83  | **5.98** |\n| Face-Diffuser   |**2.13**| 2.63| 2.33 | 1.92  | **6.88**|\n| Face2Diffusion* |**1.13**| 1.25| 1.47| 1.23  | **3.95** |\n| PhotoMaker     |**6.52**| 7.04| 10.03| 6.1   | **23.17** |\n| InstantID*          |**7.12**| 7.63| 10.65| 6.23  |**24.51** |\n| PuLID-SDXL    |**7.43**| 8.02| 10.96| 7.28  | **26.26**  |\n| PuLID-FLUX*    |**12.84**| 14.89| 16.86| 12.95 | **44.7**  |\n| InfineteYou*      | **8.34**| 10.23| 13.53| 9.03  | **32.79**  |\n\nWhile we acknowledge that introducing extra calculation can lead to longer runtime, we also emphasize that most training-free methods introduce extra processes, including inversion operation, denoising processes, and extra attention calculation. Plus, comparing to the fact that most encoder-based personalization methods require a long range time of data collection, preprocessing and tuning (this may consume a large array of GPUs and thousands of GPU-hours), our training-free method provides an innovative perspective for prompt consistency improvement by the knowledge in themselves, without the need for designing new model architecture and dataset curation. \n\n## 2. Robustness of Segmentation Model\nThanks for pointing this out. We would like to address your concern from two aspects:\n\n+ **Robustness of FreeCure when dealing with inaccurate masks.** FreeCure demonstrate consistent performance even when masks are inaccurate. To validate this, we design two-way validation experiments that introduce inaccuracy to attribute masks. First, we apply a dilation operation on masks, which might include some area which does not belong to the target attribute area. Second, we apply an erosion operation, which might cause loss of some useful information about attributes. To create significant distortion for masks, we set the receptive field of both dilation and erosion to 25 pixels. We conduct experiments on several baselines, and the table below reports the results for respective prompt consistency (PC) and ideneity fidelity (IF):\n\n> **Table 2: PC and IF Performance for Inaccurate Masks**\n|            | PC with original masks | IF with original masks | PC with dilated masks | IF with dilated masks | PC with eroded masks | IF with eroded masks |\n|---|----|------|---|----|----|----|\n| InstantID  | 23.62    | 62.01   | 23.45  | 62.23  | 23.21  | 62.23  |\n| PhotoMaker | 24.91   | 50.15   | 24.87  | 50.01    | 24.34 | 50.44   |\n| PuLID-SDXL | 26.05  | 56.95  | 25.51  | 56.34 | 25.36  | 57.01 |\n| PuLID-FLUX | 24.78   | 72.61  | 24.52   | 72.39   | 24.32  | 72.97  |\n\nThe results demonstrate that identity fidelity (IF) remains largely consistent across both dilated and eroded mask conditions, with slight improvements observed in eroded masks. This occurs because mask erosion preserves certain attributes similar to the reference image. As for prompt consistency, both dilation and erosion cause minimal degradation, though erosion exhibits marginally greater impact. We consider that this stems from cases where fine-grained attribute restoration (e.g., iris color, earrings) may fail when the mask completely degenerates due to erosion.\n\nGenerally, the performance degradation of FreeCure due to inaccurate masks is limited. Moreover, in practical applications, specialized segmentation models (e.g., BiSeNet, Segment Anything) exhibit high robustness, making failure cases caused by mask inaccuracies even rarer than observed in these deliberately designed experiments. This further validates the inherent robustness of the FreeCure framework.\n\n+ **Automatic Extraction for Attribute Masks.** FreeCure’s mask extraction process is auto-driven without any case-specific design.  Our system employs a lookup table encompassing a broad spectrum of facial attributes or other attributes to align with the input prompt. When target attributes are matched, the face parsing model (e.g., BiSeNet) or Segment Anything Model (SAM) automatically generates the corresponding masks in a fully automated manner. Specifically, BiSeNet inherently predicts masks for all facial attributes, so only the matched masks need to be extracted. In the Segment Anything configuration, the model takes the matched attributes as input to produce the final detection mask. A merging process will gather all masks together and generate a mask relevant to all attributes: \n$$\n\\mathcal{M}(j,k) = max(\\mathcal{M}_{i}(j,k)).\n$$\nThanks to these models’ expertise and generalization capabilities in segmentation, the entire process operates without manual intervention, eliminating the need for specialized handling of particular attributes.\n\n## 3. Engineering Implementation\nFirst, as discussed in **Section 3** of our submission, our architectural innovation on self-attention is grounded in solid theoretical foundations: the application of attention theory in diffusion models – where cross-attention controls identity injection while self-attention regulates spatial feature arrangements. This constitutes FreeCure’s core rationale for achieving attribute improvement without compromising identity fidelity. Thus, our method is driven by clear motivations and rigorous theoretical reasoning, rather than accidental parameter tuning or architectural trial-and-error. Second, our approach does not rely on ad hoc parameter design, demonstrating its generalizability as a performance-enhancing framework rather than case-specific engineering. As evidenced throughout our experiments, we maintained identical hyperparameter configurations across all baselines ($\\omega=2.0$ and $\\gamma=0.5$). These two aspects collectively distinguish our method from engineering-driven solutions.\n\n## 4. Implementation details on FLUX\nThanks for pointing out this. The implementation of FASA on FLUX's full-attention is almost identical to U-Net's self-attention but with some slight differences in mask strategy. The input of full-attention is a joint sequence of noisy latent and text condition: $[X; C], X\\in \\mathbf{R}^{b\\times l\\_1 \\times h}, C \\in \\mathbf{R}^{b \\times l\\_2 \\times h}$. Since PD and FD share the identity text condition, we extract noisy latent relevant $K\\_f^X, V\\_f^X$ from the FD process and concatenate it to $K\\_p^X, V\\_p^X$ of the PD process, while $Q$ keeps consistent:\n$$\n\\hat{Q\\_p} = [Q\\_p^X; Q\\_p^C]; \\hat{K\\_p} = [K\\_p^X;K\\_p^C;K\\_f^X]; \\hat{V\\_p} = [V\\_p^X;V\\_p^C;V\\_f^X]\n$$\nThe main difference lies in the mask strategy. Aligning with U-Net's implementation, we apply the all-one matrix to PD's original attention part. Specially, we assign scaling masks $\\omega \\mathcal{M}$ to attention map of $Q\\_p^X and K\\_f^X$ and assign a all-zero mask to attention map of $Q\\_p^C and K\\_f^X$ to preserve PD's original full-attention patterns:\n$$\n\\mbox{FASA}(\\hat{Q\\_p}, \\hat{K\\_p}, \\hat{V\\_p}) = \\mbox{Softmax}(\\frac{\\mathcal{M}(\\omega)\\_{flux}\\odot \\hat{Q\\_{p}}\\hat{K\\_p}^{T}}{\\sqrt{d}})\\hat{V\\_p}\n$$\n\n$$\n\\mathcal{M}(\\omega)\\_{flux} = \\begin{pmatrix}\n  \\mathbf{1}\\_{l\\_1\\times l\\_1} & \\mathbf{1}\\_{l\\_1\\times l\\_2} & \\omega \\mathcal{M}\\_{l\\_1\\times l\\_1} \\\\\\\\\n  \\mathbf{1}\\_{l\\_2\\times l\\_1} & \\mathbf{1}\\_{l\\_2\\times l\\_2} & \\mathbf{0}\\_{l\\_2\\times l\\_1}\n\\end{pmatrix}\n$$\nThis mask pattern highly resembles FASA's implementation in U-Net, which injects attribute-aware features from FD to PD while keeping the original full-attention's pattern. We will update this formula for FLUX in the updated version of the submission."}, {"review_id": "ZHfgBXW8xm", "reviewer": "Reviewer_tTVJ", "summary": "This paper aims to address the balance between identity preservation and prompt consistency in personalization diffusion models. \nFirst, this paper figures out that the limited prompt consistency comes from the ID embeddings in cross-attention. To address it, this paper propose a training-free framework that first generate faces with/without id embeddings, and then modify the self-attention module to restore facial attributes. The proposed method can be seamlessly integrated into any personalization models, and bring a significant improvement on prompt alignment.", "questions": "Please address the weaknesses above, especially weaknesses 1 and 2.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper presents an insightful finding that well-tuned personalization models can retain the intrinsic capabilities of the foundation model. This suggests that degraded prompt consistency may stem from other sources rather than from the model itself — an observation that could influence future personalization strategies.\n\n2. The proposed method  can be seamlessly integrated into any encoder-based tuning-free personalized models, suggesting strong generalization ability. \n\n3. The results are promising, as it significantly improves the prompt consistency. Although the IF metric becomes lower, I totally understand, as this metric is biased by copy-paste behavior to some degree. \n\n4. This paper provides clear technical details in terms of implementation and detailed ablation studies to verify the effectiveness each component. \n\n**Weaknesses**\n\n1. To my best knowledge, the balance between ID preservation and prompt consistency is a general  a general challenge beyond facial attributes. Thus, it is preferred that the proposed method can also. Currently, it is limited by the face parsing model and thus can only handle facial attributes. What happens for failure cases \n\n2. The idea that using zero id embedding vectors is similar to the classifier-free guidance (CFG) strategy. Most personalization models already takes a 3-way strategy that set id vectors to 0 and text embeddings to zero separately. Thus, it would be better to provide details of the cfg setting in all proposed modules. For example, does the dual inference module already use 3-way cfg? Plus, this paper should also highlight the difference between cfg strategy and the proposed dual inference as they all share the idea of zero face embeddings. \n\n3. It seems that the proposed method requires the identity embedding is injected into the model via cross-attention. It would be better if this paper could share some thoughts on full self-attention DiT models.", "was_revised": true, "final_justification": "Thanks for your detailed response. The rebuttal solves most of my concerns. \n\nRegarding the cfg comparison, this paper applies the 2-way cfg for baselines (in Table 3). However, when we have multiple conditions, a common way is to use the 3-way cfg, which also has a branch that sets only one condition to empty (See Equation 3 and Appendix B in [A]). By adjusting the two cfg parameters, the prompt consistency and reference image preservation can be easily adjusted. I'd like to see the discussion between the proposed method and the 3-way cfg, not just the 2-way cfg baseline. \n\n\n[A] Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. \"Instructpix2pix: Learning to follow image editing instructions.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.", "rebuttal": "Thanks for your constructive suggestions and positive comments on our insightful finding, strong generalization ability and promising results. We will raise the following points to address all your concerns.\n\n## 1. More Diverse Segmentation Models and Attributes\nThank you for raising this point. First, our method can leverage any attribute segmentation model to handle different types of features. In **Section A.2.4** of the supplementary materials, we discuss the performance of general semantic segmentation models (e.g., SAM) within our framework and address attributes that fall outside facial features. It is important to note that since most current encoder-based facial personalization methods extract and use the facial region as a conditional input, the copy-paste issue is most pronounced in facial attributes—which is the primary focus of our paper. In contrast, non-facial attributes, which are not subject to additional constraints during training, exhibit a less significant copy-paste effect. \n\nHowever, we fully understand your concern regarding the completeness of this discussion and have conducted supplementary experiments to address it. Following PuLID’s prompt set, we evaluated prompts involving common objects, clothing, and other typical non-facial attributes, using Segment Anything for mask generation. The experimental results across several baselines are as follows:\n\n> **Table 1: Additional Prompt Set with Non-facial Attributes**\n|   | new prompts                        |\n|---|------------------------------------|\n| 1 | a person wearing a *black headphone* |\n| 2 | a person wearing a *doctoral hat*    |\n| 3 | a person holding a *beer cup*        |\n| 4 | a person wearing a *spacesuit*       |\n| 5 | a person wearing a *yellow scarf*    |\n\n> **Table 2: Perfomance on Additional Prompt Set with Non-facial Attributes**\n|                           | PC    | IF    | Face Div. | IF x PC (hMean) |\n|---------------------------|-------|-------|-----------|-----------------|\n| Face2Diffusion            | 22.87 | 41.08 | 41.34     | 29.38           |\n| Face2Diffusion + FreeCure | **23.34 (+2.06%)** | 40.75 (-0.80%)| **41.91 (+1.38%)**    | **29.68(+1.01%)**|\n| InstantID                 | 22.43 | 65.55 | 47.73     | 33.42           |\n| InstantID + FreeCure      | **23.51(+4.81%)** | 65.34(-0.32%) | **48.06(+0.69%)**  | **34.58(+3.46%)**|\n| PhotoMaker                | 24.67 | 52.14 | 46.02     | 33.49           |\n| PhotoMaker + FreeCure     | **25.37(+2.84%)** | 51.77(-0.71%) | **46.58(+1.21%)**| **34.05(+1.67%)** |\n| PuLID (SDXL)              | 27.43 | 59.41 | 41.97     | 37.53           |\n| PuLID (SDXL) + FreeCure   | **28.13(+2.55%)** | 59.03(-0.64%) | **42.05(+0.19%)**| **38.10(+1.52%)**|\n\nExperimental results indicate that the facial personalization model exhibits a weaker copy-paste effect in non-facial regions, leading to less pronounced improvements in prompt consistency compared to the facial attributes specifically addressed in our submission. However, we observed that FreeCure still demonstrates significant restoration effects for attributes such as headphones and hats, which are in close proximity to the face (additional comparative images will be included in the updated version). Moreover, since the restoration occurs outside the facial region, FreeCure has a more negligible negative impact on identity fidelity. In summary, while our method remains applicable to various non-facial attributes due to its generalizability, the optimization effects are less pronounced but also noticeable compared to facial features. \n\n**Analysis of Failure Cases**. Although FreeCure shows consistent improvements in wide range of baselines and attributes. We observe that when enhancing transparent objects (e.g., cups), the restored images may exhibit optical distortions in transparent regions. We will include a discussion of this failure case in the Limitations section and explore potential optimizations for such cases in future work.\n\n## 2. Comparison with Classifier-free Guidance\nWe agree that this detail should be clarified. First, all our experiments do not modify the classifier-free guidance configuration in each baseline's official implementation, matching with our method's plug-and-play manner. That is to say, PD and FD process works when classifier-free guidance is included. For original inference with classifier-free guidance, the batch-wise information can be shown in the following table:\n\n> **Table 3: CFG Condition before Applying Dual Inference**\n| Batch Index| 0| 1|\n|--|--|--|\n| Condition Type | {${\\emptyset}\\_{id}, {\\emptyset}\\_{txt}$} | {${C}\\_{id}, {C}\\_{txt}$} |\n\n>  **Table 4: CFG Condition after Applying Dual Inference**\n| Batch Index    | 0| 1| 2| 3|\n|---|--|--|-----|--|\n| Condition Type | {${\\emptyset}\\_{id}, {\\emptyset}\\_{txt}$}  | {${\\emptyset}\\_{id}, {\\emptyset}\\_{txt}$}  | {${C}\\_{id}, {C}\\_{txt}$} | {${\\emptyset}\\_{id}, {C}\\_{txt}$} |\n\nwhere ${C}\\_{id}$ \\& ${C}\\_{txt}$ denote original identity and prompt conditions and ${\\emptyset}\\_{id}$ \\& ${\\emptyset}\\_{txt}$ denote the null or zero-value conditions. This is similar to the situation that dual inference in running an inference with a batch size equal to 2 (the function of *chunk*() will operate correctly). Without the techniques, attention manipulation techniques like FreeCure, FD, and PD are independent. What should be emphasized is that such independence helps us to identify the foundational knowledge where high prompt consistency still exists in these personalization models.\n\nThe core difference between CFG and FreeCure's dual inference lies in how they use zero face embeddings in their corresponding paradigm. First, during the training process of encoder-based personalization models with CFG, their basic target is still to reconstruct the reference faces. That is to say, even when the model receives a zero face embedding during some training step, their learnable modules (e.g., adapters, learnable embeddings) do not explicitly receive knowledge or constraints from the zero face embedding's denoising process. This also explains why nearly all encoder-based personalization models still face the copy-paste problem, even though the CFG strategy is introduced. In contrast, FreeCure uses zero face embedding to explicitly trigger information from the denosing process with zero face embedding (e.g., via FASA's self-attention map manipulation) and proposes a mechanism to inject such information into the original personalization process to improve its prompt consistency. We hope the explanation above can highlight the core difference and address your concerns.\n\n## 3. Implementation details on Full-attention of DiTs\nThanks for pointing out this. The implementation of FASA on DiT's full-attention is almost identical to U-Net's self-attention but with some slight differences in mask strategy. The input of full-attention is a joint sequence of noisy latent and text condition: $[X; C], X\\in \\mathbf{R}^{b\\times l\\_1 \\times h}, C \\in \\mathbf{R}^{b \\times l\\_2 \\times h}$. Since PD and FD share the identity text condition, we extract noisy latent relevant $K\\_f^X, V\\_f^X$ from the FD process and concatenate it to $K\\_p^X, V\\_p^X$ of the PD process, while $Q$ keeps consistent:\n$$\n\\hat{Q\\_p} = [Q\\_p^X; Q\\_p^C]; \\hat{K\\_p} = [K\\_p^X;K\\_p^C;K\\_f^X]; \\hat{V\\_p} = [V\\_p^X;V\\_p^C;V\\_f^X]\n$$\nThe main difference lies in the mask strategy. Aligning with U-Net's implementation, we apply the all-one matrix to PD's original attention part. Specially, we assign scaling masks $\\omega \\mathcal{M}$ to attention map of $Q\\_p^X and K\\_f^X$ and assign a all-zero mask to attention map of $Q\\_p^C and K\\_f^X$ to preserve PD's original full-attention patterns:\n$$\n\\mbox{FASA}(\\hat{Q\\_p}, \\hat{K\\_p}, \\hat{V\\_p}) = \\mbox{Softmax}(\\frac{\\mathcal{M}(\\omega)\\_{flux}\\odot \\hat{Q\\_{p}}\\hat{K\\_p}^{T}}{\\sqrt{d}})\\hat{V\\_p}\n$$\n\n$$\n\\mathcal{M}(\\omega)\\_{flux} = \\begin{pmatrix}\n  \\mathbf{1}\\_{l\\_1\\times l\\_1} & \\mathbf{1}\\_{l\\_1\\times l\\_2} & \\omega \\mathcal{M}\\_{l\\_1\\times l\\_1} \\\\\\\\\n  \\mathbf{1}\\_{l\\_2\\times l\\_1} & \\mathbf{1}\\_{l\\_2\\times l\\_2} & \\mathbf{0}\\_{l\\_2\\times l\\_1}\n\\end{pmatrix}\n$$\nThis mask pattern highly resembles FASA's implementation in U-Net, which injects attribute-aware features from FD to PD while keeping the original full-attention's pattern. We will update this formula for FLUX in the updated version of the submission."}]

Author rebuttal:

Meta-review: This paper introduces FreeCure, an innovative and practical training-free framework designed to tackle a well-known trade-off in facial personalization: maintaining a subject's identity while accurately following textual prompts. The authors point out that strong identity embeddings can suppress the model's ability to render other attributes. Their solution leverages a dual-inference paradigm, using the foundation model's superior attribute control to guide the personalized generation process without any retraining. The method is generally simple, effective, and broadly applicable to popular models like Stable Diffusion and FLUX. Given the strong empirical analysis, positive results, applicability to broad models, and the overall positive feedback from the reviews after a thorough discussion period, this paper gets a clear accept recommendation from me.

I would also like to note that the paper was flagged for an ethics review regarding the source of its non-celebrity dataset and a discussion of potential misuse, but the authors successfully addressed these points by clarifying their data sources and expanding the limitations section, satisfying the ethics reviewer. I'm confident that the final camera-ready version, incorporating the thoughtful suggestions from all reviewers, will be a strong and well-rounded contribution to the conference.
Decision: Accept (poster)