Mitigating Semantic Collapse in Generative Personalization with Test-Time Embedding Adjustment
Authors: Anh Tuan Bui
Thuy-Trang Vu
Trung Le
Junae Kim
Tamas Abraham
Rollin Omari
Amardeep Kaur
Dinh Phung
Venue: ICLR 2026 Poster (2026)
Keywords: Generative Personalization
Diffusion Models
Test-Time Computing
Primary area: generative models

Abstract: In this paper, we investigate the semantic collapsing problem in generative personalization, an under-explored topic where the learned visual concept ($V$) gradually shifts from its original textual meaning and comes to dominate other concepts in multi-concept input prompts. This issue not only reduces the semantic richness of complex input prompts like "a photo of $V$ wearing glasses and playing guitar" into simpler, less contextually rich forms such as "a photo of $V$" but also leads to simplified output images that fail to capture the intended concept. We identify the root cause as unconstrained optimisation, which allows the learned embedding $V$ to drift arbitrarily in the embedding space, both in direction and magnitude. To address this, we propose a simple yet effective training-free method that adjusts the magnitude and direction of pre-trained embedding at inference time, effectively mitigating the semantic collapsing problem. Our method is broadly applicable across different personalization methods and demonstrates significant improvements in text-image alignment in diverse use cases. Our code is published at \url{https://github.com/tuananhbui89/Embedding-Adjustment}.

Published as a conference paper at ICLR 2026
MITIGATING SEMANTIC COLLAPSING PROBLEM IN GENERATIVE PERSONALIZATION WITH TEST-TIME EM- BEDDING ADJUSTMENT
Anh Bui' Trang Vul Trung Le' Junae Kim²
Tamas Abraham Rollin Omari Amar Kaur² Dinh Phung'
Monash University
Defence Science and Technology Group, Australia
And CreamBooth
And Creambboth with TEA
ReVersion
ReVersion with TEA
EnsyContol
CasyContral with TEA
Prompt ' weaing ploses to ding ouf
Promee Cal V conor ina parite! seam Mealt panerelzed soncept Ccaeed
Pronct "V" conttuctionn ouff Maning someed conatrction
lous Dpo and risting conteat a
Dissiiilusion
Plssion with TEA
Texkal Inverabo slon
Testual Invension with TEA
Cuatom Diflurion
Custom Dillusion
TEA
Phanpt:
sunfiower
ProFUS Vn canyor"
Pronpt: with
Mouth Mactoo 100000000000 consoat (hant)
Beam Meaig Sontec goand conyen!
Iital Miric www.otalooo.ooooom con Y". (160))
Figure I: Our Test-time Embedding Adjustment (TEA) method consistently enhances text-image alignment across diverse personalization approaches (Textual Inversion, DreamBooth, and their variants) and architectures (Stable Diffusion, Flux). Notably, TEA also counteracts the anti- personalization effect of Anti- motiDrreaBooth and restores the protected concept.
ABSTRACT
In this paper, we investigate the semantic collapsing problem in generative person- alization, under-explored topic where the leamned visual concept (V") gradually shifts from its original textual meaning and comes to dominate other concepts multi-concept input prompts. This issue not only reduces the semantic richness of complex input prompts like "a photo of /' wearing glasses and playing gui- tar" into simpler, less contextually rich forms such as photo of V- but also leads to simplified output images that fail to capture the intended concept. We identify the root cause as unconstrained optimisation, which allows the learned embedding V* to drift arbitrarily in the embedding space, both in direction and magnitude. To address this, we propose simple yet effective training-free method that adjusts the magnitude and direction of pre-trained embedding inference time, effectively mitigating the semantic collapsing problem. Our method is broadly applicable across different personalization methods and demonstrates significant improvements in text-image alignment in diverse use cases. Our code published at wttttss://// thub. com htmmmmuuaaaate nhoui 89/ Embedd ng- Ad jus t.eet.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Text-to-image (T2I) diffusion models have achieved unprecedented fidelity and flexibility in image generation and sparked growing interest in generative personalization (Gal et al. 2022 Ruiz et al. 2023). This emerging problem aims to generate images of a specific user-defined visual concept (e.g. a particular person, pet, or object) in different contexts (e.g. on the beach) using a small set of user-provided reference images paired with text prompts describing the desired context. The core objective is to generate visually compelling images that faithfully preserve the unique characteristics of the personal concept while remaining semantically aligned with the textual prompt. Despite recent progress, misalignment between the generated image and the textual prompt is still a major concern.
robust generative personalization method should allow the user-defined visual concept to be com- posed with arbitrary contexts in the text prompt without losing fidelity expressiveness. However, existing approaches often struggle to maintain prompt and generated image alignment, particularly with complex or multi-concept prompts (Kong al., 2024: Zhu et al. (025). As illustrated in Figure I the user-defined concept can overpower or distort other elements in the prompt, leading to unsatisfactory generations. This issue has commonly been attributed to language drift- a phenomenon where the model gradually forgets how to generate its pretrained concepts and instead becomes overly focused on the user-defined ones (Lee et al. 2019). This drift typically stems from overfitting limited number of reference images (Ruiz et al. 2023). Beyond overfitting, other factors have also been attributed, such as the limited expressiveness textual embeddings which compress complex visual concepts into single tokens (Zhang et al., 2023) Mou et al. 2024) and the entangled nature of reference sets, where samples may contain co-occurring objects or irrelevant contextual features (Avrahami et al. 2023; Jin et al. 2024). While misalignment between textual prompt and generated images is a well-recognized challenge in generative personalization, with numerous mitigation strate- gies proposed, including latent optimization (Rassin et al. (023). regularization (Han et al. 2023
Qiu et al. 2023 Arar et al. 2024) and concept disentanglemer (Motamed et al. 20244] Huang et al. 2024a), the underlying causes and mechanisms remain relatively underexplored.
To gain deeper insight into the misalignment problem, this paper presents an empirical investigation
into the dynamics of learned personalised tokens throughout the personalisation process. Our analysis reveals an intriguing phenomenon: the personalised token gradually loses its original textual semantic meaning while acquiring increased visual information from the reference images. When such token is used in a prompt with rich descriptive text, the generated image becomes disproportionately semantically dominated by the personalized concept, often neglecting the other intended elements in the prompt. For example. in Figure I if one learns token V to represent a particular cat, a prompt like "V" in grand canyon" " may yield an image that vividly depicts the cat but fails to properly render the grand canyon background. Essentially, the prompt's 's semantic complexity collapses to simplified form centred on V We refer to this phenomenon as the semantic collapsing problem (SCP). Unlike language drift where the personalized model overfits to learned concept and loses its ability to generate other pretrained concepts, SCP arises when the personalized embedding collapses, no longer retaining meaningful textual semantics and instead encoding primarily the visual information of the reference concept. While SCP might not severely affect trivial prompts (c.g., "a photo of V*"), undermines the compositionality of the T21 diffusion model on complex prompts.
We identify the root cause of SCP as unconstrained optimisation, which allows the learned embedding to drift arbitrarily in the embedding space, both in direction and magnitude. We propose a simple yet effective remedy: a training-free test-time embedding adjustment (TEA) strategy that realigns the learned concept embedding with its original semantic meaning at inference time. The key idea is to calibrate the embedding's 's magnitude and direction to be closer to that of its reference concept. This adjustment is done without altering the model weights or requiring any additional training. The embedding is modified on on-the-fly before image generation. By enforcing a small rotation and rescaling in the text encoder's 's latent space, we constrain the personalized token to behave more like a regular word, ensuring that it contributes to the image generation in balance with other tokens in the prompt. Our approach is lightweight and broadly compatible with almost all existing personalization methods and demonstrates significant improvements in text-image alignment in diverse use cases. Surprisingly, beyond improving personalization quality, we also uncover a surprising vulnerability in anti- personalization frameworks like mati-DreamBooth (Van Le et al. (023) under the lens of SCP. In particular, when applying TEA to models poisoned by Anti- Mati-DreamBooth we find that TEA can partially reverse adversarial corruption and restore more faithful generations of the protected
2

---

Published as a conference paper at ICLR 2026
concept (see Figure 1). This result highlights false sense of security in current anti anti-personalization defenses and provides new insights into their limitations.
In summary, our contributions in generative personalization problem, and provide an empirical analysis of its existence in both textual and image spaces. Our analysis reveals its root cause, which is the unconstrained optimisation. We propose the test-time embedding adjustment, a novel solution that requires no additional training, mitigate SCP by aligning the learned embedding's direction and norm with its original semantic concept. @ We demonstrate that our proposed approach can be applied into almost all existing personalization frameworks such as Textual Inversion (Gal al. 2022). DreamBooth (Ruiz et al. (023). Custom Diffusion (Kumari et al. (023)) EasyControl (Zhang et al. (025). Versinn (Huang et al. 2024b), and CifssDiffusion (Huang et al., (024a) to significantly improve image
follows: We define the semantic collapsing problem (SCP)
origgn
generation in complex prompts across a wide range of scenarios. 4 We show that TEA unexpectedly mitigates adversarial corruption introduced by Anti-DreamBooth, revealing an overlooked weakness in anti-personalization defences.
2
SEMANTIC COLLAPSING PROBLEM IN GENERATIVE PERSONALIZATION
2.1 TERMINOLOGIES
Given set of personal images X' and a pre-trained T2I model Eg. the goal of generative personalization is to identify a textual embedding v" associated with specific verbalizable keyword V (e.g., 'sks', '<new>', etc.). This keyword represents the implicit visual concept shared in the reference set X, enabling the model to generate images with the personal concept using any textual prompt p containing the keyword V*, e.g., [p, V*J= 'A photo of V* playing on a beach', where L-.J the sentence construction operator. We denote the reference semantic concept of the keyword V*.
We denote M is the embedding matrix of the entire vocabulary of the text encoder т, and Mx is specific row of the matrix corresponding to the token k E vocab,, where k can be V (i.e., v² = My..) ) or any arbitrary token like 'dog'. 'cat', etc. We denote à = G(p) as the image generation model that takes a prompt p input and outputs image ft.
2.2 SEMANTIC COLLAPSING PROBLEM
Problem Statement. The semantic collapsing problem (SCP) refers to the phenomenon where the keyword V loses its original textual semantic meaning while acquiring increased visual information from the reference concept during the personalization process. As a result, a prompt [p. V. consisting of a context p and the concept V*, becomes dominated by the learned concept V eventually collapsing to a simplified form.
Why SCP Matters. We argue that SCP may not pose a serious issue for simple prompts, e.g.. (p. V*J = 'a photo of V*', where the primary information conveyed is still the visual concept V" However, for more complex prompts where the context p contributes meaningfully to the overall semantics, such as 'a photo of V wearing glasses and writing in a red notebook', SCP becomes more problematic illustrated in Figure 10 In such cases, the generated image is more likely to be dominated by the learned concept V* and less likely to reflect the intended context p.
Comparison with Other Challenges in Generative personalization. First, emphasise that SCP is not specific to the two representative methods we study (TI and DB), but is a general issue in personalization. Second, SCP is distinct from other recognised challenges (ref. Section (A) such as the language drift problem (Ruiz et al. 2023). which describes how the personalized model eœ overfits a learned concept and loses generalisation. Third, while SCP contributes the broader challenge of misalignment between generated images and prompts, a major concern in generative personalization, it stems from specific cause: the unconstrained optimisation of the embedding during personalization, which has not been thoroughly studied in prior work.
2.3 EMPIRICAL HUNTING FOR SCP
In this section, present empirical evidence supporting the existence of the semantic collapsing problem and its impact on generation quality. Our key findings are as follows:
Existence of SCP. SCP exists in the textual domain, where the prompt [p, V*] is dominated by the learned embedding V* and the semantic meaning of the entire prompt gradually collapses to the learned embedding V*, i.e., r(Up, V*J) + T(V*).
3

---

Published as a conference paper at ICLR 2026
12.0
4-
olP-
10G Sec soicce ath Pur)
inter Sec and intre (betterces
122
33.50
20-
un .g)
23.25-
ote Arl
28
13.00-
26-
12-
30.25
20"
500
1000
500
1500
2000
2500
3300
Time Step
Time Step
Figure 2: (a/left) The inter-set distance d(Pv., Po) and intra-set distance d(Py-, Py-) over the per-
sonalization process, and (b) The distance between all possible pairs of sets, notably di (Pv pammple)
@ Negative Impact on Generation Quality, SCP leads to the degraddtioon/missiignnnnn in generation quality in the image space, i.c., G(Lp, [*] + G(V)), particularly for prompts with complex semantic structures.
Surprisingly Positive Impact. SCP can also lead to the positive impact on generation quality, particularly for prompts where the concept requires strong visual presence be recognisable.
Root Cause of SCP. SCP arises from unconstrained optimisation during personalization, which leads to arbitrary shifts (both in magnitude and direction) in the embedding of V away from its original semantic concept c.
2.3.1 EMPIRICAL EVIDENCE FOR SCP IN TEXTUAL SPACE
Recall our hypothesis: a keyword V initialised from a concept to capture visual target Uos will lose its semantic meaning and dominate any arbitrary context p when combined into prompt.
To verify this, we propose measuring the difference between V and in the presence of diverse set of contextual prompts A {a1, d2 .... an). These are generated by querying an LLM with the instruction: "Write 200 sentences with diverse topics and contents. Each sentence should be 10-30 words long and must include the keyword c. Sample sentences are provided in Table We then construct two sets of prompts: Pv- [La, V* Ja,eA and Pa {La;, cJ JaEEA, which are used to assess the contextualised difference between V and
To quantify this difference, we compute distances between the sets Pv- and P. using four met- rics: Euclidean, Hausdorff, Mahalanobis, and KL divergence. We also evaluate intra-set variability d(Py.,P Pv.) measuring the separation among items within Py.. The distance metrics are sum- marised in Table We use Textual Inversion (TI) and DreamBooth (DB) to learn a personalized human face concept. The training data comprises 16 images from the CelebA dataset (Liu et al. 2015) (subject ID: 342, 'Henry Cavill'). The embedding v* is initialised using the concept "man" for T1, and 'a photo of sks man' for DB.
Results. Figure a(a) shows the average inter-set and intra-set distances, d(Pv-, Pa) and d(.y.,Pv.). measured over training iterations. It can be that the inter-set distance d( PP) increases steadily over time, indicating that the learned embedding e progressively diverges from its initial textual semantic meaning c. Interestingly, the intra-set distance d(Pv-, Pv-) decreases over time, suggesting that the embeddings of prompts la,, V* within Pv- become less diverse and more similar to one another. This reflects a growing dominance of the learned embedding v* across prompts, effectively overriding the contextual variations in a and becoming the principal semantic component of each prompt.
To further verify this dominance effect, we introduce an additional set of simple prompts, denoted psimple which consists of 200 concise sentences such as 'a photo of a V** 'a portrait of a V etc., where V* clearly the central concept. As shown in Figure 2)b), the distance d(Py., ((Py.,asimpl which captures the difference between complex prompts [p. V* and simple prompts [V'], decreases over time. This trend indicates that the representations of complex prompts become increasingly similar to those of simple prompts in psimple further supporting the hypothesis that v* gradually dominates and collapses the semantic contribution of contextual components.
4

---

Published as a conference paper at ICLR 2026
prompt Srip
Contect Snjpl
Sylt
ander StAp
Contect
Concapt-So
wearng writin
22 22 -
Imsont Poamo Pem.ssAiraaa
72.5
70
67%
18
15
300
Figure 3: Analysis of the SCP TI (left) and DB (right). Aligmment with the ground-truth image (5 ggt) Q) increases over time, while alignment with the contextual part (S($, p) D) decreases.
2.3.2 THE Two-Wa WAY IMPACTS ON PERSONALIZATION
In this section, we extend analysis investigate how SCP impacts image generation quality. Specifically, we generate personalized images å = G(Ip, J* using a list of prompts P (e.g., photo of V* man holding cat'), with 100 images generated per prompt. We evaluate the generation quality using the CLIP-Image-Imag alignment score S ( g.t), which measures the similarity between the generated image & and the ground-truth image Egt of the reference concept. In addition, we compute three TLIP-Text-Image alignment scores: (i) CLIPE or ST = S(i, p) alignment with the contextual part p (e.g., 'holding cat'). ), and (ii) CLIP, or Sj- S , e) alignment with the original concept e (e.g.. "a man'). ). (iii) CLIP/ or S/- = S(ē Lp, el) alignment with the full prompt.
Interestingly, observe that the SCP has both negative and positive effects on generation quality, depending on the nature of the prompt. Our key findings (illustrated in Figure 3 are as follows:
(Unsurprising) The image-to- im-ge alignment score S₁ increases V effectively captures the visual appearance of the target concept. This confirms that the learned embedding V successfully personalizes the visual identity from the reference set.
time (+ line), indicating that
(Surprising Negative Impact) The context-to iome alignment score St. decreases over time (Σ)
line), showing that generated images increasingly lose alignment with the contextual component p. This highlights the negative impact of semantic collapsing: as V* dominates the prompt, the image generator pays less attention to the surrounding context.
(Unexpected Positive Effect) For some prompts, the concept-to-image alignment score Sj. actually increases over time (6 line). Early in training, prompts with (e.g., man) often generate images dominated by the context p. As training progresses, the learned embedding V* reduces this context dominance, strengthening the representation of c. This effect is most evident when demands a strong visual presence (e.g., man in "man writing in red notebook"), where close-up subject renderings naturally downplay the surrounding context.
2.3.3 THE RoOT CAUSE OF SCP
In the previous sections, we demonstrated the semantic collapsing problem in both textual and image generation spaces. In this section, investigate the underlying cause of this phenomenon. Our key findings are summarised below:
① The semantic collapsing problem arises from the unconstrained optimisation process used in Equation and Equation Without any regularisation, the learned embedding V can deviate significantly from the original semantic meaning e in both magnitude and direction. Specifically, the embedding norm becomes much larger ((My~ > MM.. and the cosine similarity between the two drops sharply (cos(My~, MM)+ < 1), leading to a semantic drift.
@ As a result of this semantic shift in V*, the embedding of the entire prompt [p. V is also affected. That is, the prompt embedding becomes nearly identical that of V*, i.e., 7(pp, V J) s T(V*) + T T(pp, cl), which directly manifests the semantic collapsing problem discussed earlier.
To support this analysis, Figure 4 presents a histogram of embedding norms for all vocabulary tokens, along with the norm of V* tracked over the course of optimisation. is evident that the norm of 4 grows significantly, placing in the long tail of the sut.rbbttioonssssttttttuaaina larger than
5

---

Published as a conference paper at ICLR 2026
BC
105
0.
175
0.7
Figure 4: Left: The distribution of the norm of the token embedding M including special token V* Right: The semantic drift of V in term magnitude and direction over time. The adjusted embedding V adjusted is obtained by using TEA with 0.2 and / 1.5. The same phenomenon is observed in DreamBooth as shown in Figure 21
Tplp,
wearinz sctebook
U-Net
5" sominetes
To
Ta (p.c)
T
TEA
pp.e|
Normatizing
man secorin glones and writing in notebcok
Beding
Rotating
Figure 5: (left) TEA framework that adjusts the embedding inference time where both U-Net and text encoder are just personalized pre-trained models. (right) the two stages of TEA: normalization and rotation with SLERP.
standard tokens such as 'man', 'woman', and 'person', and approaching that of special tokens like "<lstartoftext>> or "<lendoftext>>.
is worth noting that in some DrcamBooth- based implementations (c.g., DreamBooth with LoRA in Diffusers (Hugging Face)), the full text encoder is fine-tuned to incorporate the personalized concept, instead of updating dedicated token embedding V* as done in Textual Inversion. In such cases, while the individual embedding vector M is not explicitly altered, the semantic shift still occurs at the prompt level, i.e., r(Ip, V* = r(V*) * c(p,,c)..
Additionally. these DreamBooth implementations already include gradient clipping mechanism (Hugging Face) to constrain parameter updates. However, this method was not designed with semantic stability in mind, and does not prevent cumulative semantic drift in practice. Even when the gradient norm is bounded, the embedding can still gradually shift over successive iterations.
To the best of our knowledge, our work is the first to identify and explain the root cause of the semantic collapsing problem as consequence of unregularised embedding dynamics.
TEST-TIME EMBEDDING ADJUSTMENT FOR SCP
As demonstrated in the previous section. SCP exhibits two-way impacts that vary depending on the nature of the context prompt. This variability makes it challenging to devise universal solution that mitigates the negative effects SCP while preserving its beneficial aspects across all inference prompts. Recall that in the earlier analysis, observed that the learned embedding V* often drifts from its original semantic anchor due to unconstrained optimisation-resullee resulting in significant shifts in both magnitude and direction. This raises natural question: Can reverse this semantic shift at test time by adjusting V*, without modifying the personalization method?
A key advantage of this approach is that it is training-free and can be applied almost all existing personalization methods, regardless of whether it is based on Textual Inversion or DreamBooth. Surprisingly. this simple adjustment proves to be highly effective.

---

Published as a conference paper at ICLR 2026
Embedding Adjustment. Given pre-trained embedding matrix M that includes learned token V (as in Textual Inversion), and a target concept è toward which we wish to regularise, we propose to adjust My- by aligning both its magnitude and direction with Mc. This is achieved by first normalising the vectors and then applying Spherical Linear Interpolation (SLERP) (Shoemake) 1985) interpolate the direction of My- towards Mc which is effective in high-dimensional vector spaces.
My- = sin( (1 - =0) My. + sin sin(a0) Mc.
(1)
(i
(i)
Here, 0 the angle between the normalized vectors M. and My-, and a E [0. 1] controls the rotation factor, where the bigger a is, the more the embedding is rotated towards Mc. The normalisation vectors are defined as My. to control the magnitude of the embedding relative to the reference concept
Mell [|
M._ where B is the scaling factor Ady- and M. - 3 Melll TM.
In Dreambooth-baseee personalization, because the embedding matrix M is not updated during the optimisation, we propose to adjust at the prompt level instead of the token level as illustrated in Figure 5 More specifically, given prompt [y, V*J and target prompt [p.c], we obtain the two embeddings T(Lp, [) and T([p, cl) from the text encoder To and then adjust the embedding of [p, V* by using the above equation every token in the prompt.
f(Lp, D = sin ))).)) F(pp, V DD + sin(a8.)
F(Ip. c]][
(2)
sin(
(0.)
where indexes each token in the prompt, and 0, is the angle between the i-th token embeddings of the two prompts after normalisation. This method enables a test-time adjustment of semantic drift without retraining, making it lightweight and broadly applicable solution to mitigating SCP effects.
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of TEA on addressing the SCP across six represen- tative and recent personalization methods, two architectures (Stable Diffusion and Flux) and three datasets (CS101, CelebA, and Relationship) consisting of total 22 concepts. Due to the space limit, present the main results here and refer readers to Appendix [nd additional quantitative and qualitative findings. Full reproducibility details are available in the GitHub repository.
4.1 EXPERIMENTAL. SETUP
Reference Images. We use subset of 9 concepts from the CustomConceptt11 (CC101) dataset as Kumari et al. (023). each of which has 3-15 images, including 'Barn', 'Tortoise plushy', "Teddy-Bear', 'Wooden Pot', 'Dog', 'Cat', 'Flower', "Table', "Chair" subjects. For the human concept, we use a subset of 10 concepts from the CelebA-HQ dataset (Liu et al. 2015), which includes 10 identities with 10-15 images per subject. Sample images are shown in Fig. 15 and I6
Prompts. We collect complex prompts from the CC101 dataset, where each prompt contains at least two concepts, e.g., watercolor painting of V tortoise plushy on mountain'. For the human concept, we create list of 17 prompts, where each prompt contains the main concept and a complex context/action, e.g., 'A photo of a V* wearing glasses and writing red notebook'. The prompts can be found in Table 4
Metrics. In addition to the CLIP-T and CLIP-I alignment scores introduced in Section 2.3 we also use the DINO image-image alignment score (Caron et al. 2021) to evaluate the alignment between the generated images and the reference images.
We also use VL.M-based evaluation metries, VIM-P and VLM--I using VLM (i.e., GPT-4o-mini) as the judge to assess the alignment between the generated images and the reference images and the prompts, respectively. Both metrics output score between 0 and 4, where 0 means there are no correspondence between the generated image and the reference image (VLM-D) or input prompt (VLM-P), while 4 means the perfectly matches. The final score for each metric is obtained by averaging all inference prompts and samples and normalizing to the range [0, 100%]. We include the full system prompts and evaluation scripts in the Github repository.
4.2 EVALUATION RESULTS
7

---

Published as a conference paper at ICLR 2026
Table Performance over EasyControl (ES) OminiControl (OC) and Re VeVession (RV) when integrat- ing with our TEA. Qualitative results are shown in Figures 28
29
31
Method
CLIPT 1
CLIF 1
CLIPAT
DINO-1
VLM-P1
VLM-I 1
CC101 Pet Deg
ES
ES+TEA
18.72 (+0.18)
26.11 (+0.09)
64.56 (+3.23) 48.32 (+4.61) 66.50
77.25 (+3.25)
CC101 Plushie Teddybear
ES
20.48
26.80
81.64
49.08
78.00
80.25
ES+TEA
20.61 (40.13)
27.3 (+0.50)
82.84 (+1.20) 51.17 (+2.09)
80.25 (+2.25)
81.50 (+1.25)
Subject Clock
OC
18.11
23.90
81.37
32.41
67.50
62.25
OC+TEA
18.78 (+0.67)
23.98 (+0.08)
83.10 (+1.73) 34.48 (+2.07)
71.75 (+4.25)
64,50 (+2.25)
Subject Oranges
OC
21.49
27.62
70.43
30.33
68.50
53.00
OC+TEA 21.60 (+0.11)
27.70 (+0.08)
71.90 (+1.47) 31.64 (+1.31) 70.00 (+1.50) 55.50 (+2.50)
Subjeet Penguin
OC
20.30
31.61
78.58
45.59
86.25
83.25
OC+TEA 20.33 (+0.03) 32.02 (+0.41)
80.64 (+2.06) 49.37 (+3.78)
90.50 (+4.25)
86.75 (+3.50)
Relationship A <Carved by> B
RV
25.64
27.74
N/A
N/A
N/A
N/A
RV+TEA 27.84 (+2.20)
30.17 (+2.43)
N/A
N/A
N/A
N/A
Relationship <Inside> N/A
RV
24.97
27.87
N/A
N/A
N/A
RV+TEA
25.15 (40.18) 28.40 (+0.53)
N/A
N/A
N/A
N/A
Relationship A <Painted on> B
RV
23.98
30.07
NIA
NIA
N/A
N/A
RV+THA 24.38 (+0.40) 30.35 (+0.28)
N/A
N/A
N/A
N/A
We evaluate the effectiveness of Test-time Embedding Ad- justment (TEA) when combined with various personaliza- tion baselines, including Textual Inversion (TI) (Gal et al.)
Table 2: Improvement of our TEA over its baseline counterparts on CC101 and CelebA datasets (positive or negative). Details are shown in Tables and
2022), DreamBooth (DB) (Ruiz et al. 2023), CustomD- iffusion (CD) (Kumari et al.) 2023), and ClassDiffusion (CL) (Huang al. 2024a). As shown in Table 2 TEA con- sistently enhances full prompt alignment (CLIP/) across all methods and datasets. Gains are particularly notable for TI and DB, with increases up to +1.87 CC101, demon- strating that TEA substantially strengthens text-image consistency. Importantly, these improvements hold across diverse concepts (Figure (). indicating that TEA is ro- bust different personalization scenarios, even with fixed, easily chosen hyper-paramene parameters.
Method
CLIP CLIP// CLIP-I DINO-I
CC101
TI+TEA
0.55
0.64
-2.34
-3.81
DB+TEA
0.77
1.87
4.57
0.59
CD+TEA
-0.13
0.34
0.37
0.24
CL+TEA
-0.12
0.42
0.67
1.24
CelebA
TI+TEA
0.33
0.57
-2.41
-1.78
DB+TEA
0.51
0.64
-2.37
-2.27
In terms of visual quality, TEA often improves CLIP-I and DINO-I scores (e.g. +4.57 CLIP-I for DB on CC101), while in some cases particulalyl on CelebA is trade-off, with modest drops in CLIP-I (c.g., -2.37 for DB+TEA). However, qualitative comparisons (Figures 24 23 particularly in Figure 22 show that
CD+TEA
-0.12
0.09
1.56
3.09
CL+TEA
0.22
0.39
-0.69
0.17
TEA reduces distortions and produces coherent, realistic outputs, when quantitative scores dip slightly. This suggests TEA better preserves semantic fidelity while mitigating common artifacts in baseline methods.
Generality across architectures and use cases. We further assess TEA on three state-of-the-ar frameworks beyond the original baselines: EasyControl (Zhang et al. 2025), OminiControl Tan et al. 2025), two Flux-based personalization methods, and Re (Huang et al.) 2024b), which targets compositional relationships, e.g., 'a cat carved by carrot'. As shown in Table I TEA again yields consistent improvements. For EontCon, TEA achieves substantial gains in image-image alignment (+3.23 CLIP-I, +4.61 DINO-I, +3.25 VLM-I), while for Versinn it strengthens prompt fidelity, with up to +2.43 CLIP/. in complex relations such 'carved by". For OniniControl, TEA
8

---

Published as a conference paper at ICLR 2026
improves VLM-P by 4.25 points and VLM-I by 3.50 points, well by 0.41 points in CLIP// and 2.06 points in CLIP,, demonstrating the consistent improvement across different metrics.
confirm these findings: While producing impressive high- quality generations, these SOTA frameworks still suffer from failure cases such as EasyControl producing spurious objects unsafe generations, OminiControl failing control the subject's action, and Version failing to accurately represent relationships. Our TEA corrects these failure cases effectively, significantly improving the prompt fidelity these frameworks without compromising the image-reference fidelity. Again, all these results are obtained in inference time without any additional training or finetuning.
Qualitative results (Figures 28 29 31
Together, these results highlight TEA as a lightweight yet broadly applicable enbancement to person alization. It systematically improves text-image alignment, generalizes across methods, datasets, and architectures, and provides qualitative corrections to baseline failure modes, establishing TEA as robust and versatile component for personalization.
Surprising Impact of TEA on Doti-DreamBooth. Anti-personalizztion (Liang et al. cct al., 2023 Salman et al. 2023) aims to protect users from malicious actors who might exploit personal images to train unauthorized personalized models. The core idea is to apply perturbation (a 'mask") ) to the user's 's data before it shared. Although attackers can still access these masked images, they are prevented from training effective personalized models. One representative approach is Bott (Van Le et al.) 2023), which employs adversarial learning (Szegedy
2023 Van Le
invisible
al. 2013 Goodfellow al. 2014) to generate such masks.
DRLOB-O%
1.0
AntOB-25%
Counre Semilerty CONTIC TA-0-
0.9
Diztacc
0.6
C
200
400
600
800
1000
Steps
Figure 6: Comparison on prompt alignment of our TEA over its baselines counterpart on CC101 dataset. Refer to Table for detailed numbers.
Figure 7: Semantic drift analysis of Dream- Booth trained with Anti-DreamBooth adversarial masks.
Given that background, we analyze mniiDreaaBBoth through the lens SCP. We hypothesize that its adversarial learning process actually amplifies the dominance of the personalized concept V** but with good implications for user privacy, by causing the prompt embedding [p, V*J to drift even further from its original concept [p, cJ. resulting distorted generations of the protected concept V*.
To verify this hypothesis, we conduct a controlled experiment. Given a set benign personal images {=:) we apply Anti-DreamBooth produce masked images [ We then train DreamBooth models using mixtures of masked and benign images, with varying proportions p E {0, 0.2, 1.0), where p corresponds to standard DreamBooth, and p 1.0 uses only masked data. We then analyze the resulting prompt embeddings as in Section 2.3.3] As shown in Figure 7 increasing leads greater embedding drift, evident in the larger norm of []=([P, J) 7 <([,,c] e)) and the lower cosine similarity between T T(Pp. V*J) and T (Lp, c[). This result confirms our hypothesis and provides an interesting perspective on why anti-personalivzatoon works.
Surprisingly, when we apply TEA to DreamBooth models poisoned by Anti-DreamBooth we observe a mitigation effect such that the generated images by TEA are less distorted and more aligned with the so-be-protecter concept V* as shown in Figure & (more results can be found in Appendix C.2) This surprising result reveals an intriguing false sense of security of Dnti-DreamBooth such that despite adversarial masking. the poisoned personalized model still retains traces of the correct/to- be-protected concept V* and the distortion of the generated images is just the consequence of the extreme Semantic Collapsing Problem magnified by the adversarial masking. The distorted

---

Published as a conference paper at ICLR 2026
(g=0.2, (p-0.3,
1p-0.8,
Pronpt
Prounge
GE
Figure 8: Effect of applying TEA to models poisoned by anti-DreamBoohh TEA is able to mitigate the corruption and recover distorted generations of the protected concept, revealing a surprising weakness in noo-hheeaBooth. Additional results and discussion can be found in Appendix
generations bring the false sense of security because the attacker can use TEA to recover partially the to-be-protected concept V*. To the best of our knowledge, this is the first work uncover such counter-intuitive vulnerability of Anti-DreaBBooth and sheds new light on the limitations of current anti-personalizztion defenses.
CONCLUSION
In this paper, we identified the Semantic Collapsing Problem (SCP) in generative personalization, where personalized tokens lose their original semantic meaning and dominate other concepts in complex prompts. We traced this issue to unconstrained optimisation, which allows the learned token embedding to drift in direction and magnitude, disrupting prompt interpretation.
To address this, we proposed a training-free test-time embedding adjustment (TEA) that realigns the personalized embedding with its original semantic context, significantly improving text-image alignment without modifying model weights. Our method is lightweight, broadly compatible with almost all existing personalization frameworks such as Textual Inversion and DreamBooth and their variants, and delivers substantial improvements in prompt consistency and image fidelity across diverse scenarios.
In addition to tackling SCP, we also provided an initial probe into the interaction between TEA and anti personaliizazion. Surprisingly, when applied models corrupted by Anti-DreamBooth, TEA partially mitigates adversarial corruption and recovers more faithful generations of the protected concept. This finding suggests that current defenses may offer false sense of security, opening an intriguing direction for future work at the intersection of personalization and privacy protection.
Overall, beyond introducing SCP and proposing a practical solution, this work lays the founda- tion for further exploration of adaptive embedding adjustments, context-aware constraints during personalization, and new perspectives on the robustness of antippersonalization methods.
ACKNOWLEDGEMENT
This work was supported by the Australian Defence Science and Technology (DST) Group through the Next Generation Technology Fund (NGTF) scheme and the Department Defence, Australia, via the Advanced Strategic Capabilities Accelerator (ASCA) program. Dinh Phung further acknowledged the support from the Australian Research Council (ARC) Discovery Project DP230101 176 and DP250100262. The authors are grateful to the anonymous reviewers for their helpful comments. This research/work was supported by Monash eResearch capabilities, including M3.
REFERENCES
Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, and Ariel Shamir. Palp: Prompt aligned personalization of text-to-image models. In
10

---

Published as a conference paper at ICLR 2026
SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024.
Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-12, 2023.
Anh Bui, Long Vuong. Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, and Dinh Phung. Erasing undesirable concepts in diffusion models with adversarial preservation. NeurIPS, 2024.
Anh Tuan Bui, Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, and Dinh Phung. Hiding and recovering knowledge in text-to-image diffusion models via learnable prompts. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025a.
Anh Tuan Bui, ThangTrang Vu, Long Tung Vuong, Trung Le, Paul Montague, Tamas Abraham, Junae Kim, and Dinh Phung. Fantastic targets for concept erasure in diffusion models and where to find them. In The Thirteenth International Conference Learning Representations, 2025b.
Tuan Anh Bui, Trung Le, Quan Tran, He Zhao, and Dinh Phung. A unified wasserstein distri- butional robustness framework for adversarial training. International Conference on Learning Representations, 2022.
Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey, arXiv preprini arXiv:2403.04279, 2024.
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security '23), pp. 3253-5270, 2023. Full paper (PDF):
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1650-9660) 2021.
Dar-Yen Chen, Hamish Tennent, and Ching-Wac Wen Hsu. Artadapter: Text-to-image style transfer using multi-level style encoder and explicit adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8619-8628. 2024a.
Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. disentangled tuning for subject-driven text-to-image generation. arXiv preprint arXiv:2305.03374 2023a.
Disenbooth:
Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding, Yangyang Song. Shizun Wang, Xu Wang, Hao Yang, Jing Liu, Kang Du, et al. Photoverse: Tuning-free image customization with text-to-image diffuuion models. arXiv preprint (tt.:2339.05993 2023b.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William Cohen. Re-imagen: Retreeval-augmenned text-to-image generator. arXiv preprint arXiv:2209 20V.:99999.9491,, 2022.
Zhuowei Chen, Shancheng Fang, Wei Liu, Qian He, Menggi Huang, and Zhendong Mao. Dreami- dentity: enhanced editability for efficient face-identity preserved image generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 1281-1289, 2024b.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 01618, 2022.
Rohit Gandikota et al. Erasing concepts from diffusion models. ICCV, 2023.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577. 2014.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar. Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7323-7334, 2023.

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6833 2020.
Jiannan Huang. Jun Hao Liew, Hanshu Yan, Yuyang Yin, Yao Zhao, Humphrey Shi, and Yunchao
Wei. Classdiffusion: More aligned personalization tuning with explicit class guidance. arXiv preprint arXiv:2405. 17532, 2024a.
Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin CK Chan, and Ziwei Liu. Reversion: Diffusion- based relation inversion from images. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024b.
Hugging Face. Dreambooth training example. http:://@ttaa...comm/uugggaggppp
Siffusers/blbbmmee wwmmmmiieeeeeeeee lesplees.. es htte///eeeetttctttttaaaaaaaaaare dev 0.33.0.
Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, and Jingfeng Zhang. Scedit: Efficient and control- lable image diffusion generation via skip connection editing. In Proceedingy of the IEEE/CVF conference on computer vision and pattern Recognition, pp. 8995-9004, 2024.
Chen Jin, Ryutaro Tanno, Amrutha Saseendran, Tom Diethe, and Philip Alexander Teare. An image is worth multiple words: Discovering object level concepts using multi-concept prompt learning. In Forty-first International Conference on Machine Learning, 2024.
Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. Omg: Occlusion-friendlyy personalized multi-concept generation diffusion models. In European Conference on Computer Vision, pp. 253-270. Springer, 2024.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1931-1941, 2023.
Jason Lee, Kyunghyun Cho, and Douwe Kicla. Countering language drift via visual grounding. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4385 4395, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D11--4445
URL
14477
Yuheng Li, Haotian Liu, Yangming Wen, and Yong Jae Lee. Generate anything anywhere any scene. arXiv preprint arXiv:2306. /7/54, 2023.
Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song. Zhengui Xue, Ruhui Ma, and Haibing Guan. Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples. In International Conference Machine Learning, pp. 20763-20786. PMLR, 2023.
Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Yibo Wang, Xintao Wang, Yujiu Yang, and Ying Shan. Stylecrafter: Enhancing stylized text-to-video generation with style adapter. arXiv preprint arXiv:2312.00333 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Saman Motamed, Danda Pani Paudel, and Luc Van Gool. Lego: Learning to disentangle and invert personalized concepts beyond object appearance in text-to-image diffusion models. In European Conference Computer Vision, pp. 1 116 133. Springer, 2024.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference artificial intelligence, volume 38, pp.
2024.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320--7366 79362, 2023.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 1(2):3 2022.
Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. In Thirty-sevnnth seventh Conference on Neural Information Processing Systems, 2023. URL pettps://openrett
net Sett:/ooommmdd=OOU-S.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Mchdi Saface, Aryan Mikacili, Or Patashnik, Daniel Cohen-Or, and Ali Mahdavi-Amiri Clic: Concept learning in context. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Hadi Salman, Alaa Khaddaj, Guillaume Leclere, Andrew Ilyas, and Aleksander Madry. Raising the cost of malicious ai ai-powered image editing. In International Conference on Machine Learning, pp. 29894 19894-29918. PMLR, 2023.
Patrick Schramowski et al. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In CVPR, 2023.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 245-254, 1985.
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation in any style. arXiv preprint 2Xiv:2306.00983 2023.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048-6058, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bcore-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Mini- mal and universal control for diffusion transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14940-14950, 2025.
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to- image personalization. In ACM SIGGRAPH 2023 conference proceedings, pp. 1-11, 2023.
13

---

Published as a conference paper at ICLR 2026
Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. FaceO: Instantaneously condi- tioning a text-to-image model on a face. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-10, 2023.
Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc N Tran, and Anh Tran. Anti- dreambooth: Protecting users from personalized text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2116 2127, 2023.
Xierui Wang. Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi-subject zero-shot image personalization with layout guidance. arXiv preprint arXiv:2406.07209, 2024.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcomposer: Tuneng-free multi-subject image generation with localized attention. International Journal of Computer Vision, pp. 1-20, 2024.
Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7754-7765, 2023.
Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi. Prompt-free diffusion: Taking" text" out of lext-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8682-8692, 2024.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023.
Yuxuan Zhang, Yirui Yuan, Yiren Song, Haofan Wang, and Jiaming Liu. Easycontrol: Adding efficient and flexible control for diffusion transformer. arXiv preprint arXiv:2503.07002, 2025.
Chenyang Zhu, Kai Li, Yue Ma, Chunming He, and Xiu Li. Multibooth: Towards generating all your concepts in image from text. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 10923-10931 2025.
14

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
Related Work
15
Diffusion Models
15
A.2 Generative Personalization.
A.3 Challenges in Generative Personalization
on Multi-Concepis Personalization
Further Quantitative Results
C.I
The Effect Hyper-paramenne parameters
.22 Surprising Impact of TEA on Anti-DreamBooth
20
Qualitative Results
23
Limitations and Future Work
23
F Experimental Setting
23
Dataset Construction
23
Evaluation Metrics
28
Computational Settings
30
STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
We utilized Large Language Models (LLMs) in this work for two primary purposes. First, employed LLMs like ChatGPT to correct grammatical errors and enhance the manuscript's clarity. Second, we leveraged these models to generate diverse context vefiry the hypothesis SCP in the main text. The instructions for the LLMs are provided in Appendix
A RELATED WORK
A.1 DIFFUSION MODELS
Given a text-to-image diffusion model eg, where (x 1, represents the predicted noise time step I given the textual embedding T(p) of prompt p and the noisy intermediate vector æ; (Ho et al. 2020; Song et al. 2020; Rombach et al. 2022). the model is trained by minimizing the following objective:
[||c [|c--c(aazz + VI- t, p)
C = E
(3)
Here, x and p denote the input image and its associated prompt, respectively, while is the Gaussian noise sampled from standard normal distribution. The intermediate input Vt,- aeæ + v1 Qre is obtained from the forward diffusion process. For simplicity, we use the notation Ez,p,, [ to represent the expectation over the input data æ. the prompt p. the diffusion time step t, and noise e.
A.2 GENERATIVE PERSONALIZATION..
Generative personalization task aims capture personal concepts which are implicitly shared in reference set of images as generative conditions and then use them as guided condition to generate new images containing the personal concept. These personal concepts very difficult express
15

---

Published as a conference paper at ICLR 2026
in the input prompt, e.g., how to express the concept of your dog that is different from a generic dog. Therefore, rather than using prompt engineering techniques to describe the concept in text, this task usually uses gradient-based method to fine-tune the model parameters capture the personal
concept. There are several categories of generative personalization (Cao et al.
2024) classified based
on the type of generative conditions, such subject-driven (Gal al. 2022 Ruiz et al. [023] Chen
itt al.
2022; Kumari et al. [2023; Wang et al., (2024). person-driven (Xiao et al. [ Valevski et al.
2023 Chen et al., 2024bb 2023b). style-driven (Sohn et al. [023 Liu et al. 2023 Chen et al. 2024a)
or image-driven (Ramesh et al. 2022, Xu et al.
Personalizing T2I diffusion model
from only a few examples presents several well keol- challenges, including language drift, limited
expressiveness of generative conditions, entanglement of concepts, and conditional misalignment.
Textual Inversion and Dreambooth. While there are many personalization methods have been proposed, they can be traced back to the two representative methods: Textual Inversion (TI) (Gal al. 2022) and DreamBooth (Ruiz et al. 2023). Mathematically, given a set of personal images and pre-trained T2I model cg, the goal is to identify textual embedding v" associated with a specific verbalizable keyword V (e.g., 'sks', '<new>', etc.). This keyword represents the implicit visual concept shared in the reference set X' enabling the model generate images with the personal concept using any textual prompt containing the keyword V", c.g.. [p, v=J= A photo of V* playing on beach', where [...] is the sentence construction operator.
X F1, P2. , Fn,
Textual Inversion (TI) Gal et al. 2022) is a pioneering method that proposes obtaining a textual embedding by minimising the following objective:
min E Exp. ||c eo (Fe,es Lo, V DIB
(4)
Here, ~ T is a template prompt sampled from a set of predefined neutral prompts T, such as ('a photo of a', 'a high-quality photo of .'...., }. In TI,, only the embedding v" is learned, while all other parameters, such as the model 69 or textual encoder т, remain fixed. Although this method
paraaeter-efffcieett, the learned embedding may not be sufficiently representative to capture the true visual concept in the reference set. Building on TI, Dreambooth (Ruiz et al.) (2023) suggests fine-tuning not only the embedding v" (i.e., by fine-tuning the textual encoder 7) but also the model parameters f by minimizing the following objective:
min E Eg Pppperprreee i ||c- cs (Ir,ert, Lp. D' +xIl +
-xlle-cc((ae co -co (ot per) [1]]
(5)
0.
In this context, R and per are the prior-preservation image and its associated prompt, respectively, which help prevent the model from overfitting to the small reference set with a tradeoff hyper- parameter A.
The rapid progress of text-to-image models such as Stable Diffusion (Rom- bach et al. (2022) and DALL-E (Ramesh al. 202T). trained on massive web-scale datasets, also brings risks of malicious misuse. These models have already been exploited to generate harmful, biased, or infringing content (Somepalli et al. 2023 Carlini et al. 2023). While such risks first surfaced with seabrities-wosse data are abundantly available online-thh rise of generative person- alization means that now anyone's 's likeness can be misused, since high-quality personalized models can be trained from only handful of images.
Anti-Personalization.
Mitigation strategies span both developer and user-side defenses. On the developer side, model providers aim to proactively remove or suppress harmful capabilities before public release. This can be achieved through machine unlearning, which erases targeted concepts from the model (Gandikota al.) 2023 Schramowski et al. 2023 Bui et al. 2024 (025b), or through parameter-editing methods that hide unintended capabilities without retraining (Bui ct al. 2025a).
On the user side, anti-personalizztion methods (Liang et al. 2023 Van Le et al. 2023 Salman et al. 2023) protect individuals from unauthorized personalization by perturbing their images before sharing. These methods apply imperceptible "masks" so that, even if adversaries access the data, they cannot train effective personalized models. A prominent example is Anti- Anti-DreamBooth (an Le
al. 2023), which leverages adversarial learning (Szegedy et al. 2013) Goodfellow et al. 2014 Bui
16

---

Published as a conference paper at ICLR 2026
al. 2022) to craft such masks. Formally, the optimization seeks perturbations 8 that maximize conditional loss Lcond while simultancously degrading DreamBooth's 's training objective Las under bounded distortion:
8-(1) -argmaxgon gis Con -00++ + vi {I, Na.
Nas
s.t. 0" -argmin
Cab(O x0)x t
(6)
and ||000 S
vi
For instance, the conditional loss can be defined via reconstruction and prior losses:
wellxs(aaxx + occ,, - x122 + +X (x Xpr + toye :pe Xpr Xprl12
(7)
Brecon
Eeproor
A.3 CHALLENGES IN GENERATIVE PERSONALIZATION
Language Drift and Overfitting. Language drift overfitting occurs due to the limited number of reference images and results in the incorporation of irrelevant elements and the neglect of the textual context within the outputs. Prior works address this issue by introducing preservation mechanisms such as rntor-preservatinn loss (Ruiz et al., (2023). locking concept-specifie parameters (Tewel et al.
2023) and regularising model weights (Han al. 2023 Qiu et al. 2023).
Limited Expressiveness of Generative Conditions. This occurs due to the limited expressiveness of the original textual format and the limited number of tokens allowed in each condition. common mitigation approach is to enable pre-trained T2I models to accept new types of conditions and generate in conjunction with the current text prompt conditioning, previous works have attempted to incorporate additional encoder (Zhang et al. 2023), or add a new adapter module to align internal knowledge the model with the new condition (Mou al. 2024 Jiang al. 2024).
modal conditions such image-image and aketch-image. To
Entanglement of Concepts. The reference image set might include samples that contain both the intended concept and other irrelevant concepts. To effectively isolate and extract the intended concept from the reference set, previous works have employed explicit masks (Avrahami et al.] 2023 Jin jt al. 2024| Safaee et al., 2024) and additional data with the personalized concept (Li et al. 2023). Alternatively, Disenbooth (Chen al. 2023a) proposed to mitigate the influence of background elements in the reference set by disentangling the identity and background of the reference set.
Conditional Misalignment. Beyond overfitting and entanglement, generative personalization faces the broader challenge of conditional misalignment where outputs deviate from the intended prompt due the limited alignment capacity of the original generative model. The trade-off between identity fidelity and semantic fidelity is a well-known and fundamental challenge in generative personalization. Most existing works address this overfitting issue during training and propose various regularization strategies, which typically require modifying the training process or adding supervision. PALP (Arar al.) (024) introduces score distillation sampling explicitly regularize the learned token toward its original class concept, preventing semantic drift. However, it operates with single fixed prompts and requires training-time modification. LEGO (Motamed et al. (024) and Re Version (Huang et al.) (024b) aim to disentangle compositional concepts (c.g., adjectives, verbs, or relationships) from exemplar images using token-based personalization. However, they are limited to token-based personalization model only.
largely unaddressed gap in prior personalization work is the potential drift or misalignment of the textual embedding itself during concept learning. We refer this phenomenon semantic collapse where the learned concept token still faithful to the visual reference, but fails to retain any meaningful textual semantics and eventually collapses simplified form. We directly address the
17

---

Published as a conference paper at ICLR 2026
20g ancher,
193 100 akge no Sooee
12.
118p
12-
21
(a) Sample generated images
Alignment (average over 50 random runs)
Figure 9: Analysis the SCP on multi-concepts personalization. The embedding of V man is fixed while the embedding of ving is varied across different training steps. Prompt: 'A photo of V man touching dog beside park bench'. See Figure 26 and 27 for more examples.
semantic drift of the learned embedding. Our method mitigates this drift without altering the training pipeline or requiring additional training. As a training-free, plug-and-play solution, our method can be seamlessly integrated into wide range of existing personalization frameworks.
B SCP ON MULTI-CONCERR PERSONALIZATION
In this section, we investigate the question: What is the SCP in the context of multi-concepl person- alization? For instance, consider a prompt like 'A photo of a Viann touching a Vioeg beside a park bench', where V Vmann and V2 represent two independently personalized concepts. Will one concept dominate the other, observed in single-concept personalization in the previous section?
To explore this, we conduct an experiment where the two concepts, Väng and Viaan (subject 342), are learned independently using Textual Inversion. We then construct a list of prompts that combine these two personalized concepts with an additional complex context, such as 'A photo Vian touching Vogg beside park bench' (refer to Table 4 for more examples).
Figure 9 illustrates the generated images where the embedding of V Viaan is held fixed, while the embedding Vtoog.ste is varied across different training steps. At early steps, Väog.step remains close to the original, generic 'dog' concept, while at later steps, it progressively captures more personalized visual information of the specific dog. This design allows us to observe how changes in the Viog embedding influence the generation of V Man within the prompt.
It can be observed from Figure that at early steps (e.g., 400), V tends to dominate the prompt, resulting in images that primarily capture the 'man' concept, consistent with the SCP observed in single-concept settings. However, as the training progresses and Vog.sttop captures more distinctive features of the personalized dog concept, it begins to overshadow Vmann leading to outputs that predominantly depict only the dog, effectively suppressing the presence of the other concept. As shown in Figure 19] the alignment score with anchor man image drops gradually while that with anchor dog image increases over time further confirming the dominance Väng over Vman-
This trend, observed consistently across multiple settings as shown in Figure 26 and 27 highlights the intricate nature of SCP in multi-concept personalization. It suggests that SCP not only persists but intensify when multiple personalized concepts are involved, presenting challenging but potentially fruitful direction for future research.
C FURTHER QUANTITATIVE RESULTS
In this section, we present additional quantitative and qualitative results that further validate our findings and broaden the scope of analysis. Specifically, include:
18

---

Published as a conference paper at ICLR 2026
Tanation Tx
Sonder 50
PJ prorent CantOrCs,
ConCOSE KO
Premett ofa (3 roening plamas writing retuacae
10.2
77.
75.0
21
70.5
(a) Intermediate generated images
(b) Alignment scores
Figure 10: Illustration of the SCP on Textual Inversion. Left: The intermediate generated images á of a prompt 'a photo of a V wearing glasses and writing in red notebook'. The generated image is gradually biased towards the personalized concept V* (i.c., easier to recognize as 'Henry Cavill') and loses the context (i.e., harder to recognize 'writing in red notebook") through out the personalization process. Right: The alignment scores (average over 100 random seeds) which empirically validate the SCP. The alignment S p) (O) with the context drops over time, while the alignment 5( gee) (+) with the ground truth Tee increases.
Figure 11: Analysis of the effect of rotation factor.
Method
CLIPS
CLIP
CLIP/
CLIP-I
TI
17.4 188
21.4 1.1 26.5
1.8 73.5 4.7
0.20
17.8 + 1.9
21.7
27.5
19
71.1
5.7
0.25
18.0
218
27.8
69.6
5.7
a 0.30
18.0 + 2.0
21.9
1.1
28.0
19
67.3
5.8
a 0.35
18.2
+ 2.0 21.9
+ 1.1 28.2
+ 1.8 63.6
+
Hyper-parameter sensitivity (Section mance.
C.1
showing how different settings affect TEA's perfor-
Unexpected effects on Anti- Boot-DreamBooth (Section C.2 where TEA reveals new insights into mitigating distortions in unlearning scenarios.
Textual evidence of semantic collapse (Figure 17 where we measure embedding drift using multiple distance metrics.
Visual evidence of collapse across methods (Figures 18 19 Inversion and DreamBooth.
confirming SCP in both Textual
Prompt-level embedding drift (Figure 21 providing direct evidence that SCP is not confined to TI variants but also affects DreamBooth and its extensions, consistent with our argument that unconstrained optimization underlies the problem (Section 2.3.3).
Detailed quantitative comparisons (Tables 7 on CustomConcept100 and CelebA, offering comprehensive view of TEA's improvements over multiple baselines.
Together, these results not only strengthen our main claims but also uncover broader implica- tions-man most notably, that SCP and TEA's corrective effect are general phenomena spanning datasets, frameworks, and experimental setups.
C.1 THE EFFECT HYPER- PEIEER--RAMEEERS
An important question is how to choose the hyper-parameters and 3 appropriately, or whether they should be adapted based on the input prompt. Interestingly, our experiments reveal clear pattern in the performance of the proposed method across range of and 3 values, providing practical guidance for their selection. For simplicity and consistency, we set a = 0.2 and 8 = 1.5 as default values, which consistently deliver robust performance across diverse prompts.
19

---

Published as a conference paper at ICLR 2026
wearing glasses handshaking with
Subey
Figure 12: Analysis of the effect scaling factor.
Figure 12 shows the performance our adjustment method over a range of 3 values with fixed 1.0 (no rotation). It can be seen that when / is too small (i.e., B 0.5, meaning || < 0.5 Me,l) or too large (i.e., 3 > 5.0, meaning || > 5.0 |MM&]. the generated images become less aligned with the ground truth indicating by the significant drop in S (2, Zyt), suggesting that V has lost its personalized information. However, the alignment (,, Fgg)) is relatively stable when is in the range of [1.0, 5.0], suggesting that the personalized concept
be effectively captured My Mell
without extreme scaling V*. As practical choice, we simply set 3 = 1.5 ==
(which is M|
middle value interpolated from |Mlll to || []), as the default setting.
Table 11 shows the performance over a range of a values with fixed 1.5. Unlike the scaling pa- rameter, the rotation factor a is more sensitive and significantly impacts prompt alignment capability. Increasing generally improves the model's ability to capture context, reflected in the CLIP/ score, which increases from 26.5 to 28.2 (an improvement of 1.7), and the CLIP score, which rises from 21.4 to 21.9 (a gain of 0.5). While this comes minor cost visual fidelity, with the CLIP-I score dropping from 73.5 to 71.1 when = 0.2, the generated images still maintain high visual quality, as shown in Figures 23 and 24
These findings highlight the critical role of the rotation factor a, which directly controls the semantic alignment between V* and the target concept c. Higher a values encourage better prompt alignment by rotating V* closer to c, while the scaling factor 3 should remain within moderate range to prevent excessive distortion of the learned visual concept.
C.2 SURPRISING IMPACT OF TEA ON ANTI-DREAMBOOTH
In this section, we present additional results highlighting the surprising impact of TEA on Anti- DreamBooth. Figures 13 and anaayye the effect of varying the rotation factor p and the scaling factor a, respectively.
Interestingly, while scaling has shown its effectiveness in mitigating the SCP in standard person- alization settings as discussed in Section C.I. when applied to Arti-DreamBooth, it does little to correct the visual distortion in generated images. By contrast, adjusting the rotation factor p shows pronounced effect such that the generated images become substainally less distorted and better aligned with the protected concept V as shown Figure 13 This contrast suggests that the se- mantic misalignment in anti- personalization settings is more sensitive to directional shift than to embedding magnitude, and that rotation-bar based corrections are inherently more effective than scaling- bcasing-based based adjustments.
We view this as a valuable and unexpected finding that not only deepens our understanding of why Anti- Aati-DreemBothh works (by amplifying the semantic collapse), but also points to promising directions for future work such as geometric interventions or adaptive adjustments during the defense process.
20

---

Published as a conference paper at ICLR 2026
(p-0.3,
Pronpet
Pounge 2
Prourggt 3
Prompe d
Pronppt 5
Prorege
GET
Pronnget 2
Prompp
Prompt a
Prosrgge to
Figure 13: Applying TEA on mitigating the Anti-Personalization Effect of DreamBooth by varying the rotation factor p
21

---

Published as a conference paper at ICLR 2026
Prompt
Prunpe 2
Prongt 3
Prompt 4
Prungee 5
Proumge 6
BEL
Prosmpp 2
Prosmpe
Prompge 6
Promnpe to
Figure 14: Applying TEA mitigating the Anti- Anti-Personalization Effect of DreamBooth by varying the scaling factor a
22

---

Published as a conference paper at ICLR 2026
D QUALITATIVE RESULTS
We provide additional qualitative results to complement the quantitative analysis:
Correcting distorted generations. Figure 22 illlstrrat how SCP in DreamBooth leads to distorted generations, while TEA corrects the semantic embedding to produce more coherent and realistic images. This connects directly to the discussion on TEA's unexpected impact on Anti-DreamBooth (Section C.2).
Cross-dataset comparisons. Figures 24 and 23 compare -EAA-augmettd variants of Textual Inversion, DreamBooth, and Custom Diffusion against their baselines on CS101 and CelebA Further comparisons on Subject Control and Relationship tasks are shown in Figures 28 30 and 31 highlighting TEA's robustness across frameworks.
SCP in multi-concept prompts. Figures and and 27 demonstrate SCP when multiple concepts are combined, with corresponding alignment scores provided in Figure 25
E LIMITATIONS AND FUTURE WORK
Limitations on Methodology. We believe that the insights and understanding provided by this paper, especially the analysis of the Semantic Collapsing Problem, are the most important contributions. Based on this analysis, we propose a simple method to adjust the embedding vectors at test time. While this method has clear advantages, such simplicity, generalizability. and no additional training requirement, the simplicity of the method itself might be a limitation. There are still trade-off's between alignment with the input prompt and alignment with the visual concept, which depend on the papee-pss,meters, suggesting that fixed pepeer-prramtters might not be optimal for all prompts.
Future Work. We believe that the insights and understanding provided by this paper. especially the analysis of the Semantic Collapsing Problem, can guide future research this topic. For example, integrating additional constraints to restrict semantic shift during the fine-tuning phase, rather than relying solely on test-time adjustment, a promising direction. This approach could directly produce adjusted and bounded embedding vectors that retain the original semantic meaning of the base concept.
In this work, we provide simple method to adjust the embedding vectors at test time. This adjustment applied equally across all dimensions of the embedding vectors. However, we believe that this not the optimal way to adjust embedding vectors, as each dimension of the embedding vector has different meanings and importance. Therefore, a more sophisticated method that considers the importance of each dimension could also be promising direction.
In Section C.I
we provide an analysis of the impact of proposed method. It has been shown that the performance of the proposed method is sensitive to the rotation factor a: the larger the a, the more the embedding vector is rotated toward the target concept, improving alignment with the generated images but potentially reducing alignment with the visual concept. In this work, we simply use the same hyper- prree-pprreeters for all settings. We believe that a search algorithm could be applied to find the optimal syper-parametent for each prompt, with stop condition based on the desired alignment with the input prompt.
on the performance of the
F EXPERIMENTAL SETTING
F.1 DATASET CONSTRUCTION
Contextual Prompts for Measuring Semantic Collapsing. Recall our hypothesis: a personalized keyword V*, initialized from base concept c to capture a specific visual target Egt, tends to lose its original semantic meaning and dominate arbitrary contexts p when used in complex prompts. This phenomenon, which we refer to as semantic shift,, can be directly assessed by comparing the embedding vectors My. and M.. However, this is challenging due to the use of contextualized text embeddings modern LLMs and diffusion models, where the surrounding context significantly shapes the final representation of each token.
23

---

Published as a conference paper at ICLR 2026
Table 3: Sample sentences for Pv- and Py..eee Set Pa can be constructed by replacing the word with V in the prompt of P. All the data and prompts can be found in the repository.
Set
Sample Sentences
Pv-
'A V walked his dog through the park every morning before sunrise" 'Despite the heavy rain, V* stood patiently waiting for the bus" 'In the small village, V* known for his kindness helped everyone'
'After twenty years of dedicated service, a V retired from his factory job
'While climbing Mount Everest, V* discovered the true meaning of perseverance'
"During the concert, a V in the front row sang along to every song' 'At the crowded marketplace, V sold handcrafted jewelry made from local materials'
Thhoughout history, V with vision has often changed the course events'
Behind every successful company, there is often V with an innovative idea'
Within the ancient temple, a prayed silently for his family's wellbeing'
p simple
'A photo of V*
'A rendering of V
'A cropped photo of a V
'A portrait of 1
'A close-up shot 1/*'
'A full-body image of a V
'A black-and-white photograph of a V
'A candid shot of a V+
'A digital illustration a V
'A stylized caricature of V
To address this, we propose evaluating the semantic shift of V relative in the presence of diverse set of contextual prompts. Specifically, we define prompt set A = {ai a2. )n
constructed by querying a large language model (L.LM) with the following instruction:
Write 200 sentences with diverse topics and contents.
Each sentence should be 10-30 words long and must include the keyword c.
This approach allows measure how well the learned embedding My- retains the original semantic characteristics c across varied contexts, providing robust test for the semantic collapsing problem.
To further examine the dominance effect V*, we introduce a complementary set simple prompts, denoted as Primpple This set consists of 200 straightforward sentences where V is the clear focal point, such "a photo of V*" "a portrait of V*" etc. These simple prompts serve as a baseline for assessing the degree to which V* overshadows other contextual elements in the generated outputs.
Sample sentences from both prompt sets are provided in Table 3 and the full dataset, along with all prompt templates, is available in the repository at ptttt::://// thub. httm:/tannnhbbui89 Embeddi iiniinn--- Ad justment.
Dataset for Evaluating Personalization Performance. We use subset of 9 concepts from the CustomConcept1 01 dataset as in the original paper (Kumari et al. 2023), each of which has 3-15 images, including 'Barn', 'Tortoise plushy', 'Teddy-Bear 'Wooden Pot', 'Dog', 'Cat', 'Flower', 'Table', 'Chair' subjects. For the human concept, use subset of 10 concepts from the CelebA-HQ dataset (Liu et al. 2015), which includes 10 identities with 10-15 images per subject. Sample images from the CustomConcept101 and CelebA-HO datasets are shown in Figure 15 and Figure 16 respectively.
To assess complex prompt handling, compile set of multi-concept prompts from the Custom- Concept101 dataset. Each prompt is designed to include two to three distinct elements, encouraging the model to balance multiple visual contexts. For instance:
24

---

Published as a conference paper at ICLR 2026
Table Sample prompts generate personalized images. Each prompt consists main subject V and context All the data and prompts can be found in the repository.
Set
Sample Sentences
CelebA
'A photo of V*
wearing glasses and handshaking with man'
'A photo of a V*
wearing glasses and handshaking with a woman'
'A photo of V
wearing glasses and handshaking with old man'
'A photo of a V*
wearing glasses and handshaking with a kid'
"A photo of V* wearing glasses and holding dog'
'A photo of a V* wearing glasses and holding a cat'
'A photo of V* wearing glasses and holding red book'
'A photo of a V* wearing glasses and holding red phone"
photo of V wearing glasses and sitting on red chair'
'A photo of a V wearing glasses and lying on red bed'
photo of V* wearing glasses and writing in red notebook'
'A photo of V* wearing glasses and drinking Coco Cola can'
'A photo a / wearing glasses and lifting weights'
'A photo of V* wearing glasses and cycling"
'A photo of a V* wearing glasses and kicking a football'
'A photo of V wearing glasses and playing guitar'
'A photo of a V* wearing glasses and eating pizza'
CustomConcept110
V in snowy ice'
V in blooming sunflower field'
VV' on boat in the sea'
VV' on top of mountain"
² made of crochet'
*V* in garden'
'a floor lamp the side of V*
'V" and a table with chocolate cake on it'
'a puppy sitting
V
'a cat sitting on a V*
'a squirrel sitting on V* 'a deer grazing near a V- teddy bear on V
photo of V* in Van Gogh style'
Multi-Concept
'A photo of V
man
wearing glasses and kissing Vdog
'A photo of a V
wearing glasses and handshaking with V
man.
'A photo of a V/4
wearing hat and hugging Vidgg
man
'A photo of a V/*
walking a Vi0, on a road with a car behind'
man
photo of V
holding a V beside a car'
'A photo of V man
touching Vion beside park bench'
'A photo of V
feeding Vidg with bowl of flowers'
man
" tortoise plushy sitting at the beach with a view of the sea"
"a watercolor painting V tortoise plushy on a mountain"
These prompts contain a primary subject V* and one or more contextual elements (context p). allowing measure the model's ability to preserve the personalized concept while maintaining accurate context alignment.
Sample prompts provided in Table 4 and the full dataset, along with all prompt templates, available the repository at Embedd ig-nnd- Ad Ang-Adjusse justment
com/ wttp://tt//tttataiiiiaaaa tuananhbui89/
25

---

Published as a conference paper at ICLR 2026
(a) Wooden Pot
(b) Flower
(c) Chair
(d) Table
(c) Dog
(f) Cat
(g) Barn
(h) Teddy Bear
ii) Tortoise Plushy
Figure Sample images from the CustomConcept1101 dataset.
26

---

Published as a conference paper at ICLR 2026
(a) Subject 124
(b) Subject 181
(c) Subject 276
(d) Subject 342
(c) Subject 351
(f) Subject 437
(g) Subject 615
(h) Subject 908
(i) Subject 1335
Figure 16: Sample images from the CelebA dataset.
27

---

Published as a conference paper at ICLR 2026
Table 5: Inter-set and intra-set distances for different distance metrics
Distance
Inter-set Distance di d(P.Q)
Intra-set Distance P)
L2
D Pi
q: 0:112
EE= Ej= Ip:
Pj l2
Hausdorff
max maxx min 9, (P: 9;);
m ppp d.2 (p. Pf)
EQ p, EP (q.,
Mahalanobis
T V (p: 2p)' (p: (P)) 2i==1
T2
m j=1
(pi Ep (p. -p))
KL
(K Q)
DKL(P
F.2 EVALUATION METRICS
Personalization Metrics. We use the CLIP-T text-image alignment score (Radford et al. 2021 to evaluate the alignment between the generated images á and the prompts. To have a better understanding of which part of the prompt contributes to construct the generated images, we break down each prompt into multiple segmentssconceets and calculate the alignment score for each segment/concept, i.e.. 7CCIP denotes the alignment score of the full prompt, the first segment-the personal concept, and the second segment-the context, respectively. We use the CLIP-I image-image alignment (Radford et al. 2021) and DINO image-image alignment score (Caron et al. 2021) to evaluate the alignment between the generated images and the reference images.
Semantic Shifting Metrics. Given the two sets P and Q (i.e., P
Py- [T([a, V*1)} and
Q P. = (r((a,,c) e])} which are the embeddings of the prompts in the prompt sets Pv- and P. respectively), we propose use the following metrics to measure the difference between the two sets.
L2 distance measures the Euclidean distance between two points in a vector space.
d12(Ps 9:) llp - -9ill2
(8)
Hausdorff distance measures the maximum distance from any point in / to the nearest point in Q. In other words, it measures the greatest distance from point in one to the nearest point in another set.
(P. max(max min d22(Pi, qj), max min dz2(g; Pj))
PeE qeE
QeQ PjEP
Mahalanobis distance measures how far the point p: is from the center of the P. taking into account the correlation between the dimensions of the set. Unlike the L2 distance which treats all dimensions equally, the Mahalanobis distance adapts the shape and spread the set
d P) V (p:- -r) nP)
(10)
where uP is the mean of the set P and Ep is the covariance matrix of the set P.
KL divergence We propose to measure the relative relationship between each data point to the entire by using the Normalized Temperature-scaled Softmax function
exp(sim(p:. (P))) p(p. Pz. P) Emep expssimppssmmpp PT)
(11)
where T is the temperature parameter. p(p; Pj, measures the the relative relationship between
Pj and the anchor p; in comparison to the entire set P. From that, we can have p(p; P) = {p(p: Pj. P) Vp, P} to represent the relative relationship between pt and the entire set P. Similarly,
28

---

Published as a conference paper at ICLR 2026
p(q: Q) {p(g: Q) Vaz Q) to represent the relative relationship between 4, and the entire set Q.
The KL divergence between P and is then defined as:
Q) Σ
p(p: P) log p(p: P) 1=1 p(g: Q)
(12)
which measures the difference between the distributions P and Q. The higher the KL divergence, the more different the two distributions are, the more semantic shifting the learned embedding u* has.
Alignment Metrics. The primary objective of generative personalization is produce visually compelling images that accurately capture the unique characteristics a personalized concept from reference set, while maintaining semantic alignment with the input textual prompt.
To evaluate this, we use two key metrics based on the CLIP model (Radford et al. 2021
Visual Fidelity: We measure the alignment between the generated image and ground-truth image using the CLIP image-image alignment score. A higher score indicates a closer match to the reference, reflecting better preservation of the personalized visual features.
Prompt Consistency: We assess the alignment between the generated image and the input textual prompt using the CLIP text-image alignment score. A higher score indicates that the generated image more accurately reflects the intended context and details of the input text.
However, complex prompts often contain multiple concepts, making single text-image alignment score insufficient to capture the nuanced relationship between the personalized concept //* and its broader context To address this, separately compute alignment scores for:
Main Concept Alignment (V*): Measuring the fidelity of the personalized concept itself.
Context Alignment (p): Evaluating how well the broader contextual elements are represented.
This multi-levar evaluation provides comprehensive understanding of how well the generated images capture both the personalized visual identity and the intended scene context.
For all evaluations, available at
the implementation provided in the TorchMetries library. clip_scoo sworeceeer .hmll
29

---

Published as a conference paper at ICLR 2026
System Prompt for VLM--ased Evaluation Metries
Task Definition
You are an expert visual evaluator subject-driven image generation, You receive:
1. Reference image A - the target subject (object or person)
2. Prompt P text describing the desired scene/context
3. Generated image the image to evaluate
Produce two independent integer scores (0-4):
1. Prompt Adherence Does match the scene, spatial relations, actions, attributes, and style
P?
Subject Identity Does subject in visually match the reference subject in
Key Principle: These scores are independent. Perfect subject in wrong scene high Subject, low Prompt. high Prompt, low Subject.
Wrong subject in correct
Prompt Adherence Score (Scene/Context)
Evaluate: Scenc/setting, spatial relationships, actions, attributes, style specified P. Ignore: Whether the specific subject from is present (only check if something fills the subject role correctly).
0 No alignment with prompt
Minimal alignment; major elements missing/wrong
Some core elements correct; important parts missing/contradicted
3 Most elements correct; minor omissions in secondary attributes
4 All major elements present and correct; full prompt satisfaction
Subject Identity Score (Visual Matching)
Evaluate: Does the subject O look like the reference A? Check shape, distinctive features, colors, recognizable characteristics.
Ignore: Whether subject is in correct scene/position (that's Prompt Adherence).
0 Subject absent completely unrecognizable
Weak resemblance; most identifying features differ
Ambiguous identity; significant changes important features
3 Clearly recognizable; some features altered but identity preserved
4 Unambiguous match; all distinctive features preserved
Output Format
Output ONLY these two lines with other text:
Pr compt Score:
<integer 0-4>
Sub jectt
<integer 0-4>
Examples:
Same subject, wrong scene - PromptScore: SubjectScore:
Wrong subject, correct scene PromptScore: SubjectScore: 0
Both perfect - PromptScore: 4, SubjectScore: 4
Now evaluate the following inputs:
VLM-based Evaluation Metrics. We use the VLM-P and VLM-I metrics to evaluate the alignment between the generated images and the reference images and the prompts, respectively. Both metrics output a score between and 4, where means there are no correspondence between the generated image and the reference image (VLM-I) input prompt (VLM-P), while 4 means the perfectly matches. The final score for each metric is obtained by averaging all inference prompts and samples (16 prompts times 50 random samples per concept/setting) and normalizing the range [0, 100%). We include the full system prompts and evaluation scripts in the Github repository.
F.3 COMPUTATIONAL SETTINGS
All experiments are conducted on a single NVIDIA RTX 4090 GPU with 24GB of memory, using the Stable Diffusion v1.5 model the base model. To prevent memory overflow, fine-tune the model with Textual Inversion (TI), DreamBooth (DB), and Custom Diffusion (CD) using a batch size of across all methods.
30

---

Published as a conference paper at ICLR 2026
(a) L2 distance
(b) Hausdorff distance
49.0
600
1100
3050
2019
(c) KL divergence
(d) Mahalanobis distance
Figure 17: Different distance metrics between the sets {T((a;. V* and Q (r([a,,)) in Textual Inversion (TI), showing the semantic shifting of the learned embedding over training iterations.
Table 6: Results on CustomConncept101 dataset, tort means tortoise plushy, teddy* means plushy teddy bear, wpot means wooden pot. The first/second metric is the /LIP, score. The blue number indicates the proposed method outperforms its baseline counterpart, while the number indicates the opposite. The GAP is the average improvement over all concepts. Qualitative results shown in Fig. 24
Method
dog
Tort
leddy
chair
table
Hlwer
wpot"
ham
GAP
26.05/
25.26/ 26.28/
THTTE
26.14/ 58.97
27.94/
77.47
30.71/ 72.74
27.68/
76.02
28.13/ 73.27
27.78/ 61.33
63.77
28.34/ 66.44
28.2/ 77.06
0.54/ -2.34
DB
20.28/ 59.93
20.29/
74,82
24,55/91,14
18.89/
52
27,87:8755
20.771 73.83
20.07/ 82.98
20.43 66.69
23.58/ 80.98
0.00 0.00
CD)
27.33/
56.09
29.31
78.37778.44
78.34
31.33/
28.47/ 71.67
27
69.88
27.88
62.84
26/7/ 62.66
30.1V 69:199111 18
26.34/
70.95
0.000 0.00
CD+TEA
27.45/
56.18
29.25/
31.06/
28.04/
78.61
28.16/
71.81
28.34
63.08
27.37/
62.63
30.15/ 68.0
27.82/
71.78
0,34/
CL
27.28/
29.19/
77.42
31.11/78.32
28.26/
76.4
28.49/
74.52
27.68/
60.34
26.76/
63.98
29.60 69.69 61.09
25.72/
69.87
CL+TEA
27.72/ 56.44 27.22/77.26
30.89/ 78.87
28.07/
77.46
29.01/
75.56
28.1
60.54
27.211 64.87
30.2/
60.72
27.65/
71,27
0.42/ 0.67
For the DreamBooth LoRA method, we follow the recommended settings from the Diffusers' example page, using a learning rate of lc-4, rank 4, and enabling text encoder training for improved performance. Textual Inversion is fine-tuned with a learning rate of 5c-4, while Custom Diffusion uses more conservative learning rate of 5e-6.
All code implementations adapted from the Hugging Face Diffusers library.
31

---

Published as a conference paper at ICLR 2026
Cannur Pojo
AoCO 416
procett 116--r7
cantet Non anderr
Image
Pernet is photo of weiting sroas and harchhiing ofti 1
Portet Aghsts df aneting gharee tcinno
75.0
Camoot
pronetSideed Contect
wournn glawo
Proppt plutooo wouren glaweo
Stores
300
15.
SrpprrrttSpppel Carvet
Orrrgg 5
40.5 51pp
+ TV pronpl Sieeee
Orteet Sipl
Coeopt
21dee Step
Imagh Scorn
150
300
625
Figure 18: Alignment scores showing the SCP on Textual Inversion with different prompts.
Table 7: Results on CelebA dataset. The first/second metric is the CLIP/ score. The blue
number indicates the proposed method outperforms its baseline counterpart, while the red number indicates the opposite. The GAP is the average improvement over all concepts. Qualitative results are shown in Fig. 23
1335
23.77699
B1.58
75.64
71.19
35.3667.35
34977
60.3
0LR0 0.00
THTRA
34.23/68 48
26
59.6
20.97/65.2
27.38/11011
26.11/65.75
73
23.93/
66.25
26.80
70.56
62.97
66.60/66.72
25.60/
59.01
0.57) -2.41
DR
25.37/54478
25.91761.75
58.7
24.37/7041
62.09
20.65/75.15
75.15
25.06
64.93/63346
25,66/
50.2
24/65/6043
0.000-0.0
DR-TRA
76.26/ 50.67
26.597 60.23
24.667666
25.58765.88
25.43/6.33 34
22.56/
68.79
26.03
60.88
25.922
60.
24.70
53.66
25.377
58.46
1.157
26.92/43.56
26.87/52.4
36.22
27.14/52.64
26.94) 53.39
26.97/
55.45
26.99
53.96
26.93/
41.55
26.88/
0.00 0.00
26.90/
27.5/56.92
29,511
59.61
27.13
59.45
20.34/
32

---

Published as a conference paper at ICLR 2026
Canettggn
GemuppSSii Silo
Srtact
Crapps 61
Perpet phol worrg
Perpet phat ofa werting srees hwrdhhillee
piots aunting glares toito koog
300
Camotts-g
Concagt Hil
ancho Stap
c Rd pronottiie
Contact Sypt
Concapts-dt
inderrtio)
Inocen Poweee
CompetS
Ornoggt
53p
Ordect Sypt
Planet aoto disces
Fron:: ".prett choss uniing redctar"
Figure 19: Alignment scores showing the SCP on DreamBooth with different prompts.
33

---

Published as a conference paper at ICLR 2026
0.
04
0.8
COCER SEREaNT
550
1002
1303
2000
2500
3000
(a) Distribution of the norm in the cat-toy setting
(b) The semantic shift in the cat-toy setting
0.9
almmii de Stminoe V/3
0.8
diriperon V71
0.7
10
F0.
550
1523
2003
2520
3320
(c) Distribution of the norm in the CelebA setting
(d) The semantic shift in the CelebA setting
Figure 20: Left: The distribution of the norm of the token embedding M including special token V*, Right: The semantic drift of V* in term of magnitude and direction over time. The same phenomenon observed in DreamBooth as shown in Figure
In(ov's - rip c)
14
Irip Soested - fie +1
0.96
coslTip +V'), T(p +c)
0.94
costT(p + Visumeel rl d)
12
Cos Cosino Similarity
L2 Distance
0.92
0.90
0.88
0.86
0.82
1000
1500
3000
Steps
Figure 21: The semantic drift of the embedding of entire prompt (p, V*J in term of magnitude and
direction over time with DreahBooth. The adjusted embedding [p, V TEA framework with a 0.2 and 1.5.
Vasjusteal obtained by using
34

---

Published as a conference paper at ICLR 2026
KUN
Figure 22: Some cool effects of our method (DB+TEA) over the bascline counterpart DB. The SCP in DB lead to the distorted generated images. Our method successfully corrects the semantic embedding and generates more coherent and realistic images.
35

---

Published as a conference paper at ICLR 2026
(a) 124
(b) 181
(c) 342
(d) 437
Figure 23: Qualitative comparison hetween baseline methods and their TEA variants on the CelebA
dataset. Column 1-2: Reference images. Column 3: TI, Column 4: TITTEA, Column 5: DB, Column 6: DB+TEA. Input prompts are shown below each image while alignment scores are shown on the top. More results be found in the repository.
36

---

Published as a conference paper at ICLR 2026
(a) DecorItems Wooden Pot
(b) Pet Dog
GAMS
(c) Pet Cat
(d) Scene Barn
Figure 24: Qualitative comparisons between baseline methods and their TEA variants on the Custom-
C0 dataset. Column 1-2: Reference images. Column 3: TI. Column 4: TI+TEA, Column 5: DB, Column 6: DB+TEA. Input prompts are shown below each image while alignment scores are shown on the top. More results can be found in the repository.
37

---

Published as a conference paper at ICLR 2026
Talprorge
anchar EndRr PM
panp:
NOW 7183
45g
1i0 conret Sa af (ogedt) wattnng (nteed)) ene rus with behiee Beee
seeer thep af CMeel Tebee bottr
14
3 Mechor Scara
Figure 25: Alignment scores showing the SCP multiple-concept personalization settings. The
embedding of V held fixed while the embedding of V /''dog is varied over fine-tuning step. See Figures 26 and 27 for the example images.
38

---

Published as a conference paper at ICLR 2026
ST0p 2000
mee 100
Clen 200
19rg TE0
09A1 830
109 123
(a) Prompt: 'A photo of a V man walking V road with behind"
Stop
Mrp
Sic0 2903
Plip 100
Piy 200
18y 300
mu 133
eop 500
DVc 100
u9 125
rep 903
0ta 000
0100 2033
2010 1000
3-22 200
Jlx 320
400
215 400
Sic 102
5020 930
242-000
3122 2900
(b) Prompt: 'A photo of
V touching Valog beside park bench"
Khpp 100
Ixp 200
111 200
0113
K2p 500
P7c 000
1723 125
2700 800
500
1300 2033
Sirp 100
3:20 200
Sig 300
1823 400
5129 500
Six 000
Mte 102
50m9 830
Xxp 900
Si03 2003
172
(c) Prompt: 'A photo of a Via. feeding a Vdog with bowl of flowers'
Figure 26: Illustration of the semantic collapsing problem on multi-concepts personalization. The embedding of V is fixed while the embedding of dog is varied across different training steps. See Figure for the detailed alignment scores.
39

---

Published as a conference paper at ICLR 2026
STOp 2003
etep 100
Che 2001
1rr TEO
OPA 830
Chec 1p0
10ep 123
<tep a01
900
(a) Prompt: 'A photo of a V walking V
road with behind'
2003
Sirp
Nep
Sic0 2903
Ktep 100
Piy 200
193 320
010 120
Pop 500
190 500
199 325
150p 803
000
0100 2030
S0o0 100
XAS 200
J1x 300
502
2/3 000
Sic 102
5122 830
2432 200
3122 2900
(b) Prompt: 'A photo of V touching Vilon beside
park bench"
Kipp 100
Exp 200
113 200
1113 833
s1p 500
170 500
172 125
2200 800
500
2300 2033
5000 100
T:20 200
Sig 300
1023 400
Sir 500
Xix 000
Ste 105
Sin0 830
5xp 900
Si03 2007
(c) Prompt: 'A photo of a Vi.. feeding a Vdog with a bowl of flowers'
Figure 27: Illustration of the semantic collapsing problem on multi-conceple personalization. The embedding of V is varied across different training steps while the embedding of Vdog is fixed. See Figure for the detailed alignment scores.
40

---

Published as a conference paper at ICLR 2026
SKS dog in construction outfit
(a) Output from EasyControl SKS dog in construction outfit
(b) Output from FasyControl with TEA
Figure 28: Comparison of the output from EasyControl pipeline with and without TEA with the same prompt: 'V 'V*en dog in construction outfit' and same random seed. EasyControl with TEA significantly improves the prompt fidelity, mitigating the failure cases of the original EasyControl pipeline, such as the dog stands beside a person or holding a gun. More results showing the same improvement of EasyControl with TEA can be found in the repository at https: / httggtttttb......g.//e.g com/ Euananhbui 89/ Embedding Adjustment,
41

---

Published as a conference paper at ICLR 2026
Output from Re Version
(b) Output from ReVersion with TEA
Figure 29: Comparison of the output from Versioo pipeline with and without TEA with the same prompt: 'cat <R> carrot in the garden' and same random seed. ReVersion with TEA significantly improves the prompt fidelity, mitigating the failure cases of the original Revessi pipeline, such as the cat is not carved by the carrot or only shows the carrot. More results showing the same improvement of Re Version with TEA can be found in the repository at wttttts:/ httpp://///ttubb...// com/ http:aanbbubasemmmmmddnnn-----ee
42

---

Published as a conference paper at ICLR 2026
SKS barn surrounded by alooming sunflower field
(a) Output from ClassDiffusion
SKS barn surrounded by alooming sunflower field
(b) Output from ClassDiffusion with TEA
Figure 30: Comparison of the output from ClassDiffusion pipeline with and without TEA with the same prompt: 'barn' and same random seed. ClassDiffusion with TEA significantly improves the prompt fidelity, mitigating the failure cases of the original ClassDiffusion pipeline, such as the image does not contain a barn but only a sunflower field. More results showing the same improvement of CiassDiffusion with TEA can be found in the repository at Intps::: /tt::///gtttt com/tuanamhbbui85 3tt9/////eeeee/n/eeeeeeeeee/e/// Aocja/aedddnnnnnnsstttnttnntttt Adjustment
43

---

Published as a conference paper at ICLR 2026
(a) Output from OminiControl
(b) Output from OminiControl with TEA
Figure 31: Comparison of the output from OminiControl pipeline with and without TEA with the same prompt: 'this item wearing glasses' and same random seed. OminiControl with TEA significantly improves the prompt fidelity, mitigating the failure cases of the original OminiControl pipeline, such as the subject (penguin) is not wearing glasses but a person does. More results showing the same improvement of OminiControl with TEA can be found in the repository at https: / / thub. wtt.//ooo//ttaaaeeee 39/Embedd -g- Adjustment
44


Reviews:
[{"review_id": "Mrzp1VYB4R", "reviewer": "Reviewer_WZt3", "summary": "This paper pioneers the definition of the Semantic Collapsing Problem (SCP)—a long-overlooked issue where personalized tokens lose their original textual semantics and dominate complex prompts, with unconstrained optimization identified as its root cause. It then proposes Test-time Embedding Adjustment (TEA), a lightweight, training-free solution that aligns the magnitude and direction of personalized embeddings with their original semantic concepts at inference, effectively mitigating SCP. TEA demonstrates strong generality, enhancing text-image alignment across diverse personalization frameworks (e.g., Textual Inversion, DreamBooth).", "questions": "How to determine the  rotation factor for each customization process?\nWhy TI+TEA and DB+TEA will degrade the customized image's alignment with reference？\nwhy the author choose SLERP interpolation to adjust the embedding?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The work proposes the Semantic Collapsing Problem (SCP) in generative personalization—an under-explored issue—and rigorously identifying unconstrained optimization as its root cause, with solid empirical evidence across textual and image spaces.\nThe proposed TEA method is lightweight and practical: it requires no additional training, avoids modifying model weights, and generalizes well across diverse frameworks (e.g., Textual Inversion, DreamBooth) and architectures (Stable Diffusion, Flux), making it easy to integrate into existing pipelines.\nThe unexpected finding that TEA mitigates Anti-DreamBooth’s adversarial corruption adds novel insights to privacy-utility tradeoffs in personalization, uncovering vulnerabilities in current anti-personalization defenses.", "weaknesses": "TEA relies on fixed hyperparameters (α=0.2, β=1.5) across all prompts, which may not be optimal for diverse scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9UckIJ0JA", "reviewer": "Reviewer_AfeA", "summary": "This paper introduces the \"Semantic Collapsing Problem\" in generative personalization, where a model over-focuses on a learned subject (e.g., a specific person) and neglects other contextual details in a prompt. The authors propose a simple, training-free solution called Test-time Embedding Adjustment (TEA) that corrects the subject's text embedding at inference time. This method significantly improves text-image alignment across various personalization techniques and surprisingly reveals a vulnerability in current anti-personalization defenses.", "questions": "1.\tHow does the paper demonstrate that the method genuinely improves performance, rather than simply acting as a trade-off between instruction alignment and reference image alignment?\n2.\tTo better understand the model's behavior, could the authors provide results for prompts that include the subject class (e.g., \"a photo of V* dog\"), not just the identifier \"V*\" alone?\n3.\tWhat is the specific effect of the prior preservation loss on the semantic collapsing phenomenon?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tHigh quality of writing and presentation.\n2.\tThe introduction of TEA, a method that is training-free and easily transferable.\n3.\tAn insightful analysis of the \"Semantic Collapsing Problem\" within generative personalization and the mechanics of anti-dreambooth methods.", "weaknesses": "1.\tThe paper lacks comparisons to recent, mainstream works, particularly those in the in-context generation paradigm (e.g., OminiControl[1], FLUX.1 Kontext[2], and Diffusion Self-Distillation[3]).\n2.\tThe evaluation is insufficient. It should be strengthened by including MLLM-based benchmarks, such as DreamBench++[4].\n3.\tTable 1 shows a performance decrease in Reference and Image alignment scores. This is presumably because TEA's objective focuses exclusively on fidelity between the learned and original subjects, potentially neglecting other alignment aspects.\n4.\tThe choice of SLERP is not justified. The paper provides no evidence or ablation study to demonstrate that SLERP is superior to other interpolation methods.\n\n[1] Tan, Z., Liu, S., Yang, X., Xue, Q., & Wang, X. (2025). Ominicontrol: Minimal and universal control for diffusion transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 14940-14950).\n[2] Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dockhorn, T., ... & Smith, L. (2025). FLUX. 1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space. arXiv e-prints, arXiv-2506.\n[3] Cai, S., Chan, E. R., Zhang, Y., Guibas, L., Wu, J., & Wetzstein, G. (2025). Diffusion self-distillation for zero-shot customized image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 18434-18443).\n[4] Peng, Y., Cui, Y., Tang, H., Qi, Z., Dong, R., Bai, J., ... & Xia, S. T. (2024). Dreambench++: A human-aligned benchmark for personalized image generation. arXiv preprint arXiv:2406.16855.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVA5KoHgJo", "reviewer": "Reviewer_1iQw", "summary": "This paper addresses the ​​Semantic Collapsing Problem in generative personalization, where personalized token embeddings drift significantly in both magnitude and direction during optimization, causing them to dominate multi-concept prompts and degrade compositional generation. The authors propose ​​Test-time Embedding Adjustment (TEA)​​, a training-free method that recalibrates the embedding’s norm and direction toward its original semantic concept during inference.", "questions": "The core operations of TEA (normalization and SLERP) are geometrically intuitive but relatively simple. Were more complex adjustment strategies explored, such as a learnable linear transformation or a small neural network? What was the rationale for choosing this specific, non-adaptive formulation? The simplicity is a strength for deployment, but it raises questions about the exhaustiveness of the exploration.\n\nThe analysis in Appendix B on multi-concept personalization is intriguing but preliminary. When two personalized concepts, V_man and V_dog, compete in a prompt, TEA is applied to which concept? Both? Does applying it to both concepts restore a better balance, or does one still dominate? This is a critical scenario for assessing SCP and TEA's utility.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The identification and empirical analysis of SCP are interesting. \n\nTEA is lightweight, requires no retraining, and is compatible with numerous existing frameworks.\n\nThe paper is well-structured and easy to follow.", "weaknesses": "While the Test-time Embedding Adjustment (TEA) method is practical and easy to deploy, its technical contribution is relatively modest. The core mechanism—adjusting the magnitude and direction of an embedding vector—is a straightforward application of existing vector space operations, lacking the novelty of a more transformative technique. The approach does not introduce new learning paradigms or architectural innovations, but rather applies a post-hoc correction to the outputs of existing models. This simplicity, though beneficial for accessibility, may limit the method's impact and long-term relevance in a fast-evolving field that often rewards more complex or foundational advancements.\n\nThe paper heavily relies on quantitative metrics like CLIP and DINO scores to demonstrate improvements, but provides limited qualitative evidence, especially for subject-driven customization tasks. For instance, while quantitative gains are reported, side-by-side visual comparisons showcasing TEA's enhancement over base methods (e.g., DreamBooth, Textual Inversion) across diverse concepts are sparse. More juxtaposed examples illustrating how TEA better preserves contextual details in complex prompts (e.g., \"V* wearing glasses and writing in a red notebook\") would strengthen the empirical claims. A broader set of qualitative results is necessary to help readers appreciate the practical improvements in output quality and alignment.\n\nA thorough discussion of scenarios where TEA underperforms or introduces new artifacts is absent. The paper does not systematically analyze failure cases, such as conditions where the embedding adjustment might over-regularize and weaken the distinctiveness of the personalized concept. Furthermore, the limitations of only adjusting the embedding at inference time—potentially being a \"shallow\" fix rather than addressing optimization issues during training—are not critically examined. A dedicated section exploring these boundaries, possibly with examples of prompts where TEA fails to mitigate semantic collapse or even degrades output, would provide a more balanced and credible evaluation.\n\nAlthough the effect of hyperparameters α (rotation factor) and β (scaling factor) is briefly studied, the analysis is conducted with a fixed base model and a limited set of prompts. The paper does not explore how these parameters interact with different model architectures (e.g., varying layers of U-Net or transformer-based text encoders) or diverse concept complexities. This narrow scope leaves open questions about the robustness and generalizability of the chosen default values across a wider range of real-world use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GE4rHsOh2p", "reviewer": "Reviewer_mTGU", "summary": "The author define semantic collapse problem and propose a method TEA to mitigate it in test-time without re-training or altering embedding weights. Experiements show that TEA perform well on top of multiple personalizated models.", "questions": "- What's CLIP$^{p}_T$ stands for? This metric is mentioned in Table 1, but not described in the experiment section. \n- Do TEA's result align with author's discuss about SCP? Could author provide similar visulization/chart on models w/ TEA?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provide a in-deep analysis about the root cause of SCP, claiming it's beacuse unconstrained optimisation, which make senses to this field.\n2. The method is simple yet effective, though the idea that using the concept token to regularize the V* token is not novel, simply using SLERP at test-time is somehow interesting.\n3. Comprehensive qualitative experiments shows the effectiveness of proposed TEA.", "weaknesses": "1. Novelty of SCP's definition: SCP is not an problem. that has not been discussed  What's the difference between SCP and semantic drift phenomenon discussed in ClassDiffusion[1]? According to the author's claim, they share a same phenomenon: can only generate \"a photo of V*\" when using the prompt \"a photo of V* with other concepts\". CoRe[2] also discuss similar problem. Also, some negative impace are somehow discussed in their paper.\n2. Insufficient evaluation: as CLIP-T and CLIP-I often fail to provide fair evaluation about fine-grained results, could author provie other quantitative results about TEA's performance? Like using user study or VLMs as judge.\n\n\n[1] Huang, Jiannan, et al. \"ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance.\" The Thirteenth International Conference on Learning Representations.\n[2] Wu, Feize, et al. \"Core: Context-regularized text embedding learning for text-to-image personalization.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 8. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)