Directional Textual Inversion for Personalized Text-to-Image Generation
Authors: Kunhee Kim
NaHyeon Park
Kibeom Hong
Hyunjung Shim
Venue: ICLR 2026 Poster (2026)
Keywords: personalized generation
text-to-image models
textual inversion
Primary area: generative models
TL;DR: We propose Directional Textual Inversion that improves text fidelity for personalized text-to-image generation.

Abstract: Textual Inversion (TI) is an efficient approach to text-to-image personalization but often fails on complex prompts. We trace these failures to embedding norm inflation: learned tokens drift to out-of-distribution magnitudes, degrading prompt conditioning in pre-norm Transformers. Empirically, we show semantics are primarily encoded by direction in CLIP token space, while inflated norms harm contextualization; theoretically, we analyze how large magnitudes attenuate positional information and hinder residual updates in pre-norm blocks. We propose Directional Textual Inversion (DTI), which fixes the embedding magnitude to an in-distribution scale and optimizes only direction on the unit hypersphere via Riemannian SGD. We cast direction learning as MAP with a von Mises-Fisher prior, yielding a constant-direction prior gradient that is simple and efficient to incorporate. Across personalization tasks, DTI improves text fidelity over TI and TI-variants while maintaining subject similarity. Crucially, DTI's hyperspherical parameterization enables smooth, semantically coherent interpolation between learned concepts (slerp), a capability that is absent in standard TI. Our findings suggest that direction-only optimization is a robust and scalable path for prompt-faithful personalization. Code is available at https://github.com/kunheek/dti.

Published as a conference paper at ICLR 2026
DIRECTIONAL TEXTUAL INVERSION FOR PERSONALIZED TEXT-TO-IMAGE GENERATION
Kunhee Kim NaHyeon Park Kibeom Hong" & Hyunjung Shim KAIST, 'Sookmyung Woman's University (kunhee. kim, julial9, kateshin, 0kaism0katttt.mmm ac. kb. wow.gesokkmyung.g....o a...ookkmmnnn................... ar.kr
ABSTRACT
Textual Inversion (TT) is efficient approach to text-to-image personalization but often fails on complex prompts. We trace these failures to embedding norm inflation: learned tokens drift out-of-distribution magnitudes. degrading prompt conditioning in pre-norm Transformers. Empirically, we show semantics are pri- marily encoded by direction in CLIP token space, while inflated norms harm contextualization; theoretically, we analyze how large magnitudes attenuate posi- tional information and hinder residual updates in pre-norm blocks. We propose Directional Textual Inversion (DTI), which fixes the embedding magnitude to an in-distribution scale and optimizes only direction on the unit hypersphere via Rie- mannian SGD, We cast direction learning as MAP with yielding a constant-direction prior gradient that simple and efficient to incorporate. Across personalization tasks, DTI improves text fidelity over TI and TI-variants while maintaining subject similarity. Crucially, DTI's hyperspherical parameteriza- tion enables smooth, semantically coherent interpolation between learned concepts (slerp), capability that is absent in standard TI. Our findings suggest that direction- only optimization robust and scalable path for prompt-faithful personalization. Code is available at httpp::// //t thub. com Konnkknheek /dii.. it
Mises-Fisher prior,
INTRODUCTION
Personalization in text-to-image generation involves the targeted adaptation of models to learn representations of novel, user-provided concepts (Gal et al., 2023a; Ruiz et al., 2023). This process allows for the creation of customized images that faithfully render specific concepts, such unique individuals, objects, or artistic styles, in new contexts.
Current personalization approaches fall into two paradigms: parameter fine-tuning and embedding optimization. Parameter fine-tuning methods, exemplified by DreamBooth (Ruiz et al., 2023), optimize entire models using few user-provided images. While effective, these approaches are computationally expensive and require significant storage per concept. In contrast, embedding optimization methods, such as Textual Inversion (Gal al., 2023a), offer a more efficient alternative by optimizing only token embeddings. This approach provides substantial advantages: minimal storage per concept and seamless workflow integration. These advantages have made a foundational component in numerous personalization frameworks (Hao al., 2023; Kumari al., 2023; Tewel et al., 2023; Lee et al., 2024) and align with a broader paradigm shared with other domains, such as LLMs (Lester et al., 2021) and VLMs (Alaluf et al., 2024).
Despite its utility, TI suffers from critical limitations. The fundamental challenge stems from the restrictive constraint of optimizing single embedding vector to encapsulate highly complex and multifaceted visual concepts. This limitation leads to two key problems. First, TI struggles to maintain high fidelity to complex prompts, compromising its controllability and expressive range. Second, the extensive fine-tuning duration required to learn each new concept hinders its practical applicability for rapid, user-driven workflows. Recent works (Voynov al., 2023; Alaluf et al., 2023) have attempted to address these limitations through enriched embedding spaces, but introduce significant
"Equal contributions.

---

Published as a conference paper at ICLR 2026
computational overhead that undermines TT's efficiency advantage. Moreover, these existing methods merely treat the symptoms rather than directly addressing the underlying optimization dynamics of TI, leaving the fundamental geometric factors that govern semantic alignment in embedding praedddig-basedd personalization largely unclear.
To bridge this gap, this paper presents a systematic, interpretability-dree driven analysis of the optimization dynamics inherent to TI, with a specific focus on the geometric characteristics of the token embedding space. By carefully dissecting the learned representations, our investigation reveals that semantic information is predominantly encoded in the direction of the embedding vectors. Furthermore, we demonstrate both theoretically and empirically that the unconstrained magnitude of these embeddings is primary source of instability; specifically, excessively high embedding norms emerge during optimization and act as a critical factor impairing image-text alignment.
Building on these findings, we introduce Directional Textual Inversion (DTI), a novel framework designed to address these fundamental limitations. Unlike conventional methods that optimize the entire token embedding, DTI decouples embeddings into their magnitude and directional components. Our approach maintains the embedding magnitude at a scale consistent with in-distribution tokens from the pre-trained model, while focusing the optimization exclusively on the embedding's direction. To enhance semantic coherence, we formulate this directional optimization as a Maximum a Posterioni (MAP) estimation problem. This formulation incorporates a directional prior, which effectively regularizes the embedding towards semantically meaningful directions in the hyperspherical latent space. The resulting framework preserves the lightweight nature of TI while significantly improving its robustness, ensuring that personalization is both computationally efficient and semantically faithful.
Mises- Fisher (vMF) distribution as
Our comprehensive evaluation demonstrates that DTI consistently outperforms conventional TI and existing enhancement methods such as CrossInit (Pang al., 2024a), achieving substantial improve- ments in semantic fidelity while maintaining computational efficiency. Beyond performance gains, the directionally optimized embeddings also enable novel applications, especially smooth interpolation between personalized concepts, expanding creative possibilities in generative Al workflows.
2 ANALYZING TOKEN EMBEDDING GEOMETRY
This section examines the token embedding space of pre-norm Transformer architectures, such the CLIP text encoder (Radford et al., 2021) and Gemma (Team et al., 2024), which are foundational to modern eext-to-image models. Our analysis establishes two key findings. First, we demonstrate that semantic information is primarily encoded in the direction of an embedding vector. Second, we identify that an excessively large embedding magnitude is common artifact of standard Textual Inversion, a phenomenon we show is detrimental to model performance. We substantiate these findings with empirical observations and subsequently develop a theoretical framework elucidate the underlying cause.
2.1 EMPIRICAL MOTIVATION: DIRECTION ENCODES SEMANTICS
Our first observation is that the semantic structure of the textual token embedding space is predominantly direc- tional. This aligns with the foundational principle of se- mantic vector spaces where meaning is encoded not in the vector's 's magnitude, but in its direction (Mikolov et al., 2013; Pennington et al., 2014). We empirically demon- strate this by comparing nearest neighbors for a given token using two different distance metrics: Fuclidean dis- tance, which is sensitive to both magnitude and direction, and cosine similarity, which sensitive only to direction. The superior semantic coherence of neighbors found using cosine similarity validates the principle that meaning in these vector spaces is encoded primarily by direction.
Table 1: Top 5 nearest tokens 'apple' under different measures.
Rank
Euclidean
Cosine
1
U+2069
apples fruit
2
altrin
3
lestwe
peach
4
heartnews
pear
5
samanthaprabhu
egg
As shown in Table I, embedding's 's nearest neighbors semantically coherent when measured by cosine similarity but not by Euclidean distance. For the token 'apple', its cosine-based neighbors include "apples', 'fruit", and 'pear', while its Fuclidean-based neighbors are often semantically
2

---

Published as a conference paper at ICLR 2026
<oat>
jungle
Norm rescale
TITI (nurs)
forist grass
gray
griin lion
Text Encoder (1)
wolf tiger
tail
Learaed Tolan (2)
leopard
ears
Count
15k
Loaanoee Tolaze
pallzting of ddog wearing
with Mouulaak the bau eground
Riccin
(ours)
sitting
walking
jumning
10k
<eat>
sad
cat
angry
5k
kitten
happy
dog
sleeping
smile
30
puppy
emmoji
Norm
ln busieet, art ztgle
(a) Norm inflation
(b) Semantic drift
Figure I: Empirical motivation for our method. Our analysis reveals two critical problems in standard TI that degrade prompt fidelity. (a) TI produces embeddings with excessive norms compared to model's original vocabulary. (b) TI also suffers from semantic drift, where learned embedding direction moves away from related concepts. These observations motivate DTI, approach designed to preserve both norm and directional integrity.
unrelated tokens that merely share a similar magnitude. This indicates that an embedding's direction the primary carrier of semantic information. More results are provided in Appendix A.
Figure lb further illustrates this principle, showing that related concepts are located proximally on the unit hypersphere. Despite this, standard TI often neglects the importance of direction. This oversight leads to semantic drift, where the learned embedding for a token like <cat> moves directionally away from related concepts like 'cat' and 'kitten', shown in the figure. This deficiency motivates the need for method that explicitly preserves the semantic direction of learned embeddings.
2.2
WHY LARGE MAGNITUDES LEAD TO LOW TEXT FIDELITY
As shown Figure la, TI produces token embeddings with norms that are drastically larger than those of the pre-trained vocabulary (often > 20 vs. R 0.4). These nut-of-distribun consistently correlate with poor prompt fidelity. For instance, a prompt like "A painting of <dog> wearing santa hat" may generate the dog but omit the hat and background details. While simply rescaling the embedding's norm after training can partially recover text alignment, does not solve the underlying issue and can degenerate subject similarity. This raises a critical question: why do large embedding norms degrade text fidelity in pre-norm Transformers?
(OOD) magnitudes
Our analysis reveals two primary mechanisms through which large-norm embeddings disrupt the Transformer's s ability to contextualize information. We analyze a standard pre-norm Transformer block, y
Fe (zorm(z))) where Norm E [LayerNorm, RMSNorm) and F/ denotes atten- tion/MLP sub-layers. We decompose the learned token (0) ||0|| = 1 (direction), and an additive positional embedding Below, we explain how a large magnitude undermines the model's performance. (For formal proofs, Appendix B).
+ p with > 0 (magnitude),
Effect Positional information attenuated (see Lemma 1). After LayerNorm/RMSNorn layer, the normalized signal that feeds attention/MLP becomes less sensitive small additive terms as m grows. Positional information contributes O(1mm) to the normalized signal Norm + p). Intuitively, very large-norm token forgets where is the sequence, weakening contextualization. resulting in omission of details such as style and background (see Figure 1).
Effect I: Residual updates stagnate (see Lemma 2). The residual updates, ((Norm(z(()) (0)), are computed from normalized inputs and thus have bounded magnitude. When this bounded update is added through the skip connection to large vector (1)), the relative change (i.c., turning angle of the hidden state's 's direction) becomes tiny, decreasing in proportion to [f] In other words, large-norm hidden states become stuck in their direction and are difficult for subsequent layers to refine. This residual stagnation accumulates across layers, severely limiting the total directional change the initial token can undergo, as formalized in the following proposition and corollary.
3

---

Published as a conference paper at ICLR 2026
Proposition (Aceumulated directional drift across L pre-norm blocks). Let x + 0 and x (f+1)) (1()) + Fe (N(Norm(z(()) for f. o,....I. 1. Let B; = suPues [F(()))22 < and St.
Σj= By. Assume || 112 > SL, then
4x x (2)
B₁ Ej<e
π
S x
2 S
t=
Corollary (Scaling = directional freezing).
With the notation of Proposition for any 1,
<(az (0) (x))
S
$
O,
2 a x St.
a-v00
where x (a) denotes the depth-L L output when the initial token is ax
Together, these two effects explain why TI struggles with text fidelity. As a token's magnitude increases, its ability to integrate contextual information from the prompt diminishes. The personalized token becomes so dominant that it overshadows other critical details, such as stylistic elements, background context, or additional subjects, from the generated output. To this end, this analysis highlights the need for method that explicitly controls the magnitude of personalized tokens, which we introduce in the next section.
2.3 EMPIRICAL VALIDATION
We empirically validate the two theoretical effects introduced in the previous sections. Effect I describes the attenuation of positional information under large embedding magnitudes, while Effect concerns residual-update stagnation in pre-norm Transformer blocks. Our experiments directly probe both behaviors on the base encoder, TI, and our proposed DTI.
Effect I (Attenuation of positional information). We evaluate whether increasing embedding magni- tude makes positional information unrecoverable after the first pre-norm normalization (LN). To validate this, train 2-layer MLP classifier on the frozen base text encoder to predict token's absolute position from the output of LN(c + p), where e and each denote token and positional embeddings. On unmodified inputs ('Normal' Figure 2), the classifier achieves 100% accuracy, confirming that LN preserves positional informa- tion. We then scale the norm of single token embedding by a factor m E (0.5, 1, 2, 4, 8, 16) before applying LN. Accuracy deteriorates rapidly Furthermore, we evaluated the classifier on TI-trained and DTI-trained personalized embeddings. TI embeddings, which have excessively large norms, collapse near-zero positional accuracy, while DTI embeddings remain fully recoverable.
1.0
@
1.000
1.000
0,846
Accuracy
0.6
0.4
0.2
0.076
0.008
0.008
0.000
Original 0.5
DTI
Setting
(ours)
Figure Position prediction accuracy from LN outputs under varying embedding magnitudes, compared with trained TI and DTI embeddings.
m exceeds the natural scale of the encoder.
This behavior directly corroborates Lemma 1: when m becomes large, LN + p) becomes dominated by mvv rendering the positional component effectively invisible. DTI avoids this failure mode by constraining magnitudes to remain in in-distribution.
Effect II (Residual-update stagnation). To test Lemma 2, we measure the internal angular change of hidden states within each pre-norm Transformer block. For each concept token, we compute the angle between the hidden state entering and exiting each block and then average across all layers. The average plo-block angular change of TI embeddings was 21.33", whereas the angular change of DTI embeddings 33.52" (1.57x larger).
These results support the theoretical prediction that excessively large norms suppress the effective residual direction in pre-norm blocks, causing the forward computation to behave nearly as identity mapping. By keeping embedding norms within the training distribution, DTI prevents such stagnation and permits substantially larger and more meaningful updates throughout the encoder.
4

---

Published as a conference paper at ICLR 2026
3 METHOD: DIRECTIONAL TEXTUAL INVERSION
Based on the observation and analysis in the previous section that token embeddings exhibit strong directional characteristics, introduce Directional Textual Inversion (DTI, a framework that optimizes an embedding's s direction with in-distribution norm to enhance text fidelity in personalized text-to-image generation.
3.1 OPTIMIZING ONLY DIRECTION ON THE HYPERSPHERE
We reformulate TI by decoupling the mag- nitude and direction of the learnable token embedding E Rd The embedding be expressed as
Algorithm 1 Directional Textual Inversion (DTI)
1: Inputs: Model ép, text encoder c(-), init token Einit magnitude m K, iterations K, learning rate
3: Vo < Eiitt ||
e m U
v E S0-1
(1)
t Ginit/ e 1
Here, S = {u E Rd = denotes the unit sphere. We fix the mag- nitude m and optimize only the direction (v). Specifically,
=
do 5: Sample minibatch (z, c)
for k: to
Idata t- V. Lanta (m (m
set m² to be an in- distribution magnitude derived from the frozen vocabulary of the text encoder (e.g., the average norm). In this way, optimiza- tion focuses on semantic information in di- rection while avoiding out-of of-disrribuutnn (OOD) norms.
7:
Jeuc t Idata kipe
(add prior gradient)
8:
t Jeuc (g U2) Up.
(tangent projection)
9:
g' t g/ /19112
(gradient scaling)
Us 79
10: Uk+1 f |us 7g'll 22
(retraction to S Sd-1)
II: end for
12: return e" = mmnuK OK
Since the parameter space is the unit sphere, Euclidean updates drift off- manifold, rendering AdamW (Loshchilov & Hutter, 2019) the default optimizer for TI-like descent (RSGD) (Bonnabel, 2013) with tangent-space projection and retraction:
To solve this, use Riemannian stochastic gradient
g = Jene (v) Jene) Us E TopSa-1
Uk+1 = Retro =
(2)
22 11 l1
Here, Sew Euclidean space gradient, 9 E T., Sd-1 is tangent-space gradient, and r > 0 is learning rate. In practice, we further scaled by its norm. This was inspired by Euclidean space optimizers (Hinton et al., 2012: Kingma & Ba, 2015; Loshchilov & Hutter, 2019), which normalize gradients based on moving averages of squared gradients. See Algorithm and Appendix C.I for further details.
3.2 MAXIMUM A POSTERIORI FORMULATION WITH A DIRECTIONAL. VMF PRIOR
To incorporate a directional prior, formulate the optimization for the optimal direction v² Maximum A Posteriori (MAP) estimation problem. Given a dataset images D {2 Zx). the MAP estimate is found by maximizing the posterior probability:
v' arg max p(v D) = arg max ((D v) + p(v).
(3)
Minimizing the negative log-posterior is equivalent to minimizing loss function composed of data term and prior term, L(v) Lutaa (m" v) + Eprioo()).
The data term, Ldsta log p( v). is the negative log- likeliieehood of the images given the direction. Following standard practice for diffusion models (Ho et al., 2020), we use the mean squared error (MSE) between the true and predicted noise as the objective:
Canta(mm v) = Ez Gex..elllee E e t,mm", c(mm))))
(4)
Here, Eg and c(-) are the diffusion model and text encoder, respectively. For notational simplicity, write c(m*v) v) as shorthand for the text conditioning obtained from sampled prompt template containing the personalized token with embedding m v, and omit the explicit dependence on text prompt throughout the paper.
5

---

Published as a conference paper at ICLR 2026
For the prior term, - llog pvv), we use von Mises-Fisher (vMF) distribution on the direction (detailed justification in Appendix C.2). The vMF distribution is a probability distribution on the (d - 1) s-ppher,, analogous to the Gaussian distribution in Euclidean space. It is parameterized by mean direction E S and a concentration parameter k. 2 0. The probability density function is given by:
42/2-1
p(vl K) = (2 (or)d/21a/221 exp(ku
(5)
where 1a/2-1 is the modified Bessel function of the first kind. Here, we work with unnormalized density: p(v) x exp(xμTTe), Ignoring constants, the negative log-prior yields our regularization term, Sprirr (t) ki v.
Constant-direction prior gradient. useful property that the Euclidean gradient of the regu- larization term is a constant: V. v.(-xuTu v) -k4.. Practically, we just add this vector to the data gradient before projecting to the tangent space and retracting. This is analogous in spirit to decoupled weight decay (Loshchilov & Hutter, 2019), but adapted for the sphere with a directional prior. The update is computationally cheap (requiring no new graph operations), numerically stable, and highly interpretable: it applies constant pull towards semantically meaningful direction.
Selection of vMF parameters. The vMF prior is defined by mean direction μi and a concentration parameter K. The mean direction lr is set to the normalized embedding of corresponding class token (e.g., 'dog') from the pre-trained text encoder and is held constant during optimization. Since estimating is non-trivial, we treat it as a hyperparameter that controls the strength of the prior. We performed a grid search and found that values in the range of 5e-5 to 2 4 work well. Based on this, we simply fixed the value of 1e-4 4 for all experiments. Further discussion on the selection of prior can be found in Appendix D.2 and D.4.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
All experiments were implemented using PyTorch (Paszke et al., 2019) and the HuggingFace di f fusers library (von Platen et al., 2022), with a single NVIDIA A6000 GPU. Detailed imple- mentation specifications are provided in Appendix D.I.
Datasets. For subject personalization, we employed all reference images from the DreamBooth dataset (Ruiz al., 2023). Additional experiments stylization and face personalization presented in Appendix D.7, utilizing StyleDrop (Sohn et al., 2023) and images from FFHQ (Karras et al., 2019). We evaluated all methods using 40 prompts, comprising the complete set of prompts from the DreamBooth dataset supplemented with 10 additional complex prompts.
Models. Unless otherwise specified, employed Stable Diffusion XL (SDXL) (Podell et al., 2024) as our primary model due to its superior performance and widespread adoption in recent literature. To demonstrate DDTTs applicability to more recent architectures, we conducted additional experiments on SANA 1.5 (Xie et al., 2025), which employs Gemma (Team et al., 2024) as the text encoder and DiT (Peebles & Xie, 2023) as the image generator.
Baselines. Our method extends Textual Inversion (TT) (Gal et al., 2023a), serving as our primary baseline for direct comparison. We additionally evaluate against CrossInit (Pang et al., 2024a), an enhanced TI variant that incorporates specialized initialization and regularization techniques. Com- prehensive comparisons with additional baselines. including P+ (Voynov et al., 2023), NeTI (Alaluf et al., 2023), CoRe (Wu et al., 2025), and DCO (Lee et al., 2024) are provided in Appendix D.3.
Metrics. Following established evaluation protocols (Ruiz et al., 2023; Kumari et al., 2023; Gal et al., 2023a), we assessed each method across two primary dimensions: subject fidelity and image- text alignment. Subject fidelity was quantified using DINOv2 (Oquab et al., 2023) feature cosine employed SigLIP (Zhai et al., 2023), a more recent variant
similarity. For image-text alignment, of CLIP, following recent work (Lee et al., 2024). For each instance, we generated samples from 40 text prompts using 4 random seeds, yielding 160 samples per instance. Complete evaluation details are provided in Appendix D.I. Results were further validated through a user study conducted via Amazon Mechanical Turk.

---

Published as a conference paper at ICLR 2026
Table 2: Our DTI consistently improves baselines by generating outputs with enhanced text fidelity while maintaining subject similarity.
Table 3: Ablation studies. We tested and confirmed the effectiveness of every com- ponent of our DTI.
SDXL
SANA 1,5-1,6B
SANA 1,5-4.88
Optimizer
K: 10
Image
Text
Methods
Image
Text
Image
Text
Image
Text
AdamW
muca
0.335
0.463
RSGU
min
0.7
00030
0.074
TI
0.561
0.292
0.480
0.621
0.446
0.646
(OOD)
Ti-rescaled
0.243
0.287
RSGD
mean
0.0
0.507
0.436
Crosslnit
0.545
0.464
0.344
0.614
0.299
0.622
RSGD
meam
0.5
0.278
0.688
DTI (ours)
0.450
0.522
0.479
0.744
0,452
0.757
RSGD
0.7
0:450
0.522
Inpar
CrossInit
DTI (ours)
A <plusshie>
A < <boot> abandoned on muusio stage, illuxinated by a spotlight
Pop- Art styll Illus:ration of vibrant neon colors
A penoil sleetsh of <emaji>> talkingg happily with sweji
witercolor painting of <sunglasses>> on the beach at sanset
A <slog>> > heving oup of Raroous painting week (ubrisate detail
A vintage-style photograph <dogg> reasting a kooke by windlow
wearing
wizarsi castuoug and holsting Jaole-o-tantemm, photorealistio digital art
Figure We compare DTI with previous methods across diverse subjects and textual prompts. spanning simple descriptions to complex variations in attributes, backgrounds, and styles (same random seeds). All results in this figure are generated with SDXL (SANA Appendix Figure 9).
4.2
MAIN RESULTS
Quantitative results. Table 2, we quantitatively evaluate DTT along two axes: subject similarity and text-prompt fidelity. DTI consistently produces outputs that adhere closely the prompt while maintaining high subject similarity. To isolate the role of embedding norm analyzed in Section 2.2, we rescaled TT's learned embeddings the in-distribution norm-speeifiiallly the average norm of the vocabulary embeddings, matching the norm scale used in DTL. Consistent with our analysis, this simple rescaling noticeably improves text fidelity but does not fully resolve the problem, image similarity. CrossInit achieves strong text fidelity SDXL. but fails to do so consistently on SANA, which we attribute to differences their text encoders; SDXI. uses a CLIP text encoder, while SANA employs the LLM-based encoder. Notably, DTI's advantage over the basclines becomes even more pronounced as the model size increases. Overall, these results clearly demonstrate the advantage of DTI over competing baselines. Additional comparisons with further baselines other Stable Diffusion variants are provided Appendix D.3.
degrades
Qualitative results. Figure 3 illustrates qualitative comparisons across various prompts. DTI consistently generates images that more accurately reflect the content of the captions, while effectively preserving subject consistency. For instance, for 'Pop-art style illustration of <cat>', Tl omits the
7

---

Published as a conference paper at ICLR 2026
cat while DTI renders the cat in the specified style. Similarly, in the second column, TI and CrossInit fail to incorporate all elements of the prompt, disregarding either the subject or details such as 'music stage' and 'spotlight'. In contrast, DTI integrates both the subject and these details, producing more complete output. Collectively, these examples highlight DTI's superior compositional fidelity and subject preservation, demonstrating that it consistently satisfies all prompt constraints. We attribute this to DTI's stable optimization within the directional space, which facilitates improved integration of multiple prompt components. DTI's ability to maintain subject fidelity and adhere to textual intent establishes it as a robust choice for wide range of text-to-image generation tasks. Additional qualitative results including those of SANA can be found in Appendix D.6.
4.3 ABLATION STUDY
We performed an ablation study to verify the effectiveness of components of our DTI, including the optimization space, the embedding magnitude m, and the concentration parameter of the vMF distribution. The results are summarized Table 3. To validate our choice of Riemannian SGD (RSGD), we compared it against baseline using the Adam optimizer. This baseline performs standard Fuclidean updates and then projects the vector back onto the unit sphere after each step, which is not true Riemannian update. The results show that RSGD substantially outperforms AdamW, highlighting the benefit respecting the geometry of the directional manifold. Next, we found that fixing the magnitude to the minimum to out-of-distributin either subject similarity or text fidelity. Setting the magnitude to an in-distribution scale yields the best results. Lastly, removing the prior (i.e., while moderate incorporation of the prior provides the most stable results. Overall, we confirm that these ablation results validate our design choices. Further analyses are provided in Appendix D.4.
scale negatively affected
0) extremely high values of x hurts the performance,
4.4 HUMAN EVALUATION
To further examine the effectiveness of our method, we conducted a large-scale user study (100 participants via Amazon Mechanical Turk) measure real-world user preferences. Each partic- ipant was asked to respond 20 questions, com- prising 10 questions assessing subject fidelity and 10 questions evaluating image-text alignment. Par- ticipants were instructed to select the output that best met the specified criteria for each question. To ensure the reliability of the study, excluded four user responses that did not adhere to the specified instructions. A fixed random seed was employed, and the answer options were shuffled for each question. The results, summarized in Table 4, show that DTI consistently outperforms the other methods on both metrics, indicating that its improvements in alignment are clearly perceived by human evaluators. More details of this user study can be found in Appendix D.5.
Table 4: We surveyed real-world user prefer- ences regarding subject fidelity aligmment. DTI ranks the top in both metrics, confirming its practical benefits.
and image-text
TI CrossInit
DTI (ours)
13.78 Text alignment
Image fidelity
42.87
43.45
10.83
22.40
66.77
4.5 EMBEDDING INTERPOLATION FOR CREATIVE APPLICATIONS
We demonstrate the creative potential of our DTI through embedding interpolation experiments. As illustrated in Figure 4, our DTI generates coherent interpolations via spherical linear interpolation (SLERP), which matches the unit-sphere parameterization. This capability is a direct result of DTT's unit-spherical embedding space, which enables smooth and effective transitions. In contrast, the linear interpolation used by TI often fails to produce coherent intermediate results.
The advantages of our approach are clearly visible across different domains. As shown in the first rows of the figure, one can seamlessly merge a dog and a teapot, resulting in imaginative hybrid objects like an adorable teapot that progressively adopts the features of the dog. This indicates that DTI excels at blending conceptually distinct subjects, significant creative application. In the second example, it can create a creative animal between a dog and cat, that merges the features of each animal in smooth manner. Lastly, DTI smoothly interpolates between the faces of a young boy and an older woman, generating a plausible progression that simultaneously alters age and appearance while maintaining facial coherence. This highlights its potential for nuanced face personalization.
8

---

Published as a conference paper at ICLR 2026
<dog>
<teehot>
DTT (oure))
<dog>
<ott>
DTT (ours)
<ohild>
<adutt>
TI
DTI (0urs)
Figure 4: We compare images generated by TI and our DTI. Two personalized subjects are inter- polated, including interpolation between inanimate and animate subjects, live subjects, and human faces. Images are generated with interpolation ratio [0.0, 0.35, 0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 1.0] for better visualization. Our DTI offers smooth interpolation between concepts, expanding personal- ization along a more creative axis.
Throughout these transitions, DTI produces visually consistent and creative outputs that retain semantic meaning, unlocking novel user-driven applications and establishing powerful tool for intuitive concept blending. We provide the results other applications, including face personalization, stylization and subject-style generation throughout Appendix D.7.
5
RELATED WORK
5.1 PERSONALIZED TEXT-TO-IMAGE GENERATION
Recent advancements in text-to-image (T2I) generation have considerably expanded the creative capabilities and flexibility of generative models (Ramesh al., 2021; Rombach et al., 2022: Nichol et al., 2022; Ramesh et al., 2022; Yu et al., 2022; Podell et al., 2024). Despite these innovations, natural language inherently struggles to precisely convey nuanced, user-specific concepts. This inherent limitation has driven the development of personalization methods, which allow users to generate images reflecting unique concepts with creative prompts.
Textual Inversion (Gal et al., 2023a), which is most well-known for its lightweight integration into many other personalization works, uses embedding optimization by introducing learnable tokens for personalized information without model modification. Subsequent work explored diverse embedding strategies (Voynov et al., 2023; Alaluf et al., 2023; Wu et al., 2025; Zhang et al., 2024a), often excessive computational cost. Among them, CrossInit (Pang al., 2024a) offered efficient initialization strategy with minimal overhead, replacing initialization tokens with the output of text encoder and using regularization loss.
In contrast, fine-tuning based methods such as DreamBooth (Ruiz et al., 2023) achieve high subject fidelity, but require significant computational resources compared to embedding optimization meth-

---

Published as a conference paper at ICLR 2026
ods (Kumari et al., 2023; Han et al., 2023; Gu et al., 2023; Chen al., 2023a; Tewel et al., 2023; Zhang et al., 2024b; Qiu et al., 2023; Pang et al., 2024b). More recently, Park et al. (2024) proposed fine-tuning the text encoder instead of the image generator for efficiency, but they still demand more parameters compared to embedding optimization methods.
Meanwhile, there exists a line of encoder-based approaches (Wei et al., 2023; Ruiz et al., 2024; Ye et al., 2023; Gal et al., 2023b; Chen et al., 2023b; Li et al., 2023; Pang et al., 2024b; Ma et al,, 2024) that offer fast inference, but they necessitate substantial pre-training.
5.2 DIRECTIONAL EMBEDDING SPACE
A number of prior works have emphasized constraining embedding representations to the hypersphere. These include using vMF mixtures for directional clustering (Jameel & Schockaert, 2019), normaliz- ing norms for face recognition (Meng et al., 2019), ongle-optimized embeddings to address cosine saturation (Li & Li, 2024), and spherical constraints for uniform document clustering (Zhang et al., 2020). Wang & Isola (2020) offered theoretical support for hyperspherical constraints in contrastive learning. Our method aligns with this trend by modeling embeddings as directional distributions but uniquely decomposes and explicitly optimizes textual embedding direction using a vMF prior within the Textual Inversion framework.
6 DISCUSSION & CONCLUSION
Our DTI primarily improves text prompt fidelity as it does not directly optimize for subject similarity. For applications where high subject fidelity is paramount, DTI can be used conjunction with complementary lightweight fine-tuning methods, such LoRA, Figure 11. Furthermore, our analysis is centered on the geometry of modern pre-norm text encoders. An interesting direction for future work would be to investigate whether our findings generalize to other types of encoders with different normalization or positional encoding schemes.
demonstrate qualitatively in
Overall, work tackles key challenge in personalized text-to-image generation: achieving strong alignment between text prompts and generated imagery. We have identified and rigorously analyzed embedding norm inflation empirical evidence of its detrimental effects. In addition, our investigation focuses on the directional characteristics of the token embedding space, an area that has been comparatively underexplored in the literature, particularly when contrasted with the extensive research dedicated to the output embedding space of text encoders. Leveraging this key insight into the semantic significance of token embedding directionality, we proposed Directional Textual Inversion (DTT), a novel framework that keeps the embedding norm at an in-distribution scale and solely optimizes the direction. We further reformulate the conventional Textual Inversion optimization process by incorporating directional priors. Our DTI demonstrably enhances prompt fidelity, thereby substantially improving the practicality of token embedding-basee based personalization and enabling innovative creative applications such as the smooth interpolation of learned concepts. We truly hope our work paves the way for more effective and versatile token umbedding-baset personalization within generative AI, unlocking enhanced capabilities for users to articulate their unique creative visions with greater precision and control.
significant bottleneck to this alignment, providing both theoretical and
ACKNOWLEDGMENTS
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP (No. mation & Communications Technology Planning & Evaluation (IIPP) grants funded by the Korea government (MSIT) (Nos. 2022-0-00680, 2022-0-01045 RS-2024-0045788) RS-2025-0221 II190075), including the National AI Research Lab Project and the Artificial Intelligence Graduate School Support Program (KAIST); and the Korea Evaluation Institute of Industrial Technol- ogy (KEIT) grants funded by the Korea government (MOTIE) (Nos. 2022-0-00680 2022-0-01045. RS-2025-022172259..
the Institute of Infor-
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the full reproducibility of our research, we provide our complete source code, experimental details, and dataset information. We utilized publicly available datasets, primarily from Dream- Booth (Ruiz et al., 2023), FFHQ (Karras et al., 2019), and StyleDrop (Sohn et al., 2023), and our repository includes scripts for all necessary preprocessing. Additionally, all software dependencies are explicitly specified in the pypro ject. to file. All experiments were conducted on single NVIDIA A6000 GPU, with training taking approximately 7 minutes per subject for SDXL-bbase and 30 minutes for SANA 1.5-1.6B. To guarantee transparency and ease of use, all hyperparameters are detailed the Appendix and are also included in the run scripts within our codebase.
LLM USAGE STATEMENT
We utilized Large Language Models (LLMs) to improve the grammar and clarity of this manuseript. The core research, including the analysis and method, is the exclusive work of the authors.
REFERENCES
Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text-to-imape personalization. ACM Transactions Graphics, 42(6):1-10, 2023.
Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvlm: Personalizing vlms for user-specific queries. In European Conference on Computer Vision, pp. 73-91. Springer, 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.0644 2016.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 139233 2025.
Silvere Bonnabel. Stochastic gradient descent on Riemannian manifolds. IEEE Transactions Automatic Control, 58(9):2217-2229) September 2013. ISSN 0018-9286, 1558-2523.
Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity- preeiryyprreeviiggg disentangled tuning for subject-driven text-to-image generation. In International Conference on Learning Representations, 2023a.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. In Advances in Neural Information Processing Systems, 2023b.
Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems, 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(7), 2011.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. In International Conference on Learning Representations, 2023a.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics, 42(4):1-133 2023b.
Yuchao Gu, Xintao Wang. Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. In Advances in Neural Information Processing Systems, 2023.
11

---

Published as a conference paper at ICLR 2026
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan- Yee K. Wong. ViCo: for personalized text-to-image generation. arXiv preprint arXiv:2306.0000 00971, 2023.
visual condition
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky, Lecture 6e: Rmsprop. Coursera, Nonlinear
aural component analysis of the cochlea, 2012. URL http:/ //t::///wwwww./ cs wwwp:ccsttrooto.... toronto.edu/ "t
p.f..
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, 2020.
Shoaib Jameel and Steven Schockaert. Word and document embedding with vmf-mixture priors on context word vectors. In Annual Meeting of the Association for Computational Linguistics, 2019.
Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi Mulci-Concept Customization of Text-to-Image Diffusion. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, and Jinwoo Shin, Direct Consistency Optimization for Compositional Text-to-Image Personalization. In Advances Neural Information Processing Systems, 2024.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104. 080991......... 2021.
Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. In Advances in Neural Information Processing Systems, 2023.
Xianming Li and Jing Li. Aoe: Angle-optimized embeddings for semantic textual similarity. In Annual Meeting of the Association for Computational Linguistics, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. International Conference on Learning Representations, 2019.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personal- ized text-to-image generation without test-time fine-tuning. In ACM SIGGRAPH Conference Proceedings, 2024.
Yu Meng, Jiaxin Huang. Guangyuan Wang, Chao Zhang, Honglei Zhuang, Lance Kaplan, and Jiawei Han. Spherical text embedding. In Advances in Neural Information Processing Systems, 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 2013.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, llya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In International Conference on Machine Learning. 2022.
12

---

Published as a conference paper at ICLR 2026
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, July 2023. ISSN 2835-8856.
Lianyu Pang. Jian Yin, Haoran Xie, Qiping Wang, Qing Li, and Xudong Mao. Cross initialization for face personalization of -ot-tt--mmage models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Lianyu Pang, Jian Yin, Baoquan Zhao, Feize Wu, Fu Lee Wang. Qing Li, and Xudong Mao. Attndreambooth: Towards text-aligned personalized text-to-image generation. In Advances in Neural Information Processing Systems, 2024b.
NaHyeon Park, Kunhee Kim, and Hyunjung Shim. TextBoost: Towards One-Shot Personalization of Text-to-Image Models via Fine-tuning Text Encoder. arXiv preprint 240ii::2009008248,, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorcc: An Imperative Style, High- High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference Computer Vision, 2023.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Conference Empirical Methods in Natural Language Processing, 2014.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In International Conference on Learning Representations, 2024.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Advances in Neural Information Processing Systems, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot text-to-Image Generation. In International Conference on Machine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 066202 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine Tuning text-tto-Image Diffusion Models for Subject- Driven Generation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfrr Aberman. Hypendreambooth: Hypemeeworks for fast personalization of eoxt-to-mmage models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
13

---

Published as a conference paper at ICLR 2026
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, Yuan Hao, Irfan Essa, Michael Rubinstein, and Dilip Krishnan. StyleDrop: Text-to-mmage Generation in Any Style. In Advances in Neural Information Processing Systems, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08225, 2024.
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to- image personalization. In ACM SIGGRAPH Conference Proceedings, 2023.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https: /ggth/bttt.////gg// www.ccoo//hhheeeeeee fggg//ggggngffce////////o diffusers, 2022.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual condition- ing in text- text-to-image generation. arXiv preprint arXiv:2303.09522, 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, 2020.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In IEEE/CVF International Conference on Computer Vision, 2023.
Feize Wu, Yun Pang, Junyi Zhang, Lianyu Pang, Jian Yin, Baoquan Zhao, Qing Li, and Xudong Mao. Core: Context-regularize text embedding learning for text-to-image personalization. In AAAI Conference on Artificial Intelligence, 2025.
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang. Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. In International Conference on Learning Representations, 2025.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for -nt-to-immae diffusion models. arXiv preprint arXiv:2330.. 06721, 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content- Text-to-Image Image Generation. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, 2023.
Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems, 2019.
Dingyi Zhang. Yingming Li, and Zhongfei Zhang. Deep metric learning with spherical embedding. In Advances in Neural Information Processing Systems, 2020.
Xulu Zhang, Xiao- Wei, Jinlin Wu, Tianyi Zhang, Zhaoxiang Zhang, Zhen Lei, and Qing Li. Compositional inversion for stable diffusion models. In AAAI Conference on Artificial Intelligence, 2024a.
Yanbing Zhang, Mengping Yang, Qin Zhou, and Zhe Wang. Attention calibration for disentangled text- to-image personalization. In IEEF/CVF Conference on Computer Vision and Patlern Recognition, 2024b.
14

---

Published as a conference paper at ICLR 2026
A EMBEDDING NORM AND DIRECTION
magnitude
Original norm
m- 0.5
m 1.0
Figure 5: Effect of magnitude change. We set the magnitude to fixed value to analyze the impact of magnitude changes. The resulting outputs show no noticeable difference.
We altered the magnitude of the token as exemplified in Figure 5. However, the resulting output re- mained mostly unchanged. This indicates that minor adjustments to the magnitude do not significantly affect the outcome.
Table 5: Nearest tokens under different measures. We show the nearest tokens to the query words "study' and 'writing" using both cosine similarity and Fuclidean distance.
Query
Cosine
Euclidean
studics, studying, research, bookelub, reading, studied, sketches, measurements, thumbnail
31 60, texanscheer, asober, instoweatherpro, mydayin, premiosmtvmiaw, taipp, thepersonslaewwork U+2412
study
writer, write, written, writ, writers, writings, recording, blogging, wrote
phdlife, poetryday, tonorowspaper urstrulmmahesh,
writing
twitterkurds, asober, fakespeare, jamiedor
In Table 5, we provide additional examples illustrating the nearest words retrieved for each query under different similarity measures, which strongly correlate with either direction or magnitude. Our analysis reveals that cosine similarity retrieves words that share semantic meaning with the query. Conversely, Euclidean distance significantly affected by embedding magnitude, often retrieving words with limited or no semantic relevance. This demonstrates that semantic meaning is predominantly associated with embedding direction rather than magnitude. Note that words beginning with U+ denote Unicode.
B PROOFS FOR THEORETICAL STATEMENTS
B.
SETUP
Pre-norm block. We study pre-norm Transformer blocks
(f) + Fc(Norm(zz!")), {= f= ae
(6)
where Norm E [LayerNorm, RMSNorm] } (with optional affine (v, 3) absorbed into F?).
Scale invariance. For normalizations, use the standard, scale-invariant definitions:
RMSN(x) = vd
x |0l|2
LN(x) = Vd. |Call
Cr
11 C d
Thus RMSN(s2) RMSN(()) and LN(sx) LN(x) for all 0. Please refer original papers (Ba et al., 2016; Zhang & Sennrich, 2019) for further details.
Token decomposition. For the input token, we denote (0) = mo + p with m > 0, ||/||| 1, and (optional) absolute positional embedding p E Rdd
Bounded sub-layers. Define S = (Norm(z) : : # 0). Since Norm maps into a fixed scale, bounded set and Fi (attention/MLP plus projections) is continuous on bounded sets,
B₁ = sup [u < co.
(8)
n
Throughout, || denotes the Fuclidean (12) norm.
15

---

Published as a conference paper at ICLR 2026
B.2 POSITIONAL ATTENUATION
Lemma 1 (Absolute positional embedding attenuates as m - xo), Let xc mo with l|o|| 1, m > O, and absolute positional embedding E TRd. Suppose Norm E [LayerNorm, RMSNorm) and non-degenerate for LayerNorm (i.e., its per-feature variance nonzero; this holds for generic token embeddings). Then
- [lllll
|Norm(mm + p) Norm(mo) |1 O
as m - oc (with v, p. fixed).
m
Hence the positional contribution shrinks linearly 1/m.
Proof. By scale invariance, Norm(me p) Norm(v +e) with : p/ m, and Norm(mo) Norm(v).
RMSNorm. With ||a|| =1,
V V+E
= e (La - vo )) + +0(l=))))
+el
hence RMSN(U + 8) - RMSN(v) [IRMS((MU p) (ARRRMSSM)))) S vd |p|| /mm ((m
Vdaaa vo vo )s + O(1I1 and
LayerNorm. Write a Cv * 0, u = a/ ||a|] Then
a + Call u (Ia uu CC
Ce
+ O(1l))),
a|
so [m +p) LN(m))] S Vd (Ly- Jipll Cpll + O(m (),, which is O(Ippl /m).
Cl|C
B.3 RESIDUAL STAGNATION
Lemma 2 (Residual stagnation in a pre-norm block). Let (C+1) x(() + Fc(Norm(zx(0)) with
(1) # 0 and Norm E {LN, RMSN], and let
B₁ sup uES
|eee())l2 < <0.
Then
|| (e:1)+1) 10(01112
B ||x
|,0(01112
If additionally Be ||2(0
B₁
2 (xx x (f+1)) < arcsin( | 1|2
Proof. Since Norm(x E S, we have || x (1) 11 [N S Be, giving the first bound. Write (f+1) = (()) + 6. The orthogonal component of is at most [|ō]|; a short (((()))((+)) < |101|// (|]12/1(=(()1
calculation shows sin ((x(1), ((++1)) angle bound.
1l2 < Be/l|=(00/l, which implies the stated
Proposition (Accumulated directional drift across L pre-norm blocks), Let (0) # and (f+1) ( + (((Nor((z(())) for = O,...,I 1. Let Bg supues [F((u))l2 co, and St. | ||2 > St, then
L-1 Bj. Assume
(x x (1)) 5
Be [x(00 1|2 Eyee Bj I
S1 |x(0|||2 2 St.
Proof. Let 0: = (x(0),)((t)) By Lemma above, 0, S aresin Bz/ |10(0111) S 2 B |1 ||- Also || > | Ejer Bj (each step can shrink the norm by at most B,). Summing angles (spherical triangle inequality) gives the first display; since [0= || Ejec B, 2 |000|| - Sz. each
fraction is Be/([==001] Sr.), yielding the last bound.
16

---

Published as a conference paper at ICLR 2026
C EXTENDED METHODS
C.I RSGD FOR TOKEN EMBEDDING OPTIMIZATION
We observe that gradient magnitudes tend to increase as training progresses, which often leads to instability in the later stages. Although standard learning rate schedules can help mitigate this issue, the gradient dynamics vary considerably across different datasets and training settings, limiting the effectiveness of fixed schedules. To address this, we draw inspiration from adaptive optimization techniques in Euclidean space (Kingma & Ba, 2015; Duchi et al., 201 I) and propose simple yet effective gradient scaling scheme based on gradient norms:
g = 9 0///199112>
99)
where is the gradient at iteration k. This normalization is equivalent to using an adaptive step size n/ [gg in Euclidean update Ux+11 Vx (1// |1 ll2]g;: the update direction still gt. but the length is fixed to 17. preventing very large gradients from causing excessively large parameter updates. Note that similar technique was previously explored in the context of Riemannian optimization (Cho & Lee, 2017).
C.2 WHY VMF OVER OTHER DISTRIBUTIONS?
We chose the Mises Fisher (vMF) distribution as it is ideally suited for modeling the directional characteristics token embeddings we identified in Section 2. Our central hypothesis is that the token embedding vocabulary can be modeled as mixture of vMF distributions, where each component corresponds to distinct semantic cluster (e.g., one for animals, another for objects). The vMF distribution is a suitable building block for this model for three key reasons:
It's natural fit. The vMF the natural analog to the Gaussian distribution a hyper- sphere, making it a principled and standard choice for modeling directional data clusters.
It's computationally efficient. The vMF's mathematical form is exceptionally convenient for optimization. In our MAP formulation, the gradient of the log-prior is a constant- direction vector (-Kμ), which provides stable and efficient semantic pull without requir- ing complex calculations. This simplicity makes more suitable for high-dimensional embeddings in large-scale models than alternatives like the Kent and Bingham distributions.
It's interpretable and controllable. The parameters are easy to understand. The mean direction 1l serves as semantic anchor to prevent the learned token from drifting away from related concepts, while the concentration g allows us to control the strength of this regularization.
These factors collectively make the vMF distribution a superior choice for our application, providing the necessary regularization way that both mathematically principled and computationally tractable.
D EXTENDED EXPERIMENTS
D.1 IMPLEMENTATION DETAILS
Following the protocol of recent studies, we primarily conducted experiments using Stable Diffusion XL. (SDLL). To demonstrate broader applicability to different models, we also conducted experiments with the recent model SANA 1.5 (Xie et al., 2025); these results are presented in Table 2.
For fair comparison, we adopted most hyperparameter settings from the Textual Inversion (TT) implementation provided by the Hugging Face c' ffusers library (von Platen et al., 2022). We used a training batch size of 4 and enabled mixed-precision training with the bfloat16 (bf16) format. For the baselines, the learning rate was set to the commonly used value method employs different optimizer, we conducted separate learning rate search and set it to 0.02. Following Rombach et al. (2022) and Gal et al. (2023a), we scaled the base learning rate proportionally to the batch size for all methods.
x 10 Since our
17

---

Published as a conference paper at ICLR 2026
All experiments were performed with fixed random seed of 42, and the maximum number of training steps was set to 500 for SDXL and 1000 for SANA. For image generation, used the DDIMScheduler with 50 inference steps for SDXL and the er with 20 inference steps for SANA.
Hyperparameters. There various approaches selecting the concentration parameter We performed a grid search and found that values the range of 5 x 10-6 to 2 x 10-4 work well. Therefore, we did not conduct an exhaustive search for a more precise optimal value. Throughout our experiments, we simply fixed k at 10-1, which generalizes well across different settings. Examples illustrating the effects of varying и are provided in Table 3.
Baselines. Throughout this paper, we compare our method with two baseline approaches: Textual Inversion (TI) (Gal et al., 2023a) and CrossInit (Pang et al., 2024a). Since the official CrossInit implementation is based on Stable Diffusion v2.1 with hyperparameters tailored to that version, we reconfigure to operate on SDXL by aligning its training setup with that of TI. Specifically, we adopt the same perparaameters as used for TI, and we set the regularization weight for CrossInit to 10 as specified the original paper.
D.2 ON THE CHOICE OF PRIOR
For all of our experiments in the main section, used the same class tokens those specified in the DreamBooth dataset as priors. However, we would like to note that since our DTI can leverage the prior, searching for better priors can lead better results. This demonstrates the effectiveness of the prior.
To test this, we experimented with having VLM recommend initial tokens. More specifically, provided reference images to the VLM and asked it to recommend 1-2 words that best describe them. For the experiments, we used Qwen-VL 2.5 (Bai et al., 2025) the VLM. The results are shown in Table 6.
The results indicate that changing the prior affects performance, although the overall effect is modest. For both TI and our DTL, Qwen- VL initialization tends to increase subject similarity, accompanied by a slight decrease in text fidelity. Practitioners may leverage VI.Ms manually craft priors with targeted terms to emphasize desired attributes. Overall, these findings demonstrate the flexibility and effectiveness of leveraging priors
Table 6: Results with VLM recommeeed priors. We compare Qwen- VL recommended initial tokens
with DreamBooth initial tokens as priors for DTT.
SDXL
SANA
Method
Initialization
Image
Text
Image
Text
DrcamBooth init
0.561
0,292
0.480
0.621
Qwen-VL VL init
0.583
0.273
0.501
0.619
DreamBooth init
0.450
0.522
0.479
0.744
DTT (ours)
Qwen-V VL init
0.520
0.391
0.504
0.697
D.3 COMPARISON WITH OTHER BASELINES
We expand our comparative analysis to include additional baselines: P+ (Voynov et al., 2023), NeTI (Alaluf et al., 2023), and CoRe (Wu et al., 2025). We run these experiments mainly on SD1.5 and SD2.1-bass
these baseline papers work on those versions. Adhering to the evaluation protocol of the main paper, we measure subject similarity using DINOv2 similarity and prompt fidelity with the CI.IP-variant, SigL.P. The results demonstrate that across both architectures, DTT consistently achieves the most favorable balance between these metrics compared to all baselines. Qualitative comparison can be found in Figure 10.
DTI drop-in replacement for TI. Although DTI is primarily designed for embedding only personalization, it also functions effectively as a drop-in replacement within fine-tuning pipelines. Recent work Direct Consistency Optimization (DCO; Lee al., 2024) typically performs joint
18

---

Published as a conference paper at ICLR 2026
Table Results on SD1.5 and SD2.1-base. We compare the baselines that improve TI on different versions of Stable Diffusion. DTI achieves the best balance between subject similarity and text fidelity compared to other baselines.
SDL55
SD2.1-base
Method
Image
Text
Image
Text
Pt (Voynov et al., 2023)
0.273
0.719
0.238
0.663
NeTI (Alaluf al., 2023)
0.408
0.579
0.565
0.517
CoRe (Wu et al., 2025)
0.340
0.661
0.357
0.654
DTI (ours)
0.418
0.554
0.469
0.568
optimization of a concept token using standard Textual Inversion (TT). To assess the impact of substituting this TI component with DTI, we conducted joint training for 250 steps using a LoRA module with rank 4.
As shown Table 8, the conventional TI-based joint training exhibits limited text-alignment (0.456), whereas replacing TI with DTI substantially improves alignment to 0.635. This guantitative improve- ment further reflected in Figure 6, where our method accurately incorporates textual attributes (e.g.. red backpack), while DCO with TI fails to do
Table DCO experiments. Comparison of DCO with standard Textual Inversion (TI) versus DCO initialized with our DTI. Integrating our method significantly improves text alignment.
Method
Image
Text
DCO
0.605
0.456
DCO + DTI (ours)
0.568
0.635
Input
DCO 11
DCO DTI
A real <brokeprcke>
A <anckpacke>> with a faturistio cityssope the kackgrroudd
A <dllgg > with a blue house in the beoeground
A <dog>> in dream of distant galaxy
A <cat>> Ge snow siil skiling doww 9 steep mcunttin
<cat>>
werning a sakta het
Figure 6: Qualitative results with DCO (Lee et al., 2024). We provide a comparison of the output images of DCO + TI and DCO DTI. The results suggest our DTT's superiority in text fidelity while preserving strong image similarity.
D.4 ABLATION STUDY
Effect of Riemannian optimization. Our DTI framework employs Riemannian optimization to ensure embeddings lie on the spherical manifold g--1, An alternative is to simply re-scale embeddings after each Fuclidean optimization step achieve this constraint. However, Table
19

---

Published as a conference paper at ICLR 2026
(first row) shows this latter Euclidean- based approach with re-scaling achieves suboptimal results, highlighting the benefit of direct Riemannian optimization.
Effect of magnitude (m). We investigated the impact of the fixed embedding magnitude, m, personalization performance. Our DTI framework, by default, sets m to the average norm observed in the pre-trained CLIP token vocabulary. We compared this "mean" strategy under the Riemannian optimization setting with x le
Setring m to the minimum vocabulary norm ("min").
Setting m to the mean vocabulary norm ("mean").
Setting to large, out-of-distribution (OOD) value of 5.0.
As shown in Table 3:
The "mean" strategy achieves the highest subject similarity and strong text fidelity.
The "min" strategy results in significantly poorer performance in both metrics.
* Using an OOD magnitude of 5.0 also leads to degradation in both metrics.
These results validate choice of fixing the magnitude to in-distribution scale, specifically the average vocabulary norm, as it provides a strong balance of subject similarity and text alignment. Both excessively small ("min") and out-of-distribution large ("OOD") magnitudes are detrimental.
Effect of concentration parameter (x). The concentration parameter k of the von Mises-Fisher (yMF) prior controls the strength of the directional regularization. We analyzed its effect by varying while using Riemannian optimization and the "mean" embedding magnitude. We tested k = 0.0 (no prior), K le- (DTI default), and 4.
The results in Table indicate:
With バ 1E 4, we observe the best balance between subject similarity and text fidelity. Setting x = 0.0, which removes the directional prior, leads to lower scores in text fidelity, which suggests that the directional prior improves semantic alignment.
Increasing the regularization strength with к = 5e 4 yields the highest text fidelity among the tested values but at the cost of reduced subject similarity.
Overall, our default choice of и = 1e 4 provides a better balance between maintaining subject similarity and ensuring text fidelity. Note that 1c 4 may not be numerically optimal across all criteria but works reasonably well by providing robust overall performance.
D.5 DETAILS OF USER STUDY
To evaluate world user preferences for image generation quality, conducted comprehensive user study involving 100 participants recruited through Amazon Mechanical Turk. Each participant completed survey consisting of 20 questions, evenly divided into two critical evaluation criteria: subject similarity and text prompt fidelity. For each question, participants were presented with three distinct image options, generated by: Textual Inversion (Gal et al., 2023a), CrossInit (Pang al., 2024a), and our proposed Directional Textual Inversion (DTI). The order of these three choices was randomized for each question, using a fixed random seed to ensure consistent shuffling across all participants. Sample questions can be found in Figure 7. We collected total of 96 valid responses, with submissions being excluded due to invalid patterns such as selecting the same answer all questions. The results, as detailed in Table 4 (in the main paper), demonstrate that our Directional Textual Inversion (DTI) consistently outperforms both Textual Inversion and CrossInit across both evaluation metrics: image subject similarity and text prompt fidelity. These findings confirm the superior performance of our proposed method in generating images that more accurately align with user expectations regarding both visual content and textual descriptions.
D.6 MORE QUALITATIVE RESULTS
We present additional qualitative comparisons with TI-based approaches (Gal et al., 2023a; Pang et al., 2024a) in Figure (SDXL.) and 9 (SANA). The results illustrate that proposed DTT consistently
20

---

Published as a conference paper at ICLR 2026
Q. Which photo most resembles the given subject?
Q. Which photo most follows the given text prompt?
"A dog exploring a vibrant reef underwater.
(b)
(a)
(b)
(a) Subject similarity
(b) Text prompt fidelity
study with 100 participants recruited via Amazon Mechanical Turk to evaluate 20 questions. The evaluation focused on two key aspects; subject similarity (10 questions) and text prompt fidelity (10 questions). To ensure fair comparison, the random seed was fixed and option order was shuffled.
Figure 7: User study design. We conducted
generates outputs that accurately align with the provided text prompts, even in challenging cases where the baseline methods fail to do so.
Our DTI serves as a drop-in replacement for TI. enhancing the model's performance when combined with LoRA. The qualitative results in Figure 11 demonstrate that DTI consistently generates outputs that both precisely follow the text prompt and accurately capture the subject's details.
D.7 MORE RESULTS ON APPLICATIONS
Stylization. We explore the combination of personalized subject embeddings and style embeddings. Our method, DTI, consistently generates images that accurately reflect both the personalized subject and the specified style. contrast, frequently fails this task, either by omitting the subject altogether (top row) by inadequately capturing the intended style subject details (bottom row) of Figure 12.
My object in my style. We also compare results in simultaneous generation of personalized subject and style. The results demonstrated in Figure 13 show that DTI successfully generates outputs that are faithful to both subject and style, while TI fails to.
Face personalization. To evaluate and showcase the capability of our DTI method in face personal- ization, we conducted experiments using randomly selected faces from the FFHQ dataset (Karras et al., 2019) as well as faces generated by DALL-E (Ramesh et al., 2021).
Since CrossInit specifically focuses on facial personalization, compare TI, CrossInit and our DTI on this task. Given that CrossInit does not explicitly provide hyperparameters (including learning rate) tailored for SDXL, we performed grid search across various learning rates. Our empirical results indicated that the learning rate used by TI yielded reasonable performance for CrossInit as well. Figure 14 illustrates a comparison between the three methods, demonstrating that all methods perform effectively for facial personalization. Nevertheless, as the complexity of text prompts increases (rows depicted the left columns), the baseline methods struggle to accurately reflect all described components of the prompts. In contrast, our DTI method consistently captures the critical components precisely, demonstrating superior performance achieving enhanced textual fidelity.
E ADDITIONAL EXPERIMENTS
We present additional experimental results in Figures 15, 16, 17, and 18. Specifically, Figure 15 compares TI using SLERP against LERP, justifying choice of the latter. In Figure 16, present an ablation study on magnitude settings. While our DTI uses the mean value of the entire vocabulary the default, further investigate initializing with the specific category describing the subject
21

---

Published as a conference paper at ICLR 2026
(ours)
(eurs)
sanddee> lucerious Interior Kuvlng
<oRAdlE> with Co the beokgrosed
A <ddgg do Lw detan. of dicLant galaxy
A <elog 2e with Rjppar-ss muplern sity bn the backgrouurd
<e00 Smang Kegydarpprr in New Youks city
<ep0p OK A sobakestove street
<olog> by vovkee theater
<cdog>> spploring veef wnderwimer
<dog?> wetring b.ack tap vanooele
<elog weering ssacesiit, planting Aag
A <d00 > police sutfit
<dog waaring saxtR
<teapot>> the skyscorders New York
A <choppre>> lesiale rusking woterfall
Figure 8: Qualitative results with SDXL. Here, provide more qualitative comparisons with TI and CrossInit, Our DTI consistently generates results that precisely reflect the user text prompts, while maintaining subject similarity.
22

---

Published as a conference paper at ICLR 2026
Input
TI
CrossInit
DTI (ours)
TI
CrossInit
DTI (ours)
<oat>> weRring police cutfit
<oat> with beauttifu ful sanaet
<oRt> with blue house in the backgrozond
A <eat> in D. ckdf cutfit
<dog explorer, wavigeting throuegh
<dog> > a mavis theatsr
Loy Arotis Lanatssope
<dog> wearing rest
<olog> fürefighter, extinguishing fire iu slaysovaper
o0
<toy> lx. Luxeriius intterior living reoka.
<tay> top of mónror
<teapot>> in the jungue
A <teepot>
with fres Avul Autumur, laaves tite boskgyound
Figure 9: Qualitative results with SANA1.5-L.6, Here, we provide more qualitative comparisons with TI and CrossInit on SANA. Our DTI consistently generates results that precisely reflect the user text prompts, maintaining subject similarity.
23

---

Published as a conference paper at ICLR 2026
Input
NeT1
CoRe
DTI (ours)
sbaoeepo in Lecurious interror Living voom.
shiny <bonkepock
<eat>> wearing ponle glasses
On cldff ouefit
<dogg>> in. disana of distont gelaxy
<oloog stanoing proudly in fiela of sanifownes
A <onndless with Japansse modera city la the baoleground
<oRAdle>> DV of a white
Figure 10: Comparison with additional baselines. We provide qualitative comparisons against additional Tl-enhancing methods Patiodd--.... NeTI, and CoRe. Because these baselines are built on SD2.1- base, apply DTI using the same pre-trained backbone to ensure fairness. The results demonstrate that DTT attains higher text fidelity while maintaining subject similarity.
24

---

Published as a conference paper at ICLR 2026
Input
DTI (ours)
A <dog> sittiing next to an ozuch
watersolor painting of dog wearing shirt, the beach sunset
syberpuent sigital paintiing of <toogg> with neow. lights
<<dogg wearing dpp riding surfboard the oceRm.
Input
BESTURE SUNTR.
H pRilice gealnnel' ' KIt
C'TY
DTI (ours)
A wäteroolor painting of <auuke <duakktoy?> wearing sawce hat, standing on frazen Lake
A <dlusk,boys photographtd A <duckk toy> dressed in in black and white ow vantagge armakacr
vintuge advertisement poster featuring a <dluol__toy> wearing
tiny suporhera cápe, with city baokoground
Figure 11: Qualitative results with TI/DII with LoRA on SDXL. We provide qualitative comparison TI DTI when combined with LoRA-based fine-tuning (rank 32). DTI consistently improves the text prompt fidelity compared to TI.
25

---

Published as a conference paper at ICLR 2026
Inputs
Input
A <dog> in <style>
TI
DTI (ours)
TI
DTI (ours)
teddy bear in <style>
Figure 12: Stylization. Qualitative comparison of personalization with diverse style inputs.
Figure 13: My subject in my style. Qualitative comparison of subject-style mixing within the same prompt.
Input
TI
CrossInit
DTI (ours)
TI
CrossInit
DTI (ours)
soulpbure <Hau>>
<HASH> wearing
Kwoman>> wearing faturistio sunglasses, sygerrank sity basegrowod
A sleetsh of Swoman>> webring hat
Figure 14: Comparison of face personalization methods. We compare our method and Textual Inversion (TT) against CrossInit, which specifically targets face personalization. To prevent bias from celebrity faces, we evaluate personalization using two alternative sources: images generated by DALL-E (Ramesh al., 2021) (top row) and randomly selected images from the FFHQ (Karras et al., 2019) (bottom row).
(e.g., cat). We demonstrate that minor variations in magnitude do not significantly alter the outcome. Figure 17 evaluates our DTI in multi-concept scenarios, illustrating both successful outcomes and limitations. Finally, we analyze specific failure cases our method in Figure 18.
F SOCIETAL IMPACTS
The rapid advancement of text-to-image diffusion models, especially in the domain of personalization techniques, raises important societal considerations. In particular, the ease of generating highly specific and detailed images raise concerns related to copyright infringement, personalized generative models may inadvertently or intentionally reproduce objects protected by intellectual property laws. Therefore, note that it is important for users and distributors of the model to develop comprehensive awareness and implement guidelines addressing copyright boundaries, fair use, and ethical content generation. Moreover, note that, since our method does modify the
26

---

Published as a conference paper at ICLR 2026
SLERP (DTI)
Linear interpolation (Tl)
Normalized > SLERP multipled with dog's norm (TI)
Normalized SLERP multipled teapot's norm (T))
<dog>
Figure 15: Interpolation options for TL. We compare several interpolation options for TI, including linear interpolation, SLERP with normalization and adjusted norms. While these approaches exhibit minor differences in behavior, none produce smooth transitions between concepts. In contrast, DTI with SLERP achieves noticeably smoother and more consistent interpolations.
Setting the adjective
Setting to average of vocab
A <cot>> in & dream of pistant gelaxy
A Soat weRring spacssult, planting flag the macon.
A Kogc>> in. movie theater
A <oot>> with city in the backgrowuu
Figure 16: Ablation on magnitude settings. For consistency and ease of use, magnitudes this
paper are set to the average value computed over the model's vocabulary (see ablation in Table 3). To evaluate the effect of using concept-specific magnitudes (e.g., the magnitude of 'cat' for the concept <cat>), ), provide ablation results under different magnitude settings. The results show that small deviations from the default magnitude do not lead to noticeable differences in output quality.
27

---

Published as a conference paper at ICLR 2026
Inputs
Output
<EOE> 2nd 8. Kippppot
A <eat> eudd <
A. <dog webking <Saagglassss>
<dag weading Saaudaeerss
weterrolor
palluiiin of <toog?
paaying with
shrag in the
awdionsing pool
Failure
A Skeboh of Seatt- Rnd rouiacc with flwatrs Dt
Sieetch of scaaec Anra a iddag?
Figure 17: Multi-concept experiments. We further evaluate DTI by combining multiple learned concepts within single prompt. The results demonstrate that DTI can successfully integrate multiple concepts, while the second column shows failure cases exhibiting attribute binding issues.
underlying parameters of the generative model but solely adjusts the token embeddings that capture personalized concepts, the quality of generated images inherently depends the capabilities of the underlying text-to-image model.
28

---

Published as a conference paper at ICLR 2026
When subject requires high visual detail
Input
A <backkpacke>> with a mountain in tikk backgrownd
A <bookppacke>> top a wooeen floor
A <backpacke>> with a futuristic citysoape in the backegroznd
<bear_pluskies besides nusking waterfall
<ebeRr plushie> in field of wildfflowers
< beay plusnies in Luxurisus living roow.
When prompt is difficult to depict
A <subject> floazing in an coean of mille
A cubse shoreed <sunbject>>>
When prompt requires change of attributes
wet <subject
A purple <subjeot>>
Figure 18: Failure cases. We present examples of three representative failure modes: (1) subjects that require high visual detail, (2) prompts that are vague or difficult to faithfully depict, and (3) prompts that involve attribute modification (e.g., color changes).
29


Reviews:
[{"review_id": "phKpDOY8Bp", "reviewer": "Reviewer_CW52", "summary": "This paper proposes DTI, based on its key motivation that the norm of the special token inflates with training. They analyze the impact of this inflation by providing theoretical analysis on positional embedding and residual updates within the transformer. To mitigate this, they propose to learn only the direction with a vMF prior, while fixing the magnitude. The proposed method is claimed to yield improvements in text fidelity of TI and its variants.", "questions": "1. The paper explains how the large norm disrupts the transformer's contextualization ability, but I'm curious why this happens. It would be more helpful to know why this phenomenon arises.\n2. Should the norm always be excluded from learning? Could this be learned, but with regularization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is useful that the norm inflation dilutes the impact of positional embedding and the residual updates within the transformer.\n2. The claims are supported with theoretical justification.\n3. The writing is clear and easy to follow.", "weaknesses": "1. The baselines are limited to heavily outdated methods. Further comparison is needed [1-3].\n2. The norm inflation of the learned token had already been explored, not only in personalization [4], but also in VLM prompt tuning [5].\nThe quantitative comparison with [4] is provided, it is not clearly explain how the proposed approach differ conceptually.\n3. Fig.3 - Is SLERP interpolation specifically available to the DTI? For fair comparison, SLERP should also be applied for TI for concept interpolation.\n4. The choice of m* (average norm of the vocaubulary) is not justified. For example, m* can be the norm of the common adjectives e.g., yellow, fluffy, or the norm of the corresponding prior concept e.g., dog, cat.\n5. Further comparison with recent optimizers e.g., Adam, AdamW should be provided to validate the effectiveness of the RSGD.\n\n\n[1] Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models, ICLR'24\n[2] Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models, Neurips'24\n[3] Controlling Text-to-Image Diffusion by Orthogonal Finetuning, Neurips'23\n[4] Cross Initialization for Personalized Text-to-Image Generation, CVPR'24\n[5] Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models, ICLR'24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mQsvkdY1vj", "reviewer": "Reviewer_Z4f5", "summary": "This paper is related to the task of text-to-image personalisation generation. The authors use Textual Inversion as the base method and claim that one problem with this method is the high norm of embedding of learned concepts, which leads to semantic drift. To address this issue, the authors propose finding such embeddings on a unit hypersphere. Additionally, they propose using a von Mises–Fisher distribution as a prior to model the distribution of these embeddings.", "questions": "1) Could you demonstrate Effect 1 in practice and explain how your method deals with it? For example, you could train a classifier that takes the trained embedding after the Layer Norm of a particular concept as input and tries to predict the position of that embedding. You could train such a classifier for the base model, TI and DTI, and demonstrate the distribution of the resulting accuracies for different concepts.\n2) Could you demonstrate Effect 2 in practice and explain how your method deals with it? For example, you could fix a concept and calculate the angle between the trained embedding and its initialization. You could show the distribution of different concepts for TI and your method. \n3) Could the authors present the metrics for the earlier TI and CrossInit checkpoints from Table 2?\n4) Could you please explain how you obtained Fig. 1b? What hyperparameters did you use to train the methods, and what was the checkpoint step?\n5) Could you clarify the results in Table 7? Why are the results for your method all in bold when, as I can see, they are not always the best?\n6) Could you clarify which settings were used to generate the images in Figure 2?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) The idea behind the presented method is clear, and the text is well written.\n2) The paper provides a solid theoretical basis for the proposed modifications.\n3) The authors demonstrate the additional capabilities of their method.\n4) The paper contains many experiments and an extensive ablation study.", "weaknesses": "1) The work contains a lot of theoretical discussion, but most of it is based on asymptotical estimations (e.g. Lemmas 1 and 2, Proposition 1). The problems described do not seem obvious to me and I am not sure they could arise in practice. (questions 1, 2)\n2) I am unsure about the correct evaluation of TI and CrossInit on SDXL (Table 2). It seems that the methods are slightly overfitted. (question 3)\n3) Some parts of the work (e.g. Fig. 1, Tab. 7) lack details on how they were obtained. (questions 4, 5, 6)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1snBbSmTsU", "reviewer": "Reviewer_LKw4", "summary": "This paper focuses on enhancing personalized text-to-image generation by addressing critical limitations in Textual Inversion (TI), specifically the issue of embedding norm inflation that leads to poor prompt fidelity in complex scenarios. The authors first identify this bottleneck through empirical observations, such as the semantic drift and out-of-distribution magnitudes in learned tokens, and provide a theoretical analysis demonstrating how large norms attenuate positional information and cause residual stagnation in pre-norm Transformers. Building on this, they propose Directional Textual Inversion (DTI), a novel framework that fixes embedding magnitudes to in-distribution scales and optimizes solely the directional component on the unit hypersphere via Riemannian SGD, incorporating a von Mises-Fisher prior for semantic regularization.", "questions": "How does DTI's performance scale with the complexity and number of concepts in a single prompt?\n\nCould the directional optimization framework be extended to multi-token representations?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper exhibits several notable strengths that enhance its scholarly contribution. \n\nFirst, the analysis of Textual Inversion (TI) limitations is particularly insightful, as it introduces a new perspective by identifying and rigorously examining embedding norm inflation—a previously underexplored issue. This is supported by empirical evidence, such as the demonstration of out-of-distribution norms and semantic drift, and complemented by theoretical foundations that explain how large magnitudes disrupt Transformer dynamics, providing a comprehensive understanding of TI's failures.\n\nSecond, the technical pathway of Directional Textual Inversion (DTI) is well-justified and methodologically sound, building logically on the identified problems. By decoupling magnitude and direction, employing Riemannian SGD for hyperspherical optimization, and incorporating a von Mises-Fisher prior within a MAP framework, the approach effectively addresses the core issues, demonstrating a coherent and innovative design.\n\nThird, the paper is exceptionally well-written, with clear organization and rigorous experimentation across diverse settings, including various personalization tasks, and comprehensive evaluations through metrics and human assessments.", "weaknesses": "The paper's introduction emphasizes TI's poor performance with complex prompts, yet the subsequent technical analysis in Section 2 appears decoupled from this specific issue. The theoretical framework primarily explains general performance degradation due to norm inflation but does not explicitly model or analyze the distinct challenges of compositional prompts, creating a slight disconnect between the stated motivation and the core technical solution.\n\nWhile the paper claims TI is an \"efficient alternative,\" this claim warrants critical examination. The requirement of approximately 7 minutes per subject for SDXL and 30 minutes for SANA, as mentioned in the reproducibility statement, along with TI's inherent limitation to optimizing a single concept per embedding, challenges its practicality for real-time applications or scenarios requiring multi-concept personalization. The efficiency argument is relative to full model fine-tuning but may not hold against more recent parameter-efficient methods.\n\nThe qualitative results, while generally supportive, exhibit inconsistencies that merit discussion. For instance, in the first column of Figure 2, the generated bears show noticeable variation in appearance, compromising subject consistency. Furthermore, the intended \"wizard costume\" attribute is not convincingly realized in the outputs, highlighting a potential limitation in capturing specific, fine-grained attributes through directional optimization alone.\n\nA more comprehensive quantitative comparison is needed to fully situate DTI within the current landscape. The evaluation primarily benchmarks against standard TI and CrossInit. Including comparisons with a wider range of contemporary methods—such as other TI variants (e.g., P+, P*, NeTI), lightweight fine-tuning approaches like LoRA, or even full fine-tuning methods like DreamBooth on key metrics—would provide a clearer picture of DTI's relative advantages and trade-offs in terms of both performance and computational cost.\n\nThe theoretical analysis, though valuable, relies on the assumption of bounded sub-layers (Section B.1). The practical validity and tightness of these bounds in modern, large-scale text encoders are not empirically verified. Exploring how these bounds behave in practice could strengthen the theoretical claims or reveal conditions under which the effects of norm inflation are more or less pronounced.\n\nThe selection of the vMF prior's concentration parameter κ, while justified via a grid search, is presented as a fixed value (1e-4). The paper does not explore whether an adaptive κ, potentially based on training dynamics or the semantic specificity of the target concept, could lead to further improvements. This fixed hyperparameter approach might limit optimal performance across diverse personalization tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x5HdXfJinJ", "reviewer": "Reviewer_r8HK", "summary": "The paper addresses the text-to-image customization problem, leveraging a textual inversion method. \n\nThe authors showcase 2 aspects about textual inversion: the OOD magnitude of the inverted vector and the direction importance for the semantic meaning. Specifically, norm inflation or direction drift causes poor text alignment or object's poor fidelity in generated images. \n\nAs a solution, the authors propose to fix the magnitude of the inverted vector (aligned with the average in the vocabulary) and optimize only the direction.", "questions": "1. Would you please provide more details about the norm inflation plot in Figure 1 (dataset, text encoders, etc.)?\n2. In Proposition 1, there is a condition when $$||x^{(0)}|| > S_{L}$$ \nWhat if this condition is not met? What is the guarantee that the initial embedding's norm will be greater than the sum of all B's?\n3. There is no description of S^{d-1} in Expression 1. Would you provide more details and descriptions about it?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Explanations for some of the known textual inversion problems (Effect I and Effect II).\n2. Theoretical support of the claims (Proposition 1 and Corollary 1).\n3. Despite some theoretical complexities, the method implementation is simple and flexible. No heavy finetuning required for diffusion models.", "weaknesses": "1. Qualitative comparisons are mostly or even only done with the baseline Textual Inversion method. Many improved or advanced methods are neglected in qualitative comparisons.\n2. The proposed method's quantitative results are very close to the results of advanced textual inversion methods (NeTI, P+, etc), and the improvement is marginal.\n3. There is no a limitations section or failed examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)