Boosting Text Encoder for Personalized Text-to-Image Generation
Authors: NaHyeon Park
Kunhee Kim
Hyunjung Shim
Venue: Accepted by TMLR ()

Abstract: In this paper, we introduce TextBoost, an efficient one-shot personalization approach for text-to-image diffusion models. Traditional personalization methods typically involve fine-tuning extensive portions of the model, leading to substantial storage requirements and slow convergence. In contrast, we propose selectively fine-tuning only the text encoder, significantly improving computational and storage efficiency. To preserve the original semantic integrity, we develop a novel causality-preserving adaptation mechanism. Additionally, lightweight adapters are employed to locally refine text embeddings immediately before their interaction with cross-attention layers, greatly enhancing the expressiveness of text embeddings with minimal computational overhead. Empirical evaluations across diverse concepts demonstrate that TextBoost achieves faster convergence and substantially reduces storage demands by minimizing the number of trainable parameters. Furthermore, TextBoost maintains comparable subject fidelity, superior text fidelity, and greater generation diversity compared to existing methods. We show that our proposed method offers an efficient, scalable, and practically applicable solution for high-quality text-to-image personalization, particularly beneficial in resource-constrained environments.

Published in Transactions on Machine Learning Research (05/2026)
Boosting Text Encoder for
Personalized Text-to-lmage Generation
NaHyeon Park
julio 19 1i0aastt....... oc.kr
KAIST
Kunhee Kim KAIST
kunhee.kinn@kaiiee oc. kr.
Hyunjung Shim KAIST
Cateshám@@kiiste oc..k
Reviewed on OpenReview:
https: /// openreview. net/ forum? <d=hsZakinHul
Abstract
In this paper, we introduce TEXTBOOST, an efficient one-shot personalization approach for text-to-image diffusion models. Traditional personalization methods typically involve fine-tuning extensive portions of the model, leading to substantial storage requirements and slow convergence. In contrast, we propose selectively fine-tuning only the text encoder, significantly improving computational and storage efficiency. To preserve the original seman- tic integrity, we develop novel mussality-rreorving adaptation mechanism. Additionally, wight adaplers are employed to locally refine text embeddings immediately before their interaction with cross- attention layers, greatly enhancing the expressiveness of text embed- dings with minimal computational overhead. Empirical evaluations across diverse concepts demonstrate that TEXTBooST achieves faster convergence and substantially reduces stor- age demands by minimizing the number of trainable parameters. Furthermore, TEXtBooST maintains comparable subject fidelity, superior text fidelity, and greater generation diver- sity compared to existing methods. We show that our proposed method offers an elficient, scalable, and practically applicable solution for high-quality text-to-image personalization, particularly beneficial in resourco-constrained environments.
1 Introduction
Recent advancements in text-to-image (T2I) diffusion models have significantly expanded the creative po- tential of AT-driven image synthesis, enabling high-fidelity images to be generated from natural language prompts (Ramesh al. 2021 Rombach al. 2022: Balaji al.] [022 Nichol et al. [023]] Chen et al., 2024c Podell al. 20024 Esser et al. 2024). Despite these advances, standard text prompts often lack the specificity required to capture fine details unique artistic styles, leading to ambiguities in the generated images. To address this, personalization techniques have emerged as a critical research direction, aiming to capture a specific concept (V*) within the token space so that the generated images faithfully reflect user-defined content (Gal et al. 2023 Ruiz et al. 2023
While existing personalization methods show promising results, they often suffer from computational and storage inefficiencies. For instance, DreamBooth (Riiz et al.) 2023) fine-tunes all parameters of the U-Net and saves distinct model for each concept, which demands substantial storage overhead. Moreover, current
"Equal contribution.

---

Published in Transactions on Machine Learning Research (05/2026)
methods often require multiple reference images (usually three to five), limiting their practicality in real- world scenarios where users can only provide a single reference image. These issues underscore the need for efficient approaches to personalization both in terms of parameter usage and training data requirements.
In this paper, we propose an efficient one-shot personalization method by focusing on an underexplored yet pivotal component of the T2T model: the text encoder. Typically, personalization approaches fix the text encoder and fine-tune components within the U-Net, such as cross-attention modules (Kumari et al. 2023, Chen et al., 2024b). However, our initial empirical analysis suggests that when all parameters of the T2I model are fine-tuned, the text encoder undergoes the largest changes, indicating significant potential benefits if adapted properly.
Building on this insight, we introduce TextBoost, a novel method that selectively fine-tunes the CLIP text encoder to achieve efficient one-shot personalization. A key challenge is to fine-tune the text encoder without disrupting its learned semantic space, particularly its auto-regressive causality: the property where loken's output embedding is conditioned only on itself and prior tokens in the sequence. To address this, we introduce Seusalltyy-Presevvel Adaptation (CPA), mechanism specifically designed to ensure that during the learning of new concept token (V*), the output embeddings of all tokens preceding V remain unchanged. This preserves the original contextual understanding of the sequence leading up to the new concept. Concretely, CPA formulates the adapted output as a mixture of original and fine-tuned embeddings, preventing the distortion of established semantic structures of the original text encoder.
To further enhance fine- fine-grained concept control, we draw inspiration from hicrarchical representations in GAN architectures (Karras et al.] 2019: 2020; Richardson et al.] 2021 Tov et al. 2021 and introduce lightweight adapters that locally refine text embeddings before they interact with cach cross-attention layer.
Unlike loken-space methods that rely on multiple forward passes (Voynov et al. 2023 Alaluf et al. 2023 our design requires only a single pass through these adapters, substantially improving computational efficiency and runtime.
We validate BoOST across various datasets, comparing to popular personalization methods such as DreamBooth (Ruiz et al. Diffusion Kumari al.
2023
Textual Inversion (Gal et al.) 2023), LoRA (Hu et al., 2022), and Custom 2023 Our results demonstrate that TEXTBooST achieves faster convergence,
reduced storage overhead, high subject and text fidelity, thus offering an improved balance between efficiency and generative quality.
In summary, our key contributions are as follows:
Fine-tuning text encoders for T2I personalization: We present a novel approach for fine- tuning text encoders specifically designed for text-to-image (T2I) personalization, a previously un- explored direction.
Causality-preseeveed adaptation (CPA): We introduce a pausality-prsserved adaptation strategy that maintains the integrity of the text encoder's original semantic properties during adaptation, lhereby preserving the causal relationships embedded within the semantic space.
Extended text embedding: We propose extending the embedding space using lightweight adapters, facilitating more precise and fine-grained adaptation of concepts through enhancements in the cross-attention mechanism.
Efficient. personalization: Our method significantly improves training elficiency and reduces stor- age requirements, enabling high-quality T21 personalization suitable for deployment in resource- constrained real-world applications.
2
Related Work
2.1
Personalized text-to-ine -o-taee generation
Recent advancements in text-to-image (T2I) generation have increasingly emphasized personalization, allow- ing models to capture user-specific concepts more accurately. Textual Inversion (Gal et al.) 2023) introduced
2

---

Published in Transactions on Machine Learning Research (05/2026)
Attn
MLP
Laver
CrossAttn
Attn
Text Encoder
Other
0.00
0.03
0.10
0.35
0.20
0.25
0.30
0.35
Misan (x 1GD)
Figure 1: Change in weights of different layers during fine-tuning of a T21 model. The mean weight change in the text encoder layers is notably larger compared to that of the U-Net parameters. This suggests that the text encoder plays pivotal role in the personalization task.
encoding personalized information into special learned tokens (V* tokens). Building upon this, (Voynov al. 2023) proposed more expressive representation with layer-wise prompt embeddings for detailed concept inversion across different model layers. (Alaluf et al. 2023) further extended this paradigm with timestep- wise inversion approach, offering greater flexibility but at the cost of increased computational overhead and longer training and inference times. Parallel efforts using fine-tuning-basee -based methods, such as Dream- Booth (Ruiz et al. 2023 achieve superior fidelity by updating significant portions the diffusion model, yet incur substantial computational and storage demands. To address these limitations, techniques like Custom Diffusion (Kumari al. 2023 and OFT (Qiu et al.) (202)) explored partial model fine-tuning, reducing computational load but still requiring modifications to the image generation component. Encoder-focused adaplations, including ELITE (Wei et al. 2023) and BLIP-Diffusion (Li et al. 2023), present another path- way to reduce computational overhead by primarily updating encoder components, thus minimizing trainable parameters while preserving personalized output quality. However, these encoder- based methods typically depend on extensive pre-training on large-scale datasets.
2.2 Parameter-efficient fine-tuning
Parameter-efficient fine-tuning (PEFT) methods have recently emerged as crucial techniques for effectively training large foundation models with limited resources. Adapter- based methods (Houlsby et al. 2019 insert lightweight, layers into pretrained models, enabling efficient adaptation, Hu al. (2022) introduced a simpler and highly efficient technique by inserting low-rank adaptation (LoRA) layers parallel existing weights, allowing the modifications to be merged back into the original model parameters after training. Due to its efficiency and effectiveness, LoRA has become popular in personalized T2I diffusion tasks, inspired subsequent studies (Qiu et al. 2023 Ruiz et al. 2024 Shah et al. 2024 and the basic component of recent personalization methods (Chen et al. 2024b Lee et al. 2024). Similarly, StyleDrop utilized adapters for transformer-basee diffusion models, demonstrating effective customization capabilities (Sohn et al.) 2023) Our proposed method also leverages tochniques from the PEFT domain, such as adapters (Houlsby et al., 2019) and LoRA (Hu al., (2022). However, unlike previous approaches primarily aimed at optimizing downstream task performance, our method emphasizes preserving the original capabilities of pretrained models by adaptively fine-tuning parameters in
manner.
3 Motivation
Unlike previous works, which typically finc-tune cither the concept token or specific components of the U-Net, we begin by systematically re-evaluating which parts of the text-to-image (T2I) diffusion model are most critical to fine-tuning for effective personalization.
3

---

Published in Transactions on Machine Learning Research (05/2026)
MCO.0.021
<START> cute puppy playing with ball in the grass <F05>
<START> cute puppy playing with batt in the geazs <E05>
<START> cute puppy playing with ball in the grass <fo5>
Original
Add noise text encoder output
Add noise token embbddding
Figure 2: Effect of perturbations early token embeddings during text-to-image generation. We compare the original generation (left) with images produced after adding small Gaussian noise to the text encoder's output embeddings (middle) and directly to the token embeddings (right). Adding even slight noise significantly affects generated image quality, highlighting the sensitivity and importance of preserving embeddings for early tokens in text-to-image generation.
Kumari et al. 2023) proposed an efficient approach by fine-tuning only the cross-attention layers of the C-Net, motivated by observing substantial parameter changes within these layers during personalization (Li et al. 2020p. Specifically, they targeted the key and value projection layers. However, their analysis did not consider potential changes in the text encoder parameters, critical component responsible for generating embeddings fed into the cross-altention layers.
Given that key and value projections directly depend on text embeddings, we hypothesized that the text encoder might also significantly influence personalization outcomes. To rigorously test this hypothesis, we extended the investigation from (Kumari el al. 2023) to include both the U-Net and the text encoder. We quantified parameter changes during fine-tuning using the following measure:
01
A
(1)
|01
where 0 and i denote the original and fine-tuned weights.
Consistent with previous findings (Kumari et al. 2023: we observed significant changes in the U-Net's cross-altention layer parameters. Surprisingly, as illustrated Fig. [ the text encoder parameters exhibited even more substantial adjustments. This finding strongly highlights the previously overlooked importance of the text encoder in personalized T2I gencration, aligning with recent insights emphasizing its central
role (Saharia el al. 2022 ?: Li et al. 2024b).
To the best of our knowledge, this study is the first systematic exploration specifically targeting fine-tuning the text encoder for personalized texl-to-image generation.
4 Boosting text encoder tuning
Building upon insights from the proceding section, we introduce TEXTBoOST, a novel approach for person- alizing text-to-image diffusion models.
4.1 Preliminaries: Text-to-image diffusion models
Diffusion models (Sohl-Dickstein al.) 2015) have recently become the most widely adopted generative mod- els for text-to-image generation (Saharia et al. 2022 Ramesh al. 2022 Rombach al., 2022). These mod- els aim to closely approximate the original data distribution g(2r)) with po(2))) Here, po po (2a) == p=(xo:)), where pa (xooT) is termed the reverse process being Markov chain with learned Gaussian transitions. The approximate posterior q(
(D) is termed a forward process, of which the noise is gradually added to the original data point 2p F: Voragg vI a4e, can be expressed as q (m))) = II' g(n, (2.-1)- The
4

---

Published in Transactions on Machine Learning Research (05/2026)
tokena
Input
Before adapiation
After adepeation
V": Learnalle token
14
Mg
M
74
Rt
Linear
Adapter
R1
Output
22
1
Byy Rz Ay Aug by ho
Ry My ft by Ao
hy By Iyy Itg At
Ny h2 As Ne Mo
1
Encoder
Text Encoder
Encoder
Focoder
6
dos
6
dos
5
dog
Causality
Naive fine-tuning
Causality-preservee finetuning
(a) Causal attention mask
(b) Causality of CLIP text encoder
(c) Causality-presenve adapter
Figure 3: (a) Causal attention mask. The CLIP text encoder employs causal masking techniques, where the output embedding h, corresponding to the input token Yi can only attend to preceding tokens. (b) Causality of CLIP text encoder. If this causality is not properly accounted for during fine-tuning (e.g. through naive fine-tuning), the embeddings of tokens preceding the new concept token V can be unintentionally altered, leading to undesired changes their representation. adapter. We attach adapters in parallel the lincar layers, and incorporate a causality mask ensure that the fine-tuned adapter influences only the new concept token V and the subsequent tokens, thereby maintaining the causality structure.
(c) Our causality-presservee
training objective is formulated with variational bound on negative log-likelihood:
S By[- log po((0):)
E - log po (xo))
= L, q(x1)100))
(2)
which is further simplified into the following objective:
Esimpte (0) = Ee, l|le ca(a 1)
(3)
Text-to-image (T21) diffusion models (Rombach et al. 2022) take text condition e as additional input, where the text prompt y is encoded with text encoder & as & (v). These models usually operate in latent space, encoding an image 2o into a latent Zg = Ex The forward diffusion process is then applied in latent space, yielding 2, = vaiz0 + vI are. The training objective hence takes an additional c and minimizes the following loss:
Ezo.t.y.c [||c - co(ze, t, c.]]]
(4)
While our approach generalizes various T2I models, we primarily employ Stable Diffusion (Rombach al. 2022), due to its extensive usage and open accessibility. Stable Diffusion adopts U-Net architecture (Ron- neberger et al.] 2015)) for its denoising diffusion process. To guide the image generation, text embeddings are injected through aross-attention layers, which dynamically integrate these embeddings into the U-Net at cach diffusion step.
Given the critical role of the CLIP text encoder (Rafford et al. 2021 in widely-used T2I diffusion models such as Stable Diffusion (SD) RRombach et al.) 2022) and Stable Diffusion XL (SDXL) (Podell et al. 2024 we specifically targeted the CLIP text encoder for effective personalization in our method.
4.2
prrseeiy adaptation
4.2.1 Challenges in fine-tuning CLIP text encoder
The CLIP text encoder, like many language models, a decoder-only transformer (Radford et al. 2021) that employs causal masking (see Fig. 3(a) This architectural feature enforces regr-segressive processing
5

---

Published in Transactions on Machine Learning Research (05/2026)
flow: the output embedding h; for an input token Ys is generated based solely on y, and all preceding tokens (U1...., 11- It is not. influenced by any subsequent tokens (yj where j > i). This inherent sequential causality is fundamental to how the encoder builds contextual representations.
When fine-tuning the encoder to learn a new concept token v", a naive update to the model's weights can inadvertenlly alter the output embeddings of tokens preceding V (as illustrated Fig. 3(b) where h1 to h4 change). This happens because weight changes, even if optimized for V*, propagate throughout the model. Such alterations disrupt the established semantic interpretations of the initial part of the text sequence (see Fig. 2 which can degrade the model's ability to understand prompts and accurately depict the concept. Our goal is to adapt the model for V while strictly preserving the original output embeddings of all tokens h; that appear before V in any given prompt.
4.2.2 prreervvd adapter
One straightforward solution would involve fully fine-tuning the text encoder and subsequently overwriting embeddings preceding V with their original values. This approach, however, requires two full forward passes, leading to inefficiencies and a failure to maintain layer-wise causality, Instead, we propose a more efficient cansaliyy-preservinn adapter (CPA) inspired by parameter-officinnt fine-tuning methods (Houlsby et al. 2019: Hu et al. 2022 from large language models (LLMs). We attach adapters in parallel to selected linear layers :......;20 unl
within the Transformer. Let U = (u;;...;
E R denote the matrix of token representations entering an adapted linear layer. The adapted output is computed
Ú UW Acra(m)
5
where W& E 1Rduxdao is the frozen pretrained weight matrix and AU)A(U) = UW down Wup represents the adapter with LoRAA-style down-projection Wown E RdaXr and up-projection Wup E Rr For a tokenized prompt Y : yyl, with the special token V located at position i*, the causal mask vector [m....., mp.] {0, 13" is defined as
if and y: is not a padding token, m; = (0, otherwise.
(6)
For a batch of B prompts, these vectors can be stacked row-wise into a batched mask matrix McPA e {0, (0,1) which is the form illustrated in Fig. 3(c) This mask ensures that the adapter affects only V and the subsequent non-padding tokens. For all tokens with i < i", we have m, = 0, so the i-th row of the adapter term in Eq, 5 is zero and the adapted output reduces to the pretrained output. This preserves the original encoder representations for the initial part of the sequence and maintains the auto-regressive causality up to V* (see Fig. 3(c).. We call this overall strategy Causality- Adaptation (CPA).
Based on empirical findings and our ablation studics (see Fig. 10), we incorporated adapters into specific layers for optimal performance. Notably, our approach is different from some previous conventions that primarily target self-attention parameters with adapter:]] Instead, our ablations indicated that focusing on other components yielded better results for our personalization task. Specifically, within Multi-Layer Perceptron (MLP) blocks of the text encoder, we add adapters to the second forw-fo layer (fc2). The rank parameter r determines the adapter's bottleneck dimension, controlling both parameter efficiency and expressivity; unless otherwise stated, we adopt r = 1 as our default configuration.
4.3 Extended textual embedding
Extended latent representations have proven effective in achieving more precise inversion in generative ad- versarial networks (Goodfellow et al. 2014 Karras et al. 2019). as demonstrated by methods like Im- age2StylcGAN (Abdal ct al. 2019) or e4c (ov et al. 2021) Motivated by these advances, we proposc
exlend the CLIP text encoder into an extended representation space.

---

Published in Transactions on Machine Learning Research (05/2026)
Y
Y
y
Text Encoder
Text Encoder
Text Encoder
U-Net
U-Net
U-Net
Naive conditioning
Extended Prompt Spuce
Extended Embedding Space
Figure 4: Extended embedding space. Unlike previous methods that operate in the prompt space, our approach introduces adapters in the embedding space, resulting in an extended embedding space. This design allows for more flexible integration of the fine-tuned text embedding from the text encoder into the image generation module. Here, Y represents the input text prompt, and A denotes the adapter for cross-atiention layer L.
Let tokenized prompt Y [y1;...; be encoded by the CLIP text encoder ε,, producing final hidden states
= = Er(( E Raxd
standard text-to-image diffusion models, this single embedding matrix H is shared across all cross- attention layers of the U-Net, where it is projected to keys and values.
We define extended tertual embedding layer-wise conditioning scheme that replaces the single text-encoder hidden state H with a set of per-cross-attention embeddings (H (0) JE= where Lea denotes the number of cross-atiention layers in the U-Net.
H H Diag(m)AA(HH)
(8)
where Ac(-) denotes wight adapter specific to cross-altention layer e, and m (0, 1 is the causality mask defined in Section 12
Each (²) is injected into cross-attention block before its key and value projection layers:
Ke H0
V V=
(9)
Importantly, the projection matrices unchanged; instead, each cross-attention layer receives its adapted textual embedding. This effectively extends the conditioning space from single embedding H to collection (H) ...., H.e.l))) increasing representational flexibility while maintaining architectural simplic- ity.
Crucially, to preserve causality, embeddings corresponding to tokens positioned before the novel concept token V remain unchanged due to the masking term Diag(m).
As illustrated in Fig. we apply pauselit adaptation (CPA) to refine text embeddings separately for each tross-attention layer. Unlike token-space methods such XTI (Voynov et al. 2023) NeTI (Alaluf et al. 2023), which require multiple forward passes through the text encoder, our approach performs only one forward pass through the text encoder followed by lightweight adapter transformations to construct /-
[H 1==1. This design significantly reduces computational overhead and improves efficiency compared to hierarchical prompt-embede
techniques.
We refer to the text-encoder fine-tuning approach with CPA as TEXTBooST, and the extended version incorporating both CPA and extended textual embedding TEXTBOOST**

---

Published in Transactions on Machine Learning Research (05/2026)
Table 1: Quantitative comparison on Stable Diffusion v2.1-base. We measure VQA scores (Lin et al, 2024) for image-text fidelity. Pairwise DINOv2 (Oquab et al. 2024) feature cosine similarity is evaluated with the reference images that are not used for training. For practicality, we compare the number of parameters.
Methods
VQA f
DINOv2 Diversity f
Params 4
Textual Inversion Gaal et al. NeTI (Alaluf et al.
2023
0.475
0.467
0.408
0.00M
2023
0.602
0.536
0.279
0.83M
DreamBooth Ruiz al. 2023
0.553
0.587
0.164
865.91M
DreamBooth LoRA LoR (Hu et al.
2022
0.528
0.589
0.190
0.83M
Custom Diffusion (Kumari et al.
2023
0.452
0.632
0,136
25,56M
DisenBooth (Chen et al. 2024b
0.675
0.486
0.305
2.93M
TEXTBOOST
0.680
0.560
0.307
0.12M
TEXTROOST**
0.607
0.598
0.229
0.15M
Table 2: Quantitative comparison with encoder-based methods. We compare TEXTBooST with encoder-based methods, which require pre-training. The results highlight the superior performance of our method over the baselines in lerms of subject similarity and text fidelity.
Methods
T21 Model
VQA f
DINOv2 1
ELITE
0.430
0.489
SDL.4
TEXTBoOST+
0.558
0.564
BLIP Diffusion
0.650
0.531
SD1.5
TEXTBoOST+
0.577
0.575
5 Experiments
5.1
Experimental setup
Datasct. We employed the benchmark introduced by Ruiz et al. (2023), which consists of 30 subjects and 25 lext prompts, with each subject having 4 to associated images. For style experiments, we use the reference images provided by StyleDrop (Sohn al.) (2023). It important note that we trained all of the models using a single reference image. Following DreamBooth, generated images per prompt (3,000 images in total) for evaluation.
Evaluation metrics. We evaluate imago-text alignment, which mcasures how well the image reflects the user prompt,, and subject fidelity, which assesses how accurately the image represents the characteristies of the given concept. Regarding image-text alignment, while prior works commonly used CLIP score, recent findings by (Lin et al. 2024) indicate that the VQA score better reflects human perception. Therefore, we opted for the VQA score for more accurate evaluation. Subject fidelity was assessed by computing the pairwise cosine similarity between generated and reference image embeddings, following the methodology of (Ruiz et al. 2023). We employed DINOv2:ViT/L-14 from DINOv2 (Oquab et al., 2024) our feature extractor, aligning with recent approaches (Lee al. 2024 Jang et al. 2024). We used the foreground of the validation images suggested by (Kim et al. 2024
To obtain foreground images, employed GroundingDINO (Liu et al, 2024) and GroundedSAM [Ren et al. 2024). Since these scores do not evaluate the extent of overfitting, memorization of reference images can result in inflated scores. Considering this, we assessed the inter- similarity between generated images using the DINOv2 (Oquab et al.) 2024) embedding. Finally, to assess the practical applicability of our method, we conducted a user study involving 50 participants on Amazon Mechanical Turk to evaluate user preferences.
Basclines. We compare our approach with recent personalization methods: DrcamBooth Ruiz et al. 2023), LoAA (Hu et al. 2022). Textual Inversion (Gal et al. 2023 Custom Diffusion (Ku-
mari et al. 2023) NeTT (Alaluf et al. 2023 and DisenBooth (Chen al. 2024b). We also compare
8

---

Published in Transactions on Machine Learning Research (05/2026)
60
50
Methods
40
30
10
Custom Ours Diftusion
Disen Ours Booth Win race
NeTI Ours
Figure 5: User study. We conducted large-scale user-preference survey with 50 participants on Amazon Mechanical Turk. We asked each participant 30 questions to choose an image that best resembles the given subject and aligns with the lext prompt at the same time (1,500 responses in lotal).
method with zero-shot personalization methods (Wei et al. 2023 Li ct al. 2023 which require pre-training beforehand.
Implementation details. We utilized the imagenet small template for the training prompt, following GGal et al., 2023). For the identifier, we employed templates of the form V [class], similar Custom Diffusion. For example, our prompt was constructed as, "a photo of a nice V dog". We also implemented weak augmentation, though significantly less aggressive than that used in Custom Diffusion (Kumari et al. did not modify the loss computation region using masking. For the T21 model, used various versions of Stable Diffusion (Rombach et al.] 2022) model, integrating the text encoder from the CLIP (Radford et al.] 2021 or OpenCLIP (Cherti et al. 2023). Our adapter is trained using the AdamW optimizer (Loshchilov & Hutter 2019) with learning rate lc-4 for text encoder and 1e-3 for V token for 100 steps. For TexTBooST++, we trained for 80 steps. All experiments were conducted with batch size on a single NVIDIA A6000 GPU.
2023 Furthermore,
5.2 Quantitative results
The quantitative comparison is presented in Tab. [ Our method performs on par with existing approaches while using significantly fewer parameters. TEXTBoOST achioves a notably high VQA score, demonstrating strong performance. Regarding subject fidelity, our method consistently produces results that are compara- ble or exceed those of other methods, suggesting that with proper tuning of the text encoder, effective personalization be achieved without sacrificing performance. Additionally, our method excels in gener- ating a diverse range of output images, indicated by the high diversity score, providing end users with diverse options.
We also compare our method, TEXTBOOST, with ELITE and BLIP Diffusion in Tab. 2 As shown in the table, TExTBooST achieves comparable or even superior image and text fidelity compared to these encoder-basod methods, which typically require extensive pro-training. This highlights a key advantage of our approach, it does require pre-training subject data, enabling rapid and efficient fine-tuning during test time.
Computational efficiency. We emphasize the practical advantages of our method, which demands sig- nificantly fewer trainable parameters, detailed in Tab. D In this regard, the fact that our method achieves performance comparable to existing approaches is a significant advantage. Morcover, personalization intro- duces storage challenges, as the fine-tuned model must be saved for each concept. Consequently, the number of parameters directly correlates with storage requirements, as all parameters related to the fine-tuned com- ponents of the T2I model must be stored. While Textual Inversion offers notable storage efficiency due its focus on fine-tuning text embeddings, its limited performance hinders practical application, evidenced by the low image fidelity and image-text alignment scores shown in the table. The compact size of our model

---

Published in Transactions on Machine Learning Research (05/2026)
Reference
TextBoost (Ouns)
image
DicenBooth
(Oura)
Image
Custom Diflusion
NaTi
sketch V dog a broycle on the street"
VA soufed animal running in anowy winter fores:"
"A'" dog flying the unlverse"
"A painning of vase the style of Moner"
Figure 6: Qualitative comparison. We compare images generated by diverse set of methods, utilizing various lypes of text prompls across different subjects. It is important to note that all models were trained using a single reference image, positioned at the leftmost of each comparison. The results demonstrate that our TEXTBOOST method can accurately generate images that adhere to the given prompt, maintaining high subject fidelity, even when compared to more computationally intensive methods with a large number of parameters.
TextBoost (Ours)
TextBoost (Ours)
Referance Custom Diffusion
NaTi
DisenBooth
Custom Diffusion
NoTI
DisenBooth
Reference Image
V stuffed animal walling
V dog wearing party han, bi tirthday cake front, surrounded by confert..
purple
style"
painting
annwy
wearing playing g park. Filld with autummn leaves"
where the pond fillad with sky fillad with
Figure 7: Comprehensive captions. To demonstrate the evenessv of TExTBOOST in handling detailed captions, we generate images from long and creative descriptions that vary in attributes, backgrounds, and styles. Compared to other bascline methods, TExTBoosT exhibits superior performance in accurately reflecting these comprehensive captions, maintaining high subject. fidelity.
allows it to be seamlessly stored in environments with limited storage capacity, such as the cloud or portable storage devices.
User study. In real-world scenarios, users need to select the output image that both satisfies (1) subject fidelity and (2) alignment between the text prompt and the generated image. To evaluate existing methods and understand user preferences, we conducted a largo-scale user study. Participants were tasked with se- lecting the best image from multiple options generated by different methods. We employed diverse subjects and text prompts sourced from (Ruiz et al., 2023) (templates in Appendix [A]. To ensure fairness, random seods were fixed, and we computed the win-rate between our method and previous works. Through Amazon Mechanical Turk, 50 participants completed 30 questions each, yielding total of 1,500 responses. As indi- cated Fig. 5 our method showed higher win-rates compared to all of the existing methods, demonstrating its superior ability to meet user demands for subject fidelity and text-image alignment in practical settings.
5.3 Qualitative results
We conducted qualitative analysis to compare the baseline models with TEXTBoOST, and the results are presented in Fig. and Fig. 7
10

---

Published in Transactions on Machine Learning Research (05/2026)
Reference
Image
DisenBooth
TextBoost
(Ours)
"A cat style"
Figure 8: Results on stylization. The output generated by TEXTBOOST effectively captures the reference style while maintaining key details such as texture, color patterns, and artistic muances. In comparison the recent method, DisenBooth, our TEXTBOOST demonstrates superior fidelity in reproducing these stylistic features.
Custom Diffusion
TextBoost (Ours)
DreamBooth
DisenBooth
NeTl
Reference image
"A V dogin the swimming poo!"
Figure 9: Diversity comparison. We evaluate the inter-similarity of images generated with identical prompts across various baselines. Our TEXT BooTT consistently generates images with high diversity.
To begin with, Fig. P illustraaes examples generated from creative text prompts. TEXTBooST consistently outperforms other methods in accurately reflecting the user's prompt. For instance, in the prompt "A v dog riding a bicycle on the street", alternative models exhibit several issues, such as the omission of the bicycle (DosenBooth), the incorrect display of only the bicycle (NeTI), or the occurrence of attribute leakage (e.g.. a blue shirt and a street in the image from Custom Diffusion). In contrast, TEXTBooST correctly includes the elements specified in the prompt. More qualitative comparison results can be found in Appendix B
We further evaluated TEXTBoOST with more complex prompts that involve simultaneous modifications to attributes, background, and style, depicted in Fig. Our findings demonstrate that TExtBooSt accurately interprets and generates images based on these comprehensive captions. For example, the term "walking" in the top-left prompt posed difficulties for baseline models in capturing the requisite details and style, whereas TExTBOOST effectively represented these elements. Tn the bottom-right example, the inclusion of the word "raccoon" caused confusion other models, resulting in artifacts or misidentiiiccttin of dog. However, TEXTBoOST accurately generated the output of both the dog and raccoon as intended.
These results indicate that TEXTBooST enhances prompt alignment and ensures subject accuracy, making it an ideal tool for users seeking greater control over the creative process. Additional results, including experiments facial datasets, can be found in Appendices and
Stylization. We also conducted experiments on style personalization using a single reference image. Lever- aging the style references from (Sohn et. al. 2023) Stable Diffusion v2.1, evaluated approach in
11

---

Published in Transactions on Machine Learning Research (05/2026)
0.56
0.56
0.55
0.54
0.52
DINOV2
0.54
DINOV2
0.50
0.53
0.48
0.48
0.52
Nalve Fine-tuning (wo/ CPA)
0.44
0.51
TextBoost (Ours)
0.42
0.670
0.672
0.674
0.676
0.678
0,680
0.682
VOA
0.50
0.55
0.60
0.65
0.70
0.75
0.80
VQA
(a) Effect of CPA
(b) Effect of adapter placement
Figure 10:
Ablation study. (a) Our proposed CPA results in better image and text fidelity (b) Placing the adapler in fc2 shows best performance.
comparison to DisenBooth CChnn et al. 2024b), as alternative methods proved ineffective for this particular task. For example, NeTI tends to overfit, resulting in images that fail to capture the intended subject- such as cat-while only reflecting the style (see Fig. i4 Appendix Bp. As demonstrated Fig. S our method effectively learns and applies the style from single reference image, maintaining fidelity the content of the text prompt.
Diversity. In Tab. 1 we mcasured the inter-similarity of output images using DINOv2, finding that TEXTBOOST exhibits greater diversity than other approaches. A visual example is shown in Fig, g where other baselines produce similar images for the same prompts, while our method generates more varied poses and backgrounds. This demonstrates TEXTBooST's advantage in generating diverse outputs, highlighting its strong potential for real-world applications. More results can be found in Appendix
5.4 Ablation study
Finally, conduct an ablation study on our causality-preservv adaptation (CPA) and the adapter place- ment.
The ablation results show the effectiveness of our CPA; its removal from TEXTBOOST markedly lowerod
VQA and DINOv2 scores, demonstrating its significant contribution (see Fig. 10(a) Note that Tab. details our extended embedding space's effect.
While parameter-effiient fine-tuning (PEFT) often largets self-attention modules (e.g., LoRA), systematic studies of optimal PEFT targets are lacking. We thus explored six linear layers in transformer blocks: self- attention query (Q), key (K), value (V), output (0), and MLP layers (fc1, fc2). As shown in Fig. 10(b) MLP layer updates results in better text-image alignment compared to adapting self-attention such as K and V projections, verifying our integration of adapters into the MLP's fc2 layer.
5.5 Results SDXL
In widely used models such Stable Ditfusion v1.5 and v2.1, single CLIP text encoder is used. However, the larger SDXL model has two text encoders: CLIP ViT-L and OpenCLIP ViT-bigG (Cherti et al. 2023 To lest the scalability of our approach, we lested TEXTBoOST on larger models. Specifically, we fine-tune only the OpenCLIP ViT-bigG encoder SDXL while keeping the CLIP ViT-L encoder frozen. This choice is motivated by the ablation results in Tab. X which suggest that adapting single text encoder provides favorable trade-off between personalization performance and computational efficiency. The results, shown in Fig. 11 demonstrate that TexrBooST method effectively captures subject and style characteristics
12

---

Published in Transactions on Machine Learning Research (05/2026)
Subject
Style
"A V the snow."
"An elephant V style"
waiking wearing senta Blue sky and palm treas are the background."
"A butterfly in V* style"
Figure 11: Results on SDXL. The results show that our proposed TEXTBOOST efficiently scales larger T2I models, which require multiple text encoders while maintaining high level of efficiency (a small number of trainable parameters and low storage requirements), which is practical, especially for large T2I models.
across a variety of prompts. In our GPU setup, methods like DreamBooth and Custom Diffusion require too many parameters, making them less efficient for personalization due to memory limits. On the other hand, our method uses far fewer parameters, enabling faster training and better T2I personalization within SDXL. This shows that our approach more practical and better suited for resource-limited environments, such as real-world applications, research, edge devices.
6 Discussion & Conclusion
The role of text encoders in large-scale text-to-image (T2I) models has attracted growing attention in recent years. Prior works (Saharia el al.) 2022; Chen et al. 2024 la Li et al.] (0244a has shown that strengthening the text side of the model can yield substantial gains in generation quality and text-image alignment. Despite these advances, the potential of text encoder adaptation for personalization has remained largely underexplored, with most existing methods focusing instead on the image generation module or on learning concept-specific token representations.
In this work, revisit this design choice and show that the text encoder is not merely auxiliary condi- tioning component, but highly effective locus for personalization. Through TEXTBoOST, demonstrate that carefully adapting the text encoder can achieve personalization performance comparable to updating the image module, while maintaining strong diversity and offering substantially more efficient parame- terization. These findings suggest that high-quality one-shot personalization does not necessarily require extensive modification the diffusion backbone; rather, significant gains can be achieved by selectively and structurally adapting the text-side representation space.
More broadly, results highlight the importance of understanding how personalization is encoded and propagated through the conditioning pathway of T21 models. We believe this perspective opens up new opportunitics for designing lightweight, scalable, and practically deployable personalization methods, par- ticularly in resource-constrained settings where storage and training efficiency are critical. Looking forward, we expect that further investigation into text-side adaptation including richer encoder architectures, more expressive conditioning schemes, and broader concopt domains such as faces and artistic styles- will lead more flexible and reliable personalization systems. We hope TEXTBooST serves as step toward this direction and stimulates further research text-centric approaches to efficient. T2I personalization.
13

---

Published in Transactions on Machine Learning Research (05/2026)
Version note
A preliminary arXiv version of this work (arXiv:2409.0822)) used an earlier formulation based on aug- mentation tokens, knowledge preservation, and weighte timestep sampling. The present TMLR version Adaptation (CPA) and the extended
substantially revises the method by introducing Causality-Preserve textual embedding variant TEXTBoOST++, together with updated experiments, ablations, and additional analyses. At the same time, both versions share the same central idea: fine-tuning the text encoder for efficient one-shot text-to-image personalization.
Acknowledgements
This work was supported by the Institute for Information & communications Technology Planning & Eval- uation (ITP)) grant funded by the Korea government (MSIT) (RS-2019-TT19007) Artificial Intelligence Graduate School Program (KAIST)); this rescarch was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP (No. RRS-2025-00520207); by the 1ITP grant funded by the Korea government (MSIT) and KEIT grant funded by the Korea govern- ment (MOTIE) (No. 2022-0-01045); by the IITP grant funded by the Korca government (MSIT) and KEIT grant funded by the Korea government (MOTIE) (No. 2022-0-00680); and by the Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. RS-2024-00457882, National Al Research Lab Project).
References
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? In IEEE/CVE /CV International Conference on Computer Vision, 2019.
Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A Neural Space Time Representation for Text-to-Image Porsonalization. ACM Transactions on Graphics (TOG), 42, 2023.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. cDiff-I: Text-to-Image Diffusion Models with an Ensemble 2022.
of Expert Denoisers. arXiv preprinl arXiv:2211.0133
Chaofeng Chen, Annan Wang, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Enhancing diffusion models with Lext-encoder reinforcement learning. In European Conference on Computer Vision, 2024a.
Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. DisenBooth: Srentity-Preservine Disentangled Tuning for Subject-Driven Text-to-Image Generation. In Internalional Conference on Learning Representations, 2024b.
Junsong Chen, Jincheng YU, Chongjian GF, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. "'xatt-':alphes Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations, 2024c.
Shangyu Chen, Zizheng Pan, Jianfei Cai, and Dinh Phung, Para: Personalizing text-to to--tto-ing text-to-image image diffusion via parameter rank reduction. In International Conference on Learning Representations, 2025.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ioharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVE Conference on Computer Vision and Pallern Recogrition, 2023.
Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. arXiv preprint arXiv:2810.0111 2023.
Patrick Essor, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Do- minik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling Rectified Flow Transformers for High- Resolutio Tmage Synthesis. In International Conference on Machine Learning, 2024.
14

---

Published in Transactions on Machine Learning Research (05/2026)
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen- Or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. Tn International Conference on Learning Representations, 2023.
lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, 2014.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- Transfer Learning for NLP. In Interna-
mundo, Mona Attariyan, and Sylvain Gelly, lional Conference on Machine Learning, pp- 2790- 2799.. PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, La Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Modcls. In International Conference Learning Represendations, 2022.
Geonhui Jang, Jin-Hwa. Kim, Yong-Hyun Park, Junho Kim, Gayoung Lee, and Yonghyun Jeong, Decoo::Dnonooositionn and Projection of Text Embeddings for Text-to-Image Customization. arxiv preprint 0rXiu2442.091000 2024.
Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of StyleGAN. In IEEE/CVF Conference on Computer Vision and Pattern Recognilion, 2020.
Jimyeong Kim, Jungwon Park, and Wonjong Rhee. Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Paltern Recognition, 2024.
Komal Kumar, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman Khan, Ivan Laptev, and Hisham Cholakkal. Deft: Decompositional efficient fine-tuning for text-to-image models. arXiv preprint arXiv:2509.22 2025.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Yan Zhu. Multi-Concept Cus- tomization of Text-to-Image Diffusion. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, and Jinwoo Shin. Direct Consistency Optimization for Compositional text-to-Image Personalization. In Advances in Neural Information Processing Systems, 2024.
Dongxu Li, Junnan Li, and Steven C. H. Hoi. BLIP-Diffusion Pre-trained Subject Representation for Con- trollable Text-to-lmage Generation and Editing, In Advances in Neural Information Processing Systems, 2023.
Yanyu Li, Xian Liu, Anil Kag, Ju Hu, Yerlan Idelbayev, Dhritiman Sagar, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Textcraftor: Your text encoder can be image quality controller. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 7985-7995, 2024a.
Yanyu Li, Xian Liu, Anil Kag, Ju Hu, Yerlan Idelbayev, Dhritiman Sagar, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. TextCraftor: Your Text Encoder Can be Image Quality Controller. In IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, 2024b.
Yijun Li, Richard Zhang, Jingwan (Cynthia) Lu, and Eli Shechtman. Few-shot Image Generation with Flastic Weight Consolidation. Tn Advances in Neural Information Processing Systems, 2020.
15

---

Published in Transactions on Machine Learning Research (05/2026)
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating Text-to- Text-to-Visual Visual Generation with Image-to- -ext Generation. In European Conference on Computer Vision, 2024.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. In European Conference on Computer Vision, 2024.
Nya Loshchilov and Frank Hutter. Decoupled Weight Docay Regularization. In International Conference on Learning Represenlations, 2019.
Byeonghu Na, Minsang Park, Gyuwon Sim, Donghyeok Shin, HeeSun Bae, Mina Kang, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Diffusion adaptive text embedding for text-to-image diffusion models. arXiv preprint ar 2to:2510.22974, 2025.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text- Guided Diffusion Models. In International Conference on Machine Learning. PMLR, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fornandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wer Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. DINOv2: Learning Robust Visual Features without Supervision. Transaction on Machine Learning Research, 2024. ISSN 2835-8856.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In International Conference on Leurning Representations, 2024.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling Text-to-Image Diffusion by Orthogonal Finetuning. In Advances in Newral Information Processing Systems, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriol Goh, Scott Gray, Chelsea Voss, Alcc Radford, Mark Chen, and Ilya Sulskever. Pero-Shot Texl-Lo-Image Generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- fextt-Conditioaal Image Generation with CLIP Latents. ar preprint ar Niv:2204.06122 2022.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Ti, Jie Yang, Hongyang Ti, Qing Jiang, and Lei Zhang, Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks, 2024.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in Style: a StyleGAN Encoder for Image-to-Tmagee Translation. In IEFE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Resolttno Image Synthesis with Latent Diffusion Models, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
16

---

Published in Transactions on Machine Learning Research (05/2026)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In International Conference on Medical Image Compuling and Compuler-Assisleee Intervention, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfr Aberman. Dream- Booth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In IEEE/CVF Con- ference Computer Vision and Pallern Recognilion, 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. HyperDreamBooth HyperNetworks for Fast. Personalization of Text-to- Image Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic Text-to-lmage Diffusion Models with Deep Language Understanding. In Advances in Newral Information Processing Systems, 2022.
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. ZipLoRA: Any Subject Any Style by Effectively Merging LoRAs In European Conference on Computer Vision, 2024.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilirrum Thermodynamics. In International Conference on Machine Learning. PMLR, 2015.
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, Yuan Hao, Irfan Essa, Michael Rubinstein, and Dilip Krishnan. Style- Drop: Text-to-Image Generation in Any Style. In Advances in Neural Information Processing Systems, 2023.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for StyleGAN image manipulation. ACM Trans. Graph., 40(44:133:11133:11 2021. ISSN 0730-0301. doi:
Soobin Um and Jong Chul Ye. Minority-focused text-to-image generation via prompt optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20926-20936, 2025.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. P+: Extended Textual Conditioning in Text-to-Image Generation. ar preprint orXiv:2303.098: 095222 2023.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Texl-to-Image Generation. In IEEE/CVP International Conference on Computer Vision, 2023.
17

---

Published in Transactions on Machine Learning Research (05/2026)
Appendix
A
User Study Details
backpack in background"
Q1.
reference imm
(a)
"A dog in police outfit"
Q2.
reference image
(a)
(b)
User study template. Each participant asked to choose an image that best meets two criteria: subject fidelity and text prompt fidelity. We asked 50 participants answer 30 questions each on Amazon Mechanical Turk and show the win-rate.
Figure 12:
To evaluate real-world user preferences for various personalization methods, conducted user study using Amazon Mechanical Turk. We recruited 50 participants, each of whom answered 30 questions. Participants were instructed to select the image that best satisfied two criteria simultaneously: (1) resembling the given subject and (2) adhering to the provided prompt. For cach question, users compared two images: one generated by our method and the other by prior work. The prior methods used for comparison included Custom Diffusion (umaar al.] 20233. NeTI (Alaluf et al., 2023), and DisenBooth (Chen al., 2024b). which represent the most recent advances in this domain. Example questions are provided in Fig. 12
To ensure fair comparison, output images were randomly chosen (random seed and prompt) from the DrcamBooth Ruiz et al. (2023) prompt set. When comparing two images (as shown in Fig. 5 options (a) and (b)) we used the same random seed. Furthermore, to avoid position bias, the image options were presented to participants a randomly shuffled order. The win rates for each method were calculated and are summarized Fig. 5 the main paper.
B Comparison with Existing Works
B.1 Positioning of our work among text-side adaptation methods
Recent work has explored the text-side of diffusson models in several different ways. Among methods that fine-tune the tezt encoder, TextCraftor [Li al. 2024a and TexForce (Chen et al. 2024a) show that adapting the text encoder can improve general image quality and text inc--imageeee alignment, but they not designed for instanco-specific onc-shot personalization. In contrast, PaRa (Chen et al.) 2025) and DEFT (Kumar et al. 2025) are efficient personalization hatpaaiiiiiiiii/aaaaallliooo 'adaptation frameworks that constrain or decompose model updates, but they are not specifically wwwe-eenoode-ceeneeeeee and do not address preserving the causal semantics of the text encoder. Mcanwhile, a line of work operates on the tezt conditioning: P2L (Ching et al.) (023) optimizes embeddings for inverse problems, while DATE (Na et al. 2025) and MinorityPrompt (Um Ye (025)) dynamically adapt text embeddings prompts during sampling to improve minority generation.
18

---

Published in Transactions on Machine Learning Research (05/2026)
Custom Diffusion
TextBoost (Ours)
Reference
DisenBooth
NeTi
"A sketch of V* chair"
"A drawing of V* cat surfing on wave"
"A watercolor painting of V* toy in the clouds"
"A V toy in the jungle"
"A V* dog wearing wizard hat"
"A V stuffed animal in the purple flowers"
Figure 13: More qualitative results. We provide additional qualitative examples in comparison with recent personalization works. We used Stable Diffusion v1.5 as the baseline T21 model.
Compared with these works, method targets one-shot personalization from single reference image by selectively fine-tuning the text encoder, introducing CausalityyPresservel Adaptation (CPA) to preserve the embeddings of tokens before V*, and further enhancing conditioning with layer-wise embeddings injected before each cross-attention layer.
B.2 More experiments for comparison
Here, we present more qualitative results compared to several recent works, as illustrated in Fig. 13
For our baseline text-to-image (T2I) model, we employed Stable Diffusion v1.5. It important to note that the results from Stable Diffusion v2.1-base and v2.1 discussed in the main paper. Our method
19

---

Published in Transactions on Machine Learning Research (05/2026)
demonstrates superior ability to generate images that closely align with the subjects of the reference images, significantly outperforming other approaches in capturing the essence of the given text prompts.
In the context of style personalization (i.e., stylization), we compared our approach with Dosen as detailed the main paper. Additionally, we explored stylization using NeTI (Alaluf et al. 2023), though these results were not included in the main paper due to NeTT's tendency to rapidly overfit. This overfitting results in the memorization of style, often at the expense of properly reflecting the input text prompts. A
comparison with NeTI is provided in Fig. 14
Reference image
NeTI
TextBoost (Ours)
"A cat in V* style"
Figure 14: Stylization. We provide style personalization results of NcTI and compare with our TEXTBoOST.
Implementation Details. We fine-tuned the baseline models following the experimental settings outlined in their respective original papers. However, since our study addresses the one-shot scenario, where only a single reference image is provided as training input, we decreased the training steps to mitigate overfitting in existing methods.
For Stable Diffusion v1.5, we set the following training steps: Custom Diffusion (Kumari et al. 2023
250
steps, NeTI (Alaluf et al. 2023): 500 steps (without bypass), and DisenBooth (Chen et al. 2024b): 1,000
steps.
For style personalization, we used Stable Diffusion v2.1. We adjusted the batch size and training steps only if explicitly specified in the original paper; otherwise, left them unchanged.
C Additional Results of TextBoost
C.1 More results
We present additional qualitative results of our method in Figures 19 ability to produce diverse outputs, we include four generated images for each text prompt. As shown in the figures, TExTBoOST generates diverse images from single reference image for given prompt. Furthermore, it consistently produces high-quality results, offering creative control through imaginative prompts with high diversity.
20 21 and 22
To demonstrate its
Furthermore, we present an ablation study (Tab. on fine-tuning the SDXL text encoders. Although fine- tuming both encoders achieves strong performance, we opt to fine-tune only the larger OpenCLIP encoder, which offers good balance between performance and computational efficiency.
20

---

Published in Transactions on Machine Learning Research (05/2026)
Table 3: Effect of fine-tuning different text encoders SDXL. We compare personalization per- formance when fine-tuning only the CLIP ViT-L encoder, only the OpenCLIP encoder, both encoders. Fine-tuning both encoders improves subject fidelity (DINO), while fine-tuning only CLIP preserves prompt adherence (VQA) better.
Fine-tuned encoder
DINO t
VQA f
CLIP ViT-L only
0.515
0.839
OpenCLIP only Both
0.565
0.763
0.615
0.690
Base text encoder
Fine- (TB)
"A painting of woman wearing a yellow hat"
"A photo a bicycle next to brick wall"
"A painting of quiet street at sunset"
"A detailed interior photo of glass greenhouse filled with rare tropical plants"
Figure 15: Preserving text encoder behavior. We compare images generated using the original (non- fine-tuned) text encoder and our TEXTBooST. The results show that the original token representations preserved, the generated images exhibit noticcable visual diffcrences.
C.2 Effect of CPA
In Fig. 3(b) demonstrate that the proposed CPA is designed to preserve the prefix tokens that appear before V* Although masking is intended to enforce this behavior, we further verify it empirically by mea- suring the cosine similarity between the prefix token representations produced by a base text encoder and those produced by TEXTBooST (which applies CPA). The mean similarity (across all subjects) between the tokens of TEXTBoOST and the base text encoder 1.00, indicating that the prefix representations are effectively preserved.
In addition, the behavior of the original text encoder remains unchanged for prompts that do not include V*. This is validated in the qualitative examples of Fig. 15 where the generated images remain visually identical.
Furthermore, we analyze the effect of CPA under different positions the V* token in the prompt. Since CPA preserves the representations of tokens precoding V*, the impact CPA becomes more noticcable when V* appears later in the sentence, where a longer prefix must be preserved. As illustrated in Fig. [6 CPA tends to better maintain prompt fidelity in such cases.
21

---

Published in Transactions on Machine Learning Research (05/2026)
Early
Early
Early: 'A dog ranning acress sandy beach sunsec!'
Early: V candle gowing on andowsildduringg rainy woining."
Earty: "A V dog bring planic blanket mountain
Late: "A beach suroet with dog ranning
misdoow, sarmunded baoket, fruit, and wilflowees" mountain meadow ying
Late: 'Arainy windowelli concle
blanket, baseet,
Figure 16: Results with varying position. We compare generations with and without CPA when the position of v in the prompt changes. Since CPA preserves the representations of tokens precoding V*, its effect becomes visible when V appears later in the senlence, where longer prefix is preserved.
DINO-VQA
Scales
0.60
10
0.50
DNO
0.40
VOA
Figure 17: DINO-VQA tradc-off under a sweep of classifier-free guidance (CFG) scales. Each point corresponds one CFG value, so the plot shows how the operating point of TEXTBoOST changes as the conditioning strength varies.
C.3 Trade-off across conditioning strength.
by sweeping the classifier-fres freef guidance (CFG) scale at inference time. Since CFG directly controls the strength of conditioning, varying it traces an operating curve for each method rather than reporting only a single best-tuned setting. This visualization, therefore, shows how the quality of personalization changes as conditioning is strengthened relaxed.
We performed a DINO-VQA trade-off experiment (Fig. 17
D Additional Results of +
To provide a comprehensive comparison of text encoder tuning approaches (naive tuning, TEXTBooST, and TExTBoosT+*) present generated images with diverse subjects and prompts in Fig. 18 As shown in the
22

---

Published in Transactions on Machine Learning Research (05/2026)
Reference image
Naive tuning
TextBoost
TextBoost++
"A painting V* dog under tree."
"A painting of V dog beach with small waves."
"A photo of V dog wearing blue bandana."
painting of V* toy near blue cup."
"A photo of V* sneaker ker sand by
Figure 18: Comprehensive comparison. We compare naive encoder fine-tuning, TEXTBoOST, and TEXTBoOSI++, Across the generated images, tb improves prompt adherence compared naive tuning, while TEXTBooST+ improves subject fidelity.
figure, TEXTBoOST improves text prompt fidelity compared to naive luning. Furthermore, TEXTBoosT+ achieves better subject fidelity than tb, which is consistent with the results reported in Tab.
23

---

Published in Transactions on Machine Learning Research (05/2026)
TextBoost (ours)
"A drawing dog'nn in Van Gogh style"
single reference image
"A'* dog riding car"
"A watercolor painting of V* dog'n Times Square"
"A dog with mountain the background"
Figure 19: More qualitative results of our Boot (dog).
24

---

Published in Transactions on Machine Learning Research (05/2026)
"Two V* cats"
a single reference image
the style watercolor painting"
"AV" the sofa" "
"A sketch of V* cat wearing tie"
Figure 20: More qualitative results of our Text (cat).
25

---

Published in Transactions on Machine Learning Research (05/2026)
TextBoost (ours)
single reference image
"A drawing of V plushie in the sky"
"A V* plushie in desert"
single reference image
"A V* cartoon in the night sky"
single reference image
"A painting o* sneaker in the jungle"
Figure 21: More qualitative results of our TextBoost (several subjects).
26

---

Published in Transactions on Machine Learning Research (05/2026)
"A V playing with ball in the
Figure 22: Diversity results. We show four generated outputs of our method with the same input text prompt. Our mothod, TEXTBoOST, consistently generates images with high diversity.
27

---

Published in Transactions on Machine Learning Research (05/2026)
E Experiments on Human Faces
painiing syle natring gardering beett cardon fall colatal freers.
wth wtth ncorton in brciyound
bing All.
A la the 5909.
XV vea hatrdd
cooved cartings
Figure 23: Qualitative results on human faces. We use 3 random faces from FFHQ and 2 generated images from DALL-E. We then use a single reference face image to train our TEXTBOOST. The reference face image is located at the upper-left corner of each generated output.
To demonstrate the versatility of TEXTBoOST on a broader range of reference images, we conduct experi- ments using facial images. To avoid using celebrity images, we select three random faces from FFHQ and two synthetic faces generated by DALL-E. The results, shown in Fig. 23 highlight that even with complex captions, TEXTBoOST produces high-fidelity images with strong text-image alignment, demonstrating clear advantages of our approach.
28


Reviews:
[{"review_id": "uOcgWkQ802", "reviewer": "Reviewer_ohby", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose TextBoost, a one-shot personalization method that focuses on fine-tuning text encoder representations instead of the diffusion model. Over their analyses and experiments, the authors argue that in the setup of a joint optimization with diffusion model and text encoder as the trainable parameters for the personalization task, the parameters of text encoder change significantly and furthermore are crucial representations to achieve accurate personalization. Overall, the authors propose a method with respect to the causality constraints of the text encoder and with careful selection of the tuned parameters, so that the encoding performance of the text encoder is not degregaded. The authors prove the effectiveness of their approach both in terms of parameter efficiency and the various concepts provided in the dreambooth dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The experiments are presented with a standardized personalization benchmark, Dreambooth dataset, with sensible choices for the evaluation metrics. \n- Since the method is suited as a fine-tuning mechanism for a personalized concept, the available comparisons are fair, which involve finetuning the model in different ways for this task. \n- The presented experiments measure the text-image alignment, and concept accuracy in a cinvincing way. \n- The authors include a user study as a perceptual metric. \n- However, as mentioned in the requested changes, the cali on perserving the text encoding structure needs additional experiments to be backed up.", "audience_yn": "Yes", "audience_explanation": "This paper would serve as an interesting study for how can a text encoder such as CLIP, trained with vision alignment can serve as an effective visual representation for the personalization task. In addition, the proposed method serves as a parameter efficient alternative to the available approaches such as DreamBooth, and thus can be useful for resource constrained environments as the authors also mention.", "requested_changes": "- In the section for the results with SDXL, the authors claim that SDXL uses CLIP and T5 as the text encoder. This is factually wrong, where SDXL uses OpenCLIP and CLIP as the two text encoders. In addition, the setup for FLUX includes the mentioned text encoders, instead of SDXL. With that mention, if it is claimed that the method works for encoders like T5, the authors should provide examples with FLUX, to back up the claim. This comparison would\n- The paper mentions the overall method as TextBoost, where they manipulate the text encoder for personalizaed image generation. However, the final approach is the mentioned as TextBoost++, which uses the extended embedding space. The authors are encouraged to explain how this extended space should be utilized in comparison to the vanilla version. As an example, are the different examples given to different layers of the cross attention, or concatenated and used as a single joint representation.\n- Regarding the adapter placement, the authors provide ablations on the moditied components of the transformer blocks in Fig. 10. In addition to the analysis provided here, I believe experiments on the blocks modified would back up the architectural choices for including the adapters.\n- The authors claim that their personalization method is motivated by preserving the embedding structure of the baseline text encoder. However, the provided experiments are more focused on the visual quality with the personalizaed embeddings. The authors should consider showing how does the performance on text encoding change (e.g., does the representation quality change if you don't include the identifier token V*).\n- WHile this is not a compulsory request given the scope of the paper, the experiments section would benefit from comparisons with human centered samples (e.g. human faces), where the concept specific details are high.", "broader_impact_concerns": "The experiments are conducted on public datasets which are open for research use. In addition, for human subjects, the authors preferred samples from public datasets and generated datasets. Furthermore, such an impact statement is not necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VjKWu01pkm", "reviewer": "Reviewer_c2UR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Text-encoder-only personalization (TextBoost): Instead of adapting the diffusion U-Net, the method personalizes by fine-tuning the CLIP text encoder, motivated by an analysis showing the text encoder tends to undergo larger effective updates during full personalization, thus you can get strong one-shot personalization with far fewer trainable parameters. \n\n2. Causality-Preserved Adaptation (CPA): Since CLIP’s text transformer is causally masked, naïve tuning can inadvertently change embeddings for tokens before the learned concept token in a prompt. CPA introduces a masked parallel adapter so only the concept token (and later tokens) are affected, preserving prefix semantics while still learning the new concept. \n\n3. Efficient adapter formulation: The adapter is injected in parallel to selected linear layers and combined via a token-level mask, which avoids the computationally heavier alternative of two forward passes, and keeps the personalization module lightweight and modular. \n\n4. TextBoost++ extended conditioning: Extended conditioning by producing layer-wise refined text embeddings right before cross-attention, aiming for better fine-grained control without repeatedly re-running the full text encoder as in some hierarchical prompt-embedding approaches. \n\n5. Experimental evaluation: Evaluated in a single reference image regime on DreamBooth-style subjects and StyleDrop-style stylization, reporting VQA-based text fidelity, DINOv2-based subject similarity and diversity (with foreground extraction), plus a MTurk preference user study comparing against multiple personalization baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- Fig. 1 supports the motivation that the text encoder is a strong leverage point for personalization: during full fine-tuning, its parameters exhibit larger effective changes than the denoising U-Net, justifying a text-encoder centric adaptation strategy.\n\n- Fig. 2 supports the need for preserving causal semantics in the CLIP text encoder: perturbations can degrade generations, motivating the design goal that adapting the learned concept token V* should not alter the output embeddings of tokens $x_i$ that precede V* in the prompt.\n\n- Tables 1 and 2 provide quantitative evidence for both efficiency (small number of trainable parameters vs U-Net tuning baselines) and performance (text/subject fidelity and diversity metrics), supporting the paper’s empirical claim.\n\n- Figs. 6–9 and appendix figures qualitatively reinforce the main outcomes: better subject fidelity and prompt adherence in more compositional/long prompts and stylization cases compared to recent personalization baselines.\n\n- Fig. 10 provides ablations that connect improvements to the proposed components, including the causality-preserved adaptation (CPA) and design choices around where/how adapters are applied for CLIP text embeddings.\n\n- The MTurk user study adds an external validation layer by comparing human preferences (win-rate) against baselines on combined criteria (subject similarity + prompt match), strengthening the empirical case beyond automated metrics.", "audience_yn": "Yes", "audience_explanation": "The authors build on the insights of other SoTA personalization methods and propose a more parameter efficient method - Textboost, that enables high-quality T2I personalization suitable for deployment in resource constrained real-world applications.\nResearchers interested in one / few - shot image concept based T2I diffusion model personalization, and broadly image-generative models’ fine-tuning, could derive value from the findings of this paper.", "requested_changes": "- I encourage the authors to add trade-off curves in the experiments: (i) personalization/subject fidelity (e.g., DINO-based similarity) versus a divergence measure (e.g., KL between the reward-tilted posterior and the diffusion prior, or text-prompt adherence) across hyperparameters controlling conditioning strength; and (ii) performance/quality of personalization versus compute (e.g., number of trainable parameters, VRAM/memory footprint). These plots would substantiate the paper’s efficiency/performance claims by enabling Pareto-optimal comparisons of TextBoost/TextBoost++ against baselines over a sweep of settings, not only at best-tuned points.\n\n- A more explicit mathematical formulation of the “textual embedding extension” in TextBoost++ would improve clarity. In particular, defining the layer-wise embedding construction and how it interfaces with cross-attention would help readers understand the mechanism beyond the high-level description.\n\n- Fig. 10(b) would benefit from clarifying what each point corresponds to (e.g., which adapter placement/configuration or hyperparameter setting). Additionally, the observed trend of the VQA decreasing as DINOv2 similarity increases should be discussed, as it suggests a prompt-subject adherence trade-off that is important for interpreting the method’s behavior.\n\n- The conceptual positioning of the proposed work would be stronger with discussion of other recent SoTA text-embedding–based personalization approaches (e.g., DEFT[1], Para[2], DATE[3], P2L[4], MinorityPrompt[5]). An empirical comparison with tradeoff curves to at least one highly relevant method (ideally DEFT given its relevance to TextBoost) would further clarify how TextBoost relates to and differs from prior text-encoder/embedding-centric SoTA.\n\n**References**\n\n[1] Kumar, Komal, et al. \"DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models.\" arXiv preprint arXiv:2509.22793 (2025).\n\n[2] Chen, Shangyu, et al. \"PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction.\" The Thirteenth International Conference on Learning Representations.\n\n[3] Na, Byeonghu, et al. \"Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models.\" arXiv preprint arXiv:2510.23974 (2025).\n\n[4] Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 8941–8967. PMLR, 21–27 Jul 2024.\n\n[5] Soobin Um and Jong Chul Ye. Minority-focused text-to-image generation via prompt optimization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20926–20936, 2025", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O5liJqNC6G", "reviewer": "Reviewer_uGBx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a one-shot personalization method for text-to-image diffusion models that avoids fine-tuning large portions of the model. The approach has three key innovations: 1. adapting the CLIP text encoder for personalization, 2. introducing Causality-Preserved Adaptation (CPA) to preserve prefix-token semantics, and 3. extending conditioning with lightweight embedding-space adapters to enable finer-grained control of cross-attention.\n\nStrengths\n1. The idea of fine-tuning the text encoder for T2I personalization is novel. To the best of my knowledge, it's previously unexplored. Most existing personalization methods focus on fine-tuning the U-Net while keeping the text encoder fixed. The authors also clearly motivate this design choice by showing the weight changes in the text encoder layers are larger than those in the U-Net in Fig. 1.\n\n2. The authors mainly validate fine-tuning the text encoder experimentally by comparing against baselines in Table 1. and demonstrating that the approach works on SDXL in Sec. 5.5, although the experiments still require stronger controlled validation and add missing ablation studies.\n\n3. The paper evaluates the method with a fairly large human user study, not only automatic metrics (Experimental setup). In Sec. 5.1, the authors state that they conducted a user study with 50 participants on Amazon Mechanical Turk to assess practical applicability, since automatic metrics (e.g., VQA/DINOv2) may not fully reflect real user preferences.\n\n\nWeaknesses\n1. The paper need to clearly explain what is Extended textual embedding\n\n**Section 4.3 is only a high-level narrative and does not provide a formal mathematical definition of the “extended textual embedding” method.**\n\nWithout a formula, readers can’t clearly understand \n\na. What exactly is different between TextBoost and TextBoost++ in implementation?\n\nb. Where the adapters sit (before K/V? before projection? on the embedding itself?)? \n\n\n2. The authors still need a clean, controlled test to show that fine-tuning the text encoder is better than fine-tuning the U-Net. In Table 1, the text-encoder-based approach is compared against baseline methods, but the comparison is confounded by additional components (e.g., CPA and the embedding-space adapters). **The paper should include an ablation that isolates only text-encoder fine-tuning (without CPA and the embedding-space adapters) and compares it to a matched budget U-Net baseline** (e.g., same number of trainable parameters if applicable, same training steps, same learning rates) to demonstrate the benefit of the tuning the text encoder itself.\n\n3. The paper still needs experiments to show CPA keeps the embeddings of the prefix tokens unchanged\nIn Sec. 4.2.2 (Causality-preserved adapter), authors mentioned \"This mask ensures that the modifications introduced by our adapter A(x) only apply to the output representations of V∗ and subsequent tokens.\" However, **the paper does not provide direct empirical evidence that the output embeddings of prefix tokens remain identical or nearly identical before and after applying CPA**. For example, in example prompt \"A cat in V* style\", the embeddings of the prefix tokens \"A\", \"cat\", \"in\" should be shown to be unchanged before and after adaptation.\n\n4. Readers may also be interested in understanding the effect of fine-tuning different text encoders in SDXL (T5 vs. CLIP vs. both).\n\nIn Sec. 5.5 (Results on SDXL), the authors note that “the larger SDXL model has two text encoders: the CLIP text encoder and a T5 model. To test the scalability of our approach, we tested TextBoost on larger models. Specifically, we fine-tuned only the CLIP encoder of SDXL, leaving the T5 model unchanged.” It would be helpful to also report results for the alternative settings: **fine-tuning T5 only, and fine-tuning both T5 and CLIP** under a matched parameter and compute budget. \n\n5. The authors need to clearly justify the use of **different numbers of training steps when comparing TextBoost and TextBoost++**.\n\nIt seems that the paper compares TextBoost and TextBoost++ in Table 1. as an ablation study to evaluate the effect of the embedding-space adapters. However, in Sec. 5.1 (Experimental setup), the authors state that \"For TextBoost++, our adapter is trained using the AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of 1e-4 for text encoder and 1e-3 for V\n∗ token for we trained for 80 steps\". This difference may confuse readers and makes the comparison less controlled. Please clarify the reason for using different training-step settings, or provide results under a matched training budget.\n\n6. The authors need to provide analysis for the adapter in each ross-attention layer contribute\n\nThe authors need to answer the following questions: **1. Do we really need adapters at every cross-attention layer? 2. are some layers more important than others?**\n\nIn Sec. 4.3 (Extended textual embedding), authors state that \"To enhance the encoder’s representation capacity, we propose adapting the text embeddings individually for each cross-attention layer.\" However, since the U-Net contains many cross-attention layers operating at different resolutions (Low/Mid/High-resolution layers), the paper currently does not clarify which layers drive the gains. Therefore **an ablation would be to run TextBoost++ with adapters placed at: a. All cross-attention layers, b. Low-res only, c. Mid-res only, d. High-res only to quantify the contribution of adapters at each resolution**.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Although I selected Yes for this question, I still expect the authors to address **Weaknesses 2–6** by adding the missing ablation studies and the missing experimental evidence. Most importantly, please address **Weakness 1: Section 4.3 is currently only a high-level narrative and does not provide a formal mathematical definition of the proposed extended textual embedding method.**", "audience_yn": "Yes", "audience_explanation": "This paper is likely of interest to readers working on diffusion models, personalization, and parameter-efficient fine-tuning", "requested_changes": "Expect the authors to address **Weaknesses 2–6** by adding the missing ablation studies and the missing experimental evidence. Most importantly, please address **Weakness 1: Section 4.3 is currently only a high-level narrative and does not provide a formal mathematical definition of the proposed extended textual embedding method.**", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SfYrmhuUMd", "reviewer": "Reviewer_GrJP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces TextBoost, a lightweight, one-shot method for personalization of T2I diffusion models by selectively finetuning the text-encoder. It begins by pinpointing the fact that the text encoder is very sensitive to perturbations. Common wisdom so far is that since the U-net is responsible for the visual diffusion process, personalization is mainly happening by finetuning the U-net parameters. However, the authors have realized that the layers of the text encoder show significant updates when finetuning the T2I model, indicating that updates to the text encoder hold more potential. By only needing to update the adapters added to the text encoder, TextBoost only contains 0.12M-0.15M params to be trained, making it more suitable for practical real-world use cases.\nThe authors present 2 main contributions to the text encoder:\n\n1. The authors take inspiration from adapter methodologies, and introduce CPA, the causality preserved adapter. The main argument made is that by masking the tokens up until the V* token, the model only updates itself from the V* token onwards. The goal of CPA is to update as little as possible in the already brittle text-encoder and only learn how the new concept token fits into the subsequent tokens.\n2. Extended textual embedding: the authors take inspiration from related works in GANs and implement extended textual embeddings for the CLIP text encoder, by adapting the text embeddings towards each cross-attention layer of the diffusion model.\nThe authors evaluate TextBoost on VQA, DINOv2, and on a user study with 50 participants. The paper shows that TextBoost and TextBoost++ outperform others on VQA and are following Custom Diffusion on DINOv2. However, TextBoost is the only method that finetunes the text-encoder, also evidencing in the fact that it is the most lightweight finetuning strategy of table 1.\n\n\n**Strengths:**\n\n•\tThe paper introduces a very elegant and lightweight method for one-shot personalization of diffusion models and presents a method which is simple and relevant for real-world use cases since the method requires little storage requirements. This is the paper’s key strength.\n\n•\tGood prompt alignment: TextBoost achieves good VQA scores compared to other methods also backed by the results from the user study.\n\n**Weaknesses:**\n\n•\tThe paper would benefit from additional theoretical or empirical intuition to clarify why the method is effective or not. A good example is the performance drop from TextBoost to TextBoost++ on VQA. TextBoost++ improves subject fidelity but results in a noticeable drop in prompt alignment, and the authors don’t give any intuition on the reason.\n\n•\tSimilarly, the authors do not provide much intuition on the effect of CPA and why it is so effective. Figure 10 is a helpful start; adding examples where naïve finetuning struggles and CPA succeeds would make the contrast more compelling and help readers better understand the benefit of the solution.", "claims_and_evidence_yn": "No", "claims_explanation": "In general, the claims in the paper are strong and convincing. However, I feel that the paper could benefit from more elaborate evidence that builds trust and intuition for the user on why the method works. See my points below:\n\nThe related work section only discusses relevant work for personalized T2I models and PEFT methods but should have included related works as evidence that specifically focus on (low rank) parameter finetuning of the text-encoder, or alternatively, methods that also focus on one-shot personalization of diffusion models. Given the growing literature on both topics at the time of writing, bringing in a few additional representative works would help readers see how this paper connects to that momentum. Example papers that could have been discussed are TextCraftor (Li et al.) and TexForce (Chen et al.) \n\nRelated, table 1 only contains works up until 2024, and I would encourage the authors to compare against more recent works.\nThe authors present a user study with 30 questions across 50 participants on Amazon Mechanical Turk. The authors describe that random seeds were fixed to ensure fair image diffusion processes. The authors mention that question prompts were selected as \"diverse subjects and text prompts sourced from Ruiz et al.\". It would be helpful to briefly describe how the prompts were selected (e.g., random with a fixed seed or curated), so readers can better assess the study design. With only 30 questions, it could be that prompts were selectively chosen towards well working cases. At present, the paper doesn’t provide evidence that rules out possible prompt selection effects; a short clarification would address this concern.\n\nChen, C., Wang, A., Wu, H., Liao, L., Sun, W., Yan, Q., & Lin, W. (2024, September). Enhancing diffusion models with text-encoder reinforcement learning. In European Conference on Computer Vision (pp. 182-198). Cham: Springer Nature Switzerland.\nLi, Y., Liu, X., Kag, A., Hu, J., Idelbayev, Y., Sagar, D., ... & Ren, J. (2024). Textcraftor: Your text encoder can be image quality controller. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7985-7995).", "audience_yn": "Yes", "audience_explanation": "The paper systematically highlights a problem in current T2I models and shows that focusing on the text-encoder for diffusion personalization is an angle that has been underexplored. The method in the paper is elegant and achieves good performance with a simple change. The understanding and control over personalization approaches in T2I models is a very relevant topic. Therefore, I believe that it is beneficial for TMLR’s audience.", "requested_changes": "While researching related works in the domain of one-shot personalization of diffusion models, I have stumbled upon the paper by Park et al. which is also called TextBoost. The figures in the papers suggest that this original paper comes from the same authors and targets the same problem, therefore I believe this is an initial implementation by the same authors. However, the methodology between that paper differs from this one and does not include the 2 main contributions of this paper. I see that the original paper already has been cited, and Seo et al. reference the SNR-based sampling approach of the original paper. Therefore, my suggestion for the authors of this paper is to change the method name, to avoid further confusion in the future.\n\nAs explained in one of the earlier questions, I think the paper would benefit from a more complete related work section with a comparison with other text-encoder finetuning methods and/or one-shot personalization methods.\n\nThe CPA method is introduced to with the goal of maintaining the auto-regressive causality or the text-encoder. Adding targeted qualitative evidence would help substantiate this claim. An experiment that would clearly show the effect of CPA would be a qualitative comparison with- and without CPA. It would be effective to see (in an image) that the model updates intended to teach the model a new subject (V∗) can ripple backward through the model's weights and interfere with the model’s semantic integrity of the preceding tokens. For instance, prompts where V∗ appears late in the sentence could make any causality effects easier to observe. So instead of: \"A painting of V* toy swimming in a pond in the snowy winter, where the pond is filled with fish and the sky is filled with stars.\", I would want to see: \"A painting of a pond in the snowy winter, where the pond is filled with fish and the sky is filled with stars and a V* toy swimming in it.\" If CPA is crucial for maintaining all the details up until the concept token, it should become clearly visible with such a detailed, descriptive prompt.\n\nThe paper emphasizes on the finetuning of the text encoder over the finetuning of the U-net. While the arguments for this are strong (both the fact that it’s sensitive to change and able to be finetuned in a lightweight way), I see examples of images in the qualitative results that have lost some aspects of their fidelity. E.g. Fig. 6: “A V* dog flying in the universe”, we see that the white stripe on the dog’s head has changed into an asymmetric stripe. Also, the result in table 1 support this claim as it ranks slightly lower in the DINOv2 score compared to competitors. Therefore, it would be very interesting to see the effect of combining TextBoost with U-net finetuning methodologies (Similar to TextCraftor, who show that fine-tuning the text-encoder is orthogonal to the U-net and that finetuning both lead to superior performance). A possible experiment could be to implement TextBoost on top of other methods such as Custom Diffusion, purely to see to what extent the results can be improved (both qualitatively and quantitatively). Of course, it is evident that it would lose its lightweight edge, but it would show its robustness and generalizability. In general, it would be good for the reader to understand the generalizability of the method towards other architectures.\n\nIn the user study, a one sentence note on prompt selection (and seed, if applicable) would improve clarity and reproducibility for readers and build trust that prompts are not hand selected.\n\nSeo, H., Jeong, W., Lee, K., & Chun, S. Y. (2025). Efficient Personalization of Quantized Diffusion Model without Backpropagation. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 7717-7727).", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)