Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis
Authors: taihang Hu
Linxuan Li
Joost van de Weijer
Hongcheng Gao
Fahad Khan
Jian Yang
Ming-Ming Cheng
Kai Wang
Yaxing Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion model
Attribute binding
Text Embedding
Primary area: diffusion_based_models

Abstract: Although text-to-image (T2I) models exhibit remarkable generation capabilities,
they frequently fail to accurately bind semantically related objects or attributes
in the input prompts; a challenge termed semantic binding. Previous approaches
either involve intensive fine-tuning of the entire T2I model or require users or
large language models to specify generation layouts, adding complexity. In this
paper, we define semantic binding as the task of associating a given object with its
attribute, termed attribute binding, or linking it to other related sub-objects, referred
to as object binding. We introduce a novel method called Token Merging (ToMe),
which enhances semantic binding by aggregating relevant tokens into a single
composite token. This ensures that the object, its attributes and sub-objects all share
the same cross-attention map. Additionally, to address potential confusion among
main objects with complex textual prompts, we propose end token substitution as
a complementary strategy. To further refine our approach in the initial stages of
T2I generation, where layouts are determined, we incorporate two auxiliary losses,
an entropy loss and a semantic binding loss, to iteratively update the composite
token to improve the generation integrity. We conducted extensive experiments to
validate the effectiveness of ToMe, comparing it against various existing methods
on the T2I-CompBench and our proposed GPT-4o object binding benchmark. Our
method is particularly effective in complex scenarios that involve multiple objects
and attributes, which previous methods often fail to address. The code will be
 publicly available at https://github.com/hutaihang/ToMe

Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis
Taihang Hu', Linxuan Li', .Joost van de Weijer Hongcheng Gao
Fahad Shahbaz Khan 3.6 Jian Yang'. Ming-Ming Cheng' -2 Kai Wang"*, Yaxing Wang 1.2
VCIP, College of Computer Science, Nankai University, "NKIARI, Shenzhen Futian
Computer Vision Center, Universitat Autònoma de Barcelona UUnieersity of Chinese Academy of Sciences
Mohaaeed bin Zayed University of AI, Linkoping University {hutaihang0O, linxuanli520, {joost, kwang)@cvc. uab es, fahad. wwan@dihaa@@. se {csjyang, cmm, cmm, yaxing) @nankai. edu.cn
geohongchengg2000Aggaaaii
Abstract
Although text-to-image (T2I) models exhibit remarkable generation capabilities, they frequently fail accurately bind semantically related objects or attributes in the input prompts; a challenge termed semantic binding. Previous approaches either involve intensive fine-tuning of the entire T2I model or require users or large language models to specify generation layouts. adding complexity. In this paper, we define semantic binding as the task of associating given object with its attribute, termed attribute binding, or linking it to other related sub-objects, referred as object binding. We introduce a novel method called Token Merging (ToMe), which enhances semantic binding by aggregating relevant tokens into single composite token. This ensures that the object, its attributes and sub-objects all share the same cross-attention map. Additionally, to address potential confusion among main objects with complex textual prompts, we propose end token substitution as complementary strategy. To further refine approach in the initial stages of T2I generation, where layouts are determined, we incorporate two auxiliary losses, entropy loss and semantic binding loss, iteratively update the composite token to improve the generation integrity. We conducted extensive experiments to validate the effectiveness of ToMe, comparing it against various existing methods on the 121-CompBench and our proposed GPT-4o object binding benchmark. Our method particularly effective in complex scenarios that involve multiple objects and attributes, which previous methods often fail to address. The code will be publicly available at http://hhhhh.ooouuuuaaaaaa///t
1 Introduction
Text-to-image generation has seen significant advancements with the recent introduction of diffusion models [57, 59, 62], with their capabilities generating high-fidelity images from text prompts. Despite these achievements, aligning the generated images with the text prompts, which is referred to as semantic alignment [30, 43], remains notable challenge. One of the most common issues observed in existing text-to-image -m-ae (T2I) generation models is the lack of proper semantic binding, where given object is not properly binding its attributes or related objects. For example, as illustrated in Fig. 1, even ofate-of-the-ar art T2I model such as SDXL [53] can struggle to generate content that accurately reflects the intended nuances of text prompts. To address the persistent challenges of aligning T2I diffusion models with the intricate semantics of text prompts, a variety of enhancement
:: Co-corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

dog wearing hat and a cat wearing sunglasses
dog with blue bow tie and a cat red scarf
SDXL
SD3
ToMe (Ours)
SDXL
PlayGround v2
ToMe (Ours)
Figure Current state-of-the-ar T2I models often struggle with semantic binding in generated images according to textual prompts. For example, hats and sunglasses are placed on incorrect objects. We introduce a novel method ToMe to address these challenges.
strategies [35, 46, 87] are proposed, either by optimizing the latent representations [69, 82, 83], guiding the generation by layout priors [54, 71, 85) or fine-tuning the T2I models [21, 34]. Despite these advancements, these methods still encounter limitations, particularly in generating high-fidelity images involving complex scenarios where an object is binding with multiple objects or attributes.
In this paper, we categorize semantic binding into two categories. First, attribute binding involves correctly associating objects with their attributes, a topic that has been studied in prior work [58]. Second, object binding, which entails effectively linking objects to their related sub-objects (for example, a 'hat' and "glasses') ), is less explored in the existing literature. Previous methods often struggled address this aspect of semantic binding. One of the main problems is the misalignment of objects with their corresponding sub-objects. Existing solutions address this through an explicit alignment process of the attention maps [7, 43] or by factorizing the generation projects into layout phases and generation phase [55]. In this paper, we propose simple solution to the attention alignment problem called token merging (ToMe). Instead of multiple attention maps, which can be misaligned, join these objects in single composite token that represents the object and its attributes and sub-objects. This composite token has a single cross-attention map that ensures semantic alignment. The composite token is simply constructed by summing the CLIP text embeddings of the various tokens it represents. For example, the phrase "a dog with hat" is abbreviated as "a dog by aggregating the text embeddings corresponding to the last three words, shown in Fig. 4. To justify the applied embedding addition in ToMe, we experimented with the semantic additivity of the text embeddings (in Fig. 3). Furthermore, mitigate potential semantic misalignment in the end tokens from the long sequences, we propose end token substitution (ETS) technique.
As the T21 generation predominantly determines the layout during earlier phases [27], we introduce an entropy loss and a semantic binding loss to update the token embeddings in early steps, integrating ToMe with an iterative update for the composite tokens. The entropy loss is defined the entropy of the cross-attention map corresponding to the updated composite token. This loss aims to enhance generation integrity by ensuring diverse attention across relevant areas of the image, thereby pre- venting focusing on non non-essential regions. The semantic binding loss encourages the new learned token infer the same noise prediction as the original corresponding phrase. This alignment further reinforces the semantic coherence between the text and the generated image.
Our final method ToMe is quantitatively assessed using the widely adopted T21-CompBench [31] and our proposed GPT-4o [1] object binding benchmark. Comparative evaluations against various types of approaches reveal that ToMe outperforms them by significant margin. Remarkably, our approach is user-friendly, requiring no dependence on large language models or specific layout information. In qualitative evaluations, we notably achieve superior generation quality, particularly in scenarios involving multi-object method. In summary, the main contributions of this paper are follows:
generation. This further underscores the superiority of our
We analyze the problem of semantic binding, and highlight the role of the [EOT] token (Fig. 2). and the problems with misaligned cross-attention maps (Fig. 7). In addition, we explore token additivity possible solution (Fig. 3).
2

---

We introduce a training-fret free approach called Token Merging (Fig. 4), denoted as ToMe. as more efficient and robust solution for semantic binding. It is further enhanced by our proposed end token substitution and iterative composite token updates techniques.
In experiments conducted the widely used T21-ComBBench benchmark and our GPT-4o object binding benchmark, we compared ToMe with various state-of-the-arr approaches and consistently outperformed them by significant margins.
Related works
critical drawback of current rext-to-image models is related to their limited ability to faithfully represent the precise semantics of input prompts, commonly referred to semantic alignment. Various studies have identified common semantic failures and proposed mitigation strategies. They can be roughly categorized into four main streams.
Optimization- based methods primarily adjust text embeddings [20, 65] or optimize noisy signals to strengthen attention maps [26, 48, 63, 69, 82, 83J. These methods are basically inspired by the observations from text-based image editing methods [27, 40, 64, 66), suggesting that the layouts of objects are determined by self-attention and cross-attention maps from the UNet of the T2I diffusion models. For example, Loennd-and-xxiitee [7] improves object existence by exciting the attention score of each object. Divide-and-Bind [43] improves by maximizing the total variation of the attention map to prompt multiple spatially distinet attention excitations. SynGen [58] syntactically analyzes the prompt to identify entities and their modifiers, and then uses attention loss functions to encourage the cross-attention maps to agrec with the linguistic binding reflected in the syntax. A-star [2] proposes to minimize concept overlap and change in attention maps through iterations. Composable Diffusion [45] decomposes complex texts into simpler segments and then composes the image from these segments. Structure Diffusion [20] attempts to address this by leveraging linguistic structures to guide the cross-attention maps. Rich-Teer Text [24] enriches textual prompts by incorporating various formatting controls and decomposes the generation task into merging inferences from multiple region- based diffusions. However, these methods often fail in complex scenarios that generate multiple objects or multiple attributes.
Layout-to- Im-aae methods [4, 9, 14, 17, 25, 32, 36, 47] are widely using layouts, particularly in the form of bounding boxes segmentation maps, as a popular intermediary to bridge the gap between text input and the generated images. For example, BoxDiff [73] encourages the desired objects to appear in the specified region by calculating losses based on the maximum values in cross-attention maps. Similarly, Attention-Refocussti
[52] modifies both cross-attention and self-attention maps control object positions. BoxNet [67] first trains a network to predict the box for each entity that possesses the attribute specified in the prompt, and then force the generation to follow the attention mask control. Additionally, Diffuson [68] enhances text-to-image models by providing extra instance-level control. There are also finetuning methods [5, 42. 50, 79] allow for additional layout conditions after fine-tuning over pair images, which are not specifically designed to solve the semantic alignment problem. Despite their promise, these methods obviously prolong the training time. Furthermore, the application of layout priors challenging when comes to global background descriptions or abstract elements. This limitation constrains the versatility of these techniques, making it difficult to deploy them effectively across real scenarios where non-specific spatial arrangements are crucial.
LLM-mummented methods are mainly following text-to- to--to-yottto---aaeeeee generation pipelines [15, 23, 33, 44. 55, 65, 80, 81, 86]. first to generate layouts from large language models (LLMs) and force the T2I generations to follow this guidance as the previous layout-guided methods. Some methods, such as RPG [75] and Mul.an [39], harness the powerful chain of thought reasoning ability of multimodal L.MS to enhance the compositionality of text-to-image diffusion models.
Finetuning- based methods [13, 76] update the model parameters over huge datasets to augment the semantic alignment. Among them, CoMat [34] proposes an end-to-end fine-tuning strategy for text-to-ing rext-to-image diffusion models by incorporating image-to-texx concept matching. ELLA [30] equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment by bridging these two pre-trained models with trainable semantic alignment connectors. More recently, Ranni [21] improves T21 generation by bridging the text and image with a semantic panel with LLMs and is fine-tuned over automatically prepared semantic panel dataset. There
3

---

DetScore "sunglasses"
60.6
50
Conditional Demoising UNet
22
20
Full Sentence
[dog] token
[EOT] tokens
(dog]
[E07]
Sentence
(a) T2I generation with various tokens
(b) DetScore probabilitiss
We generate images with various input prompts in (a): "a wearing sunglasses and dog wearing a hat"; the single-token embedding [dog]; the end token [EOT] (b) After that, we compute the probability of containing "sunglasses" in the generated images in subfigure.
Figure 2:
are also improved T2I models [10, 11, 51] learning from scratch over huge datasets. These methods improve semantic alignment implicitly by better architecture design and larger amount of training data. They further demand marvelous computational resources to achieve the purpose.
In this paper, we tackle the semantic binding problem, which is broad subcase of semantic alignment, in a training-free manner, neither needing the LLMs any training over additional datasets. Furthermore, we achieve better performance when facing complex T2I generation scenarios where users require multiple objects or multiple attributes related to specific object.
Methods
Semantic binding in T2I generation refers the crucial requirement of establishing accurate associa- tions between objects and their relevant attributes or related sub-objects. This process avoids semantic misalignment the generated images, ensuring that each visual element aligns correctly with its descriptive cues in the text. In this section, we begin by providing the preliminaries. Subsequently, we illustrate the motivation through a series of experimental analyses (Sec. 3.1). Finally, we elaborate on our methods in detail (Sec. 3.2). An illustration of our method ToMe is shown in Fig. 4.
Latent Diffusion Models. We build our novel approach for semantic aligmment on the standard SDXL. [53] model. The model is composed of two main parts: autoencoder (i.e., encoder & and a decoder D ) and a diffusion model (i.e., co with parameter 0). The model co updated by the loss:
LLDM := E2) (0.1)(00.))..)........ a- 1,7) [[|c co(2,, t, (P))12
where Eg is UNet, conditioning latent input text embedding Tę (P) and timestep t ~ Uniform(1, T). More specifically, text-guided diffusion models aim to generate an image from random noise 27 and conditional input prompt P. To distinguish from the general conditions in LDMs, we itemize the textual condition as (P)) where Te the CLIP text encoder [56]*. The cross-attention map is obtained from 60(2,, t, C). Let fa, be feature map output of the network 6g. We get a query matrix Qt = (o(fz, with projection network lg- Similarly, given a textual embedding C, we compute key matrix K. (C) with projection network bc. Then the attention map is computed according to: A, so tmar Q. KTT / /Vd) where d is the latent dimension, and the cell [At]i defines the weight of the j-th token on the i-th token.
3.1 Text Embedding Analysis
To address the semantic binding problem, we concentrate on the text embeddings utilized during the diffusion model generation process, as they predominantly dictate the content of the generated images. For a given text prompt P, tokenized by the CLIP text model by padding a start
'SXXL two CLIP text encoders and concatenate the embeddings the final text embedding.

---

(a) Additivity Text-to-lmage Generations
(b) PCA plot text embeddings
man
0.4
Iqueen- woman-man?
0.2
dog
[dog+hat]
doctor
[doetor-glasses]
0.0
womani
king
0.2
queen
-0.4
[king-man
+woman]
king
[king crown]
queen
[queen
0.6
Figure 3: (a) Image generations with the property of token additivity. All images are generated by the prompt template "a photo of (ohject)." (b) PCA plot for additivity of text embeddings.
token [SOT] and several end tokens [EOT] to extend its length to M(=77 by default). After the
CLIP text encoder тę, the condition is formulated as C = (P). Each row in C represents corresponding token embedding after the CLIP text transformers. For example, the text embedding for the sentence P ='a cat wearing sunglasses and dog wearing a hat" represented as: =
EOT In the following analysis, we take this [e5OT cf. cq , dog weering chat EOT ,.. -M--1 as default example (except when defined differently).
Information Coupling. We begin by generating images conditioning on the textual embedding C. as illustrated in the first two columns at the bottom of Fig. 2-(a). We observe that the attributes appear dog in misalignment between the dog and the cat. Subsequently, we extract the token embedding c, from the textual embedding and input it to the UNet cæ (i.e., columns of Fig. 2-(a). The dog object is frequently wearing glasses, further highlighting the semantic leakage issue. Furthermore, when we take C(EÓT] [o ... c images closely resemble all information obtained using the entire textual embedding C. As the [EOT] interacts with all tokens, it often encapsulates the entire semantic information [41, 72J. We further report the DetScore [12] to show the probability containing the corresponding object ("sunglasses") in the generated 100 images. As illustrated in Fig. 2-(b), for these three different cases, the DetScore is 22.6%, 69.6% and 75.0%, respectively. These findings also align with our observations above.
(2 As depicted in the middle
-1 as input, the generated
We
Additivity Property. Inspired by the semantic additivity of the text embeddings in previ- ous research[6, 49], we experiment the additive property of the CLIP textual embedding. We represent the textual embedding corresponding to the prompt "a photo of a dog" C1 (o50T.c9,... ... dag CB EOT M- The textual embedding for the prompt "a photo of hat" is represented as C2 [c [ogo .f. "..".,,," C c6 .... EOTT Next, we perform element-wiss, addi- tion between the object tokens (i.e., gog and chat) and the corresponding [EOT] tokens. Specifically, the resulting new embedding is C' Concat [0 4). C. (5: M - 1] C2(5 M 1]). Afterward, the textual embeddings C' are input into the diffusion UNet to generate the images shown in Fig. 3-(a). We can observe that this additivity property allows adding objects (up-left), removing objects (up- right, down-left) and even complex semantic computations (down-right). To explore the mechanism behind this phenomenon, we conducted PCA dimensionality reduction visualization on the token representations of each prompt, illustrated in Fig. 3-(b). The directional vector obtained from "queen-king" is approximately identical to that of "woman- with the cosine similarity of 0.998.
In conclusion, our analysis shows that the semantic content of text tokens is coupled and entangled, resulting attribute confusion across different subjects. Moreover, we found that in diffusion models, text embeddings exhibit semantically additive properties. This implies that the diffusion model is capable of interpreting a composite token, derived from the summation of multiple individual tokens, integrating the semantic attributes of the combined tokens.
Nooe in this case, the size of the input textual embedding is x 2048 instead 77 x 2048.
5

---

Prompt
"aleat wearingg glasses"
dady with hut"
"a cat wearing sunglasses and dog with hat
"a cat and dog")
UNet
E1
L sem
E2
Text Eneoder
cast
Or
dog
C'
axd
dog'
BDT
[SOT]
Wear- Ing
plasers
and
dog
with
bar
EOTT
UNet
XO
C'
[SOT]
a
caz"
and
a
dog
[DTT]
Lent
Cross-Attn
(a) Token Merging and End Taken Substitution
(b) Iterative Composite Token Update
Figure 4: ToMe is composed two parts: one with Token Merging and end token substitution, and the other token updating part with two auxiliary losses for iterative composite token update.
3.2 ToMe: Token Merging
Suppose the initial prompt P contains K entities indicated by noun words and their corresponding tokens as {n' ...., nr".... ..., n* J. Each entity is often related to a token with relevant objects or attributes set as (n a²). For example, in the sentence "a cat wearing glasses and dog with hat", n² n² <dog>, a² {<with>,<a>
<eat>, a² <<wearing>.<glaseess
3.2.1 Token Merging techniques
The semantic additivity of token embeddings inspires us to achieve co-expression of entities and attributes by explicitly binding tokens together. We employ element- wise addition to accomplish semantic merging of tokens. For a prompt P containing entities, we fuse each subject-attribute pair (n², .dt) into é no + Eat, referred to a composite token. This innovative approach introduces an additional benefit by utilizing a single composite token to condense a lengthy prompt sequence, resulting in unified cross-attention map, thus avoid semantic misalignment. Such observations further shown in the ablation study and appendix.
End Token Substitution (ETS). Meanwhile, as the semantic information contained in [EOT] can interfere with attribute expression, we mitigate this interference by replacing [EOT] to eliminate attribute information contained within them, retaining only the semantic information of each subject. For instance, when the prompt is "a cat wearing hat and a dog wearing sunglasses," we use the (EOT) obtained from the prompt "a cat and dog" replace the original [EOT] As illustrated in Fig. 4-a, the final text embedding after subject-attribute enhancement and EOT replacement is dog-
[= SOT .4.c2 C2
Ca EOT=
EOT> Here, dog* and EOT* respectively denote
tokens after token merging and end token substitution.
3.2.2 Iterative composite Token Update
Semantic binding loss. As stated in section 3.1, the semantic information of each token embedding is inherently linked. After strengthening the relationship between subjects and their attributes, it becomes crucial to eliminate any irrelevant semantic information within the composite tokens to prevent misrepresentation of attributes. As illustrated in Fig. 4-(b), to ensure that the semantics of the composite tokens correspond accurately to the noun phrases they are meant to represent, we employ clean prompt supervisory signal. Specifically, for composite token embedding adog which corresponds to the noun phrase "a dog wearing hat", we aim for the diffusion model to exhibit consistent noise prediction for this composite token and the full phrase. In mathematical terms, this objective can be expressed as ensuring that co( 2t, adog t) ss c0(2,, C, t). This effectively aligns Vz, log (o [2d0g) e V., Pa(( [18, 28]. At time step I, we use the semantic binding loss to align token semantics Leem EeeLLKK [| Cke t) (z, C, 1l.

---

Entropy loss. Following that, we calculate the information carried by each token embedding through entropy statistics. As shown in Fig. 7. we extract the cross-attention map A₁, corresponding to the k-th token[27]. After normalizing the cross-attention map as Ep,eA, P: = 1, we compute the entropy of each token as entropy(tokeng, the cross-attention maps can help ensure that tokens focus exclusively on their designated regions, thereby preventing the cross-attention map from becoming overly divergent. This is further depicted in Fig. 7, where observe instances of attribute confusion, characterized by different tokens inappropriately influencing the same image region. The entropy regularization loss is defined as Cent ExE(I,K] EpEEAx -P; log(p:) during time step t.
EpieAA -P; log(p:). Decreasing the entropy of
Finally, the overall L = -ent + X Luem is computed by these two novel losses to update the composite token during each time < Topt and À the trade-off hyperparameter.
4 Experiments
4.1 Experimental Setups
Evaluation Benchmarks and Metrics. We evaluate the effectiveness of ToMe over T2I- CompBench [31], a comprehensive benchmark for open-world compositional T21 generations, encompassing attribute binding and object relationships. We focus the semantic binding problem, where T21-CompBench predominantly evaluates through three attribute subsets (i.e., color, shape, and texture). We follow the evaluation protocol [21, 30, 34] that using 300 validation prompts for evaluation under each subset and the BLIP-VQA score[31] as the evaluation metrics, Following that, we adopt the ImageReward [74] model to evaluate human preference scores, which comprehensively measure image quality and prompt alignment. To comprehensively evaluate object binding perfor- mance, we introduce new GPT-4o Benchmark of 50 prompts using the template [objectA] with a [itemA] and a [objectB] with [itemB].". J,". For example, objectA and objectB are objects like "cat" and "dog" while itemA and itemB are associated items "hat" and "glasses". Afterward, we used the multimodal model GPT-4o [1] to compute the consistency score between the generated images and the prompts for objective assessment. More details are available in the Appendix C.5.
Implementation Details. We used SDXL [53] as our base model. To automate image generation for evaluation, we employed SpaCy [29] for syntactic parsing of prompts to identify each object and its corresponding attributes for token merging. The iterative composite token update is performed during the first 20% of the denoising steps Topt 0.2T.
Comparison Methods. To evaluate our method's effectiveness, we compared the current state-of-
the-art methods. These primarily encompass: (1) state-of-the-at T2I diffusion models, including SDXL [53], Playground-v2 [37] (2) Finetuning- based methods, including CoMat [34], ELLA [30] (3) Optimization- based method SynGen [58] (4) LLM- augmented finetuning based method Ranni [21].
More comparison results are shown in the Appendix E.
4.2 Experimental Results
Quantitative Comparison. As shown in Table 1, ToMe consistently outperforms or performs comparably to existing methods in BLIP-VQA scores across the color, texture, and shape attribute binding subsets, indicating its effectiveness in avoiding attribute confusion. Hluman-preferenn evaluated through the ImageReward[74] model(note that the model scores are logits and can be negative) suggest that images generated by ToMe can better align with prompts. Specifically, despite ELLAAss33) use of L.Lama or T--XL. to replace the CLIP Text Encoder for stronger text embeddings, our method still achieves higher BLIP-VQL VQA scores compared to ELLA. The significant improvement in GPT-40 scores also demonstrates the effectivenes of ToMe in object binding.
scores
Qualitative Comparison. Following SynGen [58], we classify the failure cases of attribute binding into three main categories. (i) Semantic leak in prompt, where the attribute a² not corresponding to its entity n²; (ii) Semantic leak out of prompt, where the attribute a² is describing the background or some entity not referred to the prompt P: (iii) Attribute neglect, where the attribute a² totally ignored in the image generation. Fig. presents our qualitative comparison results with other methods. The first three rows show more complex object binding results, while the last two rows demonstrate attribute binding results. The semantic binding errors in images generated by SDXL[53]
7

---

ToMe (Ours)
SDXL
PlayGround-22
ELLA1.5
Rann 1.5
SynGen Xz
a snullve! helding canora and o put
wocaring hat
glil with necidace and mith glasze
caT with sunglezzes and dog Wanniio a. hat
yallam annis and a. s.cood pear
and
20000e bod
Figure 5: Qualitative comparison among various T21 generation methods with complex prompts.
can largely be attributed to (i) semantic leak in the prompt, as evidenced the first and second row. Playground-v2[31 confronts similar semantic binding issue SDXL. ELLA[30] can occasionally succeed in simple attribute binding as in the fifth row, but frequently encounters (i) semantic leak in the prompt and (iïi) attribute neglect errors shown in the first three prompts. Ranni [21] generates images based on layouts created by a large language model, which can partially address more complex object binding (second row). However, layout-based methods may encounter constrains in achieving proper image layouts, such as shown in the first row with complex descriptions. SynGie [58], which focus on attribute binding problems, achieves good results color and shape binding but fails in object binding, exhibiting varying degrees of (i) and (iii) failures. Compared to these methods, our
Table Quantitative results for semantic binding assessment various benchmarking subsets. We denote the best score in blue, and the second- best score green
Base Model
BLIP-VQA
Human-preference t
Method
Train
GPT-4o
Color
Texture
Shape
Color
Texture
Shape
SDXL[[53]
0.6369
0.5637
0.5408
0.7798
0.5140
0.4029
0.4907
PlayG-v2(37]
0.6208
0.6125
0,5087
0.5417
Ranni[21]
0.2414
0.3029
0.2857
-0.8554
-0.6853
-0.8051
0.4166
ELLA[30]
0.6911
0.6308
0.4938
0.6586
0.2963
0.0565
0.6481
SD1.5
SynGen [58]
X
0.6619
0.6451
0.4661
0.4326
0,5072
0.0426
0.5545
CoMat[34]
0.6561
0.6190
0.4975
Ranni[21] ELLA[30]
0.6893
0.6325
0.4934
0.7260
0.6686
0,5634
SDXL.
SynGen[58]
X
0.7010
0.6044
0.5069
1.016
0.7867
0.4016
0.6458
[oM
0.7774
0.6591
0.5262
ToMe (Ours)
SDXL
X
0.7656
0.6894
0.6051
1.074
0.9281
0.5916
0,9549

---

Table 2: Ablation Study conducted on the T2I- CompBench benchmark.
BLIP-VQA Color Texture Shape
Conf. ToMe Lent Lieem
0.6369 0.5637 0.5408
B
X
X
0.6577
0.5828 0.5437
A B
DDD
0.7525
0.6775 0.5797
E p
V
0.5881
0.6194 0.5386
E
X
0.5983 0.5798 0,5125
Figure 6: Text-to- Texi-to-Image generation with various configurations.
x
0.6804
0.6263 0.5645
Ours
0.7656 0.6894 0.6051
Config
Config
ToMe (Ours)
cal
sunglasses
hat
[ear"]
[dog
[eat"]
[dog
Figure
Cross-Attention maps visualization with various configurations.
Add objects
Remove objects
Eliminate generation bias
elephent
dag
girl earrings
glasses
queen
nurse
[elephant-glasses]
[cat<glasses] with dog
girl with sarcings
with glasses
[queen-crown] with cats
Figure Additional applications of semantic additivity in text embedding.
approach ToMe shows improved performance in both object and attribute binding scenarios, which is consistent with the quantitative metrics reflected in Table 1.
Ablation Study over each component is quantitatively shown Table We can observe that using only token merging techniques (with ToMe and ETS config.B) B) results in a slight performance improvement, which is consistent with the qualitative results Fig. 6. However, token merging serve the foundation for subsequent optimizations. When they are combined with the entropy loss Lent as config.C. C, the performance improves significantly. We hypothesize that is partly due to the more regularized cross-attention maps as shown in Fig. 7. Nevertheless, conifg. C without the semantic binding loss still leads to worse generation performance in Fig. 6, as the dog on the right side still exhibits cat-like features. Incorporating the semantic alignment loss Leem (as our default configuration) ensures that the two subjects correctly bind their respective attributes without appearance confusion, achieving the best results quantitatively and qualitatively. Suppose token merging is ignored, and we only apply the optimization (Config and Config E), the performances only comparable to the baseline. Removing Cent from ToMe (Config F) also improve over the baseline, but the generation is with noticeable artifacts, which is mainly due to the less regularized cross-attention map. In conclusion, each element of these three novel techniques in ToMe contributes to achieving state-of-the-an performance. See Appendix D for more detailed ablation experiments.
Additional Applications of ToMe shown in Fig. 8. ToMe not only successfully address the semantic binding problem, it can also be applied to other problems widely exist T2Í generations, including adding objects [84, 70], removing objects [3, 22] and even bias mitigation [16, 61. 77, 78].

---

Conclusion
In this paper, investigate critical issue in text-to-image (T2l) generation models known as semantic binding. This phenomenon refers to instances where T2I models struggle to accurately interpret and visually bind the related semantics. Recognizing that previous methods often entail extensive fine-tuning of the entire T2I model or necessitate explicit specification of generation layouts by large language models, we introduce a novel training-free approach called Token Merging, denoted as ToMe, to tackle semantic binding issues in T2I generation. ToMe incorporates innovative techniques by stacking up the object token with its relevant tokens into a single composite token. This mechanism eliminate the semantic misalignment by unifying the cross- coss-attention maps. Furthermore, we assist the ToMe with end token substitution, and iterative composite token updates technique to strengthen the semantic binding. In extensive experiments, we quantitatively compare it against various existing methods using the T21-Comphench and our proposed GPT-4o benchmarks. The results demonstrate its ability to handle intricate and demanding generation tasks more effectively than current methods, especially for object binding cases that are ignored previous research.
Acknowledgements
We acknowledge project PID2022-143257NB-IBG financed by the Spanish Government htt./AE1/11.1303/////. 10001 1033 and FEDER. We acknowledge project "Science and Tech- nology Yongjiang 2035" key technology breakthrough plan project (2024Z120). The Supercomputing Center of Nankai University supports computation.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 23Xi::23003.88774. 2023.
[2] Aishwarya Agarwal, Srikrishna Karanam, KJ Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for text-to- image synthesis. In Proceedings of the International Conference Computer Vision, pages 2283-2293, 2023.
[3] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 18208-18218, 2022.
[4] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 2022.
[5] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 843-852, June 2023.
[6] Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing text-to-image models using semantic guidance. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 25365-25389, Inc., 2023.
Curran Associates,
[7] Hila Chefer, Yuval Alaluf, Yacl Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention- based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 2023.
[8] Chieh-Yun Chen, Li-Wu Tsao, Chiang Tseng, and Hong-Han Shuai. A cat is cat (not dog!): Unraveling information mix-ups in text-to-image encoders through causal analysis and embedding optimization. arXiv preprint arXiv:2410.003: 2024.
10

---

[9] Hongyu Chen, Yiqi Gao, Min Zhou, Peng Wang, Xubin Li, Tiezheng Ge, and Bo Zheng. Enhancing prompt following with visual control through training- -ree mask-guided diffusion. arXiv preprini arXiv:2404. 14768, 2024.
[10] Junsong Chen, Chongjian Ge, Enze Xie. Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang. Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024.
[11] Junsong Chen, Jincheng Yu. Chongjian Ge, Lewei Yao. Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic rext-to-image synthesis, 2023.
[12] Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark, arXiv preprint (eXXiv/906.000 0555, 2019.
[13] Kai Chen, Enze Xie, Zhe Chen, Yibo Wang, Lanqing HONG, Zhenguo Li, and Dit- Yan Yeung. Geodiffusion: Text-prompted geometric control for object detection data generation. In The Twelfth International Conference on Learning Representations, 2024.
[14] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3343-5353, January 2024.
[15] Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text- text-to-image generation and evaluation. In Advances in Neural Information Processing Systems, 2023.
[16] Ching- Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023.
[17] Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2174-2183. October 2023.
[18] Bradley Efron. Tweedie's s formula and selection bias. Journal of the American Statistical Association, 1064496):1002-1611 2011.
[19] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222-11239, 2023.
[20] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In International Conference on Learning Representations, 2023.
[21] Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffusion for accurate instruction following. arXiv preprint 1XXx:23311170000 2023.
[22] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2426-2436, 2023.
[23] Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. Llm blueprint: Enabling text-to-image generation with complex and detailed prompts. International Conference on Learning Representations, 2024.
[24] songwei Ge, Taesung Park, Jun Yan Zhu, and Jia-Bin Huang. Expressive text-to-image generation with rich text. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 7545-7556, 2023.

---

[25] Biao Gong, Siteng Huang, Yutong Feng, Shiwei Zhang, Yuyuan Li, and Yu Liu. Check, locate, rectify: A training-fie free layout calibration system for rext-to-image generation. arXiv preprint arXiv:2311.1577, 2023.
[26] Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2024.
[27] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-pronee image editing with cross attention control. International Conference Learning Representations, 2023.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 156840-6851,, 2020.
[29] Matthew Honnibal and Ines Montani. spacy 2: Natural language understanding with bloom
embeddings, convolutional neural networks and incremental parsing. To appear, 7(1):411-475 2017.
[30] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with IIm for enhanced semantic alignment. arXiv preprint 1xXii:2443305135,, 2024.
[3]] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehen- sive benchmark for open-world compositional text-to-image generation. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 78723-78747.
Curran Associates, Inc., 2023.
[32] Vikram Jamwal and S Ramaneswaran. Composite diffusion: whole>= oparts. In 2024 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 7206-7215 IEEE, 2024.
[33] Yuhao Jia and Wenhan Tan. Divcon: Divide and conquer for progressive text-to-image genera- tion. arXiv preprint arXiv:2403.064000 2024.
[34] Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. arXiv preprint arXiv:2404.03653 2024.
[35] Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. If at first you don't succeed, try, try again: Faithful diffusion-based text-to-image generation by selection. arXiv preprint arXiv: 20..2335533300 13008, 2023.
[36] Yunji Kim, Jiyoung Lee, Jin--Ha Kim, Jung- Woo Ha, and Jun- Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7701-7711, 2023.
[37] Daiqing Li, Aleks Kamko, Ali Sabet, Ehsan Akhgari, Linmiao Xu, and Suhail Doshi. Playground v2.
[38] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[39] Sen Li, Ruochen Wang, Cho-Jui Hsich, Minhao Cheng, and Tianyi Zhou. Mulan: Multimodal- Ilm agent for progressive multi-object diffusion. arXiv preprint 2rXiv:2402.12744 2024.
[40] Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embedding inversion for text-based editing, 2023.
[41] Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Get what you want, not what you don't: Image content suppression for text-to- image diffusion models. In International Conference on Learning Representations, 2024.
12

---

[42] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 22511 -25221,,, June 2023.
[43] Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing. Proceedings of the British Machine Vision Conference, 2023.
[44] Long Lian. Boyi Li, Adam Yala, and Trevor Darrell. Llm- Am-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. Transactions on Machine Learning Research (TMLR)) 2024.
[45] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439. Springer, 2022.
[46] Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Correcting diffusion generation through resampling. arXiv preprint 0rXiv:2312.06038, 2023.
[47] Wan-Duo Kurt Ma, Avisek Lahiri, JP Lewis, Thomas Leung, and W Bastiaan Kleijn. Directed diffusion: Direct control of object placement through attention guidance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4098-4106, 2024.
[48] Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high-fidelity text-to-image diffusion models. arXiv preprint 12X00:23 2023.
[49] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[50] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[51] Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-in extttto-image diffusion models. In Interna- tional Conference on Learning Representations, 2024.
[52] Quynh Phung, Songwei Ge, and Jia-Bin Bin Huang. Grounded text-to-image synthesis with attention refocusing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[53] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint 20X.v:2307.01952 2023.
[54] Zipeng Qi, Guoxi Huang, Zebin Huang, Qin Guo, Jinwen Chen, Junyu Han, Jian Wang, Gang Zhang, Lufei Liu, Errui Ding, et al. Layered rendering diffusion model for zero-shot guided image synthesis. arXiv preprint arXiv:2311.
2023.
[55] Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua. Layoutlmm-1i:: Eliciting layout guidance from I1m for text-to-image generation. In Proceedings of the ACM International Conference on Multimedia, pages 643 2023.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
13

---

[57] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
[58] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems, 36, 2023.
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 06 2022.
[60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image symthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[61] Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairness. International Conference on Learning Representations, 2024.
[62] Alex Shonenkov, Misha Konstantinov, Daria Bakshandaeva, Christoph Schuhmann, Ksenia Ivanova, and Nadiia Klokova. Deepfloyd-if. https://ge /ttp://ggttbb.../// cartuuu.commddeee
2023.
[63] Maria Mihaela Trusca, Wolf Nuyts, Jonathan Thomm, Robert Honig, Thomas Hofmann, Tinne Tuytelaars, and Marie-Francine Moens. Object-atrribute binding in text-to-image generation: Evaluation and control. arXiv preprint htti:2404.137660 2024.
[64] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven iage-to-imagee translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[65] Hazarapet Tumanyan, Dejia Xu, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. Multi-concept 12i-zero: Tweaking only the text embeddings and nothing else. arXiv preprint arXiv:2310.07419, 2023.
[66] Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, and Joost van de Weijer. Dynamic prompt learming: Addressing cross-attention leakage for text-based image editing. Advances in Neural Information Processing Systems, 2023.
[67] Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma. Haonan Lu, and Xiaodong Lin. Composi- tional text-to-image synthesis with attention map control of diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5544-5552, 2024.
[68] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. In- stancediffusion: Instance-level control for image generation. arXiv preprint 240..0090, 2024.
[69] Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Ground- ing diffusion with token-level supervision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[70] Navve Wasserman, Noam Rotstein, Roy Ganz, and Ron Kimmel. Paint by inpaint: Learning to add image objects by removing them first. arXiv preprint arXiv:2404.182. 182122. 2024.
[71] Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang. Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 7766-7776, 2023.
[72] Yinwei Wu, Xingyi Yang, and Xinchao Wang. Relation rectification in diffusion model, 2024.
[73] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffu- sion. In Proceedings of the International Conference on Computer Vision pages 7452-7461, 2023.
14

---

[74] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[75] Ling Yang. Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Master- ing text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. International Conference Machine Learning. 2024.
[76] Zhengyuan Yang, Jianfeng Wang. Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE Conference on Compuler Vision and Puttern Recognition, June 2023.
pages
[77] Hidir Yesiltepe, Kiymet Akdemir, and Pinar Yanardag. Mist: Mitigating intersectional bias with disentangled attsnttt editing text-to-image diffusion models. arXiv preprint arXiv:2403. 19738, 2024.
[78] Cheng Zhang, Xuanbai Chen, Siqi Chai, Chen Henry Wu, Dmitry Lagun, Thabo Beeler, and Fernando De la Torre. Iti-gen: Inclusive text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3969-3980, 2023.
[79] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[80] Tianjun Zhang. Yi Zhang, Vibhav Vineet, Neel Joshi. and Xin Wang. Controllable text-to-image generation with gpt-4. arXiv preprint arXiv:2305.18588 2023.
[81] Xinchen Zhang, Ling Yang, Yaqi Cai, Zhaochen Yu, Jiake Xie, Ye Tian, Minkai Xu, Yong Tang, Yujuu Yang, and Bin Cui. Realcompo: Dynamic equilibrium between realism and compositionality improves text-to-image diffusion models. arXiv preprint 240.020 2024.
[82] Yang Zhang, Teoh Tze Tzun, Lim Wei Hern, Tiviatis Sim, and Kenji Kawaguchi, Enhancing semantic fidelity in text-to-image synthesis: Attention regulation in diffusion models. arXiv preprint arXivv2403.06381, 2024.
[83] Yasi Zhang, Peiyu Yu, and Ying Nian Wu. Object-conditioned energy-based attention map alignment text-to-image diffusion models. arXiv preprint arXiv:2404.07388 2024.
[84] Ziyue Zhang, Mingbao Lin, and Rongrong Ji. Objectadd: Adding objects into image via training-free free diffusion modification fashion. arXiv preprint 1:Xiv:2404.172330, 2024.
[85] Peiang Zhao, Han Li, Ruiyang Jin, and S Kevin Zhou. Loco: Locally constrained training- layout-to-image synthesis. arXiv preprint arXiv:2311.12342 2023.
[86] Dewei Zhou, You Li, Fan Ma, Zongxin Yang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2024.
[87] Yupeng Zhou, Daquan Zhou, Zuo-Liang Zhu, Yaxing Wang, Qibin Hou, and Jiashi Feng. Maskdiffusion: Boosting text-to-image consistency with conditional mask. arXiv preprini dxXiv:2309.04399, 2023.
15

---

Appendix
A Limitations
Since method is optimized for inference based on SDXL, it inherits some inherent limitations of SDXL. For example, may produce artifacts in generated images and unable to create images with complex layouts. Additionally, the ToMe technique relies on the CLIP text encoder to generate text embeddings, which may be subject to the limitations of the encoder itself. For instance, the CLIP encoder might not fully capture all the subtle semantic nuances in the text, which could restrict the performance of ToMe when processing certain types of text prompts. Addressing these limitations and ad advancing our understanding in these areas will help improve image generation technology.
B Broader Impacts
ToMe enhances the semantic binding capability in text-to-image synthesis by enhancing text embed- dings. However, it also carries potential negative implications. could be used to generate false or misleading images, thereby spreading misinformation. If ToMe is applied to generate images of public figures, poses a risk of infringing on personal privacy. Additionally, the automatically generated images may also touch upon copyright and intellectual property issues.
C
Implementation Details
C.1 Method details
We extract the cross attention maps from the first three layers of the decoder in the UNet backbone, which contain rich semantic information, with resolution of 32 x 32. For Iterative composite Token Update, since the early timesteps the denoising process determine the layout of the image[27), we exccute it only during the first 20% of the denoising process. All experiments conducted on an NVIDIA-A44 GPU.
C.2 Baseline methods implementation
For the quantitative comparison in Tab. 1, used the official implementations of Ranni ELLA[30], SyGen[58], and CoMat[34]. Since the SDXL versions of the Ranni[21], ELLA[30], and CoMar[34] methods have not been open-sourced, we refer to the BLIP-VQA scores reported in their respective papers. SynGen[58], like our method, performs optimization during inference. To ensure a fairer comparison, adapted SynGen SDXL.
C.3 Text embedding analysis
Fig. 9's statistical analysis further demonstrates the information coupling property and semantic additivity of text embeddings. We employed [MMDetection[[2]andd GLIP[38] to detect the probability of specified objects in images, referred to as DetScore, as shown in Fig. 9-(a). Fig. 9-(b) presents statistical results on 100 generated images, showing that the probability of detecting hat in images generated from the text embedding corresponding to "a dog" is 0%. However, in images generated from the element-we wise "[tog+hat]" additive embedding. the probability of detecting a 68.6 (8.61% %, which is close to the probability of 73.12% for images generated using the prompt dog wearing hat'.
The information coupling of token embeddings is also reflected in the entropy of cross-attention for each token. Taking the prompt "a cat wearing sunglasses and a dog wearing hat" an example, we can extract the cross-attn map A& E IR 1024 for each token, averaged over 50 time steps and multiple heads. After normalizing each map to 1.0(.ee, A+ E.en.s Av calculate the token's 's
Aeti)
infomation entropy Epees -P; log(p:). As shown in Fig. 9-(c), we conducted statistics on 100 generated images and found that tokens positioned later in the prompt tend to have higher entropy, indicating more dispersed cross-attn maps. This phenomenon might be attributed to CI.IP's[56] masked attention mechanism, where each token can interact with all preceding tokens, and tokens
16

---

73.12%
1.0
3.9335
0.995
0.9355
0.6
0.8
0.4
0.2
0.1
0,0358
dog
dog*
dog weoring hat
SOT
cat sunglesses dog hat EOT
(a) Examples Detscore
(b) Detscore for hat
Information entropy of each token
Figure 9: Additional statistical analyses, all statistical values are averaged results from 100 images. (a) An example of DetScore visualization. (b) By fusing the dog and hat token, we obtain dog*, and the generated images often include a hat. The DetScore value for dog is close to the DetScore value obtained using the complete prompt dog wearing a hat". (c) calculated the entropy of the cross-attention maps for each token and found that tokens positioned later in the sequence generally have higher entropy, indicating that their cross-attention maps are more dispersed.
positioned later can interact with more tokens, thus containing more information. Consequently, we employ an entropy regularization loss to constrain each attention map be as concentrated as possible, thereby reducing the amount of irrelevant information contained in each token embedding.
C.4
Time complexity
Table 3: Time Complexity of various methods. The results of our method are highlighted in bold.
Method
Inference Steps
Time Cost
Color
Texture
Shape
SDXL
20
18s
0.6136
0.5449
0.5260
ToMe (Config C)
20
23s
0.7419
0.6581
0.5742
ToMe (Ours)
20
45s
0.7612
0.6653
0.5974
Ranni (SDXL)
50
87s
0.6893
0.6325
0.4934
ELLA (SDXL)
50
51s
0.7260
0.6686
0.5634
SynGen (SDXL)
50
67s
0.7010
0.6044
0.5069
SDXL
50
42s
0.6369
0.5637
0.5408
ToMe (Config C)
50
56s
0.7525
0.6775
0.5797
ToMe (Ours)
50
83s
0.7656
0.6894
0.6051
Tab. 3 reports the inference time costs of various methods, all measured on a single NVIDIA-A40 GPU, We demonstrate that our method does significantly increase inference time while improving semantic binding performance with 50 inference steps. We further extend this analysis by measuring the time cost with 20 inference steps and various ToMe configurations, as shown the Tab. 3. We report the time cost (by seconds) along with BLIP-VQA scores across the color, texture, and shape attribute binding subsets. From this table, we can observe that using the token merging (ToMe) technique and entropy loss (Config.C) C), our method achieves excellent performance with minimal additional time cost. Additionally, even with only 20 inference steps, our method, ToMe, maintains high performance with very little degradation.
C.5 GPT-40 Score
In order to better demonstrate the binding ability of our model for complex prompts. We have constructed a set of high-difficulty prompts, where the content primarily uses nouns to describe the subject. We use OpenAl's latest release, GPT-4o, to evaluate the quality of images generated by various models because GPT-4o excels image discernment, allowing for precise evaluation of the generated outputs. As show in Fig. 10, We designed nine scoring levels, ranging from 0 to 100 points,
17

---

Prompt: wearing sung lagosss a dog wearing
Promptt wear rngi
pleture prompt soare the picture below.
Based our sicture prompt give the
suojects promet corresponding ontectives
subjects prompt corresponcing are adjectives
ortributes
sunjects prompt hody
The promet body.
contains
explanation
rating, line.
explonation
the second
The ortteria fellows: 100: Both sutjects only
160: Buth sutjects fellows: attributes,
their
attributes, not attrib bute of the
attr butibtue the other
07.5: fiith sutjeets
their attributes
i: lluth sutjects
their aitributes.
But only wubsest that possauxess the attributes of onother subject.
But only subyes that possesses cttributcs of onother subject
7t: When two subjects pooseso their own attributes, they both possess the attributes other subject
75: When two subjects pecooss their own attributes, they both possess the attributes of other sudject
62.5: One surpect possecses witributes of own without wit buses the wnother wubject
67.5 sutject possesses aitributes own. without attributes of the another subject.
cat wearing sunginsces and day weartig Aat"
The other sulject only passescess altributes of another sudject.
The other wubject only passeusem attribatee another sudtect.
50: Une subject possesses attriuutes
52: Une subject postesses attrib
presesses attributes itsell or other party.
own. other sutject
own The other sutjjct not possesses attritutes lhsel the other party.
37.5 subjects not possess oern altribuless woar one sutject has
37.5 wubjects possess own attributex moat subject has
attributes other party.
party.
subject attributess party.
Ne subject attributes
party.
subject
Missing wabject
100
Eeplanation:
75
The zat weering sunglasses, which is iss own attribute.
Bath the and the women possaess their attributes: the man wearing hot, and the woman is wearing reckiace However both supjects aiso have the attributes of the other the woman is also wearing a hat, and the man also weer necilace.
The dog is wwaring hatt which in ias own attribute.
Neither the nor the dog wearing the attritute other sunject.
man incoring and womar wawring owcklaee"
Therefore, the picture perfectly matchees the prompt pt corning score of 100.
Scorw0.75
Figure 10: Evaluation Metrie: GPT-4o
based on factors such as whether the objects correctly possess their attributes, the mixing of attributes between objects, and whether the objects are correctly generated, to distinguish different levels of generation quality.
sunglasses
dog
Tear
[dog
Config A (SDXL)
Config B (SDXL, Token Merging)
Config (SDXL=E..))
Cunfiig CS Menging)
Figure 11: Cross maps visualization with various configurations, with the input prompt "a cat wearing sunglasses and a dog wearing hat"
D Additional Ablation Studies
D.1 More Configures and ETS ablation
As an example in Fig. 11, the original SDXL (Config.A) suffered from attribute binding errors due to divergent cross-attentis maps. When only applying token merging (Config B), the co-expression of entities and attributes resulted dog wearing hat in the image, but the attribute leakage issue remained due to the divergent cross-attention maps. When only applying the entropy loss Lent (Config F), although the cross-attention maps corresponding to each token are more concentrated, they may focus on wrong regions. Only by applying both token merging and Lent techniques (Config
18

---

[boy*]
feorgi"]
[BOT]
[OO
SDXL
Ours w/o ETS
Ours ETS
Figure 12: Ablation study of our proposed end token substitution (ETS) technique, with the input prompt "a boy wearing hat and dog weairng sunglasses"
(4) Different prompt splice
Prompt
Prompt
Prompt
uearing plasses dog
day
sowaring glasses
Yo fngg mevring but"
Text Encoder
[50T
ced
Year
plows
and
dog
with
hat
BOTT
19OT
(POT
(OTT
cal
dop
(b) Generated Images by Different prompt splice
(b) Generated Images ToMe(Ours)
Figure 13: Comparison of images generated by different prompts splice
C), the cross-attention map of the composite token becomes better concentrated on the correct areas and thus leading to more satisfactory semantic binding of entities and attributes.
The end token substitution (ETS) technique is proposed to address potential semantic misalignment in the final tokens of long sequences. As the [EOT] token interacts with all tokens, it often encapsulates the entire semantic information, shown in Fig. Therefore, the semantic information in [EOT] can interfere with attribute expressions, we mitigate this by replacing (EOT) to remove the attribute information contains from the original prompts, retaining only the semantic information for each subject.
For example, as the cross-attention maps and T2I generation performance shown in Fig. 12, when ToMe not combined with the EST technique, the 'sunglasses' semantics contained in the EOT token cause the boy incorrectly wear sunglasses. However, when combined with ETS, the unwanted semantic binding is relieved.
D.2 Different prompts splice
In Sec. 3,2.1, we fuse each object and its corresponding attributes. At this stage, both the object token embedding and the attribute token embedding are derived from the text embedding obtained by processing the same prompt through the CLIP Text Encoder, potentially causing the information between them to be coupled. We also experimented with splicing token embeddings from different prompts, as illustrated in Fig. 13. While keeping other components of ToMe unchanged, the resulting images often exhibit a missing the object. We hypothesize that this may due to the lack of contextual semantics between token embeddings from different prompts[8].
19

---

SDXL
ToMe(Ours)
SDXL
ToMe(Ours)
SDXL
Tore((uus))
A dog with and a cat
A bay with glasses and girl
buy with hat and Coryl with sunglasses
cat with scarf and dog with tie
bear with hut and
man with glussies
fox with sunglasses and deer with crown
man with hat and girl with necklace
A Liger with glasses and with bat
squirrel holding and bear with hat
white cat with glasses a black dog with hat
cat with pink and dog with blue sunglasses
A Don with vellow crown
sheep with white bandanas
with sunglasses are sitting on a green bench
wearing
man wearing earrings standing the red sandy beach
wearing
A black chocolate cake and fruit plater table
Figure 14: Additional semantic binding results. Our method not only achieves good results object binding but is also effective for composite binding of objects and their adjective attributes.
E
Additional Results
As shown in Tab. have added quantitative comparison results with additional methods. Our method consistently outperforms is on with the existing methods. Fig. 14 presents more qualitative comparison results, demonstrating that our method achieves good performance in attribute binding, object binding, and the composite binding of attribute and objects. ToMe can also generate images with subjects or backgrounds featuring multiple attributes(Fig. 14, the last line), in this scenario, we find that using an additional positional loss[19] 19] based the attention map is effective.
We also conduct user study with 20 participants to enrich the evaluation. Here we compare our method ToMe with SDXL[53], SynGen[58] Ramni[21] and ELLA[30]. As shown in Fig. 15, we ask the participants rate the semantic binding into 4 levels and calculate the distribution of each comparison method over these four diverse levels. We can observe that our method better achieve the semantic binding performance by mainly distribute in the highest level I, while the other methods struggle to obtain user satisfactory results.
20

---

Gubjects netar comnaccniin nouns pocmpt, auch dag penen, discas tels ats aq the cenciptirne frr saddeds belonging another
SDXL
Tuuns(Uus)
SynGen
LE0%
20.5%
30,5%
"lanar provda sooNe Phrrpt: Alboy
3035%
36.5%
33,0%
Level
14.Y%
Lovel
11.8%
Livel firt suteeccss sorneclly expressed withved contusiion
10.5%
32.0%
Lovel
Lecil One suppect cornedly rogromsed but the cteer ixconed.
35.5%%
Level
sutgoot messing
ELLA
Ranni
Figure 15: User study with 20 participants, ask users to rate the semantic binding into four levels.
Table 4: Comparison of BLIP-VQA Scores
BLIP-VQA
Method
Base Model
Train
Color
Texture
Shape
SD v1.5[60]
0.3750
0.4159
0.3724
SD v2[60]
0.5065
0.4922
0.4221
DALL-E2(57]
0.5750
0.6374
0.5464
SDXL[53]
0.6369
0.5637
0.5408
PlayG-v2(37]
0.6208
0.6125
0.5087
Ranni[21]
SD1.5
0.2414
0.3029
0.2857
ELLA
SD1.5
0.6911
0.6308
0.4938
SynGen[58]
SD1.5
0.6619
0.6451
0.4666
CoMat[34]
SD1.5
0.6561
0.6190
0.4975
Composable v2[45]
SD2.0
0.4063
0.3645
0.3299
Structured v2[20]
SD2.0
0.4990
0.4900
0.4218
Attn-Exct v2[7]
SD2.0
0.6400
0.5963
0.4517
GORS[31]
SD2.0
0.6603
0.6287
0.4785
Ranni
SDXL
0.6893
0.6325
0.4934
ELLA[30]
SDXL
0.7260
0.6686
0.5634
SynGen[58]
SDXL
0.7010
0.6044
0.5069
CoMat[34]
SDXL
0.7774
0.6591
0.5262
ToMe (Ours)
SDXL
0.7656
0.6894
0.6051
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification: Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
Supplementary Material
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
oniioiicpyc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececceccccccct /tte:///ggiiieeeeeeeuuubiiiiii ssisnnoillllo onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerrpp.ee/ee//ee cc http/pllcc/thhcsuiiii.eooooo
Answer: [Yes]
Justification:
We have carefully checked the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We politely cited the existing assets and read their usage license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Appendix E
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "cKU3nx8cUb", "reviewer": "Reviewer_jwvK", "summary": "This paper focuses on the problem of lack of semantic binding in text-to-image generation models, and specifically on the misalignment between objects and their sub-objects. The paper introduces a training-free T2I method named ToMe after analyzing the properties of CLIP text embeddings and diffusion models. Utilizing the composition recognition ability of diffusion models, ToMe binds the embeddings of objects with their associated sub-objects together, and retains only the semantic information in the [EOT] tokens. ToMe is shown to be simple yet effective from the experimental results.", "questions": "1. Can the proposed method generalize to more objects, or background with various properties? It would be great if the authors can explain how generating more objects will change the pipeline, or if this is not feasible, please give an explanation why.\n2. Please demonstrate how the two losses are integrated in the T2I pipeline, and how they are used to update the tokens.", "limitations": "The authors presented the inherent limitation of SDXL that ToMe is based on like producing artifacts in generated images, and unable to generate images with complex layouts. Also the limitation of CLIP to produce text embeddings could restrict the performance of ToMe.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The authors present detailed analysis of the CLIP text embeddings including their coupled and entangled properties and the semantically additive properties of text embeddings shown in diffusion models. \n3. The structure of the proposed ToMe model is training-free, simple and mostly clear.\n4. The authors provide extensive experimental results including quantitative, qualitative and ablation studies on the T2I-CompBench and GPT-4o Benchmark (introduced in this paper).", "weaknesses": "1. The Iterative composite Token Update (Sec. 3.2.2) is not clear enough to me. The authors introduce two losses: semantic binding loss and entropy loss. And since ToMe is training-free, it is not clear in the paper how these losses help the update the tokens at test time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n**W1&Q2: Token Update**\n\nBy training-free, we mean that our method ToMe does not involve training over datasets, unlike finetuning-based methods such as Ranni[19], ELLA[28], and CoMat[32]. Like other works (AnE[7], D&B[41], SynGen[56], etc.), methods that only backpropagate during inference time (on the generation of a single image) are considered to be training-free.\n\nToMe distinguishes itself from other optimization-based methods by updating token embeddings, whereas most existing methods (e.g., AnE[7], D&B[41], SynGen[56]) update the latent space using designed loss functions. Instead of back-propagating the gradients to the T2I model parameters or the latent variables of the previous timestep latent $z_t$, we compute the loss and apply the gradients to the composite token embeddings in each inference step during the first $T_{opt} = 0.2T$ time steps.\n\nMore specifically, at time step $ t $, the latent variable $ z_t $ and the text embedding $ \\mathcal{C} $ are fed into the diffusion model. After computing the loss, the gradient is back propagated to update $ \\mathcal{C} $. The latent variable $ z_t $ and the updated text embedding $ \\mathcal{C} $ are then fed into the diffusion model again to predict the noise and obtain $ z_{t-1} $.\n\n**Q1: Complex prompts generation**\n\nAs we have demonstrated in the General Response 2, our method ToMe is not limited to only simple cases. In the main paper Table 2, we show experiments on the T2I-CompBench benchmark, where there are long captions with various complex prompts. There are also complex prompts T2I generations in Fig.12 in the Appendix. \n\nTo further show that our method ToMe can be applied to the T2I generation with complex prompts. We adopt the suggestions from the reviewer. That is to achieve T2I generation with prompts including multiple objects or backgrounds with multiple attributes. The first row of T2I generation results shown in Fig.13 (in the rebuttal file) demonstrate the effectiveness of our method ToMe under complex scenarios, where more objects and background with various properties are demanded.\n\n\n**L1: Generalizability**\n\nAs we discuss in the General Response 4, we agree with you on this point. T2I models are facing generation limitations for various specific cases. We aim, as a future goal, to generalize our method to various generative models to counter this issue. Current T2I generation models are built on various language models and vision-language models, such as CLIP and T5, and incorporate diverse architectures, including UNet, GAN, and DiT. We are also curious to see if the additivity property truly exists across all these cases, and whether this property is a generalizable feature.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iHs5moEgNm", "reviewer": "Reviewer_mo93", "summary": "This paper aims to solve the semantic binding problem in T2I models. The authors introduce a semantic binding method by merging tokens of entities and related attributes. Besides, several other tricks like semantic binding loss and entropy loss are introduced to improve the performance of semantic binding.", "questions": "- 1. From Table 2, we know that entropy loss + token merging bring the biggest impact. This is reasonable because token merging actually changes nothing, all information still exists in the composition token, and semantic leakage will also happen. However, the reviewer does not fully understand what happens when doing token merging + entropy loss. Why do they work so well? Actually, we don't change the information and the way the information interacts.\n    - 1.1. The reviewer thinks Figure 7 is quite clear for the motivation, considering entities and their related attributes as one object. The reviewer is confused about what function entropy loss works.\n\n- 2. What is the impact of substituting the EOT token? Is it important in the whole process?\n\n- 3. How about the efficiency compared to the baseline in Table 1?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is straightforward and it is also reasonable that it will work with some other tricks.\n- The paper is well-written.", "weaknesses": "- The importance of a few tricks like entropy loss is not well-emphasized.\n- The motivation for some tricks like substituting the EOT token is not well-explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n**W1&Q1: Entropy loss**\n\n In Appendix C.3, we demonstrate that the information coupling of token embeddings is also reflected in the entropy of cross-attention for each token. In Fig. 9-(c), we calculated the entropy of cross-attention maps for each token and found that tokens appearing later in the sequence generally have higher entropy, indicating that their cross-attention maps are more dispersed. \n\nBased on this observations, in this paper, we combine the entropy loss with our proposed ToMe approach. Reducing the entropy of these cross-attention maps of composite tokens helps to ensure that these tokens focus exclusively on their designated regions, preventing the cross-attention maps from becoming overly divergent, thus avoiding cross-attention leakage [64]. The experimental results comparing scenarios with and without entropy loss are presented in Table 2, Fig.6, Fig.7 and Fig.14 (in the rebuttal file).\n\nAs an example in Fig.14, the original SDXL (Config A) suffered from attribute binding errors due to divergent cross-attention maps. When only applying token merging (Config B), the co-expression of entities and attributes resulted in a dog wearing a hat in the image, but the attribute leakage issue remained due to the divergent cross-attention maps. When only applying the entropy loss (Config E), although the cross-attention maps corresponding to each token are more concentrated, they may focus on wrong regions. Only by applying both token merging and $\\mathcal{L}_{ent}$ techniques (Config C), the cross-attention map of the composite token becomes better concentrated on the correct areas and thus leading to more satisfactory semantic binding of entities and attributes.\n\n**W2&Q2: EOT ablation study**\n\nThe end token substitution (ETS) technique is proposed to address potential semantic misalignment in the final tokens of long sequences. As the [EOT] token interacts with all tokens, it often encapsulates the entire semantic information, as shown in Fig. 2. Therefore, the semantic information in [EOT] can interfere with attribute expressions, we mitigate this by replacing [EOT] to remove the attribute information it contains from the original prompts, retaining only the semantic information for each subject.\n\nFor example, as the cross-attention maps and T2I generation performance shown in Fig.15 (in the rebuttal file), when ToMe is not combined with the EST technique, the ‘sunglasses’ semantics contained in the EOT token cause the boy to incorrectly wear sunglasses. However, when combined with ETS, the unwanted semantic binding is relieved.\n\n**Q3: Efficiency**\n\nAs detailed in the General Response 3, we present a time cost comparison in Table 3 in Appendix C.4, where we compare performance using 50 inference steps using the SDXL model with float32 version on an A40 GPU. We demonstrate that our method does not significantly increase inference time while improving semantic binding performance. We further extend this analysis by measuring the time cost with 20 inference steps and various ToMe configurations, as shown in the Table below. We report the time cost (by seconds) along with BLIP-VQA scores across the color, texture, and shape attribute binding subsets. From this table, we can observe that using the token merging (ToMe) technique and entropy loss (Config.C in Table 2), our method achieves excellent performance with minimal additional time cost. Additionally, even with only 20 inference steps, our method, ToMe, maintains high performance with very little degradation.\n\n| method | inference steps | Time Cost | Color | Texture | Shape |\n| :--------: | :--------: |:--------: |:--------: |:--------: |:--------: |\nSDXL                    |  20        |  18s  |  0.6136 |  0.5449  |  0.5260 | \n*ToMe (Config C)*  |   20    |   23s   |   *0.7419*  |  *0.6581*  |   *0.5742* | \n**ToMe (Ours)**         |   20    |   45s   |   **0.7612**  |   **0.6653**   |  **0.5974**  | \nRanni (SDXL)     |  50    |  87s    |  0.6893   |  0.6325   |  0.4934 | \nELLA (SDXL)     |  50    |  51s    |  0.7260   |  0.6686   |  0.5634 | \nSynGen (SDXL)   |  50    |  67s    |  0.7010   |  0.6044   |  0.5069 | \nSDXL                    |  50    |  42s    |  0.6369   |  0.5637   |  0.5408 | \n*ToMe (Config C)* |   50    |   56s   |   *0.7525*  |  *0.6775*  |   *0.5797* | \n**ToMe (Ours)**         |   50    |   83s   |   **0.7656**  |   **0.6894**   |   **0.6051**  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFVpJlhZRG", "reviewer": "Reviewer_85Q2", "summary": "The authors propose a method to mitigate semantic binding, a common phenomenon in text-to-image models. While previous methods explicitly control the attention maps so that nouns and attributes attend to the same regions, the authors propose combining the nouns and attributes into a single token. This approach enforces the attention maps to concentrate in the same area. Additionally, they introduce an inference-time optimization loss to ensure the attention maps are focused, and that each composed token retains the same semantic meaning as all the different tokens that comprise it. The authors also analyze the information encoded in different tokens at the output of the text encoder.", "questions": "1. How does the method perform on more complex prompts? For example, for a prompt like “A blue cat wearing sunglasses and a yellow dog wearing a pink hat”. From my understanding, the method should not succeed in separating “pink hat” and “yellow dog” as the whole expression will be combined into a single token.\n2. Is it not clear to me what prevents semantic binding across nouns, e.g. both the dog* and the cat* tokens to have the same appearance, (e.g. a cat-dog hybrid), which is a common problem in text-to-image models. What prevents the dog* and cat* attention maps from attending to the same regions?", "limitations": "The authors propose limitations relating to the underlying models, but I would be interested to learn in which areas of semantic binding the paper still struggles, such as with more complex prompts or a larger number of nouns, etc..", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors present an interesting analysis of the information encoded tokens, specifically they show that much of the semantic information is also encoded in the EOT tokens.\n2. The authors provide a method to mitigate semantic binding in prompts of the form: “A <noun_A> <Attribute_A> and <noun_B> <Attribute_B>”, that outperforms or performs comparably to existing methods across various benchmarking subsets.", "weaknesses": "1. It is not clear how the model performs on more complicated prompts where both the noun and its sub-objects have attributes, such as: “A blue cat wearing sunglasses and a yellow dog wearing a pink hat.”\n2. The method is on the slower side compared to other existing methods, due to inference time optimization.\n3. This work is missing a user study, as the proposed automatic metrics can be inaccurate, and human evaluation is common in previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n**W1&Q1&L1: Complex prompts generation**\n\nAs demonstrate in General Response 2, our method ToMe is not limited to only simple cases. In the main paper Table 2, we show experiments on the T2I-CompBench benchmark, where there are long captions with various prompt templates. There are also complex prompts T2I generations in Fig.12 in the Appendix.  To further show that our method ToMe can be applied to the T2I generation with complex prompts. We adopt the complicated long caption templates suggested from the reviewer. The generation results are shown in Fig.13 in the rebuttal file.\n\nIn this case, we use our method ToMe in an iterative updating way. As an instance, we generate the fourth image example in Fig.13 (left down) with the prompt “a blue cat wearing sunglasses and a yellow dog wearing a pink hat”. To apply ToMe, we merge ‘yellow dog’ into token1 (dog*) and ‘pink hat’ into token2 (hat*). Following that, we merge token1 and token2 into token3. The noun phrases for computing the semantic binding loss corresponding to these three tokens are ‘a yellow dog’, ‘a pink hat’ and ‘a dog wearing a hat’, respectively. Then we update token1 and token2 by our losses proposed in Section 3.2.2.\n\n**W2: Inference time cost**\n\nAs detailed in the General Response 3, we present a time cost comparison in Table 3 in Appendix C.4, where we compare performance using 50 inference steps using the SDXL model with float32 version on an A40 GPU.  We demonstrate that our method does not significantly increase inference time while improving semantic binding performance. We further extend this analysis by measuring the time cost with 20 inference steps and various ToMe configurations, as shown in the Table below. We report the time cost (by seconds) along with BLIP-VQA scores across the color, texture, and shape attribute binding subsets. From this table, we can observe that using the token merging (ToMe) technique and entropy loss (Config.C in Table 2), our method achieves excellent performance with minimal additional time cost. Additionally, even with only 20 inference steps, our method, ToMe, maintains high performance with very little degradation.\n\n| method | inference steps | Time Cost | Color | Texture | Shape |\n| :--------: | :--------: |:--------: |:--------: |:--------: |:--------: |\nSDXL                    |  20        |  18s  |  0.6136 |  0.5449  |  0.5260 | \n*ToMe (Config C)*  |   20    |   23s   |   *0.7419*  |  *0.6581*  |   *0.5742* | \n**ToMe (Ours)**         |   20    |   45s   |   **0.7612**  |   **0.6653**   |  **0.5974**  | \nRanni (SDXL)     |  50    |  87s    |  0.6893   |  0.6325   |  0.4934 | \nELLA (SDXL)     |  50    |  51s    |  0.7260   |  0.6686   |  0.5634 | \nSynGen (SDXL)   |  50    |  67s    |  0.7010   |  0.6044   |  0.5069 | \nSDXL                    |  50    |  42s    |  0.6369   |  0.5637   |  0.5408 | \n*ToMe (Config C)* |   50    |   56s   |   *0.7525*  |  *0.6775*  |   *0.5797* | \n**ToMe (Ours)**         |   50    |   83s   |   **0.7656**  |   **0.6894**   |   **0.6051**  | \n\n\n**W3: User study**\n\nPlease also refer to the General Response 1. In this paper, we use the ImageReward[72] model to evaluate human preference scores, which comprehensively measures image quality and prompt alignment. However, since quantitative results may not fully capture these aspects, we now also conduct a user study with 20 participants to enrich the evaluation. Here we compare our method ToMe with SDXL[51], SynGen[56], Ranni[19] and ELLA[28]. Each participant is questioned with 100 images, each generated from these five methods and twenty prompts.\n\nThe results are presented in Fig.16 of the rebuttal file. Similar to our proposed GPT-4o benchmark (detailed in Appendix C.5 and Fig.10), we ask the participants to rate the semantic binding into 4 levels and calculate the distribution of each comparison method over these four diverse levels. We can observe that our method better achieve the semantic binding performance by mainly distribute in the highest level 1, while the other methods struggle to obtain user satisfactory results.\n\n\n**Q2: Mechanism to avoid semantic misalignment**\n\nIn this paper, we propose two losses to avoid the semantic misalignment and cross-attention leakage among objects and attributes.\n\nFirst, the semantic binding loss assist in purifying the token semantics. Building on the ToMe technique, we decrease the semantic binding loss to enhance the cross-attention. It encourages the newly learned token to infer the same noise prediction as the original corresponding phrase, reinforcing the semantic coherence between the text and the generated image. To ensure that the semantics of the composite tokens accurately correspond to the noun phrases they represent, we use a clean prompt as a supervisory signal. The semantic binding loss is computed separately, ensuring no interference among noun words. \n\nIn addition, we reduce the entropy loss to help ensure that tokens focus exclusively on their designated regions, preventing the cross-attention maps from becoming overly divergent. The experimental results comparing scenarios with and without entropy loss are presented in Table 2, Fig. 6, and Fig. 7. Both losses are functioning together to improve the semantic binding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3C4f3nVw0b", "reviewer": "Reviewer_8gpY", "summary": "The authors tackle semantic binding, where T2I models often fail to correctly reflect the relations between objects (object binding) or objects and their attributes (attribute binding). To address this, they introduce Token Merging (ToMe), a method that aggregates related tokens to a single composite token, which ensures they share the same cross-attention map. To do that, two training-free optimizations are introduced. The first, Semantic Binding loss, which makes sure the composite token leads to noise prediction that is consistent with the full phrase the token is based on. The second, Entropy loss, which helps the tokens focus exclusively on their designated regions.", "questions": "1. What input text is used in the global encoding part?\n2. Do you expect your method to extend to SD-3?\n3. I worry that ToMe suffers from too many gradient updates in the event of highly complex prompts (like all training-free approaches), which would push the latent out of distribution. \n4. Does ToMe work on long captions, with multiple sentences? How?\n5. You say in the appendix that SynGen was modified to work with SDXL. Can you elaborate on the modifications? What would be needed to reproduce this experiment too?", "limitations": "See questions 3 and 4.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Creating a composite token and then optimizing it is a creative and elegant approach to the problem. I'm curious to see what future work can be built on top of this contribution.\n* The focus on object binding is indeed missing from the literature, a topic that is well-addressed in this work.\n* The idea of end token substitution to remove attribute information is simple.", "weaknesses": "* The related work underplays the role of Attend-and-Excite and Linguistic Binding in Diffusion Models (SynGen) have on this paper. The former introduced the idea of semantic guidance, while the latter used dependency graphs to extract syntactically related tokens from prompts and presented a loss that encourages cross-attention maps of syntactically related tokens to agree. Both of which are ideas this paper meaningfully builds on. \n\n* You do not describe how you obtain the syntactic information: entities and their relevant objects nor if it is automatic or an input constraint. This is particularly confusing because in section 3.2 you begin describing exactly that, but do not delve into the specifics. Furthermore, in section 4.1 (line 248), you mention spaCy and that you \"identify each object and its corresponding attributes for token merging\", but do not provide actual details about your method. Does it mean it can accurately capture *all* syntactic structures? It is possible that I misunderstand, but it feels like the syntactic analysis portion of this work borrows from the SynGen paper. If this is the case, give appropriate credit. If it is not, then add details, as this is part of the method and is quite unclear.\n\n* If I understand correctly, there are no human evaluation experiments, a user study would provide merit to ToMe’s superior performance. As a side note, in table 1, ‘Human-preference’ is a slightly misleading title. If I understand correctly, it is the output of the Image Reward human-preference model, but it sounds as if it is a human-evaluation result.\n\n* I find that despite the claim that ToMe works well on highly complex prompts, there are no such examples or experiments given. For example, Figure 5 depicts prompts that are supposed to be complex, but they are no different from the simplistic prompts in Attend-and-Excite (“a {color_1} {subject_1} and {a color_2} {subject_2}”), only that they address object binding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and will incorporate the discussions mentioned below to enhance the quality of our paper. Note that we utilize the numerical references to cite sources within the main paper.\n\n**W1: Related work**\n\nThese two methods generally fall under optimization-based approaches. They primarily adjust noisy signals to enhance attention maps and strengthen semantic binding. Attend-and-Excite improves object presence by increasing the attention score for each object. SynGen performs a syntactic analysis of the prompt to identify entities and their modifiers, using attention loss functions to ensure that cross-attention maps align with the linguistic binding reflected in the syntax. These methods have indeed provided a solid foundation for future semantic binding work. Different from these two works, our training-free method ToMe is based on the text embedding additivity property. Instead of updating the latent space, we update the composite token embeddings. We further introduce two auxiliary losses, i.e. the entropy loss and semantic binding loss, to augment the ToMe performance.\nWe will include more detailed introductions to these two methods in the introduction and related work sections in any future version.\n\n**W2: Syntactic information**\n\nIn Section 4.1 on the implementation details, we mention that we use SpaCy[27] to detect relationships, following the approach outlined in the SynGen. More specifically, we use SpaCy’s transformer-based dependency parser to analyze the prompt and identify all entity nouns, including both proper nouns and common nouns, that are not serving as direct modifiers of other nouns. This process helps pinpoint entity nouns and their corresponding modifiers. We will further detail this process in the method and implementation details section.\n\n**W3: User study**\n\nAs discussed in the General Response 1. In this paper, we use the ImageReward[72] model to evaluate human preference scores, which measures image quality and prompt alignment. However, since quantitative results may not fully capture these aspects, we now conduct a user study with 20 participants to enrich the evaluation. Here we compare our method ToMe with SDXL, SynGen, Ranni and ELLA. Each participant is questioned with 100 images, each generated from five methods and twenty prompts.  The results are presented in Fig.16 of the rebuttal file. In the user study, we ask the participants to rate the semantic binding into 4 levels. We then calculate the distribution of each method over the four diverse levels. We can observe that ToMe better achieve the semantic binding performance by mainly distribute in the highest level 1, while the other methods struggle to obtain satisfactory results.\n\n**W4&Q4: Complex prompts generation**\n\nAs illustrated in General Response 2, ToMe is not limited to only simple cases. In the main paper Table 2, we show experiments on the T2I-CompBench benchmark, where there are long captions with various prompt templates. There are also complex prompts T2I generations in Fig.12. To further show that our method ToMe can be applied to the T2I generation with complex prompts. We adopt the suggestions from the reviewer to generate with more complicated prompt templates, that includes longer prompts with multiple objects, multiple attributes or even multiple sentences. The generation results are shown in Fig.13 in the rebuttal file.\n\n\n**Q1: Global encoding text**\n\nSDXL is using a conditioning concatenation for text embeddings with two prompts. To keep our method ToMe applicable and generalizable to other T2I generative models, we keep both prompts the same. For example, with prompt “A blue cat and a green dog”, we assign this prompt as both the local and global encoding prompt and input it to the two text encoders and concatenate the text embeddings along the channels. And in our method, ToMe is updating the concatenated text embedding conditions with our proposed losses. We will add this to the implementation details. \n\n**Q2: Generalizability to other T2I models**\n\nAs claimed in General Response 4, T2I models are facing generation limitations for various specific cases. We aim, as a future goal, to generalize our method to various generative models (such as DeepFloyd-IF, SD3, PixArt, etc.) to counter this issue. Current T2I generation models are built on various language models and vision-language models, such as CLIP and T5, and incorporate diverse architectures, including UNet, GAN, and DiT. We are also curious to see if the additivity property truly exists across all these cases, and whether this property is a generalizable feature.\n\n\n**Q3: Latent Distribution Drifts**\n\nYou are correct; large gradients can cause significant changes in the latent space, which is a common issue in training-free semantic binding methods[7,41,56]. The primary goal of these methods is to update the latent or text embeddings to align the semantics between texts and images, often resulting in varying degrees of distribution change. To mitigate this issue, we only update the composite token representations during the first $T_{\\text{opt}} = 0.2T$ time steps. This technique helps our method ToMe experience fewer distribution changes from the base model SDXL, as demonstrated in multiple cases in Fig. 5, Fig. 12 and Fig.13.\n\n**Q5: SynGen reproduction details**\n\nThe original SynGen paper is based on the SD1.5 model. To adapt it to use SDXL as the base model, we utilize the 32×32 cross-attention map from the first three layers of the UNet-decoder, which serves as the most semantic layer, similar to the 16×16 cross-attention in SD1.5. We also performed a grid search to determine the optimal learning rate for SynGen in this context. All other details remain consistent with the original paper. After tuning the hyperparameters, the SDXL-based SynGen outperforms the SD1.5-based SynGen, as shown in Table 1 and Table 4. We will release the SynGen adapted code in the future for reproduction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers (**R1**=**jwvK**, **R2**=**mo93**, **R3**=**85Q2**, **R4**=**8gpY**) for their positive feedbacks. They note that this paper is well-written (**R1,R2**); the idea is simple and straightforward (**R1, R2, R4**); that we present interesting analysis over the token properties (**R1, R3**); that we provide extensive experiments over various benchmarks (**R1, R3**); and we well address the object binding in this work (**R4**). Below we respond to general questions raised by reviewers.  We use **W** to abbreviate Weaknesses, **Q** to represent Questions and **L** for Limitations. Note that we utilize the numerical references to cite sources within the main paper.


**General Response 1: User study (R3-W3, R4-W3)**

In this paper, we use the ImageReward [72] model to evaluate human preference scores, which comprehensively measures image quality and prompt alignment. However, since quantitative results may not fully capture these aspects, we now also conduct a user study with 20 participants to enrich the evaluation. Here we compare our method ToMe with SDXL[51], SynGen[56], Ranni[19] and ELLA[28]. Each participant is questioned with 100 images, each generated from these five methods and twenty prompts.

The results are presented in Fig.16 of the rebuttal file. Similar to our proposed GPT-4o benchmark (detailed in Appendix C.5 and Fig.10), we ask the participants (instead of GPT-4o model as we used) to rate the semantic binding into 4 levels and calculate the distribution of each comparison method over these four diverse levels. We can observe that our method better achieve the semantic binding performance by mainly distribute in the highest level 1, while the other methods struggle to obtain user satisfactory results.

**General Response 2: Complex prompts generation (R1-Q1, R3-W1&Q1&L1, R4-W4&Q4)**

Actually, our method ToMe is not limited to only simple cases. In the main paper Table 2, we show experiments on the T2I-CompBench benchmark, where there are long captions with various prompt templates. There are also complex prompts T2I generations in Fig.12 in the Appendix. 

To further show that our method ToMe can be applied to the T2I generation with complex prompts. We adopt the suggestions from the reviews. That includes *(1) R1 (jwvK) and R3 (85Q2)*: multiple objects or backgrounds with multiple attributes; *(2) R4 (8gpY)*: long captions with multiple sentences. The generation results are shown in Fig.13 in the rebuttal file.

For the second case, we apply the ToMe method directly, as shown in the last example of Fig.13.
For the first case, we use our method ToMe in an iterative updating way. As an instance, we generate the fourth image example in Fig.13 (left down) with the prompt “a blue cat wearing sunglasses and a yellow dog wearing a pink hat”. To apply ToMe, we merge ‘yellow dog’ into token1 (dog*) and ‘pink hat’ into token2 (hat*). Following that, we merge token1 and token2 into token3. The noun phrases for computing the semantic binding loss corresponding to these three tokens are ‘a yellow dog’, ‘a pink hat’ and ‘a dog wearing a hat’, respectively. Then we update token1 and token2 by our losses proposed in Section 3.2.2.

**General Response 3: Time cost (R2-Q3, R3-W2)**

In Appendix C.4, we present a time cost comparison in Table 3, where we compare performance using 50 inference steps using the SDXL model with float32 version on an A40 GPU.  We demonstrate that our method does not significantly increase inference time while improving semantic binding performance. We further extend this analysis by measuring the time cost with 20 inference steps and various ToMe configurations, as shown in the Table below. We report the time cost (by seconds) along with BLIP-VQA scores across the color, texture, and shape attribute binding subsets. From this table, we can observe that using the token merging (ToMe) technique and entropy loss (Config.C in Table 2), our method achieves excellent performance with minimal additional time cost. Additionally, even with only 20 inference steps, our method, ToMe, maintains high performance with very little degradation.

| method | inference steps | Time Cost | Color | Texture | Shape |
| :--------: | :--------: |:--------: |:--------: |:--------: |:--------: |
SDXL                    |  20        |  18s  |  0.6136 |  0.5449  |  0.5260 | 
*ToMe (Config C)*  |   20    |   23s   |   *0.7419*  |  *0.6581*  |   *0.5742* | 
**ToMe (Ours)**         |   20    |   45s   |   **0.7612**  |   **0.6653**   |  **0.5974**  | 
Ranni (SDXL)     |  50    |  87s    |  0.6893   |  0.6325   |  0.4934 | 
ELLA (SDXL)     |  50    |  51s    |  0.7260   |  0.6686   |  0.5634 | 
SynGen (SDXL)   |  50    |  67s    |  0.7010   |  0.6044   |  0.5069 | 
SDXL                    |  50    |  42s    |  0.6369   |  0.5637   |  0.5408 | 
*ToMe (Config C)* |   50    |   56s   |   *0.7525*  |  *0.6775*  |   *0.5797* | 
**ToMe (Ours)**         |   50    |   83s   |   **0.7656**  |   **0.6894**   |   **0.6051**  | 

**General Response 4: Generalizability to other T2I models (R1-L1, R4-Q2)**

We agree with you on this point. T2I models are facing generation limitations for various specific cases. We aim, as a future goal, to generalize our method to various generative models to counter this issue. Current T2I generation models are built on various language models and vision-language models, such as CLIP and T5, and incorporate diverse architectures, including UNet, GAN, and DiT. We are also curious to see if the additivity property truly exists across all these cases, and whether this property is a generalizable feature.

Meta-review: This paper introduces a novel approach on semantic binding during T2I, this is to associate a given object with its attribute or with other related objects. The approach uses token merging to aggregate relevant tokens into a single token. All reviewers feel positive about the paper. Reviewers think the idea is straightforward and effective, paper is well written with extensive experiments comparing benchmarks. There are a few interesting questions raised by reviewers on complex prompt and importance of entropy loss, authors have provided answers to clarify. The AC agree with all the reviewers and recommend to accept this paper. The AC encourages the authors to add these clarification to the final paper.
Decision: Accept (poster)