Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis

Authors: taihang  hu
linxuan  li
joost  van de weijer
hongcheng  gao
fahad  shahbaz khan
jian  yang
ming-ming  cheng
kai  wang
yaxing  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 05:58:24

Abstract: Although text-to-image (T2I) models exhibit remarkable generation capabilities,
they frequently fail to accurately bind semantically related objects or attributes
in the input prompts; a challenge termed semantic binding. Previous approaches
either involve intensive fine-tuning of the entire T2I model or require users or
large language models to specify generation layouts, adding complexity. In this
paper, we define semantic binding as the task of associating a given object with its
attribute, termed attribute binding, or linking it to other related sub-objects, referred
to as object binding. We introduce a novel method called Token Merging (ToMe),
which enhances semantic binding by aggregating relevant tokens into a single
composite token. This ensures that the object, its attributes and sub-objects all share
the same cross-attention map. Additionally, to address potential confusion among
main objects with complex textual prompts, we propose end token substitution as
a complementary strategy. To further refine our approach in the initial stages of
T2I generation, where layouts are determined, we incorporate two auxiliary losses,
an entropy loss and a semantic binding loss, to iteratively update the composite
token to improve the generation integrity. We conducted extensive experiments to
validate the effectiveness of ToMe, comparing it against various existing methods
on the T2I-CompBench and our proposed GPT-4o object binding benchmark. Our
method is particularly effective in complex scenarios that involve multiple objects
and attributes, which previous methods often fail to address. The code will be
 publicly available at https://github.com/hutaihang/ToMe

Main paper: 
 introduction: 
 Text-to-image generation has seen significant advancements with the recent introduction of diffusion models [57, 59, 62] , with their capabilities of generating high-fidelity images from text prompts. Despite these achievements, aligning the generated images with the text prompts, which is referred to as semantic alignment [30, 43] , remains a notable challenge. One of the most common issues observed in existing text-to-image (T2I) generation models is the lack of proper semantic binding, where a given object is not properly binding to its attributes or related objects. For example, as illustrated in Fig. 1 , even a state-of-the-art T2I model such as SDXL [53] can struggle to generate content that accurately reflects the intended nuances of text prompts. To address the persistent challenges of aligning T2I diffusion models with the intricate semantics of text prompts, a variety of enhancement Figure 1 : Current state-of-the-art T2I models often struggle with semantic binding in generated images according to textual prompts. For example, hats and sunglasses are placed on incorrect objects. We introduce a novel method ToMe to address these challenges.strategies [35, 46, 87] are proposed, either by optimizing the latent representations [69, 82, 83] , guiding the generation by layout priors [54, 71, 85] or fine-tuning the T2I models [21, 34] . Despite these advancements, these methods still encounter limitations, particularly in generating high-fidelity images involving complex scenarios where an object is binding with multiple objects or attributes.In this paper, we categorize semantic binding into two categories. First, attribute binding involves correctly associating objects with their attributes, a topic that has been studied in prior work [58] . Second, object binding, which entails effectively linking objects to their related sub-objects (for example, a 'hat' and 'glasses'), is less explored in the existing literature. Previous methods often struggled to address this aspect of semantic binding. One of the main problems is the misalignment of objects with their corresponding sub-objects. Existing solutions address this through an explicit alignment process of the attention maps [7, 43] or by factorizing the generation projects into layout phases and generation phase [55] . In this paper, we propose a simple solution to the attention alignment problem called token merging (ToMe). Instead of multiple attention maps, which can be misaligned, we join these objects in a single composite token that represents the object and its attributes and sub-objects. This composite token has a single cross-attention map that ensures semantic alignment. The composite token is simply constructed by summing the CLIP text embeddings of the various tokens it represents. For example, the phrase "a dog with hat" is abbreviated as "a dog*" by aggregating the text embeddings corresponding to the last three words, as shown in Fig. 4 . To justify the applied embedding addition in ToMe, we experimented with the semantic additivity of the text embeddings (in Fig. 3 ). Furthermore, to mitigate potential semantic misalignment in the end tokens from the long sequences, we propose end token substitution (ETS) technique.As the T2I generation predominantly determines the layout during earlier phases [27] , we introduce an entropy loss and a semantic binding loss to update the token embeddings in early steps, integrating ToMe with an iterative update for the composite tokens. The entropy loss is defined as the entropy of the cross-attention map corresponding to the updated composite token. This loss aims to enhance generation integrity by ensuring diverse attention across relevant areas of the image, thereby preventing focusing on non-essential regions. The semantic binding loss encourages the new learned token to infer the same noise prediction as the original corresponding phrase. This alignment further reinforces the semantic coherence between the text and the generated image.Our final method ToMe is quantitatively assessed using the widely adopted T2I-CompBench [31] and our proposed GPT-4o [1] object binding benchmark. Comparative evaluations against various types of approaches reveal that ToMe outperforms them by a significant margin. Remarkably, our approach is user-friendly, requiring no dependence on large language models or specific layout information. In qualitative evaluations, we notably achieve superior generation quality, particularly in scenarios involving multi-object multi-attribute generation. This further underscores the superiority of our method. In summary, the main contributions of this paper are as follows:• We analyze the problem of semantic binding, and highlight the role of the [EOT] token (Fig. 2 ), and the problems with misaligned cross-attention maps (Fig. 7 ). In addition, we explore token additivity as a possible solution (Fig. 3 ).• We introduce a training-free approach called Token Merging (Fig. 4 ), denoted as ToMe, as a more efficient and robust solution for semantic binding. It is further enhanced by our proposed end token substitution and iterative composite token updates techniques.• In experiments conducted on the widely used T2I-CompBench benchmark and our GPT-4o object binding benchmark, we compared ToMe with various state-of-the-art approaches and consistently outperformed them by significant margins. 
related works: 
 A critical drawback of current text-to-image models is related to their limited ability to faithfully represent the precise semantics of input prompts, commonly referred to as semantic alignment.Various studies have identified common semantic failures and proposed mitigation strategies. They can be roughly categorized into four main streams.Optimization-based methods primarily adjust text embeddings [20, 65] or optimize noisy signals to strengthen attention maps [26, 48, 63, 69, 82, 83] . These methods are basically inspired by the observations from text-based image editing methods [27, 40, 64, 66] , suggesting that the layouts of objects are determined by self-attention and cross-attention maps from the UNet of the T2I diffusion models. For example, Attend-and-Excite [7] improves object existence by exciting the attention score of each object. Divide-and-Bind [43] improves by maximizing the total variation of the attention map to prompt multiple spatially distinct attention excitations. SynGen [58] syntactically analyzes the prompt to identify entities and their modifiers, and then uses attention loss functions to encourage the cross-attention maps to agree with the linguistic binding reflected in the syntax. A-star [2] proposes to minimize concept overlap and change in attention maps through iterations. Composable Diffusion [45] decomposes complex texts into simpler segments and then composes the image from these segments. Structure Diffusion [20] attempts to address this by leveraging linguistic structures to guide the cross-attention maps. Rich-Text [24] enriches textual prompts by incorporating various formatting controls and decomposes the generation task into merging inferences from multiple regionbased diffusions. However, these methods often fail in complex scenarios that generate multiple objects or multiple attributes.Layout-to-Image methods [4, 9, 14, 17, 25, 32, 36, 47] are widely using layouts, particularly in the form of bounding boxes or segmentation maps, as a popular intermediary to bridge the gap between text input and the generated images. For example, BoxDiff [73] encourages the desired objects to appear in the specified region by calculating losses based on the maximum values in cross-attention maps. Similarly, Attention-Refocusing [52] modifies both cross-attention and self-attention maps to control object positions. BoxNet [67] first trains a network to predict the box for each entity that possesses the attribute specified in the prompt, and then force the generation to follow the attention mask control. Additionally, InstanceDiffusion [68] enhances text-to-image models by providing extra instance-level control. There are also finetuning methods [5, 42, 50, 79] allow for additional layout conditions after fine-tuning over pair images, which are not specifically designed to solve the semantic alignment problem. Despite their promise, these methods obviously prolong the training time. Furthermore, the application of layout priors is challenging when it comes to global background descriptions or abstract elements. This limitation constrains the versatility of these techniques, making it difficult to deploy them effectively across real scenarios where non-specific spatial arrangements are crucial.LLM-augmented methods are mainly following text-to-layout-to-image generation pipelines [15, 23, 33, 44, 55, 65, 80, 81, 86] , first to generate layouts from large language models (LLMs) and force the T2I generations to follow this guidance as the previous layout-guided methods. Some methods, such as RPG [75] and MuLan [39] , harness the powerful chain of thought reasoning ability of multimodal LLMs to enhance the compositionality of text-to-image diffusion models.Finetuning-based methods [13, 76] update the model parameters over huge datasets to augment the semantic alignment. Among them, CoMat [34] proposes an end-to-end fine-tuning strategy for text-to-image diffusion models by incorporating image-to-text concept matching. ELLA [30] equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment by bridging these two pre-trained models with trainable semantic alignment connectors. More recently, Ranni [21] improves T2I generation by bridging the text and image with a semantic panel with LLMs and is fine-tuned over an automatically prepared semantic panel dataset. There are also improved T2I models [10, 11, 51] learning from scratch over huge datasets. These methods improve semantic alignment implicitly by better architecture design and larger amount of training data. They further demand marvelous computational resources to achieve the purpose.In this paper, we tackle the semantic binding problem, which is a broad subcase of semantic alignment, in a training-free manner, neither needing the LLMs nor any training over additional datasets. Furthermore, we achieve better performance when facing complex T2I generation scenarios where users require multiple objects or multiple attributes related to a specific object. 
methods: 
 Semantic binding in T2I generation refers to the crucial requirement of establishing accurate associations between objects and their relevant attributes or related sub-objects. This process avoids semantic misalignment in the generated images, ensuring that each visual element aligns correctly with its descriptive cues in the text. In this section, we begin by providing the preliminaries. Subsequently, we illustrate the motivation through a series of experimental analyses (Sec. 3.1). Finally, we elaborate on our methods in detail (Sec. 3.2). An illustration of our method ToMe is shown in Fig. 4 .Latent Diffusion Models. We build our novel approach for semantic alignment on the standard SDXL [53] model. The model is composed of two main parts: an autoencoder (i.e., a encoder E and a decoder D ) and a diffusion model (i.e., ϵ θ with parameter θ). The model ϵ θ is updated by the loss:EQUATIONwhere ϵ θ is a UNet, conditioning a latent input z t , a text embedding τ ξ (P) and a timestep t ∼ Uniform(1, T ). More specifically, text-guided diffusion models aim to generate an image from random noise z T and a conditional input prompt P. To distinguish from the general conditions in LDMs, we itemize the textual condition as C = τ ξ (P), where τ ξ is the CLIP text encoder [56] † . The cross-attention map is obtained from ϵ θ (z t , t, C). Let f zt be a feature map output of the network ϵ θ . We get a query matrix Q t = l Q (f zt ) with projection network l Q . Similarly, given a textual embedding C, we compute a key matrix K = l K (C) with projection network l K . Then the attention map is computed according to:A t = sof tmax(Q t • K T / √ d)where d is the latent dimension, and the cell [A t ] ij defines the weight of the j-th token on the i-th token.To address the semantic binding problem, we concentrate on the text embeddings utilized during the diffusion model generation process, as they predominantly dictate the content of the generated images. For a given text prompt P, it is tokenized by the CLIP text model by padding a start token [SOT] and several end tokens [EOT] to extend its length to M (=77 by default). After the CLIP text encoder τ ξ , the condition is formulated as C = τ ξ (P). Each row in C represents a corresponding token embedding after the CLIP text transformers. For example, the text embedding for the sentence P ="a cat wearing sunglasses and a dog wearing a hat" is represented as:C = [c SOT 0 , c a 1 , c cat 2 , • • • , c dog 7 , c wearing 8 , c hat 9 , c EOT 10 , • • • , c EOT M -1 ].In the following analysis, we take this as a default example (except when defined differently).Information Coupling. We begin by generating images conditioning on the textual embedding C, as illustrated in the first two columns at the bottom of Fig. 2-(a) . We observe that the attributes appear in a misalignment between the dog and the cat. Subsequently, we extract the token embedding c dog 7 from the textual embedding and input it to the UNet ϵ θ (i.e., C = [c dog 7 ]) ‡ . As depicted in the middle columns of Fig. 2-(a) . The dog object is frequently wearing glasses, further highlighting the semantic leakage issue. Furthermore, when we takeC [EOT ] = [c EOT 10 , • • • , c EOT M -1 ]as input, the generated images closely resemble all information obtained using the entire textual embedding C. As the [EOT] interacts with all tokens, it often encapsulates the entire semantic information [41, 72] .We further report the DetScore [12] to show the probability of containing the corresponding object ("sunglasses") in the generated 100 images. As illustrated in Fig. 2-(b) , for these three different cases, the DetScore is 22.6%, 69.6% and 75.0%, respectively. These findings also align with our observations above.Additivity Property. Inspired by the semantic additivity of the text embeddings in previous research [6, 49] , we experiment the additive property of the CLIP textual embedding. We represent the textual embedding corresponding to the prompt "a photo of a dog" asC 1 = [c SOT 0 , c a 1 , • • • , c dog 5 , c EOT 6 , • • • , c SOT M -1 ].The textual embedding for the prompt "a photo of a hat" is represented asC 2 = [c SOT 0 , c a 1 , • • • , c hat 5 , c EOT 6 , • • • , c EOT M -1 ]. Next, we perform element-wise addition between the object tokens (i.e., c dog 5 and c hat 5 ) and the corresponding [EOT] tokens. Specifically, the resulting new embedding isC ′ = Concat (C 1 [0 : 4], C 1 [5 : M -1] + C 2 [5 : M -1]). Afterward, the textual embeddings C ′ are input into the diffusion UNet to generate the images shown in Fig. 3-(a) . We can observe that this additivity property allows adding objects (up-left), removing objects (upright, down-left) and even complex semantic computations (down-right). To explore the mechanism behind this phenomenon, we conducted PCA dimensionality reduction visualization on the token representations of each prompt, as illustrated in Fig. 3-(b ). The directional vector obtained from "queen-king" is approximately identical to that of "woman-man" with the cosine similarity of 0.998.In conclusion, our analysis shows that the semantic content of text tokens is coupled and entangled, resulting in attribute confusion across different subjects. Moreover, we found that in diffusion models, text embeddings exhibit semantically additive properties. This implies that the diffusion model is capable of interpreting a composite token, derived from the summation of multiple individual tokens, integrating the semantic attributes of the combined tokens. "a cat and a dog" [SOT] [EOT]Suppose the initial prompt P contains K entities indicated by noun words and their corresponding tokens as {n 1 , ..., n k ..., n K }. Each entity is often related to a token with relevant objects or attributes set as (n k , a k ). For example, in the sentence "a cat wearing glasses and a dog with a hat", n 1 = <cat>, a 1 = {<wearing>,<glasses>}, n 2 = <dog>, a 2 = {<with>,<a>,<hat>}.The semantic additivity of token embeddings inspires us to achieve co-expression of entities and attributes by explicitly binding tokens together. We employ element-wise addition to accomplish semantic merging of tokens. For a prompt P containing K entities, we fuse each subject-attribute pair (n k , a k ) into ĉk = n k + a k , referred to as a composite token. This innovative approach introduces an additional benefit by utilizing a single composite token to condense a lengthy prompt sequence, resulting in a unified cross-attention map, thus avoid semantic misalignment. Such observations are further shown in the ablation study and appendix.End Token Substitution (ETS). Meanwhile, as the semantic information contained in [EOT] can interfere with attribute expression, we mitigate this interference by replacing [EOT] to eliminate attribute information contained within them, retaining only the semantic information of each subject. For instance, when the prompt is "a cat wearing hat and a dog wearing sunglasses," we use the [EOT] obtained from the prompt "a cat and a dog" to replace the original [EOT] . As illustrated in Fig. 4 -a, the final text embedding after subject-attribute enhancement and EOT replacement isC = c SOT 0 , c a 1 , c dog * 2 , • • • , c cat * 5 , c EOT * 6 , • • • , c EOT * 76 .Here, dog* and EOT* respectively denote tokens after token merging and end token substitution.Semantic binding loss. As stated in section 3.1, the semantic information of each token embedding is inherently linked. After strengthening the relationship between subjects and their attributes, it becomes crucial to eliminate any irrelevant semantic information within the composite tokens to prevent misrepresentation of attributes. As illustrated in Fig. 4-(b) , to ensure that the semantics of the composite tokens correspond accurately to the noun phrases they are meant to represent, we employ a clean prompt as a supervisory signal. Specifically, for a composite token embedding ĉdog , which corresponds to the noun phrase "a dog wearing hat", we aim for the diffusion model to exhibit consistent noise prediction for this composite token and the full phrase. In mathematical terms, this objective can be expressed as ensuring that ϵ θ (z t , ĉdog , t) ≈ ϵ θ (z t , C, t). This effectively aligns ∇ zt logP θ (z t |ĉ dog ) ≈ ∇ zt logP θ (z t |C) [18, 28] . At time step t, we use the semantic binding loss to align token semanticsL sem = k∈[1,K] ∥ϵ θ (z t , ĉk , t) -ϵ θ (z t , C, t)∥ 2 2 .Entropy loss. Following that, we calculate the information carried by each token embedding through entropy statistics. As shown in Fig. 7 , we extract the cross-attention map A k corresponding to the k-th token [27] . After normalizing the cross-attention map as pi∈A k p i = 1, we compute the entropy of each token as entropy(token k ) = pi∈A k -p i log(p i ). Decreasing the entropy of the cross-attention maps can help ensure that tokens focus exclusively on their designated regions, thereby preventing the cross-attention map from becoming overly divergent. This is further depicted in Fig. 7 , where we observe instances of attribute confusion, characterized by different tokens inappropriately influencing the same image region. The entropy regularization loss is defined asL ent = k∈[1,K] pi∈A k -p i log(p i ) during time step t.Finally, the overall L = L ent + λ • L sem is computed by these two novel losses to update the composite token during each time t < T opt and λ is the trade-off hyperparameter. 
experimental setups: 
 Evaluation Benchmarks and Metrics. We evaluate the effectiveness of ToMe over T2I-CompBench [31] , a comprehensive benchmark for open-world compositional T2I generations, encompassing attribute binding and object relationships. We focus on the semantic binding problem, where T2I-CompBench predominantly evaluates through three attribute subsets (i.e., color, shape, and texture). We follow the evaluation protocol [21, 30, 34] that using 300 validation prompts for evaluation under each subset and the BLIP-VQA score [31] as the evaluation metrics. Following that, we adopt the ImageReward [74] model to evaluate human preference scores, which comprehensively measure image quality and prompt alignment. To comprehensively evaluate object binding performance, we introduce a new GPT-4o Benchmark of 50 prompts using the template "a [objectA] with a [itemA] and a [objectB] with a [itemB].". For example, objectA and objectB are objects like "cat" and "dog" while itemA and itemB are associated items "hat" and "glasses". Afterward, we used the multimodal model GPT-4o [1] to compute the consistency score between the generated images and the prompts for objective assessment. More details are available in the Appendix C.5.Implementation Details. We used SDXL [53] as our base model. To automate image generation for evaluation, we employed SpaCy [29] for syntactic parsing of prompts to identify each object and its corresponding attributes for token merging. The iterative composite token update is performed during the first 20% of the denoising steps T opt = 0.2T .Comparison Methods. To evaluate our method's effectiveness, we compared the current state-ofthe-art methods. These primarily encompass: (1) state-of-the-art T2I diffusion models, including SDXL [53] , Playground-v2 [37] (2) Finetuning-based methods, including CoMat [34] , ELLA [30] (3) Optimization-based method SynGen [58] (4) LLM-augmented finetuning-based method Ranni [21] . More comparison results are shown in the Appendix E.Quantitative Comparison. As shown in Table 1 , ToMe consistently outperforms or performs comparably to existing methods in BLIP-VQA scores across the color, texture, and shape attribute binding subsets, indicating its effectiveness in avoiding attribute confusion. Human-preference scores evaluated through the ImageReward [74] model(note that the model scores are logits and can be negative) suggest that images generated by ToMe can better align with prompts. Specifically, despite ELLA's [30] use of LLama or T5-XL to replace the CLIP Text Encoder for stronger text embeddings, our method still achieves higher BLIP-VQA scores compared to ELLA. The significant improvement in GPT-4o scores also demonstrates the effectivenes of ToMe in object binding.Qualitative Comparison. Following SynGen [58] , we classify the failure cases of attribute binding into three main categories. (i) Semantic leak in prompt, where the attribute a k is not corresponding to its entity n k ; (ii) Semantic leak out of prompt, where the attribute a k is describing the background or some entity not referred to in the prompt P; (iii) Attribute neglect, where the attribute a k is totally ignored in the image generation. Fig. 5 presents our qualitative comparison results with other methods. The first three rows show more complex object binding results, while the last two rows demonstrate attribute binding results. The semantic binding errors in images generated by SDXL [53] Figure 5 : Qualitative comparison among various T2I generation methods with complex prompts. can largely be attributed to (i) semantic leak in the prompt, as evidenced in the first and second row. Playground-v2 [37] confronts similar semantic binding issue as SDXL. ELLA [30] can occasionally succeed in simple attribute binding as in the fifth row, but it frequently encounters (i) semantic leak in the prompt and (iii) attribute neglect errors as shown in the first three prompts. Ranni [21] generates images based on layouts created by a large language model, which can partially address more complex object binding (second row). However, layout-based methods may encounter constrains in achieving proper image layouts, such as shown in the first row with complex descriptions. SynGen [58] , which focus on attribute binding problems, achieves good results in color and shape binding but fails in object binding, exhibiting varying degrees of (i) and (iii) failures. Compared to these methods, our approach ToMe shows improved performance in both object and attribute binding scenarios, which is consistent with the quantitative metrics reflected in Table 1 .Ablation Study over each component is quantitatively shown in Table 2 . We can observe that using only token merging techniques (with ToMe and ETS as config.B) results in a slight performance improvement, which is consistent with the qualitative results in Fig. 6 . However, token merging serve as the foundation for subsequent optimizations. When they are combined with the entropy loss L ent as config.C, the performance improves significantly. We hypothesize that is partly due to the more regularized cross-attention maps as shown in Fig. 7 . Nevertheless, conifg.C without the semantic binding loss still leads to worse generation performance in Fig. 6 , as the dog on the right side still exhibits cat-like features. Incorporating the semantic alignment loss L sem (as our default configuration) ensures that the two subjects correctly bind to their respective attributes without appearance confusion, achieving the best results quantitatively and qualitatively. Suppose token merging is ignored, and we only apply the optimization (Config D and Config E), the performances are only comparable to the baseline. Removing L ent from ToMe (Config F) can also improve over the baseline, but the generation is with noticeable artifacts, which is mainly due to the less regularized cross-attention map. In conclusion, each element of these three novel techniques in ToMe contributes to achieving state-of-the-art performance. See Appendix D for more detailed ablation experiments.Additional Applications of ToMe are shown in Fig. 8 . ToMe can not only successfully address the semantic binding problem, it can also be applied to other problems widely exist in T2I generations, including adding objects [84, 70] , removing objects [3, 22] and even bias mitigation [16, 61, 77, 78] . 
conclusion: 
 In this paper, we investigate a critical issue in text-to-image (T2I) generation models known as semantic binding. This phenomenon refers to instances where T2I models struggle to accurately interpret and visually bind the related semantics. Recognizing that previous methods often entail extensive fine-tuning of the entire T2I model or necessitate explicit specification of generation layouts by large language models, we introduce a novel training-free approach called Token Merging, denoted as ToMe, to tackle semantic binding issues in T2I generation. ToMe incorporates innovative techniques by stacking up the object token with its relevant tokens into a single composite token. This mechanism eliminate the semantic misalignment by unifying the cross-attention maps. Furthermore, we assist the ToMe with end token substitution, and iterative composite token updates technique to strengthen the semantic binding. In extensive experiments, we quantitatively compare it against various existing methods using the T2I-Compbench and our proposed GPT-4o benchmarks. The results demonstrate its ability to handle intricate and demanding generation tasks more effectively than current methods, especially for object binding cases that are ignored in previous research. positioned later can interact with more tokens, thus containing more information. Consequently, we employ an entropy regularization loss to constrain each attention map to be as concentrated as possible, thereby reducing the amount of irrelevant information contained in each token embedding. Tab. 3 reports the inference time costs of various methods, all measured on a single NVIDIA-A40 GPU. We demonstrate that our method does not significantly increase inference time while improving semantic binding performance with 50 inference steps. We further extend this analysis by measuring the time cost with 20 inference steps and various ToMe configurations, as shown in the Tab. 3. We report the time cost (by seconds) along with BLIP-VQA scores across the color, texture, and shape attribute binding subsets. From this table, we can observe that using the token merging (ToMe) technique and entropy loss (Config.C), our method achieves excellent performance with minimal additional time cost. Additionally, even with only 20 inference steps, our method, ToMe, maintains high performance with very little degradation. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Not applicable Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We politely cited the existing assets and read their usage license. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: In order to better demonstrate the binding ability of our model for complex prompts. We have constructed a set of high-difficulty prompts, where the content primarily uses nouns to describe the subject. We use OpenAI's latest release, GPT-4o, to evaluate the quality of images generated by various models because GPT-4o excels in image discernment, allowing for precise evaluation of the generated outputs. As show in Fig. 10 , We designed nine scoring levels, ranging from 0 to 100 points, Prompt: A cat wearing sunglasses and a dog wearing hat Based on our picture and prompt, give the score of the picture below. The subjects are the two subjects of the prompt words, and the attributes are the adjectives or nouns corresponding to the subjects in the prompt body. The first line of the answer contains only the rating, and then the explanation is given starting from the second line.The scoring criteria are as follows: 100: Both subjects only possess their own attributes, not the attributes of the other subject.87.5: Both subjects possesses their attributes. But only one subject that possesses the attributes of another subject.75: When two subjects possess their own attributes, they both possess the attributes of the other subject.62.5: One subject possesses attributes of its own, without attributes of the another subject. The other subject only possesses attributes of another subject.50: One subject possesses attributes of its own. The other subject do not possesses attributes of itself or the other party.37.5: Both subjects not possess its own attributes. But exist one subject has the attributes of the other party.25: Neither subject has attributes of itself or the other party.12.5: Missing one subject 0: Missing two subject 100 Explanation:-The cat is wearing sunglasses, which is its own attribute.-The dog is wearing a hat, which is its own attribute.-Neither the cat nor the dog is wearing the attribute of the other subject. Therefore, the picture perfectly matches the prompt, earning a score of 100.Prompt: A man wearing hat and a woman wearing necklace Based on our picture and prompt, give the score of the picture below. The subjects are the two subjects of the prompt words, and the attributes are the adjectives or nouns corresponding to the subjects in the prompt body. The first line of the answer contains only the rating, and then the explanation is given starting from the second line.The scoring criteria are as follows: 100: Both subjects only possess their own attributes, not the attributes of the other subject.87.5: Both subjects possesses their attributes. But only one subject that possesses the attributes of another subject.75: When two subjects possess their own attributes, they both possess the attributes of the other subject.62.5: One subject possesses attributes of its own, without attributes of the another subject. The other subject only possesses attributes of another subject.50: One subject possesses attributes of its own. The other subject do not possesses attributes of itself or the other party.37.5: Both subjects not possess its own attributes. But exist one subject has the attributes of the other party.25: Neither subject has attributes of itself or the other party.12.5: Missing one subject 0: Missing two subject 75 Both the man and the woman possess their own attributes: the man is wearing a hat, and the woman is wearing a necklace. However, both subjects also have the attributes of the other: the woman is also wearing a hat, and the man is also wearing a necklace."A cat wearing sunglasses and a dog wearing hat""A man wearing hat and a woman wearing necklace" Score:1.00 Score:0.75 The end token substitution (ETS) technique is proposed to address potential semantic misalignment in the final tokens of long sequences. As the [EOT] token interacts with all tokens, it often encapsulates the entire semantic information, as shown in Fig. 2 . Therefore, the semantic information in [EOT] can interfere with attribute expressions, we mitigate this by replacing [EOT] to remove the attribute information it contains from the original prompts, retaining only the semantic information for each subject.For example, as the cross-attention maps and T2I generation performance shown in Fig. 12 , when ToMe is not combined with the EST technique, the 'sunglasses' semantics contained in the EOT token cause the boy to incorrectly wear sunglasses. However, when combined with ETS, the unwanted semantic binding is relieved.In Sec. 3.2.1, we fuse each object and its corresponding attributes. At this stage, both the object token embedding and the attribute token embedding are derived from the text embedding obtained by processing the same prompt through the CLIP Text Encoder, potentially causing the information between them to be coupled. We also experimented with splicing token embeddings from different prompts, as illustrated in Fig. 13 . While keeping other components of ToMe unchanged, the resulting images often exhibit a missing of the object. We hypothesize that this may be due to the lack of contextual semantics between token embeddings from different prompts [8] . 14 : Additional semantic binding results. Our method not only achieves good results in object binding but is also effective for composite binding of objects and their adjective attributes.As shown in Tab. 4, we have added quantitative comparison results with additional methods. Our method consistently outperforms or is on par with the existing methods. Fig. 14 presents more qualitative comparison results, demonstrating that our method achieves good performance in attribute binding, object binding, and the composite binding of attribute and objects. ToMe can also generate images with subjects or backgrounds featuring multiple attributes(Fig. 14 , the last line), in this scenario, we find that using an additional positional loss [19] based on the attention map is effective.We also conduct a user study with 20 participants to enrich the evaluation. Here we compare our method ToMe with SDXL [53] , SynGen [58] , Ranni [21] and ELLA [30] . As shown in Fig. 15 , we ask the participants to rate the semantic binding into 4 levels and calculate the distribution of each comparison method over these four diverse levels. We can observe that our method better achieve the semantic binding performance by mainly distribute in the highest level 1, while the other methods struggle to obtain user satisfactory results. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).† SDXL uses two CLIP text encoders and concatenate the two text embeddings as the final text embedding.‡ Note in this case, the size of the input textual embedding is 1 × 2048 instead 77 × 2048. 


Decision: This paper introduces a novel approach on semantic binding during T2I, this is to associate a given object with its attribute or with other related objects. The approach uses token merging to aggregate relevant tokens into a single token. All reviewers feel positive about the paper. Reviewers think the idea is straightforward and effective, paper is well written with extensive experiments comparing benchmarks. There are a few interesting questions raised by reviewers on complex prompt and importance of entropy loss, authors have provided answers to clarify. The AC agree with all the reviewers and recommend to accept this paper. The AC encourages the authors to add these clarification to the final paper.

Reviews: review: paper_summary: paper_summary: This paper focuses on the problem of lack of semantic binding in text-to-image generation models, and specifically on the misalignment between objects and their sub-objects. The paper introduces a training-free T2I method named ToMe after analyzing the properties of CLIP text embeddings and diffusion models. Utilizing the composition recognition ability of diffusion models, ToMe binds the embeddings of objects with their associated sub-objects together, and retains only the semantic information in the [EOT] tokens. ToMe is shown to be simple yet effective from the experimental results.
strength_weakness: strength_weakness: 1. The Iterative composite Token Update (Sec. 3.2.2) is not clear enough to me. The authors introduce two losses: semantic binding loss and entropy loss. And since ToMe is training-free, it is not clear in the paper how these losses help the update the tokens at test time.
questions: questions: 1. Can the proposed method generalize to more objects, or background with various properties? It would be great if the authors can explain how generating more objects will change the pipeline, or if this is not feasible, please give an explanation why.
2. Please demonstrate how the two losses are integrated in the T2I pipeline, and how they are used to update the tokens.
limitations: limitations: The authors presented the inherent limitation of SDXL that ToMe is based on like producing artifacts in generated images, and unable to generate images with complex layouts. Also the limitation of CLIP to produce text embeddings could restrict the performance of ToMe.
paper_summary: paper_summary: This paper aims to solve the semantic binding problem in T2I models. The authors introduce a semantic binding method by merging tokens of entities and related attributes. Besides, several other tricks like semantic binding loss and entropy loss are introduced to improve the performance of semantic binding.
strength_weakness: strength_weakness: - The importance of a few tricks like entropy loss is not well-emphasized.
- The motivation for some tricks like substituting the EOT token is not well-explained.
questions: questions: - 1. From Table 2, we know that entropy loss + token merging bring the biggest impact. This is reasonable because token merging actually changes nothing, all information still exists in the composition token, and semantic leakage will also happen. However, the reviewer does not fully understand what happens when doing token merging + entropy loss. Why do they work so well? Actually, we don't change the information and the way the information interacts.
    - 1.1. The reviewer thinks Figure 7 is quite clear for the motivation, considering entities and their related attributes as one object. The reviewer is confused about what function entropy loss works.

- 2. What is the impact of substituting the EOT token? Is it important in the whole process?

- 3. How about the efficiency compared to the baseline in Table 1?
limitations: limitations: Yes
paper_summary: paper_summary: The authors propose a method to mitigate semantic binding, a common phenomenon in text-to-image models. While previous methods explicitly control the attention maps so that nouns and attributes attend to the same regions, the authors propose combining the nouns and attributes into a single token. This approach enforces the attention maps to concentrate in the same area. Additionally, they introduce an inference-time optimization loss to ensure the attention maps are focused, and that each composed token retains the same semantic meaning as all the different tokens that comprise it. The authors also analyze the information encoded in different tokens at the output of the text encoder.
strength_weakness: strength_weakness: 1. It is not clear how the model performs on more complicated prompts where both the noun and its sub-objects have attributes, such as: “A blue cat wearing sunglasses and a yellow dog wearing a pink hat.”
2. The method is on the slower side compared to other existing methods, due to inference time optimization.
3. This work is missing a user study, as the proposed automatic metrics can be inaccurate, and human evaluation is common in previous works.
questions: questions: 1. How does the method perform on more complex prompts? For example, for a prompt like “A blue cat wearing sunglasses and a yellow dog wearing a pink hat”. From my understanding, the method should not succeed in separating “pink hat” and “yellow dog” as the whole expression will be combined into a single token.
2. Is it not clear to me what prevents semantic binding across nouns, e.g. both the dog* and the cat* tokens to have the same appearance, (e.g. a cat-dog hybrid), which is a common problem in text-to-image models. What prevents the dog* and cat* attention maps from attending to the same regions?
limitations: limitations: The authors propose limitations relating to the underlying models, but I would be interested to learn in which areas of semantic binding the paper still struggles, such as with more complex prompts or a larger number of nouns, etc..
paper_summary: paper_summary: The authors tackle semantic binding, where T2I models often fail to correctly reflect the relations between objects (object binding) or objects and their attributes (attribute binding). To address this, they introduce Token Merging (ToMe), a method that aggregates related tokens to a single composite token, which ensures they share the same cross-attention map. To do that, two training-free optimizations are introduced. The first, Semantic Binding loss, which makes sure the composite token leads to noise prediction that is consistent with the full phrase the token is based on. The second, Entropy loss, which helps the tokens focus exclusively on their designated regions.
strength_weakness: strength_weakness: * The related work underplays the role of Attend-and-Excite and Linguistic Binding in Diffusion Models (SynGen) have on this paper. The former introduced the idea of semantic guidance, while the latter used dependency graphs to extract syntactically related tokens from prompts and presented a loss that encourages cross-attention maps of syntactically related tokens to agree. Both of which are ideas this paper meaningfully builds on. 

* You do not describe how you obtain the syntactic information: entities and their relevant objects nor if it is automatic or an input constraint. This is particularly confusing because in section 3.2 you begin describing exactly that, but do not delve into the specifics. Furthermore, in section 4.1 (line 248), you mention spaCy and that you "identify each object and its corresponding attributes for token merging", but do not provide actual details about your method. Does it mean it can accurately capture *all* syntactic structures? It is possible that I misunderstand, but it feels like the syntactic analysis portion of this work borrows from the SynGen paper. If this is the case, give appropriate credit. If it is not, then add details, as this is part of the method and is quite unclear.

* If I understand correctly, there are no human evaluation experiments, a user study would provide merit to ToMe’s superior performance. As a side note, in table 1, ‘Human-preference’ is a slightly misleading title. If I understand correctly, it is the output of the Image Reward human-preference model, but it sounds as if it is a human-evaluation result.

* I find that despite the claim that ToMe works well on highly complex prompts, there are no such examples or experiments given. For example, Figure 5 depicts prompts that are supposed to be complex, but they are no different from the simplistic prompts in Attend-and-Excite (“a {color_1} {subject_1} and {a color_2} {subject_2}”), only that they address object binding.
questions: questions: 1. What input text is used in the global encoding part?
2. Do you expect your method to extend to SD-3?
3. I worry that ToMe suffers from too many gradient updates in the event of highly complex prompts (like all training-free approaches), which would push the latent out of distribution. 
4. Does ToMe work on long captions, with multiple sentences? How?
5. You say in the appendix that SynGen was modified to work with SDXL. Can you elaborate on the modifications? What would be needed to reproduce this experiment too?
limitations: limitations: See questions 3 and 4.

Comments: title: Discussion
Thank you
Thanks for your comments and look forward to further discussion
Thanks for your comments and look forward to further discussion
comment: Thank you very much for taking the time to review our paper and offering insightful feedback. We're pleased that our response addressed your concerns, and we will incorporate the recommended experiments and thorough discussion in the revised manuscript. We truly appreciate your time and effort.
Dear authors:

Thanks for your response. They address my concerns.

I will raise my score.

Best,

Reviewer mo93
Thank you very much for reviewing our paper and providing valuable feedback. We're glad our rebuttal addressed your concerns, and we'll include the suggested experiments and detailed discussion in the revised manuscript. Thanks again for your time and effort.
Thank you for taking the time to write back and answering some of my questions.

**W2** Note that the current 4.1 section does not mention the SynGen paper, but I trust that this will be clarified in a revised version.

**Q1** Please add it to the paper.

**Q2** Great. For your consideration: it would be interesting to see a FLUX or SD-3 implementation in a revised manuscript.

**Q3** Thanks, please add that to the paper, as these optimization methods are indeed quite sensitive, and understanding the tradeoffs is important.

**Q5** Please specify the hyperparameters that were used and any other detail that can be used to reproduce your calibration of this baseline.

There are lots of details that were not provided in the discussion, but I trust that you will incorporate everything in great detail in a revised manuscript. I'm raising my score to reflect this.
Dear Reviewer 8gpY:

Thank you for your valuable feedback on our paper. We deeply appreciate the time and effort you’ve put into reviewing our work, and your constructive comments are invaluable to us.

Regarding the weaknesses you mentioned, we include the corresponding experiments and discussion as follows:

1. Expanded discussion on related work, particularly Attend-and-Excite and SynGen.

2. Added detailed explanations on syntactic information extraction using SpaCy.

3. Conducted a user study with 20 participants to validate ToMe’s performance.

4. Provided additional examples of complex prompts generation.

5. Clarified the input text used in global encoding.

6. Discussed the generalizability of our method to other T2I models.

7. Discussed the potential latent distribution drifts.

8. Detailed the modifications made to adapt SynGen for SDXL.

Your feedback is incredibly important to us, and we sincerely thank you for considering our rebuttal. Please let us know if there are any further concerns or questions—we are more than happy to discuss them.

Thank you again for your time, and we look forward to your response.

Best regards,
Authors of submission 1763
Dear Reviewer mo93:

Thank you for your valuable feedback on our work. Your constructive comments on our work are invaluable, and we genuinely hope to get feedback from you.

Regarding the weaknesses you mentioned, we include the corresponding experiments and discussion as follows:

1. Entropy Loss & Token Merging: Our experiments (Fig. 14 in the rebuttal file) show that combining these techniques effectively focuses cross-attention maps, improving semantic binding.

2. End Token Substitution (ETS): As illustrated in Fig. 15 in the rebuttal file, ETS is also can prevent unintended semantic interference, leading to more accurate attribute expressions.

3. Efficiency Compare: We've provided a time cost analysis , showing that our method improves performance with minimal additional time.

Your feedback is incredibly important to us, and we sincerely thank you for considering our rebuttal. We are more than happy to discuss them if you have any further concerns or questions.

Thank you again for your time and effort to review our work and looking forward to your response.

Best Regards,
 Authors of submission 1763