Title: Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion model

Attribute binding

Text Embedding
Abstract: Although text-to-image (T2I) models exhibit remarkable generation capabilities,
they frequently fail to accurately bind semantically related objects or attributes
in the input prompts; a challenge termed semantic binding. Previous approaches
either involve intensive fine-tuning of the entire T2I model or require users or
large language models to specify generation layouts, adding complexity. In this
paper, we define semantic binding as the task of associating a given object with its
attribute, termed attribute binding, or linking it to other related sub-objects, referred
to as object binding. We introduce a novel method called Token Merging (ToMe),
which enhances semantic binding by aggregating relevant tokens into a single
composite token. This ensures that the object, its attributes and sub-objects all share
the same cross-attention map. Additionally, to address potential confusion among
main objects with complex textual prompts, we propose end token substitution as
a complementary strategy. To further refine our approach in the initial stages of
T2I generation, where layouts are determined, we incorporate two auxiliary losses,
an entropy loss and a semantic binding loss, to iteratively update the composite
token to improve the generation integrity. We conducted extensive experiments to
validate the effectiveness of ToMe, comparing it against various existing methods
on the T2I-CompBench and our proposed GPT-4o object binding benchmark. Our
method is particularly effective in complex scenarios that involve multiple objects
and attributes, which previous methods often fail to address. The code will be
 publicly available at https://github.com/hutaihang/ToMe


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle semantic binding, where T2I models often fail to correctly reflect the relations between objects (object binding) or objects and their attributes (attribute binding). To address this, they introduce Token Merging (ToMe), a method that aggregates related tokens to a single composite token, which ensures they share the same cross-attention map. To do that, two training-free optimizations are introduced. The first, Semantic Binding loss, which makes sure the composite token leads to noise prediction that is consistent with the full phrase the token is based on. The second, Entropy loss, which helps the tokens focus exclusively on their designated regions.\n\nSTRENGTHS:\n* Creating a composite token and then optimizing it is a creative and elegant approach to the problem. I'm curious to see what future work can be built on top of this contribution.\n* The focus on object binding is indeed missing from the literature, a topic that is well-addressed in this work.\n* The idea of end token substitution to remove attribute information is simple.\n\nWEAKNESSES:\n* The related work underplays the role of Attend-and-Excite and Linguistic Binding in Diffusion Models (SynGen) have on this paper. The former introduced the idea of semantic guidance, while the latter used dependency graphs to extract syntactically related tokens from prompts and presented a loss that encourages cross-attention maps of syntactically related tokens to agree. Both of which are ideas this paper meaningfully builds on. \n\n* You do not describe how you obtain the syntactic information: entities and their relevant objects nor if it is automatic or an input constraint. This is particularly confusing because in section 3.2 you begin describing exactly that, but do not delve into the specifics. Furthermore, in section 4.1 (line 248), you mention spaCy and that you \"identify each object and its corresponding attributes for token merging\", but do not provide actual details about your method. Does it mean it can accurately capture *all* syntactic structures? It is possible that I misunderstand, but it feels like the syntactic analysis portion of this work borrows from the SynGen paper. If this is the case, give appropriate credit. If it is not, then add details, as this is part of the method and is quite unclear.\n\n* If I understand correctly, there are no human evaluation experiments, a user study would provide merit to ToMe’s superior performance. As a side note, in table 1, ‘Human-preference’ is a slightly misleading title. If I understand correctly, it is the output of the Image Reward human-preference model, but it sounds as if it is a human-evaluation result.\n\n* I find that despite the claim that ToMe works well on highly complex prompts, there are no such examples or experiments given. For example, Figure 5 depicts prompts that are supposed to be complex, but they are no different from the simplistic prompts in Attend-and-Excite (“a {color_1} {subject_1} and {a color_2} {subject_2}”), only that they address object binding.\n\nQUESTIONS:\n1. What input text is used in the global encoding part?\n2. Do you expect your method to extend to SD-3?\n3. I worry that ToMe suffers from too many gradient updates in the event of highly complex prompts (like all training-free approaches), which would push the latent out of distribution. \n4. Does ToMe work on long captions, with multiple sentences? How?\n5. You say in the appendix that SynGen was modified to work with SDXL. Can you elaborate on the modifications? What would be needed to reproduce this experiment too?\n\nLIMITATIONS:\nSee questions 3 and 4.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to mitigate semantic binding, a common phenomenon in text-to-image models. While previous methods explicitly control the attention maps so that nouns and attributes attend to the same regions, the authors propose combining the nouns and attributes into a single token. This approach enforces the attention maps to concentrate in the same area. Additionally, they introduce an inference-time optimization loss to ensure the attention maps are focused, and that each composed token retains the same semantic meaning as all the different tokens that comprise it. The authors also analyze the information encoded in different tokens at the output of the text encoder.\n\nSTRENGTHS:\n1. The authors present an interesting analysis of the information encoded tokens, specifically they show that much of the semantic information is also encoded in the EOT tokens.\n2. The authors provide a method to mitigate semantic binding in prompts of the form: “A <noun_A> <Attribute_A> and <noun_B> <Attribute_B>”, that outperforms or performs comparably to existing methods across various benchmarking subsets.\n\nWEAKNESSES:\n1. It is not clear how the model performs on more complicated prompts where both the noun and its sub-objects have attributes, such as: “A blue cat wearing sunglasses and a yellow dog wearing a pink hat.”\n2. The method is on the slower side compared to other existing methods, due to inference time optimization.\n3. This work is missing a user study, as the proposed automatic metrics can be inaccurate, and human evaluation is common in previous works.\n\nQUESTIONS:\n1. How does the method perform on more complex prompts? For example, for a prompt like “A blue cat wearing sunglasses and a yellow dog wearing a pink hat”. From my understanding, the method should not succeed in separating “pink hat” and “yellow dog” as the whole expression will be combined into a single token.\n2. Is it not clear to me what prevents semantic binding across nouns, e.g. both the dog* and the cat* tokens to have the same appearance, (e.g. a cat-dog hybrid), which is a common problem in text-to-image models. What prevents the dog* and cat* attention maps from attending to the same regions?\n\nLIMITATIONS:\nThe authors propose limitations relating to the underlying models, but I would be interested to learn in which areas of semantic binding the paper still struggles, such as with more complex prompts or a larger number of nouns, etc..", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper aims to solve the semantic binding problem in T2I models. The authors introduce a semantic binding method by merging tokens of entities and related attributes. Besides, several other tricks like semantic binding loss and entropy loss are introduced to improve the performance of semantic binding.

STRENGTHS:
- The idea is straightforward and it is also reasonable that it will work with some other tricks.
- The paper is well-written.

WEAKNESSES:
- The importance of a few tricks like entropy loss is not well-emphasized.
- The motivation for some tricks like substituting the EOT token is not well-explained.

QUESTIONS:
- 1. From Table 2, we know that entropy loss + token merging bring the biggest impact. This is reasonable because token merging actually changes nothing, all information still exists in the composition token, and semantic leakage will also happen. However, the reviewer does not fully understand what happens when doing token merging + entropy loss. Why do they work so well? Actually, we don't change the information and the way the information interacts.
    - 1.1. The reviewer thinks Figure 7 is quite clear for the motivation, considering entities and their related attributes as one object. The reviewer is confused about what function entropy loss works.

- 2. What is the impact of substituting the EOT token? Is it important in the whole process?

- 3. How about the efficiency compared to the baseline in Table 1?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of lack of semantic binding in text-to-image generation models, and specifically on the misalignment between objects and their sub-objects. The paper introduces a training-free T2I method named ToMe after analyzing the properties of CLIP text embeddings and diffusion models. Utilizing the composition recognition ability of diffusion models, ToMe binds the embeddings of objects with their associated sub-objects together, and retains only the semantic information in the [EOT] tokens. ToMe is shown to be simple yet effective from the experimental results.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n2. The authors present detailed analysis of the CLIP text embeddings including their coupled and entangled properties and the semantically additive properties of text embeddings shown in diffusion models. \n3. The structure of the proposed ToMe model is training-free, simple and mostly clear.\n4. The authors provide extensive experimental results including quantitative, qualitative and ablation studies on the T2I-CompBench and GPT-4o Benchmark (introduced in this paper).\n\nWEAKNESSES:\n1. The Iterative composite Token Update (Sec. 3.2.2) is not clear enough to me. The authors introduce two losses: semantic binding loss and entropy loss. And since ToMe is training-free, it is not clear in the paper how these losses help the update the tokens at test time.\n\nQUESTIONS:\n1. Can the proposed method generalize to more objects, or background with various properties? It would be great if the authors can explain how generating more objects will change the pipeline, or if this is not feasible, please give an explanation why.\n2. Please demonstrate how the two losses are integrated in the T2I pipeline, and how they are used to update the tokens.\n\nLIMITATIONS:\nThe authors presented the inherent limitation of SDXL that ToMe is based on like producing artifacts in generated images, and unable to generate images with complex layouts. Also the limitation of CLIP to produce text embeddings could restrict the performance of ToMe.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}