Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function

Authors: chenyi  zhuang
ying  hu
pan  gao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:13:58

Abstract: Text-to-image diffusion models particularly Stable Diffusion, have revolutionized the field of computer vision. However, the synthesis quality often deteriorates when asked to generate images that faithfully represent complex prompts involving multiple attributes and objects. While previous studies suggest that blended text embeddings lead to improper attribute binding, few have explored this in depth. In this work, we critically examine the limitations of the CLIP text encoder in understanding attributes and investigate how this affects diffusion models. We discern a phenomenon of attribute bias in the text space and highlight a contextual issue in padding embeddings that entangle different concepts. We propose Magnet, a novel training-free approach to tackle the attribute binding problem. We introduce positive and negative binding vectors to enhance disentanglement, further with a neighbor strategy to increase accuracy. Extensive experiments show that Magnet significantly improves synthesis quality and binding accuracy with negligible computational cost, enabling the generation of unconventional and unnatural concepts.

Main paper: 
 introduction: 
 Recently, Text-to-Image (T2I) diffusion models [1, 2, 3, 4] have drawn considerable attention from both the research community and industry. Among these models, Stable Diffusion (SD) [2] uses the CLIP text encoder [5] to encode the given prompt, which is relatively lightweight than other diffusion models that adopt T5 [6] . Unfortunately, generating text-aligned images is still challenging for SD and requires multiple runs to achieve the desirable results. Several works [7, 8, 9, 10] have pointed out that the blended context by the CLIP text encoder causes improper binding. However, few have analyzed in detail how the text encoder affects the generation of the diffusion model.Step back and refocus on the CLIP text encoder-an integral part of the Vision-Language Model (VLM). Prior studies [11, 12] have observed VLMs lacking compositional understanding and investigated them on image-to-text retrieval benchmarks. In this work, we are motivated to answer how the text encoder understands attribute, and how it affects the attribute binding of T2I diffusion models. Upon closer inspection, we observe a phenomenon of attribute bias and discern a contextual problem in padding embeddings, leading to a well-known T2I issue-concept bleeding [10] .Based on the observation, we introduce the binding vector, which is applied to the text embedding of each object. With positive and negative binding vectors, each object can pull target attributes and push unrelated attributes to distinguish them from each other. We further introduce a neighbor strategy to ensure an accurate estimation of the binding vector. Our manipulation is performed strictly in the textual space, without training, fine-tuning, or additional datasets and inputs. Overall, the main  
analysis of the clip text encoder and the diffusion model: 
 In this section, we aim to recognize the pattern of the CLIP text encoder for understanding attributes, then go deep into the diffusion model to analyze the underlying reason for improper attribute binding.The CLIP text encoder uses the causal mask mechanism to produce a unidirectional context, i.e., each word can only consider the words to their left [13, 14] . It performs contrastive learning on a specific End of Text ([EOT]) embedding without word-level supervision, while prior studies [15, 16, 17] suggest each word has a semantic effect on the generated image. In this case, we categorize two types of embedding as word and [EOT] for fine-grained analysis. Consider a prompt encoded to text embeddings c = {c SOT , c p1 , ..., c p N , c EOT } consisting N word embeddings. Different from the [EOT] embedding c EOT , word embeddings c p1 , ..., c p N are unsupervised during training. We skip the embedding of Start of Text ([SOT]) c SOT for simplicity. To study how the two types of embedding understand attributes, we select 60 familiar objects and 7 common colors to obtain text embeddings c = {c object , c EOT } and c ′ = {c ′ color , c ′ object , c ′ EOT }, i.e., without and with the color context, respectively. We aim to compare: (1) contextualized word embedding c object with the original c ′ object ; (2) contextualized [EOT] embedding c EOT with the original c ′ EOT . How do two types of embedding understand attributes? Fig. 1 (a) compares the Euclidean distance and the cosine similarity between embeddings with and without the color context. The pattern varies between cases or objects. As to the word embedding, the similarity curve of "chair" is relatively smooth, but that of "sheep" has a large gap between "black" and "white". Similarly, "blue apple" diverges from others. The above phenomenon, which we call attribute bias, describes the tendency of an object to favor certain attributes over others. We compare the attribute bias per object for two embedding types in Fig. 1 (b ). If the object has a natural composition in human knowledge, it presents a serious attribute bias (e.g., "yellow banana" v.s. "blue banana"). Meanwhile, the word embedding is more volatile than the [EOT] embedding, which shows less dramatic change. Our hypothesis is the absence of word-level supervision during CLIP training, as well as the bags-of-words behavior of VLMs [11, 12] : the [EOT] embedding is trying to remember all important words in the given prompt, including adjectives and nouns. However, it leads to an inaccurate textual representation of the [EOT] embedding, affecting the interaction between the image latent and semantic word embeddings. We have provided more analysis and examples in Appendix A.1 (see Fig. 12, 13 ).How do two types of embedding affect SD? In practice, SD pads the input prompt to a fixed length L = 77 using additional [EOT] embeddings (i.e., the padding token is initialized by the symbol [EOT]), denoted as c pad l , where l = 1, ..., L-N -2. To study the attribute binding during generation, we modify the above two text embeddings c and c ′ with L -N -2 padding token embeddings, then design 4 fine-grained cases: (1) standard generation conditioned on c ′ where all text embeddings contain the color context; (2) " red chair "(2) (4) (3) " blue apple " " black sheep " (1) (2)EQUATION(4) (3) including a natural concept "red chair", unnatural concepts "black sheep" and "blue apple". Cases 1-2 can be observed in all examples with less realistic and painting-like images. Note that the used SD is trained to generate photo-realistic images. These results indicate a deviation from the learned distribution. Conversely, cases 3-4 produce realistic images, but neglect the target color when the concept is unnatural. This suggests that the context in [EOT] and padding embeddings do have a significant impact on attribute generation. In Appendix A.2, we describe the above 4 cases in detail and provide more examples, as well as 3 additional cases (see Fig. 14 ).We posit that the first [EOT] has a close-knit context with word embeddings. The padding embedding, however, may deviate due to the causal attention mechanism. We then compute the cosine similarity between [EOT] and each padding embedding l = 1, ..., L, and dive into their cross-attention activations on single-and multi-concept scenarios. Fig. 2 (b) studies a prompt with only one object. The curve drops more drastically on the unnatural concept "blue apple" than the natural one "red chair". Cross-attention shows that "apple" overlaps with padding embeddings (e.g., pad 73 ) rather than the [EOT] embedding. It is like these padding embeddings have forgotten part of the context remembered in the first [EOT] . Without interference from other concepts, the inaccurate context in these padding embeddings causes out-of-distribution, or the binding of another attribute if the model learns an underlying bias in the training dataset (e.g., "apple" prefers "red"). Fig. 2 (c ) further studies the context issue on multiple concepts. For two objects "bird" and "car", even though the activations of their word embeddings do not overlap, cross-attention shows obvious entanglement in these padding embeddings. This multi-concept context issue in padding embeddings, i.e., entangled concept representations, explains color leakage and object sticking. We refer the reader to Appendix A.3 for a comprehensive analysis of this context issue.How to disentangle different concepts? Prior studies [8, 18] prove that these padding embeddings are essential for image quality and can not simply be removed. Also, it is impossible to manipulate one single concept in these padding embeddings due to their entangled property. On the other hand, these naturally separated word embeddings show editability. For instance, Fig. 2 (a) "black sheep" from case 2 to case 1 changes only the word embedding of "sheep" while encouraging the desired attribute. We are inspired to manipulate the word embedding of each object, therefore strengthening the binding within each concept and enhancing the distinction between concepts.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of this work have been discussed in Section 6 and Appendix F (see Figure 21 ).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
magnet: disentangling concepts with the binding vector: 
 Our approach is based on two key observations: the context issue of the padding embedding, and the controllability of the word embedding. We introduce the binding vector, which can be applied on the object embedding to attract the target attribute and repulse other attributes, analogous to a Magnet. Preliminary: Given a prompt P, we use Stanza's dependency parsing module [19] to extract each concept, denoted A&E, where E is the object word and its target attribute as A. The dependency For each object E i in D with the word embedding c Ei , we aim to estimate its positive binding vector v pos i to pull the target attribute A i , and its negative binding vector v neg i to push other attributes.Instinctively, the binding vector can be estimated by the object itself. To be specific, we compose new concepts out of the current context of P, which are: (1) unconditional concepts,Puc i = {∅&E i }, where ∅ is a blank text ""; (2) positive concepts, Ppos i = {A i &E i }; (3) negative concepts, Pneg i = {A j &E i |j = 1, ..., M, j ̸ = i}.The positive and negative binding vectors are estimated by:EQUATIONEQUATIONwhere F(•) extracts the word embedding of the object E i in a specific decontextualized prompt. Note each object has M -1 negative concepts, resulting in M -1 negative binding vectors to punish all unrelated attributes A j , j = 1, ..., M, j ̸ = i. Note that these positive and negative attributes are prompt-dependent2 . We introduce the unconditional concept as a pivot to avoid the need for manual definition or semantic contrast between positive and negative attributes.Based on our analysis of the context issue in the padding embedding in Section 2, we hypothesize an association between the attribute bias and the strength. Intuitively, unnatural concepts (e.g., "blue banana") suffer more attribute bias and their padding embeddings are more tend to forget the concept.In this case, we need to manipulate the word embedding significantly to ensure strong binding. We introduce the adaptive strength of the binding vector for each object E i :EQUATIONwhere G(•), H(•) extract the first [EOT] embedding and the last padding embedding in text embeddings E( Ppos i ), respectively. λ is a positive constant. Please refer to Appendix B.3 for the inspiration of the formula, and the statistical analysis for the choice of the hyperparameter λ.Finally, the object embedding c Ei in the initial text embeddings c = E(P) is modified by:ĉEi = c Ei + α i • v pos i -β i • v neg i (4)In practice, we find that using a single object to estimate the binding vector can be inaccurate and fail to disentangle concepts (see Fig. 7 and Fig. 19 ). In this case, we introduce the neighbor strategy to ensure an accurate estimation. These neighbor objects should have similar representations to the target object in the learned textual space. We define a candidate set S = {B 1 , ..., B R } with R objects that has pre-processed to {c B1 , ..., c B R }, which is the collection of the word embedding c Br in E(B r ) = {c SOT , c Br , c EOT , ...}, r = 1, ..., R. The top-K neighbor objects for the target object E i are determined by d(c Br , F(E i , Puc i )), where B r ∈ S, d(•) denotes the cosine similarity. In Appendix B.4, we describe this neighbor strategy in detail, and further discuss a way to predict semantic neighbors using pre-trained large language models.With the selected neighbors of the target object E i , denoted{B (i) k } K k=1 , we compose the unconditional concepts Puc k = {∅&B (i) k }, positive concepts Ppos k = {A i &B (i)k }, and negative concepts Pnegk = {A j &B (i) k |j = 1, ..., M, j ̸ = i}.The estimation of the binding vector is then rewritten as:EQUATIONEQUATIONFig. 3 depicts the workflow of Magnet. The target text embedding ĉ can be obtained after replacing all object embeddings c Ei with ĉEi , i = 1, ..., M . To generate the image, the pre-trained U-Net denoises the latent z t-1 = ϵ θ (z t , t, ĉ), where timesteps t = T, ..., 1. We set the hyperparameters λ = 0.6, K = 5. Please refer to Appendix C for implementation details.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the used two datasets (ABC-6K and CC-500) and a piece of code with sufficient instructions in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
datasets: 
 We evaluate our proposed Magnet on two existing benchmarks:(1) Attribute Binding Contrast set (ABC-6K) [8] . This dataset consists of natural compositional prompts from MS-COCO [20] , each prompt includes at least two concepts (e.g., "a bathroom with a tan sink and white toilet", "a brown cow standing in a lush green field"). We randomly sample 600 prompts from this dataset and generate 5 images per prompt to compare all methods.(2) Concept Conjunction 500 (CC-500) [8] . The dataset contains prompts that conjunct two concepts, each with one color attribute. Following [7] , objects are divided into two types: living (i.e., animals and plants) and other non-living nouns. Prompts type is categorized into (1) two living objects, (2) one living object and one non-living object, and (3) two non-living objects. We adopt 80 prompts for each case to avoid bias and maintain fairness. In total, we have used 240 prompts and generated 10 images for each prompt to compare all methods.Both datasets are augmented using contrast settings [21] . The position of attribute words for different objects is swapped (e.g., "a red chair and a blue cup" ↔ "a blue chair and a red cup").We mainly rely on human evaluation since the common metrics (e.g. CLIP text-image similarity) are unreliable for assessing attribute binding, which is discussed in Appendix D.Coarse-grained comparison. We assess the generated image for image quality and concept disentanglement on two adopted datasets. To measure image quality, human evaluators were asked "Which image is more realistic or visually appealing?". The evaluation of concept disentanglement is divided into two types: (1) object disentanglement by asking "Which image shows different objects more clearly?"; (2) attribute disentanglement by asking "Which image shows different attributes more clearly?". If all images are equally good or bad, evaluators can indicate "no winner". We randomly sample one image for each prompt from ABC-6K and two images for each prompt from CC-500 to conduct this coarse-grained comparison. Table 2 : Fine-grained comparison on the CC-500 dataset. For reference, we provide the average confidence (Conf.) of GroundingDINO [22] to detect the object (Det.). Manual evaluation concerns the object existence (Obj.) and the attribute alignment (Attr.). Fine-grained comparison. This comparison is conducted on the CC-500 dataset based on two key criteria: (1) object existence, counting the target objects in the generated images; (2) attribute alignment, concerning the correct binding between the object and its attribute. We ask annotators to identify the object mentioned in the prompt per generated image. Take prompt "a red car and a yellow cat" as an example, each image will be indicated the number two (show both objects), one (show either "car" or "cat"), or zero (no distinct object). Attribute alignment is assessed by counting whether the generated object presents the desired attribute (maximum to the number of the generated objects). All generated images on CC-500 are used for this fine-grained comparison. In addition, we adopt the phrase grounding model GrondingDINO [22] to detect the target objects automatically.Note that this automatic detection can not reflect the proper binding.Coarse-grained comparison. In Tab. 1, we present the human evaluation results of Magnet compared to three baseline methods: SD V1.4 [23] , Structure Diffusion [8] and Attend-and-Excite [7] . Note that Magnet and Structure Diffusion are both training-free. The ABC-6K benchmark has more complicated and challenging prompts. In this case, all methods may fail to include all objects and attributes, resulting in a higher number of no winner. Overall, Magnet achieves the best scores in terms of image quality and attribute disentanglement on both datasets.Fine-grained comparison. As shown in Tab. 2, Magnet alleviates the missing problem more than Structure Diffusion on both automatic and manual evaluation, with 3.8% (Det.) and 4.6% (Obj.) improvement. We are inferior to the optimization method, Attend-and-Excite in object existence.In attribute alignment (Manual Attr.), Magnet outperforms all baseline methods. In addition, we compare the runtime and memory used for generation. The data is obtained by generating 100 prompts each with two images. Obviously, Attend-and-Excite requires more resources which affects efficiency. Conversely, Magnet only adds 2.9% to runtime and 1% to the memory.Evaluation on image quality metric. We also evaluate Magnet on the commonly used metric FID [24] for two SD versions (V1.4 [23] and V2.1 [25] ). We follow the standard evaluation process and generate 10k images from randomly sampled MS-COCO [20] Fig. 4 shows the qualitative comparison of the ABC-6K and CC-500 datasets. The results demonstrate that baselines suffer from the entanglement of objects and attributes.Object entanglement includes the neglect of the object or sticking structures. In columns 1-2, baselines struggle to be faithful to the complex prompt with 4 objects, missing "fries" or "tile".In columns 5-6, the objects "banana" and "stickers" are indistinguishable. Similarly, SD presents blended objects "dog" and "chair" in columns 7-8 and neglects the target object "green apple" in columns 9-10. Note that the results of Structure Diffusion resemble that of SD. On the other hand, the optimization of Attend-and-Excite encourages the attendance of objects but leads to out-of-distribution results, showing strong artifacts (e.g., "green apple" in columns 9-10).Attribute entanglement includes the generation of incorrect attributes or the leakage of attributes. For instance, for the prompt "a pink cake with white roses on silver plate" with three colors in columns 3-4, SD and Structure Diffusion generate "white cake" and "pink roses". In columns 7-8, they generate "chair" with mixed colors "yellow" and "red". On the other hand, Attend-and-Excite may produce less aesthetic images, which can be attributed to the over-optimized image latent.Notice that baselines fail to produce unnatural concepts like "blue banana" in columns 5-6 in Fig. 4 . Instead, they generate "yellow banana", which is a natural concept learned as the prior knowledge. Conversely, Magnet is capable of disentangling different concepts and hence generating unnatural concepts, which we call the anti-prior ability. Fig. 5 displays the results on prompts with anti-prior concepts. We skip Structure Diffusion for its limited improvement over SD.Figure 6 : Ablation study on the hyperparameter λ given the prompt "a pink cake with white roses on silver plate". A small value of λ can not well disentangle different concepts, while a large value causes artifacts in the generated image (best viewed zoomed in). We empirically set λ = 0.6. The neighbor strategy improves the binding vector estimation, separating different attributes ("cup" is purely "blue") and objects ("backpack" and "apple" are distinguishable). Hyperparameter λ. We study the effect of λ in Fig. 6 . When setting λ = 0, α, β are still positive numbers but the manipulation is in relatively low strength. In this case, concepts are still entangled: "roses" appear in shades of "white" and "pink". When setting λ = 1, the result presents artifacts: distorted "plate" and watermarked background. We find using λ = 0.6 can achieve the balance between concept disentanglement and image quality based on the statistic analysis in Fig. 16 .Selection strategy of the neighbor strategy. The effectiveness of the neighbor strategy is shown in Fig. 7 . The neighbors improve the estimation accuracy and the disentanglement of concepts. In Tab.3, we ask human evaluators to evaluate both settings using the disentanglement criteria. Evaluators indicate the generated images using the neighbor strategy more disentanglement. This verifies the effectiveness of the neighbor-guided vector estimation.Effectiveness of the binding vector. In Fig. 8 , we verify the effectiveness of the binding vector by manually changing α, β instead of adaptively calculating by Eq. ( 3). The value of α, β changed from positive to negative shows a swapped binding between objects and attributes. This is because that the context problem in padding embeddings has caused the entanglement of concepts. Our proposed binding vectors can improve the discrimination between objects and lead to designated attributes.We have conducted additional ablation experiments for the hyperparameter K (Appendix E.1, Fig. 19 ), and the importance of using both positive and negative binding vectors (Appendix E.2, Fig. 20 ).Incorporate with optimization-based methods. Manipulated in the textual space, Magnet can be readily integrated with Attend-and-Excite. multiple CLIP text encoders, and PixArt [26] uses the T5 encoder [6] . We use the same setting of all hyperparameters and equations for all CLIP-based models while using fixed strength for PixArt. The redesign of the strength formula for the adaptation of T5 is a matter for future work.Incorporate with T2I controlling modules. In Fig. 10 (c ) and (d), we investigate the plug-and-play nature of Magnet. Magnet shows compatibility when integrated with existing controlling modules:(1) layout-guidance [27] , which constrains the image layout by bounding boxes and intervenes cross-attention layer, and (2) ControlNet [28] conditioned on Depth Map [29] to add spatial control.Image editing. In Fig. 11 , we compare the image editing ability of Magnet to Prompt-to-Prompt (P2P) [15] , which edits the generated image by manipulating the cross-attention layers. Given the source prompt "a car on the side of the street", we aim to change the attribute of the object "car" or "street". In column 1, Magnet applies a positive binding vector v pos (here, the strength α is stated manually) on the word embedding c Ecar toward the attribute "old". With no control of the attention maps, Magnet surprisingly edits the image with fewer changes in the background than P2P. 
related work: 
 Text-to-image diffusion models. Diffusion models that [30] pioneered, have emerged with great improvement in both unconditional [31, 32] or conditional [28, 33] image generation, together with the advance in synthesis quality [34, 35] and sampling speed [36, 37, 38] . However, the semantic flaw of the text encoder affects the performance of the diffusion models [7, 10, 39] . In this work, we discern the attribute bias and the context issue, providing novel insights about attribute binding. Attribute binding. The binding problem occurs when the model blends improper concepts. To tackle complicated prompts, [9] collaborates different pre-trained diffusion models. [8] suggests word embeddings with blended context and manipulate cross-attention features. In contrast, we highlight the entanglement of the padding embedding and modify solely the text embedding. [7] optimizes the latent to guarantee the attendance of each object. Yet, the optimization may lead to out-of-distribution and require more resources to generate images. Other works [40, 41, 42] introduce layout constraints in the attention layers. Magnet differs from the above approaches in that it can be executed entirely in the textual space. This distinguishes it as a more efficient solution.It is noteworthy that a line of works [15, 43] achieves image editing on specific visual aspects. However, none have gone as far as this paper in exploring the contextual influences on SD from the perspective of text embedding. Most are subject to a subset of attributes (e.g., texture [44] ), control the global object rather than fine-grained attributes [45, 46] , or depend on a predefined text pair [47] , requiring a learning process or additional datasets. Conversely, our method enhances binding towards arbitrary attributes without the need for new inputs to the standard pipeline.  
limitations: 
 While we have demonstrated improvement in the synthesis quality and text alignment, Magnet is still subject to a few limitations (see Fig. 21 ). First, it still suffers from the missing problem. In some cases, the manipulation may be overstrength and cause artifacts. An interesting phenomenon is that Magnet generates the correct concepts while rendering errors in positional relations. Finally, it is still challenging to generate an unnatural concept when the object is strongly biased towards one specific attribute. (e.g., "broccoli"). We have described the limitations of Magnet in detail in Appendix F. 
conclusion: 
 In this work, we propose a novel training-free method, Magnet, to tackle the attribute binding issue. First, we conduct a fine-grained analysis of the CLIP text encoder. We observe the phenomenon of attribute bias and point out the context issue of padding embeddings, where the representations of different concepts are entangled, and hence provide potential explanations for existing T2I issues. Second, we introduce the positive and negative binding vectors to enhance the binding within the concept and strengthen the distinction between concepts. Further with the neighbor strategy, the vector estimation can be more accurate. Evaluated in various ways, Magnet shows the ability to disentangle different attributes and generate anti-prior concepts. Performed in the textual space, Magnet improves the synthesis quality and text alignment, with an impressively low increase in computational cost. We sincerely hope that this work will motivate the exploration of generative diffusion models and the discovery of other interesting phenomena. Attribute bias analysis. Fig. 13 investigates the phenomenon we call attribute bias on two types of text embedding, obtained from the text encoder of CLIP ViT-L/14 and ViT-H/14, respectively. The word embedding without supervision during training has shown severe attribute bias. For example, the word embedding of the object "tiger" indicates an extreme preference for the color "yellow". Conversely, the [EOT] embedding produces a relatively small variation in the similarity curve. In the main paper, we conjecture that VLMs' poor compositional understanding and the behavior of bags-of-words [12, 11] on [EOT] lead to an inaccurate textual representation, which affects the interaction between the image latent and semantic word embeddings.Interestingly, we find that the learned representations of two text encoders are quite different. For example, to encode "car" and "vase" with the context of different attributes, CLIP ViT-L/14 gets the cosine similarity around 0.6 v.s. 0.7, while CLIP ViT-H/14 gets 0.2 v.s. 0.5, showing a discrepancy. We conjecture that ViT-L/14 in a large-sized network and dimension may have exacerbated the bias. Yet, it is beyond the scope of our research and we may leave it for future study.Despite the above difference, both encoders demonstrate the discrepancy between the word and [EOT] embeddings. The stability of the [EOT] embedding can also be explained by entangled context. In contrast, the word embedding without supervision during training may suffer less from entanglement. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Section 5, Appendix E.1 and F have discussed both potential and negative societal impacts of the proposed method.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We adopt the ABC-6K and CC-500 datasets from [8] . Our method is built upon Stable Diffusion 1.4 [2] with the pre-trained text encoder of CLIP ViT-L/14 [2] .Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The assets introduced in the paper have been documented. In the supplemental material, we have provided a Jupiter file with visualized results for reference.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: The instructions given to human evaluators are described in Section 4.2, and a screenshot instance in Figure 18 .Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes]Justification: We have obtained approval from an equivalent to the Institutional Review Board to conduct our quantitative experiments.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Take the concept "red chair" as an example. The CLIP text encoder maps it into embeddings NOTICE: we propose Magnet based on two key observations on the word embedding. First, the target color is invisible in case 4 for concepts "green car", "red cat". However, these colors can be observed in case 3 (arrive at Eq. ( 1) for computing v pos ). Second, cases 1-2 and cases 3-4 for the concept "blue banana" both generate catastrophic images. This indicates the vector estimated by the object itself be inaccurate. In this case, we introduce the neighbor-guided vector estimation.c ′ = {c ′ SOT , c ′ red , c ′ chair , c ′ EOT , c ′ pad1 , ..., c ′ pad73 } (Recall our hypothesis that [EOT] and padding embeddings are trying to remember all important information (e.g., attributes, objects, and positions) in the given prompt due to the learning and bags-of-words behavior of CLIP. In Fig. 15 , we investigate the entangled context in the padding embeddings under two scenarios for prompts with a single object or multiple objects.Single-concept scenario aims to generate one object with specific attributes. Fig. 15 (a) shows that the context issue in padding embeddings leads to (1) out-of-distribution and inaccurate structures, e.g., "cat" is painting-like in row 1, "banana" is unrecognizable in row 2, though presenting correct attributes. Or (2) generate the object with another attribute that can compose a natural concept, e.g., "broccoli" binds to the prior attribute "green" rather than "black" in row One potential explanation is the image latent is contaminated by inaccurate representation in padding embeddings, as evidenced by the overlapped activation of latter padding embeddings with the word embedding of each object. The generation of natural concepts proves our hypothesis that latter padding embeddings forget attribute context if the object has a preference for certain attributes based on the training dataset. In row 4, we present an interesting observation that padding embeddings are aligned with the attribute word rather than the object "strawberry". It seems that the word "gold" is interpreted as an entity instead of a visual feature, leading to (3) a split of the target object.Multi-concept scenario aims to generate multiple objects with the desired attributes. Fig. 15 (b) shows that the context issue in padding embeddings leads to (4) color leakage, i.e., one object presents the attribute belonging to another object in row 1. Or (5) objects stick together, e.g., a strange creature with the head of a "horse" but the body of a "bag" in row 2. All the above phenomena can be attributed to the evident entanglement in padding embeddings with overlapped cross-attention activations, which provides inaccurate object representation and indistinguishable binding relationships for each concept. Note that the above effects can occur simultaneously on a single instance: row 3 indicates an inaccurate "sheep" structure, a binding between "banana" and the prior color "yellow", a split of the object "banana", as well as a sticking problem between two objects "banana" and "sheep". In row 4, we find that the context issue of padding embeddings also explains (6) the issue of missing objects, i.e., the context loses the object "sheep" and contains a dominant representation of the object "car".[18] also discussed the semantic information in padded [EOT] embeddings. While their main concern is to remove one specific object content, our focus is the understanding of attribute. To extract the dependency set D = {A 1 &E 1 , ..., A M &E M } in the given prompt, we adopt an offthe-shelf dependency parsing module in Stanza Library [19] and construct syntax trees using NLTK. Following [8] , the pair is searched by noun phrases (NPs) in the syntax tree and their corresponding adjective words. For instance, given the prompt "a black cat sitting in a white bowl", the object "cat" is extracted according to the label NN or NNs, then allocated its attribute "black" in the subtree. Similarly, the object "bowl" and its attribute "white" can be obtained. However, the parser may fail to extract the concepts out of the "[attribute] [object]" format. For instance, it can not process the prompt "a photo of a streetlight that is green" with dependency "green"&"streetlight", or "apples of green are in white baskets" with dependency "green"&"apples". We leave this for future work.The conventional diffusion model [30] works in two steps: (1) forward diffusion that gradually adds noise to the image x; (2) reverse diffusion that removes noise from noisy image x t step-by-step.Latent Diffusion Models (LDMs) [2] perform the denoising in the latent space. The pre-trained encoder ϕ compresses the image x to the latent z = ϕ(x), and the pre-trained decoder ψ reconstructs the latent as ψ(z) ≈ x. The forward diffusion produces the noisy latent z t for the step t = 1, ..., T . The denoising network ϵ θ is trained to remove the added noise at each step by minimizing ||ϵ θ (z t , t) -ϵ|| 2 , where z t is the noisy latent at timestep t, ϵ ∼ N (0, 1) is the added Gaussian noise. The noisy latent z T is sampled from Gaussian noise N (0, 1) during inference. Finally, the reversed latent z 0 is decoded to produce the image x = ψ(z 0 ).The proposed Magnet is applied over Stable Diffusion (SD) conditioned on text prompts. The pre-trained CLIP text encoder E maps the prompt to the text embedding c = E(P). SD appends several cross-attention layers to inject the text condition into the latent z t . The loss function of the text-image latent diffusion model can be rewritten as ||ϵ θ (z t , t, v) -ϵ|| 2 .The use of the exponential function is inspired by [18] . But in a different way, Eq. (3) that determines the strength α i , β i is based on our observation in Fig. 2 (b ) and (c) in Section 2.The formula ω = cos(G( Ppos i ), H( Ppos i )) calculates the cosine similarity between the first [EOT] embedding and the last padding embedding of the concept Ppos i . In Fig. 16 , we have conducted a statistical analysis using Numpy's histogram to bin the data. Different values ω are obtained from 19648 samples encoded by CLIP ViT-L/14. The highest counts are at the values 0.66 and 0.71. Observe that the count drops when ω < 0.6 or ω > 0.82. Intuitively, smaller ω indicates a larger deviation from the target context. Empirically, we set λ = 0.6 in Eq. ( 3) to enhance the weak binding (i.e., α i > 1 when ω i < 0.6 in e λ-ωi ). We have conducted an ablation study of the value ω in Fig. 6 . For the strength β i of the negative binding vector, we suggest a relatively slight control to avoid strong deviation when the concept number M is large, i.e., β i = 1 -ω 2 .Feature Neighbors. The candidate set S = {B 1 , ..., B R } used for the feature neighbor strategy includes R words. In practice, we gathered 614 object nouns generated from ChatGPT [13] and checked manually. We extract the word embedding c B R of each candidate object. For example, the candidate "truck" is mapped into P("truck") = {c SOT , c truck , c EOT , ...}. The embedding c truck is extracted and used in the formula d(c Br , F(E i ), Puc i ). Notice the [EOT] embedding is not used. This procedure of extracting candidates' embeddings is one-for-all, i.e., we compute 641 embeddings once for each new text encoder and save them to the local path.Semantic Neighbors. These neighbor objects are semantically related to the target object. We adopt ChatGPT [13] to predict the semantic neighbors. The instruction follows the sentence pattern of "Which objects are highly related to the word <*> ?". Optionally, the large language model BERT [48] for fill-mask is considered. We mask the object in the prompt to get its neighbors. For example, to predict the neighbor object for "brown bear", the masked prompt is composed as "brown bear and a [MASK].". We hypothesize the conjecture "and" can implicitly restrict the close relation. The first two nouns output by BERT are "wolf " and "lion", which are similar objects to "bear".Configure. All experiments are conducted on RTX 3090 in a single GPU. Our proposed Magnet is built upon SD V1.4 [23] with the pre-trained text encoder of CLIP ViT-L/14 [5] .Hyperparameters. The choice of λ = 0.6 is explained in Appendix B.3 and verified by the ablation study in Fig. 6 . We set K = 5 to conduct qualitative and quantitative experiments. We have discussed other choices of K in Appendix E.1. We generate images with 50 diffusion steps with a fixed classifier-free guidance scale of 7.5.Baselines. We compare Magnet to SD V1.4 [2] , the training-free method, Structure Diffusion [49] , and the optimization method, Attend-and-Excite [7] . Since the official Attend-and-Excite does not provide an automatic parsing process, we extract the required object words (in bold) using the Stanza's package (same to Magnet, see Appendix B.1).Datasets. We have conducted statistics on the CC-500 dataset based on the three types of classification. We find the number of valid prompts for each type are 84, 212, and 136, respectively. This data bias may lead to unfair comparisons. In this case, we randomly select 80 prompts per type and obtain 240 prompts in total.Resource. The runtime to generate an image and the required maximum GPU resources for each method are listed in Tab. 2. The data of each method is obtained by generating 200 images (randomly sampling 50 prompts from each dataset and generating 2 images per prompt). Each method is tested under the same setting to maintain fairness.We rely on human evaluation since the commonly used metrics for text-to-image synthesis are unreliable for our concern about attribute binding. We discuss three models as the automatic evaluation metrics, which are retrieval models CLIP [5] and BLIP [50] , as well as the phrase grounding model GroundingDINO [22] . Fig. 17 (a) shows the drawback of CLIP score, which computes the cosine similarity between the text and the image embeddings. Failure and success cases present relatively equal values. The [EOT] embedding suffers from attribute bias and can not measure the unnatural concept "blue apple".Similar to CLIP, the metric of BLIP score in Fig. 17 (b ) diverges from the human evaluator. Given the target prompt with multiple concepts, the image of SD (top) presents entangled attributes and objects. In this case, human evaluators indicate no instance of object existence and attribute alignment. However, BLIP text-image similarity can not align with the assessment of human evaluators. In the main paper, we adopt GrondingDINO [22] to detect the object in the generated image. However, it fails to capture the structural deviation and suffers from attribute bias. As shown in Fig. 17 (c ), the entangled concepts "bird" and "cat" are detected by GroundingDINO, which diverges from the human evaluator. Conversely, the model can not detect "gold cake". This may be attributed to the attribute bias, which we have discussed in the main paper.Additionally, we follow Attend-and-Excite [7] and compare the full prompt similarity and minimum object similarity using CLIP and BLIP. The quantitative comparison is listed in Tab. 4. Magnet shows improvement on all metrics compared to SD and Structure Diffusion. Meanwhile, we compare the text-text similarity [7] using the BLIP model for image captioning, resulting in SD (66.08), Structure Diffusion (65.71), Magnet (68.22), and Attend-and-Excite (71.22) as the highest. However, we do emphasize that the above quantitative metrics can not reflect the disentanglement of objects and attributes that we are concerned about.In conclusion, we refer to the human evaluation to ensure a fair and reliable comparison. A screenshot example of the coarse-grained comparison is given in Fig. 18 .In Fig. 19 , we have conducted an ablation study on the hyperparameter K to select neighbor objects. Note that the positive and negative vectors are estimated by each object E i itself when K = 1 as Eq. ( 1). The difference is slight if concepts in the target prompt are relatively natural. For example, in row 2, using K = 1, 3, 5 (column 2-4) can generate the correct concept "red ball" compared to "white ball" in SD. However, the results of K = 1 (column 2) in rows 3-4 present a catastrophic structure of "blue bananas". This verifies the effectiveness of the neighbor strategy. On the other hand, K in a large number can lead to inaccurate binding vectors. For example, in rows 1-2, results of K ≥ 10 are similar to SD. This can be attributed to the introduction of a multitude of unrelated objects that have an impact on the estimation accuracy. Similarly, "stickers" are indistinguishable in row 3, columns 7-8 using K = 20, 50. Figure 19 : Ablation study on the hyperparameter K. We emphasize that K = 5 may not always be the best choice because of the randomly initialized latent. For example, the result of K = 3 is more appealing than K = 5. We choose K = 5 which can stabilize the generate of unnatural concepts (e.g., "blue bananas" and "yellow stickers" can be more distinguishable in K = 5 than K = 3), as well as balance the processing time.Interestingly, when using different seeds, the most visually appealing image may not always come from the same K. For example, we subjectively prefer the result of K = 3 in row 1, but in row 2 the result of K = 5 is more appealing. This is due to the randomly initialized latent. Since Magnet's resource requirements are relatively low, we believe it is possible to use different K for the same prompt and generate images simultaneously for freedom of choice to the user.In conclusion, the reason for the use of K = 5 is the balance between synthesis quality and preprocessing time for manipulation. Here, we obtain the data of time by processing 20 prompts, i.e., adding 0.25s to SD to generate an image using K = 5. Meanwhile, our code can be improved to shorten the time, which is left for future work. In Fig. 20 and Tab. 5, we conduct an ablation study on the proposed positive and negative vectors. The object disentanglement is assessed by asking "Which image shows different objects more clearly?", and the attribute disentanglement by asking "Which image shows different attributes more clearly?". We randomly select 12 prompts from CC-500 and 20 prompts from ABC-6K, generating 25 images per prompt (800 images in total) for three settings.Both qualitative and quantitative comparisons verify the importance of both vectors. For instance, the concept "green bench" is interpreted to "green grass" when using only one type of the binding vectors. This occurs because of the entanglement of two objects. For the attribute disentanglement, using both vectors is capable of generating objects with desired attributes. Notice that the negative vector improves the object disentanglement (presents "bench" in column 5), while the positive vector improves the attribute disentanglement (presents "orange dog" in column 3). The human evaluation results are consistent with the above analysis, i.e., negative only (5.1) overpasses positive only (3.9) in terms of object disentanglement, and positive only (3.0) overpasses negative only (1.3) in terms of attribute disentanglement. In conclusion, using both vectors significantly improves text alignment.Although Magnet provides an efficient and effective way to address the attribute binding problem, we acknowledge our technique is subject to a few limitations.Fig. 21 displays the failure cases of Magnet. First, the neglect of the object (columns 1-2), may be attributed to the model's limited ability to foreground limited subjects. Second, the excessive manipulation of the object embedding leads to out-of-distribution (columns 3-4). An interesting observation is that Magnet sometimes generates images with correct concepts, but incorrect positional relations (columns 5-6). We suspect that the color layout has been determined in the early stage. In this case, Magnet maps the object to the position of the attribute in the image, rather than blending the attribute with the object. Magnet inherits the well-known issue of T2I models, presenting merged objects (columns 7-8). Finally, it is still challenging to generate an unnatural concept when the object has a strong attribute bias (columns 9-10). Additionally, Fig. 22 displays examples that Magnet generates similar images with SD. Most happen when SD has produced relatively faithful images (columns 1-2), or prompts with excessively detailed concepts (columns 3-4), as well as the generation of two unrelated concepts (columns 5-7).We consider combining Magnet with optimization-based methods to tackle the neglect of objects, e.g., the integration of Attend-and-Excite and Magnet (see Fig. 9 and Fig. 23 ). Magnet is also compatible with existing T2I controlling modules to address the inability to change spatial relationships, e.g., the integration of ControlNet [28] or layout-guidance [27] and Magnet (see Fig. 10 ). The excessive or insufficient manipulation may be addressed by improving the formula in Eq. ( 3), or simply stating the strength α i , β i manually. We leave these for our future work.Fig. 24 provides examples that Magnet improves the image quality compared to SD.In Fig. 25 , we compare Magnet to SD by visualizing the cross-attention activation.Fig. 26 provides examples of typical indoor scenes using prompts from the ABC-6K dataset. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.The same object in different prompts may have different positive and negative attributes. 


Decision: This paper investigates the attribute binding limitation in text-to-image diffusion models and introduces the Magnet approach, which enhances attribute-object alignment by interpolating object features with their designated attributes (positively) and attributes designated with other objects (negatively) specified in the prompt.

Summary Of Reasons To Publish:

1) Novel approach with interesting and in-depth analysis of the attribute binding problem

2) Well-organized and well-motivated work

3) Strong evidence of utility of proposed approach with human evaluations

Summary Of Suggested Revisions:

All the major concerns raised by the reviewers are addressed in the rebuttal to some extent. I suggest authors to incorporate the feedback on the presentation and clarity issues in the final version.

Reviews: review: paper_summary: paper_summary: The paper analyzes and improves upon the “(attribute) binding problem” in VLMs such as CLIP, and the focus is primarily on the text encoder side.
First the authors analyze how the individual text embeddings behave in a diagnostic setting when encoding a two-word text “COLOR OBJECT”. With these insights they propose to modify the original embedding of an object by calculating a “binding vector” that uses a positive attribute (the one actually in the prompt, i.e. “yellow”) and negative attributes (other colors).
They show improvements to previous methods on top of which they build such as Structure Diffusion.
strength_weakness: strength_weakness: While the method clearly improves upon previous work, it requires pre-computing or manually defining a lot of components such as the negative attribute set or the neighbor set, with additional computation of the adaptive strength. With pre-computation this is not too expensive but it feels somewhat hacked together.

The abstract mentions too many terms where it is unclear whether they are established terms and if not, the abstract is not the right place to mention them all: blended text embeddings, attribute bias, object embedding, binding vector. 

Since this papers main focus is a thorough analysis and interpretability insights, it has to be clearer with its definition, i.e.:
“The above phenomenon, which we call attribute bias, occurs when one object prefers or dislikes
a specific attribute” → refer/dislike are not well-defined.

The 4 experiments with swapping various embeddings seem very interesting but it is not motivated why we need all 4, maybe 2 cases would have been more insightful and easier for the reader to understand what’s going on?

Finally, having more quantitative statistically significant experiments and less showing of examples would have strengthened the paper's contributions further!
questions: questions: It says in the abstract “blended text embeddings as the cause of improper binding… “. Is “blended text embedding” embeddings a term people are expected to know? I work in vision-and-language and have not heard of it.

Again in abstract “we observe the attribute bias phenomenon”: was this meant to say attribute binding?

How are negative attributes exactly determined? Most things are not as easy as color.

Is there an explanation why the Structure Diffusion baseline is so bad?
Does your method differ to Stucture Diffusion only in the proposed Magnet method and nothing else? Same exact SD version, CLIP version etc.? This is important to clarify in the paper so that it is clear that your method alone leads to this improvement.

Note: Fig. 1b is very small and also low-resolution, please fix!
limitations: limitations: Yes, addressed.
paper_summary: paper_summary: The authors propose _Magnet_ to solve the attribute binding problem. (1) Initially, they specifically analyze how the improper binding problem occurs in text embeddings. By comparing embeddings for each token, they demonstrate the attribute bias phenomenon where attributes do not bind well to the object token $c_{object}$ in rare concepts like "blue apple." Additionally, through cosine similarity analysis of padding embeddings, they hypothesize that the latter padding tokens, e.g., $pad_{73}$, tend to represent prior bindings (like red apple) rather than binding the target concept (blue) with the object (apple). (2) Magnet suggests a method to slightly edit the embedding of the object to align with the desired concept, based on these observations. They demonstrate their method's efficacy through ARC-6k and CC-500 datasets.
strength_weakness: strength_weakness: Overall, I think it's a good paper. It would be better if the evaluation were a bit more solid. 

- Evaluation

I agree that a user study w.r.t performance comparison as shown in Table 1 is necessary, but for this task, it is necessary to evaluate whether each method achieved binding rather than comparing different methods. Since we don't know the precise performance of each technique, it's difficult to understand the actual performance of the Magnet and how much it has improved.

The results in Table 2 are weak. It's disappointing that the automatic results are quantitatively worse compared to attend-and-excite. While I agree that manual inspection is the most accurate, trying out a recent VLM like GPT-4o, which can discern characteristics that image encoder-based evaluations fail to capture, might be worthwhile.
questions: questions: Q1: How is the analysis in lines 64-78 and Figure 2-(a) connected to the method?

Q2: Is there no need to modify the padding? If the padding tokens positioned at the end contain prior knowledge, it seems they might also interfere with binding.

Q3: How did you perform detection using GrondingDINO in Table 2? I am curious whether you detected "blue apple" or just "apple." It might be somewhat out-of-scope, but I am also interested in how different the results are between detecting "blue apple" and "apple."

Q4: As mentioned in the limitations (Figure 20), Magnet seems relatively weak at handling positional information. I suspect this issue arises not so much from a problem with Magnet itself but rather because CLIP's embedding does not effectively bind positional information to the desired token. Is there an analysis similar to Figure 1 for positional information? It seems only color is present in the appendix. Showing this could further demonstrate, as per the title of your paper, an understanding of VLM’s function, providing clarity on what can and cannot be bound.

---

- Relevant references

Before this paper was submitted, methods for editing text tokens were proposed in papers like [1, 2]. Although these papers also deal with a different task, and attribute binding, and were not considered in the evaluation due to being on Arxiv, I recommend citing them in the camera-ready version.

[1]: Uncovering the Text Embedding in Text-to-Image Diffusion Models, https://arxiv.org/abs/2404.01154

[2]: TexSliders: Diffusion-Based Texture Editing in CLIP Space, https://arxiv.org/abs/2405.00672
limitations: limitations: The authors describe the failure cases of Magnet in the appendix. They have adequately addressed the limitations.
paper_summary: paper_summary: - This work studies how CLIP text embeddings commonly used in text-to-image diffusion models affect attributes in generated images, and how attributes can be bound to the correct objects during generation.
- There is an analysis of the (a) CLIP text encoder and how it interacts with the padding used during T2I diffusion (b) how text embeddings of different nouns relate to different attributes w.r.t to distance (c) how the previous two observations interact during diffusion-based generation.
- An algorithmic innovation — Magnet — is proposed. Magnet introduces a binding vector that can be applied to the embedding of a noun-object to bind an attribute to it so that it is faithfully applied during generation.
- There is a human evaluation (amongst other evaluations) which shows that Magnet improves attribute-object bindings.
strength_weakness: strength_weakness: I had great difficulty understanding most of the figures. Even after understanding the method, I cannot understand what Fig 3. is showing and how it relates to the method. Similarly, I found the equations and the notations cumbersome, illegible, and frustrating. 

I am honestly unsure if I understand the method, because the writing is very unclear (though well organized) and the notation is completely overwrought. 

One minor weakness is that the analysis is highly specific to the CLIP text encoder. I don't know how much of a weakness this is practically since the CLIP text encoder is a de-facto standard, but I am writing it here anyway for completeness.
questions: questions: - Have at least one *simple* figure that explains the *intuition* behind the approach.
- Rewrite 3.1 to be more clear or at least provide examples of all of the $\mathcal{P}$-terms, I am not even sure the notation is correct here.
- More generally, provide a comprehensible explanation of the method so I can confirm my understanding is correct. I'm uncomfortable providing a higher score otherwise, though I would like to, given the amount of work that seems to have gone into the paper.
limitations: limitations: Not applicable.
paper_summary: paper_summary: The paper investigates the text embedding representation of text-to-image models in the context of stable diffusion. In particular, the authors find that object features often bind to their commonly associated attributes and propose the Magnet approach that interpolates object features with their designated attributes (positively) and attributes designated with other objects (negatively) specified in the prompt. To determine the strength of the interpolation, the authors leverage the similarity between the EOT and the last PAD token. In addition, for any object word, the authors retrieve their neighboring words based on feature and semantic similarity to further twist the binding vectors, resulting in enhanced concept disentanglement.
strength_weakness: strength_weakness: 1. The scope of the work is limited -- it only covers fixing attribute binding problems in text-to-image generation, one downstream task that is commonly encountered in compositional generation [1].
2. The method is model-specific. While it alleviates text-to-image models with a CLIP-based text encoder to a significant degree, it is unknown if this method also improves text-to-image models that use multiple text encoders [2] or different text encoders [3].
3. I find the neighbor finding procedure a bit awkward. For example, the authors have to manually gather 614 object nouns as feature neighbors and have to prompt GPT-3 to gather semantic neighbors, limiting the applicability of the work toward a completely automatic pipeline (e.g., without manual curation or LLM-assisted selection).
  
[1] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation  
[2] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis  
[3] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
questions: questions: 1. Could authors add reference to Stanza’s dependency parsing module in line 99?
2. Given attribute binding is a core task in compositional text-to-image generation, could authors also discuss more recent and/or relevant in related works, such as [4, 5, 6, 7]?
3. Can authors also evaluate on image quality metrics such as FID [9] and compare with other methods?
4. I wonder if authors could compare Magnet to this concurrent work [8]? As its preprint came out this March, there's no need to experimentally compare and contrast. Some discussions would be appreciated.

[4] Training-Free Layout Control with Cross-Attention Guidance  
[5] TokenCompose: Text-to-Image Diffusion with Token-level Supervision  
[6] RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models  
[7] Compositional Visual Generation with Composable Diffusion Models  
[8] Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions  
[9] GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
limitations: limitations: The paper discussed limitations in missing objects, possibilities of over- and under-manipulation as well as failures in correcting position relationships of the objects with specific examples, which are justifiable given the scope of their method.

Comments: title: Thanks for your response
Response to Comment about Flexibility of Magnet
Response to author's rebuttal
comment: Thank you for your effort. Unfortunately, your explanation in the comments did not help. I strongly encourage you to choose different notation. I will keep my rating; I think there is a clear technical contribution in this paper, but the comment did not clear anything up for me. I have put my confidence at a 2; the AC can disregard my review at their discretion.
Thank you to the authors for their detailed response. I have carefully considered the points raised.

> [W1] "Our comparison experiment has been designed to evaluate whether each method achieved binding. In Table 1, attribute disentanglement is assessed by asking, 'Which image shows different attributes more clearly?' Only the method achieving the best binding will be voted for; otherwise, evaluators tend to choose no winner."

While I understand the authors' approach, I still believe that this method inherently involves a comparison. Evaluating A vs. B seems likely to yield different insights than evaluating A and B separately. The authors suggest that both approaches could lead to similar results, but I respectfully disagree, as I believe pairwise comparisons and individual method evaluations may not produce the same outcomes.

> [W2] "Since there is no available API at this moment, we cannot evaluate Magnet with GPT-4o. But we will definitely try out GPT-4o whenever possible."

I appreciate the authors' intent to evaluate with GPT-4o when possible. However, isn't the GPT-4o API already available? 

> [Q3] "This comparison shows that GroundingDINO is not reliable for measuring attribute alignment."

I understand this concern, and I recognize the challenges it presents. It is unfortunate that there currently isn't a more robust evaluation method beyond manual inspection. I had hoped that GPT-4o might offer some supplementary capabilities in this area.

Given these unresolved concerns regarding the evaluation, I will maintain my score as a weak accept.
Thanks for your response! Given the authors' response, it would be interesting to explore Magnet with a broader scope of text encoding strategies in the future. Regardless, given that mitigating attribute binding issues from the text encoder (as opposed to the diffusion backbone) is under-explored and the authors have done a great job proposing an effective method with some interesting analysis, I would maintain my current rating and confidence and lean toward acceptance.
Thank you for your thoughtful follow-up and your interest in the flexibility of Magnet.

When applying Magnet to SDXL, we did not modify any hyperparameters or formulas, keeping all settings consistent with the main paper (e.g., $K=5$, $\lambda=0.6$). The adaptive strength formula (Eq. 3) is designed based on our analysis of the CLIP encoder and naturally adapts to SDXL.

However, the T5 encoder in PixArt operates differently by modeling bidirectional context. Consequently, it is necessary to conduct further analysis of T5 and redesign the strength formula. Due to the time constraints of the rebuttal process, we were unable to conduct a detailed analysis and therefore used fixed strengths ($\alpha=2$, $\beta=0.5$) for all objects. We also found that $K=10$ is more robust for PixArt.

We will incorporate these considerations in future work to further demonstrate the flexibility and impact of our method.
Thank you for the explanations and the new PDF! It is good to see you provided new results and engaged with all the questions. I still think my score of 6 is appropriate and think this paper is a weak accept.
I would like to thank the authors for the rebuttal -- it has addressed all of my concerns to a considerable extent. I'm also impressed that Magnet works just as well on different types of models (e.g., SD 2.1, SDXL, PixArt) and/or for different constrained generation (e.g., Layout-Guidance, ControlNet) based on the provided rebuttal file.

Since SDXL and PixArt use different text encoding strategies (e.g., multimodal text encoder or different text encoder), I wonder if the authors could discuss any necessary modifications (e.g., hyperparameters, Magnet formula, etc) to adapt your original setting for the CLIP encoder to work for two encoders (i.e., SDXL) or a T5 encoder (PixArt) to achieve considerable improvement in attribute binding? This would help me better assess the flexibility of your method as well as the scope of impact. Thanks!