Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function
Authors: Chenyi Zhuang
Ying Hu
Pan Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Generation
Diffusion Models
Vision-Language Models
Attribute Binding
Primary area: diffusion_based_models

Abstract: Text-to-image diffusion models particularly Stable Diffusion, have revolutionized the field of computer vision. However, the synthesis quality often deteriorates when asked to generate images that faithfully represent complex prompts involving multiple attributes and objects. While previous studies suggest that blended text embeddings lead to improper attribute binding, few have explored this in depth. In this work, we critically examine the limitations of the CLIP text encoder in understanding attributes and investigate how this affects diffusion models. We discern a phenomenon of attribute bias in the text space and highlight a contextual issue in padding embeddings that entangle different concepts. We propose Magnet, a novel training-free approach to tackle the attribute binding problem. We introduce positive and negative binding vectors to enhance disentanglement, further with a neighbor strategy to increase accuracy. Extensive experiments show that Magnet significantly improves synthesis quality and binding accuracy with negligible computational cost, enabling the generation of unconventional and unnatural concepts.

Magnet: We Never Know How Text-to-lmage Diffusion Models Work, Until We Learn How Vision-Language Models Function
Chenyi Zhuang'
Ying Hu'
Pan Gao
Nanjing University Aeronautics and Astromautics
Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education {chenyi .zuangg ying, hun,ann.ee gan. (aao)nuaa.ccemmmm edu..aaaeuu.... cn
Abstract
Text-to-image diffusion models particularly Stable Diffusion, have revolutionized the field of computer vision. However, the synthesis quality often deteriorates when asked generate images that faithfully represent complex prompts involving multiple attributes and objects. While previous studies suggest that blended text embeddings lead improper attribute binding, few have explored this in depth. In this work, we critically examine the limitations of the CLIP text encoder in understanding attributes and investigate how this affects diffusion models. We discern phenomenon of attribute bias in the text space and highlight a contextual issue in padding embeddings that entangle different concepts. propose Magnet, a novel training-free approach to tackle the attribute binding problem. We introduce positive and negative binding vectors enhance disentanglement, further with a neighbor strategy to increase accuracy. Extensive experiments show that Magnet significantly improves synthesis quality and binding accuracy with negligible com- putational cost, enabling the generation of unconventional and unnatural concepts. Code is available at https: /tte:///gtttttt com II/ MultMMulle timediaa a Lab/MagMa Magnet.
1 Introduction
Recently, Text-to-Image (T2I) diffusion models [1, 2, 3, 4] have drawn considerable attention from both the research community and industry. Among these models, Stable Diffusion (SD) [2] uses the CLIP text encoder [5] to encode the given prompt, which is relatively lightweight than other diffusion models that adopt T5 [6]. Unfortunately, generating text-aligned images is still challenging for SD and requires multiple runs to achieve the desirable results. Several works [7, 8, 9, 10] have pointed out that the blended context by the CLIP text encoder causes improper binding. However, few have analyzed in detail how the text encoder affects the generation of the diffusion model.
Step back and refocus on the CLIP text encoder- an integral part of the Vision- Language Model (VLM). Prior studies [11, 12] have observed VLMS lacking compositional understanding and investi- gated them image-to-text retrieval benchmarks. In this work, we are motivated to answer how the text encoder understands attribute, and how it affects the attribute binding of T2I diffusion models. Upon closer inspection, observe phenomenon of attribute bias and discern contextual problem in padding embeddings, leading to a well-known T2I issue-concen bleeding [10].
Based on the observation, we introduce the binding vector, which is applied to the text embedding of each object. With positive and negative binding vectors, each object can pull target attributes and push unrelated attributes to distinguish them from each other. We further introduce neighbor strategy ensure an accurate estimation of the binding vector. Our manipulation is performed strictly in the textual space, without training, fine-tuning, or additional datasets and inputs. Overall, the main
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Pronpe: "Iestec chair"
Prompp: "laskl shoop'
Pronpe: TLakkl apple" Capio
char
apple
banona
dsclncc
Sabrrr
Word kno
Figure Analysis of the CLIP text encoder for understanding attributes. There a discrepancy between the word and [EOT] embeddings of the attribute bias on different objects.
contributions of this work are: (1) We highlight a contextual issue in the text encoder, which impacts diffusion-based image generation. (2) We propose a novel training-free method to address the binding problem. (3) Extensive experiments are conducted to verify the effectiveness of Magnet.
2 Analysis of the CLIP text encoder and the diffusion model
In this section, we aim to recognize the pattern of the CLIP text encoder for understanding attributes, then go deep into the diffusion model to analyze the underlying reason for improper attribute binding.
The CLIP text encoder uses the causal mask mechanism to produce unidirectional context, i.e., each word can only consider the words to their left [13, 14]. It performs contrastive learning on a specific End of Text ([EOT) embedding without word-level supervision, while prior studies [15, 16, 17] suggest each word has a semantic effect on the generated image. In this case, we categorize two types of embedding as word and [EOT] for fine-grained analysis. Consider a prompt encoded to text embeddings e = {CSOT, Cpis Cpy CFOT} consisting N word embeddings. Different from the [EOT] embedding сgOr, word embeddings Gp1 -.: Cpx are unsupervised during training. We skip the embedding of Start of Text ([SOT)) csoT for simplicity. To study how the two types of embedding understand attributes, we select 60 familiar objects and common colors obtain text embeddings e = {Cobject, CЕOT] } and c' = {Cootor Cobjects EFOT) i.c., without and with the color context, respectively. We aim to compare: (1) contextualized word embedding dobject with the original Cobject (2) contextualized [EOT] embedding ĆεOT with the original CEOT-
How do two types of embedding understand attributes? Fig. 1 (a) compares the Euclidean distance and the cosine similarity between embeddings with and without the color context. The pattern varies between cases or objects. As to the word embedding. the similarity curve of "chair" is relatively smooth, but that of "sheep" has large gap between "black" and "white". " Similarly, "blue apple" diverges from others. The above phenomenon, which we call attribute bias, describes the tendency of an object to favor certain attributes over others. We compare the attribute bias per object for two embedding types in Fig. (b). the object has a natural composition in human knowledge, presents serious attribute bias (e.g.. "yellow banana" v.s. "blue banana"), Meanwhile, the word embedding is more volatile than the [EOT] embedding. which shows less dramatic change. Our hypothesis is the absence of word-level supervision during CLIP training. well as the bags-of-words behavior of VLMs [11, 12]: the [EOT] embedding is trying to remember all important words the given prompt, including adjectives and nouns. However, it leads to inaccurate textual representation of the [EOT] embedding, affecting the interaction between the image latent and semantic word embeddings. We have provided more analysis and examples in Appendix A.1 (see Fig. 12, 13).
How do two types of embedding affect SD? practice, SD pads the input prompt to a fixed length == 77 using additional [EOT] embeddings (i.e., the padding token is initialized by the symbol (EOT)), denoted as Cpad, where 7 1..., I, N 22. To study the attribute binding during generation, modify the above two text embeddings and c' with L - N 2 padding token embeddings, then design 4 fine-grained cases: (1) standard generation conditioned on where all text embeddings contain the color context; (2) only replace Lorject with Cobject eliminate the context the word embedding; (3) only replace e'EOT with CEOT, as well as padding embeddings cpad: with Cpadi (4) climinate the color context word, [EOT], and padding embeddings. Fig. 2 (a) presents examples,

---

rodd chair
block sheep
fube apple
(
Faua Caias clace Soue
(2) thass Gareng Clasos S/y
(2) daauss Swer Sanse
(b) dlaw Plpple classr Paust
(2) daaw caasn Sutt
(3)
Frets Seate Sipcr Spad
Sest Sdwat Raoon Spad
(s) daasi Sabees
Soatt Satapy 9pot
(3) dawg Slevio Caors Spod
Aeee Seppl vasoos Spad
(a) Fine-grained 4 cases
Coivre Smikry
Con no Eritentt
bluu
apple
EOT
Nuue
Férd
frown
car
ane eactes tomate
Padding Broan Indix
goad
puulys
parko
peddys
Padddng Tilenn Index
poad
podas
padly
pade
(b) Single-concepi context issue
(c) Multi-concept context issue
Figure 2: (a) Fine- Fine-grained study through our designed embedding swapping experiment. The context issue in padding embeddings for (b) single-concept scenario, and (c) multi-concept scenario.
including a natural concept "red chair", unnatural concepts "black sheep" and "blue apple". Cases 1-2 can be observed in all examples with less realistic and painting-like images. Note that the used SD trained to generate photo-realistie images. These results indicate deviation from the learned distribution. Conversely, cases 3-4 produce realistic images, but neglect the target color when the concept unnatural. This suggests that the context in [EOT] and padding embeddings do have a significant impact attribute generation. In Appendix A.2, describe the above 4 cases in detail and provide more examples, as well as additional cases (see Fig. 14).
Why improper binding? We posit that the first [EOT] has a close-knit context with word embeddings. The padding embedding, however, may deviate due the causal attention mechanism. We then compute the cosine similarity between [EOT] and each padding embedding 1,.., L, and dive into their cross-attention activations on single- and multi-concept scenarios. Fig. (b) studies prompt with only one object. The curve drops more drastically on the unnatural concept "blue upple" than the natural one "red chair". Cross-attention shows that "upple" overlaps with padding embeddings (e.g., pad7s) rather than the [EOT] embedding. It is like these padding embeddings have forgotten part the context remembered in the first [EOT]. Without interference from other concepts, the inaccurate context in these padding embeddings causes diitiiiiiitiiitinnn, or the binding of another attriiute if the model learns an underlying bias in the training dataset (e.g.. "apple" prefers "red"). Fig. (c) further studies the context issue multiple concepts. For two objects "bird" and "car". even though the activations their word embeddings do not overlap, cross coss-attention shows obvious entanglement these padding embeddings. This multi-concept context issue in padding embeddings, i.e. entangled concept representations, explains color leakage and object sticking. We refer the reader Appendix A.3 for comprehensive analysis of this context issue.
How to disentangle different concepts? Prior studies [8, 18] prove that these padding embeddings essential for image quality and can not simply be removed. Also, it is impossible to manipulate one single concept these padding embeddings due to their entangled property. On the other hand, these naturally separated word embeddings show editability. For instance, Fig. (a) "black sheep" from case 2 to case changes only the word embedding of "sheep" while encouraging the desired attribute. We inspired to manipulate the word embedding of each object, therefore strengthening the binding within each concept and enhancing the distinction between concepts.
Magnet: disentangling concepts with the binding vector
Our approach is based two key observations: the context issue of the padding embedding, and the controllability of the word embedding. We introduce the binding vector, which can be applied on the object embedding attract the target attribute and repulse other attributes, analogous a Magnet.
Preliminary: Given a prompt P, we use Stanza's dependency parsing module [19] to extract each concept, denoted A&&E, where E is the object word and its target attribute A. The dependency

---

= (Ba... Ba)
[SOT]
guean A
Sky
Froen
FiR ((9)))
Text Butaderg
KANN A.
bear A
B
wall
e fed
bosssn 4
Eq. (4)
browm Ag
[BOT]
D Bon
appley?
apple
Big.
[SOT]
[sOr]]
[PAD)
graant A
[SOMT]
Iessen 4
Unet
woir p
wott F
wolf 2
Magnet
9
Generation pipeline
Adaptive strength
(c) Neighbor-guided vector estimation
Figure 3: Overview of the proposed Magnet. We manipulate the object embedding with the positive and negative binding vectors, which are estimated with the guidance of neighbor objects.
set with M concepts is D = {A AM&EEM Detailed dependency extraction is given in Appendix B.1. Then the pre-trained CLIP text encoder applied map to the text embed- ding c = (CSOT, CA r CA CE CEOT; Cpady..... Cpadz- -22 linking words. We treat the diffusion model black box and leave its background in Appendix B.2.
J. For simplicity, we omit the
For each object E, in with the word embedding CE., we aim to estimate its positive binding vector uf to pull the target attribute A,, and its negative binding vector oy oe to push other attributes.
3.1 Apply the binding vector on the object embedding
Instinctively, the binding vector can be estimated by the object itself. specific, we compose new concepts out of the current context of P. which are: (1) unconditional concepts, P.r { where Ø is a blank text -"; (2) positive concepts, P = &A,&E,) J; (3) negative concepts, P.100 &A&EE]
j.j + i}. The positive and negative binding vectors are estimated by:
up F(E,, P. F(E,, PIC)
U neg F(E,, P.rra) - F(E,, P.O))
(2)
where F()) extracts the word embedding of the object E, in specific decontextualized prompt. Note each object has M negative concepts, resulting in M negative binding vectors punish all unrelated attributes Aj.j I,., M.j j شf i. Note that these positive and negative attributes are prompt-dependent We introduce the unconditional concept as a pivot to avoid the need for manual definition or semantic contrast between positive and negative attributes.
Based on our analysis the context issue the padding embedding Section 2, we hypothesize an association between the attribute bias and the strength. Intuitively, unnatural concepts (c.g., "blue banana") ") suffer more attribute bias and their padding embeddings are more tend to forget the concept. In this case, we need to manipulate the word embedding significantly to ensure strong binding. We introduce the adaptive strength of the binding vector for each object E;:
ag A- B
w where
W, cos(G ((PPos))
where ((.), H(-) extract the first (EOT) embedding and the last padding embedding in text embed- dings E(PPos), respectively. is positive constant. Please refer to Appendix B.3 for the inspiration of the formula, and the statistical analysis for the choice of the hyperparameter X.
Finally, the object embedding cæ. in the initial text embeddings E(P) is modified by:
čg CE. + aq t0e 3 v.
3.2 Neighbor-guided vector estimation
In practice, we find that using a single object to estimate the binding vector can be inaccurate and fail to disentangle concepts (see Fig. 7 and Fig. 19). In this case, we introduce the neighbor strategy
Thh same object in different prompts may have different positive and negative attributes.

---

to ensure an accurate estimation. These neighbor objects should have similar representations to the target object in the learned textual space. We define a candidate set = {Bi.... BR} with R objects that has pre-processed to {CB, ; .... CBR J. which is the collection of the word embedding cn. in E (B,) (esor,CEs CEOT,
I,.... R. The top- neighbor objects for the target object E; are determined by d(cB,,;(E,, P,,)), where B, E S, d(-) denotes the cosine similarity.
In Appendix B.4, we describe this neighbor strategy in detail, and further discuss a way to predict semantic neighbors using pre-trained large language models.
With the selected neighbors of the target object E,, denoted { (B 1K 1k-1 we compose the unconditional concepts Pr {Ø& B positive concepts Phass [A, ). and negative concepts Pius
{AA& = 1,... M,j # i}. The estimation of the binding vector is then rewritten as:
(F(B (1) F .P F(BB)) )K())
(5)
K
K
neg 0/
(F(B)) , P2 F(B(,) Pr
(6)
3.3 Overall workflow
Fig. 3 depicts the workflow of Magnet. The target text embedding è can be obtained after replacing all object embeddings Cg, with & i I.... ... To generate the image, the pre-trained U-Net denoises the latent 4t-1 Eæ(z,, é), where timesteps t T,.... 1. We set the hyperparameters 0.6, K 5. Please refer to Appendix C for implementation details.
Experiments
4.1 Datasets
We evaluate our proposed Magnet on two existing benchmarks:
(1) Attribute Binding Contrast set (ABC-6K) [8]. This dataset consists of natural compositional prompts from MS-CÓCO [20], each prompt includes at least two concepts (e.g., "a bathroom with tan sink and white toiler", "a brown cow standing in lush green field"). We randomly sample 600 prompts from this dataset and generate 5 images per prompt to compare all methods.
(2) Concept Conjunction 500 (CC-500) [8]. The dataset contains prompts that conjunct two concepts, each with one color attribute. Following [7]. objects are divided into two types: living (i.e., animals and plants) and other non-living nouns. Prompts type is categorized into (1) two living objects, (2) one living object and one non-living object, and (3) two non non-living objects. We adopt 80 prompts for each case to avoid bias and maintain fairness. In total, we have used 240 prompts and generated 10 images for each prompt to compare all methods.
Both datasets are augmented using contrast settings [21]. The position of attribute words for different objects is swapped (e.g., "a red chair and blue cup" <> "a blue chair and red cup").
4.2 Metrics
We mainly rely on human evaluation since the common metrics (c.g. CLIP text-image similarity) are unreliable for assessing attribute binding, which is discussed in Appendix D.
Coarse-grained comparison. We assess the generated image for image quality and concept disentan- glement on two adopted datasets. To measure image quality, human evaluators were asked "Which image is more realistic or visually appealing?". The evaluation of concept disentanglement is divided into two types: (1) object disentanglement by asking "Which image shows different objects more clearly?": (2) attribute disentanglement by asking "Which image shows different attributes more clearly?". If all images are equally good or bad, evaluators can indicate "no winner". We randomly sample one image for each prompt from ABC-6K and two images for each prompt from CC-500 to conduct this coarse- grained comparison.
5

---

Table 1: Coarse- gaarse-graindddeee comparison on the ABC-6k and CC-500 datasets for image quality, object disentanglement, and attribute disentanglement. Values are normalized to sum to 100.
ABC-6K
CC-500
Image Quality
Disentanglement Attribute
Image Quality
Disentanglement Attribute
Object
Object
Magnet (Ours)
26.57
25.71
27.14
25.43
24.86
29.43
Attend-and-Eiet
15.43
21.43
19.71
22.86
26.29
18.57
Structure Diffusion
12.28
7.14
10,29
12.29
6.86
11.14
Stable Diffusion
10.29
6.57
8.57
11.14
7.71
13.42
No Winner
35.43
39.15
34.29
28.28
34.28
27.44
Table 2: Fine-grained comparison on the CC-500 dataset. For reference, we provide the average confidence (Conf.) of GroundingDINO [22] to detect the object (Det.). Manual evaluation concerns the object existence (Obj.) and the attribute alignment (Attr.).
Automatic
Manual
Runtime
Memory Usage
Method
Det.
Conf.
Obj.
Attr.
(s)
(GB)
Stable Diffusion
71.5
56.4
65.8
59.1
6.62
6.1
Structure Diffusion
72.1
56.0
64.0
63.9
7.94 (+20.0%)
7.0 (+14.7%)
Attend-and- Excite
84.3
62.6
84.6
66.2
13.4 (+102.4%)
15.6 (+155.7%)
Magnet (Ours)
76.5
59.8
68.6
74.0
6.81 (+2.9%)
6,5 (+6.5%)
Fine-grained comparison. This comparison is conducted on the CC-500 dataset based on two key criteria: (1) object existence, counting the target objects in the generated images; (2) attribute alignment, concerning the correct binding between the object and its attribute. We ask annotators to identify the object mentioned in the prompt per generated image. Take prompt "a red car and yellow cat" as an example, each image will be indicated the number two (show both objects), one (show either "car" or "cat"), zero (no distinct object). Attribute alignment is assessed by counting whether the generated object presents the desired attribute (maximum to the number of the generated objects). All generated images CC-500 used for this fine-grained comparison. In addition, we adopt the phrase grounding model GrondingDINO [22] to detect the target objects automatically. Note that this automatic detection can not reflect the proper binding.
4.3 Quantitative comparison
Coarse-grained comparison. In Tab, 1, we present the human evaluation results of Magnet compared to three baseline methods: SD V1.4 [23], Structure Diffusson [8] and Attend-and-Excitt [7]. Note that Magnet and Structure Diffusion are both training- The ABC-6K benchmark has more complicated and challenging prompts. In this case, all methods may fail to include all objects and attributes, resulting in a higher number of no winner. Overall, Magnet achieves the best scores in terms of image quality and attribute disentanglement on both datasets.
Fine-grained comparison. As shown in Tab. 2, Magnet alleviates the missing problem more than Structure Diffusion on both automatic and manual evaluation, with 3.8% (Det.) and 4.6% (Obj.) improvement. We are inferior to the optimization method, Attend-and-Excin in object existence. In attribute alignment (Manual Attr.), Magnet outperforms all baseline methods. In addition, compare the runtime and memory used for generation. The data obtained by generating 100 prompts each with two images. Obviously, Attend-and-En -xectte requires more resources which affects efficiency. Conversely, Magnet only adds 2.9% to runtime and 6.5% to memory.
Evaluation on image quality metric. We also evaluate Magnet on the commonly used metric FID [24] for two SD versions (V1.4 [23] and V2.1 [25]). We follow the standard evaluation process and generate 10k images from randomly sampled MS-COCO [20] captions. SD V1.4 gets 19.04, with Magnet 18.92; SD V2.1 gets 19.76, with Magnet 19.20, the lower the better. This shows that Magnet will not deteriorate the image quality while improving the text alignment.

---

hot dog with tries yullon plate on aod lile
pink cake with whiie rowes on sibser plate
some bise banamas with Intle sallion stickers them
red chade and yellone dog
sree apple and brown whoep
Atterd
Mtguett
Figure 4: Qualitative comparison using prompts from ABC-6K and CC-500 datasets. For each prompt, we show the image generated by each method under the same seed.
Jame, stasa fire hydrant sits in zod grass
selllme shelf with bunches of Line surnle hananas placed together
two green bears nost roch other on patch of keunca grass
axsllan apple and rnd bonanas
bing strawberry on re plate
Stile Dillisus
Affindd
Miggett (Dara)
Figure 5: Prompts with unnatural concepts. Baselines generate exchanged colors (row I) or unwanted artifacts (row 2) while Magnet demonstrates the anti-prior ability with high-quality outputs.
4.4 Qualitative comparison
Fig. 4 shows the qualitative comparison of the ABC-6K and CC-500 datasets. The results demonstrate that baselines suffer from the entanglement of objects and attributes.
Object entanglement includes the neglect of the object sticking structures. In columns 1-2, baselines struggle to be faithful to the complex prompt with 4 objects, missing "fries" or "tile". In columns 5-6, the objects "banana" and "stickers" are indistigguishable. Similarly, SD presents blended objects "dog" and "chair" in columns 7-8 and neglects the target object "green apple" in columns 9-10. Note that the results of Structure Diffusion resemble that of SD. On the other hand, the optimization of Attend-and-Excin encourages the attendance of objects but leads to out-of-distribution results, showing strong artifacts (e.g.. "green apple" in columns 9-10).
Attribute entanglement includes the generation of incorrect attributes or the lcakage of attributes. For instance, for the prompt "a pink cake with white roses silver plate" with three colors in columns 3-4, SD and Structure Diffusion generate "white cake" and "pink roses". columns 7-8, they generate "chair" with mixed colors "yellow" and "red". On the other hand, Attend-and-Eas Excite may produce less aesthetic images, which can be attributed to the over-optimized image latent.
Notice that baselines fail to produce unnatural concepts like "blue banana" in columns 5-6 in Fig. Instead, they generate "yellow banana", which natural concept learned as the prior knowledge. Conversely, Magnet is capable of disentangling different concepts and hence generating unnatural concepts, which we call the anti-prior ability. Fig. 5 displays the results on prompts with anti-prior concepts. We skip Structure Diffusion for its limited improvement over SD.
7

---

Stable
Diffusion
= 0.0
À 1.0
À = 0.6 (Ours)
cake
rOaC4
plate
cake
ToR04
plate
cake
Poac4
plate
cake
PO864
plaae
Figure 6: Ablation study on the hyperparameter À given the prompt "a pink cake with white roses on silver plate". A small value of À can not well disentangle different concepts, while large value causes artifacts in the generated image (best viewed zoomed in). We empirically set À = 0.6.
red chair and blue cup
bloe apple and green backpack
Table 3: Ablation study. Human evalua- tors were asked to indicate which image can better separate attributes objects.
Disentanglement Attribute
Object
w/o neughoor
neighbrr
w/o neigh bor (b) object disentanglement
neighoor
(a) attribute disentanglement
w/ neighbor
27.1
28.6
w/o neighbor
9.1
6.4
Figure 7: Ablation study. The neighbor strategy im- proves the binding vector estimation, separating dif- ferent attributes ("cup" is purely "blue") and objects ("backpack" and "apple" are distinguishable).
Stable Diffusion
2.1
No winner
61.5
62.9
4.5 Ablation study
Hy Hyperparameter x. We study the effect of À in Fig. When setting À = 0, a, B are still positive numbers but the manipulation is in relatively low strength. In this case, concepts are still entangled: "roses" appear in shades of "white" and "pink". When setting À = the result presents artifacts: distorted "plate" and watermarked background. We find using À = 0.6 can achieve the balance between concept disentanglement and image quality based on the statistic analysis Fig. 16.
Selection strategy of the neighbor strategy. The effectiveness of the neighbor strategy is shown in Fig. 7. The neighbors improve the estimation accuracy and the disentanglement of concepts. In Tab. 3, we ask human evaluators to evaluate both settings using the disentanglement criteria. Evaluators indicate the generated images using the neighbor strategy more disentanglement. This verifies the effectiveness of the reighbor-guider vector estimation.
Effectiveness of the binding vector. In Fig. 8, verify the effectiveness of the binding vector by manually changing a, 3 instead of adaptively calculating by Eq. (3). The value of a, 3 changed from positive to negative shows swapped binding between objects and attributes. This is because that the context problem in padding embeddings has caused the entanglement of concepts. Our proposed binding vectors improve the discrimination between objects and lead designated attributes.
We have conducted additional ablation experiments for the hyperparameter K (Appendix E. Fig. 19), and the importance of using both positive and negative binding vectors (Appendix E.2, Fig. 20).
4.6 Extensions
Incorporate with optimization- methods. Manipulated in the textual space, Magnet can be readily integrated with Attend-and -n-id--nn--xiiteeee. Fig. 9 (a) compares the optimization loss of Attend-and- Excite with and without Magnet. The loss can start at a lower value with Magnet to strengthen the distinction between concepts. Fig. 9 (b) shows vanilla
with strong artifacts or inaccurate colors, which should be attributed to the entangled concept representations in padding embeddings. More examples displayed in Fig. 23 in the Appendix.
Different text encoders. Fig. 10 (a) and (b), we assess Magnet on three T2I models with different text encoders to SD V1.4. Specifically, SD V2.1 [25] adopts CLIP VIT-H/14, SDXL [10] combines

---

a yellow (4a) towel (Bi) and white (4g) bowl (Bg)
01,81 og. P3
yellown
towel
white
bow!
01,, 81 -1
yellow
towel
white
bowl
crg 8g
Figure 8: Ablation study on the effectiveness the binding vector.
1.00
green bench and a yellow cal
a blue apple and green vase
Artend-and-Excite Magnet
3.75
2.50
3.2
3.00
Iteration Incex
21
Attend- Sooctee
+Maggnst
Attend-and--iwcie
+Magret
(a) Loss Comparison
(b) Qualitative Comparison
Figure 9: Magnet can be combined with the optimization method, Attend-and-Excin [7]. (a) Magnet
improves the loss during optimization. (b) Magnet improves the disentanglement of concepts.
multiple CLIPP text encoders, and PixArt [26] uses the T5 encoder [6]. We use the same setting of all hyperparameters and equations for all CLIP-based models while using fixed strength for Art The redesign of the strength formula for the adaptation of T5 is matter for future work.
Incorporate with T2I controlling modules. In Fig. 10 (c) and (d), we investigate the plug-and-play nature of Magnet. Magnet shows compatibility when integrated with existing controlling modules: (1) layout-guidance [27], which constrains the image layout by bounding boxes and intervenes cross-attention layer, and (2) ControlNet [28] conditioned on Depth Map [29] to add spatial control.
Image editing. In Fig. 11, compare the image editing ability of Magnet to Coompt-to-Promen (P2P) [15], which edits the generated image by manipulating the cross -ross-attention layers. Given the source prompt "a "street". column 1, Magnet applies a positive binding vector yo (here, the strength a is stated manually) the word embedding CE... toward the attribute "old". With control of the attention maps, Magnet surprisingly edits the image with fewer changes in the background than P2P.
the side of the street",
aim to change the attribute of the object "car" or
Related work
Text-to-image diffusion models. Diffusion models that [30] pioneered, have emerged with great
improvement in both unconditional [31, 32] or conditional [28, 33] image generation, together with the advance in synthesis quality [34, 35] and sampling speed [36, 37, 38]. However, the semantic flaw of the text encoder affects the performance of the diffusion models [7, 10, 391. In this work, discern the attribute bias and the context issue, providing novel insights about attribute binding.
Attribute binding. The binding problem occurs when the model blends improper concepts. To tackle complicated prompts, [9] collaborates different pre-trained diffusion models. [8] suggests word embeddings with blended context and manipulate cross-attention features. In contrast, we highlight the entanglement of the padding embedding and modify solely the text embedding. [7] optimizes the latent guarantee the attendance of each object. Yet, the optimization may lead to outtof-distribution and require more resources to generate images. Other works [40, 41, 42] introduce layout constraints the attention layers. Magnet differs from the above approaches in that it can be executed entirely in the textual space. This distinguishes it as more efficient solution.
noteworthy that line of works [15, 43] achieves image editing on specific visual aspects. However, none have gone as far as this paper in exploring the contextual influences on SD from the perspective of text embedding. Most are subject to subset of attributes (e.g., texture [44]), control the global object rather than fine-grained attributes [45, 46), or depend on predefined text pair [47], requiring learning process or additional datasets. Conversely, our method enhances binding towards arbitrary attributes without the need for inputs to the standard pipeline.

---

backpack
lons yau hydrant grawn
lid etiiing in
foser pot
Magnec
stoco grass
amall cuctus with happy face the Sabura desert
Besnding Busa
Inyout guuddance
a Audyy sofa with leather pillow on
suallas teilyy bear with white Bass
-Maguet
SDXL
-Magnet
PixArt
-Magned
Controlert
+Magnet
Figure 10: Magnet can be integrated into other T2I models and with existing controlling modules.
old crushed
flooed street"
forest street"
Source Prompi: "a cear on the side of
Prompe-oo -o- Prompt
Magnoe (Ous))
Stable Diffusion
Figure 11: Image editing comparison using prompts from Compt-to-Promph [15].
Limitations
While we have demonstrated improvement subject to few limitations (see Fig. 21). First, still suffers from the missing problem. In some cases, the manipulation may be overstrength and cause artifacts. An interesting phenomenon that Magnet generates the correct concepts while rendering enrors in positional relations. Finally, it is still challenging to generate an unnatural concept when the object strongly biased towards one specific attribute. (e.g., "broccoli"). We have described the limitations of Magnet in detail in Appendix F.
synthesis quality and text alignment, Magnet is still
7 Conclusion
In this work, we propose a novel training-free method, Magnet, to tackle the attribute binding issue. First, we conduct fine-grained analysis of the CLIP text encoder. We observe the phenomenon of attribute bias and point out the context issue of padding embeddings, where the representations of different concepts are entangled, and hence provide potential explanations for existing T2I issues. Second, we introduce the positive and negative binding vectors to enhance the binding within the concept and strengthen the distinction between concepts. Further with the neighbor strategy, the vector estimation can be more accurate. Evaluated in various ways, Magnet shows the ability to disentangle different attributes and generate anti-prior concepts. Performed in the textual space, Magnet improves the synthesis quality and text alignment, with an impressively low increase in computational cost. We sincerely hope that this work will motivate the exploration of generative diffusion models and the discovery of other interesting phenomena.
Acknowledgements
This work was supported in part by the Natural Science Foundation China (No. 62272227), and the Postgraduate Research & Practice Innovation Program of NUAA (No. xcxjh20231604)
10

---

References
[1] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06//25, 1(2):3, 2022.
[2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[3] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-3644 2022.
[4] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich cext-to-image generation. arXiv preprint arXiv:2206.10786 2(3):5, 2022.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages $748-8763 PMLR, 2021.
[6] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[7] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions Graphics (TOG), 42(4):1-10, 2023.
[8] Weixi Feng, Xuchai Hc, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang. and William Yang Wang. Training-fire free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv: 250022 2022.
[9] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference Computer Vision, pages 423-439. Springer, 2022.
[10] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[11] Yingtian Tang, Yutaro Yamada, Yoyo Zhang, and llker Yildirim. When are lemons purple? the concept association bias of vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14333-14348, 2023.
[12] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
[13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1900 2020.
[14] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977, 2023.
[15] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. image editing with cross attention control, arXiv preprint arXiv:2208.01626
2022.

---

[16] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pages 6038 6047, 2023.
[17] Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao. Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et al. Proxedit: Improving tuning-free real image editing with proximal guidance. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 4291 -3011, 2024.
[18) Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Get what you want, not what you don't: Image content suppression for text-to- image diffusion models, arXiv preprint arXiv:2402.05375. 2024.
[19] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D) Manning. Stanza: python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082 2020.
[20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[21] Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. arXiv preprint urXiv:2004.02709 2020.
[22] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint wrXiv:2303.054990 2023.
[23] Compvis. Stable Diffusion v1-4 Model Card, https: http:///ggggggacccco/ooooo/ scable-diffusion--.. 2022.
[24] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information prucessing systems, 30, 2017.
[25] Stabilityai. Stable Diffusion v2-1 Model Card, http:://buginngfce..//tttatttttt stable-diffusion 2-1. 2022.
[26] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint 10.00046. 2023.
[27] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceedingy of the IEEE/CVF Winter Conference on Applicutions of Computer Vision, pages 5343-5353, 2024.
[28] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[29] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1633-1637, 2020.
[30] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851, 2020.
[31] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13450 2020.
12

---

[32] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[33] Ziqi Huang, Kelvin CK Chan, Yuming Jiang, and Ziwei Liu. Collaborative diffusion for multi- modal face generation and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6080-6090, 2023.
[34] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 14:8780-8794 2021.
[35] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 2259,, 2022.
[36] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion--ased generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[37] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1972-1981, 2023.
[38] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[39] Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou Ranni: Taming text-to-image diffusion for accurate instruction following. arXiv preprint arXiv:2311.1700.. 2023.
[40] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung- Woo Ha, and Jun- Yan Zhu. Dense text-to- eext-to-image generation with attention modulation. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 7701-7711, 2023.
[41] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffu- sion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452-7461, 2023.
[42] Xinchen Zhang, Ling Yang, Yaqi Cai. Zhaochen Yu, Jiake Xie. Ye Tian, Minkai Xu, Yong Tang, Yujiu Yang, and Bin Cui. Realcompo: Dynamic equilibrium between realism and compositionality improves text-to-image diffusion models. arXiv preprint arXiv:2402. /2908, 2024.
[43] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-plas diffusion features for text-driven image-to-image translation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[44] Julia Guerrero- Viu, Milos Hasan, Arthur Roullier, Midhun Harikumar, Yiwei Hu, Paul Guerrero, Diego Gutierrez, Belen Masia, and Valentin Deschaintre. Texsliders: Diffusion-basse texture editing in clip space. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[45] Hu Yu, Hao Luo, Fan Wang, and Feng Zhao. Uncovering the text embedding in text-to-image diffusion models. arXiv preprint 2rXiv:2404.0113 2024.
[46] Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Text-to- image diffusion with token-level supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8553-8564, 2024.
[47] Stefan Andreas Baumann, Felix Krause. Michael Neumayr, Nick Stracke. Vincent Tao Hu, and Björn Ommer. Continuous, subject-specifie attribute control t2i models by identifying semantic directions. arXiv preprint arXiv:2403. / 17064, 2024.
[48] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of naacI-HHLT, volume 1, page 2, 2019.
13

---

[49] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content- content-guided video synthesis with diffussin models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7346-7356, 2023.
[50] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference Machine Learning, pages
PMLR, 2022.
14

---

Word embeddings
Word embrediings
[EoT] embeddngss
[EOT] embeddings
(a) CLIP VIT-L/14
(b) CLIP VIT-H/14
Figure 12: Principal Component Analysis (PCA) analysis of CLIP ViT-H/14 and CLIP ViT-L/14.
The word embedding and the [EOT] embedding have a different understanding of the attribute.
A
Additional analysis of the CLIP text encoder and the diffusion model
A.1 Analysis of the CLIP text encoder
Principal Component Analysis (PCA). We study two types of text embedding through the PCA technique in Fig. 12 for a low-dimensional comparison. We analyze two CLIP text encoders, which are (a) ViT-L/14 (here, the dimension of embedding is d = 768), and (b) ViT-H/14 (here, = 1024). We obtain text embedding c = {cSOT, Cobjects CεOT} without the context of the attribute from 60 object nouns (including animals, plants, and non-living entities). We have extended the number of attributes 16 (including colors and materials), and ended up with 960 text embeddings c' = (csOr Cattribute Cobject: "gorr) We use 60 object embeddings Cobject or Ceor without the attribute context to fit the model, and then transform contextualized embeddings d'bjecct or CEOT to the same space. This setting allows us to observe how two types of text embedding understand different attributes. The result indicates that the word and [EOT] embeddings produce different feature spaces with the attribute context. Overall, the distribution of the word embedding is denser, while the [EOT] embedding with attribute context distributed dispersedly.
Attribute bias analysis. Fig. 13 investigates the phenomenon we call attribute bias on two types of text embedding, obtained from the text encoder of CLIP ViT-L/14 and ViT-H/14, respectively. The word embedding without supervision during training has shown severe attribute bias. For example, the word embedding of the object "tiger" indicates an extreme preference for the color "yellow". Conversely, the [EOT] embedding produces a relatively small variation in the similarity curve. In the main paper, we conjecture that VLMs' poor compositional understanding and the behavior of oogs-of-words [12, 11] on [HOT] lead to an inaccurate textual representation, which affects the interaction between the image latent and semantic word embeddings.
Interestingly, we find that the learned representations of two text encoders are quite different. For example, to encode "car" and "vase" with the context of different attributes, CLIP ViT-L/14 gets the cosine similarity around 0.6 v.s. 0.7, while CLIP ViT-H/14 gets 0.2 v.s. 0.5, showing discrepancy. We conjecture that ViT-I/144 in large-sized network and dimension may have exacerbated the bias. Yet, it is beyond the scope of our research and we may leave it for future study.
Despite the above difference, both encoders demonstrate the discrepancy between the word and [EOT] embeddings. The stability of the [EOT) embedding can also be explained by entangled context. In contrast, the word embedding without supervision during training may suffer less from entanglement.
A.2 Fine-grained cases analysis
Take the concept "red chair" as an example. The CLIP text encoder maps it into em- beddings c' = {csoT, Credy Scheer CEOT Spod, ... Epadyy } (here, 73 for L - 4. L 77). The counterpart text embedding of the concept "chair" without the color modifier is
15

---

Tonler? tuse"
Yaular! car
Yanlor? bananaa
Taaiur broccoli"
Yaular! cut"
Yaninr tiger" Sayel
(Pomew Saral
VIT-1/1
VIT-R/14
Saaun Kaorl
Ceaeen Hamm
(aeth Farr
Smllartt
Figure 13: The attribute bias of different objects encoded by CLIP ViT-H/14 and CLIP ViT-I/14. The word and [FOT] embeddings show large discrepancies of attribute bias for the objects "banana" "broccoli", etc. Observe that the extracted embeddings by different text encoders differ significantly.
pod out
"bux banona
Aprea Vage Kasrs
Zagh Capt Srer
Eees Soil s Shaa
éiner Samemer Savay Frme
Saer Spe
spani teee Aire
dest fow dlar
(3) daaa Ausern Eaerry Are
(4) dase Ourona, Ogon
CowaN
(A) Keuns Rear dl
(II) Typas 48 18.
(A)
(15) dh Roee dr
(A] Aamm
(10 Aiea dE
(C) Zprrs Caa Rg-
(C) Eoep Sata 50,5
(C) Sinur Cooumas Ciy 52
Figure 14: Fine- tine-grained 4 cases described in the main paper, well as 3 additional cases.
(CSOT, Cchair; CEOT, Cpady:-.., C (here, 74 for 3). The designed 4 cases are de- fined as: standard generation conditioned on vanilla embeddings of the concept, i.c.. Coasel = e'; (2) replace the contextualized word embedding in c'. i.c., Ccase2 = (csor,Cus Cohairs ĆEOT: Opad, ... Spadoy (3) replace all [EOT] and padding embeddings, i.c., Cease3 = (cSor, Credt Cchairr CEOT, Cpadi Cpadts }; (4) replace the contextualized word, [EOT] and padding embeddings, i.e., Ceased (CSOT, Creds Cehnir, CEOT, Cpody: ..., Cpadrs Note that maintain the attribute word embedding "entor to observe whether the model can capture the color information without contextual information in other text embeddings. The results are displayed in Fig. 14. As we discussed in the main paper, cases 1-2 where padding embeddings with the color context are still realistic when the concept is natural (i.c., "green car"). However, they generate out-of-distribution images for the examples "red cat" and "blue banana".
In addition, we have designed new cases to verify that the color information has been gradually forgotten in the padding embedding. We divide [ÉOT] and padding embeddings into groups: ex [CEETT..., Cpedaz }. Cy = {Cpadas: .... Spadia }. = (Cpaalg:....,pauee not have the color context), and their counterparts , (here, these embeddings with the color context). The results in Fig. 14 (bottom) are consistent with our hypothesis. To be specific, cases and B show light "green" invisible "red" compared to the successful binding results in case C, where embeddings {CgoT,.... Cpades are contextualized with the target color.
(here, these embeddings do
16

---

podd
podgs
padgpp
green
apple
BOT
pods
podgs
podas
Mise
BDI
pood
poodss
puuls
baokiroc
blsic
horac
BOT
goodaa
podes
Maac
Ireusali
yad,
poldas
porka
srecns
Sesamo
Imoww
s/oorgy
BCT
pudd
goudoy
podks
pold
vtrasberry
BOT
pod:
poodas
pordge
schile
carr
red
shoup
BOT
podd
porlly
podk
(a) Single-coneppt
concept
Figure 15: Several effects of the context issue in padding embeddings under the scenarios of (a) single-concept and (b) multi-concept. We refer to the detailed analysis in Appendix A.3.
we propose Magnet based on two key observations on the word embedding. First, the target color is invisible in case for concepts "green car", "red car". However, these colors can be observed in case (arrive at Eq. (1) for computing u.od). Second, cases 1-2 and cases 3-4 for the concept "blue banana" " both generate catastrophic images. This indicates the vector estimated by the object itself can be inaccurate, In this case, we introduce the neighbor-guided vector estimation.
NOTICE:
A.3
Visualization-bant analysis of the context issue
Recall our hypothesis that [EOT] and padding embeddings are trying to remember all important information (e.g., attributes, objects, and positions) in the given prompt due to the contrastive learning and bags-of-words behavior CLIP. In Fig. 15, we investigate the entangled context in the padding embeddings under two scenarios for prompts with single object multiple objects.
Single-concept scenario aims to generate one object with specific attributes. Fig. 15 (a) shows that the context issue in padding embeddings leads to (1) out-of-distribution and inaccurate object structures, e.g., "cat" is painting-like like in row 1, "banana" is unrecognizable row though presenting correct attributes. Or (2) generate the object with another attribute that can compose a natural concept, e.g., "broccoli" binds to the prior attribute "green" rather than "black" " row One potential explanation is the image latent is contaminated by inaccurate representation in padding embeddings, as evidenced by the overlapped activation latter padding embeddings with the word embedding of each object. The generation of natural concepts proves hypothesis that latter padding embeddings forget attribute context if the object has a preference for certain attributes based on the training dataset. In row 4, we present interesting observation that padding embeddings aligned with the attribute word rather than the object "strawberry". It seems that the word "gold" entity instead of visual feature, leading to (3) split of the target object.
interpreted
Multi-concept scenario aims to generate multiple objects with the desired attributes. Fig. 15 (b) shows that the context issue in padding embeddings leads to (4) color leakage, i.e.. one object presents the attribute belonging to another object row Or (5) objects stick together, e.g., strange creature with the head of "horse" but the body of "bug" in row 2. All the above phenomena can be attributed to the evident entanglement in padding embeddings with overlapped cross-attention activations, which provides inaccurate object representation and indistinguishable binding relationships for each concept. Note that the above effects can occur simultaneously on single instance: row 3 indicates
inaccurate "sheep" structure, binding between "banana" and the prior color "yellow". a split of the object "banana", as well as a sticking problem between two objects "banana" and "sheep". In row 4, find that the context issue of padding embeddings also explains (6) the issue of missing objects, i.e., the context loses the object "sheep" and contains a dominant representation of the object "car".
[18] also discussed the semantic information in padded [EOT) embeddings. While their main concern to remove specific object content, our focus is the understanding of attribute.
17

---

Count
5000
2000
0.4
0.5
0.0
costct? 2x( )(()/")))
Figure 16: Statistical analysis of w cos H (P!!o)) obtained from 19648 samples (614
objects and 32 attributes). We set À = 0.6 where the count drops.
B Detail of the proposed method
B.1 Dependency parser
To extract the dependency set D - { AM&EEM)
in the given prompt, we adopt off- the-shelf dependency parsing module Stanza Library [19] and construct syntax trees using NLTK. Following [8], the pair is searched by noun phrases (NPs) in the syntax tree and their corresponding adjective words. For instance, given the prompt "a black cal sitting a white bowl", the object "cat" is extracted according to the label NN or NNs, then allocated its attribute "black" in the subtree. Similarly, the object "bowl" and its attribute "white" can be obtained. However, the parser may fail to extract the concepts out of the "[attribute] [object]" format. For instance, it not process the prompt "a photo of a streetlight that green" with dependency "green "streettlghht" ", or "apples of green in white baskets" with dependency "green" "& "apples". We leave this for future work.
B.2 Background of diffusion models
The conventional diffusion model [30] works in two steps: (1) forward diffusion that gradually adds noise to the image x; (2) reverse diffusion that removes noise from noisy image x, step-by-step.
Latent Diffusion Models (LDMs)) [2] perform the denoising the latent space. The pre-trained encoder á compresses the image to the latent ((x), and the pre-trained decoder ià reconstructs the latent as (z) x. The forward diffusion produces the noisy latent 2t for the step = I..., The denoising network is trained to remove the added noise each step by minimizing ||ea(ze, t) elI², where 2, is the noisy latent timestep f, (00 1) is added Gaussian noise. The noisy latent 2y is sampled from Gaussian noise N N 1) during inference. Finally, the reversed latent žy is decoded to produce the image x = v((2).
The proposed Magnet is applied over Stable Diffusion (SD) conditioned on text prompts. The pre-trained CLIP text encoder & maps the prompt the text embedding e = E(P). SD appends several cross-attention layers to inject the text condition into the latent The loss function of the text-image latent diffusion model can be rewritten as []c,(2,, f, clp
B.3 Strength of the binding vector
The use of the exponential function is inspired by [18]. But in different way, Eq. (3) that determines the strength or,, 3, based on our observation Fig. (b) and (c) in Section
The formula
cos (G( H(P("))) calculates the cosine similarity between the first [EOT] embedding and the last padding embedding of the concept Pros Fig. 16, we have conducted statistical analysis using Numpy's 's histogram to bin the data. Different values w are obtained from 19648 samples encoded by CLIP ViT-L/14. The highest counts are at the values 0.66 and 0.71. Observe that the count drops when < 0.6 or > 0.82. Intuitively, smaller w indicates a larger deviation from the target context. Empirically, we set 0.6 Eq. (3) enhance the weak binding (i.c.. a4 1 when wi < 0.6 in e-w)), We have conducted ablation study of the value in Fig.
18

---

6. For the strength 3, of the negative binding vector, we suggest relatively slight control to avoid strong deviation when the concept number M is large, i.., 3 w²
B.4 Neighbor-guided vector estimation
Feature Neighbors. The candidate set S = { ..., Bx) used for the feature neighbor strategy includes R words. In practice, we gathered 614 object nouns generated from ChatGPT [13] and checked manually. We extract the word embedding CBn of each candidate object. For example, the candidate "truck" is mapped into P("truck") = {CSOT, Ctruck, CEOT, The embedding Caruck is extracted and used in the formula d(cB,, F Prr)). Notice the [EOT] embedding is not used. This procedure of extracting candidates' embeddings is one-for-all, i.e., compute 641 embeddings once for each new text encoder and save them to the local path.
Semantic Neighbors. These neighbor objects semantically related to the target object. We adopt ChatGPT [13] to predict the semantic neighbors. The instruction follows the sentence pattern of "Which objects are highly related to the word < ?". Optionally, the large language model BERT [48] for fill-mask is considered. We mask the object in the prompt to get its neighbors. For example, predict the neighbor object for "brown bear", the masked prompt is composed as "brown bear and a [MASK].". We hypothesize the conjecture "and" can implicitly restrict the close relation. The first two nouns output by BERT are "wolf" and "lion", which are similar objects to "bear".
C Implementation details
Configure. All experiments are conducted on RTX 3090 in a single GPU. Our proposed Magnet is built upon SD V1.4 [23] with the pre-trained text encoder of CLIP ViT-L/14 [5].
Hyperparameters. The choice of À 0.6 is explained in Appendix B.3 and verified by the ablation study in Fig. 6. We set K = to conduct qualitative and quantitative experiments. We have discussed other choices of K in Appendix E.I. We generate images with 50 diffusion steps with a fixed classifier-fies free guidance scale of 7.5.
Baselines. We compare Magnet to SD V1.4 [2], the training-fiee free method, Structure Diffusion [49], and the optimization method, Attend-and-Excits [7]. Since the official Attend-and ann-------xiitee does not provide an automatic parsing process, we extract the required object words (in bold) using the Stanza's 's package (same to Magnet, see Appendix B.
Datasets. We have conducted statistics on the CC-500 dataset based on the three types of classification. We find the number of valid prompts for each type are 84, 212, and 136. respectively. This data bias may lead to unfair comparisons. In this case, we randomly select 80 prompts per type and obtain 240 prompts in total.
The runtime to generate an image and the required maximum GPU resources for each method are listed in Tab. 2. The data of each method is obtained by generating 200 images (randomly sampling 50 prompts from each dataset and generating 2 images per prompt). Each method is tested under the same setting maintain fairness.
Resource.
D Metric discussion
We rely human evaluation since the commonly used metrics for text-to-image synthesis unreliable for our concern about attribute binding. We discuss three models as the automatic evaluation metrics, which are retrieval models CLIP [5] and BLIP [50], as well as the phrase grounding model GroundingDINO [22].
Fig. 17 (a) shows the drawback of CLIP score, which computes the cosine similarity between the text and the image embeddings. Failure and success cases present relatively equal values. The [EOT] embedding suffers from attribute bias and can not measure the unnatural concept "blue apple".
Similar to CLIP, the metric BLIP score in Fig. 17 (b) diverges from the human evaluator. Given the target prompt with multiple concepts, the image of SD (top) presents entangled attributes and objects. this case, human evaluators indicate no instance of object existence and attribute alignment. However, BLIP text-image similarity can not align with the assessment of human evaluators.
19

---

Targer Prompt: bluz apple
Target Promget:
pink
and browm wheep
bruam brid and yellow
BLIP
Humna
Objert
Model
Iluman
Match
99.97%
Object
sheep(X)
bind
Sauree 17.6
Suurie 1782
Laag-
Sim.
0,496
Altribute Alignment
shorp(X)
call X
cal
x
gold cake and a blue clock
BLIP
Human
Object
Model
Human
99.95%
Object Existencce
Abeop(s/
catl
cake
Soore: 17,22
Score: 17.14
Megnet
Test Image Sim.
0.504
Attribute Aligmment
sheep(t/)
cut((/
clock
CLTP
GroendingDINo
Figure 17: Failure cases of three automatic metrics: (a) CLIP text-image similarity can not assess the binding of unnatural concepts. (b) BLIP text-image similarity fails to capture the entanglement of concepts. (c) The detection of GroundingDINO diverges from human annotators.
Table 4: Quantitative comparison following Attend-and-Excitt [7].
BLIP
Type
Method
Full Prompt
Min. Object
Full Prompt
Min. Object
Stable Diffusion
32.40
22.40
45.32
30.35
Structure Diffusion
32.24
22.38
44.68
30.28
Training-free
Magnet( Ourss)
33.11
22.79
46.53
30.84
Optimization
Attend-ande Fxc-iette
34.12
24.63
49.65
34.83
In the main paper, we adopt GrondingDINO [22] to detect the object in the generated image. However, fails to capture the structural deviation and suffers from attribute bias. As shown Fig. 17 (c), the entangled concepts "bird" and "cat" are detected by GroundingDINO, which diverges from the human evaluator. Conversely, the model can detect "gold cake". This may be attributed the attribute bias, which we have discussed in the main paper.
Additionally, follow Attend-and-Exeite [7] and compare the full prompt similarity and minimum object similarity using CLIP and BIII. The quantitative comparison is listed Tab. 4. Magnet shows improvement on all metrics compared to SD and Structure Diffusion. Meanwhile, we compare the text-text similarity [7] using the BLIP model for image captioning, resulting in SD (66.08), Structure Diffusion (65.7 (6). Magnet (68. 22), and Attend-and-Ea Excite (71.22) as the highest. However, do emphasize that the above quantitative metrics can not reflect the disentanglement of objects and attributes that we are concerned about.
In conclusion, we refer to the human evaluation to ensure a fair and reliable comparison. A screenshot example of the coarse-grained comparison is given in Fig. 18.
E
Additional ablation experiments
E.1
Hyperparameter K
In Fig. 19, we have conducted an ablation study on the hyperparameter to select neighbor objects. Note that the positive and negative vectors are estimated by each object E, itself when K = 1 as Eq. (I). The difference is slight if concepts in the target prompt are relatively natural. For example, in row 2, using K = 1, 3. 5 (column 2-4) can generate the correct concept "red ball" compared to "white ball" SD. However, the results of K (column in rows 3-4 present a catastrophic structure of "blue bananas". This verifies the effectiveness of the neighbor strategy. On the other hand, K in a large number can lead to inaccurate binding vectors. For example, in rows 1-2, results of K > 10 are similar to SD. This can be attributed to the introduction of multitude of unrelated objects that have an impact on the estimation accuracy. Similarly, "stickery" are indistinguishable in row 3, columns 7-8 using K 20,50 50.
20

---

Description: Assiver
frashian nearby"
F no winwer
+1 Which image more realishic or visually appealing?
O
O
22. Which image shows difterent objects more clearty?
O
Which image shows differant more clearly?
A
C
0
O
Figure 18: A screenshot of the human evaluation for assessing image quality, disentanglement of objects and attributes. For each question, the order of images generated by Magnet and other methods is randomized to maintain fairness.
brown teddy bear sitting wooden table red ball
seed
some blue bananas with little vellow stickers on them
seed
Stable
K (0.33))
Diffusion
(0.14s)
(0.18s)
(0.25s)
(0.445)
(0.81s)
(1.9)
Figure 19: Ablation study the hyperparameter K. We emphasize that K = 5 may not always be the best choice because of the randomly initialized latent. For example, the result of K 3 is more appealing than K = 5. We choose K = 5 which can stabilize the generate of unnatural concepts (e.g., "blue bananas" and "yellow stickers" can be more distinguishable in 5 than 3), as well as balance the processing time.
Interestingly, when using different seeds, the most visually appealing image may not always come from the same K. For example, we subjectively prefer the result K = row but row 2 the result of K is more appealing. This is due to the randomly initialized latent. Since Magnet's resource requirements are relatively low, we believe it is possible to use different for the same prompt and generate images simultaneously for freedom of choice to the user.
In conclusion, the reason for use of K 5 is the balance between synthesis quality and pre- processing time for manipulation. Here, we obtain the data of time by processing 20 prompts. i.e., adding 0.25s SD to generate an image using 5. Meanwhile, our code can be improved to shorten the time, which is left for future work.
21

---

brown cat and red cup
(a)
brown purse is sitting green bench
(b)
Table Ablation study on negative and positive binding vectors. In most cases, the images generated by three cases are equally good or bad, resulting in high number no winner.
orange dog wearing gray bow tie laying sofa
(c)
Disentanglement Attribute
Object
Stable
positive only
regotive
both
Dif-uusion
only
both positive only
13.8
9.4
3.9
3.0
Figure 20: Ablation study on negative and positive bind- ing vectors. (a) depicts similar results. (b) verifies using both vectors can alleviate the missing object (i.e., "green bench"). (c) verifies using both vectors can enhance the binding ("orange dog" and "gray bow tie").
negative only
5.1
1.3
Stable Diffusion
1.4
0.2
no winner
75.8
86.1
E.2 Importance of both positive and negative vectors
In Fig. 20 and Tab. 5, we conduct ablation study on the proposed positive and negative vectors. The object disentanglement is assessed by asking "Which image shows different objects more clearly?". and the attribute disentanglement by asking "Which image shows different attributes more clearly?". We randomly select 12 prompts from CC-500 and 20 prompts from ABC-6K, generating 25 images per prompt (800 images in total) for three settings.
Both qualitative and quantitative comparisons verify the importance of both vectors. For instance, the concept "green bench" " is interpreted to "green grass" when using only one type the binding vectors. This occurs because of the entanglement of two objects. For the attribute disentanglement using both vectors is capable of generating objects with desired attributes. Notice that the negative vector improves the object disentanglement (presents "bench" in column 5), while the positive vector improves the attribute disentanglement (presents "orange dog" in column 3). The human evaluation results are consistent with the above analysis, i.e., negative only (5.1) overpasses positive only (3.9) terms of object disentanglement, and positive only (3.0) overpasses negative only (1.3) in terms of attribute disentanglement. In conclusion, using both vectors significantly improves text alignment.
F
Limitations
Although Magnet provides efficient and effective way to address the attribute binding problem, we acknowledge our technique subject to few limitations.
Fig. 21 displays the failure cases of Magnet. First, the neglect of the object (columns 1-2), may be attributed to the model's limited ability foreground limited subjects. Second, the excessive manipulation of the object embedding leads to out-of out-of-distribution (columns 3-4). An interesting observation is that Magnet sometimes generates images with correct concepts, but incorrect positional relations (columns 5-6). We suspect that the color layout has been determined in the early stage. In this case, Magnet maps the object to the position of the attribute the image, rather than blending the attribute with the object. Magnet inherits the well-known issue of T2I models, presenting merged objects (columns 7-8). Finally, it still challenging to generate an unnatural concept when the object has strong attribute bias (columns 9-10).
22

---

Aoe Eries agoied
andd sudled
pol wall
sou bord browak
ale
grese
neglect objeet
(b) manipulation
posiitinna relation
concept entangiement
(e) attribute bias
Figure 21: Limitations of the proposed Magnet. (a) shows two cases that amend the concept, while still missing one object; (b) includes (c) depicts an interesting phenomenon that Magnet correctly disentangles concepts while failing to in accordance with the location word "in". (d) shows Magnet will produce entangled concepts due to the limited power of SD. (e) provides two fail cases to generate unnatural concepts.
results caused by the excessive value of a, 5;
black
photo shows a group
long narrow
red and white train and lots
of people dressed black tie attire playing having drinks
vellow kitchen
white dog standing front of brown cabinets
black
sullow cranes
broan elephant
ldkue Inuwn backpack and blue cow
Boor tiles
and wire
and red car
Moggnet
Figure 22: Similar images generated by Magnet and SD.
Additionally, Fig. 22 displays examples that Magnet generates similar images with SD. Most happen when SD has produced relatively faithful images (columns 1-2), or prompts with excessively detailed concepts (columns 3-4), well the generation of two unrelated concepts (columns 5-7).
We consider combining Magnet with optimization- based methods to tackle the neglect of objects, e.g., the integration of Attend-and-En Excite and Magnet (see Fig. 9 and Fig. 23). Magnet is also compatible with existing T2I controlling modules to address the inability to change spatial relationships, e.g., the [27] and Magnet (see Fig. 10). The excessive insufficient manipulation may be addressed by improving the formula Eq. (3), or simply stating the strength oi, Br manually. We leave these for our future work.
integration of ControlNet [28] or aayout-guidant
G Additional results
Fig. 24 provides examples that Magnet improves the image quality compared to SD.
In Fig. 25, compare Magnet to SD by visualizing the cross-attention activation.
Fig. 26 provides examples of typical indoor scenes using prompts from the ABC-6K dataset.
Fig. 27 and Fig. 28 provide additional qualitative comparisons on the ABC-6K and CC-500 datasets, respectively.
23

---

vellow sheets on a blanket with white headboard
birds
riding
chasing after kusen frishee on top of purple lawn
brown has rui ball in it's mouth
sitting on REEN stick
kusen horse
jush black field
SD
Magnet only
ALE
+Magnet
Figure 23: Additional results of extension to Attend-and-Excite. In columns 1-2, Magnet only may neglect the object (e.g., "gray stick") "). In columns 3-4, Magnet can generate images with unnatural concepts but would be painting-like. The combination (row 4) demonstrates improvement. Column displays failure case. The parameters may need to be modified to fit Magnet.
brown bear wearing pair of red glasses
a black cat lying on brown suitcase
a yellow and brown butterfly sitting of
large silver pot sitting on a counter next red brick wall
rhile dog sitting inside a ped car next string flowers hanging off the mirror
meat with writh vegetables sitting green plate
Stable Diffusion
Magnet
Stable Diffusion
Magnet
Figure 24: Magnet improves the synthesis quality by disentangling different concepts. Best viewed zoomed in.
24

---

Prompt: blue bananas with welloe stickers them
Stable Iictusoo
Mlappe
dog fries red plate yellow
Stabie
A
Magnet
Prompt: broccoli red
Stuble
Magnet
Promut:: green plate pink frosting
stable Difrisiion
Stagne
Prompt:a hyclrant sitting patch grass
Stable Dittulon
Magnet
Figure 25: Visualisation of attention maps. The activations of different object more distinct in Magnet compared to SD. For instance, bananas are overlapped with stickers in row 1, while row indicates disentangled concepts.
25

---

clean living with green sofa with brown pillows
furnished decorated living room with white walls, paintings and teal sofa
orange kitchen with white refrigerator stove and dishwasher
the kitchen with white atop green tiled floor
Stable Diffusion
Structure Diffusion
Attend-and Boaud-and--xcoe
Magnet (Ours)
Figure 26: Qualitative comparison using prompts from the ABC-6K dataset. We provide some typical indoor scene prompts and compare Magnet to baseline methods. Best viewed zoomed in.
26

---

black train with three cars blowing green smoke
red bench in front of stone style wall and bush with blue flowers to the side of
white fire hydrant sitting in front yellow fence
yellow teddy bear in white shirt against red wall
Stable Diffusion
Structure Diffusion
Attend-and-
Magnet (Ours)
Figure 27: Additional results using prompts from the ABC-6K dataset.
27

---

blue sheep and brown vase
gold car and red clock
II
XII
11G
10
2
10
2
IX
IIII
9
5
I
8
4
7
6
black apple and green backpack
blue dog and a brown suitcase
Stable Diffusion
Structure Diffusion
Attend-and-Exccte
Magnet (Ours)
Figure 28: Additional results using prompts from the CC-500 dataset.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main contributions and scope have been included in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of this work have been discussed in Section 6 and Appendix F (see Figure 21).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: Section 2 and Appendix A provide the full set assumptions of the context
issue in the padding embedding, which is the motivation of the proposed method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details of the proposed method are described in Section 3 and Appendix B. We clarify that the main experimental results can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
30

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: provide the used two datasets (ABC-6K and CC-500) and a piece of code with sufficient instructions in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbpcccccceecce heroolicgguieeeee Codeeubbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4.1 describes the detail of datasets. Section 4.2 and Appendix have described evaluation metrics. We provide the implementation details in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
The information is provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https htt////eurripp.eeeee httpps.cc/publll htt.:i///ehhissueeeeeeeee
didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
Our research has been conducted following the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 5, Appendix E.1 and F have discussed both potential and negative societal impacts of the proposed method.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We adopt the ABC-6K and CC-500 datasets from [8]. Our method is built upon Stable Diffusion 1.4 [2] with the pre-trained text encoder of CLIP ViT-L/14 [2].
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced in the paper have been documented. In the supplemental material, we have provided Jupiter file with visualized results for reference.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The instructions given to human evaluators are described in Section 4.2, and screenshot instance in Figure 18.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have obtained approval from equivalent to the Institutional Review Board to conduct our quantitative experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "Jyq0IcTpMO", "reviewer": "Reviewer_2c6i", "summary": "The paper analyzes and improves upon the “(attribute) binding problem” in VLMs such as CLIP, and the focus is primarily on the text encoder side.\nFirst the authors analyze how the individual text embeddings behave in a diagnostic setting when encoding a two-word text “COLOR OBJECT”. With these insights they propose to modify the original embedding of an object by calculating a “binding vector” that uses a positive attribute (the one actually in the prompt, i.e. “yellow”) and negative attributes (other colors).\nThey show improvements to previous methods on top of which they build such as Structure Diffusion.", "questions": "It says in the abstract “blended text embeddings as the cause of improper binding… “. Is “blended text embedding” embeddings a term people are expected to know? I work in vision-and-language and have not heard of it.\n\nAgain in abstract “we observe the attribute bias phenomenon”: was this meant to say attribute binding?\n\nHow are negative attributes exactly determined? Most things are not as easy as color.\n\nIs there an explanation why the Structure Diffusion baseline is so bad?\nDoes your method differ to Stucture Diffusion only in the proposed Magnet method and nothing else? Same exact SD version, CLIP version etc.? This is important to clarify in the paper so that it is clear that your method alone leads to this improvement.\n\nNote: Fig. 1b is very small and also low-resolution, please fix!", "limitations": "Yes, addressed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "The paper is very practical and intuitive, motivating the final method with analysis of the embedding space. It has a clear contribution and the whole story ties around this contribution, providing additional insights.\n\nOn top, the evaluation is thorough and uses human judgment. (there is no automatic metric here, I agree).\nIt is also good to see that the neighbor strategy was quantitatively ablated with human judgment, and section 4.6 adds additional nice insights, albeit not beyond a few examples as far as I can tell.\n\nOverall a very interesting and creative exploration, and the empirical results do show that the method works! That is exciting, given that this is a persistent major problem many papers have addressed.", "weaknesses": "While the method clearly improves upon previous work, it requires pre-computing or manually defining a lot of components such as the negative attribute set or the neighbor set, with additional computation of the adaptive strength. With pre-computation this is not too expensive but it feels somewhat hacked together.\n\nThe abstract mentions too many terms where it is unclear whether they are established terms and if not, the abstract is not the right place to mention them all: blended text embeddings, attribute bias, object embedding, binding vector. \n\nSince this papers main focus is a thorough analysis and interpretability insights, it has to be clearer with its definition, i.e.:\n“The above phenomenon, which we call attribute bias, occurs when one object prefers or dislikes\na specific attribute” → refer/dislike are not well-defined.\n\nThe 4 experiments with swapping various embeddings seem very interesting but it is not motivated why we need all 4, maybe 2 cases would have been more insightful and easier for the reader to understand what’s going on?\n\nFinally, having more quantitative statistically significant experiments and less showing of examples would have strengthened the paper's contributions further!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your careful review and the valuable suggestions provided. Our responses to the Weaknesses (W) and Questions (Q) are outlined below.\n\n**W1**: Magnet is fully automatic during evaluation and does not require manual definition of these positive/negative attributes. We have introduced these components to ensure robustness. Various ablation experiments have proved the effectiveness of each component, including the parameter $\\lambda$ (see Appendix Fig. 17), the neighbor strategy (see Fig. 18), and the positive and negative vectors (see Fig. 19). Meanwhile, our method can be simplified by using fixed strengths and estimating the binding vector without neighbors (i.e., by the object itself). Furthermore, as you pointed out, it is not expensive at all.\n\n**W2&Q1&Q2**: Your comments on our abstract are very valuable to us. The following is our explanation of the terms used in the abstract:\n1. the term \"blended text embeddings\" in other words means \"text embeddings are blended\", with reference to [1] \"tokens in the later part of a sequence are *blended* with the token semantics before them\"; \n2. the term \"attribute bias phenomenon\" appears in line 60, i.e., our discovery based on the analysis of the CLIP text encoder, rather than the \"attribute binding problem\". \n\nWe will check all the terms in the abstract, and improve it to be clear and professional.\n\n**W3**: This sentence can be explained with an example in Fig. 1(b) (apology for its blur, we have provided a high-resolution one in Fig. 1 in the attached PDF). For the object \"banana\", both word and EOT embeddings show high similarity on the color \"yellow\" but low similarity on \"blue\", as if \"banana\" *prefers* the attribute \"yellow\" but *dislikes* \"blue\". We will improve the analysis section and clarify all definitions in the final version.\n\n**W4**: The 4-case experiment is deeply connected to our method. As discussed in lines 439-443, our motivation is based on several observations between 4 cases. Specifically, case 1 is a reference case w.r.t. standard generation, case 2 shows how the context information of word embeddings affects generation, case 3 shows how padding embeddings' context information affects generation, and case 4 tests whether the model can capture attribute information in adjectives (described in lines 428-429). Using all 4 cases would be more convincing to introduce our proposed binding vector and adaptive strength. In addition, we add 3 new cases in the Appendix to justify the information-forgotten problem in the latter padding embeddings (see Fig. 12, cases A-C). We will add more descriptions of this experiment to the main paper and improve readability.\n\n**Q3**: We define \"negative attributes\" as adjectives in the given prompt that do not belong to the current object. For instance, given the prompt \"a green apple on a wooden table and a red chair\" with three concepts, when operating the object \"apple\", its positive attribute is \"green\", then negative attributes are \"wooden\" and \"red\" (belong to \"table\" and \"chair\", respectively). It should be noted that negative and positive attributes do not need to be semantically opposed to each other. Magnet is not limited to colors and can work on prompts with extensive attributes (see Fig. 2(a) row 3 and (b) row 2 in the attached PDF). We will add more non-color examples in the final version.\n\n**Q4**: We assure that our experiments are completely fair through consistent control of all settings (e.g., the same SD&CLIP version and seeds). Its poor performance can be explained by our discovery in the text encoder: the improper binding is caused not only by the word embeddings, but also by the entangled padding embeddings. StructureDiffusion simply separating the word embeddings of different concepts is insufficient to offset the entanglement in the padding embeddings. Conversely, we introduce binding vectors to reinforce the difference between each concept. This improves disentanglement. Additionally, StructureDiffusion also performed poorly in [2]'s comparison experiments - Structure Diffusion may even get worse scores than Stable Diffusion.  \n\nFinally, we sincerely apologize for Fig. 1 and will fix it in the final version. Your feedback is very helpful to us. We will improve our paper to strengthen our contribution, reducing visual examples and adding quantitative results (e.g., FID-10k: SD 19.04, Magnet 18.92).\n\n[1] Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis\n\n[2] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IV8As9xFpY", "reviewer": "Reviewer_iZtx", "summary": "The authors propose _Magnet_ to solve the attribute binding problem. (1) Initially, they specifically analyze how the improper binding problem occurs in text embeddings. By comparing embeddings for each token, they demonstrate the attribute bias phenomenon where attributes do not bind well to the object token $c_{object}$ in rare concepts like \"blue apple.\" Additionally, through cosine similarity analysis of padding embeddings, they hypothesize that the latter padding tokens, e.g., $pad_{73}$, tend to represent prior bindings (like red apple) rather than binding the target concept (blue) with the object (apple). (2) Magnet suggests a method to slightly edit the embedding of the object to align with the desired concept, based on these observations. They demonstrate their method's efficacy through ARC-6k and CC-500 datasets.", "questions": "Q1: How is the analysis in lines 64-78 and Figure 2-(a) connected to the method?\n\nQ2: Is there no need to modify the padding? If the padding tokens positioned at the end contain prior knowledge, it seems they might also interfere with binding.\n\nQ3: How did you perform detection using GrondingDINO in Table 2? I am curious whether you detected \"blue apple\" or just \"apple.\" It might be somewhat out-of-scope, but I am also interested in how different the results are between detecting \"blue apple\" and \"apple.\"\n\nQ4: As mentioned in the limitations (Figure 20), Magnet seems relatively weak at handling positional information. I suspect this issue arises not so much from a problem with Magnet itself but rather because CLIP's embedding does not effectively bind positional information to the desired token. Is there an analysis similar to Figure 1 for positional information? It seems only color is present in the appendix. Showing this could further demonstrate, as per the title of your paper, an understanding of VLM’s function, providing clarity on what can and cannot be bound.\n\n---\n\n- Relevant references\n\nBefore this paper was submitted, methods for editing text tokens were proposed in papers like [1, 2]. Although these papers also deal with a different task, and attribute binding, and were not considered in the evaluation due to being on Arxiv, I recommend citing them in the camera-ready version.\n\n[1]: Uncovering the Text Embedding in Text-to-Image Diffusion Models, https://arxiv.org/abs/2404.01154\n\n[2]: TexSliders: Diffusion-Based Texture Editing in CLIP Space, https://arxiv.org/abs/2405.00672", "limitations": "The authors describe the failure cases of Magnet in the appendix. They have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- Writing quality \n\nThe writing quality is good for understanding the core motivation and method. Analyzing embeddings using cosine similarity is intuitive and effectively demonstrates how attribute bias negatively impacts attribute binding. The flow from the initial analysis to the method is also easy to follow. \n\n- Novelty\n\nResolving attribute binding in the text space, as done by Magnet, seems quite plausible and novel. Identifying the limitations of the T2I diffusion model from the VLM model has been necessary but has not been extensively conducted until now.", "weaknesses": "Overall, I think it's a good paper. It would be better if the evaluation were a bit more solid. \n\n- Evaluation\n\nI agree that a user study w.r.t performance comparison as shown in Table 1 is necessary, but for this task, it is necessary to evaluate whether each method achieved binding rather than comparing different methods. Since we don't know the precise performance of each technique, it's difficult to understand the actual performance of the Magnet and how much it has improved.\n\nThe results in Table 2 are weak. It's disappointing that the automatic results are quantitatively worse compared to attend-and-excite. While I agree that manual inspection is the most accurate, trying out a recent VLM like GPT-4o, which can discern characteristics that image encoder-based evaluations fail to capture, might be worthwhile.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review of our manuscript and the insightful comments provided. Here are our detailed responses to the identified Weaknesses (W) and Questions (Q).\n\n**W1**: Our comparison experiment has been designed to evaluate whether each method achieved binding. In Tab. 1, *attribute disentanglement* is assessed by asking \"Which image shows different attributes more clearly?\". Only the method achieving the best binding will be voted, otherwise evaluators tend to choose *no winner*. In Tab. 2 *attribute alignment* (Attr.), human annotators annotate a precise number of successful bindings (whether one object presents the desired attribute). This is done separately for each method, without comparing the others. Lines 161-178 have explained each criterion in detail.\n\n**W2**: Automatic results in Tab. 2 only assess *object existence* (see Q3). Magnet is inferior to Attend-and-Excite (Attend) since it optimizes the noisy latent to encourage the model to attend to all objects. As discussed in limitations, we acknowledge Magnet suffering from missing objects. Though designed to enhance *attribute alignment* (has outperformed all baselines in Tab. 2), Magnet shows improvement in *object existence* (Det. $+5$ on SD), which is a bonus of disentangling different concepts (see Fig. 6(b)). This still holds when integrating Magnet with Attend. We tested Attend+Magnet on CC-500 via GroundingDINO. The Det.($\\uparrow$) result is 87.7, compared to 84.3 w.r.t the original Attend. Attend+Magnet can encourage all objects and improve attributes simultaneously (see Fig. 8). Since there is no available API at this moment, we cannot evaluate Magnet with GPT-4o. But we will definitely try out GPT-4o whenever possible. \n\n**Q1**: The 4-case experiment is deeply connected to our method. Lines 64-78 and Fig. 2(a) focus on comparing how the context information in the word embedding affects generation (i.e., from case 1 to 2, or case 3 to 4), and how paddings' context information affects (i.e., from case 1 to 3, or case 2 to 4). Appendix A.2 provides a detailed analysis of these fine-grained cases. We discern that the attribute information is rich in word embeddings but entangled in padding embeddings, and modifying these word embeddings will not change the image layout as significantly as modifying padding embeddings. These have motivated us to introduce the binding vector, adaptive strength, and neighbor strategy (see lines 439-443). \n\n**Q2**: We did consider modifying the padding tokens, but for the following reasons it didn't work: \n1. These padding embeddings strongly entangle different concepts, especially given complex prompts. It is hard to manipulate one specific object from these paddings; \n2. As shown in Fig. 2 and Fig. 12 in the main paper, modifying the padding (from cases 1 to 3) will change the image significantly compared to modifying the word embedding (from cases 1 to 2). We consider not changing the non-attribute part can better maintain the pre-trained model's capability;\n3. As shown in Fig. 1 in the attached PDF, the padding embeddings are less activated than word embeddings, especially in U-Net's last two upsampling blocks and the later diffusion steps, which are more crucial for generating semantic features. \n\n**Q3**: In our comparison experiment, we input all object words in the prompt to GroundingDINO. For example, given the prompt \"a blue apple and a green vase\" with two objects, GroundingDINO's input is *\"apple . vase .\"* without any adjective. We have considered inputting adjectives, i.e., *\"blue apple . green vase .\"*. However, no matter what color the apple is (e.g., blue, red, or green), GroundingDINO in both settings will detect it. That's why we use GroundingDINO to assess *object existence* only, but not *attribute alignment* (see lines 177-178). Appendix Fig. 15(c) presents GroundingDINO's failure cases. We further conducted the following experiment to make our judgment sound. Given 50 images (generated by the prompt \"a blue apple and a green vase\"), GroundingDINO detected $38$ (w/o adjective), $38$ (w/ adjective), human annotated $\\sim 20$ \"blue apple\", maximum is 1 per image. This comparison shows that GroundingDINO is not reliable for measuring *attribute alignment*.\n\n**Q4**: We agree with your idea that the CLIP's embedding does not effectively bind positional information to the desired token. However, this makes an analysis like Fig. 1 difficult - we don't know which token to identify. We conjecture that the positional information is used in the early diffusion steps, where all tokens have relatively small attention values (see Fig. 1 in the attached PDF). In this case, the entangled padding embeddings may affect generation. This also explains why Magnet modifying the word embeddings cannot address the positional problem. However, we can integrate Magnet with layout-based methods to handle position (see Fig. 2(c) in the PDF). Except for colors, Magnet is capable of binding other attributes (see Fig. 2(a) row 3 and 2(b) row 2 in the PDF). The final version will include a detailed discussion of positional relationships and more examples.\n\nFinally, the two textual-based image editing works are impressive. We will cite the given papers and discuss the differences in the final version.\n\n[1] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aW6s822T0g", "reviewer": "Reviewer_5FTv", "summary": "- This work studies how CLIP text embeddings commonly used in text-to-image diffusion models affect attributes in generated images, and how attributes can be bound to the correct objects during generation.\n- There is an analysis of the (a) CLIP text encoder and how it interacts with the padding used during T2I diffusion (b) how text embeddings of different nouns relate to different attributes w.r.t to distance (c) how the previous two observations interact during diffusion-based generation.\n- An algorithmic innovation — Magnet — is proposed. Magnet introduces a binding vector that can be applied to the embedding of a noun-object to bind an attribute to it so that it is faithfully applied during generation.\n- There is a human evaluation (amongst other evaluations) which shows that Magnet improves attribute-object bindings.", "questions": "- Have at least one *simple* figure that explains the *intuition* behind the approach.\n- Rewrite 3.1 to be more clear or at least provide examples of all of the $\\mathcal{P}$-terms, I am not even sure the notation is correct here.\n- More generally, provide a comprehensible explanation of the method so I can confirm my understanding is correct. I'm uncomfortable providing a higher score otherwise, though I would like to, given the amount of work that seems to have gone into the paper.", "limitations": "Not applicable.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is organized very well. I appreciated the bolding.\n- The in-depth analysis of what causes the attribute binding problem was interesting and would be widely useful to the community. \n- Human evaluation shows that attribute alignment is substantially improved by Magnet. I appreciated the use of a human evaluation rather than merely using automated metrics.\n- Magnet is much cheaper w.r.t runtime and memory usage than competing methods.", "weaknesses": "I had great difficulty understanding most of the figures. Even after understanding the method, I cannot understand what Fig 3. is showing and how it relates to the method. Similarly, I found the equations and the notations cumbersome, illegible, and frustrating. \n\nI am honestly unsure if I understand the method, because the writing is very unclear (though well organized) and the notation is completely overwrought. \n\nOne minor weakness is that the analysis is highly specific to the CLIP text encoder. I don't know how much of a weakness this is practically since the CLIP text encoder is a de-facto standard, but I am writing it here anyway for completeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your positive comment on the contribution of this work is much appreciated. We apologize for any difficulty you may have experienced in following this paper. Hope the following example and the illustration in Fig. 3 in the attached PDF will help you to understand these $\\mathcal{P}$-terms and our method. \n\nConsider an input prompt $\\mathcal{P}=$\"a blue book and a red cup\", we use an off-the-shelf parser to extract concepts \"blue book\"($A_1\\\\&E_1$) and \"red cup\"($A_2\\\\&E_2$). The term $\\mathcal{\\tilde{P}}$ refers to a new prompt that involves only one object (in line 110, described as \"out of the current context of $\\mathcal{P}$\"). For instance, when operating the object \"cup\" (i.e., $i=2$), we define positive concept $\\mathcal{\\tilde{P}}^{pos}_2$=\"red cup\"($A_2\\\\&E_2$), negative concept $\\mathcal{\\tilde{P}}^{neg}_2$=\"blue cup\"($A_1\\\\&E_2$), and unconditional concept $\\mathcal{\\tilde{P}}^{uc}_2$=\"cup\"($\\varnothing\\\\&E_2$). The function $\\mathcal{F}$ extracts the embedding w.r.t. a word in one specific prompt (illustrated by the red box). **Notice that the same word in different prompts with varied contexts will produce different word embeddings**, i.e., $\\mathcal{F}(E_2,\\mathcal{\\tilde{P}}^{pos}_2)\\neq \\mathcal{F}(E_2,\\mathcal{P})$. Based on this fact, we introduce Eq. (1)(2) to estimate the binding vectors $v^{pos}_2,v^{neg}_2$ for \"cup\". The vector $v^{pos}_2$ identifies the direction towards \"red\", while $v^{neg}_2$ is the direction towards \"blue\", specifically for \"cup\". Similarly, we obtain the binding vectors $v^{pos}_1,v^{neg}_1$ specifically for the object \"book\"($E_1$). \n\nWhen involving neighbors, we compute $K$ objects close to the \"cup\" embedding in the feature space (lines 129-132), denoted $\\\\{B^{(2)}\\_k\\\\}^{K}\\_{k=1}$, where superscript $(2)$ refers to the current object \"cup\" $i=2$. Next, we replace \"cup\" in three constructed prompts with each neighbor object. For instance, the second neighbor \"mug\" ($B^{(2)}_2$, where subscript refers to $r=2$) will produce the positive concept $\\mathcal{\\tilde{P}}^{pos}_2$=\"red mug\"($A_2\\\\&B^{(2)}_2$), negative concept $\\mathcal{\\tilde{P}}^{neg}_2$=\"blue mug\"($A_1\\\\&B^{(2)}_2$) and unconditional concept $\\mathcal{\\tilde{P}}^{uc}_2$=\"mug\"($\\varnothing\\\\&B^{(2)}_2$). The same process is for the remaining neighbors to obtain $K$ positive and $K$ negative vectors. The final binding vectors $v^{pos}_2,v^{neg}_2$ used to modify \"cup\" are averaged as Eq. (5)(6).\n\nThe adaptive strength in Eq. (3) improves robustness for practical use. For the object \"cup\" ($i=2$), we extract the first [EOT], i.e., $\\mathcal{G}(\\mathcal{\\tilde{P}}^{pos}_2)$, and the last padding embedding, i.e., $\\mathcal{H}(\\mathcal{\\tilde{P}}^{pos}_2)$, to compute $\\omega_2$ and two strengths $\\alpha_2,\\beta_2$. Note the used prompt to obtain $\\omega_2$ is not the input prompt $\\mathcal{P}$ but the constructed positive concept $\\mathcal{\\tilde{P}}^{pos}_2$. Our motivation for this strategy is described in lines 116-119. Similarly, $\\omega_1, \\alpha_1,\\beta_1$ are calculated for the object \"book\" ($i=1$).\n\nFinally, we modulate the original object embedding $c\\_{E\\_i}$ using the adaptive strengths $\\alpha_i,\\beta_i$ and the estimated positive and negative vectors $v^{pos}_i,v^{neg}_i$, denoted $\\hat{c}\\_{E\\_i}$ as Eq. (4). Note Magnet does not manipulate the cross-attentional activations as existing works do. As described in Section 3.3, Magnet only modifies the embedding of each object word in the input prompt and treats the denoising U-Net as a black box to generate the image.\n\nMagnet provides a simple but effective way to identify the attribute direction for each object, as we said in line 98, to attract the target attribute (i.e., positive vector) and repulse other attributes (i.e., negative vector). Experiments show Magnet disentangling different concepts and improving attribute alignment (see Fig. 4,5 in the main paper).\n\nIn Fig. 2(b) in the attached PDF, we show Magnet is not limited to the CLIP text encoder, improving the text alignment and synthesis quality of PixArt [1], which uses T5 as the text encoder. \n\nWe sincerely hope that the above explanation will clear up the confusion. We will improve the method section to make this paper more readable.\n\n[1] PixArt-$\\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wFOEpgfuO", "reviewer": "Reviewer_eb4q", "summary": "The paper investigates the text embedding representation of text-to-image models in the context of stable diffusion. In particular, the authors find that object features often bind to their commonly associated attributes and propose the Magnet approach that interpolates object features with their designated attributes (positively) and attributes designated with other objects (negatively) specified in the prompt. To determine the strength of the interpolation, the authors leverage the similarity between the EOT and the last PAD token. In addition, for any object word, the authors retrieve their neighboring words based on feature and semantic similarity to further twist the binding vectors, resulting in enhanced concept disentanglement.", "questions": "1. Could authors add reference to Stanza’s dependency parsing module in line 99?\n2. Given attribute binding is a core task in compositional text-to-image generation, could authors also discuss more recent and/or relevant in related works, such as [4, 5, 6, 7]?\n3. Can authors also evaluate on image quality metrics such as FID [9] and compare with other methods?\n4. I wonder if authors could compare Magnet to this concurrent work [8]? As its preprint came out this March, there's no need to experimentally compare and contrast. Some discussions would be appreciated.\n\n[4] Training-Free Layout Control with Cross-Attention Guidance  \n[5] TokenCompose: Text-to-Image Diffusion with Token-level Supervision  \n[6] RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models  \n[7] Compositional Visual Generation with Composable Diffusion Models  \n[8] Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions  \n[9] GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium", "limitations": "The paper discussed limitations in missing objects, possibilities of over- and under-manipulation as well as failures in correcting position relationships of the objects with specific examples, which are justifiable given the scope of their method.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is novel in that it mitigates attribute binding problem from the text-encoder perspective instead of iteratively refining the cross-attention activations that most previous works focused on. This brings both enhanced results and improved efficiency comparatively.\n2. The paper's idea is well-motivated with empirical evidence (e.g., using the cosine similarity of PAD and EOT to decide the strength of embedding interpolation by showing that there's more decay in attribute-specific information in later PAD tokens when such attributes are uncommon).\n3. The paper is well-written with many design choices are well-justified (e.g., use of human evaluators, strength formula, necessity of both positive and negative binding vectors, etc).", "weaknesses": "1. The scope of the work is limited -- it only covers fixing attribute binding problems in text-to-image generation, one downstream task that is commonly encountered in compositional generation [1].\n2. The method is model-specific. While it alleviates text-to-image models with a CLIP-based text encoder to a significant degree, it is unknown if this method also improves text-to-image models that use multiple text encoders [2] or different text encoders [3].\n3. I find the neighbor finding procedure a bit awkward. For example, the authors have to manually gather 614 object nouns as feature neighbors and have to prompt GPT-3 to gather semantic neighbors, limiting the applicability of the work toward a completely automatic pipeline (e.g., without manual curation or LLM-assisted selection).\n  \n[1] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation  \n[2] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis  \n[3] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we would like to express our sincere gratitude for reviewing our manuscript and providing valuable feedback. Below are our responses to the weaknesses (W) and Questions (Q).\n\n**W1**: We acknowledge that attribute binding, as our main focus, is part of compositional generation. However, since Magnet performs outside the U-Net, it is plug-and-play for layout-based methods (see Fig. 2(c), layout-guidance+Magnet) or add spatial control (see Fig. 2(d), ControlNet+Magnet) to address the object relationships. Additionally, we have compared Magnet with GORS [1] (see Fig. 2(e) in PDF). Whether trained on color or complex datasets, GORS cannot demonstrate the anti-prior ability and disentangle concepts like Magnet. Meanwhile, it fine-tunes both the CLIP text encoder and the U-Net with LoRA is time-consuming to adapt to different T2I models, while Magnet is plug-and-play. Although the focus of this work is on attribute understanding, we sincerely hope that our work can provide new insights to the community and motivate further work investigating compositional understanding of T2I models.\n\n**W2**: We have applied Magnet to SD 2.1 (another version of the CLIP text encoder), SDXL [2] (multiple text encoders), and PixArt [3] (T5 model) in Fig. 2(a)(b) in the attached PDF. Magnet also improves text alignment and image quality, showing the anti-prior ability. We will add these results to our final version.\n\n**W3**: Magnet is completely automatic during evaluation. Though we describe that we gathered 641 nouns \"manually\", this procedure is one-for-all. We compute 641 embeddings once for each new text encoder and save them to the local path. In practice, given any prompt, only need to load this local file to search neighbors and encode a few new prompts (e.g., a prompt with 2 concepts, $\\sim5\\times2\\times2$ new prompts are automatically constructed and need to be encoded if $K=5$). This process is fast as shown in Tab. 2 where Magnet has a negligible increase in runtime and memory usage compared to StructureDiffusion and Attend-and-Excite. \n\n**Q1**: Your careful review is much appreciated. We will cite Stanza correctly in the main paper, the same as line 466.\n\n**Q2**: Here is our justification for Magnet different from other works: [4], [6] rely on the layout constraints, which is an additional condition and not included in the vanilla diffusion models, while Magnet only requires the input prompt. We cited two similar works in lines 252-253. [5] controls *objects* in prompts, cannot improve the text alignment for *attributes* like Magnet. [7] composes different noise latents and needs multiple diffusion processes to obtain the target latents. Magnet operates the text embedding outside the U-Net and performs only one diffusion process, which adds negligible cost to the original model and is more efficient. We will discuss these mentioned works in related works in the final version. \n\n**Q3**: We have evaluated on FID for two SD versions (v1.4 and v2.1). We follow the standard evaluation process and generate 10k images from randomly sampled MSCOCO captions. The result is SD v1.4 (19.04), +Magnet (18.92); SD v2.1 (19.76), +Magnet (19.20). This shows that Magnet will not deteriorate the image quality while improving the text alignment.\n\n**Q4**: Thanks for recommending this impressive work. We found some ideas in this paper aligned with ours. But here are some major differences to point out: \n\n1. It lacks interpretability. In our paper, we analyze the CLIP text encoder and the diffusion model by the 4-case experiment in Fig. 2(a), pointing out the entanglement of the padding embeddings. This has motivated our method and explains why Magnet works (i.e., using the binding vector to enhance the distinction between concepts and improve disentanglement); \n\n2. It defines \"positive and negative prompts\" that are semantically contrastive, e.g., \"young\" vs. \"old\" for the \"age\" attribute (may need to manually specify per attribute). In Magnet, positive and negative attributes are obtained from the given prompt automatically, and no need to be semantically opposite. We also introduce unconditional prompts as pivots, which is suitable for attributes that are hard to define their opposite attributes (e.g., what is the contrastive adjective of the color \"yellow\"?); \n\n3. Its direction vector is learned with a loss function to ensure robustness. According to the experiment setting, it uses fixed strength (edit delta). Differently, we propose the neighbor strategy and adaptive strength to improve the vector estimation. Encountered with a new attribute, [7] needs a training process to seek this direction, while Magnet can be directly applied. \n\nIn the final version, we will discuss this concurrent work [8] in more detail.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In the attached PDF file, we provide a new perspective on how the word and padding embeddings affect generation (see Fig. 1), additional examples applying Magnet to different T2I models and other techniques (see Fig. 2), and a new Magnet pipeline figure for ease of understanding (see Fig. 3).

We have read all the papers mentioned by the reviewers and would like to highlight some differences as follows: 

1. Existing works analyzing both word and EOT/pad embeddings emphasize the semantic effect of the word embedding, while we point out the entanglement of the pad embeddings to explain existing problems of SD (e.g., incorrect attributes, indistinguishable objects, see Appendix Fig. 13); 

2. Related works require additional inputs or fine-tune the denoising U-Net, whereas Magnet takes only the given text as input and can be applied directly to any prompt; 

3. Operating outside the U-Net, Magnet provides a plug-and-play capability and can be readily integrated to existing T2I models (e.g., SD, SDXL [1], PixArt [2]) and controlling techniques (e.g, Attend-and-Excite, layout-guidance [3], ControlNet [4]); 

4. Magnet shows the anti-prior ability, i.e., to generate high-quality images with unnatural concepts, while related works are incompetent (see Appendix Fig. 24, and Fig. 2(e) compared to GORS [5] in the attached PDF). 

Last but not least, we sincerely hope that our work will motivate the community to explore generative diffusion models and discover other interesting phenomena.

[1] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

[2] PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

[3] Training-Free Layout Control with Cross-Attention Guidance

[4] Adding Conditional Control to Text-to-Image Diffusion Models

[5] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation

Meta-review: This paper investigates the attribute binding limitation in text-to-image diffusion models and introduces the Magnet approach, which enhances attribute-object alignment by interpolating object features with their designated attributes (positively) and attributes designated with other objects (negatively) specified in the prompt.

Summary Of Reasons To Publish:

1) Novel approach with interesting and in-depth analysis of the attribute binding problem

2) Well-organized and well-motivated work

3) Strong evidence of utility of proposed approach with human evaluations

Summary Of Suggested Revisions:

All the major concerns raised by the reviewers are addressed in the rebuttal to some extent. I suggest authors to incorporate the feedback on the presentation and clarity issues in the final version.
Decision: Accept (poster)