A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization
Authors: Chieh-Yun Chen
Chiang Tseng
Li-Wu Tsao
Hong-Han Shuai
Venue: NeurIPS 2024 poster (2024)
Keywords: Causal manner
Embedding optimization
Information mix-ups
Information loss
Text-to-image generative model
Primary area: generative_models
TL;DR: Analyzing the effect of causal manner and text embedding in text encoders inside text-to-image diffusion models

Abstract: This paper analyzes the impact of causal manner in the text encoder of text-to-image (T2I) diffusion models, which can lead to information bias and loss. Previous works have focused on addressing the issues through the denoising process. However, there is no research discussing how text embedding contributes to T2I models, especially when generating more than one object. In this paper, we share a comprehensive analysis of text embedding: i) how text embedding contributes to the generated images and ii) why information gets lost and biases towards the first-mentioned object. Accordingly, we propose a simple but effective text embedding balance optimization method, which is training-free, with an improvement of 125.42\% on information balance in stable diffusion. Furthermore, we propose a new automatic evaluation metric that quantifies information loss more accurately than existing methods, achieving 81\% concordance with human assessments. This metric effectively measures the presence and accuracy of objects, addressing the limitations of current distribution scores like CLIP's text-image similarities.

A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-lmage Encoders through Causal Analysis and Embedding Optimization
Chich-Yun Chen R
Chiang Tseng x
Li-Wu Tsao
Hong-Han Shuai
& National Yang Ming Chiao Tung University Georgia Institute of Technology
Pcchen859@gatech.com edu & {chiang. ee11,
hhshuai]@nycu, edu. tw
Abstract
This paper analyzes the impact of causal manner in the text encoder of text-to-image (T2I) diffuusion models, which can lead information bias and loss. Previous works have focused on addressing the issues through the denoising process. However, there is no research discussing how text embedding contributes to T2I models, espe- cially when generating more than one object. In this paper, we share comprehen- sive analysis of text embedding: i) how text embedding contributes to the generated images and ii) why information gets lost and biases towards the first-mentioned object. Accordingly, we propose a simple but effective text embedding balance optimization method, which is training-free, with an improvement of 125.42% on information balance stable diffusion. Furthermore, we propose new automatic evaluation metric that quantifies information loss more accurately than existing methods, achieving 81 % concordance with human assessments. This metric effec- tively measures the presence and accuracy of objects, addressing the limitations of current distribution scores like CLIP's s text-eimaaee image similarities. The code is available: htttps://githue wtw.thu..commaaaa http///aaaccabbbbaaallliin. /ur//vuaavennnn.. Inf information. Mix ups
Introduction
Object mixture
Obiect missiig
lion
elephant
chicken
dog
Text-to-image (T2I) diffusion models []]
12
14 17 18] have recently captured significant
attention. Subsequent research [4]
g
13
16
20, 3 has extensively explored the roles of cross-attention and self-attention mechanisms in the denoising process to enhance image con- trol. Despite these advancements, there remains a critical gap in understanding the role of text embedding within T2I models, particularly in scenarios involving the generation of multiple objects. For instance, Fig. Iillstraaes that when prompted with "a lion and an elephant," mod- els like Stable Diffusion often generate an am- biguous creature that blends features of both (or object missing on the right), highlighting issues of semantic interpretation and token embedding. This paper delves into how text embeddings in- fluence the semantic outcomes of generated images, identifying specific problems of information bias and loss due to the causal nature of the self-attention mechanism in text encoders.
lion and an elephant chicken and dog Figure Visualization cross-attention maps
when object mixture and missing occur.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The existing literature [ 16 20 13 has worked in image latents to address information loss, e.g.. objects missing; however, there is no research on the main cause of the problem- text embedding. Therefore, this paper focuses on text embedding to investigate the issue. We first analyze the information bias in text embedding and find that the generated objects tend to bias towards the first mentioned object, as Table shows, when the given prompt contains more than one object. The reason for biasing towards the first mentioned object is due to the causal manner in which making nt token embedding contains the weighted attention of token embeddings between 0 to (n - -1)th tokens, as illustrated in the right-bottom side of the Fig. 2 Moreover, we analyze the semantic contribution of the token embeddings by masking different tokens. Table and Fig. B demonstrate that the causal manner would contribute to the accumulation of general information in the special token, e.g., end of token (<eot>), and padding token (<pad>). Masking the embeddings of given tokens still allows the T2I model to generate the expected information using the remaining special tokens' embeddings.
Due to the significant information loss issues in T2I models, such as object mixing and missing, we dissect the generative process of the T2I model to pinpoint the origins of these losses. In the text embedding, the causal manner would make the embedding of the ntth token mixed with the token embedding between to (n - 1)th, which makes the later token embedding similar to the earlier embedding. In the case of prompt "a cat and a dog", the causal manner would mix the <dog> embedding with the <cat> embedding. This similarity in embeddings results in similar distributions on cross-atted attention maps, detailed in See 4.4, When denoising directions these maps align too closely throughout the denoising steps, it can cause a mixed representation if responses are equally low, or object may overshadow the other if its map elicits stronger response, leading object disappearance. These phenomena are visualized in Fig. [ To address the issue of information bias and loss, we propose the Text Embedding Balance Optimization (TEBOpt) to promote distinctiveness between embeddings of equally important objects for preventing mixing and working alongside existing image latent optimization techniques to address object disappearance. The main contributions of this paper are outlined as follows:
This paper examines how text embedding contributes generated images in text-to-image diffusion models and demystifies how the causal manner leads to information bias and loss while contributing to general information.
We propose the Text Embedding Balance Optimization solution containing one positive and one negative loss to optimize text embedding for tackling information bias with 125.42% improvement in stable diffusion.
We propose evaluation metric to measure information loss. Compared to the CLIP score for evaluating text-image similarity, and the CLIP-BLIP score for evaluating text-text similarity, our evaluation metric provides concrete number for identifying whether the specified object exists in the generated image.
2 Preliminaries
Text-to-image diffusion models [|] 12 14 17 18 typically contain the text encoder 115 6 variational autoencoder (VAE), and a denoising UNet, Fig. demonstrates. Given text prompt, the text encoder would first obtain the text hidden states h, E R-ND from the sum of token embedding and positional embedding. where N the maximum token length in the text encoder and D represents the embedding dimension; then, it calculates the text embedding by going through the encoder layers with self-attention mechanism and causal masking manner. Next, given the text embedding E and the initial image noise 2,, the denoising UNet co(2,, E, t) would gradually denoise latents each timestep f to get the final image by iteratively predicting the noise residuals conditioned on the text embedding and the previous denoised latents.
Notably, the causal masking manner in the text encoder makes every token have information only from its previous tokens, which causes the text embedding to have information bias. The bottom of
Fig. 2 iillsstraees how the causal manner works in the self-attention mechanism 21 key K, and value V are calculated as follows:
The query Q.
Q= (o(he),
K kx(x(h)), V V= (ho),, where K,Q, V E R XNxD/h.
(I)
2

---

Text Embedding Balancing (Ta.
Denoising UNet
Decoder
"A photo of cat"
CLIP text encoder
Ep(t)
Zg
"A photo of dog"
CLIP text encoder
Ep(O)
Traa
Text-to-image
generative process
Obsain embeedking
Token Embodding
"A cut and a dog"
ho
Text
2t
Encoder
Positional Embodding
(cuuall manrer)
CLIP
Denaistng UNet
Decoder
Causal manner in text encoder (Demonstrating in single uttention head)
Text embedding
<300>>
A1n Ar
Raz
Saz
Oae
Foi
A
R1:
olsetos di clasto
cat
Ras R2s
and
41g 03g
Pas
Eg1
d'echto + tha d' + td +222 + +
d'oePow + dl
Softmax
Ras
V
dog
4oa Oss
52007
Rsa 0.
Des Pas Cas
dla + Pan to sellon t +a
Egt dog
tn +iaa aa
Spad
Pig Bus
Hes Rog
Part
QI + +M
Egg: spad>
d'
Σ dana
Σ d' Vin
Figure 2: Overview the text-to-image generative model, including the details the causal manner in attention mechanism. Because of the causal nature of the embedding, information is accumulated from the starting token through the end of the sequence, resulting in bias in the earlier token. To balance the critical information, we propose token with equal weights within their corresponding embedding dimension.
embedding optimization for purifying the object
(QK Vdx
+M
if S i if > i -00,
0,
Attention(Q... (Q.KV)) = softmax
V,
Mi
(2)
where la2 lx, and yy are learned linear projections, and lt, represents the head number. de the dimension of and M is the causal mask. Upon receiving the attention weight QK E NNN the mechanism applies the causal mask M. Then, taking V weight metric to get the weighted sum of attentions, representing the embedding for each token. Due to the causal manner, the embedding of the th token (,,, contains weighted information for the tokens 0 n-11 1. This representation causes two issues: First, the earlier token information accumulates the most since every subsequent token has it. Furthermore, the later token's embedding not pure representation of itself, potentially causing identity loss. For example, given text prompt "a cat and dog", the earlier token <cat>'s information would be mixed into the embedding of the later token <dog>. It makes the generated image have higher possibility of generating two cats, as the middle row of Fig. 2 shows.
3 Analysis and method
In this section, we analyze how causal manner affects the text embedding as well as the generated images in text-to-image diffusion models.
3.1 Information bias in the text embedding
We investigated 400 different prompts with different random seeds with structures (a) "a/an <objectl> and a/an <object2>" and (b) "a/an <object2> and a/an <objectl>" by exchanging the position of <objectl> and <object2> in prompt (a), where objects are randomly sampled from 17 different animals.

---

<sot> A dog and a cat <cot>
<sof dog and eat <eoT>
<
dog and cat <eot>
<sor> A dog and car eeo?>
<8of> A cut and a dog <eot> <sof>
A cat and a dog <eop>
<2of>
eat and a dog <eot>
<20f> A eat and dog <eot>
Figure 3: Masking text embedding to identify the contribution of critical tokens, e.g., cat/dog, and special tokens, e.g., <sot>, <eot>, <pad>. The first row and the second row both contain cat and dog inside prompt but in different order. The analysis shows that special tokens contain general information about the given prompt. However, the cat/dog tokens carry more weight than the special tokens. In the last two columns, where one the animal token embeddings is masked while retaining the special tokens' embedding, the generated image is predominantly influenced by the remaining animal's token embedding.
Table shows that within the same set of two animals and the same random seed, both bias
(a) A/An <objl>
(b) A/An <obj2> and a/an <objl>
Prompt
and a/an <obj2>
on the earlier mentioned animal. In prompt (a), the statistics with pink background show that causes 67.5% missing issue, including 46.0% generating only object 1. Similarly, in prompt (b), it causes 69.5% missing issue, including 47 47.0% generating only object 2. The informa- of obji exist tion bias (Info bias) is defined by #of only obj2 exist and, both prompts, they 2.30 and 0.46 respectively, which are far from balanced (Info bias 1).
2 objects exist mixtures only obj1 exist
12.25%
11.75%
20.25%
18.75%
46.00%
21.75%
only obj2 exist
20.00%
47.00%
no target object
1.50%
0.75%
Info bias
2.30
0.46
Table 1: Both prompts strongly bias towards the first mentioned object. The bias generally exists in more objects, reported in Supplement D
3.1.1 Causal effect for generating images with more than one object
Driven by the observation of information bias, we investigated the text embedding contribution of critical tokens. In Stable Diffusion, we first utilize CLIP [5] as the text encoder to obtain the text embedding E; E N XD where i E [0, N 1]. Before the denoising UNet gets the text embedding and image latent, we mask the chosen embedding and generated the corresponding images. Table 2 provides the quantitative analysis in 400 different prompts with the same setting of (a) used in Sec. 3.I extending with three masking settings corresponding to Fig. 3 each column.
Firstly, the default experiment in Table. demonstrate stable diffusion has 20.25% mixture, 67.50% missing ratio, and 2.3x information bias to the first mentioned object. Regarding the visualization sample of the default experiment, the first column in Fig. shows that animals (cat and dog) within one prompt in different orders with the same initial image latent would generate different results. The bottom one demonstrates the mixture issue, making the right animal simultaneously similar to dog and cat. Regarding the second experiment, we mask the embeddings of the given tokens, where -inf,, i [1, 5]. It reduces 2.5% mixture rate but increases 11.75% missing rate. With only

---

one object existing, the existing rate for object 1 and object 2 is more balanced as the embeddings of all given tokens are masked. suggests that the remaining embeddings, which are special tokens. including <soD, <eot>, and <pad>, contain the information of the masked embeddings due to causal manner in the text encoder.
An important insight here is that eliminating dominant embeddings reduces the mixture rate. One of the causes of the mixture issue is the attention mixture, as the text embedding in Fig. 2 shows. Given a prompt "a cat and a dog", as the text embedding is projected as key and value multiplied with the query to get cross-attention maps during denoising, the <cat> em- bedding would trigger the <cat> response while the <dog> embed- ding would trigger both <dog> and <eat> responses, causing imal mixtures. Once one of the object's 's responses dominates, the other animal disappears, leading to missing issue. In the third and fourth columns, we mask one of the animals' text embeddings, resulting in generating only the remaining animal. These experiments suggest that, firstly, causal manner makes the information accumulate from earlier token to later token. Even masking critical text embeddings, special tokens, i.e., <sot>, <eot>, <pad>, can generate the given prompt information. In addition, although special tokens accumulate information to form general embeddings, when the text embeddings contains critical tokens' embeddings, the critical token would lead the generated results, as the last two columns show.
Analysis
Default
Mask 1-5
Mask 1-3
Mask 3-5
2 objects exist only mixture
12.25%
3.00%
2.75%
0.50%
10.00%
11.50%
5.25%
3.50%
obji + mixture
6.50%
2.75%
0.00%%
0.75%
obj2 + mixture mixture sum
3.75%
3.50%
3.75%
0.00%
20.25%
17.75%
9.00%
4.25%
only objl exist 46.00%
33.25%
3,00%
89.00%
only obj2 exist
20.00%
42.00%
82.00%
1.75%
no target objects
1.50%
4.00%
3.25%
4.50%
missing sum
67.50%
79.25%
88.25%
95.23%
Info bias
2.30
0.79
0.04
50.86
Table 2: Analysis of masking token embeddings. Masking all the given token would reduce the mixture issue but increase the missing issue with balanced object and object 2 existing rate. Masking one the objects would not completely eliminate the masked object's information but would significantly reduce its existing rate. The implementation details are in Supplement
3.1.2 De-causal effect for generating images with more than one object
Hypothesis. Replacing the token embedding of later mentioned object from the corresponding pure embedding, the hypothesis expects to solve the bias problem and information loss.
Following the setting of the prompt structure (a) in Sec. 3.1 with text prompt "a/an <objl> and a/an <obj2>", the token index of <objl> and <obj2> are taken into the critical token set O f2, 5). first generates the pure embedding of the later mentioned one, "a photo of <obj2>" (Eroug)) Then, replace the corresponding token embedding <obj2> in the original embedding (3) with the pure one. The combined text embedding (E') is as follows:
Prompt
Default Hypothesis
objects exist
12.25%
7.00%
only mixture
10.00%
10.50%
objl mixture
6.50%
3.25%
obj2 mixture
3.75%
3.50%
only objl exist
46.00%
44.00%
only obj2 exist
20.00%
30.25%
no target object
1.50%
1.50%
Info bias
2.30
1.45
#i or i= =Om else
e' c'=
(3)
Table 3: Analysis of Hypothesis. Replacing the token embedding of later mentioned object from the cor- responding pure embedding can bal- ance the information but lead to large drop of two objects coexistence.
Eppatnn 5
where refers to the th object. Table that directly replacing original embedding with the pure embedding would balance the information of object and 2; however, it would result in a 5.25% loss in 2 objects coexistence. nearly equal probability of generating only objects and 2 proves that replacing token embedding with pure embedding eliminates accumulated information about object
3.2
Method: balancing critical information in the text embedding
While the causal manner results in information bias and information loss, it contributes to generate image content aligned with the general prompt information. To eliminate the accumulated information
5

---

but retain general information, we design a Text Embedding Balancing Optimization, called TEBOpt (uppermost Fig. 2). cooperating with image latent optimization address these issues. The TEBOpt tackles information bias and object mixture issues while the latent optimization tackles object mixture and missing issues. Since the missing object caused by an insufficient response value and an inadequately activated region in the cross-attention map corresponding to that object, and the text embedding is unable to precisely determine the activated position, our method cooperates with existing latent optimization methods to address this issue.
Regarding a prompt containing two objects, we expect their corresponding text embedding to be unmixed instead of mixing earlier tokens' embeddings. Since the ultimate goal is to preserve the general information of the two objects, we cannot directly replace the original embedding with the pure object's 's text embedding. This would result in high probability of losing one of the objects. Thus, our proposed TEBOpt contains TEB loss in order to encourage the later mentioned token's embedding to be less similar to the earlier token's 's embedding, while at the same time being similar its pure embedding. Considering text prompt with & objects in a set O, we first obtain each text embeddings En N and take the critical token embedding Ep.i E R R the pure embedding. For example, the prompt "a dog and a cat" contains two objects and the pure prompt embedding is calculated in format prompt of ("a photo of <dog>", "a photo of a <cat>"]. In summary, the TEB loss is as follows:
C TEB Ep) min sim(E iEO
(4)
122
Chey CTEB(E)
sim(Er,
(5)
k(m-1)
iE0
j=1
8#1
CTBB TEE
LTEB
(6)
where sim o) = |u vill and m means the effective token count, where we do not include <sot> and <eot> for optimization. The implementation details are included in Supplement B After the
text embedding is optimized, the image latent would be updated conditioned on the loss design for cross-attention maps during the denoising process. For example, A&E [ contains a loss function to ensure that each selected token activates some image patches in the cross-attention map. SynGen [16] designs a loss function to encourage the cross- attention map of the relative token to similar and make the cross-attention map of the unrelative token dissimilar.
4 Experiment
4.1 Experimental settings
Baselines. We compare our proposed method with the default Stable Diffusion 1.4 [17]. and ofate-of-the-art (SOTA) baselines, including Structure Diffusion [3], Attend-and-Ea
[1], and SynGen [16]. All of them focus improving attribute bindings solving object missing in text-to-image diffusion models. However, there is no one considering the information bias. this literature, the objective is to analyze information balance caused by pretrained text encoders instead of surpassing existing SOTAs on solving object missing. Therefore, we would provide the experimental results of our proposed method on top of the baselines.
Data. We follow previous methods [ [20 to create a set of 400 prompts with the format "a/an <objl> and a/an <obj2>" with corresponding random seeds. The objects are sampled from 17 different animals defined by previous methods [ [0], including cat, dog, bird, bear, lion, horse, elephant, monkey, frog, turtle, rabbit, mouse, panda, zebra, gorilla, penguin and chicken.
4.2 Evaluation metrics
To analyse the issues of mixed objects and missing objects in generated images, we designed an automated evaluation method since existing metrics, e.g., text-image similarity using CLIP [155 or text-text similarity using CLIP and BLIP (7], used in SOTAs 20 cannot provide the exact counting number to indicate whether the object exists not. Detailed discussions are provided in Supplement

---

First, we employed pre-trained object detection model, OWL-ViT [TO], open- open-vocabulary object detection. We separate the text prompt into k objects and the model separately predicts bounding boxes and confidence scores for the corresponding objects. Mixture status is determined when the overlap the two bounding boxes for different objects exceeds 90%. Here, we use two different thresholds to detect mixture objects and single objects, ensuring detection accuracy. Additionally, to validate the effectiveness of this automatic metric, we conducted human evaluation to demonstrate that its results are highly correlated with human perception. We asked users to label 400 generated images, each categorized into one of five options: i) two objects exist, ii) mixture exists, iii) missing object iv) missing object 2, or v) no objects exist. Our automatic evaluation metric achieves an accuracy of 81 % based on human responses, demonstrating its effectiveness.
which is a SOTA in
4.3 Qualitative results
Stable Diffisiion
Stable Diffusion Lysa
Structure Diffusion
A&E
A&E + LyeB
SynGen
SyaGen
Lyen
u. dedegg b.urre
at E.hoe and da
Figure 4: Qualitative comparison of all methods. Every prompt uses the same seed.
Object mixture
Object missing
elephun! and a penguun
zebra and a monkey
dog and horie
penguir and an elephani
chicken and dog
monkey and an elaphant
wi Lras
Vo Lras
Figure 5: Qualitative comparison for the generated image with vs. without LTEB in Stable Diffusion 1.4. Every prompt uses the same seed.
Fig. 4 demonstrated the visual comparison between all methods. The same color of the bounding box and the underline for the prompt indicate the same object. The bounding boxes predicted based on our proposed evaluation method discussed in Sec. 4.2 As an object is wrapped by two bounding boxes with different colors, e.g., A&E (T] in the first row or Structure Diffusion (3] in the second row, it has a high potential to contain mixed objects. In the first row in Fig. 4 our TEBOpt (CTEB) helps Stable Diffusion generate the specified dog and help A&E make the horse dissimilar from the dog. In the second row, our TEBOpt (CreB) helps Stable Diffusion generate cat less similar to lion, which is initially equipped with a mane as a visual signal of lion. Regarding Fig.[5) we demonstrate the generated image with and without our proposed TEBOpt (CrEB). Using our TEBOpt, T2I models solved both the object mixture and missing issues, especially when trying to solve the problem of object mixtures. It is worth noting that objects mixture or missing are also affected by the denoising process in T2I models. Our text optimization mainly contributes to balancing the information in the text embedding and further reducing mixed and missing issue.
7

---

4.4
Quantitative results
Table provides the comparative experiment for object mixture, missing and information bias. As Structure Diffusion [3] manipulates key and value from text embeddings in denoising UNet's s cross-attention calculation, which works on text embedding as ours, our method working on top of Stable Diffusion can directly compare with it.
Among all methods with our TEBOpt (CtEB) can further improve the objects' existing balance with 125,42% in Stable
Stable Diffusion
A&E
SynGen
Structure Diffusion
Method
+LrEB
+CrEB
+LTEB
2 objects exist
14.5%
+0.5%
34.0%
+1.0%
52.0%
+4.5%
7.0%
only mixture
9.5%
-1.0%
11.5%
-4.0%
10.5%
-1.0%
11.5%
Diffusion, 78.43% in A&E, and 10.65% in SynGen. Though the balance of Structure Diffusion is better than ours, it causes a 7.5% decrease in the probability of 2 objects coexisting.
obj1 + mixture
6.5%
-1.5%
9.0%
-1.5%
6.5%
-3.5%
4.0%
obj2 mixture
5.5%
-2.0%
6.0%
+0.5%
2.5%
+1.5%
3.0%
only objl exist
45.0%
-6.0%
26.0%
-2.5%
10.5%
-0.0%
39.0%
only obj2 exist
17.0%
+11.0%
12.0%
+5.0%
17.0%
-2.5%
34.5%
no target objs
2.0%
-1.0%
1.5%
+1.5%
1.0%
+1.0%
1.0%
Info bias
2.65
1.39
2.17
1.38
0.62
0.72
1.13
Table 4: Analysis of balancing performance. Within the cases generating one object, we highlight the better balanced results in blue and red.
Since it directly combines pure text embeddings and original embedding, it loses general information with two objects in the text embedding, as the Sec. 3.1.2 discusses.
Furthermore, SynGen 16 generates a reverse trend between object 1 and 2 since it works to make the two objects' cross-attention maps' distance as far as possible, which would contribute to separating two objects leading to large improvement in object missing and information balance. With our text optimization, we can further solve mixture issue and making the balance better since we tackle the issue from the front of the problem in text embedding. Thus, text embedding optimization and image latent optimization reach out a good cooperation for solving information bias and loss. The generalizability of CtEB in information bias for more than 2 objects is reported in Supplement()
In Table 5 we evaluate the similarity of token embeddings and the distance of cross- -ntsstattention maps between two objects within one prompt. Token embedding similarity (Token sim) is calcu- lated by cosine similarity while the cross-attention map distance (Map dist) calculated by the symmetric Kullback-Leibbe divergence between two normalized cross- atteetttttnionn maps M; and Mj: (M| + D (M)), where D (Mr)l = Enizett M, log (M,) With our CTEB. the token embedding similarity between two objects reduces 36.98% and the cross-attention map distance increases 13.62%.
Token sim 4
SD 1.4
+LTEB
Map dist f
SD
+CtEB
Average
0.454
0.286 (-36.98%
Average
2.321 [0.287,
2.637 (+13.62%)
[min, max]
[0.291. 0.669]
[0.137. 0.537]
[uin, max]
[0.553, 6.340]
Table 5: Analysis of optimized token embedding similarity (Token sim) and cross-attention map distance (Map dist) between two objects within one prompt.
Discussion of how the similarity of text embedding affects cross-attention maps' distance.
We calculated the cosine similarity between various text embeddings and displayed the results in Fig. (a). The data indicates that objects with similar colors or sizes, e.g., Languin-pande or turtle- frog, tend to exhibit higher similarity in their text embeddings, which can be attributed to the training mechanism of CLIP. Additionally, we computed the distance between the cross-attention maps, using
'The info bias value indicates less bias when it is closer to 1, different from the intuitive assumption that lower values represent less bias. Here's the detailed calculation: First, we calculate the bias distance between the info bias value and the balanced value (which is 1). The bias distances of SD and SD + TEBOpt are (2.647- ly11 = 164.71 %% and (1.393-1)/ 11/1 = 39.29%. Note that we have rounded the values for information bias in Table 4 reporting 2.647 as 2.65 and 1.393 as 1.39. Secondly, the balance improvement is then calculated as (64.71%-39.2999 39.2%% = 125.42%.
8

---

the same function in Table generated by the two objects' text embeddings with the same initial latent in the early denoising steps. As shown in Fig. (b), there is positive correlation between the similarity of text embeddings and the distance of the cross-attention maps triggered by the two objects. Specifically, objects with similar text embeddings are more likely to activate overlapping areas during the denoising process. This confirms that similar text embeddings contribute to object mixture, while the short distance of cross-attention maps leading to object missing has been proven by SynGen [16].
5 Related works
Textt erroeddingg siilrrty desonding,
Object mixture or missing. Stable Diffusion [14] pointed out that stable diffusion models have the issue of con- cept bleeding, which occurs by unin- tended merging or overlap of distinct visual elements, leading object mix-
33-
0004
par
ture or missing. Also, the root cause
lies in the usage of pretrained text encoders, including CLIP [15] and
exptat-
63-
get * OM
ross-
231 car
OpenCLIP [). However, all the ex- isting methods investigate the issue in the denoising process instead of text encoders. For instance, Attend-and- Excite I1 proposed an optimization process to ensure every selected token triggers some image patches when cal- culating the cross-attention maps be- tween text embedding and image fea- tures. SynGen [16] designed a loss function to increase cross-attention maps' similarity between modifier-entity pairs and enhance attribute binding. Structure Diffusion 3 leveraged linguistic structure separate the given prompt into several noun phrases and modify the corresponding value to manipulate text embedding when calculating cross-attention maps in denoising steps. Predicated Diffusion [20] designed decompose the prompt containing several objects into independent objects and minimize the loss between the generated image based on the combined independent prompt and that of the original prompt. Refocus [ proposed to define each object's positional bounding boxes with GPT-4 and designed two loss functions to make the object only denoised inside its given region.
Figure 6: (a) The cosine similarity of text embedding from single word. (b) The KL distance of coss-attention maps that are triggered by two words. The data is ordered by their text embedding similarity.
However, there is no task investigating the causal manner in the pre-trained text encoders. Therefore, we share a comprehensive analysis and propose a simple solution to tackle the issue of information bias, object mixture and missing, e.g., generating one animal with bear head and turtle shell when prompting "a turtle and a bear" or generating two cats when prompting "a cat and a dog".
6 Conclusion
In this study, we conducted detailed analysis of text embedding's impact on text-to-image diffusion models, topic rarely explored. Our findings indicate that the causal processing of text embed- ding leads to information accumulation, causing biases and loss. Directly replacing accumulated embeddings with purified embeddings, though resulting in decreased coexistence of two objects, enhances the balance between generating either object. We introduce training-fre -fee Text Embedding Balance Optimization (TEBOpt) method that effectively eliminates problematic information in critical token embeddings, improving information balance handling in stable diffusion by 125.42% while preserving object coexistence performance. Additionally, due to the unreliability of existing metrics for assessing inaccuracies in generated images, we propose a new automatic evaluation metric to more effectively measure information loss.

---

7 Acknowledgments
This work is partially supported by the National Science and Technology Council, Taiwan under Grants NSTC-112-222-EEA444000- and NSTC-112-2221-E-444999-----
References
[1] Hila Chefer, Yuval Alaluf, Yacl Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite Attention-bent based semantic guidance for text-to-image diffusion models. In SIGGRAPH, 2023.
[2] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. ICML, 2024.
[3] Weixi Feng. Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- free structured diffusion guidance for compositional text-to-imas -iagg synthesis. In ICLR, 2023.
[4] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In ICLR, 2023.
[5] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A com- prehensive benchmark for open-world compositional text-to-image generation. In NewrIPS Datasets and Benchmarks Track, 2023.
[6] Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, 2021. 2021.
[7] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML, 2022.
[8] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects context. In ECCV, 2014.
[9] Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, and Jun Huang. Towards understanding cross and self-attention in stable diffusion for text-guided image editing, In CVPR, 2024.
[10] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. NeurlPS, 2023.
[11] OpenAl. Hierarchical text-conditional image generation with clip latents. In arXiv, 2022.
[12] OpenAl. DALL-E 3, 2023.
[13] Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. In CVPR, 2024.
[14] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high nigh-resolution image synthesis. In ICLR, 2024.
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[16] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. In NewrIPS, 2023.
10

---

[17] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[18] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[19] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In ECCV, 2024.
[20] Kota Sucyoshi and Takashi Matsubara. Predicated diffusion: Predicate logic-based attention guidance for text-to-image diffusion models. In CVPR, 2024.
[21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez,
F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.

---

Supplementary Material
A Broader impact and safeguards
This study deepens our understanding text embedding in texl-to-image (T2I) models, showing that better insights into pre-trained text encoders markedly enhance accurate content generation. We identify that text embeddings accumulating too much information can introduce biases and data loss, insights that are vital for developers of large vision- asion-language models to effectively mitigate these problems. For safeguards, we have adopted the safeguard provided in Stable Diffusion [17], aligning with ethical standards and preventing misuse of generative models.
B Implementation details
Computing resources and method efficiency. All experiments are run one NVIDIA RTX 3090 24 GB GPU. We calculated the average inference time from 400 images. The average inference time of default Stable Diffusion 1.4 is 8.21 seconds/image while our proposed LrÆR in Stable Diffusion 1.4 is 9.14 seconds/image.
Model architecture. All the analyses are conducted on Stable Diffusion 1.4, where the text encoder the pre-trained CLIP VIT-L/14 [13] and the scheduler is the PNDMSchedular. In the text encoder, CLIP, the maximum token length (N) is 77, the embedding dimension (D) is 768, and the attention head number (h,,) is 12. We use fixed guidance scale of 7.5 and the denoising step to 50. For performing the TEB loss in Equation [ we set a maximum optimization time as 20 and a threshold for LTEB -0.7, which comes from LTEB 0.95 and L = 0.25. The optimization target for LTEB is -0.7 but if the optimization time exceeds 20, it will stop optimizing the text embedding. Moreover, all baselines are conducted with their official source codes.
Cross-attention map visualization. We aggregate and take the average of all cross-attention maps with a size not larger than 32x32 in all 50 denoising steps. Note: when calculating the distance between cross-attention maps, we do not accumulate the cross- gross-attention maps within 50 denoising steps. We only take the cross-attention map in corresponding denoising step.
Masking token embedding. We set the selected dimension of encoder attention pacietttoommasskk in the denosing UNet as 0 while the unselected dimension is 1. The working mechanism for encoder _attrttttttttttttee mask is that after key multiplying query to get the attention matrix, the dimen- sion with 0 inside encoder incoderaattention attention intentiio_mmakk mask would add -10,000 to the attention metric while the dimension with would remain unaffected. After processing the attention metric with sof olmaa, the selected dimension in the attention matrix would become 0.
C Existing evaluation methods discussion
The existing evaluation metries for missing objects are two-fold: text-image similarity and text-text similarity.
Text-image similarity is calculated by the CLIP cosine similarity between the text prompt and the corresponding generated images. A&E I further separated the metric into Full Prompt Similarity and Minimum object Similarity. Since full prompt similarity may not accurately reflect the existence of missing objects, minimum object similarity is proposed to evaluate the most neglected subjects independently. Take "a dog and a cat" an example. It first separates the prompt into dog" and "a cat", evaluates both separately and uses the minimum score as the metric to indict missing objects.
Text-text similarity is calculated by the CLIP cosine similarity between the text prompt and the caption of the generated image obtained by the pretrained BLIP image-captioning model [7].
However, all these metrics cannot reflect the mixed objects issue and cannot accurately reflect how many missing objects are in the generated images. Thus, we propose a new evaluation metric for reporting the number of mixed and missing objects discussed in Sec 4.2
12

---

Furthermore, we discuss the metrics that previous methods used, including full prompt similarity. minimum object similarity and text-text similarity in Figs. and Fig. given the prompt "a cat and penguin", the generated image on the left contains penguin and one mixture of penguin and cat, while the generated image on the right contains penguin and cats. However, the left one, which contained the mixture object, has 3.57% higher in the full prompt similarity than the right one without mixture objects. It indicates that the full prompt similarity cannot properly measure the mixture cases. Moreover, the text-fext similarity of the left one is 59.56% higher than the right one, which indicates that cannot identify the mixture issue. contrast, our proposed metric can accurately predict mixture/indepeeden objects with specific number.
Prompt: penguin"
Full prompt similarity:
0.3712
0.3584
Minimum Object similarity:
0.2668
0. 0.2333
Text-aaxttimillriyy
0.8830
0.5534
cat sitting next to penguin on the ground
cat standing next to penguin brown background
Generated caption by BLIP:
Our proposed metric:
penguin and mixture
cats and penguin
Figure 7: Full prompt similarity and text-text similarity cannot properly evaluate the mixture cases. As indicated in red color, the image on the left contains mixture cat and penguin but its full prompt similarity and text-text similarity are higher than which of the image the right.
In Fig- g the image the left generates bear and a mixture of bear and cat, while the image the right has bear and two cats. In terms of text-text similarity, the left one is 8.68% higher than the right one. Again, indicates that cannot identify the mixture issue. In contrast, our proposed metric can accurately report that the left one contains bear and mixture, while the right one contains bear and 2 cats.
Prompt: "a bear and cat"
Full prompt similarity:
0.3175
0.3313
Minimum Object similarity:
0.2092
0.2617
Text-text similarity:
0. 0484
0.77
painting of bear and a baby bear
a bear, a cat, a cat on a grass field
Generated caption by BLIP:
Our proposed metric:
bear and mixture
bear and 2 cats
Figure Text-text similarity the left one 8.68% higher than that of the right one. It indicates that the metric cannot identify the mixture issue.
13

---

In Fig.9 both images generate mixture of bear and lion but the left one has 33.48% higher text-text similarity than the right one. terms of full prompt similarity and minimum objecr similarity, they have a difference of 7.01 % and 15.27% respectively. These variations make the evaluation metrics unreliable to measure object mixture and missing. In contrast, our proposed metric can accurately report that both images contain one mixed object.
Prompt: "a bear and lion"
Full prompt similarity:
0.2753
0 29496
Minimum Object similarity:
0.2416
0 2785
Text-text similarity:
0.7851
0.5882
close up of lion with blurry background
large brown bear laying on top of "uush green field
Generated caption by BLIP:
Our proposed metric:
mixture
mixture
Figure 9: In two images both with mixed objects. full prompt similarity, minimum object similarity, and text-text similarity all vary greatly, making the evaluation metrics unreliable for object mixture and missing.
C.1 Demonstration of bounding boxes interaction corresponding to the evaluation status
Considering the working mechanism of SOTA text-to-image models, when the cross-attention maps of 2 objects respond closely during denoising, there is high probability of generating a mixture object, Fig. shows. Consequently, within the nature of the Owl-ViT detector, these mixtures can be identified by their high overlap with high confidence.
Detect Bioxes
Cross attention mup
Mixture exists (lefi)
<SOT>
cat
and
rabbit
<EOT>
<0T>>
and
bird
<EOT>
Mixture exists
<0T>>
bird
rabbit
<EOT>
Figure 10: Demonstrating the 90% bounding box overlapping and corresponding object mixture in generated image and cross-aad attsnttton maps during denoising steps.
14

---

D Analysis of information bias with more than 2 objects
In 6 we demonstrate the information bias on three objects, where the problem could extend to general scenarios with a larger number of objects. In each prompt from (a) to (f), the first mentioned object always has an existence of over 20%, while the existence of the second and third mentioned objects gradually decreases to below 20% and 10%. This confirms our observation regarding information bias, which gradually overlooks the later mentioned objects. By incorporating our custom loss function LTEB, we can achieve more balanced representation of the three objects, with the existence values ranging between 10% and 20%, resulting in the information bias closer to 1.
(a) A/An cobj 1> and a/an cohj2> and alan cobj3>
(b) AlAn cohj2> and a/an cubji
(c) A/An <obj3>
(d) A/An cobj 1>
(c) A/An <ohj2> and a/an cobj3s
( A/An <ohj3> and wan <abjl>
and alan cobj2s
and alan cobj3>
Prompt
and a/an <obj3>
and alan <obji>
and a/an <obj2>
and alan <obji>
and a/an <abj2>
+Crsa
Cyss
+brga
+Grsg
+Cygg
+Crgs
objects exist
0.0%
+0.8%
0.5%
+0.0%
0.8%
-0.3%
0.5%
+0.0%
0.3%
+0.0%
1.0%
+0.0%
only mixture
24.7%
-0.4%
20.0%
21 1%
+0.7%
20.4%
+4.0%
25.8%
-0.4%
abjl misture
8.6%
-2.1%
4.3%
-0.6%%
3.0%
+1.3%
9.5%
-3.0%
2.8%
+0.5%
4.5%
-1.7%
3.0%
10.4%
-3. 33.4%
4.9%
-1.1%
5.3%
40.0%
9.3%
-4.0%
+0.8%
abj3 mixture
4.5%
-1.3%
3.8%
+0.5%
11.2%
-4.1%
3.0%
+0.7%
5.5%
-1.8%
8.3%
-0.1%
obil obj2 mixture
1.0%
-0.35%
0.8%
-0.5%
0.0%%
+0.3%
1.8%
-1.0%
2.5%
-0.3%
0.8%
-0.5%
abjl obi mixture
1.5%
-0.5%
+0.2%
2.5%
1.3%
+0.0%
0.3%
-0.3%
-1.0%
obj2 obj3 + mixture
0.3%
+0.0%
2.3%
-0.3%
3.0%
-2.2%
0.5%
+0.7%
0.8%
+0.0%
1.8%
+0.5%
only obji
24.7%
-5.4%
10.1%
3.8
27.
8%
3.3%
3%
11.0%
+0.9%
only obj2 exist
14.4%
+3.1%
29.9%
-4.6%%
12.8%
+3.6%
6.8%
+6.0%
30.7%
-2.8%
4.8%
+7.2%
only obj3 exist
4.8%
+6.7%
4.8%
+4.9%
25. 1%
-5.7%
10.3%
7.7
11.1%
+2.7%
23 83%
-3,5%
abjl oby2
0
+0.2%
-1.5%
0.0%
abjl obj exist
4.8%
-0.8%
0.3%
-0.3%
4.4%
+0.4%
6.8%
-0.5%
0.0%
+0.0%
7.3%
-0.9%
abj2 obj3 exist
0.3%
-0.3%
4.3%
-0.8%%
6.6%
+0.3%
0.0%
+0.3%
7.5%
-2.8%
6.0%
-1.4%
no target object
0.0%
+0.0%
0.0%
+0.0%
0.0%
+0.0%
0.0%
+0.0%
0.0%
+0.0%
0.0%
+0.0%
Info bias (obj). obj2)
1.72
1.10
0.34
0,45
0.30
0.58
4.04
1.69
0.11
0.34
2.32
1.00
Info bias (obj obj3)
5.16
1.67
2.11
1.15
0.49
66
1.79
0.30
0.46
0.59
Info bias (obj2, obj3)
3,00
1.52
6.21
2.59
0.51
0.84
0.66
1.06
2.77
2.02
0.20
0.59
Table 6: Analysis of information bias in multiple objects. When there are more objects in the prompt, the bias might gradually enlarge as the order of the mentioned objects gets farther from the first mentioned object. In this setting, we utilize the same evaluation metric to define the information bias among three objects, which is divided into pairs of (obj1, obj2), (objl, obj3), and (obj2, obj3). As the info bias approaches 1, it yields more balanced results regarding the existence of the object.
E More qualitative and quantitative comparisons in T21-CompBench [5]
More qualitative results are in Fig. Fig. I2 Fig. 13 and Fig. 14 The public benchmark,
e.g.. T21-CompBench [5], contains more diverse prompts, e.g., adjective binding with noun, that would make the experiment distract. Thus. in the main paper, we follow related works, e.g., A&E (SIGGRAPH'23) [ and Predicated diffusion (CVPR'24) 24) (20), to create the most suitable benchmark for our task. While we respectfully disagree with the directness to conduct public T21-CompBende to demonstrate our performance, we still conduct quantitative and qualitative comparisons in T21- CompBench and prove our effectiveness.
E.1
Color set
We include a quantitative comparison in Table 7 compare SD 1.4, SD 1.5, ELLA (ArXiv'24: 224; on the only released version, SD 1.5), SDXI--Turbo [19) and SD3 (2 the color set within the T21-CompBench [S] with 1,000 cases. Within all the baseline methods, our TEBOpt improves the object co-existence rate with 7.9% on SD3 by addressing object mixture and object missing. Also, TEBOpt makes the generated information more balanced, where the info bias is from 2.07 to 1.30.
F.2 Spatial set
We conducted our TEBOpt experiments using Stable Diffusion (SD) 1.4 on the set of spatial relation- ships (e-g.. "next to." "on the side of," etc.) within the T21-CompBench (5). This dataset includes nouns representing 5 types of people (man, girl, etc.), 16 types of animals (giraffe, turtle, etc.), and
15

---

SD 1.4
SD 1.5
SDXL Turbo
SD3
Method
+CTEB
ELLA ELLA+CTEB
+Lres
+rEE
objects exist
25.4
30.8
24.5 %
57.0 %
63.1
57.1
60.5 %
68.4
only mixture
2.9
3.3
3.8 %
1.5 %
2.6
3.6 %
2.6 %
2.1
2.3
obj1 mixture
1.4 %
1.9 %
2.4 %
2.6 %
0.9 %
2.5 %
2.4%
2.9 %
2.5
obj2 mixture
2.5 %
1.5 %
1.3 %
2.1 %
0.6 %
1.3 %
1.6 %
2.1 %
0.9 %
only ohj] exist
43.8 %
35.7 %
41.1 %
20.3 %
16.9 %
19.8 %
17.0 %
15.3 %
9.8 %
only obj2 cxist
19.5 98
22.6 %
21.3 %
13.9 %
13.2 %
13.6 %
14.9 %
7.4 %
7.5 %
no larget objs
4.5
4.2
5.6 %
2.6 %
2.7
2.1 %
1.0 %
1.8 %
0.7
Info bias
2.24
1,58
1.94
1.46
1.29
1.46
1.14
2.07
1.30
Table 7: Quantitative comparison with SOTA methods the color set in the 221-CompBench. Reference: ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (ArXiv 24)
30 types of objects (table, car, etc.), encompassing a total of 1,000 cases. Table shows that our method demonstrates improvement in increasing the 2 object co-existence with 6.8% and reducing the information bias from 1.43 to 1.21. In this experiment, we further prove that when two nouns in larger
the given prompt are from different categories, such as "a woman and chair," resulting text embedding distance, it causes the mixture issue in text-to-image models to be concealed beneath surface. Thus, while our main paper focuses on the task handling only animals, contains different challenges compared to handling more diverse objects in different categories. In our focus, we reveal and address both the mixture and missing issue. Furthermore, in the following experiment, demonstrate that our method is effective across a more diverse set of categories.
Stable Diffusion 1.4
Method
+LTEB
2 objects exist
40.4%
+6.8%
only mixture
0.2%
-0.0%
objl mixture
0.0%
+0.1%
obj2 + mixture
0.1%
-0.1%
only obj1 exist
32.3%
-5.5%
only obj2 exist
22.6%
-0.5%
target objs
4.4%
-0.8%
Info bias
1.43
1.21
Table 8: Quantitative Comparison on the spatial set in the T2I-CompBench.
Image quality evaluation
We follow SD3 2 to conduct the image quality evaluation on Fréchet Inception Distance (FID) with CLIP L/14 image features on the generated images and the COCO 2017 val dataset g 5,000 samples. The FID of (SD 1.4, SD 1.4 + TEBOpt), (SDXL-Turbo, (SD3, SD3 TEBOpt) are (133.08, 133.30), (202.50. 200.71), and (143.77, 142.20), where FIDs are higher than we usually see from text-to-imag models is because the 5,000 generated sets are based on the plain prompt structure "a <objA> and <objB>". This experiment proves that visual performance is mainly affected by the selected text-to-image model as the FID for the generated images w/ or w/o TEBOpt are within marginal differences in the same model. When these models work with TEBOpt, only SD 1.4 gets 0.22 increase in FID score, while SDXL-TTubo and SD3 result in 1. 7.79 and 1. 1.57 decrease FID scores. It proves that TEBOpt would improve the general image quality.
DDXL-Turbo + TEBOpt), and
16

---

SD 1.4 Frog Missing
SD 1.4 LTEB
SD 1.4 Boy Missing
SD 1.4 TEBB
SD 1.4 Bowl Missing
SD 1.4 + TEB
bear and a frog
phone on the left of a boy
blue bench and a green bowl
SD 1.4 Bird Missing
SD 1.4 TEBB
SD 1.4 Car Missing
SD 1.4 LTEB
SD 1.4 Vase Missing
SD 1.4 + LEE
A red bench and a green bird
sheep near car
brown bear and blue vase
SD 1.4 Boat Missing
SD 1.4 LTBB
SD 1.4 Missing
SD 1.4 LTEB
SD 1.4 Mixture
SD 1.4 TEB
A red bird and a brown boat
brown and a red suitcase
A cow side of a candle
SD 1.4 Mixture
SD 1.4 + LTEB
SD 1.4 Mixture
SD 1.4 + LTEB
SD 1.4 Mixture
SD 1.4 LTEB
A dog and a cat Figure 11: More qualitative results SD 1.4 in complex prompts from color and spatial sets within T2l-CompBench [5].
A person the top of a bicycle
A red car and brown giraffe
G Interesting questions aroused by reviewers
G.1 Would the same happen for the words may change their meaning due to nearby words? E.g., mouse?
We experimented with 1,000 samples on the effect of words that may change their meaning due to nearby words, including "mouse". "horn", "jaguar", "falcon". and "palm". Specifically, we use the prompt "an <animal/object A> and <B>" and evaluate the result by detecting targets <animal A> and cobject A> using Owl-ViT detector. Our TEBOpt can address 3.67% object missing issue in animal prompts while the optimized results may lean towards the main meaning of the word in object prompts. For example, "jaguar" tends represent an animal rather than car, resulting in 1.29% decrease in generating "object jaguar" after optimization.
17

---

SD1.5
ELLA
ELLA + LTEB
SD1.5
ELLA
ELLA + +LEB
Color Mixture Dog Missing
Dog Missing
Banana Missing
Banana Missing
brown cat a yellow dog
A green banana a brown giraffe
SDI.5 Cat Missing
ELLA Cat Missing
ELLA + TEB
SD1.5 Color Mixture
ELLA Color Mixture
ELLA LTBB
A brown dog and black cat
A yellow highlighter and a pink note
Figure 12: More qualitative results on ELLA on SD 1.5 in complex prompts from color set within T2I-CompBench [5). Reference: ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (ArXiv'24)
G.2 As revealed by the analysis, not only the latter object text embedding contains the earlier object information, all the latter words all may have similar impacts to strength the text embedding of the earlier object information. Has the author considered how to resolve such possible influence?
We explored this question when we hypothesized the solution our proposed problem. We conducted an experiment on the same 400-prompt set described in the paper, eliminating "all" earlier information from accumulating in subsequent tokens. The results are presented in Table Without shared embeddings across tokens, the generation process failed to produce co-existing objects. Specifically, during the denoising process, each object token responded in the central region, as observed in the cross-attention maps, resulting in no object co-existence. In conclusion, maintaining proper proportion of earlier object token information in the latter tokens (excluding those with concrete meanings) has more positive than negative effects, especially in generating co-existing objects within a given prompt. Therefore, we propose to optimize the critical tokens' embeddings in the paper.
SD 1.4
Method
w/o info accumulation
objects exist
0.00%
only mixture
11.75%
objl + mixture
0.00%
obj2 + mixture
0,25%
only objl exist
46.50%
only obj2 exist
39.50%
no target objs
2.00%
Info bias
1.18
Table 9: Evaluation for eliminating all information accumulation.
18

---

SDXL-Turbo
SDXL-Turbo
SDXL-Turbo Fork Missing
SDXL-Turbo
SDXL-Turbo
SDXI-Turbo
Cat Missing
+ LEBB
+ TEBB
Dog Missing
+ +
black dog and brown cal
wooden spoon and metal fork
A yellow cat and brown dog
SDXL-Turbo
SDXL-Turbo LTEB
SDXL-Turbo Color Mixture
SDXL-Turbo LTEB
SDXL-Turbo Structure Mixture
SDXL-Turbo
Color Mixture
+ LEEB
A blue chair and red cup
green snake and a brown log
A green bench and red book
Figure 13: More qualitative results on SDXI--Turbo 121-CompBench S
19
complex prompts from color set within
SD 3
SD LTEB
SD 3
SD 3 LTEB
SD 3 Object Mixture
SD 3 LYEB
Clock Missing
Object Mixture
A brown bench and green clock A brown backpack and a blue cow
A brown giraffe and a red train
3
SD + LTEB
SD
SD + LTEB
Color Mixture Orange Missing
+ LEEB
Color Mixture Orange Missing
Color Mixture
blue bowl and a yellow orange green acorn and a brown leaf green apple and yellow orange
Figure 14: More qualitative results on SD3 in complex prompts from color set within T21- CompBench (5).
H The instruction for participants in human evaluation for our proposed evaluation metric
"You will be given an image and text description. The text is described as "a/an <objectl> and a/an <object2>". Please determine whether the objects in the image are a
Full-fext instruction:
combination of object I and object 2, of any object is missing. Select the corresponding answer from the options: i) two objects exist, ii) mixture exists, iii) missing object 1, iv) missing object or v) no objects exist." The screenshot of the human evaluation is demonstrated in Fig. 15
19

---

Prompt: bird and cat
Two objects exist
Mixture exists
Missing object
Missing object 2
No objects exist
Figure 15: The screenshot of the human evaluation, containing the information and options that given to participants.
I Limitation
Despite the effectiveness of our proposed text embedding balance optimization in balancing infor- mation, whether the generated images contain mixed objects or lost objects is still affected by the denoising process. Furthermore, there is still room to improve in identifying the importance of information within one complex prompt. For example, this literature investigates in the prompt with equally importance objects, e.g., <object 1> and a <object 2>, and it can be extended to more objects. more complex prompts, such as a dog and cat playing with mouse front of a yard, in which the mouse is not a device, the text embedding might indicate that the device is less important.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide clear bullet points in the introduction to reflect the paper's contri-
bution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Supplement
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
There is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Supplement
B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: We open- souree the codes for TEBOpt, all analyses, and the proposed evalua- tion metric, and the sample data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec. 4.1 and Supplement
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We work on analysis with fixed seed randomly paired with each text prompt for fair comparison within all baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Supplement B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have carefully followed the code of ethics in the paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Supplement A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please refer to Supplement A
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all the codes that we have referenced in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer to the README. md in the GitHub repository. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Please refer Sec. 4.2 and Supplement
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: We conducted human evaluation experiment to prove the effectiveness of our proposed evaluation metric. There no risk labeling the generated images to identify mixed objects or missing objects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "gEEks8NTlO", "reviewer": "Reviewer_G9At", "summary": "Background: Text-to-image diffusion models, such as Stable Diffusion, frequently encounter difficulties in accurately generating images from textual descriptions, especially when there are multiple objects -> object mixing and missing.\n\nThis paper focuses text embeddings to solve the problems.\n\nHypothesis: self-attention mixes text embeddings of multiple objects.\n\nAnalyses:\n* 400 prompts of \"a/an <animal 1/2> and a/an <animal 2/1>\" with 17 animals produce lots of mixture of objects and missing objects\n* Contribution of text embeddings to the generated images\n    * Masking the dominant embeddings reduces the mixture rate \n    * Replacing the embedding after the text encoder with the pure text embedding does not solve the problem.\n* Process of conditions getting lost and biasing towards the first-mentioned object due to masks in self-attention in the text encoder.\n\nMethod:\n* Optimize the target text embedding toward pure embedding and against earlier (first-mentioned) token’s embedding", "questions": "Q1. Would the same happen for the words may change their meaning due to nearby words? E.g., mouse?\n\nQ2. I do not think the self-attention layer models \"causal\" relationship between words. I recommend replacing \"causal manner\" with \"dependency\". Thoughts?", "limitations": "Some are covered.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper provides useful analyses on text embeddings for text-to-image diffusion models. \n\nThe experiment design corresponds to the hypothesis.\n\nThe proposed metric is a good effort for quantitative evaluation.", "weaknesses": "(Overall, I like the idea of this paper but it needs more work to be a solid paper.)\n\nW1. This paper covers only nouns, especially animals. The research community has tackled different aspects of text prompts [composable, masactrl]. How about using other classes of objects to prove generality? For instance, could this method work with text prompts like 'A <cat> standing next to the <tower>' or examples from A&E such as 'A <lion> with a <crown>'? It would be valuable to see if the approach is equally effective with these more diverse classes and prompts that users might want to apply.\n\nW2. Bboxes overlapping more than 90% not always indicate mixture.\n\nW3. The terms should be more clearly defined.\n* Information bias and object missing are confusing.\n* The critical dimension?\n\nW4. Initial latent optimization is poorly described. How can the initial latent be updated during the denoising process? I guess intermediate latents can be updated just like [self-guidance].\n\nW5. The baseline of Figure 5 is not defined. \n\nW6. The appendix should provide large number of examples to ensure the proposed method is free from cherry picking.\n\nW7. Improvements in Table 4 is marginal.\n\nW8. An important paper is missing: [composable], [custom diffusion] and [hiper]\n\nW9. The method requires optimization for each sentence, which can be inefficient for practical use.\n\nMisc.\n    1. Placing the caption of Table 5 under the table would be better for consistency.\n    2. Some numerical values in the main text are difficult to understand at first glance. To improve readability, I think it would be better to highlight or mark the key values within the table itself. For example, the main text mentions 125.42%, but I couldn't immediately identify which part in Table 4 shows a difference of 125.42%. \n    3. The result shows that the balance performance in SynGen decreased by 10.65%, rather than improved.\n    4. I think Figure 7 is not an appropriate example for demonstrating the inadequacy of full prompt similarity, as neither image correctly matches the given prompt 'a cat and a penguin'. The left image shows a mixture of the two animals, while the right image displays two cats instead of one.\n\n[composable] Compositional visual generation with composable diffusion models, Liu+\n\n[masactrl] MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, Cao+\n\n[self-guidance] Diffusion Self-Guidance for Controllable Image Generation, Epstein+\n\n[custom diffusion] Multi-Concept Customization of Text-to-Image Diffusion, Kumari+\n\n[hiper] Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion, Han+", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to review our work and providing us with your valuable suggestions and insightful comments. We have carefully considered your feedback, and we would like to address your concerns. In the following section, we respond to each weakness (W) and question (Q), while weakness 1 is included in the overall rebuttal at the top of this page.\n\n**W2:** We agree that 2 objects in the real-world overlapping >90% might not mean mixture, such as overlapping persons. However, in diffusion-based generated images, this high overlap does indicate mixtures due to the nature of the models used. Considering the working mechanism of SOTA text-to-image models, when the cross-attention maps of 2 objects respond closely during denoising, there is a high probability of generating a mixture object, as Fig. 2 in the rebuttal PDF shown. Consequently, within the nature of the Owl-ViT detector, these mixtures can be identified by their high overlap.\n\n**W3:** \n- Clarification for Information bias and object missing: Due to the causal masking manner, the earlier token information accumulates to all the later token embeddings, making the overall text embedding biased towards the token information mentioned earlier in the text prompt. We call this scenario information bias. Then, biased text embedding, having more information of the earlier mentioned token, makes the denoising process to have higher probability to generate images with the object mentioned earlier and leads to object missing.\n- Thank you for raising the question. We will change the term critical \"dimension\" to critical \"token embedding\". Take the prompt “a dog and a cat” as an example. There are 2 critical tokens, <dog> and <cat>, and we generate the pure text embeddings for them by the prompts \"a photo of a <dog>\" and \"a photo of a <cat>\", respectively. Then, we use the <dog>'s and <cat>'s token embeddings as critical (pure) token embeddings for calculating positive loss in TEBOpt.\n\n**W4:** Our TEBOpt optimizes the text embedding based on pure token embedding and original token embedding \"before\" the denoising process while the text embedding is fixed during denoising process for fixing the generated meaning guidance. Regarding the “initial” image latent mentioned in L#197, we will polish the wording to reduce confusion.\n\n**W5:** The baseline in Fig. 5 is SD 1.4.\n\n**W6:** No problem. Please refer to Fig. 1 in the rebuttal PDF.\n\n**W7:** We statistically calculated the p-value by the chi-square test in Table 4 to prove significance. The p-values for SD 1.4 vs. SD 1.4 + TEBOpt on info bias and overall performance, including 2 objects exist, object mixture, and object missing, are 0.0052 and 0.0108, which are far below the alpha value of 0.05, proving significant improvement. \n\n**W8:** As suggested, we will discuss the related works. Specifically, [composable] separates the given prompt into different objects and generates images using a set of diffusion models. [custom diffusion] focuses on preserving the customized concept for text-to-image (T2I) generation while its limitation (Fig. 9) is exactly our main focus that pretrained T2I models struggle with similar compositions, e.g., dog and cat. [Hiper] reconstructs text embedding for image editing, while its main target for image editing is different from our task.\n\n**W9:** Compared to retraining the text encoder directly, our approach is highly cost-efficient as it employs a training-free inference design, serving as a proof of concept. Specifically, our optimization adds only 3.11% to the inference time, based on an average of 1,000 cases using a single A100 GPU.\n\n**Mics (M) 1:** Thank you for the suggestion. We will make the tables consistent.\n\n**M2:** To improve numerical readability, we acknowledge that 125.42% is calculated in 2 steps. The info bias value indicates less bias when it is closer to 1, different from the intuitive assumption that lower values represent less bias. Here’s the detailed calculation:\n1. We calculate the bias distance between the info bias value and the balanced value (which is 1). The bias distances of SD and SD + TEBOpt are (2.647-1)/1 = 164.71% and (1.393-1)/1 = 39.29%. *Note that we have rounded the values for information bias in Table 4, reporting 2.647 as 2.65 and 1.393 as 1.39.*\n\n2. The balance improvement: 164.71%-39.29% = 125.42%.\n\n**M3:** The balance performance is assessed by how close the information bias is to 1, as this indicates a more balanced state. Thus, SynGen has a lower info bias value while SynGen + TEBOpt is indeed improved from 0.62 to 0.72 because 0.72 is closer to 1.\n\n**M4:** We considered that *full prompt similarity* is insensitive to numerical information. To describe the case more properly, we will update the image with 1 cat as in Fig. 3 in the rebuttal PDF. Still, it has a lower score than the mixture one in Fig. 7, indicating the inadequacy of the metric.\n\n**Q1:** We experimented with 1,000 samples on the effect of words that may change their meaning due to nearby words, including \"mouse\", \"horn\", \"jaguar\", \"falcon\", and \"palm\". Specifically, we use the prompt \"an <animal/object A> and a <B>\" and evaluate the result by detecting 2 targets <animal A> and <object A> using Owl-ViT detector. Our TEBOpt can address 3.67% object missing issue in animal prompts while the optimized results may lean towards the main meaning of the word in object prompts. For example, \"jaguar\" tends to represent an animal rather than a car, resulting in a 1.29% decrease in generating \"object jaguar\" after optimization.\n\n**Q2:** While we agree that \"dependency\" is a more accurate term, the term \"causal\" in this context refers to the causal masking manner used in the self-attention layer. This is described on P. 5 of the CLIP paper, where the source code function is also named \"causal_attention_mask\" (line 288 in modeling_clip.py). Thus, we used \"causal\" to maintain consistency with the terminology used in the CLIP documentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZqbLKTBEJR", "reviewer": "Reviewer_ACAY", "summary": "This paper proposes Text Embedding Balance Optimization (TEBOpt) to enhance the distinctiveness between text embeddings of equally important objects when using texts as conditions for diffusion models. It begins with an intriguing and informative experiment to investigate why a prompt like \"An <object1> and an <object2>\" given to a standard diffusion model, such as Stable Diffusion, often results in a higher likelihood of ignoring <object2> during image generation. The masking experiment elucidates the role of causal masking in contributing to this phenomenon.\nBased on these observations and analyses, the authors propose replacing the latter embedding with a pure embedding as a baseline. They then learn to increase the distance between the embeddings of the two objects while preserving the original meaning. Both qualitative and quantitative experimental results demonstrate that the proposed method helps to reduce bias towards the first object, leading to improved generation results, even compared to existing approaches that aim to address this issue.", "questions": "- As revealed by the analysis, not only the latter object text embedding contains the earlier object information, all the latter words all may have similar impacts to strength the text embedding of the earlier object information. Has the author considered how to resolve such possible influence?", "limitations": "The limitations are potential impact are discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is well-motivated by the detailed error analysis and masking experiments.\n- The solution proposed by the authors are novel and interesting.\n- Experimental results show that the method successfully help to improve the generation quality.\n- The paper is well-organized and easy-to-follow.\n- Code and models are provided to better support the reproductivity.", "weaknesses": "- Some of the wrapped table's layout can be improved (e.g. Table 1, Table 5)\n- The data used in this work can be systematically extended and organized to form a larger and good place for community to continue conducting research on this.\n- The research by far focuses on a few relatively simple scenarios, where only two or three objects in a very simple sentence pattern are presented. Explorations on more challenging scenarios would be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe sincerely appreciate your valuable suggestions and supportive feedback, as well as the time you took to review our work! We love your summary of our work, demonstrating a thorough understanding. In the following section, we respond to each weakness and question, while weakness 3 is included in the overall rebuttal at the top of this page.\n\n**Weakness 1:** Thank you for the suggestion. We will enhance the aesthetics of the table.\n\n**Weakness 2:** Thank you for the suggestion. We have already built an automatic data generator and an evaluation metric, and we plan to extend and organize the data for community use. In the supplementary materials, you can find the code for systematically extending the data in 'code/data/gen\\_prompt.py', and the code for the automatic evaluation metric in 'code/eval\\_metrics'.\n\n**Question 1:** Thank you for your insightful comment. We explored this question when we hypothesized the solution to our proposed problem. We conducted an experiment on the same 400-prompt set described in the paper, eliminating \"all\" earlier information from accumulating in subsequent tokens. The results are presented in the table below. Without shared embeddings across tokens, the generation process failed to produce co-existing objects. Specifically, during the denoising process, each object token responded in the central region, as observed in the cross-attention maps, resulting in no object co-existence. In conclusion, maintaining a proper proportion of earlier object token information in the latter tokens (excluding those with concrete meanings) has more positive than negative effects, especially in generating co-existing objects within a given prompt. Therefore, we propose to optimize the critical tokens' embeddings in the paper.\n\n\n| Method |  SD 1.4 w/o info accumulation |\n|:---------:|:--------:|\n|   2 object exist   |   0.00% | \n|  only mixture      |   11.75% | \n|   obj1 + mixture |   0.00% |  \n|   obj2 + mixture |   0.25% | \n|   only obj1 exists |   46.50% |\n|   only obj2 exists |   39.50% |   \n |   no target object |   2.00% |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4yqFmIicow", "reviewer": "Reviewer_LGCP", "summary": "The paper conducts an in-depth analysis of the impact of causal mechanisms in text encoders of text-to-image (T2I) diffusion models, which can lead to information bias and loss. The authors propose the Text Embedding Balance Optimization (TEBOpt) method, a training-free method to optimize text embeddings, resulting in a 90.05% improvement in information balance. Additionally, it presents a new metric for more accurate evaluation of information loss in generated images.", "questions": "a. According to the description in the text, Equation 6 is intended to encourage later tokens to be as dissimilar as possible from earlier tokens’ embedding. However, in the formula, the similarity is calculated between the object token and all tokens, with the summation range being 0 to n. This seems to contradict the goal of constraining the subsequent tokens to be dissimilar from earlier tokens?\nb. In the qualitative results shown in the second row of Figure 4, it can be observed that the results using the TEBOpt method do not seem to retain the visual features of the cat very well. Does this indicate that optimizing and adjusting text embeddings is likely to cause some uncontrollable changes in semantics? Additionally, when combined with other methods, it also seems to affect the overall layout of the generated images.", "limitations": "The paper summarizes the limitations of the proposed method in sec6, while mentioning that the case of multiple objects as well as complex prompts can be further explored. In addition, the experiments in this paper are based on simple sentences, which can be followed up to explore whether the case with complex adjectives affects the generation effects.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a. Originality: The paper offers a new perspective on T2I models by addressing the causal nature of text embeddings, which has been overlooked. The proposed TEBOpt represents a new approach to balancing information in text embeddings without additional training.\nb. Quality: Technically sound, the paper is well-supported by comprehensive analysis and experimental results. \nc. Clarity: The submission is clearly written and well-organized, making complex concepts accessible. \nd. Significance: The results are important for advancing the field of T2I models, offering a demonstrable improvement in handling information bias and loss. \n\nminors:\nsome relevant works should be cited and dicsussed:\nZou, Siyu, et al. \"Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 7. 2024.\nDu, Xiaoxiong, et al. \"Pixelface+: Towards controllable face generation and manipulation with text descriptions and segmentation masks.\" Proceedings of the 31st acm international conference on multimedia. 2023.", "weaknesses": "a. The paper has limited qualitative results and mentions that the analysis was conducted on a total of 400 prompts. Providing more qualitative results in the main text or appendix could better demonstrate the validity of the proposed method.\nb. Some of the formulas used in the paper are not thoroughly explained, making it difficult to understand the proposed method.\nc. The paper proposes a novel perspective to analyze object omissions and fusion issues in the T2I model. However, the method in the paper is rather straightforward, which to some extent may affect the quality of the generated images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe appreciate your valuable feedback and taking the time to review our work! We have carefully considered your suggestions and we would like to address your concerns. In the following section, we respond to each weakness and question, while weaknesses a and c are included in the overall rebuttal at the top of this page. Furthermore, we include corresponding qualitative and quantitative results with more complex prompts for 1,000 samples in Fig. 1 and Table 1 in the rebuttal PDF.\n\n**Minor in Strengths:** As suggested, we will discuss the related works. Specifically, Zou et al. [A] focuses on image editing based on the cross-attention maps' similarity during denoising process while Pixelface+ [B] focuses on facial manipulation based on text description and segmentation masks.\n\n**Weakness b and Question a:** Thank you for pointing out the questions about formulas. We clarify every formula related to our analysis (Eq. 3) and proposed method (Eq. 4 to 6) to provide more detailed information. \n\n1. Eq. 3 is the hypothesis equation, *expecting to solve the balance problem by replacing the token embedding of the later mentioned object with the corresponding pure token embedding.* Due to formatting difficulty in OpenReview, we change Eq. 3 into one line and correct one mistaken condition. $$ \\varepsilon' = \\varepsilon_i     \\textit{     if   }  i \\notin (O) \\textit{ or  } i = \\min (O) \\textit{ else  }  \\varepsilon_{p_{obj_n},5}, \\tag{3} $$ where n refers to the $n^{th}$ object. Eq. 3 shows how the text embedding $\\varepsilon'$ combined. Given a prompt \"a dog and a cat\", there are two critical tokens (<dog> and <cat>) with token indexes 2 and 5 in the critical token set $O$. Since we only replace the later mentioned token with a pure token embedding $\\varepsilon_{p_{obj_n},5}$, where 5 indicates the <cat> token index in the pure prompt \"a photo of a <cat>\", the equation indicates to remain original token embedding when token index i is not in the set $O$ or is equal to the minimum index in the set $O$. In this case, when i is equal to 5 and its corresponding n is equal to 2, the embedding would be replaced.\n\n2. Eq. 4, 5 and 6 are our proposed method's equations: *For rendering the correct formula, we omit the \"TEB\" originally on the right-bottom side for each loss.*\n$$ \\mathcal{L}^{pos}(\\varepsilon, \\varepsilon_p) =  \\underset{i \\in O}{\\min} sim(\\varepsilon_i, \\varepsilon_{p(i)}), \\tag{4}$$\n$$ \\mathcal{L}^{neg}(\\varepsilon) = \\frac{_1}{^{k(m-1)}} \\underset{i \\in O}{\\sum} \\underset{j=1; j \\neq i}{\\sum^{m-1}} sim(\\varepsilon_i, \\varepsilon_j),  \\tag{5} $$\n$$ \\mathcal{L} = -\\mathcal{L}^{pos}+\\mathcal{L}^{neg}, \\tag{6} $$\nwhere $sim (u, v) = \\frac{\\mathbf{u}_i \\cdot \\mathbf{v}_i}{\\|\\mathbf{u}_i\\| \\|\\mathbf{v}_i\\|}$ and m means the effective token count.\n- Eq. 4 encourages critical token embeddings $\\varepsilon_i$ from given prompt (a dog and a cat) to be more similar to pure critical token embeddings $\\varepsilon_{p(i)}$ from pure prompt (\"a photo of a <dog>\" and \"a photo of a <cat>\"). We design to select one with minimum similarity to update the text embedding since token with the minimum similarity requires more efforts to purify its embedding. \n\n- Thank you for pointing out the confusion. Eq. 5 encourages critical token embeddings to be less similar to other token embeddings, which has better performance than calculating only with its earlier tokens since the current formula encourages its later tokens not to accumulate all of its information. We believe it is not a conflict with our description since similarity is a mutual result between an earlier token and a later token. Here we calculate the average as the negative loss for update. Also, we correct the equation to \"j=1 to m-1\" since we do not use <sot> and <eot> for optimization.\n\n- Eq. 6 is the overall loss function, where the lower the better. For achieving this goal, we give a negative sign to the positive loss since it is encouraged to have a larger value/similarity. While we give a positive sign to the negative loss since it is encouraged to have a lower value.\n\n**Question b:** Thank you for your observation. Compared to the SD 1.4 result in Fig. 4 in the main paper, applying our TEBOpt method significantly reduces the lion’s visual features on the cat, although some features might not be fully eliminated. For example, within the region above the cat's head, distinct and upright strands of hair are eliminated but it still retains some shadow.\n\nThe layout of the generated images is controlled by the denoising process, which is influenced by the interaction between the text embedding and the image latent. While we use the same seed for the image latent in all methods with or without our TEBOpt, ensuring the initial image latent remains the same, the optimized text embeddings can still cause changes in the layout. When TEBOpt is combined with A&E or SynGen, it will change the layout compared to SD 1.4 method because A&E and SynGen optimize the image latent during the denoising process. Our primary goal is to address issues of object mixture and omission rather than directly controlling the layout. \n\n\n**References:**\n\n[A] Zou, Siyu, et al. \"Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 7. 2024. \n\n[B] Du, Xiaoxiong, et al. \"Pixelface+: Towards controllable face generation and manipulation with text descriptions and segmentation masks.\" Proceedings of the 31st acm international conference on multimedia. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hxpHu8vRsh", "reviewer": "Reviewer_fTPW", "summary": "This papers address the bia problem of text-to-image generation. Unlike the previous work which focus on processing of the attention map, this work explores the text embedding, which is a key factor.  Authors conduct expensive experiments to analysis the text embedding, then propose a interesting method to address the alignment  of the text-to-image generation.  A simple text embedding alignment loss is proposed by introducing other pure  text information.", "questions": "Please check the weaknesses.", "limitations": "Testing the presented problem in latest version of SD, like SD3.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Authors conduct lots of interesting experiments to analysis the alignment problem, including masking the special token. \n\n2. Exploring the text embedding instead of (stable diffusion ) UNet  network is good.\n\n3. Authors provide the code to show the effectiveness of the presented method.", "weaknesses": "1. The present problem (e.g., Fig1) seems just happy the lower version of SD1. I test SD3 with the prompt : 'a lion and an elephant', the issues does not happen. In the paper, authors show the reason is from the text encoder, which I  agree. However,  the presented reason maybe does not lead to the shown challenges. \n\n2. For text embedding, the EOT also play a key role[1]. This paper does not consider it. Only considering the soft replacing maybe harder.\n\n3. The visual results are not good. The generated images has poor quality. Does the propose lead to other negative influence?\n\n4. The baselines are  a little out of  fashion.  Authors should consider more recent methods: SDXL, SD3,  PlayG-v2, Ranni[2] and Ella[3].\n\n5. Authors should use a public benchmarks to compare the proposed method, like T2I-CompBench[4].\n\n6. The proposed idea seems limited for NeurIPS, which is not enough.\n\n[1] Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models\n[2] Ranni: Taming text-to-image diffusion for accurate instruction following.\n[3] Ella: Equip diffusionmodels with llm for enhanced semantic alignment.\n[4] T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe appreciate your valuable suggestions and taking the time to review our work! We have carefully considered your feedback, and we would like to address your concerns. In the following section, we respond to each weakness, while weakness 3 is included in the overall rebuttal at the top of this page. Furthermore, we include corresponding quantitative and qualitative results in Table 1 and Fig. 1 in the rebuttal PDF.\n\n**Weakness 1:** \nWe acknowledge that the issue occurs less frequently in SD3, as indicated in Table 1 in the rebuttal PDF. While SD3 was announced on Feb. 22, it was only open-sourced on June 12, which was after our paper was submitted. However, the problem is not completely resolved in SD3. In our experiments with the prompt \"a lion and an elephant\" using 1,000 random seeds, SD3 still exhibited the mixture issue in 2.9\\% of cases and the missing issue in 3.6\\% of cases. From SD 1 through SD 3, including SD XL, more text encoders with additional parameters have been integrated into stable diffusion models, with CLIP being a common choice. This underscores the importance of researching how the fundamental CLIP text encoder contributes to T2I models, a topic that has received little attention but is thoroughly researched in our paper.\n\n**Weakness 2:**\nThank you for the suggestion. We analyzed the role of EOT in the Fig. 3 in main paper. In our experiment, when we masked the token embeddings for \"a,\" \"cat,\" \"and,\" \"a,\" and \"dog,\" the remaining embeddings still generated both a dog and a cat in the image due to the information accumulated in the EOT, as discussed in line 154. EOT positively contributes to object coexistence, which aligns with [1] that discusses EOT's role in eliminating negative concepts in prompts. Therefore, we chose not to optimize EOT in our proposed method.\n\n**Weakness 4 and 5:**\nWe include a quantitative comparison in the rebuttal PDF as Table 1 to compare SD 1.4, SD 1.5, ELLA [3] (on the only released version, SD 1.5), SDXL-Turbo, and SD3 in the color set in T2I-CompBench [4] with 1,000 cases. Within all the baseline methods, our TEBOpt improves the 2 object co-existence rate with 7.9\\% on SD3 by addressing object mixture and object missing. Also, TEBOpt makes the generated information more balanced, where the info bias is from 2.07 to 1.30. More visual results are provided in Fig. 1 in the rebuttal PDF.\n\n**Weakness 6:**\nWe carefully consider this feedback and would like to share more thoughts on the decision of the basic model and benchmark. \n\nInitially, we focused on the SD 1.4 model because it contains one commonly-used text encoder, CLIP, the research findings of which could apply to other latest models. For example, current SOTAs, e.g., SDXL and SD3, are equipped with the CLIP text encoder. Regarding the evaluation benchmark, we created our own benchmark for effectively analyzing our proposed problem because there is no suitable benchmark.\n\nTo respond, we apply TEBOpt to the latest models, including SD 1.5 with ELLA, SDXL-Turbo, and SD3, on the public T2I-CompBench. We demonstrate that the proposed problem is still not addressed in the latest models, e.g., SD3. Our TEBOpt indeed addresses our proposed problem in all 3 models as well as in the more complex prompt. The visual results are in Fig. 1 in the rebuttal PDF. Specifically, on SD3, our TEBOpt increases 7.9% improvement for 2 object co-existence and makes information bias from 2.07 to 1.30. \n\nFurthermore, our paper conducts a thorough experiment on the contribution of the text encoder to text-to-image models, which is critical but has received little attention. Given the extensive discussion and rapid development of T2I models, and the emerging development of text-to-video models, we believe our task contributes significantly to the community.\n\n\n\n**References:**\n\n[1] Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models. ICLR, 2024.\n\n[3] Ella: Equip diffusionmodels with llm for enhanced semantic alignment. ArXiv, 2024.\n\n[4] T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation. NeurIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all reviewers,

We greatly appreciate the insightful suggestions and valuable comments from each reviewer. These have been immensely helpful and enlightening for refining this paper. In this section, we respond to common weaknesses and provide the PDF to include more qualitative and quantitative results on corresponding suggestions from reviewers. Enjoy!

1. **reviewer G9At (Weakness 1), reviewer ACAY (Weakness 3) & reviewer LGCP (Weakness a):** Thank you for your suggestions. We started with a plain setting to make sure the experimental findings were effective. In response, we conducted our TEBOpt experiments using Stable Diffusion (SD) 1.4 on the set of spatial relationships (e.g., "next to," "on the side of," etc.) within the T2I-CompBench [A]. This dataset includes nouns representing 5 types of people (man, girl, etc.), 16 types of animals (giraffe, turtle, etc.), and 30 types of objects (table, car, etc.), encompassing a total of 1,000 cases. The table below shows that our method demonstrates improvement in increasing the 2 object co-existence with 6.8% and reducing the information bias from 1.43 to 1.21. In this experiment, we further prove that when two nouns in the given prompt are from different categories, such as "a woman and a chair," resulting in a larger text embedding distance, it causes the mixture issue in text-to-image models to be concealed beneath the surface. Thus, while our paper focuses on the task of handling only animals, we reveal and address both the mixture and missing issue. Furthermore, we demonstrate that our method is effective across a more diverse set of categories. More qualitative results are in the Fig. 1 in the rebuttal PDF.

|          | SD 1.4 |SD 1.4 + TEBOpt |
|:--:|:--:|:--:|
|2 objects exist| 40.4% | **+6.8%** |
|only mixture | 0.2% | **-0.0%**|
|obj1+mixture | 0.0% | **+0.1%**|
|obj2+mixture | 0.1% | **-0.1%**|
|only obj1 exist | 32.3% | **-5.5%**|
|only obj2 exist | 22.6% | **-0.5%**|
|no target objs | 4.4% | **-0.8%** |
|info bias| 1.43| **1.21**|

2. **reviewer LGCP (Weakness c) & reviewer fTPW (Weakness 3):** Thank you for your thoughtful feedback. We follow SD3 [B] to conduct the image quality evaluation on Fréchet Inception Distance (FID) with CLIP L/14 image features on the generated images and the COCO 2017 val dataset [C] in 5,000 samples. The FID of (SD 1.4, SD 1.4 + TEBOpt), (SDXL-Turbo, SDXL-Turbo + TEBOpt), and (SD3, SD3 + TEBOpt) are (133.08, 133.30), (202.50, 200.71), and (143.77, 142.20), where *FIDs are higher than we usually see from text-to-image models is because the 5,000 generated sets are based on the plain prompt structure "a <objA> and a <objB>".* Visual performance is mainly affected by the selected text-to-image model as the FID for the generated images w/ or w/o TEBOpt are within marginal differences in the same model. When these 3 models work with TEBOpt, only SD 1.4 gets a 0.22 increase in FID score, while SDXL-Turbo and SD3 result in a 1.79 and 1.57 decrease in FID scores. It proves that TEBOpt tends to improve general image quality.



**References:**

[A] Huang et al. T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation. NeurIPS, 2023.

[B] Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ArXiv, 2024.

[C] Lin et al. Microsoft COCO: Common Objects in Context. ECCV, 2014.

Meta-review: This paper examines the role of text encoders for text-to-image (T2I) diffusion models, which can cause information bias and loss. This study analyzes text embedding's impact, especially when generating multiple objects. It introduces a training-free text embedding balance optimization method, improving information balance in stable diffusion. A new automatic evaluation metric is proposed, which measures information loss more accurately than existing methods, showing better agreements with human assessments.

We may summarize its strengths like:
- Motivation: 
  - "Proposed metric is a good effort" (G9At)
  - "Well-motivated," "novel and interesting" (ACAY)
  - "Originality" and "important" results (LGCP)
  - "Exploring the text embedding instead of UNet network is good" (fTPW)
- Results:
  - "Useful analyses," "detailed error analysis"  (G9At, ACAY)
  - "Experiment design cooresponds to the hypothesis" (G9At)
  - "Technically sound" (LGCP)
- Clarity:
  - "Well-organized and easy-to-follow" (ACAY)
  - "Clearly written and well-organized" (LGCP)
- Reproductivity:
  - "Code and models are provided to better support the reproductivity" (ACAY)
  - "Authors provide the code to show the effectiveness" (fTPW)

Initially, the reviewers pointed out the following notable weaknesses:
- Applicable coverage w.r.t. class, object count, and prompts (G9At, ACAY, LGCP)
- Bad qualitative results (fTPW) and weak baselines and insufficient comparison (fTPW)

Following the author-reviewer and reviewer-AC discussions, the authors successfully resolved the major issues by providing clarifications, additional results, and a reasonable rationale for their choice of baselines. The authors' thorough responses and consistently positive feedback from the reviewers sufficiently support a recommendation for acceptance.
Decision: Accept (poster)