A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization

Authors: chieh-yun  chen
chiang  tseng
li-wu  tsao
hong-han  shuai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 15:40:29

Abstract: This paper analyzes the impact of causal manner in the text encoder of text-to-image (T2I) diffusion models, which can lead to information bias and loss. Previous works have focused on addressing the issues through the denoising process. However, there is no research discussing how text embedding contributes to T2I models, especially when generating more than one object. In this paper, we share a comprehensive analysis of text embedding: i) how text embedding contributes to the generated images and ii) why information gets lost and biases towards the first-mentioned object. Accordingly, we propose a simple but effective text embedding balance optimization method, which is training-free, with an improvement of 125.42\% on information balance in stable diffusion. Furthermore, we propose a new automatic evaluation metric that quantifies information loss more accurately than existing methods, achieving 81\% concordance with human assessments. This metric effectively measures the presence and accuracy of objects, addressing the limitations of current distribution scores like CLIP's text-image similarities.

Main paper: 
 introduction: 
 Text-to-image (T2I) diffusion models [11, 12, 14, 17, 18] have recently captured significant attention. Subsequent research [4, 9, 13, 16, 1, 20, 3] has extensively explored the roles of cross-attention and self-attention mechanisms in the denoising process to enhance image control. Despite these advancements, there remains a critical gap in understanding the role of text embedding within T2I models, particularly in scenarios involving the generation of multiple objects. For instance, Fig. 1 illustrates that when prompted with "a lion and an elephant," models like Stable Diffusion often generate an ambiguous creature that blends features of both (or object missing on the right), highlighting issues of semantic interpretation and token embedding. This paper delves into how text embeddings influence the semantic outcomes of generated images, identifying specific problems of information bias and loss due to the causal nature of the self-attention mechanism in text encoders.The existing literature [1, 3, 16, 20, 13] has worked in image latents to address information loss, e.g., objects missing; however, there is no research on the main cause of the problem--text embedding. Therefore, this paper focuses on text embedding to investigate the issue. We first analyze the information bias in text embedding and find that the generated objects tend to bias towards the first mentioned object, as Table 1 shows, when the given prompt contains more than one object. The reason for biasing towards the first mentioned object is due to the causal manner in which making n th token embedding contains the weighted attention of token embeddings between 0 to (n -1) th tokens, as illustrated in the right-bottom side of the Fig. 2 . Moreover, we analyze the semantic contribution of the token embeddings by masking different tokens. Table 2 and Fig. 3 demonstrate that the causal manner would contribute to the accumulation of general information in the special token, e.g., end of token (<eot>), and padding token (<pad>). Masking the embeddings of given tokens still allows the T2I model to generate the expected information using the remaining special tokens' embeddings.Due to the significant information loss issues in T2I models, such as object mixing and missing, we dissect the generative process of the T2I model to pinpoint the origins of these losses. In the text embedding, the causal manner would make the embedding of the n th token mixed with the token embedding between 0 to (n -1) th , which makes the later token embedding similar to the earlier embedding. In the case of a prompt "a cat and a dog", the causal manner would mix the <dog> embedding with the <cat> embedding. This similarity in embeddings results in similar distributions on cross-attention maps, as detailed in Sec.4.4. When denoising directions on these maps align too closely throughout the denoising steps, it can cause a mixed representation if responses are equally low, or one object may overshadow the other if its map elicits a stronger response, leading to object disappearance. These phenomena are visualized in Fig. 1 . To address the issue of information bias and loss, we propose the Text Embedding Balance Optimization (TEBOpt) to promote distinctiveness between embeddings of equally important objects for preventing mixing and working alongside existing image latent optimization techniques to address object disappearance. The main contributions of this paper are outlined as follows:• This paper examines how text embedding contributes to generated images in text-to-image diffusion models and demystifies how the causal manner leads to information bias and loss while contributing to general information.• We propose the Text Embedding Balance Optimization solution containing one positive and one negative loss to optimize text embedding for tackling information bias with 125.42% improvement in stable diffusion.• We propose an evaluation metric to measure information loss. Compared to the CLIP score for evaluating text-image similarity, and the CLIP-BLIP score for evaluating text-text similarity, our evaluation metric provides a concrete number for identifying whether the specified object exists in the generated image. 
preliminaries: 
 Text-to-image diffusion models [11, 12, 14, 17, 18] typically contain the text encoder [15, 6] , a variational autoencoder (VAE), and a denoising UNet, as Fig. 2 demonstrates. Given a text prompt, the text encoder would first obtain the text hidden states h s ∈ R N ×D from the sum of token embedding and positional embedding, where N is the maximum token length in the text encoder and D represents the embedding dimension; then, it calculates the text embedding by going through the encoder layers with self-attention mechanism and causal masking manner. Next, given the text embedding ε and the initial image noise z t , the denoising UNet ϵ θ (z t , ε, t) would gradually denoise latents in each timestep t to get the final image by iteratively predicting the noise residuals conditioned on the text embedding and the previous denoised latents.Notably, the causal masking manner in the text encoder makes every token have information only from its previous tokens, which causes the text embedding to have information bias. The bottom of Fig. 2 illustrates how the causal manner works in the self-attention mechanism [21] . The query Q, key K, and value V are calculated as follows:Q = ℓ Q (h s ), K = ℓ K (h s ), V = ℓ V (h s ), where K, Q, V ∈ R hn×N ×D/hn (1)Figure 2 : Overview of the text-to-image generative model, including the details of the causal manner in attention mechanism. Because of the causal nature of the embedding, information is accumulated from the starting token through the end of the sequence, resulting in bias in the earlier token. To balance the critical information, we propose text embedding optimization for purifying the object token with equal weights within their corresponding embedding dimension.EQUATIONwhere ℓ Q , ℓ K , and ℓ V are learned linear projections, and h n represents the head number. d k is the dimension of K and M is the causal mask. Upon receiving the attention weight QK ⊤ ∈ R N ×N , the mechanism applies the causal mask M. Then, taking V as a weight metric to get the weighted sum of attentions, representing the embedding for each token. Due to the causal manner, the embedding of the n th token (ε n ) contains weighted information for the tokens 0 to n -1. This representation causes two issues: First, the earlier token information accumulates the most since every subsequent token has it. Furthermore, the later token's embedding is not a pure representation of itself, potentially causing identity loss. For example, given a text prompt "a cat and a dog", the earlier token <cat>'s information would be mixed into the embedding of the later token <dog>. It makes the generated image have a higher possibility of generating two cats, as the middle row of Fig. 2 shows. 
analysis and method: 
 In this section, we analyze how causal manner affects the text embedding as well as the generated images in text-to-image diffusion models.We investigated 400 different prompts with different random seeds with structures (a) "a/an <object1> and a/an <object2>" and (b) "a/an <object2> and a/an <object1>" by exchanging the position of <object1> and <object2> in prompt (a), where objects are randomly sampled from 17 different animals.Figure 3 : Masking text embedding to identify the contribution of critical tokens, e.g., cat/dog, and special tokens, e.g., <sot>, <eot>, <pad>. The first row and the second row both contain cat and dog inside prompt but in different order. The analysis shows that special tokens contain general information about the given prompt. However, the cat/dog tokens carry more weight than the special tokens. In the last two columns, where one of the animal token embeddings is masked while retaining the special tokens' embedding, the generated image is predominantly influenced by the remaining animal's token embedding. Table 1 shows that within the same set of two animals and the same random seed, both bias on the earlier mentioned animal. In prompt (a), the statistics with pink background show that it causes 67.5% missing issue, including 46.0% generating only object 1. Similarly, in prompt (b), it causes 69.5% missing issue, including 47.0% generating only object 2. The information bias (Info bias) is defined by # of only obj1 exist # of only obj2 exist and, in both prompts, they are 2.30 and 0.46 respectively, which are far from balanced (Info bias = 1).Driven by the observation of information bias, we investigated the text embedding contribution of critical tokens. In Stable Diffusion, we first utilize CLIP [15] as the text encoder to obtain the text embeddingε i ∈ R N ×D , where i ∈ [0, N -1].Before the denoising UNet gets the text embedding and image latent, we mask the chosen embedding and generated the corresponding images. Table 2 provides the quantitative analysis in 400 different prompts with the same setting of (a) used in Sec. 3.1, extending with three masking settings corresponding to Fig. 3 's each column.Firstly, the default experiment in Table . 2 demonstrate stable diffusion has 20.25% mixture, 67.50% missing ratio, and 2.3x information bias to the first mentioned object. Regarding the visualization sample of the default experiment, the first column in Fig. 3 shows that animals (cat and dog) within one prompt in different orders with the same initial image latent would generate different results. The bottom one demonstrates the mixture issue, making the right animal simultaneously similar to dog and cat. Regarding the second experiment, we mask all the embeddings of the given tokens, where Replacing the token embedding of later mentioned object from the corresponding pure embedding can balance the information but lead to a large drop of two objects coexistence.ε i = -inf, i ∈ [1,Following the setting of the prompt structure (a) in Sec. 3.1, with a text prompt "a/an <obj1> and a/an <obj2>", the token index of <obj1> and <obj2> are taken into the critical token set O = {2, 5}. It first generates the pure embedding of the later mentioned one, "a photo of a <obj2>" (ε p obj2 ). Then, replace the corresponding token embedding of <obj2> in the original embedding (ε) with the pure one. The combined text embedding (ε ′ ) is as follows:EQUATIONwhere n refers to the n th object. Table 3 reflects that directly replacing original embedding with the pure embedding would balance the information of object 1 and 2; however, it would result in a 5.25% loss in 2 objects coexistence. A nearly equal probability of generating only objects 1 and 2 proves that replacing token embedding with pure embedding eliminates accumulated information about object 1.While the causal manner results in information bias and information loss, it contributes to generate image content aligned with the general prompt information. To eliminate the accumulated information but retain general information, we design a Text Embedding Balancing Optimization, called TEBOpt (uppermost in Fig. 2 ), cooperating with image latent optimization to address these issues. The TEBOpt tackles information bias and object mixture issues while the latent optimization tackles object mixture and missing issues. Since the missing object is caused by an insufficient response value and an inadequately activated region in the cross-attention map corresponding to that object, and the text embedding is unable to precisely determine the activated position, our method cooperates with existing latent optimization methods to address this issue.Regarding a prompt containing two objects, we expect their corresponding text embedding to be unmixed instead of mixing earlier tokens' embeddings. Since the ultimate goal is to preserve the general information of the two objects, we cannot directly replace the original embedding with the pure object's text embedding. This would result in a high probability of losing one of the objects. Thus, our proposed TEBOpt contains a TEB loss in order to encourage the later mentioned token's embedding to be less similar to the earlier token's embedding, while at the same time being as similar to its pure embedding. Considering a text prompt with k objects in a set O, we first obtain each text embeddings ε p ∈ R N ×D and take the critical token embedding ε p,i ∈ R 1×D as the pure embedding. For example, the prompt "a dog and a cat" contains two objects and the pure prompt embedding is calculated in a format prompt of ["a photo of a <dog>", "a photo of a <cat>"]. In summary, the TEB loss is as follows:EQUATIONEQUATIONEQUATIONwhere sim(u, v) = ui•vi ∥ui∥∥vi∥ and m means the effective token count, where we do not include <sot> and <eot> for optimization. The implementation details are included in Supplement B. After the text embedding is optimized, the image latent would be updated conditioned on the loss design for cross-attention maps during the denoising process. For example, A&E [1] contains a loss function to ensure that each selected token activates some image patches in the cross-attention map. SynGen [16] designs a loss function to encourage the cross-attention map of the relative token to be similar and make the cross-attention map of the unrelative token dissimilar. 
experimental settings: 
 Baselines. We compare our proposed method with the default Stable Diffusion 1.4 [17] , and 3 state-of-the-art (SOTA) baselines, including Structure Diffusion [3] , Attend-and-Excite [1] , and SynGen [16] . All of them focus on improving attribute bindings or solving object missing in text-to-image diffusion models. However, there is no one considering the information bias. In this literature, the objective is to analyze information balance caused by pretrained text encoders instead of surpassing existing SOTAs on solving object missing. Therefore, we would provide the experimental results of our proposed method on top of the baselines.Data. We follow previous methods [1, 20] to create a set of 400 prompts with the format "a/an <obj1> and a/an <obj2>" with corresponding random seeds. The objects are sampled from 17 different animals defined by previous methods [1, 20] , including cat, dog, bird, bear, lion, horse, elephant, monkey, frog, turtle, rabbit, mouse, panda, zebra, gorilla, penguin and chicken.To analyse the issues of mixed objects and missing objects in generated images, we designed an automated evaluation method since existing metrics, e.g., text-image similarity using CLIP [15] or text-text similarity using CLIP and BLIP [7] , used in SOTAs [1, 20] cannot provide the exact counting number to indicate whether the object exists or not. Detailed discussions are provided in Supplement C. First, we employed a pre-trained object detection model, OWL-ViT [10] , which is a SOTA in open-vocabulary object detection. We separate the text prompt into k objects and the model separately predicts bounding boxes and confidence scores for the corresponding objects. Mixture status is determined when the overlap of the two bounding boxes for different objects exceeds 90%. Here, we use two different thresholds to detect mixture objects and single objects, ensuring detection accuracy. Additionally, to validate the effectiveness of this automatic metric, we conducted a human evaluation to demonstrate that its results are highly correlated with human perception. We asked users to label 400 generated images, each categorized into one of five options: i) two objects exist, ii) mixture exists, iii) missing object 1, iv) missing object 2, or v) no objects exist. Our automatic evaluation metric achieves an accuracy of 81% based on human responses, demonstrating its effectiveness. As an object is wrapped by two bounding boxes with different colors, e.g., A&E [1] in the first row or Structure Diffusion [3] in the second row, it has a high potential to contain mixed objects. In the first row in Fig. 4 , our TEBOpt (L T EB ) helps Stable Diffusion generate the specified dog and help A&E to make the horse dissimilar from the dog. In the second row, our TEBOpt (L T EB ) helps Stable Diffusion generate a cat less similar to a lion, which is initially equipped with a mane as a visual signal of a lion. Regarding Fig. 5 , we demonstrate the generated image with and without our proposed TEBOpt (L T EB ). Using our TEBOpt, T2I models solved both the object mixture and missing issues, especially when trying to solve the problem of object mixtures. It is worth noting that objects mixture or missing are also affected by the denoising process in T2I models. Our text optimization mainly contributes to balancing the information in the text embedding and further reducing mixed and missing issue.Table 4 provides the comparative experiment for object mixture, missing and information bias. As Structure Diffusion [3] manipulates key and value from text embeddings in denoising UNet's cross-attention calculation, which works on text embedding as ours, our method working on top of Stable Diffusion can directly compare with it. Furthermore, SynGen [16] generates a reverse trend between object 1 and 2 since it works to make the two objects' cross-attention maps' distance as far as possible, which would contribute to separating two objects leading to a large improvement in object missing and information balance. With our text optimization, we can further solve mixture issue and making the balance better since we tackle the issue from the front of the problem in text embedding. Thus, text embedding optimization and image latent optimization reach out a good cooperation for solving information bias and loss. The generalizability of L T EB in information bias for more than 2 objects is reported in Supplement D.In Table 5 , we evaluate the similarity of token embeddings and the distance of cross-attention maps between two objects within one prompt. Token embedding similarity (Token sim) is calculated by cosine similarity while the cross-attention map distance (Map dist) is calculated by the symmetric Kullback-Leibler divergence between two normalized cross-attention maps M i and M j :1 2 (D KL (M i ||M j ) + D KL (M j ||M i )), where D KL (M i ||M j ) = pixels M i log(M i /M j ).With our L T EB , the token embedding similarity between two objects reduces 36.98% and the cross-attention map distance increases 13.62%. Discussion of how the similarity of text embedding affects cross-attention maps' distance.We calculated the cosine similarity between various text embeddings and displayed the results in Fig. 6 (a). The data indicates that objects with similar colors or sizes, e.g., penguin-panda or turtlefrog, tend to exhibit higher similarity in their text embeddings, which can be attributed to the training mechanism of CLIP. Additionally, we computed the distance between the cross-attention maps, using 1 The info bias value indicates less bias when it is closer to 1, different from the intuitive assumption that lower values represent less bias. Here's the detailed calculation: First, we calculate the bias distance between the info bias value and the balanced value (which is 1). The bias distances of SD and SD + TEBOpt are (2.647-1)/1 = 164.71% and (1.393-1)/1 = 39.29%. Note that we have rounded the values for information bias in Table 4 , reporting 2.647 as 2.65 and 1.393 as 1.39. Secondly, the balance improvement is then calculated as 164.71%-39.29% = 125.42%. the same function in Table 5 , generated by the two objects' text embeddings with the same initial latent in the early denoising steps. As shown in Fig. 6 (b), there is a positive correlation between the similarity of text embeddings and the distance of the cross-attention maps triggered by the two objects. Specifically, objects with similar text embeddings are more likely to activate overlapping areas during the denoising process. This confirms that similar text embeddings contribute to object mixture, while the short distance of cross-attention maps leading to object missing has been proven by SynGen [16] . Object mixture or missing. Stable Diffusion [14] pointed out that stable diffusion models have the issue of concept bleeding, which occurs by unintended merging or overlap of distinct visual elements, leading to object mixture or missing. Also, the root cause lies in the usage of pretrained text encoders, including CLIP [15] and OpenCLIP [6] . However, all the existing methods investigate the issue in the denoising process instead of text encoders. For instance, Attend-and-Excite [1] proposed an optimization process to ensure every selected token triggers some image patches when calculating the cross-attention maps between text embedding and image features. SynGen [16] designed a loss function to increase cross-attention maps' similarity between modifier-entity pairs and enhance attribute binding. Structure Diffusion [3] leveraged linguistic structure to separate the given prompt into several noun phrases and modify the corresponding value to manipulate text embedding when calculating cross-attention maps in denoising steps. Predicated Diffusion [20] designed to decompose the prompt containing several objects into independent objects and minimize the loss between the generated image based on the combined independent prompt and that of the original prompt. Refocus [13] proposed to define each object's positional bounding boxes with GPT-4 and designed two loss functions to make the object only denoised inside its given region. 
related works: 
 However, there is no task investigating the causal manner in the pre-trained text encoders. Therefore, we share a comprehensive analysis and propose a simple solution to tackle the issue of information bias, object mixture and missing, e.g., generating one animal with bear head and turtle shell when prompting "a turtle and a bear" or generating two cats when prompting "a cat and a dog". 
conclusion: 
 In this study, we conducted a detailed analysis of text embedding's impact on text-to-image diffusion models, a topic rarely explored. Our findings indicate that the causal processing of text embedding leads to information accumulation, causing biases and loss. Directly replacing accumulated embeddings with purified embeddings, though resulting in decreased coexistence of two objects, enhances the balance between generating either object. We introduce a training-free Text Embedding Balance Optimization (TEBOpt) method that effectively eliminates problematic information in critical token embeddings, improving information balance handling in stable diffusion by 125.42% while preserving object coexistence performance. Additionally, due to the unreliability of existing metrics for assessing inaccuracies in generated images, we propose a new automatic evaluation metric to more effectively measure information loss. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please refer to Supplement A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Please refer to Supplement A.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly cited all the codes that we have referenced in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. Justification: We conducted a human evaluation experiment to prove the effectiveness of our proposed evaluation metric. There is no risk in labeling the generated images to identify mixed objects or missing objects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This study deepens our understanding of text embedding in text-to-image (T2I) models, showing that better insights into pre-trained text encoders can markedly enhance accurate content generation. We identify that text embeddings accumulating too much information can introduce biases and data loss, insights that are vital for developers of large vision-language models to effectively mitigate these problems. For safeguards, we have adopted the safeguard provided in Stable Diffusion [17] , aligning with ethical standards and preventing misuse of generative models.Computing resources and method efficiency. All experiments are run on one NVIDIA RTX 3090 24 GB GPU. We calculated the average inference time from 400 images. The average inference time of default Stable Diffusion 1.4 is 8.21 seconds/image while our proposed L T EB in Stable Diffusion 1.4 is 9.14 seconds/image. Model architecture. All the analyses are conducted on Stable Diffusion 1.4, where the text encoder is the pre-trained CLIP ViT-L/14 [15] and the scheduler is the PNDMSchedular. In the text encoder, CLIP, the maximum token length (N) is 77, the embedding dimension (D) is 768, and the attention head number (h n ) is 12. We use a fixed guidance scale of 7.5 and set the denoising step to 50. For performing the TEB loss in Equation 6, we set a maximum optimization time as 20 and a threshold for L T EB as -0.7, which comes from L pos T EB = 0.95 and L neg T EB = 0.25. The optimization target for L T EB is -0.7 but if the optimization time exceeds 20, it will stop optimizing the text embedding. Moreover, all baselines are conducted with their official source codes.Cross-attention map visualization. We aggregate and take the average of all cross-attention maps with a size not larger than 32x32 in all 50 denoising steps. Note: when calculating the distance between cross-attention maps, we do not accumulate the cross-attention maps within 50 denoising steps. We only take the cross-attention map in corresponding denoising step.Masking token embedding. We set the selected dimension of encoder_attention_mask in the denosing UNet as 0 while the unselected dimension is 1. The working mechanism for encoder_attention_mask is that after key multiplying query to get the attention matrix, the dimension with 0 inside encoder_attention_mask would add -10,000 to the attention metric while the dimension with 1 would remain unaffected. After processing the attention metric with sof tmax, the selected dimension in the attention matrix would become 0.The existing evaluation metrics for missing objects are two-fold: text-image similarity and text-text similarity.Text-image similarity is calculated by the CLIP cosine similarity between the text prompt and the corresponding generated images. A&E [1] further separated the metric into Full Prompt Similarity and Minimum object Similarity. Since full prompt similarity may not accurately reflect the existence of missing objects, minimum object similarity is proposed to evaluate the most neglected subjects independently. Take "a dog and a cat" as an example. It first separates the prompt into "a dog" and "a cat", evaluates both separately and uses the minimum score as the metric to indict missing objects.Text-text similarity is calculated by the CLIP cosine similarity between the text prompt and the caption of the generated image obtained by the pretrained BLIP image-captioning model [7] . However, all these metrics cannot reflect the mixed objects issue and cannot accurately reflect how many missing objects are in the generated images. Thus, we propose a new evaluation metric for reporting the number of mixed and missing objects as discussed in Sec.4.2. Furthermore, we discuss the 3 metrics that previous methods used, including full prompt similarity, minimum object similarity and text-text similarity in Figs. 7, 8 and 9. In Fig. 7 , given the prompt "a cat and a penguin", the generated image on the left contains a penguin and one mixture of penguin and cat, while the generated image on the right contains a penguin and a cats. However, the left one, which contained the mixture object, has 3.57% higher in the full prompt similarity than the right one without mixture objects. It indicates that the full prompt similarity cannot properly measure the mixture cases. Moreover, the text-text similarity of the left one is 59.56% higher than the right one, which indicates that it cannot identify the mixture issue. In contrast, our proposed metric can accurately predict mixture/independent objects with a specific number. As indicated in red color, the image on the left contains a mixture of cat and penguin but its full prompt similarity and text-text similarity are higher than which of the image on the right.In Fig. 8 , the image on the left generates a bear and a mixture of bear and cat, while the image on the right has a bear and two cats. In terms of text-text similarity, the left one is 8.68% higher than the right one. Again, it indicates that it cannot identify the mixture issue. In contrast, our proposed metric can accurately report that the left one contains 1 bear and 1 mixture, while the right one contains 1 bear and 2 cats. In Fig. 9 , both images generate a mixture of bear and lion but the left one has a 33.48% higher text-text similarity than the right one. In terms of full prompt similarity and minimum object similarity, they have a difference of 7.01% and 15.27% respectively. These variations make the evaluation metrics unreliable to measure object mixture and missing. In contrast, our proposed metric can accurately report that both images contain one mixed object.Figure 9 : In two images both with mixed objects, full prompt similarity, minimum object similarity, and text-text similarity all vary greatly, making the evaluation metrics unreliable for object mixture and missing.Considering the working mechanism of SOTA text-to-image models, when the cross-attention maps of 2 objects respond closely during denoising, there is a high probability of generating a mixture object, as Fig. 10 shows. Consequently, within the nature of the Owl-ViT detector, these mixtures can be identified by their high overlap with high confidence. In Table 6 , we demonstrate the information bias on three objects, where the problem could extend to general scenarios with a larger number of objects. In each prompt from (a) to (f), the first mentioned object always has an existence of over 20%, while the existence of the second and third mentioned objects gradually decreases to below 20% and 10%. This confirms our observation regarding information bias, which gradually overlooks the later mentioned objects. By incorporating our custom loss function L T EB , we can achieve more balanced representation of the three objects, with the existence values ranging between 10% and 20%, resulting in the information bias closer to 1. Prompt (a) A/An <obj1> and a/an <obj2> and a/an <obj3> (b) A/An <obj2> and a/an <obj1> and a/an <obj3> (c) A/An <obj3> and a/an <obj2> and a/an <obj1> (d) A/An <obj1> and a/an <obj3> and a/an <obj2> (e) A/An <obj2> and a/an <obj3> and a/an <obj1> (f) A/An <obj3> and a/an <obj1> and a/an <obj2> Table 6 : Analysis of information bias in multiple objects. When there are more objects in the prompt, the bias might gradually enlarge as the order of the mentioned objects gets farther from the first mentioned object. In this setting, we utilize the same evaluation metric to define the information bias among three objects, which is divided into pairs of (obj1, obj2), (obj1, obj3), and (obj2, obj3).As the info bias approaches 1, it yields more balanced results regarding the existence of the object.E More qualitative and quantitative comparisons in T2I-CompBench [5] More qualitative results are in Fig. 11 , Fig. 12 , Fig. 13 , and Fig. 14 . The public benchmark, e.g., T2I-CompBench [5] , contains more diverse prompts, e.g., adjective binding with noun, that would make the experiment distract. Thus, in the main paper, we follow related works, e.g., A&E (SIGGRAPH'23) [1] and Predicated diffusion (CVPR'24) [20] , to create the most suitable benchmark for our task. While we respectfully disagree with the directness to conduct public T2I-CompBench to demonstrate our performance, we still conduct quantitative and qualitative comparisons in T2I-CompBench and prove our effectiveness.We include a quantitative comparison in Table 7 to compare SD 1.4, SD 1.5, ELLA (ArXiv'24; on the only released version, SD 1.5), SDXL-Turbo [19] , and SD3 [2] on the color set within the T2I-CompBench [5] with 1,000 cases. Within all the baseline methods, our TEBOpt improves the 2 object co-existence rate with 7.9% on SD3 by addressing object mixture and object missing. Also, TEBOpt makes the generated information more balanced, where the info bias is from 2.07 to 1.30.We conducted our TEBOpt experiments using Stable Diffusion (SD) 1.4 on the set of spatial relationships (e.g., "next to," "on the side of," etc.) within the T2I-CompBench [5] . This dataset includes nouns representing 5 types of people (man, girl, etc.), 16 types of animals (giraffe, turtle, etc.), and 30 types of objects (table, car, etc.), encompassing a total of 1,000 cases. Table 8 shows that our method demonstrates improvement in increasing the 2 object co-existence with 6.8% and reducing the information bias from 1.43 to 1.21. In this experiment, we further prove that when two nouns in the given prompt are from different categories, such as "a woman and a chair," resulting in a larger text embedding distance, it causes the mixture issue in text-to-image models to be concealed beneath the surface. Thus, while our main paper focuses on the task of handling only animals, it contains different challenges compared to handling more diverse objects in different categories. In our focus, we reveal and address both the mixture and missing issue. Furthermore, in the following experiment, we demonstrate that our method is effective across a more diverse set of categories. We follow SD3 [2] to conduct the image quality evaluation on Fréchet Inception Distance (FID) with CLIP L/14 image features on the generated images and the COCO 2017 val dataset [8] in 5,000 samples. The FID of (SD 1.4, SD 1.4 + TEBOpt), (SDXL-Turbo, SDXL-Turbo + TEBOpt), and (SD3, SD3 + TEBOpt) are (133.08, 133.30), (202.50, 200.71), and (143.77, 142.20), where FIDs are higher than we usually see from text-to-image models is because the 5,000 generated sets are based on the plain prompt structure "a <objA> and a <objB>". This experiment proves that visual performance is mainly affected by the selected text-to-image model as the FID for the generated images w/ or w/o TEBOpt are within marginal differences in the same model. When these 3 models work with TEBOpt, only SD 1.4 gets a 0.22 increase in FID score, while SDXL-Turbo and SD3 result in a 1.79 and 1.57 decrease in FID scores. It proves that TEBOpt would improve the general image quality. G.1 Would the same happen for the words may change their meaning due to nearby words? E.g., mouse?We experimented with 1,000 samples on the effect of words that may change their meaning due to nearby words, including "mouse", "horn", "jaguar", "falcon", and "palm". Specifically, we use the prompt "an <animal/object A> and a <B>" and evaluate the result by detecting 2 targets <animal A> and <object A> using Owl-ViT detector. Our TEBOpt can address 3.67% object missing issue in animal prompts while the optimized results may lean towards the main meaning of the word in object prompts. For example, "jaguar" tends to represent an animal rather than a car, resulting in a 1.29% decrease in generating "object jaguar" after optimization. G.2 As revealed by the analysis, not only the latter object text embedding contains the earlier object information, all the latter words all may have similar impacts to strength the text embedding of the earlier object information. Has the author considered how to resolve such possible influence?We explored this question when we hypothesized the solution to our proposed problem. We conducted an experiment on the same 400-prompt set described in the paper, eliminating "all" earlier information from accumulating in subsequent tokens. The results are presented in Table 9 . Without shared embeddings across tokens, the generation process failed to produce co-existing objects. Specifically, during the denoising process, each object token responded in the central region, as observed in the cross-attention maps, resulting in no object co-existence. In conclusion, maintaining a proper proportion of earlier object token information in the latter tokens (excluding those with concrete meanings) has more positive than negative effects, especially in generating co-existing objects within a given prompt. Therefore, we propose to optimize the critical tokens' embeddings in the paper. Figure 13 : More qualitative results on SDXL-Turbo [19] in complex prompts from color set within T2I-CompBench [5] .Figure 14 : More qualitative results on SD3 [2] in complex prompts from color set within T2I-CompBench [5] .H The instruction for participants in human evaluation for our proposed evaluation metricFull-text instruction: "You will be given an image and a text description. The text is described as "a/an <object1> and a/an <object2>". Please determine whether the objects in the image are a combination of object 1 and object 2, or if any object is missing. Select the corresponding answer from the 5 options: i) two objects exist, ii) mixture exists, iii) missing object 1, iv) missing object 2, or v) no objects exist." The screenshot of the human evaluation is demonstrated in Fig. 15 .Figure 15 : The screenshot of the human evaluation, containing the information and options that are given to participants.Despite the effectiveness of our proposed text embedding balance optimization in balancing information, whether the generated images contain mixed objects or lost objects is still affected by the denoising process. Furthermore, there is still room to improve in identifying the importance of information within one complex prompt. For example, this literature investigates in the prompt with equally importance objects, e.g., a <object 1> and a <object 2>, and it can be extended to more objects. In more complex prompts, such as a dog and cat playing with a mouse in front of a yard, in which the mouse is not a device, the text embedding might indicate that the device is less important.Answer Justification: We have carefully followed the code of ethics in the paper.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper examines the role of text encoders for text-to-image (T2I) diffusion models, which can cause information bias and loss. This study analyzes text embedding's impact, especially when generating multiple objects. It introduces a training-free text embedding balance optimization method, improving information balance in stable diffusion. A new automatic evaluation metric is proposed, which measures information loss more accurately than existing methods, showing better agreements with human assessments.

We may summarize its strengths like:
- Motivation: 
  - "Proposed metric is a good effort" (G9At)
  - "Well-motivated," "novel and interesting" (ACAY)
  - "Originality" and "important" results (LGCP)
  - "Exploring the text embedding instead of UNet network is good" (fTPW)
- Results:
  - "Useful analyses," "detailed error analysis"  (G9At, ACAY)
  - "Experiment design cooresponds to the hypothesis" (G9At)
  - "Technically sound" (LGCP)
- Clarity:
  - "Well-organized and easy-to-follow" (ACAY)
  - "Clearly written and well-organized" (LGCP)
- Reproductivity:
  - "Code and models are provided to better support the reproductivity" (ACAY)
  - "Authors provide the code to show the effectiveness" (fTPW)

Initially, the reviewers pointed out the following notable weaknesses:
- Applicable coverage w.r.t. class, object count, and prompts (G9At, ACAY, LGCP)
- Bad qualitative results (fTPW) and weak baselines and insufficient comparison (fTPW)

Following the author-reviewer and reviewer-AC discussions, the authors successfully resolved the major issues by providing clarifications, additional results, and a reasonable rationale for their choice of baselines. The authors' thorough responses and consistently positive feedback from the reviewers sufficiently support a recommendation for acceptance.

Reviews: review: paper_summary: paper_summary: Background: Text-to-image diffusion models, such as Stable Diffusion, frequently encounter difficulties in accurately generating images from textual descriptions, especially when there are multiple objects -> object mixing and missing.

This paper focuses text embeddings to solve the problems.

Hypothesis: self-attention mixes text embeddings of multiple objects.

Analyses:
* 400 prompts of "a/an <animal 1/2> and a/an <animal 2/1>" with 17 animals produce lots of mixture of objects and missing objects
* Contribution of text embeddings to the generated images
    * Masking the dominant embeddings reduces the mixture rate 
    * Replacing the embedding after the text encoder with the pure text embedding does not solve the problem.
* Process of conditions getting lost and biasing towards the first-mentioned object due to masks in self-attention in the text encoder.

Method:
* Optimize the target text embedding toward pure embedding and against earlier (first-mentioned) token’s embedding
strength_weakness: strength_weakness: (Overall, I like the idea of this paper but it needs more work to be a solid paper.)

W1. This paper covers only nouns, especially animals. The research community has tackled different aspects of text prompts [composable, masactrl]. How about using other classes of objects to prove generality? For instance, could this method work with text prompts like 'A <cat> standing next to the <tower>' or examples from A&E such as 'A <lion> with a <crown>'? It would be valuable to see if the approach is equally effective with these more diverse classes and prompts that users might want to apply.

W2. Bboxes overlapping more than 90% not always indicate mixture.

W3. The terms should be more clearly defined.
* Information bias and object missing are confusing.
* The critical dimension?

W4. Initial latent optimization is poorly described. How can the initial latent be updated during the denoising process? I guess intermediate latents can be updated just like [self-guidance].

W5. The baseline of Figure 5 is not defined. 

W6. The appendix should provide large number of examples to ensure the proposed method is free from cherry picking.

W7. Improvements in Table 4 is marginal.

W8. An important paper is missing: [composable], [custom diffusion] and [hiper]

W9. The method requires optimization for each sentence, which can be inefficient for practical use.

Misc.
    1. Placing the caption of Table 5 under the table would be better for consistency.
    2. Some numerical values in the main text are difficult to understand at first glance. To improve readability, I think it would be better to highlight or mark the key values within the table itself. For example, the main text mentions 125.42%, but I couldn't immediately identify which part in Table 4 shows a difference of 125.42%. 
    3. The result shows that the balance performance in SynGen decreased by 10.65%, rather than improved.
    4. I think Figure 7 is not an appropriate example for demonstrating the inadequacy of full prompt similarity, as neither image correctly matches the given prompt 'a cat and a penguin'. The left image shows a mixture of the two animals, while the right image displays two cats instead of one.

[composable] Compositional visual generation with composable diffusion models, Liu+

[masactrl] MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, Cao+

[self-guidance] Diffusion Self-Guidance for Controllable Image Generation, Epstein+

[custom diffusion] Multi-Concept Customization of Text-to-Image Diffusion, Kumari+

[hiper] Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion, Han+
questions: questions: Q1. Would the same happen for the words may change their meaning due to nearby words? E.g., mouse?

Q2. I do not think the self-attention layer models "causal" relationship between words. I recommend replacing "causal manner" with "dependency". Thoughts?
limitations: limitations: Some are covered.
paper_summary: paper_summary: This paper proposes Text Embedding Balance Optimization (TEBOpt) to enhance the distinctiveness between text embeddings of equally important objects when using texts as conditions for diffusion models. It begins with an intriguing and informative experiment to investigate why a prompt like "An <object1> and an <object2>" given to a standard diffusion model, such as Stable Diffusion, often results in a higher likelihood of ignoring <object2> during image generation. The masking experiment elucidates the role of causal masking in contributing to this phenomenon.
Based on these observations and analyses, the authors propose replacing the latter embedding with a pure embedding as a baseline. They then learn to increase the distance between the embeddings of the two objects while preserving the original meaning. Both qualitative and quantitative experimental results demonstrate that the proposed method helps to reduce bias towards the first object, leading to improved generation results, even compared to existing approaches that aim to address this issue.
strength_weakness: strength_weakness: - Some of the wrapped table's layout can be improved (e.g. Table 1, Table 5)
- The data used in this work can be systematically extended and organized to form a larger and good place for community to continue conducting research on this.
- The research by far focuses on a few relatively simple scenarios, where only two or three objects in a very simple sentence pattern are presented. Explorations on more challenging scenarios would be interesting.
questions: questions: - As revealed by the analysis, not only the latter object text embedding contains the earlier object information, all the latter words all may have similar impacts to strength the text embedding of the earlier object information. Has the author considered how to resolve such possible influence?
limitations: limitations: The limitations are potential impact are discussed in the paper.
paper_summary: paper_summary: The paper conducts an in-depth analysis of the impact of causal mechanisms in text encoders of text-to-image (T2I) diffusion models, which can lead to information bias and loss. The authors propose the Text Embedding Balance Optimization (TEBOpt) method, a training-free method to optimize text embeddings, resulting in a 90.05% improvement in information balance. Additionally, it presents a new metric for more accurate evaluation of information loss in generated images.
strength_weakness: strength_weakness: a. The paper has limited qualitative results and mentions that the analysis was conducted on a total of 400 prompts. Providing more qualitative results in the main text or appendix could better demonstrate the validity of the proposed method.
b. Some of the formulas used in the paper are not thoroughly explained, making it difficult to understand the proposed method.
c. The paper proposes a novel perspective to analyze object omissions and fusion issues in the T2I model. However, the method in the paper is rather straightforward, which to some extent may affect the quality of the generated images.
questions: questions: a. According to the description in the text, Equation 6 is intended to encourage later tokens to be as dissimilar as possible from earlier tokens’ embedding. However, in the formula, the similarity is calculated between the object token and all tokens, with the summation range being 0 to n. This seems to contradict the goal of constraining the subsequent tokens to be dissimilar from earlier tokens?
b. In the qualitative results shown in the second row of Figure 4, it can be observed that the results using the TEBOpt method do not seem to retain the visual features of the cat very well. Does this indicate that optimizing and adjusting text embeddings is likely to cause some uncontrollable changes in semantics? Additionally, when combined with other methods, it also seems to affect the overall layout of the generated images.
limitations: limitations: The paper summarizes the limitations of the proposed method in sec6, while mentioning that the case of multiple objects as well as complex prompts can be further explored. In addition, the experiments in this paper are based on simple sentences, which can be followed up to explore whether the case with complex adjectives affects the generation effects.
paper_summary: paper_summary: This papers address the bia problem of text-to-image generation. Unlike the previous work which focus on processing of the attention map, this work explores the text embedding, which is a key factor.  Authors conduct expensive experiments to analysis the text embedding, then propose a interesting method to address the alignment  of the text-to-image generation.  A simple text embedding alignment loss is proposed by introducing other pure  text information.
strength_weakness: strength_weakness: 1. The present problem (e.g., Fig1) seems just happy the lower version of SD1. I test SD3 with the prompt : 'a lion and an elephant', the issues does not happen. In the paper, authors show the reason is from the text encoder, which I  agree. However,  the presented reason maybe does not lead to the shown challenges. 

2. For text embedding, the EOT also play a key role[1]. This paper does not consider it. Only considering the soft replacing maybe harder.

3. The visual results are not good. The generated images has poor quality. Does the propose lead to other negative influence?

4. The baselines are  a little out of  fashion.  Authors should consider more recent methods: SDXL, SD3,  PlayG-v2, Ranni[2] and Ella[3].

5. Authors should use a public benchmarks to compare the proposed method, like T2I-CompBench[4].

6. The proposed idea seems limited for NeurIPS, which is not enough.

[1] Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models
[2] Ranni: Taming text-to-image diffusion for accurate instruction following.
[3] Ella: Equip diffusionmodels with llm for enhanced semantic alignment.
[4] T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation
questions: questions: Please check the weaknesses.
limitations: limitations: Testing the presented problem in latest version of SD, like SD3.

Comments: title: improving my score
Official comments by  Reviewer fTPW
comment: Thanks Ac and authors for your effort. I have improved my score. For SD3, I reviewed a few papers for neurIPS2024  also consider SD3 for the base model instead ignoring it like this papers.
I greatly appreciate the thorough rebuttal. All of my concerns are clearly resolved except below.

In W7, I think it is marginal because there are only 0.5%p and 1%p improvement from 14.5% and 34%, respectively.

I raised my score from 4 to 6. I anticipate an upgraded version of this paper in the camera ready.

BTW, I value clear definition of everything. I think the word "causal" for attention mechanism is being abused considering the causal inference.
Dear Reviewer,

We greatly appreciate your recognition of our work and your valuable feedback. Thank you for helping us make our formulas clearer!
Thank the authors for their detailed rebuttal. I keep my original rating for the acceptance of this paper.
Dear AC,

**PlayG-v2** has the same architecture as SDXL (https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic). Both of them contain 2 text encoders: CLIP and OpenCLIP. Therefore, we compare SDXL-Turbo as the representative for comparison. Due to its inference efficiency, we choose SDXL-Turbo over SDXL, as SDXL requires more than 10 seconds per image on a single 3090 GPU, while SDXL-Turbo only requires less than 1 second per image.

**Ranni** has the same text encoder as SD 1.4 and SD 1.5, which all utilize CLIP; therefore, we use SD 1.5 as a representative. Moreover, Ranni incorporates additional features such as bounding boxes, learnable color embeddings, and keypoints as conditions, focusing primarily on image editing. This differs significantly from our main focus, which is to investigate the contributions of text embedding and causal manners in text encoders to text-to-image models. Thus, comparing our model with Ranni is inappropriate because it does not address the central questions of our research.

Overall, text-to-image models are experiencing explosive growth, with dozens of new models being released every day. We focus on comparing the most relevant models that align with our primary goal—exploring causal mechanisms and text embeddings in T2I models—rather than every available T2I model with diverse input conditions, as our main target is not image editing.
Dear authors, do you have any comments on PlayG-v2 and Ranni? Why do you exclude them?
Dear Reviewer fTPW, 

After reading the authors' feedback, do you still hold your thoughts on "both the baseline and benchmark are not convincing"? What baseline and benchmark would resolve your concerns? Do you think the missing results on PlayG-v2 and Ranni were significant? Some elaborations and constructive feedback would help the authors to refute and improve their work.
Dear reviewer,

**old version SD:** The SD3 you suggest is released at 12 Jun., which is later than the date our task submit to NeurIPS. In the rebuttal, we still provide the experiments to prove our effectiveness that our method outperforms SD3. Please refer to the Fig. 1 and Table 1 in the global rebuttal PDF.

**latest baselines:** In the initial submission, we compared with latest tasks addressing object missing, e.g., A&E (SIGGRAPH’23) [1], SynGen (NeurIPS’23) [13]. To respond to your suggestion, we further compare to Stable Diffusion XL-Turbo, Ella, and SD3. All the qualitative and quantitative results prove our effectiveness. Please refer to Fig. 1 and Table 1 in the global rebuttal PDF.

**Benchmark:** the public benchmark, e.g., T2I-CompBench, contains more diverse prompts, e.g., adjective binding with noun, that would make the experiment distract. Thus, we follow related works, e.g., A&E (SIGGRAPH’23) [1] and Predicated diffusion (CVPR’24) [16], to create the most suitable benchmark for our task. While we respectfully disagree with the directness to conduct public T2I-CompBench to demonstrate our performance, we still conduct quantitative and qualitative comparisons in T2I-CompBench in the rebuttal and prove our effectiveness. Please refer to Fig. 1 and Table 1 in the global rebuttal PDF.

**We sincerely look forward to more concrete details about which part of the rebuttal does not address your concern.**


[1] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH, 2023

[13] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. In NeurIPS, 2023.

[16] Kota Sueyoshi and Takashi Matsubara. Predicated diffusion: Predicate logic-based attention guidance for text-to-image diffusion models. In CVPR, 2024.
Thanks authors for rebuttal. Based on the mentioned problems (e.g., old version SD, latest baselines, and public benchmark). I think this paper is not ready for NeurIPS.  For one submitted paper, since both the baseline and benchmark are not convincing,  I can not be positive.
Dear our reviewers,

Now, we are approaching the very end of the reviewer-author discussion period. Are there any further questions for the authors regarding major issues, or are your concerns sufficiently resolved? Only a limited amount of time is available for interaction with the authors.

Thank you again for your dedication.

Your AC
Dear Reviewer,

We appreciate your valuable suggestions and the positive feedback! We are eager to share further information about our task with you in the submitted rebuttal. As the author-reviewer discussion will end soon, we are looking forward to your further consideration of our submitted rebuttal. Thank you again for taking the time to foster a thorough review of our task!

Sincerely,
authors
Dear Reviewer,

We appreciate your valuable suggestions and we are eager to address your concerns with the submitted rebuttal. As the author-reviewer discussion will end soon, we are looking forward to your further consideration of our submitted rebuttal. Thank you again for taking the time to foster a thorough review of our task!

Sincerely,
authors
Dear Reviewer,

We appreciate your valuable suggestions and we are eager to address your concerns with the submitted rebuttal. As the author-reviewer discussion will end soon, we are looking forward to your further consideration of our submitted rebuttal. Thank you again for taking the time to foster a thorough review of our task!

Sincerely,
authors
Dear Reviewer,

We appreciate your recognition of our work as a valuable starting point! Thank you also for your thorough and thoughtful comments. We will incorporate your valuable suggestions into an updated version and continue contributing to the community.

Thank you again for your support!
I appreciate the authors' efforts in addressing my concerns. After reviewing the authors' rebuttal and the comments from other reviewers, I maintain my initial stance that this paper serves as a valuable starting point in this direction and offers insights into the phenomenon through error analysis and initial results. Therefore, I will keep my original rating as 'accept.'

However, there is still much room for improvement. As noted by all the reviewers, the current version examines the problem in a relatively simple setting, and there are opportunities to further explore the problem setting, model choices, and evaluation benchmarks more systematically (though some have been provided in the rebuttal). I hope the authors can continue refining their work to better benefit the community.
Dear Reviewers,

Our eagerness for your further consideration has led us to gently remind you of the review of our submitted rebuttal. We understand that your time is valuable, and we appreciate your willingness to read our rebuttal to foster thorough consideration. We sincerely look forward to your further feedback.



Sincerely,

The authors