Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!
Authors: Arash Mari Oriyad
Mohammadali Banayeeanzade
Reza Abbasi
Mohammad Hossein Rohban
Mahdieh Soleymani Baghshah
Venue: Accepted by TMLR ()

Abstract: Text-to-image diffusion models such as Stable Diffusion and DALL-E have exhibited impressive capabilities in producing high-quality, diverse, and realistic images based on textual prompts. Nevertheless, a common issue arises where these models encounter difficulties in faithfully generating every entity specified in the prompt, leading to a recognized challenge known as entity missing in visual compositional generation. While previous studies indicated that actively adjusting cross-attention maps during inference could potentially resolve the issue, there has been a lack of systematic investigation into the specific objective function required for this task. In this work, we thoroughly investigate three potential causes of entity missing from the perspective of cross-attention maps: insufficient attention intensity, excessive attention spread, and significant overlap between attention maps of different entities.
Through comprehensive empirical analysis, we found that optimizing metrics that quantify the overlap between attention maps of entities is highly effective at mitigating entity missing. We hypothesize that during the denoising process, entity-related tokens engage in a form of competition for attention toward specific regions through the cross-attention mechanism. This competition may result in the attention of a spatial location being divided among multiple tokens, leading to difficulties in accurately generating the entities associated with those tokens. Building on this insight, we propose four overlap-based loss functions that can be used to implicitly manipulate the latent embeddings of the diffusion model during inference: Intersection over union (IoU), center-of-mass (CoM) distance, Kullback–Leibler (KL) divergence, and clustering compactness (CC). Extensive experiments on a diverse set of prompts demonstrate that our proposed training-free methods substantially outperform previous approaches on a range of compositional alignment metrics, including visual question-answering, captioning score, CLIP similarity, and human evaluation. Notably, our method outperforms the best baseline by $9\%$ in human evaluation.

Published in Transactions on Machine Learning Research (03/2025)
Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!
Arash Marioriyad
ansimmarioriyo@@@aall.om
Deportment of Computer Engineering Sharif University of Technology
Mohammadali Banayccanzade
a. b.banaayeann@@mmii.ccmmm
Department of Computer Engineering
Sharif University of Technology
Reza Abbasi
reza.abbasi, www.Bmaalllmcmmmmmmmmmmmmmmmoooo
Department of Computer Engincering
Sharif University of Technology
Mohammad Hossein Rohban
rohban@sharif.co edu
Deportment of Computer Engineering Sharif University of Technology
Mahdich Soleymani Baghshah
soleymami@sharrifeeeie
Department of Computer Engineering Shorif University of Technology
Reviewed OpenReview: /ttps://openreviiee net/ forum? id= id=Xo32rFay.1
Abstract
Text-to-imas diffusion models such as Stable Diffusion and DALL-E have exhibited impres- sive capabilities producing high-quality, diverse, and realistic images based on textual prompts. Nevertheless, a common issue arises where these models encounter difficulties in faithfully generating every entity specified in the prompt, leading to recognized challenge known entity missing in visual compositional generation. While previous studies indicated that actively adjusting cross-attention maps during inference could potentially resolve the issue, there has been a lack of systematic investigation into the specific objective function required for this task. In this work, we thoroughly investigate three potential causes of entity missing from the perspective cross-attention maps: insulficient attention intensily, excessive attention spread, and significant overlap between attention maps of different entities. Through comprehensive empirical analysis, we found that optimizing metrics that quantify the overlap between attention maps of entities is highly effective at mitigating entity missing, We hypothesize that during the denoising process, entity-related tokens engage in form of competition for attention toward specific regions through the cross-attention mechanism. This competition may result the attention of a spatial location being divided among multiple tokens, leading to difficulties in accurately generating the entities associated with those tokens. Building on this insight, we propose four pverla--based loss functions that can be used implicitly manipulate the latent embeddings of the diffusion model during inference: Intersection over union (IoU), center-of-mase (CoM) distance, Kullback-Leible (KT.) divergence, and clustering compactness (CC). Extensive experiments on diverse set of prompts demonstrate that our proposed training- free methods substantially outperform previous approaches on a range compositional alignment metrics, including visual question- answering, captioning score, CLIP similarity, and human evaluation. Notably, our method outperforms the best bascline by 9% in human evaluation.

---

Published in Transactions on Machine Learning Research (03/2025)
1
Introduction
"a banama and chair and an orange"
"a keyboard and pizza and a plate"
"a cup and donut and an orange"
"a clock and frisbee and skateboard"
"a book and umbrella"
"a bowl and window"
"a sandwich and a sheep"
Stable Diffusion
Attend & excite
CoM Distance
(Ours)
Figure 1: Comparison of compositional generation capabilities between Stable Diffusion, Attend-and-Exee and one of our proposed overlap-based methods (CoM Distance) for textual prompts containing two and three entities while using SD-1.4 as the backbone: Stable Diffusion (first row) and Attend-and-Ease Excite (second row) often fail to generate all the specified entities in the input prompt, a problem known as entity missing. Our training- free approach (third row) addresses this issue by employing an overlap- based objective function (CoM Distance) cross-attention maps during the denoising steps, resulting in more faithful generation of all the entities mentioned in the input prompt.
In recent years, text-to-image (T2I) diffusion-based (Ho al. 2020) models such Stable Diffusion
(Rombach et al. 2022 Podell el al. 2023 and DALL-E [Ramesh el al.) 2021 2022 Betker et al. 2023 have shown promising performance in generating realistic, creative, diverse, and high-quality images from textual descriptions. These models leverage the itcrative denoising process, along with text embeddings through the cross-attention mechanism, to generate images used many real-world applications across various domains,
such as controllable image synthesis (Ho Salimans 2021 Epstein et al. 2023 Zheng et al. 2023 image
editing (Kawar al. 2023 Zhang et al. (2023), 3D scene production (Poole et al. 2023 Lin et al. 2023
Richardson et al. 2023 video generation (Ho et al. 2022a b).
However, despite their impressive capabilities, these models often struggle to faithfully capture all the entities, attributes, and relationships described in the input prompt, leading to various compositional misalignments
such as entity missing (Chefer et al. 2023 Sucyoshi & Matsubara 2024 Zhang et al. 2024 Liu et al. 2022
Kim al. 2023), incorrect attribute binding (Feng al.} 2023 Li al. 2023 Rassin et al. 2023
Wang
et al.) 2024). incorrect spatial relationships (Chatterjoe et al. 2024
Gokhale et al. 2022 Chen et al.
2024).
and numeercy--rllaeed problems (Binyamin al., 2024 Zafar al. 2024).
Entity missing, in particular, is prevalent issue where the model fails to faithfully generate
more
entitics specified in the textual prompt, significantly hurting the compositional generation performance and limiting the practical usability of T21 models. Previous works have explored both training-bused (Ruiz el al.
2022; Wen et al.] 2023 Kumari et al.] 2022 Zarei et al.) 2024) and inference- based Chefer et al. (2023 ):
Sucyoshi & Matsubar bara 2024): Zhang et al. (2024); Liu al. (2022); Kim et al. (2023): Karthik ct al. (2023) ;
Eyring al. (2024) approaches mitigate this problem. While fine-tuning the model on carefully curated
datasets improve compositional alignment, is computationally expensive, may not generalize well to
2

---

Published in Transactions on Machine Learning Research (03/2025)
unseen concepts, and can potentially introduce undesirable properties compared to the primal training phase. On the other hand, inference-time approaches that manipulate the intermediate latent code by optimizing an objective function over cross-attention maps between the input tokens and the image regions have shown promise in addressing entity missing without requiring model retraining. However, there has been a lack of systematic investigation inlo the optimization objectives that are most effective for this task.
In this work, we thoroughly examine three potential sources of the entity missing problem from the perspective of coss-attention maps between input tokens and image regions: (1) insufficient attention intensity, (2) excessive attention spread, and (3) significant overlap between attention maps of different entities. We discovered that while the two former phenomena can contribute to the entity missing problem, the latter is the root cause. More precisely, through extensive empirical analysis, we demonstrate that overlap-based metrics that quantify the pixel-level overlap between attention maps of different entities are highly predictive of entity missing. Building on this insight, we propose four overlap-based objective functions that can optimize the latent code of the diffusion model during inference: Intersection over union (IoU), center-of renter-of-mass of-mass (CoM) distance, Kullback- Leihler (KI. diveryence, and clustering compuclness (CC). By minimizing these objectives, our method effectively reduces the overlap between attention maps of entities and improves the model's ability to generate all the entitics specified in the textual prompt faithfully.
We conducted a series of experiments using a synthetic set of prompts encompassing all combinations of object categories derived from the COCO dataset (Lin et al. 2014), which we refer to as COCO-Comp. For evaluation, we employed both human judgments and automatic metrics, including visual question answering (VQA) score (Hu et al.) 2023 Li et al. 2022), captioning score (Li et al., 2022), and CLIP score Hessel et al. (021). As shown in Figure 1 our results demonstrate that the proposed overlap-based optimization method significantly outperforms previous inference-tine approaches, achieving state-of-the-art performance in compositional aligmment. Notably, we found that optimizing the CoM distance between attention maps produced the best outcomes, increasing the success rate of entity generation by about 7% in scenarios with two entities and by more than 17% in scenarios with three entities, compared to existing statc-of-tho-art methods.
To validate the robustness of our werlap-based metrics, we tested our approach using diverse set of model backbones containing SD-1.4, SD-2 (Rombach al. 2022 SD-XL (Podell et al. 2023) and well-sstablished compositional generation benchmarkk, such as et al. 2023), in addition to the freestyle captions of COCO dataset, all demonstrating the superiority of our overlap-based approach over baselines. Furthermore, manipulating the latent code during the denoising process can lead to undesirable effects, such as reduced image quality and unnatural appearance. To address this issue, we assessed our methods using the Frechet Inception Distance (FTD) score (Heusel et al. 2017), Coverage, and Density (Naeem et al. 2020), all demonstrating only slight degradation in quality compared to the best-performing baseline. Finally, we evaluated the effectiveness of the proposed overlap-based methods in addressing other challenges in visual compositional generation, including incorrect attribute binding, incorrect spatial relationship, and mumeracy-related issues. The results demonstrated that solely improving entity preservation can effectively contribute to mitigating these compositional generation challenges.
(Huang et al. 2023 and Rench (Bakr
Hence, our primary contributions include a systematic investigation into three potential causes of entity missing in T2I diffusion models, identifying the overlap between attention maps of entitics as the most critical factor. Building on the insights gained from this analysis, we introduce four novel overlap-based objective functions for inference-time optimization of latent codes during the denoising process. Lastly, comprehensive experiments across diverse model backbones and benchmarks demonstrate that our proposed methods significantly surpass previous approaches in various compositional alignment metrics.
2
Related Works
2.1 Visual Compositional Generation
Compositionality, the study of how basic components combine to form entirely new concepts (Peters et al., 2017), is a fundamental characteristic of the natural world and a crucial aspect of human cognition, allowing individuals to reason and adapt to novel situations (Phillips & Wilson 2010 Frankland & Greene 2020
3

---

Published in Transactions on Machine Learning Research (03/2025)
2012 Building on this concept, compositional generation has emerged as significant area of research in recent years, particularly in developing T21 models. Numerous studies have focused on advancing and evaluating T21 models to improve and assess their ability to handle compositional generation (Huang ct al. 2023 Bakr et al., 2023 Li et al. 2024p. These studics indicate that despite the impressive capabilitics of T21 models in generating realistic and high-quality images, they sometimes fail to produce outputs that fully align with the provided textual input. This misalignment is particularly cvident when the model encounters complex or novel compositions poorly represented in the training data, leading to compositional generation failures such as entity missing, incorrect attribute binding, incorrect spatial relationships, and mumeracy-relate issues.
Reverberi et al.
Entity missing occurs when at least one entity specified in the textual prompt is not completely represented in the generated image (Chefer et al. 2023 Sueyoshi & Matsubara, 2024 Zhang et al., 2024 Liu et al. 2022 Kim et al. 2023). For example, considering the prompt "A cat sitting on a chair next to a dog", the generated image may only contain the chair, with the cat either missing entirely or only partially visible. This problem lypically arises when the model does not allocate sufficient allention to certain entities during the generation process, leading to their partial total absence in the final output (Chefer et al. 2023 On
the other hand, incorrect attribute binding refers to the mis-association of an attribute, such as color, texture, shape, and size, with the corresponding entity (Feng et al. 2023 Li et al.) 2023: Rassin et al. 2023 Wang et al. 2024 Wu et al. 2023: Park et al. 2024). For example, in response to the prompt "A green apple and a red ball" the T21 model might generate an image where the apple is red and the ball is green, incorrectly swapping the attributes. This problem often stems from improper alignment in the attention maps of the entity and its corresponding attribute during the generation process.
Moreover, incorrect spatial relationships occur when the relative positioning of entities in the generated image does not correspond to the spatial arrangement specified in the imput text (Chatterjee et al. 2024 Gokhale et al. 2022 Chen et al., 2024 Feng et al., 2024 Yang et al.) 2022). For example, if the prompt states, "A person standing behind a car," the model might generate an image with the person in front of the car instead. This problem often arises from the model's 's limited capacity to interpret and apply geometric relationships such as depth, perspective, or relative positioning. Finally, numeracy- when the number of instances of a specific entity in the generated image does not match the count specified in the prompt (Binyamin et al., 2024 Zafar et al. 2024 Kang et al. 2023). For example, if the prompt is 'Three dogs playing in a park," the model might generate an image with only two dogs or more than three, failing to meet the specified number. This issue often results from the model's poor ability to reason about numbers and generalize beyond the examples it encountered during the training phase.
problems occur
2.2 Training- Free Methods for Compositional Generation
While several fine-tuning-based approaches (Sun et al. 2023 Jiang et al. 2024 Zarei et al. 2024 Guo el al.j 2024
Wen et al. 2023) have been proposed to mitigate failure modes visual compositional generation, these methods are often resource-intensive, requiring the careful curation of appropriate datasets. Additionally, they may introduce new challenges, such as overfitting to the fine-tuning data and the potential loss of beneficial properties inherent in the T2T model from its inl training phase.
On the other hand, training-free methods provide a promising, on-the-fly solution for addressing compositional generation failures, such as missing entities, in T2I diffusion models. These approaches climinate the need for costly fine-tuning while avoiding its potential drawbacks while preserving the generation capability of the original model. Some of these methods leverage the multiple generation paradigm (Karthik et al., 2023 Liu eet al., 2024), allowing the T2I model to produce several parallel latent codes during the denoising process. The optimal result is then selected based on an evaluation metric applied either to the final image to intermediate latent embeddings. However, generating multiple instances significantly increases inference time and makes these methods resource-intensive.
A separate line of research focuses on enhancing compositional alignment between the input text and the generated image by manipulating intermediate latent embeddings during the denoising process. This approach, known as generative semantic nursing (GSN) (Chefer et al. (003)) involves defining a loss function over the cross-attention maps between input tokens and image regions, as these maps encode valuable spatial
4

---

Published in Transactions on Machine Learning Research (03/2025)
information about entities and attributes. By updating the latent embeddings in the opposite direction of the loss function's 8 gradient, compositional generation is expected to improve.
For example, the Attend-and- Exticte method (Chefer et al. 2023) employs an intensity-based loss function designed to prevent the weakening of the entity with the lowest maximum attention score, while Divide-and Bind (Li et al., 2023) uses a Kullback-Leibler (KT) divergence loss function to ensure the proper association of attributes with corresponding entities. Additionally, Predicated Diffusion (Sueyoshi & Matsubara 2024 interprets the intended meaning of the imput prompt propositions based on predicate logic, leading to differentiable loss function that guides the image generation process to better satisfy the propositions. Lastly, Attention Regulation (Zhang et al. 2024) formulates a constrained optimization problem to minimize attention mismatches, aligning attention maps more closely with the input text prompt.
On the other hand, training-free approaches such as Structured Diffusion Feng et al. 2023 incorporate linguistic structures obtained from constituency trees directly into the cross-attention layers. More prociscly, they extract noun phrases from the input text prompt, feed them separately to the CLIP text-encoder to obtain corresponding embedding, and augment the value matrix of the cross-attention mechanism with the obtained embeddings to promote the role of noun phrases.
Notably, number of studies have taken the complexity of imput prompts into account by adjusting inference time according to the intricacy of the input. For example, Composable Difflusion (Liu et al. 2022 introduces a method that first divides the input text prompt into distinct concepts, using the "and" token as a delimiter. During each denoising step, multiple instances of the denoiser independently generate latent embeddings corresponding to each concept. These latent embeddings are then combined to form unified latent code, ultimately producing an image that integrates all specified concepts.
3
Preliminaries
3.1
Latent Diffusion Model
Latent diffusion models (LDMs) (Rombach et al. 2022 are a class of generative models designed to map a latent representation of an image to the corresponding nigh-dimensional pixel space. In contrast to traditional diffusion models that operate directly in pixel space, LDMs first encode an image x into a lower-dimensional latent space using an encoder E, and then apply the diffusion process on the latent embedding = = E(x). To reconstruct the image from the denoised latent representation, a decoder D maps the denoised latent back pixel space, generating the final image a.
The diffusion process in LDMs consists of a forward (diffusion) process and reverse (denoising) process. In the forward process, Gaussian noise (e) is gradually added to the latent embedding (2) over a fixed number of time steps, transforming the original latent 20 into a noisy latent 24 at timestep t. This forward process can be described as equation
24 = Varzo + VI- oie, e~ N(0,1
(1)
where cg = I on, and ag are predefined noise schedule parameters that control the amount of noise added at each time step.
In the reverse process, the noisy latent 2, is gradually denoised using a learnable denoising network co parameterized by U-Net architecture (Ronneberger et al. 2015). More precisely, the denoising process starts from latent representation sampled from standard Gaussian distribution, ie., 2r ~ -N(0,)) T), and iteratively refines it using equation 2
1 ( 1 ai Va,
2t-1
co(2,, t)) + toe, e~~
(2)
/1 dz
where or is the standard deviation of the Gaussian noise added at timestep t, and co t) is the output of the denoiser netwrok.
5

---

Published in Transactions on Machine Learning Research (03/2025)
The denoiser is trained using a simple loss function that minimizes the L2 distance between the predicted noise 6g and the ground truth noise as described in equation 3
C E [ eØ( ce(Varb(zo) + ai, t)1l
where To is the original image, is a timestep sampled uniformly from (1....,7), and 6~ (O,,))
The use of a lower-dimensional latent space in LDMs offers several advantages over pixel-space diffusion models. First, allows for more efficient training and inference, as the diffusion process operates on a more compact representation. Second, the latent space provides a semantically meaningfil representation of the image, which can lead to improved sample quality and controllability (Rombach et al. 2022
3.2 Text-to-Image Diffusion Models
T2T diffiusion models such as Stable Diffusion (Rombach et al. 2022 extend the capabilities of latent diffussin models by conditioning the generation process on textual descriptions. This allows for creating images that accurately reffect the semantic content provided in the input text. The conditioning is typically achieved through the use cross-attention layers within the denoising U-Net architecture.
More precisely, given an input text prompt, the first step in T21 diffusion model is obtain the text embedding P E Dp using pre-trained text encoder, such as CLIP (Radford et al. 2021); ) where N is the maximum context length and Dp is the dimensionality of the text embedding space. Then, at each denoising step i, the cross-attention mechanism takes the text embedding P as the key and value matrices and the noisy latent represenlation 2 as the query matrix to produce cross-allention maps according to equation
(Woz:"W Wv.
At = Softmax
(4)
ddl
where A, HXxN is the resulting cross-attention map at time step t. Moreover, Wq. WK. and Wy are Icarnable weight matrices for the query, key, and value, respectively, and d, is scaling factor typically set to the square root of the key dimensionality.
Notably, j.] represents the attention score between the latent representation at spatial location (i,j) and the n-th text token. These attention scores indicate the relevance of each text token to different regions of the image being generated. Through this work, we denote the attention map corresponding the entity at time step AG E HxW
4 When and Why Does Entity Missing Happen?
The entity missing problem, where one more entities mentioned in the imput text prompt are not faithfully generated in the output image, is significant challenge in T2T diffusion models. Understanding the potential causes of this problem is crucial to developing effective solutions to compositional generation issues. In this section, we analyze the entity missing problem from the perspective of cross-attention maps, which capture the relationships between image regions and text tokens.
We hypothesize that the entity missing problem may stem from three key factors related to cross-attention maps: (1) attention intensity, (2) attention spread, and (3) attention overlap. The first two factors focus the evolution of the attention scores of each entity separately during denoising steps, while the third factor investigates how two or more entities affect and interact with each other. To examine these factors, we introduce several metrics that quantify different aspects of cross-allention maps, such as atlention intensity (Int), attention spread (Var), intersection over union (IoU), center-of-mass (CoM) distance (DCam), Lulbback-Loblle (KI.) divergence (Dкz); and clustering compactness (CC). We then conduct. several experiments on diverse set of prompts and analyze the correlation of each metric with the success rate of the T2T model to generate entities mentioned in the prompts.

---

Published in Transactions on Machine Learning Research (03/2025)
4.1 Attention Intensity of Entity
Attention intensity refers to the average strength of the attention scores assigned to specific entity token across different spatial locations in the cross-attention map. We hypothesize that insufficient. attention intensity could lead to the entity missing problem, as the model may fail to allocate enough attention to the corresponding entity during the image generation process. To quantify attention intensity, we propose the Int metric, which measures the average attention score an entity e across all spatial locations at a given time step t, according to equation
Inl(AA))
Σ AfC
(5)
H x W
where A the attention map of entity at time step 6, and H and W the height and width of the atlention map, respectively.
Figure illustrates how attention intensity evolves over time for the entity "bird" in two different scenarios. In the successful case, the attention intensity of the entity "bird" starts from a higher value and remains relatively high throughout the generation process compared with the failure case, enabling the model to faithfully generate the entity "bird' in the output image. This insightful example highlights the role of attention intensity, espocially in the initial steps of the denoising process, in avoiding entity missing problems.
Prompt: "a cup and bird"
Attention Intensity of 'bird"
(seed - 41)
2,00
WW
Failure (seed - 12)
5004
Image 5002
Three attention maps of entity "bird" are drawn at different time steps (0, 25, and 50) along with the generated image for both success and failure cases. Right: While in both success and failure cases, the attention intensity (Equation 5 of entity "bird" decreased over time, the bad initialization of zy with seed 12 resulted in the vanishing of the attention scores of "bird" 50.
Figure 2: Left:
4.2 Attention Spread of An Entity
In addition to attention intensity, the spread of attention scores related to entity across different spatial locations in the gross-attention map also contribute to the entity missing problem. We hypothesize that excessive attention spread could lead to the entity missing problem, the model may struggle to generate a coherent representation of the entity in the output image. To quantify the attention spread of an entity, we introduce the Var metric, which measures the variance of attention scores around the center of mass (CoM) of its cross-ateention map as described equation
Var(P.) =
- 2 Ji [cM((PF)
(6)

---

Published in Transactions on Machine Learning Research (03/2025)
where Pf is the normalized attention map of entity at time in such way that for each spatial location (i.j), Pf = Ex,. AgGis]] A[(k,a] and CoM (PT) = Pf(ia]] represents the center of mass of the normalized
attention map, calculated as the weighted average of spatial coordinates using the normalized attention scores as weights.
As illustrated in Figure the failure exhibits significantly higher attention spread in terms of the Var metric compared to the success case, meaning the attention scores are dispersed across wider area around the CoM. We believe that the excessive spread hinders the model's ability to coherently represent the entity "snowboard", leading to the entity missing problem.
Prompt: horse and a snowboard"
"snowbeard"
40-
Success (seed = 5)
-4-
Failure (seed 41)
t=
==
Image
Figure 3: Left: Three attention maps of entity "snowboard" are drawn at different time steps (0, 25, and 50) along with the generated image for both success and failure cases. Right: During most of the time steps, the attention spread (Equation ( of entity "snowboard" in the failure case is much higher than in the success case, resulting in the entity missing problem.
4.3 Attention Overlap Between Entities
From the viewpoint of the cross-altention mechanism, each entity token of the input prompt attends to spatial locations during denoising steps to form the corresponding entity in the final image. This process induces an implicit competition between the entities to assign spatial locations to themselves. However, because of the unavoidable randomnoss the latent initialization and the drawbacks the CLIP text-encoder representing complex prompts, some entities may not be able properly allend enough adjacent spatial locations form the entity through the competition, resulting in the entity missing problem.
As illustrated in Figure in the successful case, the attention overlap between the entities "cat" and "vase" remains relatively low throughout the demoising process, which allows the T2T model to generate coherent representation of both entities in the output image. In contrast, the failure case exhibits a significantly higher attention overlap, with the attention maps of the two entities competing for the same spatial regions. This competition leads to one entity (in this case, the "cat") being underrepresented or missing in the generated image.
One way to study this competition is by measuring how much the attention maps of different entities overlap with each other across spatial locations. Hence, propose four werlap-based metrics: intersection over union (IoU), CoM distance, symmetric KL divergence, and clustering compactness (CC), as described in table
The IoU metric explicitly measures the degree overlap between the attention maps of two entities by calculating the ratio of the intersection of their attention scores the union of their attention scores across all spatial locations, where the intersection and union are computed using the product and summation operations, respectively.
8

---

Published in Transactions on Machine Learning Research (03/2025)
Prompt: "a cat and vase"
6:45
Success (seed - 41)
Attention Overlap
0:33
Failure (seed 71)
==
t=
Image
033
Figure Left: For both the success and the failure cases and at three time steps (0, 25, and 50), the attention maps of entities 'cat' and 'vase' depicted together in one image, with the atlention scores of cat and vase being red and blue, respectively. Right: During most of the time steps, the attention overlap measured by IoU metric (Table D between two entities in the failure case is much higher than that in the success case, resulting in the missing entity "cat" in the generated image.
Table Four overlap-bor wrased metrics are formally introduced, where PT and P. are the normalized version of attention maps A and AT" corresponding to two entities e1 and e2. For the sake of simplicity, we defined the metrics over only two entitics. However, the extended version of these metrics is available in the Appendix.
Metric
Formulation
Intersection over Union (loU)
= (PU P E P BJTPPP
CoM Distance
DCoM(PP) , |ICoM(PP)) CoM (CMM(())
Symmetric KL Divergence
p.t [i,) (Pr Ei, Pet [. log
P [i.
log p [.
p. P.
Clustering Compactness (CC)
CC(PP) P.3) = HxW Σi, max P. (i,
Pr
P (0
On the other hand, the CoM distance measures the divergence the focal points of attention maps using Euclidean distance. More precisely, it quantifies the spatial separation between the CoM of the attention maps, indicating how far apart the entities are focused average.
Moreover, inspired by KL divergence in information theory, we propose the symmetric KL divergence between two normalized attention maps, where each normalized attention map is considered a probability distribution of attention scores over spatial locations. More precisely, this metric captures how similarly the entities allocate attention across the spatial locations from a probabilistic perspective.
Finally, clustering compactness (CC) is a measure usually used in unsupervised machine learning assess the tightness of clusters formed by grouping data points based their similarity in latent space. We believe that the cross-attention mechanism during denoising steps is similar to clustering process, where each key vector related to entity is center, and each query vector associated with the latent embedding at spatial location is a data point in the clustering process. Considering this analogy, we propose the CC metric on the attention maps two entities as the intra-cluster similarity where the similarity measured by the inner-product operation (Table Db. Note that the inner product of the cross-attention mechanism between the key vector related to the entity and the query vector associated with the spatial location (i.j) is (] Moreover, for each query vector (data point), the mar operation in the CC metric selects the most.

---

Published in Transactions on Machine Learning Research (03/2025)
similar (nearest) key vector (centers), similar to clustering process. Put them all together, the CC' metric quantifies the degree to which spatial locations are fully assigned to single entity, promoting distinct and well-separated entity representations.
4.4 Attention Overlap Is the Root Cause of Entity Missing
We analyzed the relationship between the model's success rate in faithfully generating entities and each introduced metric by running the Stable Diffusion version v1.4 model on validation set of our proposed dataset COCO-Comp. which contains 190 prompts including 20 entities in the format of "a {entity) } and a fentity) }", where entities are from the COCO dataset and the success rate is measured using a visual question-answering (VQA) approach called TIFA-score Hu et al. (2023). More details about the COCO-Comp dataset are provided in the Appendix [ Notably, when dealing with prompts containing more than one entity, the Int metric is the minimum of Tnt over entities, and the Var metric is the average of Var across all entities.
As illustrated in Figures 5 an unignorable correlation exists between the performance of the T2I model in avoiding entity missing and the proposed metrics, especially the overlap-based ones. We believe that the lower overlap value between two attention maps measured by IoU and DKL induces discrepancy in the attention distribution of entities and leads the entities to focus on distinct regions rather than being conflated, resulting in higher VQA score. Moreover, a higher distance between the DCoM of entities implicitly makes more room for each entity attend to different regions of the image without worrying about overshadowing or masking others. Similarly, higher CC values imply that at each spatial location, (i, one entity completely dominates others in the sense of attention scores, which results in less possibility of the entity missing.
Moreover, while Figure 5 demonstrates that higher attention intensity (Int) and lower attention spread (Var) results in a reduced possibility of entity missing in the generated image, a weaker correlation value between these metrics and the success rate of compositional generation reported compared to the overlap-based metrics. These observations underscore the importance of minimizing overlap between cross-attention maps of entities compared to other metrics, as enables the weaker charraceriead by lower allention scores across overlapped pixels-to to evade dominance by the stronger one.
5
Method
The investigation presented in Section 4.4 aimed to clarify the relationship between various metrics applicd to cross-altention maps and the occurrence of missing entities. The analysis revealed compelling evidence highlighting the importance of overlap- based metrics, such as IoU, DCOM, DKL, and CC, in reliably assessing performance related to mitigating missing entities. Building on insights from prior studies (Chefer et al. 2023 Li et al. 2023 Sucyoshi & Matsubara 2024 Zhang al., 2024) and leveraging the advantages of training- free methods, we propose loss function-basee based approach. This method minimizes a loss function during denoising steps to reduce the overlap between the attention maps of different entities, thereby improving the model's ability to faithfully represent all entities mentioned in the input text prompt.
Specifically, for a prompt containing two entities e₁ and e2, an overlap-based loss function is computed over the normalized attention maps of these entitics (P. and P(a) at cach denoising step t. The loss function L(P"), ,P²)) can take various forms, including (P P(²), -Doos P(t), (Pr P(?), -CC(P"",
Following the generative semantic mursing paradigm introduced in Chefer et al. 2023 in order to minimize the loss function L, we update the latent representation 2, in the direction opposite to the gradient of L with respect to 2, at each denoising step t. This iterative update process is performed using the gradient descent algorithm, as described in equation 7
24 x 24 aV
10

---

Published in Transactions on Machine Learning Research (03/2025)
VQAA
KL Divergnned
16U
0.50
100
0.25
VQL
Codl Dizance
0,7
0.7
-0.7
0.00
CCAM
Clastering
Mia
Vanunce Inaesiit
0.4
-0.4
0.3
VQA.
VQA Score
0..55
11,00
Divergener
08.94
KL Divergence
CC
InU
CoM
Intendly
Virimee
Score
VQJ
Figure 5: Left: The relationships between the proposed metrics and the VQA score: A higher value of CoM distance, KT divergence, and CC, and attention intensity results in higher VQA score, while lower value of ToU and attention spread (Var) improves the possibility of faithfully generating entities. Right: The correlation matrix showing the correlation values between the proposed metrics and the VQA score: The results reveal strong correlations between the overlap-based metrics (IoU, CoM distance, KL divergence, and CC) and the success rate of the model in faithfully generating the entities. In particular, the CoM distance, K divergence, and CC metrics exhibit strong positive correlations, while the loU metric shows a strong negative correlation with the VQA score. Moreover, there strong correlation among each pair of the overlap-based metrics as well.
11

---

Published in Transactions on Machine Learning Research (03/2025)
where denotes the learning rate, and Vz, represents the gradient of the loss function L with respect to the latent embedding
Notably, these overlap-based metrics were selected for their diverse perspectives on the attention overlap issue, thereby providing a comprehensive understanding of the problem. For instance, IoU and DCOM are computed in spatial space, where ToU emphasizes the reduction of common areas between entities, while DCOM focuses on increasing the distance between them. On the other hand, DK probabilistic measure that interprets the attention map as probability density function, quantifying the distance between distributions. Lastly, CC is novel metric that operates in the feature space, derived from the concept of clustering compactness.
Minimizing the overlap between the attention maps of entities has at least two distinct yet interconnected benefits, which can help mitigate the entity missing problem. Firstly, by reducing the attention overlap, the weaker entity, which is recognized by lower attention scores across overlapped spatial locations, can avoid being dominated by the stronger entity, thus lessening the probability of one entity vanishing. Secondly, the reduction in overlap engenders a broader spatial coverage across the image for all entities. With reduced overlap between attention maps, entities have greater freedom to focus adjacent pixels from various regions of the image, minimizing concerns about interference with other entities. The evolution of attention maps during denoising steps for each of the introduced pverlap-based losses can be found in Appendix F
Note that while the described metrics and loss functions can be easily extended to more than two attention maps, we have presented the case where the functions are applied to two attention maps only for the sake of simplicity. More information about the extension of these overlap-basee metrics is available in the Appendix
6 Results
6.1 Experimental Setup
Datasets. We evaluate our approach on four datasets: (1) our set of carefully curated prompts called COCO-Comp featuring to 4 entities in structured format, (2) a subset of 1000 captions from the COCO dataset (Lin et al. 2014) with more complex and varied linguistic structures, (3) the validation section of Tel-CompBench (Huang et al. 2023), and (4) the validation section of HRRS-Bench (Bakr et al. 2023). For COCO-Comp, we generated prompts using 71 entities obtained from the COCO validation set, strictly split into validation and test sets so that there are no overlapping entities between them, which is neglected in many previous works. This resulted in 190 validation prompts used for analysis (Section 4 and 1275 test prompts used for main experiments in the two-entity case. Morcover, we randomly selected 1000 tost prompts for three-entity cases. More details on the prompt generation process and the complete list of entities in the COCO-Comp dataset can be found in Appendix A
Evaluation Metrics For Compositional Alignment. We employed a diverse set of evaluation metrics to comprehensively assess the compositional alignment of generated images with the input prompts (Appendix
B These include:
1. Soft. and harsh human evaluation, where in soft case raters assess each entity in the prompt fully present (1.0), partially present (0.5), or missing (0.0) from the generated image, while in harsh case, the partially present state is considered as missing;
Soft and harsh visual question answering (VQA), which are computed using the BLIP Li et al, (222)) and TIFA Hu al. (2023) models. In the harsh evaluation, a score of 1.0 is awarded only if the VQA model accuratcly identifies all entities; otherwise, score of 0 is assigned. On the other hand, in the soft. VQA evaluation, the score is proportional to the number of entities that are generated in their entirety.
3. CLIP similarity (Hessel et al. 2021). , which considers the cosine similarity between the prompt embedding and generated image embedding in the CLIP embedding space;
12

---

Published in Transactions on Machine Learning Research (03/2025)
4. Captioning score, which measures the similarity between the prompt and the caption generated by the BLIP captioning model (Li et al. 2022) from the final image in the CLIP text embedding space Hessel et al. 2021
Evaluation Metrics For Quality and Naturalness. Additionally, manipulating intermediate latent embeddings during the denoising process can cause the embeddings to fall outside the distribution expected by the U-Net denoiser, resulting in additional challenges such as unnatural artifacts and reduced image quality. To address this issue, we utilized the Fréchet inception distance (FID) score (Heusel et al. 2017) to evaluate the quality of images generated by T2I methods in comparison to real images from the COCO dataset (Lin et al., 2014). More precisely, the FID score measures the distance between the generated images by T2I model and the real images in the feature space of pre-trained Inception v3 network (Szegedy et al. 2016) in such a way that lower FID scores indicate higher quality and naturalness of the generated images. Furthermore, we adopted the Coverage and Density (Naeem et al.) 2020) evaluation metrics which measure the diversity and quality of the generated images compared to the COCO real images (Lin et al.) 2014) as reference, respectively. More preciscly, A higher value of Density indicates better fidelity and quality, showing that the generated images are concent.ated in regions where reference images (COCO images) are densely packed, A higher value of Coverage indicates better diversity, showing that the generated images capture the variability of the reference images (COCO images).
Hyperparameters. We initially set the learning rate to 20 for our overlap-based optimization approach and decreased it linearly. Based on the validation set performance, the number of optimization steps per denoising step is set to 1. Moreover, we applied the gradient descent-based optimization at the first 25 denoising steps, while models are run for 50 denoising steps with guidance scale 7.5 and 512 x 512 image resolution for Stable Diffusion v1.4 and v2, and 1024 x 1024 image resolution for Stable Diffusion XL. The latent noise schedule is set to the default configuration in the Stable Diffusion codebase.
We compared our method to several existing training-free free methods, including saate-of-the-art ones for improving compositional generation in T21 diffusion models, including Attend-andd and-Ercite (Chefer
Baselines.
ct al. 2023). Dividc-and-Biin (Li ct al.) 2023), Structured Diffusion (Feng et al. 2023), Predicated Diffusion
(Sueyoshi & Matsubara, 2024), Allention-Ieen Reg (Zhang et al.) 2024). and Composable Diffusion Liu et al. (002)).
Moreover, we evaluated our method against a well-known fine-tuning-base based approach, Dense Diffusion (Kim ct al., 2023), which incorporates user-defined layouts along with the textual input. It important to note that our proposed overlap- based approach requires no additional input and functions as plug-in module for any diffusion-based backbone. For all baseline methods, we utilized the official implementations provided by the authors, following their recommended hyperparameter settings. Further details about each baseline are
provided in Scction 2.2 and Appendix C.1
6.2 Quantitative Results
COco-Comp Dataset. Tables 2 and 3 present the main quantitative results of our proposed overlap- based methods on the Stable Diffusion v1.4 model compared to several state-of-the-art baselines on the test set of two-entity and three-entity prompts from the COCO-Comp dataset, respectively. Our method significantly outperforms the vanilla Stable Diffusion model and all baselines on every metric, demonstrating the effectiveness of overlap-based optimization for mitigating entity missing issues. Particularly in the two-entity case, using the CoM distance the overlap metric yields the best results, with an absolute improvement of 23.3% in soft human score and 24.1% in soft TIFA score over the Stable Diffusion baseline. Notably, the loss function DCOM surpasses the strongest bascline, Predicated Diffusion, by 9% in soft human score and 3.6% in soft TIFA score. The other overlap-based metrics, IOU, CC, and Dxt, also perform comparably to DCOM: all outperforming other state-of-the-ate methods. Importantly, our approach's 's effectiveness increases with prompt complexity. Moving from two to threo-entity prompts the COCO-Comp dataset, our overlap-based methods' performance advantage over the strongest baseline widens, with the soft TIFA score gap growing by 17.6%, underscoring their ability to handle complex prompts.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Table 2: Quantitative results of our overlap-based methods and the state-of-the-art baselines on the Stable Diffusion v1.4 model the backbone using the two-entity prompts from the COCO-Comp dataset. The CoM Distance method (underlined) significantly outperforms all baselines (best in bold), demonstrating the effectiveness of overlap-based latent optimization for mitigating the entity missing problem. Note that the letters 'S' and 'H' in both Human and TIFA evaluations denote soft and harsh, respectively.
Method
Human (1)
Human Score (H)
TIFA (t)
TIFA Score (H) (t)
Captioning Score
CLIP Similarity
Score (8)
Score (8)
Stable Diffusion
57.5%
54.7%
69.9%
40.0%
0.789
0.297
Attend-and-Exet
69.6%
57.9%
87.6%
76.5%
0.811
0.312
Divide-and- Bind
68.7%
56.5%
86.5%
73.6%
0.815
0.309
Structure Diffusion
61.5%
52.2%
74.3%
49.6%
0.792
0.299
Predicated Diffusion
71.8%
60.3%
90.6%
81.6%
0.823
0.314
Attention Regulation
69.4%
58.3%
88.6%
78.4%
0.818
0.316
Composable Diffusion
47.9%
36.8%
66.2%
37.6%
0.761
0.297
IoU
78.5%
67.6%
92.7%
85.3%
0.824
0.317
CoM Distance
80.8%
70.2%
94.0%
88.2%
0.825
0.318
KL Divergence
77.9%
66.3%
91.7%
83.5%
0.818
0.316
CC
77.8%
66.4%
92.2%
84.6%
0.821
0.315
Table 3: Quantitative results of our overlap-basod methods and the statc-of-tho-art basclines on the Stable Diffusion v1.4 model as the backbone using the three-entity prompts from the COCO-Comp dataset. The CoM Distance method (underlined) significantly outperforms all baselines (best in bold), demonstrating the effectiveness of overlap-based latent optimization for mitigating the entity missing problem. Note that the letters 'S' and 'H' in both TIFA and BLIP evaluations denote soft and harsh, respectivcly.
Method
TIFA Score (s)(1)
TIFA Score (H) (t)
BLIP-VQA (t)
BLIP-
Score (H) (t)
Captioning()) Score
CLIP Similarity
(1)
Score (S)
Stable Diffusion
62.6%
15.2%
65.6%
21.6%
0.763
0.305
Attend-and- Ennd-xxiite
80.7%
52.9%
88.3%
69.1% 1
0.780
0.321
Divide-and- and-Bind
82.6%
56.0%
88.1%
68.8%
0.790
0.323
Structure Diffusion
63.1%
16.6%
67.6%
23.5%
0.757
0.306
Predicated Diffusion
82.6%
53.1%
87.2%
64.2%
0.800
0.327
Composable Diffusion
46.2%
5.7%
60.4%
18.1%
0.710
0.288
IoU
88.3%
67.4%
92.8%
79.3%
0.799
0.331
CoM Distance
89.9%
73.6%
93.3%
83.3%
0.790
0.329
KL Divergence
86.7%
63.4%
91.1%
75.2%
0.799
0.330
CC
87.6%
65.6%
93.4%
81.2%
0.792
0.328
HRS-Bench and T2I-CompBench Datasets. Additionally, a quantitative comparison between our overlap-based methods and the state-of-the-art training-free approach, Attend-and-Fxoite (Chefer al. 2023) is presented in Tables 5 and ! using the color-related prompts of well-known and challenging compositional generation benchmarks HRS-Bench (Bakr et al. 2023 and T21-CompBench (Huang et al. 2023 The results clearly highlight the superiority of the proposed overlap-based methods across all evaluation metrics. Notably, the methods improved the harsh TIFA score by 28.6% and 3.3% over the FIRS-Bendd and the T21-CompBench, respectively. Notably, both our proposed dataset, COCO-Comp. and the HRS- Bench include a mix of common (e.g., "cat and dog") and rare (e.g., "giraffe and pizza") entity combinations, whereas the T21-CompBench primarily focuses on more common pairings. This highlights that our proposed method not only achieves improvements in natural compositions but also demonstrates greater enhancement in rare compositions, which unlikely to be encountered during the training phase.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Table 4: Quantitative comparison between our pverlap-based methods and the state-of-the-art training-free method, Attend-and-Exxcite, the HRS--Bench where Stable Diffusion v2 is used the backbone. The overlap-based approaches outperformed the baseline methods across all evaluation metrics.
Method
Score (S) (1)
TIFA
Score (H) (1)
TIFA
BLIP-VQA (1)
BLIP-VQA (t) Score (H)
Captioning (f)
CLIP
(t)
Score (S)
Score
Similarity
Stable Diffusion
58.7%
23.2%
63.8%
28.7%
0.660
0.332
Attend-and- Exxcite
64.5%
31.9%
70.8%
39.5%
0.645
0.335
ToU CoM Distance
81.1%
52.3%
83.7%
57.3%
0.688
0.345
82.0%
60.5%
86.7%
67.1%
0.657
0.340
CC
80.5%
51.7%
82.1%
55.9%
0.685
0.348
Table Quantitative comparison between our overlap-based methods and the state-of-the-arl training-Tree method, Attend-and-Exeee on the T21-CompBench where Stable Diffusion v2 used as the backbone. The overlap-based approaches outperformed the bascline methods across all evaluation metrics.
Method
TIFA Score (S) (1)
Score (H) (t)
TIFA
BLIP-VQA Score (S)
(t)
BLIP-VQA Score (H)
(f)
Captioning Score
(t)
Similarity (t)
CLIP
Stable Diffusion
84.8%
71.3%
86.7%
74.6%
0.767
0.325
Attend-and- Exciie
95.6%
91.3%
95.8%
91.7%
0.769
0.333
IoU CoM Distance CC
96.8%
93.8%
97.7%
95.4%
0.789
0.331
97.3%
94.6%
97.7%
95.4%
0.794
0.334
96.0%
92.1%
97.1%
94.2%
0.788
0.334
Background Entites. While well-known benchmarks for visual compositional generation typically exclude background entities, we extended the scope of evaluation by conducting an experiment incorporating background elements. Specifically, we utilized the CoM distance as the loss function over 96 prompts structured the format: "a fforeground entity) and a (background entity)". Stable Diffusion XL served as the backbone model, the foreground entity was chosen from the COCO dataset, and the background entity was varied across six categories: 'sea', 'beach', 'desert', 'jungle', 'sky', and 'mountain', Table f presents the rosults from human asscssments, revealing over 4% improvement in both harsh and soft evaluation scenarios.
Table 6: Quantitative comparison between the werlap-based method, CoM distance, and the state-of-tae the-thttt--rttt T2I model, Stable Diffusion XL, over prompts containing background entities such as 'sca', 'beach', 'desert', 'jungle,, 'sky', and "mountain' using soft and harsh human judgment.
Method
Score (S) (t)
Human
Score (H)) (t)
Human
Stable Diffusion
91.7%
89.6%
CoM Distance
96.1%
94.8%
Quality and Naturalness. To further assess the quality of the generated images, calculated the FID score for each method using the COCO Captions dataset (Lin et al., 2014). Table 7 presents the FID scores for the proposed overlap-based methods alongside various bascline models on two-cntity and throc-entity COCO prompls. Among the baselines, Structure Diffusion achieves the lowest. FID scores 3.26 for two-entity prompts and 4.06 for three-entity prompts indicating that it generates images with the most realistic overall appearance. However, our overlap-based optimization methods also demonstrate competitive FID scores. Notably, the KL divergence loss achieves the best results among our methods, with scores of 4.99 for two-entity prompts and 5.78 for three-entity prompts. Although these scores slightly higher than those of Structure
15

---

Published in Transactions on Machine Learning Research (03/2025)
Diffusion, they still indicate that our approach produces images of high quality and realism. Additionally, the FID scores for other overlap-based losses (IoU, CoM Distance, and CC) are significantly lower than those of the Stable Diffusion model and most other baselines, further highlighting the effectiveness of our method in preserving the quality of the generated images. Moreover, we adopted Coverage and Density NNccm et al., 2020) metrics to assess the diversity and quality of the generated images, respectively. Table demonstrales that, excluding Structure Diffusion, our overlap-based methods purpororr the other basclines in terms of both Coverage and Density metrics. Notably, while Structure Diffusion achieved the highest performance in terms of quality and naturalness, there is significant gap in compositional generation capabilities between this model and our proposed overlap-based methods, as shown in Tables 2 and This demonstrates that the overlap-based approach offers a balanced trade-off between enhancing compositional generation and maintaining overall quality.
Table 7: FID scores on two-entity and three-entity COCO captions using Stable Diffusion v1.4 as the backbone for image generation. Lower FID scores indicate higher quality and more realistic generated images. Structure Diffusion achieves the lowest FID scores among the basclincs, while our KL Divergence loss yiclds the best FID scores among the overlap-based optimization methods. Furthermore, in terms of the Coverage and Density metrics, the proposed overlap- based metrics outperform most of the baseline models, with only Structure Diffusion demonstrating superior performance.
Method
FID Score
(4)
(Three Entities) (4)
FID Score
Coverage (t)
Density (t)
(Two Entities)
Stable Diffusion
7.85
8.09
0.84
0.66
Attend-and-Eae Excite
6.57
7.54
0.80
0.65
Divide-and-Bing
7.21
7.40
0.86
0.72
Predicated Diffusion
6.27
7.50
0.70
0.70
Structure Diffusion
3.26
4.06
0.92
0.90
ToU CoM Distance
5.09
5.85
0.83
0.73
6.21
6.39
0.85
0.67
KL Divergence
4.99
5.78
0.84
0.76
CC
5.44
6.36
0.81
0.72
Robustness to Complex and Free-style Prompts. To demonstrate the robustness our overlap-based approach in handling complex, lengthy, and free-form prompts, we conducted additional experiments using four-entity prompts from the COCO-Comp dataset as well as free-form COCO captions (Lin et al.] 2014). The results, provided in Appendices D.I and D.2 highlight the effectiveness of our overlap-based methods, showing improvements of 12% and 9.6% in the harsh TIFA score for the COCO captions and four- four-entity prompts of COCO-Comp, respectively.
Robustness to Different Backbones. To further showcase the robustness of our approach across various Stable Diffusion backbones, we repeated the previous experiments using Stable Diffusion v2-base, Stable Diffusion v2, and Stable Diffusson XL. Our approach consistently outperformed the baselines across all backbones, underscoring its adaptability to different model sizes. Notably, we observed improvements of 14.9% with Stable Diffusion v2 and 14.2% with Stable Diffusion XL. Additional details on these experiments be found in Appendix D.3
Combining Overlap-based Loss Functions with Other Metrics. As discussed in Section 4.4 attention overlap exhibits the strongest negative correlation with the success rate of generating all entities, making it key factor in the entily-missing problem. In addition to applying overlap-based loss functions (ToU, DCAM, DKL, and CC) independently, we also explored their combination with other loss terms, such as intensity (Inf) and variance (Var), detailed in Scctions 4.1 and 4.2 respectively. Appendix D.4 reports the results of these experiments on two-entity prompts from our COCO-Comp dataset using the Stable Diffusion v1.4 model as the backbone. Combining the overlap-based loss functions with intensity and variance provides
16

---

Published in Transactions on Machine Learning Research (03/2025)
some additional improvements certain metrics compared to the overlap-based loss functions alone, with the combination of CoM distance and intensity performing best overall. However, the gains from combining losses were insignificant, emphasizing the pivotal role of attention overlap in the entity missing problem. Specifically, utilizing DCOM as the sole loss function led to a 24.1% improvement over the bascline Stable Diffusion. However, when combined with the intensity-based loss function (Iul), the additional improvement. was only 0.8% compared to using Doo alone.
Other Visual Compositional Generation Failure Modes. We further evaluated the effectiveness of the proposed overlap-based metrics in addressing other compositional generation challenges, including incorrect attribute binding, incorrect spatial relationship, and numeracy-related issues. For attribute binding and spatial relationship tasks, we used 100 randomly selected prompts from the color and 2D-spatial sections of the TeeeChhmBenchh benchmark, respectively. To generate numeracy-related images, we utilized 100 prompts in the format "number entity and number entity", where the number tokens were one of 'one', 'two', 'three', or 'four', and the entity tokens were selected from the COCO dataset. The human assessment-basee based results presented in Table X were obtained using the CoM distance metric the loss function, applied exclusively to entity-related attention maps, with Stable Diffusion v2 serving as the backbone. Notably, for numeracy-related problems, the mean absolute error (MAE) was calculated as the absolute difference between the ground truth number in the prompt and the number of generated entitics in the image across all prompts. As expected, entity missing is a fundamental issue in visual compositional generation, and solely enhancing entity preservation can significantly contribute to mitigating other related challenges.
Table 8: Quantitative comparison between the overlap-based method, CoM distance, and the state-of-the-arl T21 model, Stable Diffusion v2, over other visual compositional generation tasks: attribute binding, spatial relationship, and numeracy. All the results are based on harsh human assessments. For the attribute binding and spatial relationship lasks, the success rate of the method in accurately generating the corresponding attribute or relation is evaluated. In contrast, for the numeracy task, the mean absolute error (MAE) between the number of entities generated in the images and the ground truth number specified in the prompts is
assessed.
Method
Attribute Binding (t)
Spatial Relationship (t)
Numeracy (4)
Stable Diffusion
59.75%
56.33%
2.74
CoM Distance
73.75%
63.66%
2.09
6.3 Qualitative Results
Figures D anddd 0 demonstrate qualitative comparison of images generated by our overlap- methods and stato-of-the-art basclines, Attend-and- -xccie and Predicated Diffusion, on a diverse set of prompts from cOcO-Comp dataset. As evident from the examples in Figure 6 grrend-ann-Exeitee and Predicated Diffusion often struggle to generate all entities distinctly and well-separated. Instead, they tend to produce images where one entity is clearly rendered while others are poorly formed or missing entirely. For the prompt "a bicycle and a mirror", these models generate an image with only the mirror appearing well-defined, while the bicycle is poorly represented. Similarly, for sa bottle and a sheep," " clear bottle is generated, but the sheep entity is obscured or ill-dcfined. This issue becomes even more pronounced in three-entity sccnarios. For example, given the prompt. 'boat, cow, and cup', both Mitend-and-Excitt and Predicated Diffusion completely fail to generate one of the three entities, omitting it entirely from the image.
In contrast, our proposed overlap-based methods consistently produce images that accurately capture the composition outlined in the input prompts. By minimizing the overlap between atlention maps of distinct entities, our approach ensures that each subject is well-separated within the attention map space and correctly associated with its corresponding attributes. Further qualitative results using additional backbones (Stable Diffusion XL, Stable Diffusion v2, Stable Diffusion 1.4, Stable Diffusion v2-base) and datasets (COCO-Comp, COCO, and the prompts containing background entities) are provided in Appendix
17

---

Published in Transactions on Machine Learning Research (03/2025)
CoM Distance (Ours)
KL CC (Ours)
Attend & excite
Predicated Diffusion
loU (Ours)
Divergence (Ours)
"a bicycle and a mirror"
"a bottle
a sheep"
and
"a boat
and a cow and cup"
"an apple and
carrot and a mouse
Figure 6: Qualitative comparison of images generated by our overlap-based methods (IoU, CoM Distance, KT Divergence, and CC) and sate--f-thhe-at baselines Srhend-and-Exeite and Predicated Diffusion) on various textual prompts. Our approach consistently generates images that faithfully reflect the composition specified in the prompts, with well-separated and correctly bound entities. In contrast, baselines like and Predicated Diffusion often struggle to generate all entities distinctly, with some entities being poorly represented or missing entirely, especially in scenarios with three entities.
Risk of Introducing Undesirable Biases. It is important to clarify that reducing overlap between attention maps does not necessarily correlate with increasing spatial distance between entitics in the generated image and, hence, generating unnatural images. Remarkably, three of our overlap-based methods (IoU, KL Divergence, and CC) only aim reduce the overlap between attention maps of entities, not to push entities apart spatially. For instance, two entities could be placed side-by-side in the generated image while the IoU value between the corresponding attention maps is zero. As real example,
illustrated in Figure 4 in A Appendix F (the plate and pizza example), even when reduce the overlap bet ween attention maps during the image generation, the spatial relationship between entities (pizza inside the plate) is maintained, resulting in a realistic and natural image being generated.
7 Conclusion
Through this work, we conducted a comprehensive study of the entity missing problem, a critical compositional generation failure mode in T2I diffusion models. By investigating three hypothetical sources of the problem through the lens of cross-attention mechanisms attention intensity, attention spread, and attention overlap we identified the crucial role that overlap between attention maps of entities play the entity missing problem. Building upon this insight, we proposed four overlap-based loss functions: IoU, DCoM DKL. and CC, which can be applied over the latent embedding training-free approaches to mitigate the entity missing
18

---

Published in Transactions on Machine Learning Research (03/2025)
issue during denoising process. Extensive experiments diverse set of prompts from our proposed dataset COCO-Comp to free-style COCO captions and well-known compositional benchmarks T21-Compbench and HRR-BBecch, which all together span various linguistic structures and entity counts, demonstrated that our proposed mothods significantly outperform statc-of-the-art baselines in faithfully generating entitics, as measured by commonly-used evaluation metrics such as human evaluation, visual question answering (VQA), image captioning score, and CLIP similarity. These results suggest that attention overlap primary factor contributing to the entity missing problem in T2I diffusson models. We also evaluate the quality and naturalness of the images generated by overlap- based methods using the FID score, Coverage, and Density metrics. All metrics indicate that the degradation in quality and diversity is negligible. Furthermore, qualitative results suggest that the application of overlap-based loss functions during the denoising steps does not introduce undesirable biases, such as unnecessary and unnatural spatial distances between entities. While there are limitations to the proposed method (Appendix G), the proposed overlap- based loss functions provide an effective and flexible plug-and-plas framework for addressing the entity missing problem without the need for expensive fine-tuning or retraining of the model. Furthermore, our experiments indicated that the exclusive enhancement of entity preservation can contribute to the alleviation of other challenges in compositional generation, such incorrect attribute binding, incorrect spatial relationship, and issues related to numeracy. We believe this work paves the way for further research into training-fre free approaches for enhancing the compositional generation capabilities T2T diffusion models, ultimately leading to more reliable and semantically coherent image synthesis.
19

---

Published in Transactions on Machine Learning Research (03/2025)
References
Fslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. 2023 IEEE/CVE
International Conference on Computer Vision (ICCV), pp. 19984-19996, 2023. URL https:
semanticscholar.co org/ wttp:rrrrrroooossoe5ee5.0.. 2580596800
James Belker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/ com/popers/oee /alle--- pdf, 2(3):8, 2023.
Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, and Gal Chechik. Make it count: Text-to-image generation with an accurate number of objects. ar preprint arXiv:2406. 10210, 2024.
Agneet Chatterjee, Gabriela Ben Melech Stan, Estelle Aflalo, Sayak Paul, Dhruba Ghosh, Tejas Gokhale, Ludwig Schmidt, Hannaneh Hajishirzi, Vasudev Lal, Chitta Baral, and Yezhou Yang. Getting right: Improving spatial consistency in text-to-image models. CoRR, abs/2404.01117, 2024. URL https:
//doi.org/ 10. 48550/ arXiv 2404. 01197
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Autend-and-eexite:: Attention-based semantic guidance for
diffusion models. ACM Transactions on Graphics (TOG), 42:1 10, 2023. URL https::/app. http://pp...emmttticccccchhhoool wwwpsihool.rr..rrooooroooo org./oorpusD::254115555
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5343 5353, 2024.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Newral Information Processing Systems, 36:16222-16258 2023.
Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexcy Dosovitskiy, and Zeynep Akata, Reno: Enhancing one-step texl-Lo-image models through reward-based noise optimization. arXiv preprinl or Xix:2406.0403 04312, 2024.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference on Learning Representations, 2023.
URL https://openreviee
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Newral Information Processing Systems, 36, 2024.
Sleven M Frankland and Joshua D Greene. Concepts and compositionality: in search of the brain's language of thought. Annual review of psychology, 71(1):273-300 2020.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. ArXiu, 22s/22122.00010 2022. URL sttpp:///pp.seenatttcccccooooo www.eeeiiccoeooe.....oe.ee ottpa//aetecccoooooooooooooooooa :0: 254877055
Jianshu Guo, Wenhao Chai, Jie Deng, Hsiang- Wei Huang, Tian Ye, Yichen Xu, Jiawei Zhang, Jenq-Neng Hwang, and Gaoang Wang. Versat2i: Improving text-to-image models with versatile reward. arxiv preprint ar 000:22433.1.93 2024.
Beiing-Wei
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528. Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v wwtp0...mmmmmmmm.mma.5.. URL sttp:://acllatthooogg.orr
2021. emnlp-main 595
20

---

Published in Transactions on Machine Learning Research (03/2025)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 2021 Workshop on Deep Gener- afive Models and Downstream Applications, 2021. URL http:s//pppaaanoooso.oeeeooaraap
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840-6855 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. ArXiv, abs/2210.02303, 2022a. URL https://api
semanticscholar wwwpaaarrrrrroooouuuaaass
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. ArXiu, abs/2204.003558, 2022b. URL see..i/cc...emmtttccccccoo..rrr org org/CorpusID;
248006185
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 20406-20417, 2023.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. In Thirly-sevente Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2023. URL / http::/pppparrreeee.ooefoooorooo net/forummid weHBzTLXpH!
Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. arXic preprinl 2xix:2404.03633 2024.
Wonjun Kang, Kevin Galim, and Hyung Koo. Counting guidance for high fidelity rext-to-image synthesis. arXiv preprint arXiv:2306.1158' 2023.
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. at first you don't succeed, try, try again: Faithful diffusion-based text-to-image generation by selection. arXiv preprint arXiv:2305.1300 133088 2023.
Bahjal Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 6007-6017, 2023.
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung- Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with allention modulation. In ICCV, 2023.
Nupur Kumari, Bin Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization text-to-image diffusion. 2023 LEEE/CVE Conference on Computer Vision and Pattern Recognition argg/oorpusDD:254400777
(CVPR), pp. 1931-1941, 2022. URL attpps://ap semantccscoolar http://hwwaaaaaeaeoooooroor/rr//
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jinyao Emily Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and
Deva Ramanan. GenAl-bench: A holistic benchmark for compositional text-to-visual generation. In Synthetic Data for Computer Vision Workshop @ CVPR 2024, 2024. URL https://openrevier. net/
forum? himmmidd////k3ym
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pro-training for unified vision-languaae understanding and generation. In International conference on machine learning, pp. 12888 12900. PMLRR 2022.
21

---

Published in Transactions on Machine Learning Research (03/2025)
Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for im- proved generative semantic nursing. In British Machine Vision Conference, 2023. URL https
/aapi.sematticscholarr oegeaa/ttcccccooo.ooooooooooaaaa org/CorppuaD::259911333
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Y Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognilion (CVPR), pp- 300 309, June 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 18th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 18, pp. 740-755. Springer, 2014.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision ECCV 2022: 17th European Conference, Tel Avio, Israel October 23-27, 2022, Proceedings, Part XVII, pp. 423-439. Springer, 2022.
Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Correcting diffusion generation through resampling. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8713 8723, 2024.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. International Conference on Machine Learning, 2020.
Dong Huk Park, Grace Luo, Clayton Toste, Samanch Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell. Shape-guided diffusion with inside-outside attention. In Proceedings of the IEFE/CVV Winter Conference on Applications of Computer Vision, pp. 4198 42884 2024.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms.. The MIT Press, 2017.
Steven Phillips and William H Wilson. Categorial compositionality: category theory explanation for the systematicity of human coguition. PLoS computational biology, 6(7):e1000858, 2010.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and
Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL
https://arxiv..rggt www.wawwwwwwr.raaaaaaaaaaaaaaaee /207 01952
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d
diffusion. In The Eleventh International Conference on Learning Representations, 2023. URL https :
//openreview. ittparrewee..et/fooommdddddd-e net//Eorum?1-=FFJyyssc
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, el al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πyaa Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp- 8821 8831. Pmlr, 2021.
Adilya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Lext-conditional image generation with clip latents. arXiv preprint arX1u:2204.06115,, 1(2):3, 2022.
Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. In shirth-seventh Conference on Neural Information Processing Systems, 2023. URL /ttps://openreeiew net/Eorrumix==ADKUAnaa
22

---

Published in Transactions on Machine Learning Research (03/2025)
Carlo Reverberi, Kai Görgen, and John-Dylan Haynes. Compositionality of rule representations in human prefrontal cortex. Cerebral cortex, 22(6):1237-1244) 2012.
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, New York, NY,
USA, 2023. Association for Computing Machinery. ISBN 9798400701597. doi: 10.1145/3588492.3551115
URL http:///d...rr11..1144//////3/32 35915503
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth:
Fine tuning text-to-image diffusion models for subject-driven generation. 2023 IEEE/CWE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22500-22510, 2022. URL https://apl
semanticscholar. orgo/oo//oo Co: 251800180
Kota Sueyoshi and Takashi Matsubara. Predicated diffusion: Predicate logic-based attention guidance for text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 8651-8660, June 2024.
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da- Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjocrd van Steenkiste, Ranjay Krishna, et al. Drcamsyne: Aligning text-to-image generation with image understanding feedback. In Synthelic Dala for Compuler Vision Workshop @ CVPR 2024, 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey Loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. 2022 IEFE/CVE Conference on Computer Vision and Poltern Recognition (CVPR), pp. 1228-5238 2022. URL
https://apl semanticscholar Caapaattissscollaaaorroorrrppppe Corg//orrssID..... :48006414
Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin. Compositional text-to- image synthesis with attention map control of diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 5554,5552, 2024.
Song Wen, Guian Fang, Renrui Zhang, Peng Gao, Hao Dong, and Dimitris Metaxas. Improving compositional text-to-image generation with large lasiunn models. ArXiv, abs/2310.06611, 2023. URL https //api. wew.mmessccicccollrr.. www.chollaa.rroooopussssllleeeee org/CorppuID::26330000
Qiucheng Wu, Yujian Liu, Handong Zhao, Trung M. Bui, Zhe Lin, Yang Zhang, and Shiyu Chang, Harnessing the spatial-temporal attention of diffusion models for high- high-fideiity text-to-image synthesis. 2023 IEFE/CVF International Conference on Computer Vision (ICCV) pp. 7732-774, 2023. URL
https://api. semanticscholart seg.aattessccooa.....oorrrrlllla wwwp/oorossI::2550...... 258048923
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Reco: Region-controlled text-to-image generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14246-14255, 2022. URL
ott.r//iisiccoooooooooooooooossa 254043880
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like lags-of-wordss and what to do about it? In International Conference Learning Representations, 2023. http:::://ppppeeeeeeeeeee/r/rrrp
23

---

Published in Transactions on Machine Learning Research (03/2025)
Oz Zafar, Lior Wolf, and Idan Schwartz. Iterative object count optimization for text-to-image diffusion models, 2024. URL http:://arrx...orrgab////// 11721
Arman Zarei, Keivan Rezaei, Samyadeep Basu, Mchrdad Saberi, Mazda Moayeri, Priyatham Kattakinda, and Soheil Feizi. Understanding and mitigating compositional issues in text-to-image generative models. arXiv preprint (86::2400.088444 2024.
Yang Zhang, Teoh Tze Tzun, Lim Wei Hern, Tiviatis Sim, and Kenji Kawaguchi. Enhancing semantic fidelity in text-to-image synthesis: Attention regulation in diffusion models. CoRR, abs/2403.06381, 2024. URL
https://dbi.orr//1 0tt.//p.....r........//v/a...... 06381
Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6027-6037, 2023.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffuusin model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 22490-22499, 2023.
24

---

Published in Transactions on Machine Learning Research (03/2025)
Appendix
A COCO-Comp Dataset
Unfortunately, many previous works have utilized datasets with limited diversity regarding the entities and categories represented. response, we emphasize the importance of covering a diverse set entities spanning wide range of categories, including animals, foods, furniture, and more, by adopting 71 objects from the COCO dataset Lin et al. (2014) as our entity set. These diverse entitics, detailed in table 9 can comprehensively assess the ability of the T2I model to encounter the complexity of real-world environments and even unseen combinations of entities, which is neglected in the previous works. Notably, filtered out 20 entities from the 91 entities represented in the COCO dataset because they were two-part words, such as "wine glass" or plural, such as "scissors".
Adopting the 71 remaining objects from the COCO dataset, we generated our dataset called COCO-Comp, containing two groups of prompts, each designed challenge the compositional generation capability of the T2I model. The first group comprised 2485 prompts featuring two entities, structured in the format of "a (entity) and (entity)", while the second group consisted of 57155 prompts containing three entities, presented "a {entity) } and {entity) and {entity)". The number of prompts for different entity counts is shown in table [O] For each group, we divided the prompt set into two sets, validation set for hyper-parameter finding and test set for evaluation, in such way that there is no common entity between these sets. This strict splitting strategy, neglected in previous works, ensures no data leakage during the testing phase. More details about the entities, prompts, and the splitting process are available in supplementary materials.
Notably, adopting these fixed formats in the COCO-Comp dataset serves least two purposes. Firstly, it standardizes the presentation of prompts, ensuring consistency over all experiments and reducing the effect of other irrelevant factors that may implicitly contribute to the evaluation of the entity missing problem. Secondly, these fixed formats enable us to systematically explore a wide range of entity combinations, from common pairings to more unusual configurations. Including common and rare combinations evaluates the modol's ability to generalize beyond familiar training data and adapt to novel scenarios. However, we also conducted experiments on the COCO captions to go beyond these fixed prompl formats.
Table 9: Entities and the corresponding categories of the COCO-Comp dataset, where black and blue ones are entities in the test and validation sets, respectively.
Animal Vehicle
Clothing Digital Device
Etable
Home & Other
bird dog
motorcycle boat
shoe
banana apple
spoon, bench cup, kite chair, frisbee desk, snowboard
handbag
remote laptop
airplane Lie horse bus backpack
cat
pizza
keyboard
cake donut
sheep train umbrella
blender, person knife, skateboard toilet, surfboard door, suitcase
cow truck hat
orange sandwich
elephant
bicycle bear car
broccoli
zebra
carrot
microwave, bottle
giraffe
plate
mouse
fork bowl
couch
bed
mirror
window
25

---

Published in Transactions on Machine Learning Research (03/2025)
Table 10: Number of prompts for different entity counts in the COCO-Comp dataset and COCO captions
Entity Count
#Prompts in COCO-Comp #COCO Captions
Two Entities (Validation)
190
Two Entities (Test)
1275
1000
Three Entities
1000
1000
B Compositional Evaluation Metrics
Evaluating compositional generation failure modes, like entity missing, can be challenging due to ambiguity in interpreting a missing occurrence. Honce, we applied four methods, from human evaluation to VLM-basod ones, to obtain & holistic evaluation framework for the entity missing problem.
Human Score: Given a prompt along with the corresponding generated image, we ask the evaluator assign each entity score from discrete set {0, 0.5, 1), where score of 0 denotes the absence of the entity in the generated image, score of 1 indicates that the entity is wholly and faithfully represented, and score of 0.5 signifies partial observability of the entity within the image. Subsequently, we provide two options for aggregating the assigned scores across all entities in prompt: soft and harsh averaging strategies. Under the soft evaluation paradigm, scores are aggregated using standard arithmetic averaging without imposing additional raists. Conversely, the harsh human evaluation, we adopt a more stringent criterion by treating entities deemed to be partially observable (assigned score of 0.5) as though they are absent from the image (assigned score of 0). Finally, all the scores across prompts are averaged normally.
Through the visual question answering (VQA) evaluation method, for each entity in given pair of prompts and generated images, we prepare a question, "Is there any {entity) in the image?" and give it to the VQA model along with the corresponding image. If the VQA model answers "yes", we consider a score 1; otherwise, is Similar to human evaluation, we proposed two options of soft and harsh averaging such a way that in a harsh scenario, the overall score of a prompt 1, if and only the VQA answers all questions are "yes". At the end, we average all the prompt scores. Notably, BLIP-VOL al. 2022) and TIFA Hu al. 2023) models used for VQA evaluation.
VQA Score:
Captioning Score: In caption evaluation, we use an image captioning model called BLIP-Captione et al. 2022 to produce caption for each generated image. Then, the cosine similarity between the caption and the corresponding original prompt in CLIP (Radford et al. 2021 embedding space is considered as the score of that prompt, and the final score is calculated by averaging across prompt scores.
CLIP Similarity: In this evaluation method, we calculate the cosine similarity between a prompt and its corresponding generated image in CL.IP's shared embedding space, treating it as the score (Hessel et al.)
2021 Similar to other metries, Averaging these scores across prompts yields the final score.
C Attention Masebased Loss Functions
C.1
State-of-the-art Loss Functions
The proposed loss functions of three well-knwon training-free approach, Attant-and-Fxoic. (Chefer et al. 2023
Divide-and-Bal Bind Li et al., 22023), and Predicated-Diffuuionn (Sueyoshi & Matsubara, 2024). are demonstrated in equations and 10 respectively. At each denoising step i, the latent embedding 2t modified in the direction opposite the gradient of these loss functions with respect to 24.
L LAttent-and-Ean Excite
max L eEc
where
LE = 1 max(A,)
(8)
Liivide- and Bind
min
|A
AFli.j + Aili 1,5] + [A
(9)
26

---

Published in Transactions on Machine Learning Research (03/2025)
LipredieateddDii Diff
Σ log 1 (1-
(10)
C.2
Extended Version of Introduced Metrics
Equations 11 and provide generalized formulations of Trl and Var metrics deal with more than two entities, respectively. The extended Int metric [1 takes the minimum intensity across all entities, while the generalized Var metric 12 averages the variance around the center of mass over all entities. Morcover, 15 and 16 are the extended version of the overlap-based metrics to handle prompls with multiple entities. The extended versions of ToU (equation 13) and KT. divergence (equation 15 are computed as the average value over all entity pairs. The CoM distance (equation 14] considers the area of the polygon formed by the CoMs of all entities. The CC (equation 16 takes the maximum attention score at each location across all entity maps. Note that equation [4] TN denotes the polygon formed by the CoMS of atiention maps the N vertices.
equalions 13
Int(A ,..., A...,AAN) min(/nt(AF)),, ..... [n
(11)
N Σ
P..,,
V (P.))
(12)
N
TOU(P .PM) 1 N
Σ
ToU P!))
(13)
k= 1=1k1
DCoM ,..., ,o...,..... P.N) Area(TN)
(14)
N Σ
DKI(PEt, P()
(15)
k=11=k
1
CC(PE) ,..., PeN)
max (Pe [i,j]. ..., PeN [i,
(16)
W
D Additional Quantitative Results
D.1 Results of More Complex Prompts
To evaluate the robustness our overlap-based methods to prompts containing more entities, we conducted an experiment using four-entity prompts from the COCO-Comp datasct with Stable Diffusion v1.4 as the backbone. Table D demonstrates the superiority of our overlap-based methods over strong baselines in dealing with complex prompts.
27

---

Published in Transactions on Machine Learning Research (03/2025)
Table 11: Performance comparison between our overlap-based methods and state-of-the-art training-free approaches over the four-entity prompts from COCO-Comp datasct, using the Stable Diffusion v1.4 as the backbone model. The proposed overlap-based methods consistently outporform the baselines.
Method
TIFA Score (S) (8)(1)
(t)
Score (H) (1)
TIFA
BLIP-VQA Score (S)
(1)
BLIP-VQA Score (H) (t)
Captioning Score
CLIP Similarity
(1)
Attend-and-Eaan Excite
70.8%
23.2%
82.0%
45.4%
0.738
0.322
Predicated Diffusion
65.4%
8.8%
70.6%
13.4%
0.760
0.320
IoU
77.2%
32.8%
84.3%
47.8%
0.757
0.329
CoM Distance
78.2%
28.2%
85.2%
45.4%
0.744
0.318
D.2 Results of Free- Style Prompts
In the previous experiments, prompts followed a fixed format "a {entity) and a fentity) To evaluate the generalizability of our method to more diverse linguistic structures, we also tested on free-form captions from the COCO dataset (Lin al. 2014p. Table 12 presents the results for COCO captions containing least three entities, utilizing Stable Diffusson 1.4 the backbone model. The overlap-based loss functions demonstrate significant improvements over the baseline methods.
Table 12: Performance comparison between our overlap-based methods and state-of-the-art training-free free approaches over the COCO captions with least, three entities, using the Stable Diffusion v1.4 as the backbone model. The proposed overlap-based methods consistently outperform the basclines, with the IoU achieving the best results in VQA-based evaluations.
Method
Score (S) (t)
TIFA
Score (H) (t)
TIFA
BLIP-VQA Score (S)
(1)
BLIP-VQA (1)
Captioning Score
CLIP Similarity
(1)
Score (H)
Stable Diffusion
67.0%
22.2%
70.3%
28.5%
0.781
0.313
Attend-and-
85.7%
62.1%
90.3%
73.2%
0.816
0.314
Divide-and-Bind
84.0%
58.3%
88.3%
68.6%
0.818
0.315
Structure Diffusion
79.1%
46.9%
83.0%
55.0%
0.816
0.310
Predicated Diffusion
79.2%
45.6%
81.6%
51.4%
0.812
0.312
ToU CoM Distance
90.3%
74.1%
93.6%
81.8%
0.821
0.317
88.7%
69.4%
92.5%
80.0%
0.820
0.314
KL Divergence
89.5%
71.9%
92.6%
79.2%
0.824
0.318
CC
89.7%
73.2%
93.2%
81.5%
0.812
0.313
D.3 Results of Other Backbones
In addition to SD v1.4, assessed our approach other recent backbones, SD v2-base, SD v2 (Rombachi ct al. 2022). and SD-XL Podell et al. 2023), to validate its effectiveness across different model sizes. Tables 13 and 14 present the results on (wo-entity and three-entity prompts from our COCO-Comp dataset using Stable Diffusion vasb Moreover, tables I6 and [6 demonstrate the obtained results on two-entity prompts of the COCO-Comp dataset, using Stable Diffusion v2 and Stable Diffusion XI, respectively. Across all the backbones, the overlap-based methods outperform other baselines in all evaluation metrics. This suggests that our overlap-based approach generalize well to different diffusion model architectures and sizes.
28

---

Published in Transactions on Machine Learning Research (03/2025)
Table 13: Performance comparison between our overlap-based methods and state-of-the-art training-free approaches using the two-entity prompts from the COCO-Comp dataset the Stable Diffusion v2-base model the backbone. The proposed overlap-based methods significantly outperform the baselines across all evaluation metrics, with the CoM Distance and CC methods achieving the highest scores.
Method
TIFA Score (S)
Score (H) (1)
TIFA
BLIP-VQA(1) Score (S)
BLIP-VOR (UIP-VQA(1))
Captioning (t)
CLIP Similarity
Score (H)
Score
Stable Diffusion
76.0%
52.3%
78.8%
57.8%
0.794
0.305
Attend-and-Excit
82.9%
66.0%
85.3%
70.7%
0.805
0.308
Divide-and-Rimg
77.9%
56.0%
80.3%
69.9%
0.802
0.307
Predicated Diffusion
70.8%
47.0%
78.0%
58.6%
0.758
0,285
Composable Diffusion
64.4%
31.7%
72.6%
46.3%
0.772
0.301
ToU
96.2%
92.4%
97.8%
95.7%
0.841
0.325
CoM Distance
96.6%
93.3%
97.7%
95.5%
0.844
0.326
KL Divergence
96.3%
92.8%
98.0%
96.0%
0.837
0.325
CC
96.2%
92.6%
98.2%
96.5%
0.838
0.323
Table 14: Performance comparison between overlap-based methods and state-of-the-arl training-free approaches using the three-entity prompts from the COCO-Comp dataset the Stable Diffusion v2-base model the backbone. The proposed overlap-based based methods significantly outperform the baselines across all evaluation metrics, with the CoM Distance method achieving the highest scores.
Method
Score (1)
Score (H) (t)
TIFA
BLIP-VQA (t)
BLIP- SLIP-VQAA/
(t)
Captioning (t) Score
CLIP Similarity (f)
Score (S)
Score (H)
Stable Diffiusion
67.0%
22.2%
70.3%
28.5%
0.781
0.313
Attend-and-Exee
70.0%
27.2%
73.8%
33.8%
0.785
0.314
Divide-and- Bind
67.7%
24.0%
71.2%
29.7%
0.742
0.314
Predicated Diffusion
62.0%
19.5%
70.6%
30.5%
0.730
0,292
Composable Diffusion
46.8%
5.1%
59.9%
17.9%
0..222
0.291
loU CoM Distance
91.9%
76.7%
94.7%
84.3%
0.819
0.340
94.5%
85.3%
96.8%
91.2%
0.823
0.343
KL. Divergence
89.2%
69.3%
0.92.2%
77.4%
0.819
0.340
CC
94.0%
82.6%
96.7%
90.3%
0.819
0.341
Table 15: Performance comparison between our overlap-based methods and the stale-of-the-art training- using the two-entity prompts from the COCO-Comp dataset the Stable Diffusion v2 model as the backbone. The proposed overlap-base based methods significantly outperform the baselines across all evaluation metrics, with the CoM Distance method achieving the highest scores.
method, Attend-and-Exen
Method
Score (S) (1)
TIFA
Score (H)
TIFA
BLIP-VQA (t) Score (S)
BLIP-VQA Score (H)
(t)
Captioning (t)
Score
Similarity (t)
CLIP
Stable Diffusion
81.4%
63.2%
83.3%
66.7%
0.818
0.308
Attend-and- Excite
Exxitee
90.1%
80.2%
92.7%
85.5%
0.824
0.314
ToU CoM Distance
93.9%
91.4%
98.3%
96.6%
0.842
0.323
97.5%
95.1%
98.9%
97.7%
0.841
0.325
CC
96.0%
92.1%
98.0%
95.9%
0.842
0.324
29

---

Published in Transactions on Machine Learning Research (03/2025)
Table 16: Performance comparison between our overlap-based methods and the strong baseline model, Stable Diffusion XL, using the two-entity prompts from the COCO-Comp dataset. The proposed overlap-basod methods significantly outperform the baselines across all evaluation metrics, with the CoM Distance method achieving the highest scores.
Method
TIFA (1)
TIFA
(H) (1)
BLIP-VQA (1)
BLIP-VQA Score (H)
(1)
Captioning Score
CLIP Similarity
(1)
Score (S)
Stable Diffusion
88.0%
76.2%
88.6%
77.6%
0.818
0.315
CoM Distance
95.2%
90.4%
96.0%
91.9%
0.826
0.322
KL Divergence
91.1%
82.4%
92.0%
84.3%
0.827
0.322
CC
90.2%
80.4%
90.9%
82.0%
0.820
0.320
D.4 Results of Combined Methods
Table demonstrates the quantitative results of combining overlap-based methods with other loss functions (Tnl and Var) on the two-entity prompls from our dataset COCO-Comp using the Stable Diffusion v1.4 model as the backbone. The improvement induced by this combination is relatively low, proposing the attention overlap as the root cause of the entity missing problem.
Table 17:
Performance comparison of overlap-based metrics and their combinations with variance and intensity loss functions on the two-entity prompts of the COCO-Comp dataset using the Stable Diffusion v1.4 model as the backbone. The proposed overlap-based methods and their combination with non-overlap-based ones consistently outperform the baselines across all evaluation metrics. While incorporating intensity and variance-based loss functions alongside overlap-based ones generally enhances the performance of compositional generation, the extent of this improvement is minimal, suggesting that attention overlap remains the primary factor contributing to the issuc of missing entities.
Method
TIFA (1)
TIFA
(H) (1)
BLIP-VQA (1)
BLIP-VQA
(t)
Captioning Score
CLIP Similarity
(1)
Score (8)
Score (S)
Score (H)
Stable Diffusion
69.9%
40.0%
80.0%
44.2%
0.789
0.297
Attend-and-Exee
87.6%
76.5%
92.2%
84.9%
0.811
0.312
Divide-and- Bind
86.5%
73.6%
91.9%
83.9%
0.815
0.309
Structure Diff
74.3%
49.6%
77.2%
54.9%
0.792
0.299
Predicated Diff'
90.6%
81.6%
92.7%
85.7%
0.823
0.314
Composable Diff
66.2%
37.6%
73.1%
49.3%
0.761
0.297
ToU CoM Distance
92.7%
85.3%
95.0%
90.1%
0.824
0.317
94.0%
88.2%
95.6%
91.3%
0.825
0.318
KT. Divergence
91.7%
83.5%
94.5%
88.9%
0.818
0.316
CC
92.2%
84.6%
95.1%
90.3%
0.821
0.315
IoU & Int
93.8%
87.7%
96.5%
92.9%
0.831
0.319
CoM Distance & Int
94.8%
89.8%
96.7%
93.5%
0.825
0.319
KL Divergence & Int
91.7%
83.7%
94.8%
89.7%
0.814
0.314
CC & Int
92.5%
85.2%
96.3%
92.8%
0.824
0.316
IoU & Var
91.4%
82.9%
94.3%
88.7%
0.822
0.321
CoM Distance Var
94.4%
89.0%
96.0%
92.2%
0.825
0.321
KT. Divergence & Var
90.8%
81.6%
94.5%
89.1%
0.813
0.315
CC & Var
91.5%
83.4%
93.9%
88.0%
0.813
0.320
E
Additional Qualitative Results
Figures
and 12 present additional qualitative comparisons of images generated between
overlap-basee based methods and various baselines. More specifically, figures 7 T0
11
and 12
contain
30

---

Published in Transactions on Machine Learning Research (03/2025)
two-object prompts from COCO-Comp dataset using Stable Diffusion XL, Stable Diffusion v2, Stable Diffusion v1.4, Stable Diffusion v2-base, respectively. Moreover, figure includes free-style and complex prompts from the COCO dataset. Finally, figure @ demonstrates the generated images from prompts containing background entities such as 'sca', 'beach', 'desert', and 'mountain'.
"a backpack and a surfboard"
"an airplane and a cow"
"a keyboard and a train"
"a backpack and a couch"
"a bowl and car"
Stable Diffusion
CoM Distance (Ous)
KL Divergence (Ours)
CC (Ours)
Figure 7: Qualitative comparison between our proposed overlap-based methods and the state-of-tho-art bascline, Stable Diffusion XL, on diverse prompts from our COCO-Comp datasct.
31

---

Published in Transactions on Machine Learning Research (03/2025)
a man that is a surfboard with dog
a person in red chopping green apples under an umbrella
splash dog chasing frisbee into a pool during a performance
A person painting colorful images vase
SD XL
CoM Distance (Ours)
Figure 8: Qualitative comparison between our proposed overlap-based method, CoM distance, and the state-of-the-ar bascline, Stable Diffusion XL, on free-style and complex prompts from the COCO datasct.
bowl and sea
plate and beach
bowl and desert
apple and mountain
banana and mountain
SD XL
CoM Distance (Ours)
Figure 9: Qualitative comparison between our proposed pverlapbbasd method, CoM distance, and the stato-of-the-art baseline, Stable Diffusion XL, on prompts containing background entities such as 'sca', 'beach', 'desert', and 'monntinn.
32

---

Published in Transactions on Machine Learning Research (03/2025)
"a backpack and bed"
"a carrot and a frisbee"
"a hat and window"
"an airplane and vase"
Stable Diffusion
Attend & excite
loU (Ours)
CoM Distance
(Ours)
(Ours)
CC
Figure 10: Qualitative comparison between all proposed overlap-based methods and the state-of-the-ard baselines on diverse prompts from our COCO-Comp dataset using Stable Diffusion v2.
33

---

Published in Transactions on Machine Learning Research (03/2025)
"a banana and bed and bowl"
"a bottle and cat and sandwich"
"an apple orange and tie"
"a car and an oven"
"a backpack
and a sink"
Stable Diffusion
Attend & excite
Divide & bind
Structure Diffusion
Predicated Diffusion
Composable Diffusion
IoU (Ours)
CoM Distance
(Ours)
KL Divergence
(Ours)
(Ours)
CC
Figure 11: Qualitative comparison between all proposed overlap- methods and the state-of-the-art baselines on diverse prompts from our COCO-Comp Loso-CC glataset using Stable Diffusion v1.4.

---

Published in Transactions on Machine Learning Research (03/2025)
"a banana and a chair and orange"
"a keyboard and a pizza and plate"
"a clock and frisbee and skateboard"
"a book and an umbrella"
"a bowl and a window"
Stable Diffusion
Attend & excite
Divide & bind
Predicated Diffusion
Composable Diffusion
loU (Ours)
CoM Distance
(Ours)
KL Divergence (Ours)
CC (Ours)
Figure 12: Qualitative comparison between all proposed overlap- based methods and the otate-of-the-art baselines on diverse prompts from our COCO-Comp dataset using Stable Diffusion v2-base.
35

---

Published in Transactions on Machine Learning Research (03/2025)
F
Attention Map Evolution
Figure 13 provides insights into how the cross-attention maps evolve during the donoising process for different models, including our overlap-based methods. The figure depicts the attention maps for two entities different denoising time steps (t=1, 3, 6, 10, 30, 50). In the initial time steps, the attention maps for the two entities are the same across all models because the random seed is fixed. This ensures fair comparison by starting from the same initial attention distribution. As the denoising progresses, the attention maps evolve differently for each model. In the baseline models the attention maps remain largely overlapping and diffuse throughout the denoising process. In contrast, when using our overlap-based optimization method, the attention maps become more distinct and localized as the denoising steps proceed. Our approach effectively guides the diffusion model to minimize the overlap welween attention maps different entities, encouraging clearer separation of entity representations. By the final time step, the attention maps for the two entities are largely non-overlapping and concentrated on the corresponding entities in the image.
Time steps which loss applied
t-1
1-3
1-6
t- 10
t- 30
1- 50
Image
Diffusion
Stable
Attend
excite
"a cat and a vase"
loU
CoM Distance
Divergence
KL
CC
Figure 13: Comparison of atlention maps during the denoising process for different training-free methods, including the proposed overlap-based methods. The attention maps for two entities are shown at various denoising time steps (t=1, 3, 6, 10, 30, 50). While the initial attention maps are identical across models due to fixed random seed, the proposed overlap-based methods (IoU, DCOM: DKL, CC) effectively minimizes overlap and encourages distinct, localized attention for each entity as denoising progresses, resulting improved compositional alignment.
Moreover
36

---

Published in Transactions on Machine Learning Research (03/2025)
Japa
23
50
Iange
25
Salte Dilloan
"A pizzzo awcz away from plate"
2 patizn anto pltte
Dongger
Figure 14: Comparison of attention maps during the denoising process for different training-free methods, including the proposed overlap-basee methods. As demonstrated for the prompt "a pizza into plate", applying our overlap-based methods does not push the entities (pizza and plate) far away from each other and, hence, gemerate unnatura images. On the other hand, when dealing with the prompt "a pizza away from a plate", our methods generate images entirely faithful to the input prompt. At the same time, Stable Diffusion and Attend-and-Exen fail to generate one entity (plate).
G Limitations
While our proposed training- free method offers a computationally efficient approach to mitigate compositional generation failure modes in T2I models, there are several limitations that should be acknowledgod and addressed in the future research.
Scalability limitations of training-free methods. Training-freo methods may not be the ultimate solution for compositional generation failure modes, particularly when dealing with highly complex text prompts containing numerous entities. As the complexity and length of the prompts increase, these methods might struggle to effectively capture and represent the intricate relationships and dependencies between the entities. In such scenarios, exlensive fine-tuning of the T2I model on carefully curated datasets tailored to the specific domain or task may be necessary to achieve optimal performance. However, this fine-tuning process often requires substantial computational resources and time, which could be a significant limitation for certain applications or users with limited computing power.
Limitations of the CLIP text encoder. The poor capability of the CLIP text encoder [Radford et al. 2021] in understanding and representing the compositionality of complex texts (Yuksekgomul et al. 2023 Thrush al.] 2022) may have significant impact on the performance of T2I models. Since most of these models rely on the CLIP text encoder as their backbone for processing textual inputs, any limitations weaknesses in the encoder's ability to capture and encode the compositional structure and semantics of the text can propagate through the entire system, leading to suboptimal results or failure modes in the generated images. Surprisingly, most of the related works have not paid sufficient attention to this potential rool cause of the failures. Instead, they often focus on proposing solutions without thoroughly investigating the underlying reasons behind the problems. They tend to attribute the issues to other factors without considering the limitations of the CLIP text encoder itself. To address this, we plan to focus our future work on investigating and improving the text encoding capabilitics of T2I models, specifically targeting their ability to handle compositiona ional complexities in text prompts.
37

---

Published in Transactions on Machine Learning Research (03/2025)
Limitations of commonly used evaluation metrics. The most commonly used evaluation metrics for assessing the performance of T2T models, such as VQA score, CLIP similarity, and captioning score, heavily rely on Vision-Language Models (VLMs) or multi-modal generative models. While these metrics provide useful insights, they inherently suffer from the limitations and biases of the underlying models, particularly in terms of their ability understand and evaluate compositional aspects of the generated images. These models often struggle to accurately capture and assess the fine-grained details, spatial relationships, and semantic coherence between different entities in the generated images. As a result, the evaluation scores obtained using these metrics may not always align with human perception and judgment of the compositional quality of the generated images. To mitigate this limitation and obtain more reliable assessment of our proposed method, we conducted extensive human evaluation studies, which provide more direct and intuitive measure of the compositional alignment between the inpul text and the generated images.
38


Reviews:
[{"review_id": "jxymXOBb84", "reviewer": "Reviewer_9HB8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The hypothesis of the paper is pretty interesting and potentially impactful.\n1. The writing is clear and captures all major facets.\n1. The paper evaluates the image generation across multiple metrics like CLIP similarity, VQA, captioning, human eval, etc.\n1. The paper evaluates the image generation on multiple datasets, including COCO-Comp, T2I-CompBench, and HRS-Bench etc\n1. The paper evaluates their hypothesis on multiple backbones, including SD-1.4, SD-2, SD-XL\n\nWeaknesses:\n1. Within the paradigm of composition, the paper studies only \"and\" type relationships, and that too for only two entities. \n1. Not clear if the three causes of the entity missing problems ((1) attention intensity, (2) attention spread, and (3) attention overlap), are mutually exclusive or even collectively exhaustive.\n1. The hypothesis is interesting and the literature review also is quite extensive. To make the paper stronger, I would suggest that the authors add evaluations of their proposed methods with datasets from other entity related problems observed in diffusion, such as incorrect spatial relations, numeracy challenges, incorrect spatial relationships, etc.\n1. Missing citation for \"This problem typically arises when the model does not allocate sufficient attention to certain entities during the generation process, leading to their partial or total absence in the final output.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Here is the extracted list of contributions:\n\n1. Insight into the competition of entity-related tokens for attention toward specific regions via the cross-attention mechanism.\n2. Proposal of four overlap-based loss functions for manipulating latent embeddings during inference:\n   - Intersection over Union (IoU)\n   - Center-of-Mass (CoM) distance\n   - Kullback–Leibler (KL) divergence\n   - Clustering\n3. Introduction of training-free methods to improve compositional alignment.\n4. Extensive experimentation on diverse prompts to evaluate performance.\n5. Demonstration of superior performance over previous approaches across metrics such as:\n   - Visual question answering\n   - Captioning score\n   - CLIP similarity\n   - Human evaluation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Indicated in weaknesses.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6Ch8HWL2CX", "reviewer": "Reviewer_q34m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and well-organized, presenting a clear and compelling motivation for the study.\n- The proposed overlap-based loss functions are creative and adaptable, delivering impressive results across multiple evaluation metrics and model architectures.\n- The framework requires no additional training or fine-tuning, making it computationally efficient and easily adaptable to existing models.\n- The experiments are comprehensive and solid, not only covering various evaluation metrics but also demonstrating the robustness of the proposed methods across different model backbones and prompt complexities.\n\n\nWeaknesses\n- The evaluation does not include prompts involving human subjects or complex object interactions, which limits the assessment of the method's versatility in real-world scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors investigate the entity missing problem in text-to-image diffusion models, introducing a systematic analysis of three potential causes: insufficient attention intensity, excessive attention spread, and overlapping attention maps between entities. To address these challenges, the authors propose overlap-based loss functions that optimize cross-attention during generation, enabling better integration of multiple entities while maintaining compositional fidelity and reducing conflicts in visual representation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In the Appendix, the authors have presented experimental results on free-style prompts, which show promising performance. I suggest including some qualitative results as well to provide a clearer visualization of the model's effectiveness in these scenarios.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ACrChc1hlY", "reviewer": "Reviewer_jij7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengthes:**\n\n1. Comprehensive validation experiments to confirm the cause of the entity missing problem.\n2. All the four metrics depicting the attention overlap bring significant improvement, proving the claim that the attention overlap is the primary cause of entity missing.\n3. Comprehensive experiments across different benchmarks, prompt types, and backbones. \n\n**Weaknesses:**\n\n1. The applicability of the proposed method is limited. The text-image alignment problem includes various aspects, while the entity missing problem is only part of it. The proposed method cannot handle many other problems like the alignment of the entity attribute, the numerical alignment, and the relation alignment.\n\n2. Lacking related work. The related work mostly introduce methods with inference time optimization. However, methods to address these misalignment issues in the training phase is missing [1,2]. \n3. The entities listed in Table 7 are all foreground objects. The paper does not include test cases for background entities like sea, forest, or sky. Also, the method also cannot handle abstract entities like music or atmosphere.\n\n[1] Sun, Jiao, et al. \"Dreamsync: Aligning text-to-image generation with image understanding feedback.\" *Synthetic Data for Computer Vision Workshop@ CVPR 2024*. 2023.\n\n[2] Jiang, Dongzhi, et al. \"CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching.\" *arXiv preprint arXiv:2404.03653* (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The authors conduct a systematic analysis into three possible reasons of entity\n  missing in T2I diffusion models. \n2. Their investigation reveals the overlap between attention maps of entities as the primary factor.\n3. Based on this finding, the authors design four criteria to measure the overlap of the attention maps of the entities. Then an inference time optimization is performed to promote less overlap.\n4. Extensive experiments showcase the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is beneficial to include some discussion about applying all three metrics, i.e., the attention intensity, the attention spread, and the attention overlap, in the inference time optimization. From the result in Figure 5, the attention intensity and spread also seem to affect the alignment. The question is whether they are minor causes of the entity missing problem. Does applying them also contribute to the text-image alignment?\n\n2. It is better to include the related work mentioned in *Strengths And Weaknesses* part.\n3. Adding test cases of some background entities can better demonstrate the effectiveness of the proposed method. Some example background entities are: mountain, sea, beach, and sky. \n4. The authors need to discuss the limitation of the proposed method of the entity type, especially the applicability of abstract or background entities.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The entity missing is an important problem in text-to-image diffusion models. This paper investigates the problem and concludes that attention overlap is the main reason with an extensive experiments. The reviewers initially raised some concerns such as the generality of the proposed method, the reasonability of evalluations, and related works, etc. The authors address most of the concerns by conducting further experiments and add more clarifications, which are appreciated by the reviewers. The paper receives unanimous positive support.
Decision: Accept (poster)