EGGen: Image Generation with Multi-entity Prior Learning through Entity Guidance

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 18:38:54

Abstract: Diffusion models have shown remarkable prowess in text-to-image synthesis and editing, yet they often stumble when tasked with interpreting complex prompts that describe multiple entities with specific attributes and interrelations. The generated images often contain inconsistent multi-entity representation (IMR), reflected as inaccurate presentations of the multiple entities and their attributes.  Although providing spatial layout guidance improves the multi-entity generation quality in existing works, it is still challenging to handle the leakage attributes and avoid unnatural characteristics.  To address the IMR challenge, we first conduct in-depth analyses of the diffusion process and attention operation, revealing that the IMR challenges largely stem from the process of cross-attention mechanisms.  According to the analyses, we introduce the entity guidance generation mechanism, which maintains the integrity of the original diffusion model parameters by integrating plug-in networks. Our work advances the stable diffusion model by segmenting comprehensive prompts into distinct entity-specific prompts with bounding boxes, enabling a transition from multi-entity to single-entity generation in cross-attention layers.  More importantly, we introduce entity-centric cross-attention layers that focus on individual entities to preserve their uniqueness and accuracy, alongside global entity alignment layers that refine cross-attention maps using multi-entity priors for precise positioning and attribute accuracy. Additionally, a linear attenuation module is integrated to progressively reduce the influence of these layers during inference, preventing oversaturation and preserving generation fidelity. Our comprehensive experiments demonstrate that this entity guidance generation enhances existing text-to-image models in generating detailed, multi-entity images.

Main paper: 
 related work: 
 Text-to-Image Generation. In the swiftly evolving field of textbased image generation, an array of model architectures and learning paradigms have surfaced, as evidenced by a series of pivotal studies [2-5, 10, 17, 19-21, 28, 29, 32] . Initially, GAN-based models [19, 28, 32] were at the forefront, setting foundational benchmarks for the quality and diversity of the images. Recently, the advent of diffusion models [18, 22, 23] marked a significant leap forward, enhancing the fidelity and realism achievable in text-to-image generation. These models operate on the principle of structured denoising [8] with latent diffusion [22] , which begins with initializing random noise in a latent space. This noise is then systematically refined through a denoising process, transforming it into visually detailed images by incorporating textual conditions. This method enhances both diversity and realism in generated images, making latent diffusion models a powerful player in generative AI.Multi-entity Generation. Multi-entity synthesis is an area of significant interest due to its potential and broad applications in industries. Most efforts [1, 12, 13, 15, 26, 30] to address the challenges of diffusion models in accurately representing multiple entities with special attributes. For instance, GLIGEN [12] adopted bounding box coordinates as grounding tokens and integrated them into a gated self-attention mechanism to enhance positioning accuracy. Furthermore, the LLM-grounded diffusion model [13] used DDIM inversion to create initial latents for each entity and then applied the GLIGEN model for precise layout arrangement. Detect guidance [15] integrated a latent object detection model to separate different objects during the generation process, then masked the conflicting prompts and enhanced related ones. Despite existing methods of generating images with correct positions, challenges persist, especially in generating images that accurately blend attributes from multiple entities. Our work is focused on investigating the underlying reasons behind the challenges of synthesizing multiple entities and conducting a divide-and-conquer mechanism to enhance entity-centric modeling in cross-attention operations. 
analyses on imr challenge: 
 Inconsistent Multi-entity Representation. Our approach begins with a thorough examination of the creation of multiple entities during the diffusion process. To understand the issues of the multi-entity generation, we conduct a statistical analysis on 1000 generated images using the prompt "A golden dog on the left and a black cat on the right are playing in the yard." with the SD V1.5 model, selecting different random seeds for each trial. The diverse outcomes of this experiment are illustrated in Figure 2 , which reveals approximately 50% cases with inaccurate entities, 50% cases with attributes leakage, and around 70% cases with position disorder. These findings indicate a challenge of inconsistent multientity representation, often resulting in a low likelihood of fully adhering to the intended multi-entity compositions of the prompts.Entity Coupling. Building on the insights from Hertz et al. [6] regarding the 32 × 32 resolution of cross-attention maps, we further investigate certain phenomena in diffusion models. This exploration involves analyzing token-wise attention maps within the U-Net architecture, as demonstrated in Figure 3 , aiming to uncover the token impact of inconsistent representations of multiple entities. Our observations highlight semantic relationships that exhibit the entity coupling of cross-attention across tokens and their impact on the accuracy of generating images with multiple entities. In good case (Figure 3a ), when one entity exhibits relatively weak signals in the attention map of another, the coupling between entities is deemed acceptable and not harmful, as each entity and its attributes are delineated by its stronger signal in the attention maps. In contrast, failing cases reveal: distinguish between entities, while entity coupling causes loss of control over the presence or absence of targets (Figure 3b and 3c ).(3) Attribute leakage: The entity coupling incorrectly aligns the entity of the "cat" strongly on the map of the "dog" token, leading to the golden attribute mistakenly associating with cat (Figure 3d ).Entity Prematurity. Previous research [1, 25] has highlighted that the minimum resolution of cross-attention layers dictates contour definition, whereas higher resolution layers are responsible for details. Our investigation extends these findings by examining the step-wise attention map in the low-resolution layers, as shown in Figure 4 . From Figure 4 , we observe that positioning and quantity of entities are established at a low resolution early in the process, sometimes as early as step 1, while detailed attributes like colors then play a pivotal role in later stages to refine the image's appearance. Furthermore, the phenomenon of entity coupling also persists throughout the inference process, contributing to inaccurate entities. The entire process appears to resemble the Entity Prematurity of cross-attention.Based on both token-wise and steps-wise analysis on attention maps, we learn that the challenge of inconsistent multi-entity representation primarily arises from the entity coupling and prematurity of cross-attention, resulting in the inaccuracy of multiple entity generation. In response, we propose the Entity Guidance Generation (EGGen) strategy to tackle these specific challenges. 
proposed approach: 
 In this section, we present an overview outlining the comprehensive mechanism of our approach. This is followed by a detailed examination of the entity-centric cross-attention and the alignment of attention refinement. We conclude with an explanation of the overall optimization strategy.In the task of text-to-image generation, diffusion models aim to accurately transform textual prompts into corresponding images. The latent diffusion architecture integrates textual information y into the image synthesis process via a cross-attention layer. Initially, textual prompts are encoded into embeddings s ∈ R 𝑛×𝑑 , which are then mapped through the cross-attention mechanism, involving query Q 𝑖 ∈ R ℎ𝑤 𝑖 ×𝑑 𝑖 , key K 𝑖 ∈ R 𝑛×𝑑 𝑖 , and value V 𝑖 ∈ R 𝑛×𝑑 𝑖 vectors, to produce attention maps A 𝑖 ∈ R ℎ𝑤 𝑖 ×𝑛 . Both key and value vectors are generated from the text-conditioned embeddings. To address the IMR challenge of the SD models highlighted in Section 3, we introduce the EGGen methodology, which builds on the strengths of the pre-trained GLIGEN SD model [12] , as detailed in Figure 5 .The process of the proposed EGGen can be divided into (1) Prompt decoupling: an LLM is utilized to reorganize the provided prompt into a global prompt, and separate entity prompts with spatial locations (marked by the bounding boxes). This organization enables the direct association of attributes with their respective entities, enhancing the model's ability to recognize each entity. These spatial locations of entities are then also fed into the gated attention to secure the precise positioning of the coordinates. ( 2) Entity-centric cross-attention: the entity-centric cross-attention layer is introduced that focuses on the entity prompts related to each entity, ensuring that the distinctiveness of each entity is maintained. Additionally, we apply box masking within each feature map to isolate sections corresponding to other entities, followed by an aggregation process yielding an integrated latent feature centered around each entity. HED images) as ground truth to guide the correct positioning of each entity and separate attributes from other entities.In the subsequent section, we will provide a detailed exposition of these modules and their underlying rationales, alongside the overall optimization process.Frequently, the challenge for diffusion models in accurately recognizing and attributing unique characteristics arises from prompt ambiguity, where entities and their attributes are intertwined. However, LLMs possess the capability to discern individual entities and predict the overall spatial layout of an image. Capitalizing on this strength, we decouple the prompt to reorganize the original prompt into a global prompt and entity prompts for each entity. Such segmentation facilitates a direct linkage of attributes to their corresponding entities, thereby improving the model's proficiency in distinctly recognizing and interpreting each entity.Specifically, we employ advanced language comprehension and inferential capabilities of the LLM (such as GPT-4) to discern the entities and their attributes within the given prompt y, leading to the generation of reorganized global prompt y and entity prompts y, expressed as:EQUATIONEQUATIONwhere 𝑁 signifies the total comprising 𝑁 -1 foreground entities and one background element, and the F 𝑟𝑒 is the re-caption operation, which enables the generation of denser, fine-grained details for each entity prompt. This global prompt offers a concise summary, highlighting key attributes for each entity. The re-caption operation enables the generated prompt of denser, fine-grained details for each entity. Additionally, we enhance each entity with bounding box attributes, predicted by the language model after it assesses the image's overall layout, allowing text-to-image model to interpret prompts with greater accuracy, as detailed below:EQUATIONAn example of the prompt decoupling is illustrated on the left part of Figure 5 . For further details on employing LLMs refer to the Appendix.Even with prompts clearly outlining entities and their attributes, diffusion models can struggle with entity coupling in cross-attention layers without a mechanism to handle this hierarchical information. In our approach, we introduce entity-centric cross-attention layers inserted ahead of the original cross-attention layers. The ECA layer shares weights with the original cross-attention, allowing for interaction between the feature latent and specific entity prompts. This ensures that each unique entity and its attributes are preserved. The process can be formulated asEQUATIONwhere 𝑖 represents the 𝑖th layer cross-attention in the UNet and Q 𝑖 ∈ R 𝑁 ×ℎ𝑤 𝑖 ×𝑑 𝑖 are generated to correspond with the transformed latent representations f 𝑖 ∈ R ℎ𝑤 𝑖 ×𝑑 𝑖 , reflected by 𝑁 duplicates. Keys K 𝑖 ∈ R 𝑁 ×𝑛×𝑑 𝑖 and values V 𝑖 ∈ R 𝑁 ×𝑛×𝑑 𝑖 are created through linear projections of entity text-conditioned embeddings s ∈ R 𝑁 ×𝑛×𝑑 , which originate from the entity prompts y. Moreover, we utilize bounding boxes B as masks within the cross-attention layers to ensure accurate spatial representation of entities. These bounding boxes are resized to match the dimensions of the attention maps, effectively transforming them to a compatible size of ℎ𝑤 𝑖 , and creating a mask M 𝑖 ∈ R 𝑁 ×ℎ𝑤 𝑖 . We then consolidate the feature latents across the 𝑁 dimension after masking, and apply 𝜙 𝐸 𝑖 network to average the entity-centric feature latents f ∈ R ℎ𝑤 𝑖 ×𝑑 𝑖 by summarizing the input f 𝑖 . The aggregation process is mathematically represented as:EQUATION)where 𝛾 is a fixed scalar setting to 1 in training and 𝛼 𝑖 is a learnable scalar which is initialized as 0. This whole design of the ECA layers tackles entity coupling by isolating each entity with its designated prompt. The aggregation with bounding masks further guarantees the precision and uniqueness of each entity's depiction, emphasizing their distinct attributes.Generating entities independently and merging them directly, without accounting for their interactions, can lead to sub-optimal integration. Merely using a cross-attention layer followed by a global prompt to interact with all entities within the same context may result in inaccuracies and attribute leakage since it does not address the problem of entity coupling. We address the sub-optimal integration by implementing a global entity alignment, as illustrated in the right part of Figure 5 . This involves refining the cross-attention maps A 𝑖 ∈ R 𝑚×ℎ𝑤 𝑖 ×𝑑 𝑖 (𝑚 represents the total number of valid tokens for N entities in the global prompt.) to better correspond with entity tokens through a CNN network 𝜙 𝐺 𝑖 . Additionally, we replicate bounding box masks M 𝑖 ∈ R 𝑁 ×ℎ𝑤 𝑖 to M 𝑖 ∈ R 𝑚×ℎ𝑤 𝑖 ensuring that the attention maps for 𝑚 entity tokens are constrained to their specific spatial positions, thereby mitigating attribute leakage, where 𝛾 is set as a fixed scalar 1 in training and 𝛼 𝑖 is a learnable scalar which is initialized as 0.EQUATIONTo further refine the attention maps for accurately capturing the intricate details of multiple entities, we employ a multi-entity prior learning strategy as guidance. We adopt HED [27] soft edge images as the prior information, which detect the contour C ∈ R ℎ×𝑤×1 over original images. According to the entity prematurity in Section 3, these HED images inform entity-focused attention maps A 𝑖 in cross-attention layers at 8 × 8 and 16 × 16 resolutions for processing efficiency. The choice of HED images is due to their ease of processing and richness in information, which benefits our finetuned layers' learning process, ensuring that the attention maps are precisely aligned. Further details on this process are depicted in Figure 6 and this alignment can be defined as:EQUATIONwhere C 𝑖 ∈ R 𝑚×ℎ𝑤 𝑖 ×1 is a segmented contour representation by B from C, resized to the target resolution of 8 × 8 or 16 × 16 and replicated to align with the 𝑚 dimension of A 𝑖 , and D (•, •) denotes the cosine similarity function, evaluated for each token's corresponding segment at the resolution ℎ𝑤 𝑖 . By minimizing this cosine distance, we refine the focus of the entity-centric attention map on prior entity-specific information, directing the synthesis of detailed entity structures while limiting refinement to areas associated with grouped tokens.Meanwhile, the denoising loss is also incorporated into the training process to further ensure the quality of the synthesized images. Therefore, the overall optimization objective can be expressed as follows:EQUATIONEQUATIONwhere 𝐿 denotes the number of U-Net layers at resolutions of 8 × 8 and 16 × 16, and 𝜆 denotes the entity guidance loss weight. Notably, the HED images are exclusively employed during the training phase to enhance the model's awareness of contours and are not utilized during inference. When inference, equally weighting ECA and GEA can excessively influence the rendering of details, occasionally resulting in oversaturated images compared to traditional text-to-image models. According to entity prematurity, entities and their contours are identified early in the inference process. To mitigate this, we implement linear attenuation during inference, gradually reducing the 𝛾 to 0:EQUATIONwhere 𝑇 𝑠 is the total steps of inference, commonly setting to 50 in the standard inference of the diffusion model and 𝑡 ∈ [1, 50] . This strategy could well remain the entity guidance within the models while decreasing the over-saturation of the generations. After the overall optimization, the EGGen effectively addresses the issue of inconsistent multi-entity depictions, ensuring the generated images are both semantically consistent and visually detailed. 
experiments 5.1 implementation details: 
 Baselines. We utilize the layout advancements from GLIGEN [12] as a base model to fine-tune our approach. We further extend our comparison with a spectrum of alternative approaches. Within the realm of training-free methods, we compare: (1) BoxDiff [26] ; (2) Backward Guidance [1] ; (3) LLM-grounded Diffusion [13] . In the domain of training-based methods, our analysis encompasses: (1) ReCo [30] ; (2) GLIGEN [12] ; (3) Detect Guidance [15] .Datasets. We use the 414K text-image pairs as training datasets, which are reorganized by ReCo [30] from COCO 2014 [14] train set. To comprehensively illustrate the effectiveness of our proposed method, we adopt T2I-CompBench [9] as the test dataset, which consists of 6,000 compositional text prompts from 3 categories (attribute binding, entity relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions).Evaluation Metrics. We follow the evaluation of T2I-CompBench [9] over the consistency between images and multi-entity prompts regarding Attribute Binding, entity Relationship, and Complex, which comprehensively utilizes various metrics, including B-VQA [11] , UniDet [31] and Clip-score [7] .Implementation Details. The total trainable parameters are the three-layer MLP from the proposed ECA layers and the four-layer CNN network in GEA. At the same time, ECA and GEA modules are exclusively implemented at resolutions of 8 × 8 and 16 × 16. The utilized LLM is the GPT-4-Vision for its robustness and exceptional performance. During training, we use the AdamW optimizer [16] with a fixed learning rate of 0.00001 and weight decay of 0.01 for 10 epochs, and we set 𝜆 = 10 for loss control. In the inference stage, we adopt DDIM sampler [24] with 50 steps and set the guidance scale to 7.5. All experiments are performed on 8 × Nvidia Tesla V100 GPUs. See the Appendix for more implementation details.Qualitative Evaluation. Given the varied capabilities of different models, we employ prompts that describe two entities to guarantee a fair comparison, and the visual comparisons are showcased in Figure 7 . In the cases examined, while most approaches demonstrate the capacity to accurately position entities at appropriate coordinates, they occasionally place the wrong type of entities and assign undetermined attributes. Conversely, our method successfully positions the correct entity along with its attributes in these scenarios. For instance, the golden dog and the black cat exhibit different vivid attitudes while strictly following the prompt requirement of the dog on the left and the cat on the right. For an illustration of our EGGen's capability to generate multiple entities, please see the examples featured in Figure 1 . Further visual examples can be found in the Appendix.Quantitative Evaluation. We conduct comparisons with prior state-of-the-art (SOTA) multi-entity text-to-image models across three key compositional scenarios on the T2I-CompBench. From the results, as shown in Table 1 , our approach demonstrates better comprehensive performance in five scenarios, exhibiting superior fidelity and precision in aligning with the text prompts. Notably, our method achieved the highest scores in scenarios involving color and texture, underscoring its remarkable capability to precisely interpret and replicate the colors and textures described in the text inputs. These benefits from our strategy of separating the prompts into a global prompt and individual entity prompts. We utilize the ECA to process each entity singularly, preserving its uniqueness and ensuring the correct types, while the GEA refines the attention map, precisely guiding the delineation of attributes. Notably, our method slightly underperformans in the shape domain over LLMgrounded [13] and Detect Guidance [15] . The shape attribute can be inherently more complex to interpret from the text than colors or textures, so it can easily be compounded by the interaction of the mask within the GEA, causing deformation of shape. We will address this complexity in future iterations.In this validation, we conducted an ablation study by individually removing various modules to assess their impact, with the results presented in Figure 9 and Table 2 . The data reveals that 1) the absence of the ECA results in the sole use of the global crossattention layer, which proves insufficient for accurately identifying and correctly positioning each entity; 2) removing GEA leads to a reliance on undifferentiated global prompts within the original cross-attention mechanism, which in turn results in attribute leakage and the generation of inaccurate entities; 3) eliminating the LA module precipitates a marked increase in image over-saturation and a discernible degradation in visual quality. This effect aligns with the observation of entity prematurity. Our ablation study highlights the critical roles of the ECA, GEA, and LA modules in enhancing the accuracy, attribute fidelity, and visual quality of our model. These findings underscore the importance of these modules in achieving superior performance in multi-entity generation.To conclude this section, we replicate the statistical analysis experiment similar to that depicted in Figure 2 , generating 1,000 examples based on the same prompt. The outcomes of this experiment are illustrated in Figure 8 . In scenarios involving the generation of images with two simple entities, such as a dog and a cat, our method demonstrates a high level of consistency between the text prompts and the resulting images, underscoring its robustness in addressing the challenge of inconsistent multi-entity representation. 
conclusion: 
 In this paper, our study addressed the challenge of IMR in diffusionbased text-to-image synthesis after analyzing the effect of entity coupling and entity prematurity. By integrating the EGGen mechanism within cross-attention operations, we effectively improved entity positioning and attribute accuracy while maintaining generation fidelity. Our approach leverages ECA layers and GEA layers to ensure precise entity isolation and attribute delineation, complemented by an LA module that mitigates the impact of these adaptations over successive generation steps. Through rigorous testing on T2I-CompBench and detailed visual case studies, our method demonstrates a substantial enhancement in handling complex multi-entity prompts, providing a promising avenue for future research in advanced image synthesis.  
: 
 A blue/red/ orange/yellow/ white/pink rose [7] [8] A white cloud [9] A blue sky BG: A green garden Ours GliGen [1] A golden dog [2] A black cat [3] [4] A green tree [5] A red house BG: in green grass Diffusion models have shown remarkable prowess in text-to-image synthesis and editing, yet they often stumble when tasked with interpreting complex prompts that describe multiple entities with specific attributes and interrelations. The generated images often contain inconsistent multi-entity representation (IMR), reflected as inaccurate presentations of the multiple entities and their attributes. Although providing spatial layout guidance improves the multientity generation quality in existing works, it is still challenging to handle the leakage attributes and avoid unnatural characteristics. To address the IMR challenge, we first conduct in-depth analyses of the diffusion process and attention operation, revealing that the IMR challenges largely stem from the process of cross-attention mechanisms. According to the analyses, we introduce the entity guidance generation mechanism, which maintains the integrity of the original diffusion model parameters by integrating plug-in networks. Our work advances the stable diffusion model by segmenting comprehensive prompts into distinct entity-specific prompts with bounding boxes, enabling a transition from multi-entity to singleentity generation in cross-attention layers. More importantly, we and varied, based on text prompts. Nevertheless, despite their outstanding performance, diffusion models like Stable Diffusion [22] sometimes face challenges in accurately interpreting prompts when they involve complex arrangements of multiple entities. These issues mainly manifest as entity position disorder, attribute leakage (i.e., misallocated attributes), and inaccurate presentation of entities (e.g., missing or redundant entities), as illustrated in Figure . 2. These issues collectively lead to a mismatch between the intended multi-entity compositions of the prompts and the actual image outputs, highlighting the challenge of Inconsistent Multi-entity Representation (IMR).Previous methods address the IMR issue mainly by relying on predefined bounding boxes (i.e., layout) to constrain the multi-entities' position and number at image spatial domain [1, 12, 13, 15, 26, 30] . While these methods afford SD models the capability to take care of the entities' specific positions and achieve improvement in the results, it is still challenging to achieve natural entity placement and precious attribute allocation and presentation without leakage. Although the bounding box-based hard assignment gives a direct restriction on the coordinates of the entity in the spatial domain, the interactions (e.g., the cross-attention operations) of the long/complex text prompts and visual representations are still handled as a whole, leading to mixture and confusion in the results [12] , as discussed in Figure 3 . Directly applying bounding box-based constraints on the image spatial domain may also result in unnatural artifacts on the generated images [13, 30] .The objective of our research is the generation of multiple entities from complex text descriptions with bounding boxes, enhancing precision by addressing IMR issues in generative models through fine-tuning adaptors while keeping the parameters of the original diffusion model fixed. To understand potential IMR issues during generation, we first analyze the cross-attention operations of text-based generative models, conducting detailed analyses of token-wise and step-wise cross-attention maps (see Sec. 3). The token-wise analysis indicates that the prompt tokens for different entities and their specific attributes are aggregated together to control the generation of visual representations in SD, which can easily lead to an entity coupling phenomenon marked by mismatched entity types and attributes blending across entities when the prompts are complex. Our step-wise analysis of the diffusion process reveals the problem of entity prematurity -the visual patterns, e.g., the positioning and characteristics of entities, are usually established prematurely in the early steps of the diffusion process, with low-resolution attention maps. The improperly mixed tokens of different entities (because of the entity coupling issue) can lead to improper attention maps at an early stage (e.g., cross attention at first several steps), resulting in generated images with IMR, due to the prematurity.Building on the outlined observations, we introduce an Entity Guidance Generation (EGGen) mechanism to address IMR issues within cross-attention layers. To handle the complex prompts including descriptions of multiple entities, we first strategically segment a comprehensive prompt into distinct entity-specific prompts with bounding boxes by the LLM, facilitating a shift from multientity to single-entity generation within cross-attention layers. To counteract entity coupling, we introduce Entity-centric Cross-Atention (ECA) layers focused on individual entity prompts instead of the general cross-attention operation, thereby safeguarding each entity's uniqueness and correctness of the type. Simultaneously, Global Entity Alignment (GEA) layers serve as the refinement of cross-attention maps within the standard cross-attention layers to use multi-entity priors (Holistically-Nested Edge Detection (HED) [27] ) as a ground truth for guiding accurate entity positioning and attribute delineation. Targeting the entity prematurity, a Linear Attenuation (LA) module is integrated to linearly decrease the impact of ECA and GEA layers as the step increases when inference, preventing oversaturation and ensuring generation fidelity. In our experiments, our EGGen model demonstrates precise positional control and attribute accuracy in generating multiple entities through entity guidance, as evidenced on T2I-CompBench [9] and visual case studies. Especially, Figure 1 illustrates the generation of multiple entities with precise attribute control in both realistic and cartoon styles.The key contributions of our work are summarized as follows: • We explored the underlying causes of IMR issues through the study of token-wise and step-wise attention maps, identifying the effect of entity coupling and entity prematurity. • Our EGGen model advances a stable diffusion approach by segmenting comprehensive prompts into entity prompts with bounding boxes, transferring the multi-entity generation to singleentity generation within cross-attention layers. • Combined with ECA, GEA, and LA, our EGGen model achieves precise positional control and attribute accuracy in the generation of multiple entities. 
 
 Appendix:  


Decision: This paper addresses the issue of inconsistent multi-entity representation (IMR) in the generated images of text-to-image diffusion models. All reviewers agreed that the problem is interesting, the proposed method to mitigate the IMR problem is effective, and the paper is well-written and easy to follow. While the reviewers noted that the technical novelty of the proposed method is not very high and there is room for improvement in its effectiveness, they concur that the paper meets the acceptance threshold.

It is strongly recommended that the authors incorporate the reviewers' feedback and rebuttal discussion into the final version of the paper to further enhance its comprehensiveness and quality.

Reviews: review: paper_summary: paper_summary: The paper proposes to address the inconsistent multi-entity representation (IMR) challenge. Specifcially, it first reveals that the IMR challenges largely stem from the process of cross-attention mechanisms and introduces the entity guidance generation mechanism by integrating plug-in networks. The mechanism segments comprehensive prompts into distinct entity-specific prompts with bounding boxes, enabling a transition from multi-entity to single-entity generation cross-attention layers. It further introduces entity-centric cross-attention layers that focus on individual entities to preserve their uniqueness and accuracy, alongside global entity alignment layers that refine cross-attention maps using multi-entity priors for precise positioning and attribute accuracy. Additionally, a linear attenuation module is integrated to progressively reduce the influence of these layers during inference, preventing oversaturation and preserving generation fidelity. Extensive empirical results on the benchmark datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: + The paper proposes to decompose the multi-entity generation problem into single-entity generation sub-problems, which is interesting.
+ Extensive experiments demonstrate the effectiveness of this approach.
+ The ablative studies are thorough.
+ The presentation is clear and easy to follow.
limitations: limitations: Not necessarily limitations, mostly open-ended questions.
- It seems that the number of objects in the generated images (especially in the qualitative results) are mostly 2. How well does the proposed method work on cases with more than 2 objects?
- How well does the proposed method work on negation of a given concept, e.g., not red?
- I would like to hear about other reviewers' opinions regarding the limitations.
paper_summary: paper_summary: This paper first analyzes the problem of entity coupling and entity prematurity of existing text-to-image, which the author classifies as multi-entity representation (IMR) challenge. To solve these problems, the author proposes entity guidance generation mechanism, which is a divide-and-conquer mechanism, uses LLM to split text prompts and regions for input. In addition, three modules are proposed in this paper, such as ECA, GEA, and LA to achieve precise positional control and attribute accuracy in the generation of multiple entities.
strength_weakness: strength_weakness: 1、This is a well-written paper, the questions raised are interesting and the technical points are easy to understand.

2、In this paper, a large number of experiments and analyses have been carried out to prove that the existing text-to-image generation method (SDv1.5) has entity coupling and entity prematurity problems, and the source of the problems is cross-attention.

3、This paper proposes a diffusion model of text-generated images for multi-entity generation. Its core idea is to decompose text prompt and regions to ensure the location and generation of a single entity.

4、This paper innovates the network structure，where the ECA for safeguarding each entity's uniqueness and correctness of the type, the GEA for guiding accurate entity positioning and attribute delineation, and the LA for preventing oversaturation and ensuring generation fidelity.
limitations: limitations: 1、Although this article analyzes and summarizes the existing issues of entity coupling and entity prematurity, its technological innovation is slightly weaker.
The proposed model complements GLIGEN(CVPR2023) by adding two main modules ECA and GEA, as well as a loss function hed loss. However, the two modules are similar in implementation: Formula 5 and formula 6 follow the same computation format, the difference is that the former uses attention to compute features, the latter uses convolution to compute features, and then their results are dot-multiplied with masks.

2、The method of controlling the position of multiple objects in an image using decomposed text prompt and segmented masks has been used in DenseDiffusion (ICCV2023). This is similar to a divide-and-conquer mechanism proposed by the authors, who can include this reference in the paper.

3、The author uses a large number of visual experiments to analyze the problems of the existing methods, but does not demonstrate the effectiveness of the proposed module through visual experiments.

4、From the results of quantitative comparison, the proposed method has poor shape effect. However, the author used edge image as shape supervision for training, and did not conduct a separate ablation experiment to verify hed loss. The authors need to supplement the ablation experiments of hed loss to verify the role of this loss.

5、In Figure 9, I think the text prompt is the same as in Figure 1, but I find that there are a lot of messy flowers in the background of ECA+LA and ECA+GEA, which should be "A green garden". Why is this phenomenon? Are there any other examples of ablation experiments?
paper_summary: paper_summary: This paper studies the the problem of inconsistent multi-entity representation (IMR) in text-to-image diffusion model, which is a valuable research challenge. The authors proposed to integrate the EGGen mechanism within cross-attention operations, which effectively improved
entity positioning and attribute accuracy while maintaining generation fidelity compared to the baseline method.
strength_weakness: strength_weakness: 1. The studied problem is interesting and the authors proposed a relative effective method to alleviate the IMR problem.
2. Experimental results show less attribute leakage of the proposed methods compared to the other methods.
3. The paper is well organized and easy to read.
limitations: limitations: 1. The generated images are not very realistic in fig.1, fig.7, etc. I think the main reason lies in the generating capabilities of the base model SD. It would be better to verify the proposed method on a more advanced model, e.g. sdxl.
2. In fig.7, attribute leakage still exists in the generated images, see the red and blue wall in the background of the second case.
3. This paper introduces HED soft edge as the prior guidance. Other method like Controlnet uses canny or HED edges to provide accurate location relationships. What's the difference between the proposed $L_{hed}$ with controlnet?
paper_summary: paper_summary: This article investigates multi-object generation, which is a significant research focus and also a current weakness in the field of visual generation. Stable diffusion is constrained by the encoder of CLIP, which can only extract the main part of the sentence, making it difficult to understand multiple objects. This article effectively addresses this issue.
strength_weakness: strength_weakness: The motivation is clear, the proposed method is effective, and it tackles a pain point problem.
The experiments are thorough.
limitations: limitations: If multi-object generation could be achieved solely at the level of stable diffusion without relying on large models, it might be a more meaningful endeavor. 
The use of cross-attention editing methods has been extensively used in previous papers. While effective for the task at hand, its application in this context may lack inspiring significance for future research or substantial structural innovation. 


In other words, while multi-object generation is a significant problem, addressing it at the level of CLIP or stable diffusion would be more impactful. However, employing a combination of large models and cross-attention may lead to some shortcomings in this work.
So I tend to give a Borderline Accept.