GOAL: Grounded text-to-image Synthesis with Joint Layout Alignment Tuning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 11:44:13

Abstract: Recent text-to-image (T2I) synthesis models have demonstrated intriguing abilities to produce high-quality images based on text prompts. However, current models still face Text-Image Misalignment problem (e.g., attribute errors and relation mistakes) for compositional generation. Existing models attempted to condition T2I models on grounding inputs to improve controllability while ignoring the explicit supervision from the layout conditions. To tackle this issue, we propose Grounded jOint lAyout aLignment (GOAL), an effective framework for T2I synthesis. Two novel modules, Discriminative semantic alignment (DSAlign) and masked attention alignment (MAAlign), are proposed and incorporated in this framework to improve the text-image alignment. DSAlign leverages discriminative tasks at the region-wise level to ensure low-level semantic alignment. MAAlign provides high-level attention alignment by guiding the model to focus on the target object. We also build a dataset GOAL2K for model fine-tuning, which composes 2000 semantically accurate image-text pairs and their layout annotations. Comprehensive evaluations on T2I-Compbench, NSR-1K, and Drawbench demonstrate the superior generation performance of our method. Especially, there are improvements of 19%, 13%, and 12% in color, shape, and texture metrics for T2I-Compbench. Additionally, Q-Align metrics demonstrate that our method can generate images of higher quality.

Main paper: 
 introduction: 
 Recently, text-to-image (T2I) diffusion models have made tremendous progress in generating high-fidelity and diverse images in response to textual prompts [8, 13, 19, [40] [41] [42] . However, models still struggle with the Text-Image Misalignment problem [4, 9, 15, 29] for compositional T2I generation, which means they often fail to compose multiple objects with various attributes (e.g., color, shape, texture) and complex spatial relations, as shown in the failure cases of GLIGEN [26] in Figure 1 .Previous works exploring compositional T2I generation can be classified into two main groups: training-free [9, 21, 34] and training-based approaches [26, 42, 52] . Specifically, training-free methods [9, 21] focus on directly altering the latent and crossattention maps or adding objects by continuous editing [27, 53] . For instance, Attention Refocusing [9] employs inference-time optimization on the cross-attention map to align intermediate outputs with the layout conditions. However, it adds considerable cost during inference and may lead to image quality degradation, as shown in the last column of Figure 1 .Meanwhile, other training-based methods [42, 52] incorporate additional modules for controllable image generation. For example, as shown in Figure 2 (b), GLIGEN [26] integrates grounding information into designed gated self-attention layers, supporting existing T2I models on layout inputs. Nevertheless, these layoutaware models ignore inherently explicit supervision in layout conditions, leaving them only as guidance for estimating the added global noise during training. We hypothesize that this objective is insufficient for complex generation tasks. Therefore, a natural question arises: Can we find fine-grained and explicit supervision with layout conditions as auxiliary training objectives to enhance text-image alignment? To achieve this, we adopt multiple phraseregion pairs in layout conditions as 'ground truth' labels, allowing the diffusion model to learn region-wise information within the image's context in an atomistic manner. Given that annotation of the "ground-truth" labels (e.g., layout) from human is costly, we borrow large language model's (e.g., GPT-4 [2] ) strong scene understanding ability for layout planning.In this work, we propose Grounded jOint lAyout aLignment (GOAL), an effective framework that utilizes layout conditions to provide explicit region-wise supervision for text-image alignment. Considering that a more detailed denoised image can provide richer information for semantic alignment, we refine image details by applying region-wise MAAlign in high-level feature space. This process involves shifting attention towards the target region while suppressing the attention of unrelated areas. As a result, sharper and more detailed images are generated, making semantic alignment more effective and further enhancing text-image alignment. By conducting extensive experiments, the model exhibits significant improvement on several benchmarks, including T2I-Compbench [20] , NSR-1K [16] and Drawbench [42] . Importantly, these enhancements are achieved without incurring additional inference costs. Along with our proposed training framework, we also publish a curated dataset GOAL2k for training, which consists of 2000 multimodal samples with layout annotation for improving text-image alignment especially in complex scenes. Moreover, images included in GOAL2k are generated by outstanding T2I model DALLE-3 [39] , demonstrating both high-detail and semantically accurate characteristics.The main contributions of this paper are summarized as follows:• We propose a Grounded jOint lAyout aLignment (GOAL), which is a novel layout-aware training framework. Discriminative semantic alignment (DSAlign) and masked attention alignment (MAAlign) are incorporated in this framework to improve the text-image alignment. • We build a dataset GOAL2K to study the effectiveness of our alignment-based objectives, which composes 2000 semantically accurate image-text pairs and their layout annotations for model fine-tuning. • We conduct comprehensive experiments on existing methods of T2I generation on T2I-Compbench, NSR-1K and Drawbench, and show that our method compares favorably against the state-of-the-art models.  
related works 2.1 text-to-image models: 
 Early methods to text-to-image (T2I) generation primarily relied on Generative Adversarial Networks (GANs) [1, 12, 38, 44, 51] . However, recent advancements have shifted the focus towards diffusion models [8, 13, 19, 32, 45] , which have gained prominence due to their exceptional capabilities in generating high-quality images.The Denoising Diffusion Probabilistic Model (DDPM) [19, 33, 46] introduces standard noise in the forward process and reconstructs the image from noise in the reverse process. Unlike denoising in the pixel space, the Latent Stable Diffusion Model (LDM) [40] conducts the denoising process in latent space, significantly reducing computational costs. Due to this remarkable progress, LDM has found wide application across various tasks, including image editing [7, 11, 18] , image super-resolution [24, 43] , inpainting [31, 50] and semantic segmentation [3, 6] . Building upon these state-of-theart models, our work aims to enhance the capabilities of text-image alignment according to given layouts.Despite the remarkable image generation capabilities demonstrated by Diffusion models, they encounter challenges in compositional generation, particularly within complex scenes. Recent methods [9, 34] primarily tackle this issue during the inference stage by incorporating layout-aware attention supervision at specific steps.BoxDiff [49] controls the noise map by adjusting cross-attention and self-attention layers. Attend-and-Excite [9] improves the generation of missing objects by maximizing the attention score for each object. Attention Refocusing [9] employs guidance functions to align intermediate outputs with layout conditions. Paint-withwords [5] enhances the cross-attention scores between image and text tokens corresponding to the same object based on segmentation masks. Continuous Layout Editing [53] disentangles various object concepts and facilitates continuous editing to align images with layouts. However, these methods incur computational costs during inference. Moreover, due to the direct optimization in the attention map without training, they may cause image distortion.Several works aim to improve text-image alignment by fine-tuning diffusion models [10, 14, 26, 52] to integrate layout conditions. GLIGEN [26] integrates grounding information by the gated selfattention mechanism, enabling existing pre-trained T2I diffusion models to be conditioned on grounding inputs. LayoutLLM-T2I [36] introduces a relation-aware attention module, integrating semantic relation to generate high-fidelity images. Frido [14] performs multi-scale coarse-to-fine denoising to generate images of complex scenes. ReCo [52] incorporates spatial coordinates to achieve precise region control for arbitrary objects. Inspired by these works, we propose layout alignment at the semantic and attention level simultaneously to effectively fine-tune pre-trained T2I models without non-negligible costs. 
method 3.1 preliminaries: 
 Latent Diffusion Models (LDM) [40] are widely used in conditional image generative tasks. Given an image 𝑥 0 ∈ R 𝐻 ×𝑊 ×3 , VAE E is adopted to encode image into latent space as 𝑧 0 = E (𝑥 0 ). Then Gaussian noise 𝜖 is added to the latent 𝑧 0 with a randomly sampled timestep 𝑡, yielding 𝑧 𝑡 as:EQUATIONwhere 𝛼 𝑡 defines the level of noise. To achieve conditional image generation, the caption describing the image is encoded by text encoder 𝜑 as the text embedding 𝜑 (𝑦) and then injected into the U-Net via cross-attention. Then U-Net 𝜖 𝜃 is trained to predict the added noise 𝜖, following the objective as:EQUATION)To ground the generation process via the additional condition, GLIGEN [26] incorporates the semantic information of grounding entity and spatial configurations through gated self-attention as:EQUATION)where 𝜈 is the visual feature, 𝛾 is a learnable scalar which is initialized as 0 and ℎ is the added condition such as layout. Following GLIGEN [26] , we optimize gated self-attention for stable training.In the current layout-aware LDM, which incorporates additional layout conditions and optimizes the loss function L 𝐿𝐷𝑀 for noise prediction, the relationships within the bounding box, such as attributes and objects, are not explicitly optimized. As depicted in Figure 1 , this results in poor semantic-level alignment in specific regions. Particularly, when different layouts overlap, the fusion of multiple conditions may result in a dissonant visual composition. Therefore, we propose discriminative semantic alignment (DSAlign), leveraging discriminative tasks at the region-wise level to refocus on refining the generated context. Specifically, with the noise predictor 𝜖 𝜃 and 𝑧 𝑡 , we reconstruct the latent noise 𝑧 0 via a single denoising step using the reverse of Equation 1, and then obtain the denoised version of the clean image 𝑥 0 by VAE decoder D. The formulation is:EQUATIONGiven the spatial layout defined by 𝐵 bounding boxes 𝑏 𝑖 ∈ Z + 1×4 , where 𝑖 ∈ [0, 𝐵), we perform DSAlign by minimizing the distance between the image embeddings of the target region and the corresponding phrase 𝑝 𝑖 describing the region of 𝑏 𝑖 . To crop the region from the image, we construct the 𝑀 𝑖 from the box 𝑏 𝑖 and obtained the masked image as x (𝑡 ) 0 • 𝑀 𝑖 , which is then encoded by the image encoder of CLIP, denoted by 𝐶𝐿𝐼𝑃 𝑖𝑚𝑔 . We adopt the text encoder from CLIP, denoted by 𝐶𝐿𝐼𝑃 𝑡𝑒𝑥𝑡 , to encode the corresponding phrase captions 𝑝 𝑖 . Then a normalization operation is performed to align the two embeddings within a unified semantic space. This process is formulated as follows:EQUATIONp𝑖 = 𝑛𝑜𝑟𝑚(𝐶𝐿𝐼𝑃 𝑡𝑒𝑥𝑡 (𝑝 𝑖 )).(6) Subsequently, the distance between the two normalized embeddings is computed using spherical distance as follows:EQUATION)A blue dog is chasing a frisbee in the air.A red banana is placed on a worn-out book. The average distance between all the 𝐵 target regions and the corresponding phrases is:EQUATIONwhere 𝐷 𝑙𝑜𝑐𝑎𝑙 represents the local-level semantic alignment guidance between target regions and local phrases. Moreover, we evaluate the global-level semantic alignment between the entire image and the text as follows:EQUATIONwhere ȳ = 𝑛𝑜𝑟𝑚(𝐶𝐿𝐼𝑃 𝑡𝑒𝑥𝑡 (𝑦)) and x (𝑡 ) 0 = 𝑛𝑜𝑟𝑚(𝐶𝐿𝐼𝑃 𝑖𝑚𝑔 ( x (𝑡 ) 0 )). To ensure the text-image alignment, we directly optimize local-level and global-level semantic alignment by L 𝐷𝑆𝐴 :EQUATIONGiven that a more detailed denoised image can provide richer information for semantic alignment performed by DSAlign, we refine image details through region-wise masked attention alignment (MAAlign) in high-level feature space. This process involves shifting attention towards the target region while suppressing the attention of unrelated areas. Specifically, for each cross-attention layer, let 𝑄 ∈ R ℎ𝑤×𝑑 be the input intermediate feature to the cross attention layer, which is obtained from the feature map of size ℎ × 𝑤 with feature dimension 𝑑, and let 𝐾 ∈ R 𝑛×𝑑 be the transformed text embedding with 𝑛 tokens via a linear map, we can obtain cross-attention map 𝐴 𝑡 at the step 𝑡 as follows:EQUATION)To ensure high responses within the masked regions, we shift attention towards the target regions by boosting the values of the masked attention map:EQUATIONwhere 𝑗 ∈ [0, 𝑛) is the token index describing the content of 𝑀 𝑖 , and 𝐴 𝑡 𝑗 represents the corresponding attention map. Moreover, we suppress attention on irrelevant areas by preventing attention from extending beyond the target regions:EQUATIONThe overall masked attention alignment loss is defined as:EQUATIONIn this way, MAAlign provides attention alignment by guiding the model to focus on the target object. Finally, the training objective is as follows:EQUATIONwhere 𝛼 and 𝛽 are trade-off parameters to balance the contributions of the L 𝐷𝑆𝐴 and L 𝑀𝐴𝐴 . Figure 4 illustrates the attention map 𝐴 𝑡 and denoised image x (𝑡 ) 0 with various combinations of loss terms. It's clear that when employing L 𝐷𝑆𝐴 (3rd row), the cross-attention maps exhibit a closer alignment with the real image. Furthermore, when adding L 𝑀𝐴𝐴 alongside L 𝐷𝑆𝐴 (4th row), the maximum value of the attention map is boosted, indicating an enhanced focus on the target object, thus resulting in a clearer denoised image. This enhances the effectiveness of DSAlign and further improves text-image alignment. 
experiment 4.1 dataset construction: 
 To study the effectiveness of our alignment-based objectives, we construct a compact yet robust dataset called GOAL2k to fine-tune our model. Initially, we first select common objects from COCO dataset [28] . Subsequently, leveraging in-context learning, we devise various templates encompassing different categories (e.g., color, shape and spatial). Then we employ GPT-4 [2] to generate relational captions according to the templates and objects, resulting in a set of 1,000 template-based prompts. Additionally, we directly choose 1,000 captions from COCO dataset [28] that contain multiple objects or spatial relations as natural prompts. To guarantee the high quality and semantic accuracy of images incorporated in GOAL2K, we employ DALLE-3 [39] for generating images in the training set. Subsequently, we utilize GroundingDINO [30] to produce layout annotations. Ultimately, we manually verify the generated imagetext pairs alongside the layout annotations to ensure their fidelity to accurate semantic information. More details about dataset construction can be found in supplemental materials.Building upon existing works [16, 20, 35] , we evaluate the precision of layout-to-image generation across various widely-adopted benchmarks, such as T2I-CompBench [20] , NSR-1K [16] and Drawbench [42] . For the evaluation of text-image alignment, we adopt the recommended protocols in T2I-CompBench [20] . For NSR-1K [16] , we utilize cross-modal similarities, indicated by CLIP score [37] , and detection-based similarities, denoted by GLIP score [25] .As Drawbench [42] doesn't provide labels or metrics for automatic assessment, we conduct evaluator-based evaluation via user study.To evaluate the image generation quality, we calculate Q-Align [48] as image quality score and aesthetic score. Additionally, PickScore [22] is employed to evaluate human preferences regarding the quality of generated images. Furthermore, NSR-1K [16] is also utilized to evaluate the accuracy of the layout generated by different textto-layout models [2, 17, 47] across spatial and counting scenarios.Our experiments are performed using GLIGEN [26] , a popular layout-to-image diffusion model built upon Stable Diffusion v1.4 [40] . GPT-4 [2] is employed for layout planning during inference. We use L 𝐷𝑆𝐴 and L 𝑀𝑆𝐴 as alignment-based objectives alongside the original denoising objective L 𝐿𝐷𝑀 to train our model. The weights (𝛼 and 𝛽) for these objectives are set as 1e-2 and 1e-3, respectively. CLIP ViT-L/14 [37] is employed in discriminative semantic alignment (DSAlign) as the encoder for the image and text. We select the attention map with size 16 × 16 obtained from the decoder of the U-Net to perform masked attention alignment (MAAlign).We employ the AdamW optimizer with a constant learning rate of 5e-5 over 14,000 steps, with a batch size of 1. During training, we fine-tune the gated self-attention layers following GLIGEN [26] . Additionally, to fully exploit DSAlign and MAAlign, we fine-tune both self-attention and cross-attention layers.To evaluate the text-image alignment, we compare the proposed method with other state-of-the-art methods in two aspects: with text-to-image (T2I) methods such as Stable Diffusion v1.4 [40] , DPT [35] , Attn-Exct v1 [9] , and layout-to-image methods such as GLI-GEN [26] , LayoutLLM-T2I [36] , LayoutGPT [16] on T2I-Compbench [20] . Table 1 illustrates that our proposed method surpasses all other methods in all metrics. Particularly, GOAL achieves improvements of 19%, 13%, and 12% in color, shape, and texture metrics respectively, compared to GLIGEN. This verifies that discriminative tasks utilized by DSAlign could ensure region-wise semantic alignment between objects and corresponding attributes, and attention alignment performed by MAAlign allowing the model focusing on the target object, which is a prerequisite for attribute binding. Additionally, we present the results of counting and spatial evaluation on the NSR-1K [16] benchmark in Table 2 . The improvement, particularly in the GLIP [30] score (4% in counting and 3% in spatial metrics), further demonstrates the effectiveness of our method. From Table 1 and Table 2 , we observe that layout-to-image methods outperform T2I methods in spatial and counting relationships by a large margin. This illustrates that employing layouts as an intermediate representation can significantly improve the controllable generation, particularly in scenarios involving counting and spatial relationships.Moreover, we demonstrate the results of the proposed method compared with training-free methods such as Structure v2 [15] , BoxDiff [49] , Attend-and-Excite [9] , and Attention Refocusing [34] Table 1 : Comparison to the state-of-the-art text-to-image methods and layout-to-image methods on T2I-Compbench.Color on T2I-Compbench. Additionally, the inference time for generating an image is presented in Table 3 . While these methods do not need additional training, they incur considerable computational costs during inference. For instance, Attention Refocusing [34] and Attend-Excite [9] take ×2.61 and ×2.52 times longer, respectively, to generate a single image compared to the proposed method. Our proposed method enables efficient inference and consistently outperforms baseline methods across all evaluation metrics. Beyond the above evaluation, we also assessed the quality of images generated by different models using the PickScore [22] test-unique set. Results suggest that training-free methods, which directly intervene with cross-modal attention and latent noise maps during sampling, may degrade image quality. For instance, the PickScore of Attend-Excite [9] decreased by 3% compared to its baseline model Stable Diffusion v1.4 [40] . In contrast, our proposed method demonstrates exceptional performance for both text-image alignment and image quality. Figure 5 demonstrates some cases from the T2I-Compbench [20] and NSR-1K [16] benchmarks across attributes such as color, spatial layout, shape, counting, and texture. We observe that 1) Layout conditions, serving as an intermediate representation, can enhance the generation controllability in scenes involving counting and spatial relationships. As shown in the last row, the T2I models Stable Diffusion [40] and Attend-and-Excite [9] fail to generate the correct number of animals, while other layout-to-image models achieve better control over the counting through guidance from the layout. 2) Our proposed method demonstrates outstanding performance across all attributes, generating images that faithfully capture semantic details. This can be attributed to the incorporation of DSAlign and MAAlign, which ensure region-wise semantic and attention alignment, respectively. 3) Compared with other methods, our proposed method can generate high-quality and aesthetic images, which verifies the improvement in text-image alignment does not result in a loss of image quality.Effect of loss terms. To explore the contribution of the loss terms L 𝐷𝑆𝐴 and L 𝑀𝐴𝐴 , we conduct experiments in T2I-Compbench [20] including color, shape, texture, spatial metrics. From Table 5 , we could see that both L 𝐷𝑆𝐴 and L 𝑀𝐴𝐴 could consistently promote alignment performance for T2I gengeration. Specifically, L 𝐷𝑆𝐴 shows considerable improvement in shape and texture metrics, while L 𝑀𝐴𝐴 exhibits significant enhancement in color metrics. It can be attributed to the fact that discriminative loss provided by semantic alignment improves shape and texture metrics, whereas attention-level alignment guides the model to focus on the target We demonstrate the effectiveness of the proposed method in text-image alignment compared with Stable Diffusion v1.4 [40] , GLIGEN [26] , LayoutLLM-T2I [36] , BoxDiff [49] , Attend-and-Excite [9] and Attention Refocusing [34] . Effect of different cross-attention layers. We analyze the effectiveness of employing different cross-attention layers with MAAlign as shown in Table 7 . It's clear that MAAlign is most effective when applied to the 16 × 16 attention map obtained from the U-Net decoder, which is consistent with findings from previous works [18, 25] .To evaluate the performance of text-to-layout models, we examine the transformer-based model LayoutTransformer [17] , as well as the latest large language models (LLMs) including GPT-4 [2] , GPT-3.5 [2] , Llama 2-7B [47] , and Llama 2-13B [47] . We assess their ability to comprehend visual concepts through spatial and counting scenes in the NSR-1K benchmark. From table 8, we know that the GPT-4 outperforms all other models in both spatial and counting scenes, thus it is employed for layout planning in this work. Moreover, we observe that LLMs exhibit significantly superior performance compared to LayoutTransformer, highlighting its robust cross-modal spatial reasoning abilities. Comparing Table 2 and Table 8 , we observe that the text-to-layout process is proficient, and the bottleneck of image generation in complex scenes primarily lies in layout-guided image control.While quantitative metrics have limitations in providing a comprehensive assessment, we complement our analysis with a user study. We selected attributes including color, description, counting, position, and conflicts from Drawbench [42] , resulting in 94 prompts. For each prompt, two images were generated by different models, which were then assigned to 20 individuals for evaluation. Evaluation of the images by the participants focused on two aspects: semantic alignment and aesthetic quality. Semantic alignment is utilized to assess whether the model can generate images that faithfully capture the semantic details from the input text prompts. Aesthetic quality is employed to determine if the images generated by the model exhibit any incoherent parts or unnatural poses.Our method is compared to several baseline models including Stable Diffusion v1.4 [40] , GLIGEN [26] , Attend-and-Excite [9] , and Figure 7 : Comparison with the training-free layout-aware method Attention Refocusing [34] .Attention Refocusing [25] . Figure 6 shows that human preferences are consistent with our evaluation outcomes both in text-image alignment (e.g., Table 1 ) and aesthetic quality (e.g., Table 4 ), with 28.1% and 50.36% improvements compared to GLIGEN in alignment and quality respectively. Additionally, Figure 7 demonstrates that some images generated by Attention Refocusing [25] exhibit incoherent parts and unnatural poses, possibly attributed to the direct intervention of latent noise maps during sampling. In contrast, the proposed method enhances alignment without compromising image quality. 
conclusion: 
 In this work, we propose Grounded jOint lAyout aLignment (GOAL) framework to handle Text-Image Misalignment issues in complex scenes. Discriminative semantic alignment (DSAlign) and masked attention alignment (MAAlign) are performed to provide explicit supervision with the layout conditions. In addition, We build a dataset GOAL2K to study the effectiveness of our alignment-based objectives, which composes 2000 semantically accurate image-text pairs and their layout annotations for model fine-tuning. Extensive experiments demonstrate that the proposed method achieves stateof-the-art performance on different benchmarks with improved image quality.  
anonymous authors: 
 A girl, dressed in a yellow long skirt, is strolling along the beach with a man clad in a blue T-shirt. The sand is marked with a blue heart.A dog is teaching math in a classroom, with the blackboard written with some mathematical formulas and the classroom packed with students. Figure 1 : Given a user-specified text prompt and layout condition of complex scenes, baseline models (e.g., GLIGEN [26] ,Attention Refocusing [34] ) fail to solve text-image misalignment problems such as attribute errors and relation mistakes. Our method exhibits outstanding generation performance for text-to-image synthesis task, which generates images faithfully capturing the details of text prompts.this issue, we propose Grounded jOint lAyout aLignment (GOAL), an effective framework for T2I synthesis. Two novel modules, discriminative semantic alignment (DSAlign) and masked attention alignment (MAAlign), are proposed and incorporated in this framework to improve the text-image alignment. DSAlign leverages discriminative tasks at the region-wise level to ensure low-level semantic alignment. MAAlign provides high-level attention alignment by guiding the model to focus on the target object. We also build a dataset GOAL2K for model fine-tuning, which composes 2000 semantically accurate image-text pairs and their layout annotations. Comprehensive evaluations on T2I-Compbench, NSR-1K, and Drawbench demonstrate the superior generation performance of our method. Especially, there are improvements of 19%, 13%, and 12% in color, shape, and texture metrics for T2I-Compbench. Additionally, Q-Align metrics demonstrate that our method can generate images of higher quality. 
 
 Appendix:  


Decision: The initial ratings are 2 BR, 1 BA, and 1 WA. After rebuttal, they are 1 BR, 2 BA, and 1 WA. Most reviewers appreciated the proposed method about text-guided layout-to-image generation. One reviewer is still not satisfied with the authors' responses for the explanation to the loss. The concern, however, is minor compared to the contribution of the paper. The decision is, therefore, to accept the work. The authors are encouraged to incorporate the material from their response to the camera ready version.

Reviews: review: paper_summary: paper_summary: The paper introduces the Grounded jOint lAyout aLignment (GOAL) framework, designed to address the Text-Image Misalignment problem in complex scene generation. GOAL incorporates two innovative modules: DSAlign focuses on low-level semantic alignment through discriminative tasks at the regional level, while MAAlign directs the model's attention towards the target object in the high-level feature space. They also created a dataset called GOAL2K to facilitate model fine-tuning. Extensive evaluations on various benchmarks. Experiments show substantial improvements over existing methods in color, shape, and texture metrics, highlighting GOAL's effectiveness in generating high-fidelity and semantically accurate images.
strength_weakness: strength_weakness: 1. This paper designs two types of loss, which guide the generation of image layouts from the low level semantic and high level feature space semantic levels, and the calculation process do not rely on additional models. The framework is plug-and-play and is suitable for most text-to-image diffusion models.
2. The comparative experiment demonstrated in detail the advantages of this method compared to the previous training-free method, and showed the improvement in effectiveness after being applied to the layout-to-image method. Additionally, the ablation experiment is sufficient, involving loss combinations at different levels.
limitations: limitations: 1. The paper adopts the CLIP model to adjust the low level semantics, which is based on single step denoising. However, the difficulty of denoising under different initial noise and text guidance varies. Therefore, the information of early single step denoising may be relatively blurry (being out of distribution for CLIP models) or even semantically unclear, and the loss calculated based on this may produce unstable effects.
2. DSAlign and MAlign loss based on feature space design are similar to previous methods, such as Training Free Layout Control with Cross Attention Guidance and Grounded Text to Image Synthesis with Attention Refocusing. Their core is to control the corresponding areas of graphics and text within the designated areas of the layout, which cannot be considered as a very innovative contribution.
paper_summary: paper_summary: This paper proposes a framework for text-guided layout-to-image generation to address misalignment when generating complex scenes. To that end, a dataset called GOAL2k is constructed for fine-tuning. The model builds on GLIGEN and is extended through discriminative semantic alignment and masked attention losses. Extensive comparison with existing models visually and quantitatively demonstrates the approach's effectiveness.
strength_weakness: strength_weakness: - The paper is well-written and structured. Figures clearly visualize the approach and results.
- The proposed modules are well motivated and ablated.
- The results are convincing and many baselines are used for comparison.
limitations: limitations: Major:
- It looks like there is missing discussion and comparison with some of the mentioned related work [21], [42], [27], [53]
- Masked latents / cross-attention maps are already proposed in [27]; how are they similar/different to MAAlign in this paper?
- What is the intuition/motivation for the spherical distance that works best?
- Dalle-3 is used to generate the training data. How well does it perform when used for comparison (upper bound)? 
- Are the comparisons against off-the-shelf baselines or also fine-tuned on the GOAL dataset? If they are not fine-tuned on the dataset, it is unclear whether the performance improvement over the baselines stems from the dataset or the modules.
- BoxDiff, a training-free method, compares very competitively without requiring any fine-tuning. How would it perform with fine-tuning?

Minor:
- Is there a typo between Table 3 and Table 4 on Attention Refocusing [34 vs 25]
- What are L𝑝𝑖𝑥𝑒𝑙 + L𝑓𝑒𝑎𝑡𝑢𝑟𝑒 losses? It seems that they were not introduced in the main text.

Justification for rating:
There are a few reasons for my rating despite the otherwise good impression. I am willing to improve my score upon clarification of the following:
1) The tables are not consistent. For example, some baselines appear in one table but not the other.
2) Some mentioned related work is not used as a baseline; see point above.
3) Is the masked attention alignment significantly new from the one in [27]?
4) Effect of fine-tuning: do baselines get a fair chance by also fine-tuning on the new dataset?
paper_summary: paper_summary: This paper proposes a novel framework named GOAL to handle Text-Image Misalignment issues in complex scenes. It improves the previous GLIGEN framework with two loss, $L_{𝐷𝑆𝐴}$ and $L_{𝑀𝑆𝐴}$. The proposed method optimize the model through text-image alignment in both the local masked part and the global attention map. Experimental results show the superior performance of GOAL.
strength_weakness: strength_weakness: 1. The paper alleiviate the misalignment problem in attention map through local and global text-image alignment.
2. Both the qualitative and quatitative results show that the proposed method is more effective than other compared methods.
3. Ablation study also demonstrates the effectiveness the proposed modules.
4. The paper is well organized and easy to read.
limitations: limitations: 1. In fig.5, the generating images are not so realistic, which might be due to the generation ability of the base model. The authors can improve the model performance using the other state-of-the-art models, such as sdxl.
2. In fig.5, the foreground objects and the background are not well integrated in the generating results. 
3. Using $L_{𝐷𝑆𝐴}$ and $L_{𝑀𝑆𝐴}$, is the model able to generate customized concept? For example, can the model generate a specific object at the specific location given an object image?
paper_summary: paper_summary: This paper focuses layout control of text-to-image diffusion models with bounding-box conditions. It proposes a novel pipeline (GOAL) to improve the text-to-image alignment, as well as a dataset (GOAL2k) to evaluate their results. The author conduct extensive experiments on various benchmarks and demonstrated state-of-the-art performance.
strength_weakness: strength_weakness: Overall, this paper is well-written with clear presentation and figures. The motivation of the paper is clear: controlling the layout in diffusion models is one of the key challenges. Using a discriminative model to aid the region-wise alignment of text-to-image (T2I) models is a reasonable idea, but there is still room for improvement in terms of presentation. The experiments in this paper are comprehensive and well-support their claims. In particular, the authors demonstrate state-of-the-art quantitative results on three benchmarks, and the qualitative results appear good.
limitations: limitations: From my perspective, there are no severe issues in this paper, but there are some paragraphs that may appear unclear and can be improved.

1. In Sec 3.2 (L369-L377), the author states that overlapping bounding boxes are a challenge for previous layout control methods and proposes DSAlign to solve this issue. However, the author does not clearly present what "dissonant visual composition" is and whether this is an existing issue with references or a newly found problem with a clear definition and experiments for support. Moreover, the author did not show that their DSAlign can avoid this issue. It is true that the author showed DSAlign can ensure better region-wise semantic alignment in Tables 1, 2, and 3, but this does not necessarily mean that it avoids the specific issue caused by overlapping bounding boxes. I expect the author to provide experiments to support this claim.

2. In Sec 3.3, the author should define what "target region" and "irrelevant areas" are; otherwise, it appears unclear within its context.

3. In Sec 4.6 (L692-L694), the author states, "It can be attributed to the fact that discriminative loss provided by semantic alignment improves shape and texture metrics," which seems to lack linkage. The author should explain why the DSA loss can improve shape and texture or provide relevant references. Moreover, it seems that the $L_{pixel}$ and $L_{feature}$ in Table 5 are not defined.

4. This is not an issue, but the paper could benefit from discussing why spherical distance is the best choice, as it appears to deviate from the default cosine distance in the original CLIP.
---
Despite the above issues, the overall presentation and experiments in this paper are good. I would recommend a weak accept for this paper in its current form.