Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation
Authors: Jiaxin Cheng
Zixu Zhao
Tong He
Tianjun Xiao
Zheng Zhang
Yicong Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: Layout-to-Image; Diffusion Model; Computer Vision
Primary area: diffusion_based_models

Abstract: Recent advancements in generative models have significantly enhanced their capacity for image generation, enabling a wide range of applications such as image editing, completion and video editing. A specialized area within generative modeling is layout-to-image (L2I) generation, where predefined layouts of objects guide the generative process. In this study, we introduce a novel regional cross-attention module tailored to enrich layout-to-image generation. This module notably improves the representation of layout regions, particularly in scenarios where existing methods struggle with highly complex and detailed textual descriptions. Moreover, while current open-vocabulary L2I methods are trained in an open-set setting, their evaluations often occur in closed-set environments. To bridge this gap, we propose two metrics to assess L2I performance in open-vocabulary scenarios. Additionally, we conduct a comprehensive user study to validate the consistency of these metrics with human preferences.

Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation
Jiaxin Cheng Zixu Zhao'
Tong He
Tianjun Xiao
Zheng Zhang' Yicong Zhou?
{yc47434, 1144434,iccongoo. cocozoooou @um. edu. mo {zhaozixu, tianjux, htong, (haaaaaamzzaa com Amazon Web Services Shanghai Al Lab
University of Macau
Layouts
Ours
A green mug
BoxDiff
R&B
GLIGEN
InstDiff
A silver and black
stuinless steel mug
A yellow mug with white dots it.
TPP
A nug with horizontal red and white strip pattern
The proposed method demonstrates the ability to accurately generate objects with complex descriptions in the correct locations while faithfully preserving the details specified in the text. In contrast, existing methods such as BoxDiff [57], R&B (56), GLIGEN (25), and InstDiff (54] struggle with the complex object descriptions, leading to errors in the generated objects.
Figure 1:
Abstract
Recent advancements generative models have significantly enhanced their ca- pacity for image generation, enabling wide range of applications such as im- age editing, completion and video editing. A specialized area within generative modeling is layout-to-image (L2I) generation, where predefined layouts of ob- jects guide the generative process. this study, we introduce a novel regional cross-attention module tailored to enrich layout-to-image generation. This mod- ule notably improves the representation of layout regions, particularly sce- narios where existing methods struggle with highly complex and detailed tex- tual descriptions. Moreover, while current open-vocabulary L2I methods are trained in open-set setting, their evaluations often occur in closed-set envi- ronments. To bridge this gap, we propose two metrics to assess L21 perfor- mance in open-vocabulary scenarios. Additionally, user study to validate the consistency of these metrics with human preferences. tttaps://gitaae http://gtttuuu.ccmpppppeet wwtp//pplwx/rrcchccommo. contee http:/ottttttl///eeeee/aaaa
conduct comprehensive
Work done during his internship Amazon
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Recent years have witnessed significant advancements image generation, with the diffusion model [2] 46] emerging as a leading method. This model has shown scalability with billion-scale web training data and has achieved remarkable quality in text-to-image generation tasks 32 35| 37 39]. However, text-to-image models that rely solely on textual descriptions face limitations, particularly in scenarios requiring precise location control.
As the diversity and complexity of model training tasks increase, there is growing demand for both accuracy and precision in generated data. Precision involves more accurate object positioning, while accuracy ensures that generated objects closely match intricate descriptions, even in highly complex scenarios. Recent approaches [25] 54]] have addressed this by incorporating precise location control into the diffusion model, enabling open- vocabulary layout-to-image (L.2I generation. Among various layout types, bounding box based layouts offer intuitive and convenient control compared to masks or keypoints [39]. Additionally, bounding box layouts provide greater flexibility for diverse and detailed descriptions. In this work, we systematically investigate layout-t--image generation from bounding box-based layouts with rich context, where the descriptions for each instance to be generated can be complex, lengthy, and diverse, aiming to produce highly accurate objects with intricate and detailed descriptions.
Revisiting existing diffusion- based layout-to-image generation methods reveals that many rely on an extended self-attention mechanism [25 [4], which applies self-attention to the combined features of visual and textual tokens. This approach condenses textual descriptions for individual objects into single vectors and aligns them with image features through a dense connected layer.
However, a closer examination of how diffusion models achieve texl-to-image generation 100 32
35
37 (99
shows that text conditions are typically integrated via cross-attention layers rather than self- attention layers. Adopting cross-attention preserves text features sequences of token embeddings instead of consolidating them into a single vector. Recent diffusion models have demonstrated improved generation results by utilizing larger p 10] or multiple [35] text encoders and more detailed image captions (5]. This underscores the significance of the cross-attention mechanism in enhancing generation quality through richer text representations and more comprehensive textual information.
Drawing an analogy between the generation of individual objects and the entire image, it is natural to consider applying similar cross-attention mechanisms to each object. Therefore, we propose introducing Regional Cross-Attention modules for layout-to-image generation, enabling each object to undergo a generation process akin to that of the entire image.
In addition to the proposed training scheme for L2I generation, have identified lack of reli- able evaluation metries for open- vocabuacayy L2I generation. While models [25] [4] can perform open-vocabulary L2I generation, evaluations are typically conducted on closed-set datasets such as COCO 7 or LVIS [18]. However, such closed-set evaluations may not accurately reflect the capabilities of vocabulary L2I models, as the text descriptions in these datasets often limited to just a few words. It remains unclear whether these models can perform effectively when presented with complex and detailed object descriptions.
To address this gap, we propose two metrics that consider object-text alignment and layout fidelity that works for rich-context descriptions. Additionally, we conduct a user study to assess the reliability of these metrics and identify the circumstances under which these metrics may fail to reflect human preferences accurately.
Our contributions can be summarized as follows: 1) We revisit the training of L2I generative models and propose regional cross-attention module to enhance rich-context L2I generation, outperforming approaches. 2) To effectively evaluate the performance of open-set set L21 models, we introduce two metrics that assess the models' capabilities with rich-context object descriptions and validate their reliability through user study. 3) Our experimental results demonstrate that our proposed solution improves generation performance, especially with rich-context prompts, while reducing computational cost in each layout-conditioning layer thanks to the use of cross- attention.
existing self-attention-baut

---

box
"This cat."
box
dog
Sequenced Grouding Encoding
"This cat." dog here."
Descriptions
"null"
Regional Cross-Attention
Merged features
Image features
Region Reonganization
Figure 2: An example of regional cross-attention with two overlapping objects. Cross-attention applied to each pair of regional visual and grounded textual tokens. The overlapping region cross-attends with the textual tokens containing both objects, while the non-object region attends to a leamable "nall" token.
2 Related Works
Diffusion-based Generative Models The emergence of diffusion models 21 46 has significantly advanced the field of image generation. Within just few years, diffusion models have made remarkable progress across various domains, including super-resolution [4]. colorization [1]. novel
view synthesis [55] 3D generation
14
image editing
6 24 29 image completion 42
and video editing 12, 45 This progress can be attributed to several factors. Enhancements in
39 have played pivotal role. Additionally, improvements have contributed to this advancement. Moreover, the ability to incorporate various conditions during image generation has broadened the impact applications of diffusion models. These conditions include elements such as segmentation [1] 4 [59], using an
network architectures 32
34
35
in training paradigms 15
3
47
48
image as a reference [16] 30 (40,, and layout [I] 25
54]
the latter of which will be the main focus
of our discussion in this work.
Layout-to-image generation: Early works 19
23
26
50 often utilized GANs
or transformers [53] for 1.21 generation. For instance, GAN based LAMA 22, LostGANs
49
19 encode layouts as style features fed into adaptive normalization layers, while Taming 223 and TwFA [60] use transformers to predict latent visual codes from pretrained VQ-
and Context L21
VAE [52]. Recent diffusion models [II
56 65 have shown promising results, extending
L21 generation to be open-set. LayoutDiffuse 11 injects objects into the image features through
learning per-class embeddings. LayoutDiffusion (65] fine-tunes pre-trained diffusion models by
mapping object labels and layout coordinates into cross-attendable embeddings for attention layers. FreestyleL21 [58), BoxDiff [57] and R&B [56), which are training-free frre methods, leverage pre-trained diffusion models inject objects into specified regions by imposing spatial constraints. GLIGEN (25] and Inst [54] explore open-set L.2I generation using grounded bounding boxes, which encodes layout locations and object descriptions into features attended by self-attention layers.
Methodology
3.1 Challenges in Rich-Context Layout-to-Image Generation
The layout-to-image (L2I) generation task can be formally defined as follows: given set of de- scription tuples S = {s/ss; = (b;, 4)), where represents the bounding box coordinates of an object, and i; denotes the corresponding text description, the objective is to generate image that accurately aligns objects with their respective descriptions while maintaining fidelity to the specified layouts. In the closed-set setting, the number of text descriptions is limited to a fixed number N,

---

i.e., .N- = l.t, where N is the total number of classes. However, in the open-set and rich- coch-conextt settings, the number of descriptions unlimited, with descriptions the rich-context setting being more diverse, complex, and lengthy.
Rich-context 1.21 encounters several challenges: 1)
The rich-context descriptions for each object can be lengthy and complex, requiring the model to correctly understand the descriptions without overlooking details. Existing open-set layouttt-immaee solutions [25] 54 typically condense and map text embeddings into a single vector, which is then mapped to the image space for layout conditioning. However, this condensation process can result in significant information loss, particularly for lengthy descriptions. 2) Fitting various text descriptions into their designated layout boxes while maintaining global consistency is challenging. Unlike simpler to-image generation with a single description, L2I generation deals with multiple objects, requiring precise matching of each description to its specific layout area without causing global inconsistency. 3) 1.21 involves objects with intersecting bounding boxes, unlike gegmentation-mas-too--ooo be efficiently handled by pixel-conditioned methods such as ControlNet [63] and Palette [42]. 1,21 models must determine the appropriate order and occlusion of overlapping objects autonomously, ensuring the proper representation and interaction of each object within the image.
tasks where object areas do not overlap and can
3.2 Regional Cross-Attention
We propose using regional cross-attention layer as a solution to rich-context layout-to-image generation, addressing the aforementioned challenges. The desired properties for an effective rich- context layout-conditionin module are as follows: 1) Flexibility: The model must accurately understand rich-context descriptions, regardless of their length complexity, ensuring that no details are overlooked. 2) Locality: Each textual token should only attend to the visual tokens within its corresponding layout region, without influencing regions beyond the layout. 3) Completeness: All visual features, including those in the background, should be properly attended by certain description to maintain consistency in the output feature distribution. 4) Collectiveness: In cases where visual token overlaps with multiple objects, it should consider all descriptions related to those intersecting objects.
Our approach differentiates itself from previous methods [25] 54 by employing cross-attention layers, rather than self-attention layers, to condition objects within the image. This design is inspired by the architecture of modern text-to-image diffusion models, which achieve fine-grained text control by incorporating pre-pooled textual features in the cross-attention layers. Analogously, one can apply cross-attention repeatedly between pre- pre-pooled object description tokens and visual tokens within the corresponding regions for all objects. However, this straightforward method, though satisfies flexibility and locality, does not fully meet the criteria of completeness and collectiveness, as may inadequately address non-object regions and overlapping objects. This limitation can result in inconsistent global appearances and challenges in managing overlapping objects effectively.
Region Reorganization. We propose region reorganization to satisfy locality, completeness and collectiveness, by creating a spatial partition of the image based on the layout. Each region is classified into one of three types: single-object region, overlapping region among objects, and background. This partitioning ensures that regions are mutually exclusive (i.e., non-overlapping) Figure [ illustrates simple case with two overlapping objects. The overlapping area becomes new, distinct region, while the non-overlapping parts of the original regions and remaining background are also treated as separate, new regions, thus ensuring completeness.
Formally, in the general case with multiple objects, the reorganized regions R == {ra} satisfy that the union of these regions will form a complete mask covering the entire visual feature space, while ensuring no intersection between any two reorganized regions:
| U 4=1
1
r nry = Ø for i + and i, E [R]
Our regional cross-attention operates within each reorganized region. We define a selection operation f(, rz) to identify the appropriate regions for cross-attention. For visual tokens V == {u;) ) finds the tokens whose locations loc(v,) lic within the i-th reorganized region. For description tuples S, it

---

filters the instances that overlap with the i-th reorganized region. This selection operation ensures that the text description is applied exclusively the visual tokens within its corresponding region. thus maintaining locality. For regions with multiple objects, / also ensures that all overlapping descriptions are included satisfy collectiveness.
f(V. (vylloc(e)) e m) ((S.)) = {8y1b O nr Arx *
The final attention result A is the aggregation of all regional attention outputs. The selected de- scriptions f(S, r.) are encoded using Sequenced Grounding Encoding (SGE) in Figure 3 and serve the key and value during cross-attention. For non-object regions where f(S, ri) 2, "null" embedding learned as a substitute for the description.
[R]
a, = CroesAtul(//Tr)),S////////
A a
(4)
-=1
Sequenced Grounding Encoding with Box Indica- tor. The selected object descriptions in each reorga- nized region are encoded into textual tokens using Sequenced Grounding Encoding in Figure 3 When multiple objects present in a region, their de- scriptions are concatenated with a separator token. However, if two objects share the same description, their encoded textual embeddings will be identical. This identical embedding makes it impossible for the cross-attention module to distinguish between dis- tinct objects. To address this issue, incorporate bounding box coordinates as additional indicators. During encoding, we concatenate the bounding box coordinates with the textual tokens. The bounding box coordinates are initially encoded using sinusoidal positional encoding [53] and then repeated to match the length of the textual tokens before concatena- tion. For separator tokens and special tokens such as [bos] and [eos]. we use an all vector for their box coordinates.
[bos] This is cat; A dog is here [eos]
Text Encoder
Grounding tokens
Positional
Encoding
box
box
box
box
box
box
Sequenced Grounding Encoding with box coordinates as indicators.
Figure
Evaluation for Rich-Context L21 Generation
4.1 Rethinking L21 Evaluation
For evaluating layout-to-image generation, what are mainly considered are two aspects: 1) Object- label alignment, which checks whether the generated object matches the corresponding descriptions. 2) Layout fidelity, which examines how well the generated object aligns with the given bounding box.
In closed-set scenarios, it is common use off-the-shel detector to evaluate L2I generation performance [ 25 (54). Object-label alignment is assessed by classifying image crops extracted from the generated image using pre-trained classifier. Similarly, layout fidelity is measured by comparing the bounding boxes detected in the generated image with the provided layouts, using a pre-trained object detector.
However, in the open-set scenario, impossible to list all the classes. Moreover, even the state- of-the-art open-set object detectors [3]
61 62 are set up to handle inputs at the word phrase level, which falls short for the sentence-level descriptions required in rich-context 1.21 generation. we introduce two metries bridge the gap in evaluating open-vocabulan L2I models.
5

---

4.2 Metrics For Rich-Context L2I
We leverage the powerful visual-textual model CLIP for measuring object-label alignment, and the Segment Anything Model (SAM) for evaluating layout fidelity of the generated objects.
Crop CLIP Similarity: In rich-context L2I, object descriptions can be diverse and complex. The CLIP model, known for its robustness in image-text alignment, is thus suitable for this evaluation. To ensure accuracy and mitigate interference from surrounding objects, compute the CLIP score after cropping the object as per the layout specifications.
An accurately generated object should closely align with its designated layout. Given the potential diversity in object shapes, we employ the SAM model, which can highlight an object's 's region in mask format within given box region, to identify the actual region of the generated object. We then determine the generated object's circumscribed rectangle as its bounding box. The layout fidelity of the generated object with the ground-trui truth layout is quantified by the interscction-over-unee (IoU) between the provided layout box and the generated object's 's circumscribed box.
SAMIoU:
Experiments
5.1 Model and Dataset
Model We leverage powerful pre-trained diffusion models as the foundation for our generative approach. Our best model is fine-tuned from Stable Diffusion XL (SDXL. [35]. We also provide the benchmarks using Stable Diffusion 1.5 (SD1.5) [39] which a widely used backbone in existing methods (54). The proposed regional cross-attention layer is inserted into the original diffusion model right after each self-attention layer. The weights of the output linear layer are initialized to zero, ensuring that the model equals to the foundational model at the very beginning. More implementation details is shown in Appendix B
Rich-Context Dataset To equip the model with the capability to be conditioned on complex and detailed layout descriptions, rich-context dataset required. While obtaining large-scale real-world datasets through human tagging is labor-intensive and expensive, synthetic training data can be more readily acquired by leveraging recent advancements in large visual-language models. Similar to and InstDiff [54]. generate synthetic data to train our model.
GLIGEN 25
GLIGEN
InstDar
6.12
Ours
15
0:15
0.10
0.4
0.39
0.06
02-
0.05
0.08
0.0
0.0
0.09
0.00
30
(a) Average Caption Length
(b) Gunning Fog Score (Complexity)
(c) Unique Words/Sample (Diversity)
(d) Object- label CLIP
Alignment Score
Statistical comparisons between the synthetic object descriptions generated by GLI- GEN 255, InstDiff (54), and our method. We measure the 1) average caption length, 2) the Gunning Fog Score, which estimates the text complexity from the education level required to understand the text, 3) the number of unique words per sample which indicates the text diversity, and 4) the object-label CLIP Alignment Score to measure object-label alignment. The results show that the pseudo-labels generated for our dataset are more complex, diverse, lengthier, and align better with objects, compared to those generated by GLIGEN and InstDiff.
Figure 4:
We adopt a locating Docating-and-labedd and-labeling strategy during pseudo-label generation. At the first step, use the Recognize Anything Model (RAM) [64] and GroundingDINO 27 to identify and locate salient objects in the image. Next, we use the visual-language model QWen S to produce the synthetic label for each object by asking it to generate detailed description of the object (see Appendix B for the prompt we used). We utilize CC3M X and COCO Stuff 7 as the image source. For COCO, we directly use the ground-truth bounding boxes rather than relying on RAM and GroundingDINO to generate synthetic labels. The final training dataset contains two million images, with 10,000 images from CC3M set aside and the 5,000-image validation set of COCO used for evaluation. We

---

Layouts
Ours
GLIGEN
InstDiff
tull bookcuse is placed aguinst wall
tull bookshelf with muny buoks on
manyy-book beoks
couch with pillows and a coffee tuble in frout of
X Fre
coffoe table with metal base
X peetall-asse
on a carpeted floor
blue red pataring of world
X blue-and-red
X blue-and-red
mop on a wooden surface
chandelier
metal frome
X frame sphere
X sphere
the shope of sphere
metal frame with roman numerols
X roman numerals
X fetal
Figure 5: Qualitative comparison of rich-context L21 generation, showcasing our method alongside open-set 1,21 approaches GLIGEN 25 and InstDiff (54, based on detailed object descriptions. Our method consistently generates more accurate representations of objects, particularly in terms of specific attributes such as colors and shapes. Strikethrough text indicates missing content in the generated objects from the descriptions. More qualitative results available in Appendix H
denote the generated dataset Rich- Context CC3M (RC CC3M) and Rich-Context COCO (RC COCO). Compared to the synthetic training data used in GLIGEN (25] and InstDiff [54], our rich-context dataset provides more diverse, complex, lengthy, and accurate descriptions, as shown in Figure
5.2 Reliability Analysis of Automatic 1,21 Evaluation
Our proposed evaluation metrics in Section 4.2 serve as substitute for the lack of precise ground- truth in open-set scenarios. Whether the set closed or open, the goal of evaluation is ensure that the measurement results align with human perception. To validate the reliability of our automatic evaluation metrics, we conducted a user study on the RC CC3M dataset, randomly selecting 1000 samples. Each synthetic sample may contain multiple objects, but only one object was randomly selected for each question. Users were asked answer two questions, each rated on a scale from 0 (bad) to 5 (good). For object-label alignment, users responded to the question: "Can the cropped object in the image be recognized as [label]?" with the label being the automatically generated object description from RC CC3M. For layout fidelity, users answered: "How well (tight) does the object align with the bounding box?" referring to the synthetic bounding box RC CC3M.
In total, we collected 300 answers for each question. We used the Pearson correlation coefficient to analyze how well the automatic evaluation metrics align with human perception. The Pearson correlation coefficient measures the correlation between two distributions with value ranging from -I 1, where indicates no relation and I indicates strong correlation. Empirically, we found that the automatic evaluation metrics sometimes failed to reflect human perception when the object size was very small or very large. We note that for small objects, the clarity of the object can be hampered, while large objects may overlap with many other objects, making the automatic measurements inaccurate. Therefore, we filtered out objects smaller than 5% or larger than 50% of the image, resulting in improved Pearson correlation between automatic metrics and user scores from 0.33 to 0.59 for CropCLIP and from 0.15 to 0.52 for SAMIoU. We applied the same filtering rule in the remaining evaluations.
7

---

Table I: Quantitative comparison of different L.21 approaches under image resolution at 512x512. "1" means that the higher the better, ' means that the lower the better.
RC COCO
RC CC3M
Method
CropCLIPt
SAMIoUT
FIDI
CropCLIP
SAMIoUT
FID
Constrained
BixDif (57
22.61 23.68
58.75
29.99 31.70
R&B 56
64.68
Open-Set
GLIGEN 25
25.20
78.66
27.62
25.27
83.64
15.81
InstDiff
27.46
80.78
30.00
28.46
85.59
17.96
Ours
SD1.5
27.36
80.78
25.81
28.45
86.04
16.49
SDXL
28.15
80.84
27 27.41
29.42
86.56
11.02
5.3 Rich-Context Layout-to-Image Generation
Evaluation Metrics. In addition to the two dedicated metrics discussed in Section 4.2 we also consider image quality by measuring the FID scores [20], which reflects how real or natural the generated images look compared to real images. While we did not observe significant changes in FID scores across different variations of our methods, we did notice change in image quality among different baseline methods.
Baseline Methods. We compare our approach with GLIGEN 25 a popular open-vocabulary L2I generative model, and InstDiff [44, recent method that achieves tfateoof-th-aae open-set L,21 performance. Besides, two training- frreeng-freeeee L2I methods Box [57] and R&B [56] are also considered for comparison. Although these methods can accept open-set words, their inputs are limited to single words or simple phrases which are not truly rich-context descriptions. Therefore, we denote them as constrained L21 methods and only evaluate them on COCO using category names.
Table I benchmarks the performance of 1.21 methods at an image sampling resolution of 512. Our model with SD1.5 achieves similar performance InstDiff while reducing the computation cost in the layout conditioning layer by half, as illustrated in Figure 6 Additionally, our model with SDXL achieves the best performance, even though the 512 resolution sub-optimal for Further experiments in Section Section 5.3 demonstrate that higher sampling resolutions can further enhance performance. Figure 5 shows that, as the complexity and length of object descriptions increase, existing open-set L2I methods tend to overlook details especially when objects are specified with colors shapes. In contrast, our method consistently generates objects that accurately represent the given descriptions.
5.4 Performance Across Various Complexity of Object Descriptions
By adopting pre-pooling textual features in the layout conditioning layer, method maximizes the retention textual information during generation. We observe that this design significantly enhances performance when dealing with complex and lengthy object descriptions. In Figure ( a), categorize object description complexity using the Gunning Fog score into three levels: easy (scores 0-4), medium (5-8), and hard (>8). Additionally, words), short sentences (515 words), and long sentences (216 words). Our results indicate that for simple and short descriptions, the performance difference between our method and state-of-the-art open-set L2I methods is close. However, as the complexity and length of the descriptions increase, our method consistently outperforms existing approaches.
classify descriptions by length into phrases (S8
5.5 Ablation Study
We investigate the effectiveness of the proposed region reorganization and the use bounding box indicators on object-label alignment and layout fidelity using RC CC3M dataset. In experiments without region reorganization, we use straightforward averaging features overlapping objects. Empirically, observe that without region reorganization, our model struggles to generate the

---

SDNL
Cua
29
Oure 5D1.5
Crep CLIP Score
-20
87
inviE
fof)
IreDDif Mis
-27
Curs
CLIP
Dun SELL
GLIGEN
26-
Mded of
Meal Disodamn Misld
CLIGEN
GLIGEN 21 SEIN
25-
CLIGEN 7M
Easy
Medium
Hard
Phrase
Short
Loog
Descripton Complexiyy
Deseriplion
GPLOPS
GPLLIPS
(b)
Figure 6: (a) Object-text alignment scores across varying description complexities and lengths on
RC CC3M. Our method shows significant advantages for complex and lengthy descriptions. (b) Object-text alignment and layout fidelity relative to computational cost in each layout-conditioning attention layer. Given that the number of textual tokens is much smaller than visual tokens, applying cross-attention substantially reduce computational costs.
correct object when there is an overlap of objects with complex descriptions, leading to significant drop in both object-label alignment and layout fidelity as shown in the Table 2
Unlike self-attention-basse solutions that use box indicators implicitly indicate object locations, our method explicitly aoss-attends visual tokens with corresponding textual tokens. This approach allows the model recognize the correct location for object conditioning even without box indicator. However, the reorganized mask in the regional cross-attention layer has lower resolution than the original image, causing misalignment near the borders of generated objects. Adding bounding box indicator not only helps the model distinguish objects with similar descriptions and but also improves layout fidelity, as validated by the improvement SAMIoU.
Additionally, we notice that sampling at higher image resolution (768x768) improves model perfor- mance, although it demands greater computational resources. It's important to note that generalization to higher resolution is not a universal capability of L21 models. Existing self-attention-bart
L21 methods like GLIGEN [25] experience performance declines when sampling at resolutions different from the training resolution. Another self uuf--ateetiion- -anddd method, InstDiff [54]. uses absolute coordi- nates for conditioning, requiring the sampling resolution to match the training resolution exactly. In Figure/6/b), we compare the InstDiff does not support flexible resolution sampling, we utilize Multi-Instance Instance Sampling (MIS) (54] instead. MIS was proposed to enhance InstDiff's s performance by sampling each instance separately, albeit with increased inference times. We demonstrate the simplest case MIS, which requires two inferences, but its computational cost scales linearly with the number of objects.
ceermannc-cooputatiin
trade-off" of open-set L21 approaches. Since
Table 2: Ablation study of the proposed methods on RC CC3M dataset with SDXL backbone. The results suggest that region reorganization plays important role for rich-context L2I generation, while using box indicator and sample at higher- resolution can further enhance performance.
Region Reorg.
Box Indicator
High Reso.
CropCLIP1
SAMIoUT
X
X
25.32
76.92
X
28.93
85.02
X
29.42
86.56
/
29.79
88.10
6 Conclusion
In this study, we introduced a novel approach to enhance layout-to-image generation by proposing Regional Cross-Attention module. This module improve the representation of layout regions, particu- larly in complex scenarios where existing methods struggle. Our method reorganizes object-region correspondence by treating overlapping regions as distinct standalone regions, allowing for more accu- rate and context-aware generation. Additionally, we addressed the gap in evaluating open-vocabulary
*Computed using 640 channels, which corresponds a 32x32 resolution for image features, assuming input resolution of 512x512.
pytorch an attention layer with

---

L2I models by proposing two novel metrics to assess their performance in open-set scenarios. Our comprehensive user study validated the consistency of these metries with human preferences. Overall, our approach improves the quality of generated images, offering precise location control and rich, detailed object descriptions, thus advancing the capabilities of generative models in various potential applications.
Acknowledgement This work was funded in part by the Science and Technology Development Fund, Macau SAR (File no. 0049/2022/A1, 0050/2024/GJJ), and in part by the University of Macau (File no. MYRG2022-00072--555
References
[1] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio- representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18370-18380, 2023.
[2] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 18208-18218. 2022.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin
Ge. Yu Han, Fei Huang. Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint Xiv: 16609, 2023.
[4] Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0132
2022.
[5] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang. Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.
Computer Science. https://cdn. openai. com/papers/daalle-3 pdf. 2(3):8, 2023.
[6] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 18392-18402, 2023.
[7] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1209-1218, 2018
[8] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-3568, 2021.
[9] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang. Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint 240: 240.00692, 04692, 2024.
[10] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint 2xXiv:231000422
2023.
[11] Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdif- fuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint axXiv:2302.08808, 2023.
[12] Jiaxin Cheng. Tianjun Xiao, and Tong He. Consistent video-to-video transfer using synthetic dataset. arXiv preprin! arXiv:231 022331..0211,. 2023.
[13] Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time voce-vocab object detection. arXiv preprint urXiv:2401. 2024.
[14] Yen-Chi Cheng, Hsin- Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Yan Gui, Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the JEEF/CVF Conference on Computer Vision and Patlern Recognition, pages 4456-4465,
2023.
[15] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, 2021.
10

---

[16] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2022. [17] Ian Goodfellow, Jean Abadie, Mchdi Mirza, Bing Xu. David Warde-Farkee Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):139-144, 2020.
[18] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 53556-3364, 2019.
[19] Sen He, Wentong Liao, Michael Ying Yang, Yongxin Yang, Yi-Zhe Song. Bodo Rosenhahn, and Tao Xiang. Context-aware layout to image generation with enhanced object appearance. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15049-15058, 2021.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 001166811111 2020.
So. [22] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint
arXiv:2207.125: 12599, 2022.
[23] Manuel Jahn, Robin Rombach, and Bjöm Ommer. High-resolution complex scene synthesis with transformers. arXiv preprint Sati:2205.004558 2021.
[24] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[25] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lec. Gligen: Open-set grounded text-to- generation. arXiv preprint 2023.
arXiv:2301.0740
[26] Zejian Li, Jingyu Wu, Immanuel Koh, Yongchuan Tang, and Lingyun Sun. Image synthesis from layout with locality-awar mask adaption. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 11819-13825 2021.
[27] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.... 054999, 2023.
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05000 2017.
[29] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108. 01073, 2021.
[30] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint ar 2332 002.08453, 2023.
[31] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171 8171. PMLR, 2021. [32] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models. In International Conference on Machine Learning. pages 11784-16804, PMLR, 2022.
[33] Taesung Park, Ming- Yu Liu, Ting-Chun Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337-2346, 2019.
pa [34] Pablo Pernias. Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The
Twelfth International Conference on Learning Representations, 2024.
[35] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint M.XXi:2337.09995 2023.
[36] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion. In The Eleventh International Conference on Learning Representations, 2022.
11

---

[37]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint wrXiv:220.006225 2022.
[38] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for amage-to-imann translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287-2796, 2021.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[41] Chitwan Saharia, William Chan, Huiwen Chang. Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: -o-age-t--maaee diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[42] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH
2022 Conference Proceedings, pages 1-10, 2022.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
[44] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions Putfern Analysis
and Machine Intelligence, (((4):71134775) 2022.
[45] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209, 14792, 2022.
[46] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine
learning, pages 2256-2265. PMLR, 2015.
[47] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[48] Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[49] Wei Sun and Tianfu Wu. Image synthesis from reconfigurable layout and style. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10531-10540, 2019. [50] Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R Devon Hjelm, and Shikhar Sharma. Object-centric image generation from layouts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2647-2655, 2021.
[51] Junshu Tang. Tengfei Wang. Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it- High-fidelity 3d creation from single image with diffusion prior. arXiv preprint a333v:2303. /4/84, 2023.
[52] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[54] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. In- stancediffusion:
Instance-level control for image generation. 2024.
[55] Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In The Eleventh International Conference on Learning Representations, 2022.
[56] Jiayu Xiao, Liang Li, Henglei Lv. Shuhui Wang, and Qingming Huang. R&b: Region and boundary aware zero-shot grounded text-to-image generation. arXiv preprint arXiv:23 /0.00887.. 2023.
[57] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-fres nox-constrained diffu- sion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
12

---

7452-7461, 2023.
[58] Han Xue, Zhiwu Huang. Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14256-14266. 2023.
[59] Binxin Yang. Shuyang Gu, Bo Zhang, Ting Zhang. Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18381-18391, 2023.
[60] Zuopeng Yang, Daqing Liu, Chaoyue Wang, Jie Yang, and Dacheng Tao. Modeling image composition for complex scene generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7764-7773, 2022.
[61] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary- -enicchee visual-concept paralleled pre-training for open-wortee world detection. Advances Neural Information Processing Systems, 35:9125 1191125-138, 2022.
[62] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 14393-14402. 2021.
[63]
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint (.xXi::302.0554444 2023.
[64]
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo
Qin, Tong Luo. Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging
model. arXiv preprint arXiv:2306. 2.0.003366.311 2023.
[65] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layout- diffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22490-22499
2023.
13

---

Supplementary Material for Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation
A
Limitations
Our work is built upon pre-trained diffusion models. Although our solution is backhone-agnostic, fine-tuning is still required when changing the model's backbone. Additionally, our training dataset generated using visual-language model, which cannot guarantee that all synthetic labels are correct; these inaccuracies may negatively impact the model's performance. Furthermore, our training images are from publicly available datasets, which often contain low-quality images. As result, the generated images may exhibit undesired artifacts, such as watermarks, during generation.
B Implementation Details
train our model using the AdamW 28 optimizer with a learning rate of 5e-5. The training process involves an accumulated batch size of 256, with each GPU handling batch size of 2 over 8 accumulated steps, for total 100,000 iterations on NVIDIA V100 GPUs. This training process takes approximately 8,000 GPU hours. During training, we apply random cropping and horizontal flip for image augmentation, a bounding box will be dropped if its remaining size is smaller than 30% of its original size after cropping. We randomly drop 10% of layout conditions (all conditions in an image are dropped when layout condition dropped) and 10% of image captions to support classifier-free guidance [22] During sampling,
a classifier-free guidance scale of 4.5 for SDXL-based model and 7.5 for our SD1.5-based model. The inference denoising step is set to 25 for models and all baseline methods. During synthetic data generation, obtain the description of the object using the following prompt for QWen model: "You are viewing an image. Please describe the content of the image in sentence, focusing specifically objects. Include detailed observations about all the objects and how they are positioned relation to other objects in the image. Your response should be limited to this description, without any additional information".
the spatial relationships between
C Throughput of Different Layout-to-Image Methods.
In addition to the FLOPs comparison presented Section 5.5 we compare the throughput of using different L2I methods and present the result in the Figure
D Layout-to-Image Generation Diversity Comparison
Following LayoutDiffusion 65
we evaluate the generation diversity using L.PPS and Inception Score and present the diversity comparison of different 1.21 methods using 1,000 RC CC3M evaluation layouts in Table
LPIPS
Inception Scoret
GLIGEN
0.65 0.10
15.74 + 1.52
InstDiff
0.67 + 0.10
15.26 + 1.75
Ours (SD1.5)
0.71 + 0.13
16.95 + 2.31
Ours (SDXL)
0.68 0.12
15.86 + 1.58
Table 3: For LPIPS computation, each layout is inferred twice, and the score is calculated using AlexNet. higher LPIPS score indicates larger feature distance between two generated images with the same layouts, signifying greater sample-wise generation diversity. A higher Inception Score suggests more varied appearance of generated images, indicating greater overall generation diversity.
14

---

0.6
SD1.5
SDXL
GLIGEN(SD1.4)
0.5
InstDiffiSD1.5)
Ours SD1.5
Throughput (img/s)
Ours SDXL
0.4
0.3
0.2
0.1
0.0
2
10 15
Number of Objects
Figure All methods are tested with floatl6 precision and 25 inference steps. The results are averaged over 20 runs. Notably, the overall throughput of our method is not significantly hampered. In typical scenario with objects, throughput our method exceeds 60% of the throughput of the original backbone model. Please note that while the official backbone of GLIGEN is SD1.4, its network structure and throughput are identical to those of SD1.5.
E Pseudo-code For Proposed Evaluation Metries
Algorithms Bandhhmwwsww the pseudo-code for computing the proposed Crop CLIP score and SAMIoU score on a single generated sample. The final performance is calculated by averaging these scores across all generated samples.
Algorithm 1 Compute Crop CL.IP Score
Require: Generated image conditioning layout boxes B and labels for each object, CLIP models
cliping and cliptezt
crop_clipsscore
2: for each (box, label) E zip(B, L) do
3: + cIop(I, box)
4: if S size < lower thres. or S size >> upper thres. then
5:
continue 6: end if
7: dlip_img_feat + cliping
8: clip_text_feat +- clip (label)
9: crop_clip_sin + cosine cossne_siiiiilleeee similarityycllpp...... hitp:clllppm__aatttttttaaaaaaaae feat, clip_text_feat
10: rrcp:clip.ooree.ppeen/ppp...pppp
11: end for
12: cmmple_croo_cli__ccor mean(crop.clip.soree))
13: return
F Effectiveness of Rich-Context Dataset and Regional Cross-Attention
The ablation study in Table 4 indicates that to condition the 1.1 model with rich-context descriptions, both a rich-context dataset and designated conditioning module for rich-context description are vital.
15

---

Algorithm 2 Compute SAMIoU Score
Require: Generated image conditioning layout boxes B for each object, Segment Anything Model SAM
1: sam sam_iou_score
<-
2: for box E B do
3: if S size < lower thres. S size upper thres. then
4:
continue
5: end if
6: sam_mss ham_mask= + SA b)
7: box_of_generated_ k get_ciinn http:/ssccreeeeeeeaeeeeeeeee
8: sam_iou compute (octnppute_ouubboo.. box_of [ew_oofgeeerraee_ooh]l
9: sa0..bo__sorres.pppeddesss
10: end for
sample_sam iou sccresscceeeee + mean(san_iou_scoree
12: return sample_sst @ample__am_iouueeeeee_e__aeeaee_ sc__ro_sccoee
Dataset
Attention Module
Backbone
CropCLIP
SAMIoU
SelfAttn
SelfAttn InstDiff
CrossAttn Ours
Word/Phrase
Rich-context
GLIGEN
SDXL SDXL.
25.40
86.76
29.79
88.10
SD1.5
25.56
82.72
SD1.5
28.36
85.58
SD1.5
28.94
86.91
Table The performance is evaluated on RC CC3M evaluation set and all methods are sampled under their best sampling resolution discussed in Section [.5 It can be noticed that even with the rich-context dataset, the performance self atte-attennion- based modules does show significant improvement over their performance in the Table in in the paper.
G Effeetiveness of Regional Cross-Aftention with Visual Example
Figure presents the visual comparison of using region reorganization compared straightforward feature averaging when dealing with overlapping objects. The model with region reorganization can accurate generated objects that better align with the designated layouts, while the feature averaging solution can result in objects in incorrect location, generating undesired instances or making the overlapping instances inseparable.
H
More Qualitative Results
During the evaluation, filtered out very small and very large regions to avoid inconsistencies between automatic evaluations and human preferences. However, this does not imply that our method incapable of generating high-quality small large objects. Our results in Figure 9 verify that our model can accurately handle both very large and very small objects.
In addition, We provide more qualitative comparison with existing open-set L2I approaches in Figure 10
16

---

Boxes
Feature Avg.
Region Reorg.
large body water with small island the distance. large rock formation sits the middle of the large rock formation the middle of the tall mountain with pointy top from distance.
Inseparable
Undesired
A black and clear brooch with silver metal base.
A black diamond sitting grey surface.
A black diamond shown in the center.
A black and clear brooch with black
stones and clear stones.
Figure 8: The model with region reorganization accurate generated objects that better align with the designated layouts, while the feature averaging solution can result in objects incorrect location, generating undesired instances making the overlapping instances inseparable.
17

---

pine cone sits on white background.
pine cone is shown on white background.
white snowflake ornament placed table with pinecones pine needles.
large pine cone sits on a pine tree branch.
A white snowflake ornament sitting white counter. group of brown cloves of garlic. Snowflakes and pinecones are sitting on a white surface. large pine cone is sitting on pine tree.
gold and white watch black leather strap. close of black leather strap with white stitching. close up of black leather strap with white stitching. An automatic watch gold band.
A grey backpack with zipper on the front. black backpack with strap the side. grey bag with zipper. close up of a black tie on a white background.
A brown house with a red door and a white chimney.
A cartoon tree with brown trunk and green leaves.
A coloring book app for kids is open page with house and tree.
lobol bluu wroe
A vase filled with water and green stems of hamboo.
A vase red and white roses with red carnations and green ornaments.
white rose sitting on bed of greenery.
A red carnation is the main focus of the image, with white
flower in the background.
A red rase is the center of the image, surrounded by
small white flowers and greenery.
A close up of a red rose with green leaves and white
flowers around it.
red rase is sitting in with green and white leafy
stem.
A red rase is in the center of the image, surrounded by greenery.
Figure 9: Additional qualitative results using random layouts from the synthetic RC CC3M dataset demonstrate model's ability accurately handle both very large and very small objects.
18

---

Layouts
Ours
GLIGEN
Inst
BOIE
black chair
X black
silver appliance
Gray cabinets
X gray
X grey
dishwasher with silver doors
X silver
bluck stove built in cabnet
X black built in eabnet
suinlesy oven
X stainless steel black glass
glase window
close up of a pair dowry, one the left and
X peir dowrs-en aighright silver-knob
the right botk with silver knob
X silver-knob
Layouts
Ours
GLIGEN
InstDiff
cute puppy lying grass
X unconcognizable
X unconcognizable
galden retriever dog with pink
X -ith----iin--onguee
tongue sitting graes
back and white dog standing on
X bleck snnd white
grave.
brawy dog with rainhons collar has its tongue out sitting on the grass.
X brown rainbow-collar
X
painbow-collar tongue
Figure 10: More qualitative comparison of rich-context L2I Generation, showcasing method alongside open-set L2I approaches GLIGEN (25) and InstDiff (54), based on detailed object descrip- tions. Our method consistently generates more accurate representations of objects, particularly in terms of specific attributes such as colors and shapes. Strikethrough text indicates missing content in the generated objects from the descriptions.
19

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have conducted extensive experiments to validate our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
20

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the implementation details in the paper for result repro- ducibility,
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
21

---

Justification:
Our code has been cleaned up and made public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ tiaaps.cc/ubllic/uuddees /Coeee/mmddebbeee ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See implementation details
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our experiment is computationally intensive thus we only report the average number on large testing set.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See implementation details
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://pww..aaa........aaaaaaaaa
Answer: [Yes]
Justification:
We have checked the website.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: On positive side, our work can provide better control for diffusion models, resulting in better application of generative models in research or industry. On negative side, it is possible our model will be misused for generating forgery content.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research do not have such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The dataset and pretrained model used are all licensed for researched purpose.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We not releasing new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We have provide the instruction for user study. Our use study is small scale with volunteers without compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our study object is not human.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "5v1X4ypS4K", "reviewer": "Reviewer_6Uy3", "summary": "The paper proposes a layout-to-image generation method based on cross-attention control. It also proposes evaluation metrics for the task.", "questions": "* How is \"open-set\" and \"rich-context\" defined? In line 95, \"open-set\" seems to refer to cases when N is not fixed. But even in these cases, an off-the-shelf object detector could be used to compute evaluation metrics. Why would it be impossible to list all classes (line 181)?", "limitations": "Limitations are discussed.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "* The paper highlights the potential effectiveness of cross-attention control and designs a learning framework using this insight. \n* The framework shows some empirical performance gain over prior methods.", "weaknesses": "* The performance improvement compared to baselines is very limited, as shown in Table 1. What is the backbone of the baseline methods? If the backbones of these baselines are swapped to be the more powerful SDXL, they could match or surpass the proposed method.\n* Section 3.2 proposes four properties but there lacks a discussion on 1) why these properties are complete descriptions of the desired properties of layout-to-image generation methods, or 2) why they are necessary and if so under what scenarios or using what kind of layout or text inputs would lead to undesirable outputs. \n* Lines 125-127 describe a cross-attention-based baseline which could be training-free. The paper claims advantages over this baseline without empirical evidence. Quantitative and qualitative comparisons would help strengthen the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her review. We use W and Q to denote bullets in weaknesses and questions\n\n**Answer to W1**: The backbone of the current SoTA L2I method, InstDiff, is based on SD1.5. Our proposed method has been validated using both SDXL and SD1.5 backbones (Ln 201). Therefore, our comparisons against baselines using the SD1.5 backbone are fair and justified.\n\nWhile the performance results in Table 1 appear close, it is crucial to consider the following points: 1) Table 1 illustrates the model’s performance across various description complexities, including easy, medium, and hard. However, our model excels when handling complex object descriptions (see Figure 6a), which shows that as the complexity and length of the descriptions increase, our model’s performance advantage over the baselines becomes more pronounced. 2) As noted in lines 290-295, the evaluation configuration in Table 1 does not fully leverage our model’s generation capabilities. Our model’s performance can be further enhanced with higher resolution. We provide the combined results on RC CC3M of Table 1 and Figure 6b in the following table. By examining our model’s performance at higher resolution (Figure 6b), it is clear that our improvements over the baselines are noteworthy. \n\n| Methods   | CropCLIP | SAMIoU |\n|-----------|----------|--------|\n| GLIGEN (512x512)    | 25.27    | 83.64  |\n| GLIGEN (768x768)    |  25.16   |  83.80   |\n| InstDiff (512x512)  | 28.46    | 85.59  |\n|           |          |        |\n| **Ours**  |          |        |\n| SD1.5 (512x512) | 28.45 | 86.04  |\n| SDXL (512x512) | 29.42  | 86.56  |\n| SD1.5 (768x768) | 28.94 | 86.91  |\n| SDXL (768x768) | 29.79  | 88.10  |\nPlease note that InstDiff only support fixed size of 512 for inference, as we noted in Ln 294-295.\n\n**Answer to W2**: Our method is not intended to be a, as the reviewer suggested, “complete” L2I solution, and there is potential for future extensions to tackle broader L2I problems. Our objective with the four properties outlined in Sec 3.2 is to address the rich-context L2I challenges detailed in Sec 3.1 (Ln 98-110).\n\nSpecifically, when “Flexibility” is satisfied, the rich-context description can be accurately understood by the model (Ln 99), the “Locality” ensures the objects are positioned correctly within the designated layout box (Ln 103) and “Completeness” guarantees the global consistency in the generated images (Ln 103-104). Finally, “Collectiveness”  allows the model to consider and properly represent the interaction of overlapping objects (Ln 109-110). \n\nIn practical terms, positioning objects accurately (Locality) and maintaining plausible image quality (Completeness) are fundamental prerequisites for L2I problems. By satisfying the Flexibility, our model can better understand descriptions and generate more satisfactory objects (Figure 1, 5). Additionally, a model that meets Collectiveness can more effectively handle interactions between overlapping objects (Rebuttal Sec E).\n\nWe will clarify and elaborate on this information in the revised paper to ensure it is more comprehensible.\n\n\n**Answer to W3**: As noted in lines 203-204, our method is a training-based approach and is not training-free. We insert the proposed regional cross-attention layers after each self-attention layer in the original diffusion model. Consequently, even if applying the cross-attention strategy mentioned in Ln 125-127, the model still requires training because the newly inserted parameters are randomly initialized. \n\n\n**Answer to Q1**: “Rich-context” can be considered as an extension of the “open-set” concept. While both “open-set” and “rich-context” scenarios deal with an unlimited number of object classes, the descriptions in the rich-context setting are notably more diverse, complex, and lengthy (Ln 95-97).\n\nWe mention “impossible to list all classes”  as the limitation of using the closed-set detectors for evaluation(see Ln 176-180). In contrast, the limitation of open-set object detectors is that they are designed to handle inputs at the word or phrase level, but not sentence-level that are required in the rich-context setting (refer Ln 182-183).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nyeix9vTE1", "reviewer": "Reviewer_2Dp7", "summary": "The field of paper is open-set layout-to-image (L2I) generation. They propose to apply regional cross-attention module to enrich layout-to-image generation, slightly surpassing existing self-attention-based approaches. The paper also proposes two new metrics to assess L2I performance in open-vocabulary scenarios instead of previous closed-set environments.", "questions": "I will consider increasing the score if the authors can address the following issues.\n1. Since the authors consider the two evaluation metrics as one of the main contributions, detailed information regarding their implementation is necessary to facilitate future work. For example, specifying the model type of SAM, Clip, among other details, would be helpful. This information can be included in the main text or supplementary materials. Ideally, the evaluation code should be made open source.\n2. The proposed regional cross-attention may not be plug-and-play. Please list the modifications and additional training required to apply it directly to existing pre-trained models. Providing pseudocode for the algorithm would significantly enhance the paper’s reproducibility and reduce the implementation difficulty.\n3. The related work section on layout-to-image is somewhat brief. Please expand this section to provide a more detailed overview of this field. For example, Freestyle Layout-to-Image Synthesis from CVPR 2023 uses attention-based interactions to achieve open-set rich-context layout-to-image generation, which is similar to this paper. Additionally, some closed-set layout-to-image generation methods based on diffusion should also be included. For example, CVPR 2023, LayoutDiffusion applys more comprehensive evaluations than previous works, using five metrics to assess generation quality, controllability (accuracy), and diversity. As diversity of generated image is also important, it's recommended to include it.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Considering the evaluation in open scenarios is meaningful. The proposed two metrics are reasonable, and due to the use of open-source tools such as CLIP and SAM, they are relatively easy to implement.\n2. In the context of rich-context layout-to-image generation, the proposed method shows a notable improvement.", "weaknesses": "1. The proposed regional cross-attention may not be plug-and-play and could be incompatible with existing pre-trained large models.\n2. The related work section on layout-to-image is somewhat brief. Please expand this section to provide a more detailed overview of this field.\n3. In terms of quantitative metrics in COCO, which is relatively close-set, the improvement of the proposed method over the latest approaches is not particularly significant.\n4. The paper lacks pseudocode for the algorithm.\n5. The paper lacks the diversity analysis of the generation images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her review. We use W and Q to denote bullets in weaknesses and questions\n\n**Answer to W1, Q2**: Our solution, like GLIGEN and InstDiff, requires inserting additional parameters into the pre-trained model and enhancing L2I ability through training. Indeed, our module is not plug-and-play as it is not training-free. However, it is compatible with existing pre-trained large models. Specifically, our cross-attention module is inserted into the original diffusion model right after each self-attention layer (Ln 204), similar to GLIGEN. This allows our model to be applied to all modern pre-trained diffusion models with similar structures, such as the Stable Diffusion family, DALLE family, DeepFloyd, GLIDE, etc.\n\nFor reproducibility, our implementation will be released upon the completion of the submission. In practical terms, we implement region reorganization in the dataloader to minimize computation overhead . This ensures that the reorganized region-text correspondence is shared across all regional cross-attention layers. Consequently, the regional cross-attention layer in the denoising model is reduced to a classical cross-attention operation with a special attention mask that indicates the permissible attention region between visual and textual tokens. Essentially, the attention operation depicted in Figure 2 is conducted by passing a pre-computed attention mask when cross-attending visual and textual tokens. We will enhance the clarity of these details in the revised paper.\n\n**Answer to W2.W5,Q3**: Per the reviewer’s suggestion, we evaluated the generation diversity using LPIPS and Inception Scores as proposed in [A], with results provided in Rebuttal Sec B. Using the same model backbone (SD1.5) as previous works, our method can generate images with greater diversity. We also observed that the diversity of the SDXL-based model is less than that of the SD1.5-based model, which we attribute to the inherent differences between the backbones.\n\nFurthermore, while [B] addresses mask-based layouts, our work focuses on box-based layouts. Unlike mask-based layouts, box-based layouts lack dense annotation for every pixel and involve overlapping regions, making it infeasible to directly apply cross-attention-based techniques as in [B]. In the revised paper, we will elaborate on related works and clearly differentiate our approach from previous methods.\n\n**Answer to W3**: Our method is designed for rich-context L2I, but it does not fall short on simpler description datasets like COCO. Our analysis in Figure 6a indicates that when the description is simple, our performance is on par with existing methods. However, as the description complexity increases, our advantage becomes more pronounced. \n\n**Answer to W4, Q1**: We provide pseudo-code for the proposed two evaluation metrics in Rebuttal Sec C. The CLIP model used is the clip-vit-base-patch32, trained by OpenAI, and the SAM model is the ViT-H checkpoint from the official SAM implementation.\n\nAbove information will be updated in the revised version paper. \n\n[A] LayoutDiffusion\n\n[B] Freestyle Layout-to-Image Synthesis", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0Z4r6ji5k", "reviewer": "Reviewer_ZVVH", "summary": "In this work, the author views both training and evaluation of the layout-to-image synthesis task. They propose regional cross attention to addresses the issues of previous works and they also introduce a new evaluation protocols for this task.", "questions": "The details of models and evaluations should be included more in appendix.", "limitations": "The author discussed the limitation and I agree with that (as I described in weaknesses as well).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I agree with the author's discussion on the desired properties for a layout-to-image model and how they achieve these properties. The contribution in terms of evaluation is also acknowledged.\n- Overall, the performance is superior compared to existing models.", "weaknesses": "Metrics\n- The author view a holistic “rethinking” for both training and evaluation; however, the novelty in evaluation seems limited. For instance, using CLIP to compute patchwise similarity is not particularly novel, and although calculating SAM-driven IoU is an interesting approach and might be meaningful in open-vocab-related tasks, it does not feel exceptionally special.\n\nMethod\n- I also agree that the one critical problem in layout-to-image tasks is overlapping issues and I am quite interested in the regional cross attention. However, despite Table 2 showing improvements with the proposed modules (including regional CA), I find the analysis of these overlapping issues lacking. Similarly, the paper argue that “desired properties for an effective rich-context layout-conditioning module” (L113) in four items, each needing more concrete analysis. Currently, the analysis seems insufficient.\n- The computational requirements for training seem high. How does the performance compare to other methods in terms of training costs, including both resources and data? In a current view, I am not sure that the improved performance is originated from the method itself or the increased training dataset and computational powers.\n\nPresentation\n- I recommend that the proposed evaluation methods be described in more detail, possibly in a more formal manner or with more detailed information such as in appendix.\n- Similarly, a more detailed explanation of the regional cross attention would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her review. We use W and Q to denote bullets in weaknesses and questions\n\n**Answer to W1**: Our proposed two metrics may not seem exceptionally special, however, we are the first to repurpose these metrics to evaluate L2I performance. They offer significant value for L2I problem as 1) ​​these metrics are specifically designed to quantitatively measure L2I performance in rich-context scenarios, where existing metrics have failed to do so (Ln 176-183); 2) in addition to proposing these metrics, our reliability analysis is crucial. It demonstrates the conditions under which these metrics are effective and consistent with human perception. We argue that this analysis should also be considered as a contribution to the field of evaluation metrics.\n\n\n**Answer to W2**: In Rebuttal Sec E, we provide a visual comparison to demonstrate the effectiveness of using region reorganization compared to straightforward feature averaging (the baseline described in Ln 279) when dealing with overlapping objects. The qualitative results validate the importance of our proposed methods, as they produce more accurate generated objects that better align with the designated layouts.\n\nOur objective with the four properties outlined in Sec 3.2 is to address the rich-context L2I challenges detailed in Sec 3.1 (Ln 98-110). Specifically, when “Flexibility” is satisfied, the rich-context description can be accurately understood by the model (Ln 99), the “Locality” ensures the objects are positioned correctly within the designated layout box (Ln 103) and “Completeness” guarantees the global consistency in the generated images (Ln 103-104). Finally, “Collectiveness”  allows the model to consider and properly represent the interaction of overlapping objects (Ln 109-110). \n\nIn practical terms, positioning objects accurately (Locality) and maintaining plausible image quality (Completeness) are fundamental prerequisites for L2I problems. By satisfying the Flexibility, our model can better understand descriptions and generate more satisfactory objects (Figure 1, 5). Additionally, a model that meets Collectiveness can more effectively handle interactions between overlapping objects (Rebuttal Sec E).\n\nWe will clarify and elaborate on this information in the revised paper to ensure it is more comprehensible.\n\n**Answer to W3**: As mentioned in Appendix Section B, our model is trained with an accumulated batch size of 256 for 100K iterations, which is half of the current state-of-the-art (SoTA) L2I method InstDiff, trained with a batch size of 512 for 100K iterations. Additionally, the number of training samples is 2M (after filtering failed downloads in CC3M), which is also half of the number of samples used in InstDiff, which has 5M samples. Overall, the training cost of our work is less than half of the existing L2I methods. Therefore, we believe the performance improvement is not due to larger computational resources or data, but rather from our design. We will include this comparison in the revised appendix.\n\n**Answer to W4 and Q**: We provide pseudo-code for the proposed two evaluation metrics in Rebuttal Sec C. The CLIP model used is the clip-vit-base-patch32, trained by OpenAI, and the SAM model is the ViT-H checkpoint from the official SAM implementation.\n\n**Answer to W5**: For reproducibility, our implementation will be released upon the completion of the submission. In practical terms, we implement region reorganization in the dataloader to minimize computation overhead . This ensures that the reorganized region-text correspondence is shared across all regional cross-attention layers. Consequently, the regional cross-attention layer in the denoising model is reduced to a classical cross-attention operation with a special attention mask that indicates the permissible attention region between visual and textual tokens. Essentially, the attention operation depicted in Figure 2 is conducted by passing a pre-computed attention mask when cross-attending visual and textual tokens. We will enhance the clarity of these details in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RkU1CpvUJ8", "reviewer": "Reviewer_SX1k", "summary": "The paper focuses on layout-to-image (L2I) generation from the perspective of rich-context scenario where the object descriptions are complex and lengthy. In the framework design, it introduces a novel regional cross-attention module to enhance the representation of layout regions. In the evaluation for open-vocabulary L2I models, this paper proposed two new metrics that assess model performance under rich-context descriptions, validated through a comprehensive user study.", "questions": "Please see above weaknesses.", "limitations": "The authors have discussed limitations in the supplementary.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "--The introduction of a regional cross-attention module is novel, improving the handling of complex layout descriptions compared to traditional self-attention approaches.\n\n--The paper provides rigorous experimental validation, demonstrating that the proposed regional cross-attention module enhances generation performance.", "weaknesses": "--This paper presents the GFLOPs of the proposed method. However, it seems that the region reorganization and regional cross attention may affect the realtime throughput. It would be better to also analyze how this proposed module affect the runtime cost.\n\n--This paper aims to target the scenario rich-context layout-to-image generation. During training, they collect a rich-context dataset for training. Therefore, I wonder the performance gain is come from the constructed rich-context dataset or the proposed regional cross attention.\n\n--Meanwhile, the comparison with other baselines may not be fair. Since the proposed method used a different training dataset which is beneficial for rich-context layout-to-image generation. It would be more fair if the compared baselines are also retrained on the same training dataset.\n\n--This paper introduced two new metrics to evaluate the object-text alignment and the layout fidelity. However, the proposed two metrics are intuitive extensions from existing metrics, which may not be strong enough to claim as a contribution.\n\n--Regarding ablations, it would be better to have visual results to illustrate the effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her review. We use W to denote bullets in weaknesses\n\n**Answer to W1**: In Rebuttal Section A, we compare the throughput of L2I methods using SD1.5 and SDXL baselines. It is noteworthy that the overall throughput of our method is not significantly hampered. In a typical scenario with 5 objects, the throughput of our method is more than 60% of the throughput of the original backbone model.\n\n**Answer to W2 and W3**: We argue that conditioning the model on rich-context descriptions requires both a rich-context dataset and a designated conditioning module for complex descriptions. Without a rich-context dataset, the generalizability of the conditioning module from word/phrase-level context to rich-context descriptions can be hindered. Conversely, without a proper conditioning module, the model may perform poorly when handling complex descriptions.\n\nTo validate the effectiveness of both the rich-context dataset and the regional cross-attention module, we conducted two additional ablation studies: 1)  We retained the regional cross-attention module but replaced the rich-context descriptions with word/phrase-level descriptions obtained using the Recognize Anything Model and GroundingDINO (Ln 213). 2) We replaced the regional cross-attention module with the self-attention modules used in GLIGEN and InstDiff, training them with the rich-context dataset. \n\nThe results, presented in Rebuttal Sec D, show that performance drops either when using word/phrase-level descriptions or when removing the regional cross-attention module. This validates the importance of both the rich-context dataset and the regional cross-attention module.\n\n**Answer to W4**: Our proposed two metrics, although they resemble extensions of existing metrics, are appropriately repurposed to offer significant value for the L2I problem as 1) ​​These metrics are specifically designed to quantitatively measure L2I performance in rich-context scenarios, where existing metrics have failed to do so (Ln 176-183). 2) In addition to proposing these metrics, our reliability analysis is crucial. It demonstrates the conditions under which these metrics are effective and consistent with human perception. We argue that this analysis should be considered a contribution to the field of evaluation metrics.\n\n**Answer to W5**: In Rebuttal Sec E, we provide a visual comparison to demonstrate the effectiveness of using region reorganization compared to straightforward feature averaging (the baseline described in Ln 279) when dealing with overlapping objects. The qualitative results validate the importance of our proposed methods, as they produce more accurate generated objects that better align with the designated layouts.\n\nFor the ablation study on box indicator and image resolution, identifying a failure generation without one of these components can be deliberated, and there may not be a strong sample-wise effect when applying them. Therefore, we recommend referring to the quantitative performance metrics for a more accurate comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
The figures, tables, and pseudo-codes for the rebuttal are presented in the PDF file. We appreciate the reviewers for taking the time to read and consider them.

Meta-review: The paper presents a layout-to-image (L2I) generation method focusing on rich-context scenarios where object descriptions are complex and lengthy. The authors introduce a regional cross-attention module designed to enhance the representation of layout regions, addressing challenges related to overlapping objects and complex descriptions. Additionally, the paper proposes two new evaluation metrics aimed at assessing model performance in open-vocabulary L2I tasks, validated through a comprehensive user study.

The reviewers in general acknowledge the novelty of the proposed method and evaluation metric, and are satisfied with the experimental thoroughness. The reviewers have largely been satisfied with the authors' rebuttals, and the scores have been raised accordingly, indicating that the concerns have been adequately addressed. However, the authors should take into account the reviewers' suggestions to improve the clarity and accessibility of the paper.

The AC agrees with the reviewers to accept the submission.
Decision: Accept (poster)