HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation
Authors: Bocheng
YuhangMa
wuliebucha
Shanyuan Liu
Ao Ma
Xiaoyu Wu
Dawei Leng
Yuhui Yin
Venue: NeurIPS 2024 poster (2024)
Keywords: Text to Image
Latent Diffusion Model
Layout to Image
Primary area: generative_models

Abstract: The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a \textbf{Hi}erarchical \textbf{Co}ntrollable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi branch structure to represent hierarchy and aggregate them in fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned. https://github.com/360CVGroup/HiCo_T2I.

HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation
Bo Cheng
Yuhang Ma
Liebucha Wu
Shanyuan Liu Yuhui Yin
Ao Ma
Xiaoyu Wu
Dawei Leng
360 AI Research
{chengbo1, mayuhang, wuliebucha, liushanyuan) cn {maao, wuxiaoyul, lengdawei, yinyuhui)@360.cm cn
(a) Layout grounded generation with closed-set short descriptions
(b) Layout grounded generation with open-ended fine- aineggrinee descriptions
(c) HiCo is compatible with different SD variants
(d) Multi-concept generation by HiCo with multi LoRAs
(e) Fast inference with HiCo-LCM / HiCooLiggtning
Figure 1: HiCo model serves to enhance layout controllability for text-to-image generation, by inte- grating bounding box condition different objects hierarchically. The proposed unique conditioning branch structure can produce more harmonious and holistic image with complex layout.
Abstract
The task of ttyout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object miss- ing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, propose Hierarchical Controllable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is achieve spatial disentanglement through hier- archical modeling of layouts. We use a multi branch structure to represent hi- erarchy and aggregate them in fusion module. To evaluate the performance multi-objective controllable layout generation in natural scenes, we introduce the
*Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned.
Introduction
Text-to-image (T2I) 1+ diffusion models like Stable Diffusion, GLIDE(], have rapidly developed for their exceptional quality and diverse generative capabilities. However, the T2I models lack fine-grained control over visual composition and spatial layout via text prompts alone.
Layout-to-image generation I 4 [5]. which aims to produce high-quality and realistic images from layout conditions. This article mainly studies the generation of layout images based on object text description and positional coordinates. Existing methods can be mainly divided into two categories: training- free methods [ and training- -aaied methods[4] [2]. Training- free methods usually use cross- attention to get the ability to control position. Training- -base methods typically utilize new network structures specific attention. As shown in the Figure 2 in complex scenarios, the training-free method represented by CAG 6
has a serious problem of object missing. The training- based methods represented by GLIGEN can alleviate the phenomenon of object missing, but the generated images often exhibit distortion.
Leyout
GroundTruth
CAG
GLIGEN
HICo
Figure 2: The generation of CAG(6) GLIGEN 4 and HiCo in complex layouts.
To address the issues, we propose the Hierarchical Controllable(HHiCo) diffusion model. The approach disentangle the spatial layouts by multiple branch networks. Specifically, the branch design of HiCo is inspired by external condition introduction methods similar to ControlNet[ and IP-Adapter [4). Hierarchical layout features are separately extracted by branches of weight sharing, and refinedly aggregated with Fuse Net. The overall architecture of our approach is shown in Figure 3 Our method demonstrates superior performance in terms of object omissions and image quality shown in Figure 2| This is attributed to our hierarchical modeling approach, which has particular advantages in generating complex layouts.
To further validate the effectiveness of our method, we conducted experiments on both the closed-set COCO dataset and the open-ended GRIT dataset, and achieved excellent performance both. Fur- thermore, our method exhibits a flexible scalability, including switching checkpoints and integrating plugins like LoRA, LCM. Refering to Figure for details.
Due to the lack of objective benchmark for multi-objective controllable layout in natural scenes, we introduce the HiCo-7K benchmark. HiCo-7K is uniformly sampled from GRIT-20M dataset, revalidated for object regions using Grounding- Boounding-DNNOIIE DINO and filtered based on semantic relevance using [T]. Furthermore, we conduct deep manual cleaning on it.
Our primary contributions are shown as following:
1. We propose the HiCo model, which achieves spatial disentanglement through hierarchical model- ing of layouts. Our method can generate more desirable images in complex scenarios, and exhibit a flexible scalability.
2. We propose benchmark HiCo-7K, which has been revalidated and cleaned by algorithms and professionals. It can objectively evaluate the task of layout image generation in natural scenes.
Our method achieves state-of-the-art performance on both the open-ended HiCo-7K dataset and the closed-set COCO-3K(I8) dataset.

---

2 Related Work
2.1 Diffusion Models
Diffusion models generative models that synthesize images from random noise by iterative image denoising, They have showed its excellent potential to generate high semantic relevance and aesthetic quality images than GAN-related models [19]. DDGAN(20), DiffusionVAE(2) study the combination of diffusion model and other generative methods. Compared to denoising diffusion probabilistic models (DDPM) 22), DDIM [23] and PLMS[24] mitigate the lengthy sampling proce- dure by reducing number of sampling steps. The latent diffusion model(LDM)]
leverages VAE to encode images to latent codes with smaller resolution, saving efforts to train super-resolution models for generating high-resolution images, (ontrolNet(33) and IP- Pe-AAdaterr[L [4] enabled additional saage-guided conditions into frozen T2I diffusion models (e.g., sketch, segmentation masks, canny edge).
2.2 Layout-to-Image Generation
Layout-to-image technology seeks to generate realistic images with corresponding spatial layouts based on graphical or textual inputs that convey layout information. Early layout-to-image techniques primarily used GAN-related technologies (25 (7) Although these works achieved encouraging progress, their generation effects and application scenarios were extremely limited.
Different guiding conditions(28]33 endow diffusion models with diverse capabilities. Researchers have proposed various methods generate layout-controllable images using object descriptions and spatial positions. They mainly design a brand new network or special attention, such as layout attention or attention redistribution. Our approach build basic pre-trained model by introducing weight-shared multi-branch structures for enhanced local controllability.
the combination of large language model(LLM) and diffusion model, can enhance strong performance in cearucctonnfollowwiin and controllable image generation and editing tasks. Unlike HiCo which requires layout and image specification directly from input, LMD 34 and SLD 35 resort to LLM for image scene description and layout arrangement via text automatically. It's worth pointing out that HiCo be integrated with LMD and SLD, by treating HiCo as the replacement of their train-free layout controllable image generation module.
Researches 32 33
3 Method
3.1 Preliminary
The SD model a diffusion model that operates within the latent space, which consists of three main components: The VAE-encoder[36) encodes images into the latent space, while the VAE-decoder reconstructs the latent representation into realistic images. The CLIP 37 text encoder projects a sequence of tokenized texts into the sequence embedding. The UNet model is trained to predict the added Gaussian noise using LDM loss.
LLDM E((.). [|le eo +
where is uniformly sampled from time steps (1....,?)) T}, 22 is 1-step latent of the input. eg is noise prediction model.
The conditional guided generation SD involves incorporating text conditions, reference images, masks, and other conditions into the SD model through various techniques. Controlnct is common method of introducing external control conditions through collateral structures, and its training goal is to predict noise at different stages with a learnable branch network, denoted eg. Given the latent 2o, the model adds noise to reach 2t, where t represents the number of noise-adding steps. Here, ca indicates the textual control condition, and ey stands for specific control condition. The objective function can be expressed as a function Leondition-
Leondition Em N(00.)) [lle t,
(2)
3

---

3.2 HiCo Diffusion Model
We adopt a common external condition introduction method similar to ControlNet and IP-Adapter, and explore its innovative application in the design of controllable layout networks. We proposed a multi- culti-bbanchhhh HiCo Net, which independently models the background and multiple foregrounds, and hierarchically expresses the local semantics and spatial layout relationship of the image fine-grained manner. On the fusion of branches, we experimented with variety of fusion methods and proposed non-parametric Fuse Net, decouples branches through masks and achieves excellent performance. The overall network structure is shown in Figure
HiCo The multi- multi--ranchh HiCo Net introduced to generate the global background and foreground instances for different layout regions. The branch of HiCo Net adopts the structure of ControlNet, independently decoupling layout conditions to hierarchically model foreground objects.
LHi.o o. [lle E, 1
Here, of represents the textual control condition of k-th sub area, and cf represents the bounding box control condition of k-th sub area. F' represents the Fuse Net.
We define the instruction payoutto-imaag model composition of sub-caption and sub-
boundingbox.
Instruction :y (c,, b,), [1, K1, with
caption [Cg. C1,..., ...,, Ci Ck)
boundingbox [bg. bı, .... bi bx]
where represents the number of regions, with cy and bg representing the textual descriptions and positions of the background image, c; and b, corresponding to the descriptions and positions of the individual regions. The HiCo Net processes different regional conditions to generate intermediate features that correspond to the textual descriptions within the predefined regions.
Timed sooder
Cacader
Siption
Apoood
DownBlock
HiOo Net
HiCo Not
HiCo Net
Weeght Share
UpBlock
Unet
Fuse
Frozen Medules
Modules
Figure 3: The overall architecture of our approach.
Fuse Net. The module fuses intermediate features from sub-regions and acts as the external features of the UNet model. It have different forms, including summation, averaging, mask, learnable-weights, and other methods. Based on different task types, various fusion structures can be selected. Our approach mainly decouples the content of different foreground and background regions through the mask fusion form. The detailed fusion process is defined as:
K
K
F(c,, cb) =Mg Chico &.
Mr Chico tt)
D
M
(5)
k=1
k=1

---

Here, of oi represent the text and position control conditions of k-th sub area, and c, of represent the textual description and position of the background image. Mr represents the k-th object area mask information, and Mg represents the background area mask information.
HiCo Net
HiCo Net,
HiCo Net,
HiCo Net
Hico Net,
HiCo
down ock lppr
down blok biee
funs byer
Diffusion Model
Figure The visualization of the features on different layers of the HiCo branch and Fuse Net.
conditions
conditions
conditions
conditions
20240515 02240111155 $1259
HiCo
Diffusion Model
(6) LNNetLLOA
(b) HiCa-LoRA
Figure 5: The model fine-tuning technique based on various positions LoRA. (a) Adding LoRA parameters on UNet. (b) Adding LoRA parameters HiCo.
3.3 Hierarchical Controllable Analysis
The introduction of external control conditions through side branch structure is common condition guidance approach in diffusion models. Our HiCo model employs an innovative weight-sharing mech- anism across its branch structures, which adeptly generates distinctive features for both foreground objects and the background image, tailored to the specific conditions of the caption and bounding box. These features strategically integrated during the upsampling phase, critical step in the diffusion model's generative workflow.
Figure Suure4depee depicts the generative process of the HiCo model for four foreground objects, showcasing the visualization of features of layer-2, layer-5, and layer-1l the 50th denoising step during the downsampling stage. The visualization of shallow features reveal that various branches exhibit more pronounced response their respective layout areas. The intermediate features indicate further refinement of object positions, contours, and semantics generated by different branches. Furthermore, the deep features underscore the model's adept handling of regional information, essential for the controlled layout of the image. The fusion process of the HiCo branches employs mask technique.

---

However, it is crucial to elucidate how this fusion contributes to the overall semantic coherence of the spatial layout.
3.4 Expansion Capability
Low-Rank Adaptation (LoRA)B8 stands out efficient fine-tuning technique. Our HiCo model exhibits excellent compatibility with rapid generation plugins, whether it's utilizing LCM-LoRA for quickly generating 512x512 resolution images HiCo-SD1.5 or Lightning- LoRA/Lightning-UN 40] for quickly producing I 1024x 1024 resolution images on HiCo-SDXI.
Multi-concep((4]443 controllable layout generation effectively blend different elements such as characters, style, and objects into a cohesive image. Integrating multi-LoRA models of the same type into the UNet can easily lead to confusion of different conceptual features. This will make it difficult to naturally generate different conceptual elements the same image. Specifically, training LoRA on the HiCo branch, as shown in Figure has been shown significantly enhance performance in scenarios requiring the injection of multiple concepts. For more results, please refer to Appendix A..5
4 Evaluation
4.1
Dataset
The HiCo model various types of grounding data achieve controllable layout generation across different scenarios.
Open-ended Grounded Text2Image Generation. For training datasets, the fine-grained detailed description data, comprises 1.2 million image-text pairs with regions and descriptions sourced from GRIT-20M [5). We performed algorithms and manual cleaning on the raw data, resulting in dataset with an average of 4.3 objects per image.
Closed-set Grounded Text2Image Generation. For training datasets, the coarse-grained categorical description data, we selecte subset of approximately images from COCO Stuff[[8] based on criteria such region size, labeled as COCO-75K. This dataset includes 171 categories and an average of 5.5 objects per image.
Evaluation Dataset. The evaluation datasets include two types of data: the coarse-grained COCO-3K and the fine- fine-grained HiCo-7K. We introduce the HiCo-7K benchmark for evaluation of multi- objective controllable layout in natural scenes. The HiCo-7K dataset is derived from GRIT-20M and has undergone iterative cleaning through both algorithmic and manual processes. It consists of 7,000 images, with an average of 3.75 objects per image. We have detailed the construction pipeline of the custom dataset HiKo-7K in Appendix A.2
4.2 Experimental Setup
We apply the HiCo architecture to various network structures such SD1.5, SD2.1, SDXL 44 to achieve controllable generation. Specifically, for SD1.5, We utilize the AdamW optimizer with a fixed learning rate of le-5 and train the model for 50,000 iterations with a batch size of 256. We train HiCo with A 100 GPUs for 3 days. Training HiCo-SDXL requires more iterations and fine-tuning on smaller set of high-quality data. After training, our HiCo model also supports rapid generation plugins with LoRA, LCM, SDXIL-Lightning.
4.3 Quantitative Results
4.3.1 Coarse-Grained Closed-set Text2Img Generation.
We train and evaluate HiCo model on COCO-stuff datasets to develop its layout-to-image capabilities. We use Frechet Inception Distance(FID)) 45 to evaluate the perceptual quality of the generated images. We use YOLO score to evaluate the recognizability
the object in the generated images. YOLO score uses pretrained YOLOv4 46 model detect the objects in the generated images.
The HiCo model achieves the best results image fidelity and spatial semantic dimensions, and the generated images have more beautiful and reasonable visual effect shown in Figure Mcanwhile,

---

it can generate images with a resolution of 512 and not just the categories of COCO-3K, which is an ability that other models do not possess. The quantitative comparison presented in Table
Leyos
Groundfrsth
Figure 6: Qualitative comparison of HiCo and other methods COCO-3K. Compared with other methods, HiCo can generate high-quality images under complex layout conditions. The cases generated by LAMA, LDM and LayoutDiffuse is referenced from LayoutDiffuse.
Quantitative comparison of HiCo and other methods COCO-3K. All generated images are evaluated under 256 resolution. f indicates that the experimental value referenced from LayoutDiffuse.
Table 1:
Methods
FID
APT
AP501
AP751
ARt
LAMA
31.12
20.61
28.54
22.69
26.48
LDM+
24.60
32.13
54.23
34.34
39.86
LayoutDiffuse
10
20.27
36.58
59.59
38.06
46.09
LayoutDiffusion 11
48.77
14.20
24.2
14.3
16.2
HiCo(Ours)
20.02
36.60
58.10
39.4
46.60
4.3.2 Fine-Grained Open-ended Text2Img Generation.
We train and evaluate HiCo using a high quality data of natural scenes. We use FID, Inception Score (47), Learned Perceptual Image Patch Similarity (LPIPS)488 to evaluate the perceptual quality of the images generated with layout information. The results are presented Table Our model can generate high-quality images with rich objects even in complex scenarios, shown in Figure 7
Furthermore, we use Frounding-DIIOO [6] to detect the instance caption and calculate the maximum loU between the detection boxes and the ground truth box. If the maximum IoU is higher than the threshod 0.5 and the Local CLIP Score [99 them is higher than 0.2, we mark as position correctly generated. We use AR, AP, AP50 and AP75 to evaluate the spatial performance. The results are presented in Table 3
Table 2: Quantitative comparison of perception dimension with other models HiCo-7K. All generated images are evaluated under 512x512 resolutio.. The results indicate that HiCo has better fidelity and perception.
Methods
FID
1ST
LPIPS
MDDM
24.83
25.27+1.54
0.76+10.05
GLIGEN
19.65
26.59+1.45
0.73
CAG
19.37
26.24++1.42
0.76+0.06
MIGC
26.92
22.17+:1.16
0.77+0.07
InstanceDift
16.99
26.19+11.09
0.7310.06
HiCo(Ours)
14.24
28.31
0.73

---

Table 3: Quantitative comparison of spatial location dimensions on HiCo-7K. Experiments show that HiCo has better positional control and image text consistency.
Methods
LLIP Scoret
LocalloU Scoret
AR
APt
AP501
AP751
GroundT 25.07
66.40
49.45
30.13
42.77
31.75
MIDM
24.81
25.02
5.21
0.59
2.45
0.13
GLIGEN
24.96
48.54
28.33
12.67
24.15
12.14
CAG
24.54
24.43
4.62
0.47
2.01
0.21
MIGC
24.94
48.03
25.8
10.18
23,42
7.56
InstanceDiff
25.09
51.58
33.92
16.76
27.45
17.14
HiCo(Ours)
25.17
59.98
41.21
21.53
33.22
22.66
Figure 7:
Qualitative comparison with other models on HiCo-7K. Compared with other methods, HiCo can generate high-quality images for both simple and complex layout information.
Zero-shot Evaluation. We further evaluate the zero-shot performance of HiCo trained in natural scenes on COCO-3K, detailed results are shown in table The preferences of HiCo controllability the best on COCO-3K. The reason for this problem that our model was trained on a 1.2M fine-grained long caption, which belongs to out of distribution data for COCO data.
Table 4: Quantitative comparison of spatial location dimensions zero-shot capability on COCO-3K.
Methods
LocalCLIP Scoref LocalloU Score
ARt
AP1
AP501
AP751
GLIGEN
25.83
73.17
53.39
36.87
65.35
38.09
MIGC
25.94
75.06
57.55
40.82
70.57
42.21
InstanceDiff
26.60
85.91
78.10
65.28
80.56
71.93
HiCo-Rcal(Ours)
26.27
79.57
68.40
52.47
78.61
57.05
4.4
Human Evaluation
We use a multi-round and multi-participant cross-evaluation approach to assess human preferences, focusing aspects such as object quantity, spacial location, and global image quality. Details the experimental setup can be found in supplementary material of Appendix [A.4]
Spatial Location & Semantics. Target quantity dimension assesses whether the number of objects in the generated image aligns with the preset value. The semantics and position dimension examine the relevance of the objects to the textual description and their regional placement within the image.

---

Global Image Quality. The image global quality dimension includes five sub- tib-dimensions: relativity. clarity, rationality, aesthetics and risk.
Table 5:
The human evaluation results encompass two dimensions: spatial semantic location and global image quality. The numerical range is from 0 to 100, with a higher score indicating better performance. It should be noted that the risk dimension is the proportion of generated risk images, the lower the better.
Spetial Location& Semantics
Global Image Quality
Method
Overall
quantity semantics
position
relativity clarity
rationality aesthete risk.
GroundT
100
100
100
93.36
90.69
97.34
92.56
0.00
95.62
SD-Real
80.56
67.41
83.58
60.62
63.67
2.00
MiDM
54.96
33.26
22.04
48.08
59.04
40.49
40.91
9.00
39,41
GLIGEN
80.44
73.56
72.78
56.78
61.67
53.67
51.89
4.44
67.76
CAG
50.89
32.11
37.89
48.67
55.11
49.00
44.89
7.33
43.94
MIGC 69.89
62.44
62.67
49.22
57.44
47.00
46.67
6.11
59.03
InstanceDiff
88.33
80.00
66.78
61.56
62.33
56.00
55.33
2.50
70.54
HiCo 89.44
86.11
82.22
63.67
82.33
58.33
60.89
2.00
78.08
Table demonstrates the human evaluation results conducted on 300 controllable layout images. The results indicate that, in terms of spatial position and semantic dimension, our approach outperforms other models. Moreover, it achieves near-par performance the Realistic V51 model(SD- Real) in the fine-grained dimension of global image quality, suggesting that despite the enhanced controllability, the generative capabilities of our model remain robust and effective.
4.5
Ablation Studies
The ablation focuses on UNe[GioaaCCaptinnUGC) GlobalBecgrrundBraachhh FuseNet(FN), Furthermore, FuseNet is con-paraettric network that includes the following types, such as (I (1)Summation. (2)aveee (3)Mask. Experiments are performed on HiCo-7K using the HiCo-base model with the same training amount. The results are presented in Table
and
Table 6: The results of ablation studies on HiCo-7K of UGC, GBB and FN.
UGC
GBB
FN
FID
ARt
APM
LocalCLIP Scoref
LocalloU Scoret
sum
20.81
33.12
16.54
25.18
52.25
X
avg
18.78
36.3
18.58
25.02
56.67
V
mask 15.26
39.51
20.02
25.22
59.07
V
mask 16.85
38.19
19.05
25.23
57.03
V
mask
21.82
30.07
12.07
25.24
49.64
4.6 Inference Performance
For inference run time and memory usage, conducted two additional comparisons. The first comparison is horizontal, among different models including: GLIGEN, InstanceDiff, MIGC, CAG, MtDM well our HiCo. Specifically, we evaluated the inference time and GPU memory usage for directly generating 512x512 resolution images on the HiCo-7K using a 24GB VRAM 3090 GPU, results in Figure (a) show that HiCo has the shortest inference time and the 2nd lowest GPU memory footprint.
The multi-branch of HiCo has two inference modes: "parallel mode" and "serial mode". " In order to verify the performance advantages of HiCo when the number of objects increases, the second comparison is vertical: we assessed the inference time and GPU memory usage for generating 512x512 resolution images on the HiCo-7K with different number objects. Since each object processed by separate branch in HiCo, the inference can be accelerated by inferring all the branches one batch, in "parallel mode". which as shown Figure (b), much faster than the "serial mode". inferring all the branches one by one in serial.

---

7000
GPU (M)
GLISEN
GPU (M)
6000
Serial
Paral
Iratanoe
5000
DMC
4000
6:10
3000
HCD
2000
1000
10
15
Time
Time: (8)
(8)
(b)
Figure 8: Quantitative comparison of inference performance. (a) Comparison between different
methods on HiCo-7K. (b) Comparison between different object quantities on HiCo.
5
Conclusion
HiCo is a controllable layout generation model based on diffusion model, guided by multiple branch structures. This approach allows users specify the location and detailed textual descriptions of target regions while maintaining the rationality and controllability of the generated content. Through training and testing data with varying degrees of granularity in natural scenarios, well as conducting algorithm metric evaluation and subjective human assessment, the superiority of this method demonstrated. However, there is still potential for further improvement, especially in the areas of image content editing and integrating multiple style concepts. By combining current controllable generation capabilities can boost the overall playability of Al-generated artwork.
Limitation. The complex interactions and occlusion order of overlapping areas are significant chal- lenges for HiCo model and even the field of layout-to-image generation. HiCo achieves hierarchical generation by decoupling each object's position and appearance information into different branch, meanwhile controlling their overall interactions through background branch with global prompt and the Fuse Net. HiCo capable of handling complex interactions in overlapping regions by FuseNet. The occlusion order of overlapping objects also specified via the global prompt by text description. But since there lacks corresponding occlusion order train data, the success rate is far from optimal. For current version of HiCo, there indeed lacks of more explicit mechanism for occlusion order controlling. For more results, please refer to Appendix A.3 We also found that the HiCo model still does handle the generation of complex layouts for multiple concepts of LoRA very well. We intend to conduct further research to explore solutions to these issues in the future.
Social impact aspect. Our model is designed assist users in generating creative images with controllable layouts. However, there is risk of misuse our method to generate inappropriate or sensitive content. Therefore, we believe that regulating the application of such models and developing risk detection tools are crucial. This will facilitate the progress of AI technology for the benefit of humanity.
10

---

References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolttinn image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[2] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 0rXiv:2204.06122 1(2)3, 2022.
[3] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv: 10741, 2021.
[4] Yuheng Li. Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jac Lec. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511 1-22521, 2023.
[5] Zejian Li, Jingyu Wu, Immanuel Koh, Yongchuan Tang, and Lingyun Sun. Image synthesis from layout with locality- aware mask adaption. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13819-13828, 2021.
[6] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross- atssnttttntionn guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5333, 2024.
Peiang Zhao, Han Li, Ruiyang Jin, and Kevin Zhou. Loco: Locally constrained training-free layout-to- image synthesis. arXiv preprint arXiv:231 1.13442, 2023.
[8] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng. and Mike Zheng Shou. Boxdiff:: Text-to-image synthesis with training- box cox-tonntra diffusion. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 7452-7461, 2023.
[9] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023.
[10] Jiaxin Cheng, Xiao Liang, Xingjian Shi. Tong He, Tianjun Xiao, and Mu Li. Layoutdiffuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint 23Xi:23302.8800 2023.
[11] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusson: Controllable diffusion model for snout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22490-22499, 2023.
[12] Dewei Zhou, You Li, Fan Ma, Zongxin Yang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. arXiv preprint 4r2.v:2402.05408, 2024.
[13] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3846-3847, 2023.
[14] Hu Ye, Jun Zhang. Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for set-to-image diffusion models, arXiv preprins arXiv:2308.0677 06721, 2023.
[15] Zhiliang Peng. Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[16] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li. Jianwei Yang. Hang Su. Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint MXXi:2333300599 2023.
[17] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763, PMLR, 2021.
[18] Holger Cacsar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1209-1218, 2018.
[19] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53 65, 2018.
11

---

[20] Yu Chen, Weida Zhan. Yichun Jiang, Depeng Zhu, Renzhong Guo, and Xiaoyu Xu. Ddgan: Dense residual module and dual-stream attention-guided generative adversarial network for colorizing near-infrared images. Infrured Physics & Technology. 133:104822, 2023.
[21] Kushagra Pandey, Avideep Mukherjee. Piyush Rai, and Abhishek Kumar. Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents, arXiv preprint arXiv:2201.0053 2022.
[22] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[23] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
[24] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 2022.
[25] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[26] Wei Sun and Tianfu Wu. Learning layout and style reconfigurable gans for controllable image synthesis. IEEE transactions on pattern analysis and machine intelligence, 440)):070-55077, 2021.
[27] Bo Wang, Tao Wu, Minfeng Zhu, and Peng Du. Interactive image synthesis with panoptic layout generation. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7783-
7792, 2022.
[28] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffuusion models with an ensemble of expert denoisers, arXiv preprint 0rXiv:2211.01322 2022.
[29] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[30] Amir Hertz, Ron Mokady, Jay Tenenbaum, Ktir Aberman. Yael Pritch, and Daniel Cohen- Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208 01666,, 2022.
[31] Dave Epstein, Allan Jabri, Ben Poole, Alexci Efros, and Aleksander Holynski. Diffusion self- aaf--guidace for controllable image generation. Advances in Neural Information Processing Systems, 36:16222-16239, 2023.
[32] Yingqing He, Zhaoyang Liu, Jingye Chen, Zeyue Tian, Hongyu Liu, Xiaowei Chi, Runtao Liu, Ruibin Yuan, Yazhou Xing, Wenhai Wang, al. Lims meet multimodal generation and editing: A survey, arXiv preprint ar Xiv:2405. 19334, 2024.
[33] Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang. and Zhe Gan. Guiding instruction- based image editing via multimodal large language models. arXiv preprint ar 2rXiv:2309. 2023.
(34] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell, Lim Am--rrunded diffusion: Enhancing prompt under- standing text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 13655, 2023.
[35] Tsung- Han Wu, Long Lian, Joseph Gonzalez, Boyi Li, and Trevor Darrel.. Self-correcting I1m- controlled diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6327-6336, 2024.
[36] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[37] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint axXiv:2010.11922 2020.
[38] Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2023.
[39] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang. Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint arXiv:2311005566 2023,
12

---

[40] Shanchuan Lin. Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint TrXXv:2402.13929 2024.
[41] Nupur Kumari, Bingliang Zhang, Richard Zhang. Eli Shechtman, and Jun-Yan Zhu, Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition. pages 1931-1941, 2023.
[42] Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arXiv preprint arXiv:2402. 2024.
[43] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311.1366 2023.
[44] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.019555 2023.
[45] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[46] Alexey Bochkovskiy, Chien- Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
[47] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[48] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586 595,5555,220 2018.
[49] Omri Avrahami, Thomas Hayes, Oran Gafni,, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Yin. Spatext: Spatio-textual representation controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18370- 18380, 2023.
[50] Xudong Wang. Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6232-6242, 2024.
13

---

A Appendix supplemental material
Overview
In this supplemental material, provide the following items:
Training and Inference Strategies.
HiCo-7K Benchmark.
Limitaion Discussion.
Manual Evaluation Criteria.
More results on HiCo, including different layout quantities, different base models, and different resolutions.
A.1 Training and Inference Strategies
During the training stage, only optimize the parameters of the HiCo Net, while keeping the SD base model parameters fixed. The Fuse Net can use either non-parametric methods, like summation, averaging. mask parametric method, like simple MLP structure.
During the inference stage, the structure of the Fuse Net can be reasonably selected according to the size and importance of the foreground and background regions achieve controlled generation effects in different scenarios. Additionally, LoRA network parameters can be added to the HiCo Net fine-tune and learn new tasks, such personalization, multi-language controllable generation and other tasks.
A.2 HiCo-7K Benchmark
We have detailed the construction pipeline of the custom dataset HiCo-7K in Figure 9 We found that GRIT-20M has some issues, such low labeling rate for objects with the same description and target descriptions being derived solely from the original captions. Compared to GRIT-20M, the pipeline of the HiCo-7K is as follows.
1. Extracting noun phrase. We use spaCy to extract nouns from captions and the LLM VQA model remove abstract noun phrases. Meanwhile, we the DrONndingDINO model extract richer phrase expressions.
2. Grounding noun phrase. We use the Grounding model to obtain the bboxes. After that, we use NMS and CLIP algorithms clean the bboxes.
Artificial correction. To address the issue of algorithmic missed detections for multiple objects with the same description in an image, artificial correction is employed to further enhance the labeling rate of similar objects.
Multi-captions with bounding box. We expand the basic text from the original captions and use GPT-4 re-caption the target regions. The dataset of HiCo-7K contains 7000 expression- bounding-bow pairs with referring expressoos and GFFT-eenrraeed-xxpresioo.
A.3 Limitaion Discussion
HiCo achieves hierarchical generation by decoupling each object's position and appearance infor- mation into different branch, meanwhile controlling their overall interactions through a background branch with global prompt and the Fuse Net. The Fuse Net combines features from foreground and background regions, well as intermediate features from side branches, then integrates them during the UNet upsampling stage. As illustrated in Figure 10 (O) HiCo is capable of handling complex interactions in overlapping regions without any difficult. The occlusion order of overlapping objects is also specified via the global prompt by text description, for example "bowl in front of vase". as illustrated in (c) data, the success rate is far from optimal. For current version of HiCo, there indeed lacks of more explicit mechanism for occlusion order controlling.
(d). But since there lacks corresponding occlusion order train
14

---

We recognize this problem as our future work. Actually we re already working on the occlusion order data curation, which is quite challenging task as requires reliable depth estimation besides the object detection bounding boxes.
248,
SparCy
Bimmhatc
aggresavely immediataly following
Dases
363,
A moose
NMS Filtoingg
Grounding
A buah
DIV
A yellow call
Moosi:
[52. 55, 248, 2081
bush:
[153, 242, 353, 200)
cow moose.
Ariffcie
smail black moose bent down to its young
conection
[92, 25. 248, 293)
Compose
Moesc:
[52, 56, 248, 203)
green gross
A bust:
badiground
VIM recastion
[153 242 353, 290 Calt
353,
[257. 251, 301, 302]
Figure 9: The pipeline of constructing HiCo-7K with grounded image caption pairs.
agn
(+)
(b)
dining
(c)
(d)
Figure 10: The image generated in overlapping and interactive scenarios. generated caption is displayed in the image.
15

---

A.4 Manual Evaluation Criteria
Evaluation Dimensions
We designed the evaluation dimensions by reviewing existing literature and soliciting opinions from professional designers and a large number of users. Specifically, we divided the evaluation dimensions into two categories: local and global. Local dimensions include spatial location and semantics, such as the quantities, semanties, and positions of the object bounding boxes. Global dimensions include global image quality, such as relativity, clarity, rationality, aesthetics, risk, and overall score.
Dimension Definitions
Spatial Location& Semantics. Local dimensions include three sub-dimensions: quantity indicates the number of generated objects; semantics measures the consistency between the objects within the bounding box and the textual description of that region; position measures the deviation between generated objects and ground truth by calculating the IoU of bounding boxes.
Global Image Quality. The image global quality dimension includes five uub-dimensions: relativity is primarily used evaluate the understanding and representation of text by image content; clarity is commonly used metric for assessing image quality; rationality is used to describe the distortion, deformation, and disarray of image content; aesthetics encompasses the overall assessment of the visual appeal of the generated image, incorporating factors such as detailed depiction, color usage, creativity, and other relatively subjective judgment elements; risk evaluates elements related to nudity, violence, terror, and other sensitive content within the image.
For risk assess the presence of such elements and represent using binary values of 1. For the other dimensions, we categorize them into four levels ranging from 0 to and then normalize them to 0-100. We use overall score to represent the comprehensive assessment of the image.
Evaluation Execution
The evaluation team consists of professional evaluators. They have rich professional knowledge and evaluation experience, allowing them to accurately execute the evaluation tasks based on the given dimensions.
Specifically, our evaluation score is computed with the following steps:
1. According to the rules provided, evaluators rate each image on each dimension. Risk is scored as either or 1, while the other dimensions range from 0 to 3.
2. Calculating the overall rate for each image: We calculate the overall rate by summing the weighted scores each dimension. For the sub-dimensions of the local dimension, the weight is 0.2, while for the sub-dimensions of the global dimension, the weight is 0.1. The final score for overall rate calculated as the weighted sum the seven dimensions multiplied by the score of risk (which is either 0 1).
We calculate the mean of each dimension across the entire evaluation set, excluding the risk dimension, and mapped the scores to scale of 100. For risk, calculate the coverage rate of this dimension.
A.5 More Generation Results
Figure I1 shows more results generated by HiCo using fast generated plug-in LoRA. The four columns from left to right represent the generated results of 50-steps, 4-steps, 6-steps, and 8-steps, respectively.
Figure (2) shows more results generated by HiCo-SDI.5 in HiCo-7K. Figure (2-(b) shows more results generated by HiCo-SDXL HiCo-7K. The number of object layouts ranges from 3 to over 10. Despite complex layouts and rich descriptions, HiCo reliably ensures that each object is generated in the correct position with the right description.
Figure 12
(C) shows more results generated by HiCo using different checkpoint from open source community. The results from the second to fourth lines are generated by the following three models, namely disneyPixare PixarCarton,, flat2Dnnmerre and dreamshaper.
16

---

Figure (d) show the generation of different layout information with LoRA. Rows 1-3 demonstrate that HiCo can effectively generate complex layouts for single concept with single LoRA. Row shows the generation of complex layouts for multiple concepts with multi LoRAs using the HiCo model.
Through further experiments, we found that the HiCo model still does not handle the generation of complex layouts for multiple concepts very well. More effective methods need be explored to address this issue.
Figure 11: Qualitative experiments. Fast generation of complex layout information with HiCo-LCM / HiCo-Lighning.
17

---

1abe proce astiv
FIn
EWN
OLITES
Generation of complex layout information of HiCo-SD1.5.
(b) Generation of complex layout information of HiCo-SDXL.
(c) Generation of complex layout information of (d) Generation of concepts by HiCo with multi HiCo with different backbones.
LoRA.
Figure 12: Qualitative experiments generated by HiCo model to show the layout controllability
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect our paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of the work Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided detailed descriptions of the training data, network architecture, and experimental details ensure the reproducibility of the results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [No]
Justification: The paper currently does not provide open-source code. However, we are actively working on and plan to release it soon as possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli///uuu guwde//ddee/eee Coc//ooodeeessseeeeeeo hwtpbbmmsssioooooooooooi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided detailed experimental details and test configurations in Section including datasets, hyperparameters, and
4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the relatively high cost associated with training and evaluating genera- tive models, is challenging for us to report the Experiment Statistical Significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:In the experiment, we utilized the GPUs from our internal cluster, specifically training with 8 A100 cards for 3 days. we include the resources for the exploratory phase, the total amount of resources consumed would be even greater.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ttp::::////uiippstt wttpiipp.ppcppppppe httppu/iiiiiiiiccciiieeeeeeeeese
Answer: [Yes]
Justification: The research conducted in the paper complies with the NeurIPS Code Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader effects of our research are examined in Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: While specific actions have been taken tackle this matter at present, are dedicated to reducing the risk before introducing our generative model to the public.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We utilize open-source assets and acknowledge them accordingly in references.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
No new assets are presented in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The research conducted for this paper does utilize crowdsoureing or engage with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research conducted for this paper does not utilize crowdsourcing or engage with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "uTHHqo06EK", "reviewer": "Reviewer_72RR", "summary": "This paper propose HiCo (Hierarchical Controllable) Diffusion Model for layout-to-image generation. HiCo Net is a multi-branch structure that is introduced to hierarchically generate the global background and foreground instances for different layout regions. The author further evaluate the performance of multi-objective controllable layout generation in natural scenes and introduce HiCo-7K benchmark.", "questions": "1.How to manage the overlapping issue mentioned above?\n\n2.How does the runtime compare with previous works ?\n\n3.There are similar works like LMD[1] and SLD[2], discuss the differences between HiCo and these two works.\n\n4.Could HiCo Net be considered a variant of ControlNet within the modality of bounding box-based layout?\n\n[1]Lian, Long, et al. \"Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.\"\n\n[2]Wu, Tsung-Han, et al. \"Self-correcting llm-controlled diffusion models.\"", "limitations": "The authors are encouraged to answer the questions and address the weakness above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The paper writing is clear and easy to follow.\n\n2.The HiCo model achieves spatial disentanglement and could generate more coherent and accurate images in complex scenarios.\n\n3.The HiCo model demonstrate excellent compatibility with rapid generation plugins LORA.", "weaknesses": "1.The generation requires more inference time and computational resources.\n\n2.The bounding-box and per-object generation process could not handle more complex interactions between entities such as ‘A man on the left and his wife on the right is holding their dog in the middle.’\n\n3.In the context of bounding-box based layout-to-image generation, addressing the issue of overlaps between bounding boxes corresponding to different entities has been a focal point of academic discourse. However, the authors omit any discussion on how such overlaps are managed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your detailed review and valuable suggestions. We address each of your comments below, additional figures can be found in the Author Rebuttal material PDF.\n\n## Question 1 & Weakness 2&3.\n\nHiCo achieves hierarchical generation by decoupling each object’s position and appearance information into different branch, meanwhile controlling their overall interactions through a background branch with global prompt and the Fuse Net. The Fuse Net combines features from foreground and background regions, as well as intermediate features from side branches, then integrates them during the UNet upsampling stage. As illustrated in Fig.1-(a)、Fig.1-(b) of Rebuttal PDF, HiCo is capable of handling complex interactions in overlapping regions without any difficult.\n\nThe occlusion order of overlapping objects is also specified via the global prompt by text description, for example “bowl in front of vase”, as illustrated in Fig.1-(c)、Fig.1-(d) of Rebuttal PDF. But since there lacks corresponding occlusion order train data, the success rate is far from optimal. For current version of HiCo, there indeed lacks of more explicit mechanism for occlusion order controlling.\n\nWe recognize this problem as our future work. Actually we’re already working on the occlusion order data curation, which is a quite challenging task as it requires reliable depth estimation besides the object detection bounding boxes. The process deserves another dedicated technical report in the future.\n\n## Question 2 & Weakness 1.\n\nFor inference run time and memory usage, we conducted two additional comparisons. The first comparison is horizontal, among 6 different models including: GLIGEN, InstanceDiff, MIGC, CAG, MtDM as well as our HiCo. Specifically, we evaluated the inference time and GPU memory usage for directly generating 512*512 resolution images on the HiCo-7K test set using a 24GB VRAM 3090 GPU, results in Fig.2-(a) of Rebuttal PDF show that HiCo has the shortest inference time and the 2nd lowest GPU memory footprint.\n\nTo make the results more complete, the second comparison is vertical: we assessed the inference time and GPU memory usage for generating 512*512 resolution images on the HiCo-7K test set with different number of objects. Since each object is processed by a separate branch in HiCo, the inference can be accelerated by inferring all the branches in one batch, i.e., in “parallel mode”, which as shown in Fig.2-(b) of Rebuttal PDF, is much faster than the “serial mode”, i.e., inferring all the branches one by one in serial.\n\n## Question 3.\n\nThank you for the valuable references. LMD and SLD represent early works that integrate large language models (LLM) with diffusion models for instruction-following enhancement and controllable image generation and edition. \n\nUnlike HiCo which is dedicated for layout-controllable image generation and requires layout and image specification directly from user input, LMD and SLD resort to LLM for image scene description and layout arrangement via text automatically. \n\nFor layout control, LMD and SLD adopt train-free approach by manipulating the latent and cross-attention map of each object. The solution is quite economic but the controlling effect is less satisfactory. In contrast, HiCo incorporates a dedicated conditioning network and learns layout condition from millions of data, providing comparatively superior controlling capability.\n\nLMD and SLD, on the other hand, can perform complex instruction understanding where HiCo cannot be directly compared with. It’s worth pointing out that HiCo can be integrated with LMD and SLD, by treating HiCo as the replacement of their train-free layout controllable image generation module, we’ll add these discussion in the main manuscript. \n\n## Question 4.\n   \nAs mentioned in introduction, adapter models such as ControlNet and IP-Adapter are representative works that introduce additional conditions into the diffusion model by incorporating “adapter network” alongside the frozen pretrained stable diffusion model. HiCo is also one kind of adapter model in this respective, introducing layout condition by incorporating separate branch network for each object. But unlike ControlNet, the side branch input of HiCo during training or inference uses paired text descriptions and conditional images, and the fusion method of multiple side branches of HiCo plays an important role in good generation performance. We believe HiCo’s current design is definitely not the most optimal, we will conduct more exploration on model structure in its next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bjyyvCAclT", "reviewer": "Reviewer_iE3j", "summary": "This paper studies layout-to-image generation. It proposes HiCo, a diffusion model that supports a complicated, hierarchical set of bounding boxes as the layout condition. The authors also constructed the HiCo-7K benchmark to provide challenging tasks for evaluations. The experiment results show that the proposed method can effectively generate images matching the layouts in various complex scenarios.", "questions": "Please see \"Weaknesses\".", "limitations": "The authors have addressed the limitations and broader impacts.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The key insight of disentangling each object with one branch of HiCo-Net, a ControlNet-like conditioning branch, and fusing them into the same image, is novel, interesting, and inspiring.\n- The authors provided various ways to augment the HiCo-Net branches with LoRA, which makes the model more powerful and extensible.\n- The authors constructed their dataset HiCo-7K for evaluation.\n- Abundant ablation study shows the reasonability of each design choice.", "weaknesses": "- The method only supports axis-aligned bounding boxes. It might be more powerful if it also supports more free-formed or precise bounding boxes like rotated squares or even polygon bounding boxes, which will make it desirable in some 3D generative tasks like layout-guided room generation.\n- Given that each bounding box requires an individual branch to compute intermediate features, the time complexity will be linearly growing with the number of bounding boxes. This makes the proposed method less efficient than previous method whose running time is constant.\n- Following the previous point, I wonder if there is a more efficient way to utilize each bounding boxes, e.g., assign a small number of nearby bounding boxes to one branch, to trade-off between efficiency and per-branch task complexity.\n- (Minor) In Fig.3, the \"Encoder\" and \"Decoder\" are not accurate terminologies in UNet. They should be \"downsampling\" and \"upsampling\" as in Fig.4.\n- (Minor) The texts in math formulas should not be italicized, which actually means the multiplication of each letter. For example, $Instruction$ means $I\\times n\\times s\\times t\\times r\\times u\\times c\\times t\\times i\\times o\\times n$ instead of $\\mathrm{Instruction}$.\n- (Minor) $\\times$ should be used instead of \"*\" in L130 and L131.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your wonderful review and detailed comments. Here we address each point of your comments, additional figures can be found in the Author Rebuttal material PDF.\n\n## Weakness 1\nWe are very grateful to the reviewer for presenting constructive ideas. Currently, HiCo only supports axis-aligned bounding boxes, mainly based on the perspective of general scene applications, which can achieve simplicity, ease of use, and excellent performance.\n\nFor scenarios involving other controllable conditions, using HiCo framework and designing reasonable training data and fusion methods can greatly expand their application scenarios. For example, in our ongoing work, we changed the input layout image to the foreground image segmented by segmentation model, and achieved outstanding performance on inpainting and outpaintint. We will also explore the scalability of conditional generation in future work.\n\n## Weakness 2\nFor inference run time and GPU usage, we conducted two additional comparisons.\nThe evaluation environment is a 24GB VRAM 3090 GPU. Generate images with a resolution of 512 * 512 using 50 step inference on the HiCO-7K.\n\nVertical dimension comparison, we assessed the inference time and GPU memory usage with different number of objects. Since each object is processed by a separate branch in HiCo, the inference can be accelerated by inferring all the branches in one batch, i.e., in “parallel mode”, which as shown in Fig.2-(b) of Rebuttal PDF, is much faster than the “serial mode”, i.e., inferring all the branches one by one in serial. \n\nHorizontal dimension comparison, among 6 different models including: GLIGEN, InstanceDiff, MIGC, CAG, MtDM as well as our HiCo. Results in Fig.2-(a) of Rebuttal PDF show that HiCo has the shortest inference time and the 2nd lowest GPU memory footprint.\n\n## Weakness 3\nThe HiCo model currently does not support merging multiple small and nearby layout boxes for generation. On one hand, the descriptions and bounding boxes of different targets within one branch cannot be distinguished. On the other hand, branches of HiCo can be inferred in parallel, which alleviates the impact of this issue to some extent.\n\nOf course, we are also actively exploring more concise and efficient structures to further balance inference efficiency and task complexity.\n\n## Weakness 4 ~ Weakness 6\nWe will thoroughly review the writing and formatting issues in the paper, and update it with the new experimental results and conclusions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U97WEOoynN", "reviewer": "Reviewer_6Tmf", "summary": "This work introduces a ControlNet-based conditioning method to enable pre-trained diffusion models to be layout-conditioned. The conditioning model processes each object in parallel and fuses them as residual features for the pre-trained diffusion model. In addition to the commonly used COCO dataset for evaluation, the authors also introduce a new dataset, HiCo-7K, which is fine-grained from the GRIT-20M dataset. The generation results look good even when there are many objects, and empirical results suggest that the model is backbone-agnostic.\n\nJustification: Although the method itself does not exhibit a high degree of novelty, there are notable contributions in dataset curation. However, the dataset part is not well elaborated, and the experiments lack comparison with cutting-edge methods.", "questions": "My major concerns are the weakness 1, 2, 4. I would like the authors to address these concerns. My final rating is subject to change based on the authors' feedback.", "limitations": "The author has discussed limitations and social impact.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The results validate that the proposed method can be applied to multiple pre-trained diffusion models, demonstrating its backbone-agnostic nature.\n* Qualitative results are good, especially for layouts with many objects.\n* The authors propose a new fine-grained dataset named HiCo-7K.", "weaknesses": "* The custom dataset, HiCo-7K, is an important contribution but has not been elaborated on sufficiently. The paper only provides information about the total number of images and the average number of objects. Details such as how the filtering was conducted and the criteria for manual cleaning should be included.\n* Table 1 does not include comparison with GLIGEN, and the SoTA method InstDiff[A], whose code was released several months before the NeurIPS submission deadline, is missing. Both of these methods can be applied on COCO and thus should be compared.\n* The local CLIP score in Tab 3 being higher than the Ground Truth potentially indicates that this metric may be unreliable. (This is a minor issue and my rating is not affected by this point, the author can skip this point during rebuttal if out of space)\n* The fuse layer is not explained in enough detail. While the sum and average cases are straightforward, the mask case requires more explanation, particularly regarding how features of overlapping objects are merged.\n\nA. Wang, Xudong, et al. \"Instancediffusion: Instance-level control for image generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your affirmation and constructive comments.We address each of your comments below, additional figures and experimental results can be found in the Author Rebuttal material PDF.\n\n## Weakness 1.\n\nWe have detailed the construction pipeline of the custom dataset HiCo-7K in Fig.3 of Rebuttal PDF. We found that GRIT-20M has some issues, such as a low labeling rate for targets with the same description and target descriptions being derived solely from the original captions. Compared to GRIT-20M, the pipeline of the HiCo-7K is as follows.\n\n1、Extracting noun phrase. We use spaCy to extract nouns from captions and the LLM VQA model to remove abstract noun phrases. Meanwhile, we use the GroundingDINO model to extract richer phrase expressions.\n\n2、Grounding noun phrase. We use the GroundingDINO model to obtain the bboxes. After that, we use NMS and CLIP algorithms to clean the bboxes.\n\n3、Artificial correction. To address the issue of algorithmic missed detections for multiple targets with the same description in an image, artificial correction is employed to further enhance the labeling rate of similar targets.\n\n4、Multi-captions with bounding box. We expand the basic text from the original captions and use GPT-4 to re-caption the target regions. The dataset of HiCo-7K contains 7000 expression-bounding-box pairs with referring-expressions and GPT-generated-expressions.\n\n## Weakness 2.\nWe conducted comparative evaluations of the mentioned methods on coco-3K and HiCo-7K, with detailed results shown in Table 1~3 of Rebuttal PDF. We also tested the inference time and GPU resource utilization of different models under the same conditions, as shown in Fig.2 of Rebuttal PDF. Due to distinct data quality and distribution between COCO and DM-generated images, GroundingModel and CLIP are better suited for COCO-3K evaluation.\n\nTable 3 shows that HiCo outperforms methods like InstanceDiffusion in terms of image quality and layout controllability on the HiCo-7K. However, on the COCO-3K dataset, our controllability is somewhat reduced.\n\nThe reason for this problem is that our model was trained on a 1.2M fine-grained long caption, which belongs to out of distribution data for coco data. HiCo has significant advantages over other methods in terms of inference time and GPU usage, with detailed results shown in Fig.2 of Rebuttal PDF.\n\t\n## Weakness 3.\nIn the evaluation based on the LocalCLIP Score dimension, the HiCo model exhibits slightly better performance compared to GroundTruth, primarily due to the following reasons:\n\n1.HiCo generates images that are clearer and more visually appealing.\n\n2.HiCo produces clearer boundaries in the generation of overlapping target regions.\n\n3.In the GroundTruth dataset, there are instances where adjacent identical targets are present, which both algorithms and human annotators have not fully addressed, leading to inaccuracies in the detection results from the Grounding-DINO detection algorithm.\n\nIn summary, both the quality of generated images and the accuracy of detection models will influence the LocalCLIP Score.\n\n## Weakness 4.\nWe designed a mask fusion method, which not only decouples different targets and backgrounds, but also provides a prerequisite for further image editing.\n\nThe main operation of the mask fusion method is as shown in the equation 1 of Rebuttal PDF, which is to multiply the features of different branches with the masks of the corresponding regions and then use the summation fusion.\n\nHowever, regarding overlapping and occlusion among different targets, the current mask fusion method handles fusion by directly adding features together. The occlusion order of overlapping targets can only be specified via the global prompt by text description, for example “bowl in front of vase”, as illustrated in Fig.1-(c)、Fig.1-(d) of Rebuttal PDF. For current version of HiCo, there indeed lacks of more explicit mechanism for occlusion order controlling. We have recognized this problem as our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their thoughtful, overall positive feedback and encouraging feedback. We are particularly pleased that the reviewers believe that our method achieves spatial disentanglement by separating each object, making it seamlessly compatible with SD community plugins (Reviewer 72RR). Our method is novel, simple, and effective, with strong model scalability (Reviewer iE3j). Our work on custom dataset HiCo-7K has made significant contributions (Reviewer 6Tmf). 

We would also like to thank the reviewers for all their insightful suggestions. We carefully read and analyzed all the weaknesses and questions, which can be summarized into the following aspects.

**Inference performance**. We tested the inference time and GPU resource utilization of different models under the same conditions, as shown in Fig.2.

**Target interaction**. Our method utilizes background branches to implicitly understand and generate complex interactions between targets, as shown in Fig.1-(a)、Fig.1-(b).

**Overlapping area**. Our method cannot display the specified layer order for generating overlapping areas, and can only use the prior knowledge of the model to autonomously and reasonably generate multi-target overlapping areas, as shown in Fig.1-(c)、Fig.1-(d).

**Fusion Network**. Our fusion network decouples different targets and backgrounds, improving the quality of complex layout image generation, and also provides feasibility for subsequent image editing.

**Dataset**. We have provided a detailed description of the processing procedure for the evaluation dataset HiCo, as shown in Fig.3.

**Comparison with other SOTA methods**. We conducted comparative evaluations of different methods on coco-3K and HiCo-7K. Refer to Table 1~3.

**Writing format**. We will conduct a detailed review and update of equations, symbols, terminology definitions, and method explanations.

Finally, we remind reviewers to refer to our Rebuttal PDF. At the same time, we will respond to each question and comment one by one. We are more than happy to discuss with the reviewers to address any other issues.

Meta-review: The paper received Accept, Weak Accept, and Borderline Accept scores. Overall, the reviewers appreciate the contributions of the technical approach and the new dataset for evaluating the problem of layout-to-image. The experimental validation is thorough and convincing. The authors' rebuttal clarified several details, including the inference performance, dataset description, and comparison with the state-of-the-art on coco-3K and HiCo-7K datasets. The AC agrees with the reviewers and recommends to accept.
Decision: Accept (poster)