R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation
Authors: Jiayu Xiao
Henglei Lv
Liang Li
Shuhui Wang
Qingming Huang
Venue: ICLR 2024 poster (2024)
Keywords: Stable Diffusion
training-free
grounded text-to-image generation
controllable generation
Primary area: generative models

Abstract: Recent text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images given text-prompts as input. However, these models fail to convey appropriate spatial composition specified by a layout instruction. In this work, we probe into zero-shot grounded T2I generation with diffusion models, that is, generating images corresponding to the input layout information without training auxiliary modules or finetuning diffusion models. We propose a **R**egion and **B**oundary (R&B) aware cross-attention guidance approach that gradually modulates the attention maps of diffusion model during generative process, and assists the model to synthesize images (1) with high fidelity, (2) highly compatible with textual input, and (3) interpreting layout instructions accurately. Specifically, we leverage the discrete sampling to bridge the gap between consecutive attention maps and discrete layout constraints, and design a region-aware loss to refine the generative layout during diffusion process. We further propose a boundary-aware loss to strengthen object discriminability within the corresponding regions. Experimental results show that our method outperforms existing state-of-the-art zero-shot grounded T2I generation methods by a large margin both qualitatively and quantitatively on several benchmarks. 
Project page: https://sagileo.github.io/Region-and-Boundary.

Published as a conference paper at ICLR 2024
R&B: REGION AND BOUNDARY AWARE ZERO-SHOT GROUNDED TEXT- TEXI-TO-IMAGE GENERATION
Jiayu Xiao 1.2- Henglei Ly Liang Li Shuhui Wang 1.3 Qingming Huang
1 Key Lab of Intelligent Information Processing, ICT, CAS, Beijing, China
2 University of Chinese Academy of Sciences, Beijing, China
3 Peng Cheng Laboratory, Shenzhen, China
j iiiaa..iii@va. xiao0vipl........ ict.ac.cc......c..
ac.cn liang. 11w.1ii1i001 Li@ict.... acc wwt.ttia.c........mm
umhuang@ucas.acco
ABSTRACT
Recent text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images given text-prompts as input. However, these models fail to convey appropriate spatial composition specified by layout in- struction. In this work, we probe into zero-shot grounded T2I generation with diffusion models, that is, generating images corresponding the input layout information without training auxiliary modules or finetuning diffusion models. We propose Region and Boundary (R&B) aware cross-attention guidance approach that gradually modulates the attention maps of diffusion model during genera- tive process, and assists the model to synthesize images (I) with high fidelity, (2) highly compatible with textual input, and (3) interpreting layout instructions accurately. Specifically, we leverage the discrete sampling to bridge the gap be- tween consecutive attention maps and discrete layout constraints, and design a region-aware loss to refine the generative layout during diffusion process. We further propose a boundary-aware loss to strengthen object discriminability within the corresponding regions. Experimental results show that our method outperforms existing state-of-ther zero-shot grounded T2I generation methods by large margin both qualitatively and quantitatively on several benchmarks. Project page: https://sagillo. www.ooiggggggguuteeeeeee iow.iioooon/nnnddeeeeeoe anduuiddaand.u..dr........
INTRODUCTION
Recently, text-to-image (T2I) models such as DALL-E (Ramesh al., 2022), Imagen (Saharia et al., 2022), and Stable Diffusion (Rombach et al., 2022) have shown remarkable capacity of synthesizing high-quality images conditioned on textual input. Trained on large-scale datasets (Schuhmann et al., 2022; Ramesh al., 2021; Lin et al., 2014) composed of image-text pairs collected from the Internet, these models learn a coherent generative space where vision and language modalities are highly unified. Many works (Ruiz al., 2023; Zhang & Agrawala, 2023; Hertz et al., 2022; Wu et al., 2022) leverage strong generative prior and zero-shot capacity of these T2I models and extend them to various downstream tasks, presenting diverse personalized generative results based user instructions.
Nevertheless, there remains challenges for grounded T2I generation. As shown in Figure 1, in some only highly relevant the
situations, users may expect outcome that the generated images given prompt but also adhere to a specified layout. Existing T21 models struggle (Gokhale et al., 2022) to correctly interpret spatial layout instructions specified by textual input, and lack the ability to localize concepts according to other spatial conditional inputs (e.g., bounding boxes). To address this, some works (Zheng et al., 2023; Li et al., 2023a; Avrahami al., 2023) propose train plug-in modules to incorporate layout conditions into the generative process. Yet this requires auxiliary training process on labelled data, and does not guarantee generalization on new model architectures.
Most recent works (Chen et al., 2023; Phung et al., 2023; Epstein et al., 2023; Couairon et al., 2023; Mou et al., 2023; Xie et al., 2023) show that self-attention and cross cross-attention maps of diffusion
1 CCoresponding Author. "qqual contribution.

---

Published as a conference paper at ICLR 2024
"A sketch of sandwich plate"
"a close up of banana and doughnut"
"A chicken stand- ing huge eg8, floating on river" bundle of balloon"
"A rabbit wearing
magician hat,
Text Prompt
sunglasses, and
Layout Instruction
Stable Diffusion
Ours
Figure 1: Illustration of zero-shot grounded text-to-image generation with Stable Diffusion. Given text input and bounding box information corresponding to the specific concepts, our method can generate images that conform the spatial instructions.
models encode rich structural information (Hertz al., 2022), and manipulation on the attention maps leads to edited results on the corresponding images. Inspired by the classifier guidance (Dhariwal & Nichol, 2021), they design energy functions on the attention maps for zero-shot layout control by transforming spatial guidance signals into gradients. While showing competitive results that interpret the layout instructions, they still suffer from two critical issues. First, these methods fail provide accurate spatial guidance, reflected by the misalignment between the synthetic images and the layout information. Second, they inherit the semantic inconsistency issues (e.g., missing objects, conceptual binding errors) from the original T2I model.
To mitigate the above issues, we propose Region and Boundary (R&B) aware cross-attention guidance for grounded T2I generation. As for region-aware guidance, find that the inaccurate outputs of previous methods owes to the neglect of the gap between cross- tross-attention maps and ground- truth bounding boxes. The former depicts shape and localization of objects fine-grained level, while the latter only provides coarse-grained spatial clues. To address this, we first leverage dynamic threshold to highlight object-related regions of cross-attention maps, and extend them to their minimum bounding rectangles (MBR). Then, inspired by theories on discrete sampling (Jang al., 2016; Sohn et al., 2015) for deep networks, we construct differentiable path from the binary MBRs to consecutive attention maps through region-aware loss that directly measures divergence between MBRs and ground-truth boxes, enforcing alignment between the distribution of attention maps and layout conditions. As for boundary- guidance, we discover that refining boundaries of attention maps in object regions enhances the semantic consistency the textual prompts, and further ensures adherence to layout constraints. We then propose a boundary- awwad loss to sharpen boundaries attention maps to encourage expression of different concepts in correct regions and enhance faithfulness of the synthetic images.
Throggh Estimator. Further, we design
Our contribution can be concluded as below:
1. To achieve zero-shot grounded T21 generation for diffusion models, introduce R&B, a novel attention guidance approach for layout generation, which requires no auxiliary modules or extra training.
2. We perform attention control of different objects from the perspectives of region and boundary respectively. Accordingly, we design region-aware loss and a boundary- aware loss. The former encourages accurate alignment between activation regions of cross-attention maps and layout instructions, while the latter enhances expressiveness different concepts in the refined localization, and better cooperates with text semantics.
2

---

Published as a conference paper at ICLR 2024
3. We conduct experiments and ablation study for better comprehension of the proposed method. Experimental results show that our proposed R&B performs well on zero-shot grounded T2I generation with high spatial accuracy and generative fidelity, and surpasses the existing state-of-the-art methods by large margin both qualitatively and quantitatively.
2 BACKGROUND
Diffusion models. A diffusion model is trained to approximate a data distribution via gradually denoising random variable drawn from unit Gaussian prior. Essentially, it learns to reverse time- dependent data perturbation process that diffuses the data distribution into a fixed prior distribution, e.g., Gaussian distribution. Basically, a denoiser eo is trained to estimate the noise ea from the corrupted image = ai + d,e, with the diffusion loss:
I((0) = Bzg Beg.teerr- c0(=,, ()1
(1)
Recently, a powerful Stable Diffusion (SD) (Rombach et al., 2022) is widely used for synthesizing images with high resolution and quality. It transforms real images into latents and performs denoising process in the latent space. It further introduces a fixed text encoder (Radford et al., 2021) for text-image interactions with cross- attsstattnntoon modules to achieve text conditional image generation.
Controllable diffusion. Previous work (Song et al., 2020b) describes diffusion models as score-based models from the perspective of stochastic differential equation (SDE), and transforms the derivation of the reversed diffusion process into estimation of the score function: V log Pe (zt). Further, the controllable diffusion process can be viewed as a composed score function to sample from richer distribution, i.e., V., lo, Pr (r y), where y is external condition. This joint score function can be decomposed as below:
V de Pe y) 22, (o) + Vo,logp(()lz))
(2)
where the first term the original unconditional model, and the second term corresponds to the classifier guidance that steers the sampling process towards the generation target (e.g., categories). Dozens of methods (Voynov et al., 2023; Bansal et al., 2023; Chen al., 2023; Mou et al., 2023; Lv et al., 2024) design different energy functions to estimate V log (y(()ze)) for controllable generation. Inspired by these works, we incorporate layout conditions into the denoising process through an energy function g(2;; t, y). where lower value of g indicates better alignment with spatial constraints.
In this way, the guidance Vz, log Pt (r(y) 2x) can be computed as 77 V (21 t. y), where ng is a scaling factor to control the guidance strength.
Text-to-image generation with layout guidance. Many recent works inject layout guidance into the powerful SD models and achieve interactive generation that satisfies users' preference. Some of them (Zhang & Agrawala, 2023; Li et al., 2023a) train auxiliary modules to embed the layout information into intermediate features of the SD UNet to bias model output. Other works provide a solution of cross-altention (Tu et al., 2023b;a; 2024) modulation to inject the layout guidance in either forward or backward manner. The forward methods (Liu al., 2023b; Kim et al., 2023; Balaji et al., 2022) directly manipulate the value of the SD cross-attention maps during the forward process, requiring more refined layout information (e.g., segmentation maps) or additional training. The backward methods (Chen et al., 2023; Xie et al., 2023; Phung et al., 2023) utilize an energy function to transform layout constraints into gradients and update the noisy latents during the sampling process. Our proposed R&B can be viewed as backward method.
METHOD
In this section, first provide detailed description of the problem definition of zero-shot grounded text-to-image generation. We then discuss the methodology our R&B for diffusion based zero-shot grounded T2I generation. Figure 2 illustrates an overview of training- generation framework.
Zero-shot grounded T2I generation. In contrast to visual grounding (Liu et al., 2022), zero-shot grounded text-to-image generation (Chen et al., 2023; Phung al., 2023; Xie al., 2023; Lian et al.,
3

---

Published as a conference paper at ICLR 2024
Leyout instruction
"handbag" (i 1)
Reglon-aware gaidanne
"bench" (i 2)
Bondary-aware guidanc
Differenial peth
Edge
Box selection
A ggregation
Zt
2t
Crsss- attention
KV
KV
KV
4. 4- 21- V. Update latent
hhandbag placed the wooden
feench
Ri-1
2 - V. V. Update labent
Generated Image
Figure 2: Overall framework of our region-aware cross-attention guidance for diffusion- zero- shot grounded T2I generation. At each time step, we update the noisy latent via optimizing the energy function g( 24; t, B), which is summation of region-aware loss L, and boundary aware loss Ls.
2023) aims at generating images conditioned text prompt and grounding information, without additional training procedure. Considering a text prompt T {TI.... Tx ). and a set of bounding boxes B (B,,..., Bx} Each bounding box B, corresponds to subset T; C T. The generated images are supposed to be highly consistent with the text prompt and satisfy the layout constraint imposed by bounding boxes. An intuitive visual example is shown in Figure 2, given the textual input "A handbag placed on the wooden bench" and layout instruction with two bounding boxes associated with "handbag" and "bench" respectively, our approach synthesizes image in which a handbag is positioned in the middle of a wooden bench, and the object sizes of the handbag and bench match the two corresponding bounding boxes.
Attention map extraction and aggregation. As shown in Figure 2, at each denoising step t, the noisy latent 2; is fed the diffusion U-Net. each cross-attention layer I, the image and features are fused for cross-modal interaction, where deep image features (0x))) and textual embeddings are projected to query matrix Q and key matrix Ki respectively (here we disregard the case of multi-heat attention for simplicity). The cross-attention Ar E IR HixXWxX with respect to N textual tokens at layer computed as below:
A Softmax QIKT )
where the dimension d is utilized to normalize the softmax values. H₁ and W/ represent the resolution of the cross-attention maps. Denoting { E R [ E T,} as the cross-attention maps with respect to the 4t concept T,, we compute an aggregated attention map (Couairon et al., 2023) M, by averaging the cross- aoss-atttntionn maps across different layers:
N LT. (T)) A/, E
M = 1
(4)
where (.) is an indicator function. Since the cross-atee coss-attention maps Ar at each layers have different resolutions, we first upsample them to unified resolution 64 x 64, then average them to get M;, Previous works (Hertz al., 2022; Epstein al., 2023; Tang et al., 2023) indicate that cross-attention maps of the diffuuson U-Net contain rich semantic and geometric information for generated images. Based on the above properties, we perform cross-attention guidance for grounded generation.
Dynamic thresholding. After adopting the aggregated cross-altention maps, goal is accurately describe the shape and localization of specific concept in zero-shot manner. Previous work (Epstein et al., 2023) applies a fixed threshold to the normalized attention map to highlight the foreground objects, and performs self-guided image editing. However, this approach is ineffective in the case of grounded generation, because the distribution of the attention map changes during the layout
4

---

Published as a conference paper at ICLR 2024
generation process. To address this, we propose dynamic thresholding approach highlight the foreground region M, of the 2th object:
M Mi ming, (M,)
(5)
maxA, (Mi) - mimn, (Mi)'
E h,w M. o B₁
(1 2) En, Mamo
B))
(6)
En Bi
En B.
(1
if (Mogom Nnw 2
(Mi)k.
0
if (M.omm )no
where T; is calculated by weighted averaging the activation values inside and outside the corresponding bounding box B, in Eq. (6). Tuple (h, w) represents a spatial entry of the attention map.
Box selection. The binary mask M, Eq. (7) highlights the foreground region associated with the 7th concept. Nonetheless, there remains a domain gap (Cai et al., 2023; Xiao et al., 2022; Liu et al., 2023a) between M, and the ground-truth bounding box B,, since the latter only describes the shape, size, and position of the object through a rectangular binary mask. To bridge the gap, we perform box selection by extending M, to its minimum bounding rectangle (MBR) B; to match B;, as illustrated in Figure 2. Even though, directly optimizing B; towards its ground-truth B; is not feasible because the MBRs hard and non-differentiable masks. Towards practical implementation, we first adopt two variants of M,; shape-aware attention map MN (Epstein et al., 2023) and an appearance-awand attention map MM (see Appendix for details). Then we construct differentiable path from the hard mask B, to the two consecutive attention maps via a region-aware cross- aessntattntionn guidance:
Estimator (STE) for the
B stopgrad - M2² + MM, 3 stopgrad B, - M,) +
(8)
(9)
Numerically, the two binary masks B3 and B in Eqs. (8) and (9) equal to B, However, since B and B3 are differentiable, can directly design loss functions on the above two binary masks to transform the layout condition into gradients, and optimize the input noisy latent through error backpropagation for layout control.
Region-aware loss for attention guidance. After obtaining the aforementioned differentiable boxes, align them with the ground-truth bounding box B, through region-aware loss:
Σn.w B 3 B, IoU; = En.w B, (1 (B)
(10)
B.)
En.w B B
C, Bi) (1 -lou)) (As(1
Σn, 13
(1I)
En By 2 Bi
+1a
0n B
The fraction in C encourages the attention map to transport top values from current high-activation regions into the target regions. loU; in Eq. (10) measures the spatial accuracy of generative layout, and controls the scale of CJ. When B; completely agrees with B, L. falls to 0. Different from energy functions proposed in previous methods (Phung et al., 2023; Chen et al., 2023; Xie et al., 2023), Ct in Eq. (11) directly measures the divergence between the predicted MBR and ground-truth B,, which provides a more accurate guidance path for the diffusion model, leading to better generation accuracy in accordance with layout constraints.
5

---

Published as a conference paper at ICLR 2024
Boundary-aware loss for attention guidance. Recent work (Li et al., 2023b) proposes enhance
the faithfulness of symthetic images by incentivizing the presence objects via maximizing the total variation of tross-attention maps, which promotes local changes in attention maps and facilitates the appearance of discriminative object-relative regions within the image. Inspired by this, we propose a boundary-awan loss to enlarge the variation of tross-attention maps within the regions corresponded to different objects:
& = Sobel(M;),
(12)
(= 13) = (1 IoU,) (1 Σn, E, B
(13)
Enyo E,
where Sobel(-) represents the Sobel Operator and & is the extracted edge map from the aggregated attention map. Intuitively, Lj enlarges the sum of changes in B; and suppresses the outer activation, thus constrains the objects to express within bounding boxes and better adheres to the semantics. To this way, the overall energy function for classifier guidance in Section 2 can be written as:
g(ect,y)) = g(z;t,
Σ Cz B,) (z:: B,EB
(14)
At each denoising step, we calculate the gradient of g B) and update the latent 2n!
2t 24 no V #. (2; B).
(15)
visual example for the controlling process of cross-attention maps is shown in Appendix D (Figure 7). The proposed R&B iteratively refines the cross-altention maps during the sampling process, and effectively enforces the high responses to concentrate within the corresponding boxes.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset. We utilize different datasets to validate the effectiveness of our approach quantitatively. First, we compare our model performance against different state-of-the-ard methods from the perspective of generation accuracy. We use two benchmarks: HRS (Bakr al., 2023) and Drawbench (Saharia et al., 2022). The HRS dataset is composed of various prompts tagged with name of objects and corresponding labels. We select three tracks validate the effectiveness of our proposed method: spatial/size/color, the number of prompts for each category are 1002/501/501 respectively. The Drawbench dataset comprises 39 prompts, with manually annotated labels assigned to each prompt. To evaluate the spatial accuracy, we select subset of 20 samples from this benchmark. The bounding box annotations for the above two benchmarks is generated according to the textual prompts with the aid of GPT-4 (Phung et al.. 2023). Second, assess all the methods in terms of their fidelity to the textual input. We select 100 samples from the MS-COCO (Lin et al., 2014) dataset and create triplets consisting of image caption, object phrases and bounding boxes. Five images are randomly generated for each samples to measure the consistency with image caption.
Metrics. In order to evaluate the generative accuracy, adopt the method used in HRS (Bakr et al., 2023). This method leverages an object-detection model to detect objects within the synthetic images. Accordingly, one image is considered as correct prediction when all detected objects are correct and their spatial relationships, sizes or color align with the corresponding phrases in the prompt. For the alignment with layout conditions, we report mean IoU score between the bounding boxes predicted by detection model and the ground-truth. For the fidelity to the textual input, we utilize CLIP-Score measure the distance between input textual features and generated image features.

---

Published as a conference paper at ICLR 2024
"A birthday cake with fruit toopiings"
"A birthday cake with fruit toppings"
"A birthday cake with fruit toppings"
candles
rose toppings
Figure 3: Illustration of visual variations. We vary the textual prompt and layout constraints to generate different images. The size and localization of the bounding boxes are changed for spatial control. The instances of the text prompt alternated for semantic editing. More visual variations for different text prompts can be found in Appendix F (Figure 14).
4.2 EXPERIMENTAL RESULTS
Visual variations. To validate the robustness of the proposed method, we generate different images by varying the textual input and bounding box constraints. Visual results are shown in Figure choose "A birthday cake with fruit toppings" as the basic text prompt. In the top row, the localization of the birthday cake changes from the bottom the top of the image according the box instructions. In the middle row, the cake grows taller from left to right, and the localization of the fruit toppings changes accordingly. These show that our synthetic images highly align with the box information across range of size and localization variations. In the bottom row, we adopt multiple changes with prompt alternation and box adjustment, while R&B presents delicate results respectively. This demonstrates that our method simulates the layout control from different aspects robustly, maintaining high generation fidelity and accuracy.
Quantitative comparison. We validate the generative accuracy of our method and existing state-of- the-art zero-shot grounded T2I approaches (details can be found in Appendix B). Comparison results shown in Table 1. For the spatial category, on HRS and 11.50% on DrawBench. For the size and color categories, our proposed method also beats the baselines by large margin of 9.77% and 8.5%. The proposed region- loss guides the generative process to follow the layout more faithfully, which is the key to improve performance on spatial and size category. Our proposed boundary-aware loss enables the generation of more objects according to layouts, which boosts performance on hard cases with 3 or 4 objects. Quantitative comparisons the alignment with layout constraints and textual input are in the Appendix (Table 5).
method surpasses the best methods by 5.69%
Methods
Stable Diffusion
BoxDiff
Layout-guidance
Attention-refocusing
R&B (Ours)
Spatial
8.48
16.31
16.47
24.45
30.14
HRS
Size
9.18
11.02
12.38
16.97
26.74
Color
12.61
13.23
14.39
23.54
32.04
DrawBench
Spatial
12.50
30.00
36.50
43.50
55.00
Table I: Quantitative comparisons with competing methods. The evaluation accuracy (%) reported. Best results are bold.
7

---

Published as a conference paper at ICLR 2024
pit on messy handhag"
"A living roos with couch chaie"
large knife on counter top"
"a wsaan with leading "Three horses on the runway"
Stable
Atlention-
Layoul-
Layout
Diffusion
BoxDiff
refocusing
guidance
R&BB
Text prompt
Figure Visual comparisons with different baselines. We compare our R&B with Stable Diffusion and several zero-shot grounded T2I baselines. Bounding boxes related to different objects are annotated on the images with dashed boxes. Zoom for better visualization.
Visual comparisons. In Figure 4 we illustrate visual comparison between our method and several competing baselines. We observe that without additional layout constraints, the vanilla Stable Diffusion suffers from missing objects and object counting error, therefore the generated images fail to convey the given layout information. Existing methods achieve zero-shot grounded T2I generation diffusion guidance, and present competitive results that better reflects the object layouts. Yet the critical problems of T21 methods mentioned above still exist. For example, BoxDiff fails to generate the cup (3 row), Attention-refocusiii fails generate the chair (2 row), and both BoxDiff and Layout- -ugidanee
fails to generate the handbag (1* row) and chair (2nd
row), Layout-guidane generate horses with incorrect number, see the last row. Furthermore, many synthetic images do not comply well with the layout constraints. In comparison, proposed R&B (1) handles the semantic
"Three cars on the street"
"A dog running with nan on horse"
Layout
C,
L
C, Lb
Text prompt
Figure 5: Ablation study for the effect of different losses. We illustrate the visual impact of C, and C, respectively. and also show the combined results.
8

---

Published as a conference paper at ICLR 2024
deficiency problems well, (2) produces images that better align with the bounding boxes, (3) better reflects the semantics of the given text prompts (e.g., rope in the woman's hands for leading a horse). The effectiivnness of our R&B method can be attributed to the reliable guidance provided by the region- aware loss and the boundary-aware loss. The region-aware loss accurately captures the discrepancy between the generated layout and the ground-truth boxes, while the boundary-aware loss calibrates the generative process by encouraging the expression of different objects in their corresponding regions. We notice that although Attention-refocusinng yields high quantitative performance on HRS and Drawbench benchmarks, the generated images do not align well with the box constraints. The reason is that the above two benchmarks only evaluate the model's generation accuracy terms of coarse-grained aspects. Images that do not align with the bounding box can still be considered as correct the evaluation. Additional generative results under different random seeds for competing methods can be found in Appendix F (Figure 9).
4.3 ABLATION STUDY
Impact of different loss components. In Figure 5, we present visual results for comprehension of our proposed region-aware loss L, and boundary-aware loss La- We observe that L, (the second column) contributes to good spatial accuracy, yet does not handle some difficult semantics well. For example, the wrong number of cars, and the missing man on the horse. On the other hand, with Co alone, the generated images (the third column) better adhere to the text semantics, but do not align well with the ground- truth bounding boxes. The fourth column shows visual result of the combination of the two losses. We witness that they complement each other well, and present boosted generative results with higher spatial and semantic accuracy. As for the quantitative analysis of the two losses, please refer to Table 4 in Appendix E.
Impact of the guidance ratio. We analyze the impact of the guidance ratio 7g in Eq. (15) for the energy function y from the quantitative aspect. Numerical results on MS-COCO are shown in Table 2. We scale V/ from 20 to 300, and report the mean IoU and T2I similarity to measure the alignment with layout and textual input respectively. We utilize UniDet (Zhou et al., 2022) to detect objects in the synthesized images, and calculate the mean IoU score between the predicted object bounding boxes and their corresponding ground-truth boxes. The upper bound of the mean IoU is 0.875, for some phrases in the textual prompts do not belong to the test classes of UniDet. As for the text-to-image similarity, we calculate the CLIP-Score of the image features and corresponding text prompt features. We observe the fact that as 7i grows, the two scores are improved at the beginning, and then experience rapid decline. The reason is that excessively strong constraints greatly impair the generative fidelity, leading to deterioration in the evaluation results. Practically, we choose 77 as 70 for all the experiments, in order to balance the generative accuracy and fidelity. Due to space constraints, we discuss the additional ablation results detailly in Appendix E.
20
50
60
70
80
100
150
300
mloU (1)
0.5075
0.5303
0.5239
0.5533
0.5460
0.5658
0.5626
0.4818
T2I-Sim (1)
0.3146
0.3141
0.3186
0.3218
0.3138
0.3158
0.3136
0.3007
Table 2: Quantitative analysis of the guidance ratio. We utilize mloU to measure the alignment with box instructions, and CLIP-Score to measure the correspondence between images and texts.
5 CONCLUSION
In this paper, we propose R&B, novel attention guidance approach for diffusion based zero-shot text-to-image generation. We point out two prevalent issues associated with the current approaches: (I) the misalignment between generated images and layout information, and (2) the semantic issues inherited from the base T2I models. Accordingly, we propose region and boundary aware guidance approach to mitigate the above problems. Experimental results show that our proposed method outperforms the existing baselines by large margin both quantitatively and qualitatively, and verify its robustness to various layout and textual conditions. Future work will focus on extensions of zero-shot grounded generation in different scenarios (e.g., 3D, video).

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This work was supported in part by National Natural Science Foundation of China: 62322211, 6193 1008, 62236008, 62336008, U2IB2038, the Key R&D Plan Project of Zhejiang Province (No. 0024C010233 and Youth Innovation Promotion Association of Chinese Academy of Sciences under
Grant 2020108.
REFERENCES
Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18370-18380, 2023.
Eslam Mohamed Bakr, Pengzhan Sun. Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. Xrviv preprint arXiv:2504.05390, 2023.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint 1rXiv:2211.01322 2022.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Shaofei Cai, Liang Li, Xinzhe Han, Shan Huang, Qi Tian, and Qingming Huang. Semantic and
correlation disentangled graph convolutions for multilabel image recognition. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Trunsactions Gruphics (TOG), 4204):1-10, 2023.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03373. 2023.
Jooyoung Choi, Jungbeom Lee, Chachun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffuuson models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11472-11481, 2022.
Guillaume Couairon, Marlène Careil, Matthicu Cord, Stéphane Lathuilière, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. arXiv preprint arXiv:2306. 13754, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Dave Epstein, Allan Jabri, Ben Poole. Alexei A Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. arXiv preprint arXiv:2306.00988 2023.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vincet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint 10010:2212.10015. 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt- to- to-prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2022.
Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang. Weakly-supervin semantic segmentation network with deep seeded region growing. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7014-7023, 2018.
10

---

Published as a conference paper at ICLR 2024
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzttion with gumbel-softmax. In International Conference on Learning Representations, 2016.
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Wor Woo Ha, and Jun- Yan Zhu. Dense text-to-image generation with attention modulation. arXiv preprint arXiv:2308. 12964, 2023.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22521, 2023a.
Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing, arXiv preprint arXiv:2330.10864, 2023b.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Alm-grounee diffusion: Enhancing prompt understanding of -it-tt diffusion models with large language models. arXiv preprint arXiv:2305. 13655, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014.
Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Zechao Li, Qi Tian, and Qingming Huang. Entity-enhanced adaptive reconstruction network for weakly supervised referring expression grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4503):3003-3088, 2022.
Zhenhuan Liu, Liang Li, Jiayu Xiao, Zheng-Jun Zha, and Qingming Huang. Text-driven generative domain adaptation with spectral consistency regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7019-7029, 2023a.
Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng. Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones 2: Customizable image synthesis with multiple subjects arXiv preprint arXiv:2305. /9327, 2023b.
Henglei Lv, Jiayu Xiao, Liang Li, and Qingming Huang. Pick-and-draw: Training- semantic guidance for text-t--image personalization. arXiv preprint arXiv:2401.16762 2024.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.0222J,, 2023.
Quynh Phung, Songwei Ge, and Jia- Ba-B Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:2306.005: 054277770777..... 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to- text-to-image diffusion models for subject-driven generation. In Proceed- ings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-223 225100 2023.
11

---

Published as a conference paper at ICLR 2024
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 45:36479-36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-252999 2022.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bcore-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020b.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. arXiv preprint arXiv:2306.0388 03881, 2023.
Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Chenggang Yan, and Qingming Huang. Self-supervised cross-view representation reconstruction for change captioning. In Proceedings the IEFF/CVF International Conference on Computer Vision, pp. 2805-2815, 2023a.
Yunbin Tu, Chang Zhou, Junjun Guo, Huafeng Li. Shengxiang Gao, and Zhengtao Yu. Relation attention for video captioning via graph learning. Pattern Recognition, 136:109200 109204, 2023b.
Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, and Qingming Huang. Smart: Syntax-calibrated multi-aspect relation transformer for change captioning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Kira Vinogradova, Alexandr Dibrov, and Gene Myers. Towards interpretable semantic segmentation via gradient-weighted class activation mapping (student abstract). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.
2020.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch- guide text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:22 1565, 2022.
Jiyy Xiao, Liang Li, Chaofei Wang, Zheng-Jun Zha, and Qingming Huang. Few shot genera- tive model adaption via relaxed spatial structural aligmment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1204-11213, 2022.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and
Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training- box-constrained diffusion. arXiv preprint arXiv:2307.10816, 2023.
Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprini arXiv:2302.05554; 2023.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 22490-22499, 2023.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7571-757777 2022.
12

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
We utilize the Stable Diffusion V-1.5 (Rombach et al., 2022) as our base generative model. We adopt the DDIM scheduler (Song al., 2020a) with 50 denoising steps. The ratio of classifier-free guidance is set as 7.5. Previous works (Balaji et al., 2022; Hertz et al., 2022; Choi et al., 2022) show that T21 diffusion models tend to form the layout of each object in early stages and refine the generative details (color, textures) in later stages. Thus we only perform layout guidance at the first 10 steps. The À for dynamic threshold in Eq. (6) is set as 0.4, the ratios X₂ and Xa Eq. (11) are set as 1.5 and 1.0 respectively.
B DETAILS OF COMPETING METHODS AND DISCUSSIONS
Layout-guidance (Chen et al., 2023) is an attention control method for zero-shot grounded T21 generation. It designs energy function according to the box constraints:
A EVEB Aus )?,
B(A ,B,)
(16)
Σn. A
where ( determines the attention value location u corresponded the ith token for the layer y. This energy function enforces high activation within the masked regions. At each time step, the gradient of E is computed to update the noisy latent through backpropagation. Although adopting energy function in Eq. (16) for layout guidance demonstrates effectiveness in many situations, there still remains some critical issues. (1) Optimizing Eq. (16) aims at concentrating all attention scores within the bounding box, which may lead to some tricky solutions (Xie et al., 2023). For example, in Figure (3td row, 5th column), the generated cup is much smaller than the corresponding box. And the number of generated horses is incorrect (5th row, 5th column), because most of the class-related areas are within the boxes, the designed energy function is fooled and does not provide accurate latent optimization directions. (2) This energy function is not sensitive the scale of the activations and may not effectively address some semantic deficiencies issues (missing objects, attribute binding). Thus, even though Layout-guidance presents some delicate visual results in Figure 4, does not perform well on the HRS and Drawbench benchmarks.
BoxDiff (Xie et al., 2023) proposes three constraints for training-free grounded T2I image generation: Inner-Box, Outer-Box, and Corner constraints. The Inner-Box constraint selects top-k elements of cross-attention map within the bounding box and maximize their values. The Outer-Bo Box constraint selects top-k elements out of the box and minimize their values. The Corner constraint projects the attention values alongside the x-axis and y-axis of the bounding boxes, and maximizes the projected values in order to cope with some tricky solutions (e.g., the object becomes much more smaller than the ground-truth box). The key insight of Boxf is to impose constraints on fewer elements with higher responses of cross cros--attnntion maps, rather than all of the elements. When reproducing Boxfiff, we find that it heavily relies on choice of initial seeds. The main reason is that the top-k selection strategy introduces uncertainty during the denoising process. As result, the generated outcomes exhibit significant instability. This is shown in Figure and 16, the generative layouts varies greatly across different seeds, picking required to adopt delicate results. We also observe that only impose constraints on the peak values may results in outcomes that only discriminative regions of objects locate within the box, see Figure 9 (upper half, 1s row), Figure 16 (bottom half, 1 row). Further, visual results (Figure 4, Figure 9 and Figure 16) show that Boffi does not effectively cope with the semantic issues.
A Attention-refocusing (Phung et al., 2023) proposes self-attention refocusing and cross-attention refocusing (SAR&CAR) approach to guide the diffusion process according to the layout constraints by manipulating both cross-attention maps and self-attention maps. As for the cross- coss-attention guidance, maximizes the top values of cross-attention maps within bounding boxes in the same way as the previous method (Chefer et al., 2023), and penalizes the top values in other object regions and background respectively to suppress incorrect expression of different objects. As for the self-attention guidance. it minimizes the similarity between the elements of background regions and the inner- regions to enforce the expression of objects in the correct places. Attention-refou refocusing is essentially an enhancement of training-basee methods (e.g., GLIGEN (Li al., 2023a)), it borrows inspiration
13

---

Published as a conference paper at ICLR 2024
"A man a suit wearing a tie"
"A pit bull dog on messy bed beside handbag"
cat surfing on sea"
Layout
w/o (a)
w/o (b)
R&B
Text Prompt
Figure 6: Visual examples for the effect of (a) Mi and (b) Mr We illustrate the two variants of cross-attention maps respectively, and also show the combined results.
from previous work (Chefer et al., 2023) to optimize the max attention values within and out of the input boxes to cope with the semantic issues in grounded generation. Quantitative results in Table and show that its proposed SAR&CAR boosts the performance of existing T2I methods like Stable Diffusion and GLIGEN. However, when regarded as training-free grounded generation approach, the generative outcomes lack spatial accuracy. See Figure 4, 9 and 16 for visual examples. Similar to BoxDiff, optimizing the max values of attention maps incorporates randomness into the generative procedure and only ensures the most discriminative regions of objects are within the layout. Meanwhile, the self-attention refocusing may not be effective during the early layout refinement process, for it is not aware of the textual condition, when the generative layout differs significantly from the ground truth, it does not provide accurate guidance for corresponding ojbects.
C VARIANTS OF AGGREGATED CROSS-ATTENTION MAPS
We mentioned two variants of M2 in Section 3: MZ and Mi respectively. Here we formulate these two variants as below:
Mi = normalize(sigmoidd (Mrorr Tz))),
(17)
M, ming, (M,)
M MM
(18)
max,, (M,) mink, (M:)
The formulation of M; is inspired by previous work (Epstein et al., 2023), which adopts soft masks from the cross-attention maps. The sigmoid(-) operation (with sharpness controlled by s) helps to eliminate the noisy background that is thresholded by т. In this manner, Mi represents a soft mask that describes the shape and size of the 7th object. Practically, we find that although imposing constraints on M helps for fast convergence of layout control, grounded T21 generation. This is because the sigmoid(-) function introduces nonlinearity to the guidance process. When the responses of elements deviate from the threshold, the gradients of the elements decrease rapidly. This leads to the insensitivity to the layout guidance in some local regions (e.g., background regions), reflected by redundant textures or catastrophic neglect in the output images. To cope with the above issue, further utilize Mf, which simply rescales the cross- aots-ntttntion maps with linear transforms, as complement for M Mi preserves the quantitative relationships between different actívation regions of the original attention map, thereby better reflecting the appearance
somehow not sufficient for
14

---

Published as a conference paper at ICLR 2024
"A black cat is chasing after white dog"
Stable Diffusion
R&B
"A looking laptop computer"
Layout-guidance
R&B
Figure 7: Visualization of cross attention maps at earlier diffusion steps. We show visual comparison between our R&B and (1) Stable Diffusion (Rombach et al., 2022), (2) Layout-guidance (Chen et al., 2023). Both Layout-guidance and our method perform layout guidance at the first 10 steps.
information associated with the attention map. For better comprehension, we illustrate visual comparison in Figure 6, (a) and (b) represent using My and M for layout guidance respectively. We find that only optimizing Mj (2nd column) leads weird visual effects in the synthetic images annotated by dashed boxes, for example, the redundant tie (1xt row) and sea waves (last row), and absence of the handbag on the bed (2nd row). By comparison, optimizing M (3rd column) overcomes the above issues, yet does not guarantee alignment with the bounding box constraints (e.g., the wrong size of man, dog, and handbag in the first two rows). When optimizing the two variants jointly during guidance process, R&B presents images with high (1) consistency with the layout conditions, and (2) fidelity to the textual semantics.
D VISUALIZATION OF CROSS-ATTENTION MAPS
The first few denoising steps of diffusion models decide the approximate spatial layout of the generated image, while cross attention maps are an intuitive way to describe To illustrate how method controls the generative layout through attention maps, we provide visualization of cross
15

---

Published as a conference paper at ICLR 2024
attention maps within 10 denoising steps in Figure 7. The attention map is gradually refined from left to right as the number of denoising steps increases. The upper half compares our method to Stable method to Layout-guidie
Diffusion (Rombach et al., 2022) and the lower half compares et al., 2023). Stable diffusion lacks guidance of spatial information, therefore both objects are region, which leads missing objects and attribute possible compete and activate the guidance restricts high activation regions within the misbinding in the generated image. Layout- Layout-guidance corresponding bounding boxes, but fails align the shape and size objects with the box constraints. By comparison, under the guidance of our region and boundary aware loss, the generative model precisely localizes objects according to their layouts in the early sampling process. With the well initialized generative layout, the T2I diffusion model produces images that align with the layout conditions precisely, and possess high generative fidelity.
(Chen
E
MORE ABLATIONS AND ANALYSIS
Dynamic thresholding. Thresholding class-related activation maps (Vinogradova et al., 2020; Huang et al., 2018) for precise foreground region extraction is a challenging task, the reason is that only weakly supervised information is available. Previous work (Epstein al., 2023) adopts fixed threshold to obtain the size and shape objects from the diffusion cross-attention maps for image editing. Yet find this effective for layout generation, for the reason that the distribution of cross-attention maps changes during guidance process. To address this, we utilize a dynamic thresholding approach highlight the foreground regions according to the mean activation values within and outside the bounding box. formulated in Eq. (6). We show visual results in Figure 8. We find that utilizing fixed threshold faces several issues: (1) struggles to accurately generate objects, which the original Stable Diffusion fails to synthesize, in the corresponding regions (e.g., the branch the 1st row and the hello kitty in the 2nd row). (2) does not solve the improper attribute binding problem, like the confusion of horse and car in the bottom row.
Discrete sampling. Inspired by previous works on deep model quantization (Jang et al., 2016; Sohn et al., 2015), we differentiablly binarize the cross-attention maps via a Straight-Through Estimator (STE), the formulation is shown in Eq. (8) and Eq. (9). This operation transforms the consecutive
"A dog is standing on brown branch"
"A cell phone placed on the table with a picture of hello kitty"
"A yellow horse and ca
Stable Diffusion
w/o Dynamic thresholding
w/ Dynamic thresholding
Layout
Text Prompt
Figure 8: Visualization of the impact of dynamic thresholding. We show visual results of the original Stable Diffusion nd column), binarizing the attention maps with threshold 0.5 (3rd column), and binarizing the attention maps with dynamic threshold (4th column), respectively.
16

---

Published as a conference paper at ICLR 2024
w/o Discrete sampling
w/ Discrete sampling
mloU (f)
0.5106
0.5533
T2I-Sim (t)
0.3142
0.3218
Table 3: Quantitative ablation of discrete sampling. We conduct experiments on the subset of MS-COCO. All the syyer-parameters are fixed for fair comparison.
attention maps into differentiable hard masks, and we design a discrete energy function in Eq. (11) to directly measure the divergence between the predicted bounding boxes B; with the ground-truth B,
We show the impact of discrete sampling approach from the quantitative perspective, and illustrate the results in Table 3. We alternate the discrete maps in Eq. (11) with M and M. to ablate the effect of discrete sampling. We observe that with the aid of discrete sampling, the generated images better align with the box instructions and adhere to the textual input semantically.
Quantitative analysis of the proposed two losses. Here we analyze the impact of our proposed two loss functions: region-aware loss L, and boundary- aware loss Ls quantitatively. The generative accuracy (%) on HRS and Drawbench reported in Table 4. We observe that both C, and La greatly boost the performance of the original Stable Diffusion because they incorporate the layout information into the generative process. Generally, using L, for guidance yields higher accuracy than using Ls, especially in terms of the size metric on HRS. This is because L, models the discrepancy between constraints more precisely, and the shape and localization of
maps and bounding-bor generated objects better align with the layout instructions. Moreover, find that the performance of the diffusion model gets further improved when combining the above two losses together. This demonstrates that these two losses cooperate with each other well during the guidance process.
Methods
Stable Diffusion w/o C,
w/o C.
R&B
Spatial
8.48
20.01
25.99
30.14
HRS
Size
9.18
15.05
23.26
26.74
Color
12.61
23.22
27.32
32.04
DrawBench
Spatial
12.50
42.50
47.50
55.00
Table 4: Quantitative results for different constraints. We evaluate two variants, where L, and Lo are removed from our full method. The generative accuracy (%) on HRS and DrawBench is reported.
F ADDITIONAL RESULTS
Qualitative comparisons. Figure we provide qualitative comparisons among different baselines and our proposed R&B. Layout-guidance suffers from severe attribute misbinding problem, while BoxDiff and DrcentionReffocusing suffer from missing object. These issues inhereted from Stable Diffusion, which reflects that guidance from these methods is insufficient for the model to understand the semantics of each object. Moreover, all baselines fail generate objects that conform to the bounding boxes accurately. Our proposed R&B mitigates the semantic issues by incentivizing object activation in region and boundary aware manner. further greatly improves the alignment between generated images and layout conditions through explicitly minimizing the difference between MBRs and ground-truth bounding boxes. As result, R&B generally produces images that better conform to user input both semantically and spatially. Additionally, find that R&B consistently produces high-quality images regardless of seed selection, while other baselines rely on cherry picking from a pile of generated images to satisfy user input.
Quantitative comparisons. We evaluate the adherence to the box constraints and textual input of generated images for different diffusion-based methods. The experiments are conducted on
17

---

Published as a conference paper at ICLR 2024
"An eagle is flying tree"
BoxDiff
Layout guidance
Atlention- refocusing
R&B
globe beside bottle of water"
BoxDiff
Layout guidance
Attention refocusing
R&B
Figure 9: Visual comparisons among Layout-guidance (Chen et al., 2023), BoxDiff (Xie et al., 2023),
Attention-Refocusing (Phung al., 2023) and the proposed R&B.
Methods
Stable Diffusion
BoxDiff
Layout-guidand Attention-refocusing
R&B (Ours)
mloU (t)
0.2700
0.3753
0.3960
0.4460
0,5533
T21-Sim (t)
0.2920
0.3003
0.3023
0.3108
0.3218
Table Quantitative comparisons with baselines. We calculate the mean IoU and CLIP similarity for different methods on the selected subset of MS-COCO. For each sample, we randomly generate five images for evaluation. Best results are bold.
18

---

Published as a conference paper at ICLR 2024
"A riding dog"
person and beside car
person, chair beneath an airplane"
Figure 10: Failure cases of our proposed method.
the manually collected subset of MS-COCO with 100 samples. We observe that the SOTA zero- shot grounded T2I methods Boxfiff (Xie al., 2023), Layout-guidance (Chen al., 2023), and Attention-refocusing (Phung et al., 2023) greatly improve the mean IoU of generated images because they effectively incorporate the layout information into the diffusion process. Meanwhile, the T2I similarity of Stable Diffusion is also boosted due to the spatial alignment. By comparison, our proposed R&B surpasses the existing methods by a large margin from the two aspects of fidelity to the text prompt and alignment with the layout condition, which agrees with the visual results shown Figure 4. Quantitative results further validate that our method helps the model to generate images with better spatial accuracy, and simultancously convey the essential semantics of the text prompts.
More qualitative results. Figure 11, 12, and 13 illustrate synthetic results the proposed R&B under different random seeds. We observe obvious textural level variations across different random seeds, which shows great diversity the synthetic results. Yet the objects in the generated images consistently align with the layout conditions. In Figure 14, given fixed bounding box condition, vary the key phrases of the original prompt "A castle stands across the lake under the blue sky" to adopt different generative results. Our proposed R&B produces images with (1) high fidelity the various prompts, and (2) adherence to the layout instructions.
G LIMITATIONS AND DISCUSSION
Figure 10 shows three possible failure cases of our proposed method. Firstly, if the prompt is too uncommon, c.g. Out-Of-Domain, our method cannot generate accurate objects according to the given layout. Secondly, unreasonable layouts also worsen the generation, leading to unrealistic objects. This is because the pretrained base model (Stable Diffusion) does not cover these extreme scenes. Thirdly, prompts with too many objects (four and more) are beyond the capacity of model and missing objects are observed.
19

---

Published as a conference paper at ICLR 2024
"A tree stands on the grass land with mountains in the background"
Figure 11: Synthetic trees, grasslands, and mountains with the same spatial conditions.
20

---

Published as a conference paper at ICLR 2024
"An upright bear riding bicycle"
Figure 12: Synthetic bears riding bicycles with the same spatial conditions.
21

---

Published as a conference paper at ICLR 2024
"A Monet painting of a woman standing on a flower field, holding an umbrella sideways"
Figure 13: Synthetic women holding umbrellas on flower fields in the style of Monet with the same spatial conditions.
22

---

Published as a conference paper at ICLR 2024
across under sky
"A terple stands across the lake under the blue sky"
across
sky"
"A village stands across the lake under the blue sky"
"A castle stands nountain under
"A castle stands across the grass land under the blue sky"
"A castle stands across the cenetery under the dark sky"
Figure 14: Synthetic different buildings on different backgrounds.
23

---

Published as a conference paper at ICLR 2024
orange fox under black ball"
orange can under black ball"
"An under a black ball"
blue under a red ball"
"An under a pink ball"
blue under a pink dog"
slice waterme on besides green apple"
"A cute dog besides
cute dog red handbag"
"A cute dog besides pink handbag"
"A fluffy dog besides pink handbag"
"A fluffy cat besides pink bowl"
blue handbag"
Figure 15: Illustration the effect of R&B attribute binding in T2I models. We vary the attributes and nouns in the textual prompts generate different images. Textual changes emphasized.
H VARIATION OF ATTRIBUTES AND NOUNS
In Figure 15 illustrate how the proposed R&B copes with the attribute binding issues. Given consistent grounding input, we vary the nouns or attributes to adopt different synthetic results. Vanilla Stable Diffusion struggles to allocate multiple attributes with respect to different objects properly, which results in semantic errors. Nevertheless, we witness that R&B helps to bind the nouns and their corresponding attributes across different variations.
COMPARISON WITH GLIGEN
In this section, we first compare our R&B with the training-based grounded T21 generation approach GLIGEN (Li et al., 2023a) from the quantitative aspect. We then illustrate some visual comparisons for better comprehension.
GLIGEN searches for embedding the spatial instructions into a pretrained T2I diffusion models (e.g. g., Stable Diffusion). It injects auxiliary gated self self-attention layer at each transformer block to absorb new grounding input. Training Stable Diffusion model on COCO detection datasets requires for 500k iteration on 16 V 100 GPUs.
Quantitative. We quantitatively compare our method with GLIGEN on HRS and Drawbench. Numerical results are shown in Table 6. We find that although well-trained on annotated data, GLIGEN does not perform well on color setting, which implies that does not effectively cope with the attribute binding issues and some out-of-domain textual prompts (e.g., "a red dog and an orange horse"). As for size and spatial categories, GLIGEN yields high accuracy because the generated images align well with the size and location of the grounding input. Notably, when
Methods
SAR&CAR
R&B
GLIGEN
GLIGEN-SAR&CAR
GLIGEN+R&B
Spatial
24.45
30.14
45.71
54.19
56.87
HRS
Size
16.97
26.74
36.13
39.72
42.69
Color
23.54
32.04
17.84
29.46
35.72
Benwh
Spatial
43.50
55.00
48.00
64.00
67.50
Table Quantitative comparisons with GLIGEN. The evaluation accuracy (%) is reported. Best results are bold. Second best results underlined.
24

---

Published as a conference paper at ICLR 2024
"As the aurora lights up the sky, a herd reindeer leisurely wanders on the grassy meadow, adniring breathtaking view, lake quietly reflects magnificent display, and in the distance, snow-capped mountain stands najestically, fantasy, 8k, highly detailed"
BoxDiff
Layout- guidancce
Atlention- refocusing
GLIGEN
R&B
11
12
13
airplane tv
BoxDiff
TV
Tayout- guidanee
refocusing
GLIGEN
R&B
Visual results between training-free free methods and training based method. The example prompts are borrowed from the paper of BoxDiff. Seeds for latent initialization are annotated the bottom. Zoom in for better view.
Figure 16:
combined with R&B, the performance GLIGEN gets consistently boosted (the rightmost column), because the region-aware loss further enhances the alignment between generative objects and the corresponding boxes, and the boundary- loss helps with the expression of different objects in their corresponding regions to better cooperate with the textual semanties. The performance of
25

---

Published as a conference paper at ICLR 2024
"A teddy bear and two cats sitting on the grass
Layout- guidance
Attention- refocusing
GLIGEN
R&B
"Four shoes on the field with football
in the background"
Layout- guidance
Attention-
refocusing
GLIGEN
0000
R&B
Figure 17:
Visual results between training-fiee methods and training- based method on object counts setting. Zoom in for better view.
strong competing method "GLIGEN+SAR&CAR" is also reported in the second rightmost colmun of Table 6. SAR&CAR the loss function adopted from Attention-refocusing (Phung al., 2023).
Qualitative. Some visual comparisons between GLIGEN and our proposed R&B are shown in Figure 16, 17 and 18. The box information and textual prompts of Figure 16 inherited from the paper of BoxDiff (Xie et al., 2023). The random seeds for latent initialization are shown in the bottom of the images. In the top half of Figure 16, all methods present relatively delicate outcomes, because the textual input is rather concrete and constitutes an in-domain prompt. In the bottom half of Figure 16, given an unusual prompt, we witness entanglement of semantics in the images generated by previous methods. For example, BoxDiff, Attention-refocusin and GLIGEN generate televisions inside the cabin an airplane, Layout-guidance and GLIGEN generate TV's with airplane the screen. By comparison, with the aid of the design region and boundary aware constraints, approach better cope with the semantic confusion issue in grounded generation, and generate images that align with the layout instructions.
Comparisons on the setting of object counts are presented in Figure 17. BoxDiff not included since it does not support the input format of multiple boxes pointing to the same object. Training-free
26

---

Published as a conference paper at ICLR 2024
Layout Instruction
GLIGEN+ R&B
GLIGEN
Text Prompt
"a person under airplane"
"a person and banana under chair and above car
photo
"a car and a airplane, the car is bigger than the airplane"
"a person which is bigger than car and dog and larger than chair"
"a yellow airplane and blue horse"
"a blue airplane, red horse and orange dog"
Figure 18: Visual results of the boost of GLIGEN combined with R&B on HRS benchmark. The up- permost, middle, lowermost two rows correspond the spatial, size and color category, respectively.
27

---

Published as a conference paper at ICLR 2024
methods like Layout-guidange and iitention-refoeusing suffer from attribute leakage and missing object problems, therefore fail to consistently generate accurate object counts. Specifically, Layout- guidance generates cats that severely inherit features from the teddy bear and Attention-refocusing often forgets to generate small cats. The same problem also happens at the case of football and shoes. GLIGEN, training-based method, generates right amount of objects, but somewhat at the cost of decreased generative quality and diversity: (1) the lighting and texture of teddy bears and cats do not match the background, and (2) the teddy bear always faces forward and the four shoes share the same appearance. In comparison, our R&B generates consistently high-quality results while preserving diversity, due the accurate layout guidance provided by the proposed the ambiguity of the word football causes two different appearances of the generated ball.
losses. Please note that
Figure 18 includes some visual examples on the HRS benchmark, We witness that GLIGEN provides strong prior to grounded multi-object generation, but still fails to accurately generate images with some unusual prompts. As an attention manipulation approach, R&B (1) boosts the generative fidelity
Layout Instruction
Zest
R&B
Text Prompt
"A brown teddy bear sitting in a red chair"
"A horse standing besides a red car"
"A girl in white shirt riding on a horse"
"A woman holding a phone on her hands"
"A giraffe wearing red magician hat, sunglasses, and a tie, a bundle of balloon"
Figure 19: Visual comparisons between Zest and our proposed R&B.
28

---

Published as a conference paper at ICLR 2024
of GLIGEN (1 row) of images, (2) helps to cope with the entanglement of different concepts (2nd row), (3) helps to fix poor semantics like size (4th row) and color (5th and 6th rows). This is confirmed by the quantitative results in Table 6.
J COMPARISON WITH ZEST
Zest (Couairon et al., 2023) is zero-shot segmentation guidance approach for T2I diffusion mod- els. adopts a binary cross-entropy (BCE) loss for each cross-attention map entry, enabling the representation of objects in their respective mask regions, formulated below:
S: |
Lest Σ (C, S.) LECE(
S.))
(19)
In Figure 19 we compare our method with Zest from the visual aspect. We observe when conditioned on grounded input like bounding boxes, which only coarsely describe the size and location of corresponding objects, Zest tends to impose too strong constraints to the diffusion models. As shown the 2nd and 4th rows, the generated horse, car, and woman simply fill the entire boxes, presenting poor fidelity. We also observe the presence of peculiar artifacts some synthetic images (2nd and 3rd rows). By comparison, our proposed R&B aligns well with the grounding input, and meanwhile presents images with better quality and semantic consistency.
29


Reviews:
[{"review_id": "8YE59ZAvoE", "reviewer": "Reviewer_Nptk", "summary": "This paper presents a training-free approach to grounded text-to-image generation. The key idea is to instrument the cross-attention weights in a pre-trained Stable Diffusion model to control image layout, thereby generating objects within the given bounding boxes. The key innovation of the proposed method lies in the energy functions in guidance. Two such functions are introduced; the region-aware loss facilitates the minimum-sized box enclosing the object to match the ground-truth box, whereas the boundary-aware loss encourages rich content variation within a box. Both qualitative and quantitative experiments validate the effectiveness of the proposed loss functions. The method outperforms several training-free baselines by a wide margin, enabling more precise layout control over text-to-image generation.", "questions": "- I want the authors to comment on the runtime of the proposed method. How is it compared to the vanilla generation process (i.e., without guidance). In addition, how sensitive is the method to hyper-parameters?\n\n- Is the method able to handle overlapping objects? Most examples in the paper show non-overlapping bounding boxes. I am curious about how cross-attention guidance behaves when two or more objects compete for the same pixel.\n\n- I want to understand the performance of the method with respect to object size. Since the attentions maps have relatively low resolution (down to 16x16), I am wondering whether the losses (especially the boundary-aware loss) are still effective in the presence of small objects. Some qualitative results and visualizations would be helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is training-free. It controls image layout by manipulating cross-attention weights using guidance. Compared to methods that train auxiliary modules for controllability, training-free methods readily support new architectures and model checkpoints without re-training.\n\n- The proposed method carefully reasons about the failure cases of existing methods and introduces key improvements to cross-attention based guidance for grounded text-to-image generation. These improvements cover different aspects of object localization, including dynamic thresholding of attention maps, differentiable box alignment and and boundary awareness. They offer key insights into the internal workings of Stable Diffusion and may be of interest to the community in a broader context.\n\n- The proposed loss terms effectively improve bounding box localization and address missing objects. The method outperforms several training-free baselines in both qualitative and quantitative experiments. Visualizations of cross-attention maps demonstrate clear localization and separation of object regions.", "weaknesses": "- Several training-free methods similarly apply cross-attention guidance to control object shape and location. To this end, the contribution of the present work is more or less incremental. While the loss functions are new, the novelty mainly lies in the implementation details, with the overall idea (cross-attention guidance) largely identical to previous methods.\n\n- Comparison with training-based methods are lacking. For example, GLIGEN (which the paper cites) is a training-based method for grounded text-to-image generation. As they solve the same task, it would be helpful to compare with those methods qualitatively and quantitatively to reveal the strength and weakness of both approaches (and also for the sake of completeness).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FThnwi4LI6", "reviewer": "Reviewer_KFuf", "summary": "The paper proposes a new method for grounded text-to-image generation based on the introduced region-aware and boundary-aware losses. These two losses help to modify the generation process of a pre-trained diffusion model in a way that generated objects adhere to the locations of provided bounding boxes. The method is compared to recent state-of-the-art backward methods such as BoxDiff, Attention refocusing, Layout-guidance. The experiments demonstrate that the method outperforms previous approaches in the accuracy of following the conditioning instructions.", "questions": "Please answer the questions or comment on concerns from the Weaknesses section.\n\n[UPD rebuttal - score raised from 5 to 6]", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an interesting and in-demand problem of grounded T2I generation with diffusion models, tailored for applications that require control over locations of objects via user-friendly inputs.\n- The proposed approach is compared to the most recent backward baselines, including BoxDiff [ICCV23].\n- The demonstrated results indicate clear improvement over the baselines in terms of accuracy of following the conditioning instructions like the size or location of specified objects.", "weaknesses": "- My main concern is unclarity in the conceptual differences of the proposed method to other methods. Training-free modification of attention map is not a new idea as other methods like (Chen et al., 2023; Xie et al., 2023; Phung et al., 2023) also design objectives that encourage the model to shift objects towards specified bounding boxes. These methods generally exhibit same motivation (align objects with bounding boxes) with similar implementation (shift cross-attention maps). It is not clear what makes the proposed approach conceptually different and where the performance gain comes from.\n   - Could the authors please elaborate their explanations in intro: 1) \"previous methods fail to provide accurate spatial guidance\" - why does this happen based on their design, and why does this not happen in R&B method based on its design? 2) \"inherit inconsistencies from the original T2I model\" - doesn't this also happen in R&B? Or what part of the design helps to avoid them?\n- Similarly, it is not clear to me what is the technical novelty of the proposed method. The narration of the method is structured as the adaptation of existing methods (Li et al., 2023b, Li et al., 2023b) to the task of interest without much new analysis. Could the authors please elaborate what could be a broader-impact technical lesson or insight for the community from Sec. 3?\n- I feel the presentation of experiments in Sec. 4 can be improved. For example, there is no presentation of baselines used in Table 1, Fig. 4, so it is difficult to the match the names of methods with references to respective papers.\n- Why are there no comparisons to some other baselines, like GLIGEN?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QMYWm330iL", "reviewer": "Reviewer_cFE6", "summary": "This paper introduces an approach to zero-shot grounded Text-to-Image (T2I) generation in diffusion models, termed Region and Boundary (R&B) aware cross-attention guidance for layout generation. Notably, this approach is training-free. By incorporating region-aware loss for guiding cross-attention maps and boundary-aware loss for refining localization, the method delivers enhanced ground-alignment performance. Empirical evidence from experiments and ablation studies underscores the method's superiority over existing state-of-the-art techniques for grounded T2I generation in diffusion models, demonstrating both qualitative and quantitative improvements.", "questions": "1. How does this model perform when provided with more intricate prompts, such as \"a pink rabbit with a white head,\" with the boundary pointing to both the \"rabbit\" and \"head\"?\n\n2. I find the result in the last row of the rightmost image in Figure 4 perplexing. The image seems to exhibit incomplete deformation. What factors may have led to this particular outcome?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper employs a dual approach, utilizing both region-aware loss and boundary loss to align the generated images with the text and boundary, presenting a rational and innovative methodology.\n\n2. By incorporating dynamic thresholding into the aggregated cross-attention maps, this method introduces a novel optimization technique for accentuating foreground objects, enhancing the utility of aggregated cross-attention maps.\n\n3. The visual results presented in this paper provide compelling evidence that the R&B method surpasses current state-of-the-art techniques in grounded Text-to-Image generation for diffusion models.", "weaknesses": "1. While the experimental results presented appear less solid, it's worth noting that Attention-refocusing is a versatile component that has demonstrated effectiveness in enhancing various methods, including Stable Diffusion, Layout-guidance, and GLIGEN. The results, as detailed in Tables 2 and 3 of the Attention-refocusing method, highlight the superiority of GLIGEN+CAR&SAR in achieving state-of-the-art (SOTA) performance, which outperforms the proposed method, as indicated in Table 1. Further clarification regarding this observation would be greatly appreciated.\n\n2. Although the LIMITATIONS AND DISCUSSION section acknowledges that the model's capacity may be exceeded when dealing with a high number of objects, providing a more in-depth analysis of this phenomenon would enhance the discussion. Notably, the BoxDiff model appears to yield more favorable results in generating a large number of objects.\n\n3. To enhance the comprehensiveness of the study, including additional visual comparisons with other methods would be advantageous. For instance, an evaluation of how our model performs with consistent boundary and prompts when compared to BoxDiff or other comparable methods could provide valuable insights.\n\n4. It is imperative to incorporate more detailed analyses, such as color and object counts. Additionally, there is a concern regarding the last row in the rightmost image of Figure 1, where the text \"A bundle of\" does not seem to be adequately performed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aPlCEZmDUD", "reviewer": "Reviewer_F2Yp", "summary": "This paper proposes R&B, a region- and boundary-aware cross-attention guidance for a zero-shot grounded Text-to-Image diffusion model. The goal of such a method is to produce an image conditioned on an input text prompt and bounding boxes for objects to be generated. While previous methods have tackled this task, they still lack in terms of accurate spatial generation and such issues such as missing objects and attribute-noun binding issues. First, attention maps for every concept are extracted. Next, bounding boxes are estimated from those attention maps and used for optimization in consecutive timesteps. The region-aware loss maximizes IoU of estimated boxes with ground-truth boxes, while the boundary-aware loss maximizes activation per object within the corresponding box.", "questions": "- There is recent work that tackles poor attribute-noun binding in Text-to-Image models. Can the authors provide examples where one object has an additional attribute such as a color and visualize how the generated image is changed when varying the attribute or noun (e.g. \"blue car\" -> \"orange car\" -> \"orange fox\")? See [2] examples.\n\n[2] Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models, https://arxiv.org/abs/2305.13921", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and well-structured.\n- The presented results are good.\n- The proposed method is novel.", "weaknesses": "- It is unclear to me how the presented work should be viewed in comparison with ZestGuide [1]. The authors cite it and use the method in [1] to compute an aggregated attention map. However, there is no visual/quant. comparison and a discussion on the strengths/weaknesses are missing. From a high-level, it seems that [1] is able to solve the same issues using a simpler method, namely a zero-shot segmentation approach to guide the diffusion process on a per-object level. Furthermore, [1] is able to use free-form masks instead while the presented approach is limited to boxes.\n\n[1] Couairon et al., 2023, Zero-shot spatial layout conditioning for text-to-image diffusion models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)