GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation
Authors: Yuseung Lee
TaeHoon Yoon
Minhyuk Sung
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Generation
Visual Grounding
Primary area: diffusion_based_models
TL;DR: We introduce a novel training-free approach for bounding box-based image generation leveraging the semantic sharing properties of Diffusion Transformers.

Abstract: We introduce GrounDiT, a novel training-free spatial grounding technique for text-to-image generation using Diffusion Transformers (DiT). Spatial grounding with bounding boxes has gained attention for its simplicity and versatility, allowing for enhanced user control in image generation. However, prior training-free approaches often rely on updating the noisy image during the reverse diffusion process via backpropagation from custom loss functions, which frequently struggle to provide precise control over individual bounding boxes. In this work, we leverage the flexibility of the Transformer architecture, demonstrating that DiT can generate noisy patches corresponding to each bounding box, fully encoding the target object and allowing for fine-grained control over each region. Our approach builds on an intriguing property of DiT, which we refer to as semantic sharing. Due to semantic sharing, when a smaller patch is jointly denoised alongside a generatable-size image, the two become semantic clones. Each patch is denoised in its own branch of the generation process and then transplanted into the corresponding region of the original noisy image at each timestep, resulting in robust spatial grounding for each bounding box. In our experiments on the HRS and DrawBench benchmarks, we achieve state-of-the-art performance compared to previous training-free approaches. Project Page: https://groundit-diffusion.github.io/.

GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation
Phillip Y. Lee"
Tachoon Yoon' KAIST
Minhyuk Sung
{phillip0701, taehoon, mhsumg) Skaisst ac..i kr
bailoon and frrework night sky, and below is hosutiful forest Waterrflll
honana and on apple are Beneath book and a flower lying book
lights up the sky and a horse and house grassy mesdemi with assustaia De background.
antique living room with plour and chiffoous while chandetier hanging from the ceiling.
car a insstie beoutfful landscupe and a Batloon sky."
Figure 1: Spatially grounded images generated by our GROUNDIT. Each image is generated based on a text prompt along with bounding boxes, which are displayed in the upper right corner of each image. Compared existing methods that often struggle accurately place objects within their designated bounding boxes, our GROUNDIT enables more precise spatial control through novel noisy patch transplantation mechanism.
Abstract
We introduce GROUNDIT, a novel training-free spatial grounding technique for text-to-image generation using Diffusion Transformers (DiT). Spatial grounding with bounding boxes has gained attention for its simplicity and versatility, allow- ing for enhanced user control in image generation. However, prior training -reen free approaches often rely on updating the noisy image during the reverse diffusion pro- cess via backpropagation from custom loss functions, which frequently struggle to provide precise control over individual bounding boxes. In this work, we leverage the flexibility of the Transformer architecture, demonstrating that DiT can generate noisy patches corresponding to each bounding box, fully encoding the target object and allowing for fine-grained control over each region. Our approach builds intriguing property of DiT, which we refer as semantic sharing. Due to semantic sharing, when smaller patch is jointly denoised alongside generatable-sizz image, the two become semantic clones. Each patch denoised in own branch of the generation process and then transplanted into the corresponding region of the original noisy image at each timestep, resulting in robust spatial grounding for each bounding box. In our experiments on the HRS and BrawBench benchmarks, achieve state-of-the-art performance compared to previous training-fro approaches. taatt-off
Project Page: https: https::/ground:::: wtt.:// wew.iiinnnnttttb..eeeeeeeeeeeeee github.ion io/
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The Transformer architecture 45] has driven breakthroughs across a wide range of applications, with diffusion models emerging significant recent beneficiaries, Despite the success of diffusion models with U-Net 42 as the denoising backbone 22
recent Transformer-based diffusion 37 have marked another leap in performance. This is demonstrated by recent state-of-the-art generative models such as Stable Diffusion 3 [13] and Sora (6). Open-source models like DiT (37 and its text-guided successor PixArt-a [8) have also achieved superior quality compared to prior U-Net-based diffusion models. Given the scalability of Transformers, Diffusion Transformers are expected to become the new standard for image generation,
43
39
models, namely Diffusion Transformers (DiT)
especially when trained on an Internet-scale dataset.
With high quality image generation achieved, the next critical step is to enhance user controllability. Among the various types of user guidance in image generation, one of the most fundamental and significant is spatial grounding. For instance, user may provide not only text prompt describing the image but also a set of bounding boxes indicating the desired positions of each object, as shown in Fig. [ Such spatial constraints can be integrated into exxt-to-image (T2I) diffusion models by adding extra modules that are designed for spatial grounding and fine-tuning the model. GLIGEN (31) is notable example, which incorporates gated self-attention module [ into the U-Net layers of Stable Diffusion 41. Although effective, such mee-tuning-base approaches incur substantial training costs each time new T2I model is introduced.
Recent training-free free approaches for spatially grounded image generation 9
36 38
26
have led to new advances, removing the high costs for fine-tuning. These methods leverage the fact that cross-attention maps in T2I diffusion models convey rich structural information about where [9]. Building on this, these
each concept from the text prompt is being generated in the image 7 approaches aim to align the cross-attention maps of specific objects with the given spatial constraints (e.g. bounding boxes), ensuring that the objects are placed within their designated regions. This alignment is typically achieved by updating the noisy image in the reverse diffusion process using backpropagation from custom loss functions. However, such loss-guided update methods often struggle to provide precise spatial control over individual bounding boxes, leading to missing objects (Fig. Row 9, Col. 5) or discrepancies between objects and their bounding boxes (Fig. 4 Row 4, Col. 4). This highlights the need for finer control over each bounding box during image generation.
We aim to provide more precise spatial control over each bounding box, addressing the limitations in previous loss- guided update approaches. A well-known technique for manipulating local regions of the noisy image during the reverse diffusion process is to directly replace merge the pixels (or latents) in those regions. This simple but effective approach has proven effective in various tasks, including compositional generation [17 50
44 32] and high-resolution generation 5 29 24
25 One could consider defining an additional branch for each bounding box, denoising with the corresponding text prompt, and then copying the noisy image into its designated area in the main image at each timestep. However, a key challenge lies in creating a noisy image patch-at the same noise level-that reliably contains the desired object while fitting within the specified bounding box. This has been impractical with existing T2I diffusion models, as they are trained on a limited set of image resolutions. While recent models such as PixArt-a
g support a wider range of image resolutions, they remain constrained by specific candidate sizes, particularly for smaller image patches. As a result, when these models are used create local image patch, they are often limited to denoising a fixed-size image and cropping the region to fit the bounding box. This approach can critically fail to include the desired object within the cropped region.
In this work, we show that by exploiting the flexibility of the Transformer architecture, DiT can generate noisy image patches that fit the size of each bounding box, thereby reliably including each desired object. This is made possible through our proposed joint denoising technique. First, we introduce an intriguing property of DiT: when a smaller noisy patch is jointly denoised with a generatable-sizz noisy image, the two gradually become semantic clones- a phenomenon we call semantic sharing. Next, building on this observation, we propose a training- free framework that involves cultivating a noisy patch for each bounding box in separate branch and then transplanting that patch into its corresponding region in the original noisy image. By iteratively transplanting the separately denoised patches into their respective bounding boxes, we achieved fine-grained spatial control over each bounding box region. This approach leads to more robust spatial grounding, particularly in cases where previous methods fail to accurately adhere to spatial constraints.

---

In our experiments on the HRS 3 and DrawBench 44
datasets, we evaluate our framework,
GROUNDIT, using PixArt-a as the base text-to-image DiT model. Our approach demonstrates superior performance in spatial grounding compared to previous training-free methods [8X 9 47 4%
especially outperforming the state-of-the approach [47]. highlighting its effectiveness in providing fine-grained spatial control.
2
Related Work
In this section, we review the two primary approaches for incorporating spatial controls into text- to-image (T2I) diffusion models: fine-tuning-basee methods (Sec. 2.1) and training- -fre guidance techniques (Sec. 2.2
2.1 Spatial Grounding via Fine- Fine-Tuning
Fine-tuming with additional modules is powerful approach for enhancing T2I models with spatial
grounding capabilities [51 31
2 53
46
16
10
54]. SpaTexx [2] introduces a spatio-textual
representation that combines segmentations and CLIP embeddings [40] ControlNet [51] incorporates
a trainable U-Net encoder that processes spatial conditions such as depth maps, sketches, and human
keypoints, guiding image generation within the main U-Net branch. GLIGEN [3] enables T2I models to accept bounding boxes by inserting gated attention module into Stable Diffusion [1]] GLIGEN's
strong spatial accuracy has led to its integration into follow-up spatial grounding methods 48
38 30)
and applications such as compositional generation [15] and video editing (23). InstanceDiffusion 46
further incorporates conditioning modules to provide finer spatial control through diverse conditions
like boxes, scribbles, and points. While these fine-tumin fin-tuning methods are effective, they require task- specific datasets and involve substantial costs, they must be retrained for each new T2I model, underscoring the need for training- free alternatives.
2.2 Spatial Grounding via Training-Free Guidance
In response to the inefficiencies of fine-tuning, training-free approaches have been introduced to incorporate spatial grounding into T2I diffusion models. One approach involves region-wise composition of noisy patches, each conditioned on a different text input [5] 50 32). These patches, extracted using binary masks, are intended to generate the object they are conditioned the generated image. However, since existing T2I diffusion models are limited to a fixed set of complete image, making it uncertain whether
within
image resolutions, each patch cannot be treated the extracted patch will contain the desired object. Another approach leverages the distinct roles of attention modules in T2I models soadels-self-tteee self-attention captures long-range interactions between image features, while cross-attention links image features with text embeddings. By using spatial constraints such bounding boxes segmentation masks, spatial grounding can be achieved either by updating the noisy image using backpropagation based on a loss calculated from cross-attention maps [48] 38 D 18 361, by directly manipulating cross- or self-attention maps follow the given spatial layouts 26 [4], While the loss-guided methods enable spatial grounding in a training-free manner, they still lack precise control over individual bounding boxes, often leading to missing objects or misalignmet between objects and their bounding boxes. In this work, we propose novel training free framework that offers fineggrained spatial control over each bounding box by harnessing the flexibility of the Transformer architecture in DiT.
Background: Diffusion Transformers
Diffusion Transformer (DiT) represents a new class of diffusion models that utilize the Trans- former architecture 45 for their denoising network. Previous diffusion models like Stable Diffu- architecture, of which each layer contains a convolutional block and
sion use the U-Net 42 attention modules. In contrast, DiT consists of sequence of DIT blocks, each containing pointwise feedforward network and attention modules, removing convolution operations and instead processing image tokens directly through attention mechanisms.
3

---

Grounding Condittinns
Update
Local Update
G [gt 3
Main Branch
Globel Update
PB (-)
DIT
by Bounding
Pi Object
(-th Ofjert Biranch
G Text embedcling
O Suseer Eacd (Au))
Local Patch Var
Vie=i
Joint Densising
Object Image U.
Mex-1
DIT
Sp
Figure 2: A single denoising step in GROUNDIT consists of two stages. The Global Update (Sec. 5.1 established coarse spatial grounding by updating the noisy image with a custom loss function. Then, the Local Update (Sec. [53) further provides fine-grained spatial control over individual bounding boxes through a novel technique called noisy patch transplantation.
DìT' follows the formulation of diffusion models real clean data X by
22 which the forward process applies noise to a
X v Vo + v1 0zg where <~ <~ 1), az E [0, 1].
The reverse process denoises the noisy data x2 through Gaussian transition
pe(x = (xxi (xx,,, t), Ee (x,, t))
(2)
where (x,. () is calculated by learned neural network trained by minimizing the negative ELBO objective [27] While Dg t) can also be learned, usually set as time dependent constants.
Positional Embeddings. As DiT is based on the Transformer architecture, it treats the noisy image X, Rixwxx as of image tokens. Specifically, x, divided into patches, each transformed into an image token via linear embedding. This results in (h/l) x (w/l) tokens, where is the patch size. Importantly, before each denoising step. 2D sine-cosine positional embeddings are assigned to each image token to provide spatial information as follows:
- Denoise(PE(xx))
Here, PE(.) applies positional embeddings, Denoise(:) represents a single denoising step in DiT at timestep and is the text embedding. This contrasts with U-Net-based diffusion models, which typically do not utilize positional embeddings for the noisy image. Detailed formulations of the positional embeddings are provided in the Appendix (Sec.
4 Problem Definition
Let P be the input text prompt (i.e., a list of words), which we refer to as the global prompt. Let cp be the text embedding of P. We define a set of N grounding conditions G = [ N where each condition specifies the coordinates of bounding box and the desired object to be placed within it. Specifically, each condition 9 := (b,, Pt, c. consists of the following: by E R21, the ry-coordinates of the bounding box's 'o's lefe and lower-right corners, Pt E P. the word the global prompt describing the desired object within the box, and c, the text embedding of The objective is generate image that aligns with the global prompt P while ensuring each specified object is accurately positioned within its corresponding bounding box b;.
GROUNDIT: Grounding Diffusion Transformers
We propose GROUNDIT, training-free framework based on DiT for generating images spatially grounded on bounding boxes. Each denoising step in GROUNDIT consists of two stages: Global Update and Local Update. Global Update ensures coarse aligmment between the noisy image and the bounding boxes through gradient descent update using cross-attention maps (Sec. 5.1). Following this, Local Update further provides fine-grained control over individual bounding boxes via a novel
noisy patch transplantation technique (Sec. 5.3) This approach leverages our key observation of
DiT's semantic sharing property, introduced Sec. 5.2
An overview of this two-stage denoising
step is provided in Fig.

---

5.1 Global Update with Cross-Attention Maps
First, the noisy image X₃ is updated to spatially align with the bounding box inputs. For this, we leverage the rich structural information encoded in cross- coss-attention mäps, as first demonstrated by Chefer al. [7]. Each cross-attention map shows how a region of the noisy image correspond to specific word in the global prompt P. Let DiT consist of M sequential DiT blocks. As passes through the m-th block, the cross-attention map d E Rhxux1 for object p; extracted. For each grounding condition Ji. the mean cross-attention map Ag. is obtained by averaging of over all M blocks as follows:
M Σ m=0
A
(4)
M
For convenience, we denote the operation of extracting A for every E G as below:
{Ai,) N-1 < Extractattenttinn(x,.t cp, G).
Then, following prior works U-Net-based diffusion models 48] 47 38 9
71 measure
the spatial alignment for each object p; by comparing its mean cross-attentin map A., with its corresponding bounding box b;, using predefined grounding loss C(A,,,, b,) as defined in R&B 477. The aggregated grounding loss LAGG is then computed by summing the grounding loss across all grounding conditions Ji G:
N-1
CAGO ( G)
Σ C(A b,).
(6)
Based on the backpropagation from LAGG, x2 is updated via gradient descent follows:
* x V CAGG
where wi is scalar weight value for gradient descent. We refer to Eq. 7 as the Global Update, as the whole noisy image x, updated based on an aggregated loss from grounding conditions in G.
The Global Update achieves reasonable accuracy in spatial grounding. However, often struggles with more complex grounding conditions. For instance, when G contains multiple bounding boxes (e.g., six boxes in Fig. 4 Row 9) or small, thin boxes (e.g., Fig. 4 Row 5), the desired objects may be missing or misaligned with the boxes. These examples show that the Global Update lacks fine-grained, box-speciff control, underscoring the need for precise controls for individual bounding boxes. In the following sections, we introduce a novel method achieve this fine-grained spatial control.
5.2 Semantic Sharing in Diffusion Transformers
In this section, present our observations on an intriguing property of DiT, semantic sharing, which will serve as a key building block for our main method in Sec. 5.3
Joint Denoising. We observed that DiT can jointly denoise two different noisy images together. For example, consider two noisy images, X; and yr. both at timestep t in the reverse diffusion process. Position embeddings are applied the image tokens according their sizes, resulting x; PE(x,) and y, = PE(y.). Notably, the two noisy images can differ in size, providing flexibility in joint denoising. The key part that the two noisy images- more precisely, two sets image tokens are merged into a single set z₁. We denote this process as Merge(-) (-) (see Alg. line 4). z, is passed through the DiT blocks, yielding the output z,-1, which is then split into the denoised versions x;-1 and yt-1 via Split(-). Joint denoising is illustrated in Fig. (A). and a pseudocode for single step of joint denoising is shown in Alg.
Semantic Sharing, Surprisingly, we found that joint denoising of two noisy images generates semantically correlated content correponding pixels, even when the initial random noise differs. Consider two noisy images, E x and yr E Xuy both initialized from unit Gaussian distribution (0, T). We experiment with reverse diffusion process in which, for the initial 100y% % of the denoising steps (y e [0, 1)), xx and yr undergo joint denoising. For the remaining
5

---

(A) oint Denoising
(B) Semantic Sharing between Equal Resolutions
X
PR
y
O0
Meege
Yo
(C) Semantic Sharing between Different Resolutions
"dog"
DiT
R1-1
X0
Spiit
Y--1
Yo
Figure 3: (A) Joint Denoising. Two different noisy images, X and Ye. are each assigned positional embeddings based on their respective sizes. The two sets of image tokens then merged and passed through DiT for denoising step. Afterward, the denoised tokens are split back into X and yr-1- (B), (C) Semantic Sharing. Denoising two noisy images using joint denoising results in semantically correlated content between the generated images. Here, y indicates that joint denoising is during the initial 100y of the timesteps, after which the images denoised for the remaining steps.
timesteps, they are denoised independently. The same text embedding is used as condition in both cases.
Fig. shows the generated images from Xy and yr across different y values. In Fig. 3B), xT and YT have the same resolution (hx hy: Uxx Wyy. while in Fig. (CC their resolutions differ (hx > hy. Wxx > w. When 0, the two noisy images are denoised completely independently. resulting clearly distinct images (leftmost column). We found that DiT models have certain range of resolutions within which they can generate plausible images-which we refer to as generatable resolutions-but face challenges when generating images far outside this range. This demonstrated in the output of in Fig. (C) with O. Further discussions and visual analyses are provided in the Appendix (Sec. D But as y increases, allowing Xy and yr be jointly denoised in the initial steps, the generated images become progressively more similar. When = 1, the images generated from Xy and yr appear almost identical. These results demonstrate that, in joint denoising, assigning identical or similar positional embeddings to different image tokens promotes strong interactions between them during the denoising process. This correlated behavior during joint denoising causes the two image tokens to converge toward semantically similar outputs phenomenon we term semantic sharing.
Notably, this pattern holds not only when both noisy images share the same resolution (Fig. 3 but even when one of the images does not have DiT's generatable resolution (Fig. ()). While self-attention sharing techniques have been explored in U-Net- based diffusion models to enhance style consistency between images 20 34]. they have been limited images of equal resolution. By leveraging the flexibility to assign positional embeddings across different resolutions, our joint
Algorithm 1: Pseudocode of Joint Denoising (Sec. 5.2
Inputs: E Xxxxxxxx Vt E R Xwy e, // Noisy images, timestep, text embedding,
patch aize.
Outputs: X1-1, Function (xintDenoiseexx.yyuu
// Noisy images at timestep
nx hwwx/f /lP ny +- hywy/l
// Store the number of image tokens.
X: PE(x:). Yt PE(y:);
// Apply positional embeddings, // Merge two sets of image tokens.
7: + Merge(X;, y.);
7:-1 + Denoise(Z,,
// Denoising step with DiT.
[x--1, Split(z.-1)) ( ny))
// Split back into two sets.
return 1t-1

---

denoising approach extends across heterogeneous resolutions, offering greater versatility. We provide further discussions and analyses on semantic sharing the Appendix (Sec. D).
5.3 Local Update with Noisy Patch Transplantation
In this section, we introduce our key technique: Local Update via noisy patch cultivation and transplantation. Building on DiT's semantic sharing property from Sec. 5.2 we show how this can be leveraged to provide precise spatial control over each bounding box.
Main & Object Branches. We propose a parallel denoising approach with multiple branches: one for the main noisy image & and additional branches for each grounding condition gi- The main branch denoises the main noisy image using the global prompt P, while each object branch is designed to denoise local regions within the bounding boxes, enabling fine-grainee spatial control over each region. For each i-th object branch, there is a distinct noisy object image u,,., initialized as u,, ~ ((O, 1). We predefine the resolution of the noisy object image u., by searching in PixArt-a's generatble resolutions that closely match the aspect ratio of the corresponding bounding box b,. With & obtained from Global Update (Sec. [5.1]. each branch performs denoising in parallel. Below we explain the denoising mechanism for each branch.
Noisy Patch Cultivation. In the main branch at timestep t, the noisy image X₂ is denoised using the global prompt P as follows: - < (eanois(PE(X:)) t, cp), where & is the output from the Global Update and cp is the text embedding of P. For the i-th object branch, there are two inputs: the noisy object image u and a subset Vi, of image tokens extracted from Xt, corresponding to the bounding box by. We denote this extraction as Vi, + Crop(X,, b.), where Vi. E Rxx
xw; xd is referred as a noisy local patch. Here, h; and w, corresponds to height and width of bounding box 6,, respectively. Joint denoising is then performed on u;, and Va, to yield their denoised versions:
[uir-11, Vi,t-13 } < JointDenoise (U,,, c4),
where e is the text embedding of the object pr.
Through semantic sharing with the noisy object image u,,, during joint denoising, the denoised local patch Vi,- is expected to gain richer semantic features of object p; than it would without joint denoising. Note that even when the noisy local patch Vi. does not meet the typical generatable resolution of DiT (since it often requires cropping small bounding box regions of &, to obtain vi.)). it offers a simple and effective way for enriching Vi, of the semantic features of object p4- We refer to this process noisy patch cultivation.
Noisy Patch Transplantation. After cultivating local patches through joint denoising in Eq. g each patch is transplanted into x obtained from the main branch. The patches are transplanted in their original bounding box regions specified by b, as follows:
x x ① (1 m.) + Uncrop(v 2 m;
Here, denotes the Hadamard product, m; is binary mask for the bounding box b, and Uncrop(V:.r-11 (v... by) applies zero-pes sero-padding to V..-1 to align its position with that of b;. This trans- plantation enables fine-grained local control for the grounding condition gi After transplanting the outputs from all N object branches, we obtain X,-1, representing the final output of GROUNDIT denoising step at timestep l. In X,-1, the image tokens within the b, region are expected possess richer semantic information about object pi compared to the initial x;- from the main branch. This process is referred to as noisy patch transplantation. We provide implementation details and full pseudocode of a single GROUNDIT denoising step in the Appendix (Sec. E).
6 Results
In this section, we present the experiment results of our method, GROUNDIT, and provide compar- isons with baselines. For the base text-to-image DiT model, we use Pix-rt-- ( which builds on the original DiT architecture (37 by incorporating an additional cross-attentionn module to condition on text prompts.
7

---

Layout
Layoul- Guidance v
Attention- Refecusing 38
Box
"A dog in the beautiful park."
"An eagle is flying over a tree."
duck wearing hat standing near a bicycle."
plastic bottle and an apple on table."
"An apple and banana cup on table."
""A dog wearing sunglasses and hat and blue tie."
"A chair and table and bed is the with photo frame the wall and ceiling lamp
"A dog and bird sitting branch while an cagle is flying in the sky."
"A cur and dog the road while horse and a chair is the grass."
Figure 4: Qualitative comparisons between our GROUNDIT and baselines. Leftmost column shows the input bounding boxes, and columns 2-6 include the baseline results. The rightmost column includes the results of our GROUNDIT.

---

6.1 Evaluation Settings
Baselines. We compare our method with state-of-the training- free approaches for bounding box-based image generation, including R&B [47], Box [48], Attention Rettrion--efousingg [38], and Layout-Guidance GuinnnGe [1]. For fair comparison, we also implement R&B using PixArt-a, which we refer to as &ixArt-R&B, and treat it as internal baseline. Note that this is identical to our method without the Local Guidance (Sec. 5.3).
Evaluation Metrics and Benchmarks.
(Grounding Accuracy) We follow the evaluation protocol of R&B 47 to assess spatial grounding on the HRS (3 and Draw Bench (43] datasets, using three criteria: spatial, size, and color. The HRS dataset consists of 1002, 501, and 501 images for each respective criterion, with bounding boxes generated using GPT-4 by Phung et al. 38 For DrawBench, we use the same 20 positional prompts in R&B [47).
(Prompt Fidelity) We use the CLIP score 21 to evaluate how well the generated images adhere to the text prompt. Additionally, we assess method using PickScore [28] and ImageReward [4), which provide human alignment scores based on the consistency between the text prompt and generated images.
HRS
DrawBende Spatial (%)
Method
Spatial (%)
Size (%)
Color (%)
Backbone: Stable Diffusion [41)
Stable Diffusion
[1]
8.48
9.18
12.61
12.50
PixArt-a: 8
17.86
11.82
19.10
20.00
Layout- Guidance
9
16.47
12.38
14.39
36.50
Attention-Refocusiing
38
24.45
16.97
23.54
43.50
Diff
16.31
11.02
13.23
30.00
R&B 47
30.14
26.74
32.04
55.00
Backbone: PixArt-a
PixArt-R&B
37.13
20.76
29.07
60.00
GROUNDIT (Ours)
45.01
27.75
35.67
60.00
Table 1: Quantitative comparisons grounding accuracy on HRS [3] and Draw Bench
43
bench-
marks. Bold represents the best, and underline represents the second best method.
6.2 Grounding Accuracy
Quantitative Comparisons. Tab. presents quantitative comparison of grounding accuracy be- tween our method, GROUNDIT, and baselines. GROUNDIT outperforms all baselines across different criteria of grounding accuracy-spation size, and color-including the state-of-the-ee R&B [47 and our internal baseline PixArt-R&B. Notably, the spatial accuracy on the HRS benchmark (Col. 1) of GROUNDIT is significantly higher, with + +14.87% improvement over R&B and +7.88% over PixArt-a. The comparison between PixArt-a [8]. PixArt-R&B and GROUNDIT highlights the effectiveness of the two-stage pipeline of GROUNDIT. First, integrating the loss- Global Update into PixArt-a results in a substantial improvement spatial accuracy (from 17.86% to 37. 37.13%) Then, incorporating our key contribution, the Local Update, further boosts accuracy (from 37.13% to 45.01%). For size accuracy (Col. 2), which evaluates how well the size of each generated object matches its corresponding bounding box. GROUNDIT shows + -1.11%% improvement over R&B. In terms of color accuracy (Col. 3), our method achieves +6.60% improvement over PixArt- FixArt-R&B and outperforms R&B by +3.63%. This underscores the effectiveness of our noisy patch transplantation technique accurately assigning color descriptions to the corresponding objects. As DrawBench 43 only contains images with two bounding boxes, which relatively easy generate, employing the Global Update is sufficient for grounding. We present additional quantitative comparisons of grounding accuracy in the Appendix (Sec.
Qualitative Comparisons. Fig. presents the qualitative comparisons. When the grounding condition involves two simple bounding boxes (Rows 1, 2), both method and the baselines

---

successfully generate objects within the designated regions. However, as the number of bounding boxes increases and the grounding conditions become more challenging. the baselines struggle to correctly place each object inside the bounding box (Rows 4, 8), or even fail to generate the object at all (Rows 5, 7, 9). In contrast, GROUNDIT successfully grounds each object within the boxes, even when the number of boxes is relatively high, such as four boxes (Rows 5, 6, 8), five boxes (Row 7) and six boxes (Row 9). This highlights that our proposed noisy patch transplantation technique provides superior control over each bounding box. addressing the limitations of previous loss-based update methods, as discussed Sec. 5.T For more qualitative comparisons, including images generated with various aspect ratios, please refer to the Appendix (Sec. G and Fig. 5)
6.3 Prompt Fidelity
Tab. presents quantitative comparison of prompt fidelity between our method and PixArt-R&B. Each metric is measured using the generated images from the HRS dataset []. GROUNDIT achieves higher CLIP score [2I] than PixArt-R&B (Col. 1), indicating that our noisy patch transplantation improves the text prompt fidelity of the generated images. Additionally, our method achieves higer RewgeReward [49] score, which measures human preference by considering both prompt fidelity and overall image quality. While GROUNDIT shows slight underperformance compared to PixArt-R&E in Pickscore (28), it remains comparable overall. We provide further comparisons of prompt fidelity with other baselines in the Appendix C).
Method
CLIP score
ImageReward
PickScore
PixArt-R&B
33.49
0.28
0.52
GROUNDIT (Ours)
33.63
0.44
0.48
Table 2: Quantitative comparisons on prompt fidelity on HRS benchmark (3. Bold represents the best method.
7 Conclusion
In this work, we introduced GROUNDIT, training-free spatial grounding technique for text-to-image generation, leveraging Diffusion Transformers (DiT). To address the limitation of prior approaches, which lacked fine-grained spatial control over individual bounding boxes, proposed novel approach that transplants a noisy patch generated in separate denoising branch into the designated area of the noisy image. By exploiting an intriguing property of DiT, semantic sharing, which arises from the flexibility of the Transformer architecture and the use of positional embeddings, GRoUNDIT generates smaller patch by simultaneously denoising two noisy image: one with smaller size and the other with a generatable resolution by DiT. Through semantic sharing, these two noisy images become semantic clones, enabling fine-graing spatial control for each bounding box. Our experiments on the HRS and PrawBench benchmarks demonstrated that GROUNDIT achieves state-of-the-ars performance compared previous training-fres grounding methods.
Limitations and Societal Impacts. A limitation of our method is the increased computation time, as it requires a separate object branch for each bounding box. We provide further analysis on the computation time in the Appendix (Sec. F). Additionally, like other generative Al techniques, our method is susceptible to misuse, such as creating deepfakes, which can raise significant concerns related to privacy, bias, and fairness. It is crucial to develop safeguards to control and mitigate these risks responsibly.
10

---

Acknowledgements
We thank Juil Koo and Jaihoon Kim for valuable discussions on Diffusion Transformers, This work was supported by the NRF grant (RS-2023-00)009723, IITP grants (RS-2019-1119000 RS- 2022-11220594, RS-2023-00227592, RS-2024-0039981177 and KEIT grant all funded by the Korean government (MSIT and MOTIE), as well as grants from the DRB-KAIST Futuure Research Center, NAVER-Intel Co-Lab, Hyundai NGV, KTT, and Samsung Electron- ics.
References
[1] Jean- Bean-Baptiste Alayrae, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
[2] Omri Avrahami. Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski. Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In CVPR, 2023.
[3] Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Faroog Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In ICCV, 2023.
[4] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine. Bryan Catanzaro, al, ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.011 1.01224 2022.
[5] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusion paths for controlled image generation. In ICML, 2023.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufci Guo, Li Jing. David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang. and Aditya Ramesh. Video generation models world simulators. attp:://opeaa..ccarrreeccc com/research/
dideo-generation-nodiie wo-rlldd- aaa--oolddsiiilttrreee 2024.
[7] Hila Chefer. Yuval Alaluf. Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention- based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics, 2023.
[8] Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.
[9] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In WACV, 2024.
[10] Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdiffuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv:2302.08908 2023.
[11] Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. arXiv preprint arXiv:2403. 13589, 2023.
[12] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self--uidance for controllable image generation. NewrlPS, 2023.
[13] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis, arxiv preprint 2tt::2403003206 2024.
[14] Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-fre free structured diffusion guidance for compositional text-to-image synthesis. In /CLR, 2023.
[15] Weixi Feng. Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang. and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305.1 13335, 2023.
[16] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. ECCV, 2022.
[17] Songwei Ge, Tacsung Park, Jun- Yan Zhu, and Jia-Bin Huang. Expressive text-to- imago generation with rich In ICCV, 2023.
[18] Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting coxt-to-image diffusion models via initial noise optimization. In CVPR, 2024.
11

---

[19] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman. Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. In ICLR, 2023.
[20] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. In CVPR, 2024.
[21] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[23] Hyeonho Jeong and Jong Chul Ye. Ground-a-video: Zero-shot grounded video editing using text-to-image diffusion models. In ICLR, 2024.
[24] Gwanghyun Kim, Hayeon Kim, Hoigi Sco, Dong Un Kang, and Se Young Chun. Beyondscene: Higher- resolution human-centric scene generation with pretrained diffusion. arXiv preprint arXiv:240.004544, 2024.
[25] Jaihoon Kim, Juil Koo, Kyeongmin Yeo, and Minhyuk Sung. Synctweedies: A general generative framework based on synchronized diffusions. arXiv preprint arXiv:2403.14377 2024.
[26] Yunji Kim. Jiyoung Lee, Jin-Hwa Kim, Jung-Wop Woo Ha, and Jun- Zhu, Dense text-to-image generation with attention modulation. In /CCV, 2023.
[27] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[28] Yuval Kirstain, Adam Polyak, Uriel Singer. Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In NeurlPS, 2023.
[29] Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. SyncDiffusion: Coherent montage via
synchronized joint diffiusions. In NewrlPS, 2023.
[30] Yuseung Lce and Minhyuk Sung. Reground: Improving textual and spatial grounding at no cost, arXiv preprint 2403 /3589, 2024.
[31] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, 2023.
[32] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-grounded diffusion: Enhancing prompt understanding of text-to to-eiageeeaaaae diffusion models with large language models. TMLR, 2024.
[33] Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
[34] Yuxin Liu, Minshan Xie, Hanyuan Liu, and Tien-Tsin Wong. Text-guided texturing by synchronized multi-view diffusion. arXiv preprint arXiv:2311.12891 2023.
[35] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfci Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022.
[36] Wan-Dus Kurt Ma. Avisck Lahiri, JP Lewis, Thomas Leung, and W Bastiaan Kleijn. Directed diffussion: Direct control of object placement through attention guidance. In AAAI, 2024.
[37] William Peebles and Saining Xie. Scalahle diffusion models with transformers. In ICCV, 2023.
[38] Quynh Phung. Songwei Ge. and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:2306.0 05420,422227. 2023.
[39] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint XrXiv:2307.0/953 2023.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assiiteed intervention-MICCAl 2015: 18th international conference, 2015.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-iman to-image diffusion models with deep language understanding. In NewrIPS, 2022.
12

---

[44] Takahiro Shirakawa and Seichh Uchida. Noisecollage: A layout-aware text-to-image diffusion model based noise cropping and merging. In CVPR, 2024.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[46] Xudong Wang. Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, Ishan Misra. Instancediffusion: Instance-level control for image generation. In CVPR, 2024.
[47] Jiay Xiao, Liang Li, Henglei Lv, Shuhui Wang. and Qingming Huang. R&b: Region and boundary aware zero-shot grounded text-to-image generation. ICLR, 2024.
[48] Jinheng Xie, Yuexiang Li. Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training- free box-constrained diffusion. In ICCV, 2023.
[49] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In NewrlPS, 2023.
[50] Ling Yang. Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal Ilms. JCML, 2024.
[51] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[52] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[53] Guangeong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In CVPR, 2023.
[54] Dewei Zhou, You Li, Fan Ma. Xiaoting Zhang, and Yi Yang. Mige: Multi-instance generation controller text-to-image synthesis. In CVPR, 2024.
[55] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühh. Simple multi-dataset detection. In CVPR, 2022.
13

---

Appendix
"A wide view picture of an antique living room with a chair table, fireplace, and bed"
"A high-definition space photo with galaxy, sun, spaceship, and Earth"
"A high- definition ocean photo with submarine, jellyfish, turtle and cora!"
"A photo of shelves with books, bananas, and vase"
Figure 5: Spatially grounded images generated by our GROUNDIT with varying aspect ratios and sizes. Each image is generated based on text prompt along with bounding boxes, which are displayed next to (or below) each image.
14

---

A Positional Embeddings in Diffusion Transformers
Diffusion Transformers (DiT) 337 handle noisy images of varying aspect ratios and resolutions by processing them as set of image tokens. For this, the noisy image is first divided into patches. with each patch subsequently converted into an image token of hidden dimension D through a linear embedding layer. DiT then applies 2D sine-cosine positional embeddings each image token, based on its coordinates (), defined as follows:
Px, = CONCAT [Px, Pyl. where
Ps == [cos(wa 2), sin(wa x)]d=o D/4
Py = [cos(wa y), sin(ws y) y)
where wy 1/ 1/10000(4//D) The positional embedding Pa, is then added each corresponding image token, denoted as PE(-).
B Additional Quantitative Comparisons: Grounding Accuracy
In addition Sec. 6.2. provide further quantitative comparisons of grounding accuracy between our GROUNDIT and the baselines. Specifically, we generated images based on text prompts and bounding boxes using each method, then calculated the mean Intersection over Union (mloU) between the detected bounding boxes from an object detection model (55) and the input bounding boxes. Below, we present the quantitative comparisons across three datasets with varying average numbers of bounding boxes: subset of MS-COC0-2014 [33], HRS-Spatial 3 and a custom dataset.
Subset of
Dataset
HRS-
Custom Dataset
MS-COCO-2014 33
Avg. of Bounding Boxes
2.06
3.11
4.48
Table 3: Average number of bounding boxes per dataset.
Subset of MS-COCO-2014. We filtered the validation of MS-COCO-2014 33 to exclude image-caption pairs where the target objects were either not mentioned the captions or duplicate objects present. From this filtered set, we randomly selected 500 pairs for evaluation.
The results are presented in Tab. column 2. GROUNDIT outperforms R&B by 0.021 (a 5.1% improvement) and PixArt-R&B by 0.014 (a 2.2% improvement). The relatively small margin can be attributed to the simplicity of the task, as this dataset has an average of 2.06 bounding boxes for the baseline methods.
(Tab. 3) making it less challenging
HRS-Spatial. Column 3 of Tab. presents the results on the Spatial subset of the HRS dataset K GROUNDIT surpasses R&B [7 by 0.046 (a 14. 1 % improvement) and PixArt-R&E by 0.038 (an 11.4% improvement). Compared to the results the MS-COCO-200 2014 subset, the higher percentage increase in mIoU highlights the robustness of GROUNDIT under more complex grounding conditions. Note that HRS-Spatial has average of 3.11 bounding boxes (Tab. which is higher than that of the MS- AIS-COCO-20) subset (2.06).
Custom Dataset. The custom dataset consists of 500 layout-text pairs, generated using the layout generation pipeline from LayoutGPT [15). As shown in column 4 of Tab, GROUNDIT outperforms R&B by 0.052 (a 26.3% improvement) and PixArt-R&B by 0.044 (a 21.4% improvement). This dataset has the highest average number of bounding boxes at 4.48 (Tab. 3 These results further emphasize the robustness and effectiveness of our approach in handling more complex grounding conditions with larger number of bounding boxes.
15

---

Subset of
Method
HRS-Spatial
3
Custom Dataset
MS-COCO-2000
Backbone: Stable Diffusion [1
Stable Diffusion
0.176
0.068
0.030
PixArt-a R
0.233
0.085
0.036
Sayout-Guidanne
0.307
0,199
0.122
Attention-Refocustiee 38
0.254
0.145
0.078
BoxDiff 448
0.324
0.164
0.106
R&B 47
0.411
0.326
0.198
Backbone: PixArt-a (
Pix
0.418
0.334
0.206
GROUNDIT (Ours)
0.432
0.372
0.250
Table 4: Quantitative comparisons of mloU (f) on subset of MS-CCCC-2011 [3
HRS-S HRS-Spatial 3
and our custom dataset. Bold represents the best, and underline represents the second best method.
C Additional Quantitative Comparisons: Prompt Fidelity
In addition to Sec. 6.3 we provide further quantitative comparisons of the prompt fidelity of the generated images between our GROUNDIT and the baselines. We evaluated the generated images from the HRS dataset [3] using three different metrics: CLIP score [21], ImageReward [49], and PickScore (28). The results are presented in Tab. 5 Since PickScore evaluates preferences between a pair of images, report the difference between our GROUNDIT and each baseline method in column Our GROUNDIT consistently outperforms the baselines in both CLIP score and ImageReward. For PickScore, GROUNDIT outperforms all baselines except PixArt- R&B, while remaining comparable.
PickScore
Method
CLIP score f
ImageReward 1
(Ours Baseline)
Backbone: Stable Diffusion [41]
Guyout-Guidance
32.48
-0.401
+0.30
Attention-Refoccuiing Refocusing
38
31.36
-0.508
+0.22
Diff 48
32.57
-0, 199
+0.30
R&B 47
33.16
-0.021
+0.26
Backbone: PixArt-a 8
PixArt-R&B
33.49
0.280
-0.04
GROUNDIT (Ours)
33.63
0.444
Table 5: Quantitative comparisons of prompt fidelity on the HRS dataset 3
Bold represents the best
method.
D Additional Analysis on Semantic Sharing
In this section, we provide further analyses on the generatable resolution and the semantic sharing property of DIT, initially introduced in Sec. 5.2
Generatable Resolution of DIT. Although recent DiT models can generate images at various resolutions, they still struggle to produce images completely arbitrary resolutions. We speculate that this limitation arises not from the model architecture itself, but from the resolution of the training images, which typically falls within specific range [ Generating images at resolutions far outside this range often results in implausible outputs, suggesting the existence of an acceptable resolution range for DIT, which we refer as generatable resolution. In Fig. [ we illustrate this phenomenon. When the noisy image size falls within DiT's generatable resolution range, the model produces plausible images (rightmost two images). However, when the image size significantly outside this range (leftmost two images), DiT fails to generate plausible image.
16

---

Prompt: A "og"
Image Size: [128x128]
[256x256]
[288x288]
[384x384]
[512x512]
Figure 6: Illustration the generatable resolution range of DiT. The images are generated using PixArt-a 8 from the text prompt "A dog". with varying resolutions.
Semantic Sharing. Even though DiT models have limited range of generatable resolutions, their Transformer architecture offers flexibility in handling varying lengths of image tokens, making feasible to merge two sets of image tokens and denoise them through single network evaluation. Leveraging this flexibility Transformers, we presented our joint denoising technique (Alg. D Our main observation was that the joint denoising between two noisy images causes the two generated images become semantically correlated, as illustrated Fig. (B) and Fig. (C))
In addition to the visualizations in Fig. we further quantify the semantic sharing property by measuring the LPIPS score [52] between two generated images. To explore the effect of joint denoising, we varied the parameter y E [0, I]. which controls the proportion of denoising steps where joint denoising applied. Specifically, 7 = 0 means no joint denoising is applied, and each image is denoised independently, while = means full joint denoising across all steps. As shown in Fig. 7 increasing 7 (i.e., applying more joint denoising steps) results in decrease in the LPIPS score between the two generated images, indicating that the images become more semantically similar as joint denoising applied for a larger portion of the denoising process.
0.63
0.60
0.55
0.7
0.50
LPIPS
0.45
0.6
0.40
0.35
0.30
0.25
10
20
30
40
50 (y=1)
10
20
30
40
50
(y=
Joint Demoising steps
(y=
Joint Denolsing steps
(y=
(a) Semantic sharing between equal resolutions
(b) Semantic sharing between different resolutions
Figure 7: LPIPS score between generated images with varying value. A gradual decrease in
LPIPS [52 indicates that joint denoising progressively enhances the similarity between the generated images.
E Implementation Details
As the base text-to-image DiT model. we used the 512-resolution version of PixArt-a [8] For sampling employed the DPM-Solver scheduler (35) with 50 steps. Out of the 50 denoising steps, we applied our GROUNDIT denoising step (Alg. 2 for the initial 25 steps, and applied the vanilla denoising step for the remaining 25 steps. For the grounding loss in Global Update of GROUNDIT,
we adopted the definition proposed R&B 47). and we set the loss scale to 10 and used a gradient descent weight of 5 for the gradient descent update in Eq. 7
As discussed Sec. 5.3 each i-th object branch we have a noisy object image Wi,t and a noisy local patch Ui,t, which is extracted from the noisy image X, in main branch via Crop & b;). We determine the resolution the noisy object image ui, by selecting from Pix 's generatable resolutions, choosing that best aligns with the aspect ratio of the corresponding bounding box 6,.
17

---

All our experiments were conducted an NVIDIA RTX 3090 GPU. In Algorithm 2 we provide the pseudocode of GROUNDIT single denoising step.
Algorithm 2: Pscudocode of GROUNDIT denoising step.
Parameters wis
// Gradient descent weight.
Inputs: x1, (uu, Cp; // Noisy images, grounding conditions, text embedding. // Noisy images at timestep
Outputs: X,-1, X,-1, (u,,-1) 1-
Function (xbbllpddatexx,
// b; holds coordinate information of bounding box, (Sec.
{A, Extrractattention(x...4
// Extract Pross-attention maps.
LAGG + Σ bi). X1 Xt We V return Xc;
// Compute aggregated grounding loss. // Gradient descent (Eq.
Function LocalUpdate (Kt, (us 1 (,rr,) G)
x Denoise(xx, cp):
// Main branch
for = O...., N do
// i-th object branch < Crop(x, b,);
// Obtain noisy local patch. // Joint denoising.
[U,,-1, V wointDenoiseeuu... &.);
for == ...... do 11
// is binary mask corresponding to b
K1-1 Xx-1 (1 m.) Uncrop(v.::1..)) m;;
// Patch Transplantation.
13
X--1 -
14
return - fup
Function (Ttep (urr)eo | W-1 tt G): GlobalUpdate (x,, cp,
15
// Global update (Sec. 5.1
17
X;-1, (u.e-1)] LocalUpdate (urc) 18 C)
// Local update
(Sec. 5.3)
return (1 N--1
18
F Analysis on Computation Time
present average inference time based on the number of bounding boxes Tab. 6 While our method shows slight increase in inference time, the rate of increase remains modest. For three bounding boxes, the inference time 1.01 times that of R&B and 1.33 times that of PixArt-R&BB Even with six bounding boxes, the inference time is only 1.41 times that of R&B and 1.90 times that of PixArt-R&es Ar
# of bounding boxes
6
R&B 47
37.52
38.96
39.03
39.15
PixArt-R&B
28.31
28.67
29.04
29.15
GROUNDIT (Ours)
37.71
41.10
47.83
55.30
Table 6: Comparison of the average inference time based on the number of bounding boxes. Values in the table are given in seconds.
G Additional Qualitative Results
We provide more qualitative comparisons in Fig. Our method demonstrates greater robustness against issues such as the missing object problem, attribute leakage, or object interruption prob- lem [47), due to its local update mechanism with semantic sharing. For instance, in Row 1, baseline methods struggle to generate certain objects (i.e. missing object problem).
Row 2, baselines generate a banana that retains features of an apple, illustrating attribute leakage. In Row 3, R&B generates bus that interrupts the generation of a couch, with part of the bus overlapping with the designate region of the couch. Similarly, in PixArt-R&B, hamburger and a donut interrupt the
18

---

Layout
R&B 47
PixAst-R&B
GROUNDIT
"A bear sitting between surfboard and a chair with a bird Bying in the sky."
"A banana and apple and elephant and backpack the meadow with bird flying in the sky."
"A realistic photo, a hamburger and a donut and a couch and a bus and a surfboard in the beach."
"A blue and wooden bowl with watermelon sit table, while bear holding an apple."
Figure 8: Additional qualitative comparisons between our GROUNDIT, the previous state-of oate--of-theart the-art,, R&B (47), and our internal baseline PixArt-R&B. Leftmost column shows the input bounding boxes, and columns 2-3 include the baseline results. The rightmost column includes the results our GROUNDIT.
generation of surfboard, demonstrating the object interruption problem. In more challenging cases, like Row combinations of these issues appear. By contrast, our method consistently generates each object accurately within specified locations, even under complex bounding box configurations, highlighting its robustness and precision. Additional results are shown in Fig. and examples of various aspect ratio images generated with grounding conditions are provided in Fig. 5
19

---

Layout
GROUNDIT
Layout
GROUNDIT
"A boat floating on calm lake."
"An upright bear riding bicycle."
"A aurora lights up the sky and a house is on the grassy meadow with a mountain in the background."
"A person is sitting on a chair and a bird is sitting horse while horse is the top of a car."
"A cat sitting on sunny windowsill."
"A bicycle standing near telephone booth in the park."
"A Monet painting of woman standing on flower field holding umbrella sideways with house in the background."
"A castle stands across the lake and the bird flies the blue sky."
Figure 9: Additional spatially grounded images generated by out GROUNDIT.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction include our key contribution of improving the spatial grounding of Diffusion Transformers via noisy patch transplantation, and mention the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Conclusion section and Section D of appendix, discuss main limitation, which is the increased computation time for image generation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work mainly based on empirical observations regarding Diffusion Transformers, and does not include theoretical findings.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the appendix, we provide the exact model that our method is build upon, and the implementation details required for reproducing the method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: While we have not provided the code yet, we provide a link to our project page, at which will provide the official code for our method.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pcccccpcccccooooooop https//gguuieeeoeeeuuubiiiiiiiii sesiisssonni siiniiossooiyccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Both in the Results section and Appendix, we provide the used benchmarks, experiment settings, and the used hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not provide error bars for the experiments as we have not run our main experiments multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the specific GPU resources and also the inference time of our generation method.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp://neurips. cwtpuuripscccpuuuui /
Answer: [Yes]
Justification: Our research process and results adhere the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the Conclusion section, we discuss the potential societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We do not discuss any additional safeguards for our method. However, we adhere the safeguards already implemented in text-to-image models that we utilize.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit the authors of Attention Refocusing 38 who have suggested the evaluation method, and also the creators of HRS and DrawBench 43
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: For the custom benchmarks that we have used additional experiments in the appendix, plan open-source them along with the official code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do include research with crowdsourcing and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not include research with crowdsourcing and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "iRTrguWce0", "reviewer": "Reviewer_LMpP", "summary": "This paper proposes a method to use a pre-trained text-to-image\ndiffusion model to guide the generation into placing objects at given\nlocations determined by bounding boxes. The challenge is to develop\nthis capability without requiring fine-tuning of the model.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The problem of guided generation without requiring fine-tuning of a\n  diffusion model is a relevant one, and the method seems effective.\n- Experimental results show the proposed method outperforms compared\n  baselines.", "weaknesses": "- The quality of the presentation can be improved. For instance, one\n  fundamental component of the method, $\\math{L}_{AGG}$, is not\n  defined. The evaluation metrics (spatial, size, color) are not\n  properly defined, unless I missed it.\n- Further analysis or discussion on why the proposed approach is\n  superior to simply directly transferring the patch would make the\n  paper more convincing.\n- Experimental comparison is limited to two datasets. Qualitatively,\n  it's not entirely clear there is a significant difference between\n  the R&B baseline and the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review, acknowledging that our work is solving a “relevant” problem of zero-shot guided generation through an “effective” method. Here, we address the concerns and questions that have been raised.\n\n**(1) Clarifications on the Main Method**\n\nWe would like to draw your attention to our general response above, where we provide detailed clarifications regarding our  method. \n\n**(2) Clarification on Grounding Loss $\\mathcal{L}\\_{AGG}$**\n\nPlease note that we provide clarifications on the definition of the grounding loss in our general response. We will include this in the revised version.\n\n**(3) Details on Evaluation Metrics**\n\nWe provide further details on the evaluation metrics from HRS-Bench [2]. We will also include this in the revised version.  \n\n* **Spatial:** Object detection is applied to the generated image. Based on the detection results, the spatial relations specified in the text prompt are checked. If the detected objects maintain the correct spatial relations as described in the text prompt, the image is classified as correct; otherwise, it is classified as incorrect.\n* **Size:** Similarly, based on the detected boxes, it is checked whether the size relations between the boxes follow the relations specified in the text prompt (e.g. “bigger than”).\n* **Color:** After obtaining a semantic segmentation for each object, the average hue color value within each segment is computed. The derived color is then compared with the color of the object specified in the text prompt to determine accuracy.\n\n**(4) Quantitative Comparisons on Additional Benchmarks**\n\nWe further conducted quantitative comparisons on two additional benchmarks: a subset of the MS-COCO and a new custom benchmark.\n\n**[MS-COCO Subset]**\n\nFollowing R&B, we conducted an experiment on a subset of MS-COCO. Since we could not obtain the exact subset utilized in R&B from the authors, we similarly prepared a subset of MS-COCO.\n\nWe first filtered out the image-caption pairs in the MS-COCO 2014 validation set where either the bounding box target objects were not specified in the caption, or duplicate objects were present in the grounding conditions. Then we randomly selected 500 pairs for evaluation. Following R&B, we measured the mIoU (mean IoU) for each bounding box condition.\n\n| |SD|PixArt-$\\alpha$|Layout-Guidance|Attention-Refocusing|BoxDiff|R&B|PixArt-R&B|GrounDiT (Ours)|\n|:--|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|mIoU ($\\uparrow$) | 0.176 | 0.233 | 0.307 | 0.254 | 0.324 | 0.411 | 0.418 | **0.432** |\n\nIn the table above, our GrounDiT achieves the highest grounding accuracy on the MS-COCO subset. Specifically, GrounDiT outperforms R&B by 0.021, representing a 5.1% increase, and PixArt-R&B by 0.014, representing a 2.2% increase. Note that the average number of bounding boxes in the MS-COCO subset is **2.06**, making it a relatively easy task even for baseline methods. Since the main advantage of GrounDiT over R&B and PixArt-R&B is its robustness when there is a higher number of bounding boxes, we provide further comparisons with a higher average number of bounding boxes below.\n\n**[HRS-Spatial Benchmark]**\n\n| |SD|PixArt-$\\alpha$|Layout-Guidance|Attention-Refocusing|BoxDiff|R&B|PixArt-R&B|GrounDiT (Ours)|\n|:--|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|mIoU ($\\uparrow$) | 0.068 | 0.085 | 0.199 | 0.145 | 0.164 | 0.326 | 0.334 | **0.372** |\n\nThe above table provides the mIoU comparisons on the Spatial subset of the HRS benchmark, which was also used in Sec. 5.2. The average number of bounding boxes in this dataset is **3.11**, which is larger than MS-COCO subset by 1.05. For mIoU, GrounDiT outperforms R&B by 0.046, indicating a 14.1% increase, and PixArt-R&B by 0.038, indicating a 11.4% increase. Compared to the 5.1% and 2.2% in the MS-COCO subset, respectively, the higher percentage of increase in mIoU demonstrates the robustness of GrounDiT on more complex grounding conditions. \n\n**[Custom Benchmark]**\n\n| |SD|PixArt-$\\alpha$|Layout-Guidance|Attention-Refocusing|BoxDiff|R&B|PixArt-R&B|GrounDiT (Ours)|\n|:--|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|mIoU ($\\uparrow$) | 0.030 | 0.036 | 0.122 | 0.078 | 0.106 | 0.198 | 0.206 | **0.250** |\n\nThe above table shows the comparisons on a new benchmark consisting of 500 layout-text pairs, generated using the layout generation pipeline from LayoutGPT [1]. The average number of bounding boxes in this benchmark is **4.48**. Here, GrounDiT outperforms RnB by 0.052, representing a 26.3% increase and PixArt-R&B by 0.044 representing a 21.4% increase. This is in line with the trend in the above datasets, **further highlighting the robustness and efficacy of our approach in handling a higher number of grounding conditions.**\n\n**(5) Additional Qualitative Comparisons with R&B**\n\nPlease refer to Fig. S1 of the attached PDF in the general response, where we provide additional qualitative comparisons between our method and the baselines, R&B and PixArt-R&B, with detailed analysis of each result.\n\n**(6) Advantage of Shared Sampling over Directly Transferring the Patches**\n\nPlease refer to our general response for a clarification on the effect of shared sampling:\n\nWe can think of each patch as an image on its own and pass it through DiT with the corresponding text condition. **However, this is not feasible in practice with existing DiT, as it has a certain set of preferred image resolutions.** This is due to the fact that the training data does not perfectly cover all resolutions. Please see Fig. S2 of the attached PDF in our general response for an illustration. But since DiT's Transformer architecture allows flexible token sequence length, we can perform shared sampling, which transfers the desired semantic features into the patch.\n\n[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models, Feng et al., NeurIPS 2023\n\n[2] HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models, Bakr et al., ICCV 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ItdiQPLec9", "reviewer": "Reviewer_bA1A", "summary": "This paper explores the zero-shot layout-to-image generation problem using bounding boxes and texts as conditional inputs. The authors propose a method called GrounDiT, which builds upon the recent Diffusion Transformers (DiT) model. Leveraging DiT's emergent property of semantic sharing, where two noisy images of different sizes can progressively converge during sampling, the authors introduce a key innovation: a separate object denoising branch for each bounding box condition, running in parallel to the main denoising branch. In each object branch, a noisy object patch is generated and then implanted into the original image. The authors compare GrounDiT with SOTA zero-shot methods on the HRS and DrawBench datasets, demonstrating that GrounDiT can generate complete objects in the correct regions while maintaining high overall image fidelity.", "questions": "Please refer to the weaknesses part.", "limitations": "In the conclusion part, the authors admit that the proposed method would increase the compuation time for image generation.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, with a clear and concise background and motivation.\n- The idea of having a separate object branch for each bounding box is simple and intuitive, making the method easy to understand and implement.\n- The authors conduct an extensive comparison with SOTA baselines, including R&B, BoxDiff, Attention-Refocusing, and Layout-Guidance, demonstrating the effectiveness of their approach.", "weaknesses": "- The paper lacks a comparison of prompt fidelity on the HRS dataset. Except for PixArt-R&B( a variant of GrounDiT), it would also be useful to include results from other baselines like R&B and BoxDiff to ensure the completeness of the work, even if they may use different backbones.\n\n- In Table 1, GrounDiT outperforms its variant PixArt-R&B in grounding accuracy, which is expected given the special design of noisy object patch cultivation and transplantation. However, in Table 2, GrounDiT continue to surpass PixArt-R&B in prompt fidelity scores like CLIP score and ImageReward. I am curious to know where these improvements come from, as object patch cultivation and transplantation should only benefit grounding ability. Did the authors perform multiple runs for the experiments?\n\n- The paper does not evaluate the MS-COCO subset presented in the R&B paper, which would be a valuable addition to the experiments.\n\n- The proposed method would increase the compuation cost for image generation. However, there is no quantitative evidence showing the exact cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your review, acknowledging that our paper is “well-written” and the proposed method is “simple and intuitive”. Here, we address the concerns and questions that have been raised.\n\n**(1) Additional Comparisons on Prompt Fidelity**\n\nHere we provide further quantitative comparisons on the prompt fidelity between our GrounDiT and baseline methods. We will include the results in the revised version.\n\n**[Prompt Fidelity]**\n\n| |Layout-Guidance|Attention-Refocusing|BoxDiff|R&B|PixArt-R&B|GrounDiT (Ours)|\n|:--|:---:|:---:|:---:|:---:|:---:|:---:|\n|CLIP score ($\\uparrow$) | 32.48 | 31.36 | 32.57 | 33.16 | 33.49 | **33.63** |\n|ImageReward ($\\uparrow$)| -0.401 | -0.508 | -0.199 | -0.021 | 0.280 | **0.444** |\n|PickScore (Ours $-$ Baseline)| +0.30 | +0.22 | +0.30 | +0.26 | -0.04 | - |\n\nThe table above shows the comparison of CLIP score, ImageReward and PickScore with the baselines. Since PickScore measures the preference among two images, we present their difference in the above table. In CLIP score and ImageReward, our GrounDiT outperforms the baselines. In PickScore, GrounDiT outperforms all baseline except PixArt-R&B, while remaining comparable.\n\n**(2) Relation between Grounding and Prompt Fidelity**\n\nAdditionally, we observed that precise local control provided by our GrounDiT method leads to the presence of target objects that are often missing in baseline methods, thereby increasing the prompt fidelity of the image. **Please refer to Fig. S1 in the attached PDF of our general response**, where we provide qualitative examples of these cases along with the corresponding CLIP scores for each image.\n\n**(3) Additional Quantitative Comparisons including MS-COCO**\n\nWe further conducted quantitative comparisons on two additional benchmarks: a subset of the MS-COCO and a new custom benchmark.\n\n**[MS-COCO Subset]**\n\nFollowing R&B, we conducted an experiment on a subset of MS-COCO. Since we could not obtain the exact subset utilized in R&B from the authors, we similarly prepared a subset from MS-COCO.\n\nFor this, we first filtered out the image-caption pairs in the MS-COCO 2014 validation set where either the bounding box target objects were not specified in the image captions, or duplicate objects were present in the grounding conditions. Subsequently, we randomly selected 500 pairs to use for evaluation. Following R&B, we measured the mIoU (mean IoU) for each bounding box condition.\n\n| |SD|PixArt-$\\alpha$|Layout-Guidance|Attention-Refocusing|BoxDiff|R&B|PixArt-R&B|GrounDiT (Ours)|\n|:--|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|mIoU ($\\uparrow$) | 0.176 | 0.233 | 0.307 | 0.254 | 0.324 | 0.411 | 0.418 | **0.432** |\n\nAs shown in the table above, our GrounDiT achieves the highest grounding accuracy on the MS-COCO subset. Specifically, GrounDiT outperforms R&B by 0.021, representing a 5.1% increase, and PixArt-R&B by 0.014, representing a 2.2% increase. Note that the average number of bounding boxes in the MS-COCO subset is **2.06**, making it a relatively easy task even for baseline methods. Since the main advantage of GrounDiT over R&B and PixArt-R&B is its robustness when there is a higher number of bounding boxes, we provide further comparisons with a higher average number of bounding boxes below.\n\n**[HRS-Spatial Benchmark]**\n\n| |SD|PixArt-$\\alpha$|Layout-Guidance|Attention-Refocusing|BoxDiff|R&B|PixArt-R&B|GrounDiT (Ours)|\n|:--|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|mIoU ($\\uparrow$) | 0.068 | 0.085 | 0.199 | 0.145 | 0.164 | 0.326 | 0.334 | **0.372** |\n\nThe above table provides the mIoU comparisons on the Spatial subset of the HRS benchmark, which was also used in Sec. 5.2. The average number of bounding boxes in this benchmark is **3.11**, which is larger than MS-COCO subset by 1.05. For mIoU, GrounDiT outperforms R&B by 0.046, indicating a 14.1% increase, and PixArt-R&B by 0.038, indicating a 11.4% increase. Compared to the 5.1% and 2.2% in the MS-COCO subset, respectively, the higher percentage of increase in mIoU demonstrates the robustness of GrounDiT on more complex grounding conditions. \n\n**[Custom Benchmark]**\n\n| |SD|PixArt-$\\alpha$|Layout-Guidance|Attention-Refocusing|BoxDiff|R&B|PixArt-R&B|GrounDiT (Ours)|\n|:--|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|mIoU ($\\uparrow$) | 0.030 | 0.036 | 0.122 | 0.078 | 0.106 | 0.198 | 0.206 | **0.250** |\n\nLastly, the above table shows the comparisons on a new benchmark consisting of 500 layout-text pairs, generated using the layout generation pipeline from LayoutGPT [1]. The average number of bounding boxes in this benchmark is **4.48**. Here, GrounDiT outperforms RnB by 0.052, representing a 26.3% increase and PixArt-R&B by 0.044 representing a 21.4% increase. This is in line with the trend in the above benchmarks, **further highlighting the robustness and efficacy of our approach in handling a higher number of grounding conditions.**\n\n**(4) Details on Computation Time**\n\nWe further provide the exact computation time for each method.\n\nThe numbers in the table below are in **seconds**.\n\n|Num. Boxes|3|4|5|6|\n|:--|:--:|:---:|:---:|:---:|\n|**R&B**| 37.52 | 38.96 | 39.03 | 39.15 |\n|**PixArt-R&B**| 28.31 | 28.67 | 29.04 | 29.15 |\n|**GrounDiT (Ours)**| 37.71 | 41.10 | 47.83 | 55.30 |\n\nWhile our method results in an increase in inference time, the rate of increase is not significant. For three bounding boxes, the inference time is 1.03 times that of R&B and 1.33 times that of PixArt-R&B for three bounding boxes. Even with 6 bounding boxes, the inference time is 1.41 times that of R&B and 1.90 times that of PixArt-R&B.\n\n[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models, Feng et al., NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RCRgJBbpym", "reviewer": "Reviewer_sEMS", "summary": "The paper targets exploiting a pre-trained PixArt-alpha (a text-to-image multi-aspect diffusion transformer) to generate images conditioned on a set of text-labeled bounding boxes.\n\nThe authors start from the idea that, at inference time, a transformer can be given an arbitrary number of tokens, and that these tokens can have positional embedding corresponding to multiple images of multiple aspect ratios at once. Empirically, they observe that in this situation, the diffusion transformer is able to handle the generation process consistently and the outputs are multiple images with shared content but with different aspect ratios corresponding to the input PEs.\n\nWhen generating the content of the bounding boxes, they take advange of this idea by simultaneously generating two version of the same bounding-box, one with positional embeddings corresponding to its location in the full image, and the other one with positional embeddings making the crop appear as a full image. \nThis ensures that the label is represented in the full-image version, and thanks to the shared content, also in the original box version, that can then be transplanted back into the image.", "questions": "I find this submission to be very interesting and show a lot of promise, but the description of the method is very confusing. For the discussion period, I hope the authors can provide the necessary clarifications and leave enough time for us to have a discussion together around their updated description.\nIndeed, until the main points are convincingly clarified, succintly reviewed again, and we are confident that changes will be made, I cannot recommend acceptance. My main question is then:\n- Can the authors clarify if anything happens in the main branch? Perhaps a full description of how the method works, including the reverse diffusion iterations would help.\n\nAlso, as a comparatively minor point, diffusion models are known for the flexibility of their sampling process. Considering how each stage and branch have compatible objectives, have the authors considered doing every update simultaneously in a single reverse diffusion process instead of multi-stage?", "limitations": "Limitations and impacts are discussed adequately.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The paper confirms an interesting property of multi-aspect diffusion transformers, in that tokens are able to interact at inference time, across combinations of positions that are not seen during training.\n- They propose a clever way to exploit this capability in the context of a multi-aspect diffusion transformer.\n- Handling spatial relations correctly is one of the major challenges that state-of-the-art text-to-image models have yet to overcome. The targetted task of grounding is very relevant as it is one way to circumvent the issue.\n- The proposed solution obtains very good performance on the considered benchmark.", "weaknesses": "Mainly, I find that the exposition of the idea not very clear and could be improved:\n- How the different branches, the two-stage process, and the diffusion scheduling interact is not clear:\n  - Appendix A.1 reveals that Stage 1 and 2 are merged in a single full 50 steps denoising schedule, with Stage 1 only being used in the 25 first steps, which is not exactly in line with what is presented in the main paper. This information needs to be in the main paper, and ideally the presentation should be made more consistent with the implementation.\n  - Figure 3 and Line 200 suggest that in the main branch x_t is denoised in parallel to the object branch, but it does not appear in Algorithm 1. This is inconsistent. And if the main brain does anything, it needs to be explained and detailed.\n- Section 3 should present how multi-aspect/size is handled in diffusion transformers, as it is a key requirement for the proposed method. Also, Eq.3 introduces $w$ without definition. \n- a smaller point that could be improved: in 4.2, while it can be derived from context and references that $\\mathcal{L}$ needs to be a grounding loss, this is nowhere explicitly stated. If I strictly adhered to what is written in this paragraph, the authors could very well be arguing that any loss function that takes as input $(x_t, \\mathcal(G)_i, \\mathcal(A)_i)$ would be effective, which is obviously not what they are trying to say.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your review, recognizing our method to be “very interesting” and “show a lot of promise”. Here, we address the concerns and questions that have been raised.\n\n**(1) Clarifications on the Main Method**\n\n**Please find the detailed clarification of the method in our general response above.** We will thorougly improve our presentation in the revised version. Moreover, please note that the notations in the below sections are also based on the above clarification.\n\n**(2) Clarification on Grounding Loss $\\mathcal{L}$**\n\nPlease note that we provide clarifications on the definition of the grounding loss in our general response. We will also include this clarification in the revised version.\n\n**(3) Parallel Denoising of the Main Branch and Object Branch**\n\nWhile in Algorithm 1 we meant to convey the parallel process of main branch and object branches, we notice that the presentation can be improved for clarity. We would like to provide clarifications below and further improve in the revised version.\n\n**Main Branch:** First, Line 13 of Algorithm 1 indicates the denoising step of the noisy image in the main branch. Importantly, the output from this step, $\\tilde{\\mathbf{x}}\\_{t-1}$ (originally $\\mathbf{x}\\_{t-1}$), is not directly used as an input for the denoising steps in each object branch.\n\n**Object Branch:** Lines 6-8 of Algorithm 1 represent the denoising steps of the object branches. Here, the output from the main branch, $\\tilde{\\mathbf{x}}\\_{t-1}$, is not directly used as input for the denoising. Instead, the output noisy patch $\\mathbf{b}\\_{i, t}$ (originally $B_{i,t}$) in Line 8 is later pasted back into the output $\\tilde{\\mathbf{x}}\\_{t-1}$ from the main branch, as shown in Line 9.\n\nTherefore, since the output from either the main branch or the object branches is not used as an input for the other branch, these processes are indeed parallel.\n\n**(4) Details on the Reverse Process of GrounDiT**\n\nWe clarify the reverse process of GrounDiT, supplementing Appendix A.1 of the main paper.\n\nOur reverse process consists of 50 steps using the DPM-Solver scheduler, where the GrounDiT denoising (Stage 1 and Stage 2) is applied for the first 25 steps. For the final 25 steps, the noisy image $\\mathbf{x}_t$ is denoised following the standard DiT denoising step. This strategy is also employed in previous works, including R&B, as the structure of the image is known to be mostly decided in the early steps. Therefore, applying our guidance in the early steps is sufficient for accurate grounding.\n\n**(5) Handling Multiple Image Sizes in DiT / Clarification on $w_d$ in Eq. (3)**\n\nWe appreciate your feedback on enhancing the clarity of our descriptions in Sec. 3. To address this, we have clarified DiT's mechanism for handling multiple resolutions below and will incorporate this into our revised version.\n\nDiT [1] handles multi-aspect/size noisy images by operating on a sequences of tokens. Given a noisy image $\\mathbf{x}\\_{t}\\in \\mathbb{R}^{H\\times W\\times C}$, DiT first divides it into patches of size $l \\times l$, creating a total of $T = (H/l) \\times (W/l)$ patches. Each patch is then transformed into a token with a hidden dimension of $D$ through linear embedding. A 2D sine-cosine positional embedding is applied to the sequence of tokens. The detailed formulation of positional embedding is given in Eq. (3) of our main paper, where $w_d$ follows original definition of positional embedding:\n$w_d = 1/10000^{(4d/D)}$ with $d$ running from $0$ to $D/4$.\nConsequently, when images with different aspect ratios or sizes are input into DiT, the primary difference is the token sequence length, which is managed by assigning appropriate positional embeddings to each sequence.\n\n**(6) Simultaneous Update in a Single Reverse Diffusion Process**\n\nPlease note that a single denoising step of our GrounDiT consists of the two-stage sequence: \"Stage 1 $\\rightarrow$ Stage 2\". During Stage 1, a global update is applied to the noisy image $\\mathbf{x}\\_t$. In Stage 2, precise local control is provided for each bounding box $b_i$ using our proposed shared sampling technique. We found it reasonable to perform the shared sampling based on the output $\\hat{\\mathbf{x}}\\_t$ obtained from Stage 1, as this allows to leverage the improved alignment resulting from the global update.\n\n[1] Scalable Diffusion Models with Transformers, Peebles et al., ICCV 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BaaHVBrHRv", "reviewer": "Reviewer_xDuk", "summary": "The paper introduces a zero-shot layout technique for spatially-grounded text-to-image generation with the power of transformer-based diffusion architecture. Prior methods tend to manipulate the latent image during reverse process for grounding, solely relying on cross-attention maps that show the alignment between specific text tokens and spatial regions of the target object to place in. This shows a drawback in generating a target object that fits within the region of bounding boxes. By utilizing transformer architecture, the method can divide the generation process into two branches: (1) first branch is to generate a target image patch of the bounding box with an arbitrary size by a transformer, (2) the second branch is the original generation process of the whole image. At every denoising step, the generated object is then copied into the generated image of the second branch.", "questions": "NA", "limitations": "Limitations and societal impacts are included in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Addressing the limitation of attention-based method in generating arbitrary object sizes within bounding boxes, the method proposes to utilize transformer architecture (namely DiT) which is free from fixed-resolution constraints and enables patch-level generation.\n- The paper is well-written and easy to follow.\n- Model efficacy is superior to other methods in Figure 4 and Table 1.", "weaknesses": "- 158-161: It is obvious that different noise inputs certainly yield different outputs. So I do not understand the main point of this part. \n- Semantic sharing part is an interesting observation. It is more helpful if the authors compare it with image interpolation. Does it work when two images have different semantic classes like dog and human?\nWang, Clinton, and Polina Golland. \"Interpolating between images with diffusion models.\" (2023). \n- Eq 5: What is formulation of $ \\mathcal{L}(x_t, G_i, A_i)$?\n- In reverse process: Have the authors applied semantic sharing on bounding boxes like concat tokens of multiple objects for simultaneous denoising? I assume that they share the same semantic class. \n- In Fig 3, I saw the size of object image is much bigger than the size of bounding box. Is it intentional? If so, please explain. How to define noisy object image $z_{i, t}$? As far as I understand, the method need to reserve the aspect ratio of the bounding box by scaling it correspondingly to match the size of main image. Is it possible to disable this scaling step and denoise an object image with the same size as the bounding box?\n- Does Noisy Patch transplantation guarantee the consistent content of the output image when pasting the generated image  with a binary mask (Eq. 9)? Again in Eq 9, the size of $x_{t-1}$ and $B_{i, t-1}$ are not the same as illustrated in Figure 3 so it is not clear how Hadamard product is possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review, acknowledging our work to be “well-written” and possess “superior efficacy to other methods”. Here, we address the concerns and questions that have been raised.\n\n**(1) Clarification on the Main Method**\n\nWe would like to draw your attention to our general response above, where we provide detailed clarifications regarding our problem definition and method. **Please note that the notations in the below sections are also based on the clarified versions.**\n\n**(2) Clarification on Lines 158-161 of Sec. 4.3**\n\nIt is indeed obvious that if two noisy images $\\mathbf{x}\\_t$ and $\\mathbf{y}\\_t$ are initialized with different noises, they will produce different outputs if denoised separately (lines 158-161). However, we reiterate this fact to emphasize that as we increase the number of shared sampling between these two noisy images, $\\mathbf{x}\\_t$ and $\\mathbf{y}\\_t$ increasingly produce semantically similar outputs, even when starting from two different initial noises (lines 170-171). This phenomenon, also visualized in Fig. 2-(A), (B) of the main paper, signifies the intriguing effect of semantic sharing when two noisy images pass through the DiT together.\n\n**(3) Comparison between Semantic Sharing and Image Interpolation**\n\nThank you for your comment on comparing our proposed method with the task of image interpolation. After reviewing the suggested paper [1], we would like to clarify the main purpose of our proposed shared sampling.\n\nUnlike image interpolation, which takes two (or more) input images and produces an output image with semantic features that lie between those of the input images, the goal of our shared sampling is to transfer the semantic features of one noisy image to the other. This process ensures that the desired concepts appear in the target noisy image.\n\n**(4) Clarification on Grounding Loss $\\mathcal{L}$**\n\nPlease note that we provide clarifications on the definition of the grounding loss in our general response. We will also include this clarification in the revised version.\n\n**(5) Clarification on Handling Multiple Objects of the Same Class**\n\nIn Stage 2 of our GrounDiT, please note that a separate object branch and a corresponding noisy object image $\\mathbf{z}\\_{i, t}$ is defined and processed in parallel for each grounding condition $g_i$. This ensures that each object is individually handled, therefore providing precise local guidance for each bounding box region $b_i$.\n\n**(6) Further Clarification on Shared Sampling Mechanism**\n\nIn Fig. 3 of the main paper, we intentionally depict the noisy object image $\\mathbf{z}\\_{i, t}$ as larger than the corresponding bounding box. This highlights that even when $\\mathbf{z}\\_{i, t}$ and the bounding box $b_i$ are not the same size, shared sampling between the two noisy images is still feasible. This is achieved by assigning positional embeddings so that each image is treated as a whole. This case is also visualized in Fig. 2-(B) of the main paper.\n\nMoreover, as describe in our general response, the noisy object image $\\mathbf{z}\\_{i,t}$ is set to satisfy the following criteria:\n\n* Size of $\\mathbf{z}\\_{i,t}\\in\\mathbb{R}^{H’_i \\times W’_i \\times D}$ is set within DiT’s preferred token sequence range.\n* $\\mathbf{z}\\_{i,t}$ has a similar aspect ratio as its corresponding bounding box $b_i$.\n\nTherefore, the noisy object image $\\mathbf{z}\\_{i, t}$ does not require a scaling process after it has been initialized. Simply passing $\\mathbf{z}\\_{i, t}$ and the nosy patch $\\mathbf{b}\\_{i,t}$ (originally $B_{i,t}$) together through the DiT for denoising is sufficient to perform shared sampling.\n\n**(7) Consistency of Images after Noisy Patch Transplantation**\n\nTo prevent potential inconsistencies between the bounding box regions and the background, our GrounDiT guidance is applied during the initial timesteps of the reverse process, as detailed in Appendix A.1 of our main paper.\n\nSpecifically, our reverse process consists of 50 steps, with GrounDiT denoising (Stage 1 and Stage 2) applied for the first 25 steps. For the final 25 steps, the noisy image $x_t$ is denoised using the standard DiT denoising step. This strategy, also employed in previous works such as R&B, leverages the fact that the image structure is primarily determined in the early steps. Therefore, applying our guidance in the early steps is sufficient for accurate grounding, while preventing inconsistencies between the bounding box regions and the background regions.\n\n**(8) Clarification on Hadamard Produce in Eq. (9)**\n\nThank you for the comment. We correct Eq. (9) below. We will incorporate the clarification in our revised version.\n\nLet $\\textbf{UNCROP}(\\cdot)$ be a function that zero-pads a patch $\\mathbf{b}_{i,t}$ at region $b_i$ to match the size of the original image. Then, the transplantation mechanism originally proposed in Eq. (9) can be clarified as follows:\n\n\\begin{align}\n& \\text{for}\\\\; i = 0, ..., N-1:\\\\\\\\\n& \\quad\\\\; \\tilde{\\mathbf{x}}\\_{t-1}\\leftarrow \\tilde{\\mathbf{x}}\\_{t-1}\\odot (1-M_i)+ \\textbf{UNCROP}(\\mathbf{b}\\_{i,t-1},b_i)\\odot M_i\\\\\\\\\n& \\mathbf{x}\\_{t-1}\\leftarrow \\tilde{\\mathbf{x}}\\_{t-1}\n\\end{align}\n\nPlease note that the binary mask $M_i$ is passed through the Hadamard product with $\\textbf{UNCROP}(\\mathbf{b}\\_{i,t-1},b_i)$, which has the same resolution as $M_i$.\n\n[1] Interpolating between Images with Diffusion Models, Wang et al., ICML 2024 Workshop", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Here we clarify our problem definition and method. For clarity, we have slightly modified the notations to improve the description of the entire framework and readability. We will continue to improve the presentation of our paper in the revision.

**[Notations]**

- Let $P$ be the input global text prompt.

* Grounding conditions are given as $G=[g_0,...,g_{N-1}]$. Each $g_i$ consists of a bounding box $b_i\in\mathbb{R}^4$ and a word $p_i$ indicating the desired object within $b_i$. **Note that $p_i$ must be one of the words in $P$**, that is, $p_i\in P$, since our method uses the cross-attention maps of DiT to compute the grounding loss $\mathcal{L}$ in Stage 1.

* Let $\mathbf{x}_t$ be a noisy image at timestep $t$. **Please note that DiT treats $\mathbf{x}_t$ as a sequence of tokens**, and applies positional embeddings at each timestep, as $\textbf{PE}(\mathbf{x}\_t)$.

**[Method Overview]**

Please note that our GrounDiT performs the reverse diffusion process **only once**, while conducting Stage 1 and 2 sequentially at every denoising step.

* **Stage 1** takes $\mathbf{x}_t$ and performs gradient descent to update $\mathbf{x}_t$ to $\hat{\mathbf{x}}_t$, which has not yet undergone a denoising step. $\hat{\mathbf{x}}_t$ is passed to Stage 2.

* **Stage 2** then denoises $\hat{\mathbf{x}}_t$ to obtain $\mathbf{x}\_{t-1}$, which moves to the next timestep of the reverse process.

**[Stage 1]**

We clarify Stage 1 as below:

* $\textbf{PE}(\mathbf{x}\_t)$ is passed into DiT, and in this process the cross-attention map $A_i$ for each word $p_i$ is extracted.

* With $A_i$, the grounding loss $\mathcal{L}(\mathbf{x}\_t,g_i,A_i)$ is computed. We use the loss functions in R&B [1], $\mathcal{L}=\mathcal{L}_r + \mathcal{L}_b$, where $\mathcal{L}_r$ (Eq. 11 in R&B) is a region-aware loss and $\mathcal{L}_b$ (Eq. 13 in R&B) is a region-aware loss.

* $\mathcal{L}\_{AGG}$ is computed by aggregating $\mathcal{L}$ for all conditions in $G$.

* Finally, we compute $\hat{\mathbf{x}}\_t \leftarrow \mathbf{x}\_t - \omega_t \nabla_{\mathbf{x}\_t} \mathcal{L}_{AGG}$.

While $\hat{\mathbf{x}}\_t$ is more likely to position each object of $p_i$ within the bounding box $b_i$, **it still struggles to accurately position every object in $b_i$.** Therefore, we introduce Stage 2, which focuses on a precise local update.

**[Stage 2]**

In Stage 2, we perform the denoising of $\hat{\mathbf{x}}\_t$ while injecting semantic features of object $p_i$ into the corresponding bounding box $b_i$.

For this, we define a single main branch as well as multiple object branches, each of which corresponds to a grounding condition $g_i\in G$, detailed below:

* **Main branch** is responsible for the denoising of main noisy image $\hat{\mathbf{x}}\_t$, **via the standard DiT denoising step** using the global prompt $P$:  $\tilde{\mathbf{x}}\_{t-1}\leftarrow \textbf{SampleDiT}(\textbf{PE}(\hat{\mathbf{x}}\_t), t, P)$. Note that $\tilde{\mathbf{x}}\_{t-1}$ is an intermediate denoised image, which is later combined with the outputs from object branches to finally obtain $\mathbf{x}\_{t-1}$.

* Each **object branch** is responsible for its corresponding grounding condition $g_i$.

   * Consider a local patch $\mathbf{b}\_{i,t}=\textbf{CROP}(\hat{\mathbf{x}}\_t, b_i)$. We want to ensure that the object $p_i$ appears within the bounding box $\mathbf{b}\_{i,t}$. 

   * One can consider treating $\mathbf{b}\_{i,t}$ as an image on its own and passing it through DiT with $p_i$ as the text condition. **However, please note that this is not feasible with a pretrained DiT model since it cannot properly handle arbitrary resolutions of images.** This is NOT due to the architecture but to the fact that the training images typically have a certain resolution (which we call a **preferred resolution**). Please refer to Fig. S2 of the attached PDF.

   * However, please note that **DiT’s Transformer is still flexible to the length of its token sequence and thus feasible to combine two sequences of tokens**, *provided* that the positional embedding for each token sequence is computed with one of the preferred resolutions. Please see Fig 2 (A) in our paper, where both image has a preferred resolution. 

   *  This leads us to our main idea: **shared sampling**. The noisy image $\mathbf{z}\_{i,t}$ at each branch has a preferred resolution (with a similar ratio to the bounding box $b_i$), while the patch $\mathbf{b}\_{i,t}$ cropped from the main branch does not. While we cannot directly denoise the patch $\mathbf{b}\_{i,t}$, we instead combine the two set of tokens from $\mathbf{z}\_{i,t}$ and $\mathbf{b}\_{i,t}$ and denoise them together while using $p_i$ as the text prompt. Our discovery is that this results in properly denoising the patch $\mathbf{b}\_{i,t}$ thanks to the interaction with the other sequence $\mathbf{z}\_{i,t}$, which has a preferred resolution, in the self-attention modules. Please see Fig 2 (B) in our paper. **Please note that $\mathbf{z}\_{i,t}$ in each object branch is NOT directly pasted onto the $\hat{\mathbf{x}}\_t$ in the main branch** but only used as an auxiliary to help properly denoise each patch in $\hat{\mathbf{x}}\_t$ with the corresponding object label $p_i$.

   * Specifically, we compute: $\\{ \mathbf{z}\_{i,t-1}, \mathbf{b}\_{i,t-1} \\}\leftarrow\textbf{SampleDiT}(\textbf{CONCAT}(\textbf{PE}(\mathbf{z}\_{i,t}),\textbf{PE}(\mathbf{b}\_{i,t})), t, p_i)$. Then, the denoised patch $\mathbf{b}\_{i,t-1}$ from $i$-th object branch is pasted back into its designated region in the main branch output $\tilde{\mathbf{x}}\_{t-1}$. The result of pasting back $\mathbf{b}\_{i,t-1}$ from all object branches into $\tilde{\mathbf{x}}\_{t-1}$ finally becomes $\mathbf{x}\_{t-1}$. The $\mathbf{x}\_{t-1}$ proceeds to the next timestep of the reverse process.

Meta-review: This paper received mixed reviews: one accept, one weak accept, one borderline accept, and one borderline reject. The primary concerns raised by the reviewers were the precision and clarity of the method’s presentation, as well as the need for further comparisons on additional benchmarks.

The rebuttal effectively addressed these issues with additional evaluations and evidence, demonstrating the method’s effectiveness compared to existing approaches. Although reviewer LMpP was unable to respond to the rebuttal, the AC concurs with all the other reviewers that the improvements made in the rebuttal and subsequent discussions significantly raise the paper’s acceptance prospects.

The AC also agrees and thinks this work to be a novel, clear, and inspiring solution for text-to-image models, free from any noticeable flaws, and recommends it for acceptance.

For the camera-ready version, it is encouraged to incorporate the detailed responses from the rebuttal and discussions. Additionally, as suggested by reviewer xDuK, it would be beneficial to include an ablation study on the direct pasting of object images from the object branch into the noisy images of the main branch. We look forward to seeing this presentation at NeurIPS.
Decision: Accept (poster)