Title: GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Text-to-Image Generation

Visual Grounding
Abstract: We introduce GrounDiT, a novel training-free spatial grounding technique for text-to-image generation using Diffusion Transformers (DiT). Spatial grounding with bounding boxes has gained attention for its simplicity and versatility, allowing for enhanced user control in image generation. However, prior training-free approaches often rely on updating the noisy image during the reverse diffusion process via backpropagation from custom loss functions, which frequently struggle to provide precise control over individual bounding boxes. In this work, we leverage the flexibility of the Transformer architecture, demonstrating that DiT can generate noisy patches corresponding to each bounding box, fully encoding the target object and allowing for fine-grained control over each region. Our approach builds on an intriguing property of DiT, which we refer to as semantic sharing. Due to semantic sharing, when a smaller patch is jointly denoised alongside a generatable-size image, the two become semantic clones. Each patch is denoised in its own branch of the generation process and then transplanted into the corresponding region of the original noisy image at each timestep, resulting in robust spatial grounding for each bounding box. In our experiments on the HRS and DrawBench benchmarks, we achieve state-of-the-art performance compared to previous training-free approaches. Project Page: https://groundit-diffusion.github.io/.

GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation

Abstract

We introduce GROUNDIT, a novel training-free spatial grounding technique for text-to-image generation using Diffusion Transformers (DiT). Spatial grounding with bounding boxes has gained attention for its simplicity and versatility, allowing for enhanced user control in image generation. However, prior training-free approaches often rely on updating the noisy image during the reverse diffusion process via backpropagation from custom loss functions, which frequently struggle to provide precise control over individual bounding boxes. In this work, we leverage the flexibility of the Transformer architecture, demonstrating that DiT can generate noisy patches corresponding to each bounding box, fully encoding the target object and allowing for fine-grained control over each region. Our approach builds on an intriguing property of DiT, which we refer to as semantic sharing . Due to semantic sharing, when a smaller patch is jointly denoised alongside a generatable-size image, the two become semantic clones . Each patch is denoised in its own branch of the generation process and then transplanted into the corresponding region of the original noisy image at each timestep, resulting in robust spatial grounding for each bounding box. In our experiments on the HRS and DrawBench benchmarks, we achieve state-of-the-art performance compared to previous training-free approaches. Project Page: https://groundit-diffusion.github.io/ .

1 Introduction

The Transformer architecture [45] has driven breakthroughs across a wide range of applications, with diffusion models emerging as significant recent beneficiaries. Despite the success of diffusion models with U-Net [42] as the denoising backbone [22, 43, 41, 39], recent Transformer-based diffusion models, namely Diffusion Transformers (DiT) [37], have marked another leap in performance. This is demonstrated by recent state-of-the-art generative models such as Stable Diffusion 3 [13] and Sora [6]. Open-source models like DiT [37] and its text-guided successor PixArtα [8] have also achieved superior quality compared to prior U-Net-based diffusion models. Given the scalability of Transformers, Diffusion Transformers are expected to become the new standard for image generation, especially when trained on an Internet-scale dataset.

With high quality image generation achieved, the next critical step is to enhance user controllability. Among the various types of user guidance in image generation, one of the most fundamental and significant is spatial grounding . For instance, a user may provide not only a text prompt describing the image but also a set of bounding boxes indicating the desired positions of each object, as shown in Fig. 1. Such spatial constraints can be integrated into text-to-image (T2I) diffusion models by adding extra modules that are designed for spatial grounding and fine-tuning the model. GLIGEN [31] is a notable example, which incorporates a gated self-attention module [1] into the U-Net layers of Stable Diffusion [41]. Although effective, such fine-tuning-based approaches incur substantial training costs each time a new T2I model is introduced.

Recent training-free approaches for spatially grounded image generation [9, 47, 11, 36, 38, 48, 12, 26] have led to new advances, removing the high costs for fine-tuning. These methods leverage the fact that cross-attention maps in T2I diffusion models convey rich structural information about where each concept from the text prompt is being generated in the image [7, 19]. Building on this, these approaches aim to align the cross-attention maps of specific objects with the given spatial constraints ( e . g . bounding boxes), ensuring that the objects are placed within their designated regions. This alignment is typically achieved by updating the noisy image in the reverse diffusion process using backpropagation from custom loss functions. However, such loss-guided update methods often struggle to provide precise spatial control over individual bounding boxes, leading to missing objects (Fig. 4, Row 9, Col. 5) or discrepancies between objects and their bounding boxes (Fig. 4, Row 4, Col. 4). This highlights the need for finer control over each bounding box during image generation.

We aim to provide more precise spatial control over each bounding box, addressing the limitations in previous loss-guided update approaches. A well-known technique for manipulating local regions of the noisy image during the reverse diffusion process is to directly replace or merge the pixels (or latents) in those regions. This simple but effective approach has proven effective in various tasks, including compositional generation [17, 50, 44, 32] and high-resolution generation [5, 29, 24, 25]. One could consider defining an additional branch for each bounding box, denoising with the corresponding text prompt, and then copying the noisy image into its designated area in the main image at each timestep. However, a key challenge lies in creating a noisy image patch-at the same noise level-that reliably contains the desired object while fitting within the specified bounding box. This has been impractical with existing T2I diffusion models, as they are trained on a limited set of image resolutions. While recent models such as PixArtα [8] support a wider range of image resolutions, they remain constrained by specific candidate sizes, particularly for smaller image patches. As a result, when these models are used to create a local image patch, they are often limited to denoising a fixed-size image and cropping the region to fit the bounding box. This approach can critically fail to include the desired object within the cropped region.

In this work, we show that by exploiting the flexibility of the Transformer architecture, DiT can generate noisy image patches that fit the size of each bounding box, thereby reliably including each desired object. This is made possible through our proposed joint denoising technique. First, we introduce an intriguing property of DiT: when a smaller noisy patch is jointly denoised with a generatable-size noisy image, the two gradually become semantic clones-a phenomenon we call semantic sharing . Next, building on this observation, we propose a training-free framework that involves cultivating a noisy patch for each bounding box in a separate branch and then transplanting that patch into its corresponding region in the original noisy image. By iteratively transplanting the separately denoised patches into their respective bounding boxes, we achieved fine-grained spatial control over each bounding box region. This approach leads to more robust spatial grounding, particularly in cases where previous methods fail to accurately adhere to spatial constraints.

In our experiments on the HRS [3] and DrawBench [43] datasets, we evaluate our framework, GROUNDIT, using PixArtα [8] as the base text-to-image DiT model. Our approach demonstrates superior performance in spatial grounding compared to previous training-free methods [38, 9, 47, 48], especially outperforming the state-of-the-art approach [47], highlighting its effectiveness in providing fine-grained spatial control.

2 Related Work

In this section, we review the two primary approaches for incorporating spatial controls into textto-image (T2I) diffusion models: fine-tuning-based methods (Sec. 2.1) and training-free guidance techniques (Sec. 2.2).

2.1 Spatial Grounding via Fine-Tuning

Fine-tuning with additional modules is a powerful approach for enhancing T2I models with spatial grounding capabilities [51, 31, 2, 53, 46, 16, 10, 54]. SpaText [2] introduces a spatio-textual representation that combines segmentations and CLIP embeddings [40]. ControlNet [51] incorporates a trainable U-Net encoder that processes spatial conditions such as depth maps, sketches, and human keypoints, guiding image generation within the main U-Net branch. GLIGEN [31] enables T2I models to accept bounding boxes by inserting a gated attention module into Stable Diffusion [41]. GLIGEN's strong spatial accuracy has led to its integration into follow-up spatial grounding methods [48, 38, 30] and applications such as compositional generation [15] and video editing [23]. InstanceDiffusion [46] further incorporates conditioning modules to provide finer spatial control through diverse conditions like boxes, scribbles, and points. While these fine-tuning methods are effective, they require taskspecific datasets and involve substantial costs, as they must be retrained for each new T2I model, underscoring the need for training-free alternatives.

2.2 Spatial Grounding via Training-Free Guidance

In response to the inefficiencies of fine-tuning, training-free approaches have been introduced to incorporate spatial grounding into T2I diffusion models. One approach involves a region-wise composition of noisy patches, each conditioned on a different text input [5, 50, 32]. These patches, extracted using binary masks, are intended to generate the object they are conditioned on within the generated image. However, since existing T2I diffusion models are limited to a fixed set of image resolutions, each patch cannot be treated as a complete image, making it uncertain whether the extracted patch will contain the desired object. Another approach leverages the distinct roles of attention modules in T2I models-self-attention captures long-range interactions between image features, while cross-attention links image features with text embeddings. By using spatial constraints such as bounding boxes or segmentation masks, spatial grounding can be achieved either by updating the noisy image using backpropagation based on a loss calculated from cross-attention maps [48, 38, 9, 7, 18, 36], or by directly manipulating cross- or self-attention maps to follow the given spatial layouts [26, 4, 14]. While the loss-guided methods enable spatial grounding in a training-free manner, they still lack precise control over individual bounding boxes, often leading to missing objects or misalignmet between objects and their bounding boxes. In this work, we propose a novel training-free framework that offers fine-grained spatial control over each bounding box by harnessing the flexibility of the Transformer architecture in DiT.

3 Background: Diffusion Transformers

Diffusion Transformer (DiT) [37] represents a new class of diffusion models that utilize the Transformer architecture [45] for their denoising network. Previous diffusion models like Stable Diffusion [41] use the U-Net [42] architecture, of which each layer contains a convolutional block and attention modules. In contrast, DiT consists of a sequence of DiT blocks, each containing a pointwise feedforward network and attention modules, removing convolution operations and instead processing image tokens directly through attention mechanisms.

CROP

Global Update

Local Update

Grounding Conditions

𝐺 = { 𝑔𝑖  }

𝑖=0

𝑁-1

𝑏𝑖

: Bounding box

𝑝𝑖

: Object

𝑐𝑖

: Text embedding

𝑔𝑖

Global

Update

DiT

Joint

Denoising

Main Branch

𝒊

-th Object Branch

DiT

PE(

⋅

)

Local Patch

v𝑖,𝑡

ො

𝐱𝑡

𝐱𝑡

v𝑖,𝑡-1

u𝑖,𝑡-1

Object Image

u𝑖,𝑡

𝐱𝑡-1

…

…

𝑐𝑃

𝑐𝑖

ExtractAttention()

𝐱𝑡

𝐺

𝑐𝑃

{𝐀𝒊,𝒕

}

𝐺 𝑐𝑃

ℒAGG

{𝐀𝒊,𝒕

}, 𝐺

1

2

3

4

(

⋅, 𝑏

𝑖

)

DiT follows the formulation of diffusion models [22], in which the forward process applies noise to a real clean data x 0 by

The reverse process denoises the noisy data x t through a Gaussian transition

where µ θ ( x t , t ) is calculated by a learned neural network trained by minimizing the negative ELBO objective [27]. While Σ θ ( x t , t ) can also be learned, it is usually set as time dependent constants.

Positional Embeddings. As DiT is based on the Transformer architecture, it treats the noisy image x t ∈ R h × w × d as a set of image tokens. Specifically, x t is divided into patches, each transformed into an image token via linear embedding. This results in ( h/l ) × ( w/l ) tokens, where l is the patch size. Importantly, before each denoising step, 2D sine-cosine positional embeddings are assigned to each image token to provide spatial information as follows:

Here, PE ( · ) applies positional embeddings, Denoise ( · ) represents a single denoising step in DiT at timestep t , and c is the text embedding. This contrasts with U-Net-based diffusion models, which typically do not utilize positional embeddings for the noisy image. Detailed formulations of the positional embeddings are provided in the Appendix (Sec. A) .

4 Problem Definition

Let P be the input text prompt ( i . e ., a list of words), which we refer to as the global prompt . Let c P be the text embedding of P . We define a set of N grounding conditions G = { g i } N -1 i =0 , where each condition specifies the coordinates of a bounding box and the desired object to be placed within it. Specifically, each condition g i := ( b i , p i , c i ) consists of the following: b i ∈ R 4 , the xy -coordinates of the bounding box's upper-left and lower-right corners, p i ∈ P , the word in the global prompt describing the desired object within the box, and c i , the text embedding of p i . The objective is to generate an image that aligns with the global prompt P while ensuring each specified object is accurately positioned within its corresponding bounding box b i .

5 GROUNDIT: Grounding Diffusion Transformers

We propose GROUNDIT, a training-free framework based on DiT for generating images spatially grounded on bounding boxes. Each denoising step in GROUNDIT consists of two stages: Global Update and Local Update. Global Update ensures coarse alignment between the noisy image and the bounding boxes through a gradient descent update using cross-attention maps (Sec. 5.1). Following this, Local Update further provides fine-grained control over individual bounding boxes via a novel noisy patch transplantation technique (Sec. 5.3). This approach leverages our key observation of DiT's semantic sharing property, introduced in Sec. 5.2. An overview of this two-stage denoising step is provided in Fig. 2.

5.1 Global Update with Cross-Attention Maps

First, the noisy image x t is updated to spatially align with the bounding box inputs. For this, we leverage the rich structural information encoded in cross-attention maps, as first demonstrated by Chefer et al. [7]. Each cross-attention map shows how a region of the noisy image correspond to a specific word in the global prompt P . Let DiT consist of M sequential DiT blocks. As x t passes through the m -th block, the cross-attention map a m i,t ∈ R h × w × 1 for object p i is extracted. For each grounding condition g i , the mean cross-attention map A i,t is obtained by averaging a m i,t over all M blocks as follows:

For convenience, we denote the operation of extracting A i,t for every g i ∈ G as below:

Then, following prior works on U-Net-based diffusion models [48, 47, 38, 9, 36, 7], we measure the spatial alignment for each object p i by comparing its mean cross-attention map A i,t with its corresponding bounding box b i , using a predefined grounding loss L ( A i,t , b i ) as defined in R&B [47]. The aggregated grounding loss L AGG is then computed by summing the grounding loss across all grounding conditions g i ∈ G :

Based on the backpropagation from L AGG, x t is updated via gradient descent as follows:

where ω t is a scalar weight value for gradient descent. We refer to Eq. 7 as the Global Update , as the whole noisy image x t is updated based on an aggregated loss from all grounding conditions in G .

The Global Update achieves reasonable accuracy in spatial grounding. However, it often struggles with more complex grounding conditions. For instance, when G contains multiple bounding boxes ( e . g ., six boxes in Fig. 4, Row 9) or small, thin boxes ( e . g ., Fig. 4, Row 5), the desired objects may be missing or misaligned with the boxes. These examples show that the Global Update lacks fine-grained, box-specific control, underscoring the need for precise controls for individual bounding boxes. In the following sections, we introduce a novel method to achieve this fine-grained spatial control.

5.2 Semantic Sharing in Diffusion Transformers

In this section, we present our observations on an intriguing property of DiT, semantic sharing , which will serve as a key building block for our main method in Sec. 5.3.

Joint Denoising. We observed that DiT can jointly denoise two different noisy images together. For example, consider two noisy images, x t and y t , both at timestep t in the reverse diffusion process. Position embeddings are applied to the image tokens according to their sizes, resulting in ¯ x t = PE ( x t ) and ¯ y t = PE ( y t ) . Notably, the two noisy images can differ in size, providing flexibility in joint denoising. The key part is that the two noisy images-more precisely, two sets of image tokens-are merged into a single set z t . We denote this process as Merge ( · ) (see Alg. 1, line 4). z t is passed through the DiT blocks, yielding the output z t -1 , which is then split into the denoised versions x t -1 and y t -1 via Split ( · ) . Joint denoising is illustrated in Fig. 3-(A), and a pseudocode for a single step of joint denoising is shown in Alg. 1.

Semantic Sharing. Surprisingly, we found that joint denoising of two noisy images generates semantically correlated content in correponding pixels, even when the initial random noise differs. Consider two noisy images, x T ∈ R h x × w x × d and y T ∈ R h y × w y × d , both initialized from a unit Gaussian distribution N (0 , I ) . We experiment with a reverse diffusion process in which, for the initial 100 γ %of the denoising steps ( γ ∈ [0 , 1] ), x T and y T undergo joint denoising. For the remaining

(B) Semantic Sharing between Equal Resolutions

(A) Joint Denoising

PE

PE

…

…

Merge()

…

…

DiT

'dog'

…

…

Split()

(C) Semantic Sharing between Different Resolutions

𝛾 = 0

𝛾 = 1

𝛾 = 0

𝛾 = 1

yo

yo

yt

xt

yt

Zt

Zt-1

Xt—1

yt-1

timesteps, they are denoised independently. The same text embedding c is used as a condition in both cases.

Fig. 3 shows the generated images from x T and y T across different γ values. In Fig. 3-(B), x T and y T have the same resolution ( h x = h y , w x = w y ), while in Fig. 3-(C) their resolutions differ ( h x > h y , w x > w y ). When γ = 0 , the two noisy images are denoised completely independently, resulting in clearly distinct images (leftmost column). We found that DiT models have a certain range of resolutions within which they can generate plausible images-which we refer to as generatable resolutions -but face challenges when generating images far outside this range. This is demonstrated in the output of y 0 in Fig. 3-(C) with γ = 0 . Further discussions and visual analyses are provided in the Appendix (Sec. D) . But as γ increases, allowing x T and y T to be jointly denoised in the initial steps, the generated images become progressively more similar. When γ = 1 , the images generated from x T and y T appear almost identical. These results demonstrate that, in joint denoising, assigning identical or similar positional embeddings to different image tokens promotes strong interactions between them during the denoising process. This correlated behavior during joint denoising causes the two image tokens to converge toward semantically similar outputs-a phenomenon we term semantic sharing .

Notably, this pattern holds not only when both noisy images share the same resolution (Fig. 3-(B)), but even when one of the images does not have DiT's generatable resolution (Fig. 3-(C)). While self-attention sharing techniques have been explored in U-Net-based diffusion models to enhance style consistency between images [20, 34], they have been limited to images of equal resolution. By leveraging the flexibility to assign positional embeddings across different resolutions, our joint

Algorithm 1: Pseudocode of Joint Denoising (Sec. 5.2).

denoising approach extends across heterogeneous resolutions, offering greater versatility. We provide further discussions and analyses on semantic sharing in the Appendix (Sec. D) .

5.3 Local Update with Noisy Patch Transplantation

In this section, we introduce our key technique: Local Update via noisy patch cultivation and transplantation. Building on DiT's semantic sharing property from Sec. 5.2, we show how this can be leveraged to provide precise spatial control over each bounding box.

Main & Object Branches. We propose a parallel denoising approach with multiple branches: one for the main noisy image ˆ x t and additional branches for each grounding condition g i . The main branch denoises the main noisy image using the global prompt P , while each object branch is designed to denoise local regions within the bounding boxes, enabling fine-grained spatial control over each region. For each i -th object branch, there is a distinct noisy object image u i,t , initialized as u i,T ∼ N (0 , I ) . We predefine the resolution of the noisy object image u i,t by searching in PixArtα 's generatble resolutions that closely match the aspect ratio of the corresponding bounding box b i . With ˆ x t obtained from Global Update (Sec. 5.1), each branch performs denoising in parallel. Below we explain the denoising mechanism for each branch.

Noisy Patch Cultivation. In the main branch at timestep t , the noisy image ˆ x t is denoised using the global prompt P as follows: ˜ x t -1 ← Denoise ( PE (ˆ x t ) , t, c P ) , where ˆ x t is the output from the Global Update and c P is the text embedding of P . For the i -th object branch, there are two inputs: the noisy object image u i,t and a subset v i,t of image tokens extracted from ˆ x t , corresponding to the bounding box b i . We denote this extraction as v i,t ← Crop (ˆ x t , b i ) , where v i,t ∈ R h i × w i × d is referred to as a noisy local patch . Here, h i and w i corresponds to height and width of bounding box b i , respectively. Joint denoising is then performed on u i,t and v i,t to yield their denoised versions:

where c i is the text embedding of the object p i .

Through semantic sharing with the noisy object image u i,t during joint denoising, the denoised local patch v i,t -1 is expected to gain richer semantic features of object p i than it would without joint denoising. Note that even when the noisy local patch v i,t does not meet the typical generatable resolution of DiT (since it often requires cropping small bounding box regions of ˆ x t to obtain v i,t ), it offers a simple and effective way for enriching v i,t of the semantic features of object p i . We refer to this process as noisy patch cultivation .

Noisy Patch Transplantation. After cultivating local patches through joint denoising in Eq. 8, each patch is transplanted into ˜ x t -1 , obtained from the main branch. The patches are transplanted in their original bounding box regions specified by b i as follows:

Here, ⊙ denotes the Hadamard product, m i is a binary mask for the bounding box b i , and Uncrop ( v i,t -1 , b i ) applies zero-padding to v i,t -1 to align its position with that of b i . This transplantation enables fine-grained local control for the grounding condition g i . After transplanting the outputs from all N object branches, we obtain x t -1 , representing the final output of GROUNDIT denoising step at timestep t . In x t -1 , the image tokens within the b i region are expected to possess richer semantic information about object p i compared to the initial ˜ x t -1 from the main branch. This process is referred to as noisy patch transplantation . We provide implementation details and full pseudocode of a single GROUNDIT denoising step in the Appendix (Sec. E).

6 Results

In this section, we present the experiment results of our method, GROUNDIT, and provide comparisons with baselines. For the base text-to-image DiT model, we use PixArtα [8], which builds on the original DiT architecture [37] by incorporating an additional cross-attention module to condition on text prompts.

Layout

Layout-

Guidance [9]

Attention-

Refocusing [38]

BoxDiff [48]

R&B [47]

PixArt-R&B

GROUNDIT

'A dog in the beautiful park. '

'An eagle is flying over a tree. '

'A duck wearing a hat standing near a bicycle. '

'A plastic bottle and an apple on a table. '

'An apple and a banana and a cup on a table. '

'"A dog wearing sunglasses and a red hat and a blue tie. '

'A chair and a table and a bed is on the room with a photo frame on the wall and a ceiling lamp [...]'

西

'A car and a dog on the road while horse and a chair is on the grass. '

6.1 Evaluation Settings

Baselines. We compare our method with state-of-the-art training-free approaches for bounding box-based image generation, including R&B [47], BoxDiff [48], Attention-Refocusing [38], and Layout-Guidance [14]. For a fair comparison, we also implement R&B using PixArtα , which we refer to as PixArt-R&B , and treat it as an internal baseline. Note that this is identical to our method without the Local Guidance (Sec. 5.3).

Evaluation Metrics and Benchmarks.

(Grounding Accuracy) We follow the evaluation protocol of R&B [47] to assess spatial grounding on the HRS [3] and DrawBench [43] datasets, using three criteria: spatial, size, and color. The HRS dataset consists of 1002, 501, and 501 images for each respective criterion, with bounding boxes generated using GPT-4 by Phung et al. [38]. For DrawBench, we use the same 20 positional prompts as in R&B [47].

(Prompt Fidelity) We use the CLIP score [21] to evaluate how well the generated images adhere to the text prompt. Additionally, we assess our method using PickScore [28] and ImageReward [49], which provide human alignment scores based on the consistency between the text prompt and generated images.

Table 1: Quantitative comparisons of grounding accuracy on HRS [3] and DrawBench [43] benchmarks. Bold represents the best, and underline represents the second best method.

6.2 Grounding Accuracy

Quantitative Comparisons. Tab. 1 presents a quantitative comparison of grounding accuracy between our method, GROUNDIT, and baselines. GROUNDIT outperforms all baselines across different criteria of grounding accuracy-spatial, size, and color-including the state-of-the-art R&B [47] and our internal baseline PixArt-R&B. Notably, the spatial accuracy on the HRS benchmark [3] (Col. 1) of GROUNDIT is significantly higher, with a +14.87% improvement over R&B and +7.88% over PixArtα . The comparison between PixArtα [8], PixArt-R&B and GROUNDIT highlights the effectiveness of the two-stage pipeline of GROUNDIT. First, integrating the loss-based Global Update into PixArtα results in a substantial improvement in spatial accuracy (from 17.86% to 37.13%). Then, incorporating our key contribution, the Local Update, further boosts accuracy (from 37.13% to 45.01%). For size accuracy (Col. 2), which evaluates how well the size of each generated object matches its corresponding bounding box, GROUNDIT shows a +1.01% improvement over R&B. In terms of color accuracy (Col. 3), our method achieves a +6.60% improvement over PixArt-R&B and outperforms R&B by +3.63%. This underscores the effectiveness of our noisy patch transplantation technique in accurately assigning color descriptions to the corresponding objects. As DrawBench [43] only contains images with two bounding boxes, which are relatively easy to generate, employing the Global Update is sufficient for grounding. We present additional quantitative comparisons of grounding accuracy in the Appendix (Sec. B) .

Qualitative Comparisons. Fig. 4 presents the qualitative comparisons. When the grounding condition involves one or two simple bounding boxes (Rows 1, 2), both our method and the baselines

successfully generate objects within the designated regions. However, as the number of bounding boxes increases and the grounding conditions become more challenging, the baselines struggle to correctly place each object inside the bounding box (Rows 4, 8), or even fail to generate the object at all (Rows 5, 7, 9). In contrast, GROUNDIT successfully grounds each object within the boxes, even when the number of boxes is relatively high, such as four boxes (Rows 5, 6, 8), five boxes (Row 7) and six boxes (Row 9). This highlights that our proposed noisy patch transplantation technique provides superior control over each bounding box, addressing the limitations of previous loss-based update methods, as discussed in Sec. 5.1. For more qualitative comparisons, including images generated with various aspect ratios, please refer to the Appendix (Sec. G and Fig. 5) .

6.3 Prompt Fidelity

Tab. 2 presents a quantitative comparison of prompt fidelity between our method and PixArt-R&B. Each metric is measured using the generated images from the HRS dataset [3]. GROUNDIT achieves higher CLIP score [21] than PixArt-R&B (Col. 1), indicating that our noisy patch transplantation improves the text prompt fidelity of the generated images. Additionally, our method achieves a higer ImageReward [49] score, which measures human preference by considering both prompt fidelity and overall image quality. While GROUNDIT shows a slight underperformance compared to PixArt-R&B in Pickscore [28], it remains comparable overall. We provide further comparisons of prompt fidelity with other baselines in the Appendix (Sec. C).

Table 2: Quantitative comparisons on prompt fidelity on HRS benchmark [3]. Bold represents the best method.

7 Conclusion

In this work, we introduced GROUNDIT, a training-free spatial grounding technique for text-to-image generation, leveraging Diffusion Transformers (DiT). To address the limitation of prior approaches, which lacked fine-grained spatial control over individual bounding boxes, we proposed a novel approach that transplants a noisy patch generated in a separate denoising branch into the designated area of the noisy image. By exploiting an intriguing property of DiT, semantic sharing, which arises from the flexibility of the Transformer architecture and the use of positional embeddings, GROUNDIT generates a smaller patch by simultaneously denoising two noisy image: one with a smaller size and the other with a generatable resolution by DiT. Through semantic sharing, these two noisy images become semantic clones, enabling fine-grained spatial control for each bounding box. Our experiments on the HRS and DrawBench benchmarks demonstrated that GROUNDIT achieves state-of-the-art performance compared to previous training-free grounding methods.

Limitations and Societal Impacts. A limitation of our method is the increased computation time, as it requires a separate object branch for each bounding box. We provide further analysis on the computation time in the Appendix (Sec. F) . Additionally, like other generative AI techniques, our method is susceptible to misuse, such as creating deepfakes, which can raise significant concerns related to privacy, bias, and fairness. It is crucial to develop safeguards to control and mitigate these risks responsibly.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022.

Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In CVPR , 2023.

Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In ICCV , 2023.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusion paths for controlled image generation. In ICML , 2023.

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. https://openai.com/research/ video-generation-models-as-world-simulators , 2024.

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attentionbased semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics , 2023.

Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixartα : Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR , 2024.

Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In WACV , 2024.

Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdiffuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv:2302.08908 , 2023.

Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. arXiv preprint arXiv:2403.13589 , 2023.

Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. In NeurIPS , 2023.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 , 2024.

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In ICLR , 2023.

Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305.15393 , 2023.

Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV , 2022.

Songwei Ge, Taesung Park, Jun-Yan Zhu, and Jia-Bin Huang. Expressive text-to-image generation with rich text. In ICCV , 2023.

Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. In CVPR , 2024.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-toprompt image editing with cross attention control. In ICLR , 2023.

Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. In CVPR , 2024.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP , 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Hyeonho Jeong and Jong Chul Ye. Ground-a-video: Zero-shot grounded video editing using text-to-image diffusion models. In ICLR , 2024.

Gwanghyun Kim, Hayeon Kim, Hoigi Seo, Dong Un Kang, and Se Young Chun. Beyondscene: Higherresolution human-centric scene generation with pretrained diffusion. arXiv preprint arXiv:2404.04544 , 2024.

Jaihoon Kim, Juil Koo, Kyeongmin Yeo, and Minhyuk Sung. Synctweedies: A general generative framework based on synchronized diffusions. arXiv preprint arXiv:2403.14370 , 2024.

Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV , 2023.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR , 2014.

Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In NeurIPS , 2023.

Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. SyncDiffusion: Coherent montage via synchronized joint diffusions. In NeurIPS , 2023.

Yuseung Lee and Minhyuk Sung. Reground: Improving textual and spatial grounding at no cost. arXiv preprint arXiv:2403.13589 , 2024.

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR , 2023.

Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. TMLR , 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.

Yuxin Liu, Minshan Xie, Hanyuan Liu, and Tien-Tsin Wong. Text-guided texturing by synchronized multi-view diffusion. arXiv preprint arXiv:2311.12891 , 2023.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS , 2022.

Wan-Duo Kurt Ma, Avisek Lahiri, JP Lewis, Thomas Leung, and W Bastiaan Kleijn. Directed diffusion: Direct control of object placement through attention guidance. In AAAI , 2024.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV , 2023.

Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:2306.05427 , 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference , 2015.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022.

Takahiro Shirakawa and Seiichi Uchida. Noisecollage: A layout-aware text-to-image diffusion model based on noise cropping and merging. In CVPR , 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. In CVPR , 2024.

Jiayu Xiao, Liang Li, Henglei Lv, Shuhui Wang, and Qingming Huang. R&b: Region and boundary aware zero-shot grounded text-to-image generation. In ICLR , 2024.

Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV , 2023.

Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In NeurIPS , 2023.

Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. In ICML , 2024.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , 2023.

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In CVPR , 2023.

Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Migc: Multi-instance generation controller for text-to-image synthesis. In CVPR , 2024.

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In CVPR , 2022.

Appendix

"A high-definition space photo

withgalaxy,sun,

spaceship,

and

Earth'

”

"A high- definition ocean photo with submarine, jellyfish, turtleand coral"

"A photo of

shelves with

books,

bananas,

andvase

A Positional Embeddings in Diffusion Transformers

Diffusion Transformers (DiT) [37] handle noisy images of varying aspect ratios and resolutions by processing them as a set of image tokens. For this, the noisy image is first divided into patches, with each patch subsequently converted into an image token of hidden dimension D through a linear embedding layer. DiT then applies 2D sine-cosine positional embeddings to each image token, based on its coordinates ( x, y ) , defined as follows:

where w d = 1 / 10000 (4 d/D ) . The positional embedding p x,y is then added to each corresponding image token, denoted as PE ( · ) .

B Additional Quantitative Comparisons: Grounding Accuracy

In addition to Sec. 6.2, we provide further quantitative comparisons of grounding accuracy between our GROUNDIT and the baselines. Specifically, we generated images based on text prompts and bounding boxes using each method, then calculated the mean Intersection over Union (mIoU) between the detected bounding boxes from an object detection model [55] and the input bounding boxes. Below, we present the quantitative comparisons across three datasets with varying average numbers of bounding boxes: subset of MS-COCO-2014 [33], HRS-Spatial [3], and a custom dataset.

Subset of MS-COCO-2014. We filtered the validation set of MS-COCO-2014 [33] to exclude image-caption pairs where the target objects were either not mentioned in the captions or duplicate objects were present. From this filtered set, we randomly selected 500 pairs for evaluation.

The results are presented in Tab. 4, column 2. GROUNDIT outperforms R&B by 0.021 (a 5.1% improvement) and PixArt-R&B by 0.014 (a 2.2% improvement). The relatively small margin can be attributed to the simplicity of the task, as this dataset has an average of 2.06 bounding boxes (Tab. 3), making it less challenging even for the baseline methods.

HRS-Spatial. Column 3 of Tab. 4 presents the results on the Spatial subset of the HRS dataset [3]. GROUNDIT surpasses R&B [47] by 0.046 (a 14.1% improvement) and PixArt-R&B by 0.038 (an 11.4% improvement). Compared to the results on the MS-COCO-2014 subset, the higher percentage increase in mIoU highlights the robustness of GROUNDIT under more complex grounding conditions. Note that HRS-Spatial has an average of 3.11 bounding boxes (Tab. 3), which is higher than that of the MS-COCO-2014 subset (2.06).

Custom Dataset. The custom dataset consists of 500 layout-text pairs, generated using the layout generation pipeline from LayoutGPT [15]. As shown in column 4 of Tab. 4, GROUNDIT outperforms R&B by 0.052 (a 26.3% improvement) and PixArt-R&B by 0.044 (a 21.4% improvement). This dataset has the highest average number of bounding boxes at 4.48 (Tab. 3). These results further emphasize the robustness and effectiveness of our approach in handling more complex grounding conditions with a larger number of bounding boxes.

Table 4: Quantitative comparisons of mIoU ( ↑ ) on a subset of MS-COCO-2014 [33], HRS-Spatial [3], and our custom dataset. Bold represents the best, and underline represents the second best method.

C Additional Quantitative Comparisons: Prompt Fidelity

In addition to Sec. 6.3, we provide further quantitative comparisons of the prompt fidelity of the generated images between our GROUNDIT and the baselines. We evaluated the generated images from the HRS dataset [3] using three different metrics: CLIP score [21], ImageReward [49], and PickScore [28]. The results are presented in Tab. 5. Since PickScore evaluates preferences between a pair of images, we report the difference between our GROUNDIT and each baseline method in column 4. Our GROUNDIT consistently outperforms the baselines in both CLIP score and ImageReward. For PickScore, GROUNDIT outperforms all baselines except PixArt-R&B, while remaining comparable.

Table 5: Quantitative comparisons of prompt fidelity on the HRS dataset [3]. Bold represents the best method.

D Additional Analysis on Semantic Sharing

In this section, we provide further analyses on the generatable resolution and the semantic sharing property of DiT, initially introduced in Sec. 5.2.

Generatable Resolution of DiT. Although recent DiT models can generate images at various resolutions, they still struggle to produce images at completely arbitrary resolutions. We speculate that this limitation arises not from the model architecture itself, but from the resolution of the training images, which typically falls within a specific range [8]. Generating images at resolutions far outside this range often results in implausible outputs, suggesting the existence of an acceptable resolution range for DiT, which we refer to as its generatable resolution . In Fig. 6, we illustrate this phenomenon. When the noisy image size falls within DiT's generatable resolution range, the model produces plausible images (rightmost two images). However, when the image size is significantly outside this range (leftmost two images), DiT fails to generate a plausible image.

Image Size: [128x128]

[256×256]

[288×288]

[384×384]

[512x512]

Semantic Sharing. Even though DiT models have a limited range of generatable resolutions, their Transformer architecture offers flexibility in handling varying lengths of image tokens, making it feasible to merge two sets of image tokens and denoise them through a single network evaluation. Leveraging this flexibility of Transformers, we presented our joint denoising technique (Alg. 1). Our main observation was that the joint denoising between two noisy images causes the two generated images to become semantically correlated, as illustrated in Fig. 3-(B) and Fig. 3-(C).

In addition to the visualizations in Fig. 3, we further quantify the semantic sharing property by measuring the LPIPS score [52] between two generated images. To explore the effect of joint denoising, we varied the parameter γ ∈ [0 , 1] , which controls the proportion of denoising steps where joint denoising is applied. Specifically, γ = 0 means no joint denoising is applied, and each image is denoised independently, while γ = 1 means full joint denoising across all steps. As shown in Fig. 7, increasing γ ( i . e ., applying more joint denoising steps) results in a decrease in the LPIPS score between the two generated images, indicating that the images become more semantically similar as joint denoising is applied for a larger portion of the denoising process.

0.65

0.60

0.55

→

0.50

S

P

0.45

5

0.40

0.35

0.30

0.25

0

10

20

30

40

50

(y=0)

Joint Denoising steps

(y =1)

(a) Semantic sharing between equal resolutions

0.9

0.8

~0.7

→

LPIPS

S

0.6

0.5

0.4

0

10

20

30

40

50

(y=0)

Joint Denoising steps

(y =1)

(b) Semantic sharing between different resolutions

E Implementation Details

As the base text-to-image DiT model, we used the 512-resolution version of PixArtα [8]. For sampling we employed the DPM-Solver scheduler [35] with 50 steps. Out of the 50 denoising steps, we applied our GROUNDIT denoising step (Alg. 2) for the initial 25 steps, and applied the vanilla denoising step for the remaining 25 steps. For the grounding loss in Global Update of GROUNDIT, we adopted the definition proposed in R&B [47], and we set the loss scale to 10 and used a gradient descent weight of 5 for the gradient descent update in Eq. 7.

As discussed in Sec. 5.3, for each i -th object branch we have a noisy object image u i,t and a noisy local patch v i,t , which is extracted from the noisy image ˆ x t in main branch via v i,t ← Crop (ˆ x t , b i ) . We determine the resolution of the noisy object image u i,t by selecting from PixArtα 's generatable resolutions, choosing one that best aligns with the aspect ratio of the corresponding bounding box b i .

All our experiments were conducted an NVIDIA RTX 3090 GPU. In Algorithm 2, we provide the pseudocode of GROUNDIT single denoising step.

F Analysis on Computation Time

We present the average inference time based on the number of bounding boxes in Tab. 6. While our method shows a slight increase in inference time, the rate of increase remains modest. For three bounding boxes, the inference time is 1.01 times that of R&B and 1.33 times that of PixArt-R&B. Even with six bounding boxes, the inference time is only 1.41 times that of R&B and 1.90 times that of PixArt-R&B.

G Additional Qualitative Results

We provide more qualitative comparisons in Fig. 8. Our method demonstrates greater robustness against issues such as the missing object problem, attribute leakage, or object interruption problem [47], due to its local update mechanism with semantic sharing. For instance, in Row 1, baseline methods struggle to generate certain objects ( i . e . missing object problem ). In Row 2, baselines generate a banana that retains features of an apple, illustrating attribute leakage . In Row 3, R&B generates a bus that interrupts the generation of a couch, with part of the bus overlapping with the designate region of the couch. Similarly, in PixArt-R&B, a hamburger and a donut interrupt the

Layout

R&B [47]

PixArt-R&B

GROUNDIT

generation of a surfboard, demonstrating the object interruption problem . In more challenging cases, like Row 4, combinations of these issues appear. By contrast, our method consistently generates each object accurately within specified locations, even under complex bounding box configurations, highlighting its robustness and precision. Additional results are shown in Fig. 9, and examples of various aspect ratio images generated with grounding conditions are provided in Fig. 5.

Layout

GROUNDIT

Layout

GROUNDIT

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction include our key contribution of improving the spatial grounding of Diffusion Transformers via noisy patch transplantation, and mention the experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the Conclusion section and Section D of appendix, we discuss our main limitation, which is the increased computation time for image generation.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work is mainly based on empirical observations regarding Diffusion Transformers, and does not include theoretical findings.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In the appendix, we provide the exact model that our method is build upon, and the implementation details required for reproducing the method.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: While we have not provided the code yet, we provide a link to our project page, at which we will provide the official code for our method.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Both in the Results section and Appendix, we provide the used benchmarks, our experiment settings, and the used hyperparameters.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not provide error bars for the experiments as we have not run our main experiments multiple times.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the specific GPU resources and also the inference time of our generation method.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research process and results adhere to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the Conclusion section, we discuss the potential societal impacts of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: We do not discuss any additional safeguards for our method. However, we adhere to the safeguards already implemented in text-to-image models that we utilize.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We credit the authors of Attention Refocusing [38] who have suggested the evaluation method, and also the creators of HRS [3] and DrawBench [43].

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: For the custom benchmarks that we have used in additional experiments in the appendix, we plan to open-source them along with the official code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not include research with crowdsourcing and human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not include research with crowdsourcing and human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a zero-shot layout technique for spatially-grounded text-to-image generation with the power of transformer-based diffusion architecture. Prior methods tend to manipulate the latent image during reverse process for grounding, solely relying on cross-attention maps that show the alignment between specific text tokens and spatial regions of the target object to place in. This shows a drawback in generating a target object that fits within the region of bounding boxes. By utilizing transformer architecture, the method can divide the generation process into two branches: (1) first branch is to generate a target image patch of the bounding box with an arbitrary size by a transformer, (2) the second branch is the original generation process of the whole image. At every denoising step, the generated object is then copied into the generated image of the second branch.\n\nSTRENGTHS:\n- Addressing the limitation of attention-based method in generating arbitrary object sizes within bounding boxes, the method proposes to utilize transformer architecture (namely DiT) which is free from fixed-resolution constraints and enables patch-level generation.\n- The paper is well-written and easy to follow.\n- Model efficacy is superior to other methods in Figure 4 and Table 1.\n\nWEAKNESSES:\n- 158-161: It is obvious that different noise inputs certainly yield different outputs. So I do not understand the main point of this part. \n- Semantic sharing part is an interesting observation. It is more helpful if the authors compare it with image interpolation. Does it work when two images have different semantic classes like dog and human?\nWang, Clinton, and Polina Golland. \"Interpolating between images with diffusion models.\" (2023). \n- Eq 5: What is formulation of $ \\mathcal{L}(x_t, G_i, A_i)$?\n- In reverse process: Have the authors applied semantic sharing on bounding boxes like concat tokens of multiple objects for simultaneous denoising? I assume that they share the same semantic class. \n- In Fig 3, I saw the size of object image is much bigger than the size of bounding box. Is it intentional? If so, please explain. How to define noisy object image $z_{i, t}$? As far as I understand, the method need to reserve the aspect ratio of the bounding box by scaling it correspondingly to match the size of main image. Is it possible to disable this scaling step and denoise an object image with the same size as the bounding box?\n- Does Noisy Patch transplantation guarantee the consistent content of the output image when pasting the generated image  with a binary mask (Eq. 9)? Again in Eq 9, the size of $x_{t-1}$ and $B_{i, t-1}$ are not the same as illustrated in Figure 3 so it is not clear how Hadamard product is possible.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nLimitations and societal impacts are included in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper targets exploiting a pre-trained PixArt-alpha (a text-to-image multi-aspect diffusion transformer) to generate images conditioned on a set of text-labeled bounding boxes.\n\nThe authors start from the idea that, at inference time, a transformer can be given an arbitrary number of tokens, and that these tokens can have positional embedding corresponding to multiple images of multiple aspect ratios at once. Empirically, they observe that in this situation, the diffusion transformer is able to handle the generation process consistently and the outputs are multiple images with shared content but with different aspect ratios corresponding to the input PEs.\n\nWhen generating the content of the bounding boxes, they take advange of this idea by simultaneously generating two version of the same bounding-box, one with positional embeddings corresponding to its location in the full image, and the other one with positional embeddings making the crop appear as a full image. \nThis ensures that the label is represented in the full-image version, and thanks to the shared content, also in the original box version, that can then be transplanted back into the image.\n\nSTRENGTHS:\n- The paper confirms an interesting property of multi-aspect diffusion transformers, in that tokens are able to interact at inference time, across combinations of positions that are not seen during training.\n- They propose a clever way to exploit this capability in the context of a multi-aspect diffusion transformer.\n- Handling spatial relations correctly is one of the major challenges that state-of-the-art text-to-image models have yet to overcome. The targetted task of grounding is very relevant as it is one way to circumvent the issue.\n- The proposed solution obtains very good performance on the considered benchmark.\n\nWEAKNESSES:\nMainly, I find that the exposition of the idea not very clear and could be improved:\n- How the different branches, the two-stage process, and the diffusion scheduling interact is not clear:\n  - Appendix A.1 reveals that Stage 1 and 2 are merged in a single full 50 steps denoising schedule, with Stage 1 only being used in the 25 first steps, which is not exactly in line with what is presented in the main paper. This information needs to be in the main paper, and ideally the presentation should be made more consistent with the implementation.\n  - Figure 3 and Line 200 suggest that in the main branch x_t is denoised in parallel to the object branch, but it does not appear in Algorithm 1. This is inconsistent. And if the main brain does anything, it needs to be explained and detailed.\n- Section 3 should present how multi-aspect/size is handled in diffusion transformers, as it is a key requirement for the proposed method. Also, Eq.3 introduces $w$ without definition. \n- a smaller point that could be improved: in 4.2, while it can be derived from context and references that $\\mathcal{L}$ needs to be a grounding loss, this is nowhere explicitly stated. If I strictly adhered to what is written in this paragraph, the authors could very well be arguing that any loss function that takes as input $(x_t, \\mathcal(G)_i, \\mathcal(A)_i)$ would be effective, which is obviously not what they are trying to say.\n\nQUESTIONS:\nI find this submission to be very interesting and show a lot of promise, but the description of the method is very confusing. For the discussion period, I hope the authors can provide the necessary clarifications and leave enough time for us to have a discussion together around their updated description.\nIndeed, until the main points are convincingly clarified, succintly reviewed again, and we are confident that changes will be made, I cannot recommend acceptance. My main question is then:\n- Can the authors clarify if anything happens in the main branch? Perhaps a full description of how the method works, including the reverse diffusion iterations would help.\n\nAlso, as a comparatively minor point, diffusion models are known for the flexibility of their sampling process. Considering how each stage and branch have compatible objectives, have the authors considered doing every update simultaneously in a single reverse diffusion process instead of multi-stage?\n\nLIMITATIONS:\nLimitations and impacts are discussed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the zero-shot layout-to-image generation problem using bounding boxes and texts as conditional inputs. The authors propose a method called GrounDiT, which builds upon the recent Diffusion Transformers (DiT) model. Leveraging DiT's emergent property of semantic sharing, where two noisy images of different sizes can progressively converge during sampling, the authors introduce a key innovation: a separate object denoising branch for each bounding box condition, running in parallel to the main denoising branch. In each object branch, a noisy object patch is generated and then implanted into the original image. The authors compare GrounDiT with SOTA zero-shot methods on the HRS and DrawBench datasets, demonstrating that GrounDiT can generate complete objects in the correct regions while maintaining high overall image fidelity.\n\nSTRENGTHS:\n- The paper is well-written, with a clear and concise background and motivation.\n- The idea of having a separate object branch for each bounding box is simple and intuitive, making the method easy to understand and implement.\n- The authors conduct an extensive comparison with SOTA baselines, including R&B, BoxDiff, Attention-Refocusing, and Layout-Guidance, demonstrating the effectiveness of their approach.\n\nWEAKNESSES:\n- The paper lacks a comparison of prompt fidelity on the HRS dataset. Except for PixArt-R&B( a variant of GrounDiT), it would also be useful to include results from other baselines like R&B and BoxDiff to ensure the completeness of the work, even if they may use different backbones.\n\n- In Table 1, GrounDiT outperforms its variant PixArt-R&B in grounding accuracy, which is expected given the special design of noisy object patch cultivation and transplantation. However, in Table 2, GrounDiT continue to surpass PixArt-R&B in prompt fidelity scores like CLIP score and ImageReward. I am curious to know where these improvements come from, as object patch cultivation and transplantation should only benefit grounding ability. Did the authors perform multiple runs for the experiments?\n\n- The paper does not evaluate the MS-COCO subset presented in the R&B paper, which would be a valuable addition to the experiments.\n\n- The proposed method would increase the compuation cost for image generation. However, there is no quantitative evidence showing the exact cost.\n\nQUESTIONS:\nPlease refer to the weaknesses part.\n\nLIMITATIONS:\nIn the conclusion part, the authors admit that the proposed method would increase the compuation time for image generation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
This paper proposes a method to use a pre-trained text-to-image
diffusion model to guide the generation into placing objects at given
locations determined by bounding boxes. The challenge is to develop
this capability without requiring fine-tuning of the model.

STRENGTHS:
- The problem of guided generation without requiring fine-tuning of a
  diffusion model is a relevant one, and the method seems effective.
- Experimental results show the proposed method outperforms compared
  baselines.

WEAKNESSES:
- The quality of the presentation can be improved. For instance, one
  fundamental component of the method, $\math{L}_{AGG}$, is not
  defined. The evaluation metrics (spatial, size, color) are not
  properly defined, unless I missed it.
- Further analysis or discussion on why the proposed approach is
  superior to simply directly transferring the patch would make the
  paper more convincing.
- Experimental comparison is limited to two datasets. Qualitatively,
  it's not entirely clear there is a significant difference between
  the R&B baseline and the proposed approach.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15