GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation

Authors: phillip y lee
taehoon  yoon
minhyuk  sung
main  branch
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-05 18:53:29

Abstract: We introduce GrounDiT, a novel training-free spatial grounding technique for text-to-image generation using Diffusion Transformers (DiT). Spatial grounding with bounding boxes has gained attention for its simplicity and versatility, allowing for enhanced user control in image generation. However, prior training-free approaches often rely on updating the noisy image during the reverse diffusion process via backpropagation from custom loss functions, which frequently struggle to provide precise control over individual bounding boxes. In this work, we leverage the flexibility of the Transformer architecture, demonstrating that DiT can generate noisy patches corresponding to each bounding box, fully encoding the target object and allowing for fine-grained control over each region. Our approach builds on an intriguing property of DiT, which we refer to as semantic sharing. Due to semantic sharing, when a smaller patch is jointly denoised alongside a generatable-size image, the two become semantic clones. Each patch is denoised in its own branch of the generation process and then transplanted into the corresponding region of the original noisy image at each timestep, resulting in robust spatial grounding for each bounding box. In our experiments on the HRS and DrawBench benchmarks, we achieve state-of-the-art performance compared to previous training-free approaches. Project Page: https://groundit-diffusion.github.io/.

Main paper: 
 introduction: 
 The Transformer architecture [45] has driven breakthroughs across a wide range of applications, with diffusion models emerging as significant recent beneficiaries. Despite the success of diffusion models with U-Net [42] as the denoising backbone [22, 43, 41, 39] , recent Transformer-based diffusion models, namely Diffusion Transformers (DiT) [37] , have marked another leap in performance. This is demonstrated by recent state-of-the-art generative models such as Stable Diffusion 3 [13] and Sora [6] . Open-source models like DiT [37] and its text-guided successor PixArt-α [8] have also achieved superior quality compared to prior U-Net-based diffusion models. Given the scalability of Transformers, Diffusion Transformers are expected to become the new standard for image generation, especially when trained on an Internet-scale dataset.With high quality image generation achieved, the next critical step is to enhance user controllability. Among the various types of user guidance in image generation, one of the most fundamental and significant is spatial grounding. For instance, a user may provide not only a text prompt describing the image but also a set of bounding boxes indicating the desired positions of each object, as shown in Fig. 1 . Such spatial constraints can be integrated into text-to-image (T2I) diffusion models by adding extra modules that are designed for spatial grounding and fine-tuning the model. GLIGEN [31] is a notable example, which incorporates a gated self-attention module [1] into the U-Net layers of Stable Diffusion [41] . Although effective, such fine-tuning-based approaches incur substantial training costs each time a new T2I model is introduced.Recent training-free approaches for spatially grounded image generation [9, 47, 11, 36, 38, 48, 12, 26] have led to new advances, removing the high costs for fine-tuning. These methods leverage the fact that cross-attention maps in T2I diffusion models convey rich structural information about where each concept from the text prompt is being generated in the image [7, 19] . Building on this, these approaches aim to align the cross-attention maps of specific objects with the given spatial constraints (e.g. bounding boxes), ensuring that the objects are placed within their designated regions. This alignment is typically achieved by updating the noisy image in the reverse diffusion process using backpropagation from custom loss functions. However, such loss-guided update methods often struggle to provide precise spatial control over individual bounding boxes, leading to missing objects (Fig. 4 , Row 9, Col. 5) or discrepancies between objects and their bounding boxes (Fig. 4 , Row 4, Col. 4). This highlights the need for finer control over each bounding box during image generation.We aim to provide more precise spatial control over each bounding box, addressing the limitations in previous loss-guided update approaches. A well-known technique for manipulating local regions of the noisy image during the reverse diffusion process is to directly replace or merge the pixels (or latents) in those regions. This simple but effective approach has proven effective in various tasks, including compositional generation [17, 50, 44, 32] and high-resolution generation [5, 29, 24, 25] . One could consider defining an additional branch for each bounding box, denoising with the corresponding text prompt, and then copying the noisy image into its designated area in the main image at each timestep. However, a key challenge lies in creating a noisy image patch-at the same noise level-that reliably contains the desired object while fitting within the specified bounding box. This has been impractical with existing T2I diffusion models, as they are trained on a limited set of image resolutions. While recent models such as PixArt-α [8] support a wider range of image resolutions, they remain constrained by specific candidate sizes, particularly for smaller image patches. As a result, when these models are used to create a local image patch, they are often limited to denoising a fixed-size image and cropping the region to fit the bounding box. This approach can critically fail to include the desired object within the cropped region.In this work, we show that by exploiting the flexibility of the Transformer architecture, DiT can generate noisy image patches that fit the size of each bounding box, thereby reliably including each desired object. This is made possible through our proposed joint denoising technique. First, we introduce an intriguing property of DiT: when a smaller noisy patch is jointly denoised with a generatable-size noisy image, the two gradually become semantic clones-a phenomenon we call semantic sharing. Next, building on this observation, we propose a training-free framework that involves cultivating a noisy patch for each bounding box in a separate branch and then transplanting that patch into its corresponding region in the original noisy image. By iteratively transplanting the separately denoised patches into their respective bounding boxes, we achieved fine-grained spatial control over each bounding box region. This approach leads to more robust spatial grounding, particularly in cases where previous methods fail to accurately adhere to spatial constraints.In our experiments on the HRS [3] and DrawBench [43] datasets, we evaluate our framework, GROUNDIT, using PixArt-α [8] as the base text-to-image DiT model. Our approach demonstrates superior performance in spatial grounding compared to previous training-free methods [38, 9, 47, 48] , especially outperforming the state-of-the-art approach [47] , highlighting its effectiveness in providing fine-grained spatial control.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our abstract and introduction include our key contribution of improving the spatial grounding of Diffusion Transformers via noisy patch transplantation, and mention the experimental results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 In this section, we review the two primary approaches for incorporating spatial controls into textto-image (T2I) diffusion models: fine-tuning-based methods (Sec. 2.1) and training-free guidance techniques (Sec. 2.2).Fine-tuning with additional modules is a powerful approach for enhancing T2I models with spatial grounding capabilities [51, 31, 2, 53, 46, 16, 10, 54] . SpaText [2] introduces a spatio-textual representation that combines segmentations and CLIP embeddings [40] . ControlNet [51] incorporates a trainable U-Net encoder that processes spatial conditions such as depth maps, sketches, and human keypoints, guiding image generation within the main U-Net branch. GLIGEN [31] enables T2I models to accept bounding boxes by inserting a gated attention module into Stable Diffusion [41] . GLIGEN's strong spatial accuracy has led to its integration into follow-up spatial grounding methods [48, 38, 30] and applications such as compositional generation [15] and video editing [23] . InstanceDiffusion [46] further incorporates conditioning modules to provide finer spatial control through diverse conditions like boxes, scribbles, and points. While these fine-tuning methods are effective, they require taskspecific datasets and involve substantial costs, as they must be retrained for each new T2I model, underscoring the need for training-free alternatives.In response to the inefficiencies of fine-tuning, training-free approaches have been introduced to incorporate spatial grounding into T2I diffusion models. One approach involves a region-wise composition of noisy patches, each conditioned on a different text input [5, 50, 32] . These patches, extracted using binary masks, are intended to generate the object they are conditioned on within the generated image. However, since existing T2I diffusion models are limited to a fixed set of image resolutions, each patch cannot be treated as a complete image, making it uncertain whether the extracted patch will contain the desired object. Another approach leverages the distinct roles of attention modules in T2I models-self-attention captures long-range interactions between image features, while cross-attention links image features with text embeddings. By using spatial constraints such as bounding boxes or segmentation masks, spatial grounding can be achieved either by updating the noisy image using backpropagation based on a loss calculated from cross-attention maps [48, 38, 9, 7, 18, 36] , or by directly manipulating cross-or self-attention maps to follow the given spatial layouts [26, 4, 14] . While the loss-guided methods enable spatial grounding in a training-free manner, they still lack precise control over individual bounding boxes, often leading to missing objects or misalignmet between objects and their bounding boxes. In this work, we propose a novel training-free framework that offers fine-grained spatial control over each bounding box by harnessing the flexibility of the Transformer architecture in DiT.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In the Conclusion section and Section D of appendix, we discuss our main limitation, which is the increased computation time for image generation.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background: diffusion transformers: 
 Diffusion Transformer (DiT) [37] represents a new class of diffusion models that utilize the Transformer architecture [45] for their denoising network. Previous diffusion models like Stable Diffusion [41] use the U-Net [42] architecture, of which each layer contains a convolutional block and attention modules. In contrast, DiT consists of a sequence of DiT blocks, each containing a pointwise feedforward network and attention modules, removing convolution operations and instead processing image tokens directly through attention mechanisms. DiT follows the formulation of diffusion models [22] , in which the forward process applies noise to a real clean data x 0 byx t = √ α t x 0 + √ 1 -α t ϵ where ϵ ∼ N (0, I), α t ∈ [0, 1].(1) The reverse process denoises the noisy data x t through a Gaussian transition p θ (x t-1 |x t ) = N (x t ; µ θ (x t , t), Σ θ (x t , t))(2) where µ θ (x t , t) is calculated by a learned neural network trained by minimizing the negative ELBO objective [27] . While Σ θ (x t , t) can also be learned, it is usually set as time dependent constants.Positional Embeddings. As DiT is based on the Transformer architecture, it treats the noisy image x t ∈ R h×w×d as a set of image tokens. Specifically, x t is divided into patches, each transformed into an image token via linear embedding. This results in (h/l) × (w/l) tokens, where l is the patch size. Importantly, before each denoising step, 2D sine-cosine positional embeddings are assigned to each image token to provide spatial information as follows:x t-1 ← Denoise(PE(x t ), t, c).(3) Here, PE(•) applies positional embeddings, Denoise(•) represents a single denoising step in DiT at timestep t, and c is the text embedding. This contrasts with U-Net-based diffusion models, which typically do not utilize positional embeddings for the noisy image. Detailed formulations of the positional embeddings are provided in the Appendix (Sec. A). 
problem definition: 
 Let P be the input text prompt (i.e., a list of words), which we refer to as the global prompt. Let c P be the text embedding of P . We define a set of N grounding conditions G = {g i } N -1 i=0 , where each condition specifies the coordinates of a bounding box and the desired object to be placed within it. Specifically, each condition g i := (b i , p i , c i ) consists of the following: b i ∈ R 4 , the xy-coordinates of the bounding box's upper-left and lower-right corners, p i ∈ P , the word in the global prompt describing the desired object within the box, and c i , the text embedding of p i . The objective is to generate an image that aligns with the global prompt P while ensuring each specified object is accurately positioned within its corresponding bounding box b i . 
groundit: grounding diffusion transformers: 
 We propose GROUNDIT, a training-free framework based on DiT for generating images spatially grounded on bounding boxes. Each denoising step in GROUNDIT consists of two stages: Global Update and Local Update. Global Update ensures coarse alignment between the noisy image and the bounding boxes through a gradient descent update using cross-attention maps (Sec. 5.1). Following this, Local Update further provides fine-grained control over individual bounding boxes via a novel noisy patch transplantation technique (Sec. 5.3) . This approach leverages our key observation of DiT's semantic sharing property, introduced in Sec. 5.2. An overview of this two-stage denoising step is provided in Fig. 2 .First, the noisy image x t is updated to spatially align with the bounding box inputs. For this, we leverage the rich structural information encoded in cross-attention maps, as first demonstrated by Chefer et al. [7] . Each cross-attention map shows how a region of the noisy image correspond to a specific word in the global prompt P . Let DiT consist of M sequential DiT blocks. As x t passes through the m-th block, the cross-attention map a m i,t ∈ R h×w×1 for object p i is extracted. For each grounding condition g i , the mean cross-attention map A i,t is obtained by averaging a m i,t over all M blocks as follows:EQUATIONFor convenience, we denote the operation of extracting A i,t for every g i ∈ G as below:EQUATIONThen, following prior works on U-Net-based diffusion models [48, 47, 38, 9, 36, 7] , we measure the spatial alignment for each object p i by comparing its mean cross-attention map A i,t with its corresponding bounding box b i , using a predefined grounding loss L(A i,t , b i ) as defined in R&B [47] .The aggregated grounding loss L AGG is then computed by summing the grounding loss across all grounding conditions g i ∈ G:EQUATIONBased on the backpropagation from L AGG , x t is updated via gradient descent as follows:EQUATIONwhere ω t is a scalar weight value for gradient descent. We refer to Eq. 7 as the Global Update, as the whole noisy image x t is updated based on an aggregated loss from all grounding conditions in G.The Global Update achieves reasonable accuracy in spatial grounding. However, it often struggles with more complex grounding conditions. For instance, when G contains multiple bounding boxes (e.g., six boxes in Fig. 4 , Row 9) or small, thin boxes (e.g., Fig. 4 , Row 5), the desired objects may be missing or misaligned with the boxes. These examples show that the Global Update lacks fine-grained, box-specific control, underscoring the need for precise controls for individual bounding boxes. In the following sections, we introduce a novel method to achieve this fine-grained spatial control.In this section, we present our observations on an intriguing property of DiT, semantic sharing, which will serve as a key building block for our main method in Sec. 5.3.Joint Denoising. We observed that DiT can jointly denoise two different noisy images together. For example, consider two noisy images, x t and y t , both at timestep t in the reverse diffusion process. Position embeddings are applied to the image tokens according to their sizes, resulting in xt = PE(x t ) and ȳt = PE(y t ). Notably, the two noisy images can differ in size, providing flexibility in joint denoising. The key part is that the two noisy images-more precisely, two sets of image tokens-are merged into a single set z t . We denote this process as Merge(•) (see Alg. 1, line 4). z t is passed through the DiT blocks, yielding the output z t-1 , which is then split into the denoised versions x t-1 and y t-1 via Split(•). Joint denoising is illustrated in Fig. 3-(A) , and a pseudocode for a single step of joint denoising is shown in Alg. 1.Semantic Sharing. Surprisingly, we found that joint denoising of two noisy images generates semantically correlated content in correponding pixels, even when the initial random noise differs. Consider two noisy images, x T ∈ R hx×wx×d and y T ∈ R hy×wy×d , both initialized from a unit Gaussian distribution N (0, I). We experiment with a reverse diffusion process in which, for the initial 100γ% of the denoising steps (γ ∈ [0, 1]), x T and y T undergo joint denoising. For the remaining (B), (C) Semantic Sharing. Denoising two noisy images using joint denoising results in semantically correlated content between the generated images. Here, γ indicates that joint denoising is during the initial 100γ% of the timesteps, after which the images are denoised for the remaining steps.timesteps, they are denoised independently. The same text embedding c is used as a condition in both cases.Fig. 3 shows the generated images from x T and y T across different γ values. In Fig. 3 -(B), x T and y T have the same resolution (h x = h y , w x = w y ), while in Fig. 3-(C ) their resolutions differ (h x > h y , w x > w y ). When γ = 0, the two noisy images are denoised completely independently, resulting in clearly distinct images (leftmost column). We found that DiT models have a certain range of resolutions within which they can generate plausible images-which we refer to as generatable resolutions-but face challenges when generating images far outside this range. This is demonstrated in the output of y 0 in Fig. 3-(C ) with γ = 0. Further discussions and visual analyses are provided in the Appendix (Sec. D). But as γ increases, allowing x T and y T to be jointly denoised in the initial steps, the generated images become progressively more similar. When γ = 1, the images generated from x T and y T appear almost identical. These results demonstrate that, in joint denoising, assigning identical or similar positional embeddings to different image tokens promotes strong interactions between them during the denoising process. This correlated behavior during joint denoising causes the two image tokens to converge toward semantically similar outputs-a phenomenon we term semantic sharing.Notably, this pattern holds not only when both noisy images share the same resolution (Fig. 3-(B) ), but even when one of the images does not have DiT's generatable resolution (Fig. 3-(C )). While self-attention sharing techniques have been explored in U-Net-based diffusion models to enhance style consistency between images [20, 34] , they have been limited to images of equal resolution. By leveraging the flexibility to assign positional embeddings across different resolutions, our joint Algorithm 1: Pseudocode of Joint Denoising (Sec. 5.2). denoising approach extends across heterogeneous resolutions, offering greater versatility. We provide further discussions and analyses on semantic sharing in the Appendix (Sec. D).Inputs: xt ∈ R hx×wx×d , yt ∈ RIn this section, we introduce our key technique: Local Update via noisy patch cultivation and transplantation. Building on DiT's semantic sharing property from Sec. 5.2, we show how this can be leveraged to provide precise spatial control over each bounding box.Main & Object Branches. We propose a parallel denoising approach with multiple branches: one for the main noisy image xt and additional branches for each grounding condition g i . The main branch denoises the main noisy image using the global prompt P , while each object branch is designed to denoise local regions within the bounding boxes, enabling fine-grained spatial control over each region. For each i-th object branch, there is a distinct noisy object image u i,t , initialized as u i,T ∼ N (0, I). We predefine the resolution of the noisy object image u EQUATIONwhere c i is the text embedding of the object p i .Through semantic sharing with the noisy object image u i,t during joint denoising, the denoised local patch v i,t-1 is expected to gain richer semantic features of object p i than it would without joint denoising. Note that even when the noisy local patch v i,t does not meet the typical generatable resolution of DiT (since it often requires cropping small bounding box regions of xt to obtain v i,t ), it offers a simple and effective way for enriching v i,t of the semantic features of object p i . We refer to this process as noisy patch cultivation.Noisy Patch Transplantation. After cultivating local patches through joint denoising in Eq. 8, each patch is transplanted into xt-1 , obtained from the main branch. The patches are transplanted in their original bounding box regions specified by b i as follows:EQUATIONHere, ⊙ denotes the Hadamard product, m i is a binary mask for the bounding box b i , and Uncrop(v i,t-1 , b i ) applies zero-padding to v i,t-1 to align its position with that of b i . This transplantation enables fine-grained local control for the grounding condition g i . After transplanting the outputs from all N object branches, we obtain x t-1 , representing the final output of GROUNDIT denoising step at timestep t. In x t-1 , the image tokens within the b i region are expected to possess richer semantic information about object p i compared to the initial xt-1 from the main branch. This process is referred to as noisy patch transplantation. We provide implementation details and full pseudocode of a single GROUNDIT denoising step in the Appendix (Sec. E).Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: While we have not provided the code yet, we provide a link to our project page, at which we will provide the official code for our method.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
results: 
 In this section, we present the experiment results of our method, GROUNDIT, and provide comparisons with baselines. For the base text-to-image DiT model, we use PixArt-α [8] , which builds on the original DiT architecture [37] by incorporating an additional cross-attention module to condition on text prompts.Layout Layout-Guidance [9] Attention-Refocusing [38] BoxDiff [48] R&B [47] PixArt-R&B GROUNDIT "A dog in the beautiful park.""An eagle is flying over a tree.""A duck wearing a hat standing near a bicycle.""A plastic bottle and an apple on a table.""An apple and a banana and a cup on a table."""A dog wearing sunglasses and a red hat and a blue tie.""A chair and a table and a bed is on the room with a photo frame on the wall and a ceiling lamp [...]""A dog and a bird sitting on a branch while an eagle is flying in the sky.""A car and a dog on the road while horse and a chair is on the grass." The rightmost column includes the results of our GROUNDIT.Baselines. We compare our method with state-of-the-art training-free approaches for bounding box-based image generation, including R&B [47] , BoxDiff [48] , Attention-Refocusing [38] , and Layout-Guidance [14] . For a fair comparison, we also implement R&B using PixArt-α, which we refer to as PixArt-R&B, and treat it as an internal baseline. Note that this is identical to our method without the Local Guidance (Sec. 5.3).Evaluation Metrics and Benchmarks.• (Grounding Accuracy) We follow the evaluation protocol of R&B [47] to assess spatial grounding on the HRS [3] and DrawBench [43] datasets, using three criteria: spatial, size, and color. The HRS dataset consists of 1002, 501, and 501 images for each respective criterion, with bounding boxes generated using GPT-4 by Phung et al. [38] . For DrawBench, we use the same 20 positional prompts as in R&B [47] . • (Prompt Fidelity) We use the CLIP score [21] to evaluate how well the generated images adhere to the text prompt. Additionally, we assess our method using PickScore [28] and ImageReward [49] , which provide human alignment scores based on the consistency between the text prompt and generated images. Table 1 : Quantitative comparisons of grounding accuracy on HRS [3] and DrawBench [43] benchmarks. Bold represents the best, and underline represents the second best method.Quantitative Comparisons. Tab. 1 presents a quantitative comparison of grounding accuracy between our method, GROUNDIT, and baselines. GROUNDIT outperforms all baselines across different criteria of grounding accuracy-spatial, size, and color-including the state-of-the-art R&B [47] and our internal baseline PixArt-R&B. Notably, the spatial accuracy on the HRS benchmark [3] (Col. 1) of GROUNDIT is significantly higher, with a +14.87% improvement over R&B and +7.88% over PixArt-α. The comparison between PixArt-α [8] , PixArt-R&B and GROUNDIT highlights the effectiveness of the two-stage pipeline of GROUNDIT. First, integrating the loss-based Global Update into PixArt-α results in a substantial improvement in spatial accuracy (from 17.86% to 37.13%). Then, incorporating our key contribution, the Local Update, further boosts accuracy (from 37.13% to 45.01%). For size accuracy (Col. 2), which evaluates how well the size of each generated object matches its corresponding bounding box, GROUNDIT shows a +1.01% improvement over R&B. In terms of color accuracy (Col. 3), our method achieves a +6.60% improvement over PixArt-R&B and outperforms R&B by +3.63%. This underscores the effectiveness of our noisy patch transplantation technique in accurately assigning color descriptions to the corresponding objects. As DrawBench [43] only contains images with two bounding boxes, which are relatively easy to generate, employing the Global Update is sufficient for grounding. We present additional quantitative comparisons of grounding accuracy in the Appendix (Sec. B).Qualitative Comparisons. Fig. 4 presents the qualitative comparisons. When the grounding condition involves one or two simple bounding boxes (Rows 1, 2), both our method and the baselines successfully generate objects within the designated regions. However, as the number of bounding boxes increases and the grounding conditions become more challenging, the baselines struggle to correctly place each object inside the bounding box (Rows 4, 8), or even fail to generate the object at all (Rows 5, 7, 9). In contrast, GROUNDIT successfully grounds each object within the boxes, even when the number of boxes is relatively high, such as four boxes (Rows 5, 6, 8), five boxes (Row 7) and six boxes (Row 9). This highlights that our proposed noisy patch transplantation technique provides superior control over each bounding box, addressing the limitations of previous loss-based update methods, as discussed in Sec. 5.1. For more qualitative comparisons, including images generated with various aspect ratios, please refer to the Appendix (Sec. G and Fig. 5 ).Tab. 2 presents a quantitative comparison of prompt fidelity between our method and PixArt-R&B. Each metric is measured using the generated images from the HRS dataset [3] . GROUNDIT achieves higher CLIP score [21] than PixArt-R&B (Col. 1), indicating that our noisy patch transplantation improves the text prompt fidelity of the generated images. Additionally, our method achieves a higer ImageReward [49] score, which measures human preference by considering both prompt fidelity and overall image quality. While GROUNDIT shows a slight underperformance compared to PixArt-R&B in Pickscore [28] , it remains comparable overall. We provide further comparisons of prompt fidelity with other baselines in the Appendix (Sec. C). Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Both in the Results section and Appendix, we provide the used benchmarks, our experiment settings, and the used hyperparameters.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this work, we introduced GROUNDIT, a training-free spatial grounding technique for text-to-image generation, leveraging Diffusion Transformers (DiT). To address the limitation of prior approaches, which lacked fine-grained spatial control over individual bounding boxes, we proposed a novel approach that transplants a noisy patch generated in a separate denoising branch into the designated area of the noisy image. By exploiting an intriguing property of DiT, semantic sharing, which arises from the flexibility of the Transformer architecture and the use of positional embeddings, GROUNDIT generates a smaller patch by simultaneously denoising two noisy image: one with a smaller size and the other with a generatable resolution by DiT. Through semantic sharing, these two noisy images become semantic clones, enabling fine-grained spatial control for each bounding box.Our experiments on the HRS and DrawBench benchmarks demonstrated that GROUNDIT achieves state-of-the-art performance compared to previous training-free grounding methods.Limitations and Societal Impacts. A limitation of our method is the increased computation time, as it requires a separate object branch for each bounding box. We provide further analysis on the computation time in the Appendix (Sec. F). Additionally, like other generative AI techniques, our method is susceptible to misuse, such as creating deepfakes, which can raise significant concerns related to privacy, bias, and fairness. It is crucial to develop safeguards to control and mitigate these risks responsibly.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We do not provide error bars for the experiments as we have not run our main experiments multiple times.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: We do not discuss any additional safeguards for our method. However, we adhere to the safeguards already implemented in text-to-image models that we utilize.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We credit the authors of Attention Refocusing [38] who have suggested the evaluation method, and also the creators of HRS [3] and DrawBench [43] .Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [No] Justification: For the custom benchmarks that we have used in additional experiments in the appendix, we plan to open-source them along with the official code.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We do not include research with crowdsourcing and human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: We do not include research with crowdsourcing and human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 "A bear is in an antique living room with a chair and a chiffonier, while a chandelier hanging from the ceiling.""A balloon and fireworks are in the night sky, and below is a beautiful forest with a car and a chair and a waterfall." "A banana and an apple are beneath a book and a flower is lying on the book in a room" "Aurora lights up the sky and a horse and a house are on the grassy meadow with a mountain in the background."Figure 1 : Spatially grounded images generated by our GROUNDIT. Each image is generated based on a text prompt along with bounding boxes, which are displayed in the upper right corner of each image. Compared to existing methods that often struggle to accurately place objects within their designated bounding boxes, our GROUNDIT enables more precise spatial control through a novel noisy patch transplantation mechanism.We introduce GROUNDIT, a novel training-free spatial grounding technique for text-to-image generation using Diffusion Transformers (DiT). Spatial grounding with bounding boxes has gained attention for its simplicity and versatility, allowing for enhanced user control in image generation. However, prior training-free approaches often rely on updating the noisy image during the reverse diffusion process via backpropagation from custom loss functions, which frequently struggle to provide precise control over individual bounding boxes. In this work, we leverage the flexibility of the Transformer architecture, demonstrating that DiT can generate noisy patches corresponding to each bounding box, fully encoding the target object and allowing for fine-grained control over each region. Our approach builds on an intriguing property of DiT, which we refer to as semantic sharing. Due to semantic sharing, when a smaller patch is jointly denoised alongside a generatable-size image, the two become semantic clones. Each patch is denoised in its own branch of the generation process and then transplanted into the corresponding region of the original noisy image at each timestep, resulting in robust spatial grounding for each bounding box. In our experiments on the HRS and DrawBench benchmarks, we achieve state-of-the-art performance compared to previous training-free approaches. Project Page: https://groundit-diffusion.github.io/. 
 
 Appendix: Diffusion Transformers (DiT) [37] handle noisy images of varying aspect ratios and resolutions by processing them as a set of image tokens. For this, the noisy image is first divided into patches, with each patch subsequently converted into an image token of hidden dimension D through a linear embedding layer. DiT then applies 2D sine-cosine positional embeddings to each image token, based on its coordinates (x, y), defined as follows: where w d = 1/10000 (4d/D) . The positional embedding p x,y is then added to each corresponding image token, denoted as PE(•).In addition to Sec. 6.2, we provide further quantitative comparisons of grounding accuracy between our GROUNDIT and the baselines. Specifically, we generated images based on text prompts and bounding boxes using each method, then calculated the mean Intersection over Union (mIoU) between the detected bounding boxes from an object detection model [55] and the input bounding boxes. Below, we present the quantitative comparisons across three datasets with varying average numbers of bounding boxes: subset of MS-COCO-2014 [33] , HRS-Spatial [3] , and a custom dataset.Dataset Subset of MS-COCO-2014 [33] HRS-Spatial [ Subset of MS-COCO-2014. We filtered the validation set of MS-COCO-2014 [33] to exclude image-caption pairs where the target objects were either not mentioned in the captions or duplicate objects were present. From this filtered set, we randomly selected 500 pairs for evaluation.The results are presented in Tab. 4, column 2. GROUNDIT outperforms R&B by 0.021 (a 5.1% improvement) and PixArt-R&B by 0.014 (a 2.2% improvement). The relatively small margin can be attributed to the simplicity of the task, as this dataset has an average of 2.06 bounding boxes (Tab. 3), making it less challenging even for the baseline methods.HRS-Spatial. Column 3 of Tab. 4 presents the results on the Spatial subset of the HRS dataset [3] . GROUNDIT surpasses R&B [47] by 0.046 (a 14.1% improvement) and PixArt-R&B by 0.038 (an 11.4% improvement). Compared to the results on the MS-COCO-2014 subset, the higher percentage increase in mIoU highlights the robustness of GROUNDIT under more complex grounding conditions. Note that HRS-Spatial has an average of 3.11 bounding boxes (Tab. 3), which is higher than that of the MS-COCO-2014 subset (2.06).Custom Dataset. The custom dataset consists of 500 layout-text pairs, generated using the layout generation pipeline from LayoutGPT [15] . As shown in column 4 of Tab. 4, GROUNDIT outperforms R&B by 0.052 (a 26.3% improvement) and PixArt-R&B by 0.044 (a 21.4% improvement). This dataset has the highest average number of bounding boxes at 4.48 (Tab. 3). These results further emphasize the robustness and effectiveness of our approach in handling more complex grounding conditions with a larger number of bounding boxes.Subset of MS-COCO-2014 [33] HRS-Spatial [3] Custom DatasetStable Diffusion [41] 0.176 0.068 0.030 PixArt-α [8] 0.233 0.085 0.036 Layout-Guidance [9] 0.307 0.199 0.122 Attention-Refocusing [38] 0.254 0.145 0.078 BoxDiff [48] 0.324 0.164 0.106 R&B [47] 0 In addition to Sec. 6.3, we provide further quantitative comparisons of the prompt fidelity of the generated images between our GROUNDIT and the baselines. We evaluated the generated images from the HRS dataset [3] using three different metrics: CLIP score [21] , ImageReward [49] , and PickScore [28] . The results are presented in Tab. 5. Since PickScore evaluates preferences between a pair of images, we report the difference between our GROUNDIT and each baseline method in column 4. Our GROUNDIT consistently outperforms the baselines in both CLIP score and ImageReward. For PickScore, GROUNDIT outperforms all baselines except PixArt-R&B, while remaining comparable.Method CLIP score ↑ ImageReward ↑ PickScore ↑ (Ours -Baseline)Backbone: Stable Diffusion [41] Layout-Guidance [9] 32.48 -0.401 +0.30 Attention-Refocusing [38] 31.36 -0.508 +0.22 BoxDiff [48] 32.57 -0.199 +0.30 R&B [47] 33.16 -0.021 +0.26Backbone: PixArt-α [8] PixArt-R&B 33.49 0.280 -0.04 GROUNDIT (Ours) 33.63 0.444 -Table 5 : Quantitative comparisons of prompt fidelity on the HRS dataset [3] . Bold represents the best method.In this section, we provide further analyses on the generatable resolution and the semantic sharing property of DiT, initially introduced in Sec. 5.2.Generatable Resolution of DiT. Although recent DiT models can generate images at various resolutions, they still struggle to produce images at completely arbitrary resolutions. We speculate that this limitation arises not from the model architecture itself, but from the resolution of the training images, which typically falls within a specific range [8] . Generating images at resolutions far outside this range often results in implausible outputs, suggesting the existence of an acceptable resolution range for DiT, which we refer to as its generatable resolution. In Fig. 6 , we illustrate this phenomenon. When the noisy image size falls within DiT's generatable resolution range, the model produces plausible images (rightmost two images). However, when the image size is significantly outside this range (leftmost two images), DiT fails to generate a plausible image. Leveraging this flexibility of Transformers, we presented our joint denoising technique (Alg. 1). Our main observation was that the joint denoising between two noisy images causes the two generated images to become semantically correlated, as illustrated in Fig. 3-(B ) and Fig. 3-(C ).In addition to the visualizations in Fig. 3 , we further quantify the semantic sharing property by measuring the LPIPS score [52] between two generated images. To explore the effect of joint denoising, we varied the parameter γ ∈ [0, 1], which controls the proportion of denoising steps where joint denoising is applied. Specifically, γ = 0 means no joint denoising is applied, and each image is denoised independently, while γ = 1 means full joint denoising across all steps. As shown in Fig. 7 , increasing γ (i.e., applying more joint denoising steps) results in a decrease in the LPIPS score between the two generated images, indicating that the images become more semantically similar as joint denoising is applied for a larger portion of the denoising process. Figure 7 : LPIPS score between two generated images with varying γ value. A gradual decrease in LPIPS [52] indicates that joint denoising progressively enhances the similarity between the generated images.As the base text-to-image DiT model, we used the 512-resolution version of PixArt-α [8] . For sampling we employed the DPM-Solver scheduler [35] with 50 steps. Out of the 50 denoising steps, we applied our GROUNDIT denoising step (Alg. 2) for the initial 25 steps, and applied the vanilla denoising step for the remaining 25 steps. For the grounding loss in Global Update of GROUNDIT, we adopted the definition proposed in R&B [47] , and we set the loss scale to 10 and used a gradient descent weight of 5 for the gradient descent update in Eq. 7.As discussed in Sec. 5.3, for each i-th object branch we have a noisy object image u i,t and a noisy local patch v i,t , which is extracted from the noisy image xt in main branch via v i,t ← Crop(x t , b i ).We determine the resolution of the noisy object image u i,t by selecting from PixArt-α's generatable resolutions, choosing one that best aligns with the aspect ratio of the corresponding bounding box b i .All our experiments were conducted an NVIDIA RTX 3090 GPU. In Algorithm 2, we provide the pseudocode of GROUNDIT single denoising step.Algorithm 2: Pseudocode of GROUNDIT denoising step. for i = 0, . . . , N -1 do // mi is a binary mask corresponding to bi xt-1 ← xt-1 ⊙ (1 -mi) + Uncrop(vi,t-1, bi) ⊙ mi; // Patch Transplantation. xt-1 ← xt-1 return xt-1, {ui,t-1} N -1 i=0 ; Function GrounDiTStep(xt, {ui,t} N -1 i=0 , t, cP , G): xt ← GlobalUpdate(xt, t, cP , G) ; // Global update (Sec. 5.1) xt-1, {ui,t-1} N -1 i=0 ← LocalUpdate(xt, {ui,t} N -1 i=0 , t, cP , G) ; // Local update (Sec. 5.3) return xt-1, {ui,t-1} N -1 i=0 ;We present the average inference time based on the number of bounding boxes in Tab. 6. While our method shows a slight increase in inference time, the rate of increase remains modest. For three bounding boxes, the inference time is 1.01 times that of R&B and 1.33 times that of PixArt-R&B.Even with six bounding boxes, the inference time is only 1.41 times that of R&B and 1.90 times that of PixArt-R&B. We provide more qualitative comparisons in Fig. 8 . Our method demonstrates greater robustness against issues such as the missing object problem, attribute leakage, or object interruption problem [47] , due to its local update mechanism with semantic sharing. For instance, in Row 1, baseline methods struggle to generate certain objects (i.e. missing object problem). In Row 2, baselines generate a banana that retains features of an apple, illustrating attribute leakage. In Row 3, R&B generates a bus that interrupts the generation of a couch, with part of the bus overlapping with the designate region of the couch. Similarly, in PixArt-R&B, a hamburger and a donut interrupt the Layout R&B [47] PixArt-R&B GROUNDIT "A bear sitting between a surfboard and a chair with a bird flying in the sky.""A banana and an apple and an elephant and a backpack in the meadow with bird flying in the sky.""A realistic photo, a hamburger and a donut and a couch and a bus and a surfboard in the beach.""A blue vase and a wooden bowl with a watermelon sit on a table, while a bear holding an apple." generation of a surfboard, demonstrating the object interruption problem. In more challenging cases, like Row 4, combinations of these issues appear. By contrast, our method consistently generates each object accurately within specified locations, even under complex bounding box configurations, highlighting its robustness and precision. Additional results are shown in Fig. 9 , and examples of various aspect ratio images generated with grounding conditions are provided in Fig. 5 ."A boat floating on a calm lake." "An upright bear riding a bicycle.""An aurora lights up the sky and a house is on the grassy meadow with a mountain in the background.""A person is sitting on a chair and a bird is sitting on a horse while horse is on the top of a car.""A cat sitting on a sunny windowsill." "A bicycle standing near a telephone booth in the park.""A castle stands across the lake and the bird flies in the blue sky.""A Monet painting of a woman standing on a flower field holding an umbrella sideways with a house in the background." NeurIPS Paper Checklist 


Decision: This paper received mixed reviews: one accept, one weak accept, one borderline accept, and one borderline reject. The primary concerns raised by the reviewers were the precision and clarity of the method’s presentation, as well as the need for further comparisons on additional benchmarks.

The rebuttal effectively addressed these issues with additional evaluations and evidence, demonstrating the method’s effectiveness compared to existing approaches. Although reviewer LMpP was unable to respond to the rebuttal, the AC concurs with all the other reviewers that the improvements made in the rebuttal and subsequent discussions significantly raise the paper’s acceptance prospects.

The AC also agrees and thinks this work to be a novel, clear, and inspiring solution for text-to-image models, free from any noticeable flaws, and recommends it for acceptance.

For the camera-ready version, it is encouraged to incorporate the detailed responses from the rebuttal and discussions. Additionally, as suggested by reviewer xDuK, it would be beneficial to include an ablation study on the direct pasting of object images from the object branch into the noisy images of the main branch. We look forward to seeing this presentation at NeurIPS.

Reviews: review: paper_summary: paper_summary: This paper proposes a method to use a pre-trained text-to-image
diffusion model to guide the generation into placing objects at given
locations determined by bounding boxes. The challenge is to develop
this capability without requiring fine-tuning of the model.
strength_weakness: strength_weakness: - The quality of the presentation can be improved. For instance, one
  fundamental component of the method, $\math{L}_{AGG}$, is not
  defined. The evaluation metrics (spatial, size, color) are not
  properly defined, unless I missed it.
- Further analysis or discussion on why the proposed approach is
  superior to simply directly transferring the patch would make the
  paper more convincing.
- Experimental comparison is limited to two datasets. Qualitatively,
  it's not entirely clear there is a significant difference between
  the R&B baseline and the proposed approach.
questions: questions: See weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This paper explores the zero-shot layout-to-image generation problem using bounding boxes and texts as conditional inputs. The authors propose a method called GrounDiT, which builds upon the recent Diffusion Transformers (DiT) model. Leveraging DiT's emergent property of semantic sharing, where two noisy images of different sizes can progressively converge during sampling, the authors introduce a key innovation: a separate object denoising branch for each bounding box condition, running in parallel to the main denoising branch. In each object branch, a noisy object patch is generated and then implanted into the original image. The authors compare GrounDiT with SOTA zero-shot methods on the HRS and DrawBench datasets, demonstrating that GrounDiT can generate complete objects in the correct regions while maintaining high overall image fidelity.
strength_weakness: strength_weakness: - The paper lacks a comparison of prompt fidelity on the HRS dataset. Except for PixArt-R&B( a variant of GrounDiT), it would also be useful to include results from other baselines like R&B and BoxDiff to ensure the completeness of the work, even if they may use different backbones.

- In Table 1, GrounDiT outperforms its variant PixArt-R&B in grounding accuracy, which is expected given the special design of noisy object patch cultivation and transplantation. However, in Table 2, GrounDiT continue to surpass PixArt-R&B in prompt fidelity scores like CLIP score and ImageReward. I am curious to know where these improvements come from, as object patch cultivation and transplantation should only benefit grounding ability. Did the authors perform multiple runs for the experiments?

- The paper does not evaluate the MS-COCO subset presented in the R&B paper, which would be a valuable addition to the experiments.

- The proposed method would increase the compuation cost for image generation. However, there is no quantitative evidence showing the exact cost.
questions: questions: Please refer to the weaknesses part.
limitations: limitations: In the conclusion part, the authors admit that the proposed method would increase the compuation time for image generation.
paper_summary: paper_summary: The paper targets exploiting a pre-trained PixArt-alpha (a text-to-image multi-aspect diffusion transformer) to generate images conditioned on a set of text-labeled bounding boxes.

The authors start from the idea that, at inference time, a transformer can be given an arbitrary number of tokens, and that these tokens can have positional embedding corresponding to multiple images of multiple aspect ratios at once. Empirically, they observe that in this situation, the diffusion transformer is able to handle the generation process consistently and the outputs are multiple images with shared content but with different aspect ratios corresponding to the input PEs.

When generating the content of the bounding boxes, they take advange of this idea by simultaneously generating two version of the same bounding-box, one with positional embeddings corresponding to its location in the full image, and the other one with positional embeddings making the crop appear as a full image. 
This ensures that the label is represented in the full-image version, and thanks to the shared content, also in the original box version, that can then be transplanted back into the image.
strength_weakness: strength_weakness: Mainly, I find that the exposition of the idea not very clear and could be improved:
- How the different branches, the two-stage process, and the diffusion scheduling interact is not clear:
  - Appendix A.1 reveals that Stage 1 and 2 are merged in a single full 50 steps denoising schedule, with Stage 1 only being used in the 25 first steps, which is not exactly in line with what is presented in the main paper. This information needs to be in the main paper, and ideally the presentation should be made more consistent with the implementation.
  - Figure 3 and Line 200 suggest that in the main branch x_t is denoised in parallel to the object branch, but it does not appear in Algorithm 1. This is inconsistent. And if the main brain does anything, it needs to be explained and detailed.
- Section 3 should present how multi-aspect/size is handled in diffusion transformers, as it is a key requirement for the proposed method. Also, Eq.3 introduces $w$ without definition. 
- a smaller point that could be improved: in 4.2, while it can be derived from context and references that $\mathcal{L}$ needs to be a grounding loss, this is nowhere explicitly stated. If I strictly adhered to what is written in this paragraph, the authors could very well be arguing that any loss function that takes as input $(x_t, \mathcal(G)_i, \mathcal(A)_i)$ would be effective, which is obviously not what they are trying to say.
questions: questions: I find this submission to be very interesting and show a lot of promise, but the description of the method is very confusing. For the discussion period, I hope the authors can provide the necessary clarifications and leave enough time for us to have a discussion together around their updated description.
Indeed, until the main points are convincingly clarified, succintly reviewed again, and we are confident that changes will be made, I cannot recommend acceptance. My main question is then:
- Can the authors clarify if anything happens in the main branch? Perhaps a full description of how the method works, including the reverse diffusion iterations would help.

Also, as a comparatively minor point, diffusion models are known for the flexibility of their sampling process. Considering how each stage and branch have compatible objectives, have the authors considered doing every update simultaneously in a single reverse diffusion process instead of multi-stage?
limitations: limitations: Limitations and impacts are discussed adequately.
paper_summary: paper_summary: The paper introduces a zero-shot layout technique for spatially-grounded text-to-image generation with the power of transformer-based diffusion architecture. Prior methods tend to manipulate the latent image during reverse process for grounding, solely relying on cross-attention maps that show the alignment between specific text tokens and spatial regions of the target object to place in. This shows a drawback in generating a target object that fits within the region of bounding boxes. By utilizing transformer architecture, the method can divide the generation process into two branches: (1) first branch is to generate a target image patch of the bounding box with an arbitrary size by a transformer, (2) the second branch is the original generation process of the whole image. At every denoising step, the generated object is then copied into the generated image of the second branch.
strength_weakness: strength_weakness: - 158-161: It is obvious that different noise inputs certainly yield different outputs. So I do not understand the main point of this part. 
- Semantic sharing part is an interesting observation. It is more helpful if the authors compare it with image interpolation. Does it work when two images have different semantic classes like dog and human?
Wang, Clinton, and Polina Golland. "Interpolating between images with diffusion models." (2023). 
- Eq 5: What is formulation of $ \mathcal{L}(x_t, G_i, A_i)$?
- In reverse process: Have the authors applied semantic sharing on bounding boxes like concat tokens of multiple objects for simultaneous denoising? I assume that they share the same semantic class. 
- In Fig 3, I saw the size of object image is much bigger than the size of bounding box. Is it intentional? If so, please explain. How to define noisy object image $z_{i, t}$? As far as I understand, the method need to reserve the aspect ratio of the bounding box by scaling it correspondingly to match the size of main image. Is it possible to disable this scaling step and denoise an object image with the same size as the bounding box?
- Does Noisy Patch transplantation guarantee the consistent content of the output image when pasting the generated image  with a binary mask (Eq. 9)? Again in Eq 9, the size of $x_{t-1}$ and $B_{i, t-1}$ are not the same as illustrated in Figure 3 so it is not clear how Hadamard product is possible.
questions: questions: NA
limitations: limitations: Limitations and societal impacts are included in the paper.

Comments: comment: Thank you for taking the time to review our submission and rebuttal. We greatly appreciate your valuable feedback and will work diligently to improve our work based on your insights.
Thanks for the clarifications. After reading all the reponses and other reviewers' comments, I am inclined to keep my original score.
Thank you sincerely for taking the time to review our submission. We greatly appreciate your valuable feedback and will work to improve our work based on your suggestions.
Thanks for the detailed response, it helps address my concerns. Most of my concerns are about the exposition and description of the method which is the main issue of the paper as can be seen in other reviewers. Apart from the writing, the additional benchmarks clearly demonstrate the effectiveness of the method against other baselines as seen from other reviewers. Hence, I still keep my initial score and vote for acceptance. Meanwhile, I encourage the authors to include the ablation of direct pasting object image of the object branch into noisy image of the main branch. This is a good evidence to support the claim in global response.
We sincerely appreciate your time and effort in reviewing our submission and rebuttal. Your valuable feedback is greatly appreciated, and we will work to improve the clarity of our work accordingly.
I have read the other reviews and rebuttal, and I thank the authors for all the clarifications to the algorithm.
While the initial submission lacked in terms of preciseness and clarity, I believe the content of the rebuttal fixes the main issues and demonstrate the willingness of the authors to update the submission accordingly.

As such, standing by my initial assessment that the proposed method is a novel, clever, and inspiring solution to a relevant problem in text-to-image models, and now with no major unadressed concerns, I updated my evaluation and recommend acceptance.