GroundingBooth: Grounding Text-to-Image Customization

## Abstract

Abstract Recent studies in text-to-image customization show great success in generating personalized object variants given several images of a subject. While existing methods focus more on preserving the identity of the subject, they often fall short of controlling the spatial relationship between objects. In this work, we introduce GroundingBooth, a framework that achieves zero-shot instance-level spatial grounding on both foreground subjects and background objects in the text-to-image customization task. Our proposed text-image grounding module and masked cross-attention layer allow us to generate personalized images with both accurate layout alignment and identity preservation while maintaining text-image coherence. With such layout control, our model inherently enables the customization of multiple subjects at once. Our model is evaluated on both layout-guided image synthesis and reference-based customization tasks, showing strong results compared to existing methods. Our work is the first work to achieve a joint grounding on both subject-driven foreground generation and text-driven background generation. The project page is available at https://groundingbooth.github.io ..

## 1 Introduction

Text-to-image customization, also known as subject-driven image generation or personalized text-to-image synthesis, is a task that requires a model to generate diverse variants of a subject given a set of images of the target subject. Text-to-image customization has achieved significant progress during the past few years, allowing for more advanced image manipulation.
For example, the test-time-finetuning based methods like Dreambooth (Ruiz et al., ), Textual Inversion (Gal et al., ), and Custom Diffusion (Kumari et al., ) use a few images of the same object to finetune a pretrained diffusion model and generate variants of the object from input prompts. The encoder-based methods such as ELITE (Wei et al., ) and InstantBooth (Shi et al., ) eliminate test-time-finetuning by learning a generalizable image encoder and attention modules. Despite their success, existing personalization methods mainly focus on generating identity-preserved images from the input prompt and fail to accurately describe the spatial relationship of objects and backgrounds. In real-world scenarios of image customization, it is a crucial user need to achieve fine-grained and accurate layout control on each of the generated objects for more flexible image manipulation.

To tackle this issue, we propose to deal with a more challenging task, grounded text-to-image customization, which extends the existing text-to-image customization task by enabling grounding controllability over both the foreground subjects and background objects. Specifically, under this new setting, the inputs usually include a prompt, images of subjects, and optional bounding boxes of the subjects and background text entities. The model aims to generate text-aligned background objects and identity-preserved foreground subjects, while the spatial location of all the grounded objects and subjects are exactly aligned with the input bounding boxes. It is non-trivial to achieve all these effects simultaneously, as we are indeed handling multiple tasks together and it is challenging for the model to be generalizable to all sub-tasks.

There are a few related works to handle our new task, which, however, show significant limitations. On the one hand, although existing grounded text-to-image diffusion models such as LayoutDiffusion (Zheng et al., ) and GLIGEN (Li et al., ) have made attempts at spatial controllability, they cannot achieve identity preservation of the subjects. On the other hand, subject-driven image generation methods mainly focus on the identity preservation of the reference objects, while limited attempts have been made on layout control of either subjects or background objects. There is another line of related works (Chen et al., ; Song et al., ; ) that achieve customized image composition. They can control the location of the input subject under the image composition setting but are neither able to achieve text-to-image synthesis nor control the spatial location of the background contents.

To fully address our new task, we propose GroundingBooth, a general framework for grounded text-to-image customization. Specifically, based on a pretrained text-to-image model, we build a new joint text-image grounding module that encourages both the foreground subjects and background objects to accurately follow the locations indicated by the input bounding boxes. To further enhance the identity preservation of the subjects, we propose a masked cross-attention layer in the transformer blocks of the diffusion U-Net, which helps to disentangle the subject-driven foreground generation and text-driven background generation in each block, effectively preventing the false blending of multiple visual concepts in the same location and enforcing the generation of clear subjects. As shown in Fig. 1, with such dedicated designs of the model structures, our framework not only achieves grounded text-to-image customization with a single subject (Fig. 1 (a)), but also supports multi-subject customization (Fig. 1 (b)), where users can input multiple subjects along with their bounding boxes, and our model can generate each subject in the exact target region with identity preservation and scene harmonization. Meanwhile, our model also allows for the grounding of multiple background objects (Fig. 1 (b)).

The key contributions of this work can be summarized as follows.

- •
We propose a general framework, GroundingBooth, that achieves grounded text-to-image customization. Specifically, our model achieves joint layout grounding of both the foreground subjects and the text-guided background, while maintaining accurate identity of the subjects. Furthermore, our model enables the customization of multiple subjects.
- •
We propose a novel layout-guided masked cross-attention module, which disentangles the foreground subject generation and text-driven background generation through cross-attention manipulation thus avoiding false context blending.
- •
Experiment results show the effectiveness of our model in text-image alignment, identity preservation, and layout alignment.

## 2 Related Work

### 2.1 Subject-driven Image Generation

Subject-driven image generation, also known as personalized text-to-image generation or image customization, aims to generate target images based on customized objects and a text prompt that describes the target context (Chen et al., ; Pan et al., ; Xiao et al., ; Wang et al., ). In this task, the specific identity of the input reference images is defined as a subject or a concept. So far, existing image customization works can be categorized into three major types. The first type is test-time-finetuning methods (Ruiz et al., ; Gal et al., ; Kumari et al., ). These methods first finetune a pretrained diffusion model on a few subject images so that the model is adapted to a new identifier token representing the new concept. Then they generate new images from prompts containing the identifier. Such test-time finetuning is computationally intensive. The second type of method is encoder-based customization methods (Arar et al., ; Wei et al., ; Shi et al., ; Zhang et al., ), which eliminates the test-time finetuning by learning a generalizable diffusion model that can adapt to new subjects on a large-scale training data. The generalizable model usually contains image encoders that map the subject images into dense tokens and attention modules that integrate vision tokens with text tokens. These methods can achieve much faster image customization during inference, while rely heavily on large-scale multi-view training data, and identity preservation may not be perfect in out-of-distribution cases. The third type of methods (Roich et al., ; Gal et al., ) is a combination of the first two methods, which first learn an image encoder to extract identity tokens of the input subject and then finetune the model on the subject images for a few steps.

Note that most existing subject-driven image generation methods focus on synthesizing personalized image variants from prompts. They show quite limited performance in controlling the layout of the generated scenes and modeling the spatial relationship between objects. On the contrary, our model performs well not only in generating identity-preserved, text-aligned images but also in controlling the layout of both the subjects and background. A recent related work Anydoor (Chen et al., ) can also control the location of the subject. We differ significantly from this work in many aspects. First, Anydoor aims at image composition, while our goal is for text-to-image generation. Second, Anydoor can only control the location of the subject, while our work controls the layout of all the objects and regions in the foreground and background.

### 2.2 Grounded Text-to-Image Generation

Given a layout containing bounding boxes labeled with object categories, grounded text-to-image generation aims to generate the corresponding image, which is the reverse object detection process. Traditional grounded text-to-image generation such as LostGAN (Sun & Wu, ), LAMA (Li et al., ) and PLGAN (Wang et al., ) are based on generative adversarial networks(GANs). Recently, diffusion-based methods (Rombach et al., ; Zheng et al., ; Li et al., ; Zhang et al., ; Wang et al., ) have made attempts to add layout control for image generation. For example, LayoutDiffusion (Zheng et al., ) uses a patch-based fusion method. GLIGEN (Li et al., ) injects grounded embeddings into gated Transformer layers. ControlNet (Zhang et al., ) uses copied encoders and zero convolutions. InstanceDiffusion (Wang et al., ) allows for multiple formats of location control. However, all these methods can only perform text-to-image generation, while object-guided generation and identity preservation cannot be achieved. In contrast, our model achieves satisfactory identity preservation on reference-guided image generation.

Figure: Figure 2: An overview of our GroundingBooth model. The whole pipeline contains two steps: (1) Feature extraction. We use the CLIP encoder and DINOv2 encoder to extract text and image embeddings, respectively. We use our proposed grounding module to extract the grounding tokens. (2) Foreground-background cross-attention control in each transformer block of U-Net. During training, we use datasets with a single subject as the reference image and only trains a single masked cross-attention layer per transformer block. During inference, our model supports the generation of multiple subjects in their corresponding locations by reusing the same masked cross-attention layer for each subject. This figure shows the inference pipeline of our model. We show the details of the grounding module and masked cross-attention layer in Fig. 3.
Refer to caption: x2.png

## 3 Our Approach

Our model is built upon Stable Diffusion v1.4 Rombach et al. ().
Given one or a few background-free(^1^11Background-free images refer to images with background removed. They can be easily obtained by segmentation methods such as SAM (Kirillov et al., ) or SAM2 (Ravi et al., )) reference images $\mathcal{X}=\{x_{1},x_{2},\cdots,x_{m}\}$ where $x_{m}\in\mathbb{R}^{h\times w\times 3}$ with their target bounding box locations $\mathcal{L}_{X}=\{l_{X}^{1},l_{X}^{2},\cdots,l_{X}^{m}\}$, text entities(^2^22Here each text entity is referred to a text tag, such as “chair” and “hat”.) $\mathcal{T}=\{t_{1},t_{2},\cdots,t_{n}\}$ with their target locations $\mathcal{L}_{T}=\{l_{T}^{1},l_{T}^{2},\cdots,l_{T}^{n}\}$, and the overall prompt $\mathcal{P}$, we aim to generate a customized image $\hat{x}$, where both the reference objects can be seamlessly placed inside the desired bounding box with natural poses and accurate identity and the background objects generated from text-box pairs are positioned at the correct location. Here $l_{X}^{m}$ or $l_{T}^{n}$ refers to the bounding box coordinates of a reference object or a text entity, which can be represented as ${l}=\left[x_{\min},y_{\min},x_{\max},y_{\max}\right]$. The reference object should be generated harmoniously with the background. The customized image $\hat{x}$ can be calculated as:

$$ $\hat{x}=\operatorname{GroundingBooth}(\mathcal{X},\mathcal{T},\mathcal{P}, \mathcal{L}_{X},\mathcal{L}_{T}).$ (1) $$

The pipeline of our proposed GroundingBooth model is shown in Fig. 2. Our work is the first attempt that enables precise spatial grounding in the customized image synthesis task, which jointly controls the size and location for both the reference-guided foreground objects and text-driven background regions. Moreover, our work adaptively harmonizes the poses of the reference objects and faithfully preserves their identity. In this section, we first introduce our feature extraction pipeline in Sec. 3.1, then introduce the foreground-background masked cross-attention control in Sec. 3.2. Finally, we introduce the training and inference pipeline in Sec. 3.3 and Sec. 3.4, respectively.

### 3.1 Feature Extraction

#### Feature Extraction of Text and Reference Images

We first extract text tokens from the input prompt using the CLIP text encoder and identity tokens from the reference images using DINOv2 (Oquab et al., ). For each reference image, we extract 257 identity tokens which are composed of a global image class token and 256 local patch tokens. We reshape the feature dimension of each image token to 768 through a linear projection layer.

Figure: Figure 3: Modules of our proposed framework: (a) Grounding Module: Our grounding module takes both the prompt-layout pairs and reference object-layout pairs as input. For the foreground reference object, both CLIP text token and the DINOv2 image class token are utilized. (b) Masked Cross-Attention: Q, K, and V are visual query, key, and value respectively, and A is the affinity matrix.
Refer to caption: x3.png

#### Grounding Module

To control the layout of the foreground and background objects, we propose a grounding module, which jointly ground text and image features through positional encoding. Fig. 3(a) shows the overall structure of our grounding module. We extract grounding information based on the joint guidance of the tagged text-box pair and the object-layout pair. Specifically, it contains two branches: 1) In the text entity branch, the bounding boxes of the background objects $\mathcal{L}_{T}$ are passed through a Fourier encoder to obtain the text Fourier embeddings for the text entities, which are then concatenated with the text tokens in the feature space to obtain the grounded text embeddings. 2) In the reference image branch, the bounding boxes of the reference objects $\mathcal{L}_{X}$ (in the target image) are also passed through a Fourier encoder to extract the object Fourier embeddings, which are then concatenated with the reference image tokens to obtain the grounded reference image embeddings. For all training images, we set a max number of boxes and the text entities and drop the rest ones. For the cases where there is no reference image or text entities, we set the input reference object layout to zero and reference object token to zero embeddings, or set the grounded text embeddings to zero embeddings, respectively. At the end of the following two branches, the grounded text embeddings and reference image embeddings are reshaped back into the original feature dimension through linear layers and then concatenated in the token space to form the final grounding tokens. Given the text entities $\mathcal{T}$ and reference images $\mathcal{X}$, the calculation of the grounding tokens is formulated as:

$$ $h^{(\mathcal{T},\mathcal{X})}=\left[\textit{MLP}\left(\psi_{\text{text }}( \mathcal{T}),\textit{Fourier}(\mathcal{L}_{T})\right),\textit{MLP}\left(\psi_{ \text{obj }}(\mathcal{X}),\textit{Fourier}(\mathcal{L}_{X})\right)\right],$ (2) $$

where Fourier represents the Fourier embedding (Tancik et al., ), $MLP(.,.)$ is a multi-layer perceptron, $[.,.]$ is concatenation operation, and $h^{(\mathcal{T},\mathcal{X})}$ is the grounding token. $\psi_{text}$ and $\psi_{obj}$ denote to the text encoder and image encoder, respectively. The generated grounding token ${h}^{(\mathcal{T},\mathcal{X})}$ contains the location features of both the reference objects and the text entities. It is then injected into the U-Net layers of the diffusion models. Specifically, inspired by GLIGEN (Li et al., ), we inject the grounding token through a gated self-attention layer located between the self-attention layer and cross-attention layer in each Transformer block of the U-Net, represented as:

$$ ${v}={v}+\tanh(\gamma)\cdot\left(\operatorname{SelfAttn}\left(\left[{v},{h}^{( \mathcal{T},\mathcal{X})}\right]\right)\right),$ (3) $$

where $\gamma$ is a learnable scalar initialized as 0 0, ${h}^{(\mathcal{T},\mathcal{X})}$ is the grounding token and $v$ is the output of the self-attention layer. During training, the model adaptively learns to adjust the weight $\gamma$ of the grounding module, which ensures stable training and balances the weight between the grounding token and the visual features.

### 3.2 Foreground-Background Cross-attention Control

Previous text-to-image generation methods usually directly concatenate the text and image tokens in the cross-attention layers, leading to two drawbacks. First, the reference objects and the text-driven background objects can be blended unnaturally. Second, for the circumstances where bounding boxes belong to the same class, the model cannot distinguish whether a bounding box belongs to a reference object or text entity, resulting in the misplacement of the reference object. To solve these issues, we propose a novel masked-cross attention module to separately generate the foreground objects and background contents. Moreover, when there are multiple reference objects, our module can clearly maintain the independence of generating each foreground object. The details of our module is illustrated in Fig. 3(b).

The original cross-attention layer can be formulated as:

$$ $f=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V,$ (4) $$

$$ $Q=\phi_{Q}(f),K=\phi_{K}\left(f_{c}\right),V=\phi_{V}\left(f_{c}\right),$ (5) $$

where $\sqrt{d}$ is the scaling factor that is set as the dimension of the queries and keys, $f$ denotes to the vision feature in the layer, $f_{c}$ refers to the embedding of the input condition. $\phi_{Q}$, $\phi_{K}$, $\phi_{V}$ are the linear layers to project the features into queries, keys and values, respectively.

In our masked cross-attention layer, both the DINOv2 image tokens and the layout of the reference object are taken as input. The queries $K$ and values $Q$ are calculated from the image tokens. We first compute the affinity matrix $A$ through $A=Q\cdot K$ and get $A\in\mathbb{R}^{hw\times hw}$, where $h\times w$ indicates the resolution of the feature map in the attention layer. As we have the object layout $l_{obj}$, it is straightforward to restrict the injection of image tokens only inside the region of the target bounding box. Therefore, we reshape the layout $l_{obj}$ to $h\times w$ and generate the cross attention mask, which is formulated as:

$$ $M_{Layout[i,j]}=\left\{\begin{array}[]{ll}0,&[i,j]\in l_{obj},\\ -\infty,&[i,j]\notin l_{obj}\end{array}\right.,$ (6) $$

where $M_{Layout[i,j]}$ represents the pixel of position $[i,j]$ in rectified attention score maps, $l_{obj}$ represents the layout region of the reference object in the feature map.

The mask contains the location information for restricting the reference object placement and avoiding information leakage. After acquiring the mask, we apply dot product operation between the feature maps and the layout to constrain the object generation and obtain the mask-rectified affinity matrix $A^{\prime}$ through $A^{\prime}=A+M_{Layout}$. Then we multiply the masked affine matrix $A^{\prime}$ with $V$ to obtain the layout-guided masked cross-attention output $f_{obj}$. The whole masked cross-attention module is formulated as:

$$ $f_{obj}=\operatorname{softmax}\left(\frac{QK^{T}+M_{Layout}}{\sqrt{d}}\right)V.$ (7) $$

For the scenarios where there is a lack of reference objects, $M_{Layout}$ is set to all 0 0, the masked-cross attention degrade into normal cross attention. Through masked cross-attention control, the injection of each reference object feature is restricted to be within the corresponding bounding box area. This ensures not only the independence between the generation of foreground and background, but also the independence of multiple reference objects. Our module prevents information leakage and ensures an accurate layout-guided subject generation.

### 3.3 Model Training

During training, for each image, we input only one subject image and its bounding box to the model, along with several text entities with their corresponding bounding boxes. The number of entities per training image is limited to 10 and we drop the rest ones. For a portion of training cases, the input may not contain subject image or text entities. We keep the text encoder and DINOv2 image encoder frozen and merely fine-tune the gated self-attention layers, the masked cross-attention layers, and the multi-layer perceptron after the DINOv2 image encoder.

### 3.4 Model Inference

Although our model is trained on single-subject data, it can be seamlessly extended to achieve multi-subject customization without retraining. As shown in Fig. 2, in the inference stage, assume we have $n$ reference objects, the reference object and paired layout information are concatenated as the grounded reference image embeddings. In each transformer block, the masked cross-attention layer will be reused for $n$ times, and each ID token and its paired layout information are injected into each masked cross-attention layer respectively. As we analyzed in section 3.2, our masked cross-attention ensures the independence of the generation of each subject, preventing potential false blending of visual concepts, e.g., the unnatural blending of two objects in the overlapping regions. It also guarantees an accurate layout control on all the subjects.

## 4 Experiment

### 4.1 Dataset

The training data of our experiments are from both multi-view datasets and single-image datasets. For multi-view data, we use MVImgNet (Yu et al., ), which contains 6.5 million frames from 219,188 videos across 238 object categories, with fine-grained annotations of object masks. In the data processing stage of MVImgNet, following AnyDoor (Chen et al., ), for each object, we randomly selected two different frames from the same video clip to form a training pair. We apply the object mask on one frame to obtain the background-free object as the input reference object. For the other frame, we use the bounding box of the object as the grounding information and use this frame as the training ground truth. For single-image data, we use LVIS (Gupta et al., ), a well-known dataset for fine-grained large vocabulary instance segmentation, including 118,287 images from 1,203 categories. For each sample, we select only the object instances with top-10 largest bounding box area as training data.

### 4.2 Evaluation Metrics

We calculate the CLIP-I (Radford et al., ) score and DINO (Caron et al., ) score to evaluate the identity preservation performance of the subjects and use CLIP-T (Radford et al., ) score to evaluate the text alignment performance of the generated image. For evaluation of the model’s grounding ability, we use $AP_{50}$ based on a pretrained YOLOv8 (Jocher et al., ) object detection model.

Figure: Figure 4: Visual comparison with existing methods on DreamBench objects for the single-subject customization task. Please zoom in to see the details.
Refer to caption: x4.png

Figure: Figure 5: Multi-concept customization on DreamBench objects. Please zoom in to see the details.
Refer to caption: x5.png

### 4.3 Single Subject Customization

We compare our work with existing state-of-the-art works on DreamBench (Ruiz et al., ) for the customization of a single subject. In this experiment, we use the bounding box of the subject in the ground-truth image as the input layout. The qualitative and quantitative results are shown in Fig. 4 and Table 1, respectively. Overall, our method shows significantly better performance in layout alignment, reference object identity preservation, and background text alignment. Existing encoder-based subject-driven text-to-image customization methods BLIP-Diffusion (Li et al., ), ELITE (Wei et al., ), $\lambda$-eclipse (Patel et al., ) and MLLM-based method Kosmos-G (Pan et al., ) fail to maintain accurate identity of the reference objects lack the ability for precise layout control. AnyDoor (Chen et al., ) is designed for image composition on a given background and lacks the ability of text-to-image generation. Although previous grounded text-to-image generation methods like GLIGEN (Li et al., ) are able to achieve layout control, it cannot preserve the identity of the subjects. CustomNet (Yuan et al., ) achieves flexible pose control. However, it highly relies on the pretrained model Zero123 (Liu et al., ), limiting the resolution of its generated image to be $256\times 256$. Moreover, there can be obvious artifacts around the boundary of the generated subject.

**Table 1: Comparison with existing methods on Dreambench.**
|  | CLIP-T ↑ | CLIP-I ↑ | DINO-I ↑ |
| --- | --- | --- | --- |
| SD V1.4 [(Rombach et al., )] | 0.3122 | 0.8413 | 0.6587 |
| BLIP-Diffusion [(Li et al., )] | 0.2824 | 0.8894 | 0.7625 |
| ELITE [(Wei et al., )] | 0.2461 | 0.8936 | 0.7557 |
| Kosmos-G [(Pan et al., )] | 0.2864 | 0.8452 | 0.6933 |
| lambda-eclipse [(Patel et al., )] | 0.2767 | 0.8973 | 0.7934 |
| AnyDoor [(Chen et al., )] | 0.2430 | 0.9062 | 0.7928 |
| GLIGEN [(Li et al., )] | 0.2898 | 0.8520 | 0.6890 |
| CustomNet [(Yuan et al., )] | 0.2815 | 0.9090 | 0.7526 |
| Ours | 0.2881 | 0.9146 | 0.7884 |

As an interesting observation, we find that previous non-grounding based customization methods are inclined to generate objects that are very large and in the center of the image, which gains benefit in CLIP-I score and DINO score during evaluation. However, in real-world scenarios, users may expect to flexibly control the size of the subject in the generated images. They may choose to generate larger background with broader textual information, where, in such cases, non-grounding customization methods cannot generate the desired result. The visual results in Fig. 4 demonstrate that our results achieves better identity preservation performance with accurate layout-alignment. We encourage the readers to view more visualizations in the Appendix.

### 4.4 Multi-Subject Customization and Multi-Entity Background Generation

With our proposed masked cross-attention module, our model seamlessly supports the customization of multiple subjects. Fig. 5 shows the qualitative results of the task where we customize multiple subjects and generate the image by grounding multiple text entities in the background. It can be observed that when inputting multiple subjects such as a bag and a boot, along with the layout of the background text entities such as the mountain and the lake, our model successfully generates the subjects and background with an accurate layout-alignment of each visual concept. The generated subjects preserve the their identity well. The overall generated image is well text-aligned and artifact free. Moreover, in several cases, even when the bounding boxes of the foreground objects have a large overlap with the background text entities, the model can distinguish subject-driven foreground generation from text-driven background generation, effectively avoiding the context blending.

Figure: Figure 6: Visaul results of reference-guided image generation with complex layout and text entities as conditions on COCO validation set. Note that LayoutDiffusion (Zheng et al., ) is only conducted on COCO dataset with filtered annotations, so some of its results are not available.
Refer to caption: x6.png

### 4.5 Customization with Complex Layout and Text Entities

We evaluate our model’s performance the COCO validation set for the task of generating customized images with complex layout and text entities as guidance. Quantitative and qualitative results are shown in Table 2 and Fig. 6, respectively. For each testing image, we use the largest object as the reference object (i.e., the subject), and the remaining text entities as background entities. To quantify the model’s grounding ability, we adopt YOLOv8 (Jocher et al., ) as the object detection method. Results show that even if we input complex layouts and text entities to the model, our model can still generate high-quality scenes with precise layout alignment of all the objects and regions, and accurate identity preservation of the reference object, while preserving the text-alignment. Compared with previous layout-to-image generation methods, our model has a competitive accuracy in grounding the visual concepts and remarkable improvement on identity preservation.

**Table 2: Quantitative results on MS-COCO validation set for the task of customized image generation with complex layout as guidance. In this setting, we finetune our model on COCO training set, and compare with previous methods that only train on COCO.**
|  | CLIP-T ↑ | CLIP-I ↑ | DINO-I ↑ | $AP_{50}$ ↑ |
| --- | --- | --- | --- | --- |
| LAMA[ (Li et al., )] | 0.2507 | 0.8441 | 0.7330 | 18.20 |
| LayoutDiffusion[ (Zheng et al., )] | 0.2738 | 0.8655 | 0.8033 | 27.40 |
| UniControl[ (Qin et al., )] | 0.3143 | 0.8425 | 0.7598 | 4.53 |
| GLIGEN[ (Li et al., )] | 0.2899 | 0.8688 | 0.7792 | 27.50 |
| Ours | 0.2946 | 0.9078 | 0.8560 | 31.10 |

### 4.6 Ablation Study

We conduct the ablation study to validate the effectiveness of our proposed components: the masked cross-attention module and the grounding module. Table 4 and Table 4 present the quantitative results on DreamBench and COCO, respectively. We observe that both the grounding module and the masked cross-attention module play a vital role in the model’s grounding ability and prevent the information leakage of the reference object. Benefiting from these two modules, the model shows stronger ability of identity preservation, text alignment and grounded generation.

**Table 3: Ablation Study for modules on Dreambench.**
|  | CLIP-T ↑ | CLIP-I ↑ | DINO-I ↑ |
| --- | --- | --- | --- |
| w/o Grounding Module | 0.2762 | 0.8578 | 0.7049 |
| w/o Masked Cross-Attention | 0.2878 | 0.8616 | 0.7065 |
| Full | 0.2881 | 0.9146 | 0.7884 |

**Table 5: User Study based on DreamBench: In the questions, the user is presented side-by-side comparisons of our generated image and another image randomly chosen from one of the baselines. The results in the table show user preference percentage.**
|  | Ours | CustomNet | Ours | AnyDoor | Ours | GLIGEN |
| --- | --- | --- | --- | --- | --- | --- |
| Identity | 60.78 | 39.22 | 59.31 | 40.69 | 72.81 | 27.19 |
| Grounding | 56.86 | 43.14 | 64.21 | 35.79 | 58.25 | 41.75 |
| Text Alignment | 51.96 | 48.03 | 73.52 | 26.47 | 55.34 | 44.66 |
| Overall Quality | 54.41 | 45.58 | 62.25 | 37.74 | 58.74 | 41.26 |

### 4.7 User Study

In Table 5, we show the user study results comparing our model with existing models (Chen et al., ; Yuan et al., ; Li et al., ) on DreamBench. Specifically, given the same input, we generate results with each model. Then we ask the users to make side-by-side comparison of our result and a randomly chosen result from the baselines regarding identity preservation, text alignment, grounding ability, and overall image quality. We collect the user responses using Amazon Mechanical Turk. Results show that participants have significantly higher preference on our method. We show details about user study in the Appendix Sec. D.

## 5 Conclusion and Future Work

We presented GroundingBooth, a general framework for the grounded text-to-image customization task. Our model has achieved a joint grounding for both reference images and prompts with precise object location and size control while preserving the identity and text-image alignment. Our strong results suggest that the proposed text-image feature grounding module and the masked cross-attention module are effective in reducing the context blending between foreground and background. We hope our research can motivate the exploration of a more identity-preserving and controllable foundation generative model, enabling more advanced visual editing.

Although our model successfully generates customized images with layout control, there are still several limitations. First, the model’s performance can be limited by the base model. We can address this by using a stronger base model. Second, the design of reusing the masked cross-attention layer for each subject could still be time-consuming during inference. This can be addressed by developing a parallel generation structure for multiple subjects. We leave this direction in future work.

## Appendix

## Appendix A Preliminary

Our model is based on Stable Diffusion v1.4 Rombach et al. (), a Latent Diffusion model (LDM) that applies the diffusion process in a latent space.
Specifically, an input image $x$ is encoded into the latent space using a pretrained autoencoder $z=\mathcal{E}(x),\hat{x}=\mathcal{D}(z)$ (with an encoder $\mathcal{E}$ and a decoder $\mathcal{D}$). Then the denoising process is achieved by training a denoiser $\epsilon_{\theta}\left(z_{t},t,f_{c}\right)$ that predicts the added noise following:

$$ $\min_{\theta}E_{z_{0},\epsilon\sim\mathcal{N}(0,1),t\sim\mathrm{U}(1,T)}\left \|\epsilon-\varepsilon_{\theta}\left(z_{t},t,f_{c}\right)\right\|_{2}^{2},$ (8) $$

where $f_{c}$ is the embedding of the condition (such as a prompt) and $z_{t}$ is the latent noise at timestamp $t$.

## Appendix B Training/Inference Details

Our model is trained on 4 NVIDIA A100 GPUs for 100k steps with a batch size of 14 and a learning rate of $5\times 10^{-5}$. During training, we randomly drop reference image embedding and text embedding both at the rate of 10%. We decently rank the area of the boxes per images, and set the max number of grounding boxes to be 10 with the largest areas. During inference, we set classifier-free guidance(CFG) (Ho & Salimans, ) as 3.

## Appendix C Details about Data Collection

For each reference image, we use the segmentation mask to mask out the background and get the background-free reference object. In inference stage, we use SAM (Kirillov et al., ) to get the mask of the reference object, and get the background-free reference object.

## Appendix D Details about User Study

Our user study is based on DreamBench, with full 30 objects and 25 prompts. We randomly generated layouts, and use them in the generation. In the user study, given the layout, the reference object, the text prompt, the result of our method and a random-selected baseline method, we request the user to answer the following four questions:

(1) Which generated image do you think that its object is more similar to the input object? Choose between Option A and B.

(2) Which generated image do you think that its object is most likely to be at the right position as the input layout? Choose between Option A and B.

(3) Which generated image do you think is most likely to match the text description? Choose between Option A and B.

(4) Which image do you think has better image quality? Choose between Option A and B.

We received more than 1200 votes from over 530 users. In the experiment, we randomly shuffle the order of baselines to improve the confidence of the user study.

## Appendix E Additional Qualititive Results

Here we show more qualitative results. In Fig. 7 we show results on DreamBench and in Fig. 8 and Fig. 9 we show more results about complex background background evaluation on coco validation set.

Figure: Figure 7: More visual results of our model.
Refer to caption: x7.png

## Appendix F Social Impact

GroundingBooth provides a flexible method for users to precisely customize the layout of both foreground and background objects based on user-provided reference subjects and text descriptions without any test-time finetuning. The support for the generation of multi-subjects provides a useful tool for users to generate images using their desired layout. Users can optionally choose reference objects or simple text inputs to generate their desired image, which significantly expand the flexibility in controllable and customized text-to-image generation.

Figure: Figure 8: More results on complex scene generation on COCO validation set.
Refer to caption: x8.png

Figure: Figure 9: More results on complex scene generation on COCO validation set.
Refer to caption: x9.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. For multi-objects cases, is each box in the layout assigned to an associated object label?
2. In your experiments, are all the layouts pre-determined or only the layout of the reference object is given?
soundness: 2
strengths: 1. The visualization results show that the proposed method can effectively preserve the identity of reference image while generating plausible images.
2. The proposed method is able to simultaneously handle multi-object synthesis even with complex layout.
summary: This paper proposes a framework which allows users to customize an image by 1) specifying the position (layout) of the object, and 2) providing a reference image of the object. It supports either single object customization or multi-object customization. They design a grounding module to ground the provided image with text entities. The produced grounding tokens are then later used as the condition in their diffusion model to generate the final image. They conduct experiments on Dreambench and MS-COCO and show that their methods could produce high quality image while preserving the detail of the user-specified (reference) images.
weaknesses: 1. My main concern is that the authors claim that they are able to ground the text entities during generation. While the CLIP-T score of the model indicates that the generated image is less coherent with the text comparing to other baseline methods.
2. While the paper claimed that they can control the spatial relationship between objects. It is difficult to evaluate this argument given the layouts are pre-determined.
3. How are the metrics computed? For example, when computing the CLIP-I score, do you only consider the image similarity between the reference object and the corresponding region in the generated image? If so, how do you extract the corresponding region? More details of how the metrics are computed (CLIP-I, DINO, CLIP-T) could improve the clarity of the paper.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: * Why does this method use SD-1.4 when there are so many newer / stronger models? Is there some limitation in using them?
soundness: 3
strengths: * The paper is well written and presented nicely
* The method improves over the baselines it does test (see first weakness)
* Such model can be useful in many real-life applications
summary: The paper proposes GroundingBooth, a model for grounded text-to-image customization. It aims to place subjects received in input (marked with bounding-boxes in images) in new backgrounds (described in the prompt), while maintaining the identity and spatial location of the subjects. The authors show GroundingBooth is capable of generating complex requests while preserving the subjects in the input images (e.g., “a [stuffed animal] and a [vase] with [plant] and [vintage lantern] on a quaint balcony”)

GroundingBooth incorporates a new Masked Cross Attention module in each block of the U-Net (Stable Diffusion 1.4’s). In addition to input from the existing Cross Attention layer, the masked layer receives as input DINO-2 features of the subject images received in the input. GroundingBooth is trained this way on a dataset curated from MVImgNet. 

Finally, the method is tested and compared to a few existing baselines, using automatic measurements such as CLIPScore and DINO, and a human study.
weaknesses: * The paper does not cover “Break-A-Scene: Extracting Multiple Concepts from a Single Image” by Avrahami et al (2023). In this work, they extract concepts from an image using textual inversion, and use it to embed them in new images. They too work with masks and can even accept them from the user as input. This is especially important since the sentence before last in the abstract states “Our work is the first work to achieve a joint grounding of both subject-driven foreground generation and text-driven background generation”, which makes this imprecise. More importantly, the difference between these projects should be clearly stated. What does this work do that Break-A-Scene does not? 

* The use of Fourier embedding should be explained. What makes it suitable to this task?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Previous research in layout-guided diffusion has demonstrated limitations in maintaining visual coherence when objects exhibit diverse textures. While these approaches often resulted in disharmonious image generation, our proposed method provides users with the capability to directly select and manipulate subjects. A comparative analysis with InstanceDiffusion would be particularly valuable, especially in terms of texture consistency and user control capabilities.
2. Due to the lack of publicly available code and data, an accurate evaluation is difficult to conduct.
3. It remains unclear why the paper emphasizes its zero-shot capability as a key strength even though the methodology clearly includes training procedures within the paper.([L37-40] & [L247-250])
soundness: 3
strengths: Unlike many existing layout-guided image generation methods that handle only single subjects, GroundingBooth supports multi-subject customization. This versatility broadens its applicability, especially for generating images where complex layouts and multiple subjects are essential.
summary: This paper introduces GroundingBooth, a novel framework designed to enhance text-to-image customization by enabling precise spatial control of both subjects and background elements based on textual prompts. While existing models in text-to-image generation maintain subject identity, they often lack control over spatial relationships. GroundingBooth addresses this gap by implementing zero-shot instance-level spatial grounding, enabling precise placement of both foreground subjects and text-defined background elements.
GroundingBooth supports complex tasks such as multi-subject customization, where multiple subjects and background entities are positioned according to input bounding boxes. Experimental results demonstrate its effectiveness in layout alignment, identity preservation, and text-image alignment, outperforming current approaches in controlled image generation.
weaknesses: 1. InstanceDiffusion does not exist in baseline comparisons. Despite its notable relevance with capabilities for free-form language conditions per instance and flexible instance localization methods (single points, scribbles, and bounding boxes), InstanceDiffusion is missing from both our quantitative and qualitative baselines. 
2. FID, in contrast to other works dealing with similar tasks, is not suggested in this paper.
3. Qualitative results demonstrating the model's performance on multi-subject generation tasks are notably absent from this paper.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: * For personalization of a single subject (Fig. 4, Table 1), how is the bounding box determined? How do you compare with methods that do not require a bounding box as input?
* How well can the method generate interactions between input subjects? For example, could it make the teddy bear wear the red backpack?
soundness: 3
strengths: * The authors tackle the important task of grounded image generation with both text and image localization conditions.
* The writing is clear, making it easy to understand the proposed method.
* The authors combine grounded generation from both reference objects and textual inputs within a single architecture, which is highly relevant for many applications.
* The authors evaluate their method against a variety of prior works and datasets.
summary: The paper presents GroundingBooth, a method for grounded text-to-image customization. Given a list of subject entities represented by images and text entities represented by textual descriptions, along with bounding-box locations, GroundingBooth aims to generate an image containing all subjects in the specified locations according to their bounding boxes.
weaknesses: * In all the qualitative examples, the generated objects remain in the same pose as in the input image, despite the claim in line 191: “Moreover, our work adaptively harmonizes the poses of the reference objects and faithfully preserves their identity.” Could you provide examples where the input subjects change their pose while maintaining their identity? I would like to see examples where the prompt requires a significant pose change from the input subject.

* The proposed Masked Cross-Attention module was presented in previous works; see, for instance:
[1] Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation, Dahary et al. ECCV 2024
[2] InstanceDiffusion: Instance-level Control for Image Generation, Wang et al. CVPR 2024

* Overall, the proposed modules seem to lack novelty. The gated self-attention mechanism is borrowed from GLIGEN, and the masked cross-attention module exists in prior work, such as in [1].

* I find the distinction between “background” and “foreground” objects confusing, as it actually separates objects based on their source (image or text) rather than their position in the background or foreground of the image.

* The quantitative results are not convincing, as GroundingBooth shows lower scores than prior work on several metrics (e.g., Tables 1 and 2).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Does this work support simpler text-to-image generation, layout-to-image, or personalization tasks?

2. Regarding the illustration of the masked cross-attention layer in Figure 2, is the number of layers determined by the number of reference objects? For example, if there are three reference objects in the input, does that mean three masked cross-attention modules are required? If so, this model design seems unreasonable. Sequential masking could result in information loss in subsequent modules, especially when reference objects have significant overlap.
soundness: 2
strengths: 1. The paper tackles the task of generating personalized objects based on specific locations, which is an interesting setup.
2. This work proposes integrating reference objects and their location prompts through a grounding module and masked cross-attention.
3. Experiments are conducted on two benchmarks, accompanied by illustrative visualizations.
summary: This paper focuses on improving the accurate generation of spatial relationships between objects and backgrounds when creating personalized object variants. Technically, the authors propose a joint text-image grounding module that encourages both foreground subjects and background objects to adhere to locations defined by input bounding boxes. They also introduce a masked cross-attention layer aimed at preventing the unintended blending of multiple visual concepts in the same location, producing clear, distinct objects. Experiments are conducted on the MVImgNet and LVIS datasets.
weaknesses: 1. The paper primarily focuses on enabling the location-controlled generation of personalized objects, a setting already explored in prior work [3], which the authors seem to overlook. Additionally, the authors introduce a rather complex module to integrate location information but seem to lose focus on core functionalities like layout-to-image generation or personalized object generation.
2. Missing References: Some relevant references in layout-to-image generation, such as [1,2] and subject-driven image generation [4], are absent.
3. There are some limitations in model design. For example, the authors note that in cases where bounding boxes belong to the same class, the model cannot distinguish between a bounding box for a reference object and one for a text entity, leading to misplacement of the reference object. However, the paper does not clarify whether or how the proposed masked cross-attention module addresses this issue.
4. Further analysis is needed on topics such as the maximum number of reference objects supported in a single input and the model’s performance on subject-driven image generation without layout information.


[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models     
[2] Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image generation    
[3] Training-Free Layout Control With Cross-Attention Guidance    
[4] Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing

Meta-review: This paper proposes a grounded text-to-image generation framework incorporating reference objects and bounding-box constraints. While  backed by comprehensive experiments, in private discussion period, the reviewers find the contributions incremental. The core techniques, such as masked cross-attention and gated self-attention, have been explored in prior works, such as "Training-Free Layout Control with Cross-Attention Guidance" and "Grounded Text-to-Image Synthesis with Attention Refocusing". object poses remain identical to the reference images, contradicting claims of flexibility. The results do not convincingly outperform existing methods, and the novelty is limited mostly a combination of known ideas without substantial new insights. the reviewers remain unconvinced of the work's originality.

Decision: Reject