MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance

## Abstract

Abstract Recent advancements in text-to-image generation models have dramatically enhanced the generation of photorealistic images from textual prompts, leading to an increased interest in personalized text-to-image applications, particularly in multi-subject scenarios. However, these advances are hindered by two main challenges: firstly, the need to accurately maintain the details of each referenced subject in accordance with the textual descriptions; and secondly, the difficulty in achieving a cohesive representation of multiple subjects in a single image without introducing inconsistencies. To address these concerns, our research introduces the MS-Diffusion framework for layout-guided zero-shot image personalization with multi-subjects. This innovative approach integrates grounding tokens with the feature resampler to maintain detail fidelity among subjects. With the layout guidance, MS-Diffusion further improves the cross-attention to adapt to the multi-subject inputs, ensuring that each subject condition acts on specific areas. The proposed multi-subject cross-attention orchestrates harmonious inter-subject compositions while preserving the control of texts. Comprehensive quantitative and qualitative experiments affirm that this method surpasses existing models in both image and text fidelity, promoting the development of personalized text-to-image generation.

## 1 Introduction

Recent advancements in text-to-image (T2I) diffusion methodologies [, , ] have propelled the field to new heights, realizing unprecedented levels of photorealism while demonstrating a refined ability to conform to textual prompts. These achievements have spurred the development of a broad spectrum of applications, most prominently in the domain of personalized T2I (P-T2I) models, which are tasked with the complex undertaking of assimilating and regenerating novel visual concepts or subjects across diverse contexts with a heightened demand for conceptual and compositional fidelity. Despite fine-tuning-based techniques such as DreamBooth [] and Textual Inversion [] yield results with considerable accuracy, they necessitate extensive resources for tuning individual instances and for the storage of multiple models, which renders them less feasible for widespread application. To circumvent these resource-intensive requirements, fine-tuning-free alternatives have come to the fore.

Single-subject driven personalization methods, IP-Adapter [] and ELITE [] for instance, introduce a specialized cross-attention mechanism that distinctly processes text and image features, thereby affording the possibility to employ reference images directly as visual prompts within the model. Furthermore, recent works have employed multi-subject driven customization methodologies to concatenate visual concepts with textual prompts, offering a glimpse of the potential in techniques like SSR-Encoder [], $\lambda$-ECLIPSE [], Emu2 [], and KOSMOS-G []. These models harness identity data and amalgamate it with text via cross-attention, exhibiting proficiency in adjusting textures. Nevertheless, above zero-shot personalization methods encounter limitations, notably in adapting a pressing question pertains to the congruence of granular details between the subject depicted in the synthesized imagery and its corresponding subjects, along with the degree of semblance between the content of the generated image and associated textual descriptions. The challenge is further amplified in scenarios requiring the personalization of multiple subjects. Especially, the challenge of ensuring harmonious representation when multiple subjects are incorporated—specifically, the elucidation of whether the resultant image manifests any discordant elements or deleterious interactions in accordance with textual directives and multi-subject referential controls. As illustrated in Figure 2, multi-subject personalization methods frequently incur notable detail inaccuracies in a fine-tuning-free framework and often lead to subject neglect, subject overcontrol, and subject-subject
conflict issue within the generated images.

To confront these identified challenges, we are the first to introduce the layout-guided zero-shot image personalization with multiple subjects (MS-Diffusion) framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities, the provision of layout guidance, and the preservation of the foundational model’s parameters.
Firstly, we design the grounding resampler to extract the subject detailed features and integrate them with grounding information containing entities and boxes. As an image projection module, the proposed grounding resampler can enhance the subject fidelity while appending semantic and positional priors. Secondly, we propose a novel cross-attention mechanism for multiple subjects, which confines subjects to represent themselves in specific areas. This confluence not only facilitates the efficacious infusion of multi-subject data into the model but also mitigates conflicts between text and image subject control conditions. Such an approach culminates in the refined granularity of control over the image’s multi-subject composition. The experimental results demonstrate our method consistently outperforms the state-of-the-art approaches on all the benchmarks. We conclude the previous P-T2I works and provide an overall comparison in Table 1. The contributions can be summarized as follows:

**Table 1: An overview of previous studies of P-T2I tasks. MS-Diffusion is the first approach to support multi-reference zero-shot P-T2I generation with layout guidance and base model freezing.**
| Method | Zero Shot | Multi Subject | Base Model Freezing | MLLM Free | Layout Guidance |
| --- | --- | --- | --- | --- | --- |
| Textual Inversion [] | ✗ | ✗ | ✓ | ✓ | ✗ |
| DreamBooth [] | ✗ | ✗ | ✗ | ✓ | ✗ |
| \hdashlineELITE [] | ✓ | ✗ | ✗ | ✓ | ✗ |
| BLIP-Diffusion [] | ✓ | ✗ | ✗ | ✗ | ✗ |
| IP-Adapter [] | ✓ | ✗ | ✓ | ✓ | ✗ |
| \hdashlineEmu2 [] | ✓ | ✓ | ✗ | ✗ | ✗ |
| Kosmos-G [] | ✓ | ✓ | ✓ | ✗ | ✗ |
| $\lambda$-ECLIPSE [] | ✓ | ✓ | ✓ | ✗ | ✗ |
| SSR-Encoder [] | ✓ | ✓ | ✓ | ✓ | ✗ |
| \hdashlineMS-Diffusion | ✓ | ✓ | ✓ | ✓ | ✓ |

Figure: Figure 2: Challenges inherent in multi-subject personalization approaches. Through the explicit layout guidance, MS-Diffusion addresses these challenges by directing subject conditioning to specific areas, simultaneously maintaining high image fidelity.
Refer to caption: x2.png

- •
We introduce a layout-guided, zero-shot multi-subject image personalization framework within the diffusion model paradigm, designated as ’MS-Diffusion’. This innovation streamlines the complex process of preserving detailed subject references. Moreover, it seamlessly integrates multiple subjects into a coherent and harmonious personalized image.
- •
The ’Grounding Resampler’ is advanced as a novel feature refinement mechanism. This construct enriches the detail extraction from images by ascertaining the correlative content and fusing it with box embeddings that demarcate the anticipated spatial zones for each subject. Additionally, we introduce a specialized multi-subject cross-attention mechanism, confronting and rectifying prevalent complications in multi-subject personalization, including inadvertent subject neglect, disproportionate subject dominance, and internecine subject conflicts.
- •
The capabilities of ’MS-Diffusion’ are empirically substantiated through its ability to synthesize a broader spectrum of images with notable fidelity. The paper further delineates comprehensive ablation studies, underpinning the rationale behind our design decisions and affirming the efficacy of our proposed approach.

## 2 Related Work

### 2.1 Text-to-image Generation

Text-to-image generative models [, , , ] are capable of producing high-quality images using user-provided text prompts. In recent times, diffusion-based models have shown strong performance in text-to-image tasks. Stable Diffusion [] proposes conducting the diffusion process in latent space rather than pixel space, which reduces the sampling steps without compromising image quality. Kandinsky [] takes both text embedding and image embedding as conditions to generate images more controllablely. DALLE-3 [] recaptions the training data pairs and utilizes T5-XXL [] as the text encoder to strengthen the prompt-following ability. StableCascade [] presents a cascaded architecture to leverage outputs of front stages as priors, further reducing the latent space. PixArt-$\alpha$ [] also employs a large T5 text encoder and replaces the original U-Net backbone with a transformer []. These models focus on the basic text-to-image ability and cannot handle the situation when users provide specific subjects.

To finely control text-to-image generation, some diffusion models [, , ] support users in providing layout guidance. Layout Diffusion [] and GLIGEN [] input positions and labels of bounding boxes into the diffusion model and train it to learn the layout information. DenseDiffusion []
develops a training-free method and modulates the attention maps in the inference phase. Instance Diffusion [] and MIGC [] extend the layout-conditioned diffusion to the instance level, enabling the model to generate multiple objects with precise quantities. While layout-guided diffusion models have robust controllability, they cannot reference specific concepts, which is emphasized in personalized text-to-image generation.

### 2.2 Text-to-image Personalization

Text-to-image personalization [, , , , ] has attracted much attention in the community for its powerful referencing capability of both text and image prompts. Textual inversion [] and DreamBooth [] utilize an identifier in the text to bind the visual concept through fine-tuning. IP-Adapter [] proposes a zero-shot personalized model by projecting the image embedding to the cross-attention layers. InstanceID [] develops an approach for identity personalization, replacing the image encoder with a face encoder and employing ControlNet [] to integrate the face landmarks. To narrow the gap between image and text prompts, Kosmos-G [] and $\lambda$-ECLIPSE [] conduct multi-modal training to unite the inputs by text-image interleaving. SSR-Encoder [] design a query network to extract a single subject from images with multiple subjects for personalization.

Though past research in this field has significantly enhanced the ability to reference single subjects, few zero-shot multi-subject personalized models. Moreover, existing related works struggle to address conflicts in personalized generation with multiple subjects and generate bad results, which is precisely the focus of our work, to discuss and resolve these issues.

Figure: Figure 3: The overall pipeline of MS-Diffusion. It introduces two pivotal enhancements to the model: the grounding resampler and multi-subject cross-attention mechanisms. Firstly, the grounding resampler adeptly assimilates visual information, correlating it with specific entities and spatial constraints. Subsequently, a targeted cross-attention mechanism facilitates precise interactions between the image condition and the diffusion latent within the multi-subject attention layers. Throughout the training phase, all components of the pre-existing diffusion model remain frozen.
Refer to caption: x3.png

## 3 Method

The pipeline of MS-Diffusion is shown in Figure 3. Through an improved data construction strategy, we can get multiple subject images together with the corresponding entities and layouts as input. We propose a grounding resampler to separately extract the image features and integrate them with phrase embedding and box embedding for condition enhancement. Inside the cross-attention layers, we further introduce the masked cross-attention to guide the generation with layout priors and alleviate conflicts of multiple subjects. The training needs no pre-trained weights to be optimized and remains plug-and-play in various base models.

### 3.1 Preliminaries

#### Stable Diffusion with Image Prompt.

As a widely used diffusion model, Stable Diffusion (SD) [] conducts the diffusion process in the latent space. Given an image and a text prompt, SD encodes them into latent code $\mathbf{z}$ and condition embedding $\mathbf{c_{t}}$ utilizing VAE [] and CLIP [] text encoder, respectively. In zero-shot image personalization architectures like IP-adapter [], images can also be considered a condition of the diffusion model. Specifically, a subject image is encoded to image embeddings by an image encoder and then projected into the original condition space of the diffusion model denoted as $\mathbf{c}_{i}$. For a timestep $t$ which is uniformly sampled from a fixed range, the model $\theta$ predicts the noise $\boldsymbol{\epsilon}_{\theta}$ and is optimized through the objective:

$$ $\mathcal{L}_{IP}=\mathbb{E}_{\mathbf{z},\mathbf{c},\epsilon,t}\left[\left\|{ \epsilon-\boldsymbol{\epsilon}}_{\theta}\left(\mathbf{z}_{t}\mid\mathbf{c_{t}} ,\mathbf{c_{i}},t\right)\right\|_{2}^{2}\right]$ (1) $$

where $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. In this work, we employ $\mathcal{L}_{IP}$ with SDXL [] as the pre-trained model, which contains two CLIP text encoders and additional condition inputs besides $\mathbf{c}$ and $t$, omitted for brevity.

#### Cross-attention.

In SD [], both $\mathbf{c}_{i}$ and $\mathbf{c}_{t}$ are integrated into the U-Net backbone through cross-attention layers:

$$ $\text{Attn}\left(\mathbf{Q},\mathbf{K}_{i},\mathbf{K}_{t},\mathbf{V}_{i}, \mathbf{V}_{t}\right)=\gamma\cdot\mathbf{z}_{img}+\mathbf{z}_{txt}=\gamma\cdot \text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}_{i}}{\sqrt{d}}\right)\mathbf{V} _{i}+\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}_{t}}{\sqrt{d}}\right) \mathbf{V}_{t}$ (2) $$

where $\mathbf{Q}=\mathbf{z}W_{q}$, $\mathbf{K_{i}}=\mathbf{c}_{i}W_{k}^{i}$, $\mathbf{K_{t}}=\mathbf{c}_{t}W_{k}^{t}$, $\mathbf{V_{i}}=\mathbf{c}_{i}W_{v}^{i}$, $\mathbf{V_{t}}=\mathbf{c}_{t}W_{v}^{t}$, and $W_{q}$, $W_{k}$, $W_{v}$ are corresponding projection weight matrices. And $d$ represents the dimensionality of the key vectors Note that the key and value matrix of $\mathbf{c}_{i}$ and $\mathbf{c}_{t}$ are independent of each other to decouple conditions of different modalities. Previous studies [, ] have found that attention maps $\mathbf{A}=\text{Softmax}\left(\mathbf{Q}\mathbf{K}/\sqrt{d}\right)$ can reflect the attribution relation between generated images and conditions, which means that they determine the effect of condition controls.

### 3.2 Data Construction

In the field of multi-subject personalization, creating a robust dataset architecture is a significant challenge, especially when no pre-existing dataset includes a variety of reference subjects with their validated truths. Our method starts with applying a Named Entity Recognition (NER) protocol to textual data to extract relevant entities. These entities are then used within a detection framework to define the corresponding bounding boxes. This step generates training samples that encompass a range of [subjects, entities, spatial layouts].

Previous studies have mostly created training examples from stand-alone images which is essentially a reconstruction task, leading the resulting models to favor replication, often resulting in ’copy-and-paste’ artifacts []. To address this issue, our enhanced approach involves extracting subjects from a single video frame and using another frame from the same sequence as a reference for the truth. This technique effectively separates the personalized references from the target images. Due to possible variations in subjects between frames, we use a specialized subject-matching algorithm to ensure accurate matching. We provide a detailed description of this data processing pipeline in Section A.

### 3.3 Grounding Resampler

Different from text embedding, image embedding generally contains more information and is sparser, making projection into the condition space challenging. Leveraging embeddings from all image patches primarily control the condition, but the pooled output from the image encoder tends to omit many details. Different from [] and [], we propose the integration of a grounding resampler that functions as an alternative form of image projector. Utilizing a set of learnable tokens, the grounding resampler queries and distills pertinent information from the image features. Specifically, with an image embedding $f_{i}$ and a learnable query $f_{q}$, the resampler comprises several attention layers:

$$ $\text{RSAttn}=\text{Softmax}\left(\frac{\mathbf{Q}\left(f_{q}\right)\mathbf{K} \left(\left[f_{i},f_{q}\right]\right)}{\sqrt{d}}\right)\mathbf{V}\left(\left[f _{i},f_{q}\right]\right)$ (3) $$

where $[f_{i},f_{q}]$ denotes the concatenation of the image embedding $f_{i}$ and the learnable query $f_{q}$. The architecture incorporates fully connected feedforward networks (FFNs), analogous to those utilized in standard vision transformers [].

As detailed in 3.2, we can obtain entities of multiple referenced subjects and their target area boxes in the generated image. We present to initialize the query $f_{q}$ with grounding tokens $f_{g}$ derived from text embedding of entities and Fourier embedding of boxes. Entities are related to the semantic information of images and boxes indicate the areas where the subjects are supposed to be. It would be helpful for the resampler to extract the image features appropriately and the cross-attention layers to condition the generation finely. To prevent the model from becoming dependent on the grounding tokens during inference, we randomly replace them with the original learnable queries in the training. For the input of n subjects, the projection processes of different subject images do not affect each other. The resulting n queries will be concatenated and input into the subsequent model as $\mathbf{c_{i}}$ with $N=n*n_{t}$ tokens, where $n_{t}$ is the token quantity per subject.

### 3.4 Multi-subject Cross-attention

In scenarios involving the generation of multiple subjects, challenges frequently arise that are not exclusive to personalization tasks. These include discordances between subjects and their backgrounds and amongst the subjects themselves. A viable solution to mitigate such conflicts leverages attention masks, contingent upon the availability of layout priors. The incorporation of attention masks within cross-attention mechanisms facilitates the exclusion of padding tokens from the condition, thus minimizing their impact.

To confine the context of each subject to a designated spatial domain, we propose an enhancement to the conventional attention mask, denoted as $\mathbf{M}$. This adjustment involves the bilateral neglection of tokens within both the query and key matrices, applied specifically for the $j$th subject as follows:

$$ $\mathbf{M}_{j}(x,y)=\begin{cases}0&\text{if }[x,y]\in B_{j}\\ -\infty&\text{if }[x,y]\notin B_{j}\end{cases}$ (4) $$

Here, $B_{j}$ signifies the coordinate set of bounding boxes pertaining to the $j$th subject. By this means, the conditional image latent $\hat{\mathbf{z}}_{img}$ is derived through:

$$ $\hat{\mathbf{z}}_{img}=\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}_{i}}{ \sqrt{d}}+\mathbf{M}\right)\mathbf{V}_{i}$ (5) $$

Herein, $\mathbf{M}$ represents the amalgamation of all subject-specific masks, $\text{Concat}(\mathbf{M}_{0},\ldots,\mathbf{M}_{n})$.

However, an inherent limitation arises when a query patch token is ubiquitously masked across all referenced subjects or remains unmasked (in instances of overlapping bounding boxes), thereby diminishing the intended efficacy of multi-subject cross-attention. To counteract this, we introduce dummy tokens initialized at random preceding the image tokens to symbolize the background. This strategy is instrumental in ensuring that text conditions predominantly govern areas devoid of any guided layout, thereby considered irrelevant to the referenced images. Following the acquisition of $\mathbf{A}$, we apply $\mathbf{M}_{bg}$ to seamlessly mask these tokens within $\hat{\mathbf{z}}_{img}$, as illustrated:

$$ $\mathbf{z}_{img}=(1-\mathbf{M}_{bg})\cdot\hat{\mathbf{z}}_{img}$ (6) $$

where $\mathbf{M}_{bg}$ is articulated as a binary mask, with elements within the subject bounding boxes designated as zero. Notably, certain studies have sought to resolve these conflicts through the application of objectives on attention maps within the cross-attention mechanism. A series of rigorous experiments have been conducted to substantiate our design, with the details elucidated in Section 4.4.

## 4 Experiments

Figure: Figure 4: Qualitative results of MS-Diffusion and baselines in single-subject personalization.
Refer to caption: x4.png

**Table 2: Quantitative comparison on MS-Diffusion and baselines. Bold and underline represent the highest and second-highest metrics, respectively. For single-subject, the results for IP-Adapter, Emu2, and Kosmos-G were obtained from [], while the rest are reported in the corresponding papers. * denotes the model is fine-tuned on DreamBench.**
| Method | Single-subject | Multi-subject |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-I | DINO | CLIP-T | CLIP-I | DINO | M-DINO | CLIP-T |  |
| Textual Inversion | 0.780 | 0.569 | 0.255 | - | - | - | - |
| DreamBooth | 0.803 | 0.668 | 0.305 | - | - | - | - |
| BLIP-Diffusion^* | 0.805 | 0.670 | 0.302 | - | - | - | - |
| $\lambda$-ECLIPSE^* | 0.796 | 0.682 | 0.304 | - | - | - | - |
| MS-Diffusion^* | 0.805 | 0.702 | 0.313 | - | - | - | - |
| \hdashlineELITE | 0.771 | 0.621 | 0.293 | - | - | - | - |
| BLIP-Diffusion | 0.779 | 0.594 | 0.300 | - | - | - | - |
| IP-Adapter | 0.810 | 0.613 | 0.292 | - | - | - | - |
| Emu2 | 0.765 | 0.563 | 0.273 | - | - | - | - |
| Kosmos-G | 0.822 | 0.618 | 0.250 | - | - | - | - |
| $\lambda$-ECLIPSE | 0.783 | 0.613 | 0.307 | 0.724 | 0.419 | 0.094 | 0.316 |
| SSR-Encoder | 0.821 | 0.612 | 0.308 | 0.725 | 0.425 | 0.107 | 0.303 |
| MS-Diffusion | 0.792 | 0.671 | 0.321 | 0.698 | 0.425 | 0.108 | 0.341 |

Figure: Figure 5: Qualitative results of MS-Diffusion and baselines in multi-subject personalization.
Refer to caption: x5.png

### 4.1 Experiment Setup

#### Datasets.

For training, we utilize an in-house video dataset that contains 3.6M video clips. For evaluation, we measure the single-subject and multi-subject performance on DreamBench [] and MS-Bench, respectively. DreamBench includes 30 subjects and 25 prompts and we preset all input boxes to [0.25, 0.25, 0.75, 0.75]. To thoroughly assess the performance of multi-subject personalization, we have established a new evaluation standard, MS-Bench, which includes 40 subjects and 1148 combinations, each combination having up to 6 prompts, totaling 4488 distinct test samples. Details of datasets are provided in Section A and Section B.

#### Evaluation metrics.

Following previous works, we measure the performance through image and text fidelity. To assess image fidelity, we employ cosine similarity measures between generated images and subject images within CLIP [] and DINO [] spaces, referred to as CLIP-I and DINO, respectively. For text fidelity, we utilize cosine similarity between generated images and text prompts in CLIP space, denoted as CLIP-T. In multi-subject personalization, using the average fidelity to reflect image fidelity is insufficient, as it fails to reveal cases of subject neglect. We further employ the product of multi-subject DINO, denoted as M-DINO, to indicate whether each subject has been recreated in the results.

#### Baselines.

For single-subject personalization, we compare our model with methods mentioned in Table 1. Emu2 [], Kosmos-G [], and $\lambda$-ECLIPSE [] are all MLLM-based methods, while $\lambda$-ECLIPSE is reported to have better performance. Therefore, we select SSR-Encoder [] and $\lambda$-ECLIPSE [] as baselines for multi-subject personalization. The implementation details of MS-Diffusion and these methods are contained in Section C.

### 4.2 Single-subject Comparison

In the single-subject comparison, a detailed examination is carried out utilizing both qualitative and quantitative comparisons to gauge the performance of different methodologies. On the qualitative front, as shown in Figure 4, MS-Diffusion shows an exceptional ability to generate single-subject images with high fidelity and detail retention. In quantitative results provided in Table 2, MS-Diffusion also achieves competitive scores, with obviously the highest DINO and CLIP-T scores at 0.671 and 0.321 respectively on zero-shot scenarios, and leading CLIP-I score of 0.792. For tuning methods, MS-Diffusion outperforms baselines in all metrics. As discussed in [], DINO more accurately captures the similarity in details between the results and the labels, whereas CLIP-I may exhibit high scores in situations of background overfitting, resulting in a clear advantage for DINO, but a slight disadvantage for CLIP-I of MS-Diffusion.

### 4.3 Multi-subject Comparison

From a qualitative perspective in Figure 5, MS-Diffusion manages to maintain natural interactions among subjects in generated images while ensuring each subject retains its distinctiveness and recognizability. Quantitatively, results in Table 2 demonstrate the strength of MS-Diffusion in DINO, M-DINO, and CLIP-T. Unlike in single-subject personalization, there is a larger gap in text fidelity between MS-Diffusion and the baselines in multi-subject personalization, demonstrating that MS-Diffusion not only effectively generates the multiple subjects outlined in the text but also excellently preserves the text control capabilities inherent to SD. Additionally, the image fidelity of MS-Diffusion is comparable, highlighting its superior ability to retain details, particularly significant as low text fidelity is commonly associated with overfitting.

**Table 3: Ablation study of MS-Diffusion. RS, GRS, MCA, TAL, IAL, and LG represent resampler, grounding resampler, multi-subject cross-attention, text attention loss, image attention loss, and layout guidance, respectively.**
| Method | Single-subject | Multi-subject |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-I | DINO | CLIP-T | CLIP-I | DINO | M-DINO | CLIP-T |  |
| MS-Diffusion | 0.792 | 0.671 | 0.321 | 0.698 | 0.425 | 0.108 | 0.341 |
| \hdashline   w/o RS | 0.775 | 0.583 | 0.320 | 0.680 | 0.372 | 0.082 | 0.336 |
| w/o GRS | 0.777 | 0.646 | 0.320 | 0.681 | 0.389 | 0.090 | 0.331 |
| w/o MCA | 0.798 | 0.662 | 0.312 | 0.693 | 0.422 | 0.100 | 0.309 |
| \hdashline   w/o LG w/ IAL | 0.761 | 0.577 | 0.284 | 0.675 | 0.377 | 0.080 | 0.305 |
| w/o LG w/ IAL&TAL | 0.809 | 0.660 | 0.293 | 0.687 | 0.413 | 0.093 | 0.316 |

### 4.4 Ablation Study

#### Module ablation.

We conduct an ablation experiment on the proposed two modules, grounding resampler (GRS) and multi-subject cross-attention (MCA), to validate their effects. For GRS, we replace it with a linear projection layer and a normal resampler [, ]. Results in Table 3 indicate that the resampler-like image projector significantly enhances the details, as evidenced by DINO being obviouly higher than the linear projector. Moreover, The substantial improvement in multi-object image fidelity by GRS reflects the critical role of the information carried by grounding tokens in multi-object generation. As a key module for resolving conflicts, removing MCA results in a noticeable degradation of text fidelity, especially in multi-subject generation. The combined use of both modules ensures that MS-Diffusion maintains high image and text fidelity simultaneously.

#### Layout guidance.

As mentioned in Section 3.4, we have explored the indispensable role of explicit layout guidance (LG), including grounding tokens and MCA. A straightforward approach to implicitly utilizing layout involves incorporating an attention loss during training. Besides the image attention loss (IAL), we also introduce text attention loss (TAL) to training by setting the original cross-attention layers trainable. The detailed loss definition is provided in Section F. As illustrated in Table 3, an objective to guide the image cross-attention helps the personalization hardly at all. TAL has somewhat resolved the conflict issues, but its performance is inferior to MS-Diffusion while introducing additional training parameters. We consider the inclusion of LG necessary and rational, not merely for the performance enhancements it offers, but also because it effectively resolves the various multi-object generation issues highlighted in Figure 2.

## 5 Conclusion

This study makes a significant contribution to the field of P-T2I diffusion models with the development of MS-Diffusion. This zero-shot framework excels at capturing intricate subject details and smoothly blending multiple subjects into a single coherent image. Equipped with the innovative Grounding Resampler and Multi-subject Cross-attention mechanisms, our model effectively overcomes common multi-subject personalization issues, such as subject neglect and conflict. Extensive ablation studies underscore MS-Diffusion’s enhanced performance in image synthesis fidelity compared to existing models. It stands as a groundbreaking approach for P-T2I applications that are free from the need for fine-tuning and require layout guidance.

## Appendix A Training Dataset Construction Pipeline

Figure 6 illustrates the construction pipeline of the training dataset. Firstly, we randomly select two frames from a video clip, one as the reference and the other as the ground truth. Both frames are captioned by BLIP-2 []. Secondly, we utilize a NER model(^1^11[https://spacy.io/](https://spacy.io/)) to extract entities from the caption. Entities and images are then input into Grounding DINO [] to obtain the boxes, which are parts of the final input of the model. Taking the boxes as prompts of SAM [], we can further obtain segmentation masks to extract subjects from the reference image. Since the entities in different frames can be different, we design a subject matcher, which finds the correspondence between the frames by conducting Hungarian Algorithm on the entity image embeddings. Frames in a video typically contain the same entities but exhibit clear differences in details such as angles and poses. This makes them highly suitable as training data for personalized image generation models, which can help mitigate the model’s tendency to copy-and-paste.

Our dataset comprises 2.8M general scenario videos and 0.8M product demonstration videos, where the former covers more scenarios and the latter has more clear subjects. 2-5 frames for each are adopted in the training. In practice, there may only be 1-2 subjects successfully matched. To ensure that the training data contains a sufficient number of reference subjects, for the targets where matching fails, we directly use the corresponding parts of the ground truth as references. Subjects that are too small, too large, or have imbalanced proportions are filtered out. Each training sample can have up to 4 subjects, and we pad in the ones with fewer than 4.

Figure: Figure 6: Data construction pipeline of our work. For the input of two frames, we can get subject images, entities, and boxes. Note that the entities and boxes are from the ground truth frame since they indicate the information in the generated result.
Refer to caption: x6.png

## Appendix B Details of MS-Bench

**Table 4: Explanation of MS-Bench. Each combination type has preset prompts and boxes. [S] represents prompt variations about the scene, including "in a room", "in the jungle", "in the snow", "on the beach", "on the grass", and "on a cobblestone street".**
| Type | Prompt | Boxes |
| --- | --- | --- |
| living+living | a {0} and a {1} [S] | [0.00, 0.25, 0.50, 0.75]<br>[0.50, 0.25, 1.00, 0.75] |
| living+object |  |  |
| object+object |  |  |
| living+upwearing | a {0} wearing a {1} [S] | [0.25, 0.25, 0.75, 0.75]<br>[0.25, 0.00, 0.75, 0.25] |
| living+midwearing | a {0} wearing a {1} [S] | [0.25, 0.25, 0.75, 0.75]<br>[0.25, 0.25, 0.75, 0.75] |
| living+wholewearing |  |  |
| midwearing+downwearing | a woman wearing a {0}<br>and a {1} [S] | [0.25, 0.25, 0.75, 0.60]<br>[0.25, 0.60, 0.75, 1.00] |
| living+scene | a {0} with a {1}<br>in the background | [0.25, 0.25, 0.75, 0.75]<br>[0.00, 0.00, 1.00, 1.00] |
| object+scene |  |  |
| living+living+living<br>object+object+object | a {0}, a {1}, and a {2} [S] | [0.00, 0.25, 0.35, 0.75]<br>[0.35, 0.25, 0.65, 0.75]<br>[0.65, 0.25, 1.00, 0.75] |
|  |  |  |
|  |  |  |
| living+object+scene | a {0} and a {1} with a {2}<br>in the background | [0.00, 0.25, 0.50, 0.75]<br>[0.50, 0.25, 1.00, 0.75]<br>[0.00, 0.00, 1.00, 1.00] |
|  |  |  |
|  |  |  |
| upwearing+midwearing+<br>downwearing | a woman wearing a {0}, a {1},<br>and a {2} [S] | [0.25, 0.00, 0.75, 0.25]<br>[0.25, 0.25, 0.75, 0.60]<br>[0.25, 0.60, 0.75, 1.00] |
|  |  |  |
|  |  |  |

To construct MS-Bench, we collect subjects from [, , ], the Internet(^2^22[https://unsplash.com/](https://unsplash.com/)), and an internal dataset that does not overlap with the training set. MS-Bench contains four data types and 13 combination types with two or three subjects. We provide the details in Table 4. Each combination type other than those related to the scene has 6 prompt variations. There are 1148 combinations and 4488 evaluation samples, where entities and boxes are subject categories and preset layouts. Compared to other multi-subject benchmarks, our MS-Bench ensures that the model performance can be reflected comprehensively in abundant cases.

## Appendix C Experiment Settings

#### Training and Inference.

The pre-trained model employed in MS-Diffusion is Stable Diffusion XL (SDXL) []. Implemented by Pytorch 2.0.1 and Diffusers 0.23.1, our model is trained on 16 A100 GPUs for 120k steps with a batch size of 8 and a learning rate of 1e-4. Following the training of [], we set $\gamma=1.0$ in cross-attention layers and dropped the text and image condition using the same probability. To ensure the model is not dependent on the grounding tokens (Section 3.3), we also randomly drop them with a probability of 0.1. We generate five images for each sample during the inference, with unconditional guidance scale and $\gamma$ set to 7.5 and 0.6, respectively, to get better results.

#### Comparative methods.

Here we provide the details of the baselines compared in qualitative and quantitative experiments:

- •
BLIP-Diffusion [] utilizes BLIP-2 [] to unite the text and image embeddings. We implement it in the qualitative comparison using Diffusers.
- •
IP-Adapter [] also uses image prompt as the condition. We run qualitative samples on their official code with the scale set to 0.5 recommended by the paper. Considering fairness, we use the result of SDXL in Table 2.
- •
SSR-Encoder [] design a query network to extract the specified subject of a single image, which enables it to finish multi-subject generation. We leverage it as one of the baselines in multi-subject personalization. For performance comparison, we employed the official code provided, alongside the default hyperparameters specified in the code repository.
- •
$\lambda$-ECLIPSE [] trains an independent multi-modal encoder and employs Kandinsky as the generative backbone. Since it outperforms other MLLM-based approaches, we choose it to be the representative. To facilitate a comparative analysis of performance, the study utilized the officially provided code, in conjunction with the default hyperparameters delineated within the corresponding code repository.

## Appendix D More Results of Single-subject Personalization

Additional qualitative results on DreamBench are provided in Figure LABEL:fig:single_app. MS-Diffusion shows excellent text fidelity in all subjects while keeping subject details, especially the living ones (dogs). It can be noticed that some elements in the background (the third line and the fourth line) also occur in the results (the grass and the teapot holder) since the entire images are referenced during the generation. Their scope of action depends on the input bounding box. In practical applications, using masked images as a condition is recommended.

## Appendix E More Results of Multi-subject Personalization

We provide additional multi-subject personalized images based on MS-Bench in Figure LABEL:fig:multi_app. The results encompass various combination types, fully demonstrating the generalizability and robustness of MS-Diffusion. When the scene changes freely according to the text, the details of the subject are preserved without being affected. In addition to common parallel combinations, MS-Diffusion also performs well in personalized generation for combinations with certain overlapping areas, such as "living+midwearing" and "object+scene".

## Appendix F Layout Guidance

Cross-attention maps can intuitively reflect the condition-image attribution relation [, ]. Recent works [, ] have studied utilizing an objective on the cross-attention maps in multi-subject generation. The objective exists only in training, considered implicit and insufficient to handle multi-subject conflicts. We have provided the performance comparison between our explicit layout guidance and attention loss in Section 4.4. For a single cross-attention layer, the attention loss of the $j$th subject is calculated by:

$$ $\mathcal{L}_{am}^{j}=\left(1-\frac{\sum_{\left[x,y\right]\in B_{j}}\mathbf{A}_ {\left[x,y\right],j}}{\sum_{\left[x,y\right]}\mathbf{A}_{\left[x,y\right],j}} \right)^{2}$ (7) $$

where $\left[x,y\right]$ corresponds to a latent token in $\mathbf{Q}$ and $B_{j}$ is the bounding box of the $j$th subject. This objective aims to promote the activation of attention maps within specific boxes. We average $\mathcal{L}_{am}^{j}$ across layers and subjects and set its weight to 0.01 in the final loss. To validate the text attention loss, we also optimize the text cross-attention layers in training, increasing approximately 70% in learnable parameters.

Although our model provides explicit layout guidance, it still significantly differs from layout-based diffusion. Firstly, the information of boxes in the grounding resampler is prior, and its conditional effect is relatively weak. We have also reduced the model’s reliance on this input by randomly dropping grounding tokens. Secondly, the multi-subject cross-attention only exists in image cross-attention, inherently controlling the action of image conditions in specified areas, but cannot determine the whole generation of the diffusion model. Our goal is not to develop a method that fully supports layout control but to utilize layout information to guide the model in resolving conflicts in multi-subject generation.

To further explore the layout control capability of MS-Diffusion, we provide qualitative results in Figure LABEL:fig:layout. It can be demonstrated that MS-Diffusion can generate images that adhere to layout conditions, even in the case of two instances of the same category. However, the generated positions are not entirely accurate, especially in "a cat and a cat on the grass", illustrating that the layout condition is relatively weak compared to text and image prompts in the personalization task.

## Appendix G Integration with ControlNet

In the realm of text-to-image diffusion models, a notable application is the enhancement of structural control within image generation. Our MS-Diffusoin maintains the original network architecture unchanged, thus ensuring full compatibility with existing controllable tools. Consequently, this allows for the generation of images that are not only prompted by images but also governed by additional conditions. By integrating our MS-Diffusion with established controllable mechanisms such as ControlNet, we demonstrate the capacity to produce images under varied structural directives. Figure LABEL:fig:control displays an array of images synthesized with image prompts coupled with distinct structural controls(depth, canny edge, openpose). This seamless cooperation between our method and these tools facilitates the creation of highly controllable images without necessitating fine-tuning.

## Appendix H Text-Image Attribution Analysis

In MS-Diffusion, our focus is primarily on resolving conflicts between subjects without altering the control mechanism of the text. While some approaches [, , , ] in non-personalized text-to-image tasks address multi-object generation conflicts through text cross-attention, this inevitably requires tuning the diffusion model’s parameters, thereby affecting the plug-and-play nature, which is not preferred by us. As demonstrated in the text-image attribution analysis presented in Figure 7, the control of multiple objects by text in our model is also quite evident. This may be related to the explicit layout guidance for subjects, since the images and text condition jointly in the generation process. We also attempted to control text cross-attention using the same mechanism as in Section 3.4, but no differences were observed in the results.

Figure: Figure 7: Text-image attribution analysis of MS-Diffusion. We average the attention maps corresponding to the subjects and translate them to normalized heat maps.
Refer to caption: x7.png

## Appendix I Subject Interpolation

The blending of two distinct subject representations to yield composite subjects with hybrid characteristics is feasible through the navigation of the embedding space linking the subjects. As depicted in Figure LABEL:fig:interpolation, linear interpolations are conducted among dog and hat representations, subsequently rendering the interpolated subject in an unaccustomed context. The visualization reveals a natural gradation of subject appearance along the interpolated trajectory that harmonizes with the surrounding environment. This technique proves beneficial when applied in subject fusion and style transfer.

## Appendix J Limitations

There are certain limitations in MS-Diffusion. The box-based indication of positions lacks precision, making it challenging to work effectively when the interaction between subjects is stronger. Moreover, the model requires explicit layout input during inference, and generating complex scenes becomes difficult. Though MS-Diffusion beats SOTA personalized diffusion methods in both single-subject and multi-subject generation, it still suffers from the influence of background in subject images. In the future, we will tackle the above limitations and extend the model to more applications.

## Appendix K Societal Impact

As an image personalization method, MS-Diffusion aims to customize images based on user-provided subjects without fine-tuning. Additionally, the multi-subject reference capability of MS-Diffusion allows users to freely combine and re-create different concepts. However, MS-Diffusion can also be used to generate deceptive images, especially those involving subject combinations that would not exist in reality, an issue that remains to be addressed in the future.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - Could the authors provide a bit more detail on how the M-DINO metric is calculated and how it measures multi-subject fidelity?

- I came across a recent paper [1] on a similar task and recommend testing its evaluation metric on a few cases to see if it could be helpful for assessing multi-subject fidelity.

- The qualitative results presented are impressive, particularly in handling prompts with complex interactions among multiple subjects. Could the authors provide additional examples demonstrating these capabilities, as well as discuss any current limitations?

[1] Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models (https://arxiv.org/pdf/2404.04243)
soundness: 3
strengths: - The paper is well-written and well-organized, presenting a clear and compelling motivation for the study.
- This method is the first to introduce layout-guided zero-shot image personalization for multi-subject scenarios.
- The paper showcases impressive qualitative results, particularly in layout control capabilities across both single- and multi-subject personalization, as well as in handling prompts with complex interactions among multiple subjects.
summary: The paper introduces the MS-Diffusion framework, a layout-guided zero-shot image personalization approach for multi-subject scenarios. The author features a grounding resampler to enhance subject fidelity with semantic and positional priors and a novel cross-attention mechanism to ensure that each subject is represented in specific areas and facilitating the integration of multi-subject data while mitigating conflicts between text and image subject control.
weaknesses: - In Section 2.2, second paragraph, the sentence "Though past research in this field has significantly enhanced the ability to reference single subjects, few zero-shot multi-subject personalized models" is a bit unclear. I suggest rephrasing this sentence for clarity.

- In Section 3.1, as well as in the rest of the paper, it appears that the transpose notation (T) is missing for K in the equations of calculating attention maps.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Comparing to method like Subject Diffusion and GLIGEN, what makes proposed layout control novel?
    2. In figure 2, the author proposed several limitation cases identified from previous methods, however, they are not further discussed or addressed in the following paragraphs, for example, how is subject overcontrol defined? It usually better to have some specific discussion to recall and address the problems mentioned in the introduction.
    3. On the multi-subject generation benchmark, the performance boost compared to the baseline SSR-encoder remains minimal, despite the proposed method being trained on a larger and more advanced base model (SDXL versus SD1.5). This outcome appears to serve as a counterexample, challenging the effectiveness of the proposed design.
    4. The seems to produce a higher CLIP-T scores (better text following) than previous methods, however, this performance boost might also come from the usage of SDXL where multiple text encoder is designed specifically for increase text adherence. Please justify this boost comes from the proposed model design rather than the base model.
soundness: 3
strengths: 1. High-quality multi-subject personalized generation.
 2. The novel design of the grounding resampler provides extraordinary performance boost compared to previous baselines.
summary: This paper introduces MS-Diffusion, a zero-shot personalized image generation method for multiple subject entities. Given one or more reference subjects and the text prompt, the model generates high-quality target image based on the inputs accordingly, containing the subject following the desired text descriptions.
The main contribution of the paper is the proposal of integrating grounding information for multi-subject personalized generation, and a newly designed grounding resampler for incorporating the given grounding information. 
By training a SDXL model with the encourage modules on a large-scale private dataset, the algorithm yield favorable results compared to previous designs.
weaknesses: • Lack of baseline comparisons and limited contributions
        ◦ The author claimed that they are the first to incorporate grounding information for multiple subject image personalization, however, plenty of existing works have proposed similar designs, for example Subject Diffusion, where the layout is also determined by bounding boxes as cross attention regularizations, while there are no related in-depth discussions.
        ◦ The comparison with previous design on the benchmark is also limited.

    • Limited performance boost
        ◦ In Table 2, the performance boost is rather trivial when comparing with SSR-Encoder on Multi-subject setting. Specifically, while SSR-encoder was trained based on SD1.5 and the proposed method is trained based on SDXL, the performance for is almost the same, except for a higher CLIP-T score.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: [1] L243-253: In the description of the cross-attention layers, in addition to the information injected from the text features, the authors show information that is injected from the reference image as well. In SD, the only features injected are those from the text. While injected image features through the cross-attention layers have been proposed (e.g, IP-Adapter), this wasn’t part of the original model. Please revise this section, and consider explicitly indicating the origin of the image features (f_i). 

[2] Grounding Resampler: the authors initialize the queries with word-embeddings and bounding-box Sinusoidal/Fourier features. But in another part of the text, the authors claim these embeddings are learnable. I am confused, is it learnable because of the query project-matrices ? Or do you pass the entity-word-embedding along with Bounding-Box features to another Projection matrix ? Can you please clarify this point.   Please provide a step-by-step explanation of how the queries are initialized and then updated during training, clearly stating which components are learnable and which are fixed.

[3] Do the results in Figure 4 correspond to the MS-Diffusion model before or after fine tuning on DreamBench ? Please clearly label all results in the paper to indicate whether they are from zero-shot or fine-tuned models.

[4] The Multi Subject Cross Attention module has been used (e.g, in ELITE). To my understanding, MSDiffusion only extends this to multi-subject use cases. Can you please highlight the differences ?

[5] It would be nice to provide dataset statistics. In particular, what percentage of the data there are multiple-subjects, excluding trivial cases like a person and clothes. At least from the qualitative samples, it looks like the model performs very well on clothing entities and a single subject. 

[6] Can you provide some information regarding the computational complexity of the method. In particular, it would be nice to report the additional overhead indeed by the method. Please report inference time and memory usage compared to baseline zero-shot models, for both single-subject and multi-subject generation tasks.
soundness: 3
strengths: [1] The paper introduces a zero-shot multi-subject personalization method. 

[2] There are interesting components to the proposed method like the data-collection from video and the Grounding Resampler Unit. However I feel like more in-depth analysis and ablation should be carried out to validate that design choices considered are indeed important.
summary: The paper proposes a zero-shot method for multi-subject personalization of text-to-image models. This is done through a pre-training phase where an adapter is trained to adapt a diffusion model on additional conditions derived from input images. In particular, the authors use 3.6M videos to extract two views of the same subject. Each training sample comprises segmented entities, their bounding boxes and a ground truth frame including all entities. 

The paper discusses mainly two design choices for their adapter, namely, a Grounded Resampler, and a Multi-Subject Cross Attention Layer. The grounded resampler is used to distill the subject-features through Preceiver like architecture. The authors propose to initialize the queries using entity word-embedding and bounding-box coordinates, to boost the localization of the sampling queries.

In addition, to avoid any information leakage between subjects, the authors use a Masked Cross attention where queries corresponding to a certain subject only attend the subject features.The proposed method also uses element-wise masking to avoid injected features from the reference images into background patches.
weaknesses: [1] The paper is hard to follow and some parts need further clarification. In particular, The “Multi-subject Cross Attention” section is not clear and needs major revision. 

Lack of ablation on critical parts of the methods: 

[2] Using videos for Data collection (e.g, in which ref. Image is different from gt. image) is claimed to be a key contribution. However, there’s a lack of ablation on how much this method helps the authors.

[3] In the Grounding Resampler; how important is the initialization (words+bounding box) to the module localization capability ?

[4] Lacking qualitative samples of interesting use-cases such as - spatial relation (e.g, On-top) and object-interactions (e.g., two subjects playing tennis). The only interesting case I find in the paper is fitted to “virtual try-on” which could be a by-product of the training data.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.	The grounding resampler proposed in the article also compresses image features. Why is it better than the image encoder used in existing methods? Can more qualitative analysis experiments be provided?

2.	The experiments in the article are based on Stable Diffusion XL. Now, newly emerging text-to-image diffusion models such as Flux inherently have better control over the generated content. Do these models not need an additionally trained structure to achieve satisfactory control?
soundness: 3
strengths: 1.	The paper introduced the first layout-guided zero-shot image personalization with multiple subjects framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities.

2.	The idea of decoupling and controlling the generation of the texture and position of the subject in the image is very reasonable.

3.	The paper is well written and easy to follow. The elaboration of the idea is very clear, and the framework of the structure diagram is also very easy to understand.

4.	The experiments are also very sufficient.
summary: Text-to-image generation models have advanced but face challenges in multi-subject scenarios. The MS-Diffusion framework is introduced to address these. It uses grounding tokens and feature resampler for detail, and layout guided cross-attention for location control. Experiments show it surpasses existing models in fidelity, promoting personalized text-to-image generation.
weaknesses: 1.	The practice of controlling image generation through local cross attention is not innovative enough. It has been widely adopted in many existing layout-guided text-to-image generation methods.

2.	Judging from the quantitative and qualitative experimental results provided in the article, the improvement of image generation effect compared to existing methods is relatively limited.

3.	The extraction method of image features used by the grounding resampler proposed in the article has also been used in past papers, and its innovation is limited.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: How does it perform in scenes where the theme is people and anime? Especially the problem of combining people with anime characters.
soundness: 3
strengths: 1.The authors alleviate the problem of combining natural objects in two thematic scenarios.
2.The framework is easy to think of and sensible.
3.The writing is clear and easy to understand.
summary: This paper proposes multiple subjects (MS-Diffusion) framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities, the provision of layout guidance, and the preservation of the foundational model's parameters. 
MS-diffusion explicitly utilizes the layout information of the reference images to extract the information of multiple subjects separately to inject into the base model.
weaknesses: 1.Missing results for multiple topics (>2).
2.Missing results for comparison with paper [1].
3.The authors do not mention the Image encoder used.To the best of our knowledge, the Image encoder if it is a CLIP may lose the details of the themes, leading to the results in the last graph of Figure 4 and Figure 5.
4.From the CLIP-I scores in Table 2, it seems that text fidelity is not significantly improved.
5.How does it perform in scenes where the theme is people and anime?
[1]Cones 2: Customizable Image Synthesis with Multiple Subjects

Meta-review: This paper introduces MS-Diffusion, a new approach for zero-shot image personalization that incorporates multiple subjects and uses layout guidance. The proposed grounding resampler and multi-subject cross-attention mechanism aim to enhance image generation tasks by focusing on particular areas and subjects within an image. The reported results suggest that the method can achieve qualitative and quantitative improvements over existing techniques.

The paper's main strengths lie in its novel contributions to the field of zero-shot image personalization. The introduction of the grounding resampler and cross-attention mechanism is innovative and has the potential to push forward the capabilities of image generation models. Additionally, the paper is well-structured and effectively communicates the proposed methodologies and their potential impact.

Reviewers have pointed out concerns regarding the level of innovation in the use of local cross-attention for controlling image generation. Questions about the comparative performance of the grounding resampler to existing methods were raised, as well as the need for more comprehensive quantitative evaluations. There were also criticisms about the loss of detail when using the CLIP image encoder.

The paper provides valuable contributions to zero-shot image personalization and has effectively addressed the main concerns raised during the review process. The remaining points of criticism are not critical enough to override the decision to accept, given the overall strength of the paper and the positive changes made in response to the review feedback.

Decision: Accept (Poster)