MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance
Authors: Xierui Wang
Siming Fu
Qihan Huang
Wanggui He
Hao Jiang
Venue: ICLR 2025 Poster (2025)
Keywords: Image Personalization
Multiple Subjects
Diffusion Models
Primary area: generative models
TL;DR: An approach of zero-shot image personalization supporting multiple subjects and layout guidance.

Abstract: Recent advancements in text-to-image generation models have dramatically enhanced the generation of photorealistic images from textual prompts, leading to an increased interest in personalized text-to-image applications, particularly in multi-subject scenarios. However, these advances are hindered by two main challenges: firstly, the need to accurately maintain the details of each referenced subject in accordance with the textual descriptions; and secondly, the difficulty in achieving a cohesive representation of multiple subjects in a single image without introducing inconsistencies. To address these concerns, our research introduces the MS-Diffusion framework for layout-guided zero-shot image personalization with multi-subjects. This innovative approach integrates grounding tokens with the feature resampler to maintain detail fidelity among subjects. With the layout guidance, MS-Diffusion further improves the cross-attention to adapt to the multi-subject inputs, ensuring that each subject condition acts on specific areas. The proposed multi-subject cross-attention orchestrates harmonious inter-subject compositions while preserving the control of texts. Comprehensive quantitative and qualitative experiments affirm that this method surpasses existing models in both image and text fidelity, promoting the development of personalized text-to-image generation.

dog and cat dog and cat Published as a conference paper at ICLR 2025
MS- MS-DIFFUSION: MULTI-SUBJECT ZERO-SHOT IMAGE PERSONALIZATION WITH LAYOUT GUIDANCE
Xierui Wang**
Siming Fu
Qihan Huang"
Wanggui He!
Hao Jiang*
Alibaba Group Zhejiang University
"quual contribution
"Corresponding author
Single-subject Personalization
Subject
dog in
dog top of purple nug
dog wearing pink glasses
wet dog
subte- Personalization
Subjects
Subjects
woman wearing jacket, and jeans
woman wearing a jocket, and jeons the
Subjects
a backpack and stuffed animal jungle
a backpack and stuffed animal the grass
Figure 1: Representative outputs showcase the capabilities The MS-Diffusion framework facilitates personalization across both single-subject scenarios (the upper panel) and multi-subject contexts (the lower panel). Notably, while preserving the intricacies of subject detail, MS-Diffusion achieves marked enhancement in textual fidelity.
Diffuffuusio in typical applications.
ABSTRACT
Recent advancements in text-to-image generation models have dramatically en- hanced generation photorealistic images from textual prompts, leading increased interest in personalized text-to-image applications, particularly in multi-subject scenarios. However, these advances are hindered by two main challenges: firstly, the need to accurately maintain the details of each referenced subject in accordance with the textual descriptions; and secondly, the difficulty achieving a cohesive representation of multiple subjects single image without introducing inconsistencies. To address these concerns, our research introduces the MS-Diffusion framework for layout-guided zero-shot image personalization with multi-subjects. This innovative approach integrates grounding tokens with the fea- ture resampler to maintain detail fidelity among subjects. With the layout guidance, MS-Diffusion further improves the cross-attention to adapt to the multi-subject inputs, ensuring that each subject condition acts on specific areas. The proposed multi-subject coss-attention orchestrates harmonious inter-subject compositions while preserving the control texts. Comprehensive quantitative and qualitative experiments affirm that this method surpasses existing models in both image and text fidelity, promoting the development of personalized text-to-image generation. The project page is https: /SS Di wwff-iifffuueee github.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Recent advancements in text-to-image (T2I) diffusion methodologies (Rombach et al. 2022 Saharia et al. [022] Betker et al.) (023) have propelled the field to new heights, realizing unprecedented levels of photorealism while demonstrating a refined ability to conform to textual prompts. These achievements have spurred the development of broad spectrum of applications, most prominently in the domain of personalized T2I (P-T2I) models, which are tasked with the complex undertaking of assimilating and regenerating novel visual concepts or subjects across diverse contexts with
heightened demand for conceptual and compositional fidelity. Despite fine-tuning based techniques such as DreamBooth (Ruiz et al. 2023) and Textual Inversion (Gal et al., 2023) yield results with considerable accuracy, they necessitate extensive resources for tuning individual instances and for the storage of multiple models, which renders them less feasible for widespread application. To circumvent these resource-intensive requirements, fine-tuning- free alternatives have come to the fore.
Single-subject driven personalization methods, IP-Adapter (Ye et al. 2023) and ELITE (Wei et al.) 2023) for instance, introduce a specialized cross-attention mechanism that distinctly processes text and image features, thereby affording the possibility to employ reference images directly as visual prompts within the model. Furthermore, recent works have employed multi-subject driven customization methodologies to concatenate visual concepts with textual prompts, offering glimpse of the potential in techniques like ctt al. 2023). and KOSMOS-G (Pan et al. 2023). ). These models harness identity data and amalgamate it with text via atssstattnnion, exhibiting proficiency in adjusting textures. Nevertheless, above zero-shot personalization methods encounter limitations, notably in adapting a pressing question pertains to the congruence of granular details between the subject depicted in the synthesized imagery and its corresponding subjects, along with the degree of semblance between the content of the generated image and associated textual descriptions. The challenge is further amplified in scenarios requiring the personalization of multiple subjects. Especially, the challenge of ensuring harmonious representation when multiple subjects are incorporated-speelie resultant image manifests any discordant elements or deleterious interactions in accordance with textual directives and multi-subject referential controls. As illustrated in Figure 2 multi-subject personalization methods frequently incur notable detail inaccuracies in a fine-tuning free framework and often lead to subject neglect, subject overcontrol, and subject-subject conflict issue within the generated images.
(Zhang et al. 2024), X-ECLIPSE (Patel et al.) (2024), Emu2 (Sun)
the elucidation of whether the
To confront these identified challenges, we are the first to introduce the layout-guided zero-shot image personalization with multiple subjects (MS-Diffusion) framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities, the provision of layout guidance, and the preservation of the foundational model's parameters. Firstly, we design the grounding resampler to extract the subject detailed features and integrate them with grounding information containing entities and boxes. As an image projection module, the proposed grounding resampler can enhance the subject fidelity while appending semantic and positional priors. Secondly, we propose novel cross-altention mechanism for multiple subjects, which confines subjects to represent themselves in specific areas. This confluence not only facilitates the efficacious infusion of multi-subject data into the model but also mitigates conflicts between text and image subject control conditions. Such an approach culminates in the refined granularity of control over the image's 's multi- subject composition. The experimental results demonstrate our method consistently outperforms the state-of-ther thettrrrrt approaches on all the benchmarks. We conclude the previous P-T21 works and provide an overall comparison in Table 1 The contributions can be summarized as follows:
We introduce a layout-guided, zero-shot multi-subject image personalization framework within the diffusion model paradigm, designated as 'MS-Diffusion'. This innovation stream- lines the complex process of preserving detailed subject references. Moreover, it seamlessly integrates multiple subjects into a coherent and harmonious personalized image.
The "Gronnding Resampler' is advanced as a novel feature refinement mechanism. This construct enriches the detail extraction from images by ascertaining the correlative content and fusing it with box embeddings that demarcate the anticipated spatial zones for each subject. Additionally, we introduce a specialized multi-subject cross-attention mechanism, confronting and rectifying prevalent complications in multi-subjec personalization, includ- ing inadvertent subject neglect, disproportionate subject dominance, and internecine subject conflicts.
2

---

Published as a conference paper at ICLR 2025
Table 1: An overview of previous studies of P-T21 tasks. MS-Diffusion the first approach to
support multi-reference zero-shot P-T2I generation with layout guidance and base model freezing.
Zero Shot
Multi Subject
Base Model
MLLM
Layout
Method
Freezing
Free
Guidance
Textual Inversion (Gal et
2023
X
x
DreamBooth (Ruiz et al.
2023)
x
x
ELITE (Wei et al. 2023
X
BLIP-Diftusion
2023a
X
X
IP-Adetter Ye et al.
2023
X
Emu2 [Sun et al. 20233)
Kosmos-Gi G (Pan al. (023)
X-ECLIPSE (Patel et al. 20244
X
EeR-Encoder (Zhang et al. 2024
MS-Diffusion
Input
Case A: subject neglect
Case B:
subject
Case C: subject subject conflict
MS-Diffusion
dog and a cat beach
Figure 2: Challenges inherent in multi-subject personalization approaches. Through the explicit layout guidance, MS-Diffusion addresses these challenges by directing subject conditioning to specific areas, simultaneously maintaining high image fidelity.
The capabilities of MS-DDifusion'' are empirically substantiated through its ability to synthesize broader spectrum of images with notable fidelity. The paper further delineates comprehensive ablation studies, underpinning the rationale behind our design decisions and affirming the efficacy of our proposed approach.
RELATED WORK
2.1 TEXT-TO-IMAGE GENERATION
2023 Esser et al. 2024 Podell al. 2023 capable of producing high-quality images using providdo text prompts. In models have shown strong performance in text-to-image tasks. Stable Diffusion (Rombach et al., (202)) proposes conducting the diffusion process latent space rather than pixel space, which reduces the sampling steps without compromising image quality. Kandinsky (Raz) 7higaev et al. 2023) takes both text embedding and image embedding as conditions to generate images more controllablely. DALLE-3 (Betker et al., 2023) recaptions the training data pairs and utilizes T5-XXL (Chung et al.
Text-to-image generative models (Saharia et al. 2022 Bao et al.
recent times, diffusion-
2022) as the text encoder to strengthen the prompt-following ability.
StableCascade (Pernias et al. 2023 presents cascaded architecture leverage outputs of front stages as priors, further reducing the latent space. PixArt-a (Chen et al. 2023a) also employs large T5 text encoder and replaces the original U-Net backbone with transformer (Peebles & Xie 2023). These models focus on the basic text-to-image ability and cannot handle the situation when users provide specific subjects.
3

---

Published as a conference paper at ICLR 2025
entities
Image Encoder
Grounding Resampler
Text Encoder
dog and the beach
baxes
Apn
15
1 Ma
freeze
Grounding Resampler
Multi-subject Cross-attention
paciding tokens
Figure 3: The overall pipeline of MS-Diffusion. It introduces two pivotal enhancements to the model: the grounding resampler and multi-subject coos-attentionn mechanisms. Firstly, the grounding resampler adeptly assimilates visual information, correlating it with specific entities and spatial constraints. Subsequently, the gross-attention mechanism facilitates precise interactions between the image condition and the diffusion latent within the multi- multi-subject attention layers. Throughout the training phase, all components of the pre-existing diffusion model remain frozen.
To finely control text-to-image generation, some diffusion models support users in providing layout guidance. Layout Diffusion (Zheng et al. 2023) and GLIGEN (Li et al. (023c) input positions and labels of bounding boxes into the diffusion model and train it to learn the layout information. DenseDiffusion (Kim et al. (2033) develops training- foaning-freeee method and modulates the attention maps
the inference phase. Instance Diffusion (Wang al., 2024b) and MIGC (Zhou et al. 2024) extend the layout-conditioned diffusion to the instance level, enabling the model to generate multiple objects with precise quantities. While syyout-guide diffusion models have robust controllability. they cannot reference specific concepts, which is emphasized in personalized text-to-image generation.
2.2 TEXT-TO-IMAGE PERSONALIZATION
Text-to-image personalization (Han et al. 2023 Quu et al. 2023 Chen et al. 2024 Shi et al. 2024 Hu 20244 has attracted much attention
community for powerful referencing capability of both text and image prompts. Textual inversion (Gal et al. (023) and DreamBooth RRuzz et al. 2023 utilize an identifier in the to bind the visual concept through fine-tuning. IP-Adapter (Ye
et al. 2023) proposes a zero-shot personalized model by projecting the image embedding to the cross- attention layers. InstanceID (Wang al. (024a) develops an approach for identity personalization. replacing the image encoder with a face encoder and employing ControlNet (Zhang et al. (023) to integrate the face landmarks. To narrow the gap between image and text prompts, Kosmos-C G (Pan al. 2023) and X-ECLIPSE (Patel et al. (2024) conduct multi-modal training unite the inputs by text-image interleaving. SSR-Encoder (Zhang et al. 2024) design query network to extract single subject from images with multiple subjects for personalization.
Though past research in this field has significantly enhanced the ability to reference single subjects, few studies have explored zero-shot multi- subject personalized models. Moreover, existing related works struggle to address conflicts in personalized generation with multiple subjects and generate bad results, which is precisely the focus of our work, discuss and resolve these issues.
4

---

Published as a conference paper at ICLR 2025
3 METHOD
The pipeline of MS-Diffusion is shown in Figure Through an improved data construction strategy. we can get multiple subject images together with the corresponding entities and layouts as input. We propose grounding resampler separately extract the image features and integrate them with phrase embedding and box embedding for condition enhancement. Inside the cross-attention layers, we further introduce the masked cross-atie aass-tattntiion to guide the generation with layout priors and alleviate conflicts of multiple subjects. The training needs no pre-trained weights to be optimized and remains plug-an and and-alay play in various base models.
3.1 PRELIMINARIES
Stable Diffusion with Image Prompt. As a widely used diffusion model, Stable Diffusion (SD) (Rombach et al., (2022) conducts the diffusion process in the latent space. Given an im- age and prompt, SD encodes them into latent code and condition embedding C utilizing VAE (van den Oord et al. 2017) and CLIP (Radford et al., 2021) text encoder, respectively. In zero-shot image personalization architectures like Adateer Ye et al. 2023). images can also be considered a condition of the diffusion model. Specifically, a subject image is encoded to image embeddings by an image encoder and then projected into the original condition space of the diffusion model denoted as c;. For timestep / which is uniformly sampled from fixed range, the model ( predicts the noise Eg and is optimized through the objective:
LIP = Ez,,,,. [|ee |le- Es (ze Ce, cr, t)]l
where (N,, I). In this work, employ LIP with SDXL (Podell al. 2023) the pre-trained model, which contains two CLIP text encoders and additional condition inputs besides a and t, omitted for brevity.
Cross-attention. In IP-Adapter Ye al. (003)), both c, and c integrated into the U-Net backbone through cross- artss-attention layers:
QK! Vd
QKI
Attn (Q, K,, V.. Vi) zy Zsmg + Zirt y Softmax
V, + Softmax
Ve
Vd
(2)
where Q zW, 4. K; c, W. K, c.W. V; c; " V. c,W. and W g. We. W. corresponding projection weight matrices, and d represents the dimensionality of the key vectors. Note that the key and value matrix of c; and e, are independent of each other to decouple conditions of different modalities. Previous studies (Hertz et al. 2023 Tang et al. 2023) have found that
attention maps A = Softmax [QK' /vd) can reflect the attribution relation between generated images and conditions, which means that they determine the effect of condition controls.
3.2 DISCUSSION ON SUBJECT IMAGE PERSONALIZATION
widely used method for achieving multi-subject image personalization involves training individual models for each subject, followed by their integration. Tuning-based methods (Gu et al. 2023
Avrahami et al. 2023 Liu et al. 2023b Kumari al. 2023) with improvements visual conflicts can produce impressive multi-subject personalized images. However, zero-shot methods eliminate the need for individual subject tuning the merging of different combinations, significantly reducing costs and enhancing the practicality of multi-subjer personalization. This makes research into zero-shot multi-subject image personalization both necessary and promising.
Most of the relevant studies (Ma et al.) 2024: Gu et al. 2023 Liu et al., 2023b Xiao et al. 2024
focus on mitigating visual conflicts text cross-attentions of the base model. While modifying text cross-attention can be effective, it presents certain limitations. First, adjustments to text cross- does not
attention can directly impact the control over text conditions. Second, text directly dictate the areas of influence for image conditions; rather, it exerts indirect influence image conditions by shaping the image layout generated by the diffusion model. This indirect control may result in low performance and increased uncertainty.
5

---

Published as a conference paper at ICLR 2025
3.3 DATA CONSTRUCTION
In the field of multi-subject personalization, creating a robust dataset architecture is a significant challenge, especially when no pre-existing dataset includes variety of reference subjects with their validated truths. Our method starts with applying a Named Entity Recognition (NER) protocol to textual data to extract relevant entities. These entities are then used within detection framework define the corresponding bounding boxes. This step generates training samples that encompass range of [subjects, entities, spatial layouts].
Previous studies have mostly created training examples from stand-alone images which is essentially reconstruction task, leading the resulting models to favor replication, often resulting in 'copy-and- paste" artifacts (Chen et al., 2023b). To address this issuc, our enhanced approach involves extracting subjects from a single video frame and using another frame from the same sequence as reference for the truth. This technique effectively separates the personalized references from the target images. Due specialized subject-matching algorithm
to possible variations in subjects between frames, to ensure accurate matching. We provide a detailed description this data processing pipeline in Section A
3.4 GROUNDING RESAMPLER
Different from text embedding, image embedding generally contains more information and is sparser, making projection into the condition space challenging. Leveraging embeddings from all image patches primarily control the condition, but the pooled output from the image encoder tends to omit many details. Different from Flamingo (Alayrac et al. 2022) and IP-Adapter (Ye et al., 2023). we propose the integration of grounding resampler that functions projector. Utilizing a set of learnable tokens, a resampler queries and distills pertinent information from the image features. Specifically, with image embedding fi and learnable query fạ. the resampler comprises several attention layers:
alternative form of image
(Qx)K (ffe.f.))) V (ffe..fg)
RSAttn Softmax
Vd
where [fi, fg] denotes the concatenation of the image embedding fi and the learnable query fg- The architecture incorporates fully connected feedforward networks (FFNs), analogous to those utilized in standard vision transformers (Dosovitskiy et al. 2021).
As detailed in 3.3 we can obtain entities of multiple referenced subjects and their target area boxes in the generated image. We present to initialize the query Jg with grounding tokens fg derived from text embedding of entities and Fourier embedding of boxes. Entities are related to the semantic information of images and boxes indicate the areas where the subjects are supposed to be. would be helpful for the resampler to extract the image features appropriately and the cross-attention layers to condition the generation finely. To prevent the model from becoming dependent on the grounding tokens during inference, randomly replace them with the original learnable queries in the training. For the input of subjects, the projection processes of different subject images do nor affect each other. The resulting queries will be concatenated and input into the subsequent model as e; with N = n * ne tokens, where ni is the token quantity per subject.
3.5 MULTI-SUBIECT CROSS-ATTENTION
In scenarios involving the generation of multiple subjects, challenges frequently arise that are not exclusive to personalization tasks. These include discordances between subjects and their backgrounds and amongst the subjects themselves. A viable solution to mitigate such conflicts leverages attention masks, contingent upon the availability of layout priors. The incorporation of attention masks within cross-attention mechanisms facilitates the exclusion of padding tokens from the condition, thus minimizing their impact.
To confine the context of each subject to a designated spatial domain, we propose an enhancement to the conventional attention mask, denoted as M. This adjustment involves the bilateral neglection of

---

Published as a conference paper at ICLR 2025
tokens within both the query and key matrices, applied specifically for the jth subject follows:
if [x. yl E Bj
M2 v)
-00 if
[ m| $ 13,
Here, By signifies the coordinate set of bounding boxes pertaining to the jth subject. By this means, the conditional image latent iin is derived through:
QK
mg
Softmax
M) V
(5)
Vd
Herein, M represents the amalgamation of all subject-specific masks, Concat(Mo...... M,). In this way, the model ensures each subject be represented in certain area, thus resolving the issues of subject neglect and conflict in Figure
However, inherent limitation arises when query patch token is ubiquitously masked across all referenced subjects or remains unmasked (in instances of overlapping bounding boxes), thereby diminishing the intended efficacy of multi-subject cross-attention. To counteract this, introduce dummy tokens initialized at random preceding the image tokens to symbolize the background. This strategy instrumental in ensuring that text conditions predominantly govern areas devoid of any guided layout, thereby solving the subject overcontrol issue in Figure 2
Following the acquisition of A, we apply Mog to seamlessly mask these tokens within 2amgs as illustrated:
Zing (1 Màg) Zing
(6)
where May is articulated binary mask, with elements within the subject bounding boxes des- ignated as zero. In contrast to the methods discussed in Section 3.2 our proposed multi-subject cross-attention directly manages the image conditions by employing masked image cross-attention in the targeted areas. While addressing multi-object confiits, our method ensures that text conditions remain unaffected, evidenced by the significantly higher text adherence capability shown in Table 2 Notably, certain studies have sought to resolve these conflicts through the application of objectives on attention maps within the cross-attention mechanism. series of rigorous experiments have been conducted to substantiate our design, with the details elucidated in Section 4.4
4
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. For training, we utilize an in-house video dataset that contains 3.6M video clips. For evaluation, we measure the single-subject and multi-subject performance on DreamBench (Ruiz et al., 2023) and MS-Bench respectively. DreamBench includes 30 subjects and 25 prompts and we preset all input boxes to [0.25, 0.25, 0.75, 0.75J. To thoroughly assess the performance of multi-subject personalization,
have established new evaluation standard, MS-Bench, which includes 40 subjects and 1 1148 combinations, each combination having up to 6 prompts, totaling 4488 distinct test samples. Details of datasets are provided in Section A and Section B
Evaluation metrics. Following previous works, we measure the performance through image and text fidelity. To assess image fidelity, we employ cosine similarity measures between generated images and subject images within CLIP (Radford et al. [2021] and DINO (Caron et al. 2021 spaces, referred as CLIP-I and DINO, respectively. For text fidelity, we utilize cosine similarity between generated images and text prompts in CLIP space, denoted as CLIP-T. In multi-subject personalization, using the average fidelity reflect image fidelity is insufficient, as it fails to reveal cases of subject neglect. We further employ the product of multi-subject DINO, denoted as M-DINO, to indicate whether each subject has been recreated in the results.
Baselines. For single-subject personalization, we compare our model with methods mentioned in Table I Emu2 (Sun et al. 2023), Kosmos-G (Pan et al. 2023), and X-ECLIPSE (Patel et al. 2024) are all MLLM-basee methods, while X-ECLIPSE is reported to have better performance. Therefore. (Zhang et al. 2024) and X-ECLIPSE (Patel et al. 2024) as baselines for we select personalization. Considering the faimesss, the qualitative results of MS-Diffl D-DDfusfuin are multi-subject generated without any fine-tuning on benchmarks. The implementation details of MS- IS-Diffusion and these methods contained in Section C
7

---

Published as a conference paper at ICLR 2025
Subject
Prompt
BLIP-Diftusion
IP-Adapter
X-ECLIPSE
3SR-Encoder
MS-Diffusion
cat wearing
stuffed animal Footing top of water
bowl top white nag
vase with city background
Figure 4: Qualitative results of MS-Diffusion and baselines in single-subject personalization.
Table Quantitative comparison on MS-Diffusion and baselines. Bold and underline represent the highest and second-highest metrics, respectively. For single-subject, the results for Id-Aterter, Emu2, and Kosmos-G were obtained from X-ECLIPSE (Patel et al.) 2024), while the rest are reported the corresponding papers. denotes the model fine-tuned on DreamBench.
Single-subject
Multi-subject
Method
CLIP-I
DINO
CLIP-T
CLIP-I DINO M-DINO CLIP-T
Textual Inversion
0.780
0.569
0.255
DreamBooth
0.803
0.668
0.305
BLIP-Diffusion
0.805
0.670
0.302
X-ECLIPSE"
0.796
0.682
0.304
MS-Diffusion'
0.805
0.702
0.313
ELITE
0.771
0.621
0.293
0.779
0.594
0.300
IP-Adapter
0.810
0.613
0.292
Emu2
0.765
0.563
0.273
Kosmos-G
0.822
0.618
0.250
X-ECLIPSE
0.783
0.613
0.307
0.724
0.419
0.094
0.316
SSR-Encoder
0.821
0.612
0.308
0.725
0.425
0.107
0.303
MS-Diffusion
0.792
0.671
0.321
0.698
0.425
0.108
0.341
4.2 SINGLE-SUBJECT COMPARISON
In the single-subject comparison, detailed examination is carried out utilizing both qualitative and quantitative comparisons to gauge the performance of different methodologies. On the qualitative front, as shown Figure 4 MS-Diffusion shows an exceptional ability to generate single-subject images with high fidelity and detail retention. In quantitative results provided in Table 2 MS- Diffusion also achieves competitive scores, with obviously the highest DINO and CLIP-T scores at 0.671 and 0.321 respectively on zero-shot scenarios, and leading CLIP-I score of 0.792. For tuning methods, MSS-Diffusion outperforms baselines in metrics. As discussed in DreamBooth (Ruiz al. 2023), DINO more accurately captures the similarity in details between the results and the
8

---

Published as a conference paper at ICLR 2025
Subjects
Prompt
A-ECLIPSE
MS-Diffusion
teapot in room
wearing dress snow
flower born background
woman wearing a 1-shirt and shorts the beach
Figure 5: Qualitative results of MS-Diffusion and baselines in multi-subject personalization.
Table 3: Ablation study of MS-Diffusion. RS, GRS, MCA, TAL., IAL, and LG represent resampler,
grounding resampler, multi-subject cross-attention, text attention loss, image attention loss, and layout guidance. respectively.
Single-subject
Multi-subject
Method
CLIP-I
DINO
CLIP-T
CLIP-I
DINO
M-DINO CLIP-T
MS-Diffusion
0.792
0.671
0.321
0.698
0.425
0.108
0.341
w/o RS
0.775
0.583
0.320
0.680
0.372
0.082
0.336
w/o GRS
0.777
0.646
0.320
0.681
0.389
0.090
0.331
w/o MCA
0.798
0.662
0.312
0.693
0.422
0.100
0.309
w/6 LG w/ WTAL
0.761
0.577
0.284
0.675
0.377
0.080
"0305
w/o LG w/ TAL
0.809
0.660
0.293
0.687
0.413
0.093
0.316
labels, whereas CLIP-I may exhibit high scores in situations of background overfitting, resulting in clear advantage for DINO, but a slight disadvantage for CLIP-I of MS-Diffusion.
4.3 SULJEI-UBJJECT COMPARISON
From qualitative perspective in Figure 5 MS--iifussion manages to maintain natural interactions among subjects generated images while ensuring each subject retains its distinctiveness and recognizability. Quantitatively, results in Table demonstrate the strength of MS-Diffusion in DINO, M-DINO, and CLIP-T. Unlike in single-subject personalization, there a larger gap in text fidelity between MS-Diffusion and the baselines in multi-subject personalization, demonstrating that MS- Diffusion not only effectively generates the multiple subjects outlined in the text but also excellently preserves the text control capabilities inherent to SD. Additionally, the image fidelity of MS-Diffusion is comparable, highlighting its superior ability to retain details, particularly significant as low text fidelity commonly associated with overfitting.
4.4 ABLATION STUDY
Module ablation. We conduct ablation experiment the proposed two modules, grounding resampler (GRS) and multi-subject cross-attention (MCA), to validate their effects. For GRS, we replace with linear projection layer and normal resampler (Alayrae et al. 2022 Ye et al. 2023).

---

Published as a conference paper at ICLR 2025
Subjects
w/o
w/o GRS
w/o MCA
MS MS-Diffusion
dog
dog
woman wearing jacket and shorts cobblestone street
Figure 6: Visualization results of module ablation. Models without RS or GRS have an obvious decrease in the detail-pro peeesl-preserving capability. For the multi-subject generation, the model without MCA cannot handle the subject conflicts.
Results in Table indicate that the resampler-lin image projector significantly enhances the details, as evidenced by DINO being obviouly higher than the linear projector. Moreover, The substantial improvement in multi-object image fidelity by GRS reflects the critical role of the information carried by grounding tokens in multi-object generation. As key module for resolving conflicts, removing MCA results in noticeable degradation of text fidelity, especially in multi-subject generation. The combined use of both modules ensures that MS-Diffusion maintains high image and text fidelity simultaneously. We provide visualization results regarding the module ablation in Figure As clearly reflected by the qualitative examples, GRS enhances the details and MCA handles the conflicts.
Layout guidance. As mentioned Section 3.5 we have explored the indispensable role of explicit layout guidance (LG), including grounding tokens and MCA. A straightforward approach to implicitly utilizing layout involves incorporating an attention loss during training. Besides the image attention loss (IAL., also introduce text attention loss (TAL) training by setting the original cross- attention layers trainable. The detailed loss definition is provided in Section G As illustrated in Table 3. an objective to guide the image cross-attention helps the personalization hardly all. TAL has somewhat resolved the conflict issues, but its performance is inferior to MS- MS-Diffusion while introducing additional training parameters. We consider the inclusion of LG necessary and rational, not merely for the performance enhancements it offers, but also because it effectively resolves the various multi-object generation issues highlighted Figure]
5 CONCLUSION
This study makes significant contribution to the field of P-T2I diffusion models with the development of MS-Diffusion. This zero-shot framework excels at capturing intricate subject details and smoothly blending multiple subjects into single coherent image. Equipped with the innovative Grounding Resampler and Multi-subject Cross-altention mechanisms, our model effectively overcomes common multi-subject personalization issues, such as subject neglect and conflict. Extensive ablation studies underscore Ii-Diifussion 's enhanced performance in image synthesis fidelity compared to existing models. stands as a groundbreaking approach for P-T2I applications that are free from the need for fine-tuning and require layout guidance.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from single image. In SIGGRAPHI Asia, pp. 96:1-96: 12. ACM, 2023.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: vit backbone for diffusion models. In CVPR, pp. 22669-22679. IEEE, 2023.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall---. pdf. 2:3, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, pp. 18392-18402 IEEE, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. ICCV, 2021.
Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation. In ICLR. OpenReview.net, 2024.
Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. CoRR, abs/2310.00426, 2023a.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. CoRR, abs/2307.0944 2023b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instructionfinetunee language models. arXiv preprint arXiv:2210.11 14111 2022.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, pp. 2019.
Computer Vision Foundation IEEE,
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICML. OpenReview.net, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or, An image is worth one word: Personalizing text-to-image vo-immae generation using textual inversion. In ICLR. OpenReview.net 2023.
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang. Weijia Wu, Yixiao Ge, Ying Shan, and Mike Zheng Shou. Mix-of- show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. In NeurlPS, 2023.
11

---

Published as a conference paper at ICLR 2025
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris N. Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In ICCV, pp. 7289-7300. IEEE, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Prompt- to-prompt image editing with cross aooss-tttentoon control. ICLR. OpenReview.net, 2023.
Hexiang Hu, Kelvin CK Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, et al. Instruct-imagen: Image generation with multi-modal instruction. In CVPR, pp. 4754-4763, 2024.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models, In CVPR, pp. 6007-6017. IEEE, 2023.
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung- Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV, 2023.
Alexander Kirillov. Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, pp. 3992-4003, IEEE, 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR, pp. 1931-1941. TEEE, 2023.
Dongxu Li, Junnan Li, and Steven C. H. Hoi. Blip-diffusion: Pre-trained subject representation for controllable eet--to-image generation and editing. In NeurlPS, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2:
bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 19742. PMLR, 2023b.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. CVPR, 2023c.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistie human photos via stacked ID embedding. In CVPR, pp. 8640-8650. IEEE, 2024.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0555 0559999999 2023a.
Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Customizable image synthesis with multiple subjects. In NeurIPS, 2023b.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. In SIGGRAPH, pp. 25. ACM, 2024.
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. CoRR, abs/23 10.02992, 2023.
Maitreya Patel, Sangmin Jung, Chitta Baral, and Yezhou Yang. X-eclipse: Multi-concept personalized text-to-image diffusion models by leveraging CLIP latent space. CoRR, abs/2402.05195,, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pp. 4172-4182. IEEE, 2023.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfih International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2025
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307.01952 2023.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In NeurIPS, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8766.. PMLR, 2021.
Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky: An improved text-to-image synthesis with image prior and latent diffusion. In EMNLP, pp. 286-295. Association for Computational Linguistics, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10674-10685. IEEE, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, pp. 22500-22510. IEEE, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. NewrlPS, 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. In CVPR, pp. 8543-8552, 2024.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. CoRR, Abs/2332.11322 2023.
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang. Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the DAAM: interpreting stable diffusion using cross attention. In ACL, pp. 5644-5659, Association for Computational Linguistics, 2023.
Aäron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NewrIPS, pp. 6306-6315, 2017.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.07519, 2024a.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffu- sion: Instance-level control for image generation. In CVPR, pp. 6232-6242, 2024b.
Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Grounding diffusion with token-level supervision. CoRR, abs/2312.0366 2023.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. ELITE: encoding visual concepts into textual embeddings for customized text-to-image generation. In ICCV, pp. 15897-15907. IEEE, 2023.
Zhichao Wei, Qingkun Su, Long Qin, and Weizhi Wang. Mm-diff: High-fidelity image personalization via multi-modal condition integration. CoRR, abs/2403. 2024.
Guangxuan Xiao, Tianwei Yin. William T Freeman, Frédo Durand, and Song Han. Fastcomposer: Tuning-free free multi-subject image generation with localized attention. International Journal of Computer Vision, pp. 1-20, 2024.
13

---

Published as a conference paper at ICLR 2025
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for sat-to-image diffusion models. CoRR, abs/2308.06721, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 3813-31331 3-3824, IEEE, 2023.
Yuxuan Zhang, Yiren Song. Jiaming Liu, Rui Wang, Jinpeng Yu, Hao Tang, Huaxia Li, Xu Tang, Yao Hu, Han Pan, al. Ssr-encoder: Encoding selective subject representation for subject-driven generation. In CVPR, pp. 8069-8078, 2024.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for tomaaee generation. In CVPR, pp. 22490-22499. IEEE, 2023.
Dewei Zhou, You Li, Fan Ma. Xiaoting Zhang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In CVPR, 6818-6822 2024.
14

---

Published as a conference paper at ICLR 2025
A TRAINING DATASET CONSTRUCTION PIPELINE
Figure illustrates the construction pipeline of the training dataset. Firstly, we randomly select two frames from a video clip, one as the reference and the other as the ground truth. Both frames captioned by BLIP-2 2 (Li et al. 2023b). Secondly, from the caption. Entities and images are then input into Grounding DINO (Liu et al., 2023a) to obtain the boxes, which are parts of the final input of the model. Taking the boxes prompts of SAM (Kirillov et al. 2.23), we can further obtain segmentation masks to extract subjects from the reference image. Since the entities in different frames which finds the correspondence between the frames by conducting Hungarian Algorithm on the entity image embeddings. Frames in video typically contain the same entities but exhibit clear differences in details such as angles and poses. This makes them highly suitable as training data for personalized image generation models, which can help mitigate the model's tendency copy-and-paste.
utilize NER mode to extract entities
be different, design subject matcher,
Our dataset comprises 2.8M general scenario videos and 0.8M product demonstration videos, where the former covers more scenarios and the latter has more clear subjects. 2-5 frames for each are adopted in the training. In practice, there may only be 1-2 subjects successfully matched. To ensure that the training data contains a sufficient number of reference subjects, for the targets where matching fails, we directly use the corresponding parts of the ground truth as references. Subjects that are too small, too large, or have imbalanced proportions are filtered out. Each training sample can have up to 4 subjects, and we pad in the ones with fewer than 4.
woman wearing a ned shirt, jeans and holding on the beach
6UP2
gt
Subject Matcher
Grounding DINO
aniities
red shirt
jeans
ref
refs
BUP2
SAM
training sample
woman wearing a ned shirt, holding badminton racket on the beach
masks
Data construction pipeline of our work. For the input of two frames, get subject images, entities, and boxes. Note that the entities and boxes are from the ground truth frame since they indicate the information in the generated result.
Figure 7:
B DETAILS OF MS-BENCH
To construct MS DS-Bench, we collect subjects from previous studies (Ruiz et al. 2023 Gal et al. 2023 Kumari al. 2023). the Internef" and an internal dataset that does not overlap with the training set. MS-Bench contains four data types and 13 combination types with two or three subjects. We provide the details in Table 4 Each combination type other than those related to the scene has
prompt variations. There are 1148 combinations and 4488 evaluation samples, where entities and
https://spacy.io. io/
https://unsplash. com/
15

---

Published as a conference paper at ICLR 2025
Table 4: Explanation of MS-Bench. Each combination type has preset prompts and boxes. [S] represents prompt variations about the scene. including "in a room",, "in the jungle", "in the snow", "on the beach", "on the grass", and "on a cobblestone street".
Type
Prompt
Boxes
living+living loving+object
[0,00, 0.25, 0.50, 0.75] [0.50, 0.25, 1.00, 0.75]
{0} and {1} [S]
object+object
[0.25, 0.25, 0.75, 0.75]
livin++upwearing
{0} wearing {1} [S]
[0.25, 0.00, 0.75, 0.25]
living+midwcaring
[0.25, 0.25, 0.75, 0.75]
(0) wearing (1) [S]
living-wholewearring
[0.25, 0.25, 0.75, 0.75]
woman wearing a (0)
[0.25, 0.25, 0.75, 0.60]
midwearing--dowwwwaring
and {1) [S]
[0.25, 0.60, 0.75, 1.00]
living+scene
{0} with {1}
[0.25, 0.25, 0.75, 0.75]
object+scene
in the background
[0.00, 0.00, 1.00, 1.00]
[0.00, 0.25, 0.35, 0.75]
livin--liing+-livnng
(0), (1}, and (2) [S]
[0.35, 0.25, 0.65, 0.75]
objectoojjcctobject
[0.65, 0.25, 1.00, 0.75]
[0.00, 0.25, 0.50, 0.75]
{0} and {1} with {2} in the background
livin=oobjectsseene
[0.50, 0.25, 1.00, 0.75]
[0.00, 0.00, 1.00, 1.00]
[0.25, 0.00, 0.75, 0.25]
upwearing--iddwerrinn- downwearing
a woman wearing {0). a (1},
[0.25, 0.25, 0.75, 0.60]
and {2] [S]
[0.25, 0.60, 0.75, 1.00]
boxes are subject categories and preset layouts. Compared to other multi-subject benchmarks, our MS-Bench ensures that the model performance can be reflected comprehensively in abundant cases.
C EXPERIMENT SETTINGS
Training and Inference. The pre-trained model employed MS-Diffusion is Stable Diffusion
XL (SDXL) (Podell et al.) 2023). Implemented by Pytorch 2.0.1 and Diffusers 0.23. 1, our model is trained on 16 A100 GPUs for 120k steps with a batch size of 8 and learning rate of 1e-4, Following the training of IP- IP-adapter Ye et al. 2023),
y 1.0 in cross crossattenntion layers and dropped the text and image condition using the same probability. To ensure the model is not dependent on the grounding tokens (Section 3.4). also randomly drop them with probability of 0.1. We generate five images for each sample during the inference, with unconditional guidance scale and y set to 7.5 and 0.6, respectively, get better results.
Comparative methods. quantitative experiments:
Here we provide the details of the baselines compared in qualitative and
BLIP-Diffusion Li
2023a utilizes BLIP-2 Li et 2023b) unite the text and image embeddings. We implement it in the qualitative comparison using Diffusers.
IP-Adapter (Ye et al.) 2023) also uses image prompt as the condition. We run qualita- tive samples their official code with the scale set to 0.5 recommended by the paper. Considering fairness, we use the result of SDXL in Table 2
SSR-Encoder (Zhang et al. 2024) design query network extract the specified subject single image, which enables to finish multi-subject generation. We leverage it as of the baselines in multi-subject personalization. For performance comparison,
16

---

Published as a conference paper at ICLR 2025
Subjects
Mix-of-Show
Cones2
MS-Diffusion
cat and teapot
a dog wearing a dress the snow
Figure 8: Qualitative comparison of Mix-of-Show, Cones2, and MS-Diffusion. MS-Diffusion outperforms Mix-of-Show and Cones2 in details preserving in the first row. Mix-of-Shoo and Cones2 struggle in handling the interactions between subjects.
employed the official code provided, alongside the default hyperparameters specified the code repository.
X-ECLIPSE (Patel et al. 2024) trains an independent multi-modal encoder and employs Kandinsky as the generative backbone. Since outperforms other MLLM-based approaches, choose it to be the representative. To facilitate comparative analysis of performance, the study utilized the officially provided code, in conjunction with the default byperparameters delineated within the corresponding code repository.
D MORE RESULTS OF SINGLE-SUBJECT PERSONALIZATION
Additional qualitative results on DreamBench are provided in Figure 12 MS-Diffusion shows excellent text fidelity in subjects while keeping subject details, especially the living ones (dogs). elements in the background (the third line and the fourth line) also occur in the results (the grass and the teapot holder) since the entire images are referenced during the generation. Their scope of action depends on the input bounding box. In practical applications, using masked images as a condition is recommended.
can be noticed that
One of the limitations in detail preservation is the insufficient capability of the image encoder. We pro- vide some uncommon examples in Figure 13 MS AS-Diffusion utilizes the CLIP image encoder, which results in the loss of some details in uncommon and complex cases. However, it still significantly outperforms state-of-the-an methods benefiting from the proposed grounding resampler.
E MORE RESULTS OF MULTI-SUBJECT PERSONALIZATION
We provide additional multi-subject personalized images based on MS-Bench in Figure 14 The results encompass various combination types. fully demonstrating the generalizability and robustness of MS-Diffusion. When the scene changes freely according to the text, the details of the subject are preserved without being affected. In addition common parallel combinations, MS-Diffusion also performs well in personalized generation for combinations with certain overlapping areas, such as "living+midwearing" and "objectsscene".
F COMPARISON WITH TUNING-BASED METHODS
While zero-shot methods like MS-Diffusion decrease the tuning cost, they may also suffer from performance degradation compared to tuning- based approaches due to the limitations of pre-training scale. However, IS-Diifussin indicates comparable performance in single-subject quantitative results
17

---

Published as a conference paper at ICLR 2025
Input
Break-A-Scene
MS-Diffusion
in the jungle
in the jungle
the beach
the beach
Figure 9: Qualitative comparison of Break-A-Scen and MS-Diffusion. MS-Diffusion gets comparable results by extracting subjects from a single image. Break-A-Scene tends to overfit the input image (the bird pose in the first row and the white creature sitting on the black bowl in the second row).
in T Since the proposed MS-Bench can be too large for tuning-based methods, we provide qualitative comparison results with Mix-of-Show (Gu et al. 2023), Cones2 (Liu et al.) 2023b), and Break-A-Scené (Avrahami et al. 2023 in Figure S and Figure The results show that MS-Diffusion achieves comparable results tuning- based methods. Mfx-of-Show and Cones2 face certain issues when handling multiple subjects with complex interactions (e.g., the example of a dog wearing a dress). Break-A A-Scene tends to overfit the original image's interactions (c.g., the pose of birds and the white creature sitting on the black bowl). While avoiding these issues, MS-Diffusion requires no test-time tuning and only subject image during inference, unlike the tuning-based methods that need additional time and multiple images for tuning.
G LAYOUT GUIDANCE
Cross-attention maps can intuitively reflect the condition-imant attribution relation (Tang al. 2023 |Hertz et al., (202)). Recent works (Wang et al., 2023 WWii et al.] (2024) have studied utilizing an objective on the aess-attention maps training, considered implicit and insufficient to handle multi-subject confficts. We have provided the performance comparison between our explicit layout guidance and attention loss Section 4.4 For single cross-attention layer, the attention loss of the jth subject is calculated by:
subject generation. The objective exists only in
C = -11- EjeWEEB A (m.sla
2
where [r. v] corresponds latent token in Q and By is the bounding box of the jth subject. This objective aims promote the activation of attention maps within specific boxes. We average C across layers and subjects and set its weight to 0.01 in the final loss. To validate the text attention loss, we also optimize the text coss-attention layers in training, increasing approximately 70% in learnable parameters.
Although our model provides explicit layout guidance, it still significantly differs from layout- based diffusion. Firstly, the information of boxes in the grounding resampler is prior, and its conditional effect relatively weak. We have also reduced the model's reliance on this input by randomly dropping grounding tokens. Secondly, the multi-subject cross-attention only exists in image cross-attention, inherently controlling the action image conditions in specified areas, but canmot determine the whole generation of the diffusion model. Our goal is not to develop method that fully supports layout control but to utilize layout information to guide the model in resolving conflicts in multi-subject generation.
18

---

Unlike image editing Kawar et al. 2023 Brooks et al., 2023 personalized image generation Published as a conference paper at ICLR 2025
Subjects
dog a backpack the grass
dog
backpack
Subjects
a woman wearing a shirt and shorts on the street
shirt
shorts
Figure 10: Text-image attribution analysis of MS-Diffusion. We average the attention maps corresponding to the subjects and translate them normalized heat maps.
To further explore the layout control capability of MS-Diffusion, we provide qualitative results in Figure [5] It can be demonstrated that MS-Diffusion can generate images that adhere to layout conditions, even in the case of two instances of the same category. However, the generated positions are not entirely accurate, especially "a cat and a cat on the grass", illustrating that the layout condition is relatively weak compared text and image prompts in the personalization task.
H INTEGRATION WITH CONTROLNET
In the realm of text-to-image diffusion models, notable application is the enhancement of structural control within image generation. Our MS-Diffusoin maintains the original network architecture unchanged, thus ensuring full compatibility with existing controllable tools. Consequently, this allows for the generation of images that are not only prompted by images but also governed by additional conditions. By integrating our MS-Diffusion with established controllable mechanisms such as ControlNet, we demonstrate the capacity to produce images under varied structural directives. Figure [6 [i] array of images synthesized with image prompts coupled with distinct structural controls(depth, canny edge, openpose). This seamless cooperation between our method and these tools facilitates the creation of highly controllable images without necessitating fine-tuning.
I MULTI-SUBIECT INTERACTION
features high degree of freedom, enabling effective handling of interactions among different elements, Benefiting from its architecture design that does not impact the base model, MS-Diffusine successfully inherits the multi-subject interaction capabilities of the base model. As illustrated in Figure [7] MS-Diffusion can flexibly manage interactions between reference subjects, even when there overlap among these objects, thereby demonstrating its potential practical applications.
J HUMAN AND ANIME PERSONALIZAATION
Human and anime subjects popular in the of personalized model. We provide results of MS- Difffususio on human and anime subjects in Figure 18 Some research (Wang et al.] 2024a Li
et al. 2024) has explored human personalization in text-to-image diffusion models. By utilizing
19

---

Published as a conference paper at ICLR 2025
Subjects
MS-Diffusion,w.
MS-Diffusion
MS-Diffusion,
MS-Diffusion
holding toy the snow
toy the sofa
dog and a cat the park
a dog on the
with waterfall in the background
standing next to waterfall
Figure 11: Qualitative examples when applying pseudo layout guidance during the inference. In this figure, pse-lg and pse respectively indicate whether layout prior is used.
face encoder (Deng al., 2019) and training the face dataset, MS-Diffusion also be extended to personalized model for these subjects.
K TEXT-IMAGE ATTRIBUTION ANALYSIS
In MS-Diffusion, our focus is primarily on resolving conflicts between subjects without altering the control mechanism of the text. While some approaches (Wang et al. 2023 2024b Zhou et al. 2024] Kim et al.) (023) in non- non-personalized text-to-in rext-to-image tasks address multi- object generation conflicts through text cross- attentinn,, this inevitably requires tuning the diffusion model's parameters, thereby affecting the plug-and-play nature, which is not preferred by us. As demonstrated in the text-image attribution analysis presented in Figure I0. the control of multiple objects by text in our model is also quite evident. This may be related to the explicit layout guidance for subjects, since the images and text condition jointly the generation process. We also attempted to control text cross-attention using the same mechanism in Section 3.5 but no differences were observed in the results.
L SUBJECT INTERPOLATION
The blending of two distinct subject representations yield composite subjects with hybrid charac- teristics is feasible through the navigation of the embedding space linking the subjects. As depicted in Figure 19 linear interpolations are conducted among dog and hat representations, subsequently rendering the interpolated subject in an unaccustomed context. The visualization reveals natural gradation of subject appearance along the interpolated trajectory that harmonizes with the surrounding environment. This technique proves beneficial when applied in subject fusion and style transfer.
M LIMITATIONS
There certain limitations in MS-Diffusion. The box-based indication of positions lacks precision, making challenging to work effectively when the interaction between subjects is stronger. Moreover, the model requires explicit layout input during inference. and generating complex scenes becomes difficult. Though MS- Diffussio beats SOTA personalized diffusion methods both single-subject and multi-subject generation, it still suffers from the influence of background in subject images.
20

---

Published as a conference paper at ICLR 2025
We explore solution for explicit layout needs during inference. MS-Diffusion supports using the text cross-attents cross-attention maps as the pseudo layout guidance. Specifically, as indicated in Figure [0] since the text cross-ate attention maps reflect the area of each text token, them during the inference. In practice, we set threshold to extract masks from text cross-attention maps and apply them after T denoising steps. Before T, experiment with completely disabling layout guidance or using rough box a layout prior. The results are presented in Figure [[] Although disabling layout guidance experiences decline subject consistency, still demonstrates that explicit layout guidance during inference can be optimized. One direction for exploration is to enable the model learn the layout during training.
replace the layout prior with
N SOCIETAL IMPACT
As an image personalization method, MS-Diffusion aims to customize images based on user-provided subjects without fine-tuning. Additionally, the multi-subject reference capability of MS-Diffusion allows users to freely combine and re-create different concepts. However, MS-Diffusion can also be used to generate deceptive images, especially those involving subject combinations that would not exist reality, an issue that remains to be addressed in the future.
21

---

Published as a conference paper at ICLR 2025
Subject
dog police outfit
top wooden floor
dog wearing yellow shirt
Subject
dog in purple wizard outfit
dog on beach
dog wearing rainbow scarf
Subject
sneaker
sneaker on top dirt
sneaker with mountain in background
Subject
teapot the jungle
teapot top sidewalk crowded street
teapot with tree and autumn leaves in the background
Figure 12: Additional qualitative results MS- Diffuusion in single- sugject personalization.
22

---

Published as a conference paper at ICLR 2025
Subjects
IP-Adapter
4-ECLIPSE
SSR-Encoder
MS-Diffusion
stuffed animal the wooden floor
toy on top of the sidewalk crowded street
toy top of dirt road
Figure 13: Qualitative comparison of MS-Diffusion and zero-shot personalized SOTAs
common subjects. Though losing some details, MS-Diffusion outperforms other SOTAs obviously.
23

---

Published as a conference paper at ICLR 2025
Subjects
dog wearing hat in a room
dog wearing coat in the snow
dog. dog. and dog in the jungle
Subjects
lantern and clock room
lantern with mountain the background
lantern, clock and backpack on cobblestone
Subjects
cat wearing shirt on beach
women wearing shirt and trousers a room
woman wearing shirt, jeans and cobblestone street
Figure 14: Additional qualitative results of MS-Diffusion in multi-subject personalization.
24

---

Published as a conference paper at ICLR 2025
Layout
Subjects
Result
Subjects
Result
dog the beach
cat the snow
clock
teapot table
on a bed
cat and cat on the grass
dog and dog in the jungle
Figure 15: Qualitative examples of MS-Diffusion about the layout control ability. Bounding boxes of different colors correspond to subjects with different color borders.
25

---

Published as a conference paper at ICLR 2025
Control
Subjects
Result
Subjects
Result
Depth
dog the beach
cat in room
Canny Edge
dog the grass
a cat in park
Openpose
woman wearing coat and trousers the street
woman wearing shirt jeans on the street
Figure 16:
Generative results when integrating different control conditions. The integrated ControlNets are composed of depth, canny edge, and openpose.
26

---

Published as a conference paper at ICLR 2025
Subjects
Interaction Subjects Generation
dog and cat playing on the beach
dog and cat
lying together
a dog wearing a t-shirt
a dog painted on t-shirt
a woman holding cat the snow
a woman lying on the grass with cat on her
a cat holding a flower
cat with flower on head
a dog on chair in a room
a dog under a chair in room
Figure 17: Examples of prompts with complex interaction multiple subjects. MS-Diffusion generate high-quality images following both the subjects and prompts.
27

---

Published as a conference paper at ICLR 2025
Subjects
Results
man walking on the road
a man standing in the church
an anime man on crowded street
an anime man flying between buildings
an anime man and an anime man fighting together
an anime man sitting besides an anime man
a man talking with an anime man wearing red hat
man and an anime man with a castle in the background
Figure 18: Personalized results of MS-Diffusion on human and anime subjects. MS-Diffusion can generate high-quality images in both single-subject and multi-subject personalization for humans and anime.
28

---

Published as a conference paper at ICLR 2025
Figure 19: Subjects interpolation in multi-subject generation. We select two dogs and two hats to conduct linear interpolation with the text set to "a dog wearing hat on the beach".
29


Reviews:
[{"review_id": "H4fqMqD3WO", "reviewer": "Reviewer_MCZ6", "summary": "The paper introduces the MS-Diffusion framework, a layout-guided zero-shot image personalization approach for multi-subject scenarios. The author features a grounding resampler to enhance subject fidelity with semantic and positional priors and a novel cross-attention mechanism to ensure that each subject is represented in specific areas and facilitating the integration of multi-subject data while mitigating conflicts between text and image subject control.", "questions": "- Could the authors provide a bit more detail on how the M-DINO metric is calculated and how it measures multi-subject fidelity?\n\n- I came across a recent paper [1] on a similar task and recommend testing its evaluation metric on a few cases to see if it could be helpful for assessing multi-subject fidelity.\n\n- The qualitative results presented are impressive, particularly in handling prompts with complex interactions among multiple subjects. Could the authors provide additional examples demonstrating these capabilities, as well as discuss any current limitations?\n\n[1] Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models (https://arxiv.org/pdf/2404.04243)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and well-organized, presenting a clear and compelling motivation for the study.\n- This method is the first to introduce layout-guided zero-shot image personalization for multi-subject scenarios.\n- The paper showcases impressive qualitative results, particularly in layout control capabilities across both single- and multi-subject personalization, as well as in handling prompts with complex interactions among multiple subjects.", "weaknesses": "- In Section 2.2, second paragraph, the sentence \"Though past research in this field has significantly enhanced the ability to reference single subjects, few zero-shot multi-subject personalized models\" is a bit unclear. I suggest rephrasing this sentence for clarity.\n\n- In Section 3.1, as well as in the rest of the paper, it appears that the transpose notation (T) is missing for K in the equations of calculating attention maps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HQvpBOHXqq", "reviewer": "Reviewer_vy6x", "summary": "This paper introduces MS-Diffusion, a zero-shot personalized image generation method for multiple subject entities. Given one or more reference subjects and the text prompt, the model generates high-quality target image based on the inputs accordingly, containing the subject following the desired text descriptions.\nThe main contribution of the paper is the proposal of integrating grounding information for multi-subject personalized generation, and a newly designed grounding resampler for incorporating the given grounding information. \nBy training a SDXL model with the encourage modules on a large-scale private dataset, the algorithm yield favorable results compared to previous designs.", "questions": "1. Comparing to method like Subject Diffusion and GLIGEN, what makes proposed layout control novel?\n    2. In figure 2, the author proposed several limitation cases identified from previous methods, however, they are not further discussed or addressed in the following paragraphs, for example, how is subject overcontrol defined? It usually better to have some specific discussion to recall and address the problems mentioned in the introduction.\n    3. On the multi-subject generation benchmark, the performance boost compared to the baseline SSR-encoder remains minimal, despite the proposed method being trained on a larger and more advanced base model (SDXL versus SD1.5). This outcome appears to serve as a counterexample, challenging the effectiveness of the proposed design.\n    4. The seems to produce a higher CLIP-T scores (better text following) than previous methods, however, this performance boost might also come from the usage of SDXL where multiple text encoder is designed specifically for increase text adherence. Please justify this boost comes from the proposed model design rather than the base model.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. High-quality multi-subject personalized generation.\n 2. The novel design of the grounding resampler provides extraordinary performance boost compared to previous baselines.", "weaknesses": "• Lack of baseline comparisons and limited contributions\n        ◦ The author claimed that they are the first to incorporate grounding information for multiple subject image personalization, however, plenty of existing works have proposed similar designs, for example Subject Diffusion, where the layout is also determined by bounding boxes as cross attention regularizations, while there are no related in-depth discussions.\n        ◦ The comparison with previous design on the benchmark is also limited.\n\n    • Limited performance boost\n        ◦ In Table 2, the performance boost is rather trivial when comparing with SSR-Encoder on Multi-subject setting. Specifically, while SSR-encoder was trained based on SD1.5 and the proposed method is trained based on SDXL, the performance for is almost the same, except for a higher CLIP-T score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CCqJaySjPx", "reviewer": "Reviewer_7QyM", "summary": "The paper proposes a zero-shot method for multi-subject personalization of text-to-image models. This is done through a pre-training phase where an adapter is trained to adapt a diffusion model on additional conditions derived from input images. In particular, the authors use 3.6M videos to extract two views of the same subject. Each training sample comprises segmented entities, their bounding boxes and a ground truth frame including all entities. \n\nThe paper discusses mainly two design choices for their adapter, namely, a Grounded Resampler, and a Multi-Subject Cross Attention Layer. The grounded resampler is used to distill the subject-features through Preceiver like architecture. The authors propose to initialize the queries using entity word-embedding and bounding-box coordinates, to boost the localization of the sampling queries.\n\nIn addition, to avoid any information leakage between subjects, the authors use a Masked Cross attention where queries corresponding to a certain subject only attend the subject features.The proposed method also uses element-wise masking to avoid injected features from the reference images into background patches.", "questions": "[1] L243-253: In the description of the cross-attention layers, in addition to the information injected from the text features, the authors show information that is injected from the reference image as well. In SD, the only features injected are those from the text. While injected image features through the cross-attention layers have been proposed (e.g, IP-Adapter), this wasn’t part of the original model. Please revise this section, and consider explicitly indicating the origin of the image features (f_i). \n\n[2] Grounding Resampler: the authors initialize the queries with word-embeddings and bounding-box Sinusoidal/Fourier features. But in another part of the text, the authors claim these embeddings are learnable. I am confused, is it learnable because of the query project-matrices ? Or do you pass the entity-word-embedding along with Bounding-Box features to another Projection matrix ? Can you please clarify this point.   Please provide a step-by-step explanation of how the queries are initialized and then updated during training, clearly stating which components are learnable and which are fixed.\n\n[3] Do the results in Figure 4 correspond to the MS-Diffusion model before or after fine tuning on DreamBench ? Please clearly label all results in the paper to indicate whether they are from zero-shot or fine-tuned models.\n\n[4] The Multi Subject Cross Attention module has been used (e.g, in ELITE). To my understanding, MSDiffusion only extends this to multi-subject use cases. Can you please highlight the differences ?\n\n[5] It would be nice to provide dataset statistics. In particular, what percentage of the data there are multiple-subjects, excluding trivial cases like a person and clothes. At least from the qualitative samples, it looks like the model performs very well on clothing entities and a single subject. \n\n[6] Can you provide some information regarding the computational complexity of the method. In particular, it would be nice to report the additional overhead indeed by the method. Please report inference time and memory usage compared to baseline zero-shot models, for both single-subject and multi-subject generation tasks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "[1] The paper introduces a zero-shot multi-subject personalization method. \n\n[2] There are interesting components to the proposed method like the data-collection from video and the Grounding Resampler Unit. However I feel like more in-depth analysis and ablation should be carried out to validate that design choices considered are indeed important.", "weaknesses": "[1] The paper is hard to follow and some parts need further clarification. In particular, The “Multi-subject Cross Attention” section is not clear and needs major revision. \n\nLack of ablation on critical parts of the methods: \n\n[2] Using videos for Data collection (e.g, in which ref. Image is different from gt. image) is claimed to be a key contribution. However, there’s a lack of ablation on how much this method helps the authors.\n\n[3] In the Grounding Resampler; how important is the initialization (words+bounding box) to the module localization capability ?\n\n[4] Lacking qualitative samples of interesting use-cases such as - spatial relation (e.g, On-top) and object-interactions (e.g., two subjects playing tennis). The only interesting case I find in the paper is fitted to “virtual try-on” which could be a by-product of the training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L1psw7BBpQ", "reviewer": "Reviewer_Ebqr", "summary": "Text-to-image generation models have advanced but face challenges in multi-subject scenarios. The MS-Diffusion framework is introduced to address these. It uses grounding tokens and feature resampler for detail, and layout guided cross-attention for location control. Experiments show it surpasses existing models in fidelity, promoting personalized text-to-image generation.", "questions": "1.\tThe grounding resampler proposed in the article also compresses image features. Why is it better than the image encoder used in existing methods? Can more qualitative analysis experiments be provided?\n\n2.\tThe experiments in the article are based on Stable Diffusion XL. Now, newly emerging text-to-image diffusion models such as Flux inherently have better control over the generated content. Do these models not need an additionally trained structure to achieve satisfactory control?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper introduced the first layout-guided zero-shot image personalization with multiple subjects framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities.\n\n2.\tThe idea of decoupling and controlling the generation of the texture and position of the subject in the image is very reasonable.\n\n3.\tThe paper is well written and easy to follow. The elaboration of the idea is very clear, and the framework of the structure diagram is also very easy to understand.\n\n4.\tThe experiments are also very sufficient.", "weaknesses": "1.\tThe practice of controlling image generation through local cross attention is not innovative enough. It has been widely adopted in many existing layout-guided text-to-image generation methods.\n\n2.\tJudging from the quantitative and qualitative experimental results provided in the article, the improvement of image generation effect compared to existing methods is relatively limited.\n\n3.\tThe extraction method of image features used by the grounding resampler proposed in the article has also been used in past papers, and its innovation is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cjeNV81KZj", "reviewer": "Reviewer_CBea", "summary": "This paper proposes multiple subjects (MS-Diffusion) framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities, the provision of layout guidance, and the preservation of the foundational model's parameters. \nMS-diffusion explicitly utilizes the layout information of the reference images to extract the information of multiple subjects separately to inject into the base model.", "questions": "How does it perform in scenes where the theme is people and anime? Especially the problem of combining people with anime characters.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The authors alleviate the problem of combining natural objects in two thematic scenarios.\n2.The framework is easy to think of and sensible.\n3.The writing is clear and easy to understand.", "weaknesses": "1.Missing results for multiple topics (>2).\n2.Missing results for comparison with paper [1].\n3.The authors do not mention the Image encoder used.To the best of our knowledge, the Image encoder if it is a CLIP may lose the details of the themes, leading to the results in the last graph of Figure 4 and Figure 5.\n4.From the CLIP-I scores in Table 2, it seems that text fidelity is not significantly improved.\n5.How does it perform in scenes where the theme is people and anime?\n[1]Cones 2: Customizable Image Synthesis with Multiple Subjects", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)