MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance

Authors: Xierui Wang, Siming Fu, Qihan Huang, Wanggui He, Hao Jiang
Primary area: generative models
Keywords: Image Personalization, Multiple Subjects, Diffusion Models

Abstract:
Recent advancements in text-to-image generation models have dramatically enhanced the generation of photorealistic images from textual prompts, leading to an increased interest in personalized text-to-image applications, particularly in multi-subject scenarios. However, these advances are hindered by two main challenges: firstly, the need to accurately maintain the details of each referenced subject in accordance with the textual descriptions; and secondly, the difficulty in achieving a cohesive representation of multiple subjects in a single image without introducing inconsistencies. To address these concerns, our research introduces the MS-Diffusion framework for layout-guided zero-shot image personalization with multi-subjects. This innovative approach integrates grounding tokens with the feature resampler to maintain detail fidelity among subjects. With the layout guidance, MS-Diffusion further improves the cross-attention to adapt to the multi-subject inputs, ensuring that each subject condition acts on specific areas. The proposed multi-subject cross-attention orchestrates harmonious inter-subject compositions while preserving the control of texts. Comprehensive quantitative and qualitative experiments affirm that this method surpasses existing models in both image and text fidelity, promoting the development of personalized text-to-image generation.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730100655350

thread_type:
official_review

<thread_object>
creator: reviewer_cbea

<content>
summary:
This paper proposes multiple subjects (MS-Diffusion) framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities, the provision of layout guidance, and the preservation of the foundational model's parameters. 
MS-diffusion explicitly utilizes the layout information of the reference images to extract the information of multiple subjects separately to inject into the base model.

strengths:
1.The authors alleviate the problem of combining natural objects in two thematic scenarios.
2.The framework is easy to think of and sensible.
3.The writing is clear and easy to understand.

weaknesses:
1.Missing results for multiple topics (>2).
2.Missing results for comparison with paper [1].
3.The authors do not mention the Image encoder used.To the best of our knowledge, the Image encoder if it is a CLIP may lose the details of the themes, leading to the results in the last graph of Figure 4 and Figure 5.
4.From the CLIP-I scores in Table 2, it seems that text fidelity is not significantly improved.
5.How does it perform in scenes where the theme is people and anime?
[1]Cones 2: Customizable Image Synthesis with Multiple Subjects

questions:
How does it perform in scenes where the theme is people and anime? Especially the problem of combining people with anime characters.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638202942

thread_type:
official_review

<thread_object>
creator: reviewer_ebqr

<content>
summary:
Text-to-image generation models have advanced but face challenges in multi-subject scenarios. The MS-Diffusion framework is introduced to address these. It uses grounding tokens and feature resampler for detail, and layout guided cross-attention for location control. Experiments show it surpasses existing models in fidelity, promoting personalized text-to-image generation.

strengths:
1.	The paper introduced the first layout-guided zero-shot image personalization with multiple subjects framework, which consolidates the accommodation of multiple subjects, the incorporation of zero-shot learning capabilities.

2.	The idea of decoupling and controlling the generation of the texture and position of the subject in the image is very reasonable.

3.	The paper is well written and easy to follow. The elaboration of the idea is very clear, and the framework of the structure diagram is also very easy to understand.

4.	The experiments are also very sufficient.

weaknesses:
1.	The practice of controlling image generation through local cross attention is not innovative enough. It has been widely adopted in many existing layout-guided text-to-image generation methods.

2.	Judging from the quantitative and qualitative experimental results provided in the article, the improvement of image generation effect compared to existing methods is relatively limited.

3.	The extraction method of image features used by the grounding resampler proposed in the article has also been used in past papers, and its innovation is limited.

questions:
1.	The grounding resampler proposed in the article also compresses image features. Why is it better than the image encoder used in existing methods? Can more qualitative analysis experiments be provided?

2.	The experiments in the article are based on Stable Diffusion XL. Now, newly emerging text-to-image diffusion models such as Flux inherently have better control over the generated content. Do these models not need an additionally trained structure to achieve satisfactory control?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676059362

thread_type:
official_review

<thread_object>
creator: reviewer_7qym

<content>
summary:
The paper proposes a zero-shot method for multi-subject personalization of text-to-image models. This is done through a pre-training phase where an adapter is trained to adapt a diffusion model on additional conditions derived from input images. In particular, the authors use 3.6M videos to extract two views of the same subject. Each training sample comprises segmented entities, their bounding boxes and a ground truth frame including all entities. 

The paper discusses mainly two design choices for their adapter, namely, a Grounded Resampler, and a Multi-Subject Cross Attention Layer. The grounded resampler is used to distill the subject-features through Preceiver like architecture. The authors propose to initialize the queries using entity word-embedding and bounding-box coordinates, to boost the localization of the sampling queries.

In addition, to avoid any information leakage between subjects, the authors use a Masked Cross attention where queries corresponding to a certain subject only attend the subject features.The proposed method also uses element-wise masking to avoid injected features from the reference images into background patches.

strengths:
[1] The paper introduces a zero-shot multi-subject personalization method. 

[2] There are interesting components to the proposed method like the data-collection from video and the Grounding Resampler Unit. However I feel like more in-depth analysis and ablation should be carried out to validate that design choices considered are indeed important.

weaknesses:
[1] The paper is hard to follow and some parts need further clarification. In particular, The “Multi-subject Cross Attention” section is not clear and needs major revision. 

Lack of ablation on critical parts of the methods: 

[2] Using videos for Data collection (e.g, in which ref. Image is different from gt. image) is claimed to be a key contribution. However, there’s a lack of ablation on how much this method helps the authors.

[3] In the Grounding Resampler; how important is the initialization (words+bounding box) to the module localization capability ?

[4] Lacking qualitative samples of interesting use-cases such as - spatial relation (e.g, On-top) and object-interactions (e.g., two subjects playing tennis). The only interesting case I find in the paper is fitted to “virtual try-on” which could be a by-product of the training data.

questions:
[1] L243-253: In the description of the cross-attention layers, in addition to the information injected from the text features, the authors show information that is injected from the reference image as well. In SD, the only features injected are those from the text. While injected image features through the cross-attention layers have been proposed (e.g, IP-Adapter), this wasn’t part of the original model. Please revise this section, and consider explicitly indicating the origin of the image features (f_i). 

[2] Grounding Resampler: the authors initialize the queries with word-embeddings and bounding-box Sinusoidal/Fourier features. But in another part of the text, the authors claim these embeddings are learnable. I am confused, is it learnable because of the query project-matrices ? Or do you pass the entity-word-embedding along with Bounding-Box features to another Projection matrix ? Can you please clarify this point.   Please provide a step-by-step explanation of how the queries are initialized and then updated during training, clearly stating which components are learnable and which are fixed.

[3] Do the results in Figure 4 correspond to the MS-Diffusion model before or after fine tuning on DreamBench ? Please clearly label all results in the paper to indicate whether they are from zero-shot or fine-tuned models.

[4] The Multi Subject Cross Attention module has been used (e.g, in ELITE). To my understanding, MSDiffusion only extends this to multi-subject use cases. Can you please highlight the differences ?

[5] It would be nice to provide dataset statistics. In particular, what percentage of the data there are multiple-subjects, excluding trivial cases like a person and clothes. At least from the qualitative samples, it looks like the model performs very well on clothing entities and a single subject. 

[6] Can you provide some information regarding the computational complexity of the method. In particular, it would be nice to report the additional overhead indeed by the method. Please report inference time and memory usage compared to baseline zero-shot models, for both single-subject and multi-subject generation tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723909625

thread_type:
official_review

<thread_object>
creator: reviewer_vy6x

<content>
summary:
This paper introduces MS-Diffusion, a zero-shot personalized image generation method for multiple subject entities. Given one or more reference subjects and the text prompt, the model generates high-quality target image based on the inputs accordingly, containing the subject following the desired text descriptions.
The main contribution of the paper is the proposal of integrating grounding information for multi-subject personalized generation, and a newly designed grounding resampler for incorporating the given grounding information. 
By training a SDXL model with the encourage modules on a large-scale private dataset, the algorithm yield favorable results compared to previous designs.

strengths:
1. High-quality multi-subject personalized generation.
 2. The novel design of the grounding resampler provides extraordinary performance boost compared to previous baselines.

weaknesses:
• Lack of baseline comparisons and limited contributions
        ◦ The author claimed that they are the first to incorporate grounding information for multiple subject image personalization, however, plenty of existing works have proposed similar designs, for example Subject Diffusion, where the layout is also determined by bounding boxes as cross attention regularizations, while there are no related in-depth discussions.
        ◦ The comparison with previous design on the benchmark is also limited.

    • Limited performance boost
        ◦ In Table 2, the performance boost is rather trivial when comparing with SSR-Encoder on Multi-subject setting. Specifically, while SSR-encoder was trained based on SD1.5 and the proposed method is trained based on SDXL, the performance for is almost the same, except for a higher CLIP-T score.

questions:
1. Comparing to method like Subject Diffusion and GLIGEN, what makes proposed layout control novel?
    2. In figure 2, the author proposed several limitation cases identified from previous methods, however, they are not further discussed or addressed in the following paragraphs, for example, how is subject overcontrol defined? It usually better to have some specific discussion to recall and address the problems mentioned in the introduction.
    3. On the multi-subject generation benchmark, the performance boost compared to the baseline SSR-encoder remains minimal, despite the proposed method being trained on a larger and more advanced base model (SDXL versus SD1.5). This outcome appears to serve as a counterexample, challenging the effectiveness of the proposed design.
    4. The seems to produce a higher CLIP-T scores (better text following) than previous methods, however, this performance boost might also come from the usage of SDXL where multiple text encoder is designed specifically for increase text adherence. Please justify this boost comes from the proposed model design rather than the base model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730956675903

thread_type:
official_review

<thread_object>
creator: reviewer_mcz6

<content>
summary:
The paper introduces the MS-Diffusion framework, a layout-guided zero-shot image personalization approach for multi-subject scenarios. The author features a grounding resampler to enhance subject fidelity with semantic and positional priors and a novel cross-attention mechanism to ensure that each subject is represented in specific areas and facilitating the integration of multi-subject data while mitigating conflicts between text and image subject control.

strengths:
- The paper is well-written and well-organized, presenting a clear and compelling motivation for the study.
- This method is the first to introduce layout-guided zero-shot image personalization for multi-subject scenarios.
- The paper showcases impressive qualitative results, particularly in layout control capabilities across both single- and multi-subject personalization, as well as in handling prompts with complex interactions among multiple subjects.

weaknesses:
- In Section 2.2, second paragraph, the sentence "Though past research in this field has significantly enhanced the ability to reference single subjects, few zero-shot multi-subject personalized models" is a bit unclear. I suggest rephrasing this sentence for clarity.

- In Section 3.1, as well as in the rest of the paper, it appears that the transpose notation (T) is missing for K in the equations of calculating attention maps.

questions:
- Could the authors provide a bit more detail on how the M-DINO metric is calculated and how it measures multi-subject fidelity?

- I came across a recent paper [1] on a similar task and recommend testing its evaluation metric on a few cases to see if it could be helpful for assessing multi-subject fidelity.

- The qualitative results presented are impressive, particularly in handling prompts with complex interactions among multiple subjects. Could the authors provide additional examples demonstrating these capabilities, as well as discuss any current limitations?

[1] Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models (https://arxiv.org/pdf/2404.04243)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732115128533

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank the reviewers for their thoughtful reviews. We are encouraged with positive feedback on the **novel method** (MCZ6, vy6x, Ebqr), **reasonable and interesting module design** (7QyM, Ebqr), **impressive and sufficient experiment results** (MCZ6, vy6x, Ebqr), **effective problem solution** (CBea), and **well-structured manuscript and presentation** (MCZ6, Ebqr, CBea).

In the revised manuscript, we update some clarifications according to the suggestions from all reviewers. To better demonstrate the ability of our model, we add more qualitative examples in the appendix:

- Figure 8 (updated): Qualitative comparison of Mix-of-Show, Cones2, and MS-Diffusion
- Figure 17 (updated): Examples of prompts with complex interaction of multiple subjects
- Figure 18 (new): Personalized results of MS-Diffusion on human and anime subjects

Moreover, there are some points we want to highlight:

- **The novelty of proposed modules.** The key modules of MS-Diffusion are Grounding Resampler and Multi-subject Cross-attention, which are both different from the existing research.
  - Grounding Resampler: Most of the layout-to-image methods have explored in injecting grounding information into the generation process of U-Net. However, to the best of our knowledge, MS-Diffusion is the **first** approach utilizing the grounding information in the image features projection. **As clarified in Sec 3.4, the purpose of this design is to indicate the resampler with semantic and positional prior, thus increasing the detail preserving ability.**
  - Multi-subject Cross-attention: Some studies focus on mitigating conflicts in text cross-attentions. However, as we discussed in Sec 3.2, there are some limitations for the text cross-attention editing, resulting in low performance and increased uncertainty. In comparison, to the best of our knowledge, MS-Diffusion is the **first** personalized text-to-image method employing **image attention masks**. **While addressing multi-object conflicts, MS-Diffusion ensures that text conditions remain unaffected, as evidenced by the significantly higher text adherence capability.**
- **The improvement of image and text fidelity.** These metrics need to be balanced in the applications. As the model tends to memorize the given subject, most of the SOTAs struggle in controlling the generation using text prompts, resulting in low text fidelity. **MS-Diffusion achieves a substantial improvement in text fidelity without compromising image fidelity and even enhances its ability to preserve image details.**

- **We are advocates of open research and strive to make everything publicly accessible. We are endeavoring to make the model available to the public before paper publication.**

Detailed responses to each reviewer are provided below. All revisions of the manuscript are marked as blue. Please let us know if any further clarification or discussion is required.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732719681795

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kindly reminder for the deadline of manuscript revision

comment:
Dear Reviewers,

As the deadline for manuscript revisions is less than a day away, we wanted to kindly follow up to confirm if there are any additional points or updates you would like us to address based on your reviews. 

We have already made revisions to the manuscript according to your feedback, and we greatly value your input to ensure the final version meets your expectations. Please feel free to let us know if there are any remaining suggestions.

Thank you once again for your time and thoughtful comments.

Best regards,  
MS-Diffusion Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces MS-Diffusion, a new approach for zero-shot image personalization that incorporates multiple subjects and uses layout guidance. The proposed grounding resampler and multi-subject cross-attention mechanism aim to enhance image generation tasks by focusing on particular areas and subjects within an image. The reported results suggest that the method can achieve qualitative and quantitative improvements over existing techniques.

The paper's main strengths lie in its novel contributions to the field of zero-shot image personalization. The introduction of the grounding resampler and cross-attention mechanism is innovative and has the potential to push forward the capabilities of image generation models. Additionally, the paper is well-structured and effectively communicates the proposed methodologies and their potential impact.

Reviewers have pointed out concerns regarding the level of innovation in the use of local cross-attention for controlling image generation. Questions about the comparative performance of the grounding resampler to existing methods were raised, as well as the need for more comprehensive quantitative evaluations. There were also criticisms about the loss of detail when using the CLIP image encoder.

The paper provides valuable contributions to zero-shot image personalization and has effectively addressed the main concerns raised during the review process. The remaining points of criticism are not critical enough to override the decision to accept, given the overall strength of the paper and the positive changes made in response to the review feedback.