ConceptFlow: Unified Framework for Personalized Image Generation

Authors: Trong-Vu Hoang, Quang-Binh Nguyen, Thanh-Toan Do, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Personalized Image Generation, Visual Guidance

Abstract:
Personalized image generation is an appealing area of research within controllable image generation due to its diverse potential applications. Despite notable advancements, generating images based on single or multiple concepts remains challenging. For single-concept generation, it is difficult to strike a balance between identity preservation and prompt alignment, especially in complex prompts. When it comes to multiple concepts, creating images from a single prompt without extra conditions, such as layout boxes or semantic masks, is problematic due to significantly identity loss and concept omission. In this paper, we introduce ConceptFlow, a comprehensive framework designed to tackle these challenges. Specifically, we propose ConceptFlow-S and ConceptFlow-M for single-concept generation and multiple-concept generation, respectively. ConceptFlow-S introduces a KronA-WED adapter, which integrates a Kronecker adapter with weight and embedding decomposition, and employs a disentangled learning approach with a novel attention regularization objective to enhance single-concept generation. On the other hand, ConceptFlow-M leverages models learned from ConceptFlow-S to directly generate multi-concept images without needed of additional conditions, proposing Subject-Adaptive Matching Attention (SAMA) module and layout consistency guidance strategy. Our extensive experiments and user study show that ConceptFlow effectively addresses the aforementioned issues, enabling its application in various real-world scenarios such as advertising and garment try-on.

Decision: reject
Reviewer scores: 3, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729760848101

thread_type:
official_review

<thread_object>
creator: reviewer_tyid

<content>
summary:
The paper proposes **ConceptFlow**, which includes two components: **ConceptFlow-S** for single-concept generation and **ConceptFlow-M** for multi-concept generation. ConceptFlow-S employs a **KronA-WED adapter** that integrates a Kronecker adapter with weight and embedding decomposition. It also introduces **attention regularization** to improve single-concept generation. ConceptFlow-M extends the models trained by ConceptFlow-S, using **SAMA** (Subject-Adaptive Matching Attention) and **layout consistency guidance** to handle multi-concept generation without requiring additional spatial conditions.

strengths:
1. **Comprehensive personalization framework:** 
  The paper covers both single and multi-concept personalization, supported by comparisons with recent works.

2. **Application insights and metrics:**
   The discussion includes practical applications, qualitative examples, and appropriate metrics for performance analysis.

weaknesses:
1. **Over-complex Framework without Substantial Innovations:**  
    - The **KronA-WED adapter** is like a simple stack of previous strategies, **Mix-of-Show’s *ED-LoRA*** [1] $+$ **Kronecker Adapter *(KronA)*** [6]  $\rightarrow$ **KronA-WED adapter**.
    - The proposed framework looks highly similar as a blend of **Mix-of-Show’s *gradient fusion*** [1] and **DisenBooth’s disentangled learning** [2], which dilutes its novelty. 
    - Furthermore, the **attention regularization (AR)** employed seems highly similar as **Break-A-Scene's attention-based loss** [3], which is also designed to encourages that each handle attends only to the image region occupied by the corresponding concept. 
    - The **SAMA module** for multi-concept generation largely reuses existing strategies, closely mirroring **DreamMatcher’s AMA** [4] (with **Eq. 6-9** resembling **Eq. 3, 4, 5, and 7** from DreamMatcher) but with minor modifications, such as adding a **foreground mask**. 
    - The design of **Layout Guidance** follows the core idea of **A-Star's Attention Segregation Loss** [5], also not a novel technique.

    This accumulation of existing techniques makes the work feel like an industrial assembly, which reduces the perceived novelty of the contribution.

2. **Identity Preservation Issues and Low-Level Aesthetics:**  
   - Several generated outputs display **identity dissimilarity** or **unsatisfactory aesthetics**, such as Justin Bieber, Emma Stone, cat in **Fig. 1**, which raise my concerns about the soundness of the proposed method and the robustness of the framework when dealing with **diverse or intricate concepts**.

3. **Dataset Limitations and Generalizability:**  
   - The selection of subjects  in the manuscript are limited (**24 subjects (Fig. 12)**), which is limited and potentially biased toward **cherry-pick cases**. I believe these limited subjects can not represent the full spectrum of real-world complexities, and a broader test set covering **more diverse concepts and edge cases** would provide stronger evidence of the framework's generalizability.

**Reference**

[1] *Gu, et al. "Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models." NeurIPS (2024).*

[2] *Chen, et al. "Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation." arXiv (2023).*

[3] *Avrahami, et al. "Break-a-scene: Extracting multiple concepts from a single image." SIGGRAPH 2023.*

[4] *Nam, et al. "Dreammatcher: Appearance matching self-attention for semantically-consistent text-to-image personalization." CVPR (2024).*

[5] *Agarwal, et al. "A-star: Test-time attention segregation and retention for text-to-image synthesis." ICCV (2023).*

[6] *Edalati, Ali, et al. "Krona: Parameter efficient tuning with kronecker adapter." arXiv (2022).*

questions:
1. **Enhancing Identity Matching in Evaluation:**  
   - The current evaluation focuses on **DINO scores** and **CLIP-T metrics**, but these might not capture **fine-grained identity preservation**, especially for faces. Have you considered incorporating face recognition tools like **FaceNet** or **ArcFace** [7]? These tools could offer bounding-box-level face similarity evaluation, providing a more precise measurement of face identity consistency.


2. **Expanding the Dataset and Subject Variety:**  
   - As stated in  **Weakness 3**, The number of subjects used for both quantitative and qualitative evaluations appears limited (**Fig. 12**). This narrow selection may affect the statistical reliability of the results. Could you **experiment on more subjects or scenarios** to validate the performance of ConceptFlow across a broader range of real-world cases?

3. **Innovative Contributions of SAMA Compared to DreamMatcher:**  
   - The *SAMA module* seems closely related to DreamMatcher’s *Appearance Matching Self-Attention (AMA)*, with the primary distinction being the use of a *concept foreground mask ($\mathbf{M}_k$)*. Could you further conduct **ablation studies on the AMA vs. SAMA** to show how it extends beyond DreamMatcher's approach?


4. **Time Efficiency and Computational Overhead:**  
   - ConceptFlow involves multiple steps, such as *gradient fusion, SAMA matching, and layout consistency guidance*, which could introduce computational overhead. Could you provide a **memory / time efficiency comparison** between ConceptFlow and other state-of-the-art methods, such as **LoKr [8], DisenBooth [2], and Mix-of-Show [1]**? This would clarify the practical trade-offs between performance and efficiency for both single and multi-concept generation.


**Reference**

[7] *https://github.com/deepinsight/insightface*.

[8] *Yeh, et al. "Navigating text-to-image customization: From lycoris fine-tuning to model evaluation." ICLR (2023)*.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650109691

thread_type:
official_review

<thread_object>
creator: reviewer_wm2m

<content>
summary:
This paper introduces ConceptFlow to balance identity preservation with prompt alignment, alleviating identity loss and concept omission.   The proposed KronA-WED adapter combined with a novel attention regularization objective balances reconstruction and editability
for single-concept generation. Additionally, the subject-adaptive matching attention module and layout consistency guidance strategy help prevent concept omission in multi-concept generation. Experimental results indicate that the proposed method has a positive impact on these issues to some extent.

strengths:
1. This paper collects a small dataset.
2. The proposed method achieves good identity preservation.

weaknesses:
1. Only a limited number of comparison methods, especially in multi-concept generation, does not adequately support the authors' claim. Why does this paper compare against so few published works? Let's leave aside arXiv papers, there are many accepted, peer-reviewed papers should be considered.

[1] FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention, IJCV 

[2] FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition, CVPR 2024 

[3] Key-Locked Rank One Editing for Text-to-Image Personalization, SIGGRAPH 2023

[4] Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models, CVPR2024

[5] OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models, ECCV2024

[6] KOSMOS-G: Generating Images in Context with Multimodal Large Language Models, ICLR2024

2. Even in comparison with the methods listed in table, the proposed trivial method does not show significant improvement. This is particularly evident in CLIP-T, which evaluates editability and does not support the authors' claims of balancing the trade-off between reconstruction and editability. Can this paper provide metrics that evaluate this trade-off directly, rather than reporting separately when one aspect is better and the other is worse.

questions:
1. It is good to see the new collected dataset. But for a comprehensive comprasion. Can the authors provide multi-concept generation results using CustomConcept101?
2. Most of papers report both CLIP-I and DINO. I believe use both metrics can better evaluate the identity preservation in different aspects.
3. I believe that extra conditions like boxes and masks can make generation more controllable. Of course, there are some situtations that condition-free generation is better. It would be appreciated to see the proposed can achieve better performance on both setting.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654009697

thread_type:
official_review

<thread_object>
creator: reviewer_jvbs

<content>
summary:
The manuscript introduces propose two strategies "ConceptFlow-S" and "ConceptFlow-M" for single-concept generation and multiple-concept generation to resolve the limitation of identity preservation, prompt alignment, and concept omission from previous methods. For single concept, the authors introduce KronA-WED adapter with the attention regularization objective. For multiple concept, the authors propose the SAMA module and layout the consistency guidance. This method facilitates rapid and efficient single and multi-subject customization. The results demonstrate that the method is competitive with leading frameworks in various image generation tasks.

strengths:
1. The work identifies the issues in both single and multiple concept personalization such as trade off between reconstruction and edibility, and identity loss and the issue of concept missing.

2. The authors have developed a framework for generating personalized images that effectively integrates strategies to solve both single (ConceptFlow-S) and multiple concept (ConceptFlow-M). 

2. The paper provides experimental results, including both quantitative and qualitative assessments, showcasing the superior performance of the framework. The results clearly highlight the effectiveness of the proposed method in facilitating personalized image generation.

weaknesses:
1. In Figure 5, the method utilizes SAMA module to enhance the identity details. However, it is unclear how it would perform with fine-grained subjects (two dogs or two cats with different breed). Also, I wonder how this module would work when the concept size increases. Clarification is needed on whether the module can effectively manage such fine distinctions and multiple diverse subjects.

2. Recent methodologies [1, 2] have demonstrated the capability to learn multi-concept personalization, it remains uncertain if the proposed work can handle multiple personalized instances (> 2), particularly for contexts involving up to five subjects. Although quantitative results for two subjects are provided in both paper and appendix, the absence of qualitative results for three or more subjects in both the main text and appendix might be a notable omission. Including these results would substantiate the method's capability in more complex scenarios.


[1] Liu, Zhiheng, et al. "Cones 2: Customizable image synthesis with multiple subjects." arXiv preprint arXiv:2305.19327 (2023).

[2] Yeh, Chun-Hsiao, et al. "Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition." arXiv preprint arXiv:2402.15504 (2024).

questions:
Given the concerns mentioned, particularly around the method's scalability to more complex multi-subject personalizations and the clarification behind module choices, I recommend a "marginally below the acceptance threshold" for this paper. Enhancements in demonstrating multi-subject capabilities, clarity in embedding visualization, and justification for the choice of technology could potentially elevate the manuscript to meet publication standards.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716085108

thread_type:
official_review

<thread_object>
creator: reviewer_2sbt

<content>
summary:
This paper introduces the ConceptFlow framework, including the ConceptFlow-S component for robust single-concept learning and generation, and the ConceptFlow-M component to create images of multiple concepts without the need for spatial guidance.

strengths:
ConceptFlow-M needs no spatial guidance and can somehow handle the missing concept issue, which is important in multi-concept personalized image generation.

weaknesses:
1. This work claims to be a unified framework of personalized image generation in the title, but there is no presentation about how it **unifies** personalized image generation.
2. The contribution of the proposed method is weak. There are too many modules that already exist with no relation to each other.
3. The comparison experiments are not rational. Since the proposed model introduces an image adapter, it should compare with the methods using the image adapter. In fact, as a fine-tune-based method, there is no need to use an image adapter since it needs an image input during the inference. I suggest the authors compare the proposed method with some zero-shot methods like IP-Adapter [1], SSR-Encoder [2], and MS-Diffusion [3], which can also solve the same issues.
4. The experiments are completely not enough with few baselines and qualitative examples.

[1] Ye, Hu, et al. "Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models." arXiv preprint arXiv:2308.06721 (2023). 

[2] Zhang, Yuxuan, et al. "Ssr-encoder: Encoding selective subject representation for subject-driven generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. 

[3] Wang, X., et al. "MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance." arXiv preprint arXiv:2406.07209 (2024).

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732294648869

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response (1/2)

comment:
We thank the reviewers for their constructive comments and suggestions. We are encouraged with positive feedback that ConceptFlow **effectively solves both single and multiple concept generation problems** (jVBs, 2SBt, TYid), has **good identity preservation capability** (wm2m), and has **potential applications** (TYid).

In our [**rebuttal manuscript**](https://anonymous.4open.science/r/ConceptFlow_ICLR25_Rebuttal-2411) (clickable), we have added additional experiments, ablation studies, analyses, and discussions based on the suggestions from all reviewers:
1. Additional experiments
2. ConceptFlow-S versus zero-shot methods in single concept generation
3. Attention Regularization (AR) in ConceptFlow-S versus Break-A-Scene's strategy
4. Subject Adaptive Matching Attention (SAMA) in ConceptFlow-M versus DreamMatcher's AMA
5. Limitation of ConceptFlow-M:
    - Generations of multiple semantically similar concepts
    - Generations of more than two concepts

Also, we would like to highlight the **contributions** of ConceptFlow, as well as how it **unifies personalized image generation**.

## A comprehensive framework for personalized image generation 

Firstly, ConceptFlow-S finetunes Stable Diffusion for **robust single concept learning and generation**. We can then combine the individual weights learned by ConceptFlow-S, enabling the generation of multi-concept images through the ConceptFlow-M **without requiring of extra conditions** (e.g. boxes, masks).

It is noteworthy that **using weights learned by ConceptFlow-S** significantly enhances the performance of ConceptFlow-M in generating multiple concepts compared to other single-concept learning methods. Additionally, each concept only needs to be finetuned once, with the trained weights saved for future multi-concept generation. Therefore, **ConceptFlow unifies personalized image generation**, seamlessly eveloving from single to multi-concept generation.

With superiority in generating interactions between **human and object**, as well as between **human and animal**, ConceptFlow showcases its **potential applications** in areas such as advertising, storytelling, and even garment synthesis.
</content>
</thread_object>

</thread>


Metareview:
metareview:
- **Strengths**:
  - Comprehensive framework covering both single and multi-concept personalization.
  - Demonstrated improvements in identity preservation and editability with robust quantitative metrics.

- **Weaknesses**:
  - Limited novelty in methodology.
  - Scalability concerns and dataset constraints impact generalizability.
  - Incremental improvements over prior art, with some claims inadequately substantiated.

Despite addressing most issues during the rebuttal, the concerns about scalability and generalizability weighed heavily in the final decision. The work was evaluated as promising but needing further development for inclusion in a top-tier venue.