DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation

Authors: Jing He, Haodong Li, huyongzhe, Guibao Shen, Yingjie CAI, Weichao Qiu, Ying-Cong Chen
Primary area: generative models
Keywords: Visual Disentanglement and Enrichment, Zero-shot Customization, Text-to-Image Generation

Abstract:
In the realm of image generation, creating customized images from visual prompt with additional textual instruction emerges as a promising endeavor. However, existing methods, both tuning-based and tuning-free, struggle with interpreting the subject-essential attributes from the visual prompt. This leads to subject-irrelevant attributes infiltrating the generation process, ultimately compromising the personalization quality in both editability and ID preservation. In this paper, we present $\textbf{DisEnvisioner}$, a novel approach for effectively extracting and enriching the subject-essential features while filtering out -irrelevant information, enabling exceptional customization performance, in a $\textbf{tuning-free}$ manner and using only $\textbf{a single image}$. Specifically, the feature of the subject and other irrelevant components are effectively separated into distinctive visual tokens, enabling a much more accurate customization. Aiming to further improving the ID consistency, we enrich the disentangled features, sculpting them into a more granular representation. Experiments demonstrate the superiority of our approach over existing methods in instruction response (editability), ID consistency, inference speed, and the overall image quality, highlighting the effectiveness and efficiency of DisEnvisioner.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730235969775

thread_type:
official_review

<thread_object>
creator: reviewer_6rcz

<content>
summary:
This paper introduces DisEnvisioner, a novel tuning-free method for generating customized images from a single visual prompt enriched with additional textual instructions. Existing image generation methods, both tuning-based and tuning-free, often struggle to isolate the essential attributes of a subject in the visual prompt, leading to unwanted, subject-irrelevant features that compromise customization quality, editability, and identity (ID) preservation. DisEnvisioner addresses this by disentangling the subject-essential features from irrelevant details, separating them into distinct visual tokens. This separation improves customization precision and allows for enhanced ID consistency. By further refining these disentangled features, DisEnvisioner creates a more granular representation of the subject, which bolsters the model's ability to maintain ID consistency across generations. Experimental results demonstrate that DisEnvisioner outperforms current methods in editability, ID consistency, inference speed, and overall image quality, establishing its effectiveness and efficiency for personalized image generation.

strengths:
DisEnvisioner demonstrates several strengths over competing methods in the areas of customization quality, editability, identity consistency, and efficiency, as shown in the provided results:

Customization Quality (C-T): With a score of 0.315, DisEnvisioner shows the highest performance in customization (C-T), indicating that it excels at incorporating subject-specific details while staying true to the given instructions.

ID Consistency (D-I): DisEnvisioner scores 0.802 for ID consistency (D-I), surpassing most other methods except for IP-Adapter. This score reflects its ability to maintain subject identity throughout image generation, reducing unwanted attribute drift.

Instruction Response (C-I): DisEnvisioner scores 0.828, slightly lower than some methods like IP-Adapter and DreamBooth but still within a strong range. This indicates good responsiveness to textual instructions while retaining visual prompt characteristics.

Inference Speed (IV): DisEnvisioner achieves a lower inference value of 0.026, indicating faster inference compared to methods like DreamBooth and DisenBooth, making it efficient for real-time or rapid customization needs.

*Runtime (T)**: DisEnvisioner has a runtime of 1.96 seconds, placing it on par with IP-Adapter, which is one of the fastest. This efficient runtime makes it more practical for applications needing quick processing.

Mean Rank (mRank): With an mRank of 2.0, DisEnvisioner achieves the highest overall ranking among the methods tested, suggesting that it consistently performs well across different evaluation metrics.

weaknesses:
DisEnvisioner, while showcasing significant advancements, has some limitations that temper its overall impact:

Lack of State-of-the-Art Results Across All Metrics: DisEnvisioner does not outperform all baseline models in every metric. For instance, its performance in instruction response (C-I) and identity consistency (D-I) does not reach the top scores achieved by IP-Adapter and DreamBooth. This mixed performance limits DisEnvisioner’s claim to outright superiority across all customization aspects.

Limited Test Dataset: The dataset used for evaluating DisEnvisioner is relatively constrained, potentially affecting the generalizability of the results. A more extensive and varied dataset would provide a clearer picture of the model's adaptability and robustness across diverse tasks and use cases.

Room for Improved Analysis on Underperformance Against IP-Adapter: Although DisEnvisioner demonstrates strengths in disentangling features, it would benefit from further analysis on why it lags behind IP-Adapter in specific tasks like instruction accuracy and ID consistency. Understanding these discrepancies could inform targeted improvements to make DisEnvisioner more competitive in these areas.

These aspects suggest that while DisEnvisioner makes notable contributions, there is room for further development to enhance its consistency and broaden its applicability across a wider range of tasks.

questions:
Question: Could the authors elaborate on the specific factors that led DisEnvisioner to underperform IP-Adapter in terms of instruction accuracy (C-I) and ID consistency (D-I)?

Suggestion: Analyzing and explaining these performance gaps would provide useful insights into DisEnvisioner's design choices. This could also help clarify if the approach is inherently less suited to certain customization aspects or if there are areas where further tuning could close the performance gap.

Question: Given the limited dataset, how confident are the authors in DisEnvisioner’s generalizability to a broader range of tasks and more diverse visual prompts?

Suggestion: Evaluating DisEnvisioner on additional datasets with varied visual content and textual instructions could better assess its robustness. Additional results on a larger dataset could substantiate the claims of effectiveness and versatility.

Question: How does the feature enrichment step improve ID consistency, and could the authors provide more details on its implementation?

Suggestion: Adding a more in-depth explanation of the enrichment process and its effect on ID consistency would be beneficial. Additionally, showing comparisons before and after this step could better demonstrate its impact on maintaining subject integrity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730250688880

thread_type:
official_review

<thread_object>
creator: reviewer_cz86

<content>
summary:
The paper, titled "DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation," presents a novel approach to generating customized images from visual prompts with additional textual descriptions. The main contributions are:

1. Proposing DisEnvisioner, a tuning-free framework that effectively disentangles subject-essential attributes from irrelevant features, improving the overall quality of customized image generation.

2. Introducing a two-stage architecture with DisVisioner to separate subject-essential and irrelevant features, and EnVisioner to enhance subject consistency.

3. Demonstrating through experiments that the proposed approach outperforms existing methods in terms of editability, identity (ID) consistency, and inference speed while requiring minimal fine-tuning or reference images.

strengths:
Originality: The introduction of DisVisioner and EnVisioner for disentangled visual prompt processing is a novel approach, effectively addressing the challenge of maintaining subject identity while generating customized images.

Quality: The experiments are well-conducted, demonstrating the effectiveness of the proposed method in preserving ID consistency and reducing subject-irrelevant features.

Significance: The approach enables high-quality, customized image generation in a tuning-free manner, which is practical and efficient for real-world applications.

Clarity: The overall structure of the paper is well-organized, and the experimental results effectively showcase the advantages of the proposed method over existing baselines.

weaknesses:
Limited Theoretical Justification: The paper lacks sufficient theoretical grounding for the proposed disentangling mechanism. For example, the method for separating subject-essential and irrelevant features relies primarily on empirical observations without rigorous theoretical backing.

Comparison with Existing Methods: The paper does not provide extensive comparisons with some of the most recent advancements in text-to-image generation (e.g., Transformer-based approaches). Including a wider range of baselines would help position this work within the current state of the field.

questions:
Could the authors provide more theoretical insights into the disentanglement mechanism and why it works well for customized image generation? The current explanation is mostly empirical.

How does DisEnvisioner compare to other recent Transformer-based text-to-image generation methods? Adding these comparisons could help readers better appreciate the novelty of the approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730610904485

thread_type:
official_review

<thread_object>
creator: reviewer_p8sz

<content>
summary:
The paper introduces DisEnvisioner, a novel approach aimed at enhancing image customization from visual prompts while addressing the limitations of existing methods in interpreting subject-essential attributes.Empirical evidence demonstrating the superiority of DisEnvisioner over existing methods in various aspects, including instruction response (editability), ID consistency, inference speed, and overall image quality.

strengths:
The paper introduces DisEnvisioner, a novel framework that addresses significant challenges in customized image generation by focusing on feature disentanglement and enrichment. The originality lies in the identification of the crucial role that subject-essential attributes play in the customization process, which is a new viewpoint that goes beyond existing methods reliant on either tuning or tuning-free approaches.

The clarity of the paper is commendable. The authors present their arguments logically, making it easy for readers to follow their reasoning and understand the significance of their contributions. Key terminologies and concepts like "subject-essential attributes" and "feature disentanglement" are well-defined

weaknesses:
1.The advancement of the methodology has not been sufficiently demonstrated; some indicators (e.g. image-alignment for ID-consistency (C-I, D-I) as shown in Table 1) are inferior to other similar methods (e.g., IP-Adapter, BLIP-Diffusion, etc.),you cloud provide more evidence to support your claim in abstract "Experiments demonstrate the superiority of our approach over existing methods in instruction response (editability), ID consistency,"

2.The EFFECT OF DISVISIONER has not been adequately explained; there is a lack of comparative experiments with and without DISVISIONER to validate its effectiveness. You cloud compare results with and without the DiVisioner component while keeping other parts of the system constant.

3.The EFFECT OF ENVISIONER has not been sufficiently substantiated; the paper's explanation regarding the EFFECT OF ENVISIONER only presents a few cases (as shown in Fig. 8), which lacks persuasiveness.You could provide a larger-scale comparison by human evaluation or image-alignment for ID-consistency (C-I, D-I) to measure the improvement in ID consistency or image quality.

4.Ablation on token’s numbers in DisVisioner lack of QUANTITATIVE result.You might measure performance across various metrics (like those in Table 1) for different token number configurations.

questions:
1.Could you please explain how subject features and irrelevant features are learned, and why they can be disentangled by transformer blocks without additional supervision?

2.Please improve the quantitative comparison experiment regarding the EFFECT OF DISVISIONER (Ablation study).

3.Please improve the quantitative comparison experiment regarding the EFFECT OF ENVISIONER (Ablation study).

4.Please improve the quantitative comparison experiment on the Ablation of token numbers in DisVisioner, and supplement the comparison experiments for $n_s=1, n_i=0$ and $n_s=0, n_i=1$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720956312

thread_type:
official_review

<thread_object>
creator: reviewer_n4js

<content>
summary:
This paper introduces DisEnvisioner, a method aimed at enhancing the customization capabilities of image diffusion models. The approach involves training two models: DisVisioner, which disentangles subject-specific features from subject-unrelated features, and EnVisioner, which enriches these disentangled features. Quantitative and qualitative experiments demonstrate that DisEnvisioner effectively extracts subject information while discarding unrelated details, significantly enhancing customization capabilities.

strengths:
- The paper is well-written and easy to follow, with figures that clearly illustrate the concepts.
- The visual results are impressive, with the original subject details well-preserved.
- The topic is engaging and holds potential for practical application.

weaknesses:
- **Missing Ablations:** The importance of CLIP prior initialization and augmentation in training DisVisioner has not been fully investigated.
- **Unclear Mechanism for Ensuring Disentanglement:** While the concept of disentangling subject-specific and unrelated features is intriguing, the method section does not clearly explain how this disentanglement is achieved. It remains unclear what guarantees that only subject-related information is retained.

questions:
- During the training of DisVisioner, how is disentanglement achieved? Given that the diffusion model’s objective imposes a loss on both the object and the background, it’s unclear how only subject information is learned for the subject token. Could the authors provide visualization examples or experiments of the learned disentangled tokens?
- What types of augmentations are used during DisVisioner’s training?
- Can this method be extended to use multiple images as conditioning to enhance reference image details?
- Since DisVisioner’s training requires the class name from ImageNet for prior initialization, does this limit its generalization to classes outside ImageNet?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732324057499

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concerns

comment:
The following are the common concerns raised by the reviewers.

## Common Concern-1: More explanation about the disentangling mechanism (for reviewer N4Js, p8Sz, and cZ86)

Beside the experiments shown in Fig. 7 of the main paper and **Sec. D, Fig. D, Fig.E and Fig. F of our supplementary material**, we also clarify the disentangling mechanism more clearly below: 

1. The image can be represented by a sequence of tokens. (Lines 257-258)
2. Furthermore, referred to Tokenizer technique[1], we use orthogonal tokens to represent different semantic concepts of the image, where the image tokens are aggregated into different groups by the query $Q$ using spatial-wise attention. This aggregation of tokens is similar to clustering algorithm that clustering the features with same semantics into the same group. Therefore, in DisEnvisioner, two tokens are separated effectively, considering as subject-essential and subject-irrelevant. (Lines 258-261)
3. Then, as we discussed in Sec. 4.4.1, one single token tends to represent a complete semantic meaning and thus, two tokens are effective enough to disentangle the whole input image into subject-essential and subject-irrelevant features. This analysis is supported by the ablation studies reported in the Gif. 7 and Sec. 4.4.1 of the main paper. In practice, DisVisioner enforce one token represents the subject and another token contains other concept without any specific supervision, which delivers the best results.
4. The order of subject-essential and -irrelevant token can be determined by the CLIP prior[2]. Further, the subject-irrelevant token will be eliminated and the subject-essential token will be used for enrichment and generation. (Lines 301-304)

[1] Learning token-based representation for image retrieval

[2] Clip surgery for better explainability with enhancement in open-vocabulary tasks

## Common Concern-2: The C-I and D-I scores are lower than those of other methods. (for reviewer p8Sz and 6rcz)

In the field of customized generation, which is conditioned on both reference image and textual description, there exists trade-offs between image-alignment and text-alignment, which reflects on C-I or D-I score. For instance:

- When the generation is mostly solely influenced by the textual description, with no contribution from the reference image, text alignment will be maximized, but image alignment will fail (low C-I and D-I scores).
- Conversely, if the reference image dominates the generation process, it becomes impossible to achieve a meaningful editing because the subject and its surroundings will remain unchanged. In this case, image alignment will be excessively high (excessively high C-I and D-I scores), but text alignment will significantly decline.

The lower C-I or D-I score of DisEnvisioner do not indicate the poor performance in identity consistency, instead, it represents the best comprehensive performance, as demonstrated by the “mRank”. As stated in lines 472-477, **we emphasize that the lower image-alignment scores of DisEnvisioner are due to its superior editability**. Modifications such as changing backgrounds, postures, and other attributes naturally affect this metric. In contrast, methods like IP-Adapter and DreamBooth achieve higher image-alignment scores because they tend to replicate large portions of the reference image, leading to excessive ID consistency and reduced flexibility for edits. This artifact of IP-Adapter and DreamBooth can be observed in Fig. 5 and Fig. 6 of our paper, where posture and background significantly impact editability. Apart from IP-Adapter and DreamBooth, DisEnvisioner outperforms other existing methods in both C-I and D-I scores. Moreover, DisEnvisioner achieves the best C-T score for editability among all methods listed in Tab. 1 of the main paper. The overall rank of DisEnvisioner is also the best.

Recent accepted related works on facial image customization like PhotoMaker [1] (CVPR2024, 81 citations and 9.6K github star), **only achieved two SoTA metrics across all seven metrics**, please see Tab. 1 of their paper. Representative baselines of DisEnvisioner, ELITE [2] (ICCV2023 Oral), **only achieved one SoTA metrics across all three metrics**; also the zero-shot (same setting as DisEnvisioner) version of BLIP-Diffusion [3] (NeurIPS 2023), termed BLIP-Diffusion (ZS), **did not achieve any SoTA metrics across all three metrics compared with its baselines**.

[1] PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding. CVPR 2024.

[2] Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. ICCV2023 Oral.

[3] Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. NeurIPS 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732324097966

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all reviewers for their encouraging and insightful comments regarding the novelty and contributions of our paper, such as “novel approach for disentangled visual prompt processing, holds potential for practical application” (**p8Sz, cZ86, N4Js**), “well-conducted experiments demonstrating the efectiveness with highest overall ranking” (**cZ86, 6rcz**), “impressive visual results” (**N4Js, cZ86**), “good settings of tuning-free and single image-based” (**6rcz, cZ86**), “clear writting” (**N4Js, p8Sz, 6rcz, cZ86**). 

We will address all reviewer comments point by point and revise our paper accordingly. 

Currently, all revisions are written in the **supplementary material**, here is the added sections:

- **Sec. D: Ablation on CLIP Prior Initialization**
- **Sec. E: Ablation on the Effect of Augmentations in DisVisioner**
- **Sec. F: Ablation on DisVisioner and EnVisioner**
- **Sec. G: Further Ablation on Token Numbers of DisVisioner**
- **Sec. H: Experiments When Using Multiple Reference Images**
- **Sec. I: Quantitative Comparisons with Transformer-based Methods**

We will carefully revise the main paper according to your concerns and our responses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to enhance the customization capabilities of image diffusion models. The authors propose a method DisEnvisioner by training the DIsVisioner and EnVisioner modules to address the limitations of existing methods in interpreting subject-essential attributes. Quantitative and qualitative experiments are performed and demonstrate the effectiveness of the proposed method. After rebuttals, all reviewers gave positive rating scores. Based on the above considerations, I recommend accepting this paper.