DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation
Authors: Jing He
Haodong Li
huyongzhe
Guibao Shen
Yingjie CAI
Weichao Qiu
Ying-Cong Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Visual Disentanglement and Enrichment
Zero-shot Customization
Text-to-Image Generation
Primary area: generative models
TL;DR: DisEnvisioner effectively identifies and enhances the subject-essential feature while filtering out other irrelevant information, enabling exceptional image customization in a tuning-free manner and using only a single image.

Abstract: In the realm of image generation, creating customized images from visual prompt with additional textual instruction emerges as a promising endeavor. However, existing methods, both tuning-based and tuning-free, struggle with interpreting the subject-essential attributes from the visual prompt. This leads to subject-irrelevant attributes infiltrating the generation process, ultimately compromising the personalization quality in both editability and ID preservation. In this paper, we present $\textbf{DisEnvisioner}$, a novel approach for effectively extracting and enriching the subject-essential features while filtering out -irrelevant information, enabling exceptional customization performance, in a $\textbf{tuning-free}$ manner and using only $\textbf{a single image}$. Specifically, the feature of the subject and other irrelevant components are effectively separated into distinctive visual tokens, enabling a much more accurate customization. Aiming to further improving the ID consistency, we enrich the disentangled features, sculpting them into a more granular representation. Experiments demonstrate the superiority of our approach over existing methods in instruction response (editability), ID consistency, inference speed, and the overall image quality, highlighting the effectiveness and efficiency of DisEnvisioner.

Published as a conference paper at ICLR 2025
DISENVISIONER: DISENTANGLED AND ENRICHED VISUAL PROMPT FOR CUSTOMIZED IMAGE GENERATION
Jing He! Haodong Li Yongzhe Hu Guibao Shen' Yingjie Cai Weichao Qiu' Yingcong Chen
HKUST(GZZ) *HKUST NNoah's Ark Lab
(jhe812, h1i736 Oconnectt hkust-ga gz. edu. en; yingcongcchnnust..o .k
Avearing Sabrordatt
"divder Seaiiall oecinal
"in p
"ootning raaadth's Srmay
Soie paliking dlyle'
Reference
Reference
siye"
Ta wit
Twih bitte bnologrsund
Refertnne
Reference
Teible & Bist fiid pottew!
Tel ouited Ghapp"
"oo cobiiist shreet
witee styld sls
the shoee stole
Swittd shaps"
"witin the the bastegoound"
Reference
Bio Bdn
Shatt Soldes
Elitl TReat
*e the zarg mlght shid"
fing
"ilii ah couptiegg velones in the Enckeground"
Figure 1: Customization examples of DisEnvisioner: Without cumbersome tuning or relying on multiple reference images, DisEnvisioner is capable of generating variety of exceptional cus- tomized images. Characterized by its emphasis on the interpretation of subject-essential attributes, DisEnvisioner effectively discerns and enhances the subject-essentia feature while filtering out ir- relevant attributes, achieving superior personalizing quality in both editability and ID consistency.
ABSTRACT
In the realm of image generation, creating customized images from visual prompt with additional textual instruction emerges promising endeavor. However, isting methods, both tuning-based and tuning-free free, struggle with interpreting the subject-essential attributes from the visual prompt. This leads subject-irrelevat attributes infiltrating the generation process, ultimately compromising the per- sonalization quality in both editability and ID preservation. In this paper, we present DisEnvisioner, novel approach for effectively extracting and enriching the subject-essential features while filtering out -irrelevant information, enabling exceptional customization performance, in tuning-free manner and using only a single image. Specifically, the feature of the subject and other irrelevant compo- nents effectively separated into distinctive visual tokens, enabling much more accurate customization. Aiming to further improving the ID consistency, we en- rich the disentangled features, sculpting them into Experiments demonstrate the superiority of our approach over existing methods in instruction response (editability), ID consistency, inference speed, and the overall image quality, highlighting the effectiveness and efficiency of DisEnvisioner.
granular representation.

---

Published as a conference paper at ICLR 2025
cat in Buckkee"
AO+
"a oat walking on the snow"
A DisEnVisionee
Reference
DreamBooth
DisenBooth
BLI--Diffusion
ELITE
IP-Adapter
Figure 2: Comparisons between DisEnvisioner and existing methods (Ruiz et al., 2023; Chen al., 2023a; Li et al., 2024; Wei et al., 2023; Ye et al., 2023) under single-image setting. We evaluate these methods on the same subject with different poses and environments. can be ob- served that irrelevant factors, such as the subject's posture and background can affect the customization quality and result in poor editability
poor II) consistency
For instance,
BLIP-Diffusie falls in both two factors, leading to poor editability and ID consistency. We denotes its performance as customization is not affected by subject's posture, and "L_" indicates good editability). We also try use masks to filter irrelevant information for these methods. However, the harmful influence of subject's posture still exists. And solid background colors (e.g., white or black) also can harm- fully impact the customization quality, leading to textureless backgrounds.
(the symbols without color filling, such as " "A"" indicates that the
INTRODUCTION
By training with billions of image-text pairs, state-of-the-art text-to-image generation models, e.g., DALL-E (Ramesh et al., 202 1). Imagen (Saharia et al., 2022), UnCLIP (Ramesh et al., 2022), Stable Diffusion (SD) (Rombach al., 2022), and PixArt-a (Chen al., 2023b), have demonstrated markable proficiency in generating contextually aligned images from textual descriptions. Despite unprecedented creative capabilities of these text-to-image models, customized image generation poses new and more intricate challenge. This task aims to synthesize life-like imagery that not only accurately responds to natural language instructions (editability) but also preserves the sub- ject's identity based on reference images (ID consistency), is garnering great attention from both academia and industry (Chen al.. 2023a; Dong et al., 2022; Shi al., 2023; Ma al., 2023: Gal et al., 2022: Kumari et al., 2023; Ruiz et al., 2023: Li et al., 2024; Wei et al., 2023; Ye et al., 2023; Li et al., 2023c; Wang et al., 2024; Chen al., 2023c; Arar al., 2024; Voynov al., 2023).
For high-quality image customization, accurate interpretation of the visual prompt (i.e., the input image) is crucial. This involves effectively extracting subject-essential attributes from the reference image while minimizing the influence of subject-irrelevant attributes. Failure to do may result in 1) overemphasis on irrelevant details: generated images may prioritize irrelevant information, sidelining the textual instructions and compromising the overall editability; 2) diminished subject identity: the feature of subject-essential attributes becomes entangled with irrelevant information, degrading the subject representation.
2

---

Published as a conference paper at ICLR 2025
(w/o En EnVisioner)
Subject- essential
En Visioner
Dis Visioner
Subject- irrelevant
Discard
"nuniing
beach"
Figure 3: Overview of DisEnvisioner, which consists two key components: DisVisioner and En- Visioner. During inference, subject-irrelevant features are discarded to avoid harmful disturbances.
Existing methods, both tuning-basee based (Ruiz et al., 2023; Gal et al., 2022; Kumari al., 2023; Chen et al., 2023a; Dong et al., 2022) and tuning-free (Wei et al., 2023; Li et al., 2024; Ye et al., 2023; Ma et al., 2023; Li et al., 2023c), struggle accurately interpret subject-essential attributes, par- ticularly given only single reference image. Specifically, prevailing tuning-based based methods like DreamBooth (Ruiz et al., 2023) and DisenBooth (Chen et al., 2023a), heavily rely on multiple refer- ence images to capture the common subject concept into the model. Despite their impressive results, is difficult to interpret the subject-essential attributes in single-image scenarios, leading to com- promised customization quality (Fig. 2). Additionally, each subject requires individual fine-tuning, which is time-consuming and hinders practical application. Recent tuning-free methods (Wei et al., 2023; Ma et al., 2023; Li et al., 2023c; Shi et al., 2023; Ye al., 2023; Li al., 2024; Chen al., 2023c), aim to offer significant boost in inference speed. and can generate customized images with single reference image. However, these approaches still fail to accurately disentangle the subject- essential features from the reference image. Among those methods, IP-adapter (Ye et al., 2023) and Verse (Chen al., 2023c) directly treat the global feature of the given image as subject- essential, inevitably introducing the subject-irrelevant information that diminishes the personalizing quality. Moreover, ELITE (Wei et al., 2023) and
(Li et al., 2024) attempt to learn subject representations, but their implicit extraction of subject features proves ineffective (please see Sec. 2.2 for detailed discussion), also leading to unsatisfactory customization performance hindered by irrelevant information. As illustrated in Fig. 2, the customization quality of these methods are notably compromised by irrelevant factors (we conclude the primary irrelevant factors in this case subject's posture, background and image tone). A direct solution of these methods prevent the irrelevant information is to use segmentation mask to remove the background, but it is inad- equate. As shown in Fig. ① factors like the subject's pose still introduce irrelevant details; @ replacing the background with solid colors (e.g., white or black) also influence the customization, potential leading to texture-less backgrounds the generated customized images. Thus, an effective disentanglement of the subject-essential features is indeed necessary.
Motivated by the above analysis, we propose DisEnvisioner, a novel framework meticulously de- signed to addresses the core issues via feature disentanglement and enrichment. As illustrated in Fig. 3, the image is tokenized into compact disentangled features, ie., subject-essential and subject- irrelevant tokens, through the Dis making sure the model only focus on essential attributes of the subject, facilitating more accurate editability during customization. The disentangled subject-essential features further enriched by En Visioner before feeding into the pre-trained SD model (Rombach et al., 2022) for customized generation, significantly boosting the co-onssiyyy and the overall customization quality. Together with above innovations, we achieve both accurate and high-quality image customization.
Thus, the subject-inrelevant features can be filtered,
In summary, key contributions
follows.
We emphasize the critical role of cebject-essential attribute in customized image genera- tion, which is the foundation of faithful subject concept reconstruction and reliable editabil- ity, thereby ensuring more accurate customization under more diverse textual instructions.
We present DisEnvisioner, a simple yet effective framework designed for single-image, tuning- free image customization, utilizing visual disentanglement and enrichment.
Comprehensive experiments validate DisEnvisioner's superiority in adhering to instruc- tions, maintaining ID consistency, and inference speed, demonstrating its superior person- alization capabilities and efficiency.
3

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
2.1 TEXT-TO-IMAGE GENERATION
In the field of text-to-image generation, the evolution of methodologies has transitioned from Gen- erative Adversarial Networks (GANs) (Goodfellow et al., 2014; Zhang et al., 2017: 2018; 2021; He et al., 2022; Karras et al., 2019; 2020; 2021) to advanced Diffusion Models (Ho et al., 2020; Ramesh
et al., 2022; Saharia et al., 2022; Ramesh et al., 2021; Nichol et al., 2021: Chen et al., 2023b; He et al., 2024; Rombach et al., 2022). Early GAN-base models like StackGAN (Zhang et al., 2017; 2018). AttGAAN (Xu et al., 2018), and XMC-GAN (Zhang et al., 2021) introduced multi- stage generation and attention mechanisms to improve image fidelity and alignment with textual descrip- tions. A significant breakthrough, DALL-E (Ramesh al., 2021), utilizes transformer and auto- regressive model to merge text and image data, trained on 250M pairs for high-quality, intuitive image synthesis. Following this, a series of diffusion-based methods such as GLIDE (Nichol al., 2021), DALL-E2 (Ramesh et al., 2022), and Imagen (Saharia et al., 2022) have been introduced, offering enhanced image quality and textual coherence. The Latent Diffusion Model (LDM) (Rom- bach et al., 2022), trained on 5 billion pairs, further enhanced training efficiency without compro- mising performance, becoming a community standard. Despite remarkable strides in generating images from textual descriptions, current methodologies fall short in rendering customized visual concepts from reference images. our paper, we dive into customized image generation, extending the capabilities of existing text-to-image techniques to craft personalized visual concepts.
2.2 CUSTOMIZED IMAGE GENERATION
Existing methods in the field of customized image generation primarily fall into two categories: tuning-based and tuning-free. Tuning-based methods (Gal et al., 2022; Ruiz et al., 2023: Dong et al., 2022; Han et al., 2023; Kumari et al., 2023; Chen et al., 2023a; Hua et al., 2023) involve fine-tuning a pre-trained generative model with several reference images of a specific concept dur- ing test-time. Despite the effectiveness of these methods, their high computational demand and scalability challenges limit the practicality. In contrast, tuning-free methods (Jia et al., 2023; Shi et al., 2023; Li al., 2024; Ma et al., 2023; Ye et al., 2023; Wei et al., 2023; Chen et al., 2023c; Li et al., 2023c; Wang et al., 2024) employ advanced encoders to represent customized visual concepts, enabling image generation without the need of test-time fine-tuning and leading to significantly im- proved efficiency. However, InstantBooth (Shi et al., 2023) and PhotoMaker (Li et al., 2023c) still rely on multiple reference images to ensure the II) fidelity, and more recent works start to focus on singe-image scenarios. IP-Adapter (Ye et al., 2023) introduces a coupled cross-attention mechanism that handles visual and textual prompts separately, allowing for fine-grained visual concept genera- tion using only a single reference image. Photo Verse (Chen et al., 2023c) employs a similar dual- attention strategy, with an additional identity loss to further enhance ID consistency. However, both IP-Adapter and Photo Verse directly treat the global feature of the given image as subject-essential inevitably introducing the subject-irrelevant information that diminishes the personalizing quality. BLIP-Diffusion (Li et al., 2024) and ELITE (Wei et al., 2023) attempt to learn subject-essential representations. DLff-Diffusion derives subject representations by querying image features with subject names using BLIP-2 (Li et al., 2023a). However BLIP-2 is pre-trained to extract global image feature aligned with global text prompts. It may struggle to accurately query clean local subject-essential features when the given subject names describe only the local information of whole image. Thus, BLIP-Diffusion may still fall short in filtering out subject-ireelevnt attributes. ELITE (Wei et al., 2023) separates subject attributes from others by utilizing CLIP features from different layers, assuming that the subject is represented by the deeper features and other concepts by shallower features. However, since the deep and shallow features are not fully disentangled, the subject attributes obtained through this multi-layer approach still remains inaccurate. Addition- ally, ELITE requires segmentation masks to combine the masked CLIP image features and the ob- tained subject feature for enhanced details. However, the segmenting process is cumbersome and the masked image feature is still inaccurate and can introduce bubbject-irrelevan factors. In this paper, al- though DisEnvisioner also adopts a two-stage pipeline following BLIP-Diffusee and ELITE, unlike these methods, DisEnvisioner focuses more on explicitly extracting the subject-essential attributes via Dis Vionser (Sec. 3.2), achieving more accurate customization. Then, we enrich the disentangled features by En Visioner (Sec. 3.3), rather than introducing additional features as in ELITE (Wei et al., 2023), to further enhance the IDD consistency without unnecessary disturbance.
4

---

Published as a conference paper at ICLR 2025
3 METHOD
The goal of customized image generation is to synthesize lifelike images that adhere to the tex- tual instructions while preserving the subject's identity from the reference image. To tackle the pivotal challenge minimizing the influence of subject-irreleaan attributes and enhancing subject- essential attributes, introduce DisEnvisioner-a novel approach focused on disentangling and enriching subject-essential attributes. As depicted in Fig. 3. DisEnvisioner initiates the process by disentangling the image features into subject-essenticg and -irrelevant attributes by the Dis Visioner. Subsequently, to bolster the consistency of the subject's 's identity, the En Visioner is further employed to refine the subject-essential features into a more fine-grained representation. Our discussion com- mences with an overview of basic concepts (Sec. 3.1), followed by an in-depth exploration of the Dis Visione (Sec. 3.2) and En Visioner (Sec. 3.3).
3.1 PRELIMINARIES
Diffusion models (Ho et al., 2020; Ramesh et al., 2022; Saharia et al., 2022; Ramesh et al., 2021; Nichol et al., 2021; Chen et al., 2023b) represent the current state-of-the-art in generative modeling for high-fidelity image generation, it involves forward diffusion and reverse denoising processes. The forward process incrementally introduces noise to the data, transforming it into a Gaussian random noise by a Markov chain over fixed number of steps T. In the reverse phase, a learned neural network is utilized to predict the added noise at each step, thereby recovering the original data from the noisy sample. This poising-denoising mechanism enables diffusion models achieve impressive results in generating high- high-quality images with fine details and realism.
In this study, we adopt Stable Diffusion (SD) (Rombach et al., 2022), a latent diffusion model built upon UNet (Romneberger et al., 2015), as our foundational generative model. Firstly, an auto- encoder (VAE) (-), { is trained map between RGB space and the latent space, i.e., (x) = And the textual conditions are obtained using pre-trained CLIP text encoder c (v), where is the given prompt. The training objective is:
(6 (x)) x
(ODM(0) B.x..c.err-W/............ ||e, Ea(Zi, c, 1)
where represents the noise introduced during the forward process, and éa (Zr, C, t) is the predicted noise. Cross-attention is adopted to introduce textual condition into SD. Specifically, the latent image feature f T (R)x)))xd, and text condition Q = Wee.q f. key K = WLosk c, and value = w_o.v o c, where w are weights of the corresponding mapping layers. Then, the cross-attention is calculated as:
Rayxd are firstly projected to obtain query
QKI
Attention K, V) = SoftMax
V.
(2)
Vdl
During inference, the model iteratively constructs images xa from random noises zr through reverse denoising process.
3.2 DISVISIONER
As revealed (Wei et al., 2023; Li et al., 2024; al., 2023), the customized image can be encoded sequence of tokens via inversion. In our work, we propose to disentangle the image feature into subject-cssential and -irrelevant tokens using an image tokenizer (Wu et al., 2022). Image tokenizer is a method to aggregate the image features into compact visual tokens, with each token corresponds to a distinct visual component. As illustrated in Fig. 4a, given a reference image Xtet firstly transform them using augmentation set. The transformed image x' ensures that the model can extract the effective subject-esseee essential feature to reconstruct the original image, rather than merely duplicating the subject. Subsenquently, employ the CLIP image encoder extract image features V (x(re) x(I The image tokenizer network M.) is then utilized to obtain the disentangled visual tokens Vy:
Va = M(ga, (x're))
where qa E tx, is the queries for feature aggregation, n, for the subject feature while n, for other irrelevant features. In image tokenizer M(.), spatial-wise cross-attention mechanism is
5

---

Published as conference paper at ICLR 2025
Input yo
Transformed x
Input Xyef
Best quality"
Dis Visione
Text Encoder u
Image Encoder v/b
Image
Queries 4d
Feature ref
Va
Subject Feature Projector Pg
Irrelevant Feature Projector P
Image Tokenizer
Spatial-wise Attention
Transformer Block
Ve
Ye
Disentangled
Features Va
Repluce S
'A photo of
X
Xx
X
k
K kivi
K*V*
Kiyi
KV
KV
KV
KV
Q
Q
Stable Diffusion
Stable Diffusion
(a) Dis Visioner
(b) Visione
Figure 4: Training pipeline of DisEnvisioner. Our approach structured into two stages: (a) Dis- Visioner firstly disentangles the features of the subject and other irrelevant components by aggregat- ing the image feature Uhe into two distinct and orthogonal tokens Va. (b) En Visioner subsequently refines and sculpts the disentangled features Va into more granular representations to produce high ID-consistency images with the input image Xref, and can improve the overall visual quality of the image. Only colored modules (orange and blue) are trainable.
firstly adopted to aggregate the image features, where qa is the query and eb (Xeer) serves as the key and value. The image features separated into two mutually independent and orthogonal sets of features according to Qa after the spatial-wise attention mechanism. This mutual independence and orthogonality
ensured by the SoftMax(() function applied spatial dimension within the spatial-wise attention. In order to determine the sequence of disentangled features, we initialize qa with the CLIP prior, as revealed in (Li al., 2023b). Specificallt, the n, queries for subject are initialized with random vectors, while the remaining queries are initialized using CLIP class name embeddings query the subject-irrelevan tokens. Transformer blocks are followed further refine the disentangled features to finally obtain Va as the output of our Visione..
To train the image tokenizer M()., we insert the disentangled features Va into textual feature space of the prompt by replacing the textual embedding of placeholder 5 and adopt the Eq. 1 as the training objective for the target of reconstruction. Following (Wei al., 2023; Kumari al., 2023), the entire SD model is frozen except the to_k and to.v layers of the cross-attention module for correct interpretation of the new disentangled tokens. In our implementation, we set n, = and n, = for subject-essente essenniall and irrelevant features, respectively. Excessive tokens will lead to inaccurate disentanglement, thus impairing the customization quality (please refer to Sec. 4.4 for more details).
Benefiting from image tokenizer, the subject-essential features accurately compressed into the tokens, clearly and accurately separated from irrelevant features via the spatial-wise attention space. During customized image generation, the iellecttirrelevant token will be discarded for exclude the unwanted disturbance, facilitating the editing accuracy and ID fidelity.
3.3 ENVISIONER
While Dis effectively extracts the subject-essential feature into single token, it may be inadequate for capturing the detailed nuances of the customized subject. En Visiionee, we map the disentangled features into sequence of granular tokens using projectors P*() and P'(.). The utilization of separate projectors also guarantees the disentanglement between the subject-essential and -irrelevant tokens.

---

Published as a conference paper at ICLR 2025
As shown in Fig. 4b, the disentangled tokens Va from Dis is enriched into multiple tokens: r: = Pa T T = P Tá
(4)
where Tåo Tá E II denote the disentangled subject-essential and the irrelevant feature through Dis (V = Til. Tē R2d( is the enriched subject-essential tokens from Tá and T Rnfxd is the irrelevant tokens projected from r4- By enriching the subject-essential token into multiple ones, the disentangled concept is further enhanced into a more granular representation, improving especially the ID consistency between the synthesized image and the reference image.
Similar to Dis Visioner we also employ Eq. I as the training objective to train the feature projectors in En Visioner while keeping the entire SD model and all Dis Visioner modules frozen. Additionally, we separatelt introduce cross-attention layers for subject-essential tokens and -irrelevant ones. This separate injection strategy further ensures that the subject feature will not be interfered with by other irrelevant factors. Specifically, given the latent diffusion feature (Hxx) and text instruction e E Ty xd,, the cross-attention output f is derived through three decoupled cross-attention layers, which can be described via the following Equation:
f = Attention Q, K, V) + A2 Attention Q K" V*) + A; Attention Q K V'), where K" w... a Te V w.... o T;; K = w... Te V -'==wa.
(5)
wia. o T.
the Attcnntoon-) is defined in Eq. 2, w wook w w to.k and w are trainable mapping layers. During training, weights X, and X; fixed 1.0. During inference, for the purpose of effectively ignoring subject-irrelevan features encoded in Yú we set X; =0 0.
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
4.1.1 TRAINING DATASET
We use the training set of Openlmages V6 (Kuznetsova et al., 2020) to train the DisEnvisioner. It contains about 14.6 14.61M annotated boxes across 1.74M images. Based on this dataset, we construct 6.82M (prompt, image] } pairs for training, where the images are cropped and resized (256 x 256 for Dis Visioner and 512 x 512 for En Visioner) according to the bounding box annotations, and the text prompts are obtained by randomly selecting CLIP ImageNet template (Radford et al., 2021) and integrating the annotated class names into it.
4.1.2 EVALUATION DATASET AND METRICS
The evaluation is carried out on the DreamBooth (Ruiz et al., 2023) dataset, which comprises 30 subjects and 158 images in total (4~66 images per subject). For quantitative evaluation, 25 editing prompts (Ruiz et al., 2023) are used for each image. We inference 40 times for each (prompt, image} pair, generating 158,000 customized images for evaluation across 6 metrics.
In alignment with previous methodologies (Ruiz et al., 2023; Kumari et al., 2023; Wei et al., 2023; Li al., 2023e), we utilize: 1) CLIP Text-alignment (C-T) to measure the instruction response fi- delity; 2) CLIP Image-alignment (C-I) and 3) DINO Image-alignment (D-I) (Caron et al., 2021) to evaluate the ID consistency with the reference image. Additionally, we introduce novel metric, i.e. Internal Variance (IV) of image-alignment, aiming at quantifying the impact of irrelevant fac- tors. IV measures the variance of customization results for images containing the same subject but different environments (like the top three rows in Fig. 2). Lower IV value indicates that the results only controlled by the subject and textual instruction, rather than the irrelevant factors from envi- ronments. In terms of efficiency, we record the 6) Inference-time (T) on single NVIDIA A800 GPU for evaluation. Moreover, we calculate the mean ranking (mRank) of all metrics for each method to show the comprehensive performance.
4.1.3 IMPLEMENTATION DETAILS.
DisEnvisioner is built upon Stable Diffusion v1.5. employing OpenCLIP ViT-H/14 model as the image/text encoder. During training, Dis Visioner configured with batch size of 160, learning rate of 5e-7 the resolution of 256. The training steps is 120K. We set the token number n,
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparisons with existing methods. The evaluation metrics include text alignment for assessing editability (C-T), image-alignment for ID-consistency (C-I, D-D, internal variance demonstrate the resistance subject-irelleant factors (IV), and inference time for effi ciency (T), DisEnvisioner demonstrates better comprehensive performance than other methods. Top results are in bold; second-best are underlined. "For equity, we assess only inference (and test-time tuning, if applicable) times, omitting I/O operations. *For equity, we consider C-I and D-I as two sub-indicators of sange-alignment the rank of each with weight of 0.5 in the mRank calculation, while the ranks of all other metrics have weight of 1.0.
Method
C-T²
C-1$t
D-I²t
IV
T" (s)
mRank!
DisenBooth (Chen et al., 2023a)
0.303
0.760
0.781
0.041
2.42e43
4.8
DreamBooth (Ruiz al., 2023)
0.286
0.842
0.849
0.039
1.12e43
4.3
ELITE (Wei al., 2023)
0.287
0.792
0,770
0.036
4.12
4.1
IP-Adapter (Ye et al., 2023)
0.275
0.883
0.912
0.033
1.98
3.3
Biff-Diffusion (Li et al., 2024)
0.295
0.785
0.765
0.029
1.10
2.9
DisEnvisioner
0.315
0.828
0.802
0.026
1.96
2.0
*ow kop of a purple ruo, forest"
"A cat weAring Savba hat"
Reference
Ours
IP-A dapter
ELITE
L.IP-Diffusion
DreamBooth
DisenBooth
Figure 5: Qualitative comparison live subjects. Comparing two subjects across various scenar- ios, DisEnvisioner excels in editability and ID-consistency. Notably, the animal's 's posture does not affect customization, showcasing our strength in capturing subject-essential features.
and 1, for subject-essential feature and -irrelevant respectively. The En Visioner employs the batch size of 40, learning rate of le-4 at the resolution of 512. The training steps also 120K. The enriched token number is n', experiments are conducted on 8 NVIDIA A800 GPUs using the Adam optimizer (Loshchilov & Hutter, 2017) with a weight decay of 0.01. To enable classifier-fres guidance (Dhariwal & Nichol, probability of 0.05 drop the condition, both textual and visual. During inference, X; set to 0 to eliminate irrelevant feature. In addition, we use the DDIM sampler (Song et al., 2020) with 50 steps and the scale of classifier-free guidance is set to 5.0.
and ni 4, with attention scale \, = 1.0 and X; = 1.0. All
2021),
8

---

Published as a conference paper at ICLR 2025
"with tree and autumu. Leaves in the background"
"with 8 mounntain the baokeground
Reference
Ours
IP-Adapter
ELITE
BLIP-Diffusion
DreamBooth
DisenBooth
Figure 6: Qualitative comparison non-live subjects. Besides offering better editability of tex- tual instructions, DisEnvisioner also excels in preserving accurate subject identity. Moreover, im- ages generated by DisEnvisioner minimally affected by irrelevant elements of reference images.
4.2 QUANTITATIVE RESULTS
We compare DisEnvisioner against five leading methods according to metrics specified in Sec. 4.1.2. For the tuning-based methods, chose DreamBooth (Ruiz al., 2023) and BosehBooth (Chen et al., 2023a), and implement them within the single-image setting. Our tuning-free comparison covers all available open-source methods, they are ELITE (Wei al., 2023), BLIP-Diffusion (Li et al., 2024), and IP-Adapter (Ye et al., 2023). As demonstrated in Tab. 1, our approach achieves the highest text-alignment score (C-T), indicating effectiveness in eliminating subject-irellevnt in- formation and significantly enhancing editability. Although IP- -d--Appter (Ye et al., 2023) and Dream- Booth (Ruiz et al., 2023) achieve high image-alignment scores (C-I and D-I), they struggle with editability. This is because they tend to replicate large portions of the reference image, resulting in excessive consistency and reduced flexibility in making edits (Fig. 5 and 6). Excluding these two methods, approach achieves the highest image-alignmen scores (C-I and D-I), demonstrating its superior consistency without compromising text-alignne aext-alignment. We also achieve lower internal vari- (IV) between the generated images created with the same subject under different conditions, further demonstrating that DisEnvisioner is unaffected by irrelevant factors, such as subject pose and its surroundings. In terms of efficiency, thanks our lightweight and effective design, elimi- nating the test-time tuning, only 1.96s is required for each customized generation. We also conduct user study, as detailed in the supplementary materials, which further demonstrates the superiority of Eiveniiiion.r.
4.3 QUALITATIVE RESULTS
To obtain deeper insights of the DisEnvisioner, we visualize its synthesized images against the selected five prevailing methods across three different images per subject. Fig. 5 and 6 clearly

---

Published as a conference paper at ICLR 2025
Inf. r. n
16
Ref. Image
Subject- evocatial
Subject-
Arreevant
Inf. of T
=
10g 4
71g
16
16 12
4
Figure 7: Ablation on token's numbers in Dis Visioner, Each attention map is calculated by the dot product of the obtained token (r rÿ) and the CLIP local image features. The results demonstrates that ni and n, = outperforms other configurations, achieving precise attention map and faithful token inference.
demonstrates DisEnvisioner's superiority in producing high-quality, editable images with strong ID-consistency. Notably, the consistency animal postures alongside the minimal impact of irrel- evant backgrounds across reference images under the same textual instruction, also showcase our robustness and resistance to subject-irrelevare elements, i.e., only the subject-essential attributes are extracted and preserved. This excellence also applies to non-live objects, seen in Fig. 6, where ex- ceptional customizations for subject the boot and sunglasses demonstrate DisEnvisioner's accurate focus subject-essential features, leading to superior editability, ID-consistency, and the overall visual quality of the generated images.
4.4 ABLATION STUDY
4.4.1 INFLUENCE OF TOKENS NUMBERS IN DISVISIONER.
Input
w/o EV
w/ EV
As described in Sec. 3.2, the disentangled tokens are injected into the textual space of CLIP encoder. thus each token tends to rep- resent a complete semantic meaning, similar to "words". Mean- while, due the mutual exclusivity between subject-essential and -irrleevant features in Dis Visiinne,,, an excessive number of tokens leads to increased competition, making disentanglement difficult and chaotic. For instance, in Wu et al. (2022), only four tokens are used to represent a complex outdoor scene, suggesting that fewer to- kens may be more effective for simpler subjects. To achieve seman- tic completeness and avoid excessive feature competition, we con- duct experiments the number of subject-essential and -irrelevant tokens, i.e., n, and n,. As visualized in Fig. 7, when n, * n,, the image is represented by the larger set without any disentanglement. When nt, n, 1, random initialized subject tokens have more "freedom"" capturing most image information and also leading reconstruction. We also observe that all n, tokens are the same, that is because they use the same prior discussed in Sec. 3.2. In summary, = 1 and performs the best.
Figure 8: Ablation of EV.
4.4.2 EFFECT OF ENVISIONER.
As shown in Fig. 8, we validate the effect enriched subject representation En Visioner (abbrevi- ated as EV). The prompt is "a photo of S* on the beack". We can observe that the images enhanced by En Visioner exhibit finer details and improved ID-consistencyy with significant boost overall image quality, emphasizing its crucial role in delivering high-quality customization.
CONCLUSION
In this paper, we propose DisEnvisioner, which is characterized by its emphasis on the interpreta- tion of subject-essential attributes for high-quality image customization. DisEnvisioner effectively identifies and enhances the subject-essential feature while filtering out other irrelevant information, enabling exceptional image customization without cumbersome tuning or relying on multiple refer- ence images. Through both the quantitative and qualitative evaluations, alongside the user study, demonstrate DisEnvisioner' superior performance in customization quality and efficient inference time, offering promising solution for practical applications.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, and Ariel Shamir. Palp: Prompt aligned personalization of text-to-image models. arXiv preprint arXiv:2401.06105. 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disen- booth: Disentangled parameter-effiieet tuning for subject-driven text-to-image generation. arXiv preprint ar arXiv:2305.0333 2023a.
Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo. Huchuan Lu, et al. Pixart-a: Fast training of diffusion transformer for photore- alistic text-to-image synthesis. arXiv preprint 1rXiv:2310.00422 2023b.
Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding, Yangyang Song, Shizun Wang. Xu Wang, Hao Yang, Jing Liu, Kang Du, et al. Photoverse: Tuning-free free image customization with text-to-image diffusion models. arXiv preprint 23Ki::2300.05793. 2023c.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning. arXiv preprint 1r 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 0rXiv:2208.016188 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ligong Han, Yınxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff:: Compact parameter space for diffusion fine-tuning, arXiv preprint or arXiv:2303. 11305, 2023.
Jing He, Yiyi Zhou, Qi Zhang, Jun Peng, Yunhang Shen, Xiaoshuai Sun, Chao Chen, and Rongrong Ji. Pixelfolder: An efficient progressive pixel synthesis network for image generation. arXiv preprint ar w0X.::22044008333, 2022.
Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Liu, Bingbing Liu, and Ying-Cong Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv preprint arXiv:2409. 18124, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, and Qian He. Dreamtuner: Single image is enough for subject-driven generation. arXiv preprint arXiv:2312. 13691, 2023.
Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304.02642, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of IEEF/CVF conference on computer vision and pattern recognition, pp. 4401-441 2019.
Tero Karras, Samuli Laine, Miia Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 111-8199, 2020.
11

---

Published as a conference paper at ICLR 2025
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Process- ing Systems, 34:852 863, 2021.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Putlern Recognition, pp. 1931-1941, 2023.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Sha- hab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. In- Iernational Journal of Computer Vision, 128(7):1956-1881) 2020.
Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for con- trollable text-to-image generation and editing. Advances in Neural Information Processing Sys- tems, 36, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a.
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement open-vocabulary tasks. arXiv preprint XXXv:23004.06653 2023b.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Pho-
tomaker: Customizing realistic human photos via stacked id embedding. arXiv preprint arXiv:2312.0446, 2023c.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05000 2017.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. arXiv preprint arXiv:2307.11411 1/410, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot -ot-to-immae generation. In International Conference Machine Learming. pp. 8821-8831, PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning rext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
12

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-364499 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411. 2023.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500. 2020.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual condi- tioning in text-to-image generation. arXiv preprint arXiv:2333.09952, 2023.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.07519 2024.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302. 13848, 2023.
Hui Wu, Min Wang, Wengang Zhou, Yang Hu, and Houqiang Li. Learning token-based repre- sentation for image retrieval. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 2703-2711, 2022.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1316-1324, 2018
Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial net- works. IEEE transactions paltern analysis and machine intelligence, 1149911947 2018.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 833-842, 2021.
A
APPENDIX
You may include other additional sections here.
13


Reviews:
[{"review_id": "pmTFF3FkOt", "reviewer": "Reviewer_N4Js", "summary": "This paper introduces DisEnvisioner, a method aimed at enhancing the customization capabilities of image diffusion models. The approach involves training two models: DisVisioner, which disentangles subject-specific features from subject-unrelated features, and EnVisioner, which enriches these disentangled features. Quantitative and qualitative experiments demonstrate that DisEnvisioner effectively extracts subject information while discarding unrelated details, significantly enhancing customization capabilities.", "questions": "- During the training of DisVisioner, how is disentanglement achieved? Given that the diffusion model’s objective imposes a loss on both the object and the background, it’s unclear how only subject information is learned for the subject token. Could the authors provide visualization examples or experiments of the learned disentangled tokens?\n- What types of augmentations are used during DisVisioner’s training?\n- Can this method be extended to use multiple images as conditioning to enhance reference image details?\n- Since DisVisioner’s training requires the class name from ImageNet for prior initialization, does this limit its generalization to classes outside ImageNet?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow, with figures that clearly illustrate the concepts.\n- The visual results are impressive, with the original subject details well-preserved.\n- The topic is engaging and holds potential for practical application.", "weaknesses": "- **Missing Ablations:** The importance of CLIP prior initialization and augmentation in training DisVisioner has not been fully investigated.\n- **Unclear Mechanism for Ensuring Disentanglement:** While the concept of disentangling subject-specific and unrelated features is intriguing, the method section does not clearly explain how this disentanglement is achieved. It remains unclear what guarantees that only subject-related information is retained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PwuzESdXcD", "reviewer": "Reviewer_p8Sz", "summary": "The paper introduces DisEnvisioner, a novel approach aimed at enhancing image customization from visual prompts while addressing the limitations of existing methods in interpreting subject-essential attributes.Empirical evidence demonstrating the superiority of DisEnvisioner over existing methods in various aspects, including instruction response (editability), ID consistency, inference speed, and overall image quality.", "questions": "1.Could you please explain how subject features and irrelevant features are learned, and why they can be disentangled by transformer blocks without additional supervision?\n\n2.Please improve the quantitative comparison experiment regarding the EFFECT OF DISVISIONER (Ablation study).\n\n3.Please improve the quantitative comparison experiment regarding the EFFECT OF ENVISIONER (Ablation study).\n\n4.Please improve the quantitative comparison experiment on the Ablation of token numbers in DisVisioner, and supplement the comparison experiments for $n_s=1, n_i=0$ and $n_s=0, n_i=1$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces DisEnvisioner, a novel framework that addresses significant challenges in customized image generation by focusing on feature disentanglement and enrichment. The originality lies in the identification of the crucial role that subject-essential attributes play in the customization process, which is a new viewpoint that goes beyond existing methods reliant on either tuning or tuning-free approaches.\n\nThe clarity of the paper is commendable. The authors present their arguments logically, making it easy for readers to follow their reasoning and understand the significance of their contributions. Key terminologies and concepts like \"subject-essential attributes\" and \"feature disentanglement\" are well-defined", "weaknesses": "1.The advancement of the methodology has not been sufficiently demonstrated; some indicators (e.g. image-alignment for ID-consistency (C-I, D-I) as shown in Table 1) are inferior to other similar methods (e.g., IP-Adapter, BLIP-Diffusion, etc.),you cloud provide more evidence to support your claim in abstract \"Experiments demonstrate the superiority of our approach over existing methods in instruction response (editability), ID consistency,\"\n\n2.The EFFECT OF DISVISIONER has not been adequately explained; there is a lack of comparative experiments with and without DISVISIONER to validate its effectiveness. You cloud compare results with and without the DiVisioner component while keeping other parts of the system constant.\n\n3.The EFFECT OF ENVISIONER has not been sufficiently substantiated; the paper's explanation regarding the EFFECT OF ENVISIONER only presents a few cases (as shown in Fig. 8), which lacks persuasiveness.You could provide a larger-scale comparison by human evaluation or image-alignment for ID-consistency (C-I, D-I) to measure the improvement in ID consistency or image quality.\n\n4.Ablation on token’s numbers in DisVisioner lack of QUANTITATIVE result.You might measure performance across various metrics (like those in Table 1) for different token number configurations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0DyxuIJfvn", "reviewer": "Reviewer_cZ86", "summary": "The paper, titled \"DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation,\" presents a novel approach to generating customized images from visual prompts with additional textual descriptions. The main contributions are:\n\n1. Proposing DisEnvisioner, a tuning-free framework that effectively disentangles subject-essential attributes from irrelevant features, improving the overall quality of customized image generation.\n\n2. Introducing a two-stage architecture with DisVisioner to separate subject-essential and irrelevant features, and EnVisioner to enhance subject consistency.\n\n3. Demonstrating through experiments that the proposed approach outperforms existing methods in terms of editability, identity (ID) consistency, and inference speed while requiring minimal fine-tuning or reference images.", "questions": "Could the authors provide more theoretical insights into the disentanglement mechanism and why it works well for customized image generation? The current explanation is mostly empirical.\n\nHow does DisEnvisioner compare to other recent Transformer-based text-to-image generation methods? Adding these comparisons could help readers better appreciate the novelty of the approach.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The introduction of DisVisioner and EnVisioner for disentangled visual prompt processing is a novel approach, effectively addressing the challenge of maintaining subject identity while generating customized images.\n\nQuality: The experiments are well-conducted, demonstrating the effectiveness of the proposed method in preserving ID consistency and reducing subject-irrelevant features.\n\nSignificance: The approach enables high-quality, customized image generation in a tuning-free manner, which is practical and efficient for real-world applications.\n\nClarity: The overall structure of the paper is well-organized, and the experimental results effectively showcase the advantages of the proposed method over existing baselines.", "weaknesses": "Limited Theoretical Justification: The paper lacks sufficient theoretical grounding for the proposed disentangling mechanism. For example, the method for separating subject-essential and irrelevant features relies primarily on empirical observations without rigorous theoretical backing.\n\nComparison with Existing Methods: The paper does not provide extensive comparisons with some of the most recent advancements in text-to-image generation (e.g., Transformer-based approaches). Including a wider range of baselines would help position this work within the current state of the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wpoQXdbQzT", "reviewer": "Reviewer_6rcz", "summary": "This paper introduces DisEnvisioner, a novel tuning-free method for generating customized images from a single visual prompt enriched with additional textual instructions. Existing image generation methods, both tuning-based and tuning-free, often struggle to isolate the essential attributes of a subject in the visual prompt, leading to unwanted, subject-irrelevant features that compromise customization quality, editability, and identity (ID) preservation. DisEnvisioner addresses this by disentangling the subject-essential features from irrelevant details, separating them into distinct visual tokens. This separation improves customization precision and allows for enhanced ID consistency. By further refining these disentangled features, DisEnvisioner creates a more granular representation of the subject, which bolsters the model's ability to maintain ID consistency across generations. Experimental results demonstrate that DisEnvisioner outperforms current methods in editability, ID consistency, inference speed, and overall image quality, establishing its effectiveness and efficiency for personalized image generation.", "questions": "Question: Could the authors elaborate on the specific factors that led DisEnvisioner to underperform IP-Adapter in terms of instruction accuracy (C-I) and ID consistency (D-I)?\n\nSuggestion: Analyzing and explaining these performance gaps would provide useful insights into DisEnvisioner's design choices. This could also help clarify if the approach is inherently less suited to certain customization aspects or if there are areas where further tuning could close the performance gap.\n\nQuestion: Given the limited dataset, how confident are the authors in DisEnvisioner’s generalizability to a broader range of tasks and more diverse visual prompts?\n\nSuggestion: Evaluating DisEnvisioner on additional datasets with varied visual content and textual instructions could better assess its robustness. Additional results on a larger dataset could substantiate the claims of effectiveness and versatility.\n\nQuestion: How does the feature enrichment step improve ID consistency, and could the authors provide more details on its implementation?\n\nSuggestion: Adding a more in-depth explanation of the enrichment process and its effect on ID consistency would be beneficial. Additionally, showing comparisons before and after this step could better demonstrate its impact on maintaining subject integrity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "DisEnvisioner demonstrates several strengths over competing methods in the areas of customization quality, editability, identity consistency, and efficiency, as shown in the provided results:\n\nCustomization Quality (C-T): With a score of 0.315, DisEnvisioner shows the highest performance in customization (C-T), indicating that it excels at incorporating subject-specific details while staying true to the given instructions.\n\nID Consistency (D-I): DisEnvisioner scores 0.802 for ID consistency (D-I), surpassing most other methods except for IP-Adapter. This score reflects its ability to maintain subject identity throughout image generation, reducing unwanted attribute drift.\n\nInstruction Response (C-I): DisEnvisioner scores 0.828, slightly lower than some methods like IP-Adapter and DreamBooth but still within a strong range. This indicates good responsiveness to textual instructions while retaining visual prompt characteristics.\n\nInference Speed (IV): DisEnvisioner achieves a lower inference value of 0.026, indicating faster inference compared to methods like DreamBooth and DisenBooth, making it efficient for real-time or rapid customization needs.\n\n*Runtime (T)**: DisEnvisioner has a runtime of 1.96 seconds, placing it on par with IP-Adapter, which is one of the fastest. This efficient runtime makes it more practical for applications needing quick processing.\n\nMean Rank (mRank): With an mRank of 2.0, DisEnvisioner achieves the highest overall ranking among the methods tested, suggesting that it consistently performs well across different evaluation metrics.", "weaknesses": "DisEnvisioner, while showcasing significant advancements, has some limitations that temper its overall impact:\n\nLack of State-of-the-Art Results Across All Metrics: DisEnvisioner does not outperform all baseline models in every metric. For instance, its performance in instruction response (C-I) and identity consistency (D-I) does not reach the top scores achieved by IP-Adapter and DreamBooth. This mixed performance limits DisEnvisioner’s claim to outright superiority across all customization aspects.\n\nLimited Test Dataset: The dataset used for evaluating DisEnvisioner is relatively constrained, potentially affecting the generalizability of the results. A more extensive and varied dataset would provide a clearer picture of the model's adaptability and robustness across diverse tasks and use cases.\n\nRoom for Improved Analysis on Underperformance Against IP-Adapter: Although DisEnvisioner demonstrates strengths in disentangling features, it would benefit from further analysis on why it lags behind IP-Adapter in specific tasks like instruction accuracy and ID consistency. Understanding these discrepancies could inform targeted improvements to make DisEnvisioner more competitive in these areas.\n\nThese aspects suggest that while DisEnvisioner makes notable contributions, there is room for further development to enhance its consistency and broaden its applicability across a wider range of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)