λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space
Authors: Maitreya Patel
Sangmin Jung
Chitta Baral
Yezhou Yang
Venue: Accepted by TMLR ()

Abstract: Despite the recent advances in personalized text-to-image (P-T2I) generative models, it remains challenging to perform finetuning-free multi-subject-driven T2I in a resource-efficient manner. Predominantly, contemporary approaches, involving the training of hypernetworks and Multimodal Large Language Models (MLLMs), require heavy computing resources that range from 600 to 12300 GPU hours of training. These subject-driven T2I methods hinge on Latent Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention layers.  While LDMs offer distinct advantages, P-T2I methods' reliance on the latent space of these diffusion models significantly escalates resource demands, leading to inconsistent results and necessitating numerous iterations for a single desired image.

Through empirical evidences we find that CLIP (vision) latent space is already expressive enough to preserve the fine-grained details. Building upon this insight, in this paper, we present λ-ECLIPSE, a diffusion-agnostic prior-training strategy that works in the latent space of a pre-trained CLIP model without relying on the diffusion UNet models. λ-ECLIPSE leverages the image-text interleaved pre-training for fast and effective multi-subject-driven P-T2I. Through extensive experiments, we establish that λ-ECLIPSE surpasses existing baselines in composition alignment while preserving concept alignment performance, even with significantly lower resource utilization. λ-ECLIPSE performs multi-subject driven P-T2I with just 34M parameters and is trained on a mere 74 GPU hours. Additionally, λ-ECLIPSE demonstrates the unique ability to perform multi-concept interpolations. Project page: \url{https://eclipse-t2i.github.io/Lambda-ECLIPSE/}

Published in Transactions on Machine Learning Research (11/2024)
À -ECCIIPE:: Multi- Conce- Personalized Text-to-lmage Dif- fusion Models by Leveraging CLIP Latent Space
Maitreya Patel* Arizona State University
matreyya.patelBssu.ed
Sangmin Jung"
sgmin.jungghasu.co eduu
Arizona State University
Chitta Baral Arizona State University
chitta@asuu edu
Yezhou Yang
yz. gr.yong@asu.co edu
Arizona State University
Reviewed on OpenReview: https: //
// operreveew..we.... net/ forum? id- 7g5Uew1AdM
Abstract
Despite the recent advances in personalized text-to-image (P-T2I) generative models, it mains challenging to perform finetuning- free multi-subject-rriven T21 in resource-efficiet manner. Predominantly, contemporary approaches, involving the training of hypernetworks and Multimodal Large Language Models (MLLMs), require heavy computing resources that range from 600 to 12300 GPU hours of training. These subject-driven T2I methods hinge on Latent Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention layers. While LDMs offer distinct advantages, P-T2I methods' reliance on the latent space of these diffusion models significantly escalates resource demands, leading to inconsistent results and necessitating numerous iterations for single desired image.
Through empirical evidences find that CLIP (vision) latent space is already expres- sive enough to preserve the fine-grained details. Building upon this insight, in this paper, present X-ECLIPSE, diffusion-agnostic prior-training strategy that operates in the la- tent space of a pre-trained CLIP model without relying on the diffusion UNet models. A -EECLIPSE leverages the image-text. interleaved pre-training for fast. and effective multi- subject-driven P-T2I. Through extensive experiments, we establish that X-ECLIPSE sur- passes existing baselines in composition aligument while preserving concept alignment per- formance, even with significantly lower resource utilization. X-ECLIPSE performs multi- subject driven P-T2I with just 34M parameters and is trained Additionally, X-ECLIPSE demonstrates the unique ability to perform multi-concept interpo- lations. Project page: https:
mere 74 GPU hours.
github. htt.:eiii.....abbba.a...... io/Lambda-ECLIIPEE/
1 Introduction
The field of text-to-image (T21) diffusion models has recently witnessed remarkable advancements, achiev- ing greater photorealism and enhanced adherence to textual prompts. This has catalyzed the emergence of diverse applications, notably subject-driven personalized T2I (P-T2I) models. In particular, this encom- passes the intricate task of learning and reproducing novel visual concepts or subjects in varied contexts requiring high concept and compositional alignment. The complexity escalates further when multi-subject personalization is desired.
"Equal contribution

---

Published in Transactions on Machine Learning Research (11/2024)
Single Subject Driven T2I
Single Subject Driven T2I w Edge Controlled
doggin chel cutlit
Adogon the beasch
Adog Gedgweaing tn Seggungaaan asseessses at beach
Adag to surling the ocean
dogwearing red hat
A Adog hop dha pappl rugin the forest
Teo dogs in the grass Field
puppy plynggtth
Multi-Subject Driven T2I
the suit of
A cat
on top of the snow mountain
in the suit
wearing
of
wearing
Figure 1: X-ECLIPSE estimate subject-specific latent image embeddings while maintaining the balance between concept and composition aligmment in the CLIP latent space itself.
Early works employed concept-specific optimization strategics involving fine-tuning certain parameters within T21 diffusion models Gal et al. (2022); Ruiz et al. (002a)); Kumari et al. (2023); Tumanyan et al. (2023): Gal et al. (2023). Although these methods achieve state-of-theeart (SOTA) performance, they struggle with generalization and are time-intensive. Contemporary research is pivoting towards fast personalization tech- niques. Within this paradigm, there are two types of popular approaches: Methods that involve training wopernetworks and inlegrating new layers parameters within pre-trained dilfusion UNet models Wei el al. (2023);
Ye et al. (2023); Tewel et al. (2023); Shah et al. ); Ruiz et al. (2023b), and 2) MLLM-based learning of prior models that focuses on leveraging text-latent space of frozen diffusion UNct model Pan
et al. 2023 Sun et al. (023)
The hypernetwork--based strategy achieves single-concept customization but has been extended multi- concepts. Moreover, when combined with additional control (i.c. Canny edge map), they struggle to maintain the concept alignment (~30% drop in performance; Section and strongly favor the edge map. At the same time, ML.LM-based approaches can perform fast multi-concept customization but require heavy computing resources. In Table [ we provide the overview of various single and multi-concept customization method- ologies in terms of total parameters, iterations, and GPU hours required to train the models. It can be observed that multi-concept customization methodologies further increase the resource requirements. For instance, Kosmos-G Pan et al. (2023) consumes 18x more resources than IP-Adapter Ye al. (2023). And Emu2 Sun et al.) (2023) requires training of 19x more parameters compared to Kosmos-G. Hence, despite MLLMs' seemingly useful scenarios, it is not viable to blindly train them.
Upon further investigation, we find that most subject-driven T2I approaches build upon variants of the Latent Diffusion Model (LDM) Rombach et al. (2022), specifically Stable Diffusion models. These LDMs employ cross-attention layers to condition diffusion models with text embeddings, necessitating a mapping of target subject images to latent spaces compatible with the diffusion models the prior training stage. This is also known as score distillation instruction tuning for MLLMs Pan al. (2023). As there is no choice but to learn this text-to-image diffusion latent space, involves backpropagation through the entire diffusion
2

---

Published in Transactions on Machine Learning Research (11/2024)
Table A quick overview of previous works on P-T2I. Our method the first to offer multi- concept-driven generation without depending on diffusion UNet models (except for inference). We provide the extended overview table in the appendix.
Multi Concepts
Finetuning Free
Diffusion
Total opt.
Training Steps
Dataset Size
GPU
Method
Free
params
Hours
Textual Inversion Gal et 2022
5000
DreamBooth Ruiz el al 2023a
X
X
0.9B
800
0.2
ELITE Wei et al. 2023
X
X
77M
135K
125K
336
Li cl
2023a
/
X
15B
500K
129M
2304
IP-Adapter Ye et al. 2023
V
X
22M
1M
672
Custom Difrusion Kumari
2023
X
X
57M
500
0.1
Subject-Diffusingg
2023a
252M
300K
76M
Kosmos-C G Pan et
[2023
L9B
8O0K
200M
12300
Emu-2 Sun el al.
2023
X
37B
70K
162M
X-ECLIPSE (ours)
/ V
34M
100K
2M
74
model often comprising over a billion parameters, contributing to the inefficiency of cxisting P-T2I methods. Therefore, in this work, we focus on answering one question: Do we really need diffusion models to train the customization models?
To answer this question and improve the resource efficiency for multi-concept image generation, we present X-ECLIPSE which leverages the properties of UnCLIP T2T models (e-g- DATII-F 2 Ramesh et al. (2022) and Kandinsky v2.2 Razzhigaev et al. (2023) and performs P-T21 in the compressed latent space of frozen CLIP model. Specifically, unlike previous MLLM-based methodologies, X-ECLIPSE aligns the output space of priors with CLIP vision space instead of the CLIP text space. A-ECLIPSE takes multiple images and text instructions as input and estimates the respective vision embeddings, which can be used by the frozen diffusion UNet model from the UnCLIP stack to generate the resulting image. This elevates the training time dependencies on diffusion models for P-I2I; significantly contributing the resource efficiency. Additionally, as diffusion or ALLLM-based priors are still compute heavy due huge number of parameters and slower convergence, we build upon ECLIPSE Patel et al. (202bb) and SEED Ge et al. (2023 which shows that text- to-image mapping can be optimized through contrastive pre-training. Here, we select ECLIPSE as preferred choice of prior architecture for best efficiency. At last, we propose subject-driven instruction tuning task based on the imago-text interleaved data as a pre-training strategy. This involves creating 2 million high- quality image-text pairs, where text embeddings linked to subjects are substituted with the respective image embeddings, which in return are considered as input to the X-ECLIPSE. While X-ECLIPSE can be plugged with these pre-trained methods, we explore the possibility of X-ECLIPSE to incorporate Canny edge as an additional coarsc-level control synergetically work with subject-driven image generation tasks. Figure [ provides the overview of X X-ECLIPSE capabilities.
Overall, we propose -ECLIPSE as an initial attempt to motivate future works on designing resource-efficient solutions for MLLM-based approaches. We summarize our main contributions as follows:
We show that P-T2T mapping can be learned in CLIP latent space without training-time dependency on the diffusion models; enabling efficient training and fast multi-subject customization. Extensive experiments Dreambench, Multibench, and ConceptBed reveal that X-ECLIPSE (34M pa- rameter model) trained
mere 74 GPU hours achieve competitive performance to that of big counterparts (having 2B-37B paramcters) and improve text-composition alignment.
At last, X-ECLIPSE inherits the smooth CLIP latent space. This allows us to perform the seamless transition between multi-concept generated images.
The designation À X-ECLIPSE iuspired by its conceptual aligument with the A-calculus. In this context, the A-ECLIPSE model functions similarly functional abstraction within X-calculus, where effectively binds variables. These variables, in case, represent novel visual concepts that integrated through composition prompts. Here, ECLIPSE indicates our architecture design choice.
3

---

Published in Transactions on Machine Learning Research (11/2024)
2 Related Works
Text-to-Image Generative Models. Pioneering efforts in image generation, notably DALL-E Ramesh et al. (021)) and CogView Ding et al. (2021). leveraged antoregressive models to achieve significant results. Recent advancements predominantly feature diffusion models, acclaimed for their high image fidelity and diversity in text-to-image (T2I) generation. A notable example is Stable Diffusion, which builds upon the Latent Diffusion Model (LDM) Rombach et al. (2022) and excels in semantic and conceptual understanding by transitioning training to latent space. Imagen Saharia et al. (2022) Pixart-a Chen et al. (2023b) and DALL-E 3 Betker et al. (2023) propose using a large T5 language model to improve language understanding.
DALL-E 2 Ramesh et al. (2022) along with its UnCLIP variation models such as Kandinsky Razzhigaev
et al. 2023 and Karlo Lee et al. (2022), uses diffusion prior and diffusion UNet modules to generate
images using the pre-trained CLIP Radford et al. (0211 model.
Approaches like Textual Inversion Gal et al. (2022), DreamBooth Ruiz al. (023a),, and Custom Diffusion Kumari et al. (2023) focus training specific parameters to encapsulate visual concepts. LoRA Hu el al. (2021) and Perlusion Tewel et al. (2023) largel efficient fine-tuning adjust- ments, particularly rank 1 modifications. However, these methods are constrained by their requirement for concept-specifie tuning. ELITE Wei et al. (2023) was the first approach addressing fast customizeed gener- ation for single-subject T2I. BLi--Diffusion Li et al. (2023a) adapts the BLIP2 encoder Li et al. (2023b). training approximately 1.5B parameters to enable zero-shot, subject-driven image generation. TP- IP-Adapter introduces a decoupled cross-attention mechanism, negating the need to train the foundational UNet model by permitting fine-tuning of a reduced number of 22M parameters.
Personalized T21 Methods.
Mix-of-Show Gu et al. (2023) and Zip-LoRA Shah et al. (2023) train individual concepts and then combine them to generate multiple subjects. Break-A-Scena Avrahami et al. 2023) shows multi-concept capability but requires single images containing diverse objects. Subject Diff'usion Ma et al. (0223a] creates a high-quality dataset and presents the precision control for fast personalized multi-subject image generation. Kosmos- G and Emu2 Sun al. (2023), akin to Subject-Diffusion Ma et al. (2023a), employs Multimodal Large Language Model (MLLM) for text-image embedding alignment, though il necessitales extensive parameter optimization (1.9B-37B). These multi-subject P-T2T methods are not only demanding in terms of parameters but also depend on massive number of frozen parameters of the diffusson UNet model, increasing training computational loads. In contrast, our model, X X-ECLIPSE, forgoes test-time fine-tuning and training-time time reliance on the diffusion UNet model for single and multi-concept, control-guided P-T2I, positioning it as a resourco-officient solution.
At last, methods like GLIGEN Li et (202cc), ControlNet Zhang et al. (0023)). and UniControl Qin et al. (2023) incorporate additional controls (i.e., edge map, depth, segmentations) into the diffusion model to generate the desired images. BLIP- Miff-Diffusion, IP-Adapter, and Kosmos-G can leverage such pre-trained controls. However, in many scenarios, these controls are too strong, making generated images lose subject- specific details. We show that X-ECLIPSE learns to balance the edge map, subjects, and composition. We offer a more comprehensive revicw of related works in the appendix.
3 Method
In this section, we introduce A-ECLIPSE, our approach to multi-subject personalized text-to-image genera- tion. Our method combines the contrastive toxt-to-image strategy from ECLIPSE with the novel imago-text interleaved pretraining strategy, notably omitting the need for explicit diffusion modeling. Our approach mainly capitalizes on the efficient utilization of the CLIP latent space. Figure outlines the end-to-end framework.
The primary objective of X X-ECLIPSE is to facilitate single and multi-subject P-T21 generation processes, accommodating edge maps as conditional guidance. Initially, we detail the problem formulation and elaborate on the UnCLIP stack design of the X-ECLIPSE. Subsoquently, we delve into the imago-text interleaved training methodology. This fine-tuning process enables the X-ECLIPSE to harness semantic correlations between CLIP image and text latent spaces while preserving subject-specific visual features.
4

---

Published in Transactions on Machine Learning Research (11/2024)
Stage 1: Feature Extraction
Stage 2: 2- A-ECLIPSE
Stage 3: Imagee Generation (inference-only)
CLiP(vision)
<cat>
Kandinsky
<cat>
A-ECLIPSE (prior)
v2.2
wearing glassess
<glasses>
"A cat is wearing glasses." M
CLiP(text)
Trained using Projection loss CLIP Contrastive loss
Figure 2: Three stages of the X-ECLIPSE pipcline. Create the image-text interleaved features using frozen CLIP. 2) Pre-train the X-ECLIPSE (34M parameters) using Eq- [ which ensures the mapping to the desired latent space given the image-text interleaved data. 3) During inference, the frozen Kandinsky v2.2 diffusion UNet model takes the output from the X-ECLIPSE and generates the image.
Input
Re-generated
Input
Re-generated
Input
Re-generated
Input
Re-generated
Figure 3: CLIP CLIP (vision) features capture the semantics and fine-grained visual details. Each input is given as input to the Kandinsky v2.2 and re-generated from the decoder. (Top: Real-images, Bottom: Canny edge)
3.1 Text-to-lmage Prior Mapping
In the UnCLIP T21 models, the objective of the text-to-image prior model (fe) is to establish a proficient text-to-image embedding mapping. This model is designed to adeptly map textual representations to their corresponding visual embeddings, denoted as (Jo Zy + za), where En/y represent the embeddings for images and text, respectively. The visual embedding predictions (&g = fo(zy)) are then effectively utilized by the diffusion image generators (ha), which inherently conditioned these vision embeddings (he. 2n x)). In our experiments, utilize the Kandinsky v2.2 diffusion UNet model as ha-
As shown in Figure 3 the CLIP vision encoder is very expressive and preserves the finegrained details in that is required to reconstruct the input image. CLIP image embedding itself achieves the high concept alignment score (DINO: O. 0.66) similar to the finetuning- based DreamBooth method Ruiz et al. (2023a).
Our goal to accurately estimate the image embedding 2x, incorporating the subject representations, thereby eliminating reliance on h& during training. Existing LDM-based P-T2I methods are limited by the LDM's singular module approach (he by + z). Consequently, mastering the latent space of hs becomes essential for effective P-T21 for the baseline methodologies, which limits the previous methodologics.
We propose new mapping function, Se, which processes text representations (2y) alongside subject (xx) specific visual representations (Zza), to derive an image embedding that encapsulates both text prompts and subject visuals (2.).. The challenge lics in harmonizing 2z, and 2, within fe (Zy. Zmy.) - 2,, ensuring alignment while preventing overemphasis on either aspect, as this could compromise composition alignment. To address this, employ the contrastive pre-training strategy after Patel et al. (023bb)
5

---

Published in Transactions on Machine Learning Research (11/2024)
N log
expl 2 EjE[N] expl (2), =/)/T)
Cprior
18 cnN P
1120 - folE, Bng 2mx 113
(1)
Here, À serves as the hyperparameter. i and j represent the index of the given input batch with the size N. ()) represents the inner-product and T the temperabure parameter. The first loss term (projection loss) measures the mean-squared error between the estimated and actual image embeddings, primarily ensuring concept alignment. However, our preliminary studies reveal that exclusive reliance on this term diminishes composition alignment. Therefore, we stick with the contrastive loss component (the sccond loss term) to bolster compositional generalization, with À balancing concept and composition aligument.
Additional Coarsc-level Control-based T2I Prior Mapping, Acknowledging the limitations in ex- isting methods, which necessitate learning the diffusion latent space even for additional control inputs, we endeavor to achieve a more nuanced balance between subject, text, and supplementary conditions. Con- sequently, we have augmented X-ECLIPSE to accommodate an additional modality, a Canny edge map, providing more refined control over subject-driven image generation. This entails modifying the prior model to accept additional conditions (J/= (Zy. 2zg =e) + Egg here 2e is the additional modality embedding).
Additionally, during training, drop 2e for 1% to improve the image generations without relying on the edge map. This enhances the stability and broadens the generalization capabilities of X-ECLIPSE, yielding benefits even in the absence of these controls during inference. Our results demonstrate that X-ECLIPSE learns a unified mapping function, accurately estimating target image reprosentations through the effective integration of text, image, and edge maps leading to learning coarse-level controls instead of hard constraints.
3.2 Image-text Interleaved Training
Our approach largets developing a versatile prior model capable of processing diverse inputs to estimate target visual outputs. Drawing from earlier methodologies, straightforward solution involves concatenating different inputs, like combining text ("a dog wearing sunglasses") with respective concept-specific images. Preliminary experiments indicated that this method does not effectively capture the intricate relationships between target text tokens (e.g. "dog") and the corresponding concept images, especially when multiple concepts are present.
To address this, we adopt the interleaved pre-training strategy used in Kosmos-G, but with a notable modification to enhance resource efficiency. We incorporate pretrained frozen CLIP text and vision encoders for extracting modality-specific refinement in our process is the substitution of subject token-specific text embeddings with corresponding vision embeddings instead of introducing additional trainable tokens handle the image embeddings via rosampler Alayrac et al. (2022). First, extract reference concept visual features (Zn, E Ix12200) from the CLIP vision encoder. Similarly, we also extract the text prompt features (z, E R?7xi280) from the last layer of the CLIP text encoder. Here, 1280 is the CLIP-specific feature dimension. At last, we replace the concept noun corresponding latent features from 2y with žzςi resulting in image-text interleaved features while preserving the contextual information of the text features. This alteration allows us to bypass the need to train the big priors models (e.g. MLLMss), significantly improving the model's proficiency in handling interleaved data.
sebeeddings-sppraatiin text-only from subject-specific images. The key
For the generation of high-quality training datasets, carefully selected 2 million high-quality images from the LAION dataset Schuhmann et al. (2022). each with a resolution of 1024x1024. Utilizing BLIP2, we generate captions for these images and employ SAM Kirillov et al. (2023) for extracting noun or subject- specific segmentation masks. Given the CLIP model's requirement for 224x224 resolution images, we avoid resizing the masks within their original resolutions. Instead, we opt for cropping the area of interest using Grounding DINO Liu et al. (2023a). followed by resizing the masked object while preserving its aspoct ratio. This technique is crucial in retaining maximum visual information for each subject during the training phase. We provide more details about the filters used in the appendix.

---

Published in Transactions on Machine Learning Research (11/2024)
BLIP-Diflusion (SDv1.5)
IP-Adapter (SDXL.
Koamos-G (8Dv1.8)
Emu2 (SDXL)
Ours (Kva.2)
Input
A backpack
a cobblestone street.
A a Santa hat.
cube shaped glasses.
A candle in the snow.
A teapot on the beach.
vase on top of a dirt road.
Figure 4: This figure illustrates a qualitative comparison of X-ECLIPSE with contemporary ap- proaches for singlc-subject T2I generations, utilizing concepts and prompts from the Dream- bench dataset. For each method, concept, and prompt, we generate four images and select the one that most accurately represents the queried concept and composition.
3.3 Additional Concept-Specific Finetuning
Due to the nature of UnCLIP models, even if X-ECLIPSE very accurate, the diffusion UNet model (ha) may not be effective in generating very unique visual representations. gencrated images do not always preciscly follow the reference image (c.g. hair style of the person). However, such behavior is common across the fast P-T2I methods and they lack in terms of maintaining performance compared to the finetuning- based methods (as outlined in Table Therefore, we extend the À -ECLIPPSE and perform concept-specific finetuning.
Figure 3 we can observe that
Compared to the traditional finetuning methodologies (e.g. DreamBooth), X-ECLIPSE provides very unique advantages. As X-ECLIPSE prior model (ff) is pre-trained for personalization, there is no need for further finctuning the fo and we need only finetune diffusion UNet model. Importantly, the fine-tuning of the hs does not depend on the text embeddings (zy). Hence, this leads to stable fine-tuning of the hg: unlike Dream Booth stable diffusion that observes catastrophic forgetting. The new fine-tuning objective is:

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Quantitative comparisons of different methodologies on Dreambench. The Bold and
underline represent the metric-specific first and second-ranked methods, respectively. re-benchmark the performance from open-source weights.
represents that we
Method
Base Model
Params
GPU Hours
DINO (1)
CLIP-I (1)
CLIP-T (t)
Finetuning
Textual Inversion
SDv1.5
768
0.569
0.780
0.255
DreamBooth
SDv1.5
0.9B
0.2
0,668
0.803
0.305
Custom Diffusion
SDv1.5
57M
0.2
0.643
0.790
0.305
BLIP-Diffusion
SDv1.5
0.9B
0.1
0.670
0.805
0.302
X-ECLIPSE*
Kv2.2
0.9B
0.2
0.682
0.796
0.304
Re-Imagen
Imagen
0.600
0.740
0.270
Finetuning free
ELITE
SDv1.4
77M
336
0.621
0.771
0.293
Subject-Diffusion
SDv1.5
252M
0.711
0.787
0.293
BLIP-Diffusion'
SDv1.5
1.5B
2304
0.603
0.793
0.291
IP-Adapter*
SDv1.5
22M
672
0.629
0.827
0.264
IP-Adapter*
SDXL
22M
672
0.613
0.810
0.292
Kosmos-
SDv1.5
L99
T2300
0.618
0.822
0.250
Emu2*
SDXL
37B
0.563
0.765
0.273
X-ECLIPSE*
Kv2.2
34M
74
0.613
0.783
0.307
[|le 1x(x ,1, ,0, 2x)
Ldecoder
E N (=
(2)
Here, is the visual feature of the reference concept image v. Notably, we do not need to use regularization from the DreamBooth as text alignment is already ensured during the pretraining stage of A-ECL IPSE. Moreover, this finetuning can be performed across the set of given visual concepts altogether a single model without degrading performance.
In summary, the prior model, trained with our image-text interleaved data and supplementary condition, presents an efficient pathway for resource-efficeet multi-snbject-drivenn image generations.
4 Experiments
In this section, we first introduce the experimental and evaluation setups. Later, we delve into the qualitative and quantitative results.
Training and inference details. We initialize model, X-ECLIPSE, equipped with 34M parameters. We train our model on an image-text interleaved dataset of 2M instances, partitioned into 1.6M for training and 0.4M for validation. The model is specifically tuned for the Kandinsky v2.2 diffusion image decoder. Therefore, we use pre-trained OpenCLIP-ViT-G/LL as the text and vision encoders, ensuring alignment with Kandinsky v2.2 image embeddings. Training is executed on x A100 GPUs, leveraging a per-GPU batch size of 512 and poak learning rate of 0.00005, across approximately 100,000 iterations, summing up to 74 GPU hours. During inference, the model employs 50 DDIM steps and 7.5 classifier-free guidance for perform the P-T21
the Kandinsky v2.2 diffusion image generator. Adhering to baseline methodologies, following the bascline papers' protocols. For X-ECLIPSE, target subject pixel regions in reference images are segmented before embedding extraction via the CLIP(vision) encoder. We drop the Canny edge map during inference unless specified explicitly. Unless specified all results (quantitative and qualitative) are without concept-specific additional fine-tuning.
Evaluation setup. We primarily utilize Dreambench (encompassing 30 unique concepts with 25 prompts per concept) for qualitative and quantitative evaluations using DINO and CLIP-based metrics Ruiz et al.)

---

Published in Transactions on Machine Learning Research (11/2024)
Table 3: Quantitative comparisons of different methodologies on ConceptBed. We present results on CCD (4) across three evaluation categories. The Bold and underline represent the metric-specific first and second-ranked methods, respectively. represents our benchmarking.
Base Model
Concept
Concept Alignment (4)
Composition Alignment (4)
Method
Average (4)
Replication (4)
Textual Inversion
SDv1.4
0.0662
0.1163
0.1436
0.1087
Dreambooth
SDv1.4
0.0880
0.3551
0.0360
0.1597
Custom Diffusion
SDv1.4
0.2309
0,4882
0.0204
0.2465
ELITE*
SDv1.4
0.3195
0.4666
0.1832
0.3231
BLIP-DIffusion*
SDv1.5
0.3510
0.3245
0.1589
0.2781
IP-Adapter*
SDXL
0.3665
0.3571
0.0641
0.2626
X-ECLIPSE*
Kv2.2
0.2853
0.3619
-0.0200
0.2091
Table 4: Quantitative comparisons of different methodologies on Multibench. The Bold and underline represent the metric-specific first and second-ranked methods on each metric, respectively.
Two Subjects
Three Subjects X-ECLIPSE
Kosmos-G
Emu2
X X-ECLIPSE
Emu2
DINO (t)
0.4549
0.4451
0.4478
0.3168
0.3420
CLIP-I (t)
0.7759
0.7397
0.7409
0.6231
0.6463
CLIP-T (t)
0.2493
0.2673
0.3327
0.2819
0.3469
Table 5: Quantitative results of Canny edge controlled P-T21 of different methodologies on Dreambench. The Bold and underline represent the metric-specific first and second-ranked methods, respectively. Red highlighted numbers indicate the relative percentage drop for concept alignment compared to Table
Method
DINO (t)
CLIP-I (1)
CLIP-T (1)
BLIP-Diffusion
0.423429. 79
0.7119
0.3152
IP-Adapter*
0.4281 31.9%
0.7315
0.3034
X-ECLIPSE=
0.517314.39%
0.7437
0.3158
2023a).. Due to their limitations, we extend our evaluations on the ConceptBed Patel et al. (023a)) bench- mark (covering 80 diverse imagenet concepts and a total of 33K composite prompts), where we report performance on concept replication, concept, and composition alignment using the Concept Confidence De- viation (CCD) metric Patel et al. (2023a). We extend Dreambench for multi-subject customization and present the Multibench dataset. Multibench contains about 24 unique concepts and 15 diverse prompts that result in 904 two-subject specific prompts and 1476 threc-subject specific prompts. We provide further details about the Multibench in the appendix.
4.1 Results & Analysis
Quantitative comparison. The quantitative assessments detailed in Table 2 and Table focus the single-concept T21 task, while Table shows the results multi-concept-drrien image generation. For Dreambench and Multibench, we generate and evaluate four images per prompt, reporting average per- formance on three metrics (DINO, CLIP-I, and CLIP-T). In the case of ConceptBed, we process each of the 33K prompts to generate a single concept image. The results, as depicted in these tables, highlight X-ECLIPSE's superior performance composition alignment while maintaining competitive concept align- ment. Analysis on ConceptBed ( (Table indicates that X-ECLIPSE exhibits notable proficiency in concept replication, albeit with marginal trade-off in concept alignment for enhanced composition ion fidelity. Com-

---

Published in Transactions on Machine Learning Research (11/2024)
paratively, all baselines prioritize concept alignment, often at the expense of composition alignment. While X-ECLIPSE improves the CLIP-T while preserving the DINO; achieved with significantly fewer resources. Notably, Multibench results (Table 4 indicate that X-ECLIPSE significantly outperforms the Kosmos-G (2B params) and Emm2 (37B params) in terms of CLIP-T while maintaining the DINO performance. Therefore, we can conclude that X-ECLIPSE is the most resource-efficient compared, especially when compared to the MT...-basod methods.
Auxiliary finetuning. We further perform concept-specific fine-tuning (as described in Sec- tion 3.3 and Section D After finetuming, shown in Table 2 A A-ECLIPSE outperforms the DreamBooth and BLIP-Diffusion in terms of concept alignment (DINO) while maintaining the performance on composition alignment (CLIP-T).
0.310
0.68
0.305
3.000
0.64
DNO
0.62
Our findings, illustrated in Figure 5 reveal that X-ECLIPSE and DreamBooth exhibit improved performance with incremental fine-tuning steps. Notably, the DINO score improved from 0.61 to 0.68 with few optimization steps and outperforms the baselines (see Table 2h. A detailed analysis indicates that while Dream Booth's DINO score improves, its CLIP-T performance diminishes, hinting at concept overfitting. Conversely, XPECLIPSE consistently improves in DINO scoring without. adversely im- pacting the CLIP-T performance, underscoring the efficacy of our image-text interleaved training approach at the prior stage. Qualitative comparisons, shown in Figure [0] further highlight the benefits of fine-tuning X-ECLIPSE with minimal steps. We provide detailed experimental seturppttttuuppppppuu in Appendix Section D
-.285
0.58
0.56
AECUPSE BRC Dreambout D-O
AECUPSE CUP-T
-0200
Dreandbott CUP
100
200
300
500 fie-ttninng Steps
400
600
700
800
Figure 5: DreamBooth (Stable Diffusion v1.5) vs. X-ECLIPSE (with fine-tuning) w.r.t. DINO and CLIP-T metrics on Dreambench.
Qualitative comparisons. In Figure 4 we present a range of single subject-specific images gener- ated by various methodologies including BLIP- Difffuiin LP-Adapter, Kosmos-G, Emu2, and X-ECLIPSE. X-ECLIPSE demonstrates exemplary proficiency in composition while ensuring concept alignment. In con- trast, the baselines often overemphasize reference images or exhibit concept. dilution, leading to higher concept alignment but compromised composition. Interestingly, we find that Emu2 can capture the single- subjects but fails to reproduce them with complex text compositions (as shown in Figure 4 Similarly, Figure (6 exhibits X-ECLIPSE" multi-concept generation prowess, in comparison to ZipLoRA (fine-tuning- based approach) along with Kosmos-G and Famu2 (Multimodal I.L.M approaches), underscoring its capability to rival compute-intensive methods. We discuss additional examples and limitations in the ap- pendix. That said, even though X-ECLIPSE improves the performance over the basolines, this still not enough and it signifies the challenges associated with fast multi-concept personalization.
Canny edge controlled image generation. As shown Figure 6b the baseline (BLIP-Difiusion) ad- heres strictly to the imposed edge maps, often at the cost of concept retention (rows 1, 3, and 4). This leads large number of unwanted artifacts in the generated images. To further ground this behavior, we first generated images using Stable Diffusion v1.5 for Dreambench prompts without customization then we extracted the Canny edge map and used this edge map to control the subject-driven image generations. At last, we report the performance in Table 5 It can be observed that both basclines IP-Adapter and BLIP- Diffuusion drop the DINO score by 30, which follows the qualitative results. While X-ECLIPSE do not follow the Canny edge precisely but preserves the concept alignment and improves the performance relatively by 21%.
Ablations. We extend our study to evaluate the individual contributions of different components in X-ECLIPSE. Initially, the model's performance with solely the projection loss (referenced [q as- sessed. Subsequent experiments involve training X-ECLIPSE variants with varying hyperparameters for the contrastive loss, specifically À values of 0.2 and 0.5. A comparative analysis of these baselines is conducted
10

---

Published in Transactions on Machine Learning Research (11/2024)
BLIP Dithasion
Oure
Ingub
Input
ZipLors.
Koemos-G
Butgesieontly
Controlied
Bottjct-only
Contrulled
Adog surfingnthh oceann
dogie wearing hat.
Acae wone rg dinglasnee the bearch
Asase as lamp.
backkackkot park.
cats
Adogin the beat
Aaneaker.
(a) Multi-subject P-T2I.
(b) Edge-guided P-T2I.
Figure 6: Qualitative comparison between X-ECLIPSE and other baselines.
Table 6: Ablation studies w.r.t. to the key components of X-ECLIPSE design. We report the concept and composition alignment for singlo-subject T2I using CCD (L) on the ConceptBed benchmark.
Concept Alignment (1)
Composition
Method
Alignment (L)
Projection loss (i.c. X=0.0)
0.394
0.008
w/ contrastive loss (A=0.5)
0.435
-0.043
contrastive loss (A=0.2)
0.402
-0.026
w/ edge conditions (X=0.2)
0.362
-0.020
against the fully equipped X-ECLIPSE model, which incorporates Cprior (Eq with À = 0.2 and utilizes Canny edge maps during training. Relying solely on projection loss results in high concept alignment but compromises compositions (Table 6 The contrastive loss variant with À 0.5 enhances composition align- ment at the expense of concept aligument, whereas À = 0.2 achieves a more balanced performance. Crucially, the integration of Canny edge maps during training optimally balances both alignments and, specifically, improves the concept alignment. The negative values indicate that the CCD oracle model is highly confident in the generated images.
Multi-subject interpolation. A key attribute of the CLIP latent space the ability to perform smooth interpolation between two sets of embeddings. We conducted experiments to demonstrate X-ECLIPSE'S ability to learn and replicate this smooth latent space transition. We selected two distinct dog breeds (<dog1>, <dog2>) and two types of hats (<hatl>, <hat2>> as the concepts. X-ECLIPSE to estimate image embeddings for all four possible combinations, cach corresponding to the input phrase "a <dog>> wearing a <hat.." Fig. 7 showcases a gradual and seamless transition in the synthesized images from the top left to the bottom right. Unlike current diffusion models, which often exhibit sensitivity to input variations requiring numerous iterations of user interactions for desired outcomes, X-ECLIPSE inherits CLIP's smooth latent space. This not only facilitates progressive changes in the conceptual domain but also extends the model's utility by enabling personalized multi-subject interpolations.
then used
11

---

Published in Transactions on Machine Learning Research (11/2024)
hat
dog
Figure 7: Interpolation between four concepts. Here, we estimate the image embedding using X-ECLIPSE corresponding each corner and then interpolate from top to bottom and left to right. At last, we use the Kandinsky v2.2 diffusion UNet model to generate the images with fixed random seeds from these sets of image embeddings.
5 Conclusion
In this paper, we have introduced a novel training-time diffusion-agnostic methodology for personalized text-to-image (P-T2I) applications, utilizing the latent space of the pre-trained CL.IP model with high efficiency. Our X-ECLIPSE model, trained on an image-text interlcaved dataset, achioves the capability to execute single-concept, multi-concept, and edge-guided controlled P-T2I tasks using a singular model framework, while simultaneously minimizing resource utilization. Notably, X-ECLIPSE sets new benchmark in achieving competitive performance in terms of concept and composition alignment. Furthermore, research illuminates the potential of X-ECLIPSE in exploring and leveraging the smooth latent space. This capability unlocks new avenues for interpolating between multiple concepts, thereby generating entirely novel concepts. Our findings underscore a promising pathway to improve MLLMs to effectively control the pro-trained diffusion image models without necessitating extra supervision.
Limitations
Primarily, despite its strengths, CLIP's inability to perfectly capture hierarchical representations adds the upper bound on performance. Hence, enhancing CLIP's representations could significantly boost our frame- work's efficacy in P-T2I mapping. Even though X-ECLIPSE model, trained on 34 million parameters and 1.6 million images, presents a substantial foundation, yet, there's potential for further refinement, increasing the quality of data and the number of parameters could yield even better outcomes. However, this is outside
12

---

Published in Transactions on Machine Learning Research (11/2024)
the scope of this paper. Additionally, validate the X-ECLIPSE only upto three concepts (due to the real-life usecases) and adding more concepts could be explored in future works.
Broader Impact
The current landscape text-to-image (T2I) generative models is dominated by approaches that rely on extensive data and large-scale models to achieve state-of-the-art (SOTA) performance, which demands sig- nificant computational resources. In contrast, our work with X-ECLIPSE demonstrates that feasible attain competitive performance relative to SOTA large models while achieving tenfold reduction in resource consumption. This advancement not only makes T2I generative models more accessible and cost-effective but also promotes sustainable AI practices by significantly lowering the environmental impact associated with large-scale model training.
References
Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. neural space-time representation for text-to-image personalization. ACM Transactions on Grophics (TOG), 42(6): 10, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23776,, 2022.
Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano. agmaiic tuning-encoder for fast personalization of text-to-image models. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-10, 2023.
Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scone: Extract- ing multiple concepts from a single image. arXio preprinl arxiv:2305. 10000.5.663111 16311, 2023.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/ /dalee-- pdf. 2:3, 2023.
Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Disentangled parameter-efficient tuning for subject-driven toxt-to-image generation. arXiv preprint arXiv:2305.0333 2023a.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-alpha: Fast training of diffusion transformer for photorcalistic text-to-image synthesis. arXiv preprint ar orXiv:2310.00/28 2023b.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval--uurnented text-to-image generator. In The Eleventh International Conference on Learning Representations, 2022.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming- Wei Chang, and William W Cohen. Subject-driven lext-to-image generation via apprenticeship learning. arXiv preprint arXiv:2304.00 2023c.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34:19822-1999 2021.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Danicl Cohen-or. An image is worth one word: Personalizing to-ig generation using textual inversion. In The Fleventh International Conference on Learning Representations, 2022.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder- based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG); 42(4):1-18 13, 2023.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arviv preprint arXiv:2310.01218. 2023.
Yuchao Gu, Xintao Wang, Jay Zhangjic Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. arXiv preprint arXiv:2305.1828 18292, 2023.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. arXiv preprint orXiv:2303.11300 2023.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, el al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Represen- tations, 2021.
Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero finc-tuning image customization with text-to-image diffusion models. arXiv preprinl urx:w:2044.00 02644, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et, al. Segment anything. arXio preprinl arXiv:2304.0260 0.0403,,.233... 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept. cus- tomization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 1931 1941, 2023.
Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Baek, and Saehoon Kim. Karlo-v1.0.alpha on coyo-100m and cc15m.
2022.
Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. arXiv preprint arXiv:2305114780 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXio:2301. 12597, 2023b.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511 22521, 2023c.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint wrXiu:2303.05400 2023a.
Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint wrXiv:2303.05122 2023b.
Zhiheng Liu, Yifei Zhang, Yujum Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Customizable image synthesis with multiple subjects. In Thirty-seuenth Conference on Neural Information Processing Systems, 2023c.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to- image generation without test-time fine-tuning. ai preprinl or orXiv:2307.11 114100 2023a.
Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jayyng Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arxiv preprint ar arXiv:2303.09311 2023b.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. arx preprint arXiv:3310002002, 2023.
Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Conceptbed: Evaluating concept learning
abilities of text-to-image diffusion models. arXiv preprint arXiv:2306. 04655, 2023a.
Maitreya Patel, Changhoon Kim, Sheng Cheng, Chitta Baral, and Yezhou Yang. Eclipse: A resource-efficient text-to-image prior for image generations. Xrviv preprint arXiv:2312, 2.0455, 2023b.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: unified diffusion model for controllable visual gener- ation in the wild. arXiv preprint arXiv:2305.112 11147, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821 8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, llya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky: An improved text-to-image synthesis with image prior and latent diffusion. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 286-2866290 290-295,22 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-rosolution image synthesis with latent dilfusion models. In Proceedinggs of the IEEE/CVF conference on compuler vision and pattern recognition, pp. 10684-10695, 10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-lo-image diffusson models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 22500-22510, 2023a.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfrr Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to- image models. arXiv preprint "XXx:23377.06940 2023b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kanyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems, 35:36479--36994 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large- scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 2922277722294, 2022.
Viraj Shah, Nataniel Ruiz, Forrester Cole, Frika La, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiw:2311.19600 2023.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint ar Xim:22304 03411, 2023.
15

---

Published in Transactions on Machine Learning Research (11/2024)
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. ar Xiv preprinl arXiv:334.00022, 2023.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprinl ar Xiv:23312. 2. 13286, 2023.
Yoad Tewel, Rinon Gal, Gal Chochik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, pp- 1-11, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for texl-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930, 2023.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to-image generation. arXiv preprint XxXi:2333.09522, 2023.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized texl-to-image generation. 2023 IEEE/CVE Internalional Conference on Computer Vision (ICCV), pp. 15897-15907, 2023. URL https://api. semantisccholll
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcomposer: Tuning- free multi-subject image generation with localized attention. arXiv preprint arXiv:2305. 10431, 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06791, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847,
2023a.
Yuxuan Zhang, Jiaming Liu, Yiren Song, Rui Wang, Hao Tang, Jinpeng Yu, Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al. Ssr-encoder: Encoding selective subject representation for subject-driven generation. arXiv preprint arXiv:2812. 16272, 2023b.
Ruoyu Zhao, Mingrui Zhu, Shiyin Dong, Nannan Wang, and Xinbo Gao. Catversion: Concatenating beddings for dilfusion-based lexi-lo-image personalization. arXio preprint ar Xrxx:22311.146. 2023.
16

---

Published in Transactions on Machine Learning Research (11/2024)
A Preliminaries for T21 Diffusion Models
As evidenced in numerous contemporary studies regarding T21 models, Stable Diffusion (SD) Rombach et al. (2022) has emerged as & predominant backbone for T2I models. SD involves training diffusion models in latent space, reversing a forward diffusion process that introduces noise into the image. A notable feature of SD is its integration of cross-attention, facilitating various conditions like text input. Operating in VQ-VAE. latent space, SD not only achieves exceptional generative performance surpassing that in pixel space but also significantly reduces the computational demands.
UnCLIP models (such as DALL-E 2) are very similar to the Stable Diffusion. In contrast, the UnCLIP takes the modular approach. UnCLIP first trains the diffusion text-to-image to the image prior (fe) to estimate the image embeddings (zz) from the text embeddings (z²)- In parallel, a UNet-like diffusion image generator (hs) is trained to generate images (x) conditioned on ground truth vision embeddings (zx).
Traditionally, T2I prior is modeled to estimate zu-prediction instead of e-prediction. Given the forward function 2Φ ~ (t, Zz), the goal of fo is to directly estimate 2x for all timesteps t ~ [0, as:
Sprior
E (, ~(((s)))))))
[||E So(=12 .4.6,.,,))) +,y)1
(3)
ECLIPSE proposes the contrastive learning strategy (Eq. 1 main paper) instead of minimizing Eq. The diffusion image generator is trained by following standard e-prediction formulation. Here, ha will estimate the ground truth added Gaussian noise (0, I), given the noise image X() for all timesteps [0, F and input conditions (such as 2z, ky).
Ldecoder
E c
||c- ,t
2z, 20)1
(4)
t~ O.
(3z. zy!
For models like Kandinsky v2.2, we drop the By condition the model on Zg. Therefore, A-ECLIPSE also only conditions the image generation with 2y in the prior stage.
B Image- Text Interleaved Training Details
Dataset Creation In constructing the dataset, we adhered to the data processing pipeline of Subject Diffusion Ma et al. (2023a). We utilized the LAION-5B High-Res dataset, requiring minimum image size of 1024x1024 resolution. Original captions were replaced with new captions generated by BLIP-2 (flan-t5-xl I Subjects were extracted using SpacyP] For each subject, we idcntified bounding boxes employing Grounding DINO Liu et al. (2023a). setting both box-threshold and text-threshold values to 0.2. We retained images with 1 8 detected bounding boxes, discarding the rest. Additionally, captions with multiple instances of identical objects were filtered, allowing maximum of identical objects. Following bounding box detection, individual subject masks were isolated using Segment-Anything (SAM) Kirillov et al. (2023). To enhance the efficiency of the training process, we pre-processed the dataset by pre-extracting features from CL.IP vision and text encoders. During this phase, images predominantly featuring a background (white portion) exceeding 50% of the total area were exchuded. We preserved bounding boxes with an area ranging from 0.08 to 0.7 of the total image area and logit scores of at least 0.3. Masks constituting less than 40% of the bounding box area were discarded. For the selection of subjects in images, we constrained the range to 1-4 subjects per image, excluding those with more than 4 subjects. At last, the interleaved image-text examples with respective ground truth images are shown in Figure g
Dataset Statistics In the final analysis, our dataset comprised total of 1,990,123 images. The distri- bution of subjects per image exhibited range from to 4, with the following breakdown: 1,479,785 images
htt.:///ic.ccccce/e/s/eeettee/pp https://spacy.1o
17

---

Published in Transactions on Machine Learning Research (11/2024)
Single Subject Driven Text-to-Image Generations
with boat
living
hoilding sword
playing
inmock
near swiminigg
with a moon the back
torcst
in snowy hildd
jungle the jungle
on cobbisstone sireet
the beach
near volcano
incar fiver
the forest
man esting
with with
with pumplun
Multi-Subject Driven Text-to-Image Generations
with
weaning
the living
sitting at
at
Edge Guided Subject-Driven Driven Text-to-Image Generations
Arobot
Four puppies in the grase
Dogin the bucket
Vasc as tcapot
tay
Figure 8: Qualitative results categorized by generative capabilities.
18

---

Published in Transactions on Machine Learning Research (11/2024)
A wearing in the park.
A wearing a Santa hat.
A on top of the near the river.
Figure 9: Examples of image-text interleaved training data. The left column shows the input of the prior model and the right images shows the ground truth corresponding images. Note: these examples are generated from X-ECLIPSE for better interpretability.
featuring one subject, 432,831 images with two subjects, 65,597 images containing three subjects, and 11,910 images showcasing four subjects. The overall count of unique subjects acquired from this dataset amounted 30,358. We partitioned our dataset. into an 80:20 split. between training and validation, reserving the remaining 1.6 million images for training and the rest for validation.
C Implementation Details
The X-ECLIPSE transformer prior architecture significantly more compact compared to other Text-to- Image (T21) methodologies. Our model employs a configuration of 16 Attention Heads, each with dimension size of 32, alongside total of 10 layers. Additionally, the embedding dimension size for our model is set 1280, supplemented by 4 auxiliary embeddings (including, one for camny edge map). As X-ECLIPSE is not diffusion prior model, do not keep time embedding layers. Overall, the X-ECLIPSE model comprises approximately 34 million parameters, establishing it as streamlined yet effective solution for Personalaized- T2I. Notably, the standard UnCLIP T2I priors contain billion paramcters.
D X-ECLIPSE with Finetuning
As demonstrated in the main paper (Table 2), the superiority of fine-tuming-based personalization method- ologies, whether applied to single-subject or multi-subject frameworks, over non non-fine-tee fine-tuning alternatives is evident. Consoquently, we have expanded our analysis through additional fine-tuning of the À -EECLIPSE.
Experimental Setup. Given that X-BCLIPSE effectively trains the T21 prior, capturing concept-specific features to significant degree, we opted not to further optimize this component. Our focus shifted to exclusively fine-tuning the diffusion UNet model (ha), employing the AdamW optimizer at a learning rate of 1e-5, without warm-up steps. For the DreamBooth application within the Stable Diffusion v1.5 model, selected learning rate of 5e-6, maintaining consistency in other hyperparameters. To simplify, we excluded the use of a prior preservation regularizer and conducted training on the Dreambench platform using single RTX A6000 GPU.
Advantages of fine-tuning X-ECLIPSE. The fine-tuning of X-ECLIPSE, in comparison to the base- lines, reveals two key benefits: 1) Achieving state-of-the-art (SOTA) performance within a few finetuning steps. 2) Unlike the Stable Diffusion model, which exhibits catastrophic forgetting of nearby concopts post- DreamBooth fine-tuning, XPECLIPSE maintains previous knowledge. This suggests that single model is sufficient to effectively fine-tune across multiple concepts together.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Table 7: The detailed overview subject-driven text-to-image generative methodologies.
represents the backbone base models listed are subject to potential updates or modifications.
Method
Multi-Subject
Finetuning-
Base-Model
# of Input Images
Re-Imagen Chen al.
2022
Imagen
Single
Textual Inversion Gal al.
2022
SDv1.4
Multiple
DreamBooth Ruiz
2023a
SDv14
Multiple
Custom Diffusion Kumari
2023
SDv1.4
Multiple
ELITE Wei ct al
2023
SDv1.4
Single
E4T Gal et al.
2123
X
X
SD
Single
Cones Litt et al.
20235
X
SDv1.4
Single
SVDiff Han
21233
X
SD
Multiple
UMM-Diflusion Ma
2023b
SDv1.5
Single
XTI Voynov a. 2023
SDv144
Multiple
Continual Diffusion Smith
al.
2023
X
Multiple
Inota
Shi et a. 2023
SDv1.4
Multiple
SuTi Chen el al.
2023c
Imagen
Multiple
Taming Jia
2023
V
Imagen
Single
BLIP-Diffusion
2023g
SDvL.5
Single
Concs
2023c
SDv2.1
Single
DisenBooth
Chen el al.
20235
SDv2.1
Single
Fast Fast Composee
Xiao ct al
2023
SDv1.5
Single
Perfusion Towel al Mix-of-Show
2023
SDv1.5
Multiple
2023
Chilloutmxx
Multiple
NeTl Alahuf
2023
SDvL44
Mulitple
Break- A-Senn- Avrahami
al.
2023
SDv2.1
Single"
ViCo Tumanyan et al
2023
SDv1.4
Mulitple
Domain- Domain-Agnosiic Arar al.
2023
X
Single
Subject-Diffusion Ma ct al.
202.4
SDv2
Single
BopothreamBooth Ruiz
2123b
SDvL.5
Single
Pr-Aderer
2023
SDv1.5
Single
Kosmos-C3 G an al.
2023
SDvL.5
Single
Shah et al. Z
2023
X
SDXL
Multiple
Cat Version
Zhao al.
2023
X
X
SDv1.5
Multiple
S.R--nncneer Zhang et
21236
SDv1.5
Single
Emm2 Sun
2023
SDXL
Single
XEECLIPSE
(ours)
Rv2.2
Single
This analysis underscores the strategic advantages and enhanced efficiency of fine-tuning X-ECLIPSE for personalized applications in complex visual data processing.
E
Extended P-T2I Baselines Comparison
We further expand our comparative analysis of P-T2I methods encompassing total of 33 approaches including ours and parallel works. Table 7 summarizes them into four crucial aspects: 1) multi-subject support, 2) fine-tuning free, 3) base model types, and 4) the required number of input images. To summarize, X-ECLIPSE the only methodology built on top the UnCLIP models while supporting multi-subject driven image generation with fine-tuning free, and only requires a single reference image input for the training. We detail the comparison below:
Multi-Subject Generation. Multi-subject generation enables users to integrate multiple personal sub- jects to generate image that follows the text prompts and aligns with all the concept visuals. In total, 15 of the 33 methods offer this capability, while 6 methods support fast multi-subject personalization, others demand separate training for each subject to be learned and then an additional fusing step for combining the learned subjects is required (i.c. Zip-LoRA, Mix-of-Show). Among these methods, only few can learn auxiliary guided information such as canny edge, depth maps, or open-pose and adapt style variation (i.c. Kosmos-G).
20

---

Published in Transactions on Machine Learning Research (11/2024)
Input
Without finetuning
Steps=100
Steps=200
Steps=300
Steps=400
A backpack with blue house in the background.
toy on the beach.
A backpack on cobblestone street.
A clock in the snow.
Figure 10: Qualitative examples of X E-ECLIPSE without finetuning and different stages of finel Eninuning.
Fine-tuning Free (Fast. Personalization). Many methods require test-time fine-tuning. Each varies which part alteration occurs, early models tend to modify the whole UNet. In contrast, recent models tune a small portion of the cross-attention layers or introduce additional layers performing as adapters. In our analysis of P-I21 methodologies, 14 out of 33 methods employ finetuning-fre free approach which enables fast personalization.
Diffusion Independent. A majority of the reviewed models utilize diffusion models, with Stable Diffusion being the predominant choice, spanning versions 1.4, 1.5, 2.1, and XL. Few adapt Imagen (SuTi, Taming) and Mix-of-show employs ChillOutMix their pro-trained model, known for its adeptness at preserving realistic concepts like human faces. A unique outlier in this landscape is our X-ECLIPSE, the only one that eschews the use of any diffusion prior model.
more user-frientt friendly model typically requires a single reference image per subject, as opposed to multiple images of the same subject. In our study, 19 methods offer P-T2I capabilities with just one input image. In contrast, others often require to images of the subject. Additionally, some methods necessitale storage space for learned concepts, ranging from few hundred kilobytes (e.g., Perfusion, HyperrreemBBoth) to several megabytes (e.g., Zip-LoRA). Our method stands out by eliminating the need for individual concept lea-ning storing any artifacts for P-T21 utilization, offering streamlined, efficient user experience.
Easiness of Use.
F Multibench Dataset
We provide additional qualitative results in Figure S For the multi-subject image benchmark, our dataset comprises 2,308 unique prompts, segmented into 904 two-subject and 1,476 three-subject prompts. This datasct integrates subjects from the original DreamBench dataset, featuring panded the dataset by incorporating additional concepts vital for two and three subject-specific prompts, such various parks, hats, glasses, and more. Prompt templates and the count of unique subject cat-
distinct concepts. We ex-
21

---

Published in Transactions on Machine Learning Research (11/2024)
Table 8: Example of prompt templates used for Multibench dataset. Subjects presented in Table are placed in f}
Two subjects
Three subjects
O in the {}
O with { and {
( wearing a 0
O is playing with 0 in O
{ chasing [}
{ with {} in front of {
0 looking at 0
O with a 0 and & view of the 0
0 sitting on a Q
O with a 0 and O the background
0 standing on {}
0 and { playing the garden
0 and {) top of the mountain
0 and {} in the jungle
0 and {} in the snow
0 and { on the beach
( and { on a cobblestone street
0 and ( standing next to each other
Table 9: Number of occurrences of unique subject categories. The left side of the table are subjects used for two subjects prompts, and the right side of the table are subjects used for three subjects prompts.
Two subjects
Three subjects
dog
76
boat
5
dog
81
rainbow
35
cat
76
park
4
stuffed animal
105
ruins
35
bird
76
ruins
9
toy
105
tower
35
horse
73
castle
5
cat
81
horse
81
guinca pig
73
descrt
4
desert
60
bird
81
glasses
5
rainbow
5
hill
60
guinea pig
81
hat
candle
5
castle
45
guitar
25
tower
10
backpack
3
backpack
65
french horn
25
can
130
vase
25
candle
65
robot
25
church
35
egories featured in prompts are detailed in Tables g and respectively. Overall, the dataset includes 217 two-subject compositions and 405 three-subject compositions, enriching the benchmark's diversity and comprchensiveness.
G Additional Ablations
In this section, we perform extra ablations studies. Specifically, first, we extend the X-ECLIPSE to other pre- trained diffusion models (namely, Stable UnCT.IP). Later, we analyze the impact of interleaved pertaining in terms of qualitative and quantitative evaluations. Finally, we perform an analysis by varying the data and model size.
G.1 Generalization to Pretrained UnCLIP Diffusion Decoders
Our approach is designed to generalize across any pretrained UnCLIP diffusion models, including Stable- Karlo, and Kandinsky v2.2. We conducted additional pretraining experiments to substan- tiate this claim and demonstrated the generalization ability of X-ECLIPSE. We would like to reiterate the core pipeline of X-ECLIPSE or UnCLIP models, as detailed in Appendix A The UnCLIP model comprises two key modules: (1) the Prior model, which maps the text embedding to the image embedding, and (2) the
22

---

Published in Transactions on Machine Learning Research (11/2024)
Table 10: Ablation study on generalization ability of X-BCLIPSE with respect different pre- trained UnCLIP models. We report performance on DreamBench dataset.
Method
DINO (t)
CLIP-I (1)
CLIP-T (1)
Stable UnCLIP (Stable Diffusion v2.1) Kandinsky v2.2
0.564
0.778
0.276
0.613
0.783
0.307
Table 11: Ablation on the effect of interleaved data for training X-ECLIPSE. We report performance DreamBench dataset.
# of Subjects
Metric
w Interleaved
w/o Interleaved
Difference
DINO
0.6130
0.6006
-2.02%
CLIP-I
0.7830
0.7882
0.66%
CLIP-T
0.3070
0.3048
-0.72%
DINO 0.4478
0.4332
-3.26%
2
CLIP-T 0.7409
0.7384
-0.34%
CLIP-T 0.3327
0.3271
-1.68%
DINO 0.3420
0.3202
-6.37%
CLIP-I
0.6463
0.6480
0.26%
CLIP-T
0.3469
0.3469
0.00%
Diffusion rendering model, which synthesizes the image based on the estimated image emboddings from the prior model. For our generalization studies, we adhered to two essential criteria:
Selection of an open-source UnCLIP (2 stage) model, where we chose Stable-UnCLIP a modified version of Stable Diffusion v2.1, to accept. image embeddings input.
Choice of pretrained CLIP model, with Stable-UnCLIP utilizing the Op model. Accordingly, we trained X-ECLIPSE with this model preliminary feature extractor.
Following these guidelines, trained X-ECLIPSE with the same parameters and evaluated its performance DreamBench (see Table Our results confirm that X-ECLIPSE effectively generalizes to different diffusion models and CLIP variants. Notably, the Ctable-UnCLI variant of X-ECLIPSE achieves performance similar to Emu2, reinforcing the superiority of our initial design choices.
G.2 Impact of Interleaved Pretraining
Section 3.2 mentions that training X-ECLIPSE without interleaved pretraining would yield similar results for "single-concept" P-T2I tasks. However, for "multi-concept" personalization, our experiments reveal that models trained without interleaved data sometimes struggle to synthesize the desired images. We conducted additional experiments by training the model without interleaved data. Specifically, concatenated the prompt embedding from the CLIP text encoder with the concept-specifie image embedding and trained the model with identical hyperparameters. The performance comparisons on DreamBench (single concept) and Multibench (multiple concepts) are shown below. Our findings indicate that without interleaved data, the model's ability to align concepts decreases as the number of concepts increases, resulting in a sharp 6% drop in the DINO score (see Table 11p. This performance degradation primarily due attribute leakago. As shown in Figure when the model is tasked with generating "A backpack the ruins" + <backpack>>> <ruins>, the non-interleaved pretrained models tend to generate the backpack with the color of the ruins.
23

---

Published in Transactions on Machine Learning Research (11/2024)
Table 12: Ablation study on pretraining data size. We report DreamBench performance of X -EECLIPSE models trained on 100k, 500k, 1M and 2M interleaved image-text pairs. denotes the pro- posed X-ECLIPSE model checkpoint.
Data Size
DINO (t)
CLIP-I (t)
CLIP-T (t)
100K
0.593
0.786
0.301
500K
0.595
0.777
0.305
1M
0.596
0.778
0.306
2M
0.613
0.783
0.307
Table 13: Ablation study pretraining model size. We report the DreamBench performance of XEECLIPSE models trained on 5M, 34M, and 70M parameters. denotes the proposed À XEECLIPEE model checkpoint.
Model Size
DINO (f)
CLIP-I (1)
CLIP-T (1)
5M
0.586
0.779
0.305
34M
0.613
0.783
0.307
70M
0.593
0.775
0.309
G.3 Effect of Data and Model Sizes
We also conduct ablation on data sizes (100k, 500k, IM, and 2M) and model parameter sizes (5M, 35M, and 70M). Tables 12 and 13 report the performance these newly trained models on DreamBench. All models were trained with identical hyperparameters as the proposed X-ECLIPSE, though this may not fully optimize the larger models (e.g., 70M parameters). The results show that data size influences model performance, with larger datasets improving concept understanding and prompt. compositions. This also follows the qualitative results in Figure 12 Notably, X-ECLIPSE with only 5M parameters deliver performance close to that of larger models, and the 34M paramcter model even surpasses the 70M parameter model in terms of DINO score. However, qualitative results (see Figure 13 show that increasing model parameters enhances qualitative performance and concept alignment. Specifically, the 70M parameter model excels in generating finer details of the reference concept while adhering closely to text prompts. The 34M model offers a more balanced trade-off between performance and resource efficiency.
H Qualitative Results & Failure Cases
In this section, we showcase collection of detailed qualitative examples from the P-T2I generation process, highlighting the challenges of crafting complex compositions within X-ECLIPSE and comparative models. As depicted in Figure 14 the complexity of the showcased examples progressively increases, illustrating noticeable escalation in the intricacy of visual concepts from the top to the bottom of the figure. With the rising complexity, we note a universal decline in the ability of all methodologies, including A-ECLIPSE, to preserve subject fidelity accurately. Interestingly, despite these challenges, X-ECLIPSE demonstrates a better grasp of compositional integrity, unlike the baseline models which falter across all complexity levels.
Morcover, we present instances demonstrating the variability in outcomes produced by P-T2I methods across different trials. As illustrated in Figure 15 while there is a semblance of consistency in generating single and multiple concepts between models, Kosmos-G specifically shows variability in rendering multiple concepts-occasionall misplacing elements of the Tromman suit on a dog failing to include it altogether. This phenomenon suggests that X-ECLIPSE minimizes image diversity to enhance result consistency, a trait. observed across the UnCLIP model family.
Figure 10 offers qualitative insights into the performance of X-ECLIPSE without and with minimal fine- tuming. It is evident that in certain edge cases, where X-ECLIPSE initially struggles fully grasp novel visual concepts without finetuning, modest application of few optimization iterations significantly enhances
24

---

Published in Transactions on Machine Learning Research (11/2024)
Without Interleaved
With Interleaved
Prompt
Subject 1
Subject 2
Subject
A backpack with a blue house in the background.
A backpack at the ruins.
A bird wearing the hat.
A dog with toy in the ruins.
Figure 11: Qualitative examples X-ECLIPSE model trained with and without the interleaved pretraining stralegy.
concept capture. Further optimization not only preserves text composition but also enriches minor, subject- specific details, underscoring the adaptability and finesse of X-ECLIPSE in nuanced image generation.
Moreover, in evaluations using the Multibench dataset, noticed that both the baseline models (Kosmos-G G and Emu2) and X-ECLIPSE encounter difficulties in precisely maintaining all subject-specifie details, as depicted in Figure 17 This underscores that zero-shot multi-subject P-T2I generation remains a significant. challenge in the field. Further, we explored how well each model preserves gen- uine human facial characteristics in various sconarios, particularly when combined with differing captions. The qualitative examples in Figure [6 shed light on this aspect. Although each model strives to maintain the original facial features, none succeeds in replicating the specific personal facial details accurately. These instances typically fall short of precisely conveying the intended compasitions, with the exception of scenario in IP-Adapter FacelD, indicating notable area for future improvements in model performance.
25

---

Published in Transactions on Machine Learning Research (11/2024)
Data Size
Input
100K
500K
1M
2M
A backpack on the dirt road.
A dog wearing pink sunglasses.
A vase on top of the mirror.
Figure 12: Qualitative examples X-ECLIPSE model trained with varying pretraining data.
Model Size
Input
5M
34M
70M
A cat in a firefighter outfit.
A sneaker with a blue house in the background.
A bowl on the beach.
Figure 13: Qualitative examples X-ECLIPSE model trained with varying model parameters.
26

---

ín the suit of Published in Transactions on Machine Learning Research (11/2024)
Input
BLIP- Diffusion
IP-Adapter (SDXL)
Kosmos-G
Ours
on beach
with the Eiffel Tower in the background
with a blue house in the background
on top of a purple rugin forest
Figure 14: Qualitative examples of the increasing complexity novel visual concepts as we move from top bottom.
surfing in the ocean.
A
Kosmos-G G
Ours
Figure 15:
Qualitative examples of showcasing the consistency comparisons between Kosmos-G and
X-ECLIPSE.
27

---

Published in Transactions on Machine Learning Research (11/2024)
IP-Adapter (SDXL)
IP-Adapter FaceID
Input
Kosmos-G
Ours
A man with the shield.
A man with the hammer.
A man drinking beer.
Figure 16: Qualitative examples of showcasing the failure cases on human faces on Kosmos-G; G, IP-Adapter (SDXL), IP-Adapter (FaceID), and A-ECLIPSE.
Input
Kosmos-G
Emu2
Ours
A backpack at the ruins.
A dog chasing the cat.
Figure 17: Qualitative examples of showcasing the failure cases on Multibench of Kosmos-C G, Emu2, and -EECLIPSE.
28


Reviews:
[{"review_id": "fK9SwdqIdl", "reviewer": "Reviewer_akk3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n$\\lambda$-ECLIPSE is efficient at test-time and requires less resources to pretrain than Kosmos-G. As shown in Table 2, $\\lambda$-ECLIPSE needs 0.2 GPU hours to finetune while Textual Inversion needs 1 GPU hour. The fast speed is due to its algorithm design. $\\lambda$-ECLIPSE trains a mapping function that encapsulates both text prompts and subject visuals into an image embedding, which is further adopted to generate image with a frozen-weight diffusion model. Since the diffusion model won't be updated, the majority of the training cost comes from training the mapping function. The author proposed to combine the concept alignment loss function and the contrastive loss function in training the mapping function, and also pretrain it on interleaved text-image dataset.\n\nApart from being efficient, the performance of $\\lambda$-ECLIPSE is comparable to baselines on Dreambench and ConceptBed, according to Table 2 and Table 3. \n\n\nWeaknesses:\n\nThe author conducted ablation study on some design choices of $\\lambda$-ECLIPSE, including the combination of projection loss (or concept alignment loss) and contrastive loss. However, I haven't seen ablation study on interleaved pretraining. How important is interleaved pretraining for $\\lambda$-ECLIPSE? In addition,  $\\lambda$-ECLIPSE is the only method that relies on Kv2.2 according to Table 7. This casts some doubts on how well the model generalizes to other pretrained text-image diffusion models. Another shortcoming is that the method underperforms BLIP-DIffusion and Textual Inversion in concept alignment. This may indicate that the algorithm is prone to hallucination.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed a new method for Personalized Text-to-Image (P-T2I) called $\\lambda$-ECLIPSE. $\\lambda$-ECLIPSE shares similar idea as UnCLIP, which generates the CLIP image embedding from the text description before mapping it to the image via the diffusion model. $\\lambda$-ECLIPSE takes multiple images and text instructions as input and estimates the respective vision embedding. The vision embedding is fed to a diffusion UNet to generate the image. Compared with other P-T2I methods, $\\lambda$-ECLIPSE is finetuning-free and is compatible with the pretrained diffusion model. It also has comparable performance with state-of-the-art P-T2I methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The author needs to justify whether $\\lambda$-ECLIPSE is generalizable to other text-image pretrained models, and conduct ablation study about the impact of interleaved pretraining.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dTRNGkIsbv", "reviewer": "Reviewer_fKg9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stregths:\n+ Method is well-defined and described. Text is well written and easy to follow.\n+ Related works are well covered including their drawbacks. Comparison to related works gives a good picture.\n+ While proposed method is light weight it achieves competitive results. Being resource efficient (both is parameter space, and the need for data) is a huge plus. \n+ Including additional modality such as Canny edge map, that provides more refined control over subject-driven text-to-image generation.\n+ Evaluation of the individual contributions of different components in $\\lambda$-ECLIPSE loss function.\n\nWeaknesses:\n- Fig4: How does proposed generatin+ selection differs if we generate 100 images instead of 4? How much of variation one may expect from different models, including proposed method?\n- Any results that support this statement in 3.2? \"Preliminary experiments indicated that this method does not effectively capture the intricate relationships between target text tokens (e.g. “dog”) and the corresponding concept images\"\n- What kind of filters are used when sampling 2M from LAION? resolution/aesthetic, etc? \n- How did you find the quality of captions generated by BLIP2? How does using stronger captioning models/VLMs? e.g. CogVLM, or MiniCPM would affect the quality of results?\n- Why did you choose Kv.2.2 ober other UNet models such as SDXL?\n- as discussed in limitations, what is missing is the effect of number of parameters and dataset size on the quality of results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies subject driven personalized/customized text-to-image generation, and in comparison to related works in this literature, specifically tries to answer this question: Do we really need diffusion models to train the customization models?\nThrough empirical evidences, they find that CLIP vision latent space is already expressive enough to preseve fine-grained details. They leverage this finding and propose $\\lambda$-Eclipse, based on the properties of UnCLIP text-to-image models. $\\lambda$-Eclipse aligns the output with CLIP vision space instead of the CLIP text space in LDMs. They claim to be the first to offer multi-concept-driven generation without depending on diffusion UNet models (except for inference). Their method is includes light finetuning using roughly small dataset, but high quality pairs of text and image that can achieve competitive results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "already discussed in weaknesses", "broader_impact_concerns": "- The method is heavily based on pre-trained CLIP models, and hence may be highly impacted by their weaknesses, such as incorporating NSFW contents. [1]\n\n\n[1] Kazemi, Hamid, et al. \"What do we learn from inverting CLIP models?.\" arXiv preprint arXiv:2403.02580 (2024).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SrU7VUMfUD", "reviewer": "Reviewer_4XuP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\nThe proposed λ-ECLIPSE requires much fewer parameters and computation resources to achieve comparable performance compared to prior subject-driven image generation methods. \n\nThe proposed method proposes to use an interleaved feature representation and trains a lightweight projector (34M parameters with 74 GPU hours) to transform the interleaved features into an aligned embedding space. The main image generation backbone is frozen. \n\nTo equip spatial control, the edge map condition is also involved during training with dropout. Coarse spatial control can be enabled with id-preservation.\n\nDue to the design of interleaved feature representation, multi-subject P-T2I and concept interpolation can be achieved naturally.\n\nAbundant experiments and ablations have been performed to demonstrate the superiority of the proposed method on the concept and composition alignment. \n\nCons:\n\nAs to the claim of diffusion-free in Table 1, I think it is just a learning paradigm instead of an advantage. Maybe something like model-agnostic is a better choice if the authors want to claim the generalization of the proposed method. BTW, the title of this work is “Multi-Concept Personalized Text-to-Image Diffusion Models”\n\nIt is claimed that the prior methods do not preserve the identity well when using edge map control. If I understand correctly, the proposed λ-ECLIPSE uses the edge maps as an extra vision embedding in the interleaved features.\nWhile the edge map in Figure 3 does not align well with the image in the re-generated columns, I wonder if the control here is just a coarse-level control for something like layout or sketch.\n\nI wonder if the proposed technique can also be applied to the diffusion models like StabeDiffusion. If so, experiments on this can make this work more comprehensive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a simple and efficient method for finetuning-free Multi-Concept Personalized Text-to-Image with only 34M parameters. The main idea is to fully use the powerful CLIP latent space. Experiment results show that the proposed method can balance the edge map, subjects, and composition well and achieve comparable performance with much less computation resources.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The explanations on the coarse alignment between the edge map and generated image.\nExperiments on diffusion models.", "broader_impact_concerns": "The broader impact has been discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The core findings and contributions of the paper is that the CLIP vision latent contains sufficiently expressive appearance details. This leads to the proposed prior-training strategy without resorting to finetuning diffusion models for personalization. 

The initial reviews recognized the interesting idea and the solid set of experiments. The rebuttal further provides additional ablation on data/model sizes and interleaved pretraining.

After the rebuttal, all reviewers are satisfied with the rebuttal and are leaning to accept the paper.
Decision: Accept (poster)