Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models
Authors: Kyungmin Lee
Sangkyung Kwak
Kihyuk Sohn
Jinwoo Shin
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-image models
diffusion models
personalization
Primary area: diffusion_based_models

Abstract: Text-to-image (T2I) diffusion models, when fine-tuned on a few personal images, can generate visuals with a high degree of consistency. However, such fine-tuned models are not robust; they often fail to compose with concepts of pretrained model or other fine-tuned models. To address this, we propose a novel fine-tuning objective, dubbed Direct Consistency Optimization, which controls the deviation between fine-tuning and pretrained models to retain the pretrained knowledge during fine-tuning. Through extensive experiments on subject and style customization, we demonstrate that our method positions itself on a superior Pareto frontier between subject (or style) consistency and image-text alignment over all previous baselines; it not only outperforms regular fine-tuning objective in image-text alignment, but also shows higher fidelity to the reference images than the method that fine-tunes with additional prior dataset. More importantly, the models fine-tuned with our method can be merged without interference, allowing us to generate custom subjects in a custom style by composing separately customized subject and style models. Notably, we show that our approach achieves better prompt fidelity and subject fidelity than those post-optimized for merging regular fine-tuned models.

Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models
Kyungmin Lee"
Sangkyung Kwak
Kihyuk Sohn Jinwoo Shin
KKIIST Meta Reality Labs
(kyungmnlee, skkwak9806, jinwoos)@kaistt ac kihyuk, www.kkk.oo@@gaa.ccoomo com
Abstract
Text-to-image (T2I) diffusion models, when fine-tuned on few personal images, generate visuals with a high degree of consistency. However, such fine-tuned models are not robust; they often fail to compose with concepts of pretrained model other fine-tuned models. To address this, we propose novel fine-tuning objective, dubbed Direct Consistency Optimization, which controls the deviation be- tween fine-tuning and pretrained models retain the pretrained knowledge during fine-tuning. Through extensive experiments on subject and style customization, we demonstrate that method positions itself superior Pareto frontier between subject (or style) consistency and image-text alignment over all previous baselines; it not only outperforms regular fine-tuning objective in image-text alignment, but also shows higher fidelity to the reference images than the method that fine-tunes with additional prior dataset. More importantly, the models fine-tuned with our method can be merged without interference, allowing us to generate custom sub- jects custom style by composing separately customized subject and style models. Notably. we show that our approach achieves better prompt fidelity and subject fidelity than those post-optimized for merging regular fine-tuned models.?
Introduction
Text-to-image (T2I) models are for image generation guided by natural language prompts and have seen rapid progress in recent years [1-8]. The compositional nature of the natural language has enabled the creation of novel images, which compose multiple subjects with varying attributes in different backgrounds styles. However, the ambiguity of natural language in describing the visual world makes difficult to create an image of specific subject, style, interaction, or background.
To overcome the lack of accuracy in natural language, there has been an emerging interest in teaching the pretrained T2I models new concepts, such as subject [9, 10], style [11], interaction [12], or background [13], whose precise visual description is given by a small set of reference images. As proposed in DreamBooth [10], the fundamental idea to fine-tune the pretrained T2I model on few images describing new concept. The adoption of LoRA [14] or adapter fine-tuning [15] to T21 models has made the process more accessible, fast and economical [16, 11]. Once fine-tuned, the model can generate images by composing new concept (e.g.. my subject) and the knowledge of the pretrained model (e.g., background, style). While these methods have shown great success, they still suffer from reduced textual alignment and compositional generation capability [17], which is problematic particularly when the number of reference images is small, e.g., one or two.
Work done while at Google Research. PProject page: https://dco- t2i, tiweioottiiiigiuubbeeeei.ee github.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

DB.p.p.
DB
DCO
Merging DB models
Merging DCO models
subject
style
Identiy Preservation
X
CreamBooth (D8)
DCO (ours)
X p. (DB+p.p)
Seddy bear (wn a bost, under cheary bicasom, ndveg a biee, drisog calent
leddy bear Nying concon stle
30 randering style
X Subject fidelity
Subject lddeity
v Bubjjctt fitiiit
X Direct merging
v/ Direcl meeging
Prompt edelity
X Pdeily
v/ Promppt
Supject SSylee Gonsistency
v Subject/ Style Gonsestency
Text Alignment
(a) Subject Personalization
(b) My Subject My Style
Figure I: Overview. (a) Direct Consistency Optimization (DCO) pushes the Pareto frontier between prompt fidelity and subject fidelity towards upper-right over DreamBooth [10], and with prior preservation loss (DreamBooth+p.p)) DCO improves generating custom subject with various visual attributes (e.g., astronaut outfits and background of Mars), various styles that pretrained model knows (e.g., flat cartoon illustration style). (b) The customized subject and style models fine-tuned generate my subject in my style [11].
with DCO can be merged is, allowing
We hypothesize that the limitation comes from the knowledge forgetting during low-shot fine-tuning, i.e., it fails to recontextualize new concepts with known concepts of pretrained models. Since fine- tuning with regular diffusion training objective suffer from overfitting, DreamBooth [10] attempted to tackle this issue by adding class-specific prior dataset for training dataset to preserve the prior knowledge (i.e., prior preservation loss). However, while this helps retaining the generalization capability, it comes the cost of less subject fidelity (e.g., Fig. 3). Especially, the loss becomes more significant, when the number reference images become smaller [18].
Contribution. In this paper, we propose more principled method to mitigate the forgetting behavior of low-shot fine-tuning of T2I diffusion models without using any additional data. Our method, coined Direct Consistency Optimization (DCO), fine-tunes T2I diffusion models by controlling the deviation between fine-tuning and pretrained model to be as minimal as possible in learning new concept. In specific, we show that our objective is equivalent to learning implicit reward function that is solution of constrained policy optimization problem [19, 20], which amounts for the consistency between generated images and reference images. Furthermore, sampling which utilizes learned consistency function during inference, so that one can control the tradeoff between subject consistency and textual alignment (e.g., see Fig. 5).
propose consistency guidance
We conduct an extensive empirical study on the personalization of T2I diffusion models (see Fig. 1). We show that the proposed method improves upon baselines that uses regular fine-tuning objective or using prior preservation loss [10] in generation of custom subject with various visual attributes or with the known style of pretrained model. To be specific, show that method positions on the superior Pareto frontier than baselines (e.g., see Fig. 5a). Moreover, our approach is applied to style personalization [1I] and the composition of separately fine-tuned subject and style T2I models to generate an image of my subject and my style [11]. Notably, direct merging of DCO fine-tuned models outperforms merging models fine-tuned with regular diffusion loss even using oost-optimization method, e.g., ZipLoRA [21], in terms of prompt fidelity and subject fidelity. We believe that the proposed method would enjoy broader usage in customization diffusion models in various domains.
2 Related Work
We briefly review the relevant literatures on personalized T2I synthesis and fine-tuning of T2I diffusion models, especially using rewards. We provide more comprehensive review and discussion on related works in Appendix C.
Personalized T2I synthesis. Several works have shown the promise of personalized T2I synthesis from a few images [9, 10, 22, 23, 11]. To improve efficiency, parameter efficient fine-tuning (PEFT) methods such as soft prompt tuning [9, 24-27J, LoRA [14, 28] and adapter tuning [15, 11]. have been proposed, To preserve the prior knowledge, Ruiz al. [10] proposed prior preservation loss, which additionally fine-tunes on class-specifi prior dataset synthesized from pretrained T2I model. We

---

present different approach to preserve prior knowledge by regularizing the model with pretrained model without using any prior dataset.
Fine-tuning T2I diffusion models with rewards. A line of works has studied fine-tuning T21 diffu- sion models using reward models such human preferences [29-31] or aesthetic scores [32]. Those methods use auxiliary reward models to update diffusion models by weighting with rewards [29], use reinforcement learning algorithms [33, 34], or differentiate through the reward models [35, 36]. On the other hand, Wallace et al. [37] have adapted direct preference optimizaton [38] to fine-tune T2I models with paired preference dataset which do not require explicit reward models. To the best of our knowledge, this paper is the first to study consistency as a reward function for T21 personalization.
3 Preliminaries
Diffusion models. For data point x, let q(x) be the density of data distribution and (x() be generative model parameterized by 0 that approximates q. Given x ~ q(x), the diffusion model considers a series of latent variables 2, for timesteps E [0, 1J. The forward process of diffusion model forms a conditional distribution q(z) xx), and the marginal distribution is given by Z, = Q2X + oie, where .V(O.)), I), and a noise scheduling functions. We denote X log(a] /o log signal-to-noise ratio (log-SNR), which is a decreasing function of t. For large enough A1. z1 is indistinguishable from pure Gaussian noise (i.e., p(z1) NO, I)), and for small enough Ao. the 2o is identical to the data x. The generative process of diffusion models starts off from random noise X1 ~O, I), and sequentially denoise it recover Zo. In theory, the generative sampling process is governed by solving SDE [39, 40] or probability flow ODE [41, 42], using the score networks So (Zr;t) that approximates the score function of marginal distribution V log q(z)). When training diffusion models, it is common practice to parameterize score network using the noise-prediction model [40], i.e., (z(2i) () (z( f), Then the training objective of diffusion model is given by
(0; (L-:v00.1).e-N00.))
w (z(; - =
(1)
where Wy is a weighting function and X; is time-derivative of A Note that Eq. (1) is a generalized weighted noise-prediction loss, which includes e-prediction loss [40] when -/ = 1.
Text-to-Image diffusion models. Text-to-lmage (T2I) diffusion models [43, 2, 4] are diffusion models of an image conditioned on text, which is often processed into embeddings using the pretrained text encoders, such as T5 [44] or CLIP [45]. Given a dataset (x,c) c) ~ D of paired image x and text prompt c, the training loss (0: D) for T2I diffusion models is given by (D D) = E(x.c)-D[Laa Le x, c)). with text conditional noise- prise-prediction model co(2; f). T21 diffusion models are often trained with classifier free guidance (CFG) [46], which jointly learns unconditional and conditional models, and interpolates them during inference. The predicted noise with CFG scale w > is given as:
éa(2;; t) (eo(Ze; c) co(2;;t) ())
Ea(2;;)) t).
(2)
where 6g (2(z; I) denotes the unconditional noise prediction, egg., with null text conditioning. It is known that higher CFG scale د improves the image-text alignment the cost of the image fidelity.
Personalizing T21 models. Recent works have shown the potential for personalization of T2I models by fine-tuning the T2I diffusion models on few samples. DreamBooth [10] optimizes the diffusion model on few subject images accompanied with a compact caption composed of rare token identifier and class noun. While fine-tuning with regular diffusion loss (i.e., LDM) works well, the authors proposed the so-called prior preservation loss to retain the prior knowledge of the pretrained model. This is achieved by optimizing the model with auxiliary training images of the same class to the subject of interest. Formally, given reference dataset Dref and prior dataset Dprior. the training loss of DreamBooth with prior preservation loss (DB+p-p.) with hyperparameter Aprior > 0 is given by
CDB+pp. (0) = LDM(() (0; Deet) + X (6; Dprioc.
(3)
Parameter efficient fine-tuning (PEFT). In practice, efrfmeter-efficient fine-tuning (PEFT) meth- ods are combined with DreamBooth to enable fast and memory- adaptation of diffusion models. In particular, low-rank adaptation (LoRA) [14] is a popular choice, where it fine-tunes the residuals AW E R x of weight matrix W E Raxm with low-rank decomposition AW = AB for A E Rex and B E Rrxm with rank & min(n,m) Alternatively, Textual Inversion (TI) [9] introduces a new token and corresponding textual embedding v to represent the concept. Then, Tl optimizes textual embedding by solving v arg min, LDm (v; D.et).
3

---

Algorithm/ Regular fine-tuning
Algorithm Fine-tuning with DCO loss
Require: Dataset Dret fine-tuning model 60. Require: Dataset Dist fine-tuning model cg. pre- trained model Eg, temperature > leam- ing rate 1) > 0
learning rate 1j 0
1: while not converged do
2: Sample (x, c) ~ Dref
1: while not converged do
3: Sample E ~ I)
2: Sample (x, c) Dee
4: Sample 1-)
Sample e ~~ (0,,1)
5:
0xx + oGE
4: Sample ( (0)
6: LDM(0) - ||elzi c, t) El
2, 02 orE
7: Update 9 - V 9 LDM
6:
((0) [Ea[ c) - =
8:
end while
A()) E c, t) 2 (no gradient)
(Dco log 0 (,(() (()))
Update A - 0 - 17 VoLccol))
10: end while
4 Method
In this section, we introduce our method for T2I personalization. For presentation clarity, focus on our demonstration with the subject customization [10], but the method can be applied to a broader context of personalization, such as style [11]. Throughout the paper, let denote E,, Py and e Pa the noise-prediction model and density for each fine-tuning and pretrained diffusion model, respectively.
4.1 Direct Consistency Optimization
Problem setup. While fine-tuning based T2I personalization methods have shown great success [10, 23, 16], it is shown that the generation quality heavily depends on the model's fitness. For example, the model suffers from image-text alignment when the model overfits to few images used for fine-tuning, making it difficult to generate images with varying attributes around the subject. On the other hand, the model cannot generate consistent subject images when the model underfits. To find the right balance between overfit vs. underfit, certain heuristics such as early stopping or training with an additional prior dataset (i.e., prior-preservation loss) have been popularly used.
Our approach. We devise an efficient training objective that seeks for minimal improvement on the ELBO of fine-tuning model pe over the reference model pg- Specifically, we seek for 0 satisfying (x)). while two quantities are still not far from each other. Since
DKL (pp (x)) < DKL (q(x) ()p))))))) computing the likelihood intractable for diffusion models, we consider the variational bound over sequence of latents 1 following [47, 48]. To this end, let us define the deviation as follows
(ppo psi x, c) DxL (g( zo:1 (x) Po Zo:1 /c)) DKL (q(z)]/x) po (Zz:1//)):
(4)
where the KI. divergence takes expectation over e (0, I) that generates 0x,X + for e (0, 1). To enforce the positiveness A(ps, Po). , we propose following log-sigmoid loss function:
La (0; x, c) log (3 Apo.Pe;X,
where (u) (1 exp(-u)) and B > 0 is temperature that controls the deviation.
Direct Consistency Optimization. Now we show that Eq. (5) is equivalent to optimizing reward function derived from the solution of constrained policy optimization problem [19, 20]. Suppose f(x, c; Drer) ;= fx, c) is function that measures the consistency between image x and reference dataset Dier given the prompt We opt to find 0 that maximizes the consistency of generated sample x Po [x] (x)c), while penalizing the deviation from the pretrained model pş. This can be written by
max (xxxx-uu(x [) c)] - PP (-p(- (=()))))
where 3 >> 0 is temperature that controls the deviation from pretrained model. Since the likelihood of diffusion model is intractable, we consider a consistency function on the latent variables Zo.1. and solve following relaxation of Eq. (6):
max -me (f (Zu.11 c) - 3DDK (po( Po 0

---

Note that the optimal solution to Eq. (7) satisfies
pe (z (c) oc ps(zon1() exp (f(zo.1 c)/3).
(8)
Then, we show that the deviation A po) in Eq. (4) is equivalent to the expected consistency over ((zo:[xx) up to constant, i.e., the following holds (see Appendix A for derivation):
By (x) (zo.1, c)] = | [Aepe. Poi X, c) + C.
for some constant C' that does not depend on x. Eq. (9) shows that our training objective can be regarded as optimizing the consistency function with respect to the reference dataset Drer- Remark that one can define explicit consistency function (reward) and fine-tune the diffusion model with such function, e.g., by using reward-weighter regression (RWR) [49, 29] or using reinforcement learning (RL) [33, 34j. However, those consistency function for personalized images hard to obtain as we only have few samples, and RL fine-tuning of diffusion models expensive. We bypass these issues by directly fine-tuning models with an implicit reward function, similar to those by Rafailov et al. [38] and Wallace al. [37]. Thus, name approach Direct Consistency Optimization (DCO), as we directly optimize the consistency function fine-tuning diffusion models.
4.2 Implementation and Analysis
Implementation. Note that computing the term for Eq. (4) is expensive as it requires computation of likelihood over all E [o. 1]. To this end, we derive an upper bound of Eq. (5) that allows efficient implementation using e-prediction loss. In Appendix A, we show that the deviation Eq. (4) can be expressed by the difference between noise- prede-ppedintion errors of fine-tuning and pretrained model:
(ppo Po) - (( (1,, c, () - E E (2;; c, ()
(10)
Then by plugging Eq. (10) into Eq. (5) and taking out the expectation outside of logarithm function using Jensen's 's inequality, we have final DCO loss (x, c) Drer defined as follows:
LDco (f;; x, c) Fr,e log a Be(1 c, - ==12 [z c, c) <[
(11)
where t ((, 1), (), and Br = - Bx 3. We use LDco in experiments, which is as practical and easy to implement as regular training objective (ie., LDM). Algorithm and Algorithm present DCO fine-tuning and regular fine-tuning side-by-side, with differences colored in red.
Gradient analysis of DCO loss. We provide gradient analysis of DCO loss fine-tuning diffusion models to better understand its effect. Given a data pair (x, c) ~ DDer, ~ NoO,, I) and e 4(00, 1), the gradient of DCO loss with respect to parameter A given as follows:
V VoLoco(O) ac (1 (d:)) Vol (z); c,t) =ll
(12)
||o where d, = -3 (leo) - -133-11e c, t) - = with stop-gradin aop-gradieett. Remark that Eq. (12) identical to the gradient diffusion loss (i.e., LDM), except that is scaled by 1 (de), which measures the incorrect reward modeling. In other words, DCO loss implicitly performs an adaptive loss weighting by computing deviation from the pretrained model; if the deviation between fine-tuning and pretrained models are large, it abstains update.
Comparison to prior preservation loss. While the prior preservation loss [10] in Eq. (3) has similar motivation to DCO loss, they work in very different ways. To elaborate, DCO directly regularizes the KL divergence with respect the samples in Diss, while prior preservation loss does not impose regularization for the reference data. While prior preservation loss may enhance the composition ability, fine-tuning on auxiliary samples from Derior often causes undesirable model shift, losing consistency to the pretrained model (e.g., Fig. 3). On the other hand, DCO is free from such an issue, as it does not require any auxiliary samples besides the reference dataset.
4.3 Consistency Guidance Sampling
During inference, it is common practice to use classifier-free guidance to control text conditioning. Similarly, to gain control over the consistency, we propose consistency guidance sampling, which is
Since de is decreasing function of X < 0. Thus, we 8 - enusre 3 0.
5

---

Compact caption
Comprehensive caption
photo dog sitting on couch covered with grey blanket in living room, indoor lighting style"
"A photo of [V] dog"
Reference
dog with lake the backgrround Figure 2: Comprehensive caption. We provide examples of compact caption [10] and our com- prehensive caption (top row) and generated images from each method (bottom row). The model fine-tuned with compact caption (left) generates images of a dog sitting on a couch though asked to be on the lake. Our comprehensive caption (right) effectively disentangles unwanted attributes, generating images that follow text prompts more faithfully.
[V] dog with lake in the background
[V] dog in cartoon line drawing style
dog in cartoon line drawing style
an additional guidance from implicitly learned consistency function. Specifically, decompose the guidance term with respect to the text and consistency as follows:
é(Z:; c, t) = E (Zz, t) + Whet (Ea(Zei t) ( t)) + Weon (Eu(Zi;c,)) - (zi; c)t.. (13)
each timestep Note that for fixed whex,s the fidelity to the reference images increases if using higher Wcon. while this comes at the cost of losing prompt fidelity. On the other hand, one can improve prompt fidelity by using small Weon- We show that varying weon controls the tradeoff between fidelity to the reference and image-text alignment (e.g., see Fig. 5). Note that a similar sampling method was introduced in [11], but for transformer-based T2I models [7]. While they do not use regularized fine-tuning objectives, the consistency guidance scheme is still valid. Thus, the consistency guidance could be implemented in any fine-tur tne-tuned model, while we show that is more effective when combined with DCO fine-tuned model (e.g., Fig. 5).
4.4 Prompt Construction for Reference Images
An important (yet often overlooked) part of the T2I personalization process is the prompt construction of reference images. Recall that Ruiz et al. [10] have proposed the use of a compact prompt in the form of "a photo of [V] [class]" with rare token identifier [V]. However, we find that the use of compact caption is prone to learning distractors, such as background or a style, as part of the fine-tuned model, illustrated in Fig.
Comprehensive caption. Instead, we propose to provide a comprehensive and visually grounded caption that not only describes the subject but also details other visual attributes, backgrounds, and styles of reference images. In Fig. 2, we show an example of a comprehensive caption and compare the synthesized results that compact caption. We find that providing detailed descriptions of the undesirable attributes, e.g., background, or style, helps anchor desirable attributes in reference images to corresponding texts, making it easier to separate between them. This method not only holds for subject customization, but also for style customization; we provide comprehensive descriptions the subject so that the model distinguishes style from the subject. Note that the use of comprehensive caption has been considered in practice," but has not been investigated from the lens of model shift and concept disentanglement, In our experiments, we vision-language models such as GPT-4 [50] or LLaVA [51] (e.g., see Fig. 11).
Experiments
We use Stable Diffusson XL [6] for experiments. We conduct experiments subject (Sec. 5.1), style (Sec. 5.2) personalization, and their combination (Sec. 5.3). Ablative studies are in Sec. 5.4.
5.1 Subject Personalization
Experimental setup. We conduct experiments on DreamBooth dataset [10], containing 30 subjects, with 4-6 images per subject. The examples of images and captions in Fig. 11 in Appendix D.I.
See this blog post example.

---

Reference
DB+p.p.
DB
DOO (ours)
Reference
DB+p.p.
DB
DCO (ours)
A dog weering ball gown, dencing in grand bairroom
An alarm clock made out of leather
A backpack cartzon line drawing style
wolf plush exploring future in isomeicc ilustration style
A cal langied with yam in docdle art style Figure 3: Custom subject generation. We show selected generations from DreamBooth (DB), DB with prior preservation (DB+p.p.), and (DCO) of custom subjects with varying attributes and styles guided by text prompts. While DB captures subjects well, does not follow text prompt well. DB+p.p. shows better textual alignment, but falls short in subject fidelity. Ours show the best in both image-text alignment and subject fidelity. Best viewed color, zoomed in on monitor.
A dog gracefully leaping origamii style
Reference
DB
DCO (ours)
A (banana, robot, cow) in modern 3D rendering style
Apenguin, coffee maker, piano) in kid crayon drawing style
Figure 4: Custom style generation. We show selected generations from DreamBooth (DB) and ours (DCO) of custom styles with varying subjects. DB is prone capturing undesirable attributes, resulting in generation of mixed concepts (e.g., the girl's outfits in the first row, the dog in the second row), whereas DCO mitigates such concept mixing. Best viewed in color, zoomed in monitor.
For baselines, we consider the training with a regular diffusion loss, ie.. DreamBooth (DB), and the with prior preservation loss (DB+p-p.). Note that we additionally optimized textual embeddings, ie., textual inversion [9], which enhances subject fidelity for all baselines, though we omit for clear context. For all experiments, fine-tune LoRA of rank 32 and textual embeddings using Adam [52] optimizer with learning rates 5e-5 and 5e-4, respectively. We use constant 3,=1000 for DCO loss.
Following Sec. 4.4, provide comprehensive and visually grounded caption that not only describes the subject but also details other visual attributes, backgrounds, and styles of reference images, for fine-tuning. Note that this is used for both baselines and ours. We observe that comprehensive captioning improves all the baselines, thus we omit the indications of its usage.
Qualitative results. Fig. 3 shows the qualitative comparison of our approach with DreamBooth (DB) and DreamBooth using prior preservation loss (DB+p.p.). We observe that our approach generates images of various visual attributes, e.g., outfits and backgrounds, or changing the material, well as into various styles, e.g., origami style or doodle art style. While DB changes the background, it lacks recontextualization in different outfits styles, especially due to the overfitting to the photographic style. DB+p-p. better than DB in prompt fidelity, but it often fails preserve subject identity More qualitative comparisons are demonstrated in Fig. 17.

---

0.65
0.71
0.60
Image Simliaity
Image Similarity
0.70
0.55
0.69
0.50
0.45
(eurs)
0.68
DreamBooth
(ours)
0.40
DreamBooth p.p.
0.67
DreamBooth
0.60
0.65
0.70
0.75 0.80 0.85 0.90
0.45
0.55
0.60
0.65
0.70
Image-Text Similarity
Image Tmage-Text Similarity
(a) Subject personalization
(b) Style personalization
Figure Quantitative results. We plot Pareto curve between subject / style fidelity (image similarity) and prompt fidelity (image-text similarity) on (a) subject personalization and (b) style personalization tasks. Scores of each point are measured with consistency guidance sampling (dots and lines) of
Weon 2.0, 3.0, 4.0, 5.0, and conventional classifier-free guidance sampling (diamond). See Sec. 5.1 and Sec. 5.2 for experimental details, and Appendix B. for full comparison.
Quantitative results. For quantitative evaluation, design two types of text prompts: subject customization, where we modify the attributes of the subject or its background. and subject stylization, where we change the visual style of subject image. For each subject and type, we generate images from 50 text prompts with 2 random seeds. For evaluation metries, we report the image similarity score using DINOv2 [53] and the image-text similarity score using SigLIP [54]. We also show the results using DINO [55] and CLIP [45] in Fig. 9, showing similar trends. See Appendix D for details.
Noting that this is multi-objective problem (i.e., maximizing image similarity and image-text similarity), we report the Pareto curve consisting of scores with varying consistency guidance scale values to show the tradeoff between two scores of each model, instead of reporting scores one operating point. If two curves overlap, two methods would likely perform similarly and the difference up change in the consistency guidance scale sampling.
Averaged results are in Fig. 5a, and the results for each subject customization and stylization are in Fig. 8a and Fig. 8b, respectively. Compared to DrcamBooth (DBared), ours (DCO;blue) positions on the upper-right frontier in both image-text similarity and image similarity, demonstrating its superiority. Compared DreamBooth with prior preservation loss (DB+p-p-4 green), ours (blue) results in significantly improved image similarity, while being comparable in image-text similarity. Interestingly, it (green) does push the frontier to the upper-right compared the ones without (red), but it shifts the operating point to the lower-right while lying on the seemingly similar Pareto frontier. This suggests that the of prior preservation loss improves prompt fidelity at the cost of losing the subject consistency. In Appendix B. we further compare with various design choices for DreamBooth, e.g., early stopping or lowering Aprior for prior preservation loss.
5.2 Style Personalization
Experimental setup. We experiment on style images from StyleDrop dataset [11]. The examples of style images and captions are in Fig. 12 in Appendix D.1. We fine-tune LoRA of rank 64 using Adam optimizer with a learning rate of 5e-5 and do not train textual embedding. In addition, we add an offset noise [56] of 0.1 during training, which empirically helps learning the solid background color of style images. We use 3==1000 for DCO loss, and compare with DreamBooth (DB).
Qualitative results. Fig. 4 shows qualitative comparisons between DreamBooth (DB) and ours (DCO). As seen in [21], DB captures the style reference image, yet it suffers from overfitting to the reference image, e.g., the attributes of the subject in style reference appear in generated images. On the other hand, DCO generates images with consistent style without being entangled with contents in the reference images. We provide additional comparisons in Fig. 18.
Quantitative results. We choose style images and generate stylized images using 190 text prompts excerpted from Parti prompts [5], following [11]. We generate 2 images per evaluation prompt, resulting in 380 images in total for each style. For evaluation metries, we report the image similarity against the style reference image and image-text similarity scores using SigLIP [54].

---

wafercolor painting style
dog
alarm clock
fringed boot
Flat cartoon style
dog
feddy clock
DB Merge
DB Merge
ZipLoRA
DB ZipLoRA
DCO Merge (aurs)
DCO
(ours)
Figure 6: My subject in my style generation. We show generated images by merging subject and style LoRAs, each trained independently with DB (DB Merge) or DCO (DCO Merge). We also show results of ZipLoRA [21] using DB models (DB ZipLoRA). DB Merge struggles generate high quality images. While DB Zip improves the quality, it less preserves fidelity. DCO Merge produces consistent images in both subject and style. Best viewed in color, zoomed in monitor.
Fig. 5b shows results. While two models operate in somewhat disjoint regimes, see that the curve of DCO (ours) is placed on the right of that of DB, showing improved image-text similarity. Nevertheless, noted in [11], the image similarity for style personalization is particularly noisy, as the score is not only guided by the style but also by the unexpected appearance of the subject in the style reference image (e.g., in Fig. 4) when the model overfits.
5.3 My Subject in My Style
Experimental setup. Following [11], we combine customized subject and style models to generate images of my subject my style. Specifically, given two LoRAs AW, and AW, for subject and style, respectively, we use an arithmetic merge (Merge) [21], i.e., AW Ti AW, + T with coefficients TΙ 72 1. We use subject and style LoRAs from Sec. 5.1 and Sec. 5.2, respectively, for both baseline (DB) and our method (DCO). We also compare with ZipLoRA [21], which finds optimal coefficients тj and T2 for each layer by jointly preserving the subject and style LoRAs. For ZipLoRA, we use DB fine-tuned subject and style models and follow the experimental setup in [21].
Qualitative results. In Fig. 6, provide qualitative comparisons between approach (DCO Merge), and baselines (DB Merge and DB Zi)LoRA). As noticed in [21], DB Merge struggles to generate high-qualit images when composing subject and style customized models. While DB ZipLoRA improves the image quality with fewer artifacts, it often loses subject or style consistency, Even using simple arithmetic merge, DCO Merge (ours) generates images with high subject and style consistency. In addition, Fig. and Fig. 20 in the appendix shows that DCO Merge succesfully generates my subjects in my styles under various contexts, guided by text prompts.
Quantitative results. We use 30 subjects from DreamBooth [10] dataset and 10 style images from StyleDrop dataset [11] from Sec. 5.1 and Sec. 5.2, respectively. For each subject and style pair, we generate images of "A [subject] in [style]" and of various text prompts that change attributes, backgrounds, actions (e.g., in Fig. 20). We compute subject similarity scores (Subject) using DINO v2 [53], style similarity (Style), and image-text similarity (Text) scores using SigLIP [54].
Tab. reports results. DCO Merge significantly outperforms DB Merge and DB ZipLoRA in subject similarity (0.462 vs. 0.386, 0.406) and image-text similarity (0.773 vs. 0.430, 0. 0.729), while retaining competitive style similarity (0.651 vs. 0.672, 0.662). This aligns with our observation in Fig. 6.

---

Table Quantitative results of my subject my style generation. We report subject, style, and image-text similarity scores of DB Merge, DB ZipLoRA, and DCO Merge (ours).
Merge
ZipLoRA
Merge
Subject
0.386
0,406
0.462
Style
0.672
0.662
0.651
Text 0.430
0.729
0.773
DB
2.5
DCO ( 500)
DCO 1000)
Image Similaity
2.0
DCO 2000)
0.60
1.5
0.55
1.0
DB
DCO (6= 500)
0.5
DCO (8 1000)
0.50
DCO 2000)
Timestep
(a) Ablation on 3
(b) Noise distance
Figure 7: Ablation studies. We conduct ablation studies on (a) regularization hyperparameter 3, and
(b) noise distance between pretrained model and fine-tuned model.
5.4 Ablation Study
Regularization parameter. We study the effect the regularization parameter 3e on the subject personalization task. We use 10 subjects from the DreamBooth dataset and conduct experiments with constant 3 = {500, 1000, 20003. As in Fig 7a, all DCO models form better Pareto frontier than the DB from Sec. 5.1. As 39 becomes larger the curve tends to move lower right, implying the loss in subject identity for better image-text alignment. This expected as 3 controls the deviation between fine-tuned and pretrained models (e.g.. Eq. (6)). We find 1000 works well overall, though the optimal / might vary across the reference dataset.
Error analysis. One of our insights is that DCO mitigates the shift in the model's generation distribution after fine-tuning. We verify this by computing the noise distance between pretrained and fine-tuned models on reference images and prompts, i.e., |Ee( c, () e c,(2,; 223 c, I)l at each timestep E [0, 1]. We simulate 100 random noises at each timestep and report the average value in Fig. 7b. We see a clear decrease in noise deviation from the pretrained model with DCO fine-tuning over DB. Moreover, as increases, the noise deviation gets further reduced as expected.
s-shot personalization. We demonstrate the capability of our method in 1-shot personalization. We refer Appendix B.3 for experimental details and qualitative results are in Fig. 15 and Fig. 16.
Conclusion
We introduce Direct Consistency Optimization (DCO), a novel training objective for robust low-shot fine-tuning of the T2I diffusion model. DCO learn new concepts by controlling the deviation from pretrained model, thus retaining capabilities of pretrained model. We show that DCO enhances the image-text alignment and sample quality of personalized T2I synthesis compared to regular diffusion fine-tuning. And together with consistency guidance sampling, DCO results in the most superior Pareto frontier in terms of image-text similarity and image similarity. Moreover, DCO induces easier composition of independently fine-tuned subject and style T2I models. Also, merging DCO models outperforms post post-processing methods on regular fine-tuned models, e.g., ZipLoRA [21]. We provide limitations and broader impact statements in Appendix E and Appendix F, respectively.
10

---

Acknowledgement
We express our gratitude to Nataniel Ruiz and Viraj Shah for their help ZipLoRA implementation and Meera Hahn for their feedback on the presentation of our paper. We also appreciate Jinyeop Kim and Younghyun Kim for their feedback and support on our manuscript and project page.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) -w--R------911111
Artificial
Intelligence Graduate School Program(KAISI) No. RS- 202220000000977 Global AI Frontier Lab; No.RS-2021 1.RSS-2021-I1211 1020--11220066 Artificial Intelligence Innovation Hub; RS- RS-2022-11220955 Self-directed Al Agents with Problem-solving Capability).
References
[1] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. 2021.
[2] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
[3] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220..06125 2022.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[5] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[6] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. International Conference on Learning Representations, 2024.
[7] Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint 1xXix:2301.00000 2023.
[8] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science, 2023.
[9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or, An image is worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations, 2023.
[10) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. IEEE Conference Computer Vision and Pattern Recognition, 2023.
[11] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, al. Styledrop: Text-to-image generation in any style. In Advances in Neural Information Processing Systems, 2023.
[12] Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin CK Chan, and Ziwei Liu. Reversion: Diffusion- based relation inversion from images. arXiv preprint arXin:2303.133495 2023.
[13] Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, et al. Reaffill: Reference-driven generation for authentic image completion. In ACM SIGGRAPH conference proceedings, 2024.
11

---

[14] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Alu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 2021.
[15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nip. In International Conference on Machine Learning, 2019.
[16] Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2023.
[17] Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, and Ariel Shamir. Palp: Prompt aligned personalization of text-to-image models. arXiv preprint arXiv:2401.06105 2024.
[18) Xingzhe He, Zhiwen Cao, Nicholas Kolkin, Lantao Yu, Helge Rhodin, and Ratheesh Kalarot. A data perspective on enhanced identity preservation for diffusion personalization. arXiv preprint arXiv:2311.04311 2023.
[19] Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Associa- tion for the Advancement of Artificial Intelligence, 2010.
[20] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, arXiv preprint
2019.
[21] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311. 13600,, 2023.
[22] Ligong Han, Yinxiao Li, Han Zhang. Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In IEEE International Conference on Computer Vision, 2023.
[23] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of text-to-image diffusion. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[24] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302. 2023.
[25] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Designing an encoder for fast personalization of text-to-image models. arXiv preprint arXivv2302. 12228, 2023.
[26] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to text-to-image generation. arXiv preprint arXiv:2303.09952. 2023.
[27] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-in text-to-image personalization. In ACM SIGGRAPH conference proceedings, 2023.
[28] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personaliza- tion of toxtato- models. IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[29] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. In International Conference on Machine Learning, 2023.
[30] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Adyances in Neural Information Processing Systems, 2023.
12

---

[31] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2023.
[32] Christoph Schuhmann and Romain Beaumont. Laion-aesthetics, 2022.
[33] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Picter Abbeel, Mohammad Ghavamzadch, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. In Advances in Neural Information Processing Systems, 2023.
[34] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference Learning Reprexentations, 2024.
[35] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint MrXiv:2304.06777 2023.
[36] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations. 2024.
[37] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In IEEE Conference on Computer Vision and Puttern Recognition, 2024.
[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances in Neural Information Processing Systems, 2023.
[39] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[40] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[42] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances Neural Information Processing Systems, 2022.
[43] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 2021.
[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[46] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 12X.:22007.12598. 2022.
[47] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, 2021.
13

---

[48] Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. In Advances in Neural Information Processing Systems, 2023.
[49] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In International Conference on Machine Learning, 2007.
[50] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint rrXiv:2303.08774. 2023.
[51] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023,
[52] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[53] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
[54] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE International Conference on Computer Vision, 2023.
[55] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE International Conference on Computer Vision, 2021.
[56] Nicholas Guttenberg. Diffusion with offset noise. link, 2023.
[57] Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein. Orthogonal adaptation for modular customization of diffusion models. In IEEE Conference Computer Vision and Pattern Recognition, 2024.
[58] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image gen- eration via shared attention. In IEEE Conference on Computer Vision and Patlern Recognition, 2024.
[59] Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free free consistent eext-to-image generation. ACM SIGGRAPH conference proceedings, 2024.
[60] Yuchao Gu, Xintao Wang. Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. In Advances in Neural Information Processing Systems, 2023.
[61] Omri Avrahami, Amir Hertz, Yael Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. The chosen one: Consistent characters in text-to-image diffusion models. In ACM SIGGRAPH conference proceedings, 2024.
14

---

Appendix
A Derivation
Derivation of Eq. (9). From Eq. (8), we have
Po (20. (c)
f(Z0:1, c) = 3 log
+ B3 log Z
Pel 20:1 cc)
Pe 20:1 (c)
Blogg Po(z0:1 (
=Blog
+ B log Z,
g Zo (x)
where a normalization constant that does not depends on x. Therefore, we have
Bg(son (zo.1 c) -Blog Z] Egtzan 3 Blog po(zo:1lc) 3 3log ((Z0a1 (x)
Pa Po (c)
1 20 xx
= ((DKL Po (0001)/))) - DKL Zo:1 (x) po (o (c))
3appo, Paj P))
which proves claim.
Derivation of Eq. (11), Let us consider subseries 21:1 of 20:1 for E (0, 1). Then, we define the deviation (p(pe, (p) for Zr:1 follows:
(e Ps) DKL (g(z:1) (x) po(z(alc)) DKL (9(z): xx) pe(zci
(14)
It is known that A.1 in [48]):
express the KI. divergence with e-eredcction follows (e.g., see Appendix
DKL (9(z)1) (x) po(Zealc)) X,E X.EE.. =l12]
(15)
Thus, the following holds:
d
1
di Ac Po: x, =
Bee w...ew.oo.lllaaaaa..... - -Ela(zzi - =l12].
(16)
Now it is straightforward to see that
0
Appo.po
Ap(ps Po)dt
Xr Ee-N (es - -el -lee/zz, -
E|2]dt
2
E, 10 (ileooo - Eli leolze c, -
which proves Eq. (9). Then since the softplus function is convex, Jensen's inequality to show that
La c) -log(BA(po..)) P+))
B8 (I [ (2) <E1² 2 (26 c, -
E log a
E.-H(01).c-N(0... log of ( ( c, -el22 (zr c, - -
(bco((;x,c))
where Br = 3X.
Remark that consider weighted deviation (w P&) that matches the general weighted e-prediction loss (i.e., Eq. (1)), by using monotonic weighting function wy as follows:
0 d
A (po. po)
A Ape.
dt
E Fu..uo..)ee-woonffffu (leoo/ <ll2 (lc <l2)].
2
Note that for DDPM [40], i.e., conventional e-prediction loss, considers - -/
15

---

0.65
0.65
0.60
0.60
Image Similarity
Image Similarity
0.55
0.55
0.45
D0
D0-+p
0.40
D8+p. (0.5)
pretrained
0.35
0.75
0.80
0.85
0.90
0.95
0.5
0.7
0.8
0.9
Image-Text Similarity
Image-Text Similarity
(a) Subject Customization
(b) Subject Stylization
Figure 8: Quantitative results. We plot image similarity and image-text similarity for each (a) subject customization and (b) subject stylization experiment. We use SigLIP [54] score for image-text similarity, and DINOv2 [53] score for image similarity. We plot the results of consistency guidance sampling (dots and solid lines), and conventional sampling (diamond). The reported reward guidance scales are weon (2.0, 3.0, 5.0.
B
Additional Experiments
B.1
Full comparison
Here we provide full ablation studies that we have conducted our experiments. For all 30 subjects in DreamBooth dataset, we follow the same experimental setup in Sec. 5.1. We report the image similarity score using DINOv2 [53], and image-text similarity score using SigLIP [54].
Ablation early stopping. Since low-shot fine-tuning methods suffer from overfitting, it is common practice to early stop the training. In Fig. 8b and Fig. 8b, we plot the results of DreamBooth with comprehensive caption with half of training steps (DB+ES). When compared to DB, it improves the image-text similarity (0.842 vs. 0.754 for customization, 0.645 vs. 0.468 for stylization), while the image similarity significantly drops (0.575 vs. 0.754 for customization, 0.630 0.638 for stylization). Notably, we remark that the frontier curve of DB+ES resides at the frontier of DB. Thus, early stopping does improve the frontier.
Ablation on prior preservation loss weight Aprior- In Sec. 5.1, we show that using prior preserva- tion loss often leads to loss of subject consistency. To further verify the effect of prior preservation loss, we vary the coefficient Aprior to be 0.5 (DB+p.p. (0.5)). As shown Fig. 8a and Fig. 8b, when compare DB+p.p., using smaller Aprio improves image similarity (0.547 customization, 0.441 vs. 0.424 for stylization), while decreases the image-text similarity (0.850 vs. 0.864 for customization, 0.839 0.851 for stylization). However, changing Aprior does not improve the frontier curve when using consistency guidance.
0.532 for
Prior preservation loss pretrained model. In Fig. 8a and Fig. 8b, notice that DB with prior preservation loss (DB+p.p.) shows higher image-tex text similarity than pretrained model. This is in partly due to that the model is fine-tuned with class-specifi prior dataset, which improves the prompt fidelity among the class. However, this does not necessarily improves the subject fidelity, and indicates the large model shift with respect pretrained model.
Evaluation metrics. Following DreamBooth [10], in Fig. 9, we provide quantitative results using (a) CLIP [45] for image-text similarity (CLIP-T) and DINO [55] for image similarity (DINO), (b) CLIP for both image-text similarity and image similarity. We observe consistent trends to that which used with SigLIP and DINOv2.
User Evaluation. We conduct a user study over the results of our method (DCO), DreamBooth (DB), and DB with prior preservation loss (DB+p.p.) subject personalization task. As in Section 5.1,
16

---

0.84
0.60
0.58
0.56
0.80
DIN
0.54
CUP4
0.50
DCO (ours)
DCO (ours)
0.46
DreamBcoth
DreamBcoth
DreamBcoth-ppp
0.74
DreamBcoth+pp
0.44
CLIP-T
CLIPT
(a) CLIP-T vs DINO
(b) CLIP-T vs CLIP-1
Figure 9: Quantitative results. We plot image similarity and image-text similarity scores using (a) CLIP-T and DINO and (b) CLIP-T and CLIP-I. We plot the results reward guidance sampling (dots and solid lines), and conventional sampling (diamond). The reported consistency guidance scales are Wicm E (2.0, 3.0, 5.0).
Table User evaluation. We asked users to select better one given two images generated by DCO vs. baselines (DB and DB+p.p.) for each question: 1) subject fidelity. 2) image-text alignment, and 3) image quality. We report the percentage of judgements in favor of DCO over baselines.
Win rate
vs.
vs. DB+p.p.
Subject fidelity
55.1 %
81.9
Prompt fidelity
72.7
58.0
Image quality
70.6 %
63.0
trained three models per subject with identical experimental setup (i.e., all models trained with comprehensive caption and textual inversion), and generate images with the same random seeds. Then construct two binary comparison tasks (90 comparisons) rank between DCO and each baseline method. For each pair, 12 participants were asked to choose the preferred image on three criteria with following questions:
Subject fidelity: Which image most accurately reproduces the identity (e.g., item type and details) of the reference item?
Prompt fidelity: Which image most closely aligns with the given prompt?
Image quality: Which image exhibits the highest quality (e.g., overall visual appeal and clarity)?
Overall, we collect 1,080 answers per query for each comparison pair, resulting a total 2, 160 responses. The results are summarized in Table 2, where for each method, we report the percentage of judgements in our favor. Compared to DB, DCO shows better prompt fidelity while maintaining subject fidelity. Most users favored DCO for subject fidelity compared DB+p.p., implying that DCO enables generating images with high subject and prompt fidelity.
17

---

Layerwise cosine similarity subject style
D6
DCO
0.00
0.04
0.22
0.00
Figure 10: Comparison on the alignment of DroahBooth and DCO fine-tuned subject and style LoRAs. We compute the average cosine similarity of column layers between subject and style LoRAs fine-tuned with each DreamBooth (DB) [10] and our method (DCO). The x-axis denotes the component of each U-Net layer. The cosine similarity measures the alignment between two LoRAs, and high cosine similarity values are considered as the interference between them. Interestingly, we find that there is no obvious difference in the cosine similarity values between models trained with DCO and DB methods, while DCO fine-tuned models can be successfully combined with arithmetic merge to generate images of my subject in my style (e.g., see Fig. 6 and Fig. 19). This may be in contrast with the findings of recent works [21, 57] and suggests further investigation on method to measure the compatibility between LoRA models.
B.2 Effect of Consistency Guidance Scale.
We have shown that the user can control the subject fidelity and textual alignment by changing the consistency guidance scale. Yet, we remark that the optimal consistency guidance scale vary among the reference dataset, and even for the input prompt that the user give during inference. As shown in Fig. 14, the optimal guidance scale should be large (e.g., 5.0) for the first row, while it should be low (c.g., 2.0) for the third row. Also, the effect of consistency guidance scale might be subtle as in second row. In practice, choosing the best consistency guidance scale is up to the user's preference.
B.3 1-shot personalization
Here we provide some qualitative examples that shows the capability of our method in learning subjects with single reference image. Specifically, we show the capability personalization with synthetic images, generated by different T2I models such pretrained SDXL [6] and DALLE-3 [8]. We follow the same setup as in Sec. 5.1 except that we fine-tune for 1000 steps. Remark that for while have access to the prompts that used for generation, we did not this prompt for fine-tuning our model; the generated image might have more details (e.g., backgrounds or attributes), or it may fail to capture all the prompts. Thus, caption the images following Sec. 4.4.
Fig. 15 shows the qualitative results of DCO fine-tuning on images generated by SDXL. Remark that our method can synthesize images into various actions and styles, while preserving the subject consistency. Notably, it is possible to convert the photographs to other styles (e.g., photo of a man to 2D animation style), and vice versa (e.g., 3D animation style of pig into photography). Also, as shown in Fig. 16, our method is able to generate various actions, backgrounds, or styles.
C Extended Related Work
Training-free consistent image set generation. Several works have demonstrated the capability of consistent image set generation without fine-tuning T2I diffusion models [58, 59]. While these methods do not require fine-tuning and hence may be conceived more time-saving, they often
18

---

take longer time at generation. On the other hand, fine-tuning is a one-time cost and can be used for generation without additional cost. Also, training-free -fee methods have difficulty in putting the same subject in different styles (as mentioned as limitation in [59]), while our approach is possible. Moreover, our method is able to combine style personalized model and subject personalized model. Lastly, our approach do not require any segmentation mask for subject personalization.
Multi-concept personalization. Given multiple fine-tuned T2I diffusion models (often using LoRAs), it is of great interest to combine them generate scene consists of multiple personalized subjects [60, 57], or generating custom subject in custom style [11, 21]. Those approaches often require post-optimization, e.g., orthogonal adaptation of LoRA layers [57] or optimization of merger coefficients for composition of subject and style LoRAs [21]. Those methods hypothesize that the interference between subject and style LoRAs (which is measured by the average cosine similarity between LoRA layers), and aim to minimize the interference. Our method seeks for a better training of LoRA for T2I diffusion models and thus complementary existing works. On the other hand, as shown in Fig. 10, we find that the cosine similarity values of DCO fine-tuned models are not necessarily smaller than those of DreamBooth fine-tuned models, while we do not observe significant interference during generation with arithmetically merged LoRAs trained with our DCO loss. This observation may suggest that we need another metric other than the cosine similarity based interference measure to evaluate the compatibility of LoRAs, which we leave as a future work.
D Experimental detail
Learning textual embeddings. The rare token identifier [V], such as "sks", conveys undesirable semantics." 5 Thus, we opt to remove the rare token identifier and use the natural language captions by default. In addition, we learn textual embeddings [9] to add more flexibility in subject personalization without changing the semantics of pretrained model. Given a word or phrase (i.e., [class]] of interest, we insert new tokens and initialize them with textual embeddings of original ones. Then, newly inserted textual embeddings are optimized with diffusion models.
D.1 Dataset
We use DreamBooth dataset [10] for subject personalization which contains 30 subjects, including pets and unique objects such as backpack, dogs, plushie, etc. We provide examples of image and comprehensive caption in Fig. 11 where the complete list of comprehensive captions are is available in the source code. A comprehensive caption encompasses not only the subject but also provides detailed information visual attributes, backgrounds, and style. In contrast, compact caption generally incorporated in model personalization [10, 9] focuses solely on the subject itself, as exemplified by "a photo of [class]". To generate comprehensive captions in practical scenarios, initially employ LLaVA [51] to generate a description of the reference image. Subsequently, we filter out unnecessary details such as non-visual attributes and make further modifications. Similarily, we use 10 images from StypeDrop dataset [11] for style personalization, where examples are presented in Fig. 12.
License. The license for DreamBooth dataset can be found in here. Also, the attributes for style images can be found in StyleDrop [11] paper as well as here.
D.2 Evaluation prompt
We construct two types of evaluation prompts; (1) subject customization, and (2) subject stylization. In evaluating subject customization, we provide the textual prompts that alter the attributes of the subject (e.g., "cube-shaped"" or its background (e.g., "on the beach") following DreamBooth [10]. In evaluating subject stylization, provide the textual prompts that stylizes the subject into different styles (e.g., "in watercolor painting style", "in origami style"). For fine- ine-grrined evaluation, we construct difffernnt prompts for each object (e.g.. "clock", "robot toy") and subject (e.g., "cat", "dog". "wolf plushic"). The examples of evaluation prompts for each category and type are in Table 3.
'See second and fourth rows of DB (baseline) in Fig. 17, where dog is surrounded by guns and a monster toy is holding gun. Note that this is aligned with the existing findings community.
19

---

Reference
LLaVA oulput
Comprehensive caption
image features a outs, fufly, appears encyying surlight smilnng tonpue The dap's roomed. element
dag on lodge possialy
"A photo af dog sitting on ledge in fromt of red wall and tree, macro style"
carod-oor for pet bodiground, there some trees, adding natural appears The overall compostion image close-up up of dog. background
igo contest and depth to the scene."
"The image features a brown loddy bear sitting on concreto lodge near a body of water. The toddy bear postiooned the foreground. cocupying significant porton of the soone. The background showcases clyscape with buildings and bests vis in the distance. The combination of the loday bear and the citscape creates a unique and interesting contract between the innocence of the toy and the urbon enstronment.
of and beats burdegrout lighting style"
sitting
chyysaae
imge features the scane, the plant appaars potted
plant it, placed
vase is posiiioneed the center tocus
vase doorway visiale, adds depth scene. Addiionnlly
photo vase with plant
image. background thore is bowl table. located the right vase. The overall compcation image creates sense belenoe and hamoory between the vase, plant, and the doorway in the background.
cup, placed on dining table door the background, natural lighting style"
Figure 11: Examples of comprehensive caption for subject images. We provide examples of LLaVA [51] output and our comprehensive captions for each reference image. With help of LLa we extract the visual attributes, backgrounds, and styles to construct comprehensive caption (e.g.. . the texts marked in red in LLaVA output are used). The class tokens that marked in bold (e.g., dog, sloth plush, vase) are additionally learned with new textual embeddings initialized from the original one.
Reference
Comprehensive caption
Reference
Comprehensive caption
Reference
Comprehensive caption
"A seascape and cliffs watercolor painting style'
"A female figure with exaggerated propartions modern rendering style"
"A woman walking puppy cartoon Illustration style"
"Row flowers in watercolor painting style"
"A person drowning
"A flower melting golden 3D rendering style
Into a amartphone in cartoon line drawing style"
Figure 12: Examples of comprehensive caption for style images. We provide the examples of reference style images and comprehensive captions for style personalization experiment. To disentangle the subject and style in the image, we provide comprehensive description the subject of the image. The texts marked in bold are used to generate image in custom style.
D.3 Additional implementation detail
We PyTorch and Huggingface Diffusers library for our codebase. Each training is performed on a single A100 40GB GPU using batch size of 1. We perform up to 2000 optimization steps. Note that our approach is robust to the length training steps compared the bascline (i.e., DreamBooth), which often requires early stopping to prevent overfitting, We fine-tune LoRA of rank 32 for subject personalization and rank 64 for style personalization. For sampling, we use DDIM [41] scheduler with 50 steps, and use CFG guidance scale of 7.5 throughout experiments.
D.4 Evaluation metric
To measure the image similarity, we use DINOv2 [53] score, which given by the mean cosine similarity between the embeddings of reference images and synthesized images. To measure the image-text similarity, we use SigLIP [54] score, which defined as
1
S
1 exp ( fimg (x) frea(() +
where fing and frest are le normalized embeddings from image and text encoders, and is a bias term that is optimized during pretraining. We opt to use SigLIP score instead of CLIP score [45], as the range of CLIP score depends on the prompt and images, while SigLIP score provides a general score that is bounded on [o, 1]. For style personalization experiment, we measure image similarity using SigLIP image similarity, by computing the embeddings with SigLIP image encoder. While we desire
Phttps://gtt htatpp///gthab...c/a/uggggggggg hwtp:///gggiggaac///aaauaouua
20

---

Table 3: Examples of evaluation prompts used to synthesize images for each object and live subject category. Subject customization are prompts to generate novel views of photo-realistic images and subject stylization aims to alter style of the subject. ()'s filled with the class token of the subject.
Cap
Subject customizatioo
Subject syliization
beach"
[1 style"
phano
"A. 1] wooden sculpture
photo staped (1"
carsonn
style"
Object
photo golden 11"
posel as style"
photo made leathers"
wirefuue 3D style"
"A photo of with photo aummm leaves background
"A. hygge
top
rug"
geometrs: an style"
photo weasing spacesuit, planting fleg moon" ghier, exinggishing photo in weseill vurting gisnt wave octa"
playing
style""
photo
skyscraper"
carved kughi wooden sculpture"
pilcing
taavll ageney logo style"
Subjact
"A plota of [I in Victoias attic, attanding les party
dlguutt ganlar"
canstructed from abstract metal shapes construtiiiis style"
"A ploto II in snowult, stlin down steep mountais"
"A an epc qusst
an atyle"
photo on explover. movigating through iry Anatic linoscape" "A photo of (I in olegant mouquende mook Venetion
[] designed as inthiate machine blurprint style"
(1 Uisttratt ofecational infographic style"
high scores, these metrics are not perfect, e.g., the image similarity can 1.0 if the model overfits, otherwise, the image-text similarity can achieve high score if the model underfits. Thus, instead of reporting the scores from single data point, we provide multiple data points of the same model while varying sampling parameters (e.g., guidance scale values of consistency guidance sampling) and show the trends (e.g., by showing the Pareto frontier).
Reference
DB+p.p.
DCO
dogt 1
A photo of a dog1 ridigg car toy
car toy
dogt and cat front of Eiffel Tower
cat
dog2
A dog1 and dog2 playing logether
Figure 13: Limitations. We show the multi-subject composition results. While DCO (ours) do better than DreamBooth (DB) and DreamBooth with prior preservation loss (DB+p.p.) multi-subject composition, the learned concepts are often mixed during generation (e.g., the dogs are mixed in third row). We found this more frequently happens when learned concepts are similar each other, e.g., dog-dog composition is more challenging than dog-cat composition, and dog-cat composition is more challenging than dog-toy composition.
21

---

E Limitations
Computational efliciency. Since our method leverages inference on pretrained model during training (e.g., Algorithm 2) and inference (when using consistency guidance sampling) (e.g., Eq. (13)), there exists few extra computational burden compared to original DreamBooth fine-tuning or CFG sampling. Specifically, we measured the optimization and inference time iteration. Compared to DreamBooth, our method (DCO) approximately takes x 1.3 longer time fine-tuning. Compared to CFG sampling, consistency guidance sampling requires x 2 longer time in sampling. We believe that our work will motivate future studies on efficient fine-tuning to enhance scalability in practical scenarios.
Multi-subject composition. We have shown that DCO effectively compose learned subject and style (i.e., my subject in my style). When composing multiple subject, found the performance is not stable. Fig 13 showcase some results multi-subject compositional generation. We found that for subjects that semantically distant, e.g., dog-car toy dog-cat composition, method is able to generate multi-subject consistent images, while DreamBooth or DreamBooth+p.pp often fails to. However, when composing semantically similar subjects, e.g., dog-dog composition, found that our method fails as well. Specifically, the concepts are mixed during generation, which results in subject inconsistency. We believe this is partly due the lack of model's capability in disentangling the newly learned concepts that are semantically similar. Thus, we believe using better pretrained model could ameliorate such issues.
F Broader Impact
This paper presents a method that enhances the performance of the personalization of T2I diffusion models. Similarly other works, the technology for personalization of T21 diffusion models comes with benefits and pitfalls the tool could be extremely effective for creative directors to efficiently generate new visual assets of various subjects styles derived from existing private visual assets. Yet, the responsible use of the technology is required for protecting the ownership and copyright of individual assets.
22

---

Reference
CFG
Woom
Weon
Weon 4.0
Woon
monster toy playing violin sticker style
A monster toy soaring through digital landscape in vector illustration style
A photo of sloth plush in graduation gown, receiving diploma on the slage
Figure 14: Effect of consistency guidance scale. We show the effect of consistency guidance scale
Weon by varying from 2.0 to 5.0. We also show the synthesized results using CFG. Note that the optimal choice of consistency guidance scale (in consideration user's preference) might varies among reference dataset, or even input prompts.
23

---

photo man wearing blue jeans and shirts
photo a man airplane
photo sttending seminar
photo man giving lecture about diffusion models
A photo of a man jogging in the park
A photo of a man doing workout the gym
A photo of a man having selfie wife wedding
A man in 2D animation style
Reference
pig playing guitar the stadium
pig eating pancakes
pig holding avocado
pig riding bicycle
photo of a pig natural lighting style
pig drinking coffee in 2D cartoon style
pig chef outfit, making food, flat illustration style
pig jogging at beach in cartoon style
1-shot personalization using synthetic images generatd by SDXL. We show the capability of our method in 1-shot subject personalization using the images generated by pretrained SDXL models. For each reference image (man and pig), DCO fine-tuned T2I models can generate subjects with different actions and styles. The prompts that used generate reference images were "a photo of a 50 years old man with curly hair" and "a 3D animation of happy pig". respectively, as used in [61].
Figure 15:
24

---

photo
snowsuit, skinng mountain
rook star performing at huge stadium
chet, cooking in the
astronaut, walking down
origamm style
laptop fiat cartoon illustration style
ofter dhinking in café logo style
papercut style
Figure 16: 1-shot personalization using synthethic images generated by DALLE-3, We show the capability of our method in 1-shot subject personalization using the images generated by DALLE- [8]. We asked DALLE-3 to generate cute baby otter image. The comprehensive caption to fine-tune this image is "A closed-up photo of an <otter> on the top of wooden log, forest in the background".. Our method is able recontextualize the subject in the reference image with various text prompts depicting accessories, background, or style.
25

---

Reference
Pretrained
DB+p.p.
DB+CC+p.p.
DCO (ours)
wolf plush emerging from colorful paper layers in paper cutout style
nar
dog a landscape of papercut art style
A photo of golden robot toy
A backpack in flat cartoon illustration style
A dog tiny figure in grand landscape in miniature model style
Figure 17: Custom subject generation. We compare our method (DCO) with pretrained model, DreamBooth (DB), and with prior preservation loss (DB+p.p.). Note that images in each row are generated using the same random seed. Our method is able to generate subject consistent images with various accessories, background or style, with better image-text alignment than baseline methods.
26

---

Reference
DB
DCO (ours)
A (Christmas tree, butterfly, piano) melting golden 3D rendering style
(cat, cow, avocado) watercolor painting style
A (butterfly, coffee maker, plano) in watercolor painting style
Figure 18: Custom style generation. Additional qualitative results on custom style generation. Our method (DCO) is able to generate style consistent image, while prior method, DreamBooth (DB), is prone to overfitting. For example, in the first row, the leaves of flower is inherited to butterfly, Christmas tree, and piano in DB, while our methods disentangle such attributes in generation. Those also shown in second and third rows.
27

---

carfoon line orawing style
sloth plush
wolf plush
baokpack
Kid crayon drawing style
monster toy
toddy bear
dog
DB Merge
Merge
DB ZipLOAA
DB ZipLoRA
DOO Merge (ours)
DCO Merge
(ours)
watercolor painling style
doog
alarm clock
fringed boot
moderm rendering style
cat
backpack
sneakers
Merge
Merge
DB ZipLoRA
DB ZipLoRA
DCO Merge (ours)
DCO Merge (ours)
Figure 19: My subject in my style generation. Additional results are shown. Our method (DCO Merge) generates an image that maintains subject and style consistency without any post- post-processing. On the other hand, DreamBooth Merge (DB Merge) shows inferior results as either overfitted to subject (e.g., sloth plush, wolf plush, backpack are not in cartoon line drawing style), or styles (e.g., monster toy, teddy bear, dog not appear kid crayon drawing style). Meanwhile, ZipLoRA shows better results than DB Merge, yet it often loses the subject or style fidelity.
28

---

Subject
DCO Merge (ours)
Style
DB ZipLoRA
drinking coffee
In boat
with crown
sleeping under chery blossom
riding bicycle
playing violin
Subject
DCO Merge (ours)
Style
DB ZipLoRA
dhinking coffee
sleeping under cherry blossom DCO Merge (ours)
riding bicyole
playing violin
mountain
Subject
Style
DB ZipLoRA
playing with ball
driving car
on the mountain
sleeping under chery
wearing hat
playing a violin
Figure 20: Text-compositional my subject in my style generation. We show more qualitative results for my subject in my style generation that compare arithmetic merge of DCO fine-tuned models (DCO Merge) and ZipLoRA [21] on DB fine-tuned models. DCO Merge generates images consistent to both subject and style without further post-optimization, while ZipLoRA often misses the subject or style consistency (e.g., teddy bear and monster toy are changed, and the style of the third example not consistent).
29

---

Subject
DCO Merge (ours)
Style
DB ZipLoRA
drinking coffee
boat
with crown
sleeping under cherry blossom DCO Merge (ours)
riding a bicycie
playing a violin
Subject
Style
DB ZIpLoRA
drinkigg coffee
boat
with
sleeping under cherr blossom DCO Merge (ours)
riding bicycle
playtng vioiin
Subject
Style
DB ZipLoRA
driving car
boat
playing with ball
with crown
Sleeping under cherry
playing a violin
Figure 20: Text-compositional my subject in my style generation. (continued)
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have stated main claims in the abstract and the introduction (Sec. 1) with descriptions on our method as well as experimental results (Fig. 1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitations of our method in Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
Our paper considers training diffusion models, which we explain the back- ground with principles in Sec. 3. We also provide derivation of our claims Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a pseudocode to implementation our method in Algorithm 2, and implementation details in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
32

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will make a related decision for this after the acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. co/
public/ Seeiii/ggides/Codeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//tppp.ccppbbliie httpccccccpcccccccccccccccccccct cambliccggiides guides/ /CodeSummm ssi onPolicy)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specified all the training dataset, hyperparameters (e.g.. optimizers, learning rate, rank of LoRA), and evaluation strategies. See Sec. 5 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
All experiments are conducted with the same and commonly used random
seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
33

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /neuriprrss.. oc htcppppcc//////cooccccoccclccclt wetec//Ehiisciio cwwuiceessiiellieeeen ines?
Answer: [Yes]
Justification: Authors carefully read the NeurIPS Code of Ethics and preserved anonymity. Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide potential negative societal impact in Appendix F,
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The license for each data is described in Sec. D.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involves any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involves any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "tN9iiRCznj", "reviewer": "Reviewer_HfJW", "summary": "This paper studies the catastrophically forgetting issue in personalizing text-to-image diffusion model. The main comparable baseline is DreamBooth, which prevents forgetting by finetuning the model on a subsample of original training data while learning new concepts. The proposed method, on the other hand, directly regularizes the deviation of reference model from original model, which resulting a new objective function that essentially penalizing overfitting to the ‘hard examples’ during personalization. Empirical evaluations are conducted on subject and style transfer datasets, where the proposed method achieves superior pareto optimality on image fidelity v.s. prompt following ability, as well as better visual results compared with DreamBooth baseline.", "questions": "1. At a high level, both DCO and DBpp are trying to learn the concept while keeping certain aspects of the new model closer to the original model. It seems (e.g. line 39 and Figure 2) that the motivation for using is based primarily on empirical performance. Could the author elaborate more on the intuition of why DBpp hurts the subject fidelity?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is conceptually simple yet yield strong empirical results (if judged solely based on the displayed results)\n2. One benefit of DCO is that it relies solely on the reference data, as the user does not have to gain access to the original training data for regularization during finetuning.\n3. The proposed method, in particular DCO, has its novelty and value to the community, if the strong performance can be further verified by extra experiments requested in the weakness section.", "weaknesses": "The empirical evaluation can benefit from extra validations.\n\n1. Lacks human evaluation. The paper could benefit from a proper human study from third-party judgers. Currently, the empirical comparisons seem to be confined by automatic evaluation and a subset of visualizations.\n2. Compounded factors are not studied separately. The Textual Inversion and comprehensive captioning techniques are deployed on top of DB and DCO. With these enabled, it is unclear what might be the interplay between these techniques and different baselines. For instance, the compatibility of DB might be less compatible with DB than DCO, since DCO only requires the reference data during training. These ablations would not necessarily diminish the proposed method; They are just very beneficial for research purposes IMO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer HfJW,\n\nThank you for valuable comments and suggestions in reviewing our work. We address your questions and concerns as follows.\n\n---\n### [W1] Human evaluation\nFollowing your suggestion, we conduct a user study to compare DCO (ours) against DreamBooth (DB) and DB with prior preservation loss (DB+p.p.) on subject personalization tasks. As in Section 5.1, we train three models (DCO, DB, DB+p.p.) per subject with the identical experimental setup (i.e., all models are trained with comprehensive caption and textual inversion, and compare by generating images with the same random seed. We asked the participants to choose the best one out of three images from each model. Raters are asked to compare images on the following three criteria: \nSubject fidelity: Which image most accurately reproduces the identity (e.g., item type and details) of the reference item?\nPrompt fidelity: Which image most closely aligns with the given prompt?\nImage quality: Which image exhibits the highest quality (e.g., overall visual appeal and clarity)?\nWe asked 22 users to evaluate 60 sets of images, resulting in a total 1320 responses per query. The table below shows the result.\n\nTable 1. Subject fidelity, prompt fidelity and image quality user preference.\n\n|                    | Subject fidelity | Prompt fidelity | Image quality |\n|---------------------|----------------------------|---------------------------|-------------------------|\n| DB              | 47.7%               | 22.6%             | 27.7%            |\n| DB+p.p.     | 6.0%                 | 27.6%              | 25.5%            |\n| DCO (ours) | 46.3%               | 49.8%              | 46.9%            |\n\nWe see that DCO (ours) largely outperforms DB and DB+p.p. in prompt fidelity and image quality, while showing comparable performance with DB on subject fidelity. This result is consistent with Figure. 1 of our manuscript, where we have shown that DCO outperforms others in image-text similarity (i.e., prompt fidelity), while exhibiting comparable performance in image similarity (i.e., subject fidelity). Interestingly, we find that the users find DCO-generated images are of better quality than others, which demonstrates that DCO mitigates image quality degradation, which often happens when fine-tuning from a few, low quality reference images.   We will include our results as well as the detailed information on our user study format in the final revision.\n\n---\n### [W2] Ablation study\nAs we mentioned in our manuscript (e.g., line 210-211), using comprehensive caption (instead of compact caption used in DreamBooth paper) improves the performance of both DreamBooth and DCO methods. To support our claim, we provide an additional ablative study on the effect of comprehensive caption. We select 10 subjects from DreamBooth dataset and compare with compact captions on both DreamBooth (DB) and DCO, using the same experimental setup as in Section 5.1. For compact captions, we do not use rare token identifiers and learn textual embeddings as in Section 5.1 Figure A in our attached file shows the Pareto curves with consistency guidance of varying scales ($\\omega_{\\textrm{con}}$=2.0, 3.0, 4.0, 5.0). We observe that a comprehensive caption (solid line) forms an upper-right frontier compared to compact caption (dashed line) for both DB and DCO. \n\n---\n### [Q1] Prior preservation loss hurts subject fidelity\nWe hypothesize that learning from both reference and synthetic images as in prior preservation loss hurts subject fidelity due to the concept leakage from the synthetic images. Note that such a behavior (e.g., losing subject fidelity at the cost of generation diversity) has been already observed from the DreamBooth paper (e.g., see Section 4.3 and Table 3 of the DreamBooth paper). On the other hand, DCO preserves the prior without synthetic data by minimizing the shift from the pretrained model while fitting on reference images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ka3fW3fOkc", "reviewer": "Reviewer_aXKh", "summary": "This paper introduces a novel fine-tuning objective called Direct Consistency Optimization, which regulates the deviation between fine-tuning and pre-trained models to preserve pre-trained knowledge during the fine-tuning process. Models fine-tuned using this method can be merged seamlessly without interference.", "questions": "As seen in weakness.", "limitations": "The manuscript includes discussions on limitations and social impacts.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The results seem good and surpass the baseline methods.\n2. The direct merging of subject and style is interesting.", "weaknesses": "1. The comparison with Lora is missing. Usually the results of Lora customization are very competitive.\n2. In fig1, the pose of the bear in different images are mostly the same as those in the reference image, which is overfitting. It needs to compare the diversity of generated images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer aXKh,\n\nThank you for valuable comments and suggestions in reviewing our work. We address each of your questions and concerns individually as follows.\n\n---\n### [W1] Experiment with LoRA\nWe clarify that all our experiments were conducted using LoRA. This has been explicitly mentioned in our manuscript, experimental setup section (e.g., see line 205 and line 242).\n\n---\n### [W2] Diversity of generated images. \nOur method is able to generate personalized subject images of various poses, e.g., see Figure. 19 and Figure. 20 in Appendix. Since our method is better than previous methods in terms of prompt fidelity and subject fidelity, one can provide prompts to change the poses or actions to generate diverse images while preserving identities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oewSinQ7cG", "reviewer": "Reviewer_ayXu", "summary": "Current personalized T2I models, like DreamBooth, are capable of generating personalized images by fine-tuning on a small set of reference images. However, these fine-tuned models often suffer from robustness issues, such as poor compositional capabilities with the pretrained model concepts and other fine-tuned models.\n\nThe paper \"Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models\" introduces a novel fine-tuning method called Direct Consistency Optimization (DCO) to enhance the robustness of such personalized text-to-image (T2I) diffusion models.\n\nDCO introduces a novel training objective that minimizes the deviation between the fine-tuning and pretrained models by optimizing a consistency function. The method involves computing a loss function that measures the deviation in noise prediction errors between the fine-tuned and pretrained models, thus allowing efficient implementation using a noise-prediction loss approach.", "questions": "have you thought about trying a kandinsky2.2/dalle2 like model which can directly condition on the image embedding for a (or multiple) reference images?", "limitations": "yes its adequately address in line 632", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Novel Objective Function: The introduction of Direct Consistency Optimization (DCO) represents a novel approach to fine-tuning text-to-image diffusion models. Unlike previous methods that primarily focus on incorporating additional datasets to retain pretrained knowledge, DCO directly controls the deviation from the pretrained model\n* The paper effectively combines concepts from constrained policy optimization and noise prediction to derive the DCO loss function. This is a novel contribution based on my knowledge\n* good amount of experiments and results visualization\n* this work could have big potential impact how the booming application of personalized t2i diffusion models", "weaknesses": "* Incremental Improvement: While the DCO method introduces a novel fine-tuning objective, the actual improvement seem marginal (based on figure 4 and figure 7)\n\n* Limited Diversity of Experiments: The experiments primarily focus on subject and style personalization within specific datasets. A broader range of experiments, including diverse set of human images. \n\n* Reproducibility Concerns: Although the methodology is well-explained, the paper does not provide detailed information on implementation specifics such as the codebase, computing resources, or detailed training schedules. This might pose challenges for reproducibility and practical adoption by the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer ayXu,\n\nThank you for valuable comments and suggestions in reviewing our work. We address each of your questions and concerns individually as follows.\n\n---\n### [W1] Improvement by using DCO.\nWe claim that DCO uses a novel fine-tuning that pushes the frontier of the pareto curve of image similarity and prompt similarity, outperforming previous baseline methods DreamBooth and DreamBooth with prior preservation loss, as well as their variants, e.g., early stopping or different prior preservation loss weights (e.g., see Figure. 7). Not only that, DCO enables direct merging of two independent trained LoRAs, where DreamBooth trained models often fail to preserve the subject or style fidelity during merging. \n\n---\n### [W2] Other experiments\nTo show the effectiveness of our method, we provided 1-shot subject personalization experiments as shown in Figure.14 and Figure.15 of Appendix. By using a single subject image, our method can generate diverse personalized images that change the style (e.g., photo to 2d animation style, or 3D animation style to photo) or visual attributes (e.g., actions or outfits) by using textual prompts. Specifically, as shown in Figure. 14, we show the capability of our method in learning from a single human image to stylize or change actions. We believe our method can be further explored for human specific datasets, which we leave for future work. \n\n---\n### [W3] Reproducibility\nFor reproducibility of our method, we provide implementation details in Appendix D, where we describe the compute resources and training schedules. For our codebase, we use PyTorch and Huggingface diffusers library for the implementation. We will release our code in the final manuscript.\n\n---\n### [Q1] Personalization of image-conditioned models\nWe believe our method can be utilized for personalization of image-conditioned diffusion models (e.g., Kandinsky 2.2 or Dalle-2). Similar to fine-tuning with DCO objective for text-to-image diffusion models, one can consider fine-tuning image conditioned diffusion models that use image embeddings. Specifically, given a pair of reference images (img1 and img2) that share the concept (e.g., subject or style), one can fine-tune the diffusion model on img1 conditioned with image embeddings from img2. Here, DCO loss could be used to restore the pretrained knowledge, which we believe would be effective to prevent overfitting and improve compositional generation of personalization image synthesis. We believe it is an interesting direction, and leave it for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uJLKfJeAly", "reviewer": "Reviewer_XWYq", "summary": "The paper introduces a novel training framework and novel sampling technique to enhance the state-of-the-art personalized and style-preserving adaptation of text-to-image diffusion models in the low-shot fine-tuning regime.", "questions": "## Suggestions \n\n* Figure 1.B is not obvious when conveying the limitations. Perhaps mention that the subject is not faithfully preserved (hat is missing) and the style is not faithfully preserved (blue background is missing) in the case of DreamBooth. \n* Line 97 - 98: It would be better to also mention how training with classifier-free guidance is implemented in practice, i.e., we randomly make captions associated with images to null (caption dropout). \n* Since the authors use LoRA for fine-tuning, I think the memory overhead could still be kept to a minimum if they enable and disable the adapter layers within the denoiser network. See how it's implemented [here](https://github.com/huggingface/diffusers/blob/3fca52022fe0ea9aaf0a0ea8a0fc13308bf69a9f/examples/research_projects/diffusion_dpo/train_diffusion_dpo_sdxl.py#L1019) and [here](https://github.com/huggingface/diffusers/blob/3fca52022fe0ea9aaf0a0ea8a0fc13308bf69a9f/examples/research_projects/diffusion_dpo/train_diffusion_dpo_sdxl.py#L1035). This probably should improve the memory requirements quite a bit. Could the authors verify this approach? \n* It would be nice to enlist and cite the libraries that were used in the codebase. They do enable productive research and citing them goes a long way. \n* Provide a loss landscape comparison when varying the amount of DCO $\\beta$. \n* Highlight the highest scores reported in Table 1. \n* If both GPT-4 and LlaVA were used for captioning, how were the combined captions used during training? \n\n## Questions\n\n* If there's a connection between the guidance scheme introduced in InstructPix2Pix [1] and the one introduced in DCO, it would be nice to have that discussed. \n* Line 204: It's not clear what do the authors mean by \"omitting. \n* For the DreamBooth and DreamBooth + PP experiments, do the authors use LoRA? If so, the text is not clear. \n* Was consistency sampling applied for the DreamBooth experiments? \n* Perhaps refer to the combination of LoRA DreamBooth + Textual Inversion as \"pivotal tuning\" as commonly referred in the community [2]?\n* With a higher value of $\\omega_{con}$, I would assume that the image similarity would improve. But with increasing $\\omega_{con}$, it seems to be dropping off as per Figure 4. Or am I looking at it wrongly? \n* Did the authors experiment with other merging techniques geared toward LoRAs [3]? \n\n## Nits\n\n* \"In specific\" could be replaced with \"specifically\". \n* Line 46: \"... policy optimization problem [19, 20], which amounts _for_ the consistency ...\"\n* Line 75: \"_To the best of our knowledge,_ this paper is the first to have studied consistency as a reward function for T2I personalization.\"\n* Line 83: Shouldn't $\\lambda_{1}$ be $\\lambda_{t}$? Similar comments for $\\lambda_{0}$. \n\n## References\n\n[1] InstructPix2Pix: Learning to Follow Image Editing Instructions, https://arxiv.org/abs/2211.09800\n\n[2] https://replicate.com/cloneofsimo/lora-training/readme\n\n[3] https://huggingface.co/blog/peft_merging", "limitations": "The authors are clear about limitations. Perhaps mentioning them a bit from the main text would be more sensible.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The training objective introduced in the paper is implicitly incentivized to learn a reward for the model for better consistency. \n* The decoupling of classifier-free guidance to include the learned consistency preserving model enhances image similarity with the reference image, thereby also improving image fidelity.", "weaknesses": "* DCO is computationally expensive since the proposed sampling scheme makes use of two networks during inference. However, the authors also mention it explicitly in the limitations section. But it would be nice to be forthright about it in the paper's main text. \n* Multiple references to existing works are missing. Examples include SVDiff [1], Custom Diffusion [2], IP-Adapters [3], and, DiffuseKrona [4] (for controllable personalized generation), and MaPO [5] (reward fine-tuning through alignment). It also didn't include references to works that successfully applied LoRA in the context of adapting text-to-image models [6], [7], [8]. \n* The paper's main text reads incomplete without the minimum details on the datasets being used for training and evaluation. For example, in lines 219 - 224, the reader doesn't know what reference images were used. Similarly, in lines 251 - 254, there is no mention of Appendix D, which sheds light on more details. This is also the case for the text accompanying Table 1 (Lines 280 - 287). \n* While reporting the results of Figure 4, it would have been better to talk about $\\omega_{text}$ as well because that substantially impacts the generation quality. From the accompanying text, the reader doesn't know how it was varied or if it was kept fixed.\n\n## References\n\n[1] SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, https://arxiv.org/abs/2303.11305.\n\n[2] Multi-Concept Customization of Text-to-Image Diffusion, https://arxiv.org/abs/2212.04488. \n\n[3] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models, https://arxiv.org/abs/2308.06721. \n\n[4] DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models, https://arxiv.org/abs/2402.17412. \n\n[5] Margin-aware Preference Optimization for Aligning Diffusion Models without Reference, https://arxiv.org/abs/2406.06424.\n\n[6] Implicit Style-Content Separation using B-LoRA, https://arxiv.org/abs/2403.14572. \n\n[7] https://github.com/cloneofsimo/lora\n\n[8] https://huggingface.co/blog/lora", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer XWYq,\n\nThank you for valuable comments and suggestions in reviewing our work. We address each of your questions and concerns individually as follows.\n\n---\n### [W1] Regarding computational cost\nWe will move the limitation into the main text in our final revision.\n\n---\n### [W2] References\nThank you for the relevant work. While several papers have proposed different adapter designs (e.g., SVDiff, DiffuseKrona, B-LoRA) for improving personalized T2I synthesis, we introduce a new fine-tuning objective that is independent of adapter design. We appreciate the mention of the MaPO paper, which was unavailable at our submission time but will be included in our final manuscript as it provides a different fine-tuning objective for T2I diffusion models.\n\n---\n### [W3] Experimental dataset\nWe elaborated the experimental dataset for each subject and style personalization experiment in line 200-211 and line 241-245, respectively. We will include details on the dataset for each figure and table in our final manuscript. \n\n---\n### [W4] The value of w_text \n$w_{text}$ is kept with 7.5 throughout the experiments (see Appendix D.4). We will revise our draft to include this information in figures.\n\n---\n### [S1] Regarding Figure 1\nWe will add details to the captions of Figure 1 to better deliver the limitations of previous works and demonstrate effectiveness of our method. \n\n---\n### [S2] Details on classifier-free guidance\nWe will add technical details of classifier-free guidance.\n\n---\n### [S3] Technical implementation\nFollowing your suggestion, we compare the training memory consumptions of suggested technical implementation and our implementation. Note that the suggested method use \n```\nunet.disable_adapters()\nunet.enable_adapters()\n```\nto compute loss from pretrained models. Originally, we used ‘cross_attention_kwargs’ to control the LoRA scale. For each method, we measure the allocated and reserved memory. All experiments were conducted on PyTorch v2.3.0, diffusers v0.27.2 on 1x A100 40GB GPU. The results are shown in the table below.\n\n|           | Mem. allocated | Mem. reserved | Max mem. allocated | Max mem. reserved |\n|-----------|----------------|---------------|--------------------|-------------------|\n| Ours      |   7812.78 MB   |  16653.48 MB  |     16021.28 MB    |    16653.48 MB    |\n| Suggested | 7812.78 MB     | 16647.19 MB   | 15933.40 MB        | 16647.19 MB       |\n\n\nThe table shows that the suggested method saves more memory than our implementation. We will update technical implementation to our code, and release it with our final manuscript. \n\n---\n### [S4] Cite libraries \nWe will add citations to the libraries that we used in our experiments.\n\n---\n### [S5] Loss landscape of DCO with different $\\beta$ \nWe provide the loss landscapes by using different values of $\\beta$ in Figure B in our attached file. For a small value of $\\beta$, the regularization strength is small, thus the loss results in higher variance. For a large value of $\\beta$, the regularization is stronger, which results in smaller variance. \n\n---\n### [S6] Highlight scores\nWe will highlight the scores in Table 1.\n\n---\n### [S7] Training prompts\nTraining prompts are manually engineered using vision language models like LLaVa or ChatGPT because their generated captions are often too long for SDXL text encoders. A more effective approach could be to use captioning models (e.g., in DALL-E 3) to generate prompts for personalized images similar to the pretraining datasets.\n\n---\n### [Q1] Consistency guidance\nInstructPix2Pix takes additional input (i.e., source image) to control the similarity of the edited image with respect to the source image. On the other hand, consistency guidance controls the fidelity to the reference image without any additional input. We will add discussion in our final revision.\n\n---\n### [Q2] Context of ‘omitting’\nTo clarify, we omit the usage of textual inversion technique and comprehensive captioning, which we used for all baseline experiments for fair comparison. \n\n---\n### [Q3] Usage of LoRA \nAs we mentioned in our paper, we used LoRA for all experiments. \n\n---\n### [Q4] Consistency sampling for DreamBooth\nFor qualitative results, we show the results by using classifier-free guidance (CFG) sampling of fine-tuned models. In our quantitative results, we provide results of using both CFG sampling (diamond) and consistency guidance sampling (dots with solid lines) for all baselines including DB and DB+p.p.\n\n---\n### [Q5] Reference for pivotal tuning \nWe will add citation of pivotal tuning for the usage of textual inversion and LoRA fine-tuning.\n\n---\n### [Q6] Regarding $w_{\\text{con}}$\nYes, as $w_{con}$ increases, the image similarities become higher. In Figure 4, the image similarities increase (i.e., towards upper left) as $w_{con}$ increases from 2.0 to 5.0. \n\n---\n### [Q7] Different merging method experiments \nFollowing your suggestion, we conduct experiments on comparing different merging methods. Specifically, we apply SVD based LoRA merging (i.e., TIES_SVD) [1]. As shown in Figure C, SVD merging improves quality for both DB and DCO compared to direct merging. We observe that LoRA trained with DCO shows better performance, particularly in following complicated prompts while maintaining identity. We will add this to our final revision.\n\n[1] Yadav, Prateek, et al. “Ties-merging: Resolving interference when merging models.” NeurIPS 2023.\n\n---\n[N1-3] Thanks for pointing this out. We will revise in our final manuscript. \\\n[N4] Note that $\\lambda_1$ is the maximum log-SNR for the noise scheduling function, which should be large enough so that the distribution of $z_1$ follows pure random Gaussian noise. Conversely, $\\lambda_0$ is the minimum log-SNR for the noise scheduling function, which should be small enough so that the distribution of $z_0$ matches the data distribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

As reviewers highlighted, we believe our work presents a novel training objective that is simple and effective (HfJW, aXKh, ayXu, XWYq) that is supported by qualitative and quantitative experimental results (HfJW, aXKh, ayXu). 

We kindly ask you to check the attached supplementary PDF file when acquiring the information provided in the rebuttal comments. Please let us know if you have any comments or concerns that we have not addressed up to your satisfaction.

Meta-review: In this work, authors propose an algorithm called direct consistency optimization where the core idea revolves around an objective that minimizes the deviation between the fine-tuning and pre-trained models by optimizing a consistency function. Overall, the reviewers tend to like the novelty present in this idea and are also encouraged by the potential amount of impact this algorithm can have. There are some concerns over computational cost increase, however, in my opinion, that is not prohibitively expensive for this algorithm to be adapted. Overall, I recommend acceptance.
Decision: Accept (poster)