Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models

Authors: kyungmin  lee
sangkyung  kwak
kihyuk  sohn
inwoo  shin
  kaist
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:56:36

Abstract: Text-to-image (T2I) diffusion models, when fine-tuned on a few personal images, can generate visuals with a high degree of consistency. However, such fine-tuned models are not robust; they often fail to compose with concepts of pretrained model or other fine-tuned models. To address this, we propose a novel fine-tuning objective, dubbed Direct Consistency Optimization, which controls the deviation between fine-tuning and pretrained models to retain the pretrained knowledge during fine-tuning. Through extensive experiments on subject and style customization, we demonstrate that our method positions itself on a superior Pareto frontier between subject (or style) consistency and image-text alignment over all previous baselines; it not only outperforms regular fine-tuning objective in image-text alignment, but also shows higher fidelity to the reference images than the method that fine-tunes with additional prior dataset. More importantly, the models fine-tuned with our method can be merged without interference, allowing us to generate custom subjects in a custom style by composing separately customized subject and style models. Notably, we show that our approach achieves better prompt fidelity and subject fidelity than those post-optimized for merging regular fine-tuned models.

Main paper: 
 introduction: 
 Text-to-image (T2I) models are for image generation guided by natural language prompts and have seen rapid progress in recent years [1] [2] [3] [4] [5] [6] [7] [8] . The compositional nature of the natural language has enabled the creation of novel images, which compose multiple subjects with varying attributes in different backgrounds or styles. However, the ambiguity of natural language in describing the visual world makes it difficult to create an image of a specific subject, style, interaction, or background.To overcome the lack of accuracy in natural language, there has been an emerging interest in teaching the pretrained T2I models new concepts, such as subject [9, 10] , style [11] , interaction [12] , or background [13] , whose precise visual description is given by a small set of reference images. As proposed in DreamBooth [10] , the fundamental idea is to fine-tune the pretrained T2I model on a few images describing a new concept. The adoption of LoRA [14] or adapter fine-tuning [15] to T2I models has made the process more accessible, fast and economical [16, 11] . Once fine-tuned, the model can generate images by composing a new concept (e.g., my subject) and the knowledge of the pretrained model (e.g., background, style). While these methods have shown great success, they still suffer from reduced textual alignment and compositional generation capability [17] , which is problematic particularly when the number of reference images is small, e.g., one or two.˚Work done while at Google Research. 2 Project page: https://dco-t2i.github.io 38th Conference on Neural Information Processing Systems (NeurIPS 2024). [10] , and with prior preservation loss (DreamBooth+p.p.). DCO improves generating custom subject with various visual attributes (e.g., astronaut outfits and background of Mars), or various styles that pretrained model knows (e.g., flat cartoon illustration style). (b) The customized subject and style models fine-tuned with DCO can be merged as is, allowing us to generate my subject in my style [11] .We hypothesize that the limitation comes from the knowledge forgetting during low-shot fine-tuning, i.e., it fails to recontextualize new concepts with known concepts of pretrained models. Since finetuning with regular diffusion training objective suffer from overfitting, DreamBooth [10] attempted to tackle this issue by adding class-specific prior dataset for training dataset to preserve the prior knowledge (i.e., prior preservation loss). However, while this helps retaining the generalization capability, it comes at the cost of less subject fidelity (e.g., see Fig. 3 ). Especially, the loss becomes more significant, when the number of reference images become smaller [18] .Contribution. In this paper, we propose a more principled method to mitigate the forgetting behavior of low-shot fine-tuning of T2I diffusion models without using any additional data. Our method, coined Direct Consistency Optimization (DCO), fine-tunes T2I diffusion models by controlling the deviation between fine-tuning and pretrained model to be as minimal as possible in learning new concept. In specific, we show that our objective is equivalent to learning an implicit reward function that is a solution of constrained policy optimization problem [19, 20] , which amounts for the consistency between generated images and reference images. Furthermore, we propose consistency guidance sampling which utilizes learned consistency function during inference, so that one can control the tradeoff between subject consistency and textual alignment (e.g., see Fig. 5 ).We conduct an extensive empirical study on the personalization of T2I diffusion models (see Fig. 1 ). We show that the proposed method improves upon baselines that uses regular fine-tuning objective or using prior preservation loss [10] in generation of custom subject with various visual attributes or with the known style of pretrained model. To be specific, we show that our method positions on the superior Pareto frontier than baselines (e.g., see Fig. 5a ). Moreover, our approach is applied to style personalization [11] and the composition of separately fine-tuned subject and style T2I models to generate an image of my subject and my style [11] . Notably, direct merging of DCO fine-tuned models outperforms merging models fine-tuned with regular diffusion loss even using post-optimization method, e.g., ZipLoRA [21] , in terms of prompt fidelity and subject fidelity. We believe that the proposed method would enjoy a broader usage in customization diffusion models in various domains.present a different approach to preserve prior knowledge by regularizing the model with pretrained model without using any prior dataset.Fine-tuning T2I diffusion models with rewards. A line of works has studied fine-tuning T2I diffusion models using reward models such as human preferences [29] [30] [31] or aesthetic scores [32] . Those methods use auxiliary reward models to update diffusion models by weighting with rewards [29] , use reinforcement learning algorithms [33, 34] , or differentiate through the reward models [35] [36] [37] . On the other hand, Wallace et al. [38] have adapted direct preference optimizaton [39] to fine-tune T2I models with paired preference dataset which do not require explicit reward models. To the best of our knowledge, this paper is the first to study consistency as a reward function for T2I personalization. 
preliminaries: 
 Diffusion models. For a data point x, let qpxq be the density of data distribution and p θ pxq be a generative model parameterized by θ that approximates q. Given x " qpxq, the diffusion model considers a series of latent variables z t for timesteps t P r0, 1s. The forward process of diffusion model forms a conditional distribution qpz t |xq, and the marginal distribution is given by z t " α t x `σt ϵ, where ϵ " N p0, Iq, and α t , σ t are noise scheduling functions. We denote λ t " logpα 2 t {σ 2 t q log signal-to-noise ratio (log-SNR), which is a decreasing function of t. For large enough λ 1 , z 1 is indistinguishable from pure Gaussian noise (i.e., ppz 1 q « N p0, Iqq, and for small enough λ 0 , the z 0 is identical to the data x. The generative process of diffusion models starts off from a random noise x 1 " N p0, Iq, and sequentially denoise it to recover z 0 . In theory, the generative sampling process is governed by solving SDE [40, 41] or probability flow ODE [42, 43] , using the score networks s θ pz t ; tq that approximates the score function of marginal distribution ∇ log qpz t q. When training diffusion models, it is common practice to parameterize score network using the noise-prediction model [41] , i.e., s θ pz; tq " ´ϵθ pz; tq{σ t . Then the training objective of diffusion model is given byEQUATIONwhere w t is a weighting function and λ 1 t is a time-derivative of λ t . Note that Eq. ( 1) is a generalized weighted noise-prediction loss, which includes ϵ-prediction loss [41] when ´1 2 w t λ 1 t " 1. Text-to-Image diffusion models. Text-to-Image (T2I) diffusion models [44, 2, 4] are diffusion models of an image conditioned on text, which is often processed into embeddings using the pretrained text encoders, such as T5 [45] or CLIP [46] . Given a dataset px, cq " D of paired image x and text prompt c, the training loss L DM pθ; Dq for T2I diffusion models is given by L DM pθ; Dq " E px,cq"D rL ϵ pθ; x, cqs, with text conditional noise-prediction model ϵ θ pz; c, tq. T2I diffusion models are often trained with classifier-free guidance (CFG) [47] , which jointly learns unconditional and conditional models, and interpolates them during inference. The predicted noise with CFG scale ω ě 1 is given as: εθ pz; c, tq " ω pϵ θ pz ; c, tq ´ϵθ pz; tqq `ϵθ pz; tq, (2) where ϵ θ pz; tq denotes the unconditional noise prediction, e.g., with null text conditioning. It is known that higher CFG scale ω improves the image-text alignment at the cost of the image fidelity.Personalizing T2I models. Recent works have shown the potential for personalization of T2I models by fine-tuning the T2I diffusion models on a few samples. DreamBooth [10] optimizes the diffusion model on a few subject images accompanied with a compact caption composed of rare token identifier and class noun. While fine-tuning with regular diffusion loss (i.e., L DM ) works well, the authors proposed the so-called prior preservation loss to retain the prior knowledge of the pretrained model. This is achieved by optimizing the model with auxiliary training images of the same class to the subject of interest. Formally, given reference dataset D ref and prior dataset D prior , the training loss of DreamBooth with prior preservation loss (DB+p.p.) with hyperparameter λ prior ą 0 is given by L DB+p.p. pθq " L DM pθ; D ref q `λprior L DM pθ; D prior q.(3)Parameter efficient fine-tuning (PEFT). In practice, parameter-efficient fine-tuning (PEFT) methods are combined with DreamBooth to enable fast and memory-efficient adaptation of diffusion models. In particular, low-rank adaptation (LoRA) [14] is a popular choice, where it fine-tunes the residuals ∆W P R nˆm of weight matrix W P R nˆm with low-rank decomposition ∆W " AB for A P R nˆr and B P R rˆm with rank r ! mintn, mu. Alternatively, Textual Inversion (TI) [9] introduces a new token and corresponding textual embedding v to represent the concept. Then, TI optimizes textual embedding by solving v ˚" arg min v L DM pv; D ref q. L DCO pθq Ð ´log σ `´β t `ℓpθq ´ℓpϕq ˘9: Update θ Ð θ ´η∇ θ L DCO pθq 10: end while 4 In this section, we introduce our method for T2I personalization. For presentation clarity, we focus on our demonstration with the subject customization [10] , but the method can be applied to a broader context of personalization, such as style [11] . Throughout the paper, let us denote ϵ θ , p θ and ϵ ϕ , p ϕ the noise-prediction model and density for each fine-tuning and pretrained diffusion model, respectively. 
direct consistency optimization: 
 Problem setup. While fine-tuning based T2I personalization methods have shown great success [10, 23, 16] , it is shown that the generation quality heavily depends on the model's fitness. For example, the model suffers from image-text alignment when the model overfits to few images used for finetuning, making it difficult to generate images with varying attributes around the subject. On the other hand, the model cannot generate consistent subject images when the model underfits. To find the right balance between overfit vs. underfit, certain heuristics such as early stopping or training with an additional prior dataset (i.e., prior-preservation loss) have been popularly used.Our approach. We devise an efficient training objective that seeks for minimal improvement on the ELBO of fine-tuning model p θ over the reference model p ϕ . Specifically, we seek for θ satisfying D KL `qpxq}p θ pxq ˘ă D KL `qpxq}p ϕ pxq ˘, while two quantities are still not far from each other. Since computing the likelihood is intractable for diffusion models, we consider the variational bound over sequence of latents z 0:1 following [48, 49] . To this end, let us define the deviation as follows ∆pp θ , p ϕ ; x, cq :" D KL `qpz 0:1 |xq } p ϕ pz 0:1 |cq ˘´D KL `qpz 0:1 |xq } p θ pz 0:EQUATIONwhere the KL divergence takes expectation over ϵ " N p0, Iq that generates z t " α t x `σt ϵ for t P r0, 1s. To enforce the positiveness of ∆pp θ , p ϕ q, we propose following log-sigmoid loss function:EQUATIONwhere σpuq " p1 `expp´uqq ´1 and β ą 0 is a temperature that controls the deviation.Direct Consistency Optimization. Now we show that Eq. ( 5) is equivalent to optimizing reward function derived from the solution of constrained policy optimization problem [19, 20] . Suppose f px, c; D ref q :" f px, cq is a function that measures the consistency between image x and reference dataset D ref given the prompt c. We opt to find θ that maximizes the consistency of generated sample x " p θ px|cq, while penalizing the deviation from the pretrained model p ϕ . This can be written byEQUATIONwhere β ą 0 is a temperature that controls the deviation from pretrained model. Since the likelihood of diffusion model is intractable, we consider a consistency function on the latent variables z 0:1 , and solve following relaxation of Eq. ( 6):EQUATIONNote that the optimal solution to Eq. ( 7) satisfies p θ pz 0:1 |cq 9 p ϕ pz 0:1 |cq exp `f pz 0:1 , cq{β ˘.Then, we show that the deviation ∆pp θ , p ϕ q in Eq. ( 4) is equivalent to the expected consistency over qpz 0:1 |xq up to constant, i.e., the following holds (see Appendix A for derivation):EQUATIONfor some constant C that does not depend on x. Eq. (9) shows that our training objective can be regarded as optimizing the consistency function with respect to the reference dataset D ref .Remark that one can define explicit consistency function (reward) and fine-tune the diffusion model with such function, e.g., by using reward-weighted regression (RWR) [50, 29] or using reinforcement learning (RL) [33, 34] . However, those consistency function for personalized images is hard to obtain as we only have few samples, and RL fine-tuning of diffusion models are expensive. We bypass these issues by directly fine-tuning models with an implicit reward function, similar to those by Rafailov et al. [39] and Wallace et al. [38] . Thus, we name our approach Direct Consistency Optimization (DCO), as we directly optimize the consistency function in fine-tuning diffusion models.Implementation. Note that computing the term for Eq. ( 4) is expensive as it requires computation of likelihood over all t P r0, 1s. To this end, we derive an upper bound of Eq. ( 5) that allows efficient implementation using ϵ-prediction loss. In Appendix A, we show that the deviation Eq. ( 4) can be expressed by the difference between noise-prediction errors of fine-tuning and pretrained model:EQUATIONThen by plugging Eq. ( 10) into Eq. ( 5) and taking out the expectation outside of logarithm function using Jensen's inequality, we have our final DCO loss at px, cq " D ref defined as follows:EQUATIONwhere t " Up0, 1q, ϵ " N p0, Iq, and β t " ´1 2 βλ 1 t3 . We use L DCO in our experiments, which is as practical and easy to implement as regular training objective (i.e., L DM ). Algorithm 2 and Algorithm 1 present DCO fine-tuning and regular fine-tuning side-by-side, with differences colored in red.Gradient analysis of DCO loss. We provide a gradient analysis of DCO loss in fine-tuning diffusion models to better understand its effect. Given a data pair px, cq " D ref , ϵ " N p0, Iq and t P Up0, 1q, the gradient of DCO loss with respect to parameter θ is given as follows:EQUATIONwhere d t " ´βt p}ϵ θ pz t ; c, tq ´ϵ} 2 2 ´}ϵ ϕ pz t ; c, tq ´ϵ} 2 2 ˘with stop-gradient. Remark that Eq. ( 12) is identical to the gradient of diffusion loss (i.e., L DM ), except that is scaled by 1 ´σpd t q, which measures the incorrect reward modeling. In other words, DCO loss implicitly performs an adaptive loss weighting by computing deviation from the pretrained model; if the deviation between fine-tuning and pretrained models are large, it abstains update.Comparison to prior preservation loss. While the prior preservation loss [10] in Eq. ( 3) has a similar motivation to DCO loss, they work in very different ways. To elaborate, DCO directly regularizes the KL divergence with respect to the samples in D ref , while prior preservation loss does not impose regularization for the reference data. While prior preservation loss may enhance the composition ability, fine-tuning on auxiliary samples from D prior often causes undesirable model shift, losing consistency to the pretrained model (e.g., Fig. 3 ). On the other hand, DCO is free from such an issue, as it does not require any auxiliary samples besides the reference dataset.During inference, it is common practice to use classifier-free guidance to control text conditioning. Similarly, to gain control over the consistency, we propose consistency guidance sampling, which is We provide examples of compact caption [10] and our comprehensive caption (top row) and generated images from each method (bottom row). The model fine-tuned with compact caption (left) generates images of a dog sitting on a couch though asked to be on the lake. Our comprehensive caption (right) effectively disentangles unwanted attributes, generating images that follow text prompts more faithfully.an additional guidance from implicitly learned consistency function. Specifically, we decompose the guidance term with respect to the text and consistency as follows: εpz t ; c, tq " ϵ ϕ pz t , tq `ωtext pϵ ϕ pz t ; c, tq ´ϵϕ pz t , tqq `ωcon `ϵθ pz t ; c, tq ´ϵϕ pz t ; c, tq ˘, (13) at each timestep t. Note that for fixed ω text , the fidelity to the reference images increases if using higher ω con , while this comes at the cost of losing prompt fidelity. On the other hand, one can improve prompt fidelity by using small ω con . We show that varying ω con controls the tradeoff between fidelity to the reference and image-text alignment (e.g., see Fig. 5 ). Note that a similar sampling method was introduced in [11] , but for transformer-based T2I models [7] . While they do not use regularized fine-tuning objectives, the consistency guidance scheme is still valid. Thus, the consistency guidance could be implemented in any fine-tuned model, while we show that it is more effective when combined with a DCO fine-tuned model (e.g., Fig. 5 ).An important (yet often overlooked) part of the T2I personalization process is the prompt construction of reference images. Recall that Ruiz et al. [10] have proposed the use of a compact prompt in the form of "a photo of [V] [class]" with a rare token identifier [V]. However, we find that the use of compact caption is prone to learning distractors, such as a background or a style, as part of the fine-tuned model, as illustrated in Fig. 2 .Comprehensive caption. Instead, we propose to provide a comprehensive and visually grounded caption that not only describes the subject but also details other visual attributes, backgrounds, and styles of reference images. In Fig. 2 , we show an example of a comprehensive caption and compare the synthesized results that use a compact caption. We find that providing detailed descriptions of the undesirable attributes, e.g., background, or style, helps anchor desirable attributes in reference images to corresponding texts, making it easier to separate between them. This method not only holds for subject customization, but also for style customization; we provide comprehensive descriptions of the subject so that the model distinguishes style from the subject. Note that the use of comprehensive caption has been considered in practice, 4 but has not been investigated from the lens of model shift and concept disentanglement. In our experiments, we use vision-language models such as GPT-4 [51] or LLaVA [52] (e.g., see Fig. 11 ). 
experiments: 
 We use Stable Diffusion XL [6] for our experiments. We conduct experiments on subject (Sec. 5.1), style (Sec. 5.2) personalization, and their combination (Sec. 5.3). Ablative studies are in Sec. 5.4.Experimental setup. We conduct experiments on DreamBooth dataset [10] , containing 30 subjects, with 4-6 images per subject. The examples of images and captions are in Fig. 11 in Appendix D.1. A {penguin, coffee maker, piano} in kid crayon drawing style Figure 4 : Custom style generation. We show selected generations from DreamBooth (DB) and ours (DCO) of custom styles with varying subjects. DB is prone to capturing undesirable attributes, resulting in generation of mixed concepts (e.g., the girl's outfits in the first row, the dog in the second row), whereas DCO mitigates such a concept mixing. Best viewed in color, zoomed in on monitor.For baselines, we consider the training with a regular diffusion loss, i.e., DreamBooth (DB), and the one with prior preservation loss (DB+p.p.). Note that we additionally optimized textual embeddings, i.e., textual inversion [9] , which enhances subject fidelity for all baselines, though we omit for clear context. For all experiments, we fine-tune LoRA of rank 32 and textual embeddings using Adam [53] optimizer with learning rates of 5e-5 and 5e-4, respectively. We use constant β t =1000 for DCO loss.Following Sec. 4.4, we provide comprehensive and visually grounded caption that not only describes the subject but also details other visual attributes, backgrounds, and styles of reference images, for fine-tuning. Note that this is used for both baselines and ours. We observe that comprehensive captioning improves all the baselines, thus we omit the indications of its usage.Qualitative results. Fig. 3 shows the qualitative comparison of our approach with DreamBooth (DB) and DreamBooth using prior preservation loss (DB+p.p.). We observe that our approach generates images of various visual attributes, e.g., outfits and backgrounds, or changing the material, as well as into various styles, e.g., in origami style or doodle art style. While DB changes the background, it lacks recontextualization in different outfits or styles, especially due to the overfitting to the photographic style. DB+p.p. is better than DB in prompt fidelity, but it often fails to preserve the subject identity. More qualitative comparisons are demonstrated in Fig. Quantitative results. For quantitative evaluation, we design two types of text prompts: subject customization, where we modify the attributes of the subject or its background, and subject stylization, where we change the visual style of a subject image. For each subject and type, we generate images from 50 text prompts with 2 random seeds. For evaluation metrics, we report the image similarity score using DINOv2 [54] and the image-text similarity score using SigLIP [55] . We also show the results using DINO [56] and CLIP [46] in Fig. 9 , showing similar trends. See Appendix D for details.Noting that this is a multi-objective problem (i.e., maximizing image similarity and image-text similarity), we report the Pareto curve consisting of scores with varying consistency guidance scale values to show the tradeoff between two scores of each model, instead of reporting scores at one operating point. If two curves overlap, two methods would likely perform similarly and the difference is up to a change in the consistency guidance scale at sampling.Averaged results are in Fig. 5a , and the results for each subject customization and stylization are in Fig. 8a and Fig. 8b , respectively. Compared to DreamBooth (DB;red), ours (DCO;blue) positions on the upper-right frontier in both image-text similarity and image similarity, demonstrating its superiority. Compared to DreamBooth with prior preservation loss (DB+p.p.; green), ours (blue) results in significantly improved image similarity, while being comparable in image-text similarity. Interestingly, it (green) does not push the frontier to the upper-right compared to the ones without it (red), but it shifts the operating point to the lower-right while lying on the seemingly similar Pareto frontier. This suggests that the use of prior preservation loss improves prompt fidelity at the cost of losing the subject consistency. In Appendix B.1, we further compare with various design choices for DreamBooth, e.g., early stopping or lowering λ prior for prior preservation loss.Experimental setup. We experiment on style images from StyleDrop dataset [11] . The examples of style images and captions are in Fig. 12 in Appendix D.1. We fine-tune LoRA of rank 64 using Adam optimizer with a learning rate of 5e-5 and do not train textual embedding. In addition, we add an offset noise [57] of 0.1 during training, which empirically helps learning the solid background color of style images. We use β t =1000 for DCO loss, and compare with DreamBooth (DB).Qualitative results. Fig. 4 shows qualitative comparisons between DreamBooth (DB) and ours (DCO). As seen in [21] , DB captures the style of a reference image, yet it suffers from overfitting to the reference image, e.g., the attributes of the subject in style reference appear in generated images.On the other hand, DCO generates images with consistent style without being entangled with contents in the reference images. We provide additional comparisons in Fig. 18 .Quantitative results. We choose 10 style images and generate stylized images using 190 text prompts excerpted from Parti prompts [5] , following [11] . We generate 2 images per evaluation prompt, resulting in 380 images in total for each style. For evaluation metrics, we report the image similarity against the style reference image and image-text similarity scores using SigLIP [55] . Nevertheless, as noted in [11] , the image similarity for style personalization is particularly noisy, as the score is not only guided by the style but also by the unexpected appearance of the subject in the style reference image (e.g., in Fig. 4 ) when the model overfits.Experimental setup. Following [11] , we combine customized subject and style models to generate images of my subject in my style. Specifically, given two LoRAs ∆W 1 and ∆W 2 for subject and style, respectively, we use an arithmetic merge (Merge) [21] , i.e., ∆W " τ 1 ∆W 1 `τ2 ∆W 2 with coefficients τ 1 " τ 2 " 1. We use subject and style LoRAs from Sec. 5.1 and Sec. 5.2, respectively, for both baseline (DB) and our method (DCO). We also compare with ZipLoRA [21] , which finds optimal coefficients τ 1 and τ 2 for each layer by jointly preserving the subject and style LoRAs. For ZipLoRA, we use DB fine-tuned subject and style models and follow the experimental setup in [21] .Qualitative results. In Fig. 6 , we provide qualitative comparisons between our approach (DCO Merge), and baselines (DB Merge and DB ZipLoRA). As noticed in [21] , DB Merge struggles to generate high-quality images when composing subject and style customized models. While DB ZipLoRA improves the image quality with fewer artifacts, it often loses subject or style consistency. Even using a simple arithmetic merge, DCO Merge (ours) generates images with high subject and style consistency. In addition, Fig. 1 and Fig. 20 in the appendix shows that DCO Merge succesfully generates my subjects in my styles under various contexts, guided by text prompts.Quantitative results. We use 30 subjects from DreamBooth [10] dataset and 10 style images from StyleDrop dataset [11] from Sec. 5.1 and Sec. 5.2, respectively. For each subject and style pair, we generate images of "A [subject] in [style]" and of various text prompts that change attributes, backgrounds, or actions (e.g., in Fig. 20 ). We compute subject similarity scores (Subject) using DINO v2 [54] , style similarity (Style), and image-text similarity (Text) scores using SigLIP [55] .Tab. 1 reports results. DCO Merge significantly outperforms DB Merge and DB ZipLoRA in subject similarity (0.462 vs. 0.386, 0.406) and image-text similarity (0.773 vs. 0.430, 0.729), while retaining competitive style similarity (0.651 vs. 0.672, 0.662). This aligns with our observation in Fig. 6 . Regularization parameter. We study the effect of the regularization parameter β t on the subject personalization task. We use 10 subjects from the DreamBooth dataset and conduct experiments with constant β " t500, 1000, 2000u. As in Fig 7a, all DCO models form a better Pareto frontier than the DB from Sec. 5.1. As β becomes larger the curve tends to move lower right, implying the loss in subject identity for better image-text alignment. This is expected as β controls the deviation between fine-tuned and pretrained models (e.g., Eq. ( 6)). We find β " 1000 works well overall, though the optimal β might vary across the reference dataset.Error analysis. One of our insights is that DCO mitigates the shift in the model's generation distribution after fine-tuning. We verify this by computing the noise distance between pretrained and fine-tuned models on reference images and prompts, i.e., }ϵ θ pz t ; c, tq ´ϵϕ pz t ; c, tq} 2 2 at each timestep t P r0, 1s. We simulate 100 random noises at each timestep and report the average value in Fig. 7b . We see a clear decrease in noise deviation from the pretrained model with DCO fine-tuning over DB. Moreover, as β increases, the noise deviation gets further reduced as expected.1-shot personalization. We demonstrate the capability of our method in 1-shot personalization. We refer to Appendix B.3 for experimental details and qualitative results are in Fig. 15 and Fig. 16 . 
conclusion: 
 We introduce a Direct Consistency Optimization (DCO), a novel training objective for robust low-shot fine-tuning of the T2I diffusion model. DCO learn new concepts by controlling the deviation from pretrained model, thus retaining capabilities of pretrained model. We show that DCO enhances the image-text alignment and sample quality of personalized T2I synthesis compared to regular diffusion fine-tuning. And together with consistency guidance sampling, DCO results in the most superior Pareto frontier in terms of image-text similarity and image similarity. Moreover, DCO induces easier composition of independently fine-tuned subject and style T2I models. Also, merging DCO models outperforms post-processing methods on regular fine-tuned models, e.g., ZipLoRA [21] . We provide limitations and broader impact statements in Appendix E and Appendix F, respectively. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The license for each data is described in Sec. D.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Derivation of Eq. (9). From Eq. ( 8), we have Derivation of Eq. (11) . Let us consider subseries z t:1 of z 0:1 for t P p0, 1q. Then, we define the deviation ∆ t pp θ , p ϕ q for z t:1 as follows:f pz 0:1 ,EQUATION)It is known that one can express the KL divergence with ϵ-prediction as follows (e.g., see Appendix A.1 in [49] ):EQUATIONThus, the following holds:EQUATION)Now it is straightforward to see that where β t " ´1 2 βλ 1 t . Remark that one can consider weighted deviation ∆ w pp θ , p ϕ q that matches the general weighted ϵ-prediction loss (i.e., Eq. ( 1)), by using monotonic weighting function w t as follows: We plot image similarity and image-text similarity for each (a) subject customization and (b) subject stylization experiment. We use SigLIP [55] score for image-text similarity, and DINOv2 [54] score for image similarity. We plot the results of consistency guidance sampling (dots and solid lines), and conventional sampling (diamond). The reported reward guidance scales are ω con P t2.0, 3.0, 4.0, 5.0u.∆pp θ , p ϕ q " ż 0 1 d dt ∆ t pp θ , p ϕ qdt " ż 0 1 1 2 λ 1 t E ϵ"N p0,∆ w pp θ , p ϕ q " ż 0 1 d dt w t ∆ t pp θ , p ϕ qdt " 1 2 E t"U p0,Here we provide full ablation studies that we have conducted in our experiments. For all 30 subjects in DreamBooth dataset, we follow the same experimental setup as in Sec. 5.1. We report the image similarity score using DINOv2 [54] , and image-text similarity score using SigLIP [55] .Ablation on early stopping. Since low-shot fine-tuning methods suffer from overfitting, it is a common practice to early stop the training. In Fig. 8b and Fig. 8b , we plot the results of DreamBooth with comprehensive caption with half of training steps (DB+ES). When compared to DB, it improves the image-text similarity (0.842 vs. 0.754 for customization, 0.645 vs. 0.468 for stylization), while the image similarity significantly drops (0.575 vs. 0.754 for customization, 0.630 vs. 0.638 for stylization). Notably, we remark that the frontier curve of DB+ES resides at the frontier of DB. Thus, early stopping does not improve the frontier.Ablation on prior preservation loss weight λ prior . In Sec. 5.1, we show that using prior preservation loss often leads to loss of subject consistency. To further verify the effect of prior preservation loss, we vary the coefficient λ prior to be 0.5 (DB+p.p. (0.5)). As shown in Fig. 8a and Fig. 8b , when compare to DB+p.p., using smaller λ prior improves image similarity (0.547 vs. 0.532 for customization, 0.441 vs. 0.424 for stylization), while decreases the image-text similarity (0.850 vs. 0.864 for customization, 0.839 vs. 0.851 for stylization). However, changing λ prior does not improve the frontier curve when using consistency guidance.Prior preservation loss vs. pretrained model. In Fig. 8a and Fig. 8b , we notice that DB with prior preservation loss (DB+p.p.) shows higher image-text similarity than pretrained model. This is in partly due to that the model is fine-tuned with class-specific prior dataset, which improves the prompt fidelity among the class. However, this does not necessarily improves the subject fidelity, and it indicates the large model shift with respect to pretrained model.Evaluation metrics. Following DreamBooth [10] , in Fig. 9 , we provide quantitative results using (a) CLIP [46] for image-text similarity (CLIP-T) and DINO [56] for image similarity (DINO), (b) CLIP for both image-text similarity and image similarity. We observe consistent trends to that which we used with SigLIP and DINOv2.User Evaluation. We conduct a user study over the results of our method (DCO), DreamBooth (DB), and DB with prior preservation loss (DB+p.p.) on subject personalization task. As in Section 5. we trained three models per subject with identical experimental setup (i.e., all models are trained with comprehensive caption and textual inversion), and generate images with the same random seeds.Then we construct two binary comparison tasks (90 comparisons) to rank between DCO and each baseline method. For each pair, 12 participants were asked to choose the preferred image on three criteria with following questions:• Subject fidelity: Which image most accurately reproduces the identity (e.g., item type and details) of the reference item? • Prompt fidelity: Which image most closely aligns with the given prompt? • Image quality: Which image exhibits the highest quality (e.g., overall visual appeal and clarity)?Overall, we collect 1,080 answers per query for each comparison pair, resulting in a total 2,160 responses. The results are summarized in Table 2 , where for each method, we report the percentage of judgements in our favor. Compared to DB, DCO shows better prompt fidelity while maintaining subject fidelity. Most users favored DCO for subject fidelity compared to DB+p.p., implying that DCO enables generating images with high subject and prompt fidelity. We compute the average cosine similarity of column layers between subject and style LoRAs fine-tuned with each DreamBooth (DB) [10] and our method (DCO). The x-axis denotes the component of each U-Net layer. The cosine similarity measures the alignment between two LoRAs, and high cosine similarity values are considered as the interference between them. Interestingly, we find that there is no obvious difference in the cosine similarity values between models trained with DCO and DB methods, while DCO fine-tuned models can be successfully combined with arithmetic merge to generate images of my subject in my style (e.g., see Fig. 6 and Fig. 19 ). This may be in contrast with the findings of recent works [21, 58] and suggests further investigation on method to measure the compatibility between LoRA models.We have shown that the user can control the subject fidelity and textual alignment by changing the consistency guidance scale. Yet, we remark that the optimal consistency guidance scale vary among the reference dataset, and even for the input prompt that the user give during inference. As shown in Fig. 14 , the optimal guidance scale should be large (e.g., 5.0) for the first row, while it should be low (e.g., 2.0) for the third row. Also, the effect of consistency guidance scale might be subtle as in second row. In practice, choosing the best consistency guidance scale is up to the user's preference.Here we provide some qualitative examples that shows the capability of our method in learning subjects with single reference image. Specifically, we show the capability of personalization with synthetic images, generated by different T2I models such as pretrained SDXL [6] and DALLE-3 [8] .We follow the same setup as in Sec. 5.1 except that we fine-tune for 1000 steps. Remark that for while we have access to the prompts that we used for generation, we did not use this prompt for fine-tuning our model; the generated image might have more details (e.g., backgrounds or attributes), or it may fail to capture all the prompts. Thus, we caption the images following Sec. 4.4.Fig. 15 shows the qualitative results of DCO fine-tuning on images generated by SDXL. Remark that our method can synthesize images into various actions and styles, while preserving the subject consistency. Notably, it is possible to convert the photographs to other styles (e.g., photo of a man to 2D animation style), and vice versa (e.g., 3D animation style of pig into photography). Also, as shown in Fig. 16 , our method is able to generate various actions, backgrounds, or styles.Training-free consistent image set generation. Several works have demonstrated the capability of consistent image set generation without fine-tuning T2I diffusion models [59, 60] . While these methods do not require fine-tuning and hence may be conceived more time-saving, they often take longer time at generation. On the other hand, fine-tuning is a one-time cost and can be used for generation without additional cost. Also, training-free methods have difficulty in putting the same subject in different styles (as mentioned as limitation in [60] ), while our approach is possible. Moreover, our method is able to combine style personalized model and subject personalized model. Lastly, our approach do not require any segmentation mask for subject personalization.Multi-concept personalization. Given multiple fine-tuned T2I diffusion models (often using LoRAs), it is of great interest to combine them to generate a scene consists of multiple personalized subjects [61, 58] , or generating custom subject in custom style [11, 21] . Those approaches often require post-optimization, e.g., orthogonal adaptation of LoRA layers [58] or optimization of merger coefficients for composition of subject and style LoRAs [21] . Those methods hypothesize that the interference between subject and style LoRAs (which is measured by the average cosine similarity between LoRA layers), and aim to minimize the interference. Our method seeks for a better training of LoRA for T2I diffusion models and thus complementary to existing works. On the other hand, as shown in Fig. 10 , we find that the cosine similarity values of DCO fine-tuned models are not necessarily smaller than those of DreamBooth fine-tuned models, while we do not observe significant interference during generation with arithmetically merged LoRAs trained with our DCO loss. This observation may suggest that we need another metric other than the cosine similarity based interference measure to evaluate the compatibility of LoRAs, which we leave as a future work.Learning textual embeddings. The rare token identifier [V], such as "sks", conveys undesirable semantics. 5 Thus, we opt to remove the rare token identifier and use the natural language captions by default. In addition, we learn textual embeddings [9] to add more flexibility in subject personalization without changing the semantics of pretrained model. Given a word or a phrase (i.e., [class]) of interest, we insert new tokens and initialize them with textual embeddings of original ones. Then, newly inserted textual embeddings are optimized with diffusion models.We use DreamBooth dataset [10] for subject personalization which contains 30 subjects, including pets and unique objects such as backpack, dogs, plushie, etc. We provide examples of image and comprehensive caption in Fig. 11 where the complete list of comprehensive captions are is available in the source code. A comprehensive caption encompasses not only the subject but also provides detailed information on visual attributes, backgrounds, and style. In contrast, a compact caption generally incorporated in model personalization [10, 9] focuses solely on the subject itself, as exemplified by "a photo of [class]". To generate comprehensive captions in practical scenarios, we initially employ LLaVA [52] to generate a description of the reference image. Subsequently, we filter out unnecessary details such as non-visual attributes and make further modifications. Similarily, we use 10 images from StyleDrop dataset [11] for style personalization, where examples are presented in Fig. 12 .License. The license for DreamBooth dataset can be found in here. Also, the attributes for style images can be found in StyleDrop [11] paper as well as here.We construct two types of evaluation prompts; (1) subject customization, and (2) subject stylization.In evaluating subject customization, we provide the textual prompts that alter the attributes of the subject (e.g., "cube-shaped") or its background (e.g., "on the beach") following DreamBooth [10] . In evaluating subject stylization, we provide the textual prompts that stylizes the subject into different styles (e.g., "in watercolor painting style", "in origami style"). For fine-grained evaluation, we construct different prompts for each object (e.g., "clock", "robot toy") and subject (e.g., "cat", "dog", "wolf plushie"). The examples of evaluation prompts for each category and type are in Table 3 ."A photo of a dog sitting on a ledge in front of red wall and tree, macro style"Reference "The image features a cute, fluffy, and white dog sitting on a ledge or a wall, possibly a stone wall. The dog appears to be enjoying the sunlight and is smiling, with its tongue out. The dog's fur is well-groomed, and it appears to be a well-cared-for pet. In the background, there are some trees, adding a natural element to the scene. The overall composition of the image is a close-up of the dog, with the background providing context and depth to the scene.""A photo of a sloth plush sitting on a concrete ledge near the lake, cityscape with buildings and boats in the background, natural lighting style""The image features a brown teddy bear sitting on a concrete ledge near a body of water. The teddy bear is positioned in the foreground, occupying a significant portion of the scene. The background showcases a cityscape with buildings and boats visible in the distance. The combination of the teddy bear and the cityscape creates a unique and interesting contrast between the innocence of the toy and the urban environment."The image features a red vase with a plant in it, placed on a table. The vase is positioned in the center of the scene, and the plant appears to be a potted plant. The table is black, and the vase is the main focus of the image. In the background, there is a doorway visible, which adds depth to the scene. Additionally, there is a bowl on the table, located to the right of the vase. The overall composition of the image creates a sense of balance and harmony between the vase, plant, and the doorway in the background.""A photo of vase with plant in it and a cup, placed on a dining table, door in the background, natural lighting style" We use PyTorch and Huggingface Diffusers library6 for our codebase. Each training is performed on a single A100 40GB GPU using a batch size of 1. We perform up to 2000 optimization steps. Note that our approach is robust to the length of training steps compared to the baseline (i.e., DreamBooth), which often requires early stopping to prevent overfitting. We fine-tune LoRA of rank 32 for subject personalization and rank 64 for style personalization. For sampling, we use DDIM [42] scheduler with 50 steps, and use CFG guidance scale of 7.5 throughout experiments.To measure the image similarity, we use DINOv2 [54] score, which is given by the mean cosine similarity between the embeddings of reference images and synthesized images. To measure the image-text similarity, we use SigLIP [55] score, which is defined asS IT px, cq " 1 1 `exp `´f img pxq J f text pcq `bwhere f img and f text are ℓ 2 normalized embeddings from image and text encoders, and b is a bias term that is optimized during pretraining. We opt to use SigLIP score instead of CLIP score [46] , as the range of CLIP score depends on the prompt and images, while SigLIP score provides a general score that is bounded on r0, 1s. For style personalization experiment, we measure image similarity using SigLIP image similarity, by computing the embeddings with SigLIP image encoder. While we desire Table 3 : Examples of evaluation prompts used to synthesize images for each object and live subject category. Subject customization are prompts to generate novel views of photo-realistic images and subject stylization aims to alter style of the subject. {}'s are filled with the class token of the subject.Object "A photo of {} on the beach" "A {} in sticker style" "A photo of {} with ribbons" "A {} in wooden sculpture" "A photo of cube-shaped {}" "A {} in flat cartoon illustration style" "A photo of golden {}" "A {} in pixel art style" "A photo of {} made out of leathers" "A {} in wireframe 3D style" "A photo of {} with a tree and autumn leaves in the background" "A {} in hygge style" "A photo of {} on top of a white rug" "A {} in geometric art style" Subject "A photo of {} wearing a spacesuit, planting a flag on the moon" "A {} playing a violin in sticker style." "A photo of {} as a firefighter, extinguishing a fire in a skyscraper" "A {} carved as a knight in wooden sculpture" "A photo of {} in a wetsuit, surfing a giant wave in the ocea" "A {} piloting a hot air balloon in travel agency logo style" "A photo of {} in Victorian attire, attending a tea party in an elegant garden" "A {} constructed from abstract metal shapes in constructivism style" "A photo of {} in a snowsuit, skiing down a steep mountain" "A {} on an epic quest in pixel art style" "A photo of {} as an explorer, navigating through an icy Arctic landscape" "A {} designed as an intricate machine in blueprint style" "A photo of {} in an elegant masquerade mask at a Venetian ball" "A {} illustrated in an educational infographic style" high scores, these metrics are not perfect, e. ) on multi-subject composition, the learned concepts are often mixed during generation (e.g., the dogs are mixed in third row). We found this more frequently happens when learned concepts are similar to each other, e.g., dog-dog composition is more challenging than dog-cat composition, and dog-cat composition is more challenging than dog-toy composition.Computational efficiency. Since our method leverages inference on pretrained model during training (e.g., Algorithm 2) and inference (when using consistency guidance sampling) (e.g., Eq. ( 13)), there exists a few extra computational burden compared to original DreamBooth fine-tuning or CFG sampling. Specifically, we measured the optimization and inference time per iteration. Compared to DreamBooth, our method (DCO) approximately takes ˆ1.3 longer time in fine-tuning. Compared to CFG sampling, consistency guidance sampling requires ˆ2 longer time in sampling. We believe that our work will motivate future studies on efficient fine-tuning to enhance scalability in practical scenarios.Multi-subject composition. We have shown that DCO effectively compose learned subject and style (i.e., my subject in my style). When composing multiple subject, we found the performance is not stable. Fig 13 showcase some results in multi-subject compositional generation. We found that for subjects that are semantically distant, e.g., dog-car toy or dog-cat composition, our method is able to generate multi-subject consistent images, while DreamBooth or DreamBooth+p.p. often fails to. However, when composing semantically similar subjects, e.g., dog-dog composition, we found that our method fails as well. Specifically, the concepts are mixed during generation, which results in subject inconsistency. We believe this is partly due to the lack of model's capability in disentangling the newly learned concepts that are semantically similar. Thus, we believe using better pretrained model could ameliorate such issues.This paper presents a method that enhances the performance of the personalization of T2I diffusion models. Similarly to other works, the technology for personalization of T2I diffusion models comes with benefits and pitfalls -the tool could be extremely effective for creative directors to efficiently generate new visual assets of various subjects or styles derived from existing private visual assets. Yet, the responsible use of the technology is required for protecting the ownership and copyright of individual assets. We show the capability of our method in 1-shot subject personalization using the images generated by pretrained SDXL models. For each reference image (man and pig), DCO fine-tuned T2I models can generate subjects with different actions and styles. The prompts that used to generate reference images were "a photo of a 50 years old man with curly hair" and "a 3D animation of happy pig", respectively, as used in [62] . We show the capability of our method in 1-shot subject personalization using the images generated by DALLE-3 [8] . We asked DALLE-3 to generate a cute baby otter image. The comprehensive caption to fine-tune this image is "A closed-up photo of an <otter> on the top of wooden log, forest in the background". Our method is able to recontextualize the subject in the reference image with various text prompts depicting accessories, background, or style. ). Note that images in each row are generated using the same random seed. Our method is able to generate subject consistent images with various accessories, background or style, with better image-text alignment than baseline methods.Reference A {Christmas tree, butterfly, piano} in melting golden 3D rendering style DB DCO (ours)A {butterfly, coffee maker, piano} in watercolor painting style Figure 18 : Custom style generation. Additional qualitative results on custom style generation. Our method (DCO) is able to generate style consistent image, while prior method, DreamBooth (DB), is prone to overfitting. For example, in the first row, the leaves of flower is inherited to butterfly, Christmas tree, and piano in DB, while our methods disentangle such attributes in generation. Those are also shown in second and third rows. On the other hand, DreamBooth Merge (DB Merge) shows inferior results as is either overfitted to subject (e.g., sloth plush, wolf plush, backpack are not in cartoon line drawing style), or styles (e.g., monster toy, teddy bear, dog do not appear in kid crayon drawing style). Meanwhile, ZipLoRA shows better results than DB Merge, yet it often loses the subject or style fidelity. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Since λt is a decreasing function of t, λ 1 t ă 0. Thus, we use βt " ´1 2 βλ 1 t to enusre βt ą 0.See this blog post as an example.See second and fourth rows of DB (baseline) in Fig.17, where a dog is surrounded by guns and a monster toy is holding a gun. Note that this is aligned with the existing findings in community.https://github.com/huggingface/diffusers 


Decision: In this work, authors propose an algorithm called direct consistency optimization where the core idea revolves around an objective that minimizes the deviation between the fine-tuning and pre-trained models by optimizing a consistency function. Overall, the reviewers tend to like the novelty present in this idea and are also encouraged by the potential amount of impact this algorithm can have. There are some concerns over computational cost increase, however, in my opinion, that is not prohibitively expensive for this algorithm to be adapted. Overall, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper studies the catastrophically forgetting issue in personalizing text-to-image diffusion model. The main comparable baseline is DreamBooth, which prevents forgetting by finetuning the model on a subsample of original training data while learning new concepts. The proposed method, on the other hand, directly regularizes the deviation of reference model from original model, which resulting a new objective function that essentially penalizing overfitting to the ‘hard examples’ during personalization. Empirical evaluations are conducted on subject and style transfer datasets, where the proposed method achieves superior pareto optimality on image fidelity v.s. prompt following ability, as well as better visual results compared with DreamBooth baseline.
strength_weakness: strength_weakness: The empirical evaluation can benefit from extra validations.

1. Lacks human evaluation. The paper could benefit from a proper human study from third-party judgers. Currently, the empirical comparisons seem to be confined by automatic evaluation and a subset of visualizations.
2. Compounded factors are not studied separately. The Textual Inversion and comprehensive captioning techniques are deployed on top of DB and DCO. With these enabled, it is unclear what might be the interplay between these techniques and different baselines. For instance, the compatibility of DB might be less compatible with DB than DCO, since DCO only requires the reference data during training. These ablations would not necessarily diminish the proposed method; They are just very beneficial for research purposes IMO.
questions: questions: 1. At a high level, both DCO and DBpp are trying to learn the concept while keeping certain aspects of the new model closer to the original model. It seems (e.g. line 39 and Figure 2) that the motivation for using is based primarily on empirical performance. Could the author elaborate more on the intuition of why DBpp hurts the subject fidelity?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a novel fine-tuning objective called Direct Consistency Optimization, which regulates the deviation between fine-tuning and pre-trained models to preserve pre-trained knowledge during the fine-tuning process. Models fine-tuned using this method can be merged seamlessly without interference.
strength_weakness: strength_weakness: 1. The comparison with Lora is missing. Usually the results of Lora customization are very competitive.
2. In fig1, the pose of the bear in different images are mostly the same as those in the reference image, which is overfitting. It needs to compare the diversity of generated images.
questions: questions: As seen in weakness.
limitations: limitations: The manuscript includes discussions on limitations and social impacts.
paper_summary: paper_summary: Current personalized T2I models, like DreamBooth, are capable of generating personalized images by fine-tuning on a small set of reference images. However, these fine-tuned models often suffer from robustness issues, such as poor compositional capabilities with the pretrained model concepts and other fine-tuned models.

The paper "Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models" introduces a novel fine-tuning method called Direct Consistency Optimization (DCO) to enhance the robustness of such personalized text-to-image (T2I) diffusion models.

DCO introduces a novel training objective that minimizes the deviation between the fine-tuning and pretrained models by optimizing a consistency function. The method involves computing a loss function that measures the deviation in noise prediction errors between the fine-tuned and pretrained models, thus allowing efficient implementation using a noise-prediction loss approach.
strength_weakness: strength_weakness: * Incremental Improvement: While the DCO method introduces a novel fine-tuning objective, the actual improvement seem marginal (based on figure 4 and figure 7)

* Limited Diversity of Experiments: The experiments primarily focus on subject and style personalization within specific datasets. A broader range of experiments, including diverse set of human images. 

* Reproducibility Concerns: Although the methodology is well-explained, the paper does not provide detailed information on implementation specifics such as the codebase, computing resources, or detailed training schedules. This might pose challenges for reproducibility and practical adoption by the community.
questions: questions: have you thought about trying a kandinsky2.2/dalle2 like model which can directly condition on the image embedding for a (or multiple) reference images?
limitations: limitations: yes its adequately address in line 632
paper_summary: paper_summary: The paper introduces a novel training framework and novel sampling technique to enhance the state-of-the-art personalized and style-preserving adaptation of text-to-image diffusion models in the low-shot fine-tuning regime.
strength_weakness: strength_weakness: * DCO is computationally expensive since the proposed sampling scheme makes use of two networks during inference. However, the authors also mention it explicitly in the limitations section. But it would be nice to be forthright about it in the paper's main text. 
* Multiple references to existing works are missing. Examples include SVDiff [1], Custom Diffusion [2], IP-Adapters [3], and, DiffuseKrona [4] (for controllable personalized generation), and MaPO [5] (reward fine-tuning through alignment). It also didn't include references to works that successfully applied LoRA in the context of adapting text-to-image models [6], [7], [8]. 
* The paper's main text reads incomplete without the minimum details on the datasets being used for training and evaluation. For example, in lines 219 - 224, the reader doesn't know what reference images were used. Similarly, in lines 251 - 254, there is no mention of Appendix D, which sheds light on more details. This is also the case for the text accompanying Table 1 (Lines 280 - 287). 
* While reporting the results of Figure 4, it would have been better to talk about $\omega_{text}$ as well because that substantially impacts the generation quality. From the accompanying text, the reader doesn't know how it was varied or if it was kept fixed.

## References

[1] SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, https://arxiv.org/abs/2303.11305.

[2] Multi-Concept Customization of Text-to-Image Diffusion, https://arxiv.org/abs/2212.04488. 

[3] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models, https://arxiv.org/abs/2308.06721. 

[4] DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models, https://arxiv.org/abs/2402.17412. 

[5] Margin-aware Preference Optimization for Aligning Diffusion Models without Reference, https://arxiv.org/abs/2406.06424.

[6] Implicit Style-Content Separation using B-LoRA, https://arxiv.org/abs/2403.14572. 

[7] https://github.com/cloneofsimo/lora

[8] https://huggingface.co/blog/lora
questions: questions: ## Suggestions 

* Figure 1.B is not obvious when conveying the limitations. Perhaps mention that the subject is not faithfully preserved (hat is missing) and the style is not faithfully preserved (blue background is missing) in the case of DreamBooth. 
* Line 97 - 98: It would be better to also mention how training with classifier-free guidance is implemented in practice, i.e., we randomly make captions associated with images to null (caption dropout). 
* Since the authors use LoRA for fine-tuning, I think the memory overhead could still be kept to a minimum if they enable and disable the adapter layers within the denoiser network. See how it's implemented [here](https://github.com/huggingface/diffusers/blob/3fca52022fe0ea9aaf0a0ea8a0fc13308bf69a9f/examples/research_projects/diffusion_dpo/train_diffusion_dpo_sdxl.py#L1019) and [here](https://github.com/huggingface/diffusers/blob/3fca52022fe0ea9aaf0a0ea8a0fc13308bf69a9f/examples/research_projects/diffusion_dpo/train_diffusion_dpo_sdxl.py#L1035). This probably should improve the memory requirements quite a bit. Could the authors verify this approach? 
* It would be nice to enlist and cite the libraries that were used in the codebase. They do enable productive research and citing them goes a long way. 
* Provide a loss landscape comparison when varying the amount of DCO $\beta$. 
* Highlight the highest scores reported in Table 1. 
* If both GPT-4 and LlaVA were used for captioning, how were the combined captions used during training? 

## Questions

* If there's a connection between the guidance scheme introduced in InstructPix2Pix [1] and the one introduced in DCO, it would be nice to have that discussed. 
* Line 204: It's not clear what do the authors mean by "omitting. 
* For the DreamBooth and DreamBooth + PP experiments, do the authors use LoRA? If so, the text is not clear. 
* Was consistency sampling applied for the DreamBooth experiments? 
* Perhaps refer to the combination of LoRA DreamBooth + Textual Inversion as "pivotal tuning" as commonly referred in the community [2]?
* With a higher value of $\omega_{con}$, I would assume that the image similarity would improve. But with increasing $\omega_{con}$, it seems to be dropping off as per Figure 4. Or am I looking at it wrongly? 
* Did the authors experiment with other merging techniques geared toward LoRAs [3]? 

## Nits

* "In specific" could be replaced with "specifically". 
* Line 46: "... policy optimization problem [19, 20], which amounts _for_ the consistency ..."
* Line 75: "_To the best of our knowledge,_ this paper is the first to have studied consistency as a reward function for T2I personalization."
* Line 83: Shouldn't $\lambda_{1}$ be $\lambda_{t}$? Similar comments for $\lambda_{0}$. 

## References

[1] InstructPix2Pix: Learning to Follow Image Editing Instructions, https://arxiv.org/abs/2211.09800

[2] https://replicate.com/cloneofsimo/lora-training/readme

[3] https://huggingface.co/blog/peft_merging
limitations: limitations: The authors are clear about limitations. Perhaps mentioning them a bit from the main text would be more sensible.

Comments: title: Reply to authors
comment: Thank you for your positive feedback and for updating your ratings. We appreciate your suggestion and will incorporate the details of the additional experiments in our final revision. Thank you once again for your valuable input!
Thank you for the responses, they addressed my concerns. I've improved my ratings accordingly. I suggest the authors to include the new experiments in the revision.
I ack the rebuttal
I acknowledge reading your comment, and I wish you all the best!
Thank you for your positive feedback and for raising the score. We appreciate your thoughtful comments and are glad the additional experiments were helpful.

Regarding [W2], we will ensure to mention and discuss the relevant works that align with our study, as they indeed target customization of T2I diffusion models at their core.

For [S3], we are pleased that the clarification was useful. We will update the technical implementation in our code release to reflect this.

In response to [S5], we will include the training loss landscape in the appendix of the main draft, as we agree it will be beneficial for practitioners to understand convergence.

For [S7], we will clarify this point further in the main text to enhance clarity.

Thank you once again for your valuable feedback, which has significantly improved the quality of our work. If you have any further comments or suggestions, please do not hesitate to share them.
Thank you for comprehensively addressing my comments and running those experiments. Hope they were not too much trouble. I only have minor comments. Furthermore, I have raised my score to 6 from 5 after looking at your comments and rebuttal. All the best! 

[W2] I still think they at least deserve a mention. All of them target customization of T2I diffusion models at their very core. Maybe pick the ones that you think are most relevant and aligned. 

[S3] Glad it helped. 

[S5] Would it be included in the appendix of the main draft? I think it will be helpful for practitioners as it gives an idea about convergence. 

[S7] Thanks for the clarification. Perhaps this could be clarified in the main text.