Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics
Authors: Anton Voronov
Mikhail Khoroshikh
Artem Babenko
Max Ryabinin
Venue: NeurIPS 2023 poster (2023)
Keywords: text-to-image generation
diffusion models
early stopping
TL;DR: We propose a simple yet efficient general criterion for early stopping in personalized text-to-image synthesis.

Abstract: Text-to-image generation models represent the next step of evolution in image synthesis, offering a natural way to achieve flexible yet fine-grained control over the result.
One emerging area of research is the fast adaptation of large text-to-image models to smaller datasets or new visual concepts.
However, many efficient methods of adaptation have a long training time, which limits their practical applications, slows down experiments, and spends excessive GPU resources.
In this work, we study the training dynamics of popular text-to-image personalization methods (such as Textual Inversion or DreamBooth), aiming to speed them up.
We observe that most concepts are learned at early stages and do not improve in quality later, but standard training convergence metrics fail to indicate that.
Instead, we propose a simple drop-in early stopping criterion that only requires computing the regular training objective on a fixed set of inputs for all training iterations.
Our experiments on Stable Diffusion for 48 different concepts and three personalization methods demonstrate the competitive performance of our approach, which makes adaptation up to 8 times faster with no significant drops in quality.

Is This Loss Informative? Faster Text-to-lmage Customization by Tracking Objective Dynamics
Anton Voronov* MIPT, Yandex
Mikhail Khoroshikh HSE University, Yandex
Artem Babenko HSE University, Yandex
Max Ryabinin" *
HSE University, Yandex
Abstract
Text-to-image generation models represent the next step of evolution in image synthesis, offering a natural way to achieve flexible yet fine-grained control over the result. One emerging area of research is the fast adaptation of large text-to- image models to smaller datasets or new visual concepts. However, many efficient methods of adaptation have long training time, which limits their practical applications, slows down experiments, and spends excessive GPU resources. In this work, study the training dynamics of popular text-to-image personalization methods (such as Textual Inversion or DreamBooth), aiming to speed them up. We observe that most concepts learned early stages and do not improve in quality later, but standard training convergence metrics fail to indicate that. Instead, we propose simple drop-in early stopping criterion that only requires computing the regular training objective on a fixed set of inputs for all training iterations. Our experiments Stable Diffusion for 48 different concepts and three personalization methods demonstrate the competitive performance of our approach, which makes adaptation up times faster with no significant drops quality.
Introduction
Large text-to-image synthesis models have recently attracted the attention of the research community due to their ability to generate high-quality and diverse images that correspond to the user's prompt in natural language 2 22333 4 5]. The success of these models has driven the development of new problem settings that leverage their ability to draw objects in novel environments. One particularly interesting task is personalization (or adaptation) of text-to-image models to a small dataset of images provided by the user. The goal of this task is to learn the precise details of a specific object or visual style captured in these images: after personalization, the model should be able to generate novel renditions this object in different contexts or imitate the style that was provided as an input.
Several recent methods, such as Textual Inversion (6). DrcamBooth (7]. and Custom Diffusion (8] offer easy and parameter-efficient personalization of -oxt-to-mmaee models. Still, major obstacle on the path to broader use of such methods is their computational inefficiency. Ideally, models should adapt user's images in real or close to real time. However, as noted by the authors of the first two papers, the training time of these methods can be prohibitively long, taking up to two hours for single concept. The reported training time for Custom Diffusion is minutes per concept on a single GPU, which is much faster but still outside the limits of many practical applications.
Eqqul contribution. Correspondence mryyabinin0@mmail.ccm com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

0,18100
0.10
2/3 iterations
0.18077
are not necessary!
0/08
0.18050
Cradlrur
0.18025
0.04
0.2
0.18001
0.02
0,17975
0.0
0.00
Stepe
Steps
Stepp
Both images are taken from the same training run.
Using the deterministic loss Lae as a convergence indicator, we can complete Textual Inversion much carlier without drops in quality.
A summary of our key findings: the quality of methods like Textual Inversion saturates early on, but the training loss does not indicate that. Evaluating the same loss batch of data fixed throughout the run makes the training dynamics significantly more informative.
Figure I:
In this work, we seek answers to the following questions: are text-to-image customization tech- niques inherently time-consuming and can decrease their runtime without major quality tradeoffs? Focusing on the training dynamics, we observe that the CLIP [9] image similarity score (often used assess image quality in such tasks (TO] grows sharply only in the early stages of Textual Inversion and hardly improves after that. However, as shown in Figure neither the training loss nor the gradient norm indicate the convergence of the concept embedding, which prevents us from stopping the adaptation process earlier. While it is possible to score samples from the model with CLIP during training, generating them can take quite long time.
With that in mind, we study the training objective itself and attempt to understand the reasons of its non-in non-informative dynamics. As we demonstrate, the primary cause lies in several sources of stochasticity (such as diffusion time steps or the diffusion noise) introducing noise to the loss function. If we sample random variables only once and fix them across training steps, the loss for these inputs reflects convergence hetter even when training with the original (fully stochastic) objective.
Motivated by this finding, propose Deterministic VA Riance Evaluation (DVAR), an early stopping criterion for text-to-image customization methods that generates a single fixed batch of inputs at the beginning of training and evaluates the model on this batch after each optimization step. This criterion straightforward to implement, has two interpretable hyperparameters. and its stopping time corresponds to convergence in terms of the CLIP image score. We validate DVAR by comparing with a range of baselines on three popular adaptation methods, showing that is possible to run methods like Textual Inversion and DreamBooth much faster with up to 8x speedups and little to no decline in quality. For Custom Diffusion, our method recovers the optimal number of training iterations, which makes it useful avoid empirical tuning of the step count for every specific dataset.
In summary, our contributions are as follows:
We investigate the difficulties in detecting the convergence of text-to-image adaptation methods from training-time metrics. As we demonstrate, the objective becomes much more interpretable across iterations if computed on the same batch of inputs without resampling any random variables.
We propose DVAR, simple early stopping criterion for text-to-image personalization techniques. This criterion easy to compute and use, does not affect the training process and correlates with convergence in terms of visual fidelity.
We compare this criterion with multiple baselines on Stable Diffusion v1.5. popular text-to-image model, across three methods: Textual Inversion. DreahBooth, and Custom Diffusion DVAR offers significant decrease in runtime while having quality comparable to both original methods and other early stopping baselines.
'The code of our experiments available githhb. htt.hb.-ccm/yaaeee..eeeerr/////

---

2 Background
2.1 Denoising Diffusion Probabilistic Models
Diffusion models are class of generative models that has become popular in recent years due to their ability to generate both diverse and high-fidelity samples [2].. They approximate the data
distribution through iterative denoising of variable 2 sampled from Gaussian distribution. In simple words, the model Eg is trained to predict the noise e, following the objective below:
min Exo e- e- --
c,
(1)
Here, 2e corresponds
Markov chain forward process ze Zz e) vaizg + v1-vllior OyE that starts from a sample 2o of the data distribution. For example, in the image domain, 2y corresponds to the target image, and 2T is its version with the highest degree of coruuption. In general, c can represent the condition embedding from any other modality. The inference (reverse) process occurs with a fixed time step and starts from 2, equal to sample from the Gaussian distribution.
2.2 Text-to-image generation
source of guidance for generative models is the natural language [3 14 [5] because of its convenience for the user, case of collecting training data, and significant improvements in text representations over the past several years [5]. The condition vector is often obtained from Transformer [16] models like BERT 17 applied to the input text.
The most natural and
Mfateoof-the-ar text-to-image models perform forward and reverse processes the latent space of an autoencoder model = E(r), = D(=o), where is the original image, and E, D are the encoder and the decoder, respectively. We experiment with Stable Diffusion v1.5 [ which uses variational autoencoder [18) for latent representations. Importantly, this class of autoencoder models is not deterministic, which makes inference even more stochastic but leads to a higher diversity of samples. In total, given image and caption distributions X, Y. the training loss can be formulated as:
Lirain Ey.,,elle eo (E(xx) c(y)) 1)12 y ~ Y, X, Ni U
(3)
2.3 Adaptation of text-to-image models
While rext-to-image generation models are flexible because of the natural language input, it is often difficult to design prompt that corresponds to an exact depiction of an object of choice. Hence, several methods for adapting such models to given collection of images were developed. In this work, we focus on the most well-known methods, discussing others Appendix [A]
Most approaches inject the target concept into the text-to- imoiemaee model by learning the representation of a new token è inserted into the language encoder vocabulary. The representation learned by optimizing the reconstruction loss from Equation 2 for a few (typically 3 - 5) reference images with respect to the embedding of o, while other parts of the model are frozen. The main advantage of these methods is the ability to flexibly operate with the "pseudo-wow word" e in natural language, for example, by placing the concept corresponding to that word into different visual environments.
Textual Inversion [6] is the simplest method for adaptation of text-to-image models that updates only the token embedding for é. While this method is parameter-efficient, its authors report that reaching acceptable inversion quality requires 6000 training iterations, which equals Rx2 GPU hours on most machines. This number is conservative estimate of the number of steps sufficient for all concepts: authors of the method propose selecting the earliest checkpoint that exhibits sufficient train image reconstruction quality" H
While Textual Inversion only learns the embedding of the target token, DreamBooth 7 does the same with fully unfrozen U-Net component, and Custom Diffusion [ updates the projection matrices of cross-attention layers that correspond to keys and values. These methods use a significantly smaller number of iterations, 1000 and 500 respectively, which is also not adaptive and can lead to overfitting.
200../sseooommmme-11-33333333333 1230831673

---

3 Understanding adaptation dynamics
As we explained previously, the goal of our study is to find ways of speeding up text-lo-image without significantly degrading the quality of learned concept representations. To accomplish this goal, we analyze the optimization process of running Textual Inversion on a given dataset. As we demonstrate in Section and in Appendix [B] the same findings hold for DreamBooth and Custom Diffusion, which makes our analysis broadly applicable to different methods for customized text-to-image generation.
We apply Textual Inversion to concepts released by Gal et al. 6 using Stable Diffusion v1.5 as the base model. We monitor several metrics during training:
1. First, one would hope to observe that optimizing the actual training objective would lead to its convergence, and thus we track the value of Ltrain.
2. Second, we monitor the gradient norm, which is often used for analyzing convergence in non-convex optimization. As the model converges to local optimum, the norm of its gradient should also decrease to zero or stabilize at a point determined by the gradient noise.
3. Lastly, every 50 training iterations, we generate 8 samples from the model using the current concept embedding and score them with the CLIP image similarity score using the training set as references. In the original Textual Inversion paper, this metric is named the reconstruction score and is used for quantitative evaluation.
Note that we do not rely on the CLIP text-image score for captions: in our preliminary experiments, we observed no identifiable dynamics for this metric when using the entire set of CLIP caption templates. Writing more specific captions and choosing the most appropriate ones for each concept takes substantial manual effort; hence, we leave caption alignment out of the scope for this study.
3.1 Initial observations
First, we would like view the training dynamics in terms of extrinsic evaluation: by measuring how the CLIP image score changes throughout training, we can at least estimate how fast the samples begin resemble the training set. For this, we perform inversion of all concepts released by (6] 7 (a total of 48 concepts): an example of such an experiment for one concept is available in Figure
From these experiments, we observe that the CLIP image score exhibits sharper growth at early point of training (often within the first 1000 iterations) and stabilizes later. This finding agrees with the results of our visual inspection: the generated samples for most concepts undergo the most drastic changes at the beginning and do not improve afterwards. Practically, this observation means that we can interrupt the inversion process much earlier without major drawbacks if we had criterion for detecting its convergence. What indicators can we use to create such a criterion?
The most straightforward idea is to consider the training loss Lrrain Unfortunately, it is not informative by default: as we demonstrate in Figure 2 the loss exhibits too much noise and has no signs of convergence. The gradient norm of the concept embedding is also hardly informative: in the same experiment, we can see that it actually increases during training instead of decreasing. As show in Appendix [ these findings generally hold even for much larger training batches, which means that the direct way of making the loss less stochastic is not practical. Still, as reported in the original work and shown by samples and their CLIP scores, the model successfully learns the input concepts. Curiously, we see no reflection of that in the dynamics of the training objective.
Another approach to early stopping is to leverage our observations about the CLIP image score and measure during training, terminating when the score fails to improve for a specific number of iterations. However, there are two downsides to this approach. First, frequently sampling images during training significantly increases the runtime of the method. Second, this criterion can be viewed as directly maximizing the CLIP score, which is known to produce adversarial examples for CLIP instead of actually improving the image quality 2
We chose this version because it was given default in the Diffusers 19 example for Textual Inversion.

---

0.40
CLI
0.00
500
1000
1500
2000
2500
1000
1500
0.12
0.00
0.12
Goodiett
0.04
0.12
0.11
0.00
1000
1500
2000
2900
1300
2000
2500
Sumples
Cenceps
Step
Step 2001
Step 1000
Step 2500
Figure 2: An overview of the convergence process for Textual Inversion with example concept. Figures and Appendix B present more examples.
3.2 Investigating the sources of randomness
We hypothesize that the cause of excessive noise in Ltrain is several factors of randomness injected each training step, as we mentioned previously in Section 2.2 Thus, we aim to estimate the influence of the following factors on the dynamics of the inversion objective:
1. Input images
2. Captions corresponding to images
3. VAE Latent representations for images (x))
4. Diffusion time steps
5. Gaussian diffusion noise
Now, our goal is identify the factors of stochasticity that affect the training loss. Importantly, do not change the training batches, as it alters the objective of Textual Inversion and might affect its outcome. Thus, we train the model in the original setting (with batches of entirely random data) but evaluate it on batches with some sources of randomness fixed across all iterations. Note that (xx) depends on æ: we resample the input images, also need to resample their latent representations.
First, we try the most direct approach of making everything deterministic: in other words, we compute fdcet which is the same Ltrains but instead of the expectation over random data and noise, compute it on the same inputs after each training step. Formally, we can define as
Laet || (E c), c(v), t)
(4)
with æ, y. E(x), f, and sampled only once in the beginning of inversion. Essentially, this means that the only argument of this function that changes across training iterations is c(y) that depends on the trained concept embedding.
As we show in Figure 2 this version of the objective becomes informative, indicating convergence across a broad range of concepts. Moreover, it displays approximately the same behavior as the CLIP score and is much less expensive to compute, which makes Ldet particularly useful as a metric for the stopping criterion.
For the next step, we aim find if any of the above sources of stochasticity have negligible impact on the noise in Ltrain or can be compensated with larger batches. We evaluate them separately and provide results Section 4.5 Our key findings are that (1) resampling captions and VAE encoder noise still preserves the convergence trend, (2) using random images or resampling diffusion noise reveals training dynamics only for large batches, and (3) sampling different diffusion timesteps leads non-informative training loss regardless of the batch size. Still, for the sake of simplicity and efficiency, we compute Late on batch of inputs sampled only once at the beginning of training for the rest of our experiments.
5

---

def DVAR(1osses, window_size threshold):
running_var losses[-wing <-inoosssiiddsssee]] var total_var losses var ()
ratio running_var total_var return ratio threshold
Figure 3: An example NumPy/PyTorch implementation of DVAR. See Appendix D for an example of its usage in the training code.
3.3 Deterministic Variance Evaluation
The results above show that fixing all random components of the textual inversion loss makes its dynamics more interpretable. To achieve our final goal of decreasing the inversion runtime, we need to design an early stopping criterion that leverages Let to indicate convergence.
We propose Deterministic Variance Evaluation (DVAR), a simple variance-based early stopping criterion. It maintains rolling variance estimate of Latt over the last N steps, and once this rolling variance becomes less than of the global variance estimate (0 < < 1), we stop training. A
pseudocode implementation of DVAR shown Figure
This criterion is easy to implement and has two hyperparameters that are casy to tune: the window size for local variance estimation N and the threshold In experiments, found {N 310, a 0.15] for Textual Inversion, {N = 440, a = 0.4) for DreamBooth, and {N = 180, a = 0.15) for Custom Diffusion work relatively well across all concepts we evaluated.
Importantly, we use this criterion while training in the regular fully stochastic setting: our goal is not to modify the objective, and using fixed random variables and data can affect the model's generalization capabilities. As we demonstrate in Section ! our approach demonstrates significant improvements when compared to baselines, even when all sources of randomness are fixed.
Along with DVAR, we tested other early stopping strategies that use Ldet and are based on different notions of loss value stabilization, such as estimating the trend coefficient or tracking changes in the mean instead of variance. As we show Appendix most of them result in less reliable convergence indicators and have hyperparameters that do not transfer as well between different image collections.
4 Experiments
In this section, we compare DVAR with several baselines in terms of sample fidelity for the learned concept, output alignment with novel prompts, and training time. Our goal is to verify that this early stopping criterion is broadly applicable and has little impact on the outcome of training.
4.1 Setup and data
We evaluate approaches to early stopping on three popular text-to-image personalization methods: Textual Inversion, DreamBooth with LoRA (20), and Custom Diffusion, all applied to Stable Diffusion v1.s9 For Textual Inversion, we use the implementation from the Diffusers library 19 and take the hyperparameters from the official repository We use the official code and hyperparameter for Custom Diffusion, and we use the implementation of DreamBooth-LoRA from the Diffusers library"
For evaluation, we combine the datasets published by authors of the three techniques above that were available as of March 2023, which results total of 48 concepts. Our main results are obtained by applying each method to all of these concepts, which in total took around 80 GPU hours. Each experiment used a single NVIDIA A100 80GB GPU. Although the methods we study can be used with several concepts Following Equation 2 the training batches are generated from a small set of original images with randomly sampled augmentations (central/horizzonall crop), captions, timesteps and diffusion noise.
time, we train the images of each concept separately,

---

serve an estimate of the minimum time required To automatically assess the quality of generated images, we employ CLIP image-to-ing image similarity for images generated from prompts used during training ("Train CLIP img"), which allows us to monitor the degree to which the target concept is learned. However, relying solely on this metric can be risky, as does not capture overfitting. To measure generalization, we utilize CLIP text-to-image similarity, evaluating the model's ability to generate the concept in new contexts with prompts that are not seen at training time ('Val CLIP txt"). For each method, we report both the number of iterations and the average adaptation runtime, since the average duration of one iteration may differ due to intermediate image sampling or additional loss computation. In addition, we evaluate the identity preservation and diversity of all methods in Appendix F
4.2 Baselines
We compare DVAR with several baselines: the original training setup of each adaptation method (named as "Baseline"), early stopping based on the CLIP similarity of intermediate samples the training set ("CLIP-s"), as well as the original setup with the reduced number of iterations and no intermediate sampling ("Few iters").
Specifically, the original setup runs for a predetermined number of steps (6100, 1000, and 500 for Textual Inversion, Dreambooth-Lokf 500/100/100 iterations and computing their CLIP similarity to the training set images. For each cus- tomization method, tune the hyperparameters of CLIP-s: sampling frequency (number of training iterations between generating samples), as well as the threshold for minimal improvement and the number of consecutive measurements without improvement before stopping. These hyperparameters are chosen on a held-out set of 4 concepts to achieve 90% of the maximum possible train CLIP image score for the least number of iterations. The final checkpoint is selected from the iteration with the best train CLIP image score.
and Custom Diffusion accordingly), sampling 8 images every
After running CLIP-s all concepts, determine the average and maximum number of steps before early stopping across concepts. Then, we run the baseline method with a reduced number of iterations and intermediate sampling. This baseline for the adaptation to converge, as has the most efficient iterations among all methods. However, in real-world scenarios and for new customization approaches, the exact number of iterations is unknown in advance, which makes this method less applicable in practice, as it would involve rerunning such large-scale experiments for every new approach.
4.3 Results
The results of our experiments are shown in Table As we can see, DVAR is more efficient than the baseline and CLIP-s in terms of the number of iterations and overall runtime, approaching the performance of "Few iters" (which can be considered an oracle method) while being fully input- adaptive and not relying on costly intermediate sampling. particular, the fixed step count of "Few Iters" results in higher standard deviations for almost all metrics: from practical standpoint, it means that this number of iterations might be either sufficient or excessive when adapting a model to a given set of images. We additionally discuss the need for the adaptive number of iterations in Appendix G
Furthermore, although CLIP-s and the original setup optimize the train CLIP image score by design (in case of the baseline, this metric is used for choosing the best final checkpoint), DVAR is able to achieve nearly the same final results. Finaly, with sufficiently long training time, the increase in the train CLIP image score leads to a decline in the validation CLIP text score, which indicates overfitting on input images: this phenomenon illustrated in more detail in Figure and Appendix As we can see from the CLIP text scores of DVAR, early stopping helps mitigate this issue.
4.4 Human evaluation
To verify the validity of our findings, we compare the quality of images obtained with DVAR with that of the original methods. To do that, we used a crowdsourcing platform, asking its users to compare samples of the baseline approach and samples from the model finetuned for the number of iterations determined by DVAR.
We conducted two surveys: the first survey compared samples derived from prompts seen by the model during training, measuring the reconstruction ability of adaptation. Participants were instructed
7

---

Table Comparison of speedup methods for three approaches to text-to-image personalization. The best value is in bold.
Method Train CLIP img Val CLIP img Val CLIP txt Iterations
Time, min
Textual Inversion
Baseline 0.840=2.051
0.786
=0.021
6100
27.020.3
CLIP-s 0.824=0.053
0.757
0.233 =0.024
666. 666.72174.5
9.642.5
Few iters (mean) 0. 0.796
0.744
0.202
475
1.6+0.0
Few iters (max)
0.767 00.011
0.219
850
2.8
DVAR
0. 0.795
0.74810.00m
0.227 10.024
566.01141.5
3.110.8
DreamBooth-LoRM.
Baseline 0.86520.045
0.833+0.067
0.203=0.019
1000
7.8+1.5
CLIP-s 00.04
0.788 10.07
0.002
353.2
6.1
Few iters (mean) 0..55
0.806
0.219 =0.02
367
2.0
Few iters (max)
00.05
0.097
0.214=0.019
500
2.64u.1
DVAR 0.784 106
0.687
0.238
665.3
4.940.7
Custom Diffusion
Baseline 0.755=0.077
0.695
0.258=0.021
500
6.540.9
CLIP-s 0.757
0.691 00069
0.258 0.003
510.41134.1
9.7
Few iters (mean) 0.751 10.078
0.691 i 10.070
0.259 10.023
450.010.1
3.4
Few iters (max) 0.754_..077
0.691 10.033
0.257 10.022
700.0 100.010.0
5.311.4
DVAR 0.742::0.074
0.693-10066
0.259 2..022
348. 146.6
3.4.1.0
to select the image that resembled the reference object the most. The second survey compared samples generated using novel prompts, measuring the final model capability for customization. Participants were asked determine which image corresponded better the given prompt. Each pair of images was assessed by multiple participants, ensuring an overlap of 10 responses. For each set of 4 responses, we paid 0.1S, and the annotation instructions can be viewed in Appendix []
Our findings obtained from these evaluations are shown in Table 2 As we can see, DVAR allows for early stopping without sacrificing the reconstruction quality for two out of three evaluated customization methods. Hence, our approach enables efficient training and reduces the computational cost of finetuning reconstruction quality, this can be attributed to the overfitting of the original method: other methods, which use significantly fewer iterations, able avoid this issue.
concepts. While applying DVAR to Textual Inversion slightly reduces the
Table 2: Human evaluation results: the percentage of cases where the quality of DVAR was better or equal to Baseline quality.
Method
Reconstruction
Customization
Textual Inversion
41.6
77.7
DreamBooth-LoRA
67.9
91.4
Custom Diffusion
69.9
93.8
4.5 Analysis and ablation
Having demonstrated the advantages of DVAR, we now perform a more detailed analysis of the effect of the changes we introduce to the procedure of text-to-image adaptation. We aim to answer series of research questions around our proposed criterion and the behavior of the training objective with all factors of variation fixed across itcrations.
Is it possible to observe convergence without determinism? To answer this question, we conduct a series of experiments where we "unfix" each component responsible for the stochasticity of the training procedure one by one. For each component, we aim to find the smallest size of the batch that preserves the indicativity of the evaluation loss. For large batch sizes, we accumulate losses from several forward passes.
8

---

Method: Textual Inversion. Validation prompt: "A photo of <clock>> on the beach"
Validation
Train
Method: DreamBooth. Validation prompt:
wearing sunglasses"
Validation
Train
Method: Custom Diffusion. Validation prompt: "An <elephant> skeleton in the National Science Museum"
Validation
Train
Reference
Baseline
CLIP-s
Few Iters
Few Iters Max
DVAR
Figure 4: Comparison of early stopping techniques applied to personalization methods.
Figure 5 shows the results of these experiments. While for some parameters increasing the batch size leads evident trend in loss, the stochasticity brought by varying the timesteps each iteration cannot be eliminated even with batches as large as 512. This can be explained by the scale of the reconstruction loss being highly dependent on the timestep of the reverse diffusion process. Thus, aggregating it over multiple different points makes it completely uninformative, as the scale of the trend several orders of magnitude less than the average values of the loss.
it possible to use smaller batches for evaluation and still detect convergence? To address this question, we ran experiments with deterministic batches of size 1,2 and for multiple concepts and generally observed the same behavior (as illustrated in Figure X Appendix(C), However, the dynamics of the objective become more dependent on the timesteps that are sampled: as a result, the early stopping criterion becomes more brittle. Hence, we use a batch size of 8 in our main experiments, as it corresponds to reasonable tradeoff between stability and computational cost.
Does the selection of diffusion timesteps influence the behavior of the deterministic objective? In our approach, timesteps are uniformly sampled from a range of 1000 once the beginning of the training process. However, according the analysis presented in prior work (15), timesteps from

---

Timosteje
Batch size 1
Batch size
Batch size 61
Batch size 512
0.200
0.150
0.130
0,000
0.000
0.120
250
3000
750
250
5010
750
250
5000
750
250
5010
750
137
Captions
0.188
Batch size 2
0.144
Batch soze
0.157
Betch size 8
Banch size
0.143-
0.136
0.187
0.143
0.136
0.136-
250
500
750
250
500
750
0
250
500
7550
250
500
750
0.145
Batch
0.140
Batch
Baach 16
0.13
Hasch
Noiso
0.140
0.138
0.135
136
0.135
0.135
250
500
750
250
500
750
0
250
500
750
250
500
750
0.262
0.179
Batch size 2
Batch size
Betch size &
Hatch size 16
0.112
0.112
0.261
W/
0.179
1 Aar
0.112
0.112
250
500
750
250
500
750
0
250
500
750
250
500
750
0.139
Images
0.150
Batch size 1
0.140
Batch size
Batch size 32
Batch size 61
0.138
0.138
0.140
0.135
0.137
0.136
300
750
250
500
750
250
500
750
250
500
750
Figure 5: Loss behavior in the semi-deterministic setup: row names correspond to inputs that are resampled for each evaluation batch.
different ranges have varying effects on the results of text-to-image generation. This raises a question of whether it is possible sample timesteps only from a specific range, as might lead to higher correlation between the loss and the target metrics. Our findings in Appendix [] lindicate that there is no significant difference in correlations when we change the sampling interval.
Conclusion
In this paper, we analyze the training process of text-to-image adaptation and the impact of different sources of stochasticity on the dynamics of its objective. We show that removing all these origins of noise makes the training loss much more informative during training. This finding motivates the creation of DVAR, a simple early stopping criterion that monitors the stabilization of the deterministic verify that DVAR reduces the
loss based on its running variance. Through extensive experiments, training time by 2 2-8 times while still achieving image quality similar to baselines.
Our work offers a robust and efficient approach for monitoring the training dynamics without requiring computationally expensive procedures. Moreover, has adaptive runtime depending on the input concept, which naturally mitigates overfitting. These advantages make DVAR useful tool for text- to-image researchers and practitioners, as provides a more efficient and controlled training process, ultimately leading to improved results while avoiding the unnecessary computational overhead.
10

---

References
[1] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. sero-shot text-to-image generation. In International Conference Machine Learning, pages 8821-8831, PMLR, 2021.
[2] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. CoRR, abs/21 12. 10741, 2021.
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 1:Xiv22204.06125, 1(2)3, 2022.
[5] Chitwan Saharia, William Chan, Saurabh Saxena. Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems, 15:36499-36494 2022.
[6] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. 2022.
[7] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 22500-22516 2023.
[8] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[9] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[10] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. CoRR, abs/2104.0877 2021.
[1]] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Associates, Inc., 2021.
[13] Jonathan Ho. Classifier-free diffusion guidance. ArXiv, abs/2207. 2022.
[14] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or Attend-and-excite: Attention- based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42:1 10, 2023.
[15] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint S..v:2211.001222 2022.
[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.

---

[17] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 04000 2018.
Ling-Wei
[18] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 14, 2013.
[19] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the- diffusion models. https
//gthhb.. cww com/huggingt haaeffac///ffff diffusers 2022.
ace/
[20] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[21] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual conditioning in text-to-image generation, 2023.
[22] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdifff: Compact parameter space for diffusion fine-tuning, 2023.
[23] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 6007-6017, 2023.
[24] Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Gradient-free textual inversion, 2023.
[25] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-On Prompt-to-pronee image editing with cross attention control. arXiv preprint vrXiv:2208.01622 2022.
[26] Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. Magicmix: Semantic mixing with diffusion models. arXiv preprint arXiv:2210. 16056, 2022.
[27] Jaaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[28] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038 00086,047-- 2023.
[29] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021.
[30] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
12

---

A Additional related work
Most existing methods for text-to-image personalization achieve the goal by finetuning diffusion models. In addition to the approaches evaluated in our work, there exist many other techniques that were recently proposed. For example, in P+ (2T the concept is inverted into a sequence of per-layer tokens, and SVDiff [22] focuses on learning the singular values of the SVD decomposition of weight matrices. Imagic [23] divides the adaptation process into two phases. the first phase, this method optimizes the embedding in the space of texts instead of tokens, and in the second one, it trains the model separately to reconstruct the target image from the embedding. The optimized and target embeddings can then be used for image manipulation by interpolating between them. Furthermore, adaptation can also performed with gradient-fre methods using evolutionary algorithms 24
Text-to-image model customization can also be broadly viewed as an instance of image editing: given the caption and the image generated from it, we want change the picture according to the new prompt. There are several methods in this field that use diffusion models and have similar problem statements. For example, cross cross-attention, and MagicMix [26] replaces the guidance vector after several steps during inference. The intrinsic disadvantage of such methods is the need for an additional inversion step for editing existing images, like DDIM 27 or Null-text Inversion 22
Comnptto-Prooppee 25 solves the task by injecting the new prompt into
B Indicators of convergence for other customization methods
We provide examples of the behavior of standard model convergence indicators for other techniques for text-to-image personalization in Figures and While the reconstruction loss and the gradient norm are still hardly informative, both the CL.IP image score and Lite exhibit a trend that corresponds to improvements in the visual quality of samples.
0.75
0.80
CLIP
0.00
1000
2001
300
500
500
100
300
500
6,000.00
0.29
8,000.00
3
0.25
Gradient
2,000.00
0.26
2003
500
100
2001
3011
0.0
100)
Sampiee
Chnoegg
Step 10
Step 100
Step 300
Step 500
Figure 6: Convergence process for Custom Diffusion applied to an example concept.
C The impact of the batch size on convergence dynamics
As we increase the batch size for our evaluation objective, we sample a more diverse range of timesteps, making the loss less dependent on the random seed and more indicative of convergence. Figure provides an illustration of this phenomenon.
Moreover, we provide the results of training in a regular Textual Inversion setup with much larger batches. As we demonstrate in Figure (a] the loss dynamics remain the same even if increase the batch size to 128, which is 64 times larger than the default one and thus requires 64x more time for a single optimization step.
Although the gradient norm of training runs with extremely large batches begins to display conver- gence (see Figure 9b), using the batches of this size is less practical than using an early stopping
13

---

0.50
0.30
CLIP
0.60
0.00
2000
400
600
1000
10,000.00
Grodeet
0.26
5,000.00
400
900
1000
2000
(C0
1000
Sumple
Censeep
Step 50
Step 250
Step 500
Step 1800
Figure 7: Convergence process for DreamBooth applied to an example concept.
00243
Datch size
Batch size
0.1245
Batch size 1
Batch size 8
Vyf
00735
H
0.3000
W/ Auh
4
00730
0.1007
1220
012360
1500
1000 1500
1000
1000
Figure 8: Ldet behavior in the fully deterministic setup with varying batch sizes.
criterion. Specifically, when training with these batch sizes, the quality of samples reaches its peak approximately 250 training steps. This corresponds to R 16,000 forward and backward passes with the batch size of (or 8,000 with the batch size of 8 the largest size that fits into 80GB GPU for Stable Diffusion) and thus gives no meaningful speedup to the inversion procedure.
D Example pseudocode of using DVAR for training
In Figure TO provide example PyTorch training loop that uses implementation of DVAR defined in Figure 3] This code samples the evaluation batch once before training and computes Ldet on after each training iteration. A full implementation of training with DVAR is given at github. commyyandex-resercch/Do
Daark aize
Teach sizo
8
0.3
ELON
6.40
403
0.3
603
LOCC
0,00
Itaich doc 92
Bosch dec 124
0.
0.05
0000
0.00
11
450
280
340
1.09
15.0
Training loss dynamics.
(b) Gradient dynamics.
Figure 9: Training dynamics of Textual Inversion for single concept with different batch sizes.
14

---

def training.loop (model train_datailn daaalddaaao,der, eval _al_sssses, eval eval_batch, optimizer, vindou_size, threshold):
eval_images, eval_captions eval_batch eval_stochastic = wwwpppeeeeeeeetinneelaaaeebeeeee
eval_latents, eval_ wwal_timesteps, eval_ eval_
for train_batch in brain_dataloader
images, captions train_batch latents, timesteps, noisy loisy_latents sample_everythingtraa
optimizer 2
mmioised_latente model (captions, noisy_ latents, timesteps)
train_loss F. f_mse_loss lattoissed_latents.. latents)
train_loss backward) optimizer.step
with torch... no_grad():
@eal_denoised_latene model eval_captions,
eval_noisy_latents, eval_timeston
eval aval_loss F F.mee__eee L eval_ vral_latnts))
eval_lossen (evaeeeeaeppedee
if OVAR(torch) OVAR(torch.stacckeee stack((eval_losses))) window_size, threshold) :
break
Figure 10: PyTorch-like pseudocode of training with DVAR.
E
Evaluation of other stopping criteria
Besides the ratio of the rolling variance to the global variance used in DVAR, also experimented with other metrics for early stopping that consider the behavior of L All of these metrics have a hyperparameter that denotes the number of last objective values to use for computing aggregates,
To obtain the EMA percentile, we calculate the exponential moving average (with 0.1) at the moment and n steps back, then apply the following formula:
MEMA = EMA(t) EMA(t n)
EMA(t- - -n)
The Hall criterion is simply the difference between the rolling maximum and the rolling minimum divided by the rolling average over n steps:
MHall max - min((
mean (C3,.)
where Ci denotes the last values of Last-
The Trend metric is the slope of linear regression trained on loss values in window of size computed at each step using the closed-form solution for linear regression. The main issue this criterion is its longer evaluation time compared others.
The dynamics of all these metries are shown in Figure [T] The main challenge we face with these metrics is their unstable behavior, which prevents the transfer of hyperparameters between concepts.
15

---

percentiio
2520
L.00
0,125
W 40 MMMMMM
10.00
6.75
11.00
0.015
0.010
0..00
-2
1000
770
2010
1.00
MM
6.75
0,1000
-0.00
1
0.50
-0.002
6.25
-0.000
0,001
6,00
700
250
5400
7150
0.75
M
A
0.003
0L002
6.50
WW
-00002
0.002
0.25
-5
0,00
250
500
750
255
5000
750
0
2250
500
750
255
500
750
Stop
Strp
Stagg
Stop
Figure 11: The dynamics metrics for additional stopping criteria.
Quantitative comparison with additional metrics
We conduct additional quantitative evaluation of different early stopping approaches with two metrics introduced in 7 named DINO and DIV.
The DINO metric represents the average cosine similarity between VIT-S/16 DINO [29] embeddings of generated and real images. DINO is more sensitive to differences between subjects of the same class than CLIP image score, which is more suitable for measuring the degree of unique feature preservation. By contrast, DIV was proposed to evaluate the diversity of generated images. is computed as the average LPIPS (30) cosine similarity between samples generated with the same prompt. Low values DIV signify the model overfitting the training images.
Table 3: Comparison of identity preservation and diversity of speedup methods for three text-to-image customization techniques. Standard deviations over 48 concepts in subscript.
Metric
Baseline
CLIP-s
Few iters (mean)
Few iters (max)
DVAR
Textual Inversion
DINO
0.635 0.590+0.110
0.559 133
0.591+0.108 0.757+0.092
0.566+0.119
DIV 0.742+0.078
0.769 097
8.766+0.000 099
0.777+0.091
DreamBooth-LoRA
DINO
0.72140.103
0.709
0.711
0.704
10.7710.206
DIV
0.565 154
0.630 100
0.63240.086
0.592440.127
0.585
Custom Diffusion
DINO
0.475
0.471+u.1a0
0.47544.14
0.488
0.45440.135
DIV 0.753
0.748 40.03
0.753
0.756
0.740 1.74040.0555
Table shows that DVAR increases the DIV metric for two personalization techniques, which indicates reduced overfitting, whereas having only minor losses in subject fidelity measured by
DINO. Overall, we conclude that our approach does not cause any significant quality degradations while decreasing customization time in an adaptive manner.
16

---

G Variance across concepts
To additionally justify the need for an adaptive early stopping technique, we conduct two experiments. First, analyze the variability of Ldet actoss concepts. Second, compare the distribution of stop iterations selected by DVAR with those selected by non-adaptive early stopping methods.
G.1 det variation over concepts
In Figure| 12 we depict the behavior of normalized Ldet for four concepts across all three personal- ization methods. Normalization is necessary, because the scale of loss varies highly across concepts.
Textual Inversion
Custom Diffusion
1.0
1.0
1.0
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
500
1000
1500
2000
4000
6000
500
1000
dog
Dower
cal
clait
Figure 12: Behavior of Last for various concepts and methods.
From this plot, we can conclude that the dynamics of Ldet moderately differ from one concept to another. For example, the objective function exhibits earlier saturation on some concepts and methods and demonstrates more unstable behavior on others.
G.2 Final steps distribution
We compare the distribution of the final steps selected by DVAR to the non-adaptive methods. As we from Figure 13, simply throwing away some of the iterations might be sub-optimal for some concepts, indicating the need for an adaptive early stopping technique.
Textual Inversion
Custom Diffusion
0.20
0.20
0.15
0.15
0.10
0.10
0.10
0.05
0.06
0.05
0.00
400
600
800
400
600
800
400
600
Few Items (menn)
Few Iters (mant)
Figure 13: DVAR stop iterations distribution across concepts in relation to non-adaptive bascline.
17

---

H Qualitative comparison
we show a side-by-side comparison of DVAR and Baseline methods. Validation images are obtained from the final training checkpoints using the validation prompt. This comparison confirms that our method achieves similar results in terms of concept reconstruction and prevents overfitting for all three adaptation approaches.
In Figure [4
Moreover, this comparison demonstrates that focusing only on the quality of samples during training can be detrimental to the model's ability to transfer concepts to unseen contexts. In such cases, the model tends to generate outputs that resemble the training images regardless of the input prompt.
Validation prompt: <A chair with <eat> sitting on near a beachy
Textual Inversion
Validation prompt: <A <grey sloth plushic> with tree and autumn leaves in the background>
Textual Inversion
Validation prompt: <A <dog>
the beachn>
DreamBooth
Validation prompt: aλ <dog> oil painting, Ghibli inspired
DrcamBooth
Validation prompt: aA purple <wolf plushie>>>
Custom Diflusion
Validation prompt: 4A <tortoise plushy> wearing Superman outfity
Custom Diffusion
Reference
Bascline Train
DVAR Train
Baseline Validation DVAR Validation
Figure 14: Side-by-side comparison of DVAR reconstruction quality and customization ability in relation to the baseline.
18

---

I Human evaluation instructions
In Figure 13 we provide the instructions for the quality assessment tasks that were shown to the annotators. We aimed to make the instructions as unambiguous and detailed as possible, minimizing the potential influence of different interpretations among individuals. All annotators were required to complete training task specifically designed to ensure a clear understanding of the decision-making algorithm. Furthermore, we incorporated control tasks with obvious correct answers to filter out bots or low- -wwlltllyy annotators who might vote randomly.
Task description:
Two neural networks tried to learn the object dopicted by the Reference image and to generate it in another envirooment (e.g. another backgrond. view angle, style or composition) described text prompt on the top of the task. Please help us understand which noural network does the job better terms of matching the text prompi by answering question for given pair of generated images. How to answer the question:
1. Carefully look the reference image.
try coursely describe class of the given image (e.g. table, dog. loy, backpack. etc.). Then attention to details, try to figure out what distinguishes this specifiic object from another from its class: its shape, texture, coloring, additional elements specific details, etc.
If the reforence object is present only one of the pictures seleet the picture where present. the reference object not present on any of the pictures select Equal. Else: compare two generated images by their resemblaanc to the reference. We suggest going from the most general traits the more specific details:
Which generated imuges beller captures object, proceed to step 4. Otherwise, please select comeeponning image.
the original object? images equally reflect vourse class
4. Which of the generated images better reflects the shape of the objoct being learned? If you think that both images equally reflect the shape of the object, proceed to step Otherwise, please select correponning image.
5. Which of the generated images better reflects the colouring of the object being leumed?? you think that both images reflect the abject's is color equally, proceed to step 6. Otherwise, please select a corresponding image.
6. Are there any details that one generated image reflected better than another? If yes, please select corresponding image. Otherwise, please select "Equally similar".
Reference:
Generated image A:
Generated image B:
<REFERENCE IMAGE>
<GENERATED IMAGE>
~GENERATED IMAGE>
Which of the generated images resembles the reference more?
(a) Instructions for Reconstruction survey.
Task description:
Two neural networks tried to learn the objoct depicted by the Reference image and to generate it in another envirooment (e.g. another background, view angle. style composition) described in text prompt on the top of the task. Please help understand which neural network does the job better in temms of mulching the text prompi by answering one question for given pair of generaled images.
How to answer the question: Fach prompi (text description that <cat- saat-statue> (this is unique identifier of the object being learned by the network) and the description of a secne in which we want to see this object depicted. Descriptions may include specifications different places, circumstances, change reference object color, texture, shape, etc.
used generate image) contains word phrase brackets like
1. First, read the prompt replacing the object identifier with the word "something", For example if the prompt "A <ent-statues front of the Eiffel Tower", read the prompt as 'A something in front of the Eiffel Tower".
2. Now try idemiiy which image better matches this modified prompt. If one image is better thun amother, please vote for the comreponning image. Flse: procedd the next step.
3. Now put back the reference objoct ideatifier into the prompt and answer the following question: "Which image have <reference- inage-objeat> incorporated into the scene?"
4. To do so, first carefully study the reference image object: try to coarsely classify it pay attention to its shape, coloring. texture, any patterns and details.
5. Now, for example our example prompe "A ccat-statues statue> in front of the Fiffel Tower" if one image depicts random object in front of the Eiffel Tower and the other depicts something similar the <eat-statues (e.g. a cat toy or a cat staructte of the similar shape and/or coloring. More specific details are not crucial), the latter image should be chosen.
6. If one image incorporates learned object into the scene and another does not or does in some details better than another, please select corresponding image. Else: please vote "equal".
Text description: <Text description Reference:
Generated Image A:
Generated image B:
RRFERRENCE IMAGES
<GENERATED IMAGES
<GENERATED IMAGES
Which of the generated images matches the text description more?
(b) Instructions for Customization survey.
Figure 15: Human annotation interface.
19

---

Correlations of timesteps sampling intervals and metrics
To investigate the impact of varied timesteps, we divide the interval from 0 1000 into three equal parts (beginning, middle, and end) and evaluate the Leet with timesteps sampled from each subinterval. Next, we run the training process over 1500 steps for concepts, with sampling images every 50 steps and calculating four CLIP-based quality metrics: train image similarity, train text similarity, validation image similarity, and validation text similarity.
Table 4: Pearson correlations between losses and quality metrics for different timestep sampling intervals. All results statistically significant (p-value 0.01)
Begin (0-333)
Middle (333-666)
End (666-1000)
Full (0-1000)
Train text score Validation text score
0.27
0.33
0.30
0.36
0,48
0.51
0.37
0.50
Train image score Validation image score
-0.63
-0.64
-0.56
-0.62
-0.58
-0.60
-0.53
-0.56
Inspired by previous research [15), we hypothesize that different timestep intervals might exhibit similarities with different metrics. However, indicated in Table our results do not reveal such behavior. The values within the rows demonstrate remarkable similarity to each other across all subintervals, even though there are different trends for particular metrics.
20


Reviews:
[{"review_id": "X2hf4mQ8x0", "reviewer": "Reviewer_5cki", "summary": "This paper studies the training dynamics of popular text-to-image personalization methods (such as Textual Inversion, DreamBooth, and Costom Diffsuion), aiming to speed them up by an early stopping approach which allows the model to optimize or fine-tune for fewer iterations. A key observation is that most concepts are learned at early stages and do not improve in quality later, but standard model convergence metrics fail to indicate that. Based on this observation, the authors propose a simple drop-in early stopping criterion that only requires computing the regular training objective on a fixed set of inputs for all training iterations. Experiments are conducted on Textual Inversion, DreamBooth, and Costom Diffusion.", "questions": "Please refer to the weakness section.", "limitations": "The authors addressed the limitations.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The observation that most concepts are learned at early stages and do not improve in quality later, but standard model convergence metrics fail to indicate that is interesting and inspiring.\n2. The proposed approach is well-motivated with key observations and in-depth analysis before deriving the method.\n3. The proposed approach improves the efficiency of personalized text-to-image models by a simple but effective early stopping scheme.", "weaknesses": "1. Although the visual results and CLIP scores indicate no further improvement after optimization for a certain number of steps, it is still not clear whether the observation is solid. The CLIP score can be biased because of the limited ability of CLIP in understanding complex and detailed information. Good visual results in generating an image similar to the input image do not mean perfect identity and detail perservation for personalized text-to-image generation.\n2. The writing and logic can be improved. For example, how does 3.2 (investigating the sources of randomness) relate to other sections is not well demonstrated.\n3. The evaluation is limited. Firstly, only 18 concepts are used for evaluation. Prior work such as DreamBooth actually used more concepts and prompts. Secondly, the CLIP image-image and CLIP image-text similarities are not enough to evaluate the concept preservation and image quality of personalized diffusion models. Prior work DreamBooth used several other evaluation metrics to reflect the ability of the models. Thirdly, the visual results in Figure 4 are not impressive, and there are no examples for personalized generation where the user provides a different text prompt with the same visual concept from the original image but in a different background or scene. The different application scenarios of DreamBooth, Textual Inversion, and Custom Diffusion are not extensively experimented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a detailed review! We would like to answer your concerns about our work below:\n\n>Good visual results in generating an image similar to the input image do not mean perfect identity and detail perservation for personalized text-to-image generation.\n\nWe agree that the CLIP image score may not fully reflect the process of the model learning complex details. However, as indicated in Table 2 of the paper, we also conducted **human evaluation** to compare our method with the baseline in two directions: concept **reconstruction** and **customization**. Our results demonstrate that 1) our method often stops at an iteration **that is better or equal in terms of reconstruction** quality 2) this iteration also proves to be **the best for image customization**, indicating the overfitting of original methods which can be overcome by DVAR. Additionally, the correlation between the CLIP image score and the visual quality of reconstruction can be partially confirmed using Figures 2, 10 – 12.\n\n>The writing and logic can be improved. For example, how does 3.2 (investigating the sources of randomness) relate to other sections is not well demonstrated.\n\nSection 3.2 is necessary to **motivate and introduce the new objective** $L_{det}$ (a key factor in our work) and to explain how it differs from the original training objective. $L_{det}$ is used for the DVAR early stopping criterion, which is one of the primary contributions of this paper.\n\n>The evaluation is limited. Firstly, only 18 concepts are used for evaluation. \n\nWe have conducted additional experiments on the DreamBooth dataset with **30 new concepts**; see the general response for details.\nPrior work DreamBooth used several other evaluation metrics to reflect the ability of the models. \nThank you for this suggestion! We have updated the table with main results from our paper with **two more metrics** used in the DreamBooth paper (DINO and DIV). Please refer to the table with updated results in our general response.\n>there are no examples for personalized generation where the user provides a different text prompt with the same visual concept from the original image but in a different background or scene.\n\nThank you for pointing that out. Actually, this setting is shown in Figure 4 of the submission, but it might be difficult to interpret because we did not specify the validation prompt. For example, in the row “Textual Inversion val”, the prompt “A photo of <clock> *on the beach*” is used to check if the learned concept can be used in a different scene. Both the baseline and CLIP-s methods **fail to depict** the learned concept in the desired background, whereas **DVAR succeeds**. Other qualitative examples are provided in Figure 13 of Appendix J. An improved version of Figure 4 and other qualitative side-by-side comparisons can be found in the PDF attached to our general response."}, {"review_id": "QmWgQglKGU", "reviewer": "Reviewer_DXLz", "summary": "This paper argues that customization techniques for diffusion-based text-to-image generation models train for longer than is needed. This is because the training loss of diffusion models is often not informative -- i.e., often looks like stationary noise -- so practitioners tend to use a fixed (often excessive) number of training steps. This paper identifies and analyzes the sources of stochasticity in the training loss, and propose simple ways to eliminate them to make the loss more informative. They also introduce a simple early stopping criteria based on this interpretable loss.", "questions": "How does L_det vary across concepts/models/methods?\n\nOne plot that could help motivate the DVAR is a plot that shows how L_det varies across concepts (and models/methods). If it's the case that L_det doesn't vary very much then maybe picking a fixed (but smaller) number of fine-tuning steps is sufficient.", "limitations": "n/a", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Paper addresses a key issue many people training diffusion models face: the loss is not informative. That is, it often behaves like stationary noise despite the model continuing to improve on auxiliary metrics of interest (FID, human evals).\n\nThe authors do a very principled analysis of the sources of stochasticity in this loss and identify the sampled time-step to be the main driver. Eliminating this randomness leads to loss that better corresponds to model performance. I could se this becoming common practice -- not just in the model customization/fine-tuning regime but also in the training of the base diffusion model.", "weaknesses": "The main weakness of the paper is that I'm not sure how useful the DVAR early stopping criteria is. See question below.\n\nHowever for this paper, this is less of an issue, since it's main contribution is a careful analysis of a problem people who train diffusion models face: uninformative loss and where it comes from.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We would like to address your concern in the following response:\n>How does L_det vary across concepts/models/methods?\n\nDifferent behavior of $L_{det}$ for different methods can be observed in Figures 2, 10 – 12 of our work. In Figure 2 of the PDF attached to our general response, we depict the behavior of normalized $L_{det}$ for 4 concepts across all three personalization methods. Normalization was necessary, because the scale of loss varies highly across concepts. From this plot, we can conclude that the dynamics of  $L_{det}$ moderately differ from one concept to another. For example, the objective function exhibits earlier saturation on some concepts and methods and demonstrates a more unstable behavior on others. We will include this illustration in the next revision of our paper — thank you for this suggestion!\n"}, {"review_id": "9RWI7PGd5t", "reviewer": "Reviewer_Gk9z", "summary": "This paper studies the training dynamics of a few state-of-the-art txt-to-image personalization methods and proposes an early stopping criteria while fine-tuning the base models to speedup their customization. They evaluate this criteria on Dreambooth, Textual inversion, and custom diffusion on 18 concepts and showing close or slightly better img-txt clip similarity scores on the validation prompts while degrading the similarity to the input image.", "questions": "1. In addition to the above questions mentioned above, it was not clear to me if the models are trained on all concepts together or if they are trained on the samples of each concept. If it is on each concept, where are 8 images per batch coming from? \nIf that has been done on multi-concepts, then I'd expect to see some evaluations on multi-concept customization.\n\n2. Very few qualitative examples are shown in the paper/supplemental. It would be interesting to add more examples comparing the results with the baseline.", "limitations": "- The proposed technique causes a natural trade-off between the number of fine-tuning iterations and identity preservation of the objects in the source images. While the latter is quite important in image personalization, it is degraded in this paper (based on Table 1) and is not studied well in human evaluations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper presents a comprehensive study on the randomness factors during training of the customization models and their training curves and propose a simple stopping criteria based on a deterministic variance evaluation of the diffusion loss.\n 2. This early stopping results in a 2x speedup on custom diffusion as well as dreambooth-LoRA and 12x speedup in textual inversion.", "weaknesses": "1. The whole idea of the paper is not very novel, and looks more like an analysis paper. \n1. There is a trade-off in the samples similarity to the source image during both training and validation versus the number of finetuning steps. Although the authors have shown more generalization ability to validation prompts, it looks like identity preservation has been degraded.\n2. The evaluations are only done on 18 concept examples. A larger set would be needed to make the results more convincing.\n3. The scores which are reported in Table 2, only measure the similarity of the generated image with the validation text prompt. It would be important to see how much of the identity is preserved in these generations using the proposed early stopping criteria. Identity preservation is an important goal in personalization methods and is not studied well in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide detailed feedback on our work. Please find our responses to your concerns and questions below:\n>The whole idea of the paper is not very novel, and looks more like an analysis paper.\n\nTo the best of our knowledge, **we are the first to to study the optimization process of text-to-image customization** in detail and to propose a method to **make the loss of diffusion models informative**. If there are **specific studies** that you can direct us to where such an analysis has been conducted and a solution has been proposed, we would be grateful to learn about them.\n\nMoreover, we politely disagree that positioning our work as an analysis paper should necessarily be viewed as its weakness. We believe that providing comprehensive analysis presents its own set of equally important contributions for the scientific community.\n>There is a trade-off in the samples similarity to the source image during both training and validation versus the number of finetuning steps.\n\nYou are correct: **we explicitly indicate in L269-270** that this trade-off exists and is one of the problems of the baseline methods we employ in our paper. However, **this is not a disadvantage of our method**. On the contrary, Table 2 reveals that although the baseline method often outperforms DVAR in terms of the reconstruction quality, it considerably lags behind our method in terms of the **customization** ability. Hence, our method allows us to identify an **optimal iteration to stop training** from the perspective of this trade-off. As evidenced in Table 1, early stopping by DVAR results in a negligible decrease in the quality of reconstruction on average across all concepts while preserving high quality of customization.\n>The evaluations are only done on 18 concept examples. A larger set would be needed to make the results more convincing.\n\nWe agree and conduct additional experiments on **30 new concepts** from the DreamBooth paper. Please see the second paragraph of our general response for the updated results.\n>The scores which are reported in Table 2, only measure the similarity of the generated image with the validation text prompt. It would be important to see how much of the identity is preserved in these generations using the proposed early stopping criteria. Identity preservation is an important goal in personalization methods and is not studied well in this paper.\n\nWe kindly disagree, because identity preservation is studied throughout the work. First, identity preservation is quantitatively evaluated by the **Val CLIP img** metric in Table 1. Similarly to the explanation of the Train CLIP img score provided in L229-231, Val CLIP img score measures identity preservation on **images generated from novel prompts**: we will emphasize that in the setup description to avoid reader confusion in the future. Judging by this metric, we see that DVAR is comparable to the baseline on DreamBooth-LoRA, slightly surpasses the baseline on Custom Diffusion, and slightly underperforms on Textual Inversion. Importantly, the possibility of identity preservation depends on the method and not just on the early stopping technique: training for more iterations leads to better identity preservation, but also to overfitting.\n\nSecond, in Appendix F, Figure 13b provides instructions according to which the annotators determine the degree of customization. The third point of the instruction asks annotators to answer the question \"Which image has a **<reference-image-object> incorporated into the scene?**\" and the fifth point asks to determine **which image preserves the identity better**. Therefore, **identity preservation is covered by the Customization metric in Table 2** as well.\n>it was not clear to me if the models are trained on all concepts together or if they are trained on the samples of each concept. If it is on each concept, where are 8 images per batch coming from? \n\nThe models were trained on the samples of each concept separately; we leave multi-concept personalization out of the scope of our work. Following Equations (2) and (3) and L100-102, the training  batches consist of images (x ~ X, where X is the 3 – 5 images for each concept) with minimal augmentations (central/horizontal crop), randomly sampled captions (y ~ Y), timesteps (t ~ U[0, T]), and random noise from the multivariate Gaussian distribution. Therefore, each batch is generated from **a small set of original images with different random inputs**: note that DVAR does not fix them for the training objective, only for evaluation. Thanks for highlighting this point of confusion, we will clarify the training process in the camera-ready version.\n>Very few qualitative examples are shown in the paper/supplemental.\n\nBesides the side-by-side comparisons in Figures 4 and 13 of our paper, we provide additional qualitative comparison in the PDF attached to our general response. "}, {"review_id": "TKjMslFTvo", "reviewer": "Reviewer_rpQb", "summary": "Text-to-image generation models offer fine-grained control over synthesized images, but fast adaptation to smaller datasets or new concepts remains a challenge. Existing efficient adaptation methods suffer from long training times, hindering practical applications and resource usage. This work addresses the issue by studying the training dynamics of popular text-to-image personalization methods and proposes a drop-in early stopping criterion that significantly speeds up adaptation (up to 8 times faster) without compromising quality, as demonstrated through experiments on Stable Diffusion and various personalization methods.", "questions": "Please address the issues highlighted in the Weaknesses section.", "limitations": "Please address the issues highlighted in the Weaknesses section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes a simple but effective method to accelerate text-to-image customization.\n2. The proposed method is well-motivated and easy to understand.", "weaknesses": "1. It is imperative to provide supporting evidence to justify the necessity of adaptive step choices. Can we simply set a fixed step number (e.g. reduce to 1/3) without losing much performance? For instance, analyzing the outcomes and plotting the distribution of selected step numbers can demonstrate the potential reduction in unnecessary iterations. This approach would enhance the validity of the proposed method.\n2. The authors should engage in a thorough discussion of pertinent literature concerning the acceleration of generative models. It is crucial to acknowledge and reference closely related research in this domain.\n\n---\n\nHaving read the author's rebuttal, I've chosen not to alter my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful suggestions! Allow us to address your concerns in a below response:\n>It is imperative to provide supporting evidence to justify the necessity of adaptive step choices. For instance, analyzing the outcomes and plotting the distribution of selected step numbers can demonstrate the potential reduction in unnecessary iterations.\n\nWe agree that the justification of adaptive number of steps is highly required. This is why in **L262-265** we mention that non-adaptive Few Iters baseline has a **greater variance of almost all reported metrics**, which indicates a need for an adaptive early stopping method. However, plotting the distribution of selected steps might indeed be even more illustrative. Thank you for this suggestion: we include such a plot in the PDF of the general response.\n>The authors should engage in a thorough discussion of pertinent literature concerning the acceleration of generative models.\n\nAcceleration of generative models is indeed closely related to our work’s area of research. However, since DVAR does not change neither the optimization objective nor the sampling procedure, **all advancements in this area are complementary to our research**. In other words, one can use any of such techniques to accelerate the training process while using $L_{det}$ to track the convergence of the model. Keeping our paper succinct yet thorough required us to be selective about the related works that we cover. Nevertheless, we agree with your suggestion, and we are happy to discuss ways of accelerating generative models in an additional paragraph of Appendix A. We would be especially grateful for your suggestions of particular research areas that we should cover."}]

Author rebuttal:
Dear reviewers, we deeply appreciate the time and effort you devoted to the review of our paper. We are glad that multiple reviewers recognize the motivation that drives our work (**5cki**, **DXLz**, **rpQb**), the depth of our analysis (Gk9z, DXLz), and the simplicity of DVAR combined with its effectiveness (**5cki**, **rpQb**). We have carefully considered your insightful feedback and have addressed the concerns raised by each reviewer in their respective responses. In this response we would like to address your collective questions and comments. 

First, we would like to address a common concern regarding the evaluation of the studied methods on only 18 concepts. In the submitted version of our work, we used all the concepts and prompts provided by the authors of two methods that we test: Custom Diffusion and Textual Inversion. The table below presents an extended evaluation on additional 30 concepts that were recently released by the DreamBooth paper authors (resulting in **48 concepts overall**). 

The evaluation is additionally augmented with two more metrics from the latest revision of DreamBooth paper (following the suggestion from Reviewer **5cki**): DINO measures the preservation of subject details in the images generated from train prompts and DIV measures the diversity of the samples generated from the same prompt.

|              |    |  |    Textual Inversion       ||            |        |     |
|--------------------|-----------------|---------------|---------------|--------------|--------------|--------------|-----------
| Method              | Train CLIP img   | Val CLIP img   | Val CLIP txt   | DINO          | DIV           | Iterations       | Time, min    |
| Baseline             | $0.840_{±0.051}$    | $0.786_{±0.075}$  | $0.209_{±0.021}$  | $0.635_{±0.094}$ | $0.742_{±0.078}$ | $6100.0_{±0.0}$  | $27.0_{±0.3}$ |
| CLIP-s | $0.824_{±0.053}$    | $0.757_{±0.067}$  | $0.233_{±0.024}$  | $0.590_{±0.110}$ | $0.769_{±0.097}$ | $666.7_{±174.5}$ | $9.6_{±2.5}$  |
| Few Iters (mean)    | $0.796_{±0.069}$    | $0.744_{±0.073}$  | $0.232_{±0.023}$  | $0.559_{±0.133}$ | $0.768_{±0.099}$ | $475.0_{±0.0}$   | $1.6_{±0.0}$  |
| Few Iters (max)     | $0.806_{±0.066}$    | $0.767_{±0.071}$  | $0.219_{±0.022}$  | $0.591_{±0.103}$ | $0.757_{±0.092}$ | $850.0_{±0.0}$   | $2.8_{±0.0}$  | 
| DVAR | $0.795_{±0.067}$    | $0.748_{±0.068}$  | $0.227_{±0.024}$  | $0.566_{±0.119}$ | $0.777_{±0.091}$ | $566.0_{±141.5}$ | $3.1_{±0.8}$  |
|              |    |  |    **DreamBooth-LoRA**       ||            |        |     |
Baseline             | $0.857_{±0.061}$    | $0.824_{ ±0.079}$  | $0.205_{±0.021}$  | $0.721_{±0.103}$ | $0.565_{±0.154}$ | $1000.0_{±0.0}$ | $8.1_{±2.2}$|
| CLIP-s | $0.862_{±0.045}$    | $0.788_{±0.075}$  | $0.225_{±0.022}$  | $0.709_{ ±0.093}$ | $0.630_{±0.100}$ | $353.2_{ ±88.1}$ | $6.1_{ ±1.5}$ |
| Few Iters (mean)    | $0.855_{ ±0.052}$    | $0.806_{ ±0.085}$  | $0.219_{±0.023}$  | $0.711_{±0.111}$ | $0.632_{±0.086}$ | $367.0_{±0.0}$  | $1.9_{ ±0.0}$ |
| Few Iters (max)     | $0.851_{ ±0.053}$    | $0.800_{ ±0.097}$  | $0.214_{ ±0.019}$  | $0.704_{ ±0.125}$ | $0.592_{±0.127}$ | $500.0_{ ±0.0}$  | $2.6_{ ±0.1}$ |
| DVAR | $0.784_{ ±0.106}$    | $0.687_{ ±0.140}$  | $0.238_{ ±0.034}$  | $0.577_{ ±0.206}$ | $0.585_{ ±0.114}$ | $665.3_{ ±94.9}$ | $4.9_{ ±0.7}$ |
|              |    |  |    **Custom Diffusion**       ||            |        |     |
Baseline             | $0.755_{ ±0.077}$    | $0.695_{ ±0.069}$  | $0.258_{ ±0.021}$  | $0.475_{ ±0.139}$ | $0.753_{ ±0.061}$ | $500.4_{ ±0.5}$   | $6.5_{ ±0.9}$ |
| CLIP-s | $0.757_{ ±0.076}$    | $0.691_{ ±0.069}$  | $0.258_{ ±0.023}$  | $0.471_{ ±0.140}$ | $0.748_{ ±0.063}$ | $510.4_{ ±134.1}$ | $9.7_{ ±3.0}$ |
| Few Iters (mean)    | $0.751_{ ±0.078}$    | $0.691_{ ±0.070}$  | $0.259_{ ±0.023}$  | $0.475_{ ±0.143}$ | $0.753_{ ±0.069}$ | $450.0_{ ±0.0}$   | $3.4_{ ±0.9}$ |
| Few Iters (max)     | $0.754_{ ±0.078}$    | $0.691_{ ±0.073}$  | $0.257_{ ±0.022}$  | $0.488_{ ±0.145}$ | $0.756_{ ±0.068}$ | $700.0_{ ±0.0}$   | $5.3_{ ±1.4}$ |
| DVAR | $0.742_{ ±0.074}$    | $0.693_{ ±0.066}$  | $0.259_{ ±0.022}$  | $0.454_{ ±0.136}$ | $0.740_{ ±0.055}$ | $348.1_{ ±46.6}$  | $3.4_{ ±1.0}$ |

Noticeably, adding 30 novel concepts does not change the relative ranking of early stopping methods: DVAR still allows for early stopping **without sacrificing the reconstruction quality** (judged by Train/Val CLIP img and DINO metrics) for two out of three evaluated customization methods. Moreover, models trained with DVAR demonstrate **higher customization quality** than the baseline (demonstrated by Val CLIP txt metric), while being **adaptive** and **non relying on the costly intermediate sampling**. Additionally, DVAR increases the DIV metric for two personalization techniques, signifying **reduced overfitting**. 

Moreover, we provide additional plots and illustrations in the attached PDF. Specifically, the attachment contains more qualitative side-by-side comparisons (**Gk9z**), a corrected version of Figure 4 with validation prompts specified (**5cki**), the distribution of final step numbers (**rpQb**), and the $L_{det}$ behavior on various concepts (**DXLz**).
 
We hope that a more extensive evaluation of our method addresses your concerns and that the additional illustrations further confirm our findings. If you have any additional questions, we would be happy to respond to them during discussion.

Meta-review: This paper argues that customization techniques for diffusion models train for longer than is needed. The authors analyze the sources of stochasticity in the training loss, and propose simple ways to eliminate them to make the loss more informative. 

The authors have done a nice job during rebuttal. After rebuttal, the paper received mixed scores of 4567. The reviewers who gave a score of 4 and 7 both gave a confidence score of 5 for their reviews. On one hand, this paper provides a careful analysis of a key problem people who train diffusion models face: uninformative loss and where it comes from. The authors do a principled analysis of the sources of stochasticity and provide simple fixes to make the loss more informative. The observations are interesting and inspiring. 

On the other hand, reviewers have questioned about the limited concepts used to evaluation the methods (18 concepts in total). During the rebuttal, the authors have added new results on an additional set of 30 concepts, and added 2 additional evaluation metrics. Another concern shared by two reviewers is that the proposed approach does not seem to demonstrate good identity and detail preservation as shown in the examples in the paper and the additional rebuttal page. The authors have argued that identity preservation is studied throughout the work. 

Overall, the AC finds this paper interesting and the rebuttal relatively convincing. On balance, the AC thinks that the merits of the paper slightly outweigh its flaws, and decided to recommend acceptance by the end. The authors are highly encouraged to add new results and discussions into the main paper based on reviewers' detailed comments.
Decision: Accept (poster)