SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, the original setup runs for a predetermined number of steps (6100, 1000, and 500 for Textual Inversion, Dreambooth-LoRA, and Custom Diffusion accordingly), sampling 8 images every 500/100/100 iterations and computing their CLIP similarity to the training set images. For each customization method, we tune the hyperparameters of CLIP-s: sampling frequency (number of training iterations between generating samples), as well as the threshold for minimal improvement and the number of consecutive measurements without improvement before stopping. These hyperparameters are chosen on a held-out set of 4 concepts to achieve  $90\%$  of the maximum possible train CLIP image score for the least number of iterations. The final checkpoint is selected from the iteration with the best train CLIP image score.
After running CLIP-s on all concepts, we determine the average and maximum number of steps before early stopping across concepts. Then, we run the baseline method with a reduced number of iterations and no intermediate sampling. This baseline can serve as an estimate of the minimum time required for the adaptation to converge, as it has the most efficient iterations among all methods. However, in real-world scenarios and for new customization approaches, the exact number of iterations is unknown in advance, which makes this method less applicable in practice, as it would involve rerunning such large-scale experiments for every new approach.
The results of our experiments are shown in Table 1. As we can see, DVAR is more efficient than the baseline and CLIP-s in terms of the number of iterations and overall runtime, approaching the performance of "Few iters" (which can be considered an oracle method) while being fully input-adaptive and not relying on costly intermediate sampling. In particular, the fixed step count of "Few Iters" results in higher standard deviations for almost all metrics: from a practical standpoint, it means that this number of iterations might be either sufficient or excessive when adapting a model to a given set of images. We additionally discuss the need for the adaptive number of iterations in Appendix G.
Furthermore, although CLIP-s and the original setup optimize the train CLIP image score by design (in case of the baseline, this metric is used for choosing the best final checkpoint), DVAR is able to achieve nearly the same final results. Finally, with a sufficiently long training time, the increase in the train CLIP image score leads to a decline in the validation CLIP text score, which indicates overfitting on input images: this phenomenon is illustrated in more detail in Figure 4 and Appendix H. As we can see from the CLIP text scores of DVAR, early stopping helps mitigate this issue.
4.4 Human evaluation
To verify the validity of our findings, we compare the quality of images obtained with DVAR with that of the original methods. To do that, we used a crowdsourcing platform, asking its users to compare samples of the baseline approach and samples from the model finetuned for the number of iterations determined by DVAR.
We conducted two surveys: the first survey compared samples derived from prompts seen by the model during training, measuring the reconstruction ability of adaptation. Participants were instructed
Table 1: Comparison of speedup methods for three approaches to text-to-image personalization. The best value is in bold.
Method	Train CLIP img	Val CLIP img	Val CLIP txt	Iterations	Time, min
Textual Inversion
Baseline	0.840±0.051	0.786±0.075	0.209±0.021	6100	27.0±0.3
CLIP-s	0.824±0.053	0.757±0.067	0.233±0.024	666.7±174.5	9.6±2.5
Few iters (mean)	0.796±0.069	0.744±0.073	0.232±0.023	475	1.6±0.0
Few iters (max)	0.806±0.066	0.767±0.071	0.219±0.022	850	2.8±0.0
DVAR	0.795±0.067	0.748±0.068	0.227±0.024	566.0±141.5	3.1±0.8
DreamBooth-LoRA
Baseline	0.865±0.045	0.833±0.067	0.203±0.019	1000	7.8±1.5
CLIP-s	0.862±0.045	0.788±0.075	0.225±0.022	353.2±88.1	6.1±1.5
Few iters (mean)	0.855±0.052	0.806±0.085	0.219±0.023	367	2.0±0.1
Few iters (max)	0.851±0.053	0.800±0.097	0.214±0.019	500	2.6±0.1
DVAR	0.784±0.106	0.687±0.140	0.238±0.034	665.3±94.9	4.9±0.7
Custom Diffusion
Baseline	0.755±0.077	0.695±0.069	0.258±0.021	500	6.5±0.9
CLIP-s	0.757±0.076	0.691±0.069	0.258±0.023	510.4±134.1	9.7±3.0
Few iters (mean)	0.751±0.078	0.691±0.070	0.259±0.023	450.0±0.0	3.4±0.9
Few iters (max)	0.754±0.078	0.691±0.073	0.257±0.022	700.0±0.0	5.3±1.4
DVAR	0.742±0.074	0.693±0.066	0.259±0.022	348.1±46.6	3.4±1.0
to select the image that resembled the reference object the most. The second survey compared samples generated using novel prompts, measuring the final model capability for customization. Participants were asked to determine which image corresponded better to the given prompt. Each pair of images was assessed by multiple participants, ensuring an overlap of 10 responses. For each set of 4 responses, we paid  $0.1$ , and the annotation instructions can be viewed in Appendix I.
Our findings obtained from these evaluations are shown in Table 2. As we can see, DVAR allows for early stopping without sacrificing the reconstruction quality for two out of three evaluated customization methods. Hence, our approach enables efficient training and reduces the computational cost of finetuning on new concepts. While applying DVAR to Textual Inversion slightly reduces the reconstruction quality, this can be attributed to the overfitting of the original method: other methods, which use significantly fewer iterations, are able to avoid this issue.
Table 2: Human evaluation results: the percentage of cases where the quality of DVAR was better or equal to Baseline quality.
Method	Reconstruction	Customization
Textual Inversion	41.6	77.7
DreamBooth-LoRA	67.9	91.4
Custom Diffusion	69.9	93.8
4.5 Analysis and ablation
Having demonstrated the advantages of DVAR, we now perform a more detailed analysis of the effect of the changes we introduce to the procedure of text-to-image adaptation. We aim to answer a series of research questions around our proposed criterion and the behavior of the training objective with all factors of variation fixed across iterations.
Is it possible to observe convergence without determinism? To answer this question, we conduct a series of experiments where we "unfix" each component responsible for the stochasticity of the training procedure one by one. For each component, we aim to find the smallest size of the batch that preserves the indicativity of the evaluation loss. For large batch sizes, we accumulate losses from several forward passes.
Method: Textual Inversion. Validation prompt: "A photo of  $<\text{clock}>$  on the beach"
Method: DreamBooth. Validation prompt: "A <cat> wearing sunglasses"
Method: Custom Diffusion. Validation prompt: "An <elephant> skeleton in the National Science Museum"
Validation
Train
Reference
Figure 4: Comparison of early stopping techniques applied to personalization methods.
CLIP-s
Few Iters Mean
Few Iters Max
DVAR
Figure 5 shows the results of these experiments. While for some parameters increasing the batch size leads to a more evident trend in loss, the stochasticity brought by varying the timesteps  $t$  on each iteration cannot be eliminated even with batches as large as 512. This can be explained by the scale of the reconstruction loss being highly dependent on the timestep of the reverse diffusion process. Thus, aggregating it over multiple different points makes it completely uninformative, as the scale of the trend is several orders of magnitude less than the average values of the loss.
Is it possible to use smaller batches for evaluation and still detect convergence? To address this question, we ran experiments with deterministic batches of size 1, 2 and 4 for multiple concepts and generally observed the same behavior (as illustrated in Figure 8 of Appendix C). However, the dynamics of the objective become more dependent on the timesteps that are sampled: as a result, the early stopping criterion becomes more brittle. Hence, we use a batch size of 8 in our main experiments, as it corresponds to a reasonable tradeoff between stability and computational cost.
Does the selection of diffusion timesteps influence the behavior of the deterministic objective? In our approach, timesteps are uniformly sampled from a range of 0 to 1000 once at the beginning of the training process. However, according to the analysis presented in prior work [15], timesteps from
Figure 5: Loss behavior in the semi-deterministic setup: row names correspond to inputs that are resampled for each evaluation batch.
different ranges have varying effects on the results of text-to-image generation. This raises a question of whether it is possible to sample timesteps only from a specific range, as it might lead to a higher correlation between the loss and the target metrics. Our findings in Appendix J indicate that there is no significant difference in correlations when we change the sampling interval.
In this paper, we analyze the training process of text-to-image adaptation and the impact of different sources of stochasticity on the dynamics of its objective. We show that removing all these origins of noise makes the training loss much more informative during training. This finding motivates the creation of DVAR, a simple early stopping criterion that monitors the stabilization of the deterministic loss based on its running variance. Through extensive experiments, we verify that DVAR reduces the training time by 2 - 8 times while still achieving image quality similar to baselines.
Our work offers a robust and efficient approach for monitoring the training dynamics without requiring computationally expensive procedures. Moreover, it has adaptive runtime depending on the input concept, which naturally mitigates overfitting. These advantages make DVAR a useful tool for text-to-image researchers and practitioners, as it provides a more efficient and controlled training process, ultimately leading to improved results while avoiding the unnecessary computational overhead.
[17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[18] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[19] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
[20] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuzhhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[21] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  $p+$ : Extended textual conditioning in text-to-image generation, 2023.
[22] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning, 2023.
[23] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[24] Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Gradient-free textual inversion, 2023.
[25] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
[26] Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. Magicmix: Semantic mixing with diffusion models. arXiv preprint arXiv:2210.16056, 2022.
[27] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[28] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023.
[29] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021.
[30] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Most existing methods for text-to-image personalization achieve the goal by finetuning diffusion models. In addition to the approaches evaluated in our work, there exist many other techniques that were recently proposed. For example, in  $P +$  [21], the concept is inverted into a sequence of per-layer tokens, and SVDiff [22] focuses on learning the singular values of the SVD decomposition of weight matrices. Imagic [23] divides the adaptation process into two phases. In the first phase, this method optimizes the embedding in the space of texts instead of tokens, and in the second one, it trains the model separately to reconstruct the target image from the embedding. The optimized and target embeddings can then be used for image manipulation by interpolating between them. Furthermore, adaptation can also be performed with gradient-free methods using evolutionary algorithms [24].
Text-to-image model customization can also be broadly viewed as an instance of image editing: given the caption and the image generated from it, we want to change the picture according to the new prompt. There are several methods in this field that use diffusion models and have similar problem statements. For example, Prompt-to-Prompt [25] solves the task by injecting the new prompt into cross-attention, and MagicMix [26] replaces the guidance vector after several steps during inference. The intrinsic disadvantage of such methods is the need for an additional inversion step for editing existing images, like DDIM [27] or Null-text Inversion [28].
B Indicators of convergence for other customization methods
We provide examples of the behavior of standard model convergence indicators for other techniques for text-to-image personalization in Figures 6 and 7. While the reconstruction loss and the gradient norm are still hardly informative, both the CLIP image score and  $\mathcal{L}_{det}$  exhibit a trend that corresponds to improvements in the visual quality of samples.
Figure 6: Convergence process for Custom Diffusion applied to an example concept.
Step 150
Step 300
Step 500
C The impact of the batch size on convergence dynamics
As we increase the batch size for our evaluation objective, we sample a more diverse range of timesteps, making the loss less dependent on the random seed and more indicative of convergence. Figure 8 provides an illustration of this phenomenon.
Moreover, we provide the results of training in a regular Textual Inversion setup with much larger batches. As we demonstrate in Figure 9a, the loss dynamics remain the same even if we increase the batch size to 128, which is 64 times larger than the default one and thus requires  $64\mathrm{x}$  more time for a single optimization step.
Although the gradient norm of training runs with extremely large batches begins to display convergence (see Figure 9b), using the batches of this size is less practical than using an early stopping
Figure 7: Convergence process for DreamBooth applied to an example concept.
Figure 8:  $\mathcal{L}_{det}$  behavior in the fully deterministic setup with varying batch sizes.
criterion. Specifically, when training with these batch sizes, the quality of samples reaches its peak at approximately 250 training steps. This corresponds to  $\approx 16,000$  forward and backward passes with the batch size of 4 (or 8,000 with the batch size of 8 — the largest size that fits into an 80GB GPU for Stable Diffusion) and thus gives no meaningful speedup to the inversion procedure.
D Example pseudocode of using DVAR for training
In Figure 10, we provide an example PyTorch training loop that uses an implementation of DVAR defined in Figure 3. This code samples the evaluation batch once before training and computes  $\mathcal{L}_{det}$  on it after each training iteration. A full implementation of training with DVAR is given at github.com/yandex-research/DVAR.
(a) Training loss dynamics.
Figure 9: Training dynamics of Textual Inversion for a single concept with different batch sizes.
(b) Gradient norm dynamics.
def training_loop(model, train_dataloguer, eval_losses, eval_batch, optimizer, window_size, threshold): eval_images,eval_capions  $=$  eval_batch eval_stochastic  $\equiv$  sampleeverything.eval_batch) eval-latents,eval_timesteps,eval_noisy-latents  $=$  eval_stochastic   
for train_batch in train_dataloguer: images,captions  $=$  train_batch latents,timesteps,noisy-latents  $=$  sampleeverything(train_batch) optimizer.zero_grad() denoised-latents  $=$  model(capions, noisy_latents, timesteps) train_loss  $=$  F.mse_loss(denoised-latents, latents) train_loss.Backward() optimizer step() with torch.no_grad(): eval_denoised-latents  $=$  model( eval_capions, eval_noisy-latents, eval_timesteps ） eval_loss  $=$  F.mse_loss(eval_denoised-latents,eval_latents) eval_losses.append(eval_loss) if DVAR(torch.stack.eval_losses),window_size,threshold): break
Figure 10: PyTorch-like pseudocode of training with DVAR.
E Evaluation of other stopping criteria
Besides the ratio of the rolling variance to the global variance used in DVAR, we also experimented with other metrics for early stopping that consider the behavior of  $\mathcal{L}_{det}$ . All of these metrics have a hyperparameter  $n$  that denotes the number of last objective values to use for computing aggregates.
To obtain the EMA percentile, we calculate the exponential moving average (with  $\alpha = 0.1$ ) at the moment  $t$  and  $n$  steps back, then apply the following formula:
$$
\mathcal {M} _ {E M A} = \frac {E M A (t) - E M A (t - n)}{E M A (t - n)}.
$$
The Hall criterion is simply the difference between the rolling maximum and the rolling minimum divided by the rolling average over n steps:
$$
\mathcal {M} _ {H a l l} = \frac {\max (\mathcal {L} _ {d e t} ^ {n}) - \min (\mathcal {L} _ {d e t} ^ {n})}{\mathrm {m e a n} (\mathcal {L} _ {d e t} ^ {n})},
$$
where  $\mathcal{L}_{det}^n$  denotes the last  $n$  values of  $\mathcal{L}_{det}$ .
The Trend metric is the slope of a linear regression trained on loss values in a window of size  $n$ , computed at each step using the closed-form solution for linear regression. The main issue of this criterion is its longer evaluation time compared to others.
The dynamics of all these metrics are shown in Figure 11. The main challenge we face with these metrics is their unstable behavior, which prevents the transfer of hyperparameters between concepts.
Figure 11: The dynamics of metrics for additional stopping criteria.
F Quantitative comparison with additional metrics
We conduct an additional quantitative evaluation of different early stopping approaches with two metrics introduced in [7] named DINO and DIV.
The DINO metric represents the average cosine similarity between ViT-S/16 DINO [29] embeddings of generated and real images. DINO is more sensitive to differences between subjects of the same class than CLIP image score, which is more suitable for measuring the degree of unique feature preservation. By contrast, DIV was proposed to evaluate the diversity of generated images. It is computed as the average LPIPS [30] cosine similarity between samples generated with the same prompt. Low values of DIV signify the model overfitting the training images.
Table 3: Comparison of identity preservation and diversity of speedup methods for three text-to-image customization techniques. Standard deviations over 48 concepts are in subscript.
Metric	Baseline	CLIP-s	Few iters (mean)	Few iters (max)	DVAR
Textual Inversion
DINO	0.635±0.094	0.590±0.110	0.559±0.133	0.591±0.103	0.566±0.119
DIV	0.742±0.078	0.769±0.097	0.768±0.099	0.757±0.092	0.777±0.091
DreamBooth-LoRA
DINO	0.721±0.103	0.709±0.093	0.711±0.111	0.704±0.125	0.577±0.206
DIV	0.565±0.154	0.630±0.100	0.632±0.086	0.592±0.127	0.585±0.114
Custom Diffusion
DINO	0.475±0.139	0.471±0.140	0.475±0.143	0.488±0.145	0.454±0.136
DIV	0.753±0.061	0.748±0.063	0.753±0.069	0.756±0.068	0.740±0.055
Table 3 shows that DVAR increases the DIV metric for two personalization techniques, which indicates reduced overfitting, whereas having only minor losses in subject fidelity as measured by DINO. Overall, we conclude that our approach does not cause any significant quality degradations while decreasing customization time in an adaptive manner.
G Variance across concepts
To additionally justify the need for an adaptive early stopping technique, we conduct two experiments. First, we analyze the variability of  $\mathcal{L}_{det}$  across concepts. Second, we compare the distribution of stop iterations selected by DVAR with those selected by non-adaptive early stopping methods.
G.1  $\mathcal{L}_{\text{det}}$  variation over concepts
In Figure 12, we depict the behavior of normalized  $\mathcal{L}_{det}$  for four concepts across all three personalization methods. Normalization is necessary, because the scale of loss varies highly across concepts.
Figure 12: Behavior of  $\mathcal{L}_{det}$  for various concepts and methods.
From this plot, we can conclude that the dynamics of  $\mathcal{L}_{det}$  moderately differ from one concept to another. For example, the objective function exhibits earlier saturation on some concepts and methods and demonstrates a more unstable behavior on others.
G.2 Final steps distribution
We compare the distribution of the final steps selected by DVAR to the non-adaptive methods. As we can see from Figure 13, simply throwing away some of the iterations might be sub-optimal for some concepts, indicating the need for an adaptive early stopping technique.
Figure 13: DVAR stop iterations distribution across concepts in relation to non-adaptive baseline.
H Qualitative comparison
In Figure 14, we show a side-by-side comparison of DVAR and Baseline methods. Validation images are obtained from the final training checkpoints using the validation prompt. This comparison confirms that our method achieves similar results in terms of concept reconstruction and prevents overfitting for all three adaptation approaches.
Moreover, this comparison demonstrates that focusing only on the quality of samples during training can be detrimental to the model's ability to transfer concepts to unseen contexts. In such cases, the model tends to generate outputs that resemble the training images regardless of the input prompt.
Validation prompt: «A chair with the <cat> sitting on it near a beach»
Validation prompt: «A <grey sloth plushie> with a tree and autumn leaves in the background»
Validation prompt: «A <dog> on the beach»
Validation prompt: «A <dog> oil painting, Ghibli inspired»
Reference
Figure 14: Side-by-side comparison of DVAR reconstruction quality and customization ability in relation to the baseline.
Baseline Train
DVAR Train
Baseline Validation
DVAR Validation
I Human evaluation instructions
In Figure 15, we provide the instructions for the quality assessment tasks that were shown to the annotators. We aimed to make the instructions as unambiguous and detailed as possible, minimizing the potential influence of different interpretations among individuals. All annotators were required to complete a training task specifically designed to ensure a clear understanding of the decision-making algorithm. Furthermore, we incorporated control tasks with obvious correct answers to filter out bots or low-quality annotators who might vote randomly.
Task description:
Two neural networks tried to learn the object depicted by the Reference image and to generate it in another environment (e.g. another background, view angle, style or composition) described in text prompt on the top of the task. Please help us understand which neural network does the job better in terms of matching the text prompt by answering one question for a given pair of generated images.
How to answer the question: