Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation
Authors: SHIH-YING YEH
Yu-Guan Hsieh
Zhidong Gao
Bernard B W Yang
Giyeong Oh
Yanmin Gong
Venue: ICLR 2024 poster (2024)
Keywords: Text-to-Image
Model Customization
Parameter-Efficient Fine-Tuning
Model Evaluation
Primary area: generative models
TL;DR: We introduce LyCORIS, an open-source library for parameter-efficient Stable Diffusion fine-tuning, propose a comprehensive evaluation framework, and perform extensive experiments to compare different methods.

Abstract: Text-to-image generative models have garnered immense attention for their ability to produce high-fidelity images from text prompts.  Among these, Stable Diffusion distinguishes itself as a leading open-source model in this fast-growing field.  However, the intricacies of fine-tuning these models pose multiple challenges from new methodology integration to systematic evaluation.  Addressing these issues, this paper introduces LyCORIS (Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion), an open-source library that offers a wide selection of fine-tuning methodologies for Stable Diffusion.  Furthermore, we present a thorough framework for the systematic assessment of varied fine-tuning techniques. This framework employs a diverse suite of metrics and delves into multiple facets of fine-tuning, including hyperparameter adjustments and the evaluation with different prompt types across various concept categories. Through this comprehensive approach, our work provides essential insights into the nuanced effects of fine-tuning parameters, bridging the gap between state-of-the-art research and practical application.

Published as a conference paper at ICLR 2024
NAVIGATING TEXT-TO-IMAGE CUSTOMIZATION: FROM LYCORIS FINE- TUNING TO MODEL EVALUATION
Shih-Ying Yeh NTHU ay@kblueleaf. net
Yu-Guan Hsieh" "t
Zhidong Gao"
Apple Syberhsieh21222 gma com com
UTSA zhidong. gao@ut.sa.edi
Bernard B W Yang
Giyeong Oh Yonsei University
Yanmin Gong
University of Toronto by39768gma com
UTSA
hard2251 @ard2251@yonsel1 ac kr
gongyanmin@gmall com
ABSTRACT
Text-to-image generative models have garnered immense attention for their ability to produce high-fidelity images from text prompts. Among these, Stable Diffusion distinguishes itself as a leading open-source model in this fast-growing field. However, the intricacies of fine-tuning these models pose multiple challenges from new methodology integration to systematic evaluation. Addressing these issues, this paper introduces LyCORIS (Lora be Yond Conventional methods, Other Rank adaptation Implementations for Stable diffusion), an open-source library that offers wide selection of fine-tuning methodologies for Stable Diffusion. Furthermore, we present a thorough framework for the systematic assessment of varied fine- tuning techniques. This framework employs a diverse suite of metrics and delves into multiple facets of fine-tuning, including hyperparameter adjustments and the evaluation with different prompt types across various concept categories. Through this comprehensive approach, our work provides essential insights into the nuanced effects of fine-tuning parameters, bridging the gap between state-of-the-an research and practical application.
1 Introduction
The recent advancements in deep generative models along with the availability of vast data on the internet have ushered in
of text-to-image synthesis (Balaji et al., 2022; Ramesh et al., 2022; Saharia et al., 2022). These models allow users to transform text prompts into high-quality, visually appealing images, revolutionizing the way we conceive of and interact with digital media (Ko et al., 2023; Zhang et al., 2023). Moreover, the models' wide accessibility and user-friendly interfaces extend their influence beyond the research community to laypeople who aspire to create their own artworks. Among these, Stable Diffusion (Rombach et al., 2022) emerges one of the pioneering open-source models offering such capabilities. Its open-source nature has served as a catalyst for multitude of advances, attracting both researchers and casual users alike. Extensions such as cross-attention control (Liu et al., 2022) and ControlNet (Zhang et al., 2023) have further enriched the landscape, broadening the model's appeal and utility.
While these models offer an extensive repertoire of image generation, they often fall short capturing burgeoning interest in model customization
highly personalized or novel concepts, leading techniques. Initiatives like DreamBooth (Ruiz et al., 2023) and Textual Inversion (Gal et al., 2023) have spearheaded efforts in this domain, allowing users imbue pretrained models like Stable Diffusion with new concepts through a small set of representative images (see Appendix A for detailed related work). Coupled with user frienfriy trainers designed customize Stable Diffusion. the ecosystem now boasts a plethora of specialized models and dedicated platforms that host them often witnessing the upload of thousands of new models to single website just one week.
"qqual contribution.
Corresponding author. Work done during the author's 's Ph.D. at Université Grenoble Alpes.

---

Published as a conference paper at ICLR 2024
In spite of this burgeoning landscape, our understanding of the intricacies involved in fine-tuning these models remains limited. The complexity of the task fask-from variations datasets, image types, and captioning strategies, to the abundance of available methods each with their sets of tyyperparametes--rendde
it a challenging terrain to navigate. While new methods proposed by researchers offer much potential, they are not always seamlessly integrated into the existing ecosystem, which can hinder comprehensive testing and wider adoption. Moreover, current evaluation paradigms lack systematic approach that covers the full depth and breadth of what fine-tuning entails. To address these gaps and bridge the divide between research innovations and casual usage, we present our contributions follows.
1. We develop LyCORIS, open source library dedicated fine-tuning of Stable Diffusion. This library encapsulates a spectrum of methodologies ranging from the most standard LoRA to number of emerging strategies such as LoHa, LoKr, GLoRA, and (IA)² that are newer and lesser-exploed in the context of text-to-image models.
2. To enable rigorous comparisons between methods, we propose comprehensive evaluation framework that incorporates a wide range of metrics, capturing key aspects such as concept fidelity, text-image alignment, image diversity, and preservation of the base model's style.
3. Through extensive experiments, we compare the performances of different fine-tuning algorithms implemented in LyCORIS and assess the impacts of various hyperparameters, offering insights into how these factors influence the results.
2
Preliminary
In this section, we briefly review the two core components of our study: Stable Diffusion and LoRA for model customization.
2.1 STABLE DIFFUSION
Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) are a family of probabilistic generative models that are trained to capture a data distribution through sequence of denoising operations. Given initial noise map XT (0, I), the models iteratively refine it by reversing the diffusion process until it is synthesized into a desired image Xy. These models can be conditioned on elements such text prompts, class labels, or low-resolution images, allowing conditioned generation.
Specifically, our work is based on Stable Diffusion, a text-to-image latent diffusion model (Rombach et al., 2022) pretrained the LAION 5-billion image dataset (Schuhmann et al., 2022). Latent diffusion models reduce the cost of diffusion models by shifting the denoising operation into the latent space of a pre-trained variational autoencoder, composed of encoder and decoder D. During training, the noise is added to the encoder's latent output z (xo) for each time step t .O,..., T,, resulting in noisy latent 2,. Then, the model is trained to predict the noise applied to ze, given text conditioning с = T()) obtained from an image description I (also known as the image's is caption) using text encoder Formally, with f denoting the parameter of the denoising U-Net and (-)-) representing the predicted noise from this model, we aim to minimize
((A) Exp.c,e |||e co(z,, t, 1)
(1)
where Xo. drawn from the dataset, ~ (O, 1), and t is uniformly drawn from (1....,Ty
2.2 MODEL CUSTOMIZATION WITH LoRA
To enable more personalized experiences, model customization has been proposed as a means to adapt foundational models to specific domains or concepts. In the case of Stable Diffusion, this frequently involves fine-tuning pretrained model by minimizing the original loss function (1) new dataset, containing as few as a single image for each target concept. this process, we introduce concept descriptor [V] for each target concept, comprising neutral trigger word [Vrigger] and an optional class word [Vauss] to denote the category to which the concept belongs. This concept descriptor is intended for use in both the image captions and text prompts. While it is possible to include a prior- prior-preservation loss by utilizing a set of regularization images (Kumari et al., 2023; Ruiz al., 2023), have chosen not employ this strategy in the current study.
2

---

Published as a conference paper at ICLR 2024
h
h
Frozen Weight
B R X
Frusen Weight
BERY
BER X
Frusen Weight
BER
X
Ce R
X
Wi e 12!19
AE
Wa E 3P4
A R"
An E R
Wa E R!
A AER'
LoRA
Lotta
Lokr
Matrix Product
Hadamard Product
Kronecker Product
Figure This ligure shows structure of the proposed LoHa and LoKr modules implemented LyCORIS.
Low-Rank Adaptation (LoRA). When integrated into the model customization process, Low-Rank Adaptation (LoRA) could substantially reduce the number of parameters that need be updated. It was originally developed for large language models (Hu et al., 2021), and later adapted for Stable Diffusion by Simo Ryu (2022). LoRA operates by constraining fine-tuning the original parameter space. More specifically, the weight update AW E R is pre-factorized into two low-rank matrixes B Rpxr, A E IRrx, where p, q the dimensions of the original model parameter, r is the dimension of the low-rank matrix, and r & min(p, q) During fine-tuning, the foundational model parameter Wo remains frozen, and only the low-rank matrices are updated. Formally, the forward pass of h' = Woh + b is modified to:
low-rank subspace of
h Woh + b + W Woh + b ABA,
(2)
where is a merge ratio that balances the retention pretrained model information and its adaptation the target concepts." Following Hu al. (2021), we further define "yr that y a/r.
The LyCORIS Library
Building upon the initiative LoRA, this section introduces LyCORIS, our open-source library that provides array of different methods for fine-tuning Stable Diffusion.
3.1 DESIGN AND OBJECTIVES
LyCORIS stands for Lora beyond Conventional methods, Other Rank adaptation Implementations for Stable diffasion. Broadly speaking, the library's main objective is to serve as test bed for users to experiment with variety of fine-tuning strategies for Stable Diffusion models. Seamlessly integrating into the existing ecosystem, LyCORIS graphic interfaces, allowing users to leverage the algorithms implemented in the library effortlessly. Additionally, native support exists in popular user interfaces designed for image generation, facilitating the of models fine-tuned through LyCORIS methods. For most of the algorithms implemented in LyCORIS, stored parameters naturally allow for the reconstruction of the weight update AW. This design brings inherent flexibility: it enables the weight updates be scaled and applied base model W/ different from those originally used for training, expressed as W W/ + AAW. Furthermore, weight update can be combined with those from other fine-tuned models, further compressed, or integrated with advanced tools like ControlNet. This opens up diverse range of possibilities for the application these fine-tuned models.
compatible with casy-to-use command- -ine tools and
3.2 IMPLEMENTED ALGORITHMS
We now discuss the core of the library-the the algorithms implemented in LyCORIS. For conciseness, we will primarily focus on three main algorithms: LoRA (LoCon), LoHa, and LoKr. The merge ratio 7 a/r introduced (2) is implemented for all these methods.
Setttin is mathematically equivalent to scaling the initialization of B and A by /7 and scaling the leamning rate by /r or 7; depending the used optimizer. See Appendix B.1 for generalization of this result.
3

---

Published as a conference paper at ICLR 2024
LoRA (LoCon). In the work of Hu et al. (2021), the focus was centered on applying the low-rank adapter to the attention layer within the large language model. In contrast, the convolutional layers play key role in Stable Diffusion. Therefore, we extend the method to the convolutional layers of diffusion models (details are provided in Appendix B.2). The intuition is with more layers getting involved during fine-tuning, the performance (generated image quality and fidelity) should be better.
LoHa. Inspired by the basic idea underlying LoRA, we explore the potential enhancements in fine-tuning methods. In particular, it is well recognized that methods based on matrix factorization suffer from the low-rank constraint. Within the LoRA framework, weight updates are confined within the low-rank space, inevitably impacting the performance of the fine-tuned model. To achieve better fine-tuning performance, we conjecture that a relatively large rank might be necessary, particularly when working with larger fine-tuning datasets when the data distribution of downstream tasks greatly deviates from the pretraining data. However, this cloud leads to increased memory usage and more storage demands.
FedPara (Hyeon-Won Woo et al., 2022) is a technique originally developed for federated learning that aims mitigate the low-rank constraint when applying low-rank decomposition methods to federated learning. One of the advantages of FedPara is that the maximum rank of the resulting matrix is larger than those derived from conventional low-rank decomposition (such as LoRA). More precisely, for AW = (B) @ (B2A2), where @ denotes the Hadamard product (element wesse product), B1, B2 E RRxx AI, A E Rrx,, and r S min(p, q). the rank of AW can be as large as r². To make fair comparison, we assume the low-rank dimension in equation (2) is 2r, such that they have the same number of trainable parameters. Then, the reconstructed matrix AW BA has a maximum rank of 2r. Clearly, 2r < r2, if 2. This implics decomposing the weight update with the Hadamard product could improve the fine-tuning capability given the same number of trainable parameters. We term this method LoHa (Low-rank adaptation with Hadamard product). The forward pass of h = Woh + b is then modified to:
h' Wih b b + AWhh Wih + b 7 [(BrA1 o (B2A2) h.
(3)
LoKr. In the same spirit of maximizing matrix rank while minimizing parameter count, our library offers LoKr (Low-rank adaptation with Kronecker product) as another viable option. This method an extension of the KronA technique, initially proposed by Edalati al. (2022) for fine-tuning of language models, and employs Kronecker products for matrix decomposition. Importantly, we have adapted this technique work with convolutional layers, similar to what we achieved with LoCon. A unique advantage of using Kronecker products lies in the multiplicative nature of their ranks, allowing us to move beyond the limitations of low-rank assumptions.
Going further, to provide finer granularity for model optional low-rank decomposition (which users can choose to apply or not) that focuses exclusively on the right block resulting from the Kronecker decomposition." 2 In summary, writing & for the Kronecker product, the forward pass h' Woh + b is modified to:
we additionally incorporate an
h' Wih + b + AWWh Woh + b + 7 (C s (BA))
The size of these matrices are determined by two user-specified hyperparameters: the factor f and the dimension r. With these, we have C' E R B E R and A E Igrx,, where
P
Up max (u S min(J, vP) mod u 0), Up
(5)
Up
The two scalars My and B/ are defined in the same way. Interestingly, LoKr has the widest range of potential parameter counts among the three methods and can yield the smallest file sizes when appropriately configured. Additionally, it can be interpreted an adapter that is composed of a number of linear layers, as detailed in Appendix B.3.
Others. In addition to LoRA, LoHa, and LoKr described earlier, our library features other algo- rithms including DyLoRA (Valipour et al., 2022), GLoRA (Chavan al., 2023), and (IA)$ (Liu et al.. 2022). Moreover, between the date of submission and the preparation of the camera-ready version for the main conference, we have further expanded LyCORIS by incorporating more recent advancements, notably OFT (Qiu et al., 2023). BOFT (Liu et al., 2024), and DoRA (Liu et al., 2024). However, the discussion of these supplementary algorithms is beyond the scope of this paper. As shown in Fq. (5), in our implementation, the right block is always the larger of the two.
4

---

Published as a conference paper at ICLR 2024
4 Evaluating Fine-Tuned Text-To-lmage Models
With the wide range of algorithmic choices and hyperparameter settings made possible by LyCORIS, one naturally wonders: Is there optimal algorithm set of hyperparameters for fine-tuning Stable Diffusion? To tackle this question in a comprehensive manner, it is essential to first establish a clear framework for model evaluation.
With this in mind, in this section, we turn our focus to two independent but intertwined components that are crucial for systematic evaluation of fine-tuned text-to-image models: 2) the types of prompts used for image generation and if) the evaluation of the generated images. While these two components are commonly considered as single entity existing literature, explicitly distinguishing between them allows for more nuanced evaluation of model performance (see Appendix A for comprehensive overview of related works on text-to-image model evaluation). Below, we explore each of these components in detail.
4.1 CLASSIFICATION OF PROMPTS FOR IMAGE GENERATION
To fully understand the model's behavior, it is important distinguish between different types of prompts that guide image generation. We categorize these into three main types as follows:
Training Prompts: These are the prompts originally used for training the model. The images generated from these prompts are expected to closely align with the training dataset, providing insight into how well the model has captured the target concepts.
These prompts seek to generate images that generalize learned concepts to broader contexts, going beyond the specific types of images encountered in the training set. This includes, for example, combining the innate knowledge of the base model with the learned concepts, combining concepts trained within the same model, and combining concepts trained across different models which are later merged together. Such prompts are particularly useful to evaluate the disentanglement of the learned representations.
Generalization Prompts:
Concept-Agnostic Prompts: These are prompts that deliberately avoid using trigger words from the training and are often employed to assess concept leak, see c.g., Kumari et al. (2023). When training also involves class words, this category can be further refined to distinguish between prompts that do and do not these class words.
4.2 EVALUATION CRITERIA
After detailing the different types of prompts that guide the image generation process, the next important step is to identify the aspects that we would like to look at when evaluating the generated images, outline below.
Fidelity measures the extent which generated images adhere to the target concept.
Controllability evaluates the model's 's ability generate images that align well with text prompts.
Diversity assesses the variety of images that are produced from a single or
prompts.
Base Model Preservation measures how much fine-tuning affects the base model's 's inherent capabilities, particularly in ways that may be undesirable. For example, if the target concept is an object, retaining the background and style as generated by the base model might be desired.
Image Quality concerns the visual appeal the generated images, focusing primarily on aspects like naturalness, absence of artifacts, and lack of weird deformations. Aesthetics, though related, are considered to be more dependent on the dataset than on the training method, and are therefore not relevant for our purpose.
Taken together, the prompt classification of Section 4.1 and the evaluation criteria listed above offer a nuanced and comprehensive framework for assessing fine-tuned text-to-image models. Notably, these tools also enable to evaluate other facets of model performance, such the ability to learn multiple distinct concepts without mutual interference and the capability for parallel training of multiple models that can later be successfully merged.
5

---

Published as a conference paper at ICLR 2024
5 Experiments
In this section, we perform extensive experiments to compare different LyCORIS algorithms and to assess the impact of the hyperparameters. Our experiments employ the non-EMA version of Diffusion 1.5 as the base model. All the experimental details not included in the main text along with presentations of additional experiments can be found in the appendix.
Stable
5.1 DATASET
Contrary to prior studies that primarily focus on single-concept fine-tuning with very few images, we consider dataset that spans across wide variety of concepts with an imbalance in the number of images for each. Our dataset is hierarchically structured, featuring 1,706 images across five categories: anime characters, movie characters, scenes, stuffed toys, and styles. These categories further break down into various classes and sub-classes. Importantly, classes under "scenes" and "stuffed toys" contain only 4 to 12 images, whereas other categories have 45 200 images per class.
The influence of training captions on the fine-tuned model is also widely acknowledged in the community. is particularly observed that training with uninformative captions such as "A photo of [v]", which are commonly employed in the literature, can lead to subpar results. In light of this, we use a publicly available tagger to tag the training images. We then remove tags that are inherently tied to each target concept. The resulting tags are combined with the concept descriptor to create more informative captions as " [V], , (tag 1),
(tag k ". To justify this choice, comparative analyses for models trained using different captions are presented in Appendix H.3.
5.2 ALGORITIIM CONFIGURATION AND EVALUATION
Our experiments focus on methods that are implemented in the LyCORIS library, and notably LoRA, LoHa, LoKr, and native fine-tuning (note that DreamBooth Ruiz et al., 2023 can be simply regarded as native fine-tuning with regularization images). For each of these four algorithms, we define a set of default hyperparameters and then individually vary one of the following hyperparameters: learning rate. trained layers, dimension and alpha for LoRA and LoHa, and factor for LoKr. This leads to 26 distinct configurations. For each configuration, three models are trained using different random seeds, and three checkpoints are saved along each fine-tuning, giving in this way 234 checkpoints in the end. While other wrrameter-efficenn fine-tuning methods exist in the literature, most of the proposed modifications are complementary to our approach. We thus do not include them for simplicity.
Data Balancing. To address dataset imbalance, we repeat each image a number of times within each epoch to ensure images from different classes are equally exposed during training.
Evaluation Procedure. To evaluate the trained models, we consider the following four types of prompts i) <train> training captions, ii) <trigger> concept descriptor alone, iii) <alter> generalization prompts with content alteration, and iv) <style> generalization prompts with style alteration. Using only the concept descriptor tests if the model can accurately reproduce the concept without using the exact training captions. As for generalization prompts, only single target concept is involved. This thus evaluates the model's 's ability to combine innate and tinettune knowledge. For each considered prompt type, we generate 100 images for each class or sub-class, resulting in total of 14,900 images per checkpoint. Note that we do not include concept- aoncept-agnostic prompts in our experiments.
Evaluation Metrics. Our evaluation metrics are designed to capture the criteria delineated in Section 4.2 and are computed on a per (sub)-class basis. We briefly describe below the metrics that used for each criterion.
Fidelity: We assess the similarity between the generated and dataset images using average cosine similarity and squared centroid distance between their DINOv2 embeddings (Oquab et al., 2023).
Controllability: The alignment between generated images and corresponding prompts measured via average cosine similarity in the CLIP feature space (Radford et al., 2021).
Diversity: Diversity of images generated with single prompt is measured by the Vendi score (Friedman & Dieng, 2023), calculated using the DINOv2 embeddings.
Base Model Preservation: This generally needs to be evaluated on a case-by-case basis, de- pending on which aspect of the base model we would like to retain. Specifically, examine

---

Published as a conference paper at ICLR 2024
Algorthim
Trained
Epoch
Caparity
0.2 SHAP for Similarty (eller)
024
Dikersae
033
Micd Pertroate (960)
Iona
Ioha
lobr
full matrix
Inm 1m-44//e-7
Irm5e-6/1e-6
Immle-3/5e-6
(a) Plots category "movie characters" (scatter plots are for 30 epoch checkpoints)
Algorthm
Trainod Layers
Epoch
Lr Level
63-00.2000 2.1 0.0 0.1.
SHAP for image Similatty (alter)
10mh
107
l8kr
Ixxtor112 17
4mm116/0 (echordd
16/3, Miphant
Plots for category "scene" (scatter plots are epoch checkpoints)
Figure 2: SHAP beeswarm charts and scatter plots for analyzing the impact of change in different algorithm components. In the beeswarm plots, LoRA is in blue, LoHa is in purple, LoKr is in purple red, and native fine- tuning in red. Model capacity adjusted by either increasing dimension (for LoRA LoHa) decreasing factor (for LoKr). the seatter plots, SCD indicates that we use squared centroid distance to measure image similarity. This removes the implicit penalization towards more diverse image sets in the computation of average cosine similarity (see Appendix D.4 for details). We believe it is thus more suitable when we are interested in the trade-off between fidelity and diversity. The error bars in the seatter plots represent standard errors of the metric values across random seeds and classes.
potential style leakage by measuring the standard style loss (Johnson et al., 2016) between base and fine-tuned model outputs for <style> prompts.
Image Quality: Although numerous methods have been developed for image quality assessment, most of them target natural images. As far as we aware, currently, there still lacks systematic approach for assessing the quality of Al-generated images. We attempted experiments with three leading pretrained quality assessment models, detailed in Appendix H.2, but found them unsuitable for our context. We thus do not include any quality metrics in our primary experiments.
Further justification for our metric choices provided in Appendices E and H.1, where we compute correlation coefficients for wider range of metrics and conduct experiments across three classification datasets to assess the sensitivity different image features to change a certain image attribute.
5.3 EXPERIMENTAL RESULTS
To carry out the analysis, we first transform each computed metric value into a normalized score, ranging from 0 I, based on its relative ranking among all the examined checkpoints (234 in total). A score closer to 1 signifies superior performance. Subsequently, these scores are averaged across sub-classes and classes generate set of metrics for each category and individual checkpoint. Alongside the scatter plots, which directly indicate the values of these metrics, we also employ SHAP (SHapley Additive exPlanations) analysis (Lundberg & Lee, 2017) in conjunction with CatBoost regressor (Prokhorenkova et al., 2018) to get clear visualization of the impact of different algorithm components on the considered metrics, as shown in Figure 2. essence, a SHAP value quantifies the impact of given feature the model's output. For more exhaustive presentation of the results, readers are referred to Appendix F.
5.3. CHALLENGES IN ALGORITHM EVALUATION
Before delving into our analysis, it is crucial to acknowledge the complexities inherent in evaluating the performance of fine-tuning algorithms. Specifically, we identify several key challenges, including f) sensitivity to hyperparameters, zi) performance discrepancy across concepts, di) influence of dataset, iv) conflicting criteria, and v) unreliability of evaluation metrics, among others. These challenges are discussed in detail in Appendix C. To mitigate some of these issues, our evaluation encompasses
7

---

Published as a conference paper at ICLR 2024
Target Concept
LoRA dim 8 alpha 4 (50 epochs)
Target Concept
Native fine-tuning (30 epochs)
Attention-only
Attention - MLP
10
5-10-6
LoHa dim alpha
LoHa dim 16 alpha 8
LoHa dim 16 alpha
LoKr factor 8
10 epochs
Figure 3: Qualitative comparison of checkpoints trained with different configurations. Samples of the top row are generated using only concept descriptors while samples of the bottom row are generated with two prompts "[Vasse scene stands against a backdrop of snow-capped mountains" and "[Vantee scene surrounded by a lush, vibrant forest". The number of training epochs is chosen according to the concept category.
large number of configurations and includes separate analyses for each category. Additionally, complement our quantitative metrics with visual inspections conducted throughout our experiments (see Appendix G for extensive set of qualitative results). Finally, we exercise caution in making any definitive claims, emphasizing that there no one-size-fite solution, and acknowledging that exceptions do exist to guiding principles.
5.3.2 ANALYSIS OF RESULTS AND INSIGHTTS FOR FINE-TUNING WITH LYCORIS
In this part, delve into a detailed examination of our experimental results, aiming to glean actionable insights for fine-tuning with LyCORIS. These insights should not be considered as rigid guidelines, but rather as empirical observations designed serve as foundational reference points.
Number of Training Epochs. To analyze the evolution of models over training, generate images from checkpoints obtained after 10, 30, and epochs of training. Due to the small number images available for "scenes" and "stuffed toys" categories and the use of data balancing, the 30 and 50 epoch checkpoints are almost universally overtrained for these concepts, explaining why increasing training epochs decreases image similarity as shown in the SHAP plot of Figure 2b.3 Otherwise, increasing the number of epochs generally improves concept fidelity while compromising text-image alignment, diversity, and base model preservation. Exceptions this trend exist. Specifically, observe the peefft-then-generalle phenomenon, as often illustrated through the double descent curve (Nakkiran al., 2021), certain situations. We explore this further in Appendix G.4.2.
Learning Rate. We consider three levels of learning rate, 5 10-7 10-6, and 5 10 10-6 for native fine-tuning, and 10-4 5 10-4, and 10-3 for the other three algorithms. Within reasonable range, increasing the learning rate seems to have the same effect as increasing the number of training epochs. qualitative example is provided Figure learning rate cannot be remedied by simply extending the training duration (see Appendix G.3.2).
is worth noting, however, that excessively low
Algorithm. A central question driving the development of LyCORIS is assess how different methods of decomposing the model update affect the final model's performance. We summarize our observations in Table We distinguish here between two learning rates for native fine-tuning as they result in models that perform very differently. In particular, Figure 2a reveals that native fine-tuning a learning rate of 5- 10-0 achieves high image similarity for training prompts and high text similarity
We also experimented without data halancing and observed undertraining even after 50 epochs.
8

---

Published as a conference paper at ICLR 2024
LoRA
LoHa
LoKr
Native (Ir 5 x 10-9)
Native (Ir 10-6)
Fidelity
3
2
4
5
1
Controllability
4
1
5
Diversity
3
4
2
5
Table I: A tentative ranking based on different evaluation criteria for the methods we explore in our experiments (the higher the number, the better the method's performance). Although this ranking reflects the general trend observed across different concept categories, deviations of varying degrees are common.
for generalization prompts. However, this strong performance in text similarity for generalization prompts also leads to a lower image similarity score for these prompts, highlighting in this way both the challenges in comprehensive method comparison and the necessity of evaluating on different types of prompts independently. As for LoRA, LoHa, and LoKr, comparisons are based on configurations with similar parameter counts and otherwise the same hyperparameters. Style preservation is not included in this comparison as no consistent trend across concept categories is observed.
Trained Layers. To investigate the effects of fine-tuning different layers, we examine three distinct presets: i) attn-only: where we only fine-tune attention layers; ūi) attn-mip: where we fine-tune both attention and feedforward layers; and iii) full network: where we fine-tune all the layers, including the convolutional ones. As can be seen from the SHAP plots in Figure 2, when all the other parameters fixed, restricting fine-tuning only the attention layers leads to a substantial decrease in image similarity while improving other metrics. This could be unfavorable in certain cases; for instance, the top-left example in Figure 3 shows that neglecting to fine-tune the feedforward layers prevents the model from correctly learning the character's ' uniform. The impact of fine-tuning the convolutional layers is less discernible, possibly because the metries we use are not sensitive enough capture subtle differences. Overall, our observations align with those made by Han et al. (2023). Moreover, it worth noting that with the "attm-only" preset, we also fine-tune the self-attention layers in addition to the cross-attention layers, but this may still be insufficient, as demonstrated above.
Dimension, Alpha, and Factor. We finally inspect number of parameters that are specific to our algorithms- dierrihmns-dmmmsiiooon dimension r. alpha a, and factor f. By default, we set the dimension and alpha of LoRA to and 4, and of LoHa to 4 and As for LoKr, we set the factor further decomposition of the second block. These configurations result in roughly the same parameter counts across the three methods. To increase model capacity, we either increase the dimension or decrease the factor. This is what we refer to "capacity" in the SHAP plots. We note that when the ratio between dimension and alpha is fixed, increasing model capacity has roughly the same effect as increasing learning rate training epochs, though expect the model's 's performance could now vary more greatly when varying other hyperparameters. We especially observe in Figure 2b that the effects could be reversed when alpha is set to 1 (these checkpoints are not included in our SHAP analysis). Some qualitative comparisons are further provided in the bottom row of Figure 3.
and do not perform
6 Concluding Remarks
In conclusion, this paper serves three main purposes. First, we introduce LyCORIS, an open-source library implementing diverse range of methods for Stable Diffusion fine-tuning. Second, we advocate for more comprehensive evaluation framework that better captures the nuances of different fine-tuning methods. Lastly, our extensive experiments shed light on the impact of the choice of the algorithm and their configuration on model performance, revealing their relative strengths and limitations. For instance, based on our experiments, LoHa seems be better suited for simple, multi-concept fine-tuning, whereas LoKr with full dimension is better for complex, single-concept tasks. This distinction in their applicability also indicates that the rank of the matrix update, often considered a key factor, may not always be the definitive predictor of method's efficacy in various fine-tuning scenarios.
Despite our extensive efforts, the scope of our study remains limited. For example, we have not explored the task of generating images with multiple learned concepts, as this aspect is highly sensitive to input prompts and more challenging evaluate. However, recent works such Huang et al. (2023) aim to address these issues, and we believe that incorporating these emerging evaluation frameworks will further enrich the comparison of fine-tuning methods in future studies.

---

Published as a conference paper at ICLR 2024
Acknowledgement
The work of Z. Gao and Y. Gong was supported in part by the US National Science Foundation under grant CNS-2047761. We extend our heartfelt appreciation to the community for their invaluable support, which has been instrumental in bringing LyCORIS to fruition. Our particular gratitude goes those who have directly contributed the library and those who have further integrated it into the existing ecosystem, helping to improve and expand its functionality and reach.
References
[1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mïika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. eDiff-l: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
[2] Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for paramcter-efficient fine-tuning, 2023.
[3] Ali Edalati, Marzieh Tahaci, Ivan Kobyzev, Vahid Partovi Nia, James Clark, and Mehdi Rezagholizadeh. KronA: Parameter efficient tuning with kronecker adapter, 2022.
[4] Dan Friedman and Adji Bousso Dieng. The Vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[5] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference Learning Representations, 2023.
[6] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. SVDiff: Compact parameter space for diffusion fine-tuning. arXiv preprint arXiv:2303. 11305, 2023.
[7] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances neural information procexsing systems, volume 33, pp. 6840-6851, 2020.
[8] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, 2021.
[9] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T21-CompBench: A com- prehensive benchmark for open- world compositional text-to- text-to-image generation. arXiv preprint arXiv:2307.06330, 2023.
[10] Nam Hleon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. FedPara: Low-rank hadamard product for communication-efficii federated learning. International Conference on Learning Representations, 2022.
[11] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Mision-ECCi 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 694-711. Springer, 2016.
[12] Hyung- -won Ko, Gwanmo Park, Hyeon Jeon, Jaemin Jo, Juho Kim, and Jinwook Seo. Large- scale text-to-image generation models for visual artists' creative works. In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 919-933, 2023.
[13] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1931-1941, 2023.
[14] Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot efficiet fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems, volume 35, pp. 1950-1965, 2022.
10

---

Published as a conference paper at ICLR 2024
[15] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. European Conference on Computer Vision, pp. 423-439 Springer. 2022.
[16] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint 44X1v:2402.09333, 2024.
[17] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng. Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In International Conference on Learning Representations, 2024.
[18] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[19] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theooy and Experiment, 202 I (12):124003) 2021.
[20] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov. Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/92. 2023.
[21] Liudmila Prokhorenkova. Gleb Gusev, Aleksandr Vorobev. Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in neural information processing systems, volume 31, 2018.
[22] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng. Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Advances neural information processing system, 2023.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. 8763. PMLR, 2021.
[24] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint NXiv:2204.061225 1(2):3 2022.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
[26] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-22510, 2023.
[27] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, volume 35, pp. 36479-36494, 2022.
[28] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion- 5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, volume 35, pp.
2022.
[29] Simo Ryu. Low-rank adaptation for fast -oxt-e diffusion fine-tuning. https:// github. - cwm./ll...o como: mo/ lora/ 2022. Accessed: 2023-08-14.

---

Published as a conference paper at ICLR 2024
[30] Jascha Sohl- Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
[31] Mojtaba Valipour, Mehdi Rezagholizadeh. Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter efficient tuning of pre-trained models using dynamic search-free low rank adaptation, 2022.
[32] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion model in generative ai: survey. arXiv preprint (xXiv:3333.0700) 2023.
[33] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE International Conference Computer Vision (ICCV), 2023.
12


Reviews:
[{"review_id": "QBqp5H2DfK", "reviewer": "Reviewer_ekPo", "summary": "The authors propose LyCORIS, an open-source library that contains multiple fine-tuning techniques for Stable Diffusion. The authors also explore many improved fine-tuning techniques such as LoCon, LoHa and LoKr. This paper also presents evaluations for different fine-tuning techniques using multiple metrics and prompt types.", "questions": "(1) Please refer to the main questions in the weakness section.\n(2) A minor question: It will be better if the authors provide the results on other versions of stable diffusion, such as SD2.0 and SDXL.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1) The theory and experiments are both solid. The paper has over 57 pages devoted to analyzing the fine-tuning techniques.\n(2) The details for experiments are very clear.\n(3) In addition to the framework, the authors also explore other fine-tuning techniques.", "weaknesses": "(1) The results of this framework combined with ControlNet can be presented in this paper.\n(2) Efficiency (time and GPU memory cost) of different approaches are not provided and analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACE8ZLRX2R", "reviewer": "Reviewer_PnHf", "summary": "This author introduces LyCORIS, an open source library dedicated to fine-tuning of Stable Diffusion, which integrates a comprehensive range of finetuning methods. For rigorous comparisons between the implemented methods, the author proposes a comprehensive evaluation framework that incorporates a wide range of metrics. Based on the evaluation framework, the author performs extensive experiments to compare different fine-tuning algorithms and to assess the impact of the hyperparameters (i.e, training epochs, learning rate, trained layers, et al). Overall, the experiments, comparisons, analyses, and results of the entire paper are very well-rounded and thorough.", "questions": "For this kind of paper that builds benchmarks based on a certain field, I would recommend the author to submit to a journal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Developing an open-source library is of great significance in fostering the advancement of a particular field. After comparing the existing open-source libraries available online, the LyCORIS library offers a relatively more comprehensive set of algorithms.\n\n2. The author has developed a comprehensive benchmark to evaluate various algorithms from multiple perspectives, addressing a significant gap in the text-to-image field. This thorough evaluation and comparison of existing finetuning methods have been lacking in the domain until now.\n\n3. The author conducted comprehensive experiments for different algorithms and parameters; in addition, the author also provided a detailed analysis of the current mainstream fine-tuning algorithms.", "weaknesses": "1. HuggingFace has also released the PEFT library, which supports a wider range of pre-trained models and includes the methods mentioned in the paper. Therefore, what are the advantages of the LyCORIS library compared to PEFT?\n\n2. The paper conducted a multitude of experiments and comparisons on existing methods and various hyperparameters, leading to certain conclusions. Based on these findings, could there be a more optimal algorithm or design compared to previous ones?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8PYHA5RVKo", "reviewer": "Reviewer_DWom", "summary": "This paper proposes a comprehensive library for evaluating text-to-image finetuning methods, typically based on LoRA. In addition to different algorithms, it also provides comprehensive evaluation criteria. Finally, some experimental results provide some insight about different finetuning methods.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is a good engineering paper that provides a library for text-to-image finetuning methods evaluation.\n2. It support different matrix factorization techniques such as LoRA, LoHa, LoKr, DyLoRA, GLoRA, GLoKr and so on.\n3. This paper also consider comprehensive evaluation metrics, including fieldity, controllability, diversity, base model preservation and image quality.", "weaknesses": "1. This paper mainly focus on LoRA-based finetuing strategies, can it be expanded to other parameter-efficient finetuning methods such as [1] and [2]? It doesn't provide a clear explanation.\n2. The conclusion about the performance of different finetuning methods is not clearly presented in the experimental section. Maybe some tables can more straightforwardly represent your final conclusions. \n\n[1] Qiu, Zeju, et al. \"Controlling Text-to-Image Diffusion by Orthogonal Finetuning.\" arXiv preprint arXiv:2306.07280 (2023).\n[2] Xie, Enze, et al. \"DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning.\" arXiv preprint arXiv:2304.06648 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Lb5rzzeE6", "reviewer": "Reviewer_EGJf", "summary": "**Summary:** \nThis paper presents an open-source toolkit based on LoRa. I believe this work might be more appropriate for the \"benchmarking and datasets\" track. Positioned here, it's challenging for me to evaluate the innovation this paper offers.", "questions": "I cannot evaluate this paper because I believe it is proper for a benchmarking and dataset track, not the main track.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Remarks:** \nWhile the improvements and variants on LoRa are relatively straightforward, the theoretical part of the paper seems sound.", "weaknesses": "**Recommendation:** \nI would advise the authors to provide clear insights through experiments and offer some specific suggestions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)