Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA
Authors: James Seale Smith
Yen-Chang Hsu
Lingyu Zhang
Ting Hua
Zsolt Kira
Yilin Shen
Hongxia Jin
Venue: Accepted by TMLR ()

Abstract: Recent works demonstrate a remarkable ability to customize text-to-image diffusion models while only providing a few example images. What happens if you try to customize such models using multiple, fine-grained concepts in a sequential (i.e., continual) manner? In our work, we show that recent state-of-the-art customization of text-to-image models suffer from catastrophic forgetting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high quality images of past, similar concepts degrade. To circumvent this forgetting, we propose a new method, C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the popular Stable Diffusion model. Furthermore, we use customization prompts which do not include the word of the customized object (i.e., "person" for a human face dataset) and are initialized as completely random embeddings. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. We show that C-LoRA not only outperforms several baselines for our proposed setting of text-to-image continual customization, which we refer to as Continual Diffusion, but that we achieve a new state-of-the-art in the well-established rehearsal-free continual learning setting for image classification. The high achieving performance of C-LoRA in two separate domains positions it as a compelling solution for a wide range of applications, and we believe it has significant potential for practical impact.

Published in Transactions on Machine Learning Research (04/2024)
Continual Diffusion: Continual Customization of Text-to-lmage Diffusion with C-LoRA
James Seale Smith Samsung Research America Georgia Institute of Technology
Srmnessellemmitt@@@tttee edu
Yen-Chang Hsu
Samsung Research America
Lingyu Zhang Samsung Research America
Ting Hua Samsung Research America
Zsolt Kira Georgia Institute of Technology
Yilin Shen Samsung Research America
Hongxia Jin
Samsung Research America
Reviewed on OpenReview: https: //pp:rv//peeeeveee
openrevie.. net/ forum? id= TZdEqu26f3
Abstract
Recent works demonstrate remarkable ability lo customize text-lo-image diffusion models while only providing a few example images. What happens if you try to customize such models using multiple, fine-grained concepts in a sequential (i.c., contimual) manner? In our work, we show that recent state-of-the-art customization of lext-lo-image models suffer from catastrophic forgetting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high quality images of past, similar concepts degrade. To circumvent this forgetting, propose new method, C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the popular Stable Diffusion model. Furthermore, we use customization prompts which do not include the word of the customized object (iee., "person" " for a human face dataset) and are initialized as completely random embeddings. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. We show that C-LoRA not only outperforms several baselines for our proposed setting of soct-to-image continual customization, which refer to as Continual Diffusion, but that achieve new -tate-of-the-art in the well-established rehearsal-free continual learning setting for image classification. The high achieving performance of C-LoRA in two scparate domains positions it as a compelling solution for a wide range of applications, and we believe it has significant potential for practical impact.

---

Published in Transactions on Machine Learning Research (04/2024)
continual learning
custom diffusion
custom diffusion (sequential)
target data
ours
(merge)
Figure We learn generate text-conditioned images of new concepts a sequential manner (i.c., con- tinual learning). Here show three concepts from the learning sequence sampled after training ten concepts sequentially. SOTA Custom Diffusion (Kumari et al. 2022) suffers from catastrophic forget- ting, so we propose new method which drastically reduces this forgetting.
1 Introduction
Text-to-image generation is a rapidly growing research area that aims to develop models capable of synthe- sizing high-quality images from textual descriptions. These models have the potential to enable a wide range of applications, such as generating realistic product images for a-commerce websites, creating personalizod avalars for virtual reality environments, and aiding in artistic and creative endeavors. Recent advances in this area have shown significant progress in generating images with fine-grain details that can be customized to specific concepts, such generating portrait of one's self or pet, while providing only few example images to "instruct" the model. We ask a simple and practical question: What happens when these concepts arrive in a sequential manner!
Specifically, we advocate that one of the main challenges in these models is to continuously customize them with new, previously unseen fne-grained concepts, while only providing a few examples of these new concepts. We refer to this new setting Continual Diffusion, and to the best of our knowledge we the first to propose this problem sctting. Continual Diffusion provides more efficient engagement with the text-to-image tools, such as visual storytelling and content creation. A key aspect is to not require the model to be re-trained on past concepts after each new concept is added, alleviating both computational and data privacy burdens. However, this is also a challenging problem as we show it induccs strong interference and catastrophic forgetting existing methods.
In this paper, we start by analyzing existing customized diffusion methods in the popular Stable Diffusion model (Rombach et al.] 2002)) showing that these models catastrophically fail for sequentially arriving fine-grained concepts (we specifically use human faces and landmarks). Motivated by the recent Cus- tomDiffusion (Kumari et al. 2022) method, well as recent advances in continual learning, we then propose & continually self-regulan rellfaegulrrized low-rank adaptation in cross attention layers, which we refer to as C-LoRA. Our method leverages the inherent structure of cross attention layers and adapts them to new concepts a low-rank manner, while simultaneously preserving the knowledge past concepts through a selfreegulaizztioon mochanism. Additionally, we propose a custom tokenization strategy that (i) removes the word of the customized object (lypically used in prior works) and (ii) is initialized as random embeddings (rather than arbitrary embeddings of lesser-used words, done previously), and show that this outperforms commonly used customization prompts.
As discussed in Figure 2 we note that existing approaches such as learning task-specific adapters or cus- tomized tokens/wal /wrdss (aal et al.] (002)) are not satisfying solutions. For the former, this type of approach would create different models per concopt, inhibiting the ability to generate content containing multiple learned concepts. For the latter, this type of approach underperforms model fine-tuning when capturing fine-grained concepts such as your specifie face instead of faces in general. Thus, it is critical to find
2

---

Published in Transactions on Machine Learning Research (04/2024)
Concept 1
Concept
Concept 1 & 2
Learn
Learn
Generate
Time
Figure 2: A use case of our work mobile app sequentially learns customized concepts. At later time, the user can generate photos of prior learned concepts. The user should be able to generate photos with multiple concepts together, thus ruling out simple methods such as per-concept adapters. Furthermore, the concepts are fine-grained (e.g. human faces), and simply learning new tokens or words as done in Textual Inversion (Gal et al. 2022 is not effective.
solution like ours which gently adapts the model to learn these new concepts without interfering with or forgetting prior seen concepts. In summary, our contributions in this paper are fourfold:
1. We propose the setting of continual customization of text-to-image models, simply Continual Diffusion, and show that current approaches suffer from catastrophic forgetting in this setting.
2. We propose C-LoRA, which efficiently adapts to new concepts while preserving the knowledge of past concepts through continual, self-regularized, low-rank adaptation in cross attention layers of Stable Dif- fusion.
3. We show that our C-LoRA alleviates catastrophic forgetting the Continual Diffusion setting with both quantitative and qualitative analysis.
4. We extend C-LoRA into the well-established setting of continual learning for image classification, and demonstrate that our gains translate to state-of-the-art performance in the widely used benchmark, demonstrating the versatility of our method.
2 Background and Related Work
Conditional Image Generation Models: Conditional image generation an active research area, with notable works including Generative Adversarial Networks (GANs) (Goodfellow et al.) 2014 Karras ct al. 2020). Variational Autocncoders (VAE) (Kingma & Welling 2013). and more recently, diffusion
models (Dhariwal & Nichol 2021 Ho et al. 2020
Sohl-Dickstein al. 2015 As GAN-based models require
delicate parameter selection and control over generation results usually comes from pre-defined class labels (Mirza & Osindero 2014), we focus on diffusion-based conditional image generation that takes free-fon form text prompts as conditions Ramesh et al.) 2022 Liu et al.) 2022a). Specifically, diffusion models operate by learning a forward pass to iteratively add noise to the original image, and a backward pass to remove the noise for a final generated image. cross attention mechanism introduced in the (transformor-basood)
U-Net (Ronneberger et al. 2015) to inject text conditions.
Recent works have gone beyond text-only guidance to generate custom concepts such unique stuffed toy or a pet. For example, Dreambooth (Ruiz et al. 2022) fine-tunes the whole parameter set in a diffusion model using the given images of the new concept, whereas Textual Inversion (Gal et al. 2022) learns only custom feature embedding "words". While these approaches are used for customizing to a single concept, Custom Diffusion (Kumari et al., (022)) learns multiple concepts using a combination of cross-attention fino-tuning, regularization, and closed-form weight merging. However, we show this approach struggles learn similar, fine-grained concepts in sequential manner (i.e., Continual Diffusion), thus motivating the
3

---

Published in Transactions on Machine Learning Research (04/2024)
need for our work. We note that concurrent work, ZipLoRA (Shah et al. 2023). bears resemblance to our approach; we compare to this method in Section 4
Continual Learning: Continual learning is the setting of training a model on a sequence of tasks, where each task has a different data distribution, without forgetting previously learned knowledge. Existing methods can be broadly classified into three categories based on their approach to mitigating catastrophic forget- ting (MeCloskey & Cohen 1989). Regularization-be based methods (Douillard et al. 2020; Kirkpatrick et all, Li & Hoiem 2017 7 Zenke et al.] 2017) add extra regularization terms to the objective function while learning a new task. For example, EWC (Kirkpatrick et al., 2017) estimates the importance of parameters
2017
and applics per-parameter weight decay. Rehearsal-based methods [Chaudhry et al.
2019a b Hou et al.
2019 Kamra et al.] 2017 Ostapenko et al.]
2019 [ham et al., 2021: Rebulli et al. 2017 Rolnick et al. 2019
Smith et al. 2021 van de Ven et al. 2020 store or generate samples from previous tasks in a data buffer
and replay them with new task data. However, this approach may not always be feasible due to privacy or copyright concerns. Architecture- basod methods (Aljundi et al. 2017 Li ct al.) 2019 Rusu et al.) 2016 Yoon et al. 2017) isolate model parameters for each lask. Recently, prompt-based continual learning methods for 2022c DualPrompt (Wang et al. 2022b), S-Prompt (Wang) 2022b have outperformed rehearsal-based methods without requiring replay buffer. These methods work well for classification type problems, but it is unclear how they would be applied to text-to-image generation given that their contributions focus on inferring discrim- inative properties of the data to form prompts for classification. However, we do compare to these methods in their original setting. We note that generative continual learning is not new (Lesort et al. 2019 Shin et al. 2017 Zhai et al., 2019), but emphasize that these are train-data hungry methods (whereas ours is few-shot fine-tuning), and they do not operate on fine-grained tasks such as individual faces.
Vision Transformers such as L2P (Wang et al.
ct al........... 2022a). and CODA-Prompt (Smith et al.
While the aforementioned work is mainly on the uni-modal continual learning, a few recent approaches have been proposed for the multimodal setting, REMIND (Hayes et al. 2020) proposed continual VQA tasks with latent replay, but their method requires storing compressed training data. CLiMB (Srinivasan ot al. ) proposed CL adaptation to coarsely different VL lasks, such as Visual Question Answering Natural Language for Visual Reasoning (NLVR), Visual Entailment (VE), and Visual Commonsense Reasoning(VCRI, assuming knowledge of the evaluated task-id at inference time. Construct-VL (Smith) et al. 2022a) focuses on the task of natural language visual reasoning. We formulate our problem continual adaptation of Stable Diffusion to multiple, fine-grained concepts, and the majority of the methods discussed in this section do not have a one-to-one translation for our problem setting.
TuninTuning: There have been several proposed approaches for fine-tuning models with fewer parameters, including adapters, low-rank adapters, prompt learning, or simply fine-tuning subsets of the model. [Huulsy et al.] (2019) introduced one of the earliest frameworks for parameter-efficien transfer learning in NLP, proposing the use of adapters to fine-tune pre-trained models efficiently. Building on these ideas, subsequent advancements have included methods like prefix tuning Li & Liang 2021 and prompt tuning (Lester et al.) 2021). : which further refine the process of adapting large models to specific tasks with minimal parameter updates. Some recent works in this area include Delta-T5 <Ding et al. 2022). UNIPELT (Mao et al.) 2021), Polyhistor (Liu et al. 2022b), and VL-ADAPTER (Sung et al., 2022): J. One promising approach is to use low-rank adapters, such as LoRA (Hu et al. 2021), which have been shown to be effective adapting models to new tasks with minimal additional parameters. In this work, we build upon this concept and propose an efficient continual learning neural architecture that layers low-rank adapters on the key and value projection matrices of Stable Diffusion.
Saaameter-fffcieen
3 Method
In our Continual Diffusion sctting, we learn N customization "tasks" t E f1,2...., - 1,N), where N is the total number of concepts that will be shown to our model. Inspired by Custom Diffusion (Kumari et al., 2022), the high level intuition of our method is to find a new and continuous way to update a small number of weights in Stable Diffusion. Our obscrvation is that we should both 1) update our model in way that does not over-fit to our new concept and "leave room" to add future concepts, and 2) not overwrite the information learned from the past concepts (i.e., avoid calastrophic forgetting)- We re-emphasize that we
4

---

Published in Transactions on Machine Learning Research (04/2024)
Diffusion Model
C-LORA (Sec.
U-Net
Initial weights
Past LoRA weight deltas
New LoRA weight deltas
W init
+
A
x
B1
x
Cross Attention
KV
Self-regularization (Sec. 3.2)
Customized Token (Sec. 3.3)
Other modules
Text Encoder
Prompt: Photo of
V=
Via
Dr
V2
or
Vi
Learnalle
Frozen at
Concept sequence:
Accessible
Inaccessible
Figure 3: Our method, C-LoRA, updates the key-value (K-V) projection in U-Net cross-atlention modules of Stable Diffusion using contimual, relulaagg low-rank weight adaptation. The past LoRA weight deltas are used to regulate the new LoRA weight dcltas by guiding which parameters are most available to be updated. Unlike prior work (Kumari et al. 2022), we initialize custom token embeddings as random features and remove the concept name (e.g., "person") from the prompt.
cannot simply use single-concepl adapters, because we desire a model which can produce images of multiple same time. For example, a user of this method may want to produce photo with a
learned concepts at the grandparent sitting with the user's now pet a novel setting single adapters per concept would not enable such a ionaliin
Thus, we propose method with contributions that are two-fold. First, we fine-tune small number of weights using continual, self-regularized low-rank adaptors (LoRA (Hu et al. 2021)), which refer to C-LoRA. We build LoRA due to (1) its parameter efficiency (allowing to store past task parameters for regularization of future tasks), (2) inference efficiency (the learned parameters can be folded back into the original model weights, and thus has zero cost to inference), (3) its unique ability to self-regularize (proposed by us), which cannot be translated to other marameter-effiiien fine-tumig methods, and (4) prior success in VL. discriminate ta (Smith et al. 2022a)..
Our C-LoRA contains key ingredients: (1) add new LoRA parameters per task, and (2) use past LoRA parameters to guide our new parameters adapt the weight matrix in parameters which are less likely to interfere. We keep personalized token embeddings far from each other in the token embedding space with random initialization and remove any object names (different from prior works). This encourages the personalized Lokens to give unique "personalized instructions" to the updated cross-allention layers, rather than having overlapping instructions which can interfere with past and future concepts. The remainder of this section describes the fine details of our approach.
3.1 C-LoRA
Following the findings of Custom Diffusion (Kumari et al.) 2022), we only modify small number of param- eters in the Stable Diffusion model. Why? In addition to its described empirical success, this is intuitively inviting for continual learning in that we reduce the spots in the model which can suffer from catastrophic for- gelting by surgically modifying highly sensitive parameters. However, rather than fine-tune these parameters (as is done in Custom Diffusion), which would induce favorable conditions for interference and catastrophic forgetting, we propose low-rank adaptations designed to self-rogularize to avoid the aforementioned issucs.
Which paramelers do we modify? Stable Diffusion contains three parts: the variational autoencoder (VAF), denoising model U-Net, and the text encoder. Kumari al. (2022) show that the cross-attention parameters
This work focuses using LoRA create pscudo-rchearsal data from past tasks for discriminative task; this is not applicable generation problem as already have generated data, however compare the "hase" part of their method in experiments: LoRA (sequential).
5

---

Published in Transactions on Machine Learning Research (04/2024)
of the U-Net are most sensitive to change during Stable Diffusion customization, and therefore propose to modily only these blocks. We modify the same module, but with a different approach.
V where a the softmax operator, Q = f are query features, K = W c are key features, V = W c are value features f are latent image features, с are text features, and d' the output dimension. The matrices W W WV map inputs f and c the query, key, and value features, respectively.
Consider the single-head cross-attention Vaswwnn et al. 2017) operation given Faum (Q, (V) QK
vd
Following Kumari al. (2022), we only modify W WV, which project the text features, and we refer to these simply WKP.
When learning customization "task" t, we minimize the following loss:
min w EO
Laifs(2, 0) + forget (W K.Y. W.K.)
(1)
where i is the input data of the new concept, Paiss is the loss function for stable diffusion w.r.t. model 0 (we do not modify this our work), fforget minimizes forgetting between old task W.Ky and new task W.K.V and À is hyperparameter chosen with a simple exponential sweep.
As previously mentioned, we parameterize the weight delta between old task W and new task W.k.v using LoRA (Hu et al. 2021 parameters, which decomposes the weight matrices into low-rank residuals, or:
W.k.V W.KY + A
K.V
B.K.V
f-1
(2)
W.K.V init
+
K.V. e
B.V.V
+ +
K.V B...
controlling the rank of init are the initial values from the pre-trained model. This parameterization creates the surface inject our novel regularization, described next.
where A, A.V. E RDir, B...V E RrxD: WKY DDxxD and r is the weight matrix update, chosen with a simple grid search. W.K.V
3.2 Self Regularization
How do we prevent forgetting? Prior experience would first suggest to provide simple replay data (real or generated) for knowledge distillation (Lesort et al. 2019 Shin et al. 2017 However, paper, generative replay induces new artifacts and catastrophic forgetting, and additionally want to avoid storing all training data for data privacy and memory efficiency. Another approach is to estimate individual parameter importance for each task, and penalize changes to these parameters (Aljundi et al. 2018 Kirk- patrick et al., 2017). However, such a strategy stores a copy of original parameters, introducing significant overhead, and additionally we directly show in our experiments that C-LoRA has superior performance.
show later in this
Instead, we devise simple regularization that works well without storing replay data and introduces zero parameters beyond LoRA parameters (Eq, D. The high level idea is that penalize the LoRA parameters A.V.V
K.V and BKY for altering spots that have been edited by previous concepts in the corresponding W.k.V. Thus, we the summed products of past LoRA parameters themselves to penalize the future changes. Specifically, we regularize using:
E-1 Σ #=1
forget =1
K.V
K.V.
O OAKV
K.V LAKBBBRI
(3)
where @ is the element-wise product (also known as the Hadamard product), |-1 is the element wise absolute value, and || ||r denotes the Frobenius norm. This simple and intuitive penalty "self-regularizes" future LoRA with high effectiveness and efficiency. We highlight that the A and B are low-rank matrices and thus only incur small costs to training and storage.
3.3 Customized Token Strategy
To further mitigate interference between concepts, we propose new custom tokenization strategy. Specifi- cally, we add N personalized tokens Vi, Vi . Vý to the input sequence, where N is the total number of

---

Published in Transactions on Machine Learning Research (04/2024)
concepts the model will learn. Unlike Custom Diffusion (Kumari et al. 2022), which initializes these tokens from lesser-used words, we initialize with random embeddings in the token embedding space. Furthermore, we remove any object names from the input sequence to encourage unique personalized embeddings. During inference, we replace the personalized tokens with the lcarned embeddings of the corresponding concept. allowing the model to produce images of learned concepts at the same time.
We found that this tokenization approach improves the model's ability to learn multiple concepts without interference. Why is it better? When including the object name (e.g., "person"), or using this word for token initialization, we found interference to be much higher. For example, many concepts are completely "overwritten" " with future task data. When initializing with random, lesscr-used tokens, we found that some initializations were better than others, leading the "good" tokens to incur less forgetting than the "bad" tokens, and thus remaining unsatisfactory and motivates our strategy.
Putting it Together. Our final approach, summarized in Figure 3 introduces a simple continual custom tokenization along with our sell-regularized, low-rank weight adaptors (C-LoRA). Our method is parameter efficient and effective, as demonstrated in two different continual learning experiment settings in the following sections.
4 Continual Text- To- Experiments
Implementation Details. For the most part, use the same implementation details as Custom Diffu- sion (Kumari et al. 2022) with 2000 training iterations, which is the best configuration for generating faces for baseline methods. To improve the fairness of our comparison with Custom Diffusion, we regularizes train- ing with auxiliary data (as done in Custom Diffusion) for all methods. However, using real images would require constant access to internet downloads, which would be highly impractical for our setting. Thus, in the spirit of "continual learning", , we instead generate regularization images from the Stable Diffusion backbone, as done in one of the Custom Diffusion variants, using the prompt "a photo of person". We tuned lyperparameters
short task sequence (5) to avoid over-fitting to the full task sequence (van de Ven & Tolias 2019). For LoRAA, we searched for the rank using simple exponential sweep and found that a rank of 16 sufficiently learns all concept. Additional training details are located in Appendix C
Metrics. Unlike qualitative comparisons for generative models, where performance has plateaued and fine- grained visual differences across methods are hard to spot, our method starkly increases image generation quality during Continual Diffusion compared to competing methods. Most methods either produce images with severe artifacts catastrophically crase past concepts and produce images with the wrong identity. However, we do provide some quantilative results calculating the similarity between training data samples and diffusion model samples. Given two batches of data samples, we first embed both into rich semantic space using CLIP (Radford et al. 2021) image embedding, and then we calculate the distribution distance using the Maximum Mcan Diserepancy (MMD) (Gretton et al. 2012) with a polynomial kernel. The dif- ference between our metric and the image similarity metric from Kumari et al. (2022) that we use a true distribution distance (MMD) instead of 1-to-1 cosine similarity calculations; however, we also show their CLIP image-alignment metric in our main tables and the remaining metrics in our Appendix. Specifi- cally, we provide CLIP image- alige-tlignment ( (Gal et al. 2022 in our main text and Appendix, and both CLIP text-alignment (Hessel et al. 2021) and KID BBnnkowski et al. 2018) in Appendix A only.
From this score, we present the mean MMD score averaged over all concepls and measured after truining the final concept as Amma, where lower is better. We also report the average change in past concept generations from task to task as a "forgetting" metric, given as Fmmd, where lower also better. The exact formulas for the metrics are found in Appendix [ We note that Amms is the more important metric, and Fromd simply provides more context (e.g., did the method learn the concept well and forget, as we see with Custom Diffusion, or did the method not capture the concept well to start with, as see with Textual Inversion?). We additionally report both the number of paramcters trained (i.c., unlocked during training task) and
Kumari et concepts.
2022 show that using generated images instead of real images leads to similar performance the target
7

---

Published in Transactions on Machine Learning Research (04/2024)
stored (i.e., stored between tasks). We discuss this more in Appendix C These are reported in % of the U-Net backbone model for easy comparison.
Baselines. We compare to recent customization methods Textual Inversion GGal et al. 2022). Dream- Booth (Ruiz et al., 2022), and Custom Diffusion (Kumari et al. 2022 For Custom Diffusion, we compare to both sequential training (denoted as sequentäal) as well as the constrained merging optimization variant (denoted as merged) which stores separate KV parameters for each concept individually and then merges the weights together into single model using closed-form optimization (see Kumari et al. more details). We also compare to continual learning methods Generative Replay (Shin et al. E (Kirkpatrick et al., 2017), which we have both engineered to work with Custom Diffusion. We further contextualize our results by drawing a comparison with a concurrent work, ZipLoRA (Shah et al. 2023 We developed an implementation of ZipLoRA from the ground up to adapt it to our continual learning framework. ZipLoRA does not perform as well as our C-LoRA, likely due to ZipLoRA being originally conccived for concept/style integration, as opposed to learning a large number of subjects in a continual learning context. Finally, we ablate our method by comparing to continual variant of LoRA (Hu et all, 2021 Smith et al. 2002a)). which we refer to LoRA (sequential). Additional ablations, including for our tokenization strategy, are located in Appendix B
2022 for
2017 and
4.1 Continual Customization of Real Faces
We first benchmark our method using the 512x512 resolution (self-generated) celebrity faces dataset, Celeb- Faces Attributes (Celeb-A) HQ (Karras et al. 2017 Liu et al., 2015). We sample 10 celebrities at random which have at least 15 individual training images each. Each celebrity customization is considered "task", and the tasks are shown to the model sequentially. Qualitative results are shown in Figure showing samples from task 1, 6, and 10 after training all 10 tasks, while quantitative results are given in Table L Here, we see quite clearly that our method has tremendous gains over the existing techniques. [a] and [b] give the target data and offine (i.e., not our setting) Custom Diffusion, respectfully. [c] Textual Inversion (Gal le al.] (022) struggles to capture the dataset individuals, though it suffers no forgetting (given the backbone is completely frozen). [e] Deep Generative Replay (Shin al. 2017) has completely collapsed we noticed that artifacts begin to appear in carly tasks, and then as the model is trained these artifacts in future lasks, the collapsing performance results in a "snowball" effect. [d] DreamBooth (Ruiz et al.] 2022) and [f] Custom Diffusion (Kumari et al.) 2022) (sequential) are prone to high catastrophic forgetting, with the former suffering more (which is intuitive given DrcamBooth trains the full modcl). Notice that the first two rows for DreamBooth and middle row for Custom Diffusion (sequential) are almost identical to their respective third rows, and more importantly are the completely wrong person. While we noticed [g] Custom Diffusion (merge) performed decent in the short 5 task sequence wrt Ammd (2.62 compared to our 1.65, yet still under-performs artifacts present in the generated images. [h] FWC (Kirkpatrick et al. 2017 has less forgetting, but we found that turning up the strength of the regularization loss prohibited learning new concepts, yet relaxing the regularization led to more forgetting, We notice that [i] LoRA (sequential) (Hu et al. 2021 Smith et al., 2022a) offers small improvements over sequential Custom Diffusion, yet still has high forgetting. Finally, we note in Table 1 that the concurrent ZipLoRA (Shah et al., 2023 performs the highest of all competitor methods, but it does not perform as well as our C-LoRA.
method), it fails to merge 10 tasks (8.89 compared to our 2.04), evident by the strong
On the other hand, we see that [i] our method produces recognizable images for all individuals, and furthermore has the best Ammd performance. Notice that our method adds significantly fewer parameters than Custom Diffusion Merge (Kumari et al. 2022) and has clear superior performance.
4.2 Continual Customization of Landmarks
As an additional dataset, we demonstrade the generality of our method and introduce an additional dataset with a different domain, benchmarking on 10 length task sequence using the Google Landmarks dataset v2 Weyand et al.) 2020), which contains landmarks of various resolutions. Specifically, we sampled North
SWWC is tuned in the same way as our method's regularization loss (simple exponential sweep) and we report the run with the best Awand-
8

---

Published in Transactions on Machine Learning Research (04/2024)
[a] target data
[b] custom diffusion (offline)
[c] textual Inversion
[d] dreambooth
[e] deep generative replay
[f] custom diffusion [g] custom diffusion
[h] custom diffusion (EWC)
[i] LoRA (sequential)
[i] ours
(sequential)
(merge)
Figure 4: Qualitative results continual customization using the Celeb-A HQ (Karras et al. 2017 Liu al,
2015) dataset. Results are shown for concepts (Lasks) 1, 6, and 10, and are sampled after training on all concepts. See Appendix for source of target images.
[a] target data
[b] custom diffusion (offline)
[c] textual Inversion
[d] dreambooth
[e] deep generative replay
[f] custom diffusion [g] custom diffusion (sequential)
[h] custom diffusion (EWC)
[i] LoRA (sequential)
O ours
(merge)
Figure 5: Qualitative results of continual customization using waterfalls from Google Landmarks Weyand
et al. 2020 Results are shown for concepts (tasks) 1, 5, and 10, and are sampled after training on all 10 concepts. See Appendix for source of target images.
America waterfall landmarks which had at least 15 example images in the dataset. We chose waterfall landmarks given that they have unique characteristics yet are similar overall, making them challenging task sequence (though not as challenging as human faces). Qualitative results are shown in Figure with samples from task 1, 5, and 10 after training all 10 tasks, while quantitative results are given in Table 2 Overall, we notice similar trends the face datasots, but custom diffusion (merge) performs better this type of dataset compared to face datasets (though still much worse than our method). Specifically, it catastrophically forgets the middle landmark, but the first and final landmarks represented well relative
9

---

Published in Transactions on Machine Learning Research (04/2024)
Table Results on Celeb-A HQ (Karras et al., 2017; Liu et al. 2015). Ammd (4) gives the average MMD score (x109) after training on all concept tasks, and Fmmm (4) gives the average forgetting. Image-alignment give the CLIP-score between generated images after training on all tasks and dataset images. N param (4) gives the % number of parameters being trained and stored. CD represents Custom
Diffusion Kumari et al. 2022). We report the mean and standard deviation over trials.
Method
Nparom (4)
Nparam (4)
A Ammd (4)
F Foomd (4)
Image (t) alignment.
Train
Store
Textual Inversion
0.0
100.0
2.98 = 0.60
0.00 = 0.00
0.69 0.02
DreamBooth
100.0
100.0
9.01 0.73
8.25 + 1.44
0.54 + 0.02
Gen. Replay
2.23
100.0
17.34 = 0.33
0.47 = 0.01
LoRA (Sequential)
0.09
100.0
5.57 = 0.29
3.96 112
0.69 + .01
CD (Sequential)
2.23
100.0
7.72 = .26
7.08
0.63
CD (Merge)
2.23
122.30
8.89 = 1.59
6.18
0.04
CD EWC
2.23
102.23
4.64 0.09
2.95
0.70
ZipLoRA
0.10
101.00
3.25 0.04
0.27 0.05
Ours
0.09
100.90
2.04 + 0.17
0.32 0.13
0.80 0.01
Table 2: Results on Google Landmarks dataset v2 (Weyand et al., 2020). Anmd (4) gives the average MMD score (x10²) after training on all concept tasks, and F Fond (1) gives the average forgetting. Imago-alignment give the CLIP-score between generated images after training on all tasks and dataset images. Nparam (L) gives the % number of parameters being trained and stored. CD represents Custom Diffusion Kurnari el al. 2022). We report the mean and standard deviation over trials.
Method
Nyarom (4) Train
N Nparam (4) Store
Ammd (L)
Fonmd (4)
Image (t) aligument
Textual Inversion
0.0
100.0
3.84 = 0.55
0.00 + 0.00
0.81 + 0.01
DreamBooth
100.0
100.0
5.49 = 0.69
4.70 + 0.39
0.77 + 0.00
Gen. Replay
2.23
100.0
14.83 + 0.85
10.54
0.55 + 0.00
LoRA (Sequential)
0.09
100.0
4.44 = 0.48
+
0.79 0.00
CD (Sequential)
2.23
100.0
5.04 = 0.74
0.75
0.78
CD (Merge)
2.23
122.30
7.59 3.21
5.14
0.70
CD EWC
2.23
102.23
4.61 = 0.41
3.25 3.25
0.79
ZipLoRA
0.10
101.00
0.51 0.21
0.82
Ours
0.09
100.90
2.82 + 0.26
0.30 + 0.27
0.83 + 0.01
to the face datasets. We emphasize that, addition to being the top-performer, our method has significant parameter storage advantages over custom diffusion (merge). Specifically, in order create "on the fly" merged models, custom diffusion (merge) requires that KV values of all tasks be stored indefinitely, and has storage costs that 25x greater than method for this 10 task sequence. However, note that merging in in a "continual manner" would not incur this large storage costs. We discuss this more in Appendix
4.3 Multi- Multi-Concept Generations
In Figure we provide results demonstrating the ability to generate photos of multiple concepts in the same picture. We found that using the prompt style "a photo of V* person. Posing with V* person" worked best. We notice that only our full method has success loken imil removes the random loken initialization and ablate prompl concept uses the prompt "a photo of y* person" instead of "a photo of V*") are "hit or miss", whereas custom diffusion (merge) completely fails (we found the sequential version also fails due to catastrophic forgetting). One drawback is that we noticed difficulty in producing multi-concept images of individuals of highest similarity, which needs be addressed in future work.
cach concept pair. The two ablations (ablate
10

---

Published in Transactions on Machine Learning Research (04/2024)
LV IV
ours ablate token init.
ours ablate prompt concept
custom diffusion (merge)
target data
ours
Figure 6: Multi-concept results after training 10 sequential tasks using Celeb-A HQ Karras et al. 2017
Liu et al. 2015p. Using standard quadrant numbering (I is upper right, II is upper left, III is lower left, IV is lower right), we label which target data belongs in which generated image by directly annotating the target data images. See Appendix J for source of target images.
ANIM
CD (Saquential)
CD (Marge)
TI
Ours
Tasks
Figure 7: Analysis extending C-LoRA to 50 concept sequence comprised of both Celeb-A HQ Karras
et al. 2017 Liu et al., 2015) and Google Landmarks dataset v2 (Weyand et al. 2020 shows that our met struggles es with long concept sequences, eventually performing worse than Textual Inversion Gal et al. 2022
5 On Longer Task Sequences
In Figure 7 we explore the scalability of C-LoRA by exlending it to handle sequence of 50 concepts, drawing from both the Celeb-A HQ dataset (Karras et al. 2017 Liu et al. 2015) and the Google Landmarks dataset v2 (Weyand et al.) 2020). We found that, while C-LoRA shows promise in handling a moderate number of concepts, its performance begins to falter as the concept sequence lengthens. Specifically, it approaches the performance of Textual Inversion (Gal et al.) 2022), very simple method which does not alter the backbone of the text-to-image model (and instead only learns custom tokens). This suggests there is a need for further refinement to enhance scalability and effectiveness across broader and more complex concept sequences for large-scale continual learning tasks. For further context, we include the results of Custom Diffusior (Kumari et al. 2022 with EWC (Kirkpatrick et al. 2017) regularization, showing that our method still outperforms this spproach. We include additional results our Appendix, including additional metrics and results (Appendix A additional method ablations (Appendix empirical analysis sequential update parameter interference (Appendix D), further analysis the merge variant of Custom Diffusion (Kumari al. 2022 (Appendix P additional multi-concept generations (Appendix F offlinc
comparison with Custom Dillusion (Kunari el al. 2022 (Appendix G C-LoRA hyperparameter sweeps (Appendix
6 Continual Image Classification Experiments
The previous section shows that our C-LoRA achieves SOTA performance in the proposed setting of continual customization of tert-to-image diffusson. While we compared with SOTA cxisting mothods, and furthermore tried our best to engineer improved baselines, we want to strengthen our claims that our
We do include the full plots for unregularized Custom Diffusion, they underperform Custom Diffusion with regular- ixation.
11

---

Published in Transactions on Machine Learning Research (04/2024)
Table 3: Results (%) on ImageNet-R for 10 tasks (20 classes per task). AN gives the accuracy averaged over tasks and FN gives the average forgetting. We report the mean and standard deviation over trials.
Method
AN (f)
EN (L)
UB (tune all)
77.13
UB (tune QKV)
82.30
FT
10.12 0.51
25.69 0.23
FT++
48.93 1.15
9.81
LwF.MC
66.73 + 1.25
3.52 + 0.39
L2P
69.00 0.72
+2P++
70.98 + 0.47
1.79 0.19
DualPrompt
74.12 :: 0.55
1.60 0.20
CODA-P
75.58 0.61
1.65
Ours Ablate Eq.
72.92 0.52
3.73 0.06
Ours
78.16 0.43
1.30 0.10
C-LoRA is indeed a SOTA technique for continual learning. Therefore, in this scction we also demonstrate that our method can achieve SOTA performance in the well-eslablished setting of rehearsal-free continual learning for image classification.
Implementation Details. We benchmark our approach using ImageNet-R (Hendrycks et al. 2021 Wang
et al. 2022b) which composed of 200 object classes with a wide collection of image sty es, incl Melluing cartoon, graffiti, and hard examples from the original ImageNet dataset (Russakovsky et al. 2015 This
benchmark is attractive because the distribution of training data has significant distance to the pre-training data (ImageNet), thus providing fair and challenging problem setting. In addition to the original 10-task benchmark, we also provide results with a smaller number of large tasks (5-task) and a larger number of small tasks (20 task) in Appendix !
We use the exact same experiment setting as the recent CODA KODA-Prompt implement method and all baselines in SyTorch(Paszkes et al. 2019 using the ViT-B 16 backbone (Doso- vitskly et al. 2020) pre-trained on ImageNet-1KK (Russakovsky
(mithh al., 2022b) paper. We
al. 2015). We compare to the following
methods (the same rehearsal-free comparisons of CODA-Prompl)) Learning without Forgetting (LwF) [Li & Hoiem 2017). Learning to Prompt (L2P) (Wang et al.) 2022ch, modified version of L2P (L2P++) (Smith ct al. 2022b), and DualPrompt (Wang et al. 2022b). Additionally, we report the upper bound (UB) per- formance, which is trained oflline (we provide two variants: one fine-tunes all parameters and the other only fine-tunes QKV projection matrices of self-attention blocks), and performance for a neural network trained only on classification loss using the new task training data (we refer to this as FT).. We use the same classi- fication head L2P, DualPrompt, and CODA-Prompt, and additionally compare to FT variant, FT++, which uses the same classifier as the prompting methods and suffers from less forgetting. For additional details, we refer the reader to original CODA-Prompt (Smith et al., 2022b paper. For our method, we add C-LoRA to the QKV projection matrices of self-attention blocks throughout the ViT model.
Metrics. We evaluate methods using (1) average accuracy AN, or the accuracy with respect to all past classes averaged over N tasks, and (2) average forgetting (Chaudhry al., 2019a Lopez-Paz & Ranzato 2017) FN, or the drop in lask performance averaged over N tasks. We emphasize that AN is the more important metric and encompasses both method plasticity and forgetting, whereas Fx simply provides additional context.
Results. In Table 3 we benchmark our C-LoRA against the popular and recent rehearsal-free continual learning methods. We found that our method achieves a high state-of-the-art in a setting that it was not designed for. Compared to the prompting methods I,P., DualPrompt, and the recent CODA-Prompt, our method has clear and significant improvements. We also ablate our forgetting loss and show that this simple yet major contribution is the main driving force behind our performance.
12

---

Published in Transactions on Machine Learning Research (04/2024)
7 Conclusion
In this paper, we addressed the problem of catastrophic lorgetting in continual customization of lext-lo-image models (Continual Diffusion). Our proposed method, C-LoRA, leverages the structure of cross attention layers and adapts them to new concepts in low-rank manner while preserving the knowledge of past concepts through a self-regularization mechanism. We showed that our approach alleviates catastrophic forget in the Stable Diffusion model with both quantitative and qualitative analysis. Additionally, we extended our method to the well-established setting of continual learning for image classification, demonstrating that our gains translate to otato-of-the-art performance in the standard benchmark as well. Our contributions provide a promising solution for the practical and critical problem of continually customizing text-lo-image models while avoiding interference and forgetting prior seen concepts.
8 Limitations & Broader Impact Statement
Despite the success of our approach in generating task sequences of up to ten faces, we acknowledge key limitations and cautions that must be addressed. Specifically, caution against training our method on large task sequences of 50 or 100 faces, as our approach suffers in longer concept sequences (as shown in Figure 7 To increase the scalability of our approach, future work should focus on exploring additional novel techmiques. Additionally, all existing methods (including ours which performs best) still struggle to generate multi-concept.ual faces on similar individuals, and we strongly urge future research to focus on improving this aspect of our approach.
In sclecting the CelebFaccs Attributes (Celeb-A) HQ Karras et al. 2017 Liu et al. 2015) and Google Landmarks dataset v2 /Weyand et al. 2020) for our benchmarks we aimed to demonstrate the versatility and effectiveness of our approach across varied domains. The choice of the Celeb-A dataset was motivated by the unique challenge that human faces present in terms of variability in fine-grained features. Similarly, the selection of the Google Landmarks dataset, specifically North American waterfall landmarks, was intended to showcase our method's ' applicability to natural scenes, which, despite their unique characteristics, maintain a degree of similarity that poses different kind of challenge. To avoid bias in the selection of specific concepts randomly selected valid concept identities for all datasots, where validity was
to use for our experiments, determined by having at least 15 individual training images each. We believe that future research should prioritize the use of similar datasets or consenting participants to ensure ethical practices.
Furthermore, we emphasize the ethical implications of our work and caution against the use of our approach to generate faces of individuals who do not consent. We specifically acknowledge the potential for C-LoRA to contribute to the creation of rcalistic images that blend multiple concepts or identities. This capability raises critical concerns regarding the potential for misuse in creating disinformation or harming individuals' reputations through unauthorized deceptive representations. We therefore stress the importance of de- veloping and adhering to ethical guidelines and regulatory frameworks that specifically address these risks, ensuring that advancements, such as ours, are used responsibly and with a clear commitment to upholding individual privacy and integrity.
Finally, we include a strong disclaimer against using our work in production environments without stringent ethical oversight and within the bounds of legal frameworks designed to protect individual privacy and integrity. Furthermore, we acknowledge the ethics concerns over using artists' images. We avoid artistic creativity in our work, and note that this is being worked out in the policy and legal worlds. Despite these cautions, we believe that our work has the potential to bring great joy to individuals in the form of mobile app entertainment. By scaling the generation of faces to multiple sequential faces, our approach enables novel entertainment applications for sequential settings. We are confident that our approach will pave the way for exciting advancements in the field of text-to-image generation.
We refer the reader to Appendix for more information on the image sources for figures.
13

---

Published in Transactions on Machine Learning Research (04/2024)
References
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytclaars. Expert gate: Lifelong learning with network of experts. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1366-3375, 2017.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXi::1801.004000 2018
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-GEM. In International Conference Learning Representations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint arXiv:1902. 1044:: 2019b.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780 8794, 2021.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yushong Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXin:2203. 06904, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXic:2010. 11929, 2020.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 28-28, 2020, Proceedings, Part XX 16, pp. 86-102. Springer, 2020.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashník, Amit Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXic:2208.0166 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680 2014.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent. catastrophic forgetting. In European, Conference on Computer Vision, pp. 466-483. Springer, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information. Processing Systems, 1556840-685/ 2020.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier in- crementally via rebalancing. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 831 839, 2019.
Neil Houlsby, Andrei Giurgin, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efliienn. transfer learning for ulp. In International Conference on Machine Learning, pp. 2790 2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Ti, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Nitin Kamra, Umang Gupla, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint Xrxi 10368, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing gans for improved 2017.
quality, stability, and variation. arXiv preprint
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.614, 2013.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Ya Yan Zhu. Multi-concept cus- iomization of text-to-image diffusion. arXiv preprinl arXiv:2212.04488 2022.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning. In 2019 International Joint Conference on Newral Networks (IJCNN), pp. 1 IEEE, 2019.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-offiieet prompt tuning. arXiv preprint Xiv:2110 08691, 2021.
Xiang Lisa Li and Percy Liang. Prefix- Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 001900. 2021.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pp.
PMLR, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. 1EEE transactions on pattern analysis and machine intelligence, 40(12):2935-2990 2017.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable dilfusion models. In Computer Vision-Foo -CCCV 2022: 17th European Conference, Tel Avie, Israel, October 27, 2022, Proceedings, Part XVII, pp. 423 439. Springer, 2022a.
Yen-Cheng Liu, Chih- Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter-officient multi-task adaptation for dense vision tasks. arXiv preprint ar Xrx::22200.0 03260.03.655. 2022b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEFE international conference on compuler vision, pp. 3730-3738, 3373,, 2015.
15

---

Published in Transactions on Machine Learning Research (04/2024)
David Lopez-Paz Paz and Marc' Aurelio Ranzato. Gradient episodic memory for contimual learning. In Proceedings of the 31st. International Conference on Newral Information Processing Systems, NIPS117, pp.
USA, 2017. Curran Associates Inc. ISBN 108-1-5108--096-111
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-cffiient language model tuning. arXiv preprint arXiv:2110.07577, 2021.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and molivalion, volume 24, pp. 109- 165. Elsevier, 1989.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.178, 2014.
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 11321-11329, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Newral Information Processing Systems, 34:16131 16144, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMI., 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:224.06112 2022.
Sylvestre-Alvise Rebuffi, Alexander Kolesníkov, Georg Sperl, and Christoph H. Lampert. icarl: Incremen- tal classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR'17, pp. 5533 5542, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pp. 348-358, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esscr, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognétion, pp. 10684 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer- Assisted Intervention-MTCCC 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241 Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.. 12242, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXio preprint arXiv:1606. 04671, 2016.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning. https://github.ccmm com/
cloneofsimo/lors
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXfe:2311.1300 2023.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep gencrative replay. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems 30, pp. 2990 2999. Curran Associates, Inc., 2017.
James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming; A new approach for data-free class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9374-9384, October 2021.
James Scale Smith, Paola Rascante-Bonilla, Assaf Arbelle, Donghyun Kim, Rameswar Panda, David Cox, Diyi Yang, Zsolt Kira, Rogerio Feris, and Leonid Karlinsky. Construct-vl: Data-free continual structured vl concepts learning. Xive preprint #xX::2211.007900 2022a.
James Scale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Ar- belle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. arXiv preprint Xrxi::2211. 13218, 2022b.
Jascha Sock-Dicks,,,n, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256 2265. PMLR, 2015.
Tejas Srinivasan, Ting-Yur Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for aisoo--ad-languaen tasks, 2022.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and -add- language tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogrition, pp. 5227-5237, 2022.
Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. 1944:099407794 2019.
arXiv preprint
Gido M van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial ncural networks. Nature communications, 11(1):1-14, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural informalion processing systems, 30, 2017.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022a.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. arXiv preprint arXiv:2204.04 799, 2022b.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022c.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks datasct v2-a largo-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognilion, pp. 2575 2584,, 2020.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expand- able networks. arXiv preprint arXi:: 708. 015401 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
Mengyao Zhai, Lei Chen, Froderick Tung, Jiawei Hc, Megha Nawhal, and Greg Mori. Lifelong gan: Continual learning for conditional image geueration. Proceedings of the IEEE/CVP international conference on computer vision, pp. 2759 2759-2768, 2019.
Appendix
A Additional Metrics and Results
Tables A and B extend the main paper results with additional metrics and results. In the original paper, we report: (i) N param, the number of parameters trained (i.e., unlocked during training a task) in % of the U-Net backbone model, (ii) Ammd, the average MMD score (x10*) after training on all concept tasks, and (iii) Fmmd, average forgetting. Consider customization "tasks" t E f1.2....,NN - 1, N,, where is the total number of concepts that will be shown our model. We denote X,, as task j images generated by the model after training task i. Furthermore, we denote XD. as original datasct images for task j. Using these terms, we calculate Ammd metric (where lower better)
1 N
Amnmdd
MMD (M Dx Fain (o
where Felip denotes a function embedding into a rich semantic space using CLIP RRdford et al. 2021 and MMD denotes the Maximun Mean Discrepancy (Gretton et al. (2012) with a polynomial kernel. To calculate our forgetting metric Fmodd, we calculate the average distance the images have changed over training, or:
N-1 Σ
F Fmmdd N
MMD (Fapp(a)(X)))) Fap (XN.))
In addition to these metrics, we include metrics reported from Kumari et al. (2022): (iv) amigge-ligmment, which calculates the imagc-image CLIP-Score (Gal ct al. 2022) between all XDS and XN,,, (v) text- aligueend, which calculates the image-lext CLIP-Score (Hessel et al. 2021 between all XN.j and the prompt object", and finally (vi) KID (Biikowski et al. 20188) score (xle3). We notice that the metrics mostly tell similar and intuitive story to analysis from the main text, with main difference: methods with high catastrophic forgetting seem to have good better tent-alignment than method this the individual concepls, just the general object (e.g.,
be attributed to this metric having no conditioning a "face" or "waterfall"). We include this metric for consistency for the related setting of offtine (i.e., not continual) customization text-to-image models, but we do not believe it insightful for our sctting of Continual Diffusion.
B
Ablation Studies
We include an ablation study using the CelebFaces Attributes (Celeb-A) HQ Karras al. 2017 Liu et al, 2015) benchmark in Table We ablate 3 key components of our method: (i) ablate token init removes the random token initialization, (ii) ablate prompt concept uses the prompt "a photo of V person" instcad of "a photo of V**, and (iii) ablate Eq. 3 removes the forgetting loss from our method. (ii) and (iii) have clear performance drops compared to the full method. At first glance, (i) (removing the random token initialization) scems to actually help performance, but, as we show in Section 4.3 (Figure D.. this design component crucial for multi- generations, and without it our method struggles correctly generale mallii-onccep image samples.
18

---

Published in Transactions on Machine Learning Research (04/2024)
Table A: Full results on Celeb-A HQ (Karras et al,, 2017; Liu et al. 2015) after training 10 concepts sequentially, N Nppram (4) gives the % mumber of parameters being trained and stored, Ammd (4) gives the average MMD score (x10%) after training on all concept tasks, Emmd (4) gives the average forgetting, Image/Text-alignaee
give the CLIP-score between generated images after training on all tasks and dataset images/ccptiins, respectively, and KID gives the Kernel Inception Distance (x10²) between generated and dataset images. CD represents Custom Diffusion (Kumari al. 2022). We report the mean and standard deviation over trials.
Method
Nparam (4) Train
Nperam (U)
Amama (4)
Fanmdd (4)
Image (1)
Text (f) alignment.
KID (4)
Store
alignment
Textual Inversion
0.0
100.0
2.98 + 0.60
0.00 + 0.00
0.69 + 0.02
0.53 0.01
49.69 + 14.45
DrcamBooth
100.0
100.0
9.01 + 0.73
8.25 + 1.44
0.54 - 0.02
0.60 0.01
173.48 34.03
Gen. Replay
2.23
100.0
17.34 0.33
11.91 + 0.31
0.47 0.01
0.60 = 0.01
422.11 12.19
LoRA (Sequential)
0.09
100.0
5.57 0.29
3.96 + 1112 .12
0
0.56 = 0.01
81.67 = 12.11
CD (Sequential)
2.23
100.0
7.72
7.08 + 0.54
0.00
0.58 0.01
90.48 7.56
CD (Merge)
2.23
122.30
8.89 1.59
4.7+1.11
0.04
0.56 0.01
139.27 43.78
CD EWC
2.23
102.23
4.64 0.09
2,95 0.07
0.70
0.56 0.01
39.09 2.15
ZipLoRA
0.10
101.00
3.25 0.04
0.27 0.05
0.00
0.55 0.00
40.30 7.20
Ours
0.09
100.90
2.04 + 0.17
0.32 + 0.13
0.80
0.55 0.00
26.50 1.53
Table B: Full results on Google Landmarks dataset v2 (Weyand et al. 2020) after training 10 concepts sequentially. Nparam (4) gives the % number of parameters being trained and stored, Ammd (L) gives the average MMD score (x109) after training on all concept lasks, Frond (4) gives the average forgetting, Image/ Teatee//maalllllmmettttt give the CLIP-score between generated images after training on all tasks and dataset images/captions, respectively, and KID gives the Kernel Inception Distance (x10²) between generated and
dataset images. CD represents Custom Diffusion (Kumari et al. 2022
We report the mean and standard
deviation over 3 trials.
Method
Narram Train
Nprram Store
Amma (+)
Frond (4)
Image alignment
Text (f)
KID (4)
alignment.
Textual Inversion
0.0
100.0
3.84 + 0.55
0.00 + 0.00
0.81 + 0.01
0.64 -64=000 0.02
46.28 = 8.99
DrcamBooth
100.0
100.0
5.49 + 0.69
4.70 0.39
0.77 + 0.00
0.68 0.01
110.13 15.79
Gen. Replay
2.23
100.0
14.83 0.85
10.54 0.59
0.55 0.00
0.65 0.00
491.88 1
LoRA (Soquential)
0.09
100.0
4.44 0.48
3.37 0.69
0.79 1 0.00
0.67 0.00
89.18 9.36
CD (Sequential)
2.23
100.0
5.01
4.19
0.78 0.02
0.69 = 0.02
141.45 77.12
CD (Merge)
2.23
122.30
7.59
5.14 1.79
0.07
0.61 0.05
141.45 77.12
CD EWC
2.23
102.23
4.61 4.61 + 0.41
3.25 + 0.81
0.79
0.69 0.02
+ 12.23
ZipLoRA
0.10
101.00
3.24 1 0.22
0.51 + 0.21
0.02
0.65 0.01
71.25 8.01
Ours
0.09
100.90
2.82 + 0.26
0.30 + 0.27
0.83
0.58 0.03
34.43 + 1.81
Table C: Ablation results on Celeb-A HQ (Karras et al.) 2017 Liu et al.) 2015) after training 10 concepts sequentially. N Nparam (4) gives the % number of parameters being trained and stored, Anmu (4) gives the average MMD score (x109) after training all concept lasks, Fmod (4) gives the average forgetting, mmaage/Text-alignmee and datasct images/captionn respectively, and KID gives the Kernel Inception Distance (x103) between generated and dataset images. We report the mean and standard deviation over 3 trials.
give the CLIP-score between generated images after training all tasks
Method
Nparam Train
Npuram (4)
Anmn (1)
F. Fmond (L)
Image (1)
Text aligunment
KID (L)
Store
alignment.
Ours
0.09
100.90
2.04 0.17
0.32 0.13
0.80 + 0.01
0.55 1: 0.00
26.50 1.53
Ablate token iuit.
100.90
1.95 0.18
0.20 0.13
0.79
0.54 1 0.00
34.96
(ii) Ablate prompt concept
0.09
100.90
2.41 0.42
0.54 0.39
0.78
0.02
0.53 + 0.01
18.95 - 0.25
(i) + (ii)
0.09
100.90
2.76 0.26
0.23 0.15
0.78 + 0.11
0.54 + 0.00
23.96 - 0.51
(ii) Ablate Eq. 3
0.09
100.90
5.12 1 0.06
1.151.0.13
0.69 1 0.09
0.56 +: 0.01
90.16 4 1.96
19

---

Published in Transactions on Machine Learning Research (04/2024)
C
Additional Implementation Details
We use 2 A100 GPUs to generate all results. All hyperparameters mentioned were scarched with an exponen- tial search (for example, learning rates were chosen in the range 5e 5e pe--3 3, 5e 4, 5e-5, 5, 56- -, -,5e-;; we directly show the X and rank sweeps in Figure Hp. We did not tune hyperparameters on the full task set because tuning hyperparameters with hold out data from all tasks may violate the principal of continual learning that states each task is visited only once (van de Ven & Tolias 2019]; instead, we used differ- ent. lask sequence of size We report the mean and standard deviation over 3 trials containing different randomly sampled task data. We found a learning rate of 5e 6 worked best for all non-LoRA methods, and a learning rate of 5e - 4 worked best for our LoRA methods. We found loss weight of 1e6 and 1e8 worked best. for EWC (Kirkpatrick et. al., 2017) and C-LoRA respectively. These values are high because the importance weighting factors are much smaller than 1, and squared, and thus the loss penalty is a very small number and is not effective without a large loss weighting. We found a rank of was sufficient for LoRA for the text-to-image experiments, and 64 for the image classification experiments. These were chosen from range of 8, 16, 32, 64, 128. All images are generated with 512x512 resolution, and we train for 2000 steps on the face datasets and 500 steps the waterfall datasets. For additional analysis on the efficacy for LoRA (Hu et al. 2021) for texl-to-inage dilfusion, we suggest the implementation by Ryu which was a concurrent project to ours.
On stored paramcters in our implementation: For our method, the number parameters trained is 0.09% of the U-Net backbone. In contrast, this number is 2.23% for Custom Diffusion (Kumari el al. 2022). For storage, we have two options for our method: store the individual LoRA weights for each past concept or store the summation. For 10-concept sequence, the storage cost would be 0.09% 10 0.9%. However, note that our storage is bounded by the size of the summed mullippied matrices. Thus, if the sum of individual LoRA weights exceeds the size of the summed multiplied matrix, only store the summed multiplied matrix long term, and thus our max cost is equivalent to the 2.23% of Custom Diffusion.
We note that there are two ways to implement Custom Diffusion (Merge) (Kumari et al., 2022 for contimual learning. One way to store the individual weights for each concept and then merge them together at inference time (as done in the original paper). A second way would be to merge the weights online without storing the individual concept weights. We chose the first option it performs better and is more faithful implementation of their method. As result, the the parameters stored is much higher for Custom Diffusion (Merge) than all other methods, including ours.
D A Closer Look at LoRA and Interference
% W Interference
[[W Intereferencel]
100
Ours
CD
CD LoRA
Ours
160
162
2
6
9
10
10
Tasks
Tasks
(a) Percent of weights that are updated in opposite direction the prior task (i.e. interference)
(b) Amount of change the opposite direction as the prior task (i.e., the magnilluee of interference)
Figure A: Sequential interference analysis (Table setting).
In Figure A we analyze sequential update interference for CD Kumari et al. 2022 CD+LoRA, and ours. The left plot shows the % of weights that are updated in the opposite direction as the prior task (i.c., interference), and we see our method has the least directional interference; however, the major difference lies in the right plot, which shows the amount of change in the opposite direction the prior task (iee. the
20

---

Published in Transactions on Machine Learning Research (04/2024)
magnitude of interference), and 1) LoRA reduces this magnitude, motivating its use in CL, and 2) our method reduces this magnitude much further.
E
A Closer Look at Custom Diffusion Merge
cd (merge)
of concepts learned
Figure B: Analysis of Custom Diffusion (CD) Merge (Kumari al. 2022 vs ours. We show generations of concept 1, starting after concept is learned (far left) and ending with after concept 10 is learned (lar right). These results show that CD (Merge) forgets concept 1 over time, whereas our method does not.
In Figure B we analyze Custom Diffusion (CD) Merge [Kumari al. 2022 our method through learning the ten concept sequence in Celeb-A HQ Karras et al.) 2017 Liu et al. 2015). On the far left, see generated example of concept 1 after learning concept 1. On the far right, of concept. after learning concept 10. We see that, for CD (Merge), the identity is retained through the first 6 concepts, and then concepts begin to interfere after 7 concepts have been learned. On the other hand, our method generates good samples of concept for the entire sequence. The takeaway from this analysis is that CD (Merge) is only effective for hand-full of concepts, unlike our method. This trend extends other datasets CD (Merge) typically performs similarly our method after 5 concepts, but has severe performance decreases before the 10th concept.
generated example
F Additional Multi-Concept Generations
C-LoRA A photo of Vi cat sitting inside V/ pot
CD
A photo of V2 dog in front of Vi barn
C-LoRA
C-LORA
CD
Single concept generations
of teddy style Vū turtle
photo of cat front of Vi
Figure C: Multi-concept results after training on sequential tasks using CustomConccept101 dataset Kumari
al. 2022 CD represents Custom Diffusion [Kumari et al. 2022
21

---

Published in Transactions on Machine Learning Research (04/2024)
In Figure present additional results showeasing the capability of generating images incorporating multi- ple concepts simultaneously, utilizing six random concepts from the simple CustomConceett10 dataset (Ku- mari et al.) 2022). We wish to clarify that this dataset is not the primary focus of our method, as it lacks closely related, fine-grained concepts. These results are offered to illustrate that our method achieves competitive performance in scenarios for which was not specifically tailored. is ob- served that both Custom Diffusion (Merge) (Kumari al.) 2022) and our C-LoRA can produce reasonable images featuring multiple concepts, following training across all six sequential tasks. However, do note that the identity of the second concept in the prompt tends to be lost in the continual learning scenario for CD (Merge), despite verifying that single-concept generations from both methods perform similarly to the results presented in Kumari et al. 20022
G Offline Comparison with Custom Diffusion
Table D: Offline results using a combination of Celeb-A HQ KKrras al., 2017 Liu et al., 2015) and Google Landmarks dataset v2 (Weyand et al., 2020). Ammd (4) gives the average MMD score (x109) after training on all concept lasks, Image/ TaangeeTTTxt-lligmmeet give the CLIP-score between generated images after training all tasks and dataset images/ capttons respectively, and KID gives the Kernel Inception Distance (x10²) between generated and dataset images. CD represents Custom Diffusion (Kumari We report the mean only across 6 concepts.
et al. 2022).
Method
Anand (4)
Image (t) alignment.
Text (f)
KID (4)
alignment
Ours
2.15
0.83
0.59
34.81
CD
2.12
0.84
0.61
39.37
In Table D we compare the "starting place" of our method vs. Custom Diffusion (CD) (Kumari et al., 2022 ensure fair comparison. Specifically, we report the relevant offline learning metrics for the first concept of all 6 combined trials in Tables A B We the CD actually performs better in this simple experiment. demonstrating that our gains not due simply to unfair advantage such using LoRA (Hu et al.
2021
H A Closer Look at Hyperparameters
4.-
15
20
Asam
1.9
162
1e4
1e5
1eU
1e10
À (log-scale)
rank (log-scaled)
Figure D: Hyperparameter analysis (Table setting, using only the first 5 concepts).
we delve into the impact of varying the À and rank values on the performance of our C-LoRA method. Our analysis reveals a range of effective values for both hyperparameters, bounded by regions where performance noticeably declines.
In Figure D
22

---

Published in Transactions on Machine Learning Research (04/2024)
Table E: Results (%) on ImageNet-R for tasks (40 classes per task), 10 tasks (20 classes per task), and 20 tasks (10 classes per task). AN gives the accuracy averaged over tasks and FN gives the average forgetting. We report the mean and standard deviation over trials.
Tasks
10
20
Method
Ay (f)
Fx (1)
Ay (f)
FN (J)
Ax (1)
FN (L)
UB (tune all)
77.13
77.13
77.13
UB (tune QKV)
82.30
82.30
82.30
FT FT++
18.74 = 0.44
41.49 +..9 0.52
10.12 + 0.51
25.69 + 0.23
4.75 = 0.40
16.31 = 0.19
60.12 0.87
0.21
48.93
9.81 + 0.31
35.98 1.38
0.11
LwF. MC
74.56 = 0.59
4.98 0.37
06.73 1.25
3.52 + 0.39
54.05 2.66
0.26
L2P
70.25 0.40
0.37
19.00=0.77
2.09 + 0.22
65.70 1.35
1.17
L2P++
71.95 = 0.38
2.52
0.447
1.19
1.05
1.22
roapPt
72.85 = 0.39
2.49 0.36
7.66
#.29
68.02
1.07 0.18
CODA-P (small)
75.38 0.20
0.15
7..12
1.60
70.43 1.30
1.00
CODA-P
76.41 0.38
2.98
75.58
1.15
72.08 1.03
0.97 0.17
Ours Ablate Eq. 3
76.62 = 0.16
5.21
72.92
3.73
05.08
0.11
Ours
79.30 + 0.34
2.45 0.30
78.16 1
1.30
71.85 0.13
1.65 0.39
Additional Results for ImageNet-R
In Table B we extend the image classification results to include smaller (5) and longer (20) task sequences. We see that, for the most. part, all trends hold. However, we notice that our C-LoRA method slightly under- performs CODA-Prompp (Smith et al., 2022b) for the long task sequence (20). We note that prompting methods such as CODA-Prompt grow in number of total paramcters with each additional task, whereas our model is constant sized (the small number LoRA parameters collapse back into the model during inference). It is possible that our method would perform better long task sequences with larger LoRA rank.
J On Image Sources for the Figures
Due to licensing constraints for sharing dataset images, instead generated similar images to display in many of our figures. In Figures 4 and concept Custom Diffussio (Kumari et al. 2022) using LoRA (Hu et al. 2021 adaptation (we refer to these as pseudo figure images for the rest of this section). We emphasize that all training and evaluation have been performed using the original datasets, and the result images were obtained through models trained directly on these datasets. For Figure 2 the images captioned "learn" " are pseudo figure images, and the multi-concept images are results produced with our method. For Figure all face images are pseudo figure images. In Figure the "target data" " were collected from httpa::///peeeereee org/ and are all licensed with either CC-0 license marked for public domain. Figures B only contain results produced from models we trained.
generated the images labeled "target data" with single-
23


Reviews:
[{"review_id": "0ZR6GS0iOt", "reviewer": "Reviewer_mJhg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper investigates an important and open problem in continual learning.\n\nThe considered continual adaptation setting -- to my knowledge -- extends previous studied settings (often classification tasks) on often smaller scale data (like MNIST, shuffled MNIST etc) to a generative computer vision problem on real images. This is an interesting contribution.\n\n## Weaknesses\n\nOverall, I find the idea and evaluation setting really interesting, and this could become a great TMLR paper. However, the execution and rigour in the evaluation is below TMLR standards. I would be happy to iterate with the authors on potential improvements of the paper.\n\nSome questions to start with:\n\n**Regarding memory:**\n\nFirst, I am not fully clear on how the parameters are stored. In Eqs (2) and (3) there is a split between concepts 1...t-1 and the new matrices for task t. My guess is, you store the summed matrices for 1...t-1 and the individual matrices A and B for training, is that correct? For inference, you only need to store the overall summed matrix? In either case, I do not follow how you arrive at the parameter count in Table 1 and 2. The summed matrix does not decompose into invidiudal low-rank matrices any more, so this would imply storing the multiplied matrices (larger memory footprint). Could you confirm and clarify, I might be missing something.\n\nSecond, is there a difference in parameter count used during training and inference? If so, it would be fair to other methods to have two columns in Table 1 and 2 for this, and distinguish both cases. I think for example that it would be fine to have more parameters than e.g. standard LoRA, but this should be pointed out (if it is the case, the my point above).\n\nThird, how are the parameters for Table 1 and 2 actually computed in practice? Do you get the numbers directly from your implementation (that would be the way), or is this an estimation not tied to your particular implementation? Just to double check the numbers, could you confirm that they actually correspond to something you can get out of your model/code?\n\nFinally, is there memory overhead for integrating the additional tokens you need for your approach?\n\n**Regarding hyperparameters:**\n\nFor hyperparameter tuning, I assume you sample a different set of concepts (identities) from the ones used for testing? Could you extend the appendix section on the tuning protocol, and potentially also add a few tables on hyperparameter variations and their impact on performance?\n\n**Regarding depth of analysis**\n\n- Why not plot performance of all methods across the 10 tasks? The temporal evolution of performance would be interesting to consider.\n- What is the criterion for selecting the \"concepts\" (i.e., person identities) in particular? For running the experiments, how many different concepts do you run over?\n- It would be interesting to see when the model reaches capacity\n\n### Additional weaknesses/comments on the paper\n\nThese are chronologically, important points are marked with an, minor points are not marked.\n\n- Is SOTA Custom Diffusion (Kumari et al., 2022) still the SOTA? Just checking, as the field moves fast -- was there no more recent work in 2023 to consider?\n- \"catastrophic forgetting\": This is central topic of the paper, but the quantitative evaluation metrics do not touch on this sufficiently.\n- abstract: \"(i.e\" seems to be a \"e.g.\"\n- \"Continual Diffusion\": I think this is not a good name, as it mixing the learning setup with an evaluation setup\n- Some icons (like the lock) in Fig 3 are overlapping with text\n- p.5 \"respectfully\" -> \"respectively\"\n- Eq. 1\n  - typeset the \"diff\", \"forget\" etc. correctly as text vs. math\n  - please write out the individual components of the loss (Ldiff and Lforget)\n- Eq. 2\n  - Why not letting the sum run till t? Is this because the first matrix is stored as a single element in the model? This was not fully clear during reading, maybe clarify in the methods.\n- \"However, such a strategy stores a copy of original parameters, introducing significant overhead,\" -> can you quantify this overhead in terms of # of model params/relative increase in model params?\n- Eq 3: Clarify that the | . | is the element wise absolute value (I suppose), and make clear that the norm is the Frobenius norm, i.e. change subscript from \"2\" to \"F\".\n- Appendix C misses the search ranges for all hyperparameters\n- \"We highlight that the A and B are low-rank matrices and thus only incur small costs to training and storage\": What is the exact memory footprint you incur during training and inference?\n- Section 3.3\n  - \"Furthermore, we remove any object names from the input sequence to encourage unique personalized embeddings\"\n  - which object name? where are these listed? how were they picked\n- Section 4\n  - \"However, using real images would require constant access to internet downloads, which would be highly impractical for our setting\"\n  - why exactly is this impractical?\n  - I would prefer to have the metrics mentioned in the paper\n  - typeset refs properly, \"see Kumari et al. (Kumari et al., 2022) for more details\" -> see \\citet{...} for more details, or similar\n  - \"Baselines\": Any new baselines that came out in 2023 and were dropped in the evaluation? If so, minimally cite them, and consider including the most recent SOTA model (if applicable)\n- Section 4.1.\n  - self-generated: Was the same self-generated data as in the referenced prior work used?\n  - please cite numbers in sec 4.1. directly, and make statements about the improvements you want to highlight\n  - why only 10 tasks? why not scaling to 100, or 1000s of tasks given that the data is self generated\n- Table 1:\n  - Bolding of 0.51 is off\n  - please add error bars to the table. Ideally, run across different variations of your dataset (e.g., 3-5 repeats). If unfeasible, please let me know the compute budget as a justification for running only a single seed.\n  - \"Nparam\" is misleading for a relative number. Also why not putting % in the table to make it more clear\n  - why is the Nparam number the same as for LoRA? shouldnt you get a linear increase in memory with each added task? (same Q for Table 2). Also, dont you need to store the product of matrices vs. just two low rank matrices?\n  - what is your baseline to study the effect of C LoRA specifically? i.e., which is the setting where the model etc. is exactly the same, just c lora is switched off? Do all methods use the exact same base model? if different models, what is the performance on the baseline task\n  - remove the vertical lines in the tables / improve table typesetting\n  - Why is the number of tasks in Table 1 not simply writen into the caption. What is the number of tasks in Table 2? (also seems to be 10?)\n  - consider merging table 1 and 2, given that all is same except for numbers/datasets (could be arranged across columns), if this looks nicer in the paper\n- In figure 4 and 5, was there a control experiment to make sure how the visual quality of the two methods matches in the non-continual setting? I.e., does the improved model performance stems from the method, or from a better baseline performance? or is that panel [b]?\n- \"requires that KV values of all tasks be stored indefinitely\" -> where? and how can CD (merge) be worse than the LoRA approach presented here? Is this due to the regularizer? Is there evidence validating that the regularizer is indeed the cause for the increase in performance?\n- Figure 6: I do not fully get the quadrant numbering. Is this done only for the figure, or also somehow for the model\n- Section 5\n  - ImageNet-R \"because the distribution of training data has significant distance to the pre-training data (ImageNet), thus providing a fair and challenging problem setting.\" -> Please cite a reference\n- \" our best to engineer improved baselines\": can you point to the table where u validate that the baselines are on par or beyond the original papers\n- Table 3: Do you want to contextualize the ImageNet-R results also with ref numbers from imagenet models?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension of low-rank adaptation (LoRA) to the setting of continual learning for image generation within diffusion models. Applied to datasets of faces (CelebA-like) and landscapes/waterfalls, the authors train models on sequences of 10 concepts (10 identities of people, or \"styles\" of waterfalls). The model learns new LoRA parameters for each of these concepts, which have an additive effect on the training. The authors demonstrate improvements over various baseline models (e.g. LoRA, or training seperate adapters, or using other continual learning techniques like EWC) using their approach, and claim improved performance with a smaller memory overhead.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please go through the list of weaknesses and comments I posted above. Some aspects are more minor, but a few would greatly improve the quality of the paper. The most important changes to make are:\n\n- Run multiple dataset seeds, and avoid overfitting to a few selected concepts. Report mean and standard deviation (or s.e.m. in all tables)\n- Evaluate what happens when more than 10 concepts are chosen (50, 100, etc), or justify why this is prohibitive (I do not really see why it would be)\n- Clarify the memory overhead of the method, and how the weight matrices are stored exactly. Validate this by getting param counts from your model implementation, ideally.\n- Rework the broader impact statement (see below)\n- Clarify choice of baselines, clarify which is the \"base algorithm\" the method is added to, and consider to add additional ablations to motivate the design choices.\n- broader impact statement:\n  - \"Specifically, we caution against training our method on large task sequences of 50 or 100 faces, as our approach may suffer\". This was an immediate thought while reading the paper: I think you should include exactly this experiment. It seems clear that at some point, performance will degrade, but it would be interesting to get some insight in when this happens (maybe also depending on the size of the adaptation matrices, etc).", "broader_impact_concerns": "The broader impact statement should be reworked, it is somewhat a bit informal. Please add a statement and references for the dataset, how the faces were selected, you may justify this with prior work etc. Provide more context like you already started in Appendix G. Justify why you chose a face dataset (vs. e.g., other objects) specifically, and touch on the implications this has for your work. Discuss how you picked the images. Write about the potential impact of generating new compositions of different persons, and put a disclaimer about how this work could and should not be used in production environments, and what the limitations are. What is your release policy on model weights, etc?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C9IftHvrf5", "reviewer": "Reviewer_Nanp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The paper provides analysis on how existing customization methods (DreamBooth, Custom Diffusion) are unable to do the continual learning task, and often collapse to a single mode.\n- In addition to performing well on continual customization of text-to-image diffusion, the paper also provides experimental results to show the effectiveness of C-LoRA on rehearsal-free learning for image classification, achieving SOTA (improving over approaches such as L2P, CODA-Prompt).\n\n\n## Weaknesses:\n- The paper proposes some new metrics for computing performance with different concepts (the MMD score). It will be very useful if the authors can also conduct some form of human evals, to ensure that the model indeed performs better than the baselines as evaluated by humans. The qualitative results presented in the paper seem to make this clear, but it will nevertheless be helpful to have a quantitative score that ascertains this.\n- Most of the results seem to be on unimodal datasets (e.g., all faces, or all landmarks). Are there any results showing composition of multiple, distinct objects? For example, adding a unique stuffed toy as well as a person. Does the proposed method do better than baselines, or does C-LoRA mostly shine when the objects are of the same type?\n- Missing references to earlier work on PEFT: (1) Houlsby, Neil, et al. \"Parameter-efficient transfer learning for NLP.\" International Conference on Machine Learning. PMLR, 2019., (2) prefix tuning and prompt tuning papers. These references would be helpful to give a more comprehensive overview in section 2 of the history of PEFT.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method to allow text-to-image generation models (e.g., Stable Diffusion) to learn multiple concepts (provided as images) in a sequential manner. They do this effectively by introducing C-LoRA, a low rank adapter on the diffusion model weight matrix for continual learning. They devise a regularization strategy for the introduced LoRA adapters and show that it is effective for learning multiple concepts. In addition to C-LoRA, the paper also proposes a customized token strategy by replacing object names with learned embeddings, to further reduce interference between concepts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Would strengthen the work:\n- Human evaluations for comparing C-LoRA against existing baselines.\n- Correlation between human judgement and the proposed MMD score metric.\n- Qualitative results on multiple distinct concepts (person + landmark, or person + stuffed animal, etc.)", "broader_impact_concerns": "This paper proposes methods to improve continual learning of multiple concepts in text-to-image generation models, which has potential negative ethical implications (e.g., deepfakes). These concerns are mostly discussed in the limitations section, but it may be helpful to include further discussion on the novel aspects of this work that are not present in prior papers (e.g., DreamBooth). In particular, I’d like to see some discussion on the implications of enabling generating realistic images with multiple concepts/people.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cwlLoHnVEW", "reviewer": "Reviewer_VgvP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- C-LoRA is intuitively designed for continual multi-concept learning.\n- C-LoRA outperforms the previous multi-concept learning method, Custom Diffusion.\n\n\n## Weakness\n\n- Continual learning of multiple concepts is not very practical.\n\t- In the most real scenario, people do not learn the concepts sequentially. Instead, different individuals create their own LoRAs, which are combined by the user in a post-hoc manner. Thus, Custom Diffusion (merge) and ZipLoRA (ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs) would be more practical.\n- Limited technical novelty.\n\t- C-LoRA learns a LoRA for each concept, which is a common practice. The performance gain comes from preventing catastrophic forgetting, as this paper considers a new setup for continual learning. Thus, it is not surprising that C-LoRA outperforms competitors that are not designed for this continual setup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the continual setup of concept learning from text-to-image diffusion models. The proposed method, C-LoRA, learns a LoRA for each concept, which is regularized to prevent catastrophic forgetting. C-LoRA outperforms the previous multi-concept learning approach like Custom Diffusion, both qualitatively and quantitatively.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Despite the limitations, I believe the paper is worth presenting in TMLR. Nevertheless, including a comparison with ZipLoRA would be informative, as the experimental cost would not be that expensive.", "broader_impact_concerns": "Well discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to allow text-to-image diffusion models to learn multiple concepts in a sequential manner. They do this effectively by introducing C-LoRA, a low rank adapter on the diffusion model weight matrix for continual learning. After author rebuttal, it received 3 Leaning Accept recommendations. 

All the reviewers are happy about the paper after the revision, commenting that (1) C-LoRA is intuitively designed for continual multi-concept learning; (2) C-LoRA outperforms the previous multi-concept learning method, Custom Diffusion, and overall, this paper has made some interesting contributions to the community. 

For the final version, there are some additional requests from Reviewer mJhg. 

1. ZipLoRA should not be hidden in Tables A and B, it should be pulled into the main paper tables (Table A and B can stay additionally, of course). Likewise, the section about ZipLoRA in p.17 should go to the main paper/related work.

2. The adaptation plot in Figure E likewise seems important. It would be nice to minimally add the performance of other methods at x=10 Tasks. It would be even better if the authors had numbers for all methods across the 50 tasks, but I understand if this is prohibitive to run (but it would be a great resource to readers on where we are with continual learning in this setting). I would encourage the authors to minimally place this figure into the main paper, add the already run results into the figure, and add discussion on it as well, given that there is some space left in the main paper.

3. (Optional) As an additional suggestion which I would strongly recommend, but I will defer to the authors to decide, I would really encourage the authors to improve the plotting and typesetting quality of their figures and tables. The captions in e.g. Fig 4/5 could better match the paper font (or at least not being bold). For tables like Table 3, E, I would suggest to remove the vertical bars for clarity. For figures like Figure A, D, E, I would suggest re-plotting with proper fonts (e.g. Fig. E uses at least 4 different font sizes and styles) and sizes matching the paper.

Overall, the Action Editor would like to recommend Accept with Minor Revision, so that the above reviewer's request can be properly addressed.
Decision: Accept (poster)