Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models
Authors: Saurav Jha
Shiqi Yang
Masato Ishii
Mengjie Zhao
christian simon
Muhammad Jehanzeb Mirza
Dong Gong
Lina Yao
Shusuke Takahashi
Yuki Mitsufuji
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-image diffusion model
Continual Learning
Personalization
Primary area: generative models
TL;DR: To continually acquire new user-defined concepts using pretrained text-to-image diffusion models, we propose leveraging their conditional density estimates, aka, Diffusion Classifier scores as regularization information.

Abstract: Personalized text-to-image diffusion models have grown popular for their ability to efficiently acquire a new concept from user-defined text descriptions and a few images. However, in the real world, a user may wish to personalize a model on multiple concepts but one at a time, with no access to the data from previous concepts due to storage/privacy concerns. When faced with this continual learning (CL) setup, most personalization methods fail to find a balance between acquiring new concepts and retaining previous ones -- a challenge that *continual personalization* (CP) aims to solve. 
Inspired by the successful CL methods that rely on class-specific information for regularization, we resort to the  inherent class-conditioned density estimates, also known as *diffusion classifier* (DC) scores, for CP of text-to-image diffusion models. 
Namely, we propose using DC scores for regularizing the parameter-space and function-space of text-to-image diffusion models.
Using several diverse evaluation setups, datasets, and  metrics, we show that our proposed regularization-based CP methods outperform the state-of-the-art C-LoRA, and other baselines. Finally, by operating in the replay-free CL setup and on low-rank adapters, our method incurs zero storage and parameter overhead, respectively, over the state-of-the-art.

Yuki Mitsufuji Published as a conference paper at ICLR 2025
MINING YOUR OWN SECRETS: DIFFUSION CLASSI- FIER SCORES FOR CONTINUAL PERSONALIZATION OF TEXT-TO-IMAGE DIFFUSION MODELS
Saurav Jha" UNSW Sydney
Shiqi Yangl Masato Ishii, Mengjie Zhao, Christian Simon
Sony Group Corporation
M. Jehanzeb Mirza MIT CSAIL
Dong Gong UNSW Sydney
Lina Yao
CSIRO's Data61 & UNSW Sydney
Shusuke Takahashi Sony Group Corporation
Sony Group Corporation & Sony Al
ABSTRACT
Personalized text-to-image diffusion models have grown popular for their ability efficiently acquire images. However, in the real world, a user may wish to personalize a model on multiple concepts but at time, with access the data from previous con- cepts due to storage/privacy concerns. When faced with this continual learning (CL) setup, most personalization methods fail find a balance between acquiring new concepts and retaining previous ones a challenge that continual person alization (CP) aims solve. Inspired by the successful CL methods that rely on class-specifee information for regularization. we resort to the inherent class- conditioned density estimates, also known as diffusion classifier (DC) scores, for CP of text-to-image diffusion models. Namely, we propose using DC scores for regularizing the parameter-space and function-space of text-to-image diffusion models, to achieve continual personalization. Using several diverse evaluation setups, datasets, and metrics, we show that our proposed regularization- methods outperform the state-of-the-ard C-LoRA, and other baselines. Finally, by operating in the replay-free CL setup and on low-rank adapters, our method incurs zero storage and parameter overhead, respectively, over C-LoRA. C-LoRA,,Project page
concept from user-defined text descriptions and few
CP
INTRODUCTION
With their photorealistic generation quality and text-guided steerability, text-to- text-to-image diffusion models (Saharia et al., 2022, Rombach et al., 2022) have emerged as one of the most flourishing areas in the computer vision community. This has led to their deployment across diverse domains involving the generation of audio/video/3D content, and has, in turn, seen boost in their commer- cial value. Despite achieving extraordinary performance, these models typically demand huge amount training resources and data. A practical user-centric personalization of these, c.g., us- ing limited data and compute, thus calls for efficient finetuning methods (Kumari al. 2023] Gal et al. (023). However, the existing finetuning methods perform poorly on a common worldworld scenario, where a model needs be personalized on sequentially arriving concepts, while being able to generate high-quality images for the previously acquired concepts.
Continual personalization (Smith al. 2024b) aims address the above challenge through contin- ual learning (CL) and incorporating of new tasks with unseen concepts while retaining the previously acquired concepts. Naively adapting the existing (non-continual) personalization methods (Kumari
et al. 2023 Gal et al. 2023) to acquire a new concept in CL setup often requires a complete re-
"Work done as an intern at Sony. Correspondence to: saurav. jhaeunsw. wet.uuuswwwuuuuaa.. edu.au
Projectt Lead.

---

Published as a conference paper at ICLR 2025
training data from all the seen concepts that the user desires to generate. However, storing user's s personal data may raise resource/privacyy concerns, and can be practically infeasible for edge devices. This calls for a replay- sepla--free CL solution, which does away with the seen data once given concept has been acquired. C-LoRA (Smith et al., 2024b) handles these challenges by learning low- rank adapters (LoRAs) (Hu et al. 2022) per task, where each CL. task acquires single concept. It tackles the forgetting of previous concepts by penalizing the modification of any low-rank matrix spots allocated to these. We find that such penalty leads to a degenerate CL solution where all LoRA parameter values are encouraged to be near zero (Sec. [.I)p. This has further catastrophic consequences for the first task wherein the LoRA parameters are modified in an unconstrained man- ner and are thus more liable to the penalty. Nevertheless, given their edge in mitigating forgetting (Biderman et al., 2024). we resort to finetuning task-specific LoRAs while still consolidating their previous task knowledge for enabling CL. To the latter end, we note that merging the LoRA param- eters based on task arithmetic (Iharco et al. 2023) remains insufficient at retaining high generation quality. Instead, we propose to exploit the clas-specific information of text-to- image diffusion mod- els, with which we can consolidate the model's discriminative semantic knowledge from previous tasks. Our inspiration for this comes from the broader CL literature classification models where class-specific information, e.g., logits and softargmax scores, are often employed in countering for-
getting with regularization (Li & Hoiem 2017 Buzzega et al. 2020 Jha et al. 2023 2024).
Namely, we exploit the diffusion classifier (DC) Li et al. 2023 Clark & Jaini, 2024 scores that
encode the semantic concept information inherent to conditional density estimates of pretrained text- to-image diffusion models. We note that leveraging DC scores directly for continual personalization non-trivial Instead, we incorporate these into two popular regularization-baat based CL approaches (Wang et al.] 2024): parameter-space and function-space. For parameter-spage space regularization, we acknowledge C-LoRA's limitation, and propose Elastic Weight Consolidation (EWC) (Kirkpatrick) et al., 2017) in the LoRA parameter space. We then employ DC scores for improving the task- specific Fisher information estimates of EWC for LoRA parameters. For function-space regulariza- tion, inspired by deep model consolidation (Zhang al., 2020). we propose a double-distillation framework that leverages DC scores and noise prediction scores of a diffusion model. Hence, we dub this distillation framework as diffusion scores consolidation (DSC). We consider the practical inefficiencies for computing DC scores in EWC and DSC, and design strategies to overcome these.
We evaluate our proposed consolidation methods qualitatively and quantitatively where the number of images per concept range from 4 to 20. In doing so, we notice the flaw in the existing forgetting metric that quantifies the relative change in previous concept generation quality. Subsequently, we propose to adopt a more robust backward transfer metric that measures the absolute forgetting over tasks. Our experiments on diverse task sequence lengths validate the effectiveness of our methods, all the while requiring zero inference time overhead over C-LoRA. In the spirit of parameter-efficient CL. we explore the compatibility of our method for VeRA (Kopiczko) et al. 2024) and multi-concept generation. Lastly, we provide detailed ablations of our design choices with the hope of aiding future personalization works in leveraging DC scores.
four datasets,
2 BACKGROUND AND RELATED WORK
Diffusion models (Sohi-Dickstein et al. 2015 Ho et al. 2020) are score-based generative models that learn to reverse gradual noising process. Given an observation Xo E Rd drawn independently from an underlying data distribution ((x)), they approximate ((x)) with variational distribution pe(x)), where 0 is the learnable parameter of the diffusion model Eg. To achieve this, a forward process corrupts Xo into increasingly noisy latent variables xi...., XT using Gaussian conditional distributions IT g(x, xx-1) )) with a time-dependent variance schedule St. A reverse process then learns Pa by starting from M(x:: O)I) and predicting the gradually decreasing noise at each step (Song& Ermon 2019 2020). Although. in general, the shape of the posterior q is un- known, when Br - O. it converges to Gaussian (Sohl-Dickstein et al. 2015). Hence, by setting or = 1 3r. (x; (x.) can be approximated by modelling the mean μág and the variance Eg of po: po (XxxT) = p(x)) I
xx), where po( (x+) = (x,-1: ss0 (xt, 1), Ex ()). The training objective involves maximizing variational lower bound on data likelihood, and is achieved by denoising score matching for noise samples ~ (O I) and timesteps t ~ u[1,
Ldeenise Exx,,, [[]e co c, 111
2

---

Published as a conference paper at ICLR 2025
Cg and is the conditioning information (e.g., class). Text-to-image diffusion (Saharia et al., 2022 Rombach et al. 2022) uses a cross-attention mecha- nism (Vaswani 2017) in the U-Net (Ronneberger et al. 2015) guide each reverse process step with a text prompt encompassing c. The keys and values V to the cross-attention are rich semantic embeddings of e obtained from pretrained text encoder (like CLIP (Radford et al. 2021a)).
where X; = X + v1- Cre. d2y
Personalization of a text-to-image diffusion model aims to embed a new concept into the model with the goal of generating novel images that incorporate the model's new and prior knowledge. This is achieved by steering the reverse process through a mapping from the textual embedding o(c) the distribution of the latent image features x (see App. A for further details). DreamBooth Ruiz et al.] (2023) and Textual Inversion Gal et al. (2023) perform single-concept personalization by finetuning either all parameters of the diffusion models or by learning a new word vector V* per new concept. Improving upon these, Custom diffusion (Kumari et al. 2023) performs parameter- efficient personalization with the goal of acquiring multiple concepts given only a few examples. They finetune only the weights W of the key K and value V projection layers in the cross-attention blocks: W = [W*, W. together with regularization on pretraining prior concept dataset.
Continual Learning (CL) (Rolnick et al. 2019 Jha et al. 2020) aims to train a deep neural network on sequentially arriving tasks' data to acquire new knowledge while retaining previously learned knowledge. A popular approach to CL comprises regularization-busee methods that mitigate forget- ting by imposing a penalty term on the learning objective. Based on how the penalty is computed, the function-space Parameter-space regulariza- tion, such as Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017), constrains the changes model weights that were important to previous tasks. EWC uses Fisher information (Fisher 1922) to measure the parameter importance. Function-space regularization, like Learning without Forget- ting (LwF) (Li & Hoiem 2017) and Deep Model Consolidation (DMC) (Zhang et al., 2020), aims to preserve the model's ss output behavior on previous tasks. These typically rely on knowledge dis- tillation to ensure that the model's predictions on previous tasks remain consistent. Our work uses FWC and DMC as representatives for the aforesaid regularization techniques.
regularization may act on the parameter-space
Continual Personalization (Smith et al., 2024b) extends CL to diffusion models for acquiring N sequentially arriving personalization tasks where each task comprises single user-defined custom concept n E [1,2...., ,,N. ..,.....,,,, To respect the real-world privacy and storage concerns, a replay- CL setup is assumed such that there is no data available from previous tasks. Under such setup, as the number of tasks grow, single-concept adapters stand as poor candidates in terms of resource efficiency and knowledge transferability across tasks. These limitations call for consolidating the n. task adapter using previous knowledge to enrich it with the nuances of various concepts collectively.
C-LoRA (Smith et al., 2024b) proposes parameter-efffcient continual personalization through quential training of low-rank adapters (LoRA) (Hu et al. 2022) acting on the nh task key and value projection layers Wn E Ied xd2 This allows decomposing W, n. into low-rank residuals: Inir + +>== B,, + A,, where A, xr B, E Rxxdd is the weight ma- trix rank, and init is the initial pretrained model weight. To tackle forgetting, self-regularization loss penalizes the n task LoRA parameters for altering any previously occupied spot in W,
Wn W.Y
K.
n- A Br o B,
Liorget =11
m/=1
where || I || is the Frobenius norm, O is the element-wise dot product, and | is the element-wise absolute value. C-LoRA exploits LoRA for CL to reduce the parameters undergoing interference in incremental training, and to maintain small training/storage overhead (Biderman et al. (024))
Classification with diffusion models (Li et al. 2023 Clark & Jaini 2024) involves predicting how
likely class c; is for an input x by using uniform Bayesian prior over all classes (C1, C2. CN c
expl-Ex.ce
c,, 1]3//}
po(c; x)
E==1 exp(-Ex.ce e, clll= ea (x,, C, 1)]////
where 7 >> 0 is the temperature, and the probabilities pe (cs Pe(c x).....,Pr po xx) together comprise the Diffusion classifier (DC) scores. The expectation E is approximated over Monte-Carlo (MC) estimates across (inference) trials. Each trial samples timestep U(1 T].
3

---

Published as a conference paper at ICLR 2025
computes noisy input 22 ~ ((x1(x)) and then denoises it using the diffusion model ég conditioned on the class C, DC scores thus help leverage a diffusson model's ' rich pretrained generation knowl- edge for classification. Unlike existing works exploiting it only for zero-shot classification, we aim to use DC score as a regularization prior during training such that it can help mitigate forgetting in CL. We note that the computational costs for deriving DC scores are subject the number of conditional inputs, and the number of trials. To circumvent these, existing works rely on iterative pruning of uninformative classes (Clark & Jaini, 2024). and appropriately choosing the diffusion timesteps across trials (Li et al. 2023). However, these methods still remain practically infeasible during training - iterative pruning per training iteration is computationally intensive while restricting the diffusion timesteps range leads to loss in the signal reconstruction information. Accordingly, we propose practical considerations for efficient computation of DC scores during training.
3 METHOD
In this section, we propose adapting the existing parameter-space and function-space regularization frameworks into our continual personalization setup with LoRA. For each framework, propose incorporating the class-specific information from DC scores to enrich their regularization. Next, brief our general CL setup structured to accommodate these frameworks. We then discuss the limitation of C-LoRA that keeps it from being our choice for parameter-space regularization method.
How do we structure our CL. framework for DC scores? Using DC scores directly while acquir- ing new concepts can incur significant additional training cost (over single forward pass) given the need for several class-conditional forward passes per training image (Eq. 3. Instead, following Custom Diffusion (Kumari et al. 2023 we learn the n concept with a new word vector Vi* and a LoRA layer by optimizing the diffusion loss (Eq. D. the prior regularization loss using common prior concept Cp. and additionally solidation. After training, we freeze the word vector, and plug DC scores into two relatively shorter consolidation phases, one for each regularization method. Fig. 2 shows that these phases can work on their own as well as in tandem. Note that we train only one LoRA per task. After consolidation, the 12th task LoRA serves two purposes: (a) handling inference-time queries for {1, .... n) tasks, (b) sequentially initializing the (n + 1)th task LoRA. Next, we detail on each consolidation phase.
grrameter-regulaiizan
loss in case of paremeter-space con-
C-LoRA weights evoluion
3.1 DC SCORES FOR PARAMETER-SPACE CONSOLIDATION
Limitation of C-LoRA. Despite being relevant parameter- space consolidation candidate, C-LoRA has been shown to exhibit loss of plasticity as the self-regularizztiio penalty
Liorget (Eq. 2 increases on longer task sequences (Smith et al., find that Lioret allows for more general degeneracy where any learning on new tasks pushes the LoRA weight values toward zero. This not only effects the plasticity but also the stability of C-LoRA, right from the first incremen- tal task (n 2). i.e., when Lorget first comes into effect. We also find that Leorget has particularly catastrophic consequences for the first task concept (n 1), where the LoRA weights are learned without any forgetting constraint (see Fig. Tap. This is
2000
400
500
900
1000
2024a). Here,
Training terasion
(a) LoRA weights A, for all tasks
C-LAA evolution
Tacksa men
shown in Fig. lb. where for task 2. Liorett decreases through- out training. thus losing most of the information learned for task 1. While imposing a sparsity constraint on the function space of the task LoRA parameters might look plausible at first, we observe that this additional penalty at best delays the degeneracy rather than resolving it (see App. Fig. 7
Training fleation
(b) Cioept for incremental tasks Figure 1: Task-wise evolution of
C-LoRA's: (a) weights, (b) losses.
In light of the above, we instead opt for Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017) as our method for parameter-space regularization. While training nh task. EWC selectively penalizes the change of parameters f,- - 8,, based on their importance to previous tasks. The importance is given by the Fisher Information Matrix (FIM), computed as the expected outer product
4

---

Published as a conference paper at ICLR 2025
inttialize LoRAw1
Algorithm 1 Function for computing Denois- ing and DC scores during consolidation
LoRAy-1
LoRAn
LoRAn
updale FIM with online EWC
1: lunction GETDCS(n,
Irain
n. number of seen concepts, e: noise, eg: diffusion model. timestep. zy: noisy inpul, Ca: a subset seen con- cepts to be considered for computing DC scores
Input:
Regularize
Function-space consolidation
Diflusion Classifer Scores
Licciss: fuzssise scores, Pşi scores, pred: Noise predictions
Output:
2:
pred[e,] = eo (xe,,, d(e,), () for e, E
3: Cammiteess]
pred[c;]] 22 for É Cc
Figure 2: Framework for Continual personaliza- tion with Diffusion Classifier (DC) scores.
4:
Ie
Dummy softmax
5: Pa [og w for E [0, n]] > DC scores placeholder
6:
po pole,]
(4/1
for
Ey=t Cammalls)//
Algorithm& 3 One iteration of DSC
7: return Ldmmise Pe. pred
8: end function
Input: Dr,; Training of task, {ea, -... ...; ,, }: teacher models
from tasks, &; number of concepts to use for compuring DC scores Output: Student difflusion model c9, that has been comsolidated
Algorithm 2 One iteration of online EWC
1: Initialize: to, 60,, and set teacher-l:
Input: D,; Training data of n. task, co," diffusiin model for
2: for xo E D., do
n. task, k: number of concepts to use computing DC scores
3: Sample 14(10, 1D. 24((, II)
Output: Task-shared Fisher Information Matrix F
4: Sumple N0 C) I)
1: Xa D,
5: Sample tencher-2 id: 24(11
2: Sample ~ L<((0, 1|]. [v)).e---)) ((O.
6:
Set teacher-2: Ed;
3: ex-2 = (er ~ [c1, Cn-1]} 1]} sample w/o replacement
7:
x; = varxa + vI aic
b For Lus
4:
e ck-2+(ca; Co
b. co prior comcept
8:
X,' Vaixo + vT aaaé
b For L.c
5:
varxo VI-
9:
e (co,cy, Cn.)
co a prior concept
6: Camiie: P8. GRTDCS(n, Fen N2. C1)
10: Poaa pred, = GETDCS(N e',
Ner, cu) [snjj]
7:
Luve
11:
Lamss = ||e ca, (xx, (. 0112
for c; E cy:
12:
Loc H(pe.: pe.) + H Hpp po.)
10:
9:
gt [0....,U, i U......00 .0 D
Luc = Hipo. gt) if clse H(ps.
13:
CASE lipred, pred 112 ||pred, pred,
1
Leve 4= Lasenee [c.] 8Eoc
14: C Cannise CMss + A.e
12: Update FIM F based on V Leve
15: Perform gradient descent on V
13:
end
16:
end
of the gradients of log-likelihood wrt the model parameters:
F R Σ,
Vo lo Vo log (o)
E, Vo (O
(4)
where c is the class label prediction for an nh task input x The rightmost approximation general- izes the negative log-like likeeliolhood to arbitrary loss function Lowe. EWC can be viewed as Laplace approximation to the true Bayesian posterior over the parameters, where the FIM is a proxy for the posterior precision. The choice for the loss function Lewe is thus crucial to approximating F. With the goal of improving on this approximation, we incorporate DC scores into Lewe:
Lewe Ldenoise 5xx
Σ (2 (i=n) - 1) - H(po c:)
log is the cross-entropy between the DC scores Pe and the class label c, The intuition behind Eq. 5 that for the n task images x,,, the DC scores distribution py should remain closer to the one-hot ground truth for concept Cn, and should be farther from all other class labels Cien. Fig. 3a and Algo. outline our FIM computation framework.
where ll is the indicator function, and (a, b)
How do we adapt EWC our CL frammwwwkkk? We emphasize that for each task, we use only one LoRA, which is initialized from the previous task LoRA. While training the nh task LoRA, we incorporate the EWC regularization term to the loss function. This regularization term uses the FIM computed using the n th task LoRA. Lastly, we rely on online EWC (Schwarz et al. 2018) to store/update thak-d FIM weights, which are computed using Eq. 5 training each task.
How compute DC scores efficiently? Deriving DC scores involves two major computational hurdles: a large number of timestep samples and forward passes for all seen concepts. Previous works using DC scores for test-time classification exploit restricting the timesteps (Li et al. 2023) and iterative pruning of uninformative classes (Clark Jaini 2024) as getarounds. However, as we detail below, our training- trining-timme time setup helps us with tackling these efficiency issues.
Large number of inference trials: For the DC scores to converge, the variance of the expectation (Eq. must be low. At test time, this is achieved by averaging the scores accumulated from
5

---

Published as a conference paper at ICLR 2025
Prompe conditiond inguts
Prodicted
Dassie
Diflusion classifer scores
Ground truth
Original image
Notey
min
photo
proa of
max,
UNet
||e 2
phate ofn
max.
pRoM of Toddy
piote of Today
Uniffo, 1]
M(o.T)
pretrained
(a) Derivation of diffusion classifier scores for FIM computation in parameter space consolidation.
phomo of photo ofa pholee al
Jeacher
photo
La
Int.
Le
phana in photo a
photo of
Past LoRA pool
photo ag
LoRA consolication
insaed oRA
(b) Illustration of diffusion classifier scores for function-space consolidation.
Figure 3: Our consolidation frameworks for: (a) parameter-space, (b) function-space.
large number (>> 100) of trials per class. However, during consolidation, we estimate the FIM as an average over multiple epochs (Masana et al.) 2203). By using single trial per class per minibatch, our estimated FIM thus incorporates DC scores from diverse range of timesteps over multiple epochs.
Large number of seen concepts: Note that the number of class-conditional forward passes for DC score derivation grows linearly with the number of concepts. Here, iteratively pruming the uninfor- mative classes (Clark & Jaini 2024) still requires multiple passes. Instead, we propose reducing the cost of forward passes constant factor using subset C₁ of the number of seen concepts for DC scores computation. This subset always comprises at least two concepts: the task-shared prior concept ca. and the ground truth current task's concept C,, On top of these, we randomly sample hy-
Ik 2| previous concepts without replacement from the set {CI,..., Ca-1). where & > perparameter chosen by grid search. Note that freezing the word vector V helps us compute the textual embedding per concept (cc.) once and reuse it throughout the consolidation phase.
3.2 DC SCORES FOR FUNCTION- CONSOLIDATION
As EWC only targets the LoRA parameter values, to fully exploit the information from DC scores, consider distilling the old LoRA knowledge through function-space consolidation. The intuition behind this (see App. fig. 8 to guide the diffusion model for generating images that exhibit traits of conditioned class Cywiniski al. (2024). For our replay-free CL setup, we use the nh task im- ages to distill the th task LoRA by matching the predictions of previous task LoRA conditioned on the corresponding previous class of the latter. This involves tackling two intertwined CL chal- lenges: (a) alleviating previous concepts' forgetting, and (b) merging the knowledge of old/current LoRAs. To this end, we turn the Deep Model Consolidation (DMC) framework (Zhang et al, 2020) that uses double distillation to consolidate student model based on two teachers: the new n! task model, and the previous (n - 1 task model (Fig. 3b). Given that our distillation uses diffusion (denoising/DCC scores, we dub our DMC adaptation as Diffusion Scores Consolidation (DSC).
How do we adapt DMC to our DSC framework? DMC relies on an external dataset that chosen to be different from the training data to prevent the consolidation bias towards old or new tasks. However, for large pretrained models like ours, it is practically infeasible to ensure if the external dataset has distribution that is different from the pre- pre-training data. Moreover, constant need for external downloads can defeat the purpose of end-to-end CL (Smith et al. (0246)). We thus fall back to the available task training data for DSC. Accordingly, introduce two structural changes

---

Published as a conference paper at ICLR 2025
to DMC to improve its effectiveness. First, we use the current n2hh task data, we initialize our student LoRA from the first teacher, i.e. the n²th task LoRA. This makes the consolidation phase a smooth continuation of the n task training. Second, given that our learners comprise LoRA rather than heavily parameterized networks, we find that only using the (n - 1)th task LoRA as the second teacher is insufficient for consolidating the student with knowledge of all previous tasks (see App. fig. 9). Instead, we employ all old task LoRA as potential teachers during consolidation. Namely, in each iteration, our second teacher is an old task LoRA sampled at random.
Fig. 36 and Algo. [ outline the working of DSC. Similar to EWC, we compute DC scores using three concepts: the readily available prior concept Co, the n2h task concept c. learned by the first teacher, and previous task concept Cj<nn learned by the second teacher. The teacher LoRAs should thus generate low-entropy DC scores for their corresponding concepts. The student LoRA trained to match the DC score distributions from both teachers by minimizing the cross-entropy H for its parameters (Caron et al., (02T). Lastly, while DC scores encode class-specific information for DSC, our primary goal lies in improving the generation quality. We find that relying solely on discriminative DC scores for consolidation is insufficient for noise estimation (see App. fig. 10). Hence, we also incorporate a noise score- motee-na loss LMsE into DSC learning objective:
DDsc = (H (po.. po.) + H (pe,. pe.)) + X(MMEE(EO. cO,) + (EosE(EO,,CC.)
(6)
where LMSE(a, teacher task ids, respectively, 8 is the student id, and 7y and À are the loss weights. Note that after this consolidation, we discard 68,, and instead use the student Eq, as the th task LoRA (see Fig. 2
||a - b122 po is the DC score, and 21[1, n - 1]) the first and second
How does DSC disfer from existing distillation models? We note a concurrent branch of distillation methods (Salimans& & Ho [222] [Meng al., 2023) designed reduce the number of sampling steps for diffusion model evaluations. While distilling helps us acquire a previous concept using a fraction of the training sample steps, our main purpose behind DSC retain the previous task knowledge in a CL setup, and not to optimize on the number of sampling steps for generation. We also note another line of work where distillation is done over a certain timestep range to learn selective features (generic vs domain-specific) from source diffusion model (Hur et al.) 2024). However, we wish to distill the overall knowledge from a teacher, and hence, sample from the entire timestep range [0, 1].
EXPERIMENTS
Baselines. We compare our method with three recent customization methods: Textual Inversion (TI) GGal et al. 2023), Custom Diffusion (CD) (Kumari et al. (023). and C-LoRA (Smith et al.) 2024b). For CD, we use the best performing variant of Kumari et al. (2023) that trains separate KV parameters per task and then merges them into single model using a constrained optimization objective; CD EWC uses EWC (Kirkpatrick et al. (017) with a sequentially trained variant of CD. LoRA sequential trains a LoRA adapter (Hu et al. 2022) for the KV parameters of the CD model in a sequential manner. LoRA merge fuses all the LoRAs with equal weights (Ilharco et al. 2023
Implementation. We use Stable Diffusion v1. 1.4 and v2.0 (Rombach et al. 2022 as backbones based on the Diffusers library (von Platen et al., 2022). Following CD (Kumari et al. 2023), we train all the models for 1000 iterations on all but the Celeb-A setup. where use 2000 training iterations to capture more fine- men-ggrined facial attributes (Smith et al.) (2024b). In favor of zero-shot generalization, finetune hyperparameters only on the six task sequence of Custom Concept. For both EWC and DSC, we set the number of consolidation iterations 1/5th th of that of the training iterations number. For computing DC scores, the temperature T is set to 1.0 for all but the teacher LoRA in DSC where we set 7 to 0.05. The cardinality of c for EWC is set 5. The DSC loss weights y and À are set to 0.1 and 1.5, respectively. We detail on implementation and hyperparameters in App. H
Evaluation. For each concept, we use DDPM sampling with 50 inference steps to generate 400 images using the prompt "a photo of a V?", where V; is the modifier token learned for the 7th task Kummar et al., 2023). For CD and TI, we additionally include the concept name after the modifier token. We encode the generated and the target (real) images using CLIP image encoder (Radford et al. 2021a), and use the features for computing our metrics. We quantify a task's 's performance us- ing the following metrics computed the average over all seen concepts: (a) AMMD: the Maximum
Mean Discrepancy (MMD) (Gretton et al.] 2012) between target and synthetic image feature distri- bution, (b) CLIP 121: the CLIP image-alignment (Gal et al. 2023) and (c) KID: the Kernel Inception
7

---

Published as a conference paper at ICLR 2025
task task
target images
CD
Textual Inversion
LoRA (sequential)
C-LORA
task
taok
LoRA EWC
Ours: EWC DC
Ours: DSC
Ours: DSC EWC
Ours: DSC EWC DC
(a) Qualitative results for Custom Concept setup with 6 tasks.
task tack
target images
cD
Textual inversion
LoRÅ (sequential)
C-LoRA
task
LoRA EWC
Ours: EWC DC
Ours: DSC
Ours: DSC EWC
Ours: DSC EWC DC
(b) Qualitative results for Landmarks setup with 10 tasks.
Figure 4: Qualitative results on CL. setups generated after training on all tasks.
Distance (KID) (Binkkowsii et al. 2018) 8). To quantify forgetting, we the forgetting metric FRODD (mith et al. 2024b). which is the average change in past task concept generations from task to task. Given the relative nature of FXMD.
opt for which measures the absolute change in MMD scores for previous concept generations. We also mention the percentage (wrt the U-Net) of parameters trained for task Nuurum Train, and the percentage parameters stored between tasks as Npurum Store. We detail these metrics in App.
robust backward transfer of MMD: BwTMMD,
4.1 RESULTS
Continual Personalization of Custom Concepts. We evaluate our method on the Custom Concept dataset (Kumari et al.) 2023) that comprises diverse categories such as plushies, wearables, and toys (see App.[D We randomly sample six such concepts to form six tasks. Figure/4a[show the samples generated for tasks 1, 3, and 6 after training on all 6 tasks: the upper
compares the baselines while the lower row compares our variants. We report the quantitative results in Table [ Marked by its low BwTMMD scores, find that CD is prone forgetting the category attributes such as the color, background and appearance while also struggling with plasticity loss that leads to the generation of incorrect backgrounds for task 6. While Textual Inversion Gal et al. (2023) has zero forgetting due to its frozen backbone, it remains poor at acquiring categories. LoRA (sequential) undergoes catastrophic forgetting of past tasks' concepts (it has the highest NMDD scores among LoRA-based methods) while C-LoRA displays a higher degree of interference for past tasks. Both these methods have reduced plasticity for acquiring the last task images sunglasses appear in- correct numbers/styles. LoRA with EWC shows improvement both plasticity and forgetting. Incorporating DC scores into it further helps reduce the forgetting, with an improvement of 3.19 points for BWTMMD- On the contrary, naive DSC undergoes huge forgetting it performs distil- lation without the previous task data. Using DSC with EWC helps remedy this forgetting, albeit at the cost of reduced plasticity, egg., task 6 sunglasses with three pairs of glasses. Incorporating DC
8

---

Published as a conference paper at ICLR 2025
Table 1: Custom Concept results the end of 6 tasks (avg. over seeds)
Nuuen Train(])
Nynn Store (4)
(1)
ASMD (L) 10")
CLIP 121 (T)
Method
110))
BwTsmm (t)
(x100)
Fanm (U)
Textual Inversion
0.0
100.0
205.69
185.74
CD
2.23
100
179.4
121.89
-273.41
69.53
0.62
CD EWC
2.23
101.34 34
177.99
121.02
-245.7
69.44
0.506
LoRA sequential
0.09
100.0
203.11
176.38
-118.56
61.30
0.052
LoRA merge
0.09
100.54
261.74
312.44
-338.06
58.29
0.043
C-LoRA
0.09
173.8
-107.47
0.034
EWC
0.09
100.54
156.91
105.07
-99.34
73.19
0.008
Ours: EWC DC
0.09
100.54
154,25
102.81
-102.53
73.41
0.0005
Ours: DSC
0.09
100.54
187.2
198.45
-05.799
73.36
0.049
DSC EWC
0.09
100.54
143,92
-94.63
0.02
Ours: DSC EWC DC
0.09
100.54
140.18
94.1
-92.44
73.17
0.003
Table 2: Landmarks results at the end of 10 tasks (avg. over 3 seeds)
Npra Train([)
Njan Store (4)
KID (L) 10")
AYMO (L) (x 110))
CLIP 121 (f) (x100)
Method
BwTsmm (t)
Fanma (3)
Textual Inversion
0.0
100.0
107.03
95.1
80.26
0
CD
2.23
100
376.3
249.85
-183.02
56.95
0.047
CD EWC
2.23
102.23
377.76
250.79
-191.2
56.88
0.007
LORA sequential
0.09
169.83
-112.79
0.063
LoRA merge
0.09
100.9
225,11
144.69
-173.5
67.81
0.009
C.LoRA
0.09
100.9
104.63
68.77
-16.47
EWC
0.09
100.9
66.76
44.53
43.06
81.84
0.008
EWC DC
0.09
56.98
-5.57
82,86
0.035
Ours: DSC
0.09
100.9
111.69
73.2
-72.07
77.44
0.056
EWC
0.09
100.9
87.57
57.75
-5.91
0.004
Ours: DSC EWC DC
0.09
100.9
51.22
36.09
-3.85
82.19
0.043
scores with DSC EWC further helps improve on these results, and is our best performing variant as it displays the least KID, AMMD, and BwT scores (see App. for taskwise performances).
Continual Personalization of Landmarks. We study the compared methods on natural landmarks, using 10 task sequence from the Google Landmarks dataset v2 (Weyand et al.) 2020). We create tasks by sampling global waterfall landmarks with 20 images each (see App. D) This is a chal- lenging setup given the distinct geographical features of the waterfall landmarks despite them being overall similar. Fig. 4b shows the qualitative results from tasks 1, 5, and 10 generated after training on all 10 tasks. Table 2 reports the quantitative results. Here, CD displays catastrophic forgetting of the middle task 5. LoRA sequential shows reduced plasticity for acquiring task 10, as also marked by its high KID and AMMD scores. Similar to Custom Concept, C-LoRA displays a high degree of forgetting for task 1 while LoRA EWC helps remedy this to some extent. Our DSC-only variant remembers the generic previous task traits but struggles to accurately generate their finer details, e.g. multiple waterfalls for task 1. Using EWC with DSC helps resolve this, and plugging in the DC scores further helps improve on the results. Namely, our DSC EWC DC variant performs the best on out of 5 performance metries including the robust backward transfer score. Finally, we note that our proposed variants have zero parameter overhead over C-LoRA for training and storage.
Continual Personalization of Household objects and Real Faces. We further compare the LoRA- based methods on finegrained images of household objects from the Textual Inversion (TT) dataset (Gal al. 2023}, and that of celebrity faces from the CelebA dataset (Liu et al. 2015). These setups comprise task sequences of length 9 and 10, respectively (see App. D for details). We leave the qualitative and quantitative results in App. as with Custom Concept and Landmarks. For both these setups, C-LoRA fails to remember the finegrained details of previous concepts, e.g., exact object/facial attributes. and instead, only retains the overall feature, e.g., the dominant color. Also, our DSC EWC DC variant performs the best here.
We find the results Tollow the same pattern
Long task sequence. We study the scalability of our proposed methods to a sequence of 50 concepts chosen at random from the Custom Concept dataset (Kumari et al.] 2023), with variable number of training images per concept. To avoid any learning bias from large early tasks with more training images, we pick all 50 tasks at random rather than adding them over our six tasks sequence. App. fig. [3 compares the results of C-LoRA, LoRA EWC, and Ours (EWC DC). Similar to Smith et al. (202bb, find that the performance of C-LoRA saturates the number of tasks grow. Instead, applying EWC on the LoRA parameters emerges as a better performer on the long run. EWC with DC retains the performance particularly on the latter (> 35) tasks (see App. F. for further results).

---

Published as a conference paper at ICLR 2025
4.2 ABLATION STUDIES
We ablate the influence of DC scores in our training objective. We list two sanity checks to ensure that our framework leverages DC scores. We discuss the impact of the number of concepts used for computing DC scores, and leave further hyperparameter ablations in App. H
Sanity check I: DC scores reduce the uncertainty
wwwn_blocks.co O.tttentions. Ltransformer__eee www.kk....aaai..... 0.attnl
in FIM estimation. We perform top-5 Eigenvalue analysis for the FIM computed with and without DC scores. Fig. [ shows that DC scores helps cap- ture larger eigenvalues for the same LoRA param- eter. Intuitively, this means that the parameters are more strongly informed by the data regarding the directions of high likelihood changes. We leave the analyses of further layers in App. fig. 17
lea of Eigenvalye
LoRA
LoAk
Eigenva ue index
Figure 5: Top-k eigenvalue analysis for FIM.
Sanity check II: DC scores help with training set classification. For DC scores to help enhance the generative quality of tasks, their classification infor- mation needs to be reliable, i.e., consolidation with wrong classification scores should interfere with the generation results. To validate this, we probe the classification accuracy of different methods on train- ing data of incremental tasks, after training on all six tasks of Custom Concept. As shown in Fig. 66 con- solidating with DC scores endows us with classifica- tion gains the overall training data.
Accuracy comparison
100
C-LaRA
Ours: FWC-DC
75
EWC-DMC+DD
50
Avg.
Task
Figure 6: Training set accuracy of different methods on the Custom Concept setup.
Impact of the number of concepts for DC scores computation. We study how the number of randomly sampled previous task concepts effects the performance of EWC DC (see App. fig. 15 We notice that excluding the common prior concept Cy in DC scores computation, i.e. = 2, leads to the worst performance overall. All setups with k > 2 perform similar until task 2 as there only one available previous concept. From task 3 onward, = still gets to sample only one previous concept per iteration while k 7 can use all previous concepts each step. We find that the performance of EWC DC saturates as k increases beyond 5. This is because not all previous concepts carry useful discriminative information for reliable DC scores. We use k 5 throughout.
Training time complexity. App. table 9 shows that our proposed consolidation methods scale linearly in the training sample size whereas C-LoRA scales bilinearly in the training sample size and the number of tasks. This implies that while on shorter task sequences, the average runtime per training iteration of our methods remains higher than C-LoRA (5.3s for EWC, 5.7s for DSC, 0.8s for C-LoRA on tasks) given the several conditional forward passes needed for DC scores computation, the time gap per iteration between C-LoRA and our methods bridges as the number of tasks grow: 5.5s for FWC, 5.72s for DSC, 3.8s for C-LoRA on 50 tasks (see App. for discussion).
we show that our proposed method is compatible with VeRA (Kopiczko al.) 2024) (where replace LoRA with VeRA, and train as usual) and with multi-concept generation (where prompts include two concepts).
Compatibility for VeRA and Multi-task generation. In App. Sec. G.2
G3
5 CONCLUSION
In this paper, we propose continual personalization of pretrained text-to-image diffusion models us- ing their inherent class-conditional density estimates, iee., Diffusion classifier (DC) scores. Namely, we alleviate forgetting using DC scores as regularizers for parameter-space and function-space con- solidation. We show the superior performance of our methods through extensive quantitative and qualitative analyses across diverse CL task lengths. While DC scores have previously been utilized for test-time classification (Clark& & Jaini 2024), we are the first to advocate that such pre-trained class- class-specific knowledge of diffusion models can further be reinforced through fine-tuning and can help enhance the performance of downstream generation tasks. We thus hope that our work paves the general way for leveraging DC scores in personalization of pretrained conditional diffusion models.
10

---

Published as a conference paper at ICLR 2025
6 ACKNONLEDGEMENT
We are grateful to Yuhta Takida for their feedback and comments. We would like to acknowledge Sander Dieleman's blog (Dieleman 2024) for insights diffusion distillation.
REFERENCES
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Con- nor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: httt:ss//oopereview.eeee net to//fooummm ? iid=
aloEru2qCG Featured Certification.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL https:
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark ex-
perience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33:15920-15930, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9630-9640, 2021.
URL https:
//api. wwwant i wwwtttischhlaar... wwwp:chhhoaar..r//ooooueee ID 233444273,
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. Advances in Neural Information Processing Systems, 36, 2024.
Bartosz Cywiński, Kamil Deja, Tomasz Trzciáski, Bartlomiej Twardowski, and Lukasz Kucinski. Guide: sindanceebassde incremental learning with diffusion models. arXiv preprint MxXi::2403.03938, 2024.
Sander Dieleman. The paradox of diffusion distillation, 2024. URL https::t////.. /tttps://sander... sander. aif 0024/02/222 ww/p://aaaaaaa/e html
Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transac- tions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, 222(594-604):309-333
1922.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
URL
net. forum?id MA-uumiidNNAQQOOOTTy
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made sim- ple, efficient and adaptable. https://github... www wwtpc////nggggggcg//caaeaeeeeeee
2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con-
ference on Learning Representations, 2022. URL https:: ptttp::://ppereeiiee www.eeeiwwwwwweeeeeeeeeeeeeeeeee net forum?
Ld=nzevkeeFYff
Jiwan Hur, Jachyun Choi, Gyojin Han, Dong- Lee, and Junmo Kim. Expanding expressiveness of diffusion models with limited data via deff-distillation based finettuning In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pp. 5028-5037, 2024.
11

---

Published as a conference paper at ICLR 2025
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. The Eleventh International Confer- ence Learning Representations, 2023. URL https:: http///ppeeeee..cce net forum?id- 6t.Ckff--jr)
/
Saurav Jha, Martin Schiemer, and Juan Yc. Continual learning in human activity recognition: an empirical analysis of regularization. ICML Workshop Continual Learning, abs/2007.03032. 2020. URL URLbttps://arrt www.vvrrivvv/aam///vv//ea.a.a.aa org wwtp://pppppppe///oo/// 03002
Saurav Jha, Dong Gong, He Zhao, and Lina Yao. Npel: Neural processes for uncertainty-aware
continual learning, Advances in Neural Information Processing Systems, 36, 2023.
Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. Advances in neural information processing systems, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Grabska-Barwinska. Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):35113352 2017.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL
net. wwt.eeewwwweeeo....eeeee www.ororriiooo........ NN Ldxr
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your dif- fusion model is secretly zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2206-2217, 2023.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, and Joost van de Weijer. Class-incremental learning: Survey and performance evaluation image clas- sification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2023. doi: 10. 10.1109/PAMM.2202.2213.2
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning. 2021a. URL https:://////....t.. httt:p::/aap..erree http:pp..ssnnntccccc..ooooo org/ /tt////pul:::21511111
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8748 0788887733.00 PMLR, 18-24 Jul 2021b. URL https: /ttpp://proceedingg mlr, wwwppppessss... v11///13//////////// radford21a. html
12

---

Published as a conference paper at ICLR 2025
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience
replay for continual learning. In H. Wallach, H. Larochelle, A. Beygekzimer F. d'Alché-Bue E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32.
Curran Associates, Inc., 2019. URL https: /tttps:///rroceeednnss mew.eeiiinnnnnuunnnuooo.o neur wwtppipp.ccpppppp......ppppppppe fil wwtpe//pppp//1/////ee.ee aaaae//ecccccaaaaaaaaaaaaaaaaaee
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18. pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning cet--t-imaae diffusion models for subject- driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In International conference machine learning, pp. 4528 4537. PMLR, 2018.
James Seale Smith, Yen-Chang Hsu, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion with stamina: Stack-and-mast incremental adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1744-1754, June 2024a.
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and
Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c- loRA. Transactions on Machine Learning Research, 2024b. ISSN 2835-8856. URL https: wwppnneeiiwwnne//eeee
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International conference on machine learn- ing. pp. 2256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-base based generative models. 2020.
Advances in neural information processing systems,
Iris Van Rooij, Mark Blokpoel, Johan Kwisthout, and Todd Wareham. Cognition and intractability: guide to classical and parameterized complexity analysis. Cambridge University Press, 2019.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Ra- sul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: Sfate-of-thh-an diffusion models. https: aattps://githute com/ huggingface//eee ffusers 2022.
13

---

Published as a conference paper at ICLR 2025
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2575-2584, 2020.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1131-1140, 2020.
14

---

Published as a conference paper at ICLR 2025
Algorithm 4 Algorithm summarizing our training and consolidation workflow.
Input: Dr: Training data of n task, {EA 60,-1.....,... J: - personalization tasks, k: the number of concepts use for computing DC scores, F the task- shared Fisher Information Matrix (FIM) encoding the Fisher information from n - 1 tasks Output: the diffusion model cØ, consolidated with the knowledge of all personalization tasks, the task-shared FIM F updated with the parameter importance for the n task
teacher models from previous n
Sequential initialization 2: 68. t Train cØ, on Dn using the standard denoising score matching objective (Eq. I and the
1: co. CO.-1
Fisher penalty (Kirkpatrick et al. (017)) based F
Parameter-space consolidation P. Function-space consolidation
3: 60, + Perform DSC based on Algo. b Replace the th task LoRA with the consolidated LoRA 5: F f Perform online EWC based on Algo. >> Update F with Fisher information for n task
4: 68, f Eg,
6: return 60,,
A PERSONALIZATION IN TEXT-TO-IMAGE DIFFUSION MODELS
Personalization of text-to-image diffusion model aims to embed new concept into the model by steering the reverse process through mapping from the textual embedding é(c) to the distribution of the latent image features x, where ø is the text encoder. To do so, the text-to-iman image cross-attention
blocks in the U-Net consider the query Q = w the key K
= W ((C), the value
w wVo(c), and perform the weighted sum operation: softmax
QK
V. where the weights Wº, V
WK and V map the input x and e to Q, K, and V, respecivelly and d' the output dimension. Custom diffusion (Kumari et al., 2023) perform parameter-efficien personalization with the goal of acquiring multiple concepts given only a few examples. They show that upon finetuning on new concept. the text-projection weights W WV of the text-to-image cross-attentin attention blocks in the U- Net undergo the highest rate of changes. Subsequently, they finetune only the cross-attention weights token embedding initialization, and constrained
W = WW W together with regularization, weight merging. C-LoRA builds upon this parameter-efficient setup and further proposes training low rank adaptrs (LoRA) (Hu al. 2022 for the cross-attention layers in the U-Net. Subsequently, we consider using LoRA as well.
How do we obtain and leverage the new word vector V* in the training process? Following TI (Gal et al. 2023) and CD (Kumari et al. 2023), to personalize our text-to-image diffusion model on new concept Cy., we introduce new token representing this concept and learn its corresponding word vector V,* by optimizing only this embedding and the nh task LoRA while keeping the rest of the model's parameters frozen. To do so, we create prompts for the th concept that include the new token (e.g.."a "a photo of [V*]"). By inputting these prompts into the model and comparing the generated images with the training images, our standard denoising score matching objective (Eq, [ measures how well the model reproduces the concept. Minimizing this loss adjusts the word vector V so that the model associates the new token with the visual characteristics c,, enabling it to generate images of the concept when the token is used in prompts. Lastly, mentioned in Sec.
3] V is acquired during the training stage and remains frozen thereafter, f.e., while we perform consolidation using the DC scores.
15

---

Published as a conference paper at ICLR 2025
B C-LORA WITH SPARSITY CONSTRAINT ON TASK-1 LORA PARAMETERS
Loliitt of L1 conce ain
10g Hoar LoS
Thai Renulion
(a) Weights evolution
First tasks
First taskss
C-uftt
15 12
(b) Loss evolution
(c) C-LoRA results with LI- norm on first task.
(d) C-LoRA results (without the
LI Ll-norm constraint).
Figure 7: C-LoRA with sparsity constraint on the first task: To overcome the catastrophic for-
getting of first task in C-LoRA (Smith et al. (024b) (see Sec.
3.I. we consider restricting LoRA
weight updates during first task by incorporating a sparsity (L norm) constraint into the training objective for the first task. However, shown in Fig. 7a and 7b this merely results in the degener- ate solution shifted by one task, i.e., now the task weights (instead of task 1) undergo significant updates, which in turn causes Liieger for task 3 to decrease throughout training (as most of the task 2 spots get edited). Fig. shows the results generated by this model for the first four tasks on our Custom Concept setup. Compared the results of C-LoRA in Fig. 7d now even task 2 image generation (for the pet cat concept) is seen to exhibit catastrophic forgetting.
C DIFFUSION SCORES CONSOLIDATION (DSC) FOR FUNCTION-SPACE REGULARIZATION
plushie tortoise
pet cat
plushie panda
Generated
Real
wearable sunglasses
Figure 8: Motivation behind function-space consolidation: (wearable sunglasses) on previous classes (those around the circumference) helps generate images that share features with the previous classes (Cywiński et al. 2024). In the absence of replay sam- ples (real images) from previous personalization tasks, we exploit the aforesaid property using the current (n task images to distill the current task LoRA (finetuned on wearable sunglasses) by matching the predictions of the LoRA corresponding to the previous tasks on their respective previ- ous task concepts. Images have been resized to highlight the subject of interest. The real images for previous concepts have been provided for the sake of reference.
conditioning current task images
16

---

Published as a conference paper at ICLR 2025
task 6 task task 1
target images
Ours: init from teacher
Ours: random init
Ours: fixed teacher-2
Figure 9: Design choices and their results for our DSC framework (from left to right): (A) the ground truth target images for tasks 1, 3, and 6 of our Custom Concept CL setup; (B) generated results for our proposed DSC EWC DC framework, as also reported in Fig. 4at (C) generated results for the DSC EWC DC framework where follow DMC (Zhang al., 2020 to initialize
student LoRA using random weights: the consolidated student fails to properly acquire the previous and current task custom categories: (D) generated results for the DSC EWC DC framework where we follow DMC (Zhang et al. 2020) to use the (n - 1)th task LoRA as our fixed second teacher, rather than randomly sampling the second teacher from the pool of all previous task LoRAs: the consolidated student undergoes catastrophic forgetting of previous concepts.
Figure 10: LoRA consolidated using only DC scores in DSC (after training on task 2) generates unintelligible images. We thus opt for using DC scores together with Denoising score matching in our DSC framework (Eq. 6)
D DATASETS AND THEIR CONCEPTS
Four our Custom Concept setup, we select the following 6 classes from the CustomConceptl10 dataset (Kumari et al. 2023) with at least nine images each: furniture sofal, plushie panda, plushie tortoise, garden, transport car 1, and wearable sunglasses 1.
For our Google Landmarks v2 /Weyand al., 2020) setup, we select 10 such geographically diverse waterfall landmarks and download 20 images for each. These landmarks (and their countries) include: Bow falls (Canada), Davis falls (Nepal), Fukuroda falls (Japan), Huka falls (New Zealand), Iguazu falls (Argentina), Korbu falls (Russia), Mang falls (China), Niagara falls (Canada), Rufabgo falls (Russia), and Shin falls (the UK).
Our Textual Inversion dataset (Gal et al. 2023 setup simply uses all nine available household categories and their images: cat statue, clock, colorful teapot, elephant, mug skulls, physics mug, red teapot, round bird, and thin bird. Note that some of these concepts contain few four images.
For our Celeb-A (Liu et al. 2015) setup, we rely on its 256 x 256 resized version from Kaggle We choose 10 such celebrities at random that have at least 15 images. Their IDs include: 2079, 3272, 4407, 4905, 5239, 5512, 5805, 7779, 8692, 9295.
zed- 256x256
17

---

Published as a conference paper at ICLR 2025
E
METRICS DEFINITION
Following Smith ) we report (i) Nparam Train as the percentage of parameters (with respect to the U-Net backbone) that are trainable while learning task and (ii) param Store as the percentage of parameters that are stored over the entire task sequence. Let N be the number of personalization tasks, where each task {11,2, {1, .......... , comprises dataset D hosting single personal concept. Let X1, be the generated images for the jth task by model trained sequentially until the Th task, and XD.j be the corresponding original dataset images for the 1th task. Then, using pretrained CLIP model Fatip (Radford et al. 2b221b) the feature extractor, we define (iii) the average of the maximum mean discrepancy (MMD) AMMD metric (where lower is better) over N tasks
AMDD
MMD (
where the MMD is computed using quadratic kernel function (Gretton al. 2012). Accordingly, (iv) the forgetting metric FOMD (where lower is better) quantifies how much the generated images have diverged due sequential training:
FM
(x)) (XN.))
N 1
As also stated in Sec. the forgetting metric FMMD is relative measure of divergence with respect to the results generated by the jthh task model. It can thus be misleading for cases where the j"h task model is itself a poor learner but does forget much (possibly because allocates only tiny fraction of the parameter space for learning new tasks). To the end goal of quantifying forgetting more reliably, we introduce (v) the backward transfer metric BwTMMD that measures how much the generated images have diverged from their absolute ground truth counterparts as result of sequential training:
BWTMMD
Fain(
Contrary to FMMD. larger BwT is desirable as it implies that learning the N task helps im- prove the generative quality of the jth task images. Following Kumari et al, (2023), we additionally include the following metrics that leverage the pretrained CLIP model's 's features: (vi) the image alignment quantifying the visual similarity of the generated images with their ground truth targets in the CLIP visual feature space, (vii) the text alignment quantifying the text-to-image similarity of the generated images with their respective prompts in the CLIP multimodal feature space, and (vili) the Kernel Inception Distance (KID) Binkkwwki al. (2018) quantifying overfitting the target concept (e.g., V* panda) due to the forgetting of the pretrained knowledge (e.g., panda).
In terms of magnitude, the higher the better (1) holds for: the image alignment (121) and the backward transfer (BwTMMD) metrics, while the lower the better (L) holds for all other metrics.
F MAIN RESULTS (CONTINUED)
We study the taskwise performance (AMMI and KID) evolution of the compared methods on our CL setups of Custom Concept and Landmarks. Fig. TIa and 1lb show that overall, our DC score- based variants perform better against their non-DC score- based counterparts as well as against other baselines every incremental task.
18

---

Published as a conference paper at ICLR 2025
A A_mmd( )
KID( 4
200
200
150-
180
160
100
140
2
3
2
3
Task
CD
LoRA EWC
Ours: DSC
Ours: DSC EWC DC
C-ORAA
Ours: EWC DC
Ours: DSC EWC
(a) Taskwise performance evolution on Custom Concept CL setup.
A_ A_mmd((
KID( )
100
150
100
50
50
25
10
2
3
6
10
Task
CD C-LORA
LoRA EWC
Ours: DSC
-
Ours: DSC EWC DC
Ours: EWC DC
Ours: DSC EWC
Taskwise performance evolution on Landmarks CL setup,
Figure 11: Taskwise performances.
F.1 RESULTS FOR LONG TASK SEQUENCE
task
task 25
task 50
target images
C-LoRA
Ours
Figure 12: Qualitative results for 50 tasks Custom Concept setup: we compare the results of
our EWC DC variant with that of C-LoRA for generating the images from tasks 1, 25, and 50. We find that C-LoRA does not only suffer from a loss of plasticity acquire the 50th task images but has also undergone catastrophic forgetting of the first task images, which is inline with our findings from Sec. 3.1 On the contrary, our method scales well to mitigate the forgetting of the previous tasks while still remaining plastic enough to acquire the 50th task.
19

---

Published as a conference paper at ICLR 2025
A mmd (t)
KID (1)
280
190
C-LoRA
LORA
170
240
Ours: EWC DC
150
200
130
160
10
20
30
40
50
1
10
20
30
40
50
Tasks
Figure 13:
Performance evolution a sequence of 50 custom concepts (Kumari et al. 2023
Shaded lines represent absolute values while solid lines denote the simple moving averages over tasks.
F.2
RESULTS ON HOUSEHOLD OBJECTS AND REAL FACES
lask Task task 5 task
target images
C-LoRA
LoRA EWC
Ours: EWC DC
Ours: DSC EWC DC
Figure 14: Qualitative results for LoRA-based methods Textual Inversion dataset (Gal al. 2023
setup with 9 tasks.
Table 3: TI results after 9 tasks (avg. over seeds): (L) indicates lower is better.
Method
puram Train(])
Nparam Store ())
KID
AMM (x 10²)
BwTxaD (f)
CLIP 12I (?)
FAMD ({)
(x 10")
(x100)
CD
2.23
100.0
296.11
187.4
-20.331
68.13
0.025
LoRA sequential
0.09
100.0
261.07
132.58
-125.03
67.1
0.008
C-LoRA
0.09
100.81
181.33
105.7
-106.16
69.04
0.047
EWC
0.09
100.81
158.93
88.21
.93.18
71.46
0,003
Ours: EWC DC
0.09
100.81
139.66
82.75
-79.0
73.08
0.047
Ours: DSC
0.09
100.81
168.09
96.41
-135.74
69.81
0.008
Ours: DSC EWC
0.09
100.81
135.61
80.6
-84.11
71.08
0.008
Ours: DSC EWC DC
0.09
100.81
121.08
78.34
-73.18
72. 66
0.005
G ADDITIONAL RESULTS
G.1 COMPATIBILITY WITH STABLE DIFFUSION V2.0
For a fair comparison with the of (Smihh et al. 2024b). we report our main results using Stable Diffusion v1.4. We further note that Stable Diffusion v1.4 and v1.5 remain architecturally
20

---

Published as a conference paper at ICLR 2025
Table Celeb-A results after 10 tasks (avg, over seeds): (L) indicates lower is better.
N puram Train(J)
N param Store (1)
KID (()
ANMD (U) (x 10")
CLIP 121 (1)
Method
BwTNMM (t)
FIIDD ())
(x 10")
(x100)
CD
2.23
100.0
317.0
219.82
-215.14
42,9
0.006
LoRA sequential
0.09
100.0
286.55
308.2
-156.22
60.79
0.061
C-LoRA
0.09
100.9
151.51
101.01
-101.97
64.38
0.012
BWC
0.09
100.9
141.16
95.07
-92.78
66.37
0.01
Ours: EWC DC
0.09
100.9
127-0
87.06
-86.43
67.34
0.004
Ours: DSC
0.09
100.9
185.62
112.44
-97.5
66.81
0.003
Ours: DSC EWC
0.09
100.9
138.69
92.49
-94.13
66.29
0.061
Ours: DSC EWC DC
0.09
100.9
119.5
91.72
-82.71
68.3
0.004
Ammd((
KID( 4
190
120
180
170
Task
Figure 15: Effect of the number of concepts k for DC scores computation the six-tasks Custom Concept setup performance (refer to Algo. 2 on the usage of k). Interpretation: k dictates the cardinality of the set сą that leverage for preventing the cost of DC scores computation grow with the number of seen concepts.
usually comprises the current nhth task concept C₁ and the readily available common prior concept Cy- On top of these, we randomly sample k previous task concepts that are to be employed in the DC scores computation. As such, all setups with & 2 perform similar until task 2 as there is only one available previous concept the pool to sample from. From task 3 onward, & 3 still gets to sample only previous concept per iteration while can use all previous concepts per iteration (note that our Custom Concept setup has a total of six concepts, and thus the maximum number of previous task concepts is 5 which is during training on the sixth task). We find that the performance EWC DC saturates as & increases beyond This because all previous concepts carry useful discriminative information for deriving reliable DC scores. We thus use k = 5 across all our setups. Lastly, note that DC scores comprise probability distribution over concepts and thus require the minimum number concepts for derivation. As result, we include the pre-trained common prior concept cu besides the current 12 task concept into C₁ on all but the iteration, which is similar to
setting. On the latter setting, we sample 1 previous concept random the k = 3 setup except for the common prior concept included Cg- We find that excluding the common prior concept leads to the worst performance overall. This could be because the prior concept images might help preserve more discriminative semantic information the DC scores when compared to other downstream task concepts.
very similar in that they both leverage the CLIP ViT-L/14 text encoder and U-Net capable of pro- cessing 64 64 latent representations corresponding to 512 512 images. As Stable Diffusion v2.0 improves upon these with better text encoder and support for higher resolution images, we report the results for v2.0 on Custom Concept and Landmarks setup in tables and 6 respectively.
Table Custom Concept results for Stable Diffusion v2.0 (avg. over 3 seeds)
Nurrm Train([)
Nparam Store ())
KID (L)
AMMd (L)
CLIP 121 (?) (x100)
Method
BwTNMD (t)
FAMD (1)
(x 10P)
(x 103)
C-LoRA
0.05
100.29
158.9
114.5
-003.88
68.10
0.015
BWC
0.05
100.29
141.3
100.97
-96.50
75.81
0.011
Ours: EWC
0.05
100.29
134.97
93.59
-88.30
77.29
0.0009
Ours: DSC
0.05
100.29
177.92
155.06
-102.55
75.01
0.003
Ours: DSC EWC
0.05
100.29
139.76
93.71
-91.22
77.90
0.006
Ours: DSC EWC DC
0.05
100.29
126.43
88.54
-82.15
78.22
0.001
21

---

Published as a conference paper at ICLR 2025
Table 6: Landmarks results for Stable Diffusion v2.0 (avg. over seeds)
Narrm Train(])
Npaam Store ())
KID (L)
AMMD (1)
CLIP (t)
Method
BwTMmD (f)
PinaD (1)
(x 10")
(x 10")
(x100)
C-LoRA
0.05
100.32
101.33
62.40
-12,99
80.50
0.009
EWC
0.05
100.32
59.21
-39.05
85.37
0.002
Ours: EWC
0.05
100.32
52.80
32.65
-5.19
87.22
0.001
Ours: DSC
0.05
100.32
116.83
67.32
-65.70
80.01
0.009
Ours: DSC EWC
0.05
100.32
45.27
-5.71
86.88
0.0007
Ours: EWC
0.05
100.32
46.15
29.13
-2.90
88.14
0.084
task 3 task
task
target
VeRA (sequential)
VeRA EWC
Ours: VeRA EWC DC
(a) VeRA results: we preserve our EWC DC framework and replace LoRA Hu ct al. 2022 with Vector-based
Random Matrix Adapters (VeRA) (Kopiczko al. 2024).
+
target images
CD
C-LoRA
Ours
(b) Multi-concept generation results: the upper row images are generated using the prompt "A photo of y plushie tortoise. Posing in front of V2 waterfall" while the lower row images are generated using prompt "A photo of V plushie tortoise. Posing with 2 sunglasses".
Figure 16: Additional results showing the compatibility of our method for (a) multi-concept gen- eration, and (b) VeRA (Kopiczko et al. 2024
G.2 COMPATIBILITY WITH VERA
In the spirit of parameter-efficien continual personalization, we explore the effectiveness of our method for Vector-based Random Matrix Adaptation (VeRA) (Kopiczko et al.) (004)). VeRA freezes the LoRA weight matrices A and B to share them across all network layers, and instead adapts two scaling vectors A, and Zr per layer. This helps VeRA retain the performance of LoRA-based finetuning with small fraction. For the U-Net, this amounts to 2 TOO reduction in the number of trainable parameters (N.puram Train) per task. We rely on the Diffusers library implementation of VeRA and use the default rank setup of 256. Fig. 16a compares the results of sequential VeRA, VeRA with EWC, and VeRA EWC with DC scores on our six task sequence of Custom Concept (Kumari et al. 2023). Here, VeRA sequential suffers from loss of plasticity, e,g., sunglasses with three glasses, besides forgetting the precise details previous tasks, e.g., distorted tortoise face/eyes in task 3. VeRA EWC helps improve over this despite struggling to retain knowledge times, e.g., task panda legs with white strips. Plugging DC scores shows a clear gain in terms of the generative quality. The quantitative results are reported in App. table 7 and tell us similar story.
22

---

Published as a conference paper at ICLR 2025
Table Custom Concept results with VeRA Kopiczko et al.
2024) after 6 tasks (avg, over 3 seeds).
Nprram Train(])
puram
KID (L)
AMMD (1)
CLIP 121 (t)
Method
BwTMMM (1)
FimD (1)
Store (1)
(x 10%)
(x 10%)
(x100)
VeRA sequential
0,0086
100.052
138
95.7
-95.71
74,41
0,006
VeRA EWC
0.0086
100.052
145.2
98.27
-98.73
73.37
0,0003
Ours: VeRA EWC DC
0.0086
100.052
140
94.7
-95.44
73.56
0.0001
G.3 SUPPORT FOR MUI.TI-CONCEP GENERATION
We study the compatibility of our proposed method for generating multiple custom concepts in the same picture. TigureTThhccmmee compare the multi-concept results of our method (LoRA EWC DSC DC) with that of C-LoRA (Smith et al., 2024b) and CD (Kumari et al. (023). Following C-LoRA, use the prompt style "a photo of V [X]. Posing with V [Y]", where V and V2 are the learnable custom tokens for the concept names X and Y, respectively. We find that for both the methods, multi-concept generation remains highly sensitive to prompt engineering. and minor prompt changes (replacing posing' with 'together') can lead to results where one of the concepts is overshadowed by another. We also notice similar effects upon removing the concept names X and Y from the prompt. On the other hand, using the concept names in the prompt can lead to interference from the model's pretrained knowledge, e.g., the waterfall in the background does not always resemble the waterfall from the target images. While both the methods struggle produce images that preserve the exact style of the target images, we find that our method undergoes relatively lower interference than C-LoRA, which for instance, produces images of the plushy tortoise and sunglasses that differ significantly in color and appearance from the original target images.
Following our standard evaluation setup (Sec. 4) to quantify the performance for multi-concept generation, we generate 400 images each for the prompts used in Fig. 16b "A photo of V plushie tortoise. Posing in front of V2 waterfall" and "A photo of V1 plushie tortoise. Posing with 1/2 sunglasses". In the absence of ground truth images that incorporate multiple concepts, we quantify the scores by first relying on the ground truth images the individual concepts in each of these prompts and then averaging out the scores. Table X reports the scores averaged over the two multi-concept generation prompts. We note that these scores are in line with Fig. [16b] where our EWC DSC DC variant performs significantly better than C-LoRA on all three performance quantifi- cation metrics.
Table 8: Results for multi-concept generation on Custom Concept setup
KID (L)
AMMD(L) (x 103)
CLIP 121 (f) (x100)
Method
(x 10")
CD
231.7
194.11
48.50
C-LoRA
193.61
130.80
52.45
Ours
163.85
113.42
64.53
H IMPLEMENTATION AND HYPERPARAMETERS
H.1 IMPLEMENTATION DETAILS
We use the Hugging Face Accelerate library (Gugger et al. 2022) for distributed training/inference of our models. Our experiments are conducted using four RTX A6000 GPUs with 48 GB memory each. For all the compared methods, we set the batch size allow for larger effective batch sizes during training, we set the gradient accumulation steps to 8. For a fair comparison with CD, we perform regularization during training using an auxiliary dataset of 200 images generated by the pretrained backbone using the prompt "a photo of person" (Smith) et al. 2024b). We follow C-LoRA for preserving a number of hyperparameter setup. Namely, we set the LoRA rank to 16, and the EWC loss coefficient 1e6 for all our experiments. We use learning rate of 5e - 6 for all non-LoRA methods, and a learning rate of 5e 4 for all LoRA-based methods. We implemented C-LoRA from scratch and following the authors, used coefficient of
during training and inference. To
23

---

Published as a conference paper at ICLR 2025
down
log of Eigenvalue
10
LoRA A
LoRA
W/o DC
DC
w/ DC
10-33
10-37_
10-45
Eigenvalue index
(a) Downsampling block layer.
mid__boodk haw.bbloc...aeetiinssscsso
log of Elgenvalue
10-34.
LoRA A
LoRA
w/o DC
w/ DC
10-44
w/o DC
w/ DC
10-36.
10
10-45
10
10-20
144
Eigenvalue index
(b) Bottleneck (mid) block layer.
up up_blocks.3 att.enntionss...assforeee..o 0.att1
Eigenvalue
LoRA A
LoRA
w/o DC
w/ DC
:3
w/o DC
w/ DC
10-34
30
10-45
10 19
10
10-42
log
10-31
3
Eigenvalue index
(c) Upsampling block layer.
Top-k eigenvalue comparison for FIM estimated with and without DC scores using LoRA parameters for randomly chosen U-Net layers belonging to: (a) downsampling block, (b) mid block, and (c) upsampling block.
Figure 17: Sanity Check for DC scores:
1e8 for the self-regularizatioon loss Lionet (see Eq. 2. Lastly, following the default setup of the
Diffusers library (von Platen et al. 2022). we set the classifier guidance scale to 7.5 to allow for higher adherence the conditional signal.
H.2 IMPACT OF HYPERPARAMETERS
H.2.1 NUMBER OF CONSOLIDATION ITERATIONS
We tune the number of consolidation iterations for EWC phase using our EWC DC variant and that for DSC phase using our EWC DSC DC variant. The number of consolidation iterations are fractions of the total training iterations, i.e., 1000 Custom Concept, and are chosen through sweep on the set: f0.1x.0.25
0.3x, 0.3x, 1x] }. As shown in Fig. and 20 a value of 0.2x the training iterations performs the best for both EWC DC and EWC DSC DC. While larger number of iterations can lead to degradation in the generative quality both the variants, we note that DSC remains more sensitive overall to the number of consolidation iterations.
24

---

Published as a conference paper at ICLR 2025
down
log of Eigenvalus
LoRA A
LoRA
2.0
w/o DC
DC
10-434
w/ DC
10-339
10-254
10
10-372
1074
Eigenvalue index
(a) 100 iterations
down htwp://ccssstttttttosss.rrasssse
log of Elgenvalue
10-32_
LoRA A
LoRA
w/o DC
w/ DC
w/o DC
w/ DC
10-1A -1-
10-15_
10-44
10-17
10-44
Eigenvalue index
(b) 200 itcrations (standard setup)
log of Eigenvalue
LoRA A
LoRA
10-137
w/o DC
w/ DC
w/o DC
w/ DC
10
144
20-343
10:34
10-153
30-11
2
Eigenvalue index
(c) 300 iterations
www.w_blockk..oo .. wtt.r/tiiss...ttaaaaeeeee.ebeooo
log Eigenvalue
LoRA
LoRA B
10-123
w/o DC
w/ DC
10-127
w/o DC
w/ DC
10-195
10-148
10-150
10-36
10
10
Eigenvalue index
(d) 400 iterations
Figure 18: Effect of varying consolidation iterations FIM estimation: Top-5 Eigenvalue com- parison for the FIM estimated with and without DC scores using the learned LoRA parameters for randomly chosen downsampling layer block of the U-Net. Upon increasing the number of con- solidation iterations to 200, we observe larger magnitudes of Eigenvalues, thus indicating that the learned parameters become more certain about the directions of high loss changes from data, i.e. reduced uncertainty in the FIM estimation. However, increasing the number of consolidation itera- tions beyond 200 leads to a saturation in the data-informed uncertainty estimation for the FIM.
25

---

Published as a conference paper at ICLR 2025
A_ A_mmd (4)
KID (4)
104.0
103.5
156
103.0
0.>
0.2
0.3
0.5
0.2
0.2
0.3
05
Iterations
Figure 19: Effect of number of EWC iterations for EWC DC
A mmdddd (4)
KID (4)
144
142
94
0.3
0.5
0.1
0.2
0.5
lterations
Figure 20: Effect of number of DSC iterations for EWC DSC DC
Distribution of 2000 Random Samples between and 1000
Distribution of 1000 Random Samp les between 0 and 1000
Mean: 493.11 11
Mean: 498.79
Std Dev. 292.61
40
Stdd Dev: 292, 36
Count
60
Count
20
20
10
200
400
600
800
1000
200
400
6000
800
1000
Value
Value
(a) 2000 iterations
(b) 1000 iterations
Distribution Samples
Distribution 200 Random Samples between and 1000
Mean:
Mean: 480.80
293.17
Std 281.99
15
Count
Count
10
1.
200
400
600
800
1000
Value
Value
(c) 400 iterations
(d) 200 iterations
Figure 21: Distribution of uniformly sampled timesteps over a varying total number of training
iterations: (a) 2000 finetuning iterations used by Smith et al. (2024b) for Celeb-A faces, (b) 1000 finetuning iterations as used by Custom Diffusion Kumari et al. (2023) that we adopt for all but the Celeb-A setup, (c) 400 consolidation iterations that adopt for the Celeb-A setup, (d) 200
consolidation iterations that we adopt for all but the Celeb-A setup. Note that the values for (c) and (d) are chosen per the hyperparameter tuning we perform in App. H.2.1
H.2.2 HYPERPARAMETER FOR EWC LOSS
Fig. 22 shows the impact of varying the hyperparameter controlling the contribution of the cross- entropy term for EWC DC (Eq. 5). We find the range [0.5, 1.0] to suitable for the loss weightage, and use 8 0.5 through experiments.
26

---

Published as a conference paper at ICLR 2025
A_ mmd (4)
KID ( 4)
103
156
102
101
0.1
0.5
1.0
15
0.1
0.5
1.0
1.5
Figure 22: Effect of 8 on performance of EWC DC
H.2.3 HYPERPARAMETTER FOR DSC LOSS
Effect of varying : For DSC, y depicts the strength with which the student model 6, follows the DC scores distribution of the nhh task teacher 6, and the previous task teacher 8y. As shown in Fig. 23 the range [0.01, 0.1] remains suitable for Accordingly, we set y to 0.1.
mmmd
KID
160
100
150
95
140
0.01
0.05
0.1
1.0
0.05
155
Figure 23: Effect of on performance of DSC EWC DC
Effeet of varying A: X in DSC guides the strength with which the student 0, matches the noise estimations of the nº task teacher 6, and the previous task teacher fy. Setting
leaves the student consolidation to be guided solely by the discriminative DC scores, setting that we find to be detrimental for the purpose of generation (see Fig. TO). In Fig. 24 we delve further into the impact of varying À on the performance of DSC EWC DC, and find the range [0.5, 1.0] to work well. Note that a high \ can lead the student to overfitting the noise estimations for the previous task teacher, for which the current task inputs remain out-of-domain. This, in turn, harms the generative quality the student.
A mmd (4
KID (4)
95.5
141.5
141.0
94.5
140.5
5.
10
15
20
2.5
3,0
35
2
2.5
3.0
Figure 24: Effect of À performance of DSC EWC DC
Effect of varying the teacher's softmax temperature :: Knowledge distillation frameworks typi- cally rely on sharp target distributions that intuitively mimic the outputs of confident teacher model (Caron et al. 2021). For our DSC framework, the teachers can be made to produce sharper targets by using a low value for the temperature in the sofimax normalization operation of their DC scores. We show the impact of varying the teacher softmax temperature in Fig. 25 Namely, a temperature beyond 0.1 leads to softer targets from both the teachers, which be harder to mimic for the stu- dent. On the contrary, т mimics extreme sharpening and produces onc-hot hard distributions. We find temperature of 0.05 to perform the best overall.
27

---

Published as a conference paper at ICLR 2025
A_mmd )
KID (
110
160
140
0.03
0.05
0.5
1.0
0.03
0.
0.2
0.5
1
Figure 25: Effect of teacher softmax temperature on performance of DSC EWC DC
A mmd 4)
KID
160
0.7
100
150
01
0.05
0.1
0.5
1.0
140
0.01
0.05
0.1
0.5
1.0
Figure 26: Effect of 8 and y on performance of DSC EWC DC
H.2.4 HYPERPARAMETER FOR EWC+DSC LOSS INTERACTIONS
I TRAINING TIME COMPLEXITY DERIVATION AND ANALYSES
We use the soft-O notation Ô (Van Rooij al. 2019) to describe the time complexity while safely ignoring the logarithmic factors. Formally, for some constant k. (f) provides the upper bound for J. like the standard big-O notation O but hides the factors involving powers of logarithms, ie., (n) could represent (n log n.), On log log n), O(n n). etc.
(f log k(m)
As also stated in the main paper, we consider continual personalization setup with number of tasks such that each task comprises on new concept to acquire. For the ease of computation, we assume that each task has fixed number of training images, [D]. Note that for our CL setup, we use the same number of training epochs for each task. This lets us ignore the factor of training epochs in deriving the training time complexity. Lastly, since both C-LoRA (Smith et al. (024b) and our setup train a single LoRA and modifier token per task, their complexity of a forward pass remains the same, and can be safely ignored. Put together, we can state time complexity as a function that grows linearly with more training samples [D]. We list the training time complexities of the compared methods in Table I and detail on their derivation below:
Table 9: Training time complexity analyses with soft-O notation Ô.
Method C-LoRA
Cost O(-) N D
Ours: EWC
Ours: EWC DSC DC
[D]
1. C-LoRA (Smith et al. 2024b) performs self-regularization using the weights of all previ- ous task LoRA (see Eq. 2). Therefore, in addition to the training sample size, the time complexity of C-LoRA is dependent the number of tasks N, i.e. ((N
2. For parameter-space consolidation, we rely on online EWC (Schwarz et al. 2018) which maintains a single set of FIM weights that are updated continuously using running average over tasks. This ensures that our EWC-based framework does not store separate importance weights for each task, and hence, the time complexity scales linearly in the factor of sample size, Ō((D)). Next, for DC scores computation, we assume a fixed number of conditional forward passes that is proportional to the size of the relevant concept set C₁ (see Sec. 3.11. Irrespective the number of tasks, G₁ always stores m + number of concepts, where is chosen through grid search and is typically low number for avoiding confusion
28

---

Published as a conference paper at ICLR 2025
from other uninformative classes. Hence, DC scores computation for EWC scales linearly with the number of training samples O((D).. Put together, the time complexity for our parameter-space consolidation framework is: O((D)) O(ID)) O(DD))
3. For function-space consolidation, we rely on a teacher and one student LoRA per consolidation iteration, irrespective of the number of seen tasks. Subsequently, the training time complexity of function-space consolidation remains ((D)). For computing DC scores, we always rely on three conditional forward passes through each of the teachers and the student. As described in Sec. 3.2 these forward passes correspond to the readily available common prior concept Co, and the concepts Cn and Cjen corresponding to the current task n and the previous task j < teacher LoRA. Hence, the time complexity of DC scores computation is also O(D)). Overall, the time complexity for our function-space consolidation framework remains: Ó((D)) + Ō((D)) O((D)).
framework, which uses
Runtime per training iteration. While our method scales better than C-LoRA (Smith et al. 2024b) with the number of tasks (see Table ( we are nevertheless bounded by the several con- ditional forward passes needed (depending on the value of k) to derive DC scores for each training minibatch. Despite our proposed considerations for efficient computation of DC scores during con- solidation (see Sec. 3.I). the computational overhead for deriving these remains dominant specifi- cally for CL setups with fewer number of tasks. For example, using an RTX A6000, each consolida- tion iteration for EWC requires z 5.3s, that for DSC requires == 5.7s, and that for C-LoRA requires 0.8s during finetuning to the 50 tasks setup. this time gap bridges as the runtime per training iteration of C-LoRA grows R 3.8s while that of our methods stay roughly the same ( 5.5s for EWC, and 5.72s for DSC for A = 5). It is worth noting that on the 50th training task, for lower values of k, the runtime per training iteration for our methods remain comparable (for 3) or significantly lower (for 2) than that of C-LoRA. Therefore, we expect that more clever ways to derive the DC scores during training can effectively reduce the runtime of methods.
the task 6 of our Custom Concept setup. As shown in Table IO scaling
Table 10: Comparison of training time per iteration (wall clock time in seconds) with varying k
Method
k= k
5
On 6th task
C-LoRA
0.8
Ours: EWC DC
1.27
3.4
5.3
Ours: DSC DC
1.65
3.61
5.7
On t0 task
C-LoRA
3.8
Ours: EWC DC
1.36
3.57
5.5
Ours: DSC DC
1.8
3.66
5.72
29

---

Published as a conference paper at ICLR 2025
J FAILURE CASES
Cuslam Concept
Landmarks
Textual Inversion
Figure 27: Failure cases showing the visual artefacts of our best performing variant EWC DSC DC on our three different dataset setups.
Despite our method retaining significantly better task-specific generation granularity compared to the state-of-the-art, it produces noticeable visual artefacts sometimes. Fig. [2 shows few such dataset-specific artefacts for EWC DSC DC, which is our overall best performing variant leveraging DC scores with EWC and DSC. Notably, for Custom Concept, the model at times generates figures that have out-of-prope proportion shapes including absence of the plushie panda's 's body (left), naturally big head for the plushic tortoise (middle), and poorly outlined frame for the wearable sunglasses (right). For the waterfall landmarks setup, we notice multiple incomplete rainbows (left), a transparent yet poorly formed bridge over the river (middle), and a mulberry colored waterfall fore- ground (right). Similarly, on the textual inversion setup, the generated clock image has incorrectly printed numers (left, with 1l replacing and 3 being confused with 9), the teapot with incorrectly assigned spout/handles (middle), and the elephant's 's body with holes that have unnaturally filled background.
30


Reviews:
[{"review_id": "xpEe7Aolfl", "reviewer": "Reviewer_CPEf", "summary": "The paper presents a novel framework for continual personalization (CP) in text-to-image diffusion models using Elastic Weight Consolidation (EWC) and Diffusion Scores Consolidation (DSC). A key part of their approach is using DC scores for regularization, which leverages class-specific information from the diffusion model.", "questions": "- In explaining the limitation of C-LoRA, it is said that “L_forget decreases throughout training, thus losing most of the information learned for task1”. If I’m not mistaken, no forgetting happens when L_forget is close to zero since the modified parts of the parameters do not overlap with the previous LoRA parameters. Is there something I’m missing?\n- Is there a difference between the results in Section 3.1 and the proposed method? Specifically, could you provide the results in Figure 1 for the proposed method? I’m curious if the changes in weight and loss differ from those observed in C-LoRA.\n- In line 172, the authors mention learning a new word vector  $V_n$ , but neither the algorithm nor the figure includes this  $V_n$ , which is important for personalization. It only appears to be used during evaluation. This creates confusion about how  $V_n$  is obtained and when it is used in the training process.\n- Including specific examples of pre-trained concept $c_0$$c$  and  $V_n$  would make it easier to understand.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The use of diffusion classifier for continual learning is novel and interesting.\n- The limitation of the previous work is well addressed.\n- Related work is well summarized.", "weaknesses": "- Variance in approximation of expectation: During consolidation and updating FIM, the expectation over DC scores is approximated with a single trial per minibatch. While the authors aim to mitigate this by averaging across epochs, this can induce a biased Monte Carlo estimate. More rigorous analysis or empirical evidence is needed to demonstrate that this approximation does not compromise stability or lead to noisy FIM updates.\n- Loss Component Interactions: The paper introduces multiple loss components, which can make hyperparameter tuning challenging. While the authors conducted an ablation study to evaluate the impact of individual losses, they did not address how these losses interact with each other. A more in-depth analysis of the interactions between different loss terms would strengthen the paper and provide clearer insights into optimizing the overall training process.\n- The link between classification and generative quality: For **Sanity Check II**, the paper lacks sufficient detail regarding the experimental setup. It does not clearly define what classifier is being used or how the classification is being performed (Is that diffusion classifier accuracy with a single trial?).\n    - Additionally, I don’t think that improved classification alone does not necessarily imply that the quality of generated images will also improve. There needs to be a clear demonstration of the connection between accurate classification and enhanced image generation to support this claim.\n- Selection of $k$ (number of sampled tasks) for figure 16**:** I think choosing the number of sampled tasks  $k$  is crucial for achieving good results. However, the paper does not provide a clear strategy for determining the optimal  $k$ . While it shows that selecting five tasks works best in an experimental setting with six total tasks, it does not offer guidance on how to select $k$ when the total number of tasks $n$ changes.\nEditorials: The notation for class labels is somewhat unclear throughout the manuscript. c is introduced as a text prompt in line 103, used as a class label in line 146, and used as a one-hot label in line 245. Since this notation is widely used throughout the manuscript, it is important to clarify its definition in the first place. Using superscript as an index is confusing, especially with exponentiation.\n- Minor comments\n    - Equation 5 needs to be fixed. Since L_denoise is an expectation over data, noise, concept, and time steps, the second term in r.h.s. of Eq (5) needs to be an expectation over data.\n    - The title of the manuscript is somewhat misleading. What does secret mean in this context?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3K5diYc28d", "reviewer": "Reviewer_76RN", "summary": "This paper introduces a novel approach for continual personalization of text-to-image diffusion models using Diffusion Classifier (DC) scores. The method focuses on integrating DC scores as regularizers to mitigate the problem of forgetting in continual learning setups. By employing parameter-space and function-space regularization techniques, the authors aim to maintain previously acquired knowledge while integrating new concepts. The approach is evaluated against several baselines across multiple datasets and scenarios, demonstrating improved performance in retaining learned concepts with minimal parameter overhead.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of the paper is straightforward and clear.\n2. The paper is technically sound, presenting a well-structured approach for continual personalization of text-to-image diffusion models using Diffusion Classifier (DC) scores.\n3. The method seems computationally efficient, supported by sufficient experimental results and details.", "weaknesses": "**Strengths:**\n\n1. The motivation of the paper is straightforward and clear.\n2. The paper is technically sound, presenting a well-structured approach for continual personalization of text-to-image diffusion models using Diffusion Classifier (DC) scores.\n3. The method seems computationally efficient, supported by sufficient experimental results and details.\n\n**Weaknesses:**\n\n1. While the paper is technically proficient, the novelty of this paper seems limited. The primary innovation appears to be the use of diffusion scores as a regularization term for continual personalization, which, while interesting, is not a significant departure from existing methods. The core of the approach builds upon well-established techniques such as Elastic Weight Consolidation, adapting them for diffusion models rather than introducing fundamentally new methodologies.\n2. Since Stable Diffusion versions 1.5, 2.0, 2.1, and XL have already been released, why not use the newer versions? At the very least, incorporating some of them would demonstrate that your method can be easily generalized across different architectures.\n3. The paper would be more interesting if it could test on more recently proposed personalization methods like LyCORIS [1] and DoRA [2]. \n4. Since multiple-concept generation is common when verifying the effectiveness of proposed personalization methods according to your cited papers [3,4], could the authors further provide experimental results on multi-concept generation results?\n\n[1] Yeh, Shih-Ying, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. \"Navigating text-to-image customization: From lycoris fine-tuning to model evaluation.\" In *The Twelfth International Conference on Learning Representations*. 2023.\n\n[2] Liu, Shih-Yang, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. \"Dora: Weight-decomposed low-rank adaptation.\" *arXiv preprint arXiv:2402.09353* (2024).\n\n[3] Ruiz, Nataniel, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.\" In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 22500-22510. 2023.\n\n[4] Kumari, Nupur, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. \"Multi-concept customization of text-to-image diffusion.\" In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 1931-1941. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gfuVmRQUPe", "reviewer": "Reviewer_w18g", "summary": "This paper focuses on the continual personalization of pretrained text-to-image diffusion models. It aims to address the challenge of balancing new concept learning and previous concept forgetting \nusing a two level optimization strategy, i.e Deep Model Consolidation (DMC) and Elastic Weight Consoliation (EWC) . Buliding on DMC and EWC, the author introduce Diffusion Classifier (DC) as an additional constrait during consolidation.", "questions": "In Algorithm 1, lines 5-6, the update of the DC scores dictionary is unclear. Line 6 shows that $P_\\theta[c^i]$ is normalized for a subset with a coordinate of $k$, while line 5 indicates it is a probability distribution over $n + 1$ concepts.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The key innovation of this paper is the combination of DC scores with the EWC algorithm. This provides an interesting approach to applying EWC in T2I diffusion models.\n2. The paper is well-written, with a clear and detailed explanation of the preliminary concepts in the related work section. The discussion on the limitations of C-LoRA naturally leads to introducing the EWC algorithm and the use of DC for likelihood estimation.", "weaknesses": "1. The optimization for the $n_{th}$ task combines a function-space objective with DSC (Eq. 6) and a parameter-space objective using EWC, but the paper lacks a detailed objective function that includes the FIM. This makes it difficult to understand how DSC and EWC work together during optimization. Consider adding extra consolidating Algorithms 1, 2, and 3 into a summary algorithm that corresponds to Fig. 2 to improve clarity. \n2. In Fig. 4 and Table 2, results show that EWC + DC and DSC + EWC + DC achieve comparable performance. Given that the DSC algorithm introduces complexity similar to that of EWC + DC, this raises questions about the necessity of DSC’s design.\n3. A key limitation of this method is computational complexity. Each iteration of DSC and EWC + DC requires multiple complete diffusion forward and backward passes (Eq. 4 and Eq. 6). In contrast, C-LoRA’s explicit regularization is more efficient. It is recommended to further discuss the complexity of a single iteration and to provide a quantitative description of the complexity reductions achieved through pruning optimizations, such as selecting a subset of seen concepts (Fig. 16).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)