How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?
Authors: Jiahua Dong
Wenqi Liang
Hongliu Li
Duzhen Zhang
Meng Cao
Henghui Ding
Salman Khan
Fahad Khan
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Diffusion
Continual Learning
Concept Customization
Primary area: diffusion_based_models
TL;DR: We aim to tackle a new practical concept customization problem named Concept-Incremental Flexible Customization (CIFC), where the main challenges are catastrophic forgetting and concept neglect.

Abstract: Custom diffusion models (CDMs) have attracted widespread attention due to their astonishing generative ability for personalized concepts. However, most existing CDMs unreasonably assume that personalized concepts are fixed and cannot change over time. Moreover, they heavily suffer from catastrophic forgetting and concept neglect on old personalized concepts when continually learning a series of new concepts. To address these challenges, we propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM), which can resolve catastrophic forgetting and concept neglect to learn new customization tasks in a concept-incremental manner. Specifically, to surmount the catastrophic forgetting of old concepts, we develop a concept consolidation loss and an elastic weight aggregation module. They can explore task-specific and task-shared knowledge during training, and aggregate all low-rank weights of old concepts based on their contributions during inference. Moreover, in order to address concept neglect, we devise a context-controllable synthesis strategy that leverages expressive region features and noise estimation to control the contexts of generated images according to user conditions. Experiments validate that our CIDM surpasses existing custom diffusion models. The source codes are available at https://github.com/JiahuaDong/CIFC.

How to Continually Adapt Text-to-lmage Diffusion Models for Flexible Customization?
Jiahua Dong' Wenqi Liang" Hongliu Li Duzhen Zhang", Meng Cao', Henghui Ding*. Salman Khan Fahad Shahbaz Khan
"Mohamed bin Zayed University of Artificial Intelligence
"Shenyang Institute of Automation, Chinese Academy of Sciences 'The Hong Kong Polytechnic University *Institute of Big Data, Fudan University
Australlin National University LLinköping University [dong]iahua1995, 1iangwenq10123, hongliuli1994 henghui ding))gmaill.............m.m com
{duzhen, {duzhen.zhang zhang, meng. cao, salman khan, fahad. Chanuambbuuii... ac
Abstract
Custom diffusion models (CDMs) have attracted widespread attention due to their astonishing generative ability for personalized concepts. However, most existing CDMs unreasonably assume that personalized concepts are fixed and cannot change over time. Moreover, they heavily suffer from catastrophic forgetting and concept neglect on old personalized concepts when continually learning a series of new concepts. To address these challenges, we propose novel text-to-image Diffusion Model (CIDM), which can resolve catastrophic forgetting and concept neglect to learn new customization tasks in concept-incremental manner. Specifically, to surmount the catastrophic forgetting of old concepts, we develop a concept consolidation loss and an elastic weight aggregation module. They can explore task-specific and task-shared knowledge during training, and aggregate all low-rank weights of old concepts based on their contributions during inference. Morcover, order to address concept neglect, we devise a context- controllable synthesis strategy that leverages expressive region features and noise estimation to control the contexts of generated images according to user conditions. Experiments validate that our CIDM surpasses existing custom diffusion models. The source codes are available at https: //github. com/ J iiahuaDong/CIF
Incremental
Introduction
Latent diffusion models (LDMs) [38, 33, 66, 19] have demonstrated unprecedented capabilities in generating high-fidelity images by incorporating large-scale collections of image-text pairs in the latent feature space. Until now, LDMs [4, 30) have achieved remarkable progress various application fields, including image editing [29, 26), art creation [8, 54], and reconstruction of fMRI brain scans [48]. order to synthesize some personalized concepts according to user preferences, custom diffusion models (CDMs) [40, 60, 11]rely on low-rank adaptation (LoRA) [14] to finetune the large-scale LDMs for multi-concept customization [22]. They extend the lingon-language dictionary of pretrained LDMs bind personalized concepts with specific subjects users need to synthesize.
Generally, most existing CDMs [60, 68, 3] assume that users' personalized concepts are fixed and cannot incrementally increase over time. However, this assumption is unrealistic in real-worls applications, where users want to continually synthesize a series of new personalized concepts from their own lives. To address this setting, CDMs [3, 57, 2] typically require storing all image-text
Equal contributions (ordered alphabetically). CCorespondiin authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

training pairs of old concepts to finetune the pretrained LDMs via LoRA [14, 59]. Nevertheless, the high computation costs and privacy concerns [52] may render CDMs impractical as the number of old personalized concepts consecutively increases. If the above CDMs retain all low-rank weights associated with old concepts that are obtained previous customization tasks and then merge them to learn new personalized concepts continually [59, 65], they may experience significant loss of individual attributes on old personalized concepts (i.e., catastrophic forgetting [36, 6)) for versatile customization. Moreover, in real-world scenarios, users may wish to control the contexts and objects associated with multiple old concepts in synthesized images according to the conditions they provide (e.g., scribble or bounding box [24, 44]). It forces CDMs [60] to heavily suffer from the challenge of concept neglect [1] (i.e., some old concepts are missing during multi montiit composition).
To handle the above real-world scenarios, in this paper, we propose a new practical problem named Concept-Increment
Flexible Customization (CIFC). the CIFC setting, as shown in Fig. (a)) CDMs can consecutively synthesize a sequence of new personalized concepts in concept-incremente manner for versatile customization (e.g., multi-concept generation [21], style transfer [56] and image editing [29]). Additionally, users can control the context and objects of the generated images based on the specific conditions they provide, As aforementioned, the CIFC problem faces two main challenges for versatile concept customization in this paper: catastrophic forgetting of old personalized concepts when learning new concepts consecutively under a inneep--nerrmmenal manner, and concept neglect when performing multi- concept composition according users- esers-provided conditions.
To resolve the challenges in CIFC, we develop novel CConcept-Incremeet text-to-image Diffusion Model (CIDM), which can effectively address catastrophic forgetting and concept neglect. On one hand, to mitigate the catastrophic forgetting of old personalized concepts, we propose novel concept consolidation loss for training and devise an elastic weight aggregation (EWA) module for inference. This loss employs learnable layer-wise concept tokens and an orthogonal subspace regularizer to explore task-specific knowledge (i.e., unique attributes of personalized concepts), while learning layer-wise common subspaces across different tasks to capture task-shared knowledge. Additionally, the EWA module utilizes learnable layer-wise concept tokens to merge all low-rank weights of old personalized concepts, based on their contributions to versatile concept customization. On the other hand, we develop context-contrrllabb. synthesis strategy to tackle concept neglect for multi multi-concept composition. It leverages layer-wise textual embeddings to enhance the expressive ability of region features and relies on region noise estimation to control the contexts of generated image, conforming to users-provided conditions. Comprehensive experiments illustrate the effectiveness of our proposed CIDM in addressing the CIFC problem. The main contributions of this paper are listed below:
We propose a new practical problem named where the main challenges are catastrophic forgetting and concept neglect. To address the challenges in the CIFC problem, we develop a novel Concept-Incremenne text-to-image Diffusion Model (CIDM), which can learn new personalized concepts continuously for versatile concept customization.
Flexible Customization (CIFC),
We devise a concept consolidation loss and an elastic weight aggregation module to mitigate the catastrophic forgetting of old personalized concepts, by exploring witsssppccc/ttskshaeeeee knowledge and aggregating all low-rank weights of old concepts based on their contributions in the CIFC.
We develop a context-controllabbe synthesis strategy to tackle concept neglect. This strategy controls the contexts of synthesized images according to user conditions by enhancing the expressive ability of region features with layer-wise textual embeddings and incorporating region noise estimation.
2 Related Work
Incremental Learning [25, 49, 20] accumulates previous experience to incrementally learn new tasks without the need for retraining from scratch. To prevent catastrophic forgetting of old tasks, most incremental or continual learning models mainly employ knowledge distillation between old and new tasks [63, 36, 5], replay some images from old tasks [16, 43], or dynamically expand network architecture to encode new knowledge [61, 15, 7]. Nevertheless, these methods [16, 61, 49, 36] are primarily designed for classifying new object categories consecutively, which cannot be directly applied to tackle continual concept customization tasks without catastrophic forgetting.
Concept Customization [11, 40, 22] focuses on extending large-scale diffusion model [31, 46, 64) to
synthesize personalized concepts for users. After [40] proposes to tackle the subject-driven generation

---

[LieeWe MOREON LOCKE FIMES
infernce
Threse
Canditooni:
The BIILOW
Vo Dog The Thrass
891
88
va Tedayour Sear The
7---
Can Mear
Stuaate EUno
LAV
CERCRETS
Learer
Theie
Www.aao CorsoA
Fiiiieadee Lon
1%
Concept Consolidtion Loss
Ebattisstic AgpretAggoeoatton
Co.Consssnnttlllall Synchesis
Figure 1: Diagram of the proposed CIDM to address the CIFC problem. It consists of (a) concept consolidation loss, (b) an elastic weight aggregation module to resolve catastrophic forgetting, and (c) context-controllable synthesis strategy to address the challenge of concept neglect.
by finetuning all network parameters of the pretrained diffusion model [48] on personalized concepts, some works use textual inversion [9, 53] to learn word embeddings of personalized concepts [50]. For multi-concept customization, [22] can jointly train multiple concepts or combine different diffusion models by optimizing few parameters in the cross-attention layers, while [27] aims to capture different clusters of concept neurons. Motivated by [22], Han et al. [12] finetune the singular values of latent encoding weights, thereby improving the efficiency for concept customization. [42, 57, 18] perform efficient test-time customization by training concept-specific encoders. Besides, [59, 11] fuse multiple low-rank weights to resolve multi-concept customization. In order to tackle continual text-to-image synthesis tasks, Sun el al. [47] devise lifelong diffusion model to accumulate concept information. Unfortunately, it cannot control the contexts of synthesized images and suffers from concept neglect in multi-concept composition [62, 58]. To address the issue of missing concepts, [66, 30] utilize spatial conditions (e.g., sketch and pose) for composition. However, these custom diffusion models [59, 47, 51, 58] cannot consecutively learn a sequence of new concepts to tackle the CIFC problem, as they face challenges related to catastrophic forgetting and concept neglect.
3 Preliminary and Problem Definition
Preliminary: Latent diffusion models (LDMs) [41, 10] rely on some conditional inputs (e.g., g., text prompt [33, 37] image [17, 32)) to control the contexts of synthesized images. They use encoder E() and decoder D--) to perform image synthesis the latent space [33]. Custom diffusion models (CDMs) [12, 9, 28] utilize low-rank adaptation (LoRA) [14, 55] to learn new personalized concepts by finetuning the pretrained LDMs [1, 38]. Given pair of personalized image x and its text prompt p. the encoder E(-) maps to latent feature and z, denotes the noisy latent feature at the t-th 11 T) timestep. After the text encoder T(-) (e.g., pretrained CL.IP [35]) maps to the textual embedding r(p), the objective to learn personalized concept {x, p) the t-th timestep is:
LCDMs = E
egr (ze |c. 1)
where egr (-) denotes the denoising UNet proposed in [38, 33J, and it can gradually denoise z, by predicting the noisy estimation eœr (ze) |c, 1) as Gaussian noise e ~ NN I). In the paper, 0' 0, + 20 consists of the pretrained parameter 0, 20 (AW'). updated by LoRA [22, 11]. W Aw' e Raxb denote the pretrained weight and low-rank weight in the 1-th (1 (1= 1, .... L) transfromer layer of 6', respectively. a and b are the row and column of matrices. As introduced in [40, 68), A Awi A 'BB' can be decomposed as two low-rank factors A E Rax and B' E x where r < min (a, b) denotes the rank.
{W! oJ in LDMs [39, 33] and low-rank parameter
However, most CDMs [11, 12, 28] assume that the number of users' personalized concepts remains constant over time. This assumption is unrealistic in real-world applications, where users wish to consecutively generate series of new personalized concepts based on their preferences. More
3

---

importantly, they significantly suffer from catastrophic forgetting [36] of old personalized concepts and concept neglect when performing versatile customization concept-ineremeet
manner.
Problem Definition: To address the above challenges, we propose a new practical problem named Concept-Incremmenaal Flexible Customization (CIFC). In the CIFC, there is series of consecutive text- guided concept customization tasks T = (T g=g where G denotes the task quantity. According to users' personal preferences, the g-th task Tg {x
f p v. y which includes ng triplets of image xy. text prompt Dl. and its concept tokens y Eyg. belongs to one of the versatile customization categories: multi-concept generation [69]. style transfer [67] and image editing [3]. Here, P indicates the textual description of x (e.g., photo of a [V.] [VdogD: D, whereas y denotes the concept tokens (e.g.. [V.] [Vdog]) in p Y/ is concept space of the g-th task, and it comprises Cg new personalized concepts Ya- = - iy in the g-th task. Particularly, the concept spaces between any two tasks have no overlap: 3 7 (A) = 0. It implies that C new concepts in the g-th task are different from Σ C, old personalized concepts from g-1 old tasks under the CIFC setting, Considering the practicality of the CIFC setting, we don't it allocate any memory storage to store or replay the training data of all tasks {Ty)ş= ensuring that all concept customization tasks are learned in a manner. The CIFC setting can continually learn new personalized concepts in a concept-ineremen manner for versatile customization while tackling the forgetting on old concepts.
The Proposed Model
Fig. shows the diagram of our concept-ineremmentll diffusion model (CIDM) tackle the CIFC problem. It includes (a) a concept consolidation loss in Sec. 4.1 and (b) an elastic weight aggregation module in Sec. 4.2 to resolve catastrophic forgetting during training and inference. Additionally, encompasses (c) a context-comttrlllabe synthesis strategy in Sec. 4.3 to address concept neglect.
4.1 Concept Consolidation Loss
In order to learn the g-th text-guided concept customization task Tg. we use the LoRA [14, 38] to finetune the pretrained denoising UNet e,, (-) on personalized data {x P y g by optimizing Eq. (1), and then obtain an updated model coz (-), where 8 = Og + A0g. A0g = { W and AW, A, Raxb is the updated low-rank weight in the 1-th layer. A E Raxr and B E Rx are low-rank factors. As introduced in [14, 60], A0g can encode most of Cg personalized concept identity within the g-th task. To address the CIFC problem, trivial solution for learning the g-th task is to store the updated low-rank weights of all tasks [T.) learned so far, and then linearly merge them by evaluating their contributions [59, 65] during training. However, it may result in substantial loss of individual characteristics within some personalized concepts, when learning new concepts continually in the CIFC setting. This phenomenon is referred to as catastrophic forgetting on old personalized concepts. To mitigate catastrophic forgetting during training, as show in Fig. (a), develop concept consolidation loss LOCL explore task-specific and task- knowledge.
Task-Speeific Knowledge indicates distinctive characteristics of personalized concepts within each concept customization task. To explore this knowledge, we introduce learnable layer-wise concept tokens to better preserve unique attributes of personalized concepts the synthesized images. is significantly different from existing textual inversion methods [9. 67, 53] that inject a unified text prompt into all transformer layers of Eg, (-). For a triplet (xx), y in the g-th task, we define L laser-wise text prompts as ( ) where p has its own learnable layer-wise concept tokens y in the 1-th transformer layer. For example, given a textual description P (photo of a [V-] [V of image x5. its text prompt of the 1-th layer is defined as k.l (photo of a [V] D. and [V:] [Viog] indicates learnable concept tokens ykd in the (-th layer. After using P to initialize ( J 1 we ( encoded via the text encoder I'(-) into the 1-th layer of eg; (+). When we train eg; (-) via Eq. (1), the learnable layer-wise concept tokens can capture unique characteristics of old personalized concepts from different layers surmount catastrophic forgetting.
[V)
inject the textual embedding ch. (
However, the discriminative ability of task-specific knowledge to distinguish different personalized concepts can significantly deteriorate as the number of concept customization tasks gradually increases

---

under the CIFC settings. To tackle this issue. we devise an orthogonal subspace regularizer to constrain the low-rank weights of different customization tasks. It can enhance the discriminative ability of task-specific knowledge by ensuring the orthogonality of concept subspaces across different tasks. Given the low-rank weight A0g- ()W)) in the g-th task, A W A can be regarded as consisting of the low-rank concept subspace A = [a]]...... : a E Raxr and its linear weighting matrix B = [by] .... b
g
g
E grx6 where by² Ie'h denotes the linear weighting coefficients of a/² E Rd 11 .r). In the g-th task, we perform the orthogonal subspace regularizer on the low-rank concept subspaces of different tasks: >E A 0. Since the orthogonal constraint is not differentiable, we propose an alternative optimization strategy that minimizes the absolute value of the inner product between different subspaces: R1 Σi Σ AT(A
Task-Shared Knowledge represents the shared semantic information across different tasks with se- mantically similar concepts, which is beneficial to address catastrophic forgetting on old personalized concepts. To capture task-shared knowledge, W E Rx shared across different tasks in the (-th layer. Given the low-rank weights {40, 1 learned so far, the learnable projection matrix H E Raxa can encode common semantic information of { 1 into W via R₂ = Σ El l H'W 1 Therefore, in the g-th task, the concept consolidation loss LECL learn both task-specific and task-shared knowledge is defined as:
propose to learn a layer-wise common subspace
LECt = (x e-.. - eog (z, cg 1) + + R1 R2),
(2)
where ot = (c 1 indicates L layer-wise textual embeddings, 71 and 7/2 are balance parameters.
Two-Step Optimization: To train the proposed CIDM via Eq. (2) in the g-th task, we devise two-step optimization strategy in each training batch. Firstly, to capture task-specific information, utilize Eq. (2) to update the learnable layer-wise concept tokens and low-rank weight 40g {A w' when fixing H. and W. Secondly, to explore ahaeed knowledge. we fix 20, and the learnable layer-wise concept tokens, and then only use R2 in Eq. (2) update H/ and W! respectively. Notably, the detailed optimization procedure is shown in the appendix section (Sec. A. After utilizing the two-step optimization strategy to learn the g-th concept customization gg
task, we can obtain the task learner Z,- {A w y where y [ Ji=1, and y is the i-th (i = 1, Cg) concept token learned at the 1-th transformer layer through Eq. (2).
Cg
4.2 Elastic Weight Aggregation
To tackle catastrophic forgetting on old personalized concepts during inference, as shown in Fig. 1(b), we store g task learners { learned far (g 2 2), and develop an elastic weight aggregation (EWA) module to adaptively merge them for versatile concept customization. Note that the memory storage of storing Z,
..... 9) only accounts for 0.25% of the pretrained model 6g. making negligible in practical applications. Specifically, given a text prompt P for inference, we use it to initialize layer- wise text prompts (p') and extract their textual embeddings
(c' E R xd } via text encoder (()), where ne and d denote the token number and feature dimension, respectively. Given the stored task learners [ we can collect all concept tokens { } learned so far, where क' = U J E Rh consists of n, = >
1=1 C2 concept tokens learned in the 1-th layer. After I'(+) encodes {4' } 1 to latent embeddings {e' E "e Ji=1 we average those latent embeddings belonging to the same task to obtain new embeddings (ē' E Rexd]__ (g is number of tasks learned so far). Subsequently, we compute semantic relations MM E TR9 between el and ē', and use M to adaptively merge low-rank weights (A (AW)) of all learned tasks in the 1-th layer. Therefore, the merged low-rank weight A W in the 1-th transformer layer can be formulated as follows:
M = max (G))), AW' Aw'=Z' -'_ AW (MM),
(3)
where max(-) denotes the maximization function along the row axis. 6(M) = M /IMM E R used to normalize the semantic relations M, and (MM; the i-th element of (MM)
Inference: After employing Eq. (3) to merge all low-rank weights learned so far, we obtain a new
denoising UNet eng (.) for inference, where 0' = 0, + A0., and A0, = (A WW Notably, Ø has encoded substantial distinctive attributes of all personalized concepts learned so far, which can effectively mitigate the catastrophic forgetting of old concepts during inference.
5

---

Task 1: V1
Task 2: V2
Task
Task 4: V4
Task 5: v5
Task
Task V7
Task B: VB
Task 9:
Task 10: V10
EWC
LWF
CLoRA
L2DM
LoRA-M
LORA-C
Ours
V2 duck toy in time square
Sun rises among the mountains in the V6 style
V5 teddy bear skateboard
A wooden house in the V10 style
A child playing with V4 backpack
Figure 2: Some qualitative comparisons of single-concept customization generated by SD-1.5 [38].
4.3 Context-Controllabie Synthesis
When we directly use eØ: (-) obtained via Sec. 4.2 to perform multi-concept customization under the CIFC setting, it cannot generate high-fidelity images according to users-provided conditions (e.g., scribble or bounding box [24]), and heavily suffers from the challenge of concept neglect [1] (i.e., some concepts are missing in the synthesized images). Thus, as shown in Fig. I(c), we devise context-controllable synthesis strategy to address the conditional generation and concept neglect.
Besides the initial text prompt defined in Sec. 4.2, users can provide U pairs of region conditions {P... Sy ) umm1 where s, is the bounding box to synthesize concepts related to the u-th region text prompt P.. Then we use I(-) to extract layer-wiss wise textual embeddings C, = (2) E Rd.x 1/1 for P.. Given the initial text prompt P. we can use the new denoising UNet eor (-) in Sec. 4.2 to obtain its feature map I' gg' xw< xd in the I-th transformer layer encoded by textual embedding e', where hi and w are height and width of f!. Different from [22], we perform layer-wise regional cross-attention between textual embedding e', and f to obtain the u-th region feature E E A x xd in the 1-th layer, where hi, and w are height and width of bounding box S.. Specifically, f ao(((Q (
Conditional Generation:
V! where a(-) is sigmoid function, Q' (2(F' @ m TRA, xw., xd K! C We E R xd and Vl è W. xd m E TR xw" is the binary region mask in the (-th layer, where the values inside the bounding box are set 1. Ω(-) can retain only

---

Region Boxes
EWC
CLoRA
L2DM
LoRA-M
LoRA-C
Ours
dog
V3
cat
ITP: anime artwork of magic castle. RTP: V1 dog V3 cat wearing wizard hat and wizard cloak, anime style.
V1 dog
V7 dog
V3
cat
ITP: bedroom. RTP: dog sitting on bed dog cat V7 cat.
V7
V5 teddy bear
V9 cat
ITP: football field with audience background. RTP: V7 dog V5 teddy bear V9 cat wearing a medal.
V1 dog
vs
beddy
con
ITP: resplendent palace. RTP: dog wearing suit, sitting throne V5 teddy bear cat.
V3 cat
V1 dog
V9 dog
ITP: the surface moon fiction style. cat dog dog walking on the surface of moon.
Figure 3: Some qualitative comparisons of multi-concept customization generated by SDXL [33), where ITP indicates the initial text prompt, and RTP denotes the region text prompt.
the features inside Su. and C is the Hardmard product. Wg. Wk, We E Raxd are mapping matrices in the new denoising UNet eg: (.). Then, the values of f inside the bounding boxes ($u)e=1 are respectively replaced with the corresponding region features { V obtain
feature map f!
We apply the layer-wise regional cross-attention to all layers of eqr (-) for conditional generation.
Multi-Concept Composition: After integrating the above conditional generation into the new denoising UNet Egg (-), inspired by [13], we can obtain the noise estimation E' EØ: (zzlt) + s (eaz (2, . () eœ; (z:lk ()) E hg xwz xd2 for the initial text prompt P. where E' is the output of the L-th transformer layer in eØ: (.) at the t-th timestep. h1 WL, dp are the height, width and channel of E'. respectively. eØy (z,[t) is the unconditional noise estimation, eœ: (2;/C, t) is the conditional noise estimation based on the textual embeddings č, and 7.5 denotes the scale. Therefore, the noise estimation E,, E T xwz xdz for the u-th region condition {P., S.) can be formulated as follows:
Ex
EB; (z) + (ce: (z.lč.., Su).t) cØ: (z, (t),
where eœ: (2rl Su). t) is the conditional noise estimation based on (č., S.,. For multi-concept customization in the CIFC, we aggregate U' region noise estimations to address concept neglect:
E aE Σ -=1 (1 a)E m
(5)
7

---

where i E R XWE is the binary region mask of the bounding box Su in the L-th layer. is the balance weight. Following [38, 33]. we forward E to the denoising process for image synthesis.
5 Experiments
5.1 Experimental Setups
V2 Duck Toy
Benchmark Dataset: Motivated by [47,
11, 45], this paper, we construct a new challenging concept-incremental learning (CIL) dataset including ten continuous text- guided concept customization tasks to il-
Cat V7
lustrate the effectiveness of our model un- der the CIFC setting. In the CIL dataset, seven customization tasks have different object concepts ei.e., V1 dog. V2 duck toy,
V5 Tedkly Bear
V3 cat, V4 backpack, V5 teddy bear, V7
dog and V9 cat) from [40, 22], and the remaining three tasks have different style concepts (ie., V6, V8 and VIO styles) col- lected from website. Considering the prac- ticality of the CIFC setting, we set about 3 ~ 5 text-image pairs for each task. Par- ticularly, we introduce some semantically similar concepts (e.g., VI and V7 dogs, V3 and V9 cats), making the CIL dataset more challenging under the CIFC setting.
B4Backgack
Input Figure 4: Comparisons of custom image editing.
CLORA
L2DM
Ours
Implementation Details: We utilize two popular diffusion models: Stable Diffusion (SD-1.5) [38] and SDXL [33] as the pre-
Style
trained models to conduct comparison ex- periments. For fair comparisons, we train all SOTA comparison methods and our model using the same backbone and Adam optimizer, where the initial learning rate is
1.0 x 10- to update textual embeddings, and 1.0 x 10-4 to optimize the denoising UNet. For the low-rank matrices, we fol- 4. We empirically set ?1 = 011, 72 = 1.0 in Eq. (2), a 0.1
low [11] to
V10 Style
Eq. (5), and the training steps are 800.
Evaluation Metrics: After learning the fi- nal concept customization task under the CIFC setting, we conduct both the qualita- tive and quantitative evaluations versa- tile generation tasks: single/multi-cone concept customization, custom image editing. and custom style transfer. For the quantita- tive evaluation, we follow [22] alignment (TA) and image-alignment (IA) as metrics. Specifically, for image-alignment (IA), we use the image encoder of CLIP [34] to evaluate the feature similarity between the synthesized image and original sample. For text-alignment (TA), we utilize the text encoder of CLIP [34] to compute the text-image similarity between the synthesized image and its corresponding prompt.
V6 Style
Ours Figure 5: Comparisons of custom style transfer.
Input
CLORA
L2DM
text-
5.2 Qualitative Comparisons
To verify the superiority of our model under the CIFC setting, we introduce extensive qualitative comparisons, including single/multi-concopl customization (see Figs. 2-3), custom image editing
8

---

Table 1: Comparisons (IA) of single-concept customization synthesized by SD-1.5 and SDXL.
SD-1,5 [38]
SDXL [33]
Methods
V1 V2
V3
V4
V5
V6
V7
V8
V9
V10l
Avg.
V1
V2
V3
V4
V5
V6
V7
V8
V9
V10
Avg.
Finetuning
77.6 79.0 77.6 79.6 62.9 71.5 53.7
81.4 72.1
73.7 62.0
70.8
79.1 73.4 76.4 67.5 76.8 57.4
77.1 74.8
71.5
EWC [20]
78.7 83.8 80.4 80.3 80.7 64.0 76.5 57.1
84.4 73.1
75.9
83.6
80.5 84.6 80.8 79.2 70.1 80.5 61.2
75.8
77.6
LWF [25]
80.4 79.7 80.9 77.4 80.9 61.8 73.2 53.5
78.1
74.7
74.1
84.0 84.2 81.7 79.7 68.1 77.1 60.1
76.3
76.5
LoRA-M [69]
80.0 84.2
65.7 70.1
79.5
74.1
74.6
82.6
84.5
80.9
74.3
LoRA-C [69]
80.1 84.1 79.8 76.6 82.9 65.9 70.8 54.9 79.9
74.4
74.9
82.8
80.4
84.8 80.0 81.0 58.2 76.8 54.5 72.2 73.9
74.5
CLoRA [45]
83.2 83.4 81.1 80.6 84.9 66.3 76,2 58,1
76.9
83,481.3
85.8 80.1 79.0 70.4 81.2 61.7
78.5 76.7
77.8
L2DM [47]
78.7 86.3 76.6 80.7 86.8 70.8 70.0 89.3
74.1
76.1
8846
79.5
81.9 75.5 82.1 69.2 80.9 63.8
77.0
76.4
77.1
CIDM (Ours) 83.6 86.4 82.9 80.8 86.5 69.5 73.7 56.9 82.4 75.9
78.0
(87.1 82.1 88.5 84.9 85.8 68.3 82.0 62.4 76.9 76.6
79.5
Table 2: Comparisons (TA) of single-concept customiiiaia synthesized by SD-1.5 and SDXL.
SD-1.5 [38]
SDXL [33]
Methods
V1
V2
V3
V4
V5
V6
V7
V8
V9
V10
Avg.
V1
V2
V3
V4
V5
V6
V7
V8
V9 V10
Avg.
Finetuning
64.4 74.6 69.4 68.6
75.0 70.0 76.7 69.2 65.4 67.2
70.0
54.8
77.5
72.2
85.0
80.5 76.2 79.7 73.6 77.6 76.3
75.3
EWC [20]
77.5 72.7 77.9
72.0
70.4
72.7
71.4
72.8 84.4 79.5 73.9
77.6
77.1
IWF [25]
70.8 70.8 71.0........11
71.7 76.3 72.9 72.5 70.0
73.4
75.8
76.0) 83.6 82.9 75.1 76.7 74.3 79.1
76.8
77.7
RPY [25]
68.1 76.2 70.1 78.4 75.7 69.3 74.8 70.5 65.8 68.6
71.8
69.3
81.0 87.9
87.3 78.8 71.5 76.4 75.9
79.7 76.2
76.8
CLoRA [45]
69.4 78.0 74.1 78.8 76.4 69.6 76.7 73.9
71.8
73.6
71.8
80.1 71.1 87.7 81.2 74.6 77,8 77.7 80.1 75.9
77.8
L2DM [47]
68.6 79.5 70.1 73.0 76.7 67.7 75.9 74.1
71.8
69.4
72.7
78.6
78.5 85.0 81.5 73.5 78.6 79.1
81.9
77.8
78.7
CIDM (Ours)
75.3 78.1 74.0 81.1 78.2 70.1 74.7 74.3 73.5
70.2
74.8
74.9
79.6
74.5 86.7 83.5 79.8 78.2 83.1 81.4 78.5
80.0
(see Fig. 4), and custom style transfer (see Fig. 5). As presented in Fig. 2, the proposed model achieves the best performance for single-concept customization by addressing catastrophic forgetting of old personalized concepts and preserving superior attributes of each learned concept. 2) For multi-concept customization, we incorporate the regionally controllable sampling module proposed in [11] into existing comparison methods to fairly compare them with our model. shown in Fig. 3, all comparison methods significantly suffer from the challenge of concept neglect, and they are difficult to generate multiple object concepts. contrast, our proposed model can effectively tackle the challenge of concept neglect via the proposed context-controllable synthesis strategy. 3) To perform custom image editing, we introduce Anydoor [3] as a plug-in for all comparison methods in Fig. 4. The qualitative comparisons in Fig. 4 reflect the effectiveness of our model in custom image editing. Such substantial improvement benefits from the superior performance of our concept consolidation loss to preserve unique identity of each learned concept under the CIFC setting, 4) Fig. 5 shows qualitative comparisons of custom style transfer. Our proposed model presents the best performance for image-to-iman integrity in the CIFC setting. The above qualitative comparisons in Figs. 2-5 verify that the proposed model can significantly surpass existing latent diffusion models to tackle the CIFC problem.
style transfer, since the elastic weight aggregation can preserve style
5.3 Quantitative Comparisons
Table Ablation studies of single-concept customization.
To analyze quantitative comparisons between our model and SOTA meth-
Variants
TSP TSH EWA
VI-V5
V6-V10 Avg.
Baseline
80.4
68.8
74.6
ods, follow [11, 22, 45, 47] to introduce 20 evaluation prompts for each concept and generate 50 images for each evaluation prompt, resulting in total of 1,000 synthesized images. CIDM (Ours)
Baseline w/ EWA
83.7
70.8
77.3
Ours w/o TSP Ours w/o TSH
/
/
83.7
71.0
77.4
/
83.9
71.4
77.7
1
84.0
71.7
77.9
Then quantitative evaluation is con- ducted on these 1,000 images. As shown in Tabs. 1 1-2, we can observe that our CIDM outperforms all comparison methods by 1.1% ~ 8.0% in terms of image-alignment (IA) and 1.2% ~ 4.8% in terms of text-alignment (TA). It illustrates that our model can effectively preserve unique identity of each learned concept to tackle the CIFC problem. Compared to other SOTA methods, our proposed model achieves better performance in mitigating catastrophic forgetting, as the introduced concept consolidation loss captures task-specific inwiifiiiinnnnnn information and task-shared knowledge through two-step

---

V7
V3 calt
dog
dog
V1 dog
VS beddyy
Input
Baseline
Figure 6: Ablation analysis of CSS in multi-concept customization.
optimization during training. Moreover, the elastic weight aggregation module proposed in this paper effectively merges previous personalized concepts for customization during inference.
5.4
Ablation Studies
This subsection analyzes the effective- ness of each module in our model: elas- tic weight aggregation (EWA), context- controllable synthesis (CCS), task-specific knowledge (TSP) and task-shared knowl- edge (TSH) the concept consolidation loss (CCL). Tab. 3 presents the ablation studies of single-concept customization terms of IA. When compared with Baseline, the performance of our model improves by 0.2% 3.3% in terms of IA, after we add the proposed TSP, TSH and EWA mod- ules. It demonstrates the effectiveness of our model in resolving the CIFC problem by addressing the catastrophic forgetting and concept neglect. As shown in Fig. 6, analyze the effectiveness of CCS in multi-concept customization, where CCS includes the layer- wise regional cross- cross-attentes attention (RCA) and the region noise estimation (RNE) mod- ules. In Fig. 6, the performance of our model decreases substantially when removing the RCA and RNE modules. It verifies the effectiveness of our CCS startegy in addressing conditional generation and concept neglect. Moreover, Fig. 7 shows ablation analysis about TSP and TSH. It illustrates that model can capture task-specific information within each customization task and explore task-shared knowledge across different tasks to tackle the CIFC problem via optimizing Eq. (2).
V9 on the chair
V7 dog
forest
teddy rainbow colors
Input
Baseline
Baseline W/ TSI
Baseline
W/S5I TSK Figure 7: Ablation studies of the TSP and TSH.
Conclusion
In this paper, we propose a novel Concept-Incremeental text-to-image Diffusion Model (CIDM) address a practical ceccpptanncemmentall Flexible Customization (CIFC) problem, where the CIFC problem has two major challenges: catastrophic forgetting and concept neglect. Specifically, we devise a concept consolidation loss and an clastic weight aggregation module to respectively resolve catastrophic forgetting during training and inference. They can capture aask-ppeciffc/task-shae knowledge and aggregate all low-rank weights of old concepts according to their contributions in the CIFC. To address concept neglect, we propose a new context-controllable synthesis strategy, which can control the contexts of synthesized images according to users provided conditions. Extensive experiments on versatile customization tasks (single/multi-coneep) customization, custom image editing and style transfer) show the superior performance of our CIDM in tackling the CIFC problem compared to SOTA methods. the future, we will leverage multi-modal large language models to address the CIFC problem and apply the proposed model to personalized video generation.
10

---

References
[1] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention- based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics, 42(4), jul 2023.
[2] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 30286-30305. Curran Associates, Inc., 2023.
[3] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. arxiv preprint 03xiv:200,00453 2023.
[4] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, (0990)10850-1866) 2023.
[5] Jiahua Dong, Hongliu Li, Yang Cong, Gan Sun, Yulun Zhang, and Luc Van Gool. No left behind: Real-world federated dass-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4))2055-200) 20700, 2024.
[6] Jiahua Dong, Lixu Wang, Zhen Fang. Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated class- incremental learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2022.
[7] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[8] Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffusion for accurate instruction following. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, June 2024.
[9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
[10] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang. Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, June 2022.
[11] Yuchao Gu. Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Chen Yunpeng. Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang. Weijia Wu, Yixiao Ge. Shan Ying. and Mike Zheng Shou. Mix-of-show: Decentralized low- rank adaptation for multi-concept customization of diffusion models. In Advances in Neural Information Processing Systems, 2023.
[12] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff:: Compact parameter space for diffusion fine-tuning. In 2023 IEEE/CVF International Conference on Computer Vision, pages 7289-7300, 2023.
[13] Jonathan Ho and Tim Salimans. Classifier-fiee free diffussonn guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[14] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[15] Steven C. Hung, Cheng Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 13677-13687 2019.
[16] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors. Computer Vision ECCV 2020, pages 699-715, 2020.
[17] Phillip Isola, Jun-Yan adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 5967-5976, 2017.
Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional
[18] Xuhui Jia, Yang Zhao, Kelvin C. K. Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arxiv preprint arxiv:2304. 23dd:2304.02642, 2023.
[19] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. In Advances in Neural Information Processing Systems, 2022.
11

---

[20] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic lorgetting neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3522 2017.
Grabska-Barwinska,
[21] Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang. Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. Omg: Declusion-friendly personalized multi-concept generation in diffusion models, arxiv preprint arxiv:2403. 10983, 2024.
[22] Nupur Kumari, Bingliang Zhang, Richard Zhang. Eli Shechtman, and Jun- Yan Zhu. Molti-concept customization of text-to-image diffusion. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- ure-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages
12900. PMLR, 2022.
[24] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22511 1-2222,,,, June 2023.
[25] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40( (0(11))2935-22) 2947, 2017.
[26] Chang Liu, Xiangtai Li, and Henghui Ding. Referring image editing: Object-level image editing via referring expressions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13128-3338, 2024.
[27] Zhiheng Liu, Ruili Feng. Kai Zhu, Yifei Zhang, Kecheng Zheng. Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: concept neurons in diffusion models for customized generation. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[28] Zhiheng Liu, Yifei Zhang. Yujun Shen, Kecheng Zheng. Kai Zhu, Ruili Feng. Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Customizable image synthesis with multiple subjects. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[29] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In 2023 IEEE/CVF Conference Computer Vision and Puttern Recognition, 2023.
[30] Chong Mou, Xintao Wang, Liangbin Xie. Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI Conference on Artificial Intelligence, 2024.
[31] Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. DiffuseVAE, Efficient, con- trollable and high-fidelity generation from low-dimensional latents. Transactions Machine Learning Research, 2022.
[32] Tacsung Park, Ming- Yu Liu, Ting- Chun Wang, and Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2332-2341, 2019.
[33] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, el al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR,
2021.
[35] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical fext-conditional image generation with clip latents. arxiv preprint 22x4v:204..0 2022.
[36] Sylvestre-Alvis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incre- mental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 5533-5542, 2017.
[37] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lec. Generative adversarial text to image synthesis. In Proceedings of The 33rd International Conference on Machine Learning, volume 48, pages 1060-1069, Jun 2016.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages
2022.
12

---

[39] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015.
pages 234-241, 2015.
[40]
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[41] Chitwan Saharia, William Chan, Saurabh Saxcna, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphacl Gontijo-Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image difrusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
[42] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arxiv preprint arxiv:2304.034111 2023.
[43] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems. page 2994-3003, 2017.
[44] Xincheng Shuai, Henghui Ding, Xingjun Ma, Rongcheng Tu, Yu-Gang Jiang, and Dacheng Tao. A survey of multimodal-guidel image editing with text-to-image diffusion models. arxiv preprint 24x.v:2406.14533 2024.
[45] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-Iora. Transactions on Machine Learning Research, 2024.
[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[47] Gan Sun, Wengi Liang. Jiahua Dong, Jun Li, Zhengming Ding, and Yang Cong. Create your world: Lifelong text-to-image diffusion. JEEE Transactions on Pottern Analysis and Machine Intelligence, 46(9):6454-6700 2024.
[48] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
14453-14463, 2023.
[49] Yujin Tang, Yifan Peng, and Weishi Zheng. Learning imagine: Diversify memory for incremental learning using unlaheled data, 2022 IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 33339-9548, 2022.
[50] Jishang Tu, Hao Fu, Fengyu Yang. Hanbin Zhao, Chao Zhang, and Hui Qian. Texttoucher: Fine-grained text-to-touch generation. arXiv preprint arXiv:2409.05422 2024.
[5I] Jiahang Tu, Wei Ji, Hanbin Zhao, Chao Zhang, Roger Zimmermann, and Hui Qian. Driveditfit: Fine-tuning diffusion transformers for autonomous driving. arXiv preprint arXiv:2407.15660 2024.
[52] Thanh Van Le. Hao Phung. Thuan Hoang Nguyen, Quan Dao, Ngoc N Tran, and Anh Tran. Anti- dreambooth: Protecting users from personalized text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2116-2127, 2023.
[53] Andrey Voynov, Qinghao Chu, Daniel Cohen Or, and Kfir Aberman. P+: Extended textual conditioning in text-to-image generation. arxiv preprint arxiv: 09552,, 2023.
[54] Boyang Wang, Fengyu Yang. Xihang Yu, Chao Zhang, and Hanbin Zhao. Apisr: Anime production inspired real-world anime super-resolution. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 25574-25584, 2024.
[55] Xiao Wang, Tianze Chen, Qiming Gc, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[56] Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In 2023 IEEF/CVF International Conference on Computer Vision, pages 7643-7655,
2023.
[57] Yuxiang Wei, Yabo Zhang. Zhilong Ji. Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943-15953, October 2023.
[58] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning eet--o--mage models with human preference. 2023 IEEE/CVF International Conference on Computer Vision, 2023.
[59] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of loRA experts. In The Twelfth International Conference Learning Representations, 2024.
13

---

[60]
Yang Yang, Wen Wang, Liang Peng. Chaotian Song. Yao Chen, Hengjia Li, Xiaolong Yang, Qinglin Lu, Deng Cai, Boxi Wu, and Wei Liu. Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models. arxiv preprint arxiv:2403. 2024.
[61] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference Learning Representations, 2018.
[62] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfci Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
[63] Duzhen Zhang, Wei Cong, Jiahua Dong, Yahan Yu, Xiuyi Chen, Yonggang Zhang, and Zhen Fang. Continual named entity recognition without catastrophic forgetting. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 8186-8197, 2023.
[64] Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. MM-LLMs: Recent advances in MultiModal large language models. In Findings of the Association for Computational Linguisticy ACL 2024. pages 12401-12430, August 2024.
[65] Jinghan Zhang. Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operation. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023.
[66] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In 2023 IEEE/CVF International Conference on Computer Vision, pages 3813-3824, 2023.
[67] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng
Xu. Inversion-bascd style transfer with diffusion models. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10156, June 2023.
[68] Rui Zhao, Yuchao Gu, Jay Zhangjic Wu, David Junhao Zhang. Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. arxiv preprint arxiv:2310.08465 2023.
[69] Ming Zhong, Yclong Shen, Shuohang Wang. Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arxiy preprint arxiv:2402.16843: 2024.
14

---

A Appendix
A.1 Optimization pipeline
As shown in Algorithm 1, we introduce the detailed algorithm pipeline of our proposed CIDM to tackle the CIFC problem. For the first text-guided concept customization task (g 1) in the training process, we only utilize Eq. (1) to optimize the textual embeddings of layer-wise concept tokens and low-rank weight 40g = {& W When g 2 we devise a two-step optimization strategy in each training batch to efficiently train our CIDM via Eq. (2). It can continually learn new personalized concepts under the CIFC setting, while overcoming catastrophic forgetting of old personalized concepts by exploring both task- and task-shared knowledge during training.
1) In the first step, we fix Hi and W to update learnable layer-wise concept tokens and low-rank weight 40, (4W g/i=1 via minimizing the loss LCCL in Eq. (2). Particularly, the loss LOCL can capture task-specific knowledge (i.e. unique concept attributes within each customization task).
2) In the second step, in order to explore task-shared knowledge across different tasks, we fix the learnable layer-wise concept tokens and low-rand weight 28g to update H' and W via minimizing the regulaizer R2 = Ei l - H w. p. As a result, the gradients of R₂ with respect to H and W are formulated as follows:
0R2 2(AW: H(W OH:
2R₂ (()) H(W.).
(6)
ow.
Given the gradients 0R. /H and 0R./OW in Eq. (6), we can iteratively update H and W via the following objective:
0R2
RR2
W = W +
HH
Ow!
where 77 denotes the learning rate update H and W In this paper, set the value of r7 to be the same as the learning rate the Adam optimizer.
A.2 Experiment Setups
This subsection introduces more details about dataset, comparison methods and evaluation metrics.
Inspired by [47, 11, 45], we construct new challenging concept-incremental learning (CIL) dataset including ten continuous text-guided concept customization tasks to verify the effectiveness of our model under the CIFC setting. In the CIL dataset, as shown in Fig. 8, seven tasks have different object concepts (i.e., V1 dog, V2 duck toy, V3 cat, V4 backpack, V5 teddy bear, V7 dog and V9 cat) from [40, 22], and the remaining three tasks have different style concepts (i.e., V6, V8 and V10 styles) collected from website. Considering the practicality of the CIFC setting, we set about 3 ~ 5 text-image pairs for each task, where the text prompts are extracted by BLIP [23]. Particularly, different from L2DM [47] and Mix-of-Show [11], we introduce some semantically similar concepts (e.g., V1 and V7 dogs, V3 and V9 cats), making the CIL dataset more challenging the CIFC setting.
Dataset:
Task V1
Task 2: V2
Task 3: V3
Task 4: V4
Task 5: V5
Task 6: V6
Task 7: V7
Task V8
Task 9: V9
Task 10: V10
Figure 8: Visualization of some examples in ten consecutive concept customization tasks.
15

---

Prompts for Pets
Prompts for Objects
Prompts for Styles
1. A [V'] in the swimming pool
1. A [V*] in the ocean
A [V*] of tree, mountain and moon
2. A [V*] in front of Eiffel tower
2. A [V'] near the Eiffel tower
2. A [V*] of wooden house
3. A [V'] near the mount fuji
3. A [V*] near the mount fuji
A [V"] of a bridge over the river with
4. [V*] the forest
A [V*] in the forest
mountain background
[V*] walking the street
5. A [V*] in times square
A [V"] of waterfalls appearing among mountains
[V'] cyberpunk 2077, 4K,
A [V'] cyberpunk 2077, 4K, 3d
3d render in unreal engine
render unreal engine
A [V"] of sun rises among mountains
7. A watercolor painting of a [V*]
A watercolor painting of [V']
A [V"] of flowers, grass and river
8. A painting of [V'] in the style
A painting of [V'] the style of Vincent Van Gogh
A [V*] of boat on lake
of Vincent Van Gogh
A [V"] of dragon with the cloud
9. A painting of [V'] in the style of Claude Monet
A painting of a [V*] in the style
background
A [V"] of cliff with sky and moon
of Claude Monet
background
10. A [V'] in the style of Pixel Art
10. A [V*] in the style of Pixel Art
10. A [V"] of mountains, moon and clouds
11. A [V*] sit on the chair
11. A [V'] on a table
11. A [V"] of smile girl
12. A [V*] on the boat
12. A [V"] on a chair
12. A [V"] of dog
13. A [V"] wearing a headphone
13. A [V*] on a skateboard
13. A [V"] of cat
[V"] wearing a sunglass
14. A child is playing a [V']
14. A [V*] of snow, tree and clouds
[V'] playing with ball
15. A [V"] on a carpet
15. A [V'] of panda
16. A sad [V*]
17. A close view [V*]
16. A [V'] in the style of Vincent Van Gogh
17. An angry [V*]
16. A top view of [V']
17. A [V"] in the style of Claude Monet
18. A running [V*]
18. A [V'] in rainbow colors
18. watercolor painting of [V"]
19. A jumping [V*]
19. A fallen [V*]
19. A [V'] in the style of Pixel Art
20. A [V*] is lying down
20. A broken [V*]
20. A [V'] cyberpunk 2077, 4K,
Figure 9: Descriptions of text prompts used in this paper.
Algorithm 1: Algorithm Pipeline of The Proposed CIDM.
Initialize: The pretrained demoising UNet co))). a sequence of text-guided concept customization tasks T= 1G1 1 and the pretrained CLIP text encoder I'());
b Training for The g-th Task:
Initialize: Low-rank weight 20,- = (AWI
for (xl. p y)) in To do
Initialize the layer-wise text prompts (pr Haii A.l Obtain the layer-wise textual embeddings cg via T(p)"));
if 2 2 then
Update the learnable layer-wise concept tokens and 20, via Eq. (2) when fixing Hi and Wi: Update H and W via Eq. (7) when fixing the learnable layer-wise concept tokens and A0y;
clse
Update the learnable layer- concept tokens and 20, via Eq. (I);
Return:
Task learner Zy ww y Yo of the g-th concept customization task Ty-
& Inference:
Initialize: All learned task leamner ( [2.]g- initial text prompt P and region conditions (P... S.. J
for l in (1,2..., L} do
Update the low-rank weights A W via Eq. (3) to obtain a new denoising UNet egg (-); for in (T,
................................ 1) do
for (P... S.. in [p.. | do for l in {1, 21,...., L) do
Obtain the w-th region feature C. Replace the values of I' inside the bounding box S, with C.:
Obtain the u-th region noise estimation E via Eq. (4);
Use E. in Eq. (5) to update the noisy latent feature from 2, to 2:-1i
Return: The generated image x = D(21).
Comparison Methods: To comprehensively demonstrate the superior performance of our model, we introduce eight SOTA comparison methods, including four continual learning-basse methods (i.e., Finetuning, EWC [20], LWF [25] and RPY [25]), two continual diffusion models (i.e., CLoRA [45] and L2DM [47]), and two multi-LoRA composition methods (LoRA-M [69] and LoRA-C [69]). Specifically, Finetuning aims to optimize LoRA layers to learn multiple personalized concepts in concept-inncemeental manner. To address catastrophic forgetting, EWC [20] applies an elastic regularizer to penalize the network parameters. Besides, LWF [25] stores the old training data to perform knowledge distillation on the current task, while RPY [25] replays the old concepts [20] to
16

---

Task 1: V1
Task 2: V2
Task & V3
Task V4
Task 5: V5
Task & V6
Task 7: V7
Task & V8
Task 9: VS
Task 10: V10
EWC
LWF
CLoRA
L2DM
LoRA-M
LoRA-C
Ours
V3 cat, cyberpunk 2077, 4K, render in unreal engine
V5 tedyy bear in rainbow colors
watercolor painting V9 cat
V1 dog walking on the street
A V7 dog wearing sunglass
V2 duck toy in the forest
V1 dog in the style of Pixel Art
Figure 10: Some qualitative comparisons of single-concept customization generated by SD-1.5 [38].
tackle the CIFC setting, CLoRA [45] proposes a self-regularized low-rank adaption to continually learn new personalized concepts. L2DM [47] builds long-term memory bank reconstruct old personalized concepts, and performs knowledge distillation to mitigate catastrophic forgetting. LoRA- M [69] equally amalgamates all LoRA layers retrain the latent diffusion model. LoRA-C [69] focuses on exploring contributions of different LoRA layers for multi multi-concept composition.
After learning the final concept customization task under the CIFC setting, we conduct both the qualitative and quantitative evaluations on versatile generation tasks: single/multi- concept customization, custom image editing, and custom style transfer. For the quantitative evalua-
Evaluation Metrics:
17

---

Task Task
Task
Task
EWC
LWF
CLoRA
L2DM
LoRA-M
dog riding a bike, wearing wizard hat
V4 backpack, cyberpunk 2077, 4K, render in unreal engine
V9 cat walking on sea, anime style
V8 style panda is climbing
style of wooden house with mountain background
Figure 11: Some qualitative comparisons single-concept customization generated by SDXL [33].
tion, follow [22] text-alignment (TA) and image-alignmet (IA) as metrics. Specifically, for image-aligee aligmment (IA), we use the image encoder of CLIP [34] to evaluate the feature similarity between the synthesized image and original sample. For text-alignment (TA), utilize the text encoder of CLIP [34] to compute the text-image similarity between the synthesized image and its corresponding prompt. As shown in Fig. 9, roughly categorize prompts into three categories according to the learned concepts. To analyze quantitative comparisons between our model and SOTA methods, inspired by [11, 22, 45, 47], input an evaluation prompt and unify negative prompt to sample 50 images. Therefore, given 20 evaluation prompts in this paper, we can generate 1,000 images for each concept evaluate performance in terms of TA and IA.
A.3 Implementation Details
In this paper, use two popular diffusion models: Stable Diffusion (SD-1.5) [38] and SDXL [33] as the pretrained models conduct comparison experiments. For fair comparisons, we train all SOTA comparison methods and our model using the same diffusion model and Adam optimizer, where
18

---

Task V1
Task 2: V2
Task 3: V3
Task V4
Task 5: vs
Task 6: V6
Task 7: V7
Task B: VB
Task 9: V9
Task 10: V10
Region Boxes
EWC
CLoRA
L2DM
LoRA-M
LoRA-C
Ours
V7 dog
V1 dog
V3 cat
ITP: a street, apanese comic style. RTP: dog v7 dog cat walking on street.
dog
V7 dog
ITP: a bar, neon, sleek, ultramodem. RTP: v1 dog v7 dog drinking cup of beer.
V1 dog
V5 teddy bear
ITP: street, anime style. RTP: v1 dog V5 teddy bear riding bike the street.
dog
V9
dog
cat
cat
ITP: bedroom. RTP: V1 dog sitting bed V7 dog V3 cat V9 cat.
V4
V5
back- pack
teddy bear
'2 duck
ITP: grass with mountain background. RTP: backpack teddy bear sitting duck toy
V1 dog
V3
ITP: anime artwork of a magic castle. RTP: dog V3 cat wearing wizard hat, anime style.
Figure 12: Some qualitative comparisons of multi-concept customization generated by SDXL [33], where ITP indicates the initial text prompt, and RTP denotes the region text prompt.
the initial learning rate is 1.0 10-3 3 to update textual embeddings, and 1.0 x 10-4 to optimize the denoising UNet. For the low-rank matrices, we follow [11] to set r = 4. Morcover, we empirically set set = 0.1, 72 1.0 in Eq. (2), 0.1 in Eq. (5), and the training steps are 800. In this paper, train our model on two NVIDIA RTX 4090 GPUs. each text-guided concept customization task, finetune both the textual embeddings of layer-wist concept tokens and low-rank weights via the proposed two-step optimization strategy in Sec. A.1. For example, given a textual description P (photo of a [V.] [Vdog]) of image x²: in the g-th task, its text prompt of the I-th layer is defined as A (photo of [V1] D. After using [V.] [Vlog] initialize [V1] Wog in the I-th transformer layer, we only update the layer-wise concept tokens (e.g., [V]] [V D. For fair comparisons in CIFC setting, all comparison methods and our model use the same prompt augmentation and image augmentation during the training phase. Additionally, we set the random seed as 0 for all comparison experiments. More importantly, we have no prior knowledge about task order, task quantity and data distributions in the CIFC problem.
19

---

Target Style
Input
CLoRA
L2DM
LORA-M
LORA-C
Ours
V6 Style
V10
V8 Style
V6 Style
v8 Style
Figure 13: Some qualitative comparisons of custom style transfer under the CIFC setting.
A.4 More Qualitative Comparisons
Single-Concept Customization: As presented in Figs. 10-11, we conduct extensive qualitative comparisons of single-concept customization, when using Stable-Diffusion (SD-1.5) [38] and SDXL [33] as the pretrained denoising UNet. In Fig. 11, given a text prompt (e.g.. "VI dog riding a bike, wearing a wizard hat")) for inference, we observe that our model can effectively follow the prompt instruction to synthesize images that preserve the superior identity of the personalized concept "VI dog". In contrast, existing comparison methods suffer from significant catastrophic forgetting, generating unpleasant images with distorted objects and unmatched identity.
Customization: Fig. 12 presents comprehensive comparison experiments of multi- concept customization. For fair comparisons with existing methods, we apply the region-aware cross-attention module proposed in Mix-of-Show [11] into all comparison methods. The quanlitative comparisons in Fig. 12 demonstrate the superior performance of our model in multi-concept cus- tomization, when continually learning new personalized concepts under the CIFC setting. Particularly, given an initial text prompt (TTP) and some region text prompts (RTP) with users- prors-provided bounding boxes, all comparison methods exhibit significant concept neglect and they are difficult to perform multi-concept customization. On the one hand, it is evident that all comparison methods fail to preserve the identity of learned concepts due to catastrophic forgetting. On the other hand, these methods neglect some important object concepts during the synthesis process. For example, users may want to generate an image of V1 dog and V7 dog drinking cup of beer, but none of the comparison methods can generate the V1 and V7 dogs, as shown in Fig. 12.
As shown in Fig. 13, we introduce some comparisons of custom style transfer to evaluate the effectiveness of our proposed CIDM addressing the CIFC problem. From Fig. 13, we conclude that our model achieves the best generation performance according to user- provided style concepts, compared to other existing methods. It verifies that the designed elastic weight aggregation module plays crucial role in preserving distinctive style attributes in the CIFC setting. In contrast, existing comparison methods struggle to explore the identity of different style concepts, due to the catastrophic forgetting and concept neglect in the CIFC setting.
Custom Style Transfer:
20

---

Table 4: Ablation studies (IA) of single-concept customization generated by SD-1.5 [38).
Variants
TSP TSH EWA
V1
V2
V3
V4
V5
V6
V7
V8
V9
V10
Avg.
Baseline
80.0
84.2
79.1
76.5
82.7
65.7
70.1
54.7
79.5
74.1
74.6
Baseline w EWA
82.7
85.8
81.9
81,4
86,5
68.2
72.8
56.1
81.8
75.2
77.3
Ours w/o TSP
82.9
85.9
81.8
81.5
86.3
68.5
72.6
56.7
82.2
75.2
77.4
Ours w/o TSH
83.3
86.3
82.9
80.6
86.6
69.4
73.2
56.7
82.3
75.6
77.7
CIDM (Ours)
83.6
86.4
82.9
80.8
86.5
69.6
73.7
57.0
82.5
75.9
77.9
A.5
More Ablation Studies
This subsection analyzes the effectiivness of each module in our model: elastic weight aggregation (EWA), task-specifie knowledge (TSP) and task-shared knowledge (TSH) in the concept consolidation loss (CCL). As shown in Tab. 4, we introduce some quantitative ablation studies (IA) of the single- concept customization generated by SD-1.5 [38]. As can be seen from Tab. 4, our model significantly outperforms other variants by 0.2% ~ 3.3% in terms of image-alignment (IA). These ablation experiments validate that our model can effectively explore task-shared knowledge across different customization tasks to tackle the CIFC problem during training. Moreover, also verifies that the proposed elastic weight aggregation module is effective to address catastrophic forgetting during inference by aggregating different low-rank weighs according to their contributions.
A.6 Societal Impact and Limitations
Societal Impact: To tackle the practical imncept-incremental flexible customization (CIFC) prob- lem, our proposed concept-incremental text-to-image diffusion model (CIDM) aims to continually learn new personalized concepts in concept-incremmenall manner for versatile customization (e.g.- single/multi-concept customization, custom image editing and custom style transfer), while tackling catastrophic forgetting and concept neglect on old concepts. In particular, it allows users to continu- ally generate a sequence images using their new personalized concepts. Additionally, users can control the contexts of synthesized images according their own preferences.
Generally, the proposed CIFC setting can enable the creation of highly personalized content for various applications such as marketing, entertainment, and education. This can lead to more engaging and relevant experiences for users. More importantly, artists, designers, and content creators can benefit from tools that adapt to their unique styles and preferences over time. This can foster innovation and creativity by providing customized suggestions and automating repetitive tasks. Additionally, it can be tailored to meet the needs of different user groups, including those with disabilities. For example, generating personalized educational materials can cater to diverse learning styles and needs. Consequently, the CIFC problem proposed In this paper is worth being studied. More importantly. our proposed CIDM can achieve state-of offtthe-rt performance addressing the CIFC problem, highlighting the importance of this work in promoting the development of text-to- image diffusion models. In the CIFC, the use of personalized data to train our model may raise privacy issues. However, this is common concern for all latent diffusion models (LDMss). Ensuring that user data is handled securely and ethically is paramount to prevent misuse or unauthorized access, safeguarding both privacy and trust.
Although we construct a concept-incremental dataset containing four semantically similar concepts to verify the effectiveness of our model, it will still suffer from catastrophic forgetting and concept neglect as the number of semantically similar concepts increases. The main limitation of this work is that model struggle to continually learn large-scale semantically similar concepts provided by users. Thus, we will explore how to increase the scalability of our model in the future.
Limitations:
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We accurately claim our contributions in the abstract and introduction (see the main paper).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Sec. A.6 for the limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide more implementation details Sec. A.3 to reproduce the main experimental results of this paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: Please check the source codes at htttps://githun com/ cow//iiahaaDong/OIre reproduce the main results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: gwwggeuiee////// /Cooeeubeiiiieeee wwwiisssssosioiiiieeeioo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Sec. 5 and Sec. A.3, we introduce experimental setups and implementation details about all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Sec. 5, we introduce extensive experiments to support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information about the computer resources in Sec. A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https /tttpp:://euuriie http///nuurrp..peeee wwtppuuccc...hccsssieeeelllelele
Answer: [Yes]
Justification:
This paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please check Sec. A.6 for both potential positive societal impacts and negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: Please check the source codes at https: btttps://githin /ttp//ggttttttteeeeeeeeeee com/ com/JiiauaDDnggCIIr for the descriptions of safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Sec. 5 for the licenses of existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please check the source codes at
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "76vvIZVCBq", "reviewer": "Reviewer_Th4Q", "summary": "This paper tackles the problem of continually adapting text-to-image diffusion customization models.  The proposed method employs a novel concept consolidation loss, elastic weight aggregation module, and context-controllable synthesis strategy. Extensive experiments demonstrate that the proposed method performs strongly compared to SOTA baseline methods.", "questions": "Please see my weaknesses section. I think 1 and 2 are crucial for acceptance (would raise my score to WA if done thoroughly). Part 3 is important, too, but more difficult to address in a short period of time.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach is innovative and presented in a principled manner.\n\n2. The problem setting is a strong area of impact for research with real-world applications, which is important for continual learning.\n\n3. The experiments and analysis are very extensive, and the paper presentation is of high quality.", "weaknesses": "1. Please include a table that compares the training, parameter, and memory costs of your method against other methods. This comparison is crucial for a comprehensive evaluation of your method.\n\n2. Although the results are clear and detailed, they lack confidence intervals or similar statistical measures in the quantitative metrics. Running the experiments with multiple random seeds would ensure robustness and provide a more reliable assessment.\n\n3. The dataset used is quite limited. To fully justify your setting, you should explore more diverse datasets and longer task sequences. For instance, with only 10 simple concepts, it would be feasible to store a LoRA version of Custom Diffusion or even the Custom Diffusion deltas in memory and merge them on the fly for the target task, achieving zero forgetting in a practical manner. To truly demonstrate the impact of your method and setting, beyond what can be trivially solved by adapters or state-of-the-art LoRA-based merging methods, you should consider long task sequences or challenging benchmarks such as celebrity faces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Please include a table that compares the training, parameter, and memory costs of your method against other methods. This comparison is crucial for a comprehensive evaluation of your method. \n\nA1: Thanks for your insightful comment. As shown in the following results, we use Stable Diffusion (SD-1.5) [38] to conduct comprehensive evaluations in terms of training time, parameters, and memory costs of each task, when the number of continuous concept customization tasks is ten. All comparisons of training time are conducted under two NVIDIA RTX A6000 GPUs. Our model has comparable parameters, memory costs, and training time to LoRA-M [64], LoRA-C [64] and CLoRA [43]. Additionally, the training parameters and memory requirements of our model are substantially lower than those of other SOTA methods. More importantly, our model significantly outperforms existing SOTA comparison models for both the qualitative and quantitative evaluations on versatile generation tasks (see Tabs. 1-2 and Figs. 2-5 in the submitted manuscript). We will consider introducing these comparisons about parameters and memory costs in the final revision. \n\n| Methods               | | Training Time (h) | | Parameters (M) | | Memeory Costs (M) | IA (SD-1.5) |     \n|-                      |-|:-:                |-|:-:             |-|:-:                |:-:          |\n| Finetuning            | |         0.65      | |         0.80   | |     3.43          | 73.5$\\pm$0.15  |  \n| EWC [19]              | |          0.83     | |         0.80   | |     6.87          | 75.8$\\pm$0.22  |\n| LWF [25]              | |          1.18     | |         0.80   | |     4.21          | 74.3$\\pm$0.09  |\n| LoRA-M [64]           | |          0.65     | |         0.40   | |     1.72          | 74.6$\\pm$0.23  |\n| LoRA-C [64]           | |           0.65    | |         0.40   | |     1.72          | 74.7$\\pm$0.06  |\n| CLoRA [43]            | |          0.90     | |         0.40   | |     1.72           | 76.8$\\pm$0.08  |\n| L2DM [46]             | |           1.35    | |         0.80   | |     3.43          | 76.3$\\pm$0.14  |\n| **CIDM (Ours)**       | |           0.72    | |         0.41   | |     1.74          | **77.9$\\pm$0.08** |\n---\n\nQ2: Although the results are clear and detailed, they lack confidence intervals or similar statistical measures in the quantitative metrics. Running the experiments with multiple random seeds would ensure robustness and provide a more reliable assessment.\n\nA2: Thanks for your constructive comment. We use five random seeds (0, 2021, 2022, 2023, 2024) to conduct five random experiments in terms of the image alignment (IA) metric and report their average results over five random seeds for evaluation. As shown in the following results, our model achieves better performance than other SOTA methods, which verifies the robustness and reliable effectiveness of our model to tackle the CIFC problem. Moreover, we will consider presenting averaged comparison results over multiple random seeds in the finial revision. \n\n| Methods               | |   SD-1.5 [38]  | |   SDXL [33]   |\n|-                      |-|:-:             |-|:-:            |\n| Finetuning            | | 73.5$\\pm$0.15  | | 71.4$\\pm$0.13 |          \n| EWC [19]              | | 75.8$\\pm$0.22  | | 77.5$\\pm$0.07 |        \n| LWF [25]              | | 74.3$\\pm$0.09  | | 76.6$\\pm$0.10 |        \n| LoRA-M [64]           | | 74.6$\\pm$0.23  | | 74.2$\\pm$0.09 |        \n| LoRA-C [64]           | | 74.7$\\pm$0.06  | | 74.6$\\pm$0.14 |         \n| CLoRA [43]            | | 76.8$\\pm$0.08  | | 77.7$\\pm$0.06 |        \n| L2DM [46]             | | 76.3$\\pm$0.14  | | 77.2$\\pm$0.11 |        \n| **CIDM (Ours)**       | | **77.9$\\pm$0.08**  | | **79.6$\\pm$0.07** |          \n---\n\nQ3: The dataset used is quite limited. To fully justify your setting, you should explore more diverse datasets and longer task sequences. To truly demonstrate the impact of your method and setting, you should consider long task sequences or challenging benchmarks such as celebrity faces.\n\nA3: Thanks for your invaluable comment. To further validate the effectiveness of our model in learning longer task sequences, we use a new challenging benchmark dataset (celebrity faces) to construct thirty continuous text-guided concept customization tasks. Compared to existing SOTA models such as LoRA-M [64], LoRA-C [64], CLoRA [43], and L2DM [46], this new setting on celebrity faces dataset includes longer task sequences (i.e., thirty text-guided customization tasks), and encompasses different object concepts. Therefore, it is more challenging and better suited to verify the effectiveness in addressing the CIFC problem. We use Stable Diffusion (SD-1.5) [38] to evaluate parameters, memory costs and image aligment (IA) metric over five random seeds. From the following results, we observe that our model significantly outperforms other SOTA methods to learn longer task sequences. More importantly, the training parameters and memory requirements of our model are comparable to those of LoRA-M [64], LoRA-C [64] and CLoRA [43], but are substantially lower than those of other SOTA methods. It verifies the effectiveness of our model to learn longer task sequences on the new challenging dataset.\n\n| Methods               | | Parameters (M) | | Memeory Costs (M) | |   IA (%)       |  \n|-                      |-|:-:             |-|:-:                |-|:-:             |            \n| EWC [19]              | |    0.80        | |  6.87              | | 72.7$\\pm$0.14  |               \n| LoRA-M [64]           | |    0.40        | |  1.72             | | 72.0$\\pm$0.17  |     \n| LoRA-C [64]           | |    0.40        | |  1.72             | | 72.3$\\pm$0.08  |          \n| CLoRA [43]            | |    0.40        | |  1.72             | | 74.1$\\pm$0.13  |         \n| L2DM [46]             | |    0.80        | |  3.43             | |73.4$\\pm$0.11  |        \n| **CIDM (Ours)**       | |    0.41        | |  1.74             | | **75.7$\\pm$0.09**  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h5yhPr9JW2", "reviewer": "Reviewer_5EBF", "summary": "This paper introduces the Concept-Incremental text-to-image Diffusion Model (CIDM), which addresses the Concept-Incremental Flexible Customization (CIFC) problem. This approach represents one of the first explorations into learning new customization tasks incrementally, effectively navigating the dual challenges of catastrophic forgetting and concept neglect. To combat catastrophic forgetting, the paper proposes a new concept consolidation loss coupled with an elastic weight aggregation module, which together helps preserve knowledge of previously learned personalized concepts. Additionally, the development of a novel context-controllable synthesis strategy specifically addresses the challenge of concept neglect, ensuring that new concepts are integrated without overshadowing existing knowledge. The effectiveness of the CIDM is validated through comprehensive experiments across a range of text-to-image generation tasks, demonstrating its capability to learn new customization tasks consecutively with notable efficacy.", "questions": "1. I recommend the authors include details on memory consumption or the number of stored parameters after each task during the rebuttal process.\n\n2. The authors should incorporate a comprehensive description of custom image editing and style transfer processes, including specific parameters and settings used, to allow for a better understanding and evaluation of the proposed methods' effectiveness in practical applications.", "limitations": "There are no potential negative societal impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper commendably defines the Concept-Incremental Flexible Customization problem, setting a precedent for addressing this emerging challenge within the field. CIFC is a practical issue crucial for continuously synthesizing personalized concepts based on user preferences in real-world applications.\n\n2. The novel model introduced by the authors incorporates a range of innovative functions and algorithms, which are thoroughly explained and convincingly motivated. The concept consolidation loss and the elastic weight aggregation module are noteworthy among these, paired with a context-controllable synthesis strategy. These elements collectively aim to mitigate issues of catastrophic forgetting and concept neglect effectively.\n\n3. The experimental section of the paper is detailed, providing a broad comparison of the model’s capabilities across tasks such as multi-concept generation, style transfer, and image editing.", "weaknesses": "1. A critical concern arises regarding the scalability of the model, specifically the accumulated memory load after learning each task. This is particularly pertinent in continual learning scenarios where efficiency is crucial. \n\n2. The paper lacks detailed implementation information on critical tasks such as custom image editing and style transfer. Providing these details would significantly enhance the clarity and replicability of the research.\n\n3. The concept of balancing task-specific and task-shared knowledge is intriguing and seems to be central to effectively addressing the CIFC problem. To substantiate the claims of its effectiveness, the authors should include ablation studies that isolate and quantify the impact of these components. Additionally, visualizations comparing the results with and without these strategies would provide a clearer, more direct illustration of their value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: I recommend the authors include details on memory consumption or the number of stored parameters after each task during the rebuttal process.\n\nA1: Thanks for your constructive suggestion. We conduct comparison experiments to evaluate memory consumption and training parameters of each task, when utilizing Stable Diffusion (SD-1.5) [38] as the pretrained model to learn ten continuous text-guided concept customization tasks. As shown in the following results, the training parameters and memory requirements of our model are comparable to those of LoRA-M [64], LoRA-C [64] and CLoRA [43], but are substantially lower than those of other SOTA methods. Moreover, our model achieves better performance than  existing SOTA diffusion models (see Tabs. 1-2 and Figs. 2-5). It illustrates that our model is efficient and effective to tackle the practical Concept-Incremental Flexible Customization (CIFC) problem. \n\n| Methods               | | Parameters (M) | | Memeory Costs (M) |     \n|-                      |-|:-:             |-|:-:                |\n| Finetuning            | |         0.80   | |     3.43          |   \n| EWC [19]              | |         0.80   | |     6.87          |\n| LWF [25]              | |         0.80   | |     4.21          |\n| LoRA-M [64]           | |         0.40   | |     1.72          | \n| LoRA-C [64]           | |         0.40   | |     1.72          |\n| CLoRA [43]            | |         0.40   | |     1.72          | \n| L2DM [46]             | |         0.80   | |     3.43          |\n| **CIDM (Ours)**       | |         0.41   | |     1.74          |\n---\n\nQ2: The paper lacks detailed implementation information on critical tasks such as custom image editing and style transfer. Providing these details would significantly enhance the clarity and replicability of the research.\n\nA2: Thanks for your insightful comments. \n\n1) As for custom image editing, we introduce Anydoor [3] as a plug-in for all comparison methods. Specifically, given a text prompt that contains the user's personalized concept, an initial image, and an editable bounding box, we first input the given text prompt into the latent diffusion model to perform custom synthesis. Then we embed the generated image into the editable bounding box of the initial image using Anydoor [3].\n\n2) We introduce T2I-adapter [30] to achieve custom style transfer. Specifically, given an initial image, we utilize BLIP [23] to extract the corresponding caption (e.g., \"photo of a city skyline\"). Then, we add the custom concept (e.g., [V6] style) to the caption to obtain the style transfer prompt (e.g., \"photo of a city skyline in [V6] style\"). Finally, we perform Canny edge detection on the initial image to extract a Canny image, and incorporate it with the style transfer prompt to achieve custom synthesis via the T2I-adapter [30].\n\nRefs:\n\n[3] AnyDoor: Zero-shot Object-level Image Customization\n\n[23] BLIP:Bootstrapping LanguageImage Pre-training\n\n[30] T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models \n\n\nQ3: The concept of balancing task-specific and task-shared knowledge is intriguing and seems to be central to effectively addressing the CIFC problem. To substantiate the claims of its effectiveness, the authors should include ablation studies that isolate and quantify the impact of these components. Additionally, visualizations comparing the results with and without these strategies would provide a clearer, more direct illustration of their value.\n\nA3: Many thanks for analyzing our model. \n\n1) Tab. 3 in the submitted manuscript shows ablation studies of single-concept customization to analyze the effectiveness of each module in our model, where TSP and TSH denote task-specific knowledge and task-shared knowledge in the concept consolidation loss, respectively. When we remove the TSP or TSH module, our model decreases 0.5% and 0.6% in terms of image alignment (IA) metric. It verifes the effectiveness of our model to resolve the CIFC problem by exploring both task-specific and task-shared knowledge.\n\n2) Fig. 6 in the submitted manuscript visualizes the ablation analysis of task-specific knowledge (TSP) and task-shared knowledge (TSH) from the perspective of image generation quality. These visualization results illustrates that our model can capture task-specifc information within each customization task and explore task-shared knowledge across different tasks to tackle the CIFC problem via optimizing Eq. (2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mhJQbVuda8", "reviewer": "Reviewer_YpEo", "summary": "This paper explores a novel and practically significant problem, namely how custom diffusion models can continuously learn new personalized concepts while avoiding catastrophic forgetting and concept neglect. The authors developed a  concept consolidation loss and an elastic weight aggregation module to explore task-specific knowledge and task-shared knowledge, and aggregated all low-rank weights of old concepts based on the contributions of old concepts. To address the problem of concept neglect, the authors propose a context-controllable synthesis strategy that can utilize expressive regional features and noise estimation to control the generated contexts. Experimental results confirm the effectiveness of the proposed method.", "questions": "Please check the above concerns in the weakness points. And If my concerns are resolved, I will consider increasing my score.", "limitations": "See the weaknesses above.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The continual learning of generative models is an interesting and meaningful task. How to mitigate catastrophic forgetting and concept neglect in concept-incremental flexible customization scenario remains insufficiently explored.\n2. The method proposed by the authors effectively mitigates the challenges of catastrophic forgetting and concept neglect, with detailed descriptions and comparisons in both methodology and analysis.\n3. The writing is good and easy to read.", "weaknesses": "1. It would be better to mention the comparison methods of experiments in the related work，such as LoRA-M, LoRA-C, CLoRA and L2DM, and emphasize the differences between them and the proposed method.\n2. This paper attempts to address two challenges in the Concept-Incremental text-to-image Diffusion Model: catastrophic forgetting and concept neglect. However, from the text, I did not understand the relationship between continual learning and concept neglect. I would like to know if concept neglect is a challenge introduced by continual learning, and what impact continual learning has on original concept neglect?\n3. The baseline choices for continual learning such as EWC and LWF are too outdated. Why not choose new and more effective continual learning algorithms? Intuitively, more advanced algorithms should be able to better mitigate catastrophic forgetting.\n4. Due to the past focus of continual learning mainly on classification tasks, I hope to see more discussion on the background of continual generation models. For example, in this setting, how severe are catastrophic forgetting and concept neglect specifically, and whether some more objective evaluation criteria can be introduced.\n5. How much do different weighting schemes between LoRAs affect the quality of generation, compared to some intuitive weighting baselines?\n6. Does the value of the rank in LoRA and its placement in the model (e.g., which layer) have an impact on the results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: It would be better to mention the comparison methods of experiments in the related work, such as LoRA-M, LoRA-C, CLoRA and L2DM, and emphasize the differences between them and the proposed method.\n\nA1: Thanks for your insightful comment. We will carefully polish the related work in the finial revision. The differences between our model and other methods are as follows: \n\n1) LoRA-M [64] equally amalgamates all LoRA layers to retrain diffusion model, and LoRA-C [64] explores contributions of different LoRA layers for multi-concept composition. When learning new concepts continually, they may experience signifcant loss of attributes on old concepts (i.e., catastrophic forgetting) for versatile customization. Different from them, we devise a concept consolidation loss to mitigate forgetting of old concepts by exploring task-specifc/task-shared knowledge. \n\n2) CLoRA [43] proposes a self-regularized low-rank adaption to continually learn new concepts, while L2DM [46] builds a long-term memory bank to reconstruct old concepts. They cannot control synthesized contexts according to user conditions and suffer from concept neglect in multi-concept composition. However, our model proposes a context-controllable synthesis to tackle concept neglect, and effectively learns new concepts continuously for versatile customization.\n\nQ2: I did not understand the relationship between continual learning and concept neglect. I would like to know if concept neglect is a challenge introduced by continual learning, and what impact continual learning has on original concept neglect?\n\nA2: As introduced in [46][64], concept neglect is a common challenge when performing multi-concept composition according to user-provided conditions, rather than being directly introduced by continual learning. However, if latent diffusion models aim to consecutively synthesize a sequence of new concepts under the CIFC setting, the catastrophic forgetting of old concepts can significantly exacerbate the degree of concept neglect when composing multiple personalized concepts. \n\nQ3: The baseline choices for continual learning such as EWC and LWF are too outdated. Why not choose new and more effective continual learning algorithms? Intuitively, more advanced algorithms should be able to better mitigate catastrophic forgetting. \n\nA3: To further verify the effectiveness of our model, we use a more advanced continual learning algorithm, InfLoRA Refs[1], published in CVPR2024, for performance comparison. As shown in the following results, our model substantially surpasses InfLoRA Refs[1] by a large margin. Note that CLoRA [43] and L2DM [46] are the two most advanced continual generation models, which are highly relevant to this paper. Particularly, our model achieves better performance than these SOTA methods. \n\n| Methods               |IA (SD-1.5) |  \n|-                      |:-:    |  \n| Finetuning            | 73.7  |        \n| EWC [19]              | 75.9  |        \n| LWF [25]              | 74.1  |                         \n| CLoRA [43]            | 76.9  |        \n| L2DM [46]             | 76.1  |     \n| InfLoRA Refs[1]       | 76.2  |\n| **CIDM (Ours)**       | **78.0** |          \n\nRefs[1]  InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning, CVPR 2024\n\nQ4:\tHow severe are catastrophic forgetting and concept neglect, and whether some more objective evaluation criteria can be introduced.\n\nA4: Following L2DM [46], we introduce a new evaluation criteria: Task Forgetting Rate of Image Alignment (TFR-IA), to evaluate the degree of catastrophic forgetting and concept neglect. For the $l$-th concept, its TFR-IA is computed by $\\frac{1}{g-1}\\sum_{i=1}^{g-1} I_{i,l} - I_{g,l}$, where $I_{i,l}$ and $I_{g,l}$ denote the image alignment (IA) of the $l$-th concept evaluated at the $i$-th and $g$-th tasks. Our model outperforms other SOTA methods in terms of the TFR-IA metric. It verifies that our model has the least forgetting and concept neglect, making it more effective for continual generation. \n\n| Methods               | TFR-IA (%) |  \n|-                      |:-:    |     \n| EWC [19]              | 2.48  |            \n| LoRA-M [64]           | 4.42  |        \n| LoRA-C [64]           | 4.30  |        \n| CLoRA [43]            | 1.56  |        \n| L2DM [46]             | 1.37  | \n| InfLoRA Refs[1]       | 1.39  |    \n| **CIDM (Ours)**       | **1.22** |          \n---\n\nQ5:\tHow much do different weighting schemes between LoRAs affect the quality of generation, compared to some intuitive weighting baselines?\n\nA5: As shown in Tab. 1 and Figs. 2-3 of the submitted manuscript, we introduce comparisons between our model and some intuitive LoRAs weighting baselines such as LoRA-M [64] and LoRA-C [64]. Specifically, LoRA-M [64] equally amalgamates all LoRA layers to retrain diffusion model, and LoRA-C [64] explores contributions of different LoRA layers for multi-concept composition. In Tab. 1 and Figs. 2-3, LoRA-M [64] and LoRA-C [64] experience signifcant loss of attributes on old concepts (i.e., catastrophic forgetting) for versatile customization. Moreover, our model significantly outperforms them in effectively resolving catastrophic forgetting.\n\nQ6:\tDoes the value of the rank in LoRA and its placement in the model (e.g., which layer) have an impact on the results?\n\nA6: Thanks for your comment. \n\n1) Following [10], we set the value of rank to 4 in this paper, and use Stable Diffusion (SD-1.5) [38] to analyze its effect on performance (IA) by setting r={2, 4, 6, 8}. As shown in the following results, the value of rank ($r\\leq 8$) has negligible impacts on the performance. Note that larger rank can increase parameters and memory costs, which violates practicality of the CIFC setting.\n\n2) For LoRA placement, each cross-attention layer in the diffusion models has its own LoRA layer. \n\n| rank     | 2   | 4   | 6   | 8   |\n|:-:       |:-:  |:-:  |:-:  |:-:  |\n| IA (%)   | 76.2|78.0 |77.8 | 77.9|                     \n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pdJr4L0uFU", "reviewer": "Reviewer_SpKd", "summary": "This paper introduces a novel approach called CIDM to address the challenge of Concept-Incremental Flexible Customization in text-to-image generation. The authors identify two key issues in CIFC: catastrophic forgetting of previously learned concepts and concept neglect during multi-concept composition. The proposed CIDM tackles these problems through several innovative components, including a concept consolidation loss, elastic weight aggregation, and a context-controllable synthesis strategy. The paper provides experiments on various customization tasks to demonstrate the effectiveness of their approach compared to existing methods.", "questions": "See weakness.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper addresses a practical problem in custom text-to-image generation, namely the ability to continually learn new concepts without forgetting old ones.\n2. The ablation studies provide insights into the contribution of each proposed component, strengthening the paper's technical depth.\n3. The approach is model-agnostic and has been tested on different backbone architectures (SD-1.5 and SDXL), showing its broad applicability.", "weaknesses": "1.  The symbol definitions are too complex which are hard to follow.\n2. The paper introduces several new hyperparameters. How sensitive is the model to these?\n3. How well does the model perform across different application domains and with varied types of personalized concepts?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The symbol definitions are too complex which are hard to follow.\n\nA1: Thanks for your valuable comment. We will carefully polish the symbol definitions and introduce a table about notation definitions to better understand this paper in the final revision process.  \n\nQ2: This paper introduces several new hyperparameters. How sensitive is the model to these?\n\nA2: Thanks for your insigntful comment. We employ Stable Diffusion (SD-1.5) [38] as the pretrained model and use image alignment (IA) as metric to investigate the effect of the hyperparameters {$\\gamma_1, \\gamma_2, r$} on the performance of our model. When setting the rank $r=4$ of LoRAs, we analyze the effect of the hyperparameters {$\\gamma_1, \\gamma_2$} over a range of {0.01, 0.1, 1, 10}. Moreover, we introduce the hyperparameter experiments for $r$ within the range {2, 4, 6, 8}, when setting the balancing weights $\\gamma_1=0.1, \\gamma_2=1.0$. From the following results, we observe that our model has stable performance over a wide range of hyperparameters {$\\gamma_1, \\gamma_2, r$}. Furthermore, our model is not sensitive to different hyperparameter selections, making it effective and robust for learning continuous text-guided concept customization tasks.\n\n\n| $\\gamma_1$ \\ $\\gamma_2$ | 0.01| 0.1 | 1.0 | 10  |  \n|:-:                      |:-:  |:-:  |:-:  |:-:  |  \n| **0.01**                   | 77.3|77.7 |77.9 |77.5 |         \n| **0.1**                     | 77.7|77.9 | 78.0|77.6 | \n| **1.0**                     | 77.5|77.8 | 77.9|77.9 | \n| **10**                      | 77.8|77.6 | 77.4| 76.6| \n---\n\n| rank     | 2   | 4   | 6   | 8   |\n|:-:       |:-:  |:-:  |:-:  |:-:  |\n| **IA (%)**   | 76.2|78.0 |77.8 | 77.9|                      \n---\n\nQ3: How well does the model perform across different application domains and with varied types of personalized concepts?\n\nA3: Thanks for your constructive comments. Inspired by [10][46], we construct a new challenging concept-incremental learning (CIL) dataset including ten continuous text-guided concept customization tasks to verify the effectiveness of our model under the CIFC setting. This dataset has varied types of personalized concepts, which are collected from different application domains. Moreover, different concepts have significant differences. Specifically, as shown in Fig. 8, seven tasks have different object concepts (i.e., V2 duck toy, V4 backpack, V5 teddy bear) and pet concepts (i.e., V1 dog, V3 cat, V7 dog and V9 cat) from [40, 22]. Besides, the remaining three tasks have different style concepts (i.e., V6, V8 and V10 styles) collected from website. Considering the practicality of the CIFC setting, we introduce some semantically similar concepts (e.g., V1 and V7 dogs, V3 and V9 cats), making the CIL dataset more challenging under the CIFC setting. Particularly, all comparison experiments of this paper are conducted on this challenging dataset. As shown in Tabs. 1-2 and Figs. 2-5 of the submitted manuscript, our model achieves better performance than other state-of-the-art (SOTA) comparison methods in resolving the practical concept-incremental flexible customization (CIFC) problem, under both qualitative and quantitative evaluations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and area chairs:

We extend our gratitude to all the reviewers and area chairs for dedicating their time and effort to evaluating our paper. We also thank the reviewers for their positive and insightful comments, which can help us improve our work.

We are encouraged that:

$\bullet$ Reviewer Th4Q and Reviewer 5EBF agree that our work is **novel** in resolving **a practical problem** named concept-incremental flexible customization (CIFC) and includes **comprehensive evaluation experiments** to validate the effectiveness of the proposed model.

$\bullet$ Reviewer YpEo thinks that our paper focuses on tackling **an interesting and meaningful task** with **extensive comparisons** in both methodology and analysis. 

$\bullet$ Reviewer SpKd believes that our model addresses **a practical problem**, has **in-depth ablation studies**, and demonstrates **broad applicability** across different backbone architectures.

All reviewers recognize that our model achieves **state-of-the-art performance under comprehensive experiments**. 

We have responded to each reviewer individually to address any comments. We would like to give a brief summary: 

**To Reviewer Th4Q:** 

1) We introduce comparison experiments on training times, parameters, and memory costs between our model and other models.

2) We conduct five random evaluation experiments and report their average results over five random seeds (0, 2021, 2022, 2023, 2024) to compare the performances.

3) We introduce longer task sequences on a new challenging dataset (celebrity faces) to further validate the effectiveness of our model. 

**To Reviewer 5EBF:**    

1) We provide comparisons on memory consumption and training parameters.

2) We introduce implementation details of custom image editing and style transfer. 

**To Reviewer YpEo:**   

1)  We clarify the significant differences between our model and other methods such as LoRA-M, LoRA-C, CLoRA, and L2DM, and explain the relationship between continual learning and concept neglect.

2) We use a more advanced continual learning algorithm, InfLoRA [1], published in CVPR2024, for performance comparison.

3) We introduce a new evaluation criteria: Task Forgetting Rate of Image Alignment (TFR-IA), to assess the degree of catastrophic forgetting and concept neglect. 

4) We compare our model with several intuitive LoRA weighting baselines, such as LoRA-M [64] and LoRA-C [64], and discuss the impact of low rank in LoRAs.

Refs[1] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning, CVPR 2024


**To Reviewer SpKd:**  

1) We will carefully polish the symbol definitions and introduce a table about notation definitions in the appendix.

2) We investigate the effect of the hyperparameters {$\gamma_1, \gamma_2, r$} on the performance of our model.

3) We clarify that our model can perform well on the challenging dataset used in this paper, which contains various types of personalized concepts collected from different application domains.

We thank all reviewers and area chairs again!

Best,

Authors of Paper #1919

Meta-review: This paper makes a significant contribution to the field of continual learning in generative models, with a particular focus on personalized text-to-image generation. The proposed CIDM method introduces innovative solutions to the challenges of catastrophic forgetting and concept neglect, addressing a problem of high relevance and complexity. The paper demonstrates a solid level of novelty and technical rigor, supported by a well-executed experimental evaluation. Based on the strength of the contributions, the clarity of presentation, and the comprehensive coverage of experiments, the review committee has reached a consensus to recommend this paper for acceptance.
Decision: Accept (poster)